Test Output Protocol Specification

Test success or failure is determined by the test output that a test emits.

The test output needs to comply to a specific protocol to allow for automatic result interpretation. This document specifies and documents the SoTest test output protocol.

While SoTest speaks different test protocols (GoogleTest, TAP), the SoTest protocol is specialized for tests that run on bare metal hardware and communicate via potentially slow and brittle connections. In such environments it is crucial to not only let the test communicate its results but also assumptions on the test infrastructure. This way the test infrastructure can improve and speed up test execution.

SoTest Protocol

The SoTest test output protocol symbols each span over one line of text (using the \n line terminator symbol).

SoTest Protocol Version 1 Terminal Symbol Index

Terminal Parsed Meaning
SOTEST VERSION <V> BEGIN <N> Beginning of SoTest protocol output:
N test cases have to follow.
Sotest protocol version is V.
SOTEST SUCCESS One test case succeeded.
SOTEST FAIL One test case failed.
SOTEST SKIP One test case was skipped (e.g. not supported on/by current HW).
SOTEST "<success>" BENCHMARK "<relation>" <N> "<UNIT>" "<name>" One benchmark along with a benchmark data point.
SOTEST TIMEOUT <N> Reset the maximum time between two output lines to N seconds.
SOTEST END End of the whole test is reached.
SOTEST PANIC / custom panic message Default panic message or user configurable panic message for premature test abort.

The parser tolerates suffixes of lines that carry SoTest protocol meaning, but no prefixes. This can become a problem when serial log intertwines with protocol messages. Therefore, it is advisable, to use more than one serial connection and separate as much serial log as possible from the protocol output.

Unidentified lines are simply ignored by the test result interpreter.

All serial lines, no matter if parsed as protocol lines or not, are written to the serial log.

Test State

Part of the protocol is the process of consuming test protocol terminal symbols and modifying a test state.

The test state contains 4 important counters: cases, successes, fails, skips. After test completion, the following must apply:

  • cases == successes + skips
  • fails == 0

Benchmark

A benchmark consists of the following parameters:

  • success: describes if the benchmark succeeded, must be “SUCCESS” or “FAIL”
  • name: benchmark name, type String
  • unit: unit of the measured benchmark value, type String
  • value: measured benchmark value, type Int
  • relation: must be “HIGHER_BETTER” or “LOWER_BETTER”

For every benchmark, each SOTEST "<success>" BENCHMARK ... line is counted as a success or failure and therefore needs to be included in the SOTEST BEGIN test count announcement. Benchmarks cannot use SKIP. Currently, only the success value is acknowledged in the test state. All other parameters are ignored and must be read from the serial log directly. This can be done with tools like Grafana.

Serial Log

A serial log contains all messages send over a single serial connection. All logs of one Test Run are displayed in the Web UI, while the part containing the protocol is highlighted.

Each serial log has pre-defined limits, that cause a Test Run to abort if reached. Currently we have the following limits:

  • Line length: 4000 chars
  • Log length: 10101 lines

These log errors are the only type of TestInfrastructureException that won’t lead to a retry of the test.

Timings

The SoTest test output interpreter saves the timestamps of certain SoTest protocol symbols, as well as of the last output line to detect crashes in case a machine goes silent during a test. This allows for analyzing test efficiency and enable for optimization potential.

Test Duration

The duration of a test is determined by recording the timestamps of the SOTEST BEGIN and SOTEST END messages and taking the difference.

General Timeout Behavior

The SoTest output parser waits a pre-configured amount of time between consecutive output lines in order to detect if a test crashes/hangs. SOTEST TIMEOUT lines can be used to control timeout behavior at any point in time, regardless of what other SOTEST protocol output has been emitted already.

This is useful for tests that take very long in order to avoid false positive crash timeouts. For very quick tests it can improve the test throughput as it allows for setting the timeout to more aggressive values and hence detect a crash earlier.

The timeout is changed automatically in the following scenarios:

  • to 5 seconds: after a SOTEST END message
  • to 3 seconds: after SOTEST PANIC, SOTEST BEGIN & SOTEST END when an error happened, as well as after matching a panic pattern

Protocol Parser

There’s a dedicated application, the sotest-protocol-parser designed to assist with validating SoTest protocol output without having to execute a test on the SoTest cluster.

The sotest-protocol-parser accepts some log as a file (--input <file>) or from the standard input (--stdin):

./sotest-protocol-parser --input protocol.txt

It will then analyze the contained protocol and indicate the protocol state in its output and in the return code. The following values exist:

  • Successful, SuccessfulWithSkips: Return code 0
  • Failed: Return Code 1\ Indicates a simple failure via SOTEST FAIL in the output
  • Incomplete: Return Code 2\ Indicates that the protocol seen so far is valid but messages are still missing for it to be complete (either further SOTEST SUCCESS messages or the SOTEST END message).
  • ProtocolError: Return Code 5\ This is returned when a user panic occurs or when the protocol output is not valid, e.g. because of an error such as multiple BEGIN msgs or END before BEGIN.

Additional optional parameters for the protocol parser can be specified at the command line:

  • --panicFile: Specify user panic parameters via a file. Each line represents one panic pattern.
  • --echo: This setting makes the protocol parser print a mangled version of the received input. The mangling escapes the SoTest protocol lines so that they are not interpreted by the protocol-parser again.
  • --verbose: Print detailed information on the final Test State after evaluating the received protocol. The format matches the Test State returned by the JSON API.

Usage of the --echo option (note that panic patterns are escaped, too):

$ echo "SOTEST VERSION 1 BEGIN 1\nSOTEST SUCCESS\nAssertion failed\nSOTEST END" \
 | sotest-protocol-parser --stdin --panicFile panicPatterns.txt --echo
S-O-T-E-S-T- -V-E-R-S-I-O-N- -1- -B-E-G-I-N- -1
S-O-T-E-S-T- -S-U-C-C-E-S-S
A-s-s-e-r-t-i-o-n- -f-a-i-l-e-d
S-O-T-E-S-T- -E-N-D
"Result: ProtocolError"

Output of the --verbose option:

$ echo "SOTEST VERSION 1 BEGIN 1\nSOTEST SUCCESS\nSOTEST END" \
 | sotest-protocol-parser --stdin --verbose
{
    "begin_line_time": "2022-07-18T13:59:39.085795884Z",
    "event_logs": [],
    "passes": 1,
    "cases": 1,
    "skips": 0,
    "abort": false,
    "boot_config": "",
    "creation_time": "2022-07-18T13:59:39.085795884Z",
    "test_logs": [],
    "mac_address": "00-00-00-00-00-00",
    "protocol": "SoTest 1",
    "error_message": [],
    "end_line_time": "2022-07-18T13:59:39.085795884Z",
    "machine_id": "",
    "post_time": null,
    "machine_name": "",
    "id": 0,
    "timeout": 5,
    "fails": 0
}
"Result: Successful"