Test Output Protocol Specification

Test output needs to comply to a specific protocol to allow for automatic result interpretation. This document specifies and documents the SoTest test output protocol.

While SoTest speaks different test procotols (GoogleTest, TAP), the SoTest protocol is specialized for tests that run on bare metal hardware and communicate via potentially slow and brittle connections. In such environments it is crucial to not only let the test communicate its results but also assumptions on the test infrastructure. This way the test infrastructure can improve and speed up test execution.

SoTest Protocol

The SoTest test output protocol symbols each span over one line of text (using the \n line terminator symbol).

SoTest Protocol Version 1 Terminal Symbol Index

Terminal Parsed Meaning
SOTEST VERSION <V> BEGIN <N> Beginning of SoTest protocol output:
N test cases have to follow.
Sotest protocol version is V.
SOTEST SUCCESS One test case succeeded.
SOTEST FAIL One test case failed.
SOTEST SKIP One test case was skipped (e.g. not supported on/by current HW).
SOTEST BENCHMARK <N> "<UNIT>" "<name>" One benchmark succeeded along with a benchmark data point.
SOTEST TIMEOUT <N> Reset the maximum time between two output lines to N seconds.
SOTEST END End of the whole test is reached.
SOTEST PANIC / custom panic message Default panic message or user configurable panic message for premature test abort.

The parser tolerates suffixes of lines that carry SoTest protocol meaning.

Unidentified lines are simply ignored by the test result interpreter.

Test State

Part of the protocol is the process of consuming test protocol terminal symbols and modifying a test state.

The test state contains 4 important counters: cases, successes, fails, skips. After test completion, the following must apply:

  • cases == successes + skips
  • fails == 0

For every benchmark, a SOTEST BENCHMARK line is counted as success. A benchmark therefore needs to be included in the SOTEST BEGIN test count announcement, and in case it fails, a SOTEST FAIL message must be emitted.

Timings

The SoTest test output interpreter saves the timestamps of certain SoTest protocol symbols, as well as of the last output line to detect crashes in case a machine goes silent during a test. This allows for analysing test efficiency and enable for optimization potential.

Test Duration

The duration of a test is determined by recording the timestamps of the SOTEST BEGIN and SOTEST END messages and taking the difference.

General Timeout Behavior

The SoTest output parser waits a preconfigured amount of time between consecutive output lines in order to detect if a test crashes/hangs. SOTEST TIMEOUT lines can be used to control timeout behaviour at any point in time, regardless of what other SOTEST protocol output has been emitted already.

This is useful for tests that take very long in order to avoid false positive crash timeouts. For very quick tests it can improve the test throughput as it allows for setting the timeout to more aggressive values and hence detect a crash earlier.

The timeout is set to 3 seconds after SOTEST BEGIN, SOTEST END, SOTEST PANIC messages, as well as after matching a panic pattern.

Timeouts Before/After SOTEST BEGIN

It makes sense to emit the SOTEST BEGIN line as early as possible in the test.

If a timeout is reached before the SOTEST BEGIN message, it is possible that this was due to an infrastructure failure, hence the test is retried multiple times. If a timeout is reached after the SOTEST BEGIN message, it is most likely that the test itself failed. In this case it does not need to be retried, which saves resources and increases the throughput of SoTest for all users.