Test Output Protocol Specification¶
Test output needs to comply to a specific protocol to allow for automatic result interpretation. This document specifies and documents the SoTest test output protocol.
While SoTest speaks different test procotols (GoogleTest, TAP), the SoTest protocol is specialized for tests that run on bare metal hardware and communicate via potentially slow and brittle connections. In such environments it is crucial to not only let the test communicate its results but also assumptions on the test infrastructure. This way the test infrastructure can improve and speed up test execution.
The SoTest test output protocol symbols each span over one line of text
\n line terminator symbol).
SoTest Protocol Version 1 Terminal Symbol Index¶
||Beginning of SoTest protocol output:
Sotest protocol version is
||One test case succeeded.|
||One test case failed.|
||One test case was skipped (e.g. not supported on/by current HW).|
||One benchmark succeeded along with a benchmark data point.|
||Reset the maximum time between two output lines to
||End of the whole test is reached.|
||Default panic message or user configurable panic message for premature test abort.|
The parser tolerates suffixes of lines that carry SoTest protocol meaning.
Unidentified lines are simply ignored by the test result interpreter.
Part of the protocol is the process of consuming test protocol terminal symbols and modifying a test state.
The test state contains 4 important counters:
skips. After test completion, the following must apply:
cases == successes + skips
fails == 0
For every benchmark, a
SOTEST BENCHMARK line is counted as success.
A benchmark therefore needs to be included in the
SOTEST BEGIN test count
announcement, and in case it fails, a
SOTEST FAIL message must be emitted.
The SoTest test output interpreter saves the timestamps of certain SoTest protocol symbols, as well as of the last output line to detect crashes in case a machine goes silent during a test. This allows for analysing test efficiency and enable for optimization potential.
The duration of a test is determined by recording the timestamps of the
SOTEST BEGIN and
SOTEST END messages and taking the difference.
General Timeout Behavior¶
The SoTest output parser waits a preconfigured amount of time between
consecutive output lines in order to detect if a test crashes/hangs.
SOTEST TIMEOUT lines can be used to control timeout behaviour at any point in
time, regardless of what other
SOTEST protocol output has been emitted
This is useful for tests that take very long in order to avoid false positive crash timeouts. For very quick tests it can improve the test throughput as it allows for setting the timeout to more aggressive values and hence detect a crash earlier.
The timeout is set to 3 seconds after
SOTEST PANIC messages, as well as after matching a
It makes sense to emit the
SOTEST BEGIN line as early as possible in the test.
If a timeout is reached before the
SOTEST BEGIN message, it is possible that
this was due to an infrastructure failure, hence the test is retried multiple
If a timeout is reached after the
SOTEST BEGIN message, it is most likely that
the test itself failed.
In this case it does not need to be retried, which saves resources and increases
the throughput of SoTest for all users.