SoTest User Documentation

The latest version of this documentation is hosted on docs.sotest.io.

The purpose of these documents is to describe to users how to interact with SoTest in software development project teams.

Overview

Testing a software product on all supported hardware platforms is typically not done during development. Instead, QA teams inspect the product for possible regression before every release. This mode of operation leads to late discovery of regressions. Development teams then need to correct these regressions under time pressure during the release phase. Bugs that are discovered late are usually more costly to fix as the process to correction at this point involves multiple departments.

SoTest is an infrastructure product developed by Cyberus Technology GmbH that automates software tests on baremetal hardware. It aims to make testing on hardware as simple as running unit tests. Software product development teams can use it to automate as many hardware tests as possible in order to integrate them into a fast synchronous end-to-end testing development workflow.

Workflow

The following diagram describes the typical workflow in projects where a SoTest deployment performs product testing on hardware:

Typical Developer Workflow with SoTest

This diagram also points out the 2 typical ways to interact with SoTest. Either, users can

  • manually schedule test runs or
  • let their CI schedule test runs automatically.

Architecture

In order to provide hardware testing service for multiple hardware platforms, a deployed SoTest system is structured roughly like this:

SoTest Architecture Overview

SoTest Database

The SoTest Database stores all test runs along with their results and makes them accessible via a web UI that can be browsed by humans, but also via a REST API.

Test runs are created against the SoTest database’s REST API. The test workloads are then scheduled to run on the selected hardware platforms.

SoTest Cluster

Hardware platforms are managed in clusters. Each cluster consists of one SoTest Controller which maintains its own private network in which it controls many individual hardware platforms. Projects can have multiple SoTest clusters distributed on different sites that all report to the same SoTest database.

The controller performs remote power cycling on a hardware platform, then provides it with all needed binaries at boot time, and finally records the relevant test output. After interpreting the test output, it posts back the test results to the SoTest database.

Concepts

In order to organize hardware tests in a useful abstract notion, SoTest introduces a few simple concepts. These become the typical vocabulary that developers use when referring to things around automated hardware testing.

Hardware Platform

A hardware platform is a specific configuration of hardware that is made available for testing by a SoTest deployment.

Hardware platforms can be different models of laptops, servers, desktop computers with specific hardware installed, etc.

Multiple instances per hardware platform may exist under the same hardware platform name. SoTest treats these as identical machines to speed up test execution.

Boot Item

A boot item describes how to boot one test in a hardware-platform agnostic way.

A boot item that has been executed on a specific hardware platform is saved in the database as a successful or failed test. The web UI visualizes test results.

Test Run

A Test Run is a list of boot items that are each to be executed on one or more hardware platforms.

Example Test Run Index

The SoTest web UI index page typically lists all existing test runs in a pageable overview that can also be searched. Test runs can also have different priorities.

Example Test Run with Results

A test run is represented as a matrix of boot items in the SoTest web UI: Every row represents a boot item, while the column shows for which hardware platforms it is configured to execute.

(The small (s) appendix to some of the boot items in the screenshot indicate that some test cases have been skipped on these hardware platforms.)