Tutorial

The purpose of this tutorial is to give you a full walkthrough from test package preparation to test run creation until test result polling.

Test Run Preparation: Binary Packaging

Let us create a very simple test run that consists of one boot item that shall be booted on all hardware platforms of our sotest cluster.

We begin with a Linux kernel and a companion initial RAM disk that can be booted together. The RAM disk contains the dmidecode app and will print many machine details along with SoTest protocol output.

Download them in a ZIP bundle here: tutorial-binaries.zip If interested, see how you can build them yourself at the end of this page.

Kernel and RAM disk act as the build output of some example project now. They were zipped into the ZIP archive with the following command, which could have been done by ourselves or CI scripts:

$ zip tutorial-binaries.zip bzimage-5.4.8 initrd

Test Run Creation: Project Config

Now we have all files together and can prepare a description for sotest on how to boot these files as a boot item. Save the following text as project_config.yaml:

boot_items:
  - name: "Linux + initial RAM disk test"
    exec: bzimage-5.4.8 console=@{linux_terminal}
    load:
      - initrd

As described in the project configuration doc page, the project configuration consists at least of a list of boot items under the field name boot_items.

Our boot item has a name, an exec line and a load line. The exec line says that we are going to boot the bzimage-5.4.8 Linux kernel file, and supply it with the command line console=@{linux_terminal}.

The expression @{linux_terminal} is a platform specific variable . The administrator of a sotest cluster can provide all configured hardware platforms with their own definitions of key-value pairs. In this case, the variable @{linux_terminal} is substituted by ttyS0 on one machine, and ttyS4 on another machine, depending on which serial device is hardwired in the setup for test output.

The load line says that we are also loading the initial RAM disk into memory, so that the Linux kernel can discover and mount it at boot time.

Test Run Creation

Both our binary package and project config need to be sent to the SoTest server.

As SoTest does not store binaries itself but only where they are stored, we need to upload them somewhere (if not done by our CI automatically).

Now we can do a SoTest REST API call in order to create a new test run:

$ curl --netrc \
     -F 'boot_files_url=https://sotest.io/tutorial-binaries.zip' \
     -F 'url=https://docs.sotest.io/tutorial' \
     -F 'name=Tutorial Test Run' \
     -F 'config=@project_config.yaml' \
     https://opensource.sotest.io/api/create
45

We can note down the 45 as this is the ID of our test run.

Test Run Results

If the hardware platform resources are currently free, we can observe how they pick up our jobs:

Animated image of the test run progress bar

While the testing takes a few minutes (which depends on how fast the hardware boots and how long our actual test programs take), we can use the REST API to poll for the test run progress, which is useful for CIs and generally scripts:

$ curl https://opensource.sotest.io/api/query/45
{"tag":"UNFINISHED"}
$ curl https://opensource.sotest.io/api/query/45
{"tag":"UNFINISHED"}
$ curl https://opensource.sotest.io/api/query/45
{"tag":"SUCCESS"}

In the test run results page, we can select individual machines and look at their output, which looks like this:

[    3.781442] Run /init as init process
[    4.448661] tsc: Refined TSC clocksource calibration: 2100.014 MHz
[    4.454848] clocksource: tsc: mask: 0xffffffffffffffff
max_cycles: 0x1e453ddc726, max_idle_ns: 440795276965 ns
[    4.464890] clocksource: Switched to clocksource tsc
Hello! Running Linux (none) 5.4.8 #1-NixOS SMP Sat Jan 4 18:19:19 UTC 2020
x86_64 GNU/Linux

SoTest protocol output
SOTEST VERSION 1 BEGIN 1

# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.7 present.
75 structures occupying 2919 bytes.
Table at 0x000EB550.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
    Vendor: American Megatrends Inc.
    Version: 2.0a
    Release Date: 02/01/2013
    Address: 0xF0000
    Runtime Size: 64 kB
    ROM Size: 11264 kB
    Characteristics:
        PCI is supported
        BIOS is upgradeable
# lots of output omitted....

SOTEST SUCCESS
SOTEST END

The lines prefixed with SOTEST ... are the ones that the SoTest interpreter uses to distinguish successful from failing tests.

Appendix: Build linux kernel and initial RAM disk yourself

In order to provide all source code and information needed to reproduce the binaries, we provide nix expressions that can be built locally on Linux.

You will need to install nix before executing the build command, which you can download at https://nixos.org/nix/:

$ nix-build \
  https://github.com/cyberus-technology/cbspkgs-public/tarball/\
  2cc63a257870dea4143333a2fa50e06a9c04b9bb \
  -A initrds.initrds.dmidecode \
  -A sotest-kernels.linux_latest

# ... lots of output omitted
/nix/store/7b8cm04m1hadiy9qvrpp4w0iy79b59ys-initrd
/nix/store/50hkk8796wx5616dw906wsww7s89x3fp-linux-5.4.8

The files will then be located in the paths that this command emitted on stdout.