Project Configuration

Every Test Run is described by a SoTest Project Configuration file. The basic idea of a project configuration is to provide an abstract list of tests along with explanations on how to execute them on a given hardware platform without being hardcoded to specific hardware setups.

A project configuration can be committed along with product code and tests. This way it is possible to check out a project on random commit states and test it.

In order to create a test run, all needed binaries are bundled in a ZIP archive and uploaded to a URL that is accessible to the SoTest Controllers. Then this URL together with a project config that references the files in the archive is used to create a SoTest test run via the REST API.

Project configurations can be represented as JSON or YAML.

Boot Items

One boot item describes one test. The key boot_items of a project configuration contains a list of boot items.

Every boot item has its own name which is later also displayed in the SoTest web UI. The rest of the boot item explains how it is booted.

The description on how to boot a test is described in terms of the Multiboot Standard. This standard is already familiar from the GRUB Bootloader.

A boot item specifies an execution item and optionally a list of load items. Each of these items can be regarded as a 2-tuple that contains the binary path and command line parameters that are intended to be consumed by the running binary.

The following keys map to this description:

  • name: The name of the boot item
  • exec: Path to a bootable binary, followed by boot parameters (separated by a space).
  • load: List of additional binaries to load, each followed by additional parameters
  • iscsi: Optional field to specify iSCSI disks that should be visible to the booted system.
  • boot_item_timeout: For boot items that need some more time until they emit their SOTEST BEGIN message, the timeout can optionally be increased. The default timeout is 120 seconds.
  • extra_files: Optional list of files that should be made available to the test via HTTP.

Extra files

Files listed in the extra_files key of a boot item will be made available via @{extra_files_http_url}/filename, where @{extra_files_http_url} is a machine variable that can be used in the exec and load sections to provide the base URL to the respective test program(s). This variable expands to the HTTP base URL of the web server where the files are made available for this test.

Extra Dependencies

While the binary archive that is provided together with the project configuration contains binaries that are under development (product binary and tests), boot items may reference certain binaries that remain unchanged over test runs. For example, a Linux kernel under development might be started with a larger initial ramdisk file which is relatively large and never changing.

The extra_dependencies field of the project configuration optionally contains a list of binaries that should be available for consumption by boot items in addition to the project binary archive.

Each item consists of the following fields:

  • name: The name under which it will be referenced by boot items
  • url: A URL that points to the location from where this binary can be obtained by a SoTest controller.
  • sha256: An optional hash that can be used to make sure that there are no unintended changes that might jeopardize test stability.

Panic Patterns

The boot_panic_patterns field in the project configuration contains an optional list of panic patterns. Every panic pattern is a case-insensitive regular expression. The test output parser tests if any of these regular expressions can be found in any output line, and in case one of the patterns matches, the whole test run is aborted.

Adequate usage of panic patterns improves test execution, because critically failing tests are detected immediately, as opposed to tests that are detected as fails by running into timeouts.

Linux kernel panic messages or application-specific assertion errors are example use cases.

Example Project Configuration

This configuration file originates from a real-life use case and demonstrates different kinds of boot items:

extra_dependencies:
  - name: bender
    url: http://my.artifacts.server/bender
    sha256: e3381a7de1d208784a4c4af903cd58879eaefc257bfd636d0a68b445d9e3b976
  - name: bzImage4.3
    url: http://my.artifacts.server/bzImage4.3
    sha256: ad66834890cfc0df695c1c7827b0a28b0d9a82a215823c1d6623a197b9fbfa2a
  - name: initramfs.cpio.xz
    url: http://my.artifacts.server/initramfs.cpio.xz
    sha256: 3cb757078989bb72a6674dce8c17d1ecbd2166a4e311bfe2afc05ca8c24da193

boot_items:
  - name: Single Multiboot Binary
    exec: build/test/guest/simple/test_simple

  - name: Simple Linux boot
    exec: bzImage4.3 console=ttyS0,115200
    load:
    - initramfs.cpio.xz

  - name: Linux as VM on Supernova
    exec: build/contrib/NOVA/src/hypervisor-x86_64 serial novga iommu
    load:
    - build/src/supernova linux
    - bzImage4.3 console=ttyS0,115200 nr_cpus=1 hpet=disable
    - initramfs.cpio.xz

  - name: Linux as VM on Supernova with Bender
    exec: bender
    load:
    - build/contrib/NOVA/src/hypervisor-x86_64 serial novga iommu
    - build/src/supernova linux
    - bzImage4.3 console=ttyS0,115200 nr_cpus=1 hpet=disable
    - initramfs.cpio.xz

  - name: Single Multiboot Binary as VM on Supernova
    exec: hypervisor-x86_64 serial novga iommu
    load:
    - build/src/supernova
    - build/test/guest/simple/test_simple fooparams

  - name: Windows 7 Hello Test from network iSCSI Drive
    exec: bender
    load:
    - build/contrib/NOVA/src/hypervisor-x86_64
    - build/src/supernova --boot=disk
    iscsi:
      - 0x80 = win7-64
      - 0x81 = hello
    boot_item_timeout: 300

boot_panic_patterns:
  - kernel panic
  - assertion failed

Paths

Files in the exec, load and extra_files items of a boot item must be referenced in one of these 2 ways:

  • As a relative path within the binary archive that was provided at test run creation time along with the project configuration.
  • As the name of an external dependency.

Example: If the content of the binary archive looks like this:

unzip -l binaries.zip
Archive:  binaries.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 12202050  12-04-2019 15:56   initrds/dmidecode.initrd
204350231  12-04-2019 15:56   initrds/kernel-compile.initrd
  4507296  12-04-2019 15:56   kernels/linux-4.19.81.bzImage
  4761984  12-04-2019 15:56   kernels/linux-5.3.8.bzImage
---------                     -------
225821561                     4 files

Then a boot item could look like this:

  - name: Linux kernel compile test
    exec: kernels/linux-5.3.8.bzImage console=ttyS0,115200
    load:
    - initrds/kernel-compile.initrd

iSCSI Disks

It is possible to attach one or more iSCSI disks to the system at boot time. The optional iscsi field in a boot item accepts drive parameters in the following format:

iscsi:
  - <BIOS drive ID> = <image name>
  - ...

The BIOS drive ID specifies the drive ID slot that is locally used to attach the drive to the system.

The image name is concatenated with the MAC address of the hardware platform instance. That means, if the MAC address of the machine is 01:23:45:67:89:ab and the image name is foobar then the resulting target name is 01-23-45-67-89-ab-foobar.

In order for such an image name to exist, the administrator of the iSCSI service must create one image per hardware platform instance. This requirement is to make it impossible that multiple hardware platform instances manipulate the same image concurrently.

For more information on the underlying mechanism see the documentation for attaching SAN devices of the iPXE project.

In order to debug iSCSI errors, have a look at the Boot Config section of the boot item in the web UI that is failing to attach an iSCSI target, and inspect the resulting iSCSI load line.

Hardware Platform Specific Variables

It is possible to parametrize exec and load lines with hardware platform specific variables.

The administrator of a SoTest cluster can configure such variables per hardware platform instance. Configured variables can be accessed using the following syntax: @{variable_name} if the variable in the hardware platform configuration is variable_name.

Example: In order to boot the same Linux kernel on multiple hardware platforms that each have different serial ports and baudrates, the following configuration would parametrize the boot item accordingly:

  - name: Linux kernel compile test
    exec: kernels/linux-5.3.8.bzImage console=@{serialport},@{serialbaudrate}

If a variable is not defined for the executing hardware platform, the boot item fails.