Project Configuration¶

The following page lists information on the use of a SoTest project configuration file, typically named project_config.yaml.

Every Test Run is described by a project configuration file. The basic idea of a project configuration is to provide an abstract list of tests along with explanations on how to execute them on a given hardware platform without being hardcoded to specific hardware setups.

More concretely, the project config defines a list of boot items with additional parameters. It is then used for running boot items in the local runner, or uploaded to the Web API via a POST request to create a new Test Run.

A project configuration can be stored alongside product code and tests. This way it is possible to check out a project on an arbitrary commit state and execute the tests.

In order to create a Test Run, all needed test binaries are bundled in a ZIP archive and uploaded to a URL that is accessible to the SoTest Controllers. Together with the project config that references the files in the archive, the URL is then used to create a Test Run via the REST API.

Project configurations can be represented as JSON or YAML.

A project configuration can have the following configuration items:

• boot_items
• extra_dependencies
• boot_panic_patterns
• fail_behavior
• local_tags_list

They are explained in detail in the following sections.

Boot Items¶

One boot item describes one test. The key boot_items of a project configuration contains a list of boot items that will be scheduled on the machines that are connected to the SoTest cluster.

Every boot item has its own name which is also displayed in the SoTest Web UI. The remaining parts of the boot item describe how it is booted.

The description on how to boot a test is derived from the Multiboot Standard. This standard is already familiar from the GRUB Bootloader.

A boot item specifies a boot source (boot_source), which is one of the following types: BIOS (bios), UEFI (uefi) or both (bifi). Each BIOS and UEFI boot source defines a mandatory execution command (exec) and optionally a list of load commands (load). Each command contains a binary path as well as command line parameters that are intended to be consumed by the running binary.

Currently, the boot_source key is optional, since we still support the legacy way of defining one—namely, defining execution and load commands at the top-level of the config. If used, a specific tag must be added to determine which type of boot source is configured. Either boot:bios or boot:uefi must be given. Failure to do so will lead to the boot item being handled as a BIFI one. This applies if both or none of these tags are given. The resulting boot config won’t be of much use though, since the configured BIOS and UEFI config will have the same exec and load values. Please use the new option to configure a boot source if you want to specify BIOS and UEFI configs at the same time with their own exec and load information.

Please note: using the legacy exec and load top-level keys overwrite any use of the boot_source key. Also, the legacy support is only temporary, we will announce any further changes ahead of time.

The following keys map to this description:

• name: The name of the boot item. Is displayed in the Web UI.
• boot_source: The boot source definition of the boot item. Optional for legacy reasons. One or both of the following keys must be defined if used:
  • bios: A boot source config of type BIOS.
  • uefi: A boot source config of type UEFI.
  • Both bios and uefi keys need to define the following values:
    • exec: Path to a bootable binary, followed by boot parameters (separated by a space). This binary will be executed during the test.
    • load: List of additional binaries to load into memory, each followed by additional parameters. This is a mandatory field but it may be empty.
  • Deprecated: exec and load can be top-level keys for legacy boot source definition
• iscsi: Optional field to specify iSCSI disks that should be visible to the booted system. Items are of the structure drive = os.
• extra_files: Optional list of files that should be made available to the test via @{extra_files_http_url}/filename. These can either be prerequisites or files from the Test Run zip archive. @{extra_files_http_url} is a machine variable that can be used in the exec and load sections to provide the base URL to the respective test program(s). This variable expands to the HTTP base URL of the web server where the files are made available for this test.
• boot_item_panic_patterns: Optional list of boot item specific panic patterns. More information about them can be found in the Test Run specific panic patterns section.
• boot_item_timeout: For boot items that need a long time until the test begins to execute and prints SoTest protocol output, it may be necessary to increase the default timeout. The default timeout is 120 seconds.
• boot_item_type: Must be all or any. Default is all. Specifies if the boot item should be executed on all available machines or just on one of all, no matter which one (any). Can be combined with the tags field to limit the available machines to a specific subset.
• tags: Optional list of tags to limit the boot item execution to specific machines. The names of the available machines, and which machine provides which tags, is a server side setting that can be obtained from your SoTest operator. An empty list will result in using all configured machines rather than none. Note: a machine needs to fulfill all listed tags to be ‘valid’.

Extra Dependencies [optional]¶

While the binary archive that is provided together with the project configuration contains binaries that are under development (product binary and tests), boot items may reference certain binaries that remain unchanged over Test Runs. For example, a Linux kernel under development might be started with a large but static initial ramdisk file.

The extra_dependencies field of the project configuration optionally contains a list of binaries that should be available for consumption by boot items in addition to the project binary archive.

Each item consists of the following fields:

• name: The name under which it will be referenced by boot items
• url: A URL that points to the location from where this binary can be obtained by a SoTest controller.
• sha256: An optional hash that can be used to make sure that there are no unintended changes that might jeopardize test stability.

Panic Patterns [optional]¶

The boot_panic_patterns field in the project configuration contains an optional list of panic patterns. These panic patterns apply to all boot items in the project config. Every panic pattern is a case-insensitive regular expression. The test output parser tests if any of these regular expressions can be found in any output line, and in case one of the patterns matches, the whole Test Run is aborted.

Adequate usage of panic patterns improves test execution, because critically failing tests are detected immediately, as opposed to tests that are detected as fails by running into timeouts.

Linux kernel panic messages or application-specific assertion errors are example use cases.

Fail behavior [optional]¶

The fail_behavior field is optional and specifies the behavior of a Test Run after one of its Boot Items failed. There are three options:

• continue: The Test Run continues as is, just like before (default).
• priority: The Test Run’s priority is set to -2, so that all other Boot Items are scheduled way later.
• abort: The complete Test Run gets aborted.

This feature helps to reduce the waiting time of other Test Runs when priority or abort is used.

Local tags [optional]¶

The local_tags_list field is optional and contains, for a specific machine, a list of tags to be used instead of the tags configured in the Controller.

This allows users to override the tag configurations for single project configs without affecting other Test Runs, for example when testing existing features on new hardware.

For each machine_name specified in this list, SoTest finds the matching machine(s) and replaces all of their tags with the ones given as local_tags (see example). Machines that are not listed here are not modified.

Details¶

Paths¶

Files in the exec, load and extra_files items of a boot item must be referenced in one of these two ways:

• As a relative path within the binary archive that was provided at Test Run creation time along with the project configuration.
• As the name of an external dependency.

Example: If the content of the binary archive looks like this:

$ unzip -l binaries.zip
Archive:  binaries.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 12202050  12-04-2019 15:56   initrds/dmidecode.initrd
204350231  12-04-2019 15:56   initrds/kernel-compile.initrd
  4507296  12-04-2019 15:56   kernels/linux-4.19.81.bzImage
  4761984  12-04-2019 15:56   kernels/linux-5.3.8.bzImage
---------                     -------
225821561                     4 files

Then a boot item could look like this:

  - name: Linux kernel compile test
    boot_source:
      bios:
        exec: kernels/linux-5.3.8.bzImage console=ttyS0,115200
        load:
        - initrds/kernel-compile.initrd

iSCSI Disks¶

It is possible to attach one or more iSCSI disks to the system at boot time. The optional iscsi field in a boot item accepts drive parameters in the following format:

iscsi:
  - <BIOS drive ID> = <image name>
  - ...

The BIOS drive ID specifies the drive ID slot that is used locally to attach the drive to the system.

The image name is internally concatenated with the MAC address of the hardware platform instance used while testing. That means if the MAC address of the machine is 01:23:45:67:89:ab and the image name in the config is foobar then the resulting target name is 01-23-45-67-89-ab-foobar. Therefore, the iscsi definition from the config itself is machine agnostic.

In order for such an image name to exist, the administrator of the iSCSI service must create one image per hardware platform instance. This way, no two hardware platform instances can manipulate the same image concurrently. If a machine switches its boot source type (UEFI to legacy BIOS or back), the image must be changed as well.

For more information on the underlying mechanism see the documentation for attaching SAN devices of the iPXE project.

In order to debug iSCSI errors, have a look at the Boot Config section of the boot item in the Web UI that is failing to attach an iSCSI target, and inspect the resulting iSCSI load line.

Hardware Platform Specific Variables¶

It is possible to parametrize exec and load lines with hardware platform specific variables.

The administrator of a SoTest cluster can configure such variables per hardware platform instance. Configured variables can be accessed using the following syntax: @{variable_name} if the variable in the hardware platform configuration is variable_name.

Example: In order to boot the same Linux kernel on multiple hardware platforms that each have different serial ports and baudrates, the following configuration would parametrize the boot item accordingly:

  - name: Linux kernel compile test
    boot_source:
      bios:
        exec: kernels/linux-5.3.8.bzImage console=@{serialport},@{serialbaudrate}

If a variable is not defined for the executing hardware platform, the boot item fails.