Changelog

All notable changes to SoTest will be documented in this file.

Release SoTest 0.34

Added

  • New field updated_at for machines in the controller config that must hold a date when the machine was updated the last time.
  • New field cache_version in the controller config that holds a number and is used to indicate a cache invalidation.
  • New way to define a boot source in the project config via the boot_source key. Now one boot item can handle BIOS and UEFI configs at once.

Release SoTest 0.33

Added

  • Option to configure a command delay for the switch off command of the Antrax power socket. This is necessary so that machines connected to the switchable PDU have enough time to properly switch off. Default is 1000ms.

Release SoTest 0.32

Fixed

  • Fixed bug in JSON representation of a TestState, where the number of fails was not correct
  • Fixed bug where retired machines wouldn’t be updated in the data base. Instead, they stayed inactive and offline.
  • Fixed bug in serial reader, where the timeout thread wouldn’t know the serial thread id when no serial line was send ever. This lead to never ending test cases.

Release SoTest 0.31

Added

  • New app sotest-protocol-parser for quickly validating if a file conforms to the SoTest protocol.
  • Machine configs can now set the retired flag. Retired machines act like inactive ones: they are not executing any tests and are ignored in Test Run creation. They are also hidden from the user, therefore not shown in the Infrastructure page.

Changed

  • The maximum size of a test log is now restricted with a byte limit of 2MB. The old line count and line length restrictions were removed.

Removed

  • The Terminal Server was outsourced into a new standalone repository.

Release SoTest 0.30

Changed

  • Added redirects for the old /results routes in the Web UI. These were removed as part of the API cleanup but the old routes are still used because Test Runs are linked in tickets.

Release SoTest 0.29

Added

  • Boot files can now be uploaded during the Test Run creation by specifying the form attribute boot_files in the POST request of the Test Run creation API. The previous method to specify boot_files_url is still supported. Boot files that are directly uploaded to SoTest will be stored for some time so tests can be retried. Eventually the boot files will be garbage collected.
  • JSON API: It’s now possible to request data from the Web UI in JSON by adding the HTTP header Accept: application/json.
    • This works on the following routes with GET:
      • /test_runs/<id> - Test Run Results. This shows the overview over a single Test Run including its Boot Items and Execution Items but excluding the Test States that contain the Serial Logs and Event Logs.
      • /test_runs/<id>/status - This shows only the status of the Test Run
      • /test_runs/<id>/exec_items/<id> - Test Run Details. This shows the Serial Logs and Event Logs for one Execution Item.
      • / - Test Run Index (list of Test Runs). Filtering and pagination work in the same way as in the HTML view. Note that this route will likely move to /test_runs at some point in the future and pagination and filtering will then be implemented using query parameters.
      • /machines - Infrastructure Interface. Displays connected machines.
      • /machines/<machine_id> - This shows only the selected machine.
    • There are also several new routes that can be used with HTTP POST:
      • /test_runs - Create a new Test Run.
      • /test_runs/<id> - Abort or reset a Test Run.
      • /test_runs/<id>/exec_items/<id> - Abort or reset an Execution Item.
      • /machines/<machine_id> - Activate or deactivate a machine.
    • Detailed information on the usage can be found in the documentation.
  • Local controller: supports tags for more production-like debugging

Changed

  • Due to the switch to the JSON API, several routes in the Web UI have moved.
    • /results –> /test_runs
    • /results/<test_run_id>/<exec_item_id> –> /test_runs/<test_run_id>/exec_items/<exec_item_id>
    • /machinestates –> /machines
    • Several of the internally used abort/reset and activate/deactivate handlers use POST forms now and could therefore be moved to simpler routes:
      • /aborttestrun/<id> –> /test_runs/<id>
      • /resettestrun/<id> –> /test_runs/<id>
      • /abortbootitem/<id> –> /test_runs/<id>/exec_items/<id>
      • /resetbootitem/<id> –> /test_runs/<id>/exec_items/<id>
      • /machinestates/<machine_id>/<ActivityState> –> /machines/<machine_id>
    • The routes at /api/create and /api/query that are used by the run_sotest script are unchanged. We’re planning to migrate the script to the new routes /test_runs and /test_runs/<id>/status in the future.
    • Please check the documentation for more information.

Release SoTest 0.28

Fixed

  • The timeout thread is now killed properly on user aborts and for all other asynchronous exceptions that might happen.

Release SoTest 0.27

Added

  • Instances of the TS now serve their OpenAPI documentation under /openapi.json.
  • Added a custom power interface to execute arbitrary commands, such as launching a QEMU machine.
  • The Test Run Details View in the SoTest Web UI used to show the machine name for the Test State but not the machine ID that the test was executed on (in the table above “MAC Address”). Now it also shows the machine ID. Since the machine ID was not stored in the database at all, the field may be empty for old entries.
  • New field local_tags_list in the project config used for overriding machine tags specified in the Controller for single Test Runs and single machines.
  • The project config is saved in the database and displayed on the results page of the Web UI.

Fixed

  • The PowerState App used for remote switching machines on and off now filters machines based on the machine ID rather than the machine name.
  • The Controller now correctly cleans up the temporary TFTP directory when an error occurs during boot file allocation, for example due to an incorrect file path.

Removed

  • Removed legacy field machines in the project config that allowed ignoring the tagging mechanism by specifying a list of machines to run the test on. It’s still possible to restrict the machines using the machine name as a tag or using the new local_tags mechanism.

Release SoTest 0.26

Changed

  • Changed the serial line limit from 400 to 4000 characters

Fixed

  • Fixed a bug that let tests that failed with a line or log limit exception seem to never end on the index and results page.

Release SoTest 0.25

Fixed

  • LogError handling: Tests won’t retry twice on LogError exceptions anymore

Release SoTest 0.24

Added

  • SoTest Web UI: Window titles and pages now contain the Test Run ID and boot item name / machines where applicable
  • SoTest Web UI: External links are now marked with a square-new-window icon

Fixed

  • SoTest Web UI: Middle clicking and CTRL-click on a row of the Test Index is now possible and opens the Test Run in a new tab. Please note: The Test Run ID is purposely kept as a link to the Test Run, as this enables all right-click features like Copy Link or Open Link in New Window. Please be aware that this means that a middle click or CTRL-click on this link on the Index page now opens two new tabs in the background.

Known Issues

  • SoTest Web UI: Middle clicking and CTRL-click on the ID of a Test Run in the Test Index opens two new tabs in the background as two click listeners are triggered—one for the test row and one for the ID. Users can instead click on any other part of the test row to avoid the issue.

Release SoTest 0.23

Added

  • Added two limits to the serial log to prevent DOS attacks via long lines or logs
    • A serial line is no longer allowed to be longer than 400 characters
    • A serial log is no longer allowed to have more than 10101 lines

Fixed

  • Fixed a group resolve error in the SoTest modules for SoTest deployment on NixOS systems
  • Fixed carriage return handling for the serial log
  • Fixed issues in the serial handling when appending log lines which caused high memory usage

Release SoTest 0.22

Fixed

  • Fixed release 0.21: added the missing unpacking dependency to the Web UI docker image

Release SoTest 0.21

Fixed

  • Fixed corner cases where the Controller used too much memory while unpacking zip archives

Release SoTest 0.20

Added

  • It’s now possible to filter for timestamps. We currently support >, <, >= and <= with either dates or complete timestamps. E.g.

    • time > 2021-10-12 11:30:00 UTC
    • time >= 2021-10-21

  • It is now possible to filter for negative numbers in the SoTest Web UI, e.g. priority = -1

  • The version field in the footer of the Web UI now links to the respective section of the public changelog.

  • Serial definitions now also have an optional display_name field to be able to distinguish between multiple serial lines of the same type. If given, it is added everywhere where the type of the serial was displayed before in the following format: <type> (<display_name>). This affects the following places in the Web UI:

    • the Infrastructure Interface “Serial Interface” column
    • the header of the raw log output of a test result, to differentiate which log comes from which serial connection
    • the event log of a test result, to make it more clear which serial connection was started when

  • All applications now have a --version flag that gives the version number of the currently deployed app.

Release SoTest 0.19

Changed

  • The database configuration fields prefixed with postgres_ have been dropped.
    • The configuration field port works as expected. Default value remains unchanged.
    • The configuration field connection_pool_size works as expected. Default value remains unchanged.
  • Improve nix derivations for all apps.
    • packages.x86.generate-history contains the app generate_history.
    • packages.x86.powerstate contains the app powerstate.
    • packages.x86.sotest-db-migration contains the app sotest_db_migration.
    • packages.x86.sotest-controller contains the apps run and sotest_local.

Release SoTest 0.18

Added

  • The infrastructure interface of the Web UI now shows the Controller version per machine
  • The user can now specify what happens after a boot item failed in a Test Run. This greatly reduces developer wait times, as Test Runs with failures are abort-ed early or reduced in their priority. Therefore all other Test Runs are executed earlier. If the user needs all results no matter of failure, then the default option to simply continue is the right choice.
  • The project config gets the new optional field: fail_behavior. There are three options:
    • continue: The TestRun continues as is, just like before (default).
    • priority: The TestRuns priority is set to -2, so that all other Boot Items are scheduled way later.
    • abort: The complete Test Run gets aborted.
  • The Web UI now shows the version of the deployed SoTest Web UI app.
  • Log lines are now prefixed with a timestamp that represents the time the line was read.

Changed

  • Error messages regarding incorrect file names have improved readability.
  • Error messages regarding connection problems now censor secrets that may show.

Release SoTest 0.17

Fixed

  • Update to persistent-2.11 might have fixed the bug “another command is already in progress” that leads to broken states sometimes.

Release SoTest 0.16

Added

  • Added 3 power interface timings that can be specified for the raspi power interface (see power configuration).
  • nix/modules/sotest.nix is now a single NixOS module definition for both the Web UI and Controller services. The central definition shares common parts and simplifies the configuration of a SoTest instance when both Web UI and Controller run on the same host.
  • All SoTest config .yaml files are now automatically generated from the declarative NixOS module interface.
  • Automatic DB Migration after updates can now be activated on the Web UI node with services.sotest-webui.autoMigration = true; and will be performed before the service is started.

Release SoTest 0.15

Added

  • Optimization feature
    • Generate a history out of the SoTest DB.
    • Train a model with the generated history.
    • Calculate the failure prediction with the trained model.
    • Multiple serial interfaces.
  • Configure multiple serial interfaces.

Changed

  • Renamed QemuSerial to CustomSerial.