Changelog¶
All notable changes to SoTest will be documented in this file.
Release SoTest 0.34¶
Added¶
- New field
updated_at
for machines in the controller config that must hold a date when the machine was updated the last time. - New field
cache_version
in the controller config that holds a number and is used to indicate a cache invalidation. - New way to define a boot source in the project config via the
boot_source
key. Now one boot item can handle BIOS and UEFI configs at once.
Release SoTest 0.33¶
Added¶
- Option to configure a command delay for the switch off command of the Antrax power socket. This is necessary so that machines connected to the switchable PDU have enough time to properly switch off. Default is 1000ms.
Release SoTest 0.32¶
Fixed¶
- Fixed bug in JSON representation of a
TestState
, where the number of fails was not correct - Fixed bug where retired machines wouldn’t be updated in the data base. Instead, they stayed inactive and offline.
- Fixed bug in serial reader, where the timeout thread wouldn’t know the serial thread id when no serial line was send ever. This lead to never ending test cases.
Release SoTest 0.31¶
Added¶
- New app
sotest-protocol-parser
for quickly validating if a file conforms to the SoTest protocol. - Machine configs can now set the
retired
flag. Retired machines act like inactive ones: they are not executing any tests and are ignored in Test Run creation. They are also hidden from the user, therefore not shown in the Infrastructure page.
Changed¶
- The maximum size of a test log is now restricted with a byte limit of 2MB. The old line count and line length restrictions were removed.
Removed¶
- The Terminal Server was outsourced into a new standalone repository.
Release SoTest 0.30¶
Changed¶
- Added redirects for the old
/results
routes in the Web UI. These were removed as part of the API cleanup but the old routes are still used because Test Runs are linked in tickets.
Release SoTest 0.29¶
Added¶
- Boot files can now be uploaded during the Test Run creation by specifying the
form attribute
boot_files
in the POST request of the Test Run creation API. The previous method to specifyboot_files_url
is still supported. Boot files that are directly uploaded to SoTest will be stored for some time so tests can be retried. Eventually the boot files will be garbage collected. - JSON API: It’s now possible to request data from the Web UI in JSON by adding
the HTTP header
Accept: application/json
.- This works on the following routes with
GET
:/test_runs/<id>
- Test Run Results. This shows the overview over a single Test Run including its Boot Items and Execution Items but excluding the Test States that contain the Serial Logs and Event Logs./test_runs/<id>/status
- This shows only the status of the Test Run/test_runs/<id>/exec_items/<id>
- Test Run Details. This shows the Serial Logs and Event Logs for one Execution Item./
- Test Run Index (list of Test Runs). Filtering and pagination work in the same way as in the HTML view. Note that this route will likely move to/test_runs
at some point in the future and pagination and filtering will then be implemented using query parameters./machines
- Infrastructure Interface. Displays connected machines./machines/<machine_id>
- This shows only the selected machine.
- There are also several new routes that can be used with HTTP
POST
:/test_runs
- Create a new Test Run./test_runs/<id>
- Abort or reset a Test Run./test_runs/<id>/exec_items/<id>
- Abort or reset an Execution Item./machines/<machine_id>
- Activate or deactivate a machine.
- Detailed information on the usage can be found in the documentation.
- This works on the following routes with
- Local controller: supports tags for more production-like debugging
Changed¶
- Due to the switch to the JSON API, several routes in the Web UI have moved.
/results
–>/test_runs
/results/<test_run_id>/<exec_item_id>
–>/test_runs/<test_run_id>/exec_items/<exec_item_id>
/machinestates
–>/machines
- Several of the internally used abort/reset and activate/deactivate
handlers use
POST
forms now and could therefore be moved to simpler routes:/aborttestrun/<id>
–>/test_runs/<id>
/resettestrun/<id>
–>/test_runs/<id>
/abortbootitem/<id>
–>/test_runs/<id>/exec_items/<id>
/resetbootitem/<id>
–>/test_runs/<id>/exec_items/<id>
/machinestates/<machine_id>/<ActivityState>
–>/machines/<machine_id>
- The routes at
/api/create
and/api/query
that are used by therun_sotest
script are unchanged. We’re planning to migrate the script to the new routes/test_runs
and/test_runs/<id>/status
in the future. - Please check the documentation for more information.
Release SoTest 0.28¶
Fixed¶
- The timeout thread is now killed properly on user aborts and for all other asynchronous exceptions that might happen.
Release SoTest 0.27¶
Added¶
- Instances of the TS now serve their OpenAPI documentation under
/openapi.json
. - Added a custom power interface to execute arbitrary commands, such as launching a QEMU machine.
- The Test Run Details View in the SoTest Web UI used to show the machine name for the Test State but not the machine ID that the test was executed on (in the table above “MAC Address”). Now it also shows the machine ID. Since the machine ID was not stored in the database at all, the field may be empty for old entries.
- New field
local_tags_list
in the project config used for overriding machine tags specified in the Controller for single Test Runs and single machines. - The project config is saved in the database and displayed on the results page of the Web UI.
Fixed¶
- The PowerState App used for remote switching machines on and off now filters machines based on the machine ID rather than the machine name.
- The Controller now correctly cleans up the temporary TFTP directory when an error occurs during boot file allocation, for example due to an incorrect file path.
Removed¶
- Removed legacy field
machines
in the project config that allowed ignoring the tagging mechanism by specifying a list of machines to run the test on. It’s still possible to restrict the machines using the machine name as a tag or using the newlocal_tags
mechanism.
Release SoTest 0.26¶
Changed¶
- Changed the serial line limit from 400 to 4000 characters
Fixed¶
- Fixed a bug that let tests that failed with a line or log limit exception seem to never end on the index and results page.
Release SoTest 0.25¶
Fixed¶
LogError
handling: Tests won’t retry twice onLogError
exceptions anymore
Release SoTest 0.24¶
Added¶
- SoTest Web UI: Window titles and pages now contain the Test Run ID and boot item name / machines where applicable
- SoTest Web UI: External links are now marked with a
square-new-window
icon
Fixed¶
- SoTest Web UI: Middle clicking and CTRL-click on a row of the Test Index is
now possible and opens the Test Run in a new tab. Please note: The Test Run
ID is purposely kept as a link to the Test Run, as this enables all
right-click features like
Copy Link
orOpen Link in New Window
. Please be aware that this means that a middle click or CTRL-click on this link on the Index page now opens two new tabs in the background.
Known Issues¶
- SoTest Web UI: Middle clicking and CTRL-click on the ID of a Test Run in the Test Index opens two new tabs in the background as two click listeners are triggered—one for the test row and one for the ID. Users can instead click on any other part of the test row to avoid the issue.
Release SoTest 0.23¶
Added¶
- Added two limits to the serial log to prevent DOS attacks via long lines or logs
- A serial line is no longer allowed to be longer than 400 characters
- A serial log is no longer allowed to have more than 10101 lines
Fixed¶
- Fixed a group resolve error in the SoTest modules for SoTest deployment on NixOS systems
- Fixed carriage return handling for the serial log
- Fixed issues in the serial handling when appending log lines which caused high memory usage
Release SoTest 0.22¶
Fixed¶
- Fixed release 0.21: added the missing unpacking dependency to the Web UI docker image
Release SoTest 0.21¶
Fixed¶
- Fixed corner cases where the Controller used too much memory while unpacking zip archives
Release SoTest 0.20¶
Added¶
-
It’s now possible to filter for timestamps. We currently support
>
,<
,>=
and<=
with either dates or complete timestamps. E.g.time > 2021-10-12 11:30:00 UTC
time >= 2021-10-21
-
It is now possible to filter for negative numbers in the SoTest Web UI, e.g. priority = -1
-
The version field in the footer of the Web UI now links to the respective section of the public changelog.
-
Serial definitions now also have an optional
display_name
field to be able to distinguish between multiple serial lines of the same type. If given, it is added everywhere where thetype
of the serial was displayed before in the following format:<type> (<display_name>)
. This affects the following places in the Web UI:- the Infrastructure Interface “Serial Interface” column
- the header of the raw log output of a test result, to differentiate which log comes from which serial connection
- the event log of a test result, to make it more clear which serial connection was started when
-
All applications now have a
--version
flag that gives the version number of the currently deployed app.
Release SoTest 0.19¶
Changed¶
- The database configuration fields prefixed with
postgres_
have been dropped.- The configuration field
port
works as expected. Default value remains unchanged. - The configuration field
connection_pool_size
works as expected. Default value remains unchanged.
- The configuration field
- Improve nix derivations for all apps.
packages.x86.generate-history
contains the appgenerate_history
.packages.x86.powerstate
contains the apppowerstate
.packages.x86.sotest-db-migration
contains the appsotest_db_migration
.packages.x86.sotest-controller
contains the appsrun
andsotest_local
.
Release SoTest 0.18¶
Added¶
- The infrastructure interface of the Web UI now shows the Controller version per machine
- The user can now specify what happens after a boot item failed in a Test Run.
This greatly reduces developer wait times, as Test Runs with failures are
abort
-ed early or reduced in theirpriority
. Therefore all other Test Runs are executed earlier. If the user needs all results no matter of failure, then the default option to simplycontinue
is the right choice. - The project config gets the new optional field:
fail_behavior
. There are three options:continue
: The TestRun continues as is, just like before (default).priority
: The TestRuns priority is set to -2, so that all other Boot Items are scheduled way later.abort
: The complete Test Run gets aborted.
- The Web UI now shows the version of the deployed SoTest Web UI app.
- Log lines are now prefixed with a timestamp that represents the time the line was read.
Changed¶
- Error messages regarding incorrect file names have improved readability.
- Error messages regarding connection problems now censor secrets that may show.
Release SoTest 0.17¶
Fixed¶
- Update to
persistent-2.11
might have fixed the bug “another command is already in progress” that leads to broken states sometimes.
Release SoTest 0.16¶
Added¶
- Added 3 power interface timings that can be specified for the raspi power interface (see power configuration).
nix/modules/sotest.nix
is now a single NixOS module definition for both the Web UI and Controller services. The central definition shares common parts and simplifies the configuration of a SoTest instance when both Web UI and Controller run on the same host.- All SoTest config .yaml files are now automatically generated from the declarative NixOS module interface.
- Automatic DB Migration after updates can now be activated on the Web UI node
with
services.sotest-webui.autoMigration = true;
and will be performed before the service is started.
Release SoTest 0.15¶
Added¶
- Optimization feature
- Generate a history out of the SoTest DB.
- Train a model with the generated history.
- Calculate the failure prediction with the trained model.
- Multiple serial interfaces.
- Configure multiple serial interfaces.
- For examples have a look at serial configuration.
Changed¶
- Renamed QemuSerial to CustomSerial.