Step-by-Step for creating a NixOS Test VM¶
This page walks you through creating a simple NixOS Test VM that runs SoTest.
The code snippets and the example files in this tutorial are available within
the SoTest source code at docs/content/snippets.
Step 1 - NixOS configuration¶
The first step is creating a NixOS configuration file. We’re using the example file provided below. You may adapt the example file to your needs.
Check out the NixOS module guide for detailed information on all module configuration options.
When modifying the configuration, taking a look at the integration tests and the arion config can help:
nix/tests/lib/single_host_test_config.nixnix/tests/lib/postgres_setup_for_sotest_db.nixnix/arion/arion-compose.nix
When starting out with a new installation, it’s easiest not to use real hardware for running the tests but instead to work with a virtualized hardware (e.g. with QEMU). You can then add real hardware once everything else is up and running.
Step 2 - Build the VM¶
Build the VM using your configuration by calling nix-build with your NixOS
config file (we also determine the correct nixpkgs src path to use beforehand):
#!/usr/bin/env bash
# File: build_nixos_vm.sh
set -eou pipefail
# Find script parent dir so we make paths relative to this location
script_dir=$(
cd "$(dirname "${BASH_SOURCE[0]}")"
pwd -P
)
# Edit to point to your own configuration if desired
CONFIG_FILE=${script_dir}/example_nixos_configuration.nix
# Determine the nixpkgs src path
nixpkgs_src=$(nix-instantiate --eval -E "builtins.toString (import ${script_dir}/../../../nix/sources.nix {}).nixpkgs" | tr -d '"')
export NIX_PATH=nixpkgs=$nixpkgs_src
# Build the VM
nix-build '<nixpkgs/nixos>' -A vm -I nixos-config="$CONFIG_FILE"
# Remove any old QEMU image file
# Otherwise the old file will be reused when the VM is started and
# configuration changes won't take effect
rm -f nixos.qcow2
Check out the file ./result/bin/run-nixos-vm that has been generated.
It starts the VM with a set of QEMU command line parameters, some of which allow
for more configuration.
Step 3 - Run the VM¶
You can start the VM using the following wrapper script:
#!/usr/bin/env bash
# File: run_nixos_vm.sh
set -eou pipefail
# Enable forwarding the port to the SoTest Web UI
# Enable forwarding the port to the webserver
# (ports have to match the ones configured in the NixOS configuration)
export QEMU_NET_OPTS="hostfwd=tcp::5000-:5000,hostfwd=tcp::8080-:80"
# Add another network card
# We're pretending this is the network the test machines are connected to
export QEMU_OPTS="-net nic,netdev=user.1,model=virtio -netdev user,id=user.1"
# Set relative directory for the temporary directory to be used by QEMU
export TMPDIR=/tmp/sotest-nixos-vm
mkdir -p $TMPDIR/xchg
export USE_TMPDIR=0
# Run the VM (script has to be executed from same directory as build_nixos_vm.sh)
./result/bin/run-nixos-vm
The script creates a TMPDIR for data exchange with the host at
/tmp/sotest-nixos-vm/xchg and forwards the ports for the SoTest Web UI and for
the configured webserver to the host.
You should now be able to access the SoTest Web UI at http://localhost:5000.
Check the available machines of the Web UI at
http://localhost:5000/machines and verify that the machines configured in
your Controller configuration are present. When the machine list is empty, the
SoTest Controller might be down due to an incorrect configuration. In that case,
log into the VM and check journalctl output for hints on what went wrong.
When everything is correctly configured, the SoTest API returns output as well:
$ curl localhost:5000/machines
[...]
The webserver at http://localhost:8080 should show some output using curl or a
browser. It is configured to serve the contents of $TMPDIR/xchg, so any files
placed in this directory on the host will be available through the webserver.
Due to the QEMU_NET_OPTS from the example, the webserver is available at port
8080 on the host and at port 80 on the guest.
Step 4 - Start a simple Test Run¶
Create a simple Test Run for the virtualized hardware. The User Tutorial has more in-depth information on this topic.
You’ll need to provide a URL to some test binaries (e.g. the SoTest tutorial
binaries) or copy the binaries to the xchg directory so they are served by the
VM’s webserver. You can download the binaries from the tutorial and then copy
them to the xchg directory set up earlier and that is served by HTTPD. This
way, they’ll show up as http://localhost/tutorial-binaries.zip on the VM.
cp tutorial-binaries.zip /tmp/sotest-nixos-vm/xchg
(Alternatively, you can run nix-build nix/release.nix -A tutorial-binaries to
obtain the binaries.)
You also need to provide the project configuration file.
Then you can run:
#!/usr/bin/env bash
# File: start_testrun_example.sh
set -eou pipefail
# Change directory to `snippets`, so we can access the project config file
cd "$(dirname "${BASH_SOURCE[0]}")"
# SoTest API call with some test data
curl \
-F 'boot_files_url=http://localhost/tutorial-binaries.zip' \
-F 'url=https://docs.sotest.io/tutorial' \
-F 'name=Tutorial Test Run' \
-F 'user=Test User' \
-F 'config=@./example_nixos_project_configuration.yaml' \
-X POST \
localhost:5000/test_runs
The API create call yields the ID of your Test Run. Check the SoTest Web UI for detailed information on your Test Run.
Example SoTest Configuration¶
# File: example_nixos_configuration.nix
{ config, pkgs, lib, ... }:
let
tftpFolder = "/tmp/tftp_root";
testDataFolder = "/tmp/testData";
machinesInterfaceIp = "192.168.1.1";
iscsiPort = 3260;
learning-pkg = pkgs.callPackage ../../../optimization/learning_pkg { };
targetIqnPrefix = "iqn.2017-09.de.example.iscsi:";
machineIqnPrefix = "iqn.2017-09.de.example:";
in
{
imports = [
# Adapt this path to point to your SoTest installation
../../../nix/modules/sotest.nix
];
# Required for including the SoTest module
# You can also use this when building the config: export NIXPKGS_ALLOW_UNFREE=1
nixpkgs.config.allowUnfree = true;
networking = {
firewall = {
# Used for webserver
allowedTCPPorts = [ 80 ];
allowedUDPPorts = [ ];
};
interfaces.eth1 = {
useDHCP = false;
ipv4.addresses = [{ address = machinesInterfaceIp; prefixLength = 24; }];
};
firewall.interfaces.eth1 = {
allowedUDPPorts = [
67 # dhcp/dnsmasq
69 # tftp
iscsiPort # iSCSI
];
allowedTCPPorts = [
80 # webserver
iscsiPort # iSCSI
5432 # db
];
};
};
services.dnsmasq = {
enable = true;
resolveLocalQueries = false;
extraConfig = builtins.replaceStrings [ "\${tftpFolder}" ] [ "${tftpFolder}" ] (builtins.readFile ./dnsmasq_example_configuration.conf);
};
systemd.services.dnsmasq = {
# Required so that dnsmasq can access the tftpFolder
serviceConfig.PrivateTmp = lib.mkForce false;
preStart = ''
# Create tftpFolder here as otherwise it'll only be created when
# the SoTest Controller starts
mkdir -m 0777 -p ${tftpFolder}
'';
};
# tgt is the iscsi service we're using
environment.systemPackages = with pkgs; [
tgt
];
# pkgs.tgt already provides a systemd service file
# systemd.packages allows systemd to find the .service file for tgt
systemd.packages = [ pkgs.tgt ];
# Enable iscsi service to start automatically
systemd.services.tgtd = {
enable = true;
# Additional "wantedBy" required because the default puts "wantedBy" into
# the "[Install]" section
# See: https://search.nixos.org/options?channel=unstable&show=systemd.services.%3Cname%3E.wantedBy&query=systemd.services.%3Cname%3E.wantedBy
wantedBy = [ "multi-user.target" ];
};
# The tgt service expects to find its config file at /etc/tgt/targets.conf
environment.etc = {
"tgt/targets.conf" = {
text = ''
<target ${targetIqnPrefix}:50-7b-9d-27-ab-77-win10-abc>
backing-store /media/storage/iscsi/win10-abc/50-7b-9d-27-ab-77.img
write-cache off
incominguser testUser testPassword
</target>
'';
mode = "0440";
};
};
# Create dummy iscsi image that can be checked for availability by iscsi service
environment.extraInit = ''
mkdir -p /media/storage/iscsi/win10-abc/
echo "test image" > /media/storage/iscsi/win10-abc/50-7b-9d-27-ab-77.img
'';
# This is the database that's used by SoTest. You may exchange this for
# another database of your choice.
services.postgresql = {
enable = true;
package = pkgs.postgresql_10;
inherit (config.services.sotest-db) port;
enableTCPIP = true;
authentication = "host all all 0.0.0.0/0 md5";
initialScript = with config.services.sotest-db; pkgs.writeText "postgres-initScript" ''
CREATE ROLE ${user} WITH LOGIN PASSWORD '${password}' CREATEDB;
CREATE DATABASE ${database};
GRANT ALL PRIVILEGES ON DATABASE ${database} TO ${user};
'';
};
services = {
# The database configuration used by the sotest-webui and the
# sotest-controller. Please make sure to adapt the values.
sotest-db = {
host = "localhost";
port = 5432;
user = "myuser";
password = "mypw";
database = "mydb";
};
# The SoTest Web UI
sotest-webui = {
enable = true;
openFirewall = true;
port = 5000;
# Set to true to automatically migrate the DB schema before the service is
# started
autoMigration = true;
# Set to true to enable failure probability prediction. Either enable
# sotest-model-generation or provide your own `predictionCommand`.
prediction = true;
storageDir = testDataFolder;
storageUrl = "http://localhost";
storageGCSizeStopMB = 0;
storageGCSizeTriggerMB = 1;
};
sotest-db-filler = {
# Enable the manual insertion of arbitrary test data into the database.
enable = true;
};
# Enable to regularly generate a prediction model from the database.
sotest-model-generation = {
enable = true;
nJobs = 1;
timer = {
enable = true;
interval = "*-*-* *:*:00";
};
};
# The SoTest Controller
# Use a recursive set so we don't have to retype tftpFolder and mac
sotest-controller = rec {
enable = true;
binaryTmpDir = "/tmp/sotest-binaries";
inherit tftpFolder;
authMethods = {
"https://gitlab.example.de/" = {
gitlab_private_token = "xxxxxxxxxxxxxxxxxxxx";
};
};
machines = [
rec {
id = "qemu_1";
name = "qemu_machine";
tags = [ "qemu" "no_power" ];
power_interface = "none";
mac = "02-03-04-05-06-07";
config_vars = [ "linux_terminal = ttyS0" ];
serial_interfaces = [
{
type = "custom";
cmd = "qemu-system-x86_64 -boot n -net user,tftp=${tftpFolder},bootfile=ipxe.kpxe -net nic,macaddr=${mac} -nographic";
}
];
updated_at = "2022-08-04T00:00:00Z";
}
];
iscsiConfig = {
targetIp = machinesInterfaceIp;
targetPort = iscsiPort;
inherit targetIqnPrefix machineIqnPrefix;
};
cacheVersion = 1;
};
# Webserver for serving own SoTest test binaries
# The NixOS QEMU VM mounts a provided host $TMPDIR at /tmp/xchg in the guest
# Use apache because nginx always returns 404 although files are present
httpd = {
enable = true;
adminAddr = "test@example.org";
virtualHosts.localhost = {
documentRoot = testDataFolder;
servedDirs = [
{ urlPath = "/"; dir = "/tmp/xchg"; }
{ urlPath = "/"; dir = testDataFolder; }
];
};
};
};
# When using a test VM, you may set an empty root password for easy access
users.users.root.initialHashedPassword = "";
# give both: the webserver and the webui access to the test data storage
users.extraGroups.TestDataAccessors.members = [
config.services.httpd.user
config.systemd.services.sotest-webui.serviceConfig.User
];
systemd.tmpfiles.rules = [ "d ${testDataFolder} 0070 - TestDataAccessors" ];
system.stateVersion = "22.05";
}