LNST in containers

LNST supports running both agents and controller in containers at the host machine. LNST uses custom RPC protocol to communicate between controller and agents which uses separate network interface to not interfere with test and so, it doesn’t matter where controller and agents are running as long as they can communicate.

With support of running LNST in containers your machine setup might look like this:

both controller and agents are running on your baremetal machines
controller is running on your baremetal machine and agents are running in containers
controller is running in container and agents are running on your baremetal machine
both controller and agents are running in containers

This article describes how to run individual parts of LNST in containers. If you want to run either controller or agents on baremetal see Install LNST and Hello world section.

Common requirements

We recommend to use Podman as this was developed and tested with Podman but should work with Docker as well.

If you want to use Podman, follow installation steps on official Podman installation page.

Containerized agents

Containers and networks are dynamically created based on recipe requirements. Containers are also automatically connected to networks.

Requirements

The first requirement is Podman, follow installation steps on official Podman installation page.

Container Networking

LNST ContainerPoolManager supports 3 different plugins to generate the networks defined in Recipe Requirements.

The first two plugins are podman native plugins netavark and CNI.

The final plugin is the recently added custom_lnst. This plugin has no requirements on your system or it’s configuration, it utilizes ip-route commands to build a network using veth and bridge interfaces manually. So the only requirement here is to run the recipe with root privileges.

To select which plugin is used for networking you can change the value of the network_plugin argument when creating the Controller class:

ctl = Controller(
    poolMgr=ContainerPoolManager,
    mapper=ContainerMapper,
    network_plugin="custom_lnst",
)

CNI system configuration

To use CNI you may need to adjust the systemwide container configurations.

Remove all the LNST’s leftovers from /etc/cni/net.d/lnst_*.conflist if you already tried to run LNST

2. Set default network backend to cni, add following block to containers.conf - documentation:

[network]
network_backend="cni"

Install CNI plugins:

dnf install containernetworking-plugins

Start your Podman instance

Enabling Podman API service:

Podman API is also required, follow the steps below:

systemctl enable --now podman.socket

and get socket URL:

systemctl status podman.socket | grep "Listen:"

Starting Podman API manually:

If you don’t want to run Podman API as a service, you can start it manually. Don’t forget to run the command below with root privileges.

podman system service --timeout 0 --log-level=debug

Socket URL could be found at the top of logs generated by this command.

The usual URL is unix:/run/podman/podman.sock

Build LNST agent image

Currently, LNST does not support automated building, so build LNST agent machine image.

Podman uses different storage locations for root-full and root-less images, so make sure you build image to root-full storage. LNST currently uses the default storage location. Build context should be a directory, where your LNST project is located.

Your local copy of LNST is used by agents in containers.

Use -t argument to name your image, this name is later used.

cd your_lnst_project_directory
podman build . -t lnst -f container_files/agent/Dockerfile

Now is everything ready to run LNST in containers. For testing purposes, we can use HelloWorldRecipe from Creating an executable “HelloWorld” test script.

Only initialization of Controller() object has to be changed:

from lnst.Controller.MachineMapper import ContainerMapper
from lnst.Controller.ContainerPoolManager import ContainerPoolManager

podman_uri = ""  # podman URI from installation step above
image_name = ""  # name of image from build step above
ctl = Controller(poolMgr=ContainerPoolManager, mapper=ContainerMapper, podman_uri=podman_uri, image=image_name)

And run the script.

Classes documentation

class lnst.Controller.MachineMapper.ContainerMapper

Implements simple matching algorithm that maps containers to requirements. Containers are created in lnst.Controller.ContainerPoolManager.ContainerPoolManager using requirements.

set_pools_manager(pool_manager): lnst.Controller.MachineMapper.ContainerMapper does not support multiple pools but it requires pool manager.

matches(): 1:1 mapping of containers to requirements

class lnst.Controller.ContainerPoolManager.ContainerPoolManager(pools, msg_dispatcher, ctl_config, podman_uri, image, network_plugin='netavark', pool_checks=True)

This class implements managing containers and networks. It uses Podman API to handle operations with containers, the API needs to be running with root privileges.

Parameters:

pools (dict) – Dictionary that contains pools. In lnst.Controller.ContainerPoolManager.ContainerPoolManager are pools dynamically created based on recipe requirements. That means this parameter is not used but it is needed to keep parameters of this class and lnst.Controller.AgentPoolManager.AgentPoolManager the same.
msg_dispatcher (lnst.Controller.MessageDispatcher.MessageDispatcher)
ctl_config (lnst.Controller.Config.CtlConfig)
pool_checks (boolean (default True)) – if False, will disable checking the online status of Agents
podman_uri (str) – Mandatory parameter
image (str) – Mandatory parameter
network_plugin (Optional[str]) – Podman network plugin, ‘cni’, ‘netavark’ or ‘custom_lnst’, if unset, the network backend is auto-detected

process_reqs(mreqs: dict): This method is called by lnst.Controller.MachineMapper.ContainerMapper, it is responsible for creating containers and networks.

Containerized controller

Using containerized agents

Before proceeding with containerized controller, you need to build LNST agent image (see Containerized agents) you also need to provide following parameters as environment variables to controller container:

PODMAN_URI - URI to Podman socket, e.g. tcp://localhost[:port]. This needs to be accessible from container.
IMAGE_NAME - name of the image you built for agents.

It expects that you use CNI as network backend for Podman.

Test environment description

Instead of manually creating machine pool XMLs, you can provide a test environment description file (test_environment.json). The controller generates pool XMLs from it at startup.

The file must be located at /lnst/container_files/controller/pool/test_environment.json inside the container (see test_environment.example.json for reference):

podman run -v /path/to/test_environment.json:/lnst/container_files/controller/pool/test_environment.json:ro ...

Format:

[
    {
        "test_system_name": "host1",
        "hostname": "machine1.example.com",
        "ssh_port": 22,
        "username": "root",
        "password": "",
        "test_nic_hw_addrs": [
            "00:00:5e:00:53:01"
        ]
    },
    {
        "test_system_name": "host2",
        "hostname": "machine2.example.com",
        "ssh_port": 22,
        "username": "root",
        "password": "",
        "test_nic_hw_addrs": [
            "00:00:5e:00:53:02"
        ]
    }
]

Fields:

test_system_name — unique identifier for the machine (used in pool XML filename)
hostname — FQDN or IP of the remote host
ssh_port — SSH port
username — SSH username
password — SSH password (leave empty for key-based authentication)
test_nic_hw_addrs — list of MAC addresses of NICs used for testing (verified at startup)

Automatic agent setup

When a test environment description file is present, the controller container automatically sets up LNST agents on remote hosts at startup. For each host defined in test_environment.json, the container:

Connects via SSH using credentials from the test environment description
Runs setup_agent.sh remotely — installs LNST agent, its dependencies, and starts the lnst-agent systemd service

The agent is installed using the same LNST version as the controller (determined by the LNST_SRC build argument). If the agent is already running on a host, the setup is skipped.

SSH authentication uses either the password from the test environment description file or SSH keys mounted into the container (e.g. -v machine_keys/:/root/.ssh:ro).

Note

Remote hosts must be reachable from the container via SSH. If using the default network mode, you may need --network=host for the container.

Using baremetal agents

If you decide to run agents on baremetal machines, you can either provide a test environment description (see above) or prepare machine pool XMLs manually (see Creating a simple machine pool) and mount them into the container at runtime.

Mount your pool directory to /root/.lnst/pool/ in the container (read-only access is sufficient):

podman run -v /path/to/pool:/root/.lnst/pool:ro ... lnst_controller

Warning

If SELinux is enforcing on the host, mounted volumes may not be accessible from inside the container. The container runner will detect this on startup and print an error message. To fix this, you can:

add the :z or :Z suffix to the volume mount (e.g. -v /host/path:/container/path:z) to let Podman relabel the directory automatically
relabel the host directory manually with chcon -Rt svirt_sandbox_file_t /host/path
disable SELinux label confinement for the container with --security-opt label=disable

Build and run controller

Build the controller image:

podman build -t lnst_controller -f container_files/controller/Dockerfile .

The image clones LNST from https://github.com/LNST-project/lnst.git into /lnst inside the container and installs its dependencies.

Pinning LNST version

By default, the image clones the latest version of LNST. To pin to a specific git reference, use the LNST_SRC build argument:

podman build --build-arg LNST_SRC="https://github.com/LNST-project/lnst.git@<ref>" \
    -t lnst_controller -f container_files/controller/Dockerfile .

Where <ref> can be a commit hash, branch name, or tag.

The same LNST_SRC value is also used to install matching LNST version on remote agents (see Automatic agent setup).

Note

If you want to use a local copy of LNST (e.g. during development), you can mount your project directory to /lnst in the container. The LNST virtual environment is located outside of /lnst/, so if your changes require reinstallation of LNST and/or its dependencies, you need to rebuild the image.

Note

The machine pool is not baked into the image. You must mount it at runtime (see Using baremetal agents above).

Before running the container, you need to provide environment variables:

RECIPE - name of recipe class, these are loaded from lnst.Recipes.ENRT as wildcard import
RECIPE_PARAMS - ; separated list of parameters for recipe. Each parameter is in format key=value
FORMATTERS - ; separated list of formatters, these are loaded from lnst.Formatters as wildcard import
DEBUG - enables/disables LNST’s debug mode

Warning

RECIPE, RECIPE_PARAMS and FORMATTERS are parsed using Python’s eval function, which is a security risk. Make sure you trust the source of these variables.

Using the test database

Instead of specifying RECIPE and RECIPE_PARAMS environment variables, you can define a list of tests in container_files/controller/test_db.json. When the RECIPE environment variable is not set, the container runner will automatically execute all tests defined in test_db.json in order.

The location of the test database can be overridden with the TEST_DB environment variable. This accepts a local file path or an HTTP(S) URL:

# Local file (default)
podman run -e TEST_DB=/path/to/my_tests.json ...

# Remote URL
podman run -e TEST_DB=https://server/tests.json ...

Each entry in the JSON array is an object with the following keys:

uuid – (optional) unique identifier for the test. When set, the results directory for this test uses the UUID as its name instead of the default {index}_{recipe_name} format.
recipe_name – string name of the recipe class (loaded from lnst.Recipes.ENRT)
params – object of parameters to pass to the recipe constructor

Example test_db.json:

[
    {
        "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
        "recipe_name": "SimpleNetworkRecipe",
        "params": {
            "perf_iterations": 1,
            "perf_duration": 10,
            "driver": "ice"
        }
    },
    {
        "recipe_name": "BondRecipe",
        "params": {
            "bonding_mode": "active-backup",
            "miimon_value": 5,
            "driver": "ice"
        }
    }
]

Tests run sequentially and each test is independent – a failure in one test does not prevent subsequent tests from running. A summary of pass/fail results is printed at the end of the run.

To use the test database, RECIPE and RECIPE_PARAMS must not be set.

Exporting results

Results are automatically exported to /root/.lnst/results/ inside the container. Each recipe run gets its own subdirectory (e.g. 0_SimpleNetworkRecipe/) containing:

controller.log – human-readable log with debug-level output
run-data-{i}.json – JSON-formatted results for each recipe run
run-data-{i}.lrc – pickled/compressed run data for each recipe run

At the end of execution, all result directories are zipped into results.zip and verified for integrity.

To access results on the host, mount a volume to /root/.lnst/results/:

podman run -e DEBUG=1 -v /host/path/to/results:/root/.lnst/results --rm --name lnst_controller lnst_controller

Now, you can run the controller:

podman run -e RECIPE=SimpleNetworkRecipe -e RECIPE_PARAMS="perf_iterations=1;perf_duration=10" -e DEBUG=1 -v /path/to/pool:/root/.lnst/pool:ro --rm --name lnst_controller lnst_controller

Note

Podman containers are by default NATed, so you may need to use some other –network mode to make agent machines reachable from controller container. If agent machines are reachable from your host machine, –network=host should do the job. Read Podman’s documentation first.

Or you can run more complex recipes:

podman run -e RECIPE=XDPDropRecipe -e RECIPE_PARAMS="perf_iterations=1;perf_tool_cpu=[0,1];multi_dev_interrupt_config={'host1':{'eth0':{'cpus':[0],'policy':'round-robin'}}}" -v /path/to/pool:/root/.lnst/pool:ro --rm --name lnst_controller lnst_controller

Classes documentation

class container_files.controller.container_runner.ContainerRunner

This class is responsible for running the LNST controller in a container.

Environment variables:

DEBUG: Set to 1 to enable debug mode
RECIPE: Name of the recipe class to run
RECIPE_PARAMS: Parameters to pass to the recipe class
MULTIMATCH: Set to 1 to enable multimatch mode

Agents in containers-specific environment variables:

PODMAN_URI: URI of the Podman socket
IMAGE_NAME: Name of the container image

run() → ResultType

Execute all tests from test_db sequentially.

Each test is independent – a failure in one test does not prevent subsequent tests from running. A summary is printed at the end.