Design: TestRunSpec

Overview

A TestRunSpec is a static, immutable description of one test group. It answers the question “which command should be run, on which class of tentacle, and how?”.

Each spec is a singleton constant defined in the testrunspecs/ package:

TESTRUNSPEC_RUNTESTS_STANDARD = TestRunSpec(
    label="RUN-TESTS_STANDARD",
    label_intuitive="run-tests.py",
    command=["run-tests.py"],
    required_fut=EnumFut.FUT_MCU_ONLY,
    requires_reference_tentacle=False,
    testrun_class=TestRunRunTests,
    timeout_s=3660.0,
    ...
)

All singletons are registered in _TESTRUN_SPECS inside util_testrunner.py and selected at startup by get_testrun_specs().

During a test run the TestRunSpec also becomes mutable in one regard: its tsvs_todo list shrinks as individual TestRun instances are completed.

Key fields

Field	Type	Purpose
label	str	Unique identifier, e.g. RUN-TESTS_EXTMOD_HARDWARE. Used by --only-test / --skip-test on the command line.
label_intuitive	str	Human-readable shorthand shown in reports, e.g. run-tests.py --via-mpy.
command	list[str]	The executable and its fixed arguments, e.g. ["run-tests.py", "--test-dirs=extmod_hardware"].
required_fut	EnumFut	The Feature Under Test that a tentacle must declare in its inventory to be eligible, e.g. FUT_MCU_ONLY, FUT_WLAN_STA.
requires_reference_tentacle	bool	When True a second tentacle is needed (e.g. WLAN-AP), and the tentacle_reference from Args.reference_board is passed to every TestRun.
testrun_class	type[TestRun]	The concrete subclass instantiated when a tentacle is ready.
tsvs_todo	TentacleSpecVariants	The remaining board-variants still to be tested. Populated by assign_tentacles() and shrinks as tests complete.
timeout_s	float	Hard wall-clock limit for one TestRun of this spec.
priority	int	Scheduling hint (higher = run earlier).

Relationship to tentacle types

The chain from hardware inventory down to an individual test execution goes through four distinct classes.

        classDiagram
    direction TB

    class TentacleSpecMicropython {
        board: str
        futs: set[EnumFut]
        build_variants: list[str]
    }
    note for TentacleSpecMicropython "Static per-board description\nlived in tentacles_inventory.py"

    class TentacleSpecVariant {
        tentacle: TentacleMicropython
        variant: str
        role: TestRole
    }
    note for TentacleSpecVariant "One (board, variant, role) tuple\ncreated at assign_tentacles() time"

    class TentacleSpecVariants {
        list[TentacleSpecVariant]
    }
    note for TentacleSpecVariants "The todo-list of a TestRunSpec"

    class TestRunSpec {
        label: str
        required_fut: EnumFut
        tsvs_todo: TentacleSpecVariants
        generate() -> TestRun
    }

    class TestRun {
        testrun_spec: TestRunSpec
        tentacle_variant: TentacleSpecVariant
        tentacle_reference: TentacleMicropython | None
    }
    note for TestRun "One concrete execution\n(one board-variant on one tentacle)"

    TentacleSpecMicropython "1" --> "0..*" TentacleSpecVariant : expanded into
    TentacleSpecVariant "0..*" --* TentacleSpecVariants : contained in
    TentacleSpecVariants "1" --* TestRunSpec : tsvs_todo
    TestRunSpec "1" --> "0..*" TestRun : generate()
    TentacleSpecVariant "1" --* TestRun : tentacle_variant
    

TentacleSpecMicropython — the static board description

Defined once per physical board type in the tentacle inventory. It holds the board name, the set of supported FUTs, and the build_variants tag which lists the firmware variants the board can run (e.g. ["", "RISCV"] for the RP2350).

TentacleSpecVariant — one (tentacle, variant, role) tuple

Created at runtime by tentacle_spec_2_tsvs() when assign_tentacles() expands each tentacle according to its build_variants. For a RP2350 this yields two entries: RPI_PICO2("") and RPI_PICO2("RISCV").

When a test requires two tentacles (e.g. WLAN STA ↔ AP), two TentacleSpecVariant objects are created for the same tentacle, one with role=ROLE_INSTANCE0 and one with role=ROLE_INSTANCE1.

TentacleSpecVariants — the todo-list

A typed list[TentacleSpecVariant]. It is stored as TestRunSpec.tsvs_todo and acts as the scheduler’s work queue for that spec. Every time a TestRun finishes, mark_as_done() removes the corresponding entry. When the list is empty, this spec is fully done.

TestRun — one concrete execution

Produced by TestRunSpec.generate() when a tentacle is available and its firmware has been built. It carries:

• testrun_spec — back-reference to the owning spec

• tentacle_variant — which physical board and variant to test

• tentacle_reference — optional second tentacle for multi-device tests

TestRun.test() (overridden per subclass) contains the actual subprocess call to e.g. run-tests.py.

Lifecycle of tsvs_todo

assign_tentacles()          # fills tsvs_todo from ConnectedTentacles
      |
      v
generate()                  # yields TestRun for each ready (tentacle, variant)
      |
      v
TestRun executes
      |
      v
mark_as_done()              # removes the TentacleSpecVariant from tsvs_todo
      |
      v
tsvs_todo empty → spec done

assign_tentacles() is called once after get_testrun_specs() resolves the active set. generate() is called repeatedly by the TestBartender scheduler – it only yields a TestRun if the required firmware is already built and the tentacle is not busy.