Flightlevel: pytest

Note

This section explains:

  • How to use pytest parameters to select the firmware to be tested.

  • How to run the different test suites.

  • How pytest selects tentacles and FUT (Feature Unter Test)

  • How pytest does testcollection.

Required knowhow:

Be warned that pytest is quite a complex beast. If you are not familiar with pytest, take time to read a pytest showcase about test collection and fixtures. This time is a good investment as pytest is extremly powerful and widely used.

Note

All tests are located in the folder <repo>/tests !

pytest command line arguments

Note

pytest already provides many command line arguments. This section is about the octoprobe specific command line arguments.

The arguments are implmented here:

tests.conftest.pytest_addoption()[source]

This function name is reserved by pytest. See https://docs.pytest.org/en/7.1.x/reference/reference.html#initialization-hooks.

It will be called to determine the program arguments.

When calling pytest --help, below arguments will be listed!

$ pytest --help
...
--firmware=FIRMWARE   The url to a git repo to be cloned and compiled, a path to a source directory. Or a json file with a download location. Syntax:
                      https://github.com/micropython/micropython.git@master.

Arguments –firmware

1{
2    "board_variant": "RPI_PICO",
3    "url": "https://micropython.org/resources/firmware/RPI_PICO-20240602-v1.23.0.uf2",
4    "micropython_full_version_text": "3.4.0; MicroPython v1.23.0 on 2024-06-02;Raspberry Pi Pico with RP2040"
5}

$ pytest –firmware=pytest_args_firmware_RPI_PICO_v1.23.0.json will

  • download the firmware (see url on line 3)

  • Install this firmware an matching tentacles (see board_variant on line 2)

  • Verify the installed version (see micropython_version_text on line 4)

Arguments –firmware-build-url

$ pytest –firmware-build-url=https://github.com/dpgeorge/micropython.git@rp2-add-rp2350 will

  • git clone https://github.com/dpgeorge/micropython.git

  • git checkout rp2-add-rp2350

  • Query the installed MCU tentacles for tags like boards=RPI_PICO2:RPI_PICO2-RISCV. From this the supported firmware / variants are collected.

  • For every supported firmware / variant:

    • build the firmware using mpbuild

    • install the firmware

    • run the tests

pytest test collection

Pytest automatically collects tests.

1 @pytest.mark.required_futs(EnumFut.FUT_I2C)
2 def test_i2c(
3     mcu: Tentacle,
4     device_potpourry: Tentacle,
5     daq_saleae: Tentacle,
6 ) -> None:
7   ...

Line 2: As test_i2c(..) starts with test_, it will be collected by pytest. Line 3-5: This test requires 3 tentacles. The parameter names are reserved and match the tentacle type. Line 1: This test will be testing FUT_I2C.

During testcollection pytest will:

  • Collect all tests starting with test_

  • Above text_i2c() will not be collected if

    • The connected testbed does not contain the required tentacle types.

  • Above text_i2c() will be collected for

    • all combinations of tentacles matching the required tentacle types.

    • all firmware version supported by these mcu

pytest –collect-only -q Will show the collected tests without executing them

 1$ pytest \
 2   --collect-only -q \
 3   --firmware-build-url=https://github.com/dpgeorge/micropython.git@rp2-add-rp2350 \
 4   tests/test_simple.py::test_i2c
 5 tests/test_simple.py::test_i2c[4429pyboard(PYBV11)-3f31potpourry-1331daq]
 6 tests/test_simple.py::test_i2c[4429pyboard(PYBV11-DP)-3f31potpourry-1331daq]
 7 tests/test_simple.py::test_i2c[4429pyboard(PYBV11-DP_THREAD)-3f31potpourry-1331daq]
 8 tests/test_simple.py::test_i2c[4429pyboard(PYBV11-THREAD)-3f31potpourry-1331daq]
 9 tests/test_simple.py::test_i2c[1831pico2(RPI_PICO2)-3f31potpourry-1331daq]
10 tests/test_simple.py::test_i2c[1831pico2(RPI_PICO2-RISCV)-3f31potpourry-1331daq]
11 ^^^^^^^^^^^^^^^^^^^^                                Filename of the test
12                       ^^^^^^^^                      Testfunction
13                                ^^^^^^^^^^^^^^^^^^^^ '-' separated list of tentacles
14                                                     assigned to the test

Above output lists all collected tests.

Line 3: 4429pyboard(PYBV11-DP): This is tentacle number 4429 which is a pyboard. The firmware under test is PYBV11-DP which stands for pyboard and double precision float. Line 7: 1831pico2(RPI_PICO2-RISCV): This is tentacle number 1831 which is a pico2. The firmware under test is RPI_PICO2-RISCV which is the riscv version.

The testcollection is implemented here:

tests.conftest.pytest_generate_tests()[source]

This is a pytest hook https://docs.pytest.org/en/7.1.x/reference/reference.html?highlight=pytest_generate_tests#std-hook-pytest_generate_tests

Give a test function like ‘test_i2c()’ in ‘metafunc’, this function will create test calls for possible combinations of tentacles and firmware versions.

Calls metafunc.parametrize which defines the tests that have been be collected.

Parameters:

metafunc (pytest.Metafunc) – See https://docs.pytest.org/en/7.1.x/reference/reference.html#metafunc

conftest.py

This file configures pytest and contains many important hooks.

This function will setup and tear down octoprobe

tests.conftest.ctxtestrun()[source]

Setup and teardown octoprobe and all connected tentacles.

Now we loop over all tests an return for every test a CtxTestRun structure. Using this structure, the test find there tentacles, git-repos etc.

This function will setup and tear down the tentacles for every test

tests.conftest.setup_tentacles()[source]

Runs setup and teardown for every single test:

  • Setup

    • powercycle the tentacles

    • Turns on the ‘active’ LED on the tentacles involved

    • Flash firmware

    • Set the relays according to @pytest.mark.required_futs(EnumFut.FUT_I2C).

  • yields to the test function

  • Teardown

    • Resets the relays.

Parameters:

testrun (CtxTestRun) – The structure created by testrun()

testbeds

There are currently 3 testbeds:

  • pytest: See tests/test_simple.py. These are tests purely written in pytest and serve as a best practice template.

  • pytest and mip: See tests/test_mip.py. This is a starting point for testing a mip or other mictopython libraries using octoprobe.

  • testbed_mictopython. Here the MicroPython interpreter is tested on the different mcus. This is implemented in https://github.com/octoprobe/testbed_micropython.