Flightlevel: pytest

Note

This section explains:

  • How to use pytest parameters to select the firmware to be tested.

  • How to run the different test suites.

  • How pytest selects tentacles and FUT (Feature Unter Test)

  • How pytest does testcollection.

Required knowhow:

Be warned that pytest is quite a complex beast. If you are not familiar with pytest, take time to read a pytest showcase about test collection and fixtures. This time is a good investment as pytest is extremly powerful and widely used.

Note

All tests are located in the folder <repo>/tests !

pytest command line arguments

Note

pytest already provides many command line arguments. This section is about the octoprobe specific command line arguments.

The arguments are implmented here:

$ pytest --help
...
--firmware=FIRMWARE   The url to a git repo to be cloned and compiled, a path to a source directory. Or a json file with a download location. Syntax:
                      https://github.com/micropython/micropython.git@master.

Arguments –firmware

1{
2    "board_variant": "RPI_PICO",
3    "url": "https://micropython.org/resources/firmware/RPI_PICO-20240602-v1.23.0.uf2",
4    "micropython_full_version_text": "3.4.0; MicroPython v1.23.0 on 2024-06-02;Raspberry Pi Pico with RP2040"
5}

$ pytest –firmware=pytest_args_firmware_RPI_PICO_v1.23.0.json will

  • download the firmware (see url on line 3)

  • Install this firmware an matching tentacles (see board_variant on line 2)

  • Verify the installed version (see micropython_version_text on line 4)

Arguments –firmware-build-url

$ pytest –firmware-build-url=https://github.com/dpgeorge/micropython.git@rp2-add-rp2350 will

  • git clone https://github.com/dpgeorge/micropython.git

  • git checkout rp2-add-rp2350

  • Query the installed MCU tentacles for tags like boards=RPI_PICO2:RPI_PICO2-RISCV. From this the supported firmware / variants are collected.

  • For every supported firmware / variant:

    • build the firmware using mpbuild

    • install the firmware

    • run the tests

pytest test collection

Pytest automatically collects tests.

1 @pytest.mark.required_futs(EnumFut.FUT_I2C)
2 def test_i2c(
3     mcu: Tentacle,
4     device_potpourry: Tentacle,
5     daq_saleae: Tentacle,
6 ) -> None:
7   ...

Line 2: As test_i2c(..) starts with test_, it will be collected by pytest. Line 3-5: This test requires 3 tentacles. The parameter names are reserved and match the tentacle type. Line 1: This test will be testing FUT_I2C.

During testcollection pytest will:

  • Collect all tests starting with test_

  • Above text_i2c() will not be collected if

    • The connected testbed does not contain the required tentacle types.

  • Above text_i2c() will be collected for

    • all combinations of tentacles matching the required tentacle types.

    • all firmware version supported by these mcu

pytest –collect-only -q Will show the collected tests without executing them

 1$ pytest \
 2   --collect-only -q \
 3   --firmware-build-url=https://github.com/dpgeorge/micropython.git@rp2-add-rp2350 \
 4   tests/test_simple.py::test_i2c
 5 tests/test_simple.py::test_i2c[4429pyboard(PYBV11)-3f31potpourry-1331daq]
 6 tests/test_simple.py::test_i2c[4429pyboard(PYBV11-DP)-3f31potpourry-1331daq]
 7 tests/test_simple.py::test_i2c[4429pyboard(PYBV11-DP_THREAD)-3f31potpourry-1331daq]
 8 tests/test_simple.py::test_i2c[4429pyboard(PYBV11-THREAD)-3f31potpourry-1331daq]
 9 tests/test_simple.py::test_i2c[1831pico2(RPI_PICO2)-3f31potpourry-1331daq]
10 tests/test_simple.py::test_i2c[1831pico2(RPI_PICO2-RISCV)-3f31potpourry-1331daq]
11 ^^^^^^^^^^^^^^^^^^^^                                Filename of the test
12                       ^^^^^^^^                      Testfunction
13                                ^^^^^^^^^^^^^^^^^^^^ '-' separated list of tentacles
14                                                     assigned to the test

Above output lists all collected tests.

Line 3: 4429pyboard(PYBV11-DP): This is tentacle number 4429 which is a pyboard. The firmware under test is PYBV11-DP which stands for pyboard and double precision float. Line 7: 1831pico2(RPI_PICO2-RISCV): This is tentacle number 1831 which is a pico2. The firmware under test is RPI_PICO2-RISCV which is the riscv version.

The testcollection is implemented here:

conftest.py

This file configures pytest and contains many important hooks.

This function will setup and tear down octoprobe

This function will setup and tear down the tentacles for every test

testbeds

There are currently 3 testbeds:

  • pytest: See tests/test_simple.py. These are tests purely written in pytest and serve as a best practice template.

  • pytest and mip: See tests/test_mip.py. This is a starting point for testing a mip or other mictopython libraries using octoprobe.

  • testbed_mictopython. Here the MicroPython interpreter is tested on the different mcus. This is implemented in https://github.com/octoprobe/testbed_micropython.