Features

Features to support testing

• Allows to test by hand

  Just connect one or more tentacles and run the tests from the micropython repo from the command line against the board on the tentacle.

  For example tests/run-perfbench.py –pyboard –device=/dev/ttyUSB2 or tests/run-tests.py -t /dev/xxx -d extmod_hardware.

• Allows to test ad hoc

  Just connect one tentacle to your computer and run tests on that tentacle.

  For example mptest test.

• Allows to test regressions

  Set up a github runner and connect many tentacles to do automated regessions tests.

• Dynamic number of tentacles

  As soon as tentacles are connected via USB, they will be used for the tests.

  Every tentacle is identified by its serial number. A inventory lookup will provide the required information about that tentacle.

• Detect flakiness –count

  Specifying –count=3 will run every test 3 times. In the Summary Report, the 3 results will be presented aside (for example: pass fail pass).

• Specify reference tentacle using –reference=ESP32_C3_DEVKIT

  WLAN and BLE tests run always against a reference tentacle. The hardcoded default is RPI_PICO_W. However this may be overriden using –reference=ESP32_C3_DEVKIT. Watch out to select a tentacle which supports WLAN and BLE!

• Tab completion

  mptest test –only-board / –only-test provide tab completion.

• Timing report

  To save time, everything should run in parallel: Building and testing. task_report.md/.txt/.html summarizes where the time was spent.

• Labels mptest labels

  The labels in front of every tentacle may be created automatically.

• Current state mptest list

  This command shows the connected tentacles and the implemented tests.

Technical features

• Verify flashed firmware

  After flashing a firmware, the strings sys.version / sys.implementation are compared against the build result. This allows to detect if the wrong firmware was flashed to the wrong tentacle. Nothing is more tedious than testing against the wrong target!

• One tentacle covering multipe boards

  For example the RPI_PICO2_W supports FUT_WLAN and is therefore a superset of RPI_PICO2. So it would be beneficial to have ONE tentacle to test both. I decided not to support this as it would increase overall complexity. The workaround is to solder TWO tentacles which is not very expensive/time consuming and speeds up the tests!

• Journalctl

  Some usb errors might pop up in dmesg or Journalctl. For example limited USB bandwidth. testbed_micropython starts journalctl in the background and terminates loudly when a critical message is detected.

• multiprocessing

  The python multiprocessing package is used to allow parallelism. The timeout feature of that package that package is used to recover from blocking which might be caused by a blocking test or mpremote.

• DAQ connector

  Every tentacle provides a 12 pin connector were a DAQ may be connected to spy on the signals.

• Bring boards in to boot mode (programming mode): mptest debugbootmode

  Figuring out how to put a board into boot mode is tedious work. mptest debugbootmode allows to run just that code.