Test strategy

Posted on
  • I hate writing tests but I'm a big fan of CI :-) What is the test strategy for Espruino? I see the tests directory and they seems to be all JS tests that are independent of hardware? It looks like the travis job only does builds but not tests? Is there a reason the travis builds are not uploaded to S3? Would setting something like this up be of interest? (I use this all the time at work and it can have an amazing effect on productivity.)

    What about tests on the embedded systems themselves? I think the following would be awesome, but some work:

    • a small test PCB for each board that someone cares about that has a small number of standard components, such as a i2c device, an spi device, a 1-wire temp sensor, and maybe a loop-back from a PWM pin through a low-pass RC filter to an ADC input
    • a test suite that exercises these components in a basic manner
    • a test runner on an RPi or ODROID that takes git web hooks and can run the test after travis has built the binary
      I'd be willing to do work on this for the esp8266 and the emw3165 if that ever really gets off the ground. Thoughts?

    NB: in terms of peripherals I was thinking really simple. For example, a small digital I/O expander could be used with some pins looped back so the test suite can output a 1 on one pin and read it back on another, and then output a 0 and read it back and declare victory.

  • Yes, all those tests are hardware independent I'm afraid. All the hardware stuff is hand-tested, which is less than ideal.

    Travis builds are uploaded to http://www.espruino.com/binaries/travis/­?C=M;O=D but as they're not padded for the bootloader the ones for the Original Espruino and Pico aren't that useful.

    ... but also, I have builds running here (which were set up before Travis), which run code on an Espruino board itself (I have one set up that puts it into bootloader mode, and then it's flashed with new firmware). Currently it's only a series of benchmarks, which actually aren't that reliable because the 32k crystal is way off when Espruino starts :) I need to fix it and measure timings using the second board.

    Builds and stats: http://www.espruino.com/binaries/git/

    But yeah, some actual hardware tests could be good - Maybe an I2C IO expander wired to an SPI IO expander? I'm not sure how much use they'd be though... The other option is to just wire one board straight to another, and to run things like SPI and I2C slow enough that 'setWatch' can be used for testing.

    Problem is, the regressions I've hit have all been really odd things that the tests wouldn't have picked up - for instance SPI on strange pin combinations. Things like GPIO or SPI on basic pins have never broken as far as I'm aware.

    Another thing - perhaps more likely to be scalable - is to have something set up whereby anyone can set up a test with a Pi + board, and can then contribute back the test results for each build? Then if someone makes a module to control some hardware, they can add a test specifically for that hardware.

  • Mhh, interesting that you haven't had regressions with the basic I/O stuff. Makes sense in that they're fairly small pieces of code and don't tend to get mucked with accidentally.

    I have to look where the binaries get uploaded. I'd really like to get the esp8266 to have automatic builds (I know, it's gonna take some work to package the toolchain).

  • Yes, it's all quite separate too. If you're doing work on SPI you tend to test that work before a commit - so the only stuff that gets broken accidentally is really when there are complex interactions between bits of hardware (timers/etc).

    ... of course for STM32 it's trying to support F1, F3 and F4 with the same source, so you could conceivably alter it for one processor and break another - although not many people tend to fiddle with jshardware.c so it's not such an issue :)

    It'd be great if you could get it running as a Travis build - and then you could just upload the result of the build. It's scary how much processing power gets wasted doing the setup for a Travis build as-is, but it seems to be ok :)

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Test strategy

Posted by Avatar for tve @tve

Actions