-
have you looked at github actions and their matrix builds? You can self host runners connected to real hardware and its all be in github.
Not sure if this is what your asking but it looks cool! :shrug:
https://docs.github.com/en/actions/using-jobs/using-a-build-matrix-for-your-jobs
https://futurestud.io/tutorials/github-actions-create-a-testing-matrix -
Thanks! The 'self-hosted runner' thing in particular looks very helpful: https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners
I didn't realise they had some code for this that 'just works' - I was expecting there'd be a lot of messing about with HTTP requests to get it working.
The Typescript PR for Bangle.js actually starts using GitHub actions (instead of Travis) for builds so soon I'll get to grips with that properly as well (since I'm using Travis for the moment, but their 'free for Open Source' plan changed to be limited).
Having the test results integrated in GitHub would be cool, but for me, the real benefit is having a way not just to say 'is it broken' - (which hopefully is rare :) ) but to look back at a graph of benchmark results and see where something got slower, and to then click through and find the commit that caused it.
Another case is firmware size - it's always a pain to keep the build going for the micro:bit, so ideally I'd be able to look at firmware size over time, see when a commit made it a lot worse, and then figure out if maybe that feature is something that didn't actually have to be in really stripped down builds.
So really it's just finding a nice way to look at the data as well...
-
maybe something like this?
https://github.com/otto-de/gitactionboard -
also discussed in forum here
Chaz-
Post a reply
About
Hardware test harness
Posted by
@Gordon
Hi! I'd mentioned on Patreon a while back that when I hit $500/month I'd start hosting something that tested real hardware for every commit to the main Espruino project.
We hit that goal in November, but with everything going on with the Bangle.js 2 KickStarter I didn't manage to get time to work on in properly, but hopefully in the next month or so I'll have time to get something live.
Several years ago I did have a hardware testing system, and it just wrote the results of all the tests into a directory for the commit, then just scanned that to come up with a CSV file which could be viewed with a simple HTML graph webpage showing things like test fails, benchmark results.
However that wasn't ideal (it was very hard to actually analyse that data to eg. find out which commit made the firmware larger or slower). So it brings up some questions....
Any thoughts? Ideally I don't want to have to host a bunch of databases, but there must be something that works statically (we're only talking of 1000s of datapoints, not millions).
thanks!