Concept: Run smoke tests / unit tests in custom built Espruino IDE docker images with GitHub actions

Hi everyone! I was thinking about the following possible workflow in order to automatically run unit tests or smoke tests in an emulator:

• build a docker image that contains the Espruino IDE (https://espruino.com/ide) or a smaller version of it
  
• when a commit in a PR gets pushed, this triggers a GitHub action
  
• optionally the smoke tests could also be run just once a day, e.g. at midnight, to save GitHub Actions minutes
  
• this GitHub action uses the Espruino IDE docker image and automatically loads the newly added/changed code in the IDE
  
• some test cases that check basic functionality of the app run and confirm the changes don't break other code
  
  

When it comes to test cases, obviously some mocking needs to be done, however I think this is perfectly doable. For example https://www.espruino.com/ReferenceBANGLE­JS2#l_E_getTemperature always returns NaN in the emulator, but the easiest way to mock it for the test suite would be to call E.getTemperature = function() { return 20; }; at the start of the test suite.

For example, here's some pseudo code I came up with for activityreminder:

// Suite setup
// Mock some functions
E.getTemperature = function() { return 27; }; // assume the user is always wearing the smart watch
settings.maxInnactivityMin = 0; // immediately trigger an inactivity warning by setting max inactivity to 0 min

// Test cases
// wait a few seconds and assert E.showPrompt() got called (which should happen)
// if not, fail the test or raise a warning
// (optionally) take a screenshot and include it as an artifact

Why would this be useful?
Around 2 weeks ago there was an issue where (if I understood correctly) some firmware changes caused a new date format which broke activityreminder because it still expected/used the old date format. We could detect sudden breaking changes like this one using test cases. For example, the test case above would have failed after the new date format changes, because it couldn't have read the new date format.

Obviously this is a huge suggestion and is probably low priority, but I'm just sharing my ideas and I'm open for feedback! What are your thoughts about this?

There is already something available - it doesn't use the IDE but does use the emulator and run it within Node.js on the command-line - it doesn't need Docker. It's at https://github.com/espruino/BangleApps/b­lob/master/bin/runapptests.js

It'll scan through all apps for test files of a certain format, load the app up and try and run through a test script - which eventually could involve comparing screenshots at different points (it's just a matter of including the code from https://github.com/espruino/BangleApps/b­lob/master/bin/thumbnailer.js).

The thing is though, there was some talk about this and I made the rough framework but nobody seems particularly bothered about actually creating any tests. For something to catch the activityreminder issue the level of tests required across all apps would be pretty high.

One thing that might be a big improvement for reliability is actually to catch any Errors created on the watch itself (or even within BangleApps) and bring them into some kind of database - but I've been resisting that as I wonder if there might be privacy implications if somehow personal data could included in the errors.

Thanks for the info. If this already exists, cool, no need to build any new images then. And yes, I get the point that automatically catching and uploading errors would definitely be a problem regarding privacy.

I had done this a while ago: https://github.com/espruino/BangleApps/b­lob/master/apps/android/test.js

Every step in that would probably map nicely to a test object in the json. I think I could refactor that.

Maybe some of the asserts would be useful as a teststep in the apptests? I could try to integrate those.

How does the emulator handle "hardware" like GPS? If it does not I probably need to find a new way to track the state of the "internal" GPS instead of checking the PIN mode like it does currently.

I think I could refactor that.

That would be great!

Maybe some of the asserts would be useful as a teststep in the apptests?

Yes, absolutely - I was expecting that as tests got added we might want to have some different steps in there.

How does the emulator handle "hardware" like GPS?

Right now it doesn't, but I guess a good option would be to have a test step like GPS that actually just called Bangle.emit("GPS",{....}) to fake the GPS, so then you know exactly which coordinates are being fed in and can then check that the code is doing the right thing.

As I understand it right now things like GPS power are always off in the emulator, but maybe that could be changed or we just upload some boilerplate before the test that replaces Bangle.setGPSPower() with some JS that does what we want

I've had a play around porting the first of the test cases to json:

https://gist.github.com/halemmerich/3d0f­1d41d8eebe76eedc8553530e5dd5

I think this should be all the asserts/features I need. I have a rough implementation for those but it is not yet working correctly.

Why did you decide on the json based format for test cases? Wouldn't javascript files be better for editing/linting/syntax highlighting?
Maybe a library that provides asserts and an easy way to setup the emulator for running testcases for a single app? Or the other way round, the test code is a module and apptests.js iterates over the exports which are test functions and handles the environment like now.

Thanks!

Well, really it was a few things:

• we might want to have some data beforehand (eg to say we need to load app X as well) - but that could be in JSON in a comment at the start of JS I guess
  
• Ideally we want to be able to write a command to the emulator and wait for it to complete so we know where the test failed if it did (but I guess if we wrote the whole JS file and then ran it, exceptions would give line numbers)
  
• Writing JS to the console isn't quite the same as normal JS - writing for (var i=0;i<10;i++) and then print("Hello") on a newline will run the loop and THEN the print.
  
• If you let people write JS they'll try to be clever and it'll no longer be a set of 'do this check that' but something far more complicated
  
• I was expecting some commands of the JSON might be doing more in the test harness than just running on the device (eg saving and comparing screenshots) - but maybe we could have the emulator send out some text like COMPARE IMAGE FOOBAR.png followed by the image's base64 and then the test harness could interpret that.
  
  

But yes, potentially we could just have JS files and work around that stuff above. There's nothing in place at the moment, so if you wanted to do a PR I'm all for it.

If you let people write JS they'll try to be clever and it'll no longer be a set of 'do this check that' but something far more complicated

I think this speaks most to me personally :D
I will try to keep it as simple as possible and in json. I guess if we need more complexity we could just run/eval a js file as a step from the json test definition.

we could just run/eval a js file as a step from the json test definition.

Yes - that sounds like a great plan for the future