Moar space for code

I finally spent some time to grok E.memoryArea so I can save code in flash and execute code from flash. This achieves two goals: have more space for code and reduce upload time when testing 'cause modules can remain loaded.

The way it works is that on esp8266 modules with at least 1MB of flash there are 9 flash pages of 4KB free (actually only 5 on modules with exactly 1MB flash). I wrote a little JS module that stores one JavaScript module per flash page indexed by name. So, as long as the minified size of the module is <4KB (minus the length of the module name and a few extra bytes) you can store the module code in one of the flash pages. Then another smaller module loads all the modules that are stored into the Modules cache so when you require them in the main app they're already there and are executed from flash.

The savings are mitigated a little bit by the fact that all the functions that are defined in the module still need to be represented in memory. Here's an example:

// encode a string for mqtt by prefixing the length (16-bits)
function mqttStr(str) {
  return sfcc(str.length >> 8, str.length&255) + str;
};

This turns into (after minimizing):

                  #380[r1,l2] Name String [2 blocks] "mqttStr"                    #382[r1,l1] Function { return
                      #383[r1,l2] Name Param "a"                         undefined
                      #385[r1,l2] Name String [1 blocks] "\xFFcod"                        #384[r1,l1] NativeString [1 blocks] "sfcc(a.length>>8,a.length&255)+a;"
                      #386[r1,l2] Name String [1 blocks] "\xFFsco"                        #270[r14,l2] ...

In this case the function header uses 2 JSvars, the parameter 1, the pointer to the code 1, and one more for I forget what. The function body is 33 characters long, which would have taken 4 JSvars, I believe. So we end up needing 5 instead of 9 JSvars. Functions that have longer bodies have more savings...

To make all this usable I modified the espruino-cli to accept a -u module argument, which minifies the module and uploads it to flash. So to prep an esp8266 for an app that uses 3 modules I'll run something like:

espruino-cli -m -s espruino:23 -u MQTT
espruino-cli -m -s espruino:23 -u Nextion
espruino-cli -m -s espruino:23 -u MCP23008_tve

Under the covers this does the same minification as a require(moduleName) would do but instead of running Modules.addCached(moduleName, moduleText) it runs require("FlashString").save(moduleName, moduleText) to save the module into flash.

To run my app, I modify the requires so espruino-cli doesn't think it needs to pull-in and upload the modules. A tiny bit of trickery suffices: require("module") becomes r=require; r("module"). All this needs to be prefixed by a very small module to actually load everything from flash. That's done by adding require("FlashLoader")(). So what happens is that the FlashLoader module gets uploaded with the app, the FlashLoader restores all the modules from flash into the Modules cache, and then require finds them there. Phew!

The storage of the modules in flash is extremely simple. Each page starts with a 4-byte header that contains the length of the module name, the length of the module text, and a magic number. This is followed by the module name and the module text. That's it.

If you want to play with any of this, the FlashString module is at https://github.com/tve/EspruinoModules/b­lob/master/FlashString.js (it knows nothing about Javascript modules, as far as its concerned it just stores named strings in flash). The FlashLoader module is at https://github.com/tve/EspruinoModules/b­lob/master/FlashLoader.js and my version of the espruino-cli is at https://github.com/tve/EspruinoTools. Note that the cli changes are not strictly required, it's fairly easy to copy and paste stuff to upload a module manually.