Moar space for code

Posted on
  • I finally spent some time to grok E.memoryArea so I can save code in flash and execute code from flash. This achieves two goals: have more space for code and reduce upload time when testing 'cause modules can remain loaded.

    The way it works is that on esp8266 modules with at least 1MB of flash there are 9 flash pages of 4KB free (actually only 5 on modules with exactly 1MB flash). I wrote a little JS module that stores one JavaScript module per flash page indexed by name. So, as long as the minified size of the module is <4KB (minus the length of the module name and a few extra bytes) you can store the module code in one of the flash pages. Then another smaller module loads all the modules that are stored into the Modules cache so when you require them in the main app they're already there and are executed from flash.

    The savings are mitigated a little bit by the fact that all the functions that are defined in the module still need to be represented in memory. Here's an example:

    // encode a string for mqtt by prefixing the length (16-bits)
    function mqttStr(str) {
      return sfcc(str.length >> 8, str.length&255) + str;
    };
    

    This turns into (after minimizing):

                      #380[r1,l2] Name String [2 blocks] "mqttStr"                    #382[r1,l1] Function { return
                          #383[r1,l2] Name Param "a"                         undefined
                          #385[r1,l2] Name String [1 blocks] "\xFFcod"                        #384[r1,l1] NativeString [1 blocks] "sfcc(a.length>>8,a.length&255)+a;"
                          #386[r1,l2] Name String [1 blocks] "\xFFsco"                        #270[r14,l2] ...
    

    In this case the function header uses 2 JSvars, the parameter 1, the pointer to the code 1, and one more for I forget what. The function body is 33 characters long, which would have taken 4 JSvars, I believe. So we end up needing 5 instead of 9 JSvars. Functions that have longer bodies have more savings...

    To make all this usable I modified the espruino-cli to accept a -u module argument, which minifies the module and uploads it to flash. So to prep an esp8266 for an app that uses 3 modules I'll run something like:

    espruino-cli -m -s espruino:23 -u MQTT
    espruino-cli -m -s espruino:23 -u Nextion
    espruino-cli -m -s espruino:23 -u MCP23008_tve
    

    Under the covers this does the same minification as a require(moduleName) would do but instead of running Modules.addCached(moduleName, moduleText) it runs require("FlashString").save(moduleName, moduleText) to save the module into flash.

    To run my app, I modify the requires so espruino-cli doesn't think it needs to pull-in and upload the modules. A tiny bit of trickery suffices: require("module") becomes r=require; r("module"). All this needs to be prefixed by a very small module to actually load everything from flash. That's done by adding require("FlashLoader")(). So what happens is that the FlashLoader module gets uploaded with the app, the FlashLoader restores all the modules from flash into the Modules cache, and then require finds them there. Phew!

    The storage of the modules in flash is extremely simple. Each page starts with a 4-byte header that contains the length of the module name, the length of the module text, and a magic number. This is followed by the module name and the module text. That's it.

    If you want to play with any of this, the FlashString module is at https://github.com/tve/EspruinoModules/b­lob/master/FlashString.js (it knows nothing about Javascript modules, as far as its concerned it just stores named strings in flash). The FlashLoader module is at https://github.com/tve/EspruinoModules/b­lob/master/FlashLoader.js and my version of the espruino-cli is at https://github.com/tve/EspruinoTools. Note that the cli changes are not strictly required, it's fairly easy to copy and paste stuff to upload a module manually.

  • To load modules automatically at boot or reset, use the following:

    // Copyright (c) 2015 Thorsten von Eicken. MIT License
    E.setBootCode(
      "(function(){" +
        "var FL=require('Flash');var FR=FL.getFree()[2];var FB=FR.addr;var FS=4096;" +
        "var FN=FR.length/FS;var FO=1075838976;" +
        "for(var c=0;c<FN;c++){" +
          "var b=FB+c*FS;var a=FL.read(4,b);var d=a[0]<<2;var e=b+4+d;var f=a[1]<<4;" +
          "if(a[0]==0||a[1]==0||a[2]!=165||a[3]!=1­95){" +
            "console.log('  nothing at',b.toString(16));continue;}" +
          "var g=E.toString(FL.read(d,b+4)).trim();" +
          "console.log('  memoryArea',e.toString(16),f,g)," +
          "Modules.addCached(g,E.memoryArea(FO+e,f­));" +
        "}"+
      "})();",
      true);
    

    it uses no JSvars itself once executed. Caveat: currently boot code doesn't work on the esp8266 due to https://github.com/espruino/Espruino/iss­ues/891

  • ...like the 'Moar' ...

  • Very interesting (and nice work!).

    IS there anything similar, at all, on the other Espruino platforms? Specifically, I'm interested in the HY-STM32F1 board, where I've been working on a GUI. If I could flash the GUI module code to flash memory, and execute it from there, it would free up a TON of RAM for actual application code.

  • The same thing should work with E.setBootCode - the issue is that on many of the STM32 boards there is very little free flash memory available, so being able to put code in there really doesn't help you much.

    Note: There's an option in the Web IDE to do setBootCode automatically - the only issue is it won't work for modules (hence @tve's stuff above). If you just stick all your code into the main editor window it'll work though

  • I think the other issue with STM32 platforms is that most of flash has very large pages IIRC and thus my simple scheme where I store one module in one string in flash isn't viable.

  • ...not much more effort is required to use it as a library/libraries: an array of module names with pointer to the entry point for each large page will do it...

  • Hmm, yeah, at some point it might be worth implementing saving code in some kind of flash filesystem... But with STM32 it is a pain... The F4 has huge pages (mostly) and all the Espruino boards tend to use up pretty much all the available flash

  • Just to update this, there's now:

    • The Storage Module built into Espruino itself which allows you to save simple 'files' - and load them in a memoryArea-style way that doesn't use up RAM.
    • Save to flash and modules as functions in the Web IDE, which automatically save your code direct to flash

    These work on all platforms.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Moar space for code

Posted by Avatar for tve @tve

Actions