You are reading a single comment by @Gordon and its replies. Click here to read the full conversation.
  • However it is not like that and never was(?)

    Correct - it can't easily because as you noted the string might be escaped (although I guess we could have checked for that when parsing). Either way we'd have to parse over it at load time, and then again when we wanted to use it - so while it saves RAM it might not be a clear win for performance.

    Then in future case of WebIDE replacing atob() with big binary strings which would be stored directly inside the file in flash, those strings would also be in flash

    Correct, yes - and that is now the case if you turn on pretokenisation in the IDE. I'm going to have to update the advice in https://www.espruino.com/Code+Style#other-suggestions !

    Update

    I've just pushed an updated Web IDE, BangleApps (in the dev app loader) and a new firmware (the raw strings exposed an interpreter issue that's been around for a while) and the results are really good.

    Ensure pretokenisation is turned on in the dev app loader, that you have the latest firmware, and click Reinstall apps to update the code for every app with the pretokenised strings.

    Quick example with the Anton clock:

    On the same build, I do reset() followed by tm=getTime();load(eval(require("Storage").read("antonclk.app.js")));getTime()-tm which loads the anton clock and measures how long it takes.

    • Not prtokenised 256ms
    • pretokenised(old) 254ms
    • pretokenised with strings 194ms

    Bear in mind that's the whole clock, so it's a big improvement. And it pretty much all comes from loading the font at the start (even though Anton is relatively modest font-wise - other clocks will see bigger benefits).

    I did a quick benchmark of just how long it takes to load the font, with a test file in Storage:

    setFontAnton = function(scale) {
    g.setFontCustom(atob("...."), 78 + (scale << 8) + (1 << 16));
    };
    
    t = getTime();for (var i=0;i<1000;i++) setFontAnton();print(getTime()-t,"ms")
    
    • Not pretokenised: 53ms per call
    • pretokenised with strings: 1.37ms

    So it's around 40x faster! Font rendering will be a little slower, but it's not going to be huge compared to that boost.

    The only downside at the moment is that Espruino will now make all strings in pretokenised code native strings, and I feel like we should probably change it so that if a string is pretokenised but it would fit inside one JsVar (eg it's not going to use any more RAM than the pointer to the string in flash) we should keep it in RAM.

  • I think I notice the improved speed - 40x is impressive!

    Ensure pretokenisation is turned on in the dev app loader, that you have the latest firmware, and click Reinstall apps to update the code for every app with the pretokenised strings.

    I did this. Then I compacted the storage via the settings app and the on screen message is incomplete (see attached image). I reinstalled the settings app without pretokenization and tried compacting again, this time the whole message was displayed.

    I am not sure that it has to do with the new changes, I have not tried with older firmwares and pretokenization. But I thought I'd mention it here anyway.

About

Avatar for Gordon @Gordon started