strange behavior with RLE compression of flash new code

Posted on
Page
of 2
/ 2
Next
  • I see some strange behavior with a feature recently introduced: RLE compression

    A complex and big (weight of js file: 19.6 KB) software that has been loaded and executed more times without problem on version 1v79, it returns an error with latest firmware (1v81):

    ERROR: Too big to save to flash (32796 vs 30716 bytes)

    however, after the flash, the script seems to be complete

  • Hi Francesco, Gordon

    Same Problem here with a pico:
    ERROR: Too big to save to flash (49803 vs 49148 bytes)

    Sacha

  • @Francesco which board was this on?

    @Sacha did the code work before?

    Can you share the code that is giving the error? I'll try and look at this next week.

    There is a bit of a balancing act. Adding compression let me increase the amount of available RAM, but then if you have a lot of static data, you can't save it all.

    Probably what is needed is better compression

  • Hi Gordon

    I never tried the code below 1.80.
    Sent you an email with the code.
    A better compression could be a very nice solution.

    Sacha

  • Any thoughts on a compression algorithm that could be used? It would be nice to have something more or less standard that could be exposed for other bits of JS code, but things like gzip seem to have pretty use a lot of RAM/flash so don't seem suitable.

    Best contender right now seems to be https://github.com/atomicobject/heatshri­nk - but that's not even remotely standard.

  • Maybe this is a dumb idea, but it seems to me that being able to store code outside of the JsVars would be helpful. A reference/handle to the allocated memory can still be stored as a JsVar. It immediately saves 25%, if I understand the 12 chars per 16 bytes correctly. But beyond that, it would also allow code to be stored in flash, which would be a big help with larger modules from a library, in particular.

  • Don't we have something like that (storing data without the 25% overhead) with "flat strings"? This sounds really familiar... Maybe it wasn't ever applied to code though... I forget the details.

  • Yes - flat strings are used for Typed Arrays, and store data with just a single 16 byte var at the start as overhead. They take a bit longer to allocate though as Espruino has to search through to find a contiguous block of variables - and you can't guarantee that it will.

    There's an issue open for using them to store function code - but we probably don't want to do it when we're creating functions inside functions as that might be too slow.

    I'd really like to avoid 're-packing' the data that's in RAM if at all possible - it just sounds like a recipe for disaster.

  • Currently the compression of the source code is done on ARM or with the IDE ?

    I think the compression of the source code during the flash operation should be optionally enabled/disabled (with a special instruction like echo/save)

    If you are looking for a good compromise between compression ratio and low memory usage you can consider the standard algorithm LZO which requires no memory buffer during the decompression process.

  • Are modules stored as flat strings? They probably should be...

    Maybe something like the "compiled"?

    function (x) {
    "flat";
    ...
    }

  • The compression is done on the ARM. The problem is that you have ~80kB of RAM and 48kB of flash. If you don't compress then what do you do? I'm not sure I really see what having the option to disable compression really gets you.

    LSO looks good, but the licence is GPL and so can't be included in the more liberal MPL licenced Espruino! I guess I could ask if he'd be willing to licence it differently.

    @DrAzzy modules are executed when they go into Espruino, so they're stored in the same way as everything else. Hopefully a tweak to store functions as flat strings would automatically include any that are in modules too.

  • If you are looking for a non GPLed algorithm you can also consider LZ4 as a candidate (more fast but using few memory during process)

    My perception about source code compression is that, on particular and huge scripts, this process renders more complex the search for a compromise between performance and memory management.

    I take as example a big script that I have completed recently that works without problems on an ARM STM32F103RCT6 and old firmware, but shows this warning (ERROR: Too big to save to flash) on latest version.
    Maybe my example can be a borderline case but I think is good to let to the developer a chance to optimize by hand his code.

  • So you're using something other than the Pico or Espruino board? You could potentially build your own firmware and allow more storage space in Flash memory for your code - then this wouldn't be an issue.

    The cause is probably not really the compression, but that I had to reduce the flash memory used to store code on the F103 boards (in order to give more to the Espruino firmware).

  • Ok, please could you try this build: http://www.espruino.com/binaries/git/com­mits/a5688ca49ad5b0512fc513b4a7919b3b627­486ec

    It's using LZ4 compression, and should hopefully manage to do a better job than simple RLE.

    edit: just wait a few minutes - it hasn't finished building yet

  • I have tryed this
    http://www.espruino.com/binaries/git/com­mits/a5688ca49ad5b0512fc513b4a7919b3b627­486ec/espruino_1v81.149_espruino_1r3.bin­
    the software has been flashed without problems or warnings, but during his normal usage (when the script isn't at his maximum load) the free memory returned by process.memor().free is about 160 vars

    before 1v80 in this situation my script provide me about 236 vars as free memory

    this is very strange
    please, can you provide me more details about your compression mechanism?.

    compression of the source code, in a binary blob, is done after calling save() function, right?
    after a reboot a function is called to decompress the source code, when the espruino has decompressed the entire source code it keeps in memory the blob?

    I'm not able now to perform a complete test but I think is a proportional trend and this solution render useless my effort done to optimize the memory usage on my script

    build my own firmware it's a good idea (and my secret target :-) ) but for now I should to find more time for analyze and understand your software
    For now our devices are working well with standard firmware, but in the future I'd like to build a version with some functionality to manage regular expression

  • The compression happens after save() and the code should be restored exactly as it was before - there should be no difference between LZ4 and regular RLE.

    Maybe you could compare with a recent build that doesn't have RLE compression like this one

    Chances are it will be exactly the same as the LZ4 version.

    My guess is that the increased memory usage is because the new Espruino builds actually store line numbers for functions (which takes one extra variable per function) in order to give you proper debugging. If you use the latest Web IDE from GitHub I just added an option so you can turn that off if you want to, which should save you some memory.

    in the future I'd like to build a version with some functionality to manage regular expression

    That'd be great - and if you are using Espruino in your company's products, anything you can contribute back to help Espruino is appreciated.

  • As a company, we will publish all improvements made on your original source code, because we believe in open source philosophy and would work in compliance with it

    I'm happy that this problem isn't related to the compression features, unfortunately I'm using the Web IDE from chrome store.
    I will wait that the new version is published on the store

  • Hi Gorden
    Thanks for the new LZ4 Version. It worked:

    save();
    =undefined
    Erasing Flash.....
    Writing.................................­...........
    Compressed 81600 bytes to 42825
    Checking...
    Done!

    ={ "free": 3040, "usage": 2060, "total": 5100, "history": 1270,
    "stackEndAddress": 536958392, "flash_start": 134217728, "flash_binary_end": 320920, "flash_code_start": 134234112, "flash_length": 393216 }

    There seems to be a lot of potential in the compressen method. Maybe there is even a better method in the future.

    Is there some documentation with hints how to reduce space ? I.e. use UintArray not strings etc.

    Many thanks

    Sacha

  • Great! There's http://www.espruino.com/Performance that talks about memory.

    Also, newer builds should improve memory usage a lot more. Yesterday I made a few more changes that reduce memory usage in general.

    I may have to find a different compression library - LZ4 seems quite big, so I can no longer build Espruino for 128kB parts like the Olimexino :( Any other thoughts? There must be something out there... Speed is not an issue at all as it only happens once at startup.

  • @Sacha a little Off Topic:
    Now I'm curious, how have you reached 5100 available variables?
    I know that some modules (like graphics, hashlib, tv) can be excluded from a build, but how to assign this extra space to increase the available variables?

  • @Francesco on the Espruino Pico that's just the amount of variables that you get. There's more RAM, so more can be used for variables.

  • I'll explain, my question is: how do you calculate/estimate the amount of free space to use for variables? How changes this space when I choose what modules include/exclude from a build?

  • You have to change the details in boards/YOURBOARD.py to tweak the amount of variables, as well as the flash used to save them. The values depends on all kinds of things, and it's really just a matter of making an educated guess once you know how much RAM and Flash you have left over.

    If you need help customising the build for your company's board I can always provide consultancy services.

  • Is compression related to the odd experience I have of sending code once, executing save and being told too big for flash, then resending exactly the same code doing save and finding it is 5000 bytes slimmer and does fit on flash? Up till now I've assumed this is something wrong with the first time the IDE processes a local module, but it still seems very odd to me.

  • I have no idea why there should be any difference between the times when code it sent - there really shouldn't be. If you have some code that you can reliably get that with, maybe you could send it over so I can check?

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

strange behavior with RLE compression of flash new code

Posted by Avatar for Francesco @Francesco

Actions