is storage.compact() working ?

Posted on
  • Hello Gordon
    I tried the storage function on FW 2.00

    I transfered some files over wifi and stored them using the storage.write();
    After deleting all files exept the saved code, i do call storage.compact().

    But i do net get the space back ?

    >ST.getFree();
    =22516
    // Transferred some files and stored: 
    >ST.list();
    =[
      ".varimg",
      "init",
      "config",
      "CORE",
      "DOUT",
      "SW",
      "reboot"
     ]
    // New Free.
    >ST.getFree();
    =9504
    
    >ST.erase('init');
    =undefined
    >ST.erase('config');
    =undefined
    >ST.erase('CORE');
    =undefined
    >ST.erase('DOUT');
    =undefined
    >ST.erase('SW');
    =undefined
    >ST.erase('reboot');
    =undefined
    >ST.list();
    =[
      ".varimg"
     ]
    > ST.compact();
    =undefined
    // Free ?
    >ST.getFree();
    =9504
    

    Thanks for the help.

    Sacha

    P.S.: A storage.eraseAll() function that removes everything exept the saved code would be nice (As an option for eraseAll() ) ?

  • There was actually an issue with writing that got fixed in firmwares after 2v00, so you could have a go with one of the 'cutting edge' builds I guess?

    The other thing is it might not have been able to compact because there wasn't enough RAM free to temporarily put the .varimg file in RAM. In that case it'd be able to compact other flash pages, just not the ones that contain that file... So if you added a bunch of files that used up more flash in other pages then you'd be able to free that, just not what's in any page containing .varimg

  • Hi Gordon
    Thanks for your fast answer. I tried the latest 'cutting edge' build. Same result.
    After deleting everything exept the .varimg and calling compact(); again, i do not get any space back.

    It would be very nice if you could implement a way to delete the flash for reuse without deleting the saved code. Like eraseAll() but without deleting the saved code.

    Thanks
    Sacha

  • I'm afraid it's a bit difficult to handle nicely. Hopefully it'll get done at some point, but about the only way of getting it to work is to use other flash pages as temporary storage, which seems a bit of a waste of flash lifetime.

    Are you really in a situation where you absolutely need that ~2kB of flash memory? As I said above, once you start using other flash pages, those pages should still be able to be cleared automatically with eraseAll.

  • Hi Gordon

    I have several EspruinoWIFI distributed in my house.
    The Espruinos are "installed", in most cases difficult to connect via USB again.

    They all communicate to each other over IP. They know each other over UDP dynamicaly.

    Each espruino has a little "core" software saved.

    The over all configuration of all espruinos and additional code for different devices (Replays, Sensores, etc.)
    are dirtibuted over wifi to the espruinos as needed by the "core" software.

    Every espruino will get the config and the individual needed additional device code to use and store it on the storage.

    So i have a full redundancy. Each espruino has stored the config of all espruinos and several espruinos has stored small code for
    different devices like sensores etc.

    They inform each other about the newest config version and the version of the small code pieces over udp.

    If a espruino has a newer version of the config for all espruinos or a newer version of code for connected
    devices (Relay, Sensor etc.), the other espruinos will request and get them over tcp too.

    That is the reason why i use a lot of the storage and get in trouble with it.

    That's why i'm looking for a way to wipe out everything except the saved code from the storage.
    In that case the config and the needed additional code can be downloaded and stored again.

    Greetings

    Sacha

  • Ahh, ok - that's tricky. I didn't realise it was Espruino WiFi.

    The Espruino WiFi is a bit oddball in that it only has one page for storage - a 64kB one - so there are no 'unused' pages it can free. If it can't get the contents of that page temporarily into RAM then it can't compact it.

    If you try:

    var s = require("Storage");
    s.list().forEach(fn=>print(fn,s.read(fn)­.length));
    

    It should show you roughly how much space is needed for each file. My guess would be that ".varimg" is using by far the most.

    By default it'll try storing the data that's needed on the stack, but that's probably only around 10k or so, and I bet .varimg is more than that.

    What you could try doing is:

    function tryCompact() {
      var s = require("Storage");
      var x = E.toString(s.read(".varimg"));
      if (x===undefined) throw new Error("Not enough RAM for varimg");
      s.erase(".varimg");
      s.compact();
      s.write(".varimg",x);
    }
    

    Which will try and allocate space for .varimg as a JS variable (rather than on the stack), will then remove it from storage, compact, then write it back. That might work?

    Otherwise .varimg is an image of what is in RAM when you save, so another slightly scarier option is to do:

    s.erase(".varimg");
    s.compact();
    save(); // but this will then reboot the Espruino WiFi
    

    But hopefully you should only ever have to do this when the WiFi board runs out of storage.

    Or... It looks like you're basically dynamically loading JS code? If that's the case then you probably don't need to use save() at all - you probably want to use the save to flash option in the IDE to save just a simple 'bootloader' that searches for init/etc and runs them in sequence?

    If you did that then you wouldn't need a .varimg file, and chances are compacting would go fine.

  • Hi Gordon
    Thanks for your answer. I got "not enough RAM for varimg".
    Save() is not an option. After the reboot, i get errors in the console. Wifi communication fails.

    With the save to flash option in IDE i'm not familiar. Do you have a link where i can read about this ?

    EDIT: Found: https://www.espruino.com/Saving

    Thanks
    Sacha

  • I've added to my wishlist a espruino wifi v2 with more RAM and more Flash ;-)

  • Hi Gordon

    After switching to "save to flash" in the IDE, i got a lot of more RAM free. After deleting all files except ".bootcde" and calling compact, nothing new. I do not get flashmemory back.

    Then i made a very ugly hack that seems to work!

    // ST is the storage object
    
    var x = E.toString(ST.read(".bootcde"));
        if (x===undefined) { throw new Error("Not enough RAM for .bootcde"); }
        else {
            // Ugly hack !
            eval('var x = E.toString(ST.read(".bootcde")); ST.eraseAll(); ST.write(".bootcde",x); console.log("survived");');
           // Here i do a reset via my custom watchdog function.
    }
    

    After the reboot everything is working fine, storage is freed and ready for reuse ;-)

    Sacha

  • That's great! Thanks for posting up the code - I'd forgotten about the issues you can hit if you're evaluating a function from Flash memory when you delete the flash! You could actually call E.reboot() to properly force a reboot without waiting for it to crash :)

    Just to add if you're using almost too much memory there is a very small chance that E.toString(ST.read(".bootcde")) will work, but will then fail later on once a bit more memory is used during eval - but it's super unlikely :)

    I could definitely add the ability to use JsVars for storage when compacting, which would make it a little less likely that you have to do this. Potentially Espruino could also scan over all the functions inside itself and adjust the addresses such that compact wouldn't cause them to break when using save to flash.

  • Hi Gordon

    Thanks for your help. How much mem will eval use ? I think i allocate some spare space ?

    var evalneed='How much is needed for eval ?';
    if (x===undefined) { throw new Error("Not enough RAM for .bootcde"); }
        else {
             evalneed=''; // Required for eval
    // Ugly hack !
            eval('var x = E.toString(ST.read(".bootcde")); ST.eraseAll(); ST.write(".bootcde",x); console.log("survived");');
           // Here i do a reset via my custom watchdog function.
    }
    
  • Honestly, I'm not sure. Mainly it'll be allocating the string, so:

    var evalneed='How much is needed for eval ?';
    var cmd = `var x = E.toString(ST.read(".bootcde")); ST.eraseAll(); ST.write(".bootcde",x); console.log("survived");'
    if (x===undefined) { throw new Error("Not enough RAM for .bootcde"); }
        else {
    evalneed=''; // Required for eval
    // Ugly hack !
            eval(cmd);
           // Here i do a reset via my custom watchdog function.
    }
    

    would probably be pretty safe.

    Or... you could just do:

    eval('var x = E.toString(ST.read(".bootcde")); if (x===undefined) throw new Error("Not enough RAM for .bootcde"); ST.eraseAll(); ST.write(".bootcde",x); console.log("survived");');
    

    Which might be tidier/safer?

  • Thanks again.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

is storage.compact() working ?

Posted by Avatar for Sacha @Sacha

Actions