You are reading a single comment by @MisterG and its replies. Click here to read the full conversation.
  • @fanoush Ah yes thank you, that is exactly the answer I was looking for. In the NRF module it's memory mapped. @Robin suggested it was I2C based which I find interesting (an I2C based flash module), but maybe unlikely for that unit. I'd love to find an I2C flash storage though, that would be fun.

    Is your new Storage code designed to read external SPI modules? Perhaps we could use it to extend the size of the current Storage module and it's limitations.

    @Robin Thanks for both of those. I reviewed the Storage vs StorageFile, and I think I'm ok. I read the files back using StorageFile.readLine(). Of course as far as I can tell the "append" pattern isn't even listed in the REFERENCE which is weird no? Maybe I just missed it. You have to go to the data logging example. This is a frustration with Espruino I experience alot. Alot of it sometimes seems kind mysterious or even mystical and the information is located in alot of unrelated different places. It could certainly use a book where everything is explained properly and laid you. Not a criticism, it's super fun, but really -- hard to find your solution sometimes.

    So backed up all my data this time, and repeated my Storage.compact() action.

    >
    >ESP32.getState()
    ={
      sdkVersion: "v3.1.3-dirty",
      freeHeap: 11780, BLE: false, Wifi: true, minHeap: 9852 }
    >process.memory()
    ={ free: 1572, usage: 2428, total: 4000, history: 855,
      gc: 0, gctime: 6.627, blocksize: 13 }
    >storage.getFree()
    =4132
    >storage.compact()
    =undefined
    >c.s()
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function "s" called from line 1 col 5
    c.s()
        ^
    >
    >begin()
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]...
           ^
    in function "begin" called from line 1 col 7
    begin()
          ^
    >reset()
    =undefined
    WARNING: gap set scan error code = 103
    WARNING: set rssi scan not implemeted yet
    WARNING: config adv data failed, error code = 103
    WARNING: check error not implemented yet:103
    Storage is corrupt.
    Erasing Storage Area...
    Erase complete.
     ____                 _
    |  __|___ ___ ___ _ _|_|___ ___
    |  __|_ -| . |  _| | | |   | . |
    |____|___|  _|_| |___|_|_|_|___|
             |_| espruino.com
     2v10.33 (c) 2021 G.Williams
    Espruino is Open Source. Our work is supported
    only by sales of official boards and donations:
    http://espruino.com/Donate
    > 
    
    

    So I'm just repeating myself from last time, but after using Storage.compact() my functions were inaccessible (as can be seen) with the dreaded "Uncaught SyntaxError: Got [ERASED] expected EOF" error.

    After reset() it AGAIN erased all my code and files I'd saved the data this time -- not a surprise of course, it's the same version of Espruino. But I honestly think we should give data preservation a higher priority in Espruino. If I write a logging module and I'm never sure if my data will survive -- then why would I use it really? Anyways, I still find this to be an insane default behaviour -- that it would "identify" my Storage is corrupt and instead of trying to help me get my data back, it would just erase everything. I'm now left with a virgin ESP32 module.

    Don't forget this exact pattern repeated itself with 2.07 and I was always able to read my log files back. So the Storage area is not REALLY so corrupted as to be unreadable.

    Anyways so the problem seems to be repeatable -- for me at least.

    I guess my next move will be to write a test program to fill up Storage and use Storage.compact() to see what happens. Also I'll try it on the Linux and maybe ESP8266 build to see if it's port dependent.

    Finally, has anyone else found that if you use Storage.compact() when you have very little storage space left, you get corruption? As you can see, I had alot of vars left for memory....

    G

  • Is your new Storage code designed to read external SPI modules? Perhaps we could use it to extend the size of the current Storage module and it's limitations.

    It is not mine, it is already part of Espruino for Bangle.js

    I just tested it and found in my case that it is becasue of watchdog. I have manual watchog enabled (E.enableWatchdog(15, false);) so I need to call E.kickWatchdog() from setInterval.

    When I fill internal storage (approx 600KB) the Storage.compact() call takes too long and watchdog reboots it. With watchdog off it works fine for me and compact() call takes approx. 50 seconds and I see no corruption. So unless you are using watchdog too or ESP32 reboots for other reason in the middle of compact() too, you probably have different issue than me.

About

Avatar for MisterG @MisterG started