You are reading a single comment by @Robin and its replies. Click here to read the full conversation.
  • Hi everyone. You know I love the storage module, but I'm still having issues with it.

    Here's my scenario. I have an app which uses this pattern:

    https://www.espruino.com/Data+Collection­

    and writes a CSV of data in "append" mode (actually to three seperate CSV files).

    So my storage gets full over a couple of days. So it appears that the standard procedure in this case is to download your data and then run storage.compact() to free up space. However this almost always corrupts my storage and I have to reflash everything. This happened constantly on v2.07. So every few days I was reflashing and reconfiguring my modules. I saw there were some changes to fix storage.compact() after in the changelog, so I have upgraded to v2.10.

    So this week my CSVs wrote their data, and the storage got full, so I did a storage.compact().

    But the same problem persists. When I ran storage.compact() I got "Got [ERASED] expected EOF". I used to get that on v2.07, but I could still get my data through the Web IDE. Now for some reason on "reset()" v2.10 ERASES everything, complaining of corruption, which isn't really a great default behavior, because often lots of files are still readable.

    I've included my console output. The sequence of events are:

    1) I get a flash full notice from my app (not shown).
    2) I do storage.compact().
    3) I get a "Got [ERASED] expected EOF" error.
    4) I do a "reset()" and the system erases my data.

    I'm using an ESP32. Any recommendations? Maybe I should "reset()" my module BEFORE doing "storage.compact()", in order to not have open "append" files?

    Anyways here's my output:

    
    >storage.compact()
    =undefined
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function called from system
    >c.sf()
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function "sf" called from line 1 col 6
    c.sf()
         ^
    >
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function called from system
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function called from system
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function called from system
    Uncaught SyntaxError: Got [ERASED] expected EOF
     at line 1 col 1
    [ERASED]
           ^
    in function called from system
    >reset()
    =undefined
    WARNING: gap set scan error code = 103
    WARNING: set rssi scan not implemeted yet
    WARNING: config adv data failed, error code = 103
    WARNING: check error not implemented yet:103
    Storage is corrupt.
    Erasing Storage Area...
    Erase complete.
     ____                 _
    |  __|___ ___ ___ _ _|_|___ ___
    |  __|_ -| . |  _| | | |   | . |
    |____|___|  _|_| |___|_|_|_|___|
             |_| espruino.com
     2v10.33 (c) 2021 G.Williams
    Espruino is Open Source. Our work is supported
    only by sales of official boards and donations:
    http://espruino.com/Donate
    >
    >
    > 
    
  • Wed 2021.10.13

    'I'm using an ESP32. Any recommendations?'

    What were the results of process.memory() before entering the compact command?
    By chance mixing Storage and StorageFile?



    Hi @MisterG while the ESP32 falls under the not 'officially supported' category, have you tried the same on an authentic Espruino or maybe consider supporting the community over Patreon?

    https://www.espruino.com/Other+Boards

    While I have no direct experience with the Storage class, I might be willing to give it a go to learn a bit myself. A project I'm attempting may require I go down this path in any event using the Storage class, but my plan was to dive in next month.

    I could attempt to duplicate the errors you are seeing using your code module, to see if the same anomalies exist on a different platform. Having another set of eyes might uncover a process/procedure type of situation.

    Up to you as I continue to incrementally slog through a bizarre conversion issue.

  • What were the results of process.memory() before entering the compact command?
    By chance mixing Storage and StorageFile?

    I had about 1000/4000 JS vars available (ie: around 3000 used). I can check again next time I have to do this which would be in a couple of days. That's when the logs would fill up.

    Yes I'm definately mixing Storage and StorageFile. I use storage for single variables or arrays and StorageFile for logs/files that will be appended to, which from what I understand is how it is supposed to be used I think. Is that supposed to be an issue? As far as I understand StorageFile is just a series of regular Storage files with a backslash and number appended.

    Hi @MisterG while the ESP32 falls under the not 'officially supported' category, have you tried the same on an authentic Espruino or maybe consider supporting the community over Patreon?

    Of course this is always an option, but often the "official" Espruino hardware has been designed with a philosophy of running for two weeks on a CR2032 and BLE. This is great, because PixlJS and BangleJS and PuckJS are really great and super fun. However, the ESP modules are really suited for when you need 20,000 JS vars and some heavy internet lifting like HTTPS. So it's hard to find an overlap sometimes. I think the Patreon route is the best idea in this case.

    I think the Storage module is really fantastic and as I explore it I enjoy it's features more and more. For examples it seems to me you can load code directly into it and Espruino will read that code if you "eval" it, and it won't take up JSVars really. Also just being able to save your state between power cycles / reboots is awesome. However it should be reliable. Also it could be bigger :P

    So what I'm saying is I encourage you to explore and we can continue the discussion. I might try and create a small program to recreate the problem over the weekend. You gave me a good idea: Espruino Linux seems to support the Storage module. I could try and recreate the scenario there and see what happens. This would give me an idea if it's a more general problem or something on the ESP32 port side.

    What is your bizarre conversion issue?

About

Avatar for Robin @Robin started