-
Wed 2021.10.13
'I'm using an ESP32. Any recommendations?'
What were the results of
process.memory()before entering the compact command?
By chance mixing Storage and StorageFile?
Hi @MisterG while the ESP32 falls under the not 'officially supported' category, have you tried the same on an authentic Espruino or maybe consider supporting the community over Patreon?While I have no direct experience with the Storage class, I might be willing to give it a go to learn a bit myself. A project I'm attempting may require I go down this path in any event using the Storage class, but my plan was to dive in next month.
I could attempt to duplicate the errors you are seeing using your code module, to see if the same anomalies exist on a different platform. Having another set of eyes might uncover a process/procedure type of situation.
Up to you as I continue to incrementally slog through a bizarre conversion issue.
-
What were the results of process.memory() before entering the compact command?
By chance mixing Storage and StorageFile?I had about 1000/4000 JS vars available (ie: around 3000 used). I can check again next time I have to do this which would be in a couple of days. That's when the logs would fill up.
Yes I'm definately mixing Storage and StorageFile. I use storage for single variables or arrays and StorageFile for logs/files that will be appended to, which from what I understand is how it is supposed to be used I think. Is that supposed to be an issue? As far as I understand StorageFile is just a series of regular Storage files with a backslash and number appended.
Hi @MisterG while the ESP32 falls under the not 'officially supported' category, have you tried the same on an authentic Espruino or maybe consider supporting the community over Patreon?
Of course this is always an option, but often the "official" Espruino hardware has been designed with a philosophy of running for two weeks on a CR2032 and BLE. This is great, because PixlJS and BangleJS and PuckJS are really great and super fun. However, the ESP modules are really suited for when you need 20,000 JS vars and some heavy internet lifting like HTTPS. So it's hard to find an overlap sometimes. I think the Patreon route is the best idea in this case.
I think the Storage module is really fantastic and as I explore it I enjoy it's features more and more. For examples it seems to me you can load code directly into it and Espruino will read that code if you "eval" it, and it won't take up JSVars really. Also just being able to save your state between power cycles / reboots is awesome. However it should be reliable. Also it could be bigger :P
So what I'm saying is I encourage you to explore and we can continue the discussion. I might try and create a small program to recreate the problem over the weekend. You gave me a good idea: Espruino Linux seems to support the Storage module. I could try and recreate the scenario there and see what happens. This would give me an idea if it's a more general problem or something on the ESP32 port side.
What is your bizarre conversion issue?
-
Interesting, just few days ago @enaon hit similar issue on Magic 3 smartwatch - this is 52840 watch (14000 vars :-)) and over 500 kB free in internal flash (and 8MB external SPI), I've made two builds of recent 2.10, one that is using internal flash for Storage an another using external. He manages to fill 500kB of storage pretty quickly and then hit corruption with compact too. With storage in external flash there is plenty of space so there it is not visible.
-
@fanoush That's super interesting. I had actually often thought that because the Storage module's space is so tiny, that of course I'm going to run into problems quickly. For example -- my logs are now filled up again just after 3 days. What do you mean exactly by "external flash"? How is this differentiated technically from "internal flash"? The PICO I'm using (M5StickC) is just an ESP32 with integrated flash -- I believe -- but I assume it's just wired up via pins like any external flash module.
Your Magic 3 smartwatch sounds interesting -- 14K vars (woo) -- > competes well with the TTGO watch @ 20K :P I'd love to know more.
@MaBe That's ALSO super interesting. It could well be related. The last save from storage huh? I'll keep that in mind. I feel like I need to write some test code for this to hammer it out. I actually wrote a small test program to evaluate the effectiveness of your /targets/esp32/main.c modification for the HEAP memory on HTTPS requests.
There had been some discussion in the GitHub around 45K, 55K, 60K, etc for the heap memory. And so I wrote this code that would run on some ESP32 modules for 24 hours and record success / failures. The technique could be used here (ie: fill up a bunch of Storage space in different ways, then run Storage.compact() and see what happens).
Tomorrow I will "Storage.compact()" my device -- as it's full -- just 8K left using Storage.getFree(). I'll take a note of some settings and stuff and see what happens, and report back...
G
-
Sun 2021.10.17
'What do you mean exactly by "external flash"? How is this differentiated technically from "internal flash"?'
External EEPROM 8 pin DIP
One example: uses I2C and not SPI as suggested in #5 post
It's likely the 52840 watch DK has the 8M of external SPI accessible
-
What do you mean exactly by "external flash"? How is this differentiated technically from "internal flash"?
Sorry, this is esp32 forum and the case I described is nrf52 specific. There is some flash directly inside the nrf52 and then in that watch there is 8MB SPI flash which I call 'external'.
Builtin flash is directly mapped to memory and is managed by nrf52 NVMC hardware while SPI flash is read/written via SPI commands -
@fanoush Ah yes thank you, that is exactly the answer I was looking for. In the NRF module it's memory mapped. @Robin suggested it was I2C based which I find interesting (an I2C based flash module), but maybe unlikely for that unit. I'd love to find an I2C flash storage though, that would be fun.
Is your new Storage code designed to read external SPI modules? Perhaps we could use it to extend the size of the current Storage module and it's limitations.
@Robin Thanks for both of those. I reviewed the Storage vs StorageFile, and I think I'm ok. I read the files back using StorageFile.readLine(). Of course as far as I can tell the "append" pattern isn't even listed in the REFERENCE which is weird no? Maybe I just missed it. You have to go to the data logging example. This is a frustration with Espruino I experience alot. Alot of it sometimes seems kind mysterious or even mystical and the information is located in alot of unrelated different places. It could certainly use a book where everything is explained properly and laid you. Not a criticism, it's super fun, but really -- hard to find your solution sometimes.
So backed up all my data this time, and repeated my Storage.compact() action.
> >ESP32.getState() ={ sdkVersion: "v3.1.3-dirty", freeHeap: 11780, BLE: false, Wifi: true, minHeap: 9852 } >process.memory() ={ free: 1572, usage: 2428, total: 4000, history: 855, gc: 0, gctime: 6.627, blocksize: 13 } >storage.getFree() =4132 >storage.compact() =undefined >c.s() Uncaught SyntaxError: Got [ERASED] expected EOF at line 1 col 1 [ERASED] ^ in function "s" called from line 1 col 5 c.s() ^ > >begin() Uncaught SyntaxError: Got [ERASED] expected EOF at line 1 col 1 [ERASED]... ^ in function "begin" called from line 1 col 7 begin() ^ >reset() =undefined WARNING: gap set scan error code = 103 WARNING: set rssi scan not implemeted yet WARNING: config adv data failed, error code = 103 WARNING: check error not implemented yet:103 Storage is corrupt. Erasing Storage Area... Erase complete. ____ _ | __|___ ___ ___ _ _|_|___ ___ | __|_ -| . | _| | | | | . | |____|___| _|_| |___|_|_|_|___| |_| espruino.com 2v10.33 (c) 2021 G.Williams Espruino is Open Source. Our work is supported only by sales of official boards and donations: http://espruino.com/Donate >So I'm just repeating myself from last time, but after using Storage.compact() my functions were inaccessible (as can be seen) with the dreaded "Uncaught SyntaxError: Got [ERASED] expected EOF" error.
After reset() it AGAIN erased all my code and files I'd saved the data this time -- not a surprise of course, it's the same version of Espruino. But I honestly think we should give data preservation a higher priority in Espruino. If I write a logging module and I'm never sure if my data will survive -- then why would I use it really? Anyways, I still find this to be an insane default behaviour -- that it would "identify" my Storage is corrupt and instead of trying to help me get my data back, it would just erase everything. I'm now left with a virgin ESP32 module.
Don't forget this exact pattern repeated itself with 2.07 and I was always able to read my log files back. So the Storage area is not REALLY so corrupted as to be unreadable.
Anyways so the problem seems to be repeatable -- for me at least.
I guess my next move will be to write a test program to fill up Storage and use Storage.compact() to see what happens. Also I'll try it on the Linux and maybe ESP8266 build to see if it's port dependent.
Finally, has anyone else found that if you use Storage.compact() when you have very little storage space left, you get corruption? As you can see, I had alot of vars left for memory....
G
-
Ah I think is relevant:
https://github.com/espruino/Espruino/issues/2009
Opinions?
Gordon asks if it's reproducible...I'm pretty sure I can reproduce it reliably. What causes it exactly is another question...
-
Is your new Storage code designed to read external SPI modules? Perhaps we could use it to extend the size of the current Storage module and it's limitations.
It is not mine, it is already part of Espruino for Bangle.js
I just tested it and found in my case that it is becasue of watchdog. I have manual watchog enabled (
E.enableWatchdog(15, false);) so I need to callE.kickWatchdog()fromsetInterval.When I fill internal storage (approx 600KB) the Storage.compact() call takes too long and watchdog reboots it. With watchdog off it works fine for me and compact() call takes approx. 50 seconds and I see no corruption. So unless you are using watchdog too or ESP32 reboots for other reason in the middle of compact() too, you probably have different issue than me.
-
@fanoush Ah I see what you mean. In earlier versions I had that problem, but the issue resolved itself more recently as work was done on Storage. Now it's ok.
I see in the GitHub that Gordon posted some code to stress test Storage, so I might start there this week and see what the results are.
Robin
fanoush
MaBe-
Post a reply
About
Still issues with storage.compact()?
Posted by
@MisterG
Hi everyone. You know I love the storage module, but I'm still having issues with it.
Here's my scenario. I have an app which uses this pattern:
https://www.espruino.com/Data+Collection
and writes a CSV of data in "append" mode (actually to three seperate CSV files).
So my storage gets full over a couple of days. So it appears that the standard procedure in this case is to download your data and then run storage.compact() to free up space. However this almost always corrupts my storage and I have to reflash everything. This happened constantly on v2.07. So every few days I was reflashing and reconfiguring my modules. I saw there were some changes to fix storage.compact() after in the changelog, so I have upgraded to v2.10.
So this week my CSVs wrote their data, and the storage got full, so I did a storage.compact().
But the same problem persists. When I ran storage.compact() I got "Got [ERASED] expected EOF". I used to get that on v2.07, but I could still get my data through the Web IDE. Now for some reason on "reset()" v2.10 ERASES everything, complaining of corruption, which isn't really a great default behavior, because often lots of files are still readable.
I've included my console output. The sequence of events are:
1) I get a flash full notice from my app (not shown).
2) I do storage.compact().
3) I get a "Got [ERASED] expected EOF" error.
4) I do a "reset()" and the system erases my data.
I'm using an ESP32. Any recommendations? Maybe I should "reset()" my module BEFORE doing "storage.compact()", in order to not have open "append" files?
Anyways here's my output: