Fake EEPROM module just added

Hi,

I just uploaded this: http://www.espruino.com/FlashEEPROM

Hopefully it'll give you nice easy access to nonvolatile memory, without having to faff around with pages.

/* No arguments mean use the default, which will work on the Original Espruino or Espruino Pico
 You can supply extra arguments to choose which flash page you'll use, and even whether you
 use external flash memory - but be careful when doing this. You can accidentally overwrite
 Espruino itself! */
var f = new (require("FlashEEPROM"))(); 

f.write(0, "Hello");
f.write(1, "World");
//.. you can write to any address between 0 and 255, with any data up to 256 chars long

f.read(0)
// returns new Uint8Array([72, 101, 108, 108, 111])

E.toString(f.read(1))
// returns "World"

f.readAll();
// returns [
//  new Uint8Array([72, 101, 108, 108, 111]),
//  new Uint8Array([87, 111, 114, 108, 100])
// ]

f.write(1, "Espruino");
// has now overwritten 'World'

E.toString(f.read(1))
// returns "Espruino"

/* You can tidy up the journal if you want, removing any values that were
overwritten by subsequent writes. It can take around a second though */
f.cleanup();

Nice! Features like this really make Espruino great, really really easy to use.

What do you mean by:

and even whether you use external flash memory

Q: what is flash.length set to? Basically are you putting the eeprom area by default just after the flashed code/data, such that it gets lots if a new version of espruino changes the length in pages? Wouldn't it be better to add a param to the board config to specify where the eeprom area should start? At least going forward that would be safer.

Have you thought of any CRC to validate that the eeprom contents is still valid and hasn't been corrupted?

This module only works on a single flash page, correct? Might be nice to state that.

It looks like the addr parameter of FlashEEPROM.prototype.readAll is not used?

What are the memory requirements of this module in order to rewrite the data? If the flash area contains, say, 1KB of data the memory requirements will be much more than that, right? Have you thought of using two pages in order to incrementally copy from one to the other? This would also avoid data loss in case the flash page write gets interrupted.

What do you mean by: and even whether you use external flash memory

You should be able to specify an object for flash as the last argument. If we have flash modules that implement read/write/getPage/erasePage (I think there was one?) then you can just use them.

are you putting the eeprom area by default just after the flashed code/data

STM32F4 has only a few pages, so there isn't any choice in the matter. It uses the page starting at 384kB - the one that the chip isn't supposed to have (but that exists anyway) - all the rest are used. I thought it said in the docs.

... and by using a page that the STM32 shouldn't have, it doesn't get overwritten by the bootloader :)

specify where the eeprom area should start?

That's the optional first argument of the module? I don't think it needs to go in the firmware...

CRC

Hmm... maybe - I wonder what the chances of it being wrong are? The flash memory seems very reliable - so much so I have now turned off verify when flashing as it just wastes time :)

Single page / two pages

It'd be nice to use a second page, but on STM32F4 there aren't enough pages, and on the F1 the pages are 1024 or 2048k, so storing the contents for that in RAM isn't a big deal (although yes, if it got interrupted while writing a page you'd lose stuff)

It depends what you've written to the EEPROM what memory gets used. You could obviously fill a 128kB page with data and it wouldn't rewrite, but if you just wrote values at maybe 100 addresses and changed them enough to fill up memory, rewriting would not be an issue.

I'll get rid of the readAll argument... That was a copy/paste bug :)

specify where the eeprom area should start?
That's the optional first argument of the module? I don't think it needs to go in the firmware...

Mhh, you seem to be taking a very stm32-centric view. On the esp8266 the flash_start and flash_length values don't make a ton of sense because there are multiple areas with settings and such. Flash_start+Flash_length=Flash_end, basically, so you can't put anything beyond that. So the way this is coded, a user has to go and look up the memory map and decide into which gap to put the 4KB eeprom page.
If you would allow process.memory() to have an "eeprom" field, or "scratch_flash" field that could be populated by the device dependent code then it could be coded to just work. As it is, it only "just works" on the pico and for the esp8266 the user has to (a) discover that there's a problem and (b) dig out the answer, and (c) modify the code. That doesn't make for portable code (consider what happens when someone publishes a module that uses the flasheeprom).

Well, as you say, something defining a spare area could be good? I'm happy to add something like that - and maybe it could be pulled out of BOARD.py. At least if it isn't defined, the module could throw an error rather than soldiering on regardless.

Just to add, it looks like there's a big rant on Gitter that this doesn't 'just work' on ESP8266.

All you have to do is var f = new (require("FlashEEPROM"))(whatever_address_works_on_esp8266);

if you want to use this at the moment.

Big rant on gitter? where? there were some questions and responses, but no rant that I see.

Ahh, sorry - I literally just scanned over it this morning while answering mail and forum posts and saw:

However, he has chosen to NOT expose the physical address within the address space where the flash is read and written

But as you spotted that's not right. So no, not much of a rant :)

I had to translate rant, interesting. Anyway lets go back to the facts:

• Flash-object works on ESP8266
  
• FlashEEPROM works on ESP8266
  
• with the actual firmware from EspruinoBuilds the free area starts at 0x6c000 and ends at 0x7c000, @tve please correct me if I'm wrong
  
• another option to load some data into flash is to create a binary file with a tool of your choice, and upload this with ESP8266Flasher.exe
  

with the actual firmware from EspruinoBuilds the free area starts at 0x6c000 and ends at 0x7c000, @tve please correct me if I'm wrong

That is probably correct if it works for you ;-). The start really depends on where the current firmware binary ends. However, with the soon-to-happen change to a single firmware image for all flash chip sizes this is likely to change and may be different between flash chip sizes.

I'd like to keep FlashEEPROM in a state where it still works on a normal 1v81 release, but I think reporting free flash areas in process.memory could be really neat - and it could be modified to check and use those if they existed.

Perhaps something like:

unused : [
 { addr : ..., length: ... },
 { addr : ..., length: ... },
]

I guess for now, only one might get reported, but in a lot of cases I imagine there may be several small areas free that could potentially be used. I don't know if we'd only report whole free pages, or maybe we could report if there were parts of pages that were free as well.

Any thoughts?

@Gordon, AFAIK, erasing can only be done on whole page(usually 4K). Therefore I would recommend to publish empty pages only. Otherwise it would happen too often, that data needed somewhere else, will be deleted.
For example, someone could store some data in a page, used partly by save(). Erase this page and you could come in trouble.
@tve, I see the problem with single firmware for all ESP8266-boards. Would Gordons list of unused flash pages help ? Would it be possible to create something like unused_flash, similar to saved_code in ESP8266_BOARD.py ?

Would Gordon's list of unused flash pages help ?

To be fair, this was @tve's idea originally ;)

The list would be created by the process.memory function, so potentially some runtime checks could be done to see what part was being used?

I think a list of free page regions would be awesome. Would it make things too complicated to be able to reserve a chunk from that? That info would have to be persisted in the save() area, perhaps?
WRT esp8266: see https://github.com/tve/EspruinoDocs/blob/master/boards/EspruinoESP8266.md#flash-map-and-access if you want a preview for what I'm working towards.

That info would have to be persisted in the save() area

I'm not sure I understand this? You mean that when you update firmware you don't want to overwrite it?

Really good documentation on the memory layout though - I need one of those for Espruino boards :)

I think if we added int jshGetFreeFlashAreas(uint32_t **addressAndLength, int maxAreas) to jshardware.c then the process.memory implementation could just use that, and report each area back as addr and length?

I guess probably returning a set of ranges (from + length) would be best, as otherwise you're going to get swamped with data items if you ask how many pages are in the 4MB parts :)

... other option is to just have JsVar *jshGetFreeFlashAreas() and let each platform return its own array of data (I guess then some platform-specific hints could be added).

...[[from,length,pageSize],...], or [[from, pageSize,numberOfPages],...]... if page size varies.

Adding the page size sounds like a good idea.

But you can just do flash.getPage(addr) on it and get the page size or work out the number of pages already? I'm not sure it's worth duplicating that work for every platform?

Ah, good point.

DO NOT USE THIS ON ESPRUINO PICO FIRMWARE 1v82 - it'll cause your Pico to overwrite part of its firmware, and you'll have to enter the bootloader and re-update it.

I'll fix this in 1v83 which I'll release shortly.

And 1v83 is released now, so it's fine to use again.