Architecture: Semantic meaning of the definition of SAVE_ON_FLASH

I'm starting to study how to integrate Flash interaction with the ESP8266 port. I have stumbled over a macro definition called "SAVE_ON_FLASH". I grepped the whole code based and searched the forum looking for a description of what it means but couldn't find one.

Does anyone know what the meaning of this definition might be?

:) It's what gets set when you're building for devices with very little flash memory - so it does things like:

• doesn't include non-vital JS functions and classes (I think waveform gets ripped out for instance)
  
• doesn't inline some functions (would have been done for speed, but saves space if they're not inlined)
  
  

... but it doesn't have anything to do with saved program code at all.

Not sure where the best place is to document it is actually - maybe we could add something to README_BuildProcess.md?

He he he .... so "SAVE_ON_FLASH" is all about "save on the usage of flash memory" as opposed to "saving data to flash memory" .... LOL ... British vs American language structure strikes again :-)

Yep! You know that meaning never occurred to me at all!

...what about 'SAVE_ONTO_FLASH(CARD)'?... shortening language expression(element)s and looking at them out of context creates 'fun'.

I'm looking into the saving of code into flash. I see the stuff in jswrap_flash.c which uses a special area that is statically defined. Is there a way to make this a bit more dynamic because esp8266's come with different flash memory sizes and it would be a shame to have to define different boards just for this.

Related question, would it be possible to execute JS programs directly from "read-only" memory? In the esp8266 a big chunk of the flash is memory mapped, so the program code could be written to flash but be readable like normal memory. There are some 4-byte alignment constraints I'd have to look into. This would mean that programs don't have to be loaded into RAM and use space there before being executed.

Aren't there currently separate Makefile targets and binaries for all the different amounts of flash? If those are absolutely required then I guess it's not a big deal if that's hard-coded?

Espruino will always have a fixed amount of RAM to play with on all ESP8266's? in which case it'll only need the same area of flash memory in each. Hopefully the rest of the memory would still be exposed to the user though (but that's fine, because it can be done via the jshardware flash read/write functions).

So right now, Espruino only executes from RAM - and when it saves, it doesn't save the raw JS code - it saves what's in RAM. So the state of the interpreter at that time - functions, variables, active timers, pin state, etc.

It's done that way so you can develop using just the 'console' if you need to - or can upload code, but then can just tweak things very slightly and save again.

Potentially it could execute from raw JS code in flash, but I don't think that helps a great deal. Whenever a function gets defined it'd go into RAM anyway, so you'd still end up losing quite a bit of memory.

Now Espruino could store read-only variables in flash, and have writable ones in RAM - but that's actually a pretty massive change.

Aren't there currently separate Makefile targets and binaries for all the different amounts of flash?

Yes. See https://github.com/tve/Espruino/blob/master/Makefile#L463-L535
But FLASH_SAVED_CODE_START is set through the board config stuff, isn't it?

it'll only need the same area of flash memory in each

Correct, but at different start addresses. With a 512KB flash we'll have to squeeze the save area in there. With 4MB it can move up beyond the firmware partition, which will help with ensuring that upgrades don't wipe it out.

Whenever a function gets defined it'd go into RAM anyway

Do you have an high-level overview of how this works? You store the source in RAM? Is that in JsVars or somewhere else? Do you parse it into an AST or something and then interpret that? And what does the compiler produce?

Yes - although ideally if you're targeting different boards with different setups then you'd have different board.py files?

With 4MB it can move up beyond the firmware partition, which will help with ensuring that upgrades don't wipe it out.

Presumably it could stay in the same place and everything would be fine? The storage of the data changes between versions - so realistically even if you kept the saved data between updates it probably wouldn't work with the new one.

Do you have an high-level overview of how this works?

Have you read the docs linked from the readme?

Internals and performance might help?

I've read these pages but there really isn't much there about how code is executed. Is the code stored as strings in JsVars or is that external to JsVars? One string or one per line or ...? Is each statement recursive-descent-parsed each time it is executed? What does the compiler produce?

Is the code stored as strings in JsVars or is that external to JsVars?

It's inside JsVars. It'd be worth using trace(...) to peer inside some datastructures so you get a feel for it?

One string or one per line or ...?

One big string,

Is each statement recursive-descent-parsed each time it is executed?

Yes - there's no bytecode at all. It's slightly quirky, but it works and saves a lot of memory.

What does the compiler produce?

You mean this compiler?

Again, there's some info in there - but proper native code is stored in a String in RAM, and is executed directly from where it sits.

One big string,

Is each statement recursive-descent-parsed each time it is executed?

Yes - there's no bytecode at all. It's slightly quirky, but it works and saves a lot of memory.

Could it save even more memory when building only references in RAM to the structural blocks - if.then.else, loops,... of the ('static'/uploaded) code (located in eprom)?

It's something I'd considered - there may be an issue on GitHub for it.

It's not easy to do though, because of the reference counting and locks which change whenever something is referenced. Also, the jsvLock implementation would need to check if the variable was in flash or not, which would slow down everything.

Also, right now, the data is stored compressed in Flash - so I couldn't access it directly.

Thanks for the pointers, Gordon. Sounds like it "shouldn't be too difficult" to (a) move the source code strings to flash and just keep a string-ref in jsVars, and (b) place compiled functions into flash as well.

What this means to me right now is that for the esp8266 I should place the "save to flash" stuff in a memory mapped area while a SPIFFS filesystem can use the non-mapped area. There's a 2MB limit to what can be mapped.

Yes, sounds good.

Actually using a stringref to reference a string somewhere else is difficult because there are only 16 bit pointers (12 if you've got less than 1024 vars). However you could initialise a JsvStringIterator with an arbitrary pointer and string length and execute straight from flash that way.

That's what I had planned for setBootCode mentioned in the other thread - and for that, having 'save to flash' memory mapped would be perfect.

There's actually a FAT32 filesystem already in Espruino (so external SD cards can be supported). IMO it'd make a lot of sense to just use that in the remaining block of memory instead of including a whole new filesystem as well.

Thanks for mentioning the FAT32 filesystem. I'll look at it.

I've spent the past couple of hours looking at the FAT32 filesystem to use the "internal" esp8266 flash. The characteristics of the serial NOR flash chips are: 4KB block size with the ability to write 256 byte pages (but only erase full blocks). The amount of flash available is 1MB to 3MB for 2MB to 4MB flash chips, and ~32KB for 1MB flash chips (and nothing for the 512KB flash chips).

The problem I'm hitting with the FAT32 implementation is that it really assumes that 512 byte sectors are writable and erasable individually, which is not the case here. Setting the FAT sector size to 4096 would make things work, but then that's the smallest file size, which means that a 1MB filesystem could only support about 200 files and a 32KB filesystem would be a no-go. Write will be really slow because each sector write will consist of an erase followed by a write. On the RAM front the whole thing may blow up 'cause buffers will be 4096 bytes and there just isn't room for even 2 buffers.

One alternative might be to implement sector remapping, i.e., what a wear-leveling controller does, but I don't think we really want to do that. Another alternative is to use a filesystem that is designed for NOR flash, such as SPIFFS: https://github.com/pellepl/spiffs. The same issue will come up to use the 2MB flash on the emw3165, as far as I can tell.

I may just implement the 4096 sector size FAT32 on the esp8266, but I believe the better solution would be to implement spiffs. Thoughts?

What are your real needs in respect to capacity? ...minimal, comfortable, luxury?

@allObjects you did have a filesystem you'd done in JS a while back didn't you? Did that cope with flash pages correctly?

It is more a memory manager with garbage collect for storing/retrieving/updating strings... it could certainly be tuned to the needs of a page oriented thing. I was thinking of it when reading @tve's post. The issue though is that the way it is built, a flash would wear out quickly. It has a pointer system for free space and freed space management that is updated on most of the writes... too much of updates for a flash. I had started it for MRAM/FRAM where no page limitations are and no real wear out. For a flash, it gets a bit more complicated, because the free space would have to be managed with bit lists (ribs - 1 bit per n bytes block)... reference to mm.html in conversation about New EEPROM module names/AT EEPROMS with an emulation / cross-implementation for developing the logic... some more explanation and rational in the conversation about Running code off of an EEPROM - includes screen shots of usage example and explananation what's goin on in the memory (all in the cross-context / browser), where visualization is easy to show what is going on.

To apply the memory manager for flash would require:

• convert pointer/chained blocks system to bitlist for free space
  
• take page constraints into account (only update 'in one direction' (go for a new page or rease it before rewrite...)
  
  

Not that simple... may become slow, and is still not holding up for frequent use.