Proposed, and implemented, some enhancements (1Mb + 1Mb firmware, resizable jsvars size, ...)

Hi all,

This looks like a great piece of work, and I am migrating some personal project to Espruino with my ESP cards.

For everything to work fine, I needed some enhancements, so I have done them (on v91), and I think it woud be worth sharing so that these can be merged into the public version:

• Resizable vars : the code is there, for the Linux version. Just patched a bit to enable it in ESP (should also work for other cards now) and fixed also some minor bugs on the save to / load from flash code (as it was, I think it would also fail on Linux, as with RESIZABLE_JSVARS the vars are stored in blocks, while the compress functions, as they were, read and wrote a full contiguous block of memory)

• 1024Kb + 1024Kb memory map : Added few lines to the Makefile to create a new target for the ESP 4Mbyte cards. With this layout, now we have 1024KB for firmware and stored vars and functions. Still neeeded to check where to fix some address mapping for the second image, as in this mode the bootloader maps 0x4020000 to either the first or second Mb of flash, and the image itself is exactly identical for user1 and user2. Just need to check in existing code if there is something harcoded around flash address and fix it.

So now, I get:

reset();
=undefined
... [ Espruino boot banner ] ...
Flash map 4MB:1024/1024, manuf 0xe0 chip 0x4016
Flash code area: 0x76000 (565248 bytes)

process.memory();
={ "free": 106, "usage": 22, "total": 128, "history": 0 }

require("ESP8266").getState();
={
"sdkVersion": "2.0.0(5a875ba)",
"cpuFrequency": 160, "freeHeap": 34232, "maxCon": 10,
"flashMap": "4MB:1024/1024",
"flashKB": 4096,
"flashChip": "0xe0 0x4016"
}

var tester = [];
=[ ]

for(var i=0 ; i < 900 ; i++) tester[i] = i;
=undefined

process.memory();
={ "free": 88, "usage": 936, "total": 1024, "history": 6 }

var more = [];
=[ ]

for(var i=0 ; i < 900 ; i++) more[i] = 2*i;
=undefined

process.memory();
={ "free": 210, "usage": 1838, "total": 2048, "history": 6 }

require("ESP8266").getState();
={
"sdkVersion": "2.0.0(5a875ba)",
"cpuFrequency": 160, "freeHeap": 3344, "maxCon": 10,
"flashMap": "4MB:1024/1024",
"flashKB": 4096,
"flashChip": "0xe0 0x4016"
}

require("ESP8266").reboot();
=undefined
... [ ESP boot info and some garbage :) ] ...
Loading 2048 jsvars from flash...

dump();
var tester = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,;
var i = 900;
var more = [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, ;
pinMode(D0, "input_pullup", true);
pinMode(D4, "input_pullup", true);
pinMode(D5, "input_pullup", true);
pinMode(D12, "input_pullup", true);
pinMode(D13, "input_pullup", true);
pinMode(D14, "input_pullup", true);
pinMode(D15, "input_pullup", true);
=undefined

TO DO:

• More testing
  
• Take benefit of this extra memory (make sure we can access code directly on it)
  
  

For the second part, what I use is as small piece of code based on something I found a couple of years ago in Internet and which I enhanced bit, so that ESP can access directly the flash mapped memory independently of alignment, or size (1, 2, 4 bytes) with no further modification to the user code. This is possible because ESP8266 is a bit of a monster, and allows virtualizing memory access through HW catchable exceptions for this kind of memory access errors, so a trap function can directly see the failing opcode and target address, access flash with the right alignment, simulate the opcode and put the result on the right register, and return control to the program (at assembler opcode level) after the offending opcode address.

This is seamless to your code, that can now access memory mapped flash as if it was in the rodata area. Of course, there is an impact in speed, which I have not benchmarked, but not noticed either, as we are not using our ESP8266s to do math intensive calculations (are we?)

Will work on other improvements.

So, if there is interest on it, I have a bunch of ".patch" files I will be happy to submit, or share, or whatever is the best way to get these into the main build.

Hi @Pedro, looks promising !

Building firmware for ESPs has a lot of compatible setting to make sure it runs for all models.

There was a switch to a single board build

So why not switch to a two board build: 512K and 4MB

Would like to jump in for testing, please share.

Hi @Pedro
The way of patching this stuff is to clone https://github.com/espruino/Espruino into your own name.
Then you can make a branch, say called 4MB, and then do your updates into the branch.

Then when you are ready, you do a pull request to the /espruino/Espruino master.

I think you might need to reserve some heap space for the wifi, as it looks like you don't have much heap space left after the vars increase. The buffer for IP are made dynamically and a free heap of 3344 might not be enough.

Thanks @MaBe, I am following @Wilberforce instructions on to create a branch so that the code cen be checked, and once I get feedback and more testing I will do a pullt request. Need to investigate how to do it : not a git master :) even if I remember doing something similar to patch Linux kernel in the past.
For the two board switch, the main issue is that platform_config.h and some ifdefs in the code must behave differently (for the RESIZABLE_JSVARS part), so I did something like the past two board switch, but the whole thing needs to be recompiled (not just relinked to get the different firmwares), so it is somehow different. Will try to submit today so that you can check it.

After reading through the source code, I think there is no need to have a new board for ESP, because with make option FLASH_4MB=1 it should be possible to handle all stuff.

FLASH_4MB=1

build "flashMap": "4MB:1024/1024" 

not sure about RESIZABLE_JSVARS

Thanks @Wilberforce, will try to branch the code so that it can be reviewed.
Regarding your comment on free heap, I just pushed it to the limit (2048 jsvars dynamically allocated). I agree at some point I need to put some check to keep a minimum heap available, failing to create a new jsvar block, but the result should be the same : memory full / bad things happen :)
Currently new jsvars are allocated in blocks of 128, so it does not go directly from 31KB heap to 3KB heap.

Ok, sounds good.

32k heap sounds like a bit more than usual - was that related to your changes?

@Wilberforce normal is 8k heap and 1700 vars

You are right. For the 4MB map we do not need a new board. For RESIZABLE_JSVARS, as it is behaving differenty as the standard ESP8266 current version, I wanted to make sure not to mess with it and create a new board. To avoid "forking" then new board just includes the standard ESP8266_BOARD.py and changes a few parameters. If there is a cleaner way, will be happy to do it.

Ahh - I see why now - 128 vars at startup, that's why so much heap is available.

Yes it is. Initially only 128 jsvars are allocated, and it grows as needed.

@Pedro can you please briefly explain how to switch to "flashMap": "4MB:1024/1024"

found a lot of information on this espressif page in chapter 4 and 5

In the Makefile:

ESP_FLASH_SIZE ?= 6
ET_FS ?= 32m-c1

And in user_main.c, change (to avoid a warning):

if ((chip == 0x4013 && map != 0) || (chip == 0x4016 && map != 4)) {

to

if ((chip == 0x4013 && map != 0) || (chip == 0x4016 && map != 4 && map != 6)) {

As soon as I have created the branch I will submit the changes for review (including this in the makefile with the proper conditionals)

In this mode, user1 and user2 bin files are identical (no different ld files) and the bootloader maps the first or second MB of flash to 0x40200000. As commented before, I need to check in the existing code that there is nothing hardcoded in current espruino build for ESP8266 when referencing flash from the first or second image

@MaBe, @Wilberforce, how do I create a new branch ? (or do I just fork into my repository). As commented, not a expert github user (mostly a newby here)

These are the hard coded sections I know so far:

https://github.com/espruino/Espruino/blob/master/boards/ESP8266_BOARD.py#L46

https://github.com/espruino/Espruino/blob/master/libs/network/esp8266/jswrap_esp8266_network.c#L1185

not sure about https://github.com/espruino/Espruino/blob/master/libs/network/esp8266/ota.c

• Create yourself a github account

• goto https://github.com/espruino/Espruino

• and click on Fork

That's it :)

1 Attachment

Next

• git clone your fork

• install a tool like "Github Desktop" an add your fork with "+"

• add your changes and commit them with "Github Desktop"

1 Attachment

Hi @MaBe,

If I did not get anything wrong :) here's the branch : https://github.com/pedro-es/Espruino/tree/esp8266-enhancements

I only did one commit for all the files, as they are interrelated. The comments for the commit covers all.

Hi @Pedro,

looks great, thanks for sharing.

Will start to implement this into my branch and run some test.

Your stuff is a great enhancement for ESP8266 with 4MB!

• got "4MB:1024/1024" running with fixed vars.

• used "FLASH_4MB_C1" instead of "ESP8266_BIGFLASH" :-)

next is the "scripts/build_platform_config.py" stuff

My implementation of FLASH_4MB_C1:

https://github.com/MaBecker/Espruino/commit/b41210ba77b5fd75bb949d7d07cc76c6528228ca

Happy it is being useful :)

Just a couple of remarks:

• No need for two different linker scripts (app1 and app2). By definition in this mode both images should be identical (which is indeed a great advantage), so we only need to generate one, and if we want to keep two for consisntency, I would point both linker scripts to the same file.
  
• We need to check the upper limit. In the "jswrap esp8266 network.c" file you pointed before as having a hardcoded flash address, I see it is pointing to the last page, looks like) so we may need to adjust the linker script to allow for one page less size, and of course adding 1Mb to the flash address if we are in image 2 (in runtime, getting the base flash address from the corresponding sdk os_* function)
  

Thanks

EDIT:

No need for two different linker scripts (app1 and app2)

good point, as it is just a linker script keep both :-)

Shrink firmware size by six pages and always use first 1MB area, five pages to save code and one page for wifi. What do you think ?

tested wiflash with "4MB:1024/1024" - works - Thanks @tve