Proposed, and implemented, some enhancements (1Mb + 1Mb firmware, resizable jsvars size, ...)

@mabe > tested wiflash with "4MB:1024/1024" - works - Thanks

Please post your complete wiflash line...

sure

• first step

  ESP8266_BOARD=1 FLASH_4MB_C1=1 make

• second step - manual way

  cd to_your_firmware_dir

  esptool.py --port "your_com_port" --baud 460800 erase_flash

  esptool.py --port "your_com_port" --baud 460800 write_flash \
  --flash_freq 80m --flash_mode qio --flash_size 4MB-c1 \
  0x0000 boot_v1.6.bin 0x1000 espruino_esp8266_user1.bin \
  0x3FC000 esp_init_data_default.bin 0x3FE000 blank.bin

• third step - send updates with wiflash.sh

  sh wiflash.sh "ip_of_esp":88 espruino_esp8266_user1.bin espruino_esp8266_user2.bin

Five pages for code save and one page for wifi save - and still using one board file :)

https://github.com/MaBecker/Espruino/commit/6421b97f5e9e27a1f886eb978e64b4dad0c56b92

Looking good. I think the change of saved code would need to be documented well.

I'm wondering if there is a better area for the saved code rather than the end of the application user Space?

I'd be a bit iffy about the RESIZABLE_JSVARS - it does make things slower, and may possibly also break flat strings/arraybuffers/etc - not to mention make the whole thing just crash with a stack overflow if you run out of memory.

I'm not entirely sure what you gain over just having set up a sensible number of JsVars beforehand.

The whole unaligned access catcher would be great though - while Espruino tries quite hard not to do unaligned accesses, having something to handle the one or two cases where it does would be really handy.

Also, with the flash map - I don't know if all 1MB is really needed at the moment? What would be great is automatically saving code to the extra areas of flash if they were detected - then finally you wouldn't get the 'out of memory' errors when saving

@Wilberforce

I'm wondering if there is a better area for the saved code rather than the end of the application user Space?

• ESP_FLASH_MAX is reduced to have extra pages for code and wifi

• afaik, this is the same technic that is used now for ESPs

@Gordon

Also, with the flash map - I don't know if all 1MB is really needed at the moment?

• Me too, but there are only a few KB left and as there are still improvements in Espruino, how long would it last until it does not fit in anymore?

• Using "4MB:1024/1024" still offers additional 2MB flash space

Is it possible to calculate how many space is needed by save() in worst case?

there are only a few KB left and as there are still improvements in Espruino

True, and you could probably add a few extra bits and bobs (encryption, etc). With the byte read hack you could simply force all read-only arrays into flash, freeing a lot of RAM. I guess a 4MB build would probably be a good plan then.

Is it possible to calculate how many space is needed by save() in worst case?

You could argue it's JsVars * 16 bytes. But with E.setBootCode - assuming you had a way to actually write the code - you could upload a 1MB JS file if you wanted.

Great! So 1MB for save code and 1MB for additional flash space, that's what I am going to configure.

Got all puzzle pieces together.

• wifi save starts at 0x200000 with 1 page

• code save starts at 0x201000 with 255 pages

• getFreeFlash returns "addr": 3145728, "length": 3129344

https://github.com/MaBecker/Espruino/commit/81204d6b15ad2a6e08e02c7f741650c9d080a158

Hi @MaBe, 1MB for save code won't work, as far as I have seen in the code. The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM :)
Also, when we already have saved code in flash, and we try to save state later, the whole code in flash will try to go to the stack (alloca) temporarily ro reflash, which for sure is not good.
In my code you will find an implementation of alloca that will check if we have enough stack and at least return NULL if not enough room, instead of overflowing.
Regarding RESIZABLE_JSVARS, the code is already there for Linux, I did not reinvent the wheel :) The idea is to leave as much heap free as possible when not needed for JsVars (better encryption, more connections, etc), and allow it to grow on demand. Of course there is some work to be done, to make sure we always leave a minimum, but it is just an "if" in the provided malloc function, and a check for NULL in where the code calls malloc (I think I did not see this check in the LINUX code, but it is quite straight forward)

Hi @Gordon, my comments on RESIZABLE_JSVARS above, in my answer to @MaBe.

Regarding the unaligned access code, here it is. I can't remember where I found it first on the internet, so I cannot credit the original idea author (even if current code does not seem too much to the original one...).

By the way, it may need some adjustements to compile (this is just copied from another project of mine, and I have only done minor changes for it to compile in Espruino, but not really checked).

Important: change everywhere ## by the a single "## hash" caracter (unless somebedy explains me how to write a single "## hash" without it being interpreted by the forum as some special character...)

To activate the code, you need to call virtual_unaligned_memory_access() early in user_init()

 ##include "espmissingincludes.h"
 ##define RODATA __attribute__((section(".rodata")))
 ##define EXCCAUSE_LOAD_STORE_ERROR 3
 ##define EXCCAUSE_UNALIGNED 9
 ##define __stringify_1(x...) ##x
 ##define __stringify(x...) __stringify_1(x)
 ##define RSR(sr) ({ \
        uint32_t r; \
        __asm__ volatile("rsr %0," __stringify(sr) : "=a"(r)); \
        r; \
})

struct xtos_saved_regs {
        uint32_t pc;    /* instruction causing the trap */
        uint32_t ps;
        uint32_t sar;
        uint32_t vpri;  /* current xtos virtual priority */
        uint32_t a0;    /* when __XTENSA_CALL0_ABI__ is true */
        uint32_t a[16]; /* a2 - a15 */
};

typedef void (_xtos_handler_func)(struct xtos_saved_regs *);
typedef _xtos_handler_func *_xtos_handler;
_xtos_handler _xtos_set_exception_handler( int n, _xtos_handler f );

const char _trap_unaligned_debug_msg[] RODATA = "Unaligned access @%p [%08x %08x]: %p\n";
uintptr_t _trap_unaligned_last;
uint32_t _trap_unaligned_read;

static void CALLED_FROM_INTERRUPT unaligned_debug(uint8_t *pc, void *vaddr) {
        uint32_t *base = (uint32_t *)((uintptr_t) pc & ~0x3);
        ets_printf(_trap_unaligned_debug_msg, pc, base[0], base[1], vaddr);
}

uint8_t inline CALLED_FROM_INTERRUPT read_unaligned_byte(uint8_t *addr) {
        uintptr_t base = (uintptr_t) addr & ~0x3;
        if(base != _trap_unaligned_last) {
                _trap_unaligned_read = * (uint32_t *) base;
                _trap_unaligned_last = base;
        }
        return ((uint8_t *)&_trap_unaligned_read)[((uintptr_t) addr & 0x3)];
}

static void CALLED_FROM_INTERRUPT flash_emul_exception_handler(struct xtos_saved_regs *frame) {
        uint32_t vaddr = RSR(EXCVADDR);
        uint32_t instr = read_unaligned_byte((uint8_t *) frame->pc) | (read_unaligned_byte((uint8_t *) frame->pc + 1) << 8);
        uint8_t at = (instr >> 4) & 0xf;
        uint32_t val;
        uint32_t count;
        uint32_t u2s;

        if (vaddr < 0x40200000 /* 0x3ffe8000 */) {
                unaligned_debug((void *)frame->pc, (void *)vaddr);
                while(1);
        }

        if ((instr & 0xf00f) == 0x0002) {
                // |     imm8       |   r   |   as  |   at  |0 0 1 0|
                // l8ui at, as, imm    r:0
                frame->pc += 3;
                count = 1;
                u2s = 0;
        } else if ((instr & 0x700f) == 0x1002) {
                // |     imm8       |   r   |   as  |   at  |0 0 1 0|
                // l16ui at, as, imm   r:1
                // l16si at, as, imm   r:9
                frame->pc += 3;
                count = 2;
                u2s = (instr & 0x8000);
        } else if ((instr & 0xf00f) == 0x2002) {
                // |     imm8       |0 0 1 0|   as  |   at  |0 0 1 0|
                // l32i at, as, 0..1020
                frame->pc += 3;
                count = 4;
                u2s = 0;
        } else if ((instr & 0x000f) == 0x0008) {
                // |  imm4 |   s   |   t   |1 0 0 0|
                // l32i.n at, as, 0..60
                frame->pc += 2;
                count = 4;
                u2s = 0;
        } else {
                unaligned_debug((void *)frame->pc, (void *)vaddr);
                while(1);
        }

        val = 0;
        do {
                val |= read_unaligned_byte((uint8_t *) vaddr + --count);
                if(count) val <<= 8;
        } while(count);

        /* a0 and a1 are never used as scratch registers */
        frame->a[at - 2] = (u2s) ? (int16_t) val : val;
}

void virtual_unaligned_memory_access(void) {
        _xtos_set_exception_handler(EXCCAUSE_LOAD_STORE_ERROR, flash_emul_exception_handler);
        _xtos_set_exception_handler(EXCCAUSE_UNALIGNED, flash_emul_exception_handler);
}

Hi @Pedro,

1MB for save code won't work, as far as I have seen in the code. The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM :)

I fully agree, it's a waste of space.

>save();
=undefined
Erasing Flash.................................................................................................................................................................................................................................................................
Writing.....
Compressed 25600 bytes to 2224
Checking...

At the end it will be a solution that calculates JsVars * 16 byte and add the rest as FlashArea.

Regarding RESIZABLE_JSVARS .....The idea is to leave as much heap free as possible when not needed for JsVars

I definitely will try it, because this is too temptingly :)

It'd be worth checking that the firmware update over WiFi can still function in the 1MB/1MB config

The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM

However as I said above, you can potentially execute JS directly from flash, so if there were a way to write larger JS (eg in chunks, or with the bootloader) then it could be an option.

Wrt RESIZABLE_JSVARS - it's fun to play with, but I think it'd be a huge mistake to make the default builds use it. As I said above, the problems with flat string support could really trip people up - not to mention issues with garbage collection (I'm not sure when that happens currently).

There is a branch for 'variable cache' which IMO would be the most exciting if it had a little work done on it. It would use the RAM only for most-used variables, and would journal the rest in flash - giving you potentially 65000 variables.

@Pedro

It'd be worth checking that the firmware update over WiFi can still function in the 1MB/1MB config

Here is the code for the over the air updates... there is a flash map address look up:
https://github.com/espruino/Espruino/blob/6da2ddd05917d98e0693a084a2d64fcfb8b800d1/libs/network/esp8266/ota.c#L171

This all sounds interesting, I have some questions:

Where are wifi settings stored? Are they now stored twice and thus have to be reset after upgrading?

What about saving javascript code to flash, is that now lost when upgrading? And "EEPROM" settings?

Hi @user74245,

• switch to new firmware requires erase_flash

• code and wifi save check link in #34

Hi @Gordon, I agree that if RESIZABLE_JSVARS is not stable, and until it is, the default builds should not use it, but I would definetively suggest to leave it there so that it can be compiled directly as an option from the main branch. Otherwise keeping it up to date and bring it up to stable would be a nightmare.
Just one question : I understand from your comment that it is not really working in LINUX ? Regarding "flat string support", do you mean that not everywhere in the code strings are accessed thorugh iterators ? (sorry if this is a naive question).

At least a couple of my code changes are bug fixes for the existing LINUX RESIZABLE_JSVARS part and should anyway be applied. What is the best way to get this done?

Regarding the "variable cache" branch, it is a different topic as I understand it in your comment : RESIZABLE_JSVARS leaves as much heap as possible free when not used for JSVars, and "variable cache" would allow (as you speak of flash cache) to have more JSvars than would fit on RAM. They are two related things, but different in approach and implementation (and complementary to each other in some way).
By the way, wouldn't this kind of "live" jsvar storage in flash kill it really fast ? (too many refresh cycles) I rendered one ESP8266 unusable by leaving it only one night in a reboot cycle that wrote to the system config area at every reboot.

I would definetively suggest to leave it there so that it can be compiled directly as an option from the main branch

It's not going anywhere. Perhaps the build_platform_config.py file could be modified to accept 0 JsVars to mean resizable? That way it's an easy tweak.

I understand from your comment that it is not really working in LINUX ?

No, it does work. However Linux can afford to have quite a big chunk size which means that flat strings can usually be allocated.

Regarding "flat string support", do you mean that not everywhere in the code strings are accessed through iterators ?

Usually they are, but if a big area of memory is needed then sometimes it'll be allocated out of a stretch of JsVars. If the block size for resizable jsvars is too small then that may not happen. At least on Linux for things like TLS/HTTPS it won't crash, but the command will fail if a flat string can't be allocated.

Also flat strings are usually allocated for types arrays to speed up accesses - if they can't be allocated then the access speed for arrays will drop right down.

At least a couple of my code changes are bug fixes for the existing LINUX RESIZABLE_JSVARS part and should anyway be applied. What is the best way to get this done?

That's great - if you could create a branch on your GitHub for those fixes and could issue a Pull Request for them that'd be great.

But it helps a great deal to submit changes in several small groups. If something gets broken it's easy enough to look back and see exactly why things were changed then - if it's part of a massive commit you have no idea what changes go with what improvement.

wouldn't this kind of "live" jsvar storage in flash kill it really fast ?

I'm not sure if it's done yet or not, but the idea was that the variable_cache branch would use journalling when writing to flash. The sale way the FlashEEPROM module works at the moment - it's actually pretty kind on flash memory.

Also in my tests I found that the majority of code didn't cause writebacks - most often you're just loading code in to execute, and aren't changing anything so have no need to write to flash.