Search results for:

Hi @Gordon, I agree that if RESIZABLE_JSVARS is not stable, and until it is, the default builds should not use it, but I would definetively suggest to leave it there so that it can be compiled directly as an option from the main branch. Otherwise keeping it up to date and bring it up to stable would be a nightmare.
Just one question : I understand from your comment that it is not really working in LINUX ? Regarding "flat string support", do you mean that not everywhere in the code strings are accessed thorugh iterators ? (sorry if this is a naive question).

At least a couple of my code changes are bug fixes for the existing LINUX RESIZABLE_JSVARS part and should anyway be applied. What is the best way to get this done?

Regarding the "variable cache" branch, it is a different topic as I understand it in your comment : RESIZABLE_JSVARS leaves as much heap as possible free when not used for JSVars, and "variable cache" would allow (as you speak of flash cache) to have more JSvars than would fit on RAM. They are two related things, but different in approach and implementation (and complementary to each other in some way).
By the way, wouldn't this kind of "live" jsvar storage in flash kill it really fast ? (too many refresh cycles) I rendered one ESP8266 unusable by leaving it only one night in a reboot cycle that wrote to the system config area at every reboot.

Hi @Gordon, my comments on RESIZABLE_JSVARS above, in my answer to @MaBe.

Regarding the unaligned access code, here it is. I can't remember where I found it first on the internet, so I cannot credit the original idea author (even if current code does not seem too much to the original one...).

By the way, it may need some adjustements to compile (this is just copied from another project of mine, and I have only done minor changes for it to compile in Espruino, but not really checked).

Important: change everywhere ## by the a single "## hash" caracter (unless somebedy explains me how to write a single "## hash" without it being interpreted by the forum as some special character...)

To activate the code, you need to call virtual_unaligned_memory_access() early in user_init()

 ##include "espmissingincludes.h"
 ##define RODATA __attribute__((section(".rodata")))
 ##define EXCCAUSE_LOAD_STORE_ERROR 3
 ##define EXCCAUSE_UNALIGNED 9
 ##define __stringify_1(x...) ##x
 ##define __stringify(x...) __stringify_1(x)
 ##define RSR(sr) ({ \
        uint32_t r; \
        __asm__ volatile("rsr %0," __stringify(sr) : "=a"(r)); \
        r; \
})

struct xtos_saved_regs {
        uint32_t pc;    /* instruction causing the trap */
        uint32_t ps;
        uint32_t sar;
        uint32_t vpri;  /* current xtos virtual priority */
        uint32_t a0;    /* when __XTENSA_CALL0_ABI__ is true */
        uint32_t a[16]; /* a2 - a15 */
};

typedef void (_xtos_handler_func)(struct xtos_saved_regs *);
typedef _xtos_handler_func *_xtos_handler;
_xtos_handler _xtos_set_exception_handler( int n, _xtos_handler f );

const char _trap_unaligned_debug_msg[] RODATA = "Unaligned access @%p [%08x %08x]: %p\n";
uintptr_t _trap_unaligned_last;
uint32_t _trap_unaligned_read;

static void CALLED_FROM_INTERRUPT unaligned_debug(uint8_t *pc, void *vaddr) {
        uint32_t *base = (uint32_t *)((uintptr_t) pc & ~0x3);
        ets_printf(_trap_unaligned_debug_msg, pc, base[0], base[1], vaddr);
}

uint8_t inline CALLED_FROM_INTERRUPT read_unaligned_byte(uint8_t *addr) {
        uintptr_t base = (uintptr_t) addr & ~0x3;
        if(base != _trap_unaligned_last) {
                _trap_unaligned_read = * (uint32_t *) base;
                _trap_unaligned_last = base;
        }
        return ((uint8_t *)&_trap_unaligned_read)[((uintptr_t) addr & 0x3)];
}

static void CALLED_FROM_INTERRUPT flash_emul_exception_handler(struct xtos_saved_regs *frame) {
        uint32_t vaddr = RSR(EXCVADDR);
        uint32_t instr = read_unaligned_byte((uint8_t *) frame->pc) | (read_unaligned_byte((uint8_t *) frame->pc + 1) << 8);
        uint8_t at = (instr >> 4) & 0xf;
        uint32_t val;
        uint32_t count;
        uint32_t u2s;

        if (vaddr < 0x40200000 /* 0x3ffe8000 */) {
                unaligned_debug((void *)frame->pc, (void *)vaddr);
                while(1);
        }

        if ((instr & 0xf00f) == 0x0002) {
                // |     imm8       |   r   |   as  |   at  |0 0 1 0|
                // l8ui at, as, imm    r:0
                frame->pc += 3;
                count = 1;
                u2s = 0;
        } else if ((instr & 0x700f) == 0x1002) {
                // |     imm8       |   r   |   as  |   at  |0 0 1 0|
                // l16ui at, as, imm   r:1
                // l16si at, as, imm   r:9
                frame->pc += 3;
                count = 2;
                u2s = (instr & 0x8000);
        } else if ((instr & 0xf00f) == 0x2002) {
                // |     imm8       |0 0 1 0|   as  |   at  |0 0 1 0|
                // l32i at, as, 0..1020
                frame->pc += 3;
                count = 4;
                u2s = 0;
        } else if ((instr & 0x000f) == 0x0008) {
                // |  imm4 |   s   |   t   |1 0 0 0|
                // l32i.n at, as, 0..60
                frame->pc += 2;
                count = 4;
                u2s = 0;
        } else {
                unaligned_debug((void *)frame->pc, (void *)vaddr);
                while(1);
        }

        val = 0;
        do {
                val |= read_unaligned_byte((uint8_t *) vaddr + --count);
                if(count) val <<= 8;
        } while(count);

        /* a0 and a1 are never used as scratch registers */
        frame->a[at - 2] = (u2s) ? (int16_t) val : val;
}

void virtual_unaligned_memory_access(void) {
        _xtos_set_exception_handler(EXCCAUSE_LOAD_STORE_ERROR, flash_emul_exception_handler);
        _xtos_set_exception_handler(EXCCAUSE_UNALIGNED, flash_emul_exception_handler);
}

Hi @MaBe, 1MB for save code won't work, as far as I have seen in the code. The main reason is that saved code must be first allocated somewhere in RAM, and we do not have so much RAM :)
Also, when we already have saved code in flash, and we try to save state later, the whole code in flash will try to go to the stack (alloca) temporarily ro reflash, which for sure is not good.
In my code you will find an implementation of alloca that will check if we have enough stack and at least return NULL if not enough room, instead of overflowing.
Regarding RESIZABLE_JSVARS, the code is already there for Linux, I did not reinvent the wheel :) The idea is to leave as much heap free as possible when not needed for JsVars (better encryption, more connections, etc), and allow it to grow on demand. Of course there is some work to be done, to make sure we always leave a minimum, but it is just an "if" in the provided malloc function, and a check for NULL in where the code calls malloc (I think I did not see this check in the LINUX code, but it is quite straight forward)

Happy it is being useful :)

Just a couple of remarks:

• No need for two different linker scripts (app1 and app2). By definition in this mode both images should be identical (which is indeed a great advantage), so we only need to generate one, and if we want to keep two for consisntency, I would point both linker scripts to the same file.
  
• We need to check the upper limit. In the "jswrap esp8266 network.c" file you pointed before as having a hardcoded flash address, I see it is pointing to the last page, looks like) so we may need to adjust the linker script to allow for one page less size, and of course adding 1Mb to the flash address if we are in image 2 (in runtime, getting the base flash address from the corresponding sdk os_* function)
  

Hi @MaBe,

If I did not get anything wrong :) here's the branch : https://github.com/pedro-es/Espruino/tree/esp8266-enhancements

I only did one commit for all the files, as they are interrelated. The comments for the commit covers all.

@MaBe, @Wilberforce, how do I create a new branch ? (or do I just fork into my repository). As commented, not a expert github user (mostly a newby here)

In the Makefile:

ESP_FLASH_SIZE ?= 6
ET_FS ?= 32m-c1

And in user_main.c, change (to avoid a warning):

if ((chip == 0x4013 && map != 0) || (chip == 0x4016 && map != 4)) {

to

if ((chip == 0x4013 && map != 0) || (chip == 0x4016 && map != 4 && map != 6)) {

As soon as I have created the branch I will submit the changes for review (including this in the makefile with the proper conditionals)

In this mode, user1 and user2 bin files are identical (no different ld files) and the bootloader maps the first or second MB of flash to 0x40200000. As commented before, I need to check in the existing code that there is nothing hardcoded in current espruino build for ESP8266 when referencing flash from the first or second image

Yes it is. Initially only 128 jsvars are allocated, and it grows as needed.

You are right. For the 4MB map we do not need a new board. For RESIZABLE_JSVARS, as it is behaving differenty as the standard ESP8266 current version, I wanted to make sure not to mess with it and create a new board. To avoid "forking" then new board just includes the standard ESP8266_BOARD.py and changes a few parameters. If there is a cleaner way, will be happy to do it.

Thanks @Wilberforce, will try to branch the code so that it can be reviewed.
Regarding your comment on free heap, I just pushed it to the limit (2048 jsvars dynamically allocated). I agree at some point I need to put some check to keep a minimum heap available, failing to create a new jsvar block, but the result should be the same : memory full / bad things happen :)
Currently new jsvars are allocated in blocks of 128, so it does not go directly from 31KB heap to 3KB heap.

Thanks @MaBe, I am following @Wilberforce instructions on to create a branch so that the code cen be checked, and once I get feedback and more testing I will do a pullt request. Need to investigate how to do it : not a git master :) even if I remember doing something similar to patch Linux kernel in the past.
For the two board switch, the main issue is that platform_config.h and some ifdefs in the code must behave differently (for the RESIZABLE_JSVARS part), so I did something like the past two board switch, but the whole thing needs to be recompiled (not just relinked to get the different firmwares), so it is somehow different. Will try to submit today so that you can check it.

Hi all,

This looks like a great piece of work, and I am migrating some personal project to Espruino with my ESP cards.

For everything to work fine, I needed some enhancements, so I have done them (on v91), and I think it woud be worth sharing so that these can be merged into the public version:

• Resizable vars : the code is there, for the Linux version. Just patched a bit to enable it in ESP (should also work for other cards now) and fixed also some minor bugs on the save to / load from flash code (as it was, I think it would also fail on Linux, as with RESIZABLE_JSVARS the vars are stored in blocks, while the compress functions, as they were, read and wrote a full contiguous block of memory)

• 1024Kb + 1024Kb memory map : Added few lines to the Makefile to create a new target for the ESP 4Mbyte cards. With this layout, now we have 1024KB for firmware and stored vars and functions. Still neeeded to check where to fix some address mapping for the second image, as in this mode the bootloader maps 0x4020000 to either the first or second Mb of flash, and the image itself is exactly identical for user1 and user2. Just need to check in existing code if there is something harcoded around flash address and fix it.

So now, I get:

reset();
=undefined
... [ Espruino boot banner ] ...
Flash map 4MB:1024/1024, manuf 0xe0 chip 0x4016
Flash code area: 0x76000 (565248 bytes)

process.memory();
={ "free": 106, "usage": 22, "total": 128, "history": 0 }

require("ESP8266").getState();
={
"sdkVersion": "2.0.0(5a875ba)",
"cpuFrequency": 160, "freeHeap": 34232, "maxCon": 10,
"flashMap": "4MB:1024/1024",
"flashKB": 4096,
"flashChip": "0xe0 0x4016"
}

var tester = [];
=[ ]

for(var i=0 ; i < 900 ; i++) tester[i] = i;
=undefined

process.memory();
={ "free": 88, "usage": 936, "total": 1024, "history": 6 }

var more = [];
=[ ]

for(var i=0 ; i < 900 ; i++) more[i] = 2*i;
=undefined

process.memory();
={ "free": 210, "usage": 1838, "total": 2048, "history": 6 }

require("ESP8266").getState();
={
"sdkVersion": "2.0.0(5a875ba)",
"cpuFrequency": 160, "freeHeap": 3344, "maxCon": 10,
"flashMap": "4MB:1024/1024",
"flashKB": 4096,
"flashChip": "0xe0 0x4016"
}

require("ESP8266").reboot();
=undefined
... [ ESP boot info and some garbage :) ] ...
Loading 2048 jsvars from flash...

dump();
var tester = [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35,;
var i = 900;
var more = [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34, 36, 38, 40, 42, 44, 46, 48, 50, 52, 54, 56, 58, 60, 62, 64, 66, 68, ;
pinMode(D0, "input_pullup", true);
pinMode(D4, "input_pullup", true);
pinMode(D5, "input_pullup", true);
pinMode(D12, "input_pullup", true);
pinMode(D13, "input_pullup", true);
pinMode(D14, "input_pullup", true);
pinMode(D15, "input_pullup", true);
=undefined

TO DO:

• More testing
  
• Take benefit of this extra memory (make sure we can access code directly on it)
  
  

For the second part, what I use is as small piece of code based on something I found a couple of years ago in Internet and which I enhanced bit, so that ESP can access directly the flash mapped memory independently of alignment, or size (1, 2, 4 bytes) with no further modification to the user code. This is possible because ESP8266 is a bit of a monster, and allows virtualizing memory access through HW catchable exceptions for this kind of memory access errors, so a trap function can directly see the failing opcode and target address, access flash with the right alignment, simulate the opcode and put the result on the right register, and return control to the program (at assembler opcode level) after the offending opcode address.

This is seamless to your code, that can now access memory mapped flash as if it was in the rodata area. Of course, there is an impact in speed, which I have not benchmarked, but not noticed either, as we are not using our ESP8266s to do math intensive calculations (are we?)

Will work on other improvements.

So, if there is interest on it, I have a bunch of ".patch" files I will be happy to submit, or share, or whatever is the best way to get these into the main build.