1. Home
  2. Other Boards
  3. ESP8266

E.memoryArea not working on ESP8266 (due to char-iterator ?)

Posted on
Page
of 2
/ 2
Next
  • • #1

    Tested this command on actual Esp8266 firmware.
    Going down to assembler listing it breaks using an 8 bit instruction in iterator.
    There could be a problem in reading bytes from Flash, reading works fine for aligned 32bits only (afaik).

  • • #2

    Yes - I think this is something we'd have to look into. It might be worth seeing if you could just tweak jsvStringIterator to read 32 bits at a time and see if that fixes it? (you'd have to change jslex.c too)

    I guess seeing if you could use peek32 to access flash would be a quick way to test whether it was that easy.

  • • #3

    peek32 works fine on flash
    there only needs to be an address translation (0x71000 to 0x40271000)
    peek16 and peek8 don't work and create an error

    Sorry my knowledge is too poor to make the changes by myself :-(
    Learned a lot, but I'm still far away from changes like this.
    See this, thats my level right now, somewhere around copy/paste from already existing sources.

  • • #4

    Out of curiosity, what are you trying to do with E.memoryArea? And where is it documented, I don't see it in http://www.espruino.com/Reference ?

  • • #5

    I think E.memoryArea is only in recent builds, so won't be in the reference (generally the reference matches the latest 'stable' version).

    The idea is that you can put strings/arrays into flash, and can even execute code direct from it - I think this was something you'd actually asked for at some point?

    Potential uses:

    • On Pico - storing keys/certificates for TLS
    • Storing modules (still not quite sure why you'd want this)
    • Storing things like Waveform or Bitmap data
    • You can even define a function and put its contents into flash
  • • #6

    In jswrap_espruino.c line 700 and following is a description for E.memoryArea.
    In my understanding this creates an additional kind of string for jsvar.
    It holds start in memory(flash) and length of string.

  • • #7

    Based on a lot of copy/paste, trial and error, E.memoryArea works for strings.
    function jsvStringIteratorGetChar in jsviterator.h is extend to handle native string

    static ALWAYS_INLINE char jsvStringIteratorGetChar(JsvStringIterator *it) {
  if (!it->ptr) return 0;
  if (jsvIsNativeString(it->var)){
    int addr = (int)it->ptr + (int)it->charIdx;
	uint32_t bytes = *(uint32_t*)(addr & ~3);
	return ((uint8_t*)&bytes)[(int)addr & 3];
  }
  else return it->ptr[it->charIdx];
}

    In next step function jslNextCh in jslex.c is extended too and first test calling a simple function works fine.

    static ALWAYS_INLINE char jslNextCh(JsLex *lex) {
  if(jsvIsNativeString(lex->it.var)){
	int addr = (int)lex->it.ptr + (int)lex->it.charIdx;
	uint32_t bytes = *(uint32_t*)(addr & ~3);
	return ((uint8_t*)&bytes)[(int)addr & 3];
  }
  else return (char)(lex->it.ptr ? lex->it.ptr[lex->it.charIdx] : 0);
}

    Pretty sure there is a nicer way doing this, and we will see this in next version.
    Anyway, it opened a new door of knowledge for me.

  • • #8

    That's great! So that works for you on ESP8266?

    Personally I'd just #ifdef ESP8266 and access by word all the time (it'll be faster than checking if you're using a native string!).

  • • #9

    @tve I know you're doing some benchmarking at the moment? Please could you try:

    static ALWAYS_INLINE char jsvStringIteratorGetChar(JsvStringIterat­or *it) {
  if (!it->ptr) return 0;
~ifdef ESP8266
  int addr = (int)it->ptr + (int)it->charIdx;
  uint32_t bytes = *(uint32_t*)(addr & ~3);
  return ((uint8_t*)&bytes)[(int)addr & 3];
~else
  return it->ptr[it->charIdx];
~endif
}

// and

static ALWAYS_INLINE char jslNextCh(JsLex *lex) {
  if (!lex->it.ptr) return 0;
~ifdef ESP8266
  int addr = (int)lex->it.ptr + (int)lex->it.charIdx;
  uint32_t bytes = *(uint32_t*)(addr & ~3);
  return ((uint8_t*)&bytes)[(int)addr & 3];
~else
  return (char)(lex->it.ptr[lex->it.charIdx]);
~endif
}

// ~ should be a hash - the forum 'eats' it otherwise

    And see if it hurts performance much? My hope is that actually it won't have that much of an effect at all.

    If not, it'd be great to get it in as it could really help with the low RAM situation...

  • • #10

    OK. This week looks very busy for me, so it will have to wait a few days.

  • • #11

    I did some testing with new function to use strings in flash.
    Testcase is a SHA1-function and an server-object in WebIDE.
    For testing a simple function calls SHA1 and creates an object.

    • sent to the board with minification free:985
    • checked memory some seconds later free:1168
    • called testing function free:1101, duration 553ms
    • checked memory some seconds later free:1133

    Next testcase has SHA1-function and server-object in Flash.
    Both, function and object are executed with eval from Flash.

    • sent to the board with minification free:1156
    • checked memory some seconds later free:1174
    • called testing function free:1113, duration 500ms
    • checked memory some seconds later free:1144

    Result from this test(IMHO) is:

    • sending from WebIDE takes some memory which is free later.
    • at the end memory consumption does not change much.
    • source from Flash executes 10% faster, no idea why
    • reading sources from flash can be helpful for state oriented apps, where functions/objects are not needed all the time. Not used functions could then be dropped and others could be reloaded. LUA on ESP8266 supports something like this.
    • I would like to have this function, especially for ESP8266 where memory is a problem
  • • #12

    One more test.
    This time the SHA1 was copied to flash without function body.
    All variables have been set to undefined at the end of source.
    2 more global variables are defined, one for input data, one for the result.

    • sent to board with minification free:1330
    • checked memory some seconds later free:1347
    • called test function free:1286, duration 500ms
    • checked memory some seconds later free:1292
    • called test function several times, no change in free memory
  • • #13

    Interesting tests, but a bit difficult to follow without any code snippets. For example, I don't know what you mean by "This time the SHA1 was copied to flash without function body." If you didn't put the SHA1 function body in flash, then what did you put in flash?

    I can see how to execute code from a memory area, what I'm not yet seeing is how to put it all together. One option could be to have a special method to load a bunch of source code into a flash area and tell the interpreter: here, load this. But that now creates a completely separate path from the way the IDE or command line tool work to load code.

    Or we could modify some stuff in jsinteractive that puts code into a memory area by default. But now we need to create a memory allocator for the flash area that can also deal with code being replaced, i.e. areas being freed, etc. On top of that then comes save() which somehow needs to make sure that whatever is in RAM that points to the flash is saved (perhaps that's in a JSvar so automatically gets saved, I haven't looked). There's probably a simple solution that goes 90% of the way, like dropping the flash area on reset() and only adding to it, never freeing until the next reset().

  • • #14

    Let me try to give a better explanation. Its more or less a proof of concept.
    The idea behind is to use it like a "firmware-extension".
    First of all, I made some changes to Espruino firmware:

  • • #15

    I've just added some code to handle this here I'm not able to test on my ESP8266 for some reason so it'd be great if someone could take a look at the builds and see if they still work ok.

  • • #16

    Hi Gordon,
    Is there any easy way to tell which builds now have the E.memoryArea update for reading esp8266 flash?
    I don't think this is any @tve builds yet?

  • • #17

    You could look at the ChangeLog on @tve's GitHub and figure out which commits have the changes in (if any). Otherwise the main travis builds for Espruino are auto-built and should be usable.

    According to the comments on this bug it may not work though. I'd kind of assumed that after all the requests someone might have tested it, but it seems not :(

  • • #18

    Ah - I'll try one of the auto builds this weekend and report back if it works.
    thanks

  • • #19

    Just tested with sources from github, for me it works fine.
    Also tested E.memoryArea, reading a string from flash and executed with eval, works fine too.

    peek32(0x3ff00054).toString(16)
="200991a"
>peek16(0x3ff00054).toString(16)
="991a"
>peek16(0x3ff00056).toString(16)
="200"
>peek8(0x3ff00054).toString(16)
="1a"
>peek8(0x3ff00055).toString(16)
="99"
>peek8(0x3ff00056).toString(16)
="0"
>peek8(0x3ff00057).toString(16)
="2"
  • • #20

    That's awesome - thanks for checking! Nice to know it is all working properly, at least in some builds :)

  • • #21

    I can confirm the peek stuff above works, however I get a reset with E.memoryArea trying to read the Esp8266 flash:

    >esp8266 = require("ESP8266");
=function () { [native code] }
>var base=esp8266.getFreeFlash()[2].area;
=1011712
>E.memoryArea(base,10);
="
 ets Jan  8 2013,rst cause:2, boot mode:(3,6)
load 0x40100000, len 1396, room 16
tail 4
chksum 0x89
load 0x3ffe8000, len 776, room 4
tail 4
chksum 0xe8
load 0x3ffe8308, len 540, room 4
tail 8
chksum 0xc0
csum 0xc0
2nd boot version : 1.4(b1)
  SPI Speed      : 80MHz
  SPI Mode       : DIO
  SPI Flash Size & Map: 32Mbit(512KB+512KB)
jump to run user1 @ 1000
don't use rtc mem data

    >process.env
={
  "VERSION": "1v84",
  "BUILD_DATE": "Feb 12 2016",
  "BUILD_TIME": "08:57:53",
  "GIT_COMMIT": "8d4df7829bc7ebc8bc701c0f04e2fc645b99bd02",
  "BOARD": "ESP8266_BOARD",
  "CHIP": "ESP8266",
  "CHIP_FAMILY": "ESP8266",
  "FLASH": 0, "RAM": 81920,
  "SERIAL": "5ccf7f00-d975",
  "CONSOLE": "Serial1",
  "EXPORTS": { "jsvLock": 1075994728, "jsvLockAgainSafe": 1075994712, "jsvUnLock": 1075981000, "jsvSkipName": 1075970120,
    "jsvMathsOp": 1075935700, "jsvMathsOpSkipNames": 1075960212, "jsvNewFromFloat": 1076000164, "jsvNewFromInteger": 1076000232, "jsvNewFromString": 1075999068,
    "jsvNewFromBool": 1076000200, "jsvGetFloat": 1075970588, "jsvGetInteger": 1075971844, "jsvGetBool": 1075971036, "jspeiFindInScopes": 1075910904,
    "jspReplaceWith": 1075904248, "jspeFunctionCall": 1076010408, "jspGetNamedVariable": 1075912700, "jspGetNamedField": 1075923460, "jspGetVarNamedField": 1075922944,
    "jsvNewWithFlags": 1075999608 }
 }

    It works with the 0x3ff00054 address:

    >peek32(0x3ff00054).toString(16)
="20000d9"
>E.memoryArea(0x3ff00054,4);
>E.memoryArea(0x3ff00054,4);
="\xD9\x00\x00\x02"

    So this seems to be associated with the address....

    https://github.com/espruino/Espruino/wiki/ESP8266-Design-Notes

  • • #22

    Flash in ESP8266 is mirrored (windowed?) to 0x40200000.
    So if you want to use E.memoryArea for 1011712 (0xf7000) please check 0x0402f7000.
    BTW, you could check for free flash, since value given by esp8266.getFreeFlash() returns a value which is designed to be free. If firmware does not use all of reserved memory for firmware, there is some more available flash, right now starting at 0x72000(0x40272000)
    One more BTW, for more confusion, flash.read does adress calculation internally.

    flash = require("Flash");
function mem(){
  for(var i = 0x70000;i < 0x80000;i+=4096){
    console.log(i.toString(16),flash.read(8,i));
  }
}

    if this returns 255 for all fields, usually this is available flash

  • • #23

    hmm, so could it be that the READ_UINT8 macro won't work graphics (As in @MaBe's bug) because the address used is wrong? That could complicate things :(

    I guess we'd need a second Macro with if (addr < RAM+ADDR) addr|=0x40200000;

  • • #24

    Thanks. Using the offset works ok- I'm wondering if there is a way of hiding this transparently like Flash() lib does?

    I was expecting the esp8266.getFlash() to return these larger regions, it doesn't look like this is implemented yet.

    I also believe these is a restriction of 1mb max, and that this is in the espurino core rather than the esp8266 implementation. Does anyone know why this is the case?

  • • #25

    if (addr < RAM+ADDR) addr|=0x40200000

    That makes sense, and would explain it.

Page
of 2
/ 2
Next
  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
Post a reply
About

E.memoryArea not working on ESP8266 (due to char-iterator ?)

Posted by Avatar for JumJum @JumJum

Actions

© Espruino, powered by microcosm.

About Terms of Use Privacy Policy Cookie Policy Report a problem