MQTT on ESP8266 with Espruino

Posted on
  • Hi guys,

    Actually it seems that using this MQTT module: on the ESP8266 (with espruino on board), causes an ERROR: Out of Memory.

    Someone can give me some hints on how to fix that?


  • Hi Rocco,

    Your best bet for now might be to use the Web IDE's 'projects' feature to set up a local directory, and to then copy into there, and to try and delete any functions you don't need.

  • LOL - but no insult... let's take a Piagio Ape or Citroën Fourgonnette, or may be just many of them, and build Hoover Dam... or go to the moon... or mars...

    I'm not saying it is not possible, but just becuase the peg actually fits the hole (shape) does not say if it really 'fits'. (It is like me at the time when digital cameras came up and I had gazillions of 3-1/2 floppy disks and Sony had a digital camera that could use those to store the images... everyone - except obviously me - kows that image quality/resolution has something to do with storage capacity and speed to put it to and get it back from storage... The idea fell pretty quickly out of favor... Yes, at that time pictures had 500...700KB... 2 pics / media... 1.3MPixels and quality defeating compression,... not to talk about 'burst rate' or 'movie shooting').

    ESP8266 - in its current capacity - may not be the platform for everything - everything very convenient of general... The very defined - constrained - resource set asks for an adequate, dedicated use.

  • @allObjects Are you saying that the @Gordon 's solution can't work?

    MQTT works well on both the NodeMCU firmware and the Arduino firmware on ESP8266.

  • It will hopefully work, but it might be a bit tight. It should be possible to get significantly more code into the ESP8266 (currently only 12k is used for code/variables) but right now the ESP8266 port is still under heavy development, so I don't think anyone has had much of a chance to play around and see what can actually be fitted in.

  • I'm afraid that the current 1023 JsVars is the absolute max you can fit in there, unless you remove wifi. 1023 JsVars only really work with the lwip_536 library that uses 536 byte packets (instead of 1460). At least that's my experience.

  • Wonder if the MQTT module was stripped back so that it could run inline with the rest of the code rather than be loaded via require(...) if compilation might help?

    Shot in the dark from me but @Gordon will be able to confirm/refute. Is compilation just a speed thing or are there memory saving advantages to be had that might help here?

  • Ouch. I wonder what's using the rest of the RAM then? I wonder if the find_big_ram script works on xtensa...

    I just committed something that should help - Espruino now doesn't store { and } in functions any more. It breaks formatting in dump() slightly but it should make things a bit more efficient.

    @roccomuso where do you get the out of memory error? during upload, or at some point after?

    MQTT works on NodeMCU and Arduino because AFAIK it's compiled as native code, not written in JavaScript.

  • Well, actually uploading the module inline with the rest of the code could really help if it's an out of memory you get during upload.

    Compiling code doesn't have quite as much of an effect as you'd think. From my random tests it looks like minified code is about as compact (if not better than!) compiled code as far as code size goes.

  • Is there a way to save the "code" (i.e. the JavaScript statements to be interpreted) into flash memory? I think as I understand it today, the JavaScript statements are saved in JsVars in RAM for interpretation at runtime.

  • There was talk a while back of storing modules in text form in flash, but when loaded they'd still have to go into RAM - so it depends on where @roccomuso is getting his error.

    It's a lot of work to work around though. Time would be much better spent figuring out where all the RAM has gone in ESP8266 and fixing that.

  • I loaded just the MQTT.js file and seem to find that has used about 900 blocks of JsVar. I have attached the file showing the trace. I am not good at reading these but I seem to be seeing that the majority of blocks (that I can see) are used to hold JavaScript text ...

    The source file of MQTT.js is about 8K.

    The amount of RAM in the ESP8266s today is very low. At best, with absolutely nothing but WiFi and TCP loaded, Espressif tell us that we can use about 40K. If we then subtract from that 1024 JsVars ... I think that takes us down to about 28K. I think when Espruino boots from fresh on an ESP8266, the heap left is about 9K and from that, the TCP/IP memory buffers need to be allocated. However there appears to be about 17K un-accounted for that is probably worthy of short term investigation.

    My hope though is that within 6 months (maybe more, maybe less) the RAM available will increase by a 10x factor and hopefully that will push the problem to the far right.

    1 Attachment

  • Can you try with a new build of Espruino from the last few minutes? the situation may have changed now :)

  • @roccomuso, I'm not saying it is not possible, but as @Gordon mentions: it get's tight with current implementations.

    @Kolban, inpressive the list and details about RAM figures.

    @Gordon, last time you mentioned executing code from EEPROM, I understood it as 'exectute directly from ROM'. Is that not an option at all? I know that some information about the code needs to be in the RAM, such as symbolic entry points and their (EEPROM) address, vars, etc. *'jump' addresses for conditional and loop consturcts, etc... What else more would have to go into the RAM? Is it too far fetched to make that step from source-interpretation of source in RAM to source-interpretation to source in EPROM? May be the overhead eats up (almost all) the yield...?

  • You can exec from flash, but the second you define a function, that has to be in RAM.

    @Kolban 17k RAM would more than double the amount of variables available - so definitely worth looking into :)

  • I built a new set of firmware using the latest code base ... attached is the new trace and memory usage. It looks like the changes made by @Gordon seem to have saved 180 blocks.

    Will upload a new binary/firmware set for the ESP8266 now.

    As for the "missing" 17K ... ...

    After booting with Espruino - free heap = ~5K
    Known JsVars = 1024*x 16 = ~16K
    Total known = 16K + 5K = ~21K
    ESP8266 Booted with no firmware but WiFi and TCP = ~40K
    Un-accounted for RAM = 40K - 21K = ~19K

    My first impression is ... and this is a guess ... that String constants are placed in RAM. So if in C we code:

    char *x = "hello";


    if (strcmp(x, "hello")) ...

    Then the string constants eat RAM. I am "guessing" that some large portion of the missing RAM is used by strings.

    For example, I looked at jsinteractive.o uses 2100 bytes and jsparse.o uses 1900 bytes and there are likely other hitters out there too. I don't know enough to know if static character string can be moved to flash or instruction RAM but I'm going to guess not.

    1 Attachment

  • Tested new version(1v81.705) immediately :-)
    For websocket SHA1 is needed, this was my choice to compare benefit:

    • old SHA1 duration was 1390ms and process.memory().free after calling was 735
    • after a lot of changes it's 1047ms and 823 blocks are free
    • now with new firmware version it takes 1290ms and 866 blocks are free
      So for this function it uses less memory (20%) and is slower (20%), interesting result.
  • ...guess the 1x1 ~~ 0.8x1.2 rule of time or memory still goes... silver lining: 4% better... if time is not really a challenge, it becomes a go for a no-go: +1000%.

  • I've build the latest Espruino code base, but this time it seems taking muuuch time to upload the sketch( 10 times slower ) and then it gives me out of memory while uploading.

    I've tried to copy the mqtt module inline and was able to upload it but then it gave me error wifi not defined

    Uncaught ReferenceError: "wifi" is not defined
     at line 1 col 5
  • @Kolban actually at 1024 vars, Espruino uses 12 byte vars so the situation is even worse.

    Very interesting about the strings - that is quite likely (it's what Arduino does). Changes to the Linker script could sort that out though.

    @JumJum thanks for the benchmarks! That's a much worse result than I was expecting in terms of speed :( The only thing that should be slower is function creation - execution should be faster. Do you think you could post up a small section of code I could benchmark on Pico builds?

    @sameh.hady maybe this is the first time you've used the new builds with the new WiFi API?

  • @Kolban the guys from nodemcu had the same problem and fixed it. Heap doubled from one day to the other.
    @Gordon source is available here­A1.js.
    I did some (crazy) things to reduce memory and speed, which did not make it easier to read:

    • use typed arrays
    • remove temporary values
    • replace switch by if...elseif...
    • removed functions and added code inline
    • reuse local vars
    • converted multiplication(like *4, *32,..) to shift
    • removed a "funny" encode64 and use btoa instead
    • moved functions to be local
      All together my plan is to use ESP8266 for web, and let Espruino boards do the hard work.
  • I just tried that code (ran it 10 times just to be sure) and the results are:

    • 1v80 - 3.83 sec
    • absolute latest - 3.80 sec, 158 used
    • latest, but before flat string changes - 3.79 sec, 193 used

    So there is a very small speed difference, but I'm not sure it's noticeable.

    I think your slowdown might have been because the GitHub builds I put online still have debug checks in them.

  • @Gordon I've used to directly use the generated bins done by nkolban, Yes this was my first time to build from latest source code and flash it to my ESP

  • Then the string constants eat RAM. I am "guessing" that some large portion of the missing RAM is used by strings.

    A while back I switched printf to put the format string into flash. It's not trivial to do that with all strings because flash only supports word-aligned accesses, so you have to be careful.

  • Ive put out a call for community involvement here ...­&t=6376 ... to see if there are some practices that might be shared on how to move strings out of RAM assuming it is possible.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview

MQTT on ESP8266 with Espruino

Posted by Avatar for roccomuso @roccomuso