Piping from http to sd card results in OOM error

Not sure why I get this error:

ERROR: Out of Memory!
done piping
at line 1 col 106
...str(0,b);f[a]=f[a].substr(b)}else d=f[a],f[a]="","DataClose"...
in function called from system

(highlighted piece of code is the substr() call)
Simplified code is:

http.request(httpOptions, function(requestResult) {
  requestResult.pipe(f, {
    chunkSize: 512,
    complete: function() {
      console.log('done piping');
    }
  });

}).end();

Is this with Espruino WiFi? And does it happen the first time you perform the request?

Please can you post up what process.memory() reports, just before the request? So add the line console.log(JSON.stringify(process.memor­y())) just before the http.request in the code you posted above.

uhhhmm

Uncaught SyntaxError: Got ?[173] expected ','
at line 36 col 41
console.log(JSON.stringify(process.memor­y()))

And yes this is on Espruino WiFi

How are you communicating with the Espruino? is it via the Web IDE or something else?

The forum inserts annoying invisible 'line break' characters in code, but the IDE should usually filter them out. The act of pasting onto the right-hand side of the IDE should definitely filter them out.

Ah, I retyped the command and now it works.

{ "free": 4862, "usage": 2286, "total": 7148, "history": 0,
"stackEndAddress": 536990532, "flash_start": 134217728, "flash_binary_end": 435488, "flash_code_start": 134283264, "flash_length": 524288 }

Out of interest: were you using the IDE, or something else?

Wow, yeah - that's loads of free memory. I've just been looking at this, and it seems that the out of memory error occurs inside the receive handler - so basically the ESP8266 is pushing data into Espruino and it's not able to handle it fast enough so it ends up getting buffered in the ESP8266 driver.

I guess you'd figured that out with your work with adding flow control anyway?

You could try to increase the chunk size to 2048, and see if that fixes it? Also, if it's possible to increase the IO speed to the SD card I'm sure that'd help a huge amount.

It's a bit difficult to work around really - the new Espruino builds will do proper flow control with ESP8266 but that's only at the low-level side. There would have to be something else added that'd push back from further inside the networking layers as well.

Possibly another option might be for me to add a way to give priority to piping that is done from sockets - but I don't think that's what is happening in this case - I think the Espruino is busy enough that the data isn't even getting out of the ESP8266 driver into the networking layer.

I edit code in the modules/ directory of the project and then flash with the IDE, so that was probably the issue.

I'll try increasing the chunk size and baud rate for the SD Card.

Anything you can do would be great, I really need the wifi connectivity to be rock solid. Thanks for your help so far.

To be honest, the OOM error has mostly been replaced by the power issue, I can't actually replicate the OOM any more.

I did a setInterval to print the memory usage and it never really drops too low.

{ "free": 3775, "usage": 3373, "total": 7148, "history": 0,
"stackEndAddress": 536990532, "flash_start": 134217728, "flash_binary_end": 435488, "flash_code_start": 134283264, "flash_length": 524288 }
{ "free": 4146, "usage": 3002, "total": 7148, "history": 0,
"stackEndAddress": 536990532, "flash_start": 134217728, "flash_binary_end": 435488, "flash_code_start": 134283264, "flash_length": 524288 }
{ "free": 4144, "usage": 3004, "total": 7148, "history": 0,

Odd - oh well, let's leave this for now. I have a branch that does ESP8266 handling in the firmware itself that's a lot faster - so hopefully that'll be ready soon and will go a long way towards fixing some of these issues.

Fixed: http://forum.espruino.com/conversations/­306379/#13693680