How to go about tracking down this error?

Hi, I've spent many hours now setting up the tests and investigating different settings. Note: this code doesn't do anything with the incoming data, just counts the bytes.

The good news is that with 'well matched' settings the pipe/stream will flow all the way from the ipdHandler to the req.on('data',... in your own code. More on what 'well matched' means later.
The bad news is that under default settings (i.e. 115200 baud usart to ESP8266 and a 64 byte CHUNK size in socketserver.c) the netCallbacks.recv never gets the chance to get going: the ipdHandler (or more likely the ser.on("data",... in AT.js) dominates and buffers all the incoming serial data from the ESP8266 into sockData[sckt] before the netCallbacks.recv gets going. The memory pressure doubles (more or less) on the following calls to netCallbacks.recv as it tries to remove a 64 byte chunk from the beginning of sockData[sckt]

    if (sockData[sckt]) {
      /** INSTRUMENTED =>**/ var len = sockData[sckt].length;
      var r;
      if (sockData[sckt].length > maxLen) {
        r = sockData[sckt].substr(0,maxLen);
        sockData[sckt] = sockData[sckt].substr(maxLen);
      } else {

that second .substr(...) is the culprit as it makes a copy of the whole buffer (less 64 (maxlen) bytes).
On a pico running only the attached code, the maximum http message body (file upload) is just a bit less than 20K before an out-of-memory error occurs. I've attached a log generated from my instrumented copy of ESP8266WIFI_0v25 (also attached) where you can see that netCallbacks.recv only gets invoked once or twice by the time ipdHandler has buffered the whole http message.

If you play around with dropping the baud rate down to say 9600 you can see that while the serial data is arriving, netCallbacks.recv gets invoked only every ~500ms.

So what are some 'well matched' settings?
Well I'd been playing around with compiling my own build of Espruino just to play around with the CHUNK size in socketserver.c.
So as an example: a CHUNK size of 1460 matched with a baudrate of 19200 gets the stream flowing nicely (actually I can bring the baudrate upto 24000 with this chunk size) and the maximum size of the sockData[sckt] never gets much larger than the CHUNK size. This isn't exactly a fantastic baudrate, but I can stream in a file of any size (tested upto 300K, but saw that the buffer always stayed small), although I'm just counting the bytes and nothing more 'busy'.

So what next?

• flow control isn't an option as the ESP8266 only does hardware flow control with AT firmware, and the USART2 on the Pico doesn't support hardware flow control
  
• using a bigger CHUNK size and a low baudrate might be an option, memory permitting for a given project
  
• rewriting the ESP8266WIFI_0v25 module in native code in Espruino for better memory efficiency (one and only one copy of the data until it is consumed?) and optimising the balance between receiving serial data and scheduling javaScript execution
  
• if this isn't an option and we have to acknowledge the maximum buffer size that can be achieved with sockData then the memory pressure can be managed by using an index into the buffer for the start of the next chunk, and only removing delivered chunks when the buffer is at least half consumed(?)
  
  

3 Attachments

• test.js
• ESP8266WiFi_0v25inst.js
• test.log