ESP8266 as serial wifi module - expected performance?

I'm using a puck.js v2 + esp8266 (ESP-01) module on Serial1 to serve files stored on an SD-card.

While I understand that this will never be a super fast file transfer method, I wonder what kind of speed could be expected? I'm currently getting between 600-700 bytes/s, and increasing the UART speed from 115200->460800 didn't really make a difference so I suspect that the UART is not the bottleneck.

Example code:

var wifi;

function mountSDCard() {
  SPI2.setup({mosi:D30, miso:D28, sck:D29, baud: 8000000});
  E.connectSDCard(SPI2, D31);
  try {
    console.log(require("fs").readdirSync());
  } catch(error){
    console.log(error);
  }
  E.setFlags({unsyncFiles:1})
}

function onInit() {
  Bluetooth.setConsole(true);
  mountSDCard();

  Serial1.setup(460800, { rx: D1, tx : D2 });
  setupWifi();
}


function setupWifi() {
  var WIFI_NAME = "UseThisOneMom";
  var WIFI_PASS = "asdf";
  wifi = require("ESP8266WiFi_0v25").connect(Serial1, function(err) {
    if (err) throw err;
    console.log("Connecting to WiFi");
    wifi.connect(WIFI_NAME, WIFI_PASS, function(err) {
      if (err) throw err;
      console.log("Connected");
      // Now you can do something, like an HTTP request
      require("http").createServer(onPageRequest).listen(8080);
    });
  });
}

function onPageRequest(req, res) {
  var a = url.parse(req.url, true);
  var f = E.openFile(a.pathname, "r");
  if (f !== undefined) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    f.pipe(res); // streams the file to the HTTP response
  } else {
    res.writeHead(404, {'Content-Type': 'text/plain'});
    res.end("404: Page "+a.pathname+" not found");
  }
}

Found this comment at Internet / Transferring files:

Note: by default .pipe will use a relatively small chunk size (the
amount of data read and written in one go). Replacing f.pipe(res) with
f.pipe(res, {chunkSize:512}); will drastically increase file transfer
speeds as it better matches the block size of SD cards.

Thanks, this made a huge difference and I now get ~4.5kB/s instead.

I wonder what the bottleneck is now, and if there's more tweaking that could be done.

Glad you got this improved! I'm not sure how much room for improvement there is here - I just checked and it seems that the WiFi stack should be able to send data out in ~530 byte chunks, so it should mean data from the file goes straight out.

Without digging down (you could attach a USB-TTL converter to the ESP8266 RX line and look at what's being sent and when) it's hard to know where the slowdown might be.

So I hooked up two USB-TTL converters and had a look at the traffic going to and from the ESP8266 and I have a few findings.

1. The network_js.c implementation limits the number of bytes sent on each AT command to ESP to 536 bytes, even though the AT command can take up to 2048 bytes of data for each command. This is probably for a good reason, since that code can be used for other transports then the ESP, but maybe it could be possible to tweak it if you know what transport that will be used.

2. The "ESP8266WiFi_0v25" module uses the AT+CIPSEND command for sending, which is a synchronous version that means that Espruino will say "I want to send X bytes", ESP: "OK", Espruino: "here's the data", ESP "Got that, and now I have sent it". Then the Espruino can send the next chunk of data. There's another command called AT+CIPSENDBUF that can be used to add data to the ESP:s send buffer, so that more data can be pushed in without waiting for all data to be sent. It's more complex to use since it requires you to make sure the buffer has space etc.

Just tweaking the chunksize so that 2048 bytes of data in sent in each AT+CIPSEND increases throughput to ~12kB/s, but in order to go above that I guess the logic needs to be rewritten to use AT+CIPSENDBUF instead so that the wifi connection have data to send at all times. I'll see if I have the possibility to make a working implementation of this.

Nice - thanks for looking into that! AT+CIPSENDBUF sounds really promising - although I guess if you have to query the buffer size it may not end up being any better.

As I remember the send buffer end up being allocated on the stack, which I think is why we don't allocate too much. However I think it would be pretty easy to make that configurable.

In an idea world we'd actually have a shortcut for the JS interface that just pushed the entire send buffer - no matter how big - to the JS interface, and it could decide how much to send.