Filesystem load to RAM vs piping

Posted on
of 2
/ 2
  • Trying to send a 16KB file from the Espruino to a HTTP client. I've hooked up an SDcard reader to the Espruino and configured it, can successfully read and write data to/from it.

    I've tried both the methods mentioned here: to send data to a HTTP client.

    The first, reading the file in its entirety to RAM and then sending to the client, is relatively fast , 4.52 seconds to send 16.1KB.

    The second, piping the file directly from the SD card is massively slower. I could understand it being a second or two slower, but 16.1KB takes 21.24 seconds.

    I don't believe the SD card interface is the bottleneck, because reading the file to RAM first happens on a page request, so if the SD card took 16.72 seconds to read the file and the remaining 4.52 seconds was just sending the data over http, then the first request would take the same amount of time as the piping request.

    Have I found another bug, am I doing something wrong, or am I misunderstanding the limitations of the platform?

  • Please can you try with ...pipe(res, {chunkSize:512});

    The issue will probably be due to the much smaller default chunk size (64 bytes IIRC) when piping data. While it shouldn't be that much slower, it can mean that the WiFi connection chooses to send many small packets rather than just a few large packets - and it can also be less efficient loading smaller amounts of data from the filesystem as well.

  • Down to 5.65 seconds, love it :)

    Something else for the docs!

  • Although 16KB is nothing, I need to stream about 2MB...

  • You could try some bigger chunk sizes and see if it helps, but I'm afraid it's never going to be super-fast. By default the WiFi is connected at 115200 baud, so realistically even with zero overhead you're still looking at only around 12k/second.

    By increasing the WiFi baud rate (you need to send a command to the ESP8266 to set it up) and also increasing the SD card bit rate you could improve things, but I think it may still not be as fast as you really want for a 2MB file.

  • So while I could set a new baud rate, I can't see a way to then tell the Espruino to communicate with the wifi module at that rate, by the look of­WiFi.js its fixed to 115200.

    Is there any chance you could alter the module so I can pass configuration information for the Wifi module if I want to?

  • For anyone as silly as me who decides to change the baud rate without waiting for a configurable wifi communication module, here is how to rescue yourself.

    AT+CIOBAUD seems to carry across reboots, which everyone says it doesn't, but it does!

    digitalWrite(A14, 0);
    Serial2.setup(9600, { rx: A3, tx : A2 });
    at = require('AT').connect(Serial2);
    at.cmd("\r\nAT+RST\r\n", 10000, function cb(d) {
      at.cmd(“AT+CIOBAUD=115200\r\n”, 1000, function () {
        at.cmd(“AT+RST\r\n”, 1000);
    digitalWrite(A13, 1);
    digitalWrite(A14, 1);
  • Argh - sorry - I hadn't realised you were on Espruino WiFi. You can do Serial2.setup after WiFi is initialised I think, but yes, coping with the changed baud rate afterwards is a pain. I thought there was a non-persistent way to change it, but I'm afraid I can't find it now.

    Thanks for posting up your code for the fix though - that's great!

  • AT+UART_CUR should do the job but I don't think it is supported on the firmware that comes on the Espruino WiFi.

  • Oh poop now I killed it by setting an unsupported baud rate.

    Gordon could you convert the instructions from for Flashing the ESP8266 so that they work for the Espruino WiFi? Looks like I need to reflash to get this working again.

  • I had hoped I'd managed to convert the instructions myself, but no luck.

    Here's what I had

    Serial2.setup(115200, { rx: A3, tx : A2 });
    Serial2.on('data', function(d) { USB.write(d); });
    USB.on('data', function(d) { Serial2.write(d); });

    but when I run I just get: v1.3
    Auto-detected Flash size: 32m
    Running Cesanta flasher stub...

    A fatal error occurred: Invalid head of packet (b'\x13')

  • Looks promising - but maybe try:

    digitalWrite(A14,0); // power off
    digitalWrite(A13,0); // boot mode
    digitalWrite(A14,1); // power on
    Serial2.setup(115200, { rx: A3, tx : A2 });
    Serial2.on('data', function(d) { USB.write(d); });
    USB.on('data', function(d) { Serial2.write(d); });

    I think you set the boot mode after powering the device up?

    Are you sure you're disconnected with the Web IDE?

  • Still no joy. Should I expect a specific output before I disconnect the IDE? At the moment I get different responses depending on the baud but nothing readable.

    USB.on('data', function(d) { Serial2.write(d); });
    -> Serial1


    -> Serial1

    By doing some mad and barely reproducible set of steps, including a baud rate of 4608000(!), I can occasionally get the flasher to connect v1.3
    Running Cesanta flasher stub...
    Flash params set to 0x0200
    Writing 520192 @ 0x0... 120832 (23 %)
    A fatal error occurred: Unexpected packet while writing:

    but never to complete a write. I get different percentages of success but 23% is the highest so far.

    I've ordered a new board because I'm pretty sure I've fried this one :(

  • Well, if you get to 23% that's a really good sign. I'm sure we'll be able to get this one going.

    Did you change the buffer sizes in the flasher program? That can make it a lot more stable.

  • Yeah I tried setting

    ESP_RAM_BLOCK and ESP_FLASH_BLOCK all the way down to 0x1 as well as ESP_ROM_BAUD (although I'm fairly sure this is overridden by the -b flag) and ESP_FLASH_SECTOR.

    No luck with any combo I could come up with.

    Regards to your code sample, don't I need to pulse on A15?

  • Don't I need to pulse on A15?

    No, Reset isn't connected on the Espruino WiFi, since you can do the same just by powering the device off with CH_PD.

    Could you try with a block size of 128 and a baud rate of 57600? It'll be really slow, but the slower it is the more reliable it should be.

    Your other option if that fails is to do:

    digitalWrite(A14,0); // power off
    digitalWrite(A13,0); // boot mode
    digitalWrite(A14,1); // power on

    and to then connect a USB-TTL adaptor directly to the ESP8266. It'll be a bit fiddly to solder, but if you use the high baud rates used for normal programming you could probably connect GND to the Espruino WiFi's pin and then just hold RX and TX on by hand.

  • Gordon is that a baud rate in esptool's settings, or in the code I'm running on the Espruino to become the serial passthrough? Or do they both need to match?

  • It's the Espruino WiFi one. The baud rate in esptool shouldn't matter since it's over USB, but it'd be better to make the two match just in case.

  • No luck there either. A different language of serial gobbledegook, but the flasher still complains :(

    Serial2.setup(57600, { rx: A3, tx : A2 });
    Serial2.on('data', function(d) { USB.write(d); });
    -> Serial1

  • Very odd - not sure what to suggest with trying to upgrade via the WiFi. Do you have a USB-TTL converter handy that you could try the upgrade with?

  • I've got one­YBHM26 on the way, should arrive today.

    Can you confirm which pins of the usb-ttl I'm hooking to which pins on the ESP8266 module on the WiFi?

    I'm guessing I can use the + and - pins on the WiFi itself and hook those to the 5v and GND on the usb-ttl. Not sure where to hook RXD and TXD.

    Assuming­d/blob/master/WiFi/pdf/espruino_wifi_sch­.pdf is oriented in the real world, I've guessed as per my attachment.

    Should I hook usb-ttl RXD to ESP8266 RXD and TXD to TXD or do I want to crossover?

    Also, while I think of it, do you have a copy of the AT 0.40 firmware? I was going to flash with 0.25 as it was the newest I could find from your downloads (­T25-SDK112-512k.bin), but I'd rather flash 0.40, I just can't find it.

    1 Attachment

    • IMG_20170509_091058183.jpg
  • Yes, looks good.

    TXD is out from the ESP8266 towards the PC, and RXD is into the ESP8266 - so depending on your USB-TTL dongle you may need to swap them. I have dongles labelled both ways here to if it doesn't work it's just best to swap the wires - but it won't damage anything if you have them around the wrong way for a minute or two.

    I wouldn't connect + at all. Just connect GND, RX and TX, and leave the WiFi board connected by USB so you can set up the pins correctly.

    With the firmware, I don't actually have 0.40 here - I'm afraid you'd just have to google and find it.

    This might be the most helpful:­T-Command_firmware

    As you're all set up, I'd try uploading the newest 'espressif firmware' from that page. It should work fine, and I know it does have some new features (the ability to advertise itself via mDNS is cool).

    However it looks like the firmware is available at the bottom of­ware/

  • The 0.40 firmware there looks to have some questionable strings in it, I've managed to find 0.60 from espressif official, will let you know how the flashing goes.

    *** Modified by ***

    GET /ota/device/at/rom/v1/?is_format_simple=­true HTTP/1.0

  • No, I lied, 0.51+ only works on devices with 8Mbit flash and I think the esp8266 we have is 4Mbit. So I have 0.50 official­=46&t=1123

    I have found some documentation that suggests even the newest firmwares will actually fit on 4Mbit, will give it a try, not sure who to believe!­stalling-latest-at-firmware-on-4mbit.htm­l

  • AT+GMR
    AT version: 14 2016 18:54:01)
    SDK version:2.0.0(656edbf)
    compile time:Jul 19 2016 18:43:55


  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview

Filesystem load to RAM vs piping

Posted by Avatar for dave_irvine @dave_irvine