Unexpected delays writing to an SD card.

Posted on
  • Using the attached test script, I've been investigating the maximum write speed I can get out of my 2GB SD micro card.

    Out of the four SD micro cards I have to hand, I can only get one to work. Also note that the write speed of every card is likely to be different.
    I find that writes are most efficient when I write a block of 2048 bytes. The data I am writing is just 'A'-'Z' repeated to fill the buffer. What I write doesn't appear to make any difference.

    The script allows you to choose software SPI or hardware SPI. If you you'd like to run the test on your Espruino and post the average speed you get here, that would be interesting for all to see.

    With my working SD micro card the best speed I can get is with hardware SPI, where I can an average write speed of ~380 kbps (with an SPI baudrate of 2,000,000 being the highest setting that makes a difference). If I switch to software SPI it drops to ~230 kbps

    The 'problem' I am having, that is affecting my application, is a series of unexpected delays in the File.write() where is takes 4-5 times the average write time at random intervals during a 200K file write in 2K blocks. I've attached an example log output from running the script. The 'long' writes are highlighted by a >>>>>>>>>>>>>>>>>>>>>>>>>>. These happen using either hardware or software SPI, so I wonder if it is the SD card itself taking longer to respond or if it is an problem with the SPI module?

    Any insights here would be good, and solutions would be better.

    P.S. all my testing is on an Espruino Pico

    2 Attachments

  • I'm afraid I think this is just the nature of most filesystems. As I understand it, FAT writes in blocks, but there's a central table that contains information about the file. While writing a block takes a fixed amount of time, the filesystem will occasionally have to update the FAT table (I think?) which will involve a read, erase and write, and it definitely has to search around to find the next free block to write.

    The SD card library itself is in 'low memory' mode, and I guess potentially moving out of there into the mode where it uses up more RAM might help with the speed issues, but I'd imagine that there will still be times when it takes a long time.

    The problem has been that the filesystem lib used (iirc?) 2k of RAM all the time, whether it was used or not. It's something that people who don't use SD cards would get fed up with. Potentially now it could be modified to dynamically allocate that memory, but I think that's quite a bit of work.

  • Such a shame that it blocks the serial interrupt during these long 'phases' causing incoming serial data to be lost.
    Its really proving to be a show stopper for dependable file uploads via http with the ESP8266

    I've had the occasional case where I've managed to transfer a file of 100K without fault at 57600 baud from the ESP8266, but more often than not I will experience dropped data during a 10K file upload at 19200 baud. Its fairly unpredictable.

  • With software SPI it shouldn't block the serial interrupt? It's possible that if Serial and SPI shared the same interrupt priority then this wouldn't happen I guess... Although to be honest it'd be a mile better if the SPI stuff could be rewritten to use DMA.

    I guess 19200 baud with software SPI might be a bit more dependable?

  • I've tried software SPI and the result is much the same. Its slower in in the first instance for each write, not as bad as half the speed of the best hardware SPI rate I can get. However it suffers delays of the same length (~0.2 sec) as in the log above with similar frequency and these delays do block the serial interrupts the same way as for hardware SPI.

  • Wow, 0.2 sec is pretty hefty. While it won't block serial IRQs during that time, it's more than enough that it'd fill the buffer.

  • Just to add, it turns out the filesystem isn't in low memory mode, so I'm afraid it can't go any faster. So I guess the only options are:

    • Increase the input buffer size
    • Re-write the SD library so that it's nonblocking - however that would require the FAT library to be rewritten too!
    • Speed up software SPI (it could be implemented like jswrap_io_shiftOut, which would really speed it up by accessing the register directly - at least on STM32). That may still not help you though
    • Don't use an SD card. For instance you could use SPI flash, and then you can create your own journalling filesystem that wouldn't have these pauses.
  • Funny you should mention using SPI flash, I'm in the process of investigating that as an option to buffer the incoming multipart/form-data and then parse that, saving the file parts to an SD card.

  • Yes, that'd work... Or there's actually 128kB of flash free at the end of the Pico's memory space, and you could just write into that.

  • So Snerkle, what happened in the end? I am doing a similar project and facing this issue. What was your workaround to this? Thanks.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview

Unexpected delays writing to an SD card.

Posted by Avatar for Snerkle @Snerkle