Using SDCard, where am I going wrong?

Hi All,

I'm building a sensor data logger with my Espruino 1.4 board with latest firmware.

Reading the reference (couldn't find a tutorial), if I wanted to write data to the onboard SDCard once a second, it looks to be as simple as:

function onInit () {
  
    var sdCard = require('fs');

    setInterval(function () {

        sdCard.appendFileSync("Test.txt", getTime() + "\r\n");
    
    }, 1000);

}

And I should get a single file named "Test.txt" containing a list of system times.

However, if I let it run for a while, disconnect power and put the SDCard in a reader, I actually end up with around (it varies) 8 files all called "Test.txt" all which contain the same information, mainly. This is strange because I can certainly not create two files with the same name via a PC.

I'm guessing from this that I missing a step, like closing a file, flushing etc. But reading the reference this looks like it should be synced.

Can someone advise please. The data logger will fly on a multicopter for atmospheric monitoring so would like any data to be written to the file as it is collected rather then storing in memory, then writing due to the inherent risks of flying these small craft.

Cheers

Ian

And to add to this, it appears to be running at half speed when it writes to the file if you look at the data. It should be once a second but after the first few values, it very accurately writes every two seconds.

5.00040340423
6.00036048889
8.00035190582
10.00035953521
12.00035476684
14.00034523010
16.00034523010
18.00039958953
20.00035858154
22.00036430358
24.00035572052
26.00037193298
28.00035285949
30.00035285949
32.00035858154
34.00038909912
36.00037765502

I moved to the Espruino from an Arduino to have built in SDcard logging and more pins. And I thought integrating the sensors was going to be the hard bit, but that has been super easy. :)

Hi Ian,

Yes, everything should be flushed after appendFileSync. If you're removing the card and putting it back in you should do E.unmountSD() though, so Espruino knows to reset the card.

Could you try just reformatting the SD card as normal FAT32? Sometimes the cards come as exFAT (or something else) and the Filesystem library isn't able to understand them properly. That could be what's happening here - sure seems like it if you have two files called the same thing.

It might also explain the write every 2 seconds. You might find the 'odd' seconds in one of the other files - although I don't know how you'd open it!

Hi Gordon,

Thanks, sounds like a possibility so tried the following.

Ok, card is freshly formatted as FAT32. Put card in Espruino and power up via LiPo battery. Let the above code run for 30 or so seconds. Disconnect power. Eject SD card and put in reader.

I have eight files all called "Test.txt". If I open them all individually they all contain the same values, I.e. Few lines into log the getTime() values are all 2 seconds apart.

I've checked the files again on a Mac (in case PC was problem) and its the same. Strange.

I'll try a different card first, but if that doesn't help, can you advise if there is a better way to do this? Ideal situation for me would be:

1. Apply power. Code runs automatically starting in the onInit() function.
   
2. Once GPS has acquired a 3D fix, create a file named from the date/time from GPS module.
   
3. Every time a new GPS position fix comes in (it's set at 1Hz), capture values from the three weather sensors and write position, date/time and sensor readings to this file as a single string.
   
4. Repeat everytime the GPS sends a new position sentence.
   
5. Remove power to end the logging.
   
   

Applying power and removing power will be a simple toggle switch in the +ve line from Lipo to Espruino. This would allow multiple flights (and date/time labelled files) to happen from just cycling the power.

Guess I just need a way to ensure a flush and dismount between each file write?

I'm rambling here, I'll try a different card first.

Cheers

Ian

Hi Ian,

Thanks for trying the formatting - that does seem strange though - people have been doing exactly this kind of thing a lot so if it were a problem I'd have thought it would have come up sooner.

I guess the issue could also be with just pulling the power - if you're unlucky and you pull power while it's writing the file then it could corrupt the SD card.

What you could do (which would be more efficient, if you cared about that) is to use var f=E.openFile(...) to open a file, and then use f.write(...) to write to it. However, you then definitely have to close the file with f.close() before the power went.

While you could detect the power going off and have a capacitor that would hold enough charge to close the file, it might be easier to implement a 'soft' on/off switch?

With setDeepSleep(1) in the code, when Espruino isn't doing anything it'll only be drawing a few microamps - so you could probably leave it connected and just have a pushbutton that enabled or disabled logging.

Actually, that's not a bad idea. The battery will be unplugged when not in use anyway (regardless of a toggle switch in).

So I could use BTN1 to start and stop the logging and go into deep sleep when not logging. That would then give the signal to close files and unmount the SD card before power is removed. Would also mean I don't need to put a toggle switch in.

That sounds really sensible and deep sleep is something I've not used before (as an Arduino user) so would be good.

As you say, if everyone was having problems, you'd be flooded with issues.

I'll try this new process and see how I get on.

Thank you Gordon, really appreciate it.

Cheers

Ian

So I could use BTN1 to start and stop the logging and go into deep sleep when not logging.

Yes, that should work great!

Literally all you need (not tested) is:

var logFile;
var logInterval;

function doLog() {
  digitalPulse(LED3,1,50); // pulse blue led as indicator
  f.write(getTime()+"\r\n");
}

setWatch(function() {
  if (logFile===undefined) {
   logFile = E.openFile("log.txt", "a");
   digitalWrite(LED1,1); // indicator
   logInterval = setInterval(doLog, 1000);
  } else {
    clearInterval(logInterval);
    logInterval = undefined;
    logFile.close();
    logFile = undefined;
    E.unmountSD(); // card can now be pulled out
    digitalWrite(LED1,0); // indicator
  }
}, BTN,  { repeat:true, debounce:50 });
setDeepSleep(1);

setDeepSleep just sets a flag, so you don't need to call it each time you need it to go to sleep. The second the 1000ms interval gets stopped it'll drop into sleep mode - to be woken only by you pressing the button.

So simple but elegant solution. Always helps to have a second pair of eyes on an issue. Thank you Gordon.

(Yes, my mindset is still arduino loop)

Cheers

Ian

Ok, confirmed. It was the lack of unmounting the SD card that was causing the problems and multiple files.

So I've taken Gordon's example code above and modified to allow use of fs.appendFileSync (as I don't want to worry if a file exists already or not) and now it works really.

As long as logging is stopped before ejecting the SD card, doesn't matter if you create five logs or one, you can take the card out and put it back in again and everything is as you would hope.

This now makes a great basis for my little sensor logging project. I'll add the deep sleep back in once testing is finished but running on a battery it does indeed work well and doesn't affect the logging (I've yet to test it with the sensors though).

So, thanks again Gordon.

Cheers

Ian

var fs = require('fs');
var logFile;

function onInit () {
  
  var logInterval;

  setWatch(function() {
    
    if (logFile === undefined) {
      
     logFile = Math.round(getTime()) + ".txt";
     digitalWrite(LED2, 1); // indicator
     logInterval = setInterval(doLog, 1000);
      
    } else {
      
      clearInterval(logInterval);
      logFile = undefined;
      E.unmountSD(); // card can now be pulled out
      digitalWrite(LED2, 0); // indicator
      
    }
  }, BTN, { repeat:true, debounce:50, edge:'rising' });
  
}

function doLog() {

  digitalPulse(LED3, 1, 100); // pulse blue led as indicator
  fs.appendFileSync(logFile, getTime() + "\r\n");
  
}

No problem, glad it's working!