Possible issues with large(ish) files [Resolved]

I think that's something that could definitely be improved on! 1 megabyte in 10 hours is around 30 bytes every second! That's insanely detailed - like you're storing 30 bytes of data for every heartbeat!

There's the enterprise dev (in me) kicking in again... I was extra cautious to have correct CSV format, so every field in every line was wrapped in an opening quote and a closing quote #faceplam...

• just getting rid of the quotes from the file reduced it by 100K. I am actually saving all the accel values so x,y,z,g and delta.
  
• Changed row header form HRM to HR and ACCL to AL, another 10K savings
  
• My Date/Time format is a mess, I am going to change that up to save some more.
  
  

The file currently has ~12K lines of data of which

• 7364 lines are accelerometer values (saved at every step event).
  
• 396 GPS entries (Loads of NAN that can be trimmed)
  
• 4728 lines of Heart Rate entries
  
• Logging started at about 09:07AM and stopped at 16:19PM.
  
  

I am saving full resolution of the floating point data I get so yeah each line is pretty detailed.

Looks like I can't really save full day of heart rate + accelerometer + GPS settings in current format. I'll see what further optimisations I can do or what data I can live without.

There is nothing on the desktop/analysis side yet apart from a prototype to map GPS points on a map using MapBox. But I want a closer look at accelerometer readings and heart rate and find co-relations between low movement + heart rate = sleep quality and stuff like that. Also correlations between heart rate and GPS values when cycling etc. Basically mimic as much of the Fitbit functionality as possible.

I'll keep working on optimising data collection until we have access to the rest of the memory via StorageFile :-). Also I'll aim to sync atleast twice a day. So it is just a matter of managing entries for 8 hours.

Update:
Looks like it clocked off at about 2:15 pm today... that too at 675KB only... hmm... very curious... I'll keep digging. First I need to fix this need for eraseAll after it has errored out!