You are reading a single comment by @cephdon and its replies.
Click here to read the full conversation.
-
@cephdon - you are absolutely right - when there is no time constraint using the practical unlimited space of the SD card is a way out. For the first of my three blocks of data this is a valid approach. For the second and third one, access and process performance are crucial. Therefore they have to live in the memory. If all options fail - space and time - I may resort to inline assembler creating the values from a very compressed, byte based/bit banged memory chunk, and pass them back to the JavaScript context.
allObjects
@cephdon
If we had the ability to do a relative seek in a file, or the ability to move to an absolute position (forward and backward) in the file, then we could build a relatively unlimited (from the perspective of memory) key value store (or simple db). It's easier if it's relative, though.
Basically, you store in a file(for each entry):
Offset to the beginning of the next entry, offset to the beginning of the previous entry, key, data. (Or, size of this entry, size of previous entry, key, data. It's the same thing when relative)
It could be line delimited, or not.
In any case, you could easily binary search the database for your data, assuming you sort it at insertion time.
Then, when that is done, you only keep the data you want in memory and leave the rest saved to disk.
It might not be as fast as an in memory array, but with a 32GB sd card, you could store and search much more than the memory available.