What is recommended practice for creating arrays of structured data?

I was playing around with the simpleDataLogger and found that the Int16Array() used in the example is efficient (especially pushing new values onto the array), however, what is the recommendation if you have a tuple like: 32 bit timestamp and 2 16 bit temperature values, and want to maintain an array of these, to make a random example?

Being a 'C' programmer, I'd define a structure containing the 3 values and simply create an array of structures, keeping head and tail indexes to add to the array. Very efficient. I suspect those who have 'internal' knowledge and experience with Espruino would come up with a more efficient implementation than I would... :-)

@TomWS, you are absolutely correct w/ the C approach... with JS this is a bit different, because the 'system' (language and implementation) does memory handling for you... In order to do so, it needs information which then leads to memory inefficiency, something you do not like very much when already memory strapped / bound. BUT there are some trickeries used all across, even in the (adafruit graphics) C world: compose 2 16 bits to a 32 bit one (or worse, 2 8 bit to a 16 bit one and then combine it with another 16... etc.. I guess you get this muddying. The array would then include a fixed number of 32 bit entries per event and when processing they have to be taking them apart, with shift and masking (and making the MSBit right to not venture into the issue of packing a positive value and unpacking a negative one and vice versa). More (backwards) advanced things are then to provide some structure information in the first byte to allow variable number of entries in a 'record' stored as 32-bit Uints... ;-) or more :\ - take it for what you look at it...

Structured data is a bit more of a pain - you are stuck doing reasonably low level work to get it stored efficiently.

The easiest way to actually access the data is to create an ArrayBuffer (byte array) of the length required and to use DataView to access it: http://www.espruino.com/Reference#DataVi­ew

If you were particularly interested in making it 'nice' you could use getters and setters:

var ELEMENT_SIZE = 3;

class Element {
  constructor(buf, idx) {
    this.d = new DataView(buf,idx*ELEMENT_SIZE,ELEMENT_SI­ZE);
  }
  
  get a() { return this.d.getInt16(0); }
  set a(v) { this.d.setInt16(0, v); }
  get b() { return this.d.getUint8(2); }
  set b(v) { this.d.setUint8(2, v); }
};


var buf = new ArrayBuffer(5*ELEMENT_SIZE);

function el(idx) {
  return new Element(buf, idx);
}

el(0).a = 257;
el(3).b = 42;
print(el(3).b);

But it's not going to be especially fast if you care about that, since it's creating a new object each time you call el(..)

Thanks both. @Gordon, define 'not going to be especially fast'. In this particular case I'll have 10 second sampling interval and only need to manipulate two of the three elements if I store in a Uint32Array. The first value is used as-is, right from the array. The second two values would be 'deconstructed' from the next 32 bit value.
I'll take a look at Dataview. I had noticed it, but hadn't pursued it yet. Thanks for the tip.

Looking at Dataview, it seems as if it really doesn't matter if the array is a typed array or simply ArrayBuffer. The setting and getting is based on the type of the set or get and alignment is apparently not a concern. The key thing seems to be keeping the byte offset correctly tracking the data. But, I think you're right, Dataview makes the code cleaner rather than cryptic data conversion code.

Update: it's too bad the byte offset isn't post incremented by the size of the object that's set or get.

@Gordon 's solution is nicely oo... I was not thinking to go that far if not needed... two (or 3) functions with upfront storage definitions would have done the job for me, as something like this or variations of it:

var storage = new ArrayBuffer(80)       // 10 x 32+16+16d bits 10 x 8 bytes
  , store08 = new Uint8Array(storage)   // 80 x  8 bits - just for display
  , store32 = new Uint32Array(storage)  // 20 x 32 bits
  , store16 = new Uint16Array(storage)  // 40 x 16 bits
  ;

// store time(ui32),temp1(u16),temp2(u16) Values
function storeVals(idx,time,temp1,temp2) {
  store32[idx <<= 1] = time; 
  store32[++idx] = (temp2<<16) | temp1; // ...or analog to retrObj()
}

// store {time(u32),temp1(u16),temp2(u16)} Object
function storeObj(idx, obj) {
  storeVals(idx,obj.time,obj.temp1,obj.tem­p2);
}

// retrieve as {time(ui32),temp1(u16),temp2(u16)} Object
function retrObj(idx) {
  return (
    { time:  store32[idx <<= 1]
    , temp1: store16[idx = ++idx << 1]
    , temp2: store16[++idx]
    } );
}

To have even more convenience methods for retrieving is adding functions like getTim(idx), getTemp1(idx) and getTemp2(idx) with idx adjusting to respective store32 and store16.

PS: tested: see post #11

I like the Dataview, the code is easier to read and takes care of endianess. I do think a variant of the getters and setters where the byte offset is passed as reference so it can be post incremented would be fantastic!

define 'not going to be especially fast'. In this particular case I'll have 10 second sampling interval

It'd be perfect. You'll hit trouble if you try 100s a second, but for that it's fine :)

If your object is simple, just using DataView does makes a lot of sense. DataView itself is based on the standard JS one so we're stuck without any kind of increment - you could always add your own functionality I guess:

DataView.prototype.setUint16i = function(v) {
  this.setUint16(this.idx,v);
  this.idx+=2;
};

var buf = new ArrayBuffer(15);
var d = new DataView(buf);
d.idx = 0;
d.setUint16i(1);
d.setUint16i(2);
d.setUint16i(3);

While you can modify byteOffset in Espruino, you can't in normal JS so I'd avoid doing that if possible :)

Hi all,

Just as an example, I am doing that sort of things with a ublox sam-m8q gps using their binary protocol.
The example below is for the paragraph 31.18.20 UBX-NAV-STATUS (0x01 0x03).
Their messages are pure packed c struct with properly aligned words, dwords and so on and most of those have a fixed length. That was designed for C, C++ definitely.

However, I am not using an array, just a buffer containing a single data packet.

I would store more than one in an Array (javascript standard) of ArrayBuffer(s) and then getters or setters with this kind of Dataview code, not pre splitted buffers with their contents separated. Thats may be slower but it's memory savy.

gps.Status = function() {
      if (this[0x0103]===undefined) return undefined;
      let d=new DataView(this[0x0103].buffer);
      return {
          iTOW: d.getUint32(6, true)/1000,
          gpsFix: d.getUint8(10, true), 
          flags: d.getUint8(5+6, true),
          fixStat: d.getUint8(6+6, true),
          flags2: d.getUint8(7+6, true),
          ttff: d.getUint32(8+6, true)/1000,
          msss: d.getUint32(12+6, true)/1000, 
          receivedAt : new Date(this.receivedAt[0x0103]*1000)
      };
};

@Gordon, I realized last night that I could probably extend Dataview, thanks for showing me how!! And, yes, I think Dataview is all I really need in this case.

Interesting approach, @asez73. Thanks for showing it. I don't think I would use that in my case, but it's good to have in the 'toolbox'. ;-)

Tom

Did some adjustments and tested code in post #6. The main advantages of using pattern in post #6 vs. the other ones is that the creation of (view) objects happens only once and only indexes are used afterwards... 1st one works only when working all the time on the same piece of memory. Latter one can be combined with the pattern @Gordon a d @asez73 suggest. In real app, I would anyway hide these things in an StructXArray 'class' and implement methods as needed rather than just functions. Doing so, multiple structured objects can then coexist in a very easy to read form, as @TomWS points out. Final verdict regarding performance always depends on the implementation of Espruino, though,...

Below is the test code (html in browser), followed by output. The output visualizes nicely what is going on with the memory locations. In addition to the actually used Uint32 and Uint16 array, also the Uint8 array of the same memory is shown. I attached the arrayOfStrict.html. You can click on it and it runs directly in your browser and should show the same output...

<html>
<head>
<script>
var storage = new ArrayBuffer(80)       // 10 x 32+16+16d bits 10 x 8 bytes
  , store08 = new Uint8Array(storage)   // 80 x  8 bits - just for display
  , store32 = new Uint32Array(storage)  // 20 x 32 bits
  , store16 = new Uint16Array(storage)  // 40 x 16 bits
  ;

// store time(ui32),temp1(u16),temp2(u16) Values
function storeVals(idx,time,temp1,temp2) {
  store32[idx <<= 1] = time; 
  store32[++idx] = (temp2<<16) | temp1;
}

// store {time(u32),temp1(u16),temp2(u16)} Object
function storeObj(idx, obj) {
  storeVals(idx,obj.time,obj.temp1,obj.tem­p2);
}

// retrieve as {time(ui32),temp1(u16),temp2(u16)} Object
function retrObj(idx) {
  return (
    { time:  store32[idx <<= 1]
    , temp1: store16[idx = ++idx << 1]
    , temp2: store16[++idx]
    } );
}
</script>
</head>
<body>
<h3>10 x Structuer w/ 32+16+16 bit</h3>
test / visualization:
<script>
document.write('<br><b>storeVals(0,65536­,255,256); // @ index 0</b>');
storeVals(0,65536,255,256);
document.write('<br>--->store32:['+store­32+']');
document.write('<br>--->store16:['+store­16+']');
document.write('<br>--->store08:['+store­08+']');

document.write('<br><b>storeObj(3,{time:­65535,temp1:128,temp2:1040}); // @ index 3</b>');
storeObj(3,{time:65535,temp1:128,temp2:1­040});
document.write('<br>--->store08:['+store­08+']');
document.write('<br>--->store32:['+store­32+']');
document.write('<br>--->store16:['+store­16+']');

document.write('<br><b>obj1 = retrObj(0); // @ index 0</b>')
var obj1 = retrObj(0);
document.write('<br>---> {time:'+obj1.time+',temp1:'+obj1.temp1+'­,temp2:'+obj1.temp2+'}');

document.write('<br><b>obj2 = retrObj(3); // @ index 3</b>')
var obj2 = retrObj(3);
document.write('<br>---> {time:'+obj2.time+',temp1:'+obj2.temp1+'­,temp2:'+obj2.temp2+'}');

document.write('<br><b>obj3 = retrObj(9); // @ index 9</b>')
var obj3 = retrObj(9);
document.write('<br>---> {time:'+obj3.time+',temp1:'+obj3.temp1+'­,temp2:'+obj3.temp2+'}');

</script>
</body>
</html>

And the output:

10 x Structuer w/ 32+16+16 bit
test / visualization: 
storeVals(0,65536,255,256); // @ index 0
--->store32:[65536,16777471,0,0,0,0,0,0,­0,0,0,0,0,0,0,0,0,0,0,0]
--->store16:[0,1,255,256,0,0,0,0,0,0,0,0­,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0­,0,0,0,0,0,0,0,0]
--->store08:[0,0,1,0,255,0,0,1,0,0,0,0,0­,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0­,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0­,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0­,0,0,0,0,0,0,0]
storeObj(3,{time:65535,temp1:128,temp2:1­040}); // @ index 3
--->store08:[0,0,1,0,255,0,0,1,0,0,0,0,0­,0,0,0,0,0,0,0,0,0,0,0,255,255,0,0,128,0­,16,4,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,­0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,­0,0,0,0,0,0,0,0,0,0,0]
--->store32:[65536,16777471,0,0,0,0,6553­5,68157568,0,0,0,0,0,0,0,0,0,0,0,0]
--->store16:[0,1,255,256,0,0,0,0,0,0,0,0­,65535,0,128,1040,0,0,0,0,0,0,0,0,0,0,0,­0,0,0,0,0,0,0,0,0,0,0,0,0]
obj1 = retrObj(0); // @ index 0
---> {time:65536,temp1:255,temp2:256}
obj2 = retrObj(3); // @ index 3
---> {time:65535,temp1:128,temp2:1040}
obj3 = retrObj(9); // @ index 9
---> {time:0,temp1:0,temp2:0}

2 Attachments

• arrayOfStruct.html
• 

As an update to this conversation, I had to make a minor change to the data buffer, changing it from ArrayBuffer to Uint8Array so that I could use buf.set(...) to push the older samples out. I presume that .set is going to be more efficient than anything I could have done programmatically.

So here are the relevant snippets from my working code:

var ELEMENT_SIZE = 10;  // 10 bytes per data structure (uint32, 3 x int16)
...
// define various setters and getters (could probably put this in a 'required' module
DataView.prototype.setInt16i = function(v) {
  this.setInt16(this.idx,v);
  this.idx+=2;
};
DataView.prototype.setUint32i = function(v) {
  this.setInt32(this.idx,v);
  this.idx+=4;
};
DataView.prototype.getInt16i = function() {
  var v = this.getInt16(this.idx);
  this.idx+=2;
  return v;
};
DataView.prototype.getUint32i = function() {
  var v = this.getInt32(this.idx);
  this.idx+=4;
  return v;
};
DataView.prototype.flush = function() {
  this.idx=0;
};

// buffer to store data samples
var buf = new Uint8Array(100*ELEMENT_SIZE);  // enough room for 100 samples
var d = new DataView(buf.buffer);
d.flush();
...
// A function to sample and store data into buffer
function log() {
  // is the buffer full?  If so, push it
  if (d.idx >= d.buffer.length-ELEMENT_SIZE) {
    // move all history values back by one element
    buf.set(new Uint8Array(buf.buffer,ELEMENT_SIZE));
    d.idx -= ELEMENT_SIZE;  
  }
  
  // sample the rest of the data and store it all in buffer
  d.setUint32i(Math.floor(Date.now()/1000)­);
  d.setInt16i(Math.floor(NRF.getBattery()*­1000));
  d.setInt16i(temps[0]);
  d.setInt16i(temps[1]);
  
}
...
// This is a utility function that will output our data
// in a form that can just be copy/pasted into a spreadsheet
function output() {
  clearInterval(timer);
  var saveIdx = d.idx;
  d.flush();
  while (d.idx < saveIdx) {
    var dt = d.getUint32i();
    var bat = d.getInt16i();
    var temp1 = d.getInt16i();
    var temp2 = d.getInt16i();
    var line = [dt,bat/1000,temp1/10,temp2/10].join(","­);
    console.log(line+"\n");
    i++;
  }
  d.idx = saveIdx;
  startSampling();
}

Note that the output() function might look a bit cumbersome, but that's mainly because I want to retain the current data set rather than output and flush the old stuff...

Thanks for everyone's help!
Tom

You are very welcome. - Extending a base class... works... including the auto increment every time you set or get a value. From a maintenance point of view though, I'd rather have defined a dataLog object with all this functionality stuck to it. Since it is though a very specific application, there is no issue with that, especially when fitting exactly the context - spread sheet row.