Improved memory usage in Espruino

Hi - I thought some of you might be interested in the packed_array branch I just uploaded. I've made a build of it at http://www.espruino.com/binaries/espruino_1v65_espruino_1r3_packed_array_beta.bin

Basically, if a value is an integer between -32768 and -32767, it packs it into the same variable block as the variable name. That means that normal sparse arrays and objects are now twice as efficient when filled with small integers.

This is a pretty big change - It passes all the tests but there may still some issues when running code - If anyone can come up with a simple code example that shows up problems it'd be greatly appreciated!

A quick example of the improvements:

var obj = {
  a:63,
  b:100,
  c:42 };

function f3(a,b,c,d) { print(process.memory().usage); }
function f2(a,b,c,d) { f3(a,b,c,d); }
function f1(a,b,c,d) { f2(a,b,c,d); }

print(process.memory().usage);

f1("Hello","There","World","");
f1(1,2,3,4);

setTimeout("print(process.memory().usage);", 1);

• Old: 53, 75, 74, 63
  
• New: 50, 72, 67, 58
  
  

So:

• Objects/arrays take up less space
  
• Calling functions with integers takes less space
  
• Timers take less space (especially if the interval is <32ms)
  
  

@BogdanG will be happy anyway - finally something that improves memory usage on the VLdiscovery!

Hi Gordon,
For sure!
Frankly I gave up the espruino latelly for pure C but it may be worth to try it again.

Thanks
Bogdan

There's another beta of this up (based on the latest version of Espruino) at:

http://www.espruino.com/binaries/git/commits/2fc9d09ff088bba2fd3d31f1b64b659be89283c1

Please can some of you give it a go with your projects and report back if there are any problems? I'd really like to get this merged into the version that everyone has - it means that a lot more code and data can be squeezed into Espruino.

It's the same to flash as any of the git builds - click the link, then right-click on espruino_1v68_espruino_1r3.bin, then "copy link address", and then paste it into the text box on the Web IDE's flasher page and click advanced flash firmware.

Nobody? :(

New build up here:

http://www.espruino.com/binaries/git/commits/ba9ce23084b74a18ccba91a0a88c3cf0c62821a9/

This one not only uses variable blocks more efficiently, I've also been able to get the block size from 20 bytes down to 16. That means that on the Espruino board, where you could have got 1800 variables you can now get 2250 (I haven't updated the specs for each board yet though).

This branch also manages to get the block size for low-end boards down to 12 bytes, while still allowing up to 1023 blocks. It means that the VLDiscovery board now stores 500 blocks instead of 250.

Not only that but as arrays are packed, you can actually get 4x as many array elements in the boards memory as you could have before!

Woah.

I just read the bit about the block size going down to 16 bytes. I'll have to give this a try . The weather here has been great every weekend, so I've had a lot less time than usual to play with electronics.

That means we could get 3250 jsvars with the bigram builds...

Eurgh. What git command would I use to get this branch so I can build it with more jsvars? I'd like to test it with my desk-lamp, but that needs >2k (even after I cut out the 8x12 font, it runs out of memory) jsvars to run.

You should be able to do git checkout origin/small_jsvar in the existing repository (after git pull).

Doing it that way won't let you commit any changes, but otherwise it'll be fine.

3250 vars would be pretty good :) The only bad point is that there's still the same fixed overhead of 4 bytes for long strings, so for big chunks of data it's only 12/16=75% efficient rather than 16/20=80%. The extra vars make up for it though :)

I guess even the value packing should help you out too - I'd be interested to see how many variables it uses in the new build relative to the old.

git pull? git pull what?

I'm sorry for being slow on this - the only git command I know is 'git clone https://github.com/espruino/espruino'

Ahh. It's probably worth trying to find a quick git howto - but git pull updates an existing repository rather than git clone that creates a new one.

Fixed

3250 jsvar build:

http://drazzy.com/espruino/espruino_1v68_espruino_1r3_smalljsvar_bigram_wiznet.bin

In order to make the new FW work at all, I had to pick through the flash memory with the STM flash module erasing debris from previous save()'s with bigram builds. I was never able to get the "magic touch" to work (where you hit btn1 at just the right moment after rst and it starts the Espruino ignoring the saved program).

This function breaks with the new build (either using your 1800jsvar build, or my 3250 jsvar build):

function getfargostatus() {
	var fargost="";
	require("http").get(fargosturl, function(res) {
		res.on('data',function (data) {fargost+=data;});
      res.on('close',function() {var tfs=JSON.parse(fargost); vtfs=tfs; for (var i=0;i<8;i++) { fargo[i]=tfs.relaystate[i].state;} if(menustate==3){uplcd();}});
	});
}

Uncaught SyntaxError: Got '{' expected '}'
 at line 1 col 20
{var b=JSON.parse(a);vtfs=b;for(var c=0;8>c;c++)fargo[c]=b.r...
                    ^
> 

This worked in v65 bigram, and the error message makes no sense to me.

My BMP180 is returning bogus values - It says the pressure is -28347 Pa?
Maybe this is a result of the 32-bit int change?

(BMP180 module code that might be relevant is reproduced here for convenience):

BMP085.prototype.getPressure = function(callback) {
  var bmp = this;
  this.readRawTemperature(function(UT) {
    bmp.readRawPressure(function(UP) {
      // Temperature compensation
      var X1 = Math.round((UT - bmp.ac6) * bmp.ac5 / 32768);
      var X2 = Math.round((bmp.mc * 2048) / (X1 + bmp.md));
      var B5 = X1 + X2;
      var compt = (B5 + 8) / 160;
      // Pressure calculation
      var B6 = B5 - 4000;
      X1 = (bmp.b2 * (B6 * B6 >> 12)) >> 11;
      X2 = bmp.ac2 * B6 >> 11;
      var X3 = X1 + X2;
      var B3 = (((bmp.ac1 * 4 + X3) << bmp.oss) + 2) >> 2;
      X1 = bmp.ac3 * B6 >> 13;
      X2 = (bmp.b1 * (B6 * B6 >> 12)) >> 16;
      X3 = ((X1 + X2) + 2) >> 2;
      var B4 = (bmp.ac4 * (X3 + 32768)) >> 15;
      var B7 = (UP - B3) * (50000 >> bmp.oss);
      var p = Math.round(B7 < 0x80000000 ? (B7 << 1) / B4 : (B7 / B4) << 1);

      X1 = (p >> 8) * (p >> 8);
      X1 = (X1 * 3038) >> 16;
      X2 = (-7357 * p) >> 16;
      var compp = p + ((X1 + X2 + 3791) >> 4);
      callback({pressure: compp, temperature: compt});
    });
  });
};

BMP085.prototype.readRawPressure = function(convert) {
  this.i2c.writeTo(0x77, [0xF4, (0x34 + (this.oss << 6))]);
  var delay;
  var bmp = this;
  switch(this.oss) {
    case 0:
      delay = 5;
      break;
    case 1:
      delay = 8;
      break;
    case 2:
      delay = 14;
      break;
    case 3:
      delay = 26;
      break;
  }
  setTimeout(function() {
    var msb = bmp.read8(0xF6);
    var lsb = bmp.read8(0xF7);
    var xlsb = bmp.read8(0xF8);
    convert(((msb << 16) + (lsb << 8) + xlsb) >> (8 - bmp.oss));
  }, delay);
};

Numbers going into that:

"ac1":7972,"ac2":-1182,"ac3":-14539,"ac4":33201,"ac5":25505,"ac6":18922,"b1":6515,"b2":47,"mb":-32768,"mc":-11786,"md":2512

bmp.read8(0xF6);
=161
bmp.read8(0xF7);
=122
bmp.read8(0xF8);
=96

As an aside, the loss in efficiency of storing strings - does that apply to function code too? I say this because code is what takes up most of the space (that's been my experience, and iirc, Sascha indicated that was the

Thanks... With the bmp180, does it also fail on 'normal' builds? If so, that is almost certainly 32 bit as you say.

For the first error, does that fail on 'normal' 1v67 too? It seems like it might - my guess is that you're parsing invalid JSON and that it now throws an exception. Perhaps it shouldn't? I'll have to look at the docs...

The loss in efficiency does affect strings as well, but I think that given the rise in available variables it probably has a net benefit.

About JSON.parse: Looks like it actually is spec compliant now :) https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/JSON/parse

Throws a SyntaxError exception if the string to parse is not valid JSON.

I'll start doing those tests tonight or tomorrow. Seeing that 3250 jsvars is such a beautiful sight; so far my code uses around 1800 just sitting there (with a few hundred used when it's doing something)

The json the page is serving is valid... I'll need to throw in some console.log() to see if it's somehow not getting the whole thing?

{"relaystate":[ {"state":0,"rdtiev":0}, {"state":1,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0} ],"fargostatus":{ "temperature":38.5, "volts":11.02 }}

Hmm - yes, I guess it might be getting corrupt data :( Could be some issue with HTTP on the new build I guess, but if you could dump the data you get it might give us some hints

Okay, the BMP gives bogus results on the normal firmware too - Looks like it must have been the 32bit int change.

Neither of these issues turned out to be specific to the improved memory usage builds.

Okay, the BMP gives bogus results on the normal firmware too - Looks like it must have been the 32bit int change.

The http response is mondo mangled - same way every time, and it comes out correctly when viewed in a normal browser, and worked correctly in v65. So somewhere between v65 and v68, there's been a really nasty regression in http. (note - these are being tested with Wiznet)

function getfargostatus() {
	var fargost="";
	require("http").get(fargosturl, function(res) {
		res.on('data',function (data) {fargost+=data;});
      res.on('close',function() {console.log(fargost); var tfs=JSON.parse(fargost); vtfs=tfs; for (var i=0;i<8;i++) { fargo[i]=tfs.relaystate[i].state;} if(menustate==3){uplcd();}});
	});
}

Expected result:

{"relaystate":[ {"state":1,"rdtiev":0}, {"state":1,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":1,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0}, {"state":0,"rdtiev":0} ],"fargostatus":{ "temperature":35.5, "volts":10.98 }}

Actual result - where did these newlines even come from?

{"relaystate":[
{"state":1,"rdtiev":0},
{"state":1,{"state":1,"rdtiev":0},
{"state":0,"rdtiev":0},
{"state":0,"rdre":44.0, "volts":10.85
}}

eep. thanks - I'll get on this... It's probably because I'm now re-using the stream code that's used for Serial devices...

Ok - fixed in Git, or it'll be in 1v68

I've just merged all this into the main branch, so it'll be in 1v70.

Please check out the latest builds though - the sooner I'm happy that nothing has regressed, the sooner I can release 1v70 which has all these features in!

Just tested latest build(http://www.espruino.com/binaries/git/commits/8e7aac987b82d5843face571789fdab1e2c6cc5b/espruino_1v70_espruino_1r3.bin) with this:

var  l = false,led = LED1;
setInterval(function() {
  l = !l;
  led.write(l);
}, 1500);

And got Uncaught Error: Field or method does not exist.....
May be an "older bug". I did not test for some time.

That bug is new. It does not reproduce under v69 (latest release) but it does under v70 builds (I reproduced it on http://drazzy.com/espruino/bigram/espruino_1v70_08-05_espruino_1r3_bigram_wiznet.bin - those are built from a git clone made at around 1:14 AM on the listed day)

I had no idea you could do LED1.write() before, honesly.

LED2.write(1); //works
var led=LED2
led.write(1); //doesn't work

Thanks for letting me know - I'll try and get a fix in for this today.

Looks like it's converting the pin number to an integer by accident.