chunks of an Uint8Array

Posted on Wed 23rd, November 2016

• #1

MaBe

this works excellent for Espruino

var C = {
 OLED_WIDTH                 : 128,
 OLED_CHAR                  : 0x40,
 OLED_CHUNK                 : 128
};
WIDTH = 128;
HEIGHT = 64;
var buffer = new Uint8Array((WIDTH * HEIGHT)/ 8);
bufferLen = buffer.length;
var chunk = new Uint8Array(C.OLED_CHUNK+1);
chunk[0] = C.OLED_CHAR;

for (var p=0; p < bufferLen; p+=C.OLED_CHUNK) {
  console.log(p +"-"+(p+C.OLED_CHUNK));
  chunk.set(new Uint8Array(buffer,p,C.OLED_CHUNK), 1);
}

but anywhere else it causes something like "RangeError: Offset/length out of range" for chunk.

Any suggestions how to rewrite this line

 chunk.set(new Uint8Array(buffer,p,C.OLED_CHUNK), 1);

to work with other javascript engines ?

• #2

Gordon
Looks like it's that chunk.set(new Uint8Array(buffer,p,C.OLED_CHUNK), 1); should really be chunk.set(new Uint8Array(buffer.buffer,p,C.OLED_CHUNK), 1);

In the browser:
```
(new Uint8Array((new Uint8Array(1000)).buffer,0,8)).length
=8

(new Uint8Array((new Uint8Array(1000)),0,8)).length
=1000
```
It's one of those times where I can't help thinking that Espruino does the more sensible thing, but it's not compliant with the spec.
• #3

Gordon

Ahh, and I just realised - the module itself that you copied that code from is perfectly JS compliant and fine.

The problem is that Graphics.buffer is an ArrayBuffer (so untyped). You replaced it with a Uint8Array :)

• #4

allObjects

How about... (corrected):

chunk.set(buffer.slice(p, p + C.OLED_CHUNK), 1);

• #5

MaBe

thanks - I was so focused on chunk.set :-(

btw : The code of module SSD1306 if the fastest i have found so far - great work !
• #6

MaBe

Oh yeah - that works too - thanks allObjects
• #7

Gordon
@allObjects there is a slight problem with that:
- As per the spec, buffer.slice creates a brand new sparse array - it uses lots of memory and time to create
- Using new Uint8Array(buffer,p,C.OLED_CHUNK) only creates a 'view' of the data - nothing is copied until the set command.
Basically the .set(new Uint8Array(buffer,x,y), z); pattern is about as close to a memcpy in JavaScript as you can get
• #8

allObjects
edited... (and see post #10)
I expected the negative side effects... unfortunately (Chrome) browser just does not behave as (EDITED: by me misread) MDN constructor specs / doc ...

In test code new Uint8Array(buffer, begin, length).length return just the length of buffer... and on .set() it reports Uncaught RangeError: Source is too large(…).

In Chrom inspec/debug console:
```
> var b = new Uint8Array(10);
< undefined
> var t = new Uint8Array(5);
< undefined
> t.set(new Uint8Array(b,0,5))
X> VM380:1 Uncaught RangeError: Source is too large(…)(anonymous function) @ VM380:1
> (new Uint8Array(b,0,5)).length
< 10
```
Found no complaint on the Web about this violation...

Write a compiled copy function, could that help speeding up an element-wise copy?
• #9

Gordon
As I'd said above, Chrome is spec compliant. See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Uint8Array

The constructor is:
```
new Uint8Array(length);
new Uint8Array(typedArray);
new Uint8Array(object);
new Uint8Array(buffer [, byteOffset [, length]]);
```
In this case, you're not passing in buffer, but typedArray - so the next 2 arguments are not being used. If you instead so (new Uint8Array(b.buffer,0,5)).length it'll work

• #10

allObjects

NICE! --- me: ddddoooohhhh (: ... LOL...

That's where implicit - wishful - typing by variable naming (trap) kicks in... the 'wrong' name has been chosen for the variable: buffer is not a buffer because it is called so... ;(

I made a rough performance test, and it shows that slice is still about 6 times slower (even with some optimization / moving the var s out of loop)... memory is for sure contributing to the time aspect... - you can directly run the attached .html file by just clicking on the link. Test on Espruino still pending.

<html><head><title>typedBuffedSliced</title></head>
<body><h3>typedBuffedSliced</h3>
<button onclick="exec()">exec()</button>
<script>
var C = {
 OLED_WIDTH                 : 128,
 OLED_CHAR                  : 0x40,
 OLED_CHUNK                 : 128
}, WIDTH = 128, HEIGHT = 64;
function exec() {
  var buf = new Uint8Array((WIDTH*HEIGHT) / 8), bLen = buf.length;
  var chunk = new Uint8Array(C.OLED_CHUNK+1), cLen = chunk.length;
  console.log("buf: " + bLen + " - chunk: " + cLen);
  chunk[0] = C.OLED_CHAR;
  var p, pe, c1, t10, t11, slice, c2, t20, t21, cL = C.OLED_CHUNK;
  for (p=0; p < bLen; p += C.OLED_CHUNK) { pe = p + C.OLED_CHUNK;
    t10 = new Date().getTime(); for (c1 = 0; c1 < 100000; c1++) {
      chunk.set(new Uint8Array(buf.buffer,p,cL), 1);
    } t11 = new Date().getTime();
    t20 = new Date().getTime(); for (c2 = 0; c2 < 100000; c2++) {
      chunk.set(buf.slice(p, pe), 1);
    } t21 = new Date().getTime();
    console.log(p+"-"+(pe)+": " + ((t21 - t20) / (t11 - t10)));
  }
}
</script>
<pre>
Console:
buf: 1024 - chunk: 129
0-128: 4.375
128-256: 5.529411764705882
256-384: 7.142857142857143
384-512: 7.142857142857143
512-640: 7.846153846153846
640-768: 5.875
768-896: 6.928571428571429
896-1024: 7.615384615384615
</pre>
</body>
</html>

PS: updated - (line 18 had buf.buf and passing undefined which did speed it up to factor 2)

2 Attachments

typedBuffedSliced.html

• #11

allObjects

Espruino test (uploaded from editor pane and invoked with exec() in the console on Espruino Wifi):

var C = {
 OLED_WIDTH                 : 128,
 OLED_CHAR                  : 0x40,
 OLED_CHUNK                 : 128
}, WIDTH = 128, HEIGHT = 64;
function exec() {
  var buf = new Uint8Array((WIDTH*HEIGHT) / 8), bLen = buf.length;
  var chunk = new Uint8Array(C.OLED_CHUNK+1), cLen = chunk.length;
  console.log("buf: " + bLen + " - chunk: " + cLen);
  chunk[0] = C.OLED_CHAR;
  var p, pe, c1, t10, t11, slice, c2, t20, t21, cL = C.OLED_CHUNK;
  for (p=0; p < bLen; p += C.OLED_CHUNK) { pe = p + C.OLED_CHUNK;
t10 = new Date().getTime(); for (c1 = 0; c1 < 100; c1++) {
chunk.set(new Uint8Array(buf.buffer,p,cL), 1);
} t11 = new Date().getTime();
t20 = new Date().getTime(); for (c2 = 0; c2 < 100; c2++) {
chunk.set(buf.slice(p, pe), 1);
} t21 = new Date().getTime();
    console.log(p+"-"+(pe)+": " + ((t21 - t20) / (t11 - t10)));
  }
}

Console:

 1v87 Copyright 2016 G.Williams
>exec()
buf: 1024 - chunk: 129
0-128: 4.07852737698
128-256: 4.52856306801
256-384: 5.24940776218
384-512: 5.27483808673
512-640: 6.57535412290
640-768: 8.54367417763
768-896: 7.12319090028
896-1024: 7.51280680688

Interesting is the relatively great variation... so I run it with HEIGHT = 128 - double the original buffer... and variation is worse:

 1v87 Copyright 2016 G.Williams
>exec()
buf: 2048 - chunk: 129
0-128: 2.42377434903
128-256: 6.01698313765
256-384: 3.10307471306
384-512: 5.11592931261
512-640: 8.91618411518
640-768: 9.59016115959
768-896: 8.67426096522
896-1024: 8.14185429263
1024-1152: 13.84030717299
1152-1280: 13.78322895572
1280-1408: 8.70759501664
1408-1536: 10.94154115774
1536-1664: 22.54689730282
1664-1792: 12.00001317154
1792-1920: 34.27794204212
1920-2048: 16.76400105146

Have no answer why...

• #12

Gordon

So you're saying that it gets worse the larger the offset in the ArrayBuffer?

I think it's due to the implementation of slice on Espruino. It's expecting that it'll run on a normal sparse array, which is a Linked List.

Because of that it's just iterating to get to the correct start index - even though with an ArrayBuffer it could just go straight there in most cases. I've just filed an issue for this.
• #13

MaBe

wow - thanks to both for your effort and those very interesting explanations and samples !
• #14

MaBe

So let me share the motivation behind this.

All npm modules out there are using Oled data updates byte by byte - so you can imagine how slow the screen updates are - takes more than 1sec for a full screen update.

The Espruino SSD1306.js does it different, updates are so quick - because it uses chunks to update.

So I decided to pimp the npm module I use for my beaglebone projects.

And this is the result: https://github.com/MaBecker/oled-i2c-bus/commit/c099f6145024ea9b7c190d16a936eba45232e2a8
• #15

Gordon

Neat - thanks for the comment in the commit!

Post a reply
- Bold
- Italics
- Link
- Image
- List
- Quote
- code
- Preview
Formatting Help

Don't worry about formatting, just type in the text and we'll take care of making sense of it. We will auto-convert links, and if you put asterisks around words we will make them bold.

Tips:

Create headers by underlining text with ==== or ----

To *italicise* text put one asterisk each side of the word

To **bold** text put two asterisks each side of the word

Embed images by entering:
![](https://www.google.co.uk/images/srpr/logo4w.png)
That's the hard one: exclamation, square brackets and then the URL to the image in brackets.

* Create lists by starting lines with asterisks

1. Create numbered lists by starting lines with a number and a dot

> Quote text by starting lines with >

Mention another user by @username

For syntax highlighting, surround the code block with three backticks:

```
Your code goes here
```
Just like Github, a blank line must precede a code block.

If you upload more than 5 files we will display all attachments as thumbnails.

For a full reference visit the Markdown syntax.

chunks of an Uint8Array

About