LedMatrix for 8*8 WS2812

I'm working on a small object for 8*8 matrix of WS2812.
The project is right now developed for ESP8266, but my problems are more general for Espruino (hope so).
Based on @nkolban video a VM for ESP8266 is running, and a lot of testing runs fine.
Since I'm a newbie in this, I learned a lot but still run into problems.
Actual there are 2 problems, and if somebody could give me a hint, ....

• JSON wrapper with more than 4 parameters (in my case setPixel(row,column,red,green,blue)) gives a warning during compiling. Sooner or later this will be changed to use a color instead of three values, therefore its not a big problem. Anyway, if somebody like me uses tons of args in the future, ...
  
• main problem is right now how to read my pixelbuffer (uint8array), change some values and store back.
  
  

Some code snippets:

• create the buffer

  JsVar *buf = jswrap_arrayBuffer_constructor(192); //192 is 8 rows with 8 columns having rgb
  jsvUnlock2(jsvAddNamedChild(parent,buf,"buffer"),buf); // I can see the buffer after that
  
  

• read buffer inside a function like setPixel

  JsVar *buf = jsvObjectGetChild(parent,"buffer",false); //read buffer
  JSV_GET_AS_CHAR_ARRAY(pixels,dataLength,buf); //convert to char Array
  uint8_t *p = (uint8_t *)pixels;//get a pointer to that
  

• doing changes like a fill

  for(int i = 0; i < 64;i++){ //loop over all fields to set colors
  *p++ = (char) green; *p++ = (char) red......
  

• and last not least store the changes back. I did only find a way by creating an JsVar for string, then convert to ArrayBuffer. This, obviously has problems with colorvalue of zero.

  JsVar *bufb = jsvNewFromString(pixels); //convert to JsVar-string, and get the zero problem (end of string)
  JsVar *bufa = jsvNewArrayBufferFromString(bufb,0); //did not find a function to create this from char array directly 
  jsvSetNamedChild(parent,bufa,"buffer");
  

JSON wrapper with more than 4 parameters

You can use JsVarArray and can decode all the arguments manually if you want more - it's not worth me wasting another byte (or more) for each function, as 99% of them have 4 or less arguments.

main problem is right now how to read my pixelbuffer (uint8array), change some values and store back.

Instead of trying to get a memory area I'd use jsvArrayBufferIterator - JSV_GET_AS_CHAR_ARRAY was added as a bit of a hack to allow other non-espruino bits of code to access data. All of Espruino's stuff is written to access the data in-place rather than copying it onto the stack first.

However... what are you trying to accomplish? Espruino already lets you access 8x8 RGB matrices of WS2812 really easily: http://www.espruino.com/RGB123

The Graphics library has everything built in, so realistically what you're doing can be accomplished in JS with:

SPI2.setup({baud:3200000, mosi:B15});
var leds = Graphics.createArrayBuffer(8,8,24,{zigzag:true}); 
leds.flip = function() { SPI2.send4bit(leds.buffer, 0b0001, 0b0011); };

leds.clear();
leds.setColor(r,g,b);
leds.fillRect(0,0,7,7);
leds.flip();

jsvArray, good point, I will try.

I want to have some animation.
I've already a JS-object supporting shifting in all directions (up/down/left/right) or rotating.
In Javascript looping throgh an Uint8array for rotating takes some time (up to 100ms).
Tested also a c-function to read a row from arraybuffer, but even using this takes some time.
Same function, in C, takes 5ms only
Some other functions similiar good old tennis game will follow, once I get the basics done.

Iterator was one of my steps, wher I gave up after a lot of crashes :-(
An example how to read a value from let me say 10th pixel and to write it to 20th pixel would be helpful.
in javascript this would be:
buffer[60] = buffer[30]; ....

You can do:

JsVar* v = jsvArrayBufferGet(arrayBuffer, 30);
jsvArrayBufferSet(arrayBuffer, 60, v);
jsvUnLock(v);

But that's not as fast. The ideal thing is to read or write data sequentially:

JsvArrayBufferIterator it;
jsvArrayBufferIteratorNew(&it, start_index);
while (pixels--) {
  jsvArrayBufferIteratorSetByteValue(&it, value);
  jsvArrayBufferIteratorNext(&it);
}
jsvArrayBufferIteratorFree(&it);

Your other option is to use 'flat strings' - if you can use them then when you use JSV_GET_AS_CHAR_ARRAY you'll get a pointer to the underlying data, so there is no need to copy back.

The ability to scroll with the Graphics library would actually be really useful, although that's a bit tricker to deal with as you need to cope with bit-based graphics where the pixels are in strange formats...

No problem if you're doing this for fun yourself - but I don't see I'll be able to include this code in Espruino itself - there's just too little flash memory available for me to be be able to use it on something that's very single-purpose.

...smells to me it would be the time for dll's --- I don't know enough about the internals, but I can imagine that it is not easy or may be even impossible... otherwise, it would already be there. On the other hand, modules and compiled deliver already a great deal of it. After all, it is not a computer with general purpose OS, it is a mC.

It's really not that hard to compile your own firmware - especially with Docker/Vagrant. Basically if you have the tools to compile a library for Espruino, you can compile the full firmware - and it'll be easier to understand than if you have to get your head around a homebrew DLL system :)

About the only thing that'd help (IMO) is a make menuconfig sort of thing that could automatically scan the libs folder for modules that you could choose whether you wanted to include or not.

Of course you can use nativeCall relatively easily to add your own compiled code like this, but you're pretty limited with how much you can interact with Espruino as you don't have access to that many of its internal functions.

I would not agree to the point "not that hard to compile".
First of all, you would need an environment to get compiling running. Without Neils video I wouldn't have tried to get this running. And even with that, a lot of those unexpected problems, that you have to expect, occured.
To get "my firmware" compiled, no changes have been made to make file. Only changes are in existing files for ESP8266. No idea right now, how actual espruino version from github could be used without loosing those changes, needed to get my firmware compiled. Lack of knowledge is obviously one part of the answer ;-)
My assumption is that more than 75% of Espruino users don't have time and/or knowledge to get this done. Next assumption is that we will have more than one extension, that some people would like to have. For example, once I got my WS2812-class running, next project will be an SHA1 implementation. For Websockets I don't need the full blown crypt with all the memory problems comin up. Latest at this time, I need to change makefile.
Both extensions will not make it into the standard firmware for Espruino, they are not really designed for that. On the other hand, I would like to share them to others. And here @allObjects comment comes in. How could we support sharing this to more Espruino users than those having the ability to compile their own version ?
BTW, just took a first look to Docker/Vagrant. From my point of view, its a first step only. Installation of developer environment with all those "unexpected problems" took a long time, and doing all of this with a script file seems to be, at least, very complex. To give an example, on my VM I never got the USB-flasher running. Or another one, never got Oracle VM running, using VMware right now, which is fine for a hobby developer but for companies ? Not even talking about using Linux, which is not that easy for windows guys.

For me, on windows, getting a custom firmware compiled can be done with these steps:

1. Install VirtualBox
   
2. Install Vagrant
   
3. start cmd && cd /path/to/Espruino
   
4. vagrant plugin install vagrant-auto_network
   
5. vagrant up
   
6. vagrant ssh
   
7. do some changes
   
8. ./scripts/create_espruino_image_1v3.sh
   
9. flash it using the best tool for the local machine, I use ST-Link Utility and I flash from Windows
   
   

Come to think of it, just follow this: https://github.com/espruino/Espruino/blob/master/README_Building.md#building-under-windowsmacos-with-a-vm-vagrant

:)

@alexanderbrevig, good point. I had in mind there was something like that, but couldn't find.
In my case I will use this for a very simple tool, to combine wifi and ws2812, I'm compiling for ESP8266.
Installation of toolchain on vmware was a kind of complex. Therefore my question:
could you please try to do ESP8266_BOARD=1 make ?
If this is running, at least some of my concerns are gone.

About the only thing that'd help (IMO) is a make menuconfig sort of thing that could automatically scan the libs folder for modules that you could choose whether you wanted to include or not. @Gordon

I was thinking of something like that too... in the form of installing/enabling (new) software in, for example, eclipse, is implemented: including the dependencies and - because memory is a hard constraint - cumulated memory requirement.

So far I did not tackle building my own firmware - shying away from getting lost in all the details as @JumJum describes. @alexanderbrevig, thanks for your outline... which for some items is a bit more than outline, but still many of the the items have their caveats. Not for nothing the forum is 'littered' with conversations of many posts to build the firmware. This is not questioning at all what @Gordon and other outstanding contributors make available through github and forum: it's just the 'crux' of the thing of many, many, heavy interdependent, moving parts. No issue for someone having grown up within, trying to enter from different fields of professions though something a daunting task. Nevertheless exciting, time (resource) is just not always on the driver's side... ;). I'm glad the forum exists and shows in - literally - deed the great willingness to share!

Vagrant would be the way to go if it works. To be honest even a script that installed the relevant tools on a basic Linux install would be good - then all you'd need is Virtualbox.

I was experimenting with jslinux and almost got it working (Fabrice Bellard gave the Ok to use it as well) - it'd be great to be able to point users at a website with the VM already on it.

On the other hand, I would like to share them to others

Maybe put them on GitHub with a script a bit like this?

git clone https://github.com/espruino/Espruino.git
cp -r mylib Espruino/libs
cd Espruino
PICO_1V3=1 make

The only thing missing for that to work is something in the Makefile that would automatically search for and include sub-makefiles in the libs folder. That could tidy a lot of things up anyway.

next project will be an SHA1 implementation

:( You know there is already require("crypto").SHA1(...)? IMO it'd make more sense to just #ifdef out SHAxxx and AES if you don't want them, rather than just reimplementing what is already there?

I think the crypto library (without HTTPS) would 'just work' on ESP8266 anyway. Worth a try - just add USE_CRYPTO=1 RELEASE=1 to the Make command-line.

Exciting and 'frustrating' at the same time:

Something taking off as 'LedMatrix for 8 x 8 WS2812' and ending up in, for example, 'Options for build environments' or 'Ways / How to build customized firmware' tells me that Forum Editors / Curators of the old days are not dead, no matter how much 'google' we throw at any 'orphaned' forum. Since nobody has the time to do edit or there are just no means to sponsor it, 'google' is the 'least worst' option.

Since creating new conversations and moving the stuff there is a lot of work and needs admin access, may be renaming this conversation is a way to go and have the topic ('issue'/'problem') introduced with 'LedMatrix for 8 x 8 WS2812' as the example at hand...

Renaming though may not be the final answer since justice has to be done to really 'LedMatrix for 8 x 8 WS2812' specifics in this thread...

I should have made a 'link' and start a new thread with 'DLL' subject, since it was not on the level of ''LedMatrix for 8 x 8 WS2812'... and @Gorden, when you see things like that, just make a link and create a conversation that show the right direction... (to all of us 'lost' sheep - sorry, no offense to any one related). Talking about direction, it would then be less a conversation or response to a post in a conversation, but more a statement... after all, you are the mastermind (Graue Eminenz - sorry for the only partially accurate Francophonism/Germanism) behind all of this.

@gordon, the copy example would copy mylib to folders, downloaded using git. Isn't there still a need to add mylib to make file ?
How would this help to share mylib ? Isn't it still on my computer only?
Creating a github for my mylibs could be done. So everybody could download, are you talking about that ? Is there a place where you would recommend to open a dictionary for tools like this ?

Is Virtualbox a placeholder for any virtual tool ? There have been a lot of problems installing VirtualBox for me, and at the end I gave up. For example, it didn't let me install 64bit version. VMware is working fine for me, even the 64bit version, and installation ran fine the first time.

In my understanding up to some minutes ago, please correct me if I'm wrong

• crypto needs a lot of memory (ram and/or flash ?), and therefore it only fits to the pico. Since ESP8266 has more restrictions for memory, I didn't expect it to work.
  
• crypto has a lot of SHAnn-functions in and can only be used as it is. To switch off some functions could be helpful.
  
  

Right now, my computer is far away, therefore I cannot check immediately.
Recommendation to use #ifdef in crypto would be a change on a standard library supported by you. Therefore I wouldn't like do any changes there. Especially for a port, still under development, which may be supported or not (there was a discussion about that). Last not least, my knowledge is too poor to feel good for changes like this.
On the other hand, I fully agree to avoid reinvention of existing libs. Its always a bad decision, since both options could have their own pros and cons.

FYI, the easiest way to build espruino once you have a linux VM (preferably ubuntu 14.04) is to follow the steps in the .travis.yml file. It comes down to just:

git clone hhttps://github.com/tve/Espruino.git
curl -Ls http://s3.voneicken.com/xtensa-lx106-elf.tgx | tar Jxf -
curl -Ls http://s3.voneicken.com/esp_iot_sdk_v1.5.0.tgx | tar Jxf -
cd Espruino

I haven't tested this, so I may be missing one command in there to set some environment variables, but a long install process is certainly not necessary. Travis runs this on ever build.

ESP8266_SDK_ROOT needs to be set to esp_iotsdk....
and xtensa-lx106.... is appended to path
Thats what I read from travis.yml.
But now its time to go to bed and get some sleep.
Will test this later today, for me its already wednesday

Isn't there still a need to add mylib to make file ?

At the moment, yes. As I'd said above, the ideal solution would be to have something in the Makefile that scanned the libs (or another) folder and included any files that it found.

How would this help to share mylib ?

You'd have to stick it on GitHub - I'd be happy to add a page to the website with instructions and a link to your (and others') code.

Is Virtualbox a placeholder for any virtual tool ?

Yes - assuming you can run a recent version of Linux on something (whatever) then it should be quite easy to make a script to install everything (in fact as @tve pointed out, you can pretty much just copy what .travis.yml does.

crypto

The crypto libs shouldn't use up any RAM until they're used - they're really not that bad at all. It's HTTPS that's the killer (it can be compiled in separately), although even that doesn't use RAM until you use it.

... having said that, due to the way the ESP8266 works, the crypto libs could end up using some RAM as any constant arrays will be stored there by default :(

@Gordon, could this be an option ?

 ifdef USE_MYLIB
   WRAPPERSOURCES += $(wildcard myLibs/jswrap*.c)
 endif

Yes... I also wondering whether it could just append to WRAPPERSOURCES, such that you could do:

WRAPPERSOURCES=mylib/jswrap_foo.js make

It would then be easy enough to have a script that just called make with whatever extra files you wanted...

Some days ago, I started to compile my own firmware with an extension to handle LEDMatrix(WS2812). Main focus was to add simple animation. This is an video, showing actual status (see animation at 3:40). video

In the video you will also see short description of available commands, and some Javascript functions that have been used for testing.