Driving WS2811, WS2812 and other NeoPixel devices from ESP8266

While working on the ESP8266 port of Espruino, we came across requests for driving WS2811, WS2812 and other NeoPixel devices. Unfortunately, hardware ESP8266 support for SPI or I2C isn't there yet. However, others in un-related ESP8266 projects have pioneered correct driving of NeoPixels using C and Assembler. These code snippets can easily be incorporated into Espruino builds as new modules. For example:

var neopixel = require("NeoPixel"); // This is made up and not real.... don't try it
neopixel.write(D1, [r1, g1, b1, r2, g2, b2, ...]);

However, just because we can do something ... never means we should. Now we have a community and project owner question. What should we do about the (hopefully few) modules that need to be implemented in the Espruino base code in C?

These are the choices I see:

1) We bake it into Espruino ... it is just "there" for folks to use ... it increases the firmware size by an amount but we live with it. (I think the WS2812 code came out to be an increase of about 150-200 bytes).

2) We don't bake it into Espruino ... but leave it in the "source" such that a custom build could be attempted by those who wish to make a custom build containing the code.

3) As (2) but multiple firmware builds are distributed for each release ... some with features and some without.

4) We don't accommodate function into Espruino that isn't "obviously" going to be used by the vast majority of Espruino users.

Let's open this up for discussion and see what we, as a community, think and can come up with. Maybe there are other ideas or solutions? However, lets be practical ... ideas such as "Let us create a build server where a custom Espruino build can be generated on demand based upon a user's selection" sure sounds awesome ... but multiply the number of boards and environments ... and monies ... that might be needed for that ... it isn't likely to fly ... and ... if it did, chances are we would have to fall back to one of the simpler ideas so that function could be delivered before such a "build machine" became practical and then we might be back at the start again.

Besides NeoPixel support, what have others heard of that might require "native C" support within Espruino?

Just to add for others - driving WS2811/etc on normal Espruino boards is absolutely no problem at all. See the WS2811 page.

@Kolban what are the chances of just fixing hardware SPI support on the ESP8266? For this specific issue that would seem to be the most sensible course of action, and would be useful to far more people?

Or, maybe you could detect the specific use of SPI.send4bit at the given speed, and could just shift over to the WS2811-specific software implementation for that?

Since it is possible to have 'native' functions - even in ESP8266 - I guess one other option is to have a set of native code functions that get built for each major release of Espruino, and which can be loaded in on demand - but again that's quite a lot of effort.

The issue for me is I think it's good to have flexible support built in to Espruino itself, but building support for one specific bit of hardware just seems like a bad idea - even more so when it's for one bit of hardware on one specific processor.

I suppose one more option is to make a simple neopixel 'wrapper' module for normal Espruino boards. We could tweak Modules.addCached so it wouldn't overwrite built-in modules, and then there could be a native equivalent module just for the ESP8266, that would take over from the normal Espruino one.

Oh.... I see why you need it on the ESP8266...

You can't use the SPI pin, since it only has one SPI, and that's used for the EEPROM, so any use of the EEPROM would confuse the WS2812's...

Ugh.

There will need to be some solution to this for the ESP8266 - it's gotta be able to drive neopixels.
I'm not sure what the right way to handle this is - I guess you need the bit of native code to do that on the ESP8266, since the normal Espruino way of controlling WS2812's won't work...

In general, I don't like the idea of doing that sort of thing, but I don't see an alternative for this specific case. On STM32 platforms, with multiple SPIs, I think we should stick to the current method, as it covers more than just the WS2812's, and I haven't heard people complaining that they don't have enough SPI pins for WS2812's on the normal Espruino boards.

I suppose one more option is to make a simple neopixel 'wrapper' module for normal Espruino boards. We could tweak Modules.addCached so it wouldn't overwrite built-in modules, and then there could be a native equivalent module just for the ESP8266, that would take over from the normal Espruino one.

This.... this might be a good, more general solution to this sort of case...

First of all, I would like to have more options to extend firmware by myself. Let me add my 2 cents:

• adding all modules to Espruino Firmware that are helpful to somebody makes development and maintenance a nightmare. Who would be (or at least feel) responsible ? If I would be Gordon, my answer would be "not me anymore".
  
• modules like neopixel can be big or small. There are some implementation for WS2812 and others that give a lot of options. Which one should be implemented in Espruino ?
  
• some modules are helpful on special platform only. For example, Espruino boards already supports WS2812, why should there be one more module ?
  
• Espruino already supports binarys, see http://www.espruino.com/Assembler and http://www.espruino.com/Compilation.
  
• binarys are machine code, therefore we will never have binarys running on all platforms.
  
• by working with WebSockets I would like to have SHA1 implemented, to give one more example
  
• something I was working on some time ago was a PID, one more example
  
• some already available function in Espruino are needed by few people only, like FFT, CC3000, tv, ...
  
  

What I could imagine is something like this:

• use compiler of WebIDE to get a code-frame (function mytest(a,b){ return a+b;}) in a cpp file
  
• add C-code to do whatever somebody wants to do inside
  
• compile and upload this binary
  
• or take binarys, that somebody else created
  
  

Right now we have some limitations, even I know about

• we need actual exports from process.env. This will change with each new release. So we need a more general way to get this information.
  
• binarys right now are stored in memory not in flash.
  
• at least in assembler we can store local variables inside the binarys, a nogo for flash
  
• we don't get C-code from actual compiler service
  
• compiler service works for Espruino(ARM?) only, there is nothing similiar available for ESP8266, EMW3165, NRF52, ...
  

Yes, being able to pull in C code would be good.

For me, the problem is the compiler service. I have to run a server with that on, and right now I do it, for free, for any ARM boards. Honestly I don't see any reason I should take on the maintenance headache of supporting other boards though.

... and the reason I don't allow C code right now is it seems like a security nightmare. I'd somehow have to find a way of making sure someone didn't include "/etc/passwd" in their files.

However (and this is something I've been thinking about for a while), why couldn't users spin up their own VM and compile a custom Espruino firmware themselves? For instance Virtual x86, which runs in the browser.

It wouldn't be fast, but I imagine it'd take 10 mins max to compile the Espruino firmware - I'm sure people would be willing to wait.

I'd think that with a relatively small amount of hacking, someone could come up with a webpage that loaded v86 with a Linux image, downloaded the zip of the latest Espruino source code from GitHub, built it, and then let you save the result to a file. Potentially all of that could even be built into the Web IDE at some point.

So someone could:

• Install arm-gcc and xtensa-gcc on an image for v86
  
• Come up with a webpage for v86 that grabbed the ZIP off GitHub, started the build, and then added a download link
  
• Add an option in the Web IDE to allow flashing from file (which would be trivial)
  
• Improve Espruino's build system. There could maybe even be a make menuconfig that gave you a nice UI where you could choose which components you wanted in your build.
  
  

To be honest I don't really have time for this right now - I could probably get the Web IDE loading from a file though.

The good news is that I have mechanically tested calling NeoPixels from JS using a combination of C and assembler as the driver and it seems to work great. Using JS as the driver means I can write net code and use JS as the driver for the pixels.

However, until and unless we reach consensus on whether or not to include it in the ESP8266 base distribution, I'll leave it out of the GIT repository.

...so this conversation is not really about NeoPixels. Can we change the title to something like Custom builds, of Builds on demand, Selective builds,...? ...and use NeiPixels as (the) practical example for pursuing this effort?

Old technology used several steps to get 'there': compiling delivers object modules that include meta data which then in a linker were used the put all the stuff together... Tubo Pascal was a great example to get rid of this multi-step process - even the need for multi-pass compiling - to get faster to the executable. I'm not familiar enough with the tool chain/build process of Espruino (and alike), but my guts tell me that something modelled after the old world could work - and it well may already work this way, just named differently.

Like every module has dependencies, module components could have dependencies... (Classes have dependencies on their members for state (data type classes) and behavior (method used classes). Java (and a lot of alikes) are stuck with atomic-granularity on the class level - matches a file. There are other implementations that go further and allow granularity on the method level: A class can be extended by adding methods without having to subclass (What I have not seen yet is comparable state extensibility). All in all, it would not be composition yet, but comes close... Javascript can do all that at runtime... but I'm looking for a build time for that: compose sources on that level.

As a concrete idea, a build system could present a catalog of component and sub-components with check-boxes, with dependency support/constraints active. Checking/unchecking a functionality will also pick(indicate check) the minimal dependencies, and at the same time say what the (minimal) hardware requirements are in RAM and EEPROM, and as an icing on the cake: allow assignment of requirment compatible hardware profiles, such as PICO, Standard, ESP8266-xx,...

If this all could be done in an IDE extension (additional page or left-hand side), why not...

I think we probably don't want to miss the wood for the trees. The reason that there is a current usefulness for native code in C/Assembler for NeoPixels is because they have sub microsecond timing requirements that can't be done at the JavaScript level. To the best of my limited knowledge, this appears to be the only example of such a story. If it is a very rare boundary case ... then I'd hate for us to set up an infrastructure for custom builds and there be no other value or need beyond NeoPixels.

If a contributor to Espruino internals development has an hour to spare ... would we want that hour spent on custom code inclusion or would we prefer that the hour be spent burning down the open Issues that already exist?

The NeoPixels story came up because there was an apparent "usage pattern" being asked for by user after user. However, I'm not hearing of other stories that seem to "require" native code (C/Assembler) support.

I think realisatically that falls under @Kolban's 'However, lets be practical ...' heading :)

But I think a v86-based build would work very well indeed, with minimal effort. Even that would be a great help, and make menuconfig could be added later, which would do exactly what @allObjects suggests (nice UI, dependency checking).

I'm lost here... Is this about neopixels? And can neopixels be driven by using SPI hardware? Then isn't the solution to use the available esp8266's SPI hardware interface? Or am I missing something?

A NeoPixel also known as the WS2812 has an ultra specific set of timing criteria ... see for example:

https://www.adafruit.com/datasheets/WS2812.pdf

We need a signal length of 0.35us to represent low and 0.7us to represent high. These have tolerances of +/- 150ns. The overall period of 1 bit is about 1.3us. As such, timing is absolutely critical. The AdaFruit sample code for an ESP8266 can be found here which uses assembly to access the ESP8266 special register called "ccount" which uses the 80MHz/160MHz clock cycle counter for this level of precision:

https://github.com/adafruit/Adafruit_NeoPixel/blob/master/esp8266.c

The equivalent Adruino code seems to be mostly written in assembler ... see:

https://github.com/adafruit/Adafruit_NeoPixel/blob/master/Adafruit_NeoPixel.cpp

Using these as references, we see both an API and an implementation that has been demonstrated to work. If it is possible to drive reliable signals at these rates using SPI, I'm afraid I personally don't know how to do that. We are also stuck on hardware SPI support on the ESP8266 and that has been an outstanding issue for a while now with no obvious resolution.

If we can achieve NeoPixel driving with hardware SPI, that would be fantastic ... but in the interim, this story talks about a potential code fragment that works today using the recipes that others (Adafruit) are advocating ... as opposed to SPI techniques. If it is possible to drive NeoPixels using hardware SPI on an ESP8266, that may indeed be a breakthrough that even folks like AdaFruit would be interested in hearing about.

Neopixels can be driven by SPI hardware, as Espruino has always done. You just need the clock speed to be about right, and then you can send the pulses as a series of 0b0011 and 0b001 nibbles that get clocked out.

The good thing with that is it gives you enough headroom to decode Espruino's array structures. To use Adafruit's method you'll have to decode the arrays into a flat area of memory and then feed through that...

Rather than adding a hack just for flashy lights on ESP8266, if hardware SPI was working it'd make ESP8266 a whole lot more faster for all kinds of things and all the existing tutorials for WS2811 should 'just work'.

Isn't the ESP8266's only SPI used for the EEPROM that holds the firmware though? Or does it have a second one?

@DrAzzy ... There is an open issue on hardware support for the ESP8266 SPI which is Issue #606 ... https://github.com/espruino/Espruino/issues/606

There are a bunch of links from there to the available docs and other forum entries. From what I can tell, it appears that one can use a hardware SPI for I/O driving without it affecting the operation of flash ... however, I for one, have not been able to comprehend the documentation to figure out a design that would allow us to include the hardware SPI support in Espruino. This is one of those areas where I had been hoping to engage an SPI skilled ESP8266 resource ... or at the very least, find others who would be willing to collaborate on bit-by-bit comprehension of the story.

Hmm.... I'm sure that could work in the case of "well behaved" SPI devices, with a CS pin and all that don't care what comes down the MOSI line when CS isn't asserted. But the WS2812's don't have that - I think any shit coming down the MOSI line would (or could) confuse them...

I have found a ready HW SPI library for ESP8266...
https://github.com/MetalPhreak/ESP8266_SPI_Driver
@Gordon @Kolban @tve can you look at it and review whether it can solve our problem with HW SPI?

It's a good point about the CS/MOSI @DrAzzy - if the HWSPI can't be connected to any other pins then it might not work for WS2811 :(

Take a look here: https://github.com/nodemcu/nodemcu-devkit-v1.0/blob/master/Documents/NODEMCU-DEVKIT-V1.0-INSTRUCTION-EN.pdf

The SPI on the left side (orange SDIO pins) is used to talk to the flash. The HSPI on the right side (bright green) can be used freely.

Awesome - thanks!

I guess this does put a spanner in the works for someone who just wants to use WS2811 with the ESP01 board though :(

Yup, no HSPI on ESP-01 boards.

The MetalPhreak library proved to be a g*d-send ... fantastic. The latest firmware build now includes the first pas at hardware SPI and Github issue #606 contains the code which has been submitted as a pull request. Now we just need some users to test and see what happens.

One can use NeoPixels with an ESP-1 ...

The first is that although the hardware SPI is not exposed on edge pins, it is still there on the physically exposed ESP8266. In "theory" one could solder on some wires directly to the SMD IC. From my perspective, I'd discount this immediately ... but Ive seen it "talked about" in some forums ... I for one couldn't solder that finely.

The other option would, of course, be the 1 wire solution written in C code that had been mentioned before ... bypassing the use of SPI.

As I was thinking about SPI driven NeoPixels on an ESP8266, we may have distaste for that solution. It strikes me that the hardware SPI will now consume 4 pins for driving the NeoPixel as opposed to the 1 that is strictly needed. SPI has 4 pins:

• MOSI
  
• MISO
  
• CLK
  
• SS
  
  

Since we will be driving data down MOSI and ignoring the other 3 pins, that may be considered a waste if users are finding themselves pin constrained. We might also find ourselves getting stuck if users need hardware SPI in addition to NeoPixel driving. For example, if the data describing the colors of the pixels over time were on an SD card which was being read by hardware SPI, we would need to introduce some form of SS to decouple it from MOSI data that it would otherwise detect when communicating with other devices.

So maybe the dedicated WS2812 driving code should just be included as well in the Espruino build for ESP8266? These are all valid points:

• Non-availability of HSPI on ESP-01 boards (soldering wires to a QFN package is practically not viable)
  
• Wasting 4 pins where only 1 is needed
  
• Drive more than one LED strip
  
• The increase in binary size is really small