Espruino on ESP8266

Ok, just done... I think? https://gitter.im/espruino

I'm afraid I can't really devote any time today - my wife will be properly fed up. I'll try and find some time to look over stuff, but I can properly spend time on this on Monday (I know you'll be at work then though most likely).

@Gordon ... awesome ... youve created a Gitter area for your account (Espruino) but haven't yet created a room for the Espuino project. What you have to do is click the "Create a Room" button at the bottom left of the Gitter tool and select the Espruino repository from within the Espruino account.

Ok, cool - just done.

The merging of the ESP8266 port and ESP8266 specific code to the master Espruino code base has now been completed. The Espruino github repository is now the exclusive place for the past work performed for an ESP8266 port and will be the exclusive place where future changes are made.

@Gordon has been an awesome owner and steward of the project and without his assistance and support, we would have been sunk. He is also extremely responsive and the merges were approved VERY quickly.

status update:

1. Gordon merged all the good work @aplikatika and @Kolban did into the main repo, yay!
   
2. I spend time bringing the esp8266 compilation in-line with other targets and went down a rabbit hole when I tried to enable function inlining as is done for other targets: things ceased to compile using gcc 4.8 but worked for Kolban in windows using gcc 5.1
   
3. In the end, by turning link-time optimization on and going through some linker gyrations to deal with iram0/iram1 sections the build worked again for me on linux with gcc 4.8; the link time fun is documented in the Makefile https://github.com/tve/Espruino/blob/master/Makefile#L1480-L1497
   
4. A big issue in all this is that gcc 4.8 uses -std=gnu89 as default and that produces lots of errors, it would be good for Gordon to state which value of -std is intended and which version of gcc needs to be used
   
5. I started to create additional esp board targets to distinguish 512KB/1MB/2B/4MB flash sizes, this becomes important for OTA update and to add a SPIFFS filesystem to store data and programs, this is still only in my private fork and not ready yet
   
6. Kolban points out that it's important to get SPI, I2C and similar other things to work
   

@Gordon: you have some 6-7 functions in jsvars.[hc] defined to be force-inlined but the function body is in the .c file instead of the .h file. The result is that this can only work with link-time-optimization where the compiler emits intermediate code into the object file and basically the second stage of the compiler runs at link time. While this is cool, given that it's just about 6-7 functions, what do you think about moving these into the .h files so LTO is not required. I have not looked into the code so I don't know whether there's an important reason for these functions to be in the .c file. At this point this is no longer a blocker for the esp8266 but the next port may well run into the same issues.

it would be good for Gordon to state which value of -std is intended and which version of gcc needs to be used

gnu89 I think? The one that doesn't allow for (int i=0;...). I'd actually quite like to change that though as without it, code is needlessly complicated.

Kolban points out that it's important to get SPI, I2C and similar other things to work

SPI is less of a big deal because Espruino can do software SPI out of the box. It can't do software I2C (yet?) or Serial though.

Also, a big one (IMO) is the utility timer. I don't even know if it's possible (it should be called from a hardware timer via an IRQ), but adding support for that would enable a lot of extra stuff - controlling servos, software PWM, all sorts.

you have some 6-7 functions in jsvars.[hc] defined to be force-inlined but the function body is in the .c file instead of the .h file.

I think the INLINE keyword used is defined in jsutils.h so that if there's no LTO they're not inlined? It just seemed tidier - it may also be that given the nature of ESP8266 it's actually faster to not inline them (maybe to just stick them in that first block of flash memory?).

You could look into it, I thought more functions were force-inlined than just 6 or 7, and I think those functions also depend on other functions defined in jsvar.c being inlined into them. There are some in jsvariterator.h too... It just seemed tidier - as the compilers will do it, I though it was better to make use of it.

Did your LTO fixes work for GCC 5 in the end? I'm hitting those kind of errors with LTO on the ARM-based NRF51/2 ports too, and I'm not sure why because as far as I can tell there's no real linker magic going on.

Nope, the LTO fixes didn't work on gcc5. The internal compiler error is fixed in gcc head and scheduled for 5.2. I did not fiddle with options a whole lot to see whether I could get around it.

The always_inline functions that cause all the trouble I believe are https://github.com/espruino/Espruino/blob/master/src/jsvar.h#L305-L321

Also, a big one (IMO) is the utility timer.

There definitely is timer support, but it goes through FreeRTOS and I believe is scheduled on "the" app thread, so it may or may not be good enough. For millisec resolution yes, for lower, dunno.

Ok, that's a shame. However it's unsurprising I guess.

What we need is someone from EspressIF... It might be there's some free timer hardware in there somewhere :)

Also, I guess setWatch isn't implemented yet? That's quite a big one that people will expect. It'd be good if the OS supported it, but if not it could be doing by polling (there's an example of that in the Linux build).

It might be there's some free timer hardware in there somewhere :)

Could you elaborate on what you expect, perhaps use-cases you expect to cover? In the end the issue is LwIP, which is not multithreading friendly. As a result, the Espressif SDK manages the application thread carefully. It would be easy to create a timer interrupt, but that will run on some random FreeRTOS thread and it won't be able to safely perform any network I/O.

My recommendation would be to use the timers provided by the SDK and see where that leads us. The step up would be to use interrupts in some form and to use them to drive some yield() type of stack switching in the interpreter loop so the code triggered by the timer can run on the normal application thread.

The esp has an RTC, but I don't know how accurate that ends up being. Fortunately with Wifi NTP usually isn't too far away.

It would be easy to create a timer interrupt, but that will run on some random FreeRTOS thread and it won't be able to safely perform any network I/O.

That's perfect - if it was accurate (ish - 0.01ms?). this code expects to set up a call to jstUtilTimerInterruptHandler after a set time period. The IRQ works off its own circular buffer, so shouldn't interfere with much and definitely doesn't call into any OS functions (apart from GPIO or analog IO).

It's for digitalPulse, waveforms, and Espruino's software PWM

I see, so it's not something that is exposed to javascript, i.e., setTimeout()?

Nope - only via digitalPulse, pwm, etc - but no JS code can run from it

I'm a bit disappointed by the way the network is hooked in from an event perspective. With the esp8266 each packets gets buffered and copied many times. As far as I can tell, for receive: LwIP buffers the packet, the SDK notices and probably copies it into a buffer of its own (not sure about that), then invokes a callback into user-code (the esp8266 esprunio "driver"), there it's copied into another buffer (required by the callback semantics) and the callback chain is dropped. Later, in the idle loop, espruino discovers "oh, look, there's some data there" and does a recv, which copies the data again, and I believe it's copied yet once more into a JS data structure?
I may not have all the details right, but is there not a way to queue an event for the "idle loop" instead of having it poll every socket all the time, which is also in the end what requires all the intermediate buffers?
On the sending side the buffer copying is not much better and the "you can send more" event is similarly dropped and the idle loop rediscovers it.

My thinking has always been that there is a current Espruino architecture in place that @Gordon designed and built to support a number of partner networking devices. I too believe that there may be alternative designs available but my thinking at this stage of the project is to adhere to the current designs in place. The alternative would be to re-architect and re-work existing designs which (opinion) would delay the availability of a good ESP8266 port.

My vote would be to get as much of the ESP8266 port working as possible adhering to the existing designs and then ... when the port has reached a stage that some might call "complete" ... then we can turn our attention (if needed) to suggestions for core Espruino internal re-designs.

Now ... as we go through the ESP8266 porting work, things like you have said are golden and should not be lost but instead, perhaps, captured as new "Issues" for subsequent evaluation and addressing.

It took me a ton of time to get my mind around how the networking subsystems worked ... but once I did that, the actual amount of design and coding needed to "integrate" with the existing design wasn't horrible. That leads me to believe that if future rework was needed at the ESP8266 level because of a change to the way Espruino core was driving it, that wouldn't be radical surgery.

(all opinions)

Yep, I agree with you both here.

It's not great right now, but it does work - if not as efficiently as it could. I think getting ESP8266 actually working reliably with hardware peripheral support should really a priority before you start optimising.

Right now the API is the way it is because it's easy and pretty efficient on 'normal' Espruino platforms - it's also a bit like a normal socket layer so it won't totally throw people. We could definitely add some kind of data_received and get_data_to_send callbacks at some point, but I'd want to make really sure they weren't overly complicating the existing stuff just for the sake of ESP8266, and that any changes got tested properly on Espruino hardware and CC3000/WIZnet/etc first.

I think getting ESP8266 actually working reliably with hardware peripheral support should really a priority before you start optimising

It's not because I ask about advanced features that I'm necessarily optimizing right now...

We could definitely add some kind of data_received and get_data_to_send callbacks at some point

I'm puzzled, isn't that what Socket.on('data' and Socket.on('drain' do?

They're for JS code - but you need a way to feed socketserver.c.

While in the case of TCP/IP sockets it goes straight through, socketserver needs to grab the data first for HTTP so it can decode the headers. When someone implements Websockets it'll be important too, as it'll have to strip the header off each packet.

Ah got it, I didn't understand your previous comment correctly.

Just saying I'm very interested in WebSockets though I don't have the technical knowledge to implement them sadly. CNLohr did a cool video about them on the ESP8266 here
but its in C. I just wounder how fast they could be when done in JS.

Yes, websockets have been asked for a bit - they could be implemented in socketserver.c in a way that worked for all internet connection types on Espruino.

Also all the work could be done and tested with a pure linux desktop build so it's actually not too hard for someone else to do, it's just that nobody has done it yet.

Unfortunately I have a pretty big To Do list, and that's not really that near the top of it at the moment. To me, they don't really add a great deal. Sure, you get a UI with a button that can do stuff in 10ms and not 200ms - but apart from toy examples, I'm not sure how many real-world things that lets you do that couldn't have been done with HTTP.

On the subject of Websockets ... there is an issue already in flight for that ... see #258

https://github.com/espruino/Espruino/issues/258

That might be a great place to capture technical and pertinent notes on Websockets as needed.