Architecture: Core processing of jsiLoop

In the Espruino port to the ESP8266 what I am doing is invoking jsiLoop() once and, when it returns, scheduling a callback the next time the ESP8266 is idle and giving control back to the ESP8266 runtime. When the callback informs me that ESP8266 is idle, I invoke jsiLoop() again and repeat this process forever.

The philosophy here is that the ESP8266 is a single threaded processor and hence can't do things in parallel. Since WiFi and TCP/IP are implemented in software and radio in the ESP8266, we can't take too long outside of the ESP8266 world. So far ... all has been working fine. Espressif claim that we should give control back to the ESP8266 every 50 msecs or less.

Now imagine that we did NOT return control to ESP8266 in a timely manner. Imagine that jsiLoop() took seconds to return. What that would mean is that the WiFi and TCP/IP layers would not be fed. So, for example, if we setup a socket listener in JS such that an external client tried to connect ... if we didn't feed ESP8266 soon enough, an external client might try and form a connection and its own TCP layer wouldn't get any IP level responses in time and the connection would appear to fail.

We are starting to see indications that we might be not feeding the ESP8266 fast enough. It is manifesting during timing critical areas such as WiFi access point connections. If we ask the ESP8266 to connect to an AP as a station while also having setInterval() timer at work, we appear to get WiFi level timeouts.

It is early days in debugging this. If we do NOTHING timer related while forming the WiFi AP connection we are good ... so that might be a work around ...

Now ... the meat of the question. Is there a description of how the main loop of Espruino works? Does it process a statement, a function or something else in a unit of jsiLoop()?

Neil

I believe that not returning control soon enough is why nodemcu panics.

If you see messages such as "wdt reset" then that is very likely. "wdt" is "Watch Dog Timer" which attempts to reboot the ESP8266 if it thinks it has stalled (such as not giving control back to ESP8266 often enough).

Is the interpreter returning control after each line of JS, or only when there's no "next line of js" to process?

Yes, I noticed this. A watchdog timer seems to kick in after around 1 sec if you're executing a particularly heavy bit of JS.

jsiLoop handles all available events and then returns - potentially it could be rewritten to execute just one event at a time and return, which would help a small amount.

However Espruino must execute entire blocks of code at once. It's just the way it's written - it uses the main execution stack for parsing and to store program state, so it can't magically jump out of execution and resume where it left off.

I don't see there's any way around it without implementing separate stacks and context switching (but there doesn't seem to be enough RAM for that). I guess esp8266 itself might provide some context switching functions (as it does seem to have an OS under the hood), but you're the ESP8266 guru :)

If there's no context switching, it'd be easiest to have Espruino as the main loop - you could hack a function into the parser such that it called into the ESP8266 OS every few milliseconds - but having it completely stop execution and return every few ms isn't an option.

It's actually pretty surprising that ESP8266 works that way - things like the Nordic chips handle all the radio-specific stuff via interrupts, and leave user code running unprivileged (which is much more sensible from a developer's point of view).

Just to add - of course anything is possible and the parser could be drastically rewritten, but it'd be a lot of work, and the act of resuming after having 'yielded' would be very slow.

In the ESP8266 community there are a few open source projects that stand higher than all the others. One of them is "esp8266/Arduino". See:

https://github.com/esp8266/Arduino

This project has enabled the Arduino IDE to be used to build ESP8266 apps and has also implemented the core of the Arduino libraries to run on the ESP8266. The result being that folks who come from the Arduino side of the house for C programming can feel right at home on the ESP8266.

The chap that runs that project is a genius with the handle @igrr.

One of the things he achieved is a "yield" mechanism. This allows code that is being executed as a result of an ESP8266 callback to "yield"control back to ESP8266 and when ESP8266 has finished its housekeeping (presumably very quickly) control returns back to where we yielded from. It is possible that THIS may be exactly what we need.

The problem is that the logic of how this framework works is not clear to me and I have asked the ESP8266/Arduino team to assist. And before anyone says "Why not read the code?" ... this one is written in Xtensa assembly language :-)

I have also asked @igrr for permission to include this fragment in the Espruino/ESP8266 port and he has said yes ... but I'd like @Gordon and @igrr to both be happy with that notion so I'd like to put them in touch with each other ... however ... that shouldn't stop us from investigating that part to see if the technologies are even compatible with each other.

"yield" mechanism ... It is possible that THIS may be exactly what we need.

Yes, that looks spot on. You'd need to stick some hooks into the Espruino parser that yielded each time around a loop or something, but after that it'd all work fine.

Ok, just checked up on this... It's here:

• header
  
• assembler
  
  

Not that I can read xtensa assembler but it's actually a pretty small bit of code so it's easy enough to get the idea...

It looks like you run your code in its own separate stack and then save all the registers when you yield. What are the specs of the ESP8266? Maybe you could reduce the size of the stack that the ESP8266 has, in order to increase the Espruino stack?

If there's enough RAM to do that then you're laughing. It might mean a few less JsVars available in Espruino but it's probably worth it.

@Gordon These source files are great ... but they aren't just "drop in and it works" ... I understand that there are limitations, setups and considerations that we can't ascertain from just a look at the source code. I've asked the Arduino team if they have any docs or are willing to work with me to write up a "guide" on how this technique and code can be applied outside of the ESP8266/Arduino project. I'm not looking to be spoon fed ... but rather to actually talk to someone who understands how the "yield" stuff works to see if there are any lurking reasons that we couldn't use this technique. I'd hate to spend the next 2 days integrating an ESP8266 yield technology only to find that it wasn't going to work (in conjunction with Espruino) and then be told at the end "Oh yeah, we knew that was a limitation if you'd asked, we could have told you.".

Yeah, from the look of it, it might 'just work', but best to check. RAM usage is the only real consideration I can think of.

From an implementation point of view, does ESP8266 have a 'SYSTICK' counter (just a 32 bit counter that ticks up every clock cycle and overflows). Then Espruino could have some relatively speedy check of that such that it didn't yield so often it was painfully slow.