Porting to FreeRtos and ESP32

Some days ago I got an ESP32 Dev Board. Based on Neil Kolbans work it was easy to get a first version of Espruino running. It lacks a lot of things, since Espressif is still working on SPI/I2C/BLE, ...
Anyway we have a new world for Espruino and this world supports tasks, queues and other nice functions.
First version of queue handling is already running

• a seperate task polls data from somewhere and writes it to a queue
  
• in Espruino a function polls this queue for new data
  Task handling is not that easy, there are some restrictions which hopefully will disappear with next versions. May be that time, we will be able to have 2 Espruino tasks running, one on each CPU talking to each other using queues.
  
  

Target is to have JS-commands like:

• queue.list()
  
• var q = new queue()
  
• q.assign(queue_name)
  
• q.newDataAvailable()
  
• q.readData()
  
• q.writeData(.....)
  
• ......
  
• task.list()
  
• var t = new task()
  
• t.assign(task_name)
  
• t.suspend()
  
• t.resume()
  
• ........
  We would need a list of queues and tasks like {name,handle} for that. This could be an internal jsi-type function which would be called from other tasks. Or we could add function calls in jshardware.c returning pointer to global arrays.
  This would be the first time, Espruino meets something like RTOS, therefore I would like to get some other thoughts before spending too much time walking into wrong direction.
  

Shame about the SPI/I2c - but if you're up to date with the current Espruino branch, you should be able to handle both of those in software pretty well?

Do the two cores share an address space, and is there any coherency? I guess that could change how you approach it pretty drastically...

Personally, I'd try and getting Espruino running on the non-Wifi core with it as much real-time control as possible (using interrupts for the utility timer and serial, and making sure Wifi doesn't use any CPU time) - then handling whatever tasks you can (eg HTTP server) on the Wifi core.

There's jsiQueueEvent and similar functions that could form quite a nice place to split between the two cores. Having an easy method to handle callbacks from the second CPU core could mean that it'd be much easier to hand tasks to it from JS.

For example things like FFT (I'm struggling to think of other intensive things :) ) could work on the second core and could then call back when complete.

Going for two Espruino execution threads - one on each core - is for sure a good idea... but there is caution to be applied: JavaScript is by definition a single threaded execution. This makes it so easy to use... This fact is already compromised in some browsers that have a separate system io thread that is for entry fields not (anymore) synchronized with the events handling of attached JavaScript handling and causes quite nice challenges for implementing a solid working type ahead... To use one core for all (most) of the hardware event handling and native implemented code and leave the other core (as much as possible) 'free' for or busy with JavaScript execution is already a great progress and does not create challenges - last but not least in memory handling etc... otherwise we can then go right away to a Java VM like implementation... for which we for sure do not have the memory resources...

JavaScript in the browser handles multiple threads of JavaScript via Web-Workers, perhaps there could be an implementation like that?

Does it even make sense to have js running in both threads? Like, I'd say it's fine if everything runs in one thread, while the other one just handles network stuff that happens under the hood? I thought the main reason we were so excited about the second thread was so that we didn't have the network - stuff watchdog biting our ass if we have to do something slow

Status: SPI support is now available and I2C should be right behind it as the puzzles are now resolved. Note that the build instructions changed a tad to get the libraries correct for SPI and I2C. Espressif has released the news that their formal SDK at the 1.0 release will be available at the start of December at which time we will have all the proper libraries that we need to complete the build out. Until then we are using libraries that "work" but which aren't final.

My 2 cents opinion on multi-threading in JavaScript is ... well ... not to do it. The dual core that is now the ESP32 is primarily one core for the app and one core for networking (Wifi and/or Bluetooth). This allows us to dedicate a core to the app without ANY worry about starving the networking. This dramatically simplified Espruino builds because we can now use FreeRTOS and berkley sockets. As for trying to leverage both cores of the ESP32 for Espruino ... I personally don't see a need. One core for Espruino (the app) and one core for ESP32 networking feels right. That said, I am always open to listening to discussions and if anyone can put forward a cogent reason for using the 2nd core for Espruino work ... I'm all ears.

For me ESP32 is like a (power hungry and power devouring EspruinoWifi on stereoids... The operation of the extra HW resources - 2 cores, higher clock rate/speed, more memory - and the extra SW resources - more versatile and more amout of code - come just not for free...

The choice of, for example, EspruinoWifi vs ESP32 boils down to all aspects of the application requirements. We know that the ESP8266 network stack and the serial communication and its ramifications pose restrictions and burdens in various ways, such as more work on Espruino side, they both ESP8266 and Espruino have at this time outgrown their growing pain in quality (and function and tooling), and that is something that ESP32 still has to go through... and more so since some of ESP32's (final - initial version 1.000000) pieces are not even 'born' yet ;).

From all the pieces that go into the ESP32 is mostlikele the Espruino piece and for sure its Javascript part the most robust and advanced one, similar to RTOS. With a successful integration of Espruino Javascript and RTOS, and RTOS and ESP32 hardware and grown up ESP networking stack, ESP32 as a whole will be a powerful and quite versatile IoT field player: the 'Onion' in JavaScript... the 'perfect and easy' to program 'EspruinoIoTHub' ... Espruino io t(h)ub... EspruinoHub ;)

Reading the comments, there is a discussion starting "do we need 2 Espruino tasks" From my point of view this would be one option only to use multiple tasks. Mention the 2-core Espruino option in was accidently begin of a wrong question.
Writing this, I'm absolutely sure, if we can do 2 tasks running Espruino, there will be some guys using that, if we like it or not.

Looking back to history, there was a time where so called mini-computers (mini not micro) have been as powerful as ESP32 is today. Several OS have been on the market and all of them supported multitasking. And there was a good reason for that.

Today we have this chip supporting multitasking, lets discuss how Espruino could support multitasking.
One big question is sharing data between tasks, queues and semaphores are a way to do this. Another question is task handling, and some more will come up.
Thats what I wanted to discuss, not "do we need 2 Espruino tasks". As I can see, my first post was misleading, sorry for that.

As @Ducky points out, JavaScript has already some kind of multi-tasking with the Web Worker implementation. Web-Workers could be implemented and bound into Espruino binary on build for ESP32.

In Espruino there are two main queues already that are relatively well tested (used for everything, and have been there since the start) - there's ioBuffer - for input events, and txBuffer - for output events.

Potentially some tasks could be handled on the second core using those if needed.

Not sure what though - I'm struggling to think of stuff that's easy to offload.