Events

I'm in my first week on this project to rewrite an existing C++ 8266 application. Admittedly very much the novice using javaScript/node.js. I need to insure that function A runs before Function B in several different places in the code. I thought I would simply things a bit by using a node.js event from the EventEmitter class and trigger Function B from the event polling. Doesn't look like that class is supported. Any guidance on how to proceed would be greatly appreciated.

You want to look into using promises if you're working on Espruino and using Javascript but I don't know if they're in ESP8266 as it's fairly constrained. If not you might be able to include the class with a custom build.

Thanks, the PROMISE is available on the 8266 and does work mostly. I would like to create the promise here:

wifi.on('connected',(details) => {
var clearQueuePromise = new Promise((resolve,reject) => { --http stuff-- }
});

However, that blows up with nothing in the way of useful information. If I just create the promise outside of the wifi.on, and let it fail until the WiFi connection completes it does what I want although a bit messy.

var clearQueuePromise = new Promise((resolve,reject) => {
  idClearQueue = setInterval(() => {
       idClearQueue = setInterval(() => {
           --do http GET until successful---
          clearInterval(idClearQueue);
         resolve("OK");
       },2000);
});

And then

wifi.on('connected', (details) => {
clearQueuePromise.then((resolve,reject) => {
    print('clear queue finished');
  });
 // This initiates the main loop 
    idMainLoop = setInterval(mainLoop, 2000);

Critical comments are welcome :-)

...critical? ...mainLoop smells Arduino loopy... It may be that there is no other way to easily convert this from a main-polling-loop over a bunch of flags that are set by interrupts of various participating sensors/inputs/... or a-like.

Each of these event producers - when written for an event driven system - allow to attach event handlers... as you see in practically all Espruino modules that provide connectivity to event producers. The reason is that the JavaScript engine - event queue server - is not allowed to block or take computing resources away to update a counter / time-comparison on the JavaScript level (because it would literally suck / hog, and more so with every additional such 'counter / timer'). If it blocks with the typical delay(...), it chokes / starves the hardware / low level service to death - and especially on a ESP8266, this is just not gona work.

I'm sure you had to move over also some of these in C written components that set global flags that are usually checked and handled in mainLoop constructs. Instead of setting such a global flag, allow to provide a callback.

You never can get completely away from polling... but if you have to do so, allow different intervals suitable to each of the components and do not enforce a single one thru the mainLoop.

In your high-level problem description you say A has to complete before B... Using promise is an elegant way to overcome call-back hell. Since you are talking of only two things with sequence dependency, a simple callback will do. I assume A is triggered by some event... and A has asynchronous behavior... (like the http operations). What you can do then is to take the (completion) event(s) of A and trigger B. With something like that you get completely rid of the mainLoop thing...

To be more detailed in the comments, I need a bit more details on the logic flow structure.

For your reading, take a look at sequencing implementations without / before availability of promises (using state machine concepts):

• Tasker - Task synchronizer - kind of a 'purposed' Promise
  
• sequencing - configure, require(), initializing w/ onInit() and save() ...and bind()
  
  

First example explores the concepts, second one is an actual application where http is involved.

Might be worth elaborating on what you are trying to achieve in your code. And if you can provide the full sketch it's more likely people can advise on your approach.

But, if it works, it is often good enough - at least for me ;)

Copying your code together this would be the code not working, right?

wifi.on('connected',(details) => {
    var clearQueuePromise = new Promise((resolve,reject) => {
      idClearQueue = setInterval(() => {
           idClearQueue = setInterval(() => {
               --do http GET until successful---
              clearInterval(idClearQueue);
             resolve("OK");
           },2000);
    });
    clearQueuePromise.then((resolve,reject) => { print('clear queue finished'); });
});

And having the exact same promise definition "var clearQueuePromise =" outside of the "wifi.on('connected')" makes it work?

Sure we can explain the application. We are a now, small charity that provides at no cost custom-made voice activated bed controllers for people with limited mobility. Essentially the 8266 becomes a remote device. You can check out the charity at http://limitedmobility.solutions .

From either an Alexa SmartHome or Google SmartHome device, our clients can voice activate their beds, sometimes doors, etc. If we can buy it off the shelf we do and our goal is to encourage bed manufacturers to provide a solution. However we estimate that there are at least 100,000 older beds that are not voice activated in use by people with severe limitations to mobility. Each bed model seems to require a custom solution. The 8266 is used to activate servos, sometimes RF, whatever it takes to operate the bed. My partner does the hardware.

We designed this application as a prototype two years ago using the most expedient approach. It has worked so well that we have not made any basic changes to design but are always open to changes. I have always been concerned about the polling aspect which I will address later.

So basically Alexa/Google sends a turnOn request to our backend. We pretend to be be a switch. The request goes into a LIFO queue via a https GET. The 8266 bed controller polls using a https GET. The remote web site returns the latest request and deletes any older ones. The 8266 does what it needs to do to operate the bed.

This particular POST was in response to the need to clean out the LIFO queue when the 8266 starts up. If the controller goes offline for some reason, a client may issue multiple voice requests before the unit comes back online. We do not what any unexpected bed movements because of residual requests in the queue.

Needless to say, the setup of a bed controller has to be super easy. We rely on caregivers to do the installation. Almost always these are remote installations with us providing phone support, videos and hard copy documentation.

One of our early decisions was to not require any router firewall changes. https outbound always works.

The reason we are experimenting with a javaScript solution is because Google is announcing a Google Mini solution where the local "device" can talk directly to the Mini by-passing the need for a persistent internet connection. The only time the current solution fails is when we lose internet connectivity to the web servers. The Google solution is based on a provided javaScript library.

Polling could eventually have some performance issues for us. But at 20 clients currently, it is really not an issue. I issue a http keep-alive to the servers which greatly reduces any certificate processing.

I am looking at some IoT technologies but so far everything that is "push" would require firewall changes. It may come to that eventually.

I didn't explore this too much but yes the above code is what I had originally tried. When running it I got some error (can't remember exactly) like "lost prompt" and then nothing.

Having it outside fails to connect to the web site until the WiFi is connected but otherwise works.

Went to check out your charity work @nhlives and on which continent, but found link is bad. Missing a 't' in protocol ref #7 link

Thanks for the heads up on the typo. Currently we are doing just the US. If someone wanted to sponsor a non-US charity, I'm sure we would be happy to help get that started. We have an older open source version of the software and plan on making the new version available publicly on GitHub. It just takes some volunteers and a modest amount of money to have a real huge impact on people's lives.

Looks like a great project.

A couple of gotchas to watch for, some might be problematic.

ESP8266 build of Espruino doesn't support TLS so you won't be able to call HTTPS endpoints. If you are having success it "might" be that ESP8266 is using the HTTP schema and the endpoint you are using is available on both. One to check.

Both ESP32 and Espruino Wifi have the resources needed to support TLS.

Another potential problem is available space. ESP8266 is quite constrained you can run out of both flash and sometimes RAM when handling your http response data if the pages are large.

Websocket & MQTT are other communication options supported in Espruino (above caveats re TLS still apply) and which would seem to fit your use case. The former works with Firewalls because it's using the same ports as HTTP/S and the latter usually works unless outbound connections are blocked, which isn't usually the default.

Alexa/Google sends a turnOn request to our backend. We pretend to be be a switch. The request goes into a LIFO queue via a https GET. The 8266 bed controller polls using a https GET. The remote web site returns the latest request and deletes any older ones. The 8266 does what it needs to do to operate the bed.

The terms our backend, We pretend, FIFO, ...this is all a site that is available for all the registered HW control devices to pull from? Neat solution.

To overcome the issues of polling can be overcome by NIOB - non io blocking http(s) application server and long lasting http(s) GET requests.

Quite a while back when it was still just HTTP(S) available, push was not really possible... but there was a startup that built a instant messaging hub and a browser client - html and JavaScript - with the setup and the technology that could serve you very well... the startup go bought by google... of course... about and the domain disappeared... it was noob.com or something like that. Cannot find things on the Web anymore... The key function was that you could have this one instant messaging client and on the hub you would subscribe to all this plethora of instant messaging sites... like a federated instant messaging.

How did it go:

• The client / browser / your remote controller would send a GET to the hub server.
  
• If an entry is waiting on the hub server, it would return immediately with the response.
  
• If no entry is waiting, the request would eventually timeout, but triggered just the next one.
  
  

This technology is against the initial idea of http protocol: request comes, gets served, and is stateless, resources are freed to serve next request / requestor. These long running request tied the resources - sessions - quite quickly: too many pending requests. That is when NIOB implemented: a pending request can be suspended in the the server, the thread is freed and handle other request, either incoming, or, when response is ready, pickup a suspended request and complete it. See http://tutorials.jenkov.com/java-nio/non-blocking-server.html - which describes this... kind-a.

Most application servers have this NIO technology implemented. At that time - quite a while ago - there was even a protocol developed and implemented in many language bindings. Using just two outbound http(s) connections provided the overall application with bidirectional push functionality.

...'found'/recalled the technologies: CometD, Bayeaux Protocol, (AJAX) Long Polling, Pub-Sub. Found also the CometD site and the salesforce developer site talking about.

In a stealth org I had this running for an app I can understandably not talk about in detail... but what I can say is that two html / js apps in two browsers on different machines could talk to each other / control each other using the JavaScript language binding of the Bayeux protocol via a Java Web app using CometD implementation approach on jetty on a 'private server at a home' at a third location - securely and authenticated - after signing into the app server. To our surprise, the latency was in the range of 250...400 ms only... out of the box, no tuning applied. We also had a 'local hub' in mind that would either forward to the 'global hub' if the 'other party' is not local to the same 'local hub'. Today I would go for a nodeJS server and hub implementation in JavaScript rather than a jetty and java setup (assuming nodeJS Web (App) server offers NIO - looks like it is possible reading this (old but still younger) article on DZone from 11'.

Today, as @Ollie mentions, Web sockets & MQTT is the way to go... but it was not (yet reliably) available then and may still today require more sophisticated setups.

To make things simple, I would quench older - not picked up messages - on the server level rather than in the client level... messages arrived on the hub and not picked up within threshold just 'evaporate' connection is not present (goes absent for too long, no long poll request is waiting or showing up).

After all, quite an interesting project...

If the controller software is simple enough and https communication software still fits on a ESP8266 (see #232 How to secure our devices using SSL (ESP8266, ESP32, Tutorial) - Andreas Spiess, youtube
), it may still work. Espruino is then sadly out of the pic... To start prototyping: yes, Espruino-Wifi is a good platform. Could you also consider using BLE for communication between HW controlling devices and local hub? - then Espruino Puck/Pixl/MDBT42Q breakout board/module are viable options too with a local hub. And for the local hub: some RaspiPi... w/ Espruino or nodeJS. With a powerful enough platform the logic - FIFO/LIFO - can reside on it and the global hub works only as a "comm switch".

I can't see why you would need a LIFO. At startup I'd simply discard all messages that queued up while offline, to keep the device simple.

Regarding security:

• When going with the Google Mini local API http is ok (ESP8266 doesn't support https).
  
• When connecting to the internet you should verify the identity of the server using https (ESP32 supports https).
  
• In both cases keep in mind that Espruino has a local telnet and OTA server running in the default firmware, that is nice for development but also a potential security risk during operations in a customer WLAN.
  
  

Instead of http/https polling you could use websockets, which are much easier to implement in nodejs/esprunion compared to C++. It's an outgoing http (or https) connection as you're using right now, no firewall change required. The only difference is that you "upgrade" the connection once, and then you can send "messages" instead of http requests. Having a tested solution with polling I wouldn't change it, unless you have to scale up.

EDIT of post #12: jetty instead of tomcat... https://github.com/eclipse/jetty.project

Thanks all. I have pretty much written off the 8266 for this effort namely Google Local home. Maybe the esp32 with WiFi but I'm thinking a Pi is in my future. I'll probably convert the existing 8266 Arduino IDE code to use MQTT just to be able to use the AWS or Azure IoT hubs. It looks like the webhook triggers there will mesh nicely with the existing webhook backend. BTW, the HTTPS issue was the final straw. Unfortunately my test back end has both HTTP and HTTPS enabled so I missed that.

"...The terms our backend, We pretend, FIFO, ...this is all a site that is available for all the registered HW control devices to pull from? Neat solution..."

The site is where all of the devices pickup requests. We have incorporated OTA, we can remotely reboot the device and get various status information. Given the way the 8266 handles certificates (or doesn't) we can push a new fingerprint to the device if necessary. There are also two middle-ware pieces, one each for Google and Alexa. Alexa requires an AWS lambda program for Smart Home. Google is any web endpoint will do. Everything is C#. Web sites are .Net Core. Everything is secured with JWT. Both of the middleware components handle "turnOn", "discovery/SYNC" and "QUERY" requests from Alexa/Google in conjunction with the web back end. We wrote our own lightweight oAuth server.

"...To make things simple, I would quench older - not picked up messages - on the server level rather than in the client level... messages arrived on the hub and not picked up within threshold just 'evaporate' connection is not present (goes absent for too long, no long poll request is waiting or showing up)...." Not a bad idea and easily enough implemented. I may incorporate that in the next build. I'll try and remember to give you attribution.

In all this is a very rewarding project. You don't often have opportunities like this that really have such positive impact on these people's quality of life.

You could get a lot of mileage from the Espruino Wifi maybe speak with @Gordon (Espruino creator and Official board supplier) before you write it off - but as you say Rasp Pi, particularly Zero W, is the other option.

One question if I may, how do you get on with approval/certification. The benefits of your solution are very clear - I checked your website - but the insurers always seem to be about risk assessment and ass-covering. If you would share how you approach this it could help others with your skills, put them to equally good use.

I can easily second @Ollie in both: still to consider Espruino for the 'interface' between a local brain device and the things to control, last but not least because it is frugal power consumption, especially when you go BLE for communication - battery driven / cordless - and for the 'decent' local brain you go for a Rasp Pi that can be power connected all the time.

I will not be making any decisions soon on a platform. I plan on getting an esp32 to test with. In any case, I putting this off for at least six weeks as I'm going on an extended vacation :-)

allObjects
...least because it is frugal power consumption, especially when you
go BLE for communication - battery driven / cordless...

Power has never been a problem as they are always AC connected.

Bill, the founder, and primary source of funding is not too keen on changing platforms. He does the hardware builds and is pretty comfortable with the ESP8266. We will see how much he wants to do the Google Local Home.

As to the liability question, corporate insurance for sure. Bill does a pretty extensive interview with perspective clients to be sure that the would not have any problems with whatever position the controller might put them in. We have declined, unfortunately, some that might not be able to handle having the bed in any maximum or minimum position. There are waivers to sign, etc.

We do not have any way to determine starting or stopping positions on these beds as they do not have any sensors that we can interrogate. If we were shipping an entire bed, we could probably do that. But that is not the current model or would I expect it to change. The normal standard is to limit bed movement to 15 seconds for any command. That normally means the clients has to issue several commands to make the bed travel to a full up position, for example. We only support "turn On". There is no "turn Off" command.

One problem I had with node.js on the ESP8266 was the limitations to what was supported. I just needed a couple more unsupported events:

response.on('finished'); 

@nhlives, nice concept of limit the time of operation to avoid run-aways. Do you have some imagery about how the ESP8266 is connected / mounted on some of the control device of the bed and other devices? ...You may know that Espruino runs also on sonoff and a-like that allow 110..220 main power control?

Since power seems not to be an issue, there is no issue with wires 'all over the place'... and also no issue with power for driving servos, solenoids, etc. That's good to know. Understandably, because beds with these options are mains connected anyway, and if at home, they are not moved around either...

Regarding the event you were looking for: 'finished' of what? - http comm? operation? etc.

I open a web server to get the SSID and password. Once the WiFi successfuly connects, I send a "connected" page and then i want to close the access point and stop listening on the web server. What is happening is the AP closes before the page is received or the event doesn't fire at all. I've tried "response.on('finish') " and as below "response.on('close') .

 console.log(SSID + '  ' + password);
                  SSID = SSID.trim();
                  password = password.trim();
                  console.log(SSID + '  ' + password);
                  options = {password: password };
                  wifi.connect(SSID,options , err => {
                    if (err !== null) {
                      msg = err;
                      index(response);  
                      throw err;
                    }
                   
                    wifi.getIP((err, info) => {
                      if (err !== null) {
                        throw err;
                      }
                      print("ESP8266 IP: "+info.ip); 
                      connected( "response.on('close') );
                      response.on('close', () => {
                        console.log('Stopping web server and AP');
                        server.close();
                        wifi.stopAP();
                    });

Right-click seems to work. BTW, the manual control is for a caregiver or as a backup if the internet is not available.

BTW, I've order a ESP32 development kit.

Right-Click on image works, see if the filename AND extension are present in the link

Regarding the code:

I do not understand lines 18..23... it also looks syntactically weird to me... may be something got lost in copy-paste... I'm also curious about the big indentation... must be some context.

Regarding the text:

open a Web server to get SSID and password

Does that mean that something is 'polling/trying to connect to that Web server to come up and then requests a page to post the SSID and password to connect to a different network? ...because servers do not request things, they serve requests, they also do not send pages by themselves, the respond to requests with a page.

stop listening on a Web server

Do you mean stop listening to requests?

Milling over my own question I guess I see what event you are locking for:

The even of having sent the response in its entirety and only then you want to shut down the server. You can setup the response with streaming... then you get an event when all data is sent only the close - response.end()with no data anymore is needed... and with a short timeout you can then shut down what ever you need (in correct sequence).

Are you thinking of the remote device operating in two modes:

First mode - for setup / boot: it is an access point with 'static/fix' ssid, pw and web server that allows to be connect to and can receive a post with and other ssid and password for Second mode.
Second mode - for regula operation: Connect as station to the network with ssid and password received in the first mode.

Another comment I want to make is: when you are already connected to a wifi network w/ a particular SSID and password - I do not know if you can can create an access point, because an access point means defining a new network with its own, different SSID and password, and I do not know if you can operate on two networks simultaneously. But:

If you have a Wifi connection, yes you can open a server and listen to requests and then shut it down after serving the pages / posts you needed to serve (to either provide data or get data, and use the gotten data to continue on the connection with something else (such as create a http request - to get a 'command' for execution, polling).

Hopefully I can help in some of the questions or incorrect assumptions:

1) The ESP8266 does SSL just fine like the ESP32 does. Proof: http://forum.espruino.com/conversations/330049/#comment14873565

2) The amazing Gordo has a write up on doing MQTT if this is what you're looking for: https://www.espruino.com/MQTT

3) If you want instant access and PUSH something to a "web client"(your device is a web client to any server it connects to) you can use websocket. They're fast and easy: https://www.espruino.com/ws

4) If you want an example of code that will "wait until it connects to wifi successfully and then hit a website" here's my code that does exactly that. http://forum.espruino.com/conversations/330049/#comment14873565

Just a heads up: in javascript we don't wait for things. (like pausing until the wifi connects or until a website is downloaded) You CAN use promises but you'll see that most of the code uses just a callback (that's just fancy talk for a function as the last parameterthat runs ONLY once something in the call has finished.

Good luck. this sounds like a great project.

The ESP8266 does SSL just fine like the ESP32 does.

It doesn't. The request is going over http / port 80. I'm pretty certain.

The ESP8266 http implementation would be better not sending the request at all IMO, because it's easy to think you've got a secured connection and if the server listens on 443 and 80 then you'd be none the wiser.

If you take it down a level into http.request(), you can see what's what in terms of available protocols.

let wifi = require("Wifi");
let http = require("http");

if (wifi.getIP().ip != "0.0.0.0") {
     let options = {
       host: 'http://www.google.co.uk', // note the editor has added http:// 
       port: 443,
       path: '/',
       method: 'GET',
       protocol: 'https:'
     };
  
  let req = http.request(options, function(res) {
      res.on('data', function(data) {
        console.log(data);
      });
    
      res.on('close', function(data) {
        console.log("Connection closed");
      });
  });

  req.on('error', function(err) {
      console.log(err);
  });
  
  req.end();
}

Note the port and the protocol/schema:

 port: 443,
...
 protocol: 'https:'

Responds with:

{ "code": -15,
  "message": "no response"
 }

Change the port to 80 and you get your response, but the protocol is still https:.

When using http.get all of this is obfuscated so it is easy to think you've got an encrypted connection.

Fact of the matter is that ESP8266 is unfortunately too limited in terms of flash/ram to handle the certificate management required for HTTPS