I've made some updates to the webSockets lib ws.js

Hi,
I've really wanted to use webSockets with the Espruino & esp8266 as an http/webSockets server
but I was having some issues, namely:

1. the webSocket handshake was not completing until the server did its first send somewhere later, leaving the client waiting and unable to send data before the server did
   
2. the client was closing the socket after 1 minute of inactivity because it was expecting the server to ping
   
3. the server didn't know or gracefully close when the the client closed the webSocket
   
   

So I learnt more than I thought I would about the webSocket protocol and made the attached, simple changes that address these requirements (renaming the module to wsX.js just to avoid conflicts). A simple diff between the attached file and the hosted module will show you the few changes and I've added comments to explain, but feel free to remove these.

I'm now finding webSockets very reliable and doing what I need. I hope the changes I'm offering make it into the hosted module.

1 Attachment

• wsX.js

Thank you!

Thanks! Were you using the latest (Git) builds of Espruino? The current (1v84) version had a bug where it didn't sent the header immediately (why you needed the write("")).

Still, if that fixed it for everyone without needing a tweaked firmware it's probably worth putting in the main websockets code :)

Ok, I just pulled your changes into GitHub, and tweaked a few things around. It works great!

I'll try and get the changes actually on the website today.

Hello Gordon,
thanks for your wonderful work, I love this javascript based hardware (use it for home automation). Unlikely with one problem concerning websocket.

My espruino is monitoring some temperatures which I would like to stream via websocket to my dataserver. I tried some firmwares (also the latest cutting edge espruino_1v84.155_espruino_1r3_wiznet.bin) but have issues with it. Generally this latest version feels better then the older ones, but it freezes my espruino console on the left side of the Chrome IDE (it is not possible to type something into it, I use ist for reset() sometimes).

My test script (as attached) is basically websocket trying to use setInterval every 2 seconds. If you run it with commented websocket part (line 17 - 37) the timer works and the script gets read in completely. If you uncomment lint 17-37 the setInterval timer never gets to run and the last console.log on line 40 sometimes shows up sometimes not (on startup).
The espruino "hangs" as I wrote above and needs to be plugged out and in again every time I want to flash a new script. But the websocket connection is correctly set and also the first message (line 27 ws.send(data); is showing up on the dataserver)

I really tried a lot of things and for me it looks like there is a problem with the event loop, maybe this ws.on('open',... ) never finishes and blocks the event loop (would explain the odd behaviour of the timer/console.logs).

It would be great if you have an idea to put me in the right direction to solve this issue (I hope I haven't missed something and the problem is all mine).

Many thanks and a happy / successful 2016!

Best regards, Manuel

var eth = require("WIZnet").connect();
eth.setIP({ ip : "192.168.0.10" });

var data = JSON.stringify([{
  topic: 'espruino',
  messages: ["199-hello\nldskjfds"]
}]);

var t = setInterval(function() {
  console.log("Write  should be done every 2 seconds");
  //ws.send(data);
}, 2000);

var WebSocket = require("ws");

/*
var ws = new WebSocket("192.168.0.6",{
  port: 8080,
  protocolVersion: 13,
  origin: 'Espruino',
  keepAlive: 30  // Ping Interval in seconds.
});

ws.on('open', function() {
  console.log("Connected to server");
  //Send message to server
  ws.send(data);
  console.log("Sent first time");
});

ws.on('message', function(msg) {
  console.log("MSG: " + msg);
});

ws.on('close', function() {
  console.log("Connection closed");
});
*/

console.log("Script read in completely");

Where is the code? I see no attachment

As @DrAzzy says, I don't see any code - but if the console on the left-hand side isn't responding it's usually a problem with the upload.

On Espruino, code is executed as it is uploaded, and if you write code that takes a long time to execute then it can cause the PC to wait too long to send data, and in the end the PC will just not send it.

So what you need to do (in fact probably what you meant to do) is to make sure that your main code (initialising the network/socket connection?) happens in the onInit function, so that when saved, it will be called each time Espruino starts up.

When you do that you'll need to manually type onInit to start it running after upload, but that may fix your problems.

Otherwise you might be able to go into the Web IDE's settings and click 'Throttle Send' - it'll slow down transmission and might avoid your problems without you having to change your code.

File Upload seems to not work, I have included the short code into my question above.

I'll try to use an onInit function in a moment. Thanks so far I will post my results.

Ahh, yes - that's definitely your issue.

With WIZnet, when you create the Websocket, I think it connects immediately - which might cause a delay which would stop everything getting uploaded correctly.

Even using a setTimeout would work:

var eth = require("WIZnet").connect();
eth.setIP({ ip : "192.168.0.10" });
var data = JSON.stringify([{
  topic: 'espruino',
  messages: ["199-hello\nldskjfds"]
}]);
var t = setInterval(function() {
  console.log("Write  should be done every 2 seconds");
  //ws.send(data);
}, 2000);
var WebSocket = require("ws");
var ws;

setTimeout(function() {
  ws = new WebSocket("192.168.0.6",{
    port: 8080,
    protocolVersion: 13,
    origin: 'Espruino',
    keepAlive: 30  // Ping Interval in seconds.
  });
  ws.on('open', function() {
    console.log("Connected to server");
    //Send message to server
    ws.send(data);
    console.log("Sent first time");
  });
  ws.on('message', function(msg) {
    console.log("MSG: " + msg);
  });
  ws.on('close', function() {
    console.log("Connection closed");
  });
}, 1000);

Thank you Gordon for your helpful reply, it (your example) still is not working and "hangs" the espruino. I even put the whole thing into an onInit function without success:

function onInit() {
  var eth = require("WIZnet").connect();
    eth.setIP({ ip : "192.168.0.10" });
    var data = JSON.stringify([{
      topic: 'espruino',
      messages: ["199-hello\nldskjfds"]
    }]);
    var t = setInterval(function() {
      console.log("Write  should be done every 2 seconds");
      //ws.send(data);
    }, 2000);
    var WebSocket = require("ws");
    var ws;
    setTimeout(function() {
      ws = new WebSocket("192.168.0.6",{
        port: 8080,
        protocolVersion: 13,
        origin: 'Espruino',
        keepAlive: 30  // Ping Interval in seconds.
      });
      ws.on('open', function() {
        console.log("Connected to server");
        //Send message to server
        ws.send(data);
        console.log("Sent first time");
      });
      ws.on('message', function(msg) {
        console.log("MSG: " + msg);
      });
      ws.on('close', function() {
        console.log("Connection closed");
      });
    }, 10000); // I extended this to 1o secs to see what below setInterval function delivers
}

setInterval(function() { console.log("Halo"); }, 1000);

onInit();

The output is as follows:

Halo
Halo
Write should be done every 2 seconds
Halo
Halo
Write should be done every 2 seconds
Halo
Halo
Write should be done every 2 seconds
Halo
Halo
Write should be done every 2 seconds
Halo
Halo
Write should be done every 2 seconds
Connected to server
Sent first time

So as long as ws = new WebSocket("192.168.0.6",... is not executed times work well and the event loop runs, but after 10 Seconds ws is created the ws.on("open", ... is executed correctly but afterwards the espruino hangs and does not execute any more, as well as I have to plug usb out and in again to renew the script. To me it seems on ws.on("open",... the event loop gets blocked.

Another thing I encountered is on the general Info page (http://www.espruino.com/Notes) it says:

If you create a function called onInit, that will be executed too:

In my case I needed to run the function manually otherwise onInit did not get fired (see last line).

Thanks for your help!

Best regards Manuel

Is the problem just that you don't have a WebSocket server running on 192.168.0.6 port 8080, or that Espruino can't connect to it for whatever reason? I think the connection takes quite a while before it times out, which will unfortunately block Espruino.

If you create a function called onInit, that will be executed too

It's executed when loading from flash (if you save with save()) but not immediately after uploading code - which can be a bit confusing.

The reason is that since setInterval is remembered, if onInit was auto-run, and then you typed save() onInit would get called again at each power-on and you'd end up with two intervals!

I checked my dataserver running on 192.168.0.6 port 8080 and this nodejs part works as expected (it even gets the initial data sent on ws.on('open') perfectly, and also saves data from other sources too). Maybe I have time to check your code in the WebSocket library, like I mentioned above I think it dies there (or maybe it needs an acknowledge message from dataserver to finish).
I cannot confirm that there is a timeout which ends the blocking (waited 15mins or more).

Tanks for your clarification about onInit.

Hmm, that's strange then - sorry, I didn't pick up that it had Connected to server.

Does Ctrl-C help? It might let you break and see where execution has stopped?

I had problems getting WS to work with the latest nodejs ws version (1.1.0) as a server.

The trick for me was to add a "return" statement in the first if block of the "parseData" method (roughly line 117 of wsx.js) to prevent the other statements to execute and produce unexpected results in some cases. After I did this change the connection was not directly terminated any longer.

My ultimate goal is to get it running together with cloudflare (as they now support websocket connections in free plans).
I did some changes that significantly improved the behaviour:

Added the hostname to the handshake (otherwise reverseproxies can not handle the request)
Removed the Origin from the handshake (otherwise correct WS servers would decline the connection because of CORS problems, however it is optional for devices like espruino)
Added XOR encryption for the send method as this is mandatory for clients to do

Further I also try handle binary data, however this is experimental and has nothing to do with cloudflare.

Now cloudflare finally accepts the connection and even forwards my first message, however the connection is terminated right after that. When not using cloudflare (and connecting directly) everything works as it should. Does anyone have a clue what could be the problem?

1 Attachment

• ws2.js

Ok, I fixed it myself. The reason why the code was disconnecting all the time when using cloudflare was the following: The handshake was transmitted in two chunks (each chunk has a wrapped around it here):

<DATA>HTTP/1.1 101 Switching Protocols
Date: Sun, 08 May 2016 08:59:10 GMT
Connection: upgrade
Set-Cookie: __cfduid=xxx; expires=Mon, 08-May-17 08:59:10 GMT; path=/; domain=.urbiworx.de; HttpOnly
Upgrade: websocket
Sec-WebSocket-</DATA>
<DATA>Accept: HSmrc0sMlYUkAGmm5OPpG2HaGWk=
Server: cloudflare-nginx
CF-RAY: 29fbb6cda43e26d2-FRA
</DATA>

So what happend here was, that during the handshake one data packet was received with an "H" at the beginning (first chunk) that did not contain the "Accept" Header (that is in the second chunk) and so was interpreted as WS message. "H" in hexadecimal is "48" which is the worst that could happen as it means opcode "8" - terminate connection.

I now changed the code accordingly so that chunks do not get interpreted as message as long as the handshake is not finished.
It is not yet implemented optimal right now however it works much better than before.

1 Attachment

• ws2.js

Thanks! Is this based on the module at http://www.espruino.com/ws, or is it completely different now?

It's be great to pull these changes back in...

I'm happy to look at the integration, if you can spell out which parts have changed. When I have time I'll try to do a diff...

My JavaScript is based on the wsx.js from the first post here which should be widely identical to the current ws.js available from the espruino site so it is compatible to what is in place now. Just give me a few days as I would like to further improve the http handshake as it could still fail under certain circumstances and I would like to add support for larger messages (more than 127 chars).

I'm looking through your updates, and trying to follow:

old line 128:

   if (opcode == 0xA)
        return this.emit('pong');

new:

 if (opcode == 0xA) {
        this.emit('pong');
        return
    }

Aren't these equivalent?

and also here line 182sh:

 this.socket.write(socketHeader.join("\r\n"));

and new:

    for (var index = 0; index < socketHeader.length; index++) {
        this.socket.write(socketHeader[index] + "\r\n");
    }