HTTPS support on Pico!

I'm not sure many more people would have tried a release candidate than tried the GitHub builds. I just released 1v82 properly - if there's a big problem I can always shift out 1v83 quite quickly.

One way that might help is opensensors.io now support TLS on mqtt and have a free tire.

Partially ported AWS IoT library.
As there is currently no support for client-cert X.509 auth yet on Espruino, use with socat.

Created pull requests on github:
github.com/espruino/EspruinoDocs­/pull/193
github.com/espruino/EspruinoDocs­/pull/194

Managed to connect to AWS IoT by offloading tls to socat.

root@192.168.2.1>
socat tcp-listen:8883,fork,reuseaddr openssl-connect:A4B6O0T19QBXJ.iot.ap-northeast-1.amazonaws.com:8883,method=TLS1.2,cert="$HOME/Downloads/61a57a0f66-certificate.pem.crt",key="$HOME/Downloads/61a57a0f66-private.pem.key",cafile="$HOME/Downloads/VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem"

function initEthernet(cb) {
  SPI2.setup({ mosi:B15, miso:B14, sck:B13 });
  var eth = require("WIZnet").connect(SPI2, B10);
  eth.setIP();
  global.eth = eth;
  if('function' === typeof cb){cb();}
}

function initMQTT(cb) {
  var mqtt = require("MQTT").create("192.168.2.1", { port: 8883 });
  mqtt.on('connected', function() {
    mqtt.subscribe("$aws/things/+/shadow/#");
    mqtt.on('publish', function (pub) {
      console.log("topic: "+pub.topic);
      console.log("message: "+pub.message);
    });
    if('function' === typeof cb){cb();}
  });
  mqtt.connect();
  global.mqtt = mqtt;
}

function initAwsIotDevice(cb) {
  var awsIot = {};
  awsIot.device = require('aws-iot-device-sdk-device');
  var device = awsIot.device({ host: '192.168.2.1', port: 8883 });

  device.on('connect', function() {
    console.log('connect');
    device.subscribe('$aws/things/+/shadow/#');
    device.publish('$aws/things/espruino1/shadow/test', JSON.stringify({test_data:1}));
    if('function' === typeof cb){cb();}
  });

  device.on('message', function(topic, payload) {
    console.log('message', topic, payload.toString());
  });
}

function initAwsIotThingShadow(cb) {
  var awsIot = {};
  awsIot.thingShadow = require('aws-iot-device-sdk-thing');

  var thingShadows = awsIot.thingShadow({ host: '192.168.2.1', port: 8883 });
  var rgbLedLampState = {"state":{"desired":{"red":187,"green":114,"blue":222}}};

  thingShadows.on('connect', function() {
    thingShadows.register('espruino1');
  });
}

//E.on('init', initEthernet);
//E.on('init', initMQTT);
//save();
initEthernet(function() {
  console.log(global.eth.getIP());
  initMQTT(function() {
    console.log("AWS IoT Ready");
  });
});

root@192.168.2.1>
mosquitto_pub -h 192.168.2.1 -p 8883 -d -t '$aws/things/espruino1/shadow/test' -m "Hello World"

AWS IoT device library is working.
Will continue to port AWS IoT shadow things library from nodejs.

For those with Espruino Pico, this opens TLS connection from espruino to socat, and TLS v1.2 from socat to AWS IoT.

root@192.168.2.1>
socat openssl-listen:8883,fork,reuseaddr,cert="$HOME/Downloads/61a57a0f66-certificate.pem.crt",key="$HOME/Downloads/61a57a0f66-private.pem.key",verify=0 openssl-connect:A4B6O0T19QBXJ.iot.ap-northeast-1.amazonaws.com:8883,method=TLS1.2,cert="$HOME/Downloads/61a57a0f66-certificate.pem.crt",key="$HOME/Downloads/61a57a0f66-private.pem.key",cafile="$HOME/Downloads/VeriSign-Class 3-Public-Primary-Certification-Authority-G5.pem",verify=1

function initEthernet(cb) {
  SPI2.setup({ mosi:B15, miso:B14, sck:B13 });
  var eth = require("WIZnet").connect(SPI2, B10);
  eth.setIP();
  global.eth = eth;
  if('function' === typeof cb){cb();}
}

function initMQTT(cb) {
  var mqtt = require("MQTT").create();
  mqtt.on('connected', function() {
    mqtt.subscribe("$aws/things/+/shadow/#");
    mqtt.on('publish', function (pub) {
      console.log("topic: "+pub.topic);
      console.log("message: "+pub.message);
    });
    if('function' === typeof cb){cb();}
  });
  require("tls").connect("mqtts://192.168.2.1:8883", function(conn) {
    mqtt.connect(conn);
    global.mqtt = mqtt;
  });
}

//E.on('init', initEthernet);
//E.on('init', initMQTT);
//save();
initEthernet(function() {
  console.log(global.eth.getIP());
  initMQTT(function() {
    console.log("AWS IoT Ready");
  });
});

Great! So what exactly is missing from the Espruino Pico implementation? Is it just that there's no check on the certificate so it's not a good idea to connect to AWS, or it's actually that you can't connect to AWS at the moment.

I think it's:

there is currently no support for client-cert X.509 auth yet on
Espruino

i.e., the client (Espruino) needs to authenticate to AWS using a client certificate and it sounds like what you have doesn't support that.

Ahh, I'd have to look into that. mbedtls will definitely handle that - I just haven't exposed it.

Pull requests welcome :)

can't connect to AWS at the moment

This is because client TLS auth is required to connect to AWS. This needs to be enabled in the espruino tls library and also I am not sure where and how to store the client-cert and private key to be passed to mbed_tls.

Ok, I just filed an issue for this: https://github.com/espruino/Espruino/issues/736

I'm not sure when I'll get around to it, but I'll see what can be done. It doesn't look like it'll be too painful to implement.

Ok, just done. For instance the following now works:

var options = url.parse("https://localhost");
options.key = atob("MIIJKQ ... OZs08C");
options.cert = atob("MIIFi ... Uf93rN+");
options.ca = atob("MIIFgDCC ... GosQML4sc=");
require("http").get(options, ... );

If you have the certificates as .pem files, you need to load these files, take the
information between the lines beginning with ----, remove the newlines from it
so you have raw base64, and then feed it into atob as above.

This will be in http://www.espruino.com/binaries/git/commits/master/ - just give it an hour for the latest build to appear in there.

Just to add, while this works in theory, in practice there could be trouble. It doesn't look like there's enough RAM in the Pico to store the all the information needed for the certificates.

how about: ...perfect place to store all this heavy read only things: fake EEPROM

If it has to be simultaneously in RAM, then it may not work. So it is time to have certain operations to be enabled to read 'streaming' from (fake/extra) EEPROM/RAM. I don't know what exactly is needed, but that would be the only way where memory could come from.

'coincidence': the FlashEEPROM is fixed in 1v83.

Thanks - I just posted here

1v83 won't do that, but it should be possible with some changes. I filed an issue for it here: https://github.com/espruino/Espruino/issues/738

Thanks @heri16 & @Gordon!

On initial testing, I'm getting a 0x4310 mbedtls_ssl_handshake error. (This does work when using another client to test, so I know my keys and policy is valid.)

Here's the test code:

SPI3.setup({ mosi:B5, miso:B4, sck:B3 });
var eth = require("WIZnet").connect(SPI3, A8);
eth.setIP();
eth.getIP();

var options = url.parse("mqtt://A0123456789AA.iot.us-east-1.amazonaws.com:8883");
options.key = atob("MIIEow ... d4/IvHs");
options.cert = atob("MIIDWj ... +NrCQ==");
options.ca = atob("MIIE0z ... dao7WNq");

var mqtt = require("MQTT").create();
mqtt.on('connected', function() {
	mqtt.subscribe("test");
});

require("tls").connect(options, function(res) {
  mqtt.connect(res);
});

And the output:

>echo(0);
> Send DHCP_DISCOVER
> Receive DHCP_OFFER
> Send DHCP_Request
> Receive DHCP_ACK
Connecting with TLS...
Loading the CA root certificate...
Loading the Client certificate...
Loading the Client Key...
Performing the SSL/TLS handshake...
=undefined
Client connected
ERROR: Failed! mbedtls_ssl_handshake returned -0x4310
ERROR: Socket error -1 while sending
ERROR: Failed! mbedtls_ssl_handshake returned -0x4310
MQTT client disconnected

Gordon, I'll message you the code snippet with test keys as this may help in debugging. I'll continue to debug from my side as well.

Thanks,
Luke

Thanks - that's an odd error. It looks like it's as I said above though - out of memory :(

Without key,cert' andca` you'd probably be ok, but right now there doesn't seem to be enough free memory for all the TLS buffers required and the keys. I'll look into @allObjects' suggestion of storing the keys in flash - that could be a big help.

It works!

I tried your suggestion of deleting the keys after initiating the connection.

Updated code example:

SPI3.setup({ mosi:B5, miso:B4, sck:B3 });
var eth = require("WIZnet").connect(SPI3, A8);
eth.setIP();
eth.getIP();

var options = url.parse("mqtt://A1XXXXXXXXX9AA.iot.us-east-1.amazonaws.com:8883");
options.key = atob("MIIEow ... d4/IvHs");
options.cert = atob("MIIDWj ... +NrCQ==");
options.ca = atob("MIIE0z ... dao7WNq");

var mqtt = require("MQTT").create();
mqtt.on("connected", function() {
  console.log("connected to AWS IoT...");
  mqtt.subscribe("test/topic");
});

mqtt.on("publish", function (pub) {
  console.log("\r\nnew message received: ");
  console.log("topic: "+pub.topic);
  console.log("message: "+pub.message);
});

require("tls").connect(options, function(res) {
  console.log("tls connected");
  mqtt.connect(res);
});

delete options.key;
delete options.cert;
delete options.ca;

And the results:

Connecting with TLS...
Loading the CA root certificate...
Loading the Client certificate...
Loading the Client Key...
Performing the SSL/TLS handshake...
tls connected
Client connected
=undefined
>Verifying peer X.509 certificate...
MQTT connection accepted
connected to AWS IoT...
>
new message received:
topic: test/topic
message: hello world!

The 'hello world' message was published from another client application.

Exciting! Thanks for making this happen so quickly.

That's great! Glad it works!

It's not the best solution, but hopefully as time goes on I'll be able to find ways of improving the memory usage.

It'd be interesting to see if that works on ESP8266, but it may not - the driver takes up quite a bit of memory, and it may end up being too much.

Fyi, I have a version working that reads the certs from the file system (sd card on Espruino). I modified the appropriate functions in network.c to attempt this if the ca/cert/key values are present, but shorter than 100 characters (assuming a path) and the USE_FILESYSTEM option is set. It fixes the memory issues and seems like a good option for many devices.

I can submit a PR if you would like to incorporate this feature into the main project.

It might be nice to have flash memory as option for those without a file system. I'll look at adding this soon.

That's a great idea! Yes, a PR for that would be great.

In terms of flash memory, what's covered here would actually sort that out in a more general kind of way, so I think we should do that first.

I guess another cool option would be to allow a function to be specified. It would then be called and whatever it returned would be used.

I just pushed a branch for E.memoryArea - I'm not sure yet how it affects execution performance, there's a slim chance it's actually an improvement.

However, you should be able to store stuff in Flash and then use myStr = E.memoryArea(addr, len) to get a string that references it

I just put in a pull request for the feature that reads from the file system. PEM files can be stored in the default format and the certificate/key data will be parsed, removing beginning, ending delimiters and newline characters.

Here's example code using the file paths instead of cert strings:

var eth;
var connection;

var options = url.parse("mqtt://A1123456789AA.iot.us-east-1.amazonaws.com:8883");
options.ca = 'certs/rootCA.pem';
options.cert = 'certs/cert.pem';
options.key = 'certs/key.pem';

var mqtt = require("MQTT").create();
mqtt.on("connected", function() {
  console.log("connected to AWS IoT Service...");
  mqtt.subscribe("test/topic");
});

mqtt.on("publish", function (pub) {
  console.log("\nNew message received: ");
  console.log("topic: "+pub.topic);
  console.log("message: "+pub.message);
});

function onInit() {

  SPI2.setup({mosi:B15,miso:B14,sck:B13,baud:1000000});
  E.connectSDCard(SPI2,B1);
  
  SPI3.setup({ mosi:B5, miso:B4, sck:B3 });
  eth = require("WIZnet").connect(SPI3, A0);
  eth.setIP();
  
  require("tls").connect(options, function(res) {
    connection = res;
    console.log("tls connected");
    setTimeout(function() {
      mqtt.connect(res);
    },2000);
  });
}

The timeout is necessary to allow the certificate exchange and verification to complete before calling mqtt.connect. Maybe this can be event driven.

Great, thanks!

Why is the timeout needed? Is there some kind of error from mqtt.connect? The network layer should delay any transmission/reception until after the connection has been made.

... This is actually something that needs sorting generally - Espruino calls the 'connected' callback before the connection has actually been made (the current internal API for networking wasn't really meant for non-blocking connections).

It appears the MQTT client connection is rejected by the server. The TLS connection has been made, but hasn't finished authenticating yet (sd->connecting is still false, but the connected event is emitted). The mqtt connect should not be called until the TLS client authentication process is completed/authorized by the server.

Here's the output without the delay and some notes:

Connecting with TLS...
Loading the CA root certificate...
Loading certificate file: "certs/rootCA.pem"
Loading the Client certificate...
Loading certificate file: "certs/cert.pem"
Loading the Client Key...
Loading certificate file: "certs/key.pem"
Performing the SSL/TLS handshake...
TLS connected  ------------------->(logged in my TLS connected event handler)
Client connected  ----------------->(logged in MQTT module connected event)
Verifying peer X.509 certificate...   --->(logged ~line 530 of network.c)
MQTT client disconnected   --------->(logged in MQTT end event)

Maybe add an authorized event that fires after the authentication process? Or delay the connected event in this scenario?

Could it be that the MQTT module itself just times out?

Good question. I'll run a few tests.