micro-mqtt

Hi,

I got some ESP8266 modules and one I already use to control my heating with pimatic and the MQTT protocol. Currently I programmed the ESP8266 the Arduino way using an Arduino Client for MQTT. It proved to be very stable, currently almost 28 days online!

I would like to program the modules with JavaScript (and preferably Typescript) so I looked into the Espruino implementation of MQTT. To be honest it appeared to lack the quality I like to see for such an important library. I also miss some features which are important for my use case, like Last Will and Testament and sending retained messages.

Since I also wanted to get my feet wet with TypeScript and the node.js stack, I decided to start a small open source project where I will rewrite the current MQTT implementation and extend it. I currently rewrote it and I also fixed some issues.

I hope it is useful for others too and I would really like to get some feedback on this project. Please let me know.

Regards,
Ronald

Awesome news! I'll give it a go next time I start or work on one of my espruino projects, I try to do all my comms for my home automation over mqtt so I'm keen on an improved client for espruino.

Thanks for your response! Glad to hear you would like to give it a go.

I'd be interested to hear what you're worried about quality-wise. Some things are done specifically to make it more space-efficient, so if you're not careful you may find that your version uses a lot more memory.

What are the fixed issues? It would at least be worth trying to pull those back into the current MQTT library.

Hi Gordon,

Here are some issues I found:

1. When there is a problem connecting it throws a JavaScript error. For this I created a pull request.
   
2. When receiving multiple Publish packets at once, it only sees the first one. This is a common scenario when subscribing and receiving some retained messages from the broker, but also if another client publishes multiple messages.
   
3. When parsing Publish packets it does not take into account that a QoS 0 message does not have a packet id, therefore the message the parser returns is malformed when receiving multiple Publish packets at once.
   
4. It does not send PubAck messages. This results in the broker re-sending QoS 1 Publish packets over and over again.
   
   

I would be happy to help getting those fixes back in, no problem. Maybe it's also a good idea if we would coo-operate on my rewrite. It's fully covered by tests, which definitively helps to keep the quality high. What do you think about that?

Of course I am also concerned about the available space on the device, therefore I would only add what is required for a stand-alone MQTT controlled device and I would remove what's not really needed. Do you have some advise for me how to keep it small and how to keep the memory usage as low as possible?

@Rovale since you're "building" the module, maybe you'd be able to generate two modules by excluding some features, a basic one and a full-featured one. Basic is pretty much publish and subscribe, doesn't do anything extra like Last Will, full feature does everything.

Hi, my purpose is to create an MQTT client which is reliable enough that I would dare to control my heating with it, for instance. It should:

• Always try to stay connected. It should reconnect if the connection is lost.
  
• Be able to publish and subscribe at QoS 1 level (actually I think it's impossibly to truly support publishing at QoS 1).
  
• Be able to publish retained messages to allow other devices which just connect to get up to date with the last status messages.
  
• Have a Last Will Testament to let other devices know it it is online or offline.
  
  

On the other hand it does not need to:

• Support QoS 2.
  
• Unsubscribe.
  
• Disconnect.
  
• Get user friendly error messages.
  
• Have a rich monkey proof API with too many overloads.
  
  

If size is a serious issue then it can be interesting to consider multiple variants of MQTT client. What would your typical use case be and what features you think it would require?

I am still very new on the Espruino field so if somebody can give me some guidance on measuring size and memory usage and on how to keep things small, then that would be helpful.

By the way, yesterday I implemented the Last Will and Testament feature. It did not take that much new code :-)

Nice, the last will really doesn't take much! I wonder whether subscribing wasn't really used/tested? I guess most people have been using MQTT just to push data - I know that's all I did for the initial proof of concept, and as you note, the message parsing looks extremely rough around the edges.

Hopefully fixes for what you suggested wouldn't be too hard though...

To check available memory, just upload and type process.memory() - .usage is what you're interested in. Check out http://www.espruino.com/Performance for some info on getting code that works well - one of the biggest things is making sure that code minifies well - which is why the MQTT lib uses private functions where it can.

I'm afraid I really don't have time to help out with a new module though. If you did want to contribute this as a new module, the fact that it's in TypeScript makes life quite difficult - I guess TypeScript compilation (and all its dependencies) would need adding to the build process, just for this one module.

I'm surprised about the need for QoS1 - surely TCP/IP handles all that for you?

In terms of staying connected, that might be better as a separate module? Ideally an MQTT lib would be capable of running over any connection type, so if possible you probably won't want transport-specific code in there.

Thanks! Available memory and size is the next thing that I want to focus on.

Good question about QoS1! So does TCP/IP quarantee that all packets are delivered at least once? I am not an expert on that field... And could the MQTT client miss packets if it is too busy at a certain point of time? Actually it might make sense to track the DUP flag, just to to be sure.

I am not sure about your remark about staying connected. In the end the client will have to sent a Connect packet again, not?

Gordon, about the new module, you're helping me more than enough with your comments. Helps me to think things trough. If TypeScript compilation is an issue than I can also consider to add the compiled/minified JavaScript to GitHub. Or mabe we can add an extra step to the TravisCI build? Whatever works best.

Thanks - yes, TCP/IP basically guarantees that all packets arrive, in the right order, and only once - so MQTT QoS is basically just reimplementing what TCP/IP does already.

Yes, there would have to be another connect packet if the connection was lost, but my point is that you might be using MQTT over WebSockets (for instance... or something else) in which case the reconnection code would be totally different.

So actually it might be interesting to drop the support for QoS1. That will save quite some space. The only problem might be that clients which actually do require QoS1 will never receive QoS1 packets from the micro-mqtt client because the QoS of a packet never upgrades. It can only downgrade.

WebSockets, or something else... Nice thought. I already wanted to get rid of the dependency on 'net'. Need to think about a more generic interface.

Question: is it fair to say that the actual usage of the running code = process().memory.usage -/- process().memory.history?

I already did some optimizations. Currently, if the following client is up and running:

"use strict";
function onInit() {
    var Client = require('micro-mqtt').Client;
    var client = new Client({
        host: '192.168.2.4',
        clientId: 'espruino',
        username: 'username', password: 'password',
        will: {
            topic: 'rovale/espruino/status',
            message: 'offline',
            qos: 1,
            retain: true
        }
    });
    client.on('connected', function () {
        client.subscribe('rovale/#', 1);
        client.publish('rovale/espruino/status', 'online', 1, true);
    });
    client.on('publish', function (pub) {
        console.log('on publish');
        console.log(pub);
    });
    client.on('debug', function (debug) {
        console.log('[debug] ' + debug);
    });
    client.on('info', function (info) {
        console.log('[info] ' + info);
    });
    client.on('error', function (error) {
        console.log('[error] ' + error);
    });
    client.connect();
    global.client = client;
}

it shows:

>process.memory();
={ "free": 626, "usage": 774, "total": 1400, "history": 58 }

I started off with:

>process.memory();
={ "free": 521, "usage": 879, "total": 1400, "history": 58 }

So that is already quite an improvement. Now I want to keep track of possible memory leaks, for instance after losing the network connection.

history is the memory used by the command history in the console - but it's not included in the number for usage or free (as the history is automatically freed if memory gets low)

Ah, I see, thanks!