TinyMQTT messages not fully delivered

Posted on
  • I'm using TinyMQTT in a home automation project on a Pico + ESP8266.

    About 20% of the time when handling new messages to subscribed topics, my message events fire but do not contain the topic or message, meaning I am unable to handle the message properly.

    Subscribing using mosquitto_sub to check what is going on shows that the messages are definitely there and look normal, so this is an issue on the Pico side.

    As a workaround I am sending an error back in the case of a null topic so that the UI can ask the user to try again but this is not satisfactory.

    The fact it happens only a proportion of the time for identical messages suggests to me that it might be some kind of latency issue reading the data.

    Anyone seen anything like this before with TinyMQTT? I am loth to switch to the much heavier MQTT, although I haven't tried it to see if it behaves better yet.

    function MQTTmessage(msg) {
      console.log("MQTT message " + msg.topic + ":" + msg.message); 
      if (msg.topic.endsWith("/topic1")) {
        //doSomething();
      }
      if (msg.topic.endsWith("/topic2")) {
        //doSomethingElse();
      }
      if (msg.topic == "" || msg.topic == null) {
        //in this case msg.message is also empty
        throwError(null, "null message received - please try again", false);
      }
    
    
  • I put some debugging code into a copy of TinyMQTT.js, and in each case with the problem MQTT messages the first byte of data (or sometimes the first two bytes of data) is/are missing.

    Good receipt:

    cmd 3 / 30:16:0:10:Coffeemaker/brewbrew

    Bad receipt, it is received as two separate calls to the data handler in TinyMQTT (onDat):

    cmd 3 / 30:0:0:0:
    cmd 1 / 16:0:10:43:offeemaker/brewbrew

    We can see that if these were received in a single call to onDat, the publish command would be good.

    Looking at the code for TinyMQTT, I don't think the issue can be in there, it must be somewhere else - any thoughts on where to look next? Or any thoughts on a sensible workaround, maybe compelling the socket to wait a little longer for data somehow - but looks like that code is maybe built into the firmware in the net module.

    In the meantime I have put in a horrible hack so that if it is a publish command (3) with a total length of less than 5 bytes - which should never happen in my setup - then it stores up to 4 bytes in state waiting for the next call, and prepends these to the second set of data it receives.

    Here is the hacked replacement onDat function for TinyMQTT.js

    function onDat(data) {
    	var i; 
    	//var log="MQTT receiving ";
    	if (_q.tmp != null) {
    		//log += ' APPLYING STORED DATA ';
    		data = _q.tmp + data;
    		_q.tmp = null;
    	}
    	
    	var length = data.length;
    	var cmd = data.charCodeAt(0) >> 4;
    	//log += " length " + data.length + " cmd " + cmd.toString() + " / ";
    	
    	if (cmd ===3 && length <5) {
    		_q.tmp = data;
    		//console.log(log + " STORING EXTRA DATA FOR NEXT TRANSMIT");
    		return;
    	}
    	
    	//for(i=0;i<4;i++) {
    	//	log = log + data.charCodeAt(i).toString(16) + ":";
    	//}
    	//log += data.substr(4,256);
    	//console.log(log);
    	//log = "";
    	if(cmd === 3) {
    		var var_len = data.charCodeAt(2) << 8 | data.charCodeAt(3);
    		var msg = {
    			topic: data.substr(4, var_len),
    			message: data.substr(4+var_len, (data.charCodeAt(1))-var_len)
    		};
    		_q.emit("message", msg);
    	}
    }
    
    

    All kinds of things could go wrong with this approach, especially if new messages are received in quick succession so I don't really want to live with it long-term.

  • Thanks - it sounds like you may be hitting this bug: https://github.com/olliephillips/tinyMQT­T/issues/30

    There's some code in there that may help? I was waiting for it to get merged into the main TinyMQTT so I could get it in Espruino's version.

  • Thanks definitely same bug. Your solution is better as I couldn't be bothered to figure out the length calculations, though with a packet size of less than 127 I could have taken it easy:->

  • @Moray can you please try again.

    https://github.com/MaBecker/tinyMQTT

  • @MaBe, do you have fully featured test bed for tinyMQTT as at https://github.com/MaBecker/tinyMQTT ?

  • do you have fully featured test bed for tinyMQTT as at

    I use a local installed mqtt server , MQTTBox to obserrve and some js snippets.

    To fix the last issue a snipped given by Gordon was used.

    var mqtt = require('mqtt');
    var client  = mqtt.connect('mqtt://localhost');
    function go(i) { client.publish('test', ""+i+'.123')}
    setInterval(function() {go(1);go(2);go(3);go(4); console.log("4xgo()");}, 5000);
    

    Simple test frame

    Modules.addCached("tinyMQTT-MAC",
    // place function here
    );
    
    Modules.addCached("tinyMQTT-MAC-MIN",
    // place minimized function here
    );
    
    
    var mqtt = require("tinyMQTT-MAC").create("127.0.0.­1");
    //var mqtt = require("tinyMQTT-MAC-MIN").create("127.­0.0.1");
    
    mqtt.on("connected", function(){mqtt.subscribe("test");});
    mqtt.on("message", function(msg){console.log('['+msg.topic+­'] : ['+ msg.message+']');});
    mqtt.on("published", function(){console.log("message sent");});
    mqtt.on("disconnected", function(){ console.log("disconnected");});
    
  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

TinyMQTT messages not fully delivered

Posted by Avatar for Moray @Moray

Actions