TinyMQTT messages not fully delivered

Posted on
  • I'm using TinyMQTT in a home automation project on a Pico + ESP8266.

    About 20% of the time when handling new messages to subscribed topics, my message events fire but do not contain the topic or message, meaning I am unable to handle the message properly.

    Subscribing using mosquitto_sub to check what is going on shows that the messages are definitely there and look normal, so this is an issue on the Pico side.

    As a workaround I am sending an error back in the case of a null topic so that the UI can ask the user to try again but this is not satisfactory.

    The fact it happens only a proportion of the time for identical messages suggests to me that it might be some kind of latency issue reading the data.

    Anyone seen anything like this before with TinyMQTT? I am loth to switch to the much heavier MQTT, although I haven't tried it to see if it behaves better yet.

    function MQTTmessage(msg) {
      console.log("MQTT message " + msg.topic + ":" + msg.message); 
      if (msg.topic.endsWith("/topic1")) {
        //doSomething();
      }
      if (msg.topic.endsWith("/topic2")) {
        //doSomethingElse();
      }
      if (msg.topic == "" || msg.topic == null) {
        //in this case msg.message is also empty
        throwError(null, "null message received - please try again", false);
      }
    
    
  • I put some debugging code into a copy of TinyMQTT.js, and in each case with the problem MQTT messages the first byte of data (or sometimes the first two bytes of data) is/are missing.

    Good receipt:

    cmd 3 / 30:16:0:10:Coffeemaker/brewbrew

    Bad receipt, it is received as two separate calls to the data handler in TinyMQTT (onDat):

    cmd 3 / 30:0:0:0:
    cmd 1 / 16:0:10:43:offeemaker/brewbrew

    We can see that if these were received in a single call to onDat, the publish command would be good.

    Looking at the code for TinyMQTT, I don't think the issue can be in there, it must be somewhere else - any thoughts on where to look next? Or any thoughts on a sensible workaround, maybe compelling the socket to wait a little longer for data somehow - but looks like that code is maybe built into the firmware in the net module.

    In the meantime I have put in a horrible hack so that if it is a publish command (3) with a total length of less than 5 bytes - which should never happen in my setup - then it stores up to 4 bytes in state waiting for the next call, and prepends these to the second set of data it receives.

    Here is the hacked replacement onDat function for TinyMQTT.js

    function onDat(data) {
        var i; 
        //var log="MQTT receiving ";
        if (_q.tmp != null) {
            //log += ' APPLYING STORED DATA ';
            data = _q.tmp + data;
            _q.tmp = null;
        }
        
        var length = data.length;
        var cmd = data.charCodeAt(0) >> 4;
        //log += " length " + data.length + " cmd " + cmd.toString() + " / ";
        
        if (cmd ===3 && length <5) {
            _q.tmp = data;
            //console.log(log + " STORING EXTRA DATA FOR NEXT TRANSMIT");
            return;
        }
        
        //for(i=0;i<4;i++) {
        //  log = log + data.charCodeAt(i).toString(16) + ":";
        //}
        //log += data.substr(4,256);
        //console.log(log);
        //log = "";
        if(cmd === 3) {
            var var_len = data.charCodeAt(2) << 8 | data.charCodeAt(3);
            var msg = {
                topic: data.substr(4, var_len),
                message: data.substr(4+var_len, (data.charCodeAt(1))-var_len)
            };
            _q.emit("message", msg);
        }
    }
    
    

    All kinds of things could go wrong with this approach, especially if new messages are received in quick succession so I don't really want to live with it long-term.

  • Thanks - it sounds like you may be hitting this bug: https://github.com/olliephillips/tinyMQT­T/issues/30

    There's some code in there that may help? I was waiting for it to get merged into the main TinyMQTT so I could get it in Espruino's version.

  • Thanks definitely same bug. Your solution is better as I couldn't be bothered to figure out the length calculations, though with a packet size of less than 127 I could have taken it easy:->

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

TinyMQTT messages not fully delivered

Posted by Avatar for Moray @Moray

Actions