TinyMQTT messages not fully delivered

I'm using TinyMQTT in a home automation project on a Pico + ESP8266.

About 20% of the time when handling new messages to subscribed topics, my message events fire but do not contain the topic or message, meaning I am unable to handle the message properly.

Subscribing using mosquitto_sub to check what is going on shows that the messages are definitely there and look normal, so this is an issue on the Pico side.

As a workaround I am sending an error back in the case of a null topic so that the UI can ask the user to try again but this is not satisfactory.

The fact it happens only a proportion of the time for identical messages suggests to me that it might be some kind of latency issue reading the data.

Anyone seen anything like this before with TinyMQTT? I am loth to switch to the much heavier MQTT, although I haven't tried it to see if it behaves better yet.

function MQTTmessage(msg) {
  console.log("MQTT message " + msg.topic + ":" + msg.message); 
  if (msg.topic.endsWith("/topic1")) {
    //doSomething();
  }
  if (msg.topic.endsWith("/topic2")) {
    //doSomethingElse();
  }
  if (msg.topic == "" || msg.topic == null) {
    //in this case msg.message is also empty
    throwError(null, "null message received - please try again", false);
  }

I put some debugging code into a copy of TinyMQTT.js, and in each case with the problem MQTT messages the first byte of data (or sometimes the first two bytes of data) is/are missing.

Good receipt:

cmd 3 / 30:16:0:10:Coffeemaker/brewbrew

Bad receipt, it is received as two separate calls to the data handler in TinyMQTT (onDat):

cmd 3 / 30:0:0:0:
cmd 1 / 16:0:10:43:offeemaker/brewbrew

We can see that if these were received in a single call to onDat, the publish command would be good.

Looking at the code for TinyMQTT, I don't think the issue can be in there, it must be somewhere else - any thoughts on where to look next? Or any thoughts on a sensible workaround, maybe compelling the socket to wait a little longer for data somehow - but looks like that code is maybe built into the firmware in the net module.

In the meantime I have put in a horrible hack so that if it is a publish command (3) with a total length of less than 5 bytes - which should never happen in my setup - then it stores up to 4 bytes in state waiting for the next call, and prepends these to the second set of data it receives.

Here is the hacked replacement onDat function for TinyMQTT.js

function onDat(data) {
	var i; 
	//var log="MQTT receiving ";
	if (_q.tmp != null) {
		//log += ' APPLYING STORED DATA ';
		data = _q.tmp + data;
		_q.tmp = null;
	}
	
	var length = data.length;
	var cmd = data.charCodeAt(0) >> 4;
	//log += " length " + data.length + " cmd " + cmd.toString() + " / ";
	
	if (cmd ===3 && length <5) {
		_q.tmp = data;
		//console.log(log + " STORING EXTRA DATA FOR NEXT TRANSMIT");
		return;
	}
	
	//for(i=0;i<4;i++) {
	//	log = log + data.charCodeAt(i).toString(16) + ":";
	//}
	//log += data.substr(4,256);
	//console.log(log);
	//log = "";
	if(cmd === 3) {
		var var_len = data.charCodeAt(2) << 8 | data.charCodeAt(3);
		var msg = {
			topic: data.substr(4, var_len),
			message: data.substr(4+var_len, (data.charCodeAt(1))-var_len)
		};
		_q.emit("message", msg);
	}
}

All kinds of things could go wrong with this approach, especially if new messages are received in quick succession so I don't really want to live with it long-term.

Thanks - it sounds like you may be hitting this bug: https://github.com/olliephillips/tinyMQTT/issues/30

There's some code in there that may help? I was waiting for it to get merged into the main TinyMQTT so I could get it in Espruino's version.

Thanks definitely same bug. Your solution is better as I couldn't be bothered to figure out the length calculations, though with a packet size of less than 127 I could have taken it easy:->

@Moray can you please try again.

https://github.com/MaBecker/tinyMQTT

@MaBe, do you have fully featured test bed for tinyMQTT as at https://github.com/MaBecker/tinyMQTT ?

do you have fully featured test bed for tinyMQTT as at

I use a local installed mqtt server , MQTTBox to obserrve and some js snippets.

To fix the last issue a snipped given by Gordon was used.

var mqtt = require('mqtt');
var client  = mqtt.connect('mqtt://localhost');
function go(i) { client.publish('test', ""+i+'.123')}
setInterval(function() {go(1);go(2);go(3);go(4); console.log("4xgo()");}, 5000);

Simple test frame

Modules.addCached("tinyMQTT-MAC",
// place function here
);

Modules.addCached("tinyMQTT-MAC-MIN",
// place minimized function here
);


var mqtt = require("tinyMQTT-MAC").create("127.0.0.1");
//var mqtt = require("tinyMQTT-MAC-MIN").create("127.0.0.1");

mqtt.on("connected", function(){mqtt.subscribe("test");});
mqtt.on("message", function(msg){console.log('['+msg.topic+'] : ['+ msg.message+']');});
mqtt.on("published", function(){console.log("message sent");});
mqtt.on("disconnected", function(){ console.log("disconnected");});