How to properly handle incoming TCP/UDP data

E.getErrorFlags() returned nothing. My stripped-down version of the code merely prints the number of bytes in each tcp on 'data' call. That version is calling dumpArray, isn't copying data, is only printing the count of data in each on 'data' call.

wifi.turbo() didn't help with respect to lost data, but boy the data arrived wicked-fast! I'm keeping that one.

I'm working on making a node version, stripped down, which is what I'll post when done.

@Gordon,

I have a minimal nodejs app that extracts the data from the audio mixer. I was way off in the total number of bytes - the real count is actually 67,786 bytes for the entire state of the mixer. Here is the code that successfully got the message dump (obfuscated IP address):

let net = require('net');

let tcpSocket = net.Socket();

let mixerIpAddress = "169.254.0.1";	// A&H Qu Series Mixer IP
let tcpPort = 51325;				// ...and port
let totalLength = 0;

let sysExAllCall = new Uint8Array([0xF0,
	0x00, 0x00, 0x1A, 	// Allen & Heath
	0x50, 0x11,			// Qu series 
	0x01, 0x00,			// Major/Minor version
	0x7F, 				// All MIDI channels
	0x10,				// ??
	0x00,				// LSB is iPad bit, zero to disable requiring ActiveSense
	0xF7]				// All-Call
);

function tcpConnect(details) {
	tcpSocket.connect({port: tcpPort, host: mixerIpAddress}, function() {
		console.log("...connected!");

		tcpSocket.on('data', tcpOnMessage.bind(this));
		tcpSocket.on('close', tcpOnDisconnected.bind(this));

		// inform the Qu that we want telemetry from the device
		tcpSocket.write(sysExAllCall);

	}.bind(this));
}

function tcpOnMessage(chunk) {
	totalLength += chunk.length;
	console.log("chunk length is " + chunk.length + ", totalLength is " + totalLength);
}

function tcpOnDisconnected() {
	console.log('client disconnected');
}

// Get connected
tcpConnect();

I've attached the file that represents the data dump (and there is no missing data from what I can tell). The data dump is a file of space-separated hex values, each representing a byte received from the mixer. At the very end, you'll see a few 'fe' single-byte messages, which is the MIDI ActiveSense byte sent once-per-second by the device. The fact that only 'fe' is being sent at the end of the data means the mixer has completed its dump of NRPN and SysEx data.

I dumped the messages as they arrived - no buffering or messing with the data. Each newline represents the end of the data for that particular message. If you need the file in a different format, let me know and I'll see what I can do.

Edit: how fast can the wifi go? For my project to work, it's gotta be pretty close to real network speeds. I saw 13Mbps was being mentioned on the internet - is that possible?

1 Attachment

• node-nrpn-hex.txt

Fri 2019.10.25

'the real count is actually 67,786 bytes'

Two observations:

1) 67786 is greater than 65535 - might be a factor

2) From #15 available JsVars 3995 (understood that program may have grown in the last week)

67786 / 16 = 4237

What is the program usage, and/or free space before the data fetch? (on EspruinoWiFi not via Node.js)

Well, I'm not storing any of this data, just accumulating the byte count, so there's no 16-bit limitation involved, in my code anyway. Regarding free space, it's no different than my comment here.

At this point, I'm hoping Gordon will be able to reproduce the problem on his end using the sample data I provided.

I'm crazy busy the next few weeks gearing up for a conference and a new product launch so I may not get time until after then I'm afraid. If I don't get around to it, maybe you could ping me after that?

Gordon, don't bother yourself with my pesky issue, it can wait. I'll ping you in mid/late-November or something. Until then, have a successful conference and product launch.

Wait, did you say new product launch?? :-)

Mon 2019.10.28

shhhhhhhhh don't let the cat out of the bag yet

We may not be totally communicating here, as I made assumptions in #28 not knowing whether the data payload was tested on the actual device at that point. (rather than node)

My observation of 4237 created is greater than 3995 available just popped out. If it was and as Gordon pointed out that Espruino may be creating an additional large string during decoding, you can see my concern.

Any way to just call half the data payload in a pseudo MIDI All-Call SysEx message call, and then while monitoring free mem, just keep increasing the payload request until the error occurs? This would iron out whether the error is caused internally by the data itself, or an overrun of memory BEFORE Espruino renders the error.

or are we at a point where the test using node will take less time than writing repetitive try test repeat trials?

'so there's no 16-bit limitation involved'

Again, are we communicating re: JsVar usage bottom of page?

http://www.espruino.com/Internals

Sixteen was used as a divisor, not a limitation

did you say new product launch?? :-)

;) Keep an eye open on Nov 11th - that's when I'm launching it!