How to properly handle incoming TCP/UDP data

@Gordon, well I'm stumped. I am failing to receive all of the data from my Allen & Heath Qu-SB mixer via MIDI-over-TCP. In short, I am seeing truncated/missing data. I am sending a MIDI All-Call SysEx message, and I expect a big response to my request, over 24k bytes of data.

This MIDI protocol document (https://www.allen-heath.com/media/Qu_MID­I_Protocol_V1.9.pdf) describes what that response contains - it is the state of the mixer: every mixer channel's level, EQ, name, etc. Here is a small snippet:

0xb0 0x63 0x20 0xb0 0x62 0x17 0xb0 0x6 0x49 0xb0 0x26 0x7 
0xb0 0x63 0x20 0xb0 0x62 0x18 0xb0 0x6 0x1 0xb0 0x26 0x7 
0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x33 0xb0 0x26 0x0 
0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x53 0xb0 0x26 0x1 
0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 
0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x1 
0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x31 0xb0 0x26 0x2 
0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x2 
0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x2 

These are MIDI NRPN messages, where each one of the lines that starts with '0xb0 0x63' is a single 12-byte NRPN message, containing information about a mixer channel (channel number, channel level, etc).

I should get 64 (12-byte) messages for each channel, and there are 32 channels to the mixer. That alone is over 24kb of data.

Meanwhile, I will also receive other SysEx data along with the NRPN data, each 20 bytes long. An example is here:

0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x20 0x4e 0x6f 0x72 0x64 0x20 0x4c 0x0 0xf7 0xfe

I should receive one of these for each of the 32 channels, because it contains that channel's name (as a 7-byte ascii string). That's only 640 bytes, practically in the noise.

According to the A&H documentation, and after consulting with A&H support, these NRPN messages and SysEx messages can in fact be intermingled - meaning, you might not receive all of the NRPN messages before you start receiving the SysEx messages. However, they will never intermingle in the middle of a message each other. In other words, you will always receive a complete NRPN message, followed by a complete SysEx message, followed by a complete NRPN message, in the response. So you might see this:

0xb0 0x63 0x2a 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x2a 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x1 
0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x20 0x4e 0x6f 0x72 0x64 0x20 0x4c 0x0 0xf7 0xfe
0xb0 0x63 0x2a 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 

...but you should never see this:

0xb0 0x63 0x2b 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x0 
0xb0 0x63 0x2b 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x2b 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x2b 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x1 
0xb0 0x63 0x2b 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 
0xb0 0x63 0x2b 
0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x21 0x4e 0x6f 0x72 0x64 0x20 0x52 0x0 0xf7 0xfe 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x65 0xb0 0x6 0x57 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x66 0xb0 0x6 0x5b 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x67 0xb0 0x6 0x0 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x68 0xb0 0x6 0x0 0xb0 0x26 0x0 
0xb0 0x63 0x38 0xb0 0x62 0x61 0xb0 0x6 0x1 0xb0 0x26 0x0 

Unfortunately, I often see the latter. Note that the third byte of the start of the NRPN message (0xb0 0x63 0x2b) jumps from 0x2b (mixer channel 11) to 0x37 (mixer channel 23). And it jumps to nearly the end of the 64-message payload, as it quickly moves on to 0x38 (mixer channel 24).

So in that response, I lost some of mixer channel 11, all of mixer channels 12-22, and most of mixer channel 23.

This occurs randomly in the response, never in the exact same place each time I issue the All-Call.

To test to see if the problem is in my code, I make the TCP connection, send the MIDI All-Call, and then dump the responses to the console. I typically receive a few hundred bytes per TCPSocket on 'data' callback, and all I do is dump that data to console. When I do that, I see the data loss described above. I'm not copying to another buffer, or doing any processing on it whatsoever.

Note that I'm not seeing data corruption - all of the data I receive is in fact real MIDI data - it's just truncated data. And this truncated data is typically NOT occurring around a TCPSocket on 'data' boundary. In fact, it's usually in the middle of a received data packet.

I realize you don't have an A&H mixer to test, so I'm putting you in a real bind, but is there anything you can think of that could cause this kind of data loss?