-
Well, I'm not storing any of this data, just accumulating the byte count, so there's no 16-bit limitation involved, in my code anyway. Regarding free space, it's no different than my comment here.
At this point, I'm hoping Gordon will be able to reproduce the problem on his end using the sample data I provided.
-
From what I can tell, it's used mainly for filtering out services you don't care about discovering. For example, Apple uses a service type called something like "_daap" (digital audio something player) from iTunes to find compatible players. So it won't find your printers or web servers or home automation devices, because it's filtering by that service type. So I can create my own service type and then discover only the devices I created, ignoring everything else on the network.
I don't think it's a very big deal when you have only a few discoverable devices on your network, but if you had hundreds, filtering by service type is kinda cool.
-
I have a minimal nodejs app that extracts the data from the audio mixer. I was way off in the total number of bytes - the real count is actually 67,786 bytes for the entire state of the mixer. Here is the code that successfully got the message dump (obfuscated IP address):
let net = require('net'); let tcpSocket = net.Socket(); let mixerIpAddress = "169.254.0.1"; // A&H Qu Series Mixer IP let tcpPort = 51325; // ...and port let totalLength = 0; let sysExAllCall = new Uint8Array([0xF0, 0x00, 0x00, 0x1A, // Allen & Heath 0x50, 0x11, // Qu series 0x01, 0x00, // Major/Minor version 0x7F, // All MIDI channels 0x10, // ?? 0x00, // LSB is iPad bit, zero to disable requiring ActiveSense 0xF7] // All-Call ); function tcpConnect(details) { tcpSocket.connect({port: tcpPort, host: mixerIpAddress}, function() { console.log("...connected!"); tcpSocket.on('data', tcpOnMessage.bind(this)); tcpSocket.on('close', tcpOnDisconnected.bind(this)); // inform the Qu that we want telemetry from the device tcpSocket.write(sysExAllCall); }.bind(this)); } function tcpOnMessage(chunk) { totalLength += chunk.length; console.log("chunk length is " + chunk.length + ", totalLength is " + totalLength); } function tcpOnDisconnected() { console.log('client disconnected'); } // Get connected tcpConnect();
I've attached the file that represents the data dump (and there is no missing data from what I can tell). The data dump is a file of space-separated hex values, each representing a byte received from the mixer. At the very end, you'll see a few 'fe' single-byte messages, which is the MIDI ActiveSense byte sent once-per-second by the device. The fact that only 'fe' is being sent at the end of the data means the mixer has completed its dump of NRPN and SysEx data.
I dumped the messages as they arrived - no buffering or messing with the data. Each newline represents the end of the data for that particular message. If you need the file in a different format, let me know and I'll see what I can do.
Edit: how fast can the wifi go? For my project to work, it's gotta be pretty close to real network speeds. I saw 13Mbps was being mentioned on the internet - is that possible?
-
I tweaked Gordon's example code snippet for flexibility:
wifi.enableMDNS = function(hostname, serviceType, port, callback) { let mdns = "AT+MDNS=1," + JSON.stringify(hostname) + "," + JSON.stringify(serviceType) + "," + JSON.stringify(port) + "\r\n"; this.at.cmd(mdns, 500, function(d) { callback(d); }); };
Now you can specify the service type and port, like this:
wifi.enableMDNS("esp-livingroom", "iot", 80, function(result) { console.log("result is " + result); });
-
E.getErrorFlags() returned nothing. My stripped-down version of the code merely prints the number of bytes in each tcp on 'data' call. That version is calling dumpArray, isn't copying data, is only printing the count of data in each on 'data' call.
wifi.turbo() didn't help with respect to lost data, but boy the data arrived wicked-fast! I'm keeping that one.
I'm working on making a node version, stripped down, which is what I'll post when done.
-
-
Cool, thanks @Gordon. So I'm not sure where 'turnOn' and 'at' objects reside. It sucks that you have to hold my hand so much. How do I use this function - I can't just paste it into my code, because 'turnOn' and 'at' are undefined.
-
I got the ESP8266 fw update installed, validated it was 1.5.4, but still no difference. So here's what I'm doing:
I'm calling wifi.setHostname() in the wifi 'connected' callback, and I get a success on that.
Then I call 'dns-sd -B _http._tcp' from the Mac Terminal to browse for the device, not using Windows.
I fear that I need to do more, but I don't know what it would be.
-
-
Thanks, @Robin. Funny, I did a search on the espruino website for "mDNS" and got zero results. Yet that string exists in wifi.getHostname and wifi.setHostname. But I digress...
I inserted that into the code and didn't notice anything happen when I ran mDNS discovery, but I'm at the learning stage. Anyway, I'm guessing that the code will respond only when in access point mode - as a response to network devices looking for a DHCP server. I'm not using it in AP mode - I'm just trying to use zeroconf to discover other devices and to be discovered. So far adding .setHostname() didn't help. I imagine there are other steps I need to accomplish, but I don't know what they are.
-
-
In the end, I don't see the value in trying to view the entire buffer in one shot, at the end, when the buffer count already tells me I have 18k instead of 24k bytes of data received. In fact, that proves that dumping each buffer to the console as I receive it not causing any problems. If it were, I would have accumulated more data by not printing the data vs. printing the data. However, I get the same amount of data in either scenario.
I've stripped the code down to its bare minimum (where 'bare minimum' is defined as the code required to do a wifi/tcp connection and nothing else). No difference.
It's possible/likely that I need to set/change some configuration field that I'm unaware of, but other than that, I no longer think the problem is with my code. So unless there's a miracle, I'll have to think of something else with which to use these cool devices.
-
No, it's not in the data stream. The code snipped I copied from you is in a function that's called on a setInterval(), called long after I know I've received all the data. That error occurs on the console.log() of the attempt to dump the whole array.
I assume the CTRL-C was some interrupt or something going on in the microcontroller when it encounters a fatal memory error. I assume that because I've seen that exact message so many times while I was coding up my data processing code (which allocates memory). But none of that processing code is in use right now, only the attempt to dump data.
So I have to issue a correction, embarrassingly. The dumpArray() doesn't strictly allocate memory (as in new XXX()), but it creates a string that it appends the entire buffer to, and returns that string (to be printed to console). So there's absolutely another big buffer allocation going on when I try to dump the data.
My problem is that if I just loop through the array, and console.log() each byte, I get newlines after each byte:
0xb0
0x63
0x20
...and so on. It becomes unreadable. Any ideas on how to print to the console without the newline? -
Yes, the memory error occurs on console.log() is in line 3 of that example, where I'm trying to dump the entire 18k buffer. But again, it kinda doesn't matter that I suck at being able to dump the data - I know it's short 6kb or so.
If I use your try/catch sample above, I get this (which is the same as what I get if don't have try/catch):
1934 ERROR: Ctrl-C while processing interval - removing it. Execution Interrupted during event processing. New interpreter error: CALLBACK,MEMORY
-
Thanks a million for your thoughtful responses...
process.memory().free is 1942 at the end of the receipt of data.
No external memory module in use, literally nothing physically attached to the Wifi module.
Re: possible memory allocation issue: in the "real" version of my app, where I'm actually processing the incoming data, I perform memory allocation, enough to hold whatever has arrived in TCPSocket on 'data', but has yet to be processed. You can see that code here. Because I suspected my own code at fault, I commented all that out, and am doing only the console.log() bit described above. So no memory allocation at all. Well, that's a lie - I'm doing 'E.toUint8Array(chunk)' in the console.log(). But still, that's allocating up to a few hundred bytes, not allocating an ever-growing buffer.
RE: looping through the data, yes, that's exactly what my real code does: the TCPSocket on 'data' callback calls handlers that process the data, and spit out a single NRPN (or SysEx) message at a time. And it works flawlessly, until it encounters a truncated NRPN message (as shown previously), then it stops being able to process. The point here is that the issue occurs in every context in which I am receiving the data. I'm not seeing buffer overflows, out-of-memory errors, etc. The behavior, excuse me, behaviour, of the incoming data stream that my processing code sees is identical to the "just dump the buffer" code.
I could modify dumpArray() to manage the large buffer, but I already know what I'm going to get: ~8kb of missing data. The 18,796 byte count (in a 32kb pre-allocated buffer) already proves that.
-
@Robin,
Board Type: Espruino Wifi
Firmware version: 2v04
Espruino is USB-connected to my Mac, powered from USB
No other physical connections, as the MIDI-over-TCP communication is via WiFi
All buffers and buffer sizes are whatever Espruino created for a TCP Socket connectionRe: possible delays, I considered waiting to dump all data, but had out-of-memory issues. Probably due to coding errors by me. I can try again, now that I'm a little less ignorant.
Memory usage at start:
{"free":3961,"usage":3187,"total":7148,"history":1731,"gc":422,"gctime":9.96589660644,"stackEndAddress":536991644,"flash_start":134217728,"flash_binary_end":460040,"flash_code_start":134283264,"flash_length":524288}
Memory usage when all data received:
{"free":3995,"usage":3153,"total":7148,"history":1731,"gc":84,"gctime":9.58442687988,"stackEndAddress":536991644,"flash_start":134217728,"flash_binary_end":460040,"flash_code_start":134283264,"flash_length":524288}
Relevant code. Here is the TCP on 'connect' callback, which makes the All-Call request:
tcpConnect(details) { this.net.connect({port: this.tcpPort, host: details.address}, function(tcpSocket) { this.tcpSocket = tcpSocket; this.connected = true; console.log("...connected!"); this.tcpSocket.on('data', this.tcpOnMessage.bind(this)); this.tcpSocket.on('close', this.tcpOnDisconnected.bind(this)); // inform the client that we've connected this.onConnectedCallback(); // inform the Qu that we want telemetry from the device this.sendSysExAllCall(); }.bind(this)); }
...and here is the on 'data' callback:
tcpOnMessage(chunk) { // dumpArray() is just prepending the data with '0x' in front, etc console.log(this.utils.dumpArray(E.toUint8Array(chunk))); }
I tried to append the responses into a pre-allocated 32k buffer, and after receiving the entire response, and I still get out-of-memory errors when I try to console.log() it. So I changed it to merely dump the size of the data I received, and it showed 18,796, which is around 6kb short of expected - that's a data loss equivalent to 8 mixer channels. Curiously, multiple attempts at this seem to always return 18,796 bytes, every single time.
-
@Gordon, well I'm stumped. I am failing to receive all of the data from my Allen & Heath Qu-SB mixer via MIDI-over-TCP. In short, I am seeing truncated/missing data. I am sending a MIDI All-Call SysEx message, and I expect a big response to my request, over 24k bytes of data.
This MIDI protocol document (https://www.allen-heath.com/media/Qu_MIDI_Protocol_V1.9.pdf) describes what that response contains - it is the state of the mixer: every mixer channel's level, EQ, name, etc. Here is a small snippet:
0xb0 0x63 0x20 0xb0 0x62 0x17 0xb0 0x6 0x49 0xb0 0x26 0x7 0xb0 0x63 0x20 0xb0 0x62 0x18 0xb0 0x6 0x1 0xb0 0x26 0x7 0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x33 0xb0 0x26 0x0 0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x0 0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x53 0xb0 0x26 0x1 0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x1 0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x31 0xb0 0x26 0x2 0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x2 0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x2
These are MIDI NRPN messages, where each one of the lines that starts with '0xb0 0x63' is a single 12-byte NRPN message, containing information about a mixer channel (channel number, channel level, etc).
I should get 64 (12-byte) messages for each channel, and there are 32 channels to the mixer. That alone is over 24kb of data.
Meanwhile, I will also receive other SysEx data along with the NRPN data, each 20 bytes long. An example is here:
0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x20 0x4e 0x6f 0x72 0x64 0x20 0x4c 0x0 0xf7 0xfe
I should receive one of these for each of the 32 channels, because it contains that channel's name (as a 7-byte ascii string). That's only 640 bytes, practically in the noise.
According to the A&H documentation, and after consulting with A&H support, these NRPN messages and SysEx messages can in fact be intermingled - meaning, you might not receive all of the NRPN messages before you start receiving the SysEx messages. However, they will never intermingle in the middle of a message each other. In other words, you will always receive a complete NRPN message, followed by a complete SysEx message, followed by a complete NRPN message, in the response. So you might see this:
0xb0 0x63 0x2a 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 0xb0 0x63 0x2a 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x1 0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x20 0x4e 0x6f 0x72 0x64 0x20 0x4c 0x0 0xf7 0xfe 0xb0 0x63 0x2a 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1
...but you should never see this:
0xb0 0x63 0x2b 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x0 0xb0 0x63 0x2b 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x0 0xb0 0x63 0x2b 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 0xb0 0x63 0x2b 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x1 0xb0 0x63 0x2b 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 0xb0 0x63 0x2b 0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x21 0x4e 0x6f 0x72 0x64 0x20 0x52 0x0 0xf7 0xfe 0xb0 0x26 0x0 0xb0 0x63 0x37 0xb0 0x62 0x65 0xb0 0x6 0x57 0xb0 0x26 0x0 0xb0 0x63 0x37 0xb0 0x62 0x66 0xb0 0x6 0x5b 0xb0 0x26 0x0 0xb0 0x63 0x37 0xb0 0x62 0x67 0xb0 0x6 0x0 0xb0 0x26 0x0 0xb0 0x63 0x37 0xb0 0x62 0x68 0xb0 0x6 0x0 0xb0 0x26 0x0 0xb0 0x63 0x38 0xb0 0x62 0x61 0xb0 0x6 0x1 0xb0 0x26 0x0
Unfortunately, I often see the latter. Note that the third byte of the start of the NRPN message (0xb0 0x63 0x2b) jumps from 0x2b (mixer channel 11) to 0x37 (mixer channel 23). And it jumps to nearly the end of the 64-message payload, as it quickly moves on to 0x38 (mixer channel 24).
So in that response, I lost some of mixer channel 11, all of mixer channels 12-22, and most of mixer channel 23.
This occurs randomly in the response, never in the exact same place each time I issue the All-Call.
To test to see if the problem is in my code, I make the TCP connection, send the MIDI All-Call, and then dump the responses to the console. I typically receive a few hundred bytes per TCPSocket on 'data' callback, and all I do is dump that data to console. When I do that, I see the data loss described above. I'm not copying to another buffer, or doing any processing on it whatsoever.
Note that I'm not seeing data corruption - all of the data I receive is in fact real MIDI data - it's just truncated data. And this truncated data is typically NOT occurring around a TCPSocket on 'data' boundary. In fact, it's usually in the middle of a received data packet.
I realize you don't have an A&H mixer to test, so I'm putting you in a real bind, but is there anything you can think of that could cause this kind of data loss?
-
-
After checking out suggestions from @maze1980 and @Robin, I've created my TCP message handler, and it kinda works. I appear to be getting data corruption in the buffer that I copy the incoming data into, and I can't figure out what I'm doing wrong. The handler:
if(this.tcpBuffer && this.tcpBuffer.byteLength) { let leftoverDataLen = this.tcpBuffer.byteLength; let tmpBuffer = new Uint8Array(leftoverDataLen + chunk.length); tmpBuffer.set(this.tcpBuffer, 0); tmpBuffer.set(E.toUint8Array(chunk), leftoverDataLen); this.tcpBuffer = tmpBuffer; } else { this.tcpBuffer = E.toUint8Array(chunk); }
So I either start off with a brand new copy of the incoming data (converted to Uint8Array), or I append the new data to the end of the current buffer. When done, this.tcpBuffer should contain all of the data that I've not yet processed. Am I doing this correctly / is there a better way to append the data?
-
@Gordon yes I was completely forgetting some of the rules of scope in javascript.
-
@allObjects I agree with @Robin.
Getting a reliable 'pipe' going may be a challenge...
Most of the time, my code will be on the receiving end of MIDI data, and most of that will be a burst of NRPN messages after initial connection, and then mostly quiet. So far, so good on that front, but it's still early - plenty of time to screw things up ;-)
-
Thanks for the awesome detail @allObjects. My data is MIDI data, so technically 7 bit, although the extended MIDI data is 8-bit. I switched between Uint8 and Uint8Clamped and my data doesn't look any different, so I think I'm safe. However, my 16-bit binary data on USART? Different animal altogether. Good ole' endianness.
-
-
I have a connection to a device that's sending binary data over a TCP connection, and of course I get the data in chunks, which must be concatenated as new data comes in. Because the data I receive is binary, I have to convert the incoming data using E.toUint8Array, which works great. But when new data comes in, I have to add the new data to the existing Uint8Array, and I'm struggling to find the right way to do this. Anyone have experience with this?
Gordon, don't bother yourself with my pesky issue, it can wait. I'll ping you in mid/late-November or something. Until then, have a successful conference and product launch.
Wait, did you say new product launch?? :-)