How to properly handle incoming TCP/UDP data

I have a connection to a device that's sending binary data over a TCP connection, and of course I get the data in chunks, which must be concatenated as new data comes in. Because the data I receive is binary, I have to convert the incoming data using E.toUint8Array, which works great. But when new data comes in, I have to add the new data to the existing Uint8Array, and I'm struggling to find the right way to do this. Anyone have experience with this?

No, but .set seems to fit:

a = new Uint8Array([1,2,3,4,5,6,7,8,9,10]);
b = new Uint8Array(7); // 0,0,0,0,0,0,0
c = new Uint8Array(a.buffer, 2, 5); // [3,4,5,6,7]
b.set(c, 2); // set b with the contents of c starting from index 2
b; // [0,0,3,4,5,6,7]

Source: https://www.espruino.com/Performance

Wed 2019.08.14

Have you considered for memory efficiency @indianajones using a clamped array?

http://www.espruino.com/Reference#Uint8C­lampedArray

In #2 @maze1980 points out the set() function which is the way I do this task. Access can even be sped up with 'inlineC', but really won't be needed for your project needs. Grasping the hierarchy took me a bit, as the above is inherited from ArrayBufferView. Uint8ClampedArray is a type of TypedArray from which the set method is defined.

Also check out heading 'STRINGS VS. FLAT STRINGS' from link in #2 above.

For your 'light' reading pleasure ;-)

https://developer.mozilla.org/en-US/docs­/Web/JavaScript/Reference/Global_Objects­/Uint8ClampedArray#Methods
https://developer.mozilla.org/en-US/docs­/Web/JavaScript/Reference/Global_Objects­/TypedArray/set
https://developer.mozilla.org/en-US/docs­/Web/JavaScript/Reference/Global_Objects­/ArrayBuffer
https://developer.mozilla.org/en-US/docs­/Web/JavaScript/Reference/Global_Objects­/DataView

@maze1980 and @Robin, thanks again for your suggestions and links. I had been using Uint8Array, or should I say mis-using it. After reading the links and trying more iterations, I settled on using Uint8ClampedArray, oh and using it correctly also helps. It's working very well now, thanks!

Watch out when you use Uint8Array and Uint8ClampedArray.

First of all, regarding memory efficiency / consummation, they are absolutely the same.

Second, they have the same protocol (state and behavioral properties / protocol).

Third, they construct all from a plain array or from an array view. Former looks at the values of each source element where latter looks at the bytes the the source array is stored.

Where they differ is handling the values of the source array when constructed, as code and output below show. A plain array with integers of different values is passed in the constructors of an Uint8Array and an Uint8ClampedArray:

• Uint8Array just takes the least significant 8 bits
  
• Uint8ClampedArray looks at the value and applies min/max function to be 0 or 255.
  
  

If you need to transmit just 8 bits and the source elements are values up to 8 bits unsigned or 8-bit characters or 8-bit chars subset of 16-bit (or more) char set, you are just fine - you can think of 0..255 range - 256 different values.

If they are signed integer numbers, you can think of only 0..127 range - still 256 different values - but you have to process them differently when interpreting the bytes back to numbers: you have to check MSB - most significant bit - and then do some math or bit manipulation for getting back the original value (in the 8+ bit format).

// ui8_vs_ui8Clamped_arrays.js
// 2019-08-15 - allObjects
var arr
  , ui8Arr
  , uiCArr
  ;

function fmtI(n,l) {
  var s = "            "
        + ((typeof n == "number")
            ? n
            : (""+n).substr(0,l)
          );
  return s.substr(s.length - (l || 12));
}
function onInit() {

arr =
    [    0
    ,    1
    ,   -1
    ,  127
    , -127
    , -128
    ,  128
    ,  255
    , -255
    , -256
    ,  256
    ];
ui8Arr = new Uint8Array(arr);
uiCArr = new Uint8ClampedArray(arr);
fmtLen = 5;
console.log(
    fmtI(" src"       ,fmtLen)
  , fmtI(" ui8"       ,fmtLen)
  , fmtI(" uiClampled",fmtLen)
  );
arr.forEach(function(e,i,a) {
  console.log(
      fmtI(arr[i]   ,fmtLen)
    , fmtI(ui8Arr[i],fmtLen)
    , fmtI(uiCArr[i],fmtLen)
    );
} );
  
}

setTimeout(onInit,500);

And respective output ('packed' into a multiline string for legible color in forum css):

 `
 ____                 _
|  __|___ ___ ___ _ _|_|___ ___
|  __|_ -| . |  _| | | |   | . |
|____|___|  _|_| |___|_|_|_|___|
         |_| espruino.com
 2v04 (c) 2019 G.Williams
>
  src   ui8  uiCl
    0     0     0
    1     1     1
   -1   255     0
  127   127   127
 -127   129     0
 -128   128     0
  128   128   128
  255   255   255
 -255     1     0
 -256     0     0
  256     0   255
><- Serial1
`

Thanks for the awesome detail @allObjects. My data is MIDI data, so technically 7 bit, although the extended MIDI data is 8-bit. I switched between Uint8 and Uint8Clamped and my data doesn't look any different, so I think I'm safe. However, my 16-bit binary data on USART? Different animal altogether. Good ole' endianness.

The conversation starts out with

sending binary data over a TCP connection

and that is what guided my response.

That MIDI - at least the initial prevailing standard and not the newest draft - deals with 7 bit values only is actually great to be able to introduce meta data in the transport layer which can easily be filtered out: MSBit 0 means destined for the MIDI layer, MSBit 1 means control data for the transportation layers. I'm aware of that the stream of MIDI data is most likely time critical and therefore too much meta data in the stream may be a challenge... except the transport layer is so much faster that it still can sustain the 31.25kbps above it (for the MIDI endpoints - source and sink).

Your experience switching between both the plain and clamped unsigned 8 bit array is explained now explained why you have same results. From a point of speed the clamping may time-wise not be noticeable since it is done internally on firmware level and not on the (much slower) JavaScript level. Great is that all arrays have the same protocol...

Getting a reliable 'pipe' going may be a challenge, because you do not have much control over the packages. Adding that would have to be with a fast implementation and I'm thinking about compile JavaScript or even embedded C. That piece of fast code could insert frame information on the sending side and filtering out on the receiving side in order to give you control over the pipe... and drop failing packages without real big consequences since the transportation layer package boundaries can be made aligned - logically - to the payload package boundaries. With sockets you could even think of retransmitting packages that are not allowed to be dropped / skipped... or double / triple down on sending them... and on the receiving side to pass on only the one...

To be frank, I do not know what is or can be critical in a MIDI stream from point of data loss / corruption and tolerance thereof, so take my thoughts as general thoughts of getting data reliably from one end-point to another one.

Fri 2019.08.16

@allObjects nicely done and demonstrates a possible trap using negative numbers. This would make a nice Tutorial posting with a heading like 'Effects of negative numbers when using Typed Arrays' . . . .

@allObjects I agree with @Robin.

Getting a reliable 'pipe' going may be a challenge...

Most of the time, my code will be on the receiving end of MIDI data, and most of that will be a burst of NRPN messages after initial connection, and then mostly quiet. So far, so good on that front, but it's still early - plenty of time to screw things up ;-)

After checking out suggestions from @maze1980 and @Robin, I've created my TCP message handler, and it kinda works. I appear to be getting data corruption in the buffer that I copy the incoming data into, and I can't figure out what I'm doing wrong. The handler:

if(this.tcpBuffer && this.tcpBuffer.byteLength) {
    let leftoverDataLen = this.tcpBuffer.byteLength;
    let tmpBuffer = new Uint8Array(leftoverDataLen + chunk.length);
    tmpBuffer.set(this.tcpBuffer, 0);
    tmpBuffer.set(E.toUint8Array(chunk), leftoverDataLen);
    this.tcpBuffer = tmpBuffer;
}
else {
    this.tcpBuffer = E.toUint8Array(chunk);
}

So I either start off with a brand new copy of the incoming data (converted to Uint8Array), or I append the new data to the end of the current buffer. When done, this.tcpBuffer should contain all of the data that I've not yet processed. Am I doing this correctly / is there a better way to append the data?

This sounds good to me... I guess if you know how much data you're expecting maximum then you could just preallocate the buffer? That would be a bit faster and more reliable.

You can even use the ArrayBuffer view

this.tcpBuffer = new Uint8Array(1024);
this.tcpLen = 0;
...
/// in handler
        var c = E.toUint8Array(chunk);
        this.tcpBuffer.set(c, this.tcpLen);
        this.tcpLen += c.length;
/// and when you're sure you are done
var result = new Uint8Array(this.tcpBuffer.buffer,0,this.­tcpLen); // make result a 'view' of the right length
this.tcpLen=0; // reset

I'm getting a datastream from the audio mixer (MIDI data over TCP), so the data never ends. I suppose I could pre-allocate a 1kb buffer and use it as a FIFO, so long as the consumer code keeps pulling data out.

@Gordon, well I'm stumped. I am failing to receive all of the data from my Allen & Heath Qu-SB mixer via MIDI-over-TCP. In short, I am seeing truncated/missing data. I am sending a MIDI All-Call SysEx message, and I expect a big response to my request, over 24k bytes of data.

This MIDI protocol document (https://www.allen-heath.com/media/Qu_MID­I_Protocol_V1.9.pdf) describes what that response contains - it is the state of the mixer: every mixer channel's level, EQ, name, etc. Here is a small snippet:

0xb0 0x63 0x20 0xb0 0x62 0x17 0xb0 0x6 0x49 0xb0 0x26 0x7 
0xb0 0x63 0x20 0xb0 0x62 0x18 0xb0 0x6 0x1 0xb0 0x26 0x7 
0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x33 0xb0 0x26 0x0 
0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x53 0xb0 0x26 0x1 
0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 
0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x1 
0xb0 0x63 0x20 0xb0 0x62 0x20 0xb0 0x6 0x31 0xb0 0x26 0x2 
0xb0 0x63 0x20 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x2 
0xb0 0x63 0x20 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x2 

These are MIDI NRPN messages, where each one of the lines that starts with '0xb0 0x63' is a single 12-byte NRPN message, containing information about a mixer channel (channel number, channel level, etc).

I should get 64 (12-byte) messages for each channel, and there are 32 channels to the mixer. That alone is over 24kb of data.

Meanwhile, I will also receive other SysEx data along with the NRPN data, each 20 bytes long. An example is here:

0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x20 0x4e 0x6f 0x72 0x64 0x20 0x4c 0x0 0xf7 0xfe

I should receive one of these for each of the 32 channels, because it contains that channel's name (as a 7-byte ascii string). That's only 640 bytes, practically in the noise.

According to the A&H documentation, and after consulting with A&H support, these NRPN messages and SysEx messages can in fact be intermingled - meaning, you might not receive all of the NRPN messages before you start receiving the SysEx messages. However, they will never intermingle in the middle of a message each other. In other words, you will always receive a complete NRPN message, followed by a complete SysEx message, followed by a complete NRPN message, in the response. So you might see this:

0xb0 0x63 0x2a 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x2a 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x1 
0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x20 0x4e 0x6f 0x72 0x64 0x20 0x4c 0x0 0xf7 0xfe
0xb0 0x63 0x2a 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 

...but you should never see this:

0xb0 0x63 0x2b 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x0 
0xb0 0x63 0x2b 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x2b 0xb0 0x62 0x50 0xb0 0x6 0x1 0xb0 0x26 0x0 
0xb0 0x63 0x2b 0xb0 0x62 0x20 0xb0 0x6 0x0 0xb0 0x26 0x1 
0xb0 0x63 0x2b 0xb0 0x62 0x55 0xb0 0x6 0x1 0xb0 0x26 0x1 
0xb0 0x63 0x2b 
0xf0 0x0 0x0 0x1a 0x50 0x11 0x1 0x0 0x0 0x2 0x21 0x4e 0x6f 0x72 0x64 0x20 0x52 0x0 0xf7 0xfe 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x65 0xb0 0x6 0x57 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x66 0xb0 0x6 0x5b 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x67 0xb0 0x6 0x0 0xb0 0x26 0x0 
0xb0 0x63 0x37 0xb0 0x62 0x68 0xb0 0x6 0x0 0xb0 0x26 0x0 
0xb0 0x63 0x38 0xb0 0x62 0x61 0xb0 0x6 0x1 0xb0 0x26 0x0 

Unfortunately, I often see the latter. Note that the third byte of the start of the NRPN message (0xb0 0x63 0x2b) jumps from 0x2b (mixer channel 11) to 0x37 (mixer channel 23). And it jumps to nearly the end of the 64-message payload, as it quickly moves on to 0x38 (mixer channel 24).

So in that response, I lost some of mixer channel 11, all of mixer channels 12-22, and most of mixer channel 23.

This occurs randomly in the response, never in the exact same place each time I issue the All-Call.

To test to see if the problem is in my code, I make the TCP connection, send the MIDI All-Call, and then dump the responses to the console. I typically receive a few hundred bytes per TCPSocket on 'data' callback, and all I do is dump that data to console. When I do that, I see the data loss described above. I'm not copying to another buffer, or doing any processing on it whatsoever.

Note that I'm not seeing data corruption - all of the data I receive is in fact real MIDI data - it's just truncated data. And this truncated data is typically NOT occurring around a TCPSocket on 'data' boundary. In fact, it's usually in the middle of a received data packet.

I realize you don't have an A&H mixer to test, so I'm putting you in a real bind, but is there anything you can think of that could cause this kind of data loss?

Sat 2019.10.19

Hi @indianajones, as it has been many weeks since the initial post, would you mind refreshing us on board type, flash version, some sample array initialization code, interconnect schematic, etc. Thanks

What is powering the device(s)? Solid common ground between all hardware? Cable length? Timers tripping in background? Array buffers too small? Noise on the clock/data line?

re: 'all I do is dump that data to console. When I do that, I see the data loss described above.'

Could the delay in attempting to output the data, be blocking the stream of incoming data?
Is it possible to delay the output to ensure all data is received?

re: 'and I expect a big response to my request, over 24k bytes of data.'

That is a load of data!

Would you mind posting process.memory(); before attempting a transfer and then again immediately after.

Does MIDI use a device select, CS or DTR line, or is the data *just there* on the data line sent by any device? Possible data collisions interrupting the sending device?

EDIT:
Found these:

https://learn.sparkfun.com/tutorials/mid­i-tutorial/hardware--electronic-implemen­tation
https://www.midi.org/specifications-old/­category/gm-specifications
https://www.designacable.com/7-pin-din-m­idi-cable-foot-switch-to-amp-lead-bi-dir­ectional-phantom-power.html
https://www.instructables.com/id/Send-an­d-Receive-MIDI-with-Arduino/

Single Out >> In or Bi-directional? 5 pin or 7 pin?

@Robin,
Board Type: Espruino Wifi
Firmware version: 2v04
Espruino is USB-connected to my Mac, powered from USB
No other physical connections, as the MIDI-over-TCP communication is via WiFi
All buffers and buffer sizes are whatever Espruino created for a TCP Socket connection

Re: possible delays, I considered waiting to dump all data, but had out-of-memory issues. Probably due to coding errors by me. I can try again, now that I'm a little less ignorant.

Memory usage at start:

{"free":3961,"usage":3187,"total":7148,"­history":1731,"gc":422,"gctime":9.965896­60644,"stackEndAddress":536991644,"flash­_start":134217728,"flash_binary_end":460­040,"flash_code_start":134283264,"flash_­length":524288}

Memory usage when all data received:

{"free":3995,"usage":3153,"total":7148,"­history":1731,"gc":84,"gctime":9.5844268­7988,"stackEndAddress":536991644,"flash_­start":134217728,"flash_binary_end":4600­40,"flash_code_start":134283264,"flash_l­ength":524288}

Relevant code. Here is the TCP on 'connect' callback, which makes the All-Call request:

	tcpConnect(details) {
		this.net.connect({port: this.tcpPort, host: details.address}, function(tcpSocket) {
			this.tcpSocket = tcpSocket;
			this.connected = true;
			console.log("...connected!");

			this.tcpSocket.on('data', this.tcpOnMessage.bind(this));
			this.tcpSocket.on('close', this.tcpOnDisconnected.bind(this));

			// inform the client that we've connected
			this.onConnectedCallback();

			// inform the Qu that we want telemetry from the device
			this.sendSysExAllCall();

		}.bind(this));
	}

...and here is the on 'data' callback:

	tcpOnMessage(chunk) {
        // dumpArray() is just prepending the data with '0x' in front, etc
		console.log(this.utils.dumpArray(E.toUin­t8Array(chunk)));
	}

I tried to append the responses into a pre-allocated 32k buffer, and after receiving the entire response, and I still get out-of-memory errors when I try to console.log() it. So I changed it to merely dump the size of the data I received, and it showed 18,796, which is around 6kb short of expected - that's a data loss equivalent to 8 mixer channels. Curiously, multiple attempts at this seem to always return 18,796 bytes, every single time.

Thanks for the detail.

Is there an external memory module in use? Something doesn't appear right here.

ref: http://www.espruino.com/Internals

With a JsVar in memory equivalent to either 12 or 16 bytes, 3187-3153 = 34 * 16 = 544 bytes.

Conservative use with 1000 JsVars remaining, 2000 x 16 = 32ooo

Possible memory allocation issue?

Now I am fumbling for how to check (view) strings in memory. Researching . . .
Does dump() reveal anything helpful?

http://www.espruino.com/Performance
Heading 70% down page:
"ARRAY BUFFERS ARE THE MOST EFFICIENT WAY TO STORE DATA"

What does console.log( process.memory().free ); reveal inserted before L3 arrayDump() of last snippet in post#15?

Has the use of try/catch blocks around the array manipulation been tried to see if there are any un-handled errors during concatenation? Would you post some array declaration sections along with the concatenation code please.

try {
  console.log( process.memory().free );
  console.log(this.utils.dumpArray(E.toUin­t8Array(chunk)));
} catch( e ) {
  console.log( "Err at line xx " + e.message );
  console.log( process.memory().free );
}

Any way of looping through the output looking for 0xb0 and just console.log() that section, building the output 'one line at a time' ? Wondering if the console.log() call makes a duplicate in memory, thus running out as only ~1000 JsVars remain before dumpArray(), but need ~2048.

Example when uploading a code block, Espruino is running in memory and needs the full upload before parsing can complete, so double the memory is needed. Wondering if the use of console.log() on the entire 32K is requiring the same?

@Robin,

Thanks a million for your thoughtful responses...

process.memory().free is 1942 at the end of the receipt of data.

No external memory module in use, literally nothing physically attached to the Wifi module.

Re: possible memory allocation issue: in the "real" version of my app, where I'm actually processing the incoming data, I perform memory allocation, enough to hold whatever has arrived in TCPSocket on 'data', but has yet to be processed. You can see that code here. Because I suspected my own code at fault, I commented all that out, and am doing only the console.log() bit described above. So no memory allocation at all. Well, that's a lie - I'm doing 'E.toUint8Array(chunk)' in the console.log(). But still, that's allocating up to a few hundred bytes, not allocating an ever-growing buffer.

RE: looping through the data, yes, that's exactly what my real code does: the TCPSocket on 'data' callback calls handlers that process the data, and spit out a single NRPN (or SysEx) message at a time. And it works flawlessly, until it encounters a truncated NRPN message (as shown previously), then it stops being able to process. The point here is that the issue occurs in every context in which I am receiving the data. I'm not seeing buffer overflows, out-of-memory errors, etc. The behavior, excuse me, behaviour, of the incoming data stream that my processing code sees is identical to the "just dump the buffer" code.

I could modify dumpArray() to manage the large buffer, but I already know what I'm going to get: ~8kb of missing data. The 18,796 byte count (in a 32kb pre-allocated buffer) already proves that.

'I could modify dumpArray() to manage the large buffer, but I already know what I'm going to get'

Nothing glaringly obvious in the code snippet, no mathematical limit stands out in the 18,796 value. Don't have the same hardware in order to test on this end. Any chance to humor me (us) and reveal the amount of free memory before and after dumpArray() as in the try/catch above? Are any errors shown then?

Although I didn't spend a ton of time studying the source:
Clicking on the right facing arrow adjacent to the heading at:

http://www.espruino.com/Reference#l_E_to­Uint8Array
hyperslinks to the source:
https://github.com/espruino/Espruino/blo­b/master/src/jswrap_espruino.c#L826

Was wondering if there are constants there that might clue us in.

'I'm not seeing buffer overflows, out-of-memory errors, etc'

It's possible that errors are being thrown, bubbling up and not caught. I find that wrapping every code block within a function, sometimes reveals tid-bits, such as 'undefined' and 'null' which otherwise escape viewing.

EDIT:
Opps, looks like out post edits crossed.

'process.memory().free is 1942 at the end of the receipt of data'

Then does the memory error occur when attempting to reveal with console.log() at that point? What remains *should* be able to be viewed; 18796 / 16 = ~1175 which should leave enough ~600 for Espruino housekeeping.

'until it encounters a truncated NRPN message'

What specific bytes are at msg end? Maybe others have seen or know of whether that sequence has been problematic in the past?

Hey, I just noticed in post #13 that some bytes are not printing both numerical values to stay in the 0-255 0x00-0xff range. Could one of those 0x0 be detected as a null and blowing up memory at that point?

Yes, the memory error occurs on console.log() is in line 3 of that example, where I'm trying to dump the entire 18k buffer. But again, it kinda doesn't matter that I suck at being able to dump the data - I know it's short 6kb or so.

If I use your try/catch sample above, I get this (which is the same as what I get if don't have try/catch):

1934
ERROR: Ctrl-C while processing interval - removing it.
Execution Interrupted during event processing.
New interpreter error: CALLBACK,MEMORY

Big Clue: 'which is the same as what I get if don't have try/catch'

What is a Ctrl+C in hex and is that in the data stream?

Could it be possible that the alignment of those single hex values (off by four bits) get interpreted as the hex equivalent of Ctrl+C ?

EDIT:

https://www.windmill.co.uk/ascii-control­-codes.html

So, we are looking for 0x03 or 0b00000011 or 0000 0011 in that stream.

No, it's not in the data stream. The code snipped I copied from you is in a function that's called on a setInterval(), called long after I know I've received all the data. That error occurs on the console.log() of the attempt to dump the whole array.

I assume the CTRL-C was some interrupt or something going on in the microcontroller when it encounters a fatal memory error. I assume that because I've seen that exact message so many times while I was coding up my data processing code (which allocates memory). But none of that processing code is in use right now, only the attempt to dump data.

So I have to issue a correction, embarrassingly. The dumpArray() doesn't strictly allocate memory (as in new XXX()), but it creates a string that it appends the entire buffer to, and returns that string (to be printed to console). So there's absolutely another big buffer allocation going on when I try to dump the data.

My problem is that if I just loop through the array, and console.log() each byte, I get newlines after each byte:
0xb0
0x63
0x20
...and so on. It becomes unreadable. Any ideas on how to print to the console without the newline?

'Any ideas on how to print to the console without the newline?'

Not off hand. I've usually built new strings just concatenating each byte, then console.log() say 32 bytes at a time generating a table. I've done some code stripping the leading '0x' let me see if I can locate that snippet. . . .

EDIT: (truncate the leading '0x')

https://www.w3schools.com/jsref/jsref_pa­rseint.asp
https://www.w3schools.com/jsref/tryit.as­p?filename=tryjsref_parseint
var byte = 0x??;
parseInt( byte, 16 );

build a columnar table of hex values

  for( var j=0; j<bytes; j++ ) {

    var decbyte = sread[j];
    //var schar = String.fromCharCode( decbyte );
    var soutcvrt = parseInt( decbyte, 16 );
    soutequiv+= " ";
    soutequiv+= ( j == nNumSepHexBytes ) ? " " : "";    // Separate each eight byte block
    soutequiv+= soutcvrt;

//  08 52 ff ff ff ff ff ff   31 32 33 61 00 ff ff ff

substitute a period marker for known value

soutequiv+= "|";
  for( var k=0; k<bytes; k++ ) {

    var decbyte = sread[k];
    if( decbyte == 255 ) decbyte = 46;  // . Period marker
    var schar = ( decbyte > dASCIICtrlCharEnd ) ? String.fromCharCode( decbyte ) : " ";

    if( bDebugPrint ) {
      if( decbyte == 0 ) {
        console.log( "[L242] schar [ " + schar + " ]" );
      }
      if( sChar == "00" ) {
        console.log( "[L242] schar [ " + schar + " ]" );
      }
    }

    soutequiv+= ( k == nNumSepHexBytes ) ? " " : "";    // Separate each eight byte block
    soutequiv+= schar;
  }
soutequiv+= "|";

    if( bDebugPrint ) {

    console.log( "[L120] str read " + sread );
    console.log( "[L121] " + sread.toString() );

    // https://www.w3schools.com/jsref/jsref_fr­omcharcode.asp
    // var str = String.fromCharCode(sread);

    // https://stackoverflow.com/questions/4014­2324/how-to-convert-an-array-of-characte­r-codes-to-string-in-javascript

    // https://developer.mozilla.org/en-US/docs­/Web/JavaScript/Reference/Global_Objects­/Function/apply

    var str = String.fromCharCode.apply(null,sread);
    console.log( "[L126] " + str );
    }

end EDIT

Here is a similar issue (Ctrl+C error) and explanation with setInterval():

http://forum.espruino.com/conversations/­321778/

In the end, I don't see the value in trying to view the entire buffer in one shot, at the end, when the buffer count already tells me I have 18k instead of 24k bytes of data received. In fact, that proves that dumping each buffer to the console as I receive it not causing any problems. If it were, I would have accumulated more data by not printing the data vs. printing the data. However, I get the same amount of data in either scenario.

I've stripped the code down to its bare minimum (where 'bare minimum' is defined as the code required to do a wifi/tcp connection and nothing else). No difference.

It's possible/likely that I need to set/change some configuration field that I'm unaware of, but other than that, I no longer think the problem is with my code. So unless there's a miracle, I'll have to think of something else with which to use these cool devices.

Mon 2019.10.21

Had a flash thought in passing, could there be a hardware watchdog that is tripping? Researching. . .

See this .pdf file in link:

http://forum.espruino.com/comments/14791­865/

Wow, odd that data is missing. It's not usually an issue as Espruino WiFi can receive quite big things (like images) over HTTP without issues.

If you run E.getErrorFlags() does it say anything?

FIFO_FULL would indicate that data has come in faster than Espruino could handle and it had to drop something, so that would be the first port of call.

You could also try adding the turbo mode (wifi.turbo(true,callback) as part of your wifi initialising code) - that'll hugely increase the baud rate from the ESP8266, which might help if the ESP8266 itself had been losing data (it shouldn't though);

Please could you also post up the complete code you're using to read data into the buffer? I'm not sure I'm seeing your tcpOnMessage that does that?

And yeah, it looks to me like the out of memory when dumping is because dumpArray is just turning the array into one big string. If you could make dumpArray call console.log itself with each line of text as it was ready, that would probably help you out a bunch - but if you're not getting the amount of data you expect then it's not worth caring about yet.

I'm not sure I can help much more unless I can reproduce it, so if you don't have any luck with the above, maybe you could:

• Cut your code down to a completely minimal example which you can post up here
  
• Try running that on Node.js on the desktop and see if it works (the code should be pretty much standard JS so it should be fine)
  
• Post up a dump of the complete data you get from Node.js - then I should be able to 'fake' the controller here by making a server which pushes all the data out, and try it with an Espruino WiFi running your code.