How to properly handle incoming TCP/UDP data

Posted on
  • I have a connection to a device that's sending binary data over a TCP connection, and of course I get the data in chunks, which must be concatenated as new data comes in. Because the data I receive is binary, I have to convert the incoming data using E.toUint8Array, which works great. But when new data comes in, I have to add the new data to the existing Uint8Array, and I'm struggling to find the right way to do this. Anyone have experience with this?

  • No, but .set seems to fit:

    a = new Uint8Array([1,2,3,4,5,6,7,8,9,10]);
    b = new Uint8Array(7); // 0,0,0,0,0,0,0
    c = new Uint8Array(a.buffer, 2, 5); // [3,4,5,6,7]
    b.set(c, 2); // set b with the contents of c starting from index 2
    b; // [0,0,3,4,5,6,7]


  • Wed 2019.08.14

    Have you considered for memory efficiency @indianajones using a clamped array?­lampedArray

    In #2 @maze1980 points out the set() function which is the way I do this task. Access can even be sped up with 'inlineC', but really won't be needed for your project needs. Grasping the hierarchy took me a bit, as the above is inherited from ArrayBufferView. Uint8ClampedArray is a type of TypedArray from which the set method is defined.

    Also check out heading 'STRINGS VS. FLAT STRINGS' from link in #2 above.

    For your 'light' reading pleasure ;-)­/Web/JavaScript/Reference/Global_Objects­/Uint8ClampedArray#Methods­/Web/JavaScript/Reference/Global_Objects­/TypedArray/set­/Web/JavaScript/Reference/Global_Objects­/ArrayBuffer­/Web/JavaScript/Reference/Global_Objects­/DataView

  • @maze1980 and @Robin, thanks again for your suggestions and links. I had been using Uint8Array, or should I say mis-using it. After reading the links and trying more iterations, I settled on using Uint8ClampedArray, oh and using it correctly also helps. It's working very well now, thanks!

  • Watch out when you use Uint8Array and Uint8ClampedArray.

    First of all, regarding memory efficiency / consummation, they are absolutely the same.

    Second, they have the same protocol (state and behavioral properties / protocol).

    Third, they construct all from a plain array or from an array view. Former looks at the values of each source element where latter looks at the bytes the the source array is stored.

    Where they differ is handling the values of the source array when constructed, as code and output below show. A plain array with integers of different values is passed in the constructors of an Uint8Array and an Uint8ClampedArray:

    • Uint8Array just takes the least significant 8 bits
    • Uint8ClampedArray looks at the value and applies min/max function to be 0 or 255.

    If you need to transmit just 8 bits and the source elements are values up to 8 bits unsigned or 8-bit characters or 8-bit chars subset of 16-bit (or more) char set, you are just fine - you can think of 0..255 range - 256 different values.

    If they are signed integer numbers, you can think of only 0..127 range - still 256 different values - but you have to process them differently when interpreting the bytes back to numbers: you have to check MSB - most significant bit - and then do some math or bit manipulation for getting back the original value (in the 8+ bit format).

    // ui8_vs_ui8Clamped_arrays.js
    // 2019-08-15 - allObjects
    var arr
      , ui8Arr
      , uiCArr
    function fmtI(n,l) {
      var s = "            "
            + ((typeof n == "number")
                ? n
                : (""+n).substr(0,l)
      return s.substr(s.length - (l || 12));
    function onInit() {
    arr =
        [    0
        ,    1
        ,   -1
        ,  127
        , -127
        , -128
        ,  128
        ,  255
        , -255
        , -256
        ,  256
    ui8Arr = new Uint8Array(arr);
    uiCArr = new Uint8ClampedArray(arr);
    fmtLen = 5;
        fmtI(" src"       ,fmtLen)
      , fmtI(" ui8"       ,fmtLen)
      , fmtI(" uiClampled",fmtLen)
    arr.forEach(function(e,i,a) {
          fmtI(arr[i]   ,fmtLen)
        , fmtI(ui8Arr[i],fmtLen)
        , fmtI(uiCArr[i],fmtLen)
    } );

    And respective output ('packed' into a multiline string for legible color in forum css):

     ____                 _
    |  __|___ ___ ___ _ _|_|___ ___
    |  __|_ -| . |  _| | | |   | . |
    |____|___|  _|_| |___|_|_|_|___|
     2v04 (c) 2019 G.Williams
      src   ui8  uiCl
        0     0     0
        1     1     1
       -1   255     0
      127   127   127
     -127   129     0
     -128   128     0
      128   128   128
      255   255   255
     -255     1     0
     -256     0     0
      256     0   255
    ><- Serial1
  • Thanks for the awesome detail @allObjects. My data is MIDI data, so technically 7 bit, although the extended MIDI data is 8-bit. I switched between Uint8 and Uint8Clamped and my data doesn't look any different, so I think I'm safe. However, my 16-bit binary data on USART? Different animal altogether. Good ole' endianness.

  • The conversation starts out with

    sending binary data over a TCP connection

    and that is what guided my response.

    That MIDI - at least the initial prevailing standard and not the newest draft - deals with 7 bit values only is actually great to be able to introduce meta data in the transport layer which can easily be filtered out: MSBit 0 means destined for the MIDI layer, MSBit 1 means control data for the transportation layers. I'm aware of that the stream of MIDI data is most likely time critical and therefore too much meta data in the stream may be a challenge... except the transport layer is so much faster that it still can sustain the 31.25kbps above it (for the MIDI endpoints - source and sink).

    Your experience switching between both the plain and clamped unsigned 8 bit array is explained now explained why you have same results. From a point of speed the clamping may time-wise not be noticeable since it is done internally on firmware level and not on the (much slower) JavaScript level. Great is that all arrays have the same protocol...

    Getting a reliable 'pipe' going may be a challenge, because you do not have much control over the packages. Adding that would have to be with a fast implementation and I'm thinking about compile JavaScript or even embedded C. That piece of fast code could insert frame information on the sending side and filtering out on the receiving side in order to give you control over the pipe... and drop failing packages without real big consequences since the transportation layer package boundaries can be made aligned - logically - to the payload package boundaries. With sockets you could even think of retransmitting packages that are not allowed to be dropped / skipped... or double / triple down on sending them... and on the receiving side to pass on only the one...

    To be frank, I do not know what is or can be critical in a MIDI stream from point of data loss / corruption and tolerance thereof, so take my thoughts as general thoughts of getting data reliably from one end-point to another one.

  • Fri 2019.08.16

    @allObjects nicely done and demonstrates a possible trap using negative numbers. This would make a nice Tutorial posting with a heading like 'Effects of negative numbers when using Typed Arrays' . . . .

  • @allObjects I agree with @Robin.

    Getting a reliable 'pipe' going may be a challenge...

    Most of the time, my code will be on the receiving end of MIDI data, and most of that will be a burst of NRPN messages after initial connection, and then mostly quiet. So far, so good on that front, but it's still early - plenty of time to screw things up ;-)

  • After checking out suggestions from @maze1980 and @Robin, I've created my TCP message handler, and it kinda works. I appear to be getting data corruption in the buffer that I copy the incoming data into, and I can't figure out what I'm doing wrong. The handler:

    if(this.tcpBuffer && this.tcpBuffer.byteLength) {
        let leftoverDataLen = this.tcpBuffer.byteLength;
        let tmpBuffer = new Uint8Array(leftoverDataLen + chunk.length);
        tmpBuffer.set(this.tcpBuffer, 0);
        tmpBuffer.set(E.toUint8Array(chunk), leftoverDataLen);
        this.tcpBuffer = tmpBuffer;
    else {
        this.tcpBuffer = E.toUint8Array(chunk);

    So I either start off with a brand new copy of the incoming data (converted to Uint8Array), or I append the new data to the end of the current buffer. When done, this.tcpBuffer should contain all of the data that I've not yet processed. Am I doing this correctly / is there a better way to append the data?

  • This sounds good to me... I guess if you know how much data you're expecting maximum then you could just preallocate the buffer? That would be a bit faster and more reliable.

    You can even use the ArrayBuffer view

    this.tcpBuffer = new Uint8Array(1024);
    this.tcpLen = 0;
    /// in handler
            var c = E.toUint8Array(chunk);
            this.tcpBuffer.set(c, this.tcpLen);
            this.tcpLen += c.length;
    /// and when you're sure you are done
    var result = new Uint8Array(this.tcpBuffer.buffer,0,this.­tcpLen); // make result a 'view' of the right length
    this.tcpLen=0; // reset
  • I'm getting a datastream from the audio mixer (MIDI data over TCP), so the data never ends. I suppose I could pre-allocate a 1kb buffer and use it as a FIFO, so long as the consumer code keeps pulling data out.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview

How to properly handle incoming TCP/UDP data

Posted by Avatar for indianajones @indianajones