How to properly handle incoming TCP/UDP data

You are reading a single comment by @indianajones and its replies. Click here to read the full conversation.

•

@Robin,
Board Type: Espruino Wifi
Firmware version: 2v04
Espruino is USB-connected to my Mac, powered from USB
No other physical connections, as the MIDI-over-TCP communication is via WiFi
All buffers and buffer sizes are whatever Espruino created for a TCP Socket connection

Re: possible delays, I considered waiting to dump all data, but had out-of-memory issues. Probably due to coding errors by me. I can try again, now that I'm a little less ignorant.

Memory usage at start:

{"free":3961,"usage":3187,"total":7148,"history":1731,"gc":422,"gctime":9.96589660644,"stackEndAddress":536991644,"flash_start":134217728,"flash_binary_end":460040,"flash_code_start":134283264,"flash_length":524288}

Memory usage when all data received:

{"free":3995,"usage":3153,"total":7148,"history":1731,"gc":84,"gctime":9.58442687988,"stackEndAddress":536991644,"flash_start":134217728,"flash_binary_end":460040,"flash_code_start":134283264,"flash_length":524288}

Relevant code. Here is the TCP on 'connect' callback, which makes the All-Call request:

	tcpConnect(details) {
		this.net.connect({port: this.tcpPort, host: details.address}, function(tcpSocket) {
			this.tcpSocket = tcpSocket;
			this.connected = true;
			console.log("...connected!");

			this.tcpSocket.on('data', this.tcpOnMessage.bind(this));
			this.tcpSocket.on('close', this.tcpOnDisconnected.bind(this));

			// inform the client that we've connected
			this.onConnectedCallback();

			// inform the Qu that we want telemetry from the device
			this.sendSysExAllCall();

		}.bind(this));
	}

...and here is the on 'data' callback:

	tcpOnMessage(chunk) {
        // dumpArray() is just prepending the data with '0x' in front, etc
		console.log(this.utils.dumpArray(E.toUint8Array(chunk)));
	}

I tried to append the responses into a pre-allocated 32k buffer, and after receiving the entire response, and I still get out-of-memory errors when I try to console.log() it. So I changed it to merely dump the size of the data I received, and it showed 18,796, which is around 6kb short of expected - that's a data loss equivalent to 8 mixer channels. Curiously, multiple attempts at this seem to always return 18,796 bytes, every single time.

•

Robin in reply to @indianajones
Thanks for the detail.

Is there an external memory module in use? Something doesn't appear right here.

ref: http://www.espruino.com/Internals

With a JsVar in memory equivalent to either 12 or 16 bytes, 3187-3153 = 34 * 16 = 544 bytes.

Conservative use with 1000 JsVars remaining, 2000 x 16 = 32ooo

Possible memory allocation issue?

Now I am fumbling for how to check (view) strings in memory. Researching . . .
Does dump() reveal anything helpful?

http://www.espruino.com/Performance
Heading 70% down page:
"ARRAY BUFFERS ARE THE MOST EFFICIENT WAY TO STORE DATA"

What does console.log( process.memory().free ); reveal inserted before L3 arrayDump() of last snippet in post#15?

Has the use of try/catch blocks around the array manipulation been tried to see if there are any un-handled errors during concatenation? Would you post some array declaration sections along with the concatenation code please.
```
try {
  console.log( process.memory().free );
  console.log(this.utils.dumpArray(E.toUint8Array(chunk)));
} catch( e ) {
  console.log( "Err at line xx " + e.message );
  console.log( process.memory().free );
}
```
Any way of looping through the output looking for 0xb0 and just console.log() that section, building the output 'one line at a time' ? Wondering if the console.log() call makes a duplicate in memory, thus running out as only ~1000 JsVars remain before dumpArray(), but need ~2048.

Example when uploading a code block, Espruino is running in memory and needs the full upload before parsing can complete, so double the memory is needed. Wondering if the use of console.log() on the entire 32K is requiring the same?

How to properly handle incoming TCP/UDP data

About

Espruino