NewBee - Espruino upload/download error checking?

When using the WEB IDE, is there any checksum/crc error checking when uploading or downloading an user application to the Espruino? What about firmware updates?

From my observation, due to my hardware configuration and many software modules dependent on each other, there is no data checking "down the USB tube". On many occasions, I have to reload or upload again just to make the hardware work.

Firmware updates seem to be very reliable.

The same is not the case for sending code. I've had the same experience - stuff routinely gets lost, seemingly at random. You can throttle sending in settings - but then it's painfully slow.

Yes, firmware has a CRC, but it never fails because I believe USB has error checking built in.

In terms of sending code, I've never heard of a corruption problem, but occasionally characters get missed out. That's usually due to modules - it takes Espruino some time to load each module in, but the Web IDE doesn't wait for it as there is very little in the way of flow control.

That's something that I have plans to sort out in the IDE - but ideally we could add flow control in the USB layer. I'm just not quite sure how to do that!

Translation ... No data validation on source code uploading. When you send a data stream to another device is has to be checked for data integrity. You cannot depend on "H&P" for data validation. (Hope and praying)

Yes, firmware has a CRC, but it never fails because I believe USB has
error checking built in.

The other day, I had to reload the firmware "again" because the "process.memory()" command did not work.

It would be nice to have a 100% guarantee that the data/firmware being sent to the Espruino is correct and validated.

Translation ... No data validation on source code uploading.

Wrong.

When you send a data stream to another device is has to be checked for data integrity

Yes, but that's what USB does. Like with TCP/IP you can be confident that you have a non-corrupt stream of data, because lower layers sort it out at the packet level.

As I said, the problem is not that there is no validation - it is that the flow control that I'm getting with USB isn't good enough right now (it can only delay data, not stop it completely). The data is accurate and is coming to Espruino, but it is just coming too fast in a few cases.

You can come up with particularly nasty bits of code that will always fail to upload without this flow control. What's needed isn't a sticking plaster over that, it's a fix for the underlying problem.

I had to reload the firmware "again" because the "process.memory()" command did not work.

And everything else worked fine? If so, it's unlikely to be due to a bad firmware update. I'd be interested to hear how it failed though.

I see the missing process.memory() sometimes. Reset() then resending code usually fixes it. It seems to happen a lot when upgrading firmware on bigram builds. I think is a problem loading the stored program (which isn't automatically erased when flashing bigram - only the first 256k, the flash the chip is speced as having, gets erased.

I think is a problem loading the stored program

Ahh - that's almost certainly it. Same problem you have on other boards if you don't do a full erase when you flash.

Espruino figures out about built-in classes by storing a pointer to the constructor function - unfortunately every time you use a new firmware it's pretty likely that the pointer changes, so it will forget that 'process' included the function! Trying delete process and the trying again might fix it.

It's probably just as well that something so obvious breaks - there are loads of other more subtle things that could break as well!

Translation ... No data validation on source code uploading. Wrong.

I stand corrected. It would still be nice to run a packet communication test over a weekend, on the Espruino, using a communication analyzer to see what is the error rate - if any. Of course, I don't have the means to purchase or to even rent this test equipment. Maybe get a loaner from Oxford?

The problem is that usually there won't be errors, but you can write code that always causes problems. For instance:

for (var i=0;i<10000;i++);
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";
"LotsOfDataLotsOfDataLotsOfDataLotsOfDat­aLotsOfDataLotsOfDataLotsOfData";

So in that case, you're writing code that is executed immediately, so it stalls Espruino. Unfortunately the strings of text keep getting sent to it and it just can't handle them.

Even with error checking, it won't fix the problem because it will always lose data.

If you're doing something like that then you can easily fix it by doing:

function doStuff() {
 // stuff that takes a while
}
....

// at end of file
setTimeout(doStuff,10);

Often the issue is that instead of user code causing problems, you're actually waiting for modules to load.

I actually added software flow control at the end of last week, but it turns out Chrome doesn't support it (it only supports hardware flow control) which makes life a bit difficult. Actual hardware flow control would be great, but ST's USB code for the chip has some bugs which seem to make it non-trivial.