Search results for:

I see that you've added heatshrink compression to Espruino and that the Flash module is setup to use it, but I'm not sure if that is active at the moment as I don't see the macro USE_HEATSHRINK being set anywhere.

Is it possible, and would it make sense, to have heatshrink jswrap-ed so that we could use it when playing with strings and typed arrays in javaScript?

Is it worth considering hardware flow control on the usart with the ESP8266?

Funny you should mention using SPI flash, I'm in the process of investigating that as an option to buffer the incoming multipart/form-data and then parse that, saving the file parts to an SD card.

I've tried software SPI and the result is much the same. Its slower in in the first instance for each write, not as bad as half the speed of the best hardware SPI rate I can get. However it suffers delays of the same length (~0.2 sec) as in the log above with similar frequency and these delays do block the serial interrupts the same way as for hardware SPI.

Thanks @Gordon, I'll try it again with the next build you post up and see if the problems are gone.

I agree with you on the buffer size, and have now observed the same thing myself.
I'll have a look at the status info and data structures you've mentioned.

I have been doing my own build's in a VM using your guide. One extra step was required from memory because recent releases of Ubuntu omit a package now for VM installs which is where the USB serial device mapping magic happens, I think the package is linux-image-extra-virtual that has to be installed or the USB serial device never shows up as /dev/tty.... So I can continue to play with buffer sizes that way if I need to.

I will look at how node behaves at res.end() and let you know.

Its a shame about the SD file writing occasionally blocking the incoming serial coms. I'd really hoped to come up with a working solution.

Thanks for your interest @allObjects.
What I take from your post is that you have particular use-cases that you choose an MC for that you can make work. While I am exploring a different use-case and trying to make it work for me.

The results so far have been very fruitful and the journey has enabled me to make a number of contributions to the Espruino project in the form of bug fixes, functional improvements and some enhancements that have been readily accepted and implemented. Making Espruino better for everyone, hopefully.

So forgive me if I feel that you are suggesting I am trying to do something for which this MC isn't intended, because I disagree. Once I find its limits then I will know what use-cases I will choose this MC for, and for what use-cases I will need to choose something else.

Such a shame that it blocks the serial interrupt during these long 'phases' causing incoming serial data to be lost.
Its really proving to be a show stopper for dependable file uploads via http with the ESP8266

I've had the occasional case where I've managed to transfer a file of 100K without fault at 57600 baud from the ESP8266, but more often than not I will experience dropped data during a 10K file upload at 19200 baud. Its fairly unpredictable.

OK, I've tried both hardware and software SPI and end up with the same problem: I've opened a new post to follow that issue separately, although I believe it is now the root cause of my problems with dropping incoming packets of serial data from the ESP8266.

Meanwhile I'll let you know that the last binary/build that I can use is e54a6b982545da111b2f24d725fb30b0279c11da , anything after that has a regression error when trying to require a module

Uncaught SyntaxError: Got ?[255] expected EOF
 at line 1 col 257
...t(),h.cmd(a[0],a[1],a[2]))}ÿsco
                              ^
in function "cmd" called from line 2 col 79
...er AT+RST");else return cb})
                              ^
in function "reset" called from line 1 col 152
...ncs.reset(connectedCallback);return wifiFuncs
                              ^
in function "connect" called from line 24 col 2
});
 ^

you'll notice that \xFF character on line 3 that isn't there, and I don't get this error with the build linked just above.

I like where its heading with detecting the end of the body by comparing the bytes received with the Content-Length header. This is what I've been doing in my javaScript in the req.on('data',... for a while now. It looks like the implementation is still unfinished, and it currently also logs the message Closing now to the console from socketserver.c. And req.pipe(...) still doesn't end as yet.

I'd like to propose a few things at this point:

1. can the (new) chunksize parameter be user-definable if using NewtworkJS type 'drivers'. The place this could go is in the 'driver' module, in my case ESP8266WIFI_0v25 by including a chunksize attribute in the object passed to require("NetworkJS").create(obj), that if present, overrides the default value of 536 specified in network_js.c. As I have shown on line 2 here:

var netCallbacks = {
  chunkSize: 1460,
  create : function(host, port) {
    /* Create a socket and return its index, host is a string, port is an integer.
    If host isn't defined, create a server socket */  
    if (host===undefined) {
.
.
.
  require("NetworkJS").create(netCallbacks);

I would try and add it myself but I still haven't got my head around handling jsVars in the Espruino source code. This would remove the trouble of having to make a new build to try out different socket buffer sizes.

2. I propose it is a good opportunity to introduce the req.on('end',... event into Espruino. This is how node works with its ReadableStream implementation. Checkout https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/#request-body for some background.

• an http transaction works on a 'half-closed' basis between client and server
  
• the client (half-)closes the transaction after transmitting the request body with a tcp FIN packet
  
• of course devices like the ESP8266 don't make this visible to us, so
  
• it is typical to count the incoming bytes and compare with the Content-Length header and req.emit('end') when everything has arrived
  
• a timeout is probably also required (reset whenever data does arrive) in case the client stops sending, or some of the data never arrives - if there is a timeout, it should req.emit('error')
  
• the server only closes the transaction when it issues a res.end() or the socket is taken down for another reason (its been 'half-closed' until now)
  
• when the client gets the result of the res.end() the http transaction is complete
  
• if the client never gets a complete response (because the socket goes down for another reason) its up to the client to decide how to deal with this
  
  

The upshot of this approach is that the http server does not and should not have to issue a res.end() just to get a close (or end) event and then determine if the the whole of the request body has been received: an end event should happen as soon as the byte count is right.

I've noticed a lot of the sample/tutorial code for an http server does this 'early' res.end() to close the transaction/socket, which is fine if you are only expecting an http request header anyway, but not if you are expecting to receive/stream in an http request body. In fact when the res.end() is done you'd probably see a req.on('close',... event fire but I'll have to check this out in node to be sure.

Gosh this post got long but I really hope to make this work on the Espruino so I can complete my larger project, which by the way is an attempt to implement a light-weight, modular express (V4) like http router for the Espruino to simplify building http server apps along with my first http server app which would be a web based UI to manage content on an attached SD storage card, including drag-and-drop file uploads - so you can see why reliably receiving long http request bodies is important to me. I've got everything else working quite well, its just receiving long request bodies that is still causing problems.

Using the attached test script, I've been investigating the maximum write speed I can get out of my 2GB SD micro card.

Out of the four SD micro cards I have to hand, I can only get one to work. Also note that the write speed of every card is likely to be different.
I find that writes are most efficient when I write a block of 2048 bytes. The data I am writing is just 'A'-'Z' repeated to fill the buffer. What I write doesn't appear to make any difference.

The script allows you to choose software SPI or hardware SPI. If you you'd like to run the test on your Espruino and post the average speed you get here, that would be interesting for all to see.

With my working SD micro card the best speed I can get is with hardware SPI, where I can an average write speed of ~380 kbps (with an SPI baudrate of 2,000,000 being the highest setting that makes a difference). If I switch to software SPI it drops to ~230 kbps

The 'problem' I am having, that is affecting my application, is a series of unexpected delays in the File.write() where is takes 4-5 times the average write time at random intervals during a 200K file write in 2K blocks. I've attached an example log output from running the script. The 'long' writes are highlighted by a >>>>>>>>>>>>>>>>>>>>>>>>>>. These happen using either hardware or software SPI, so I wonder if it is the SD card itself taking longer to respond or if it is an problem with the SPI module?

Any insights here would be good, and solutions would be better.

P.S. all my testing is on an Espruino Pico

OK, I've been doing a lot of testing in the meantime and found:

• it all works nicely if you don't do anything that blocks in req.on('data',...
  
  • so counting bytes; buffering into a string; etc... all works well
    
• things you'd like to do like file i/o don't work so well, causing incoming serial packets from the ESP8266 to get dropped
  
  • creating a file with E.openFile() takes a long time and blocks
    
  • file write with File.write() can cause packet loss, but if you rebuffer into 2048 blocks its quite good, File.pipe(...,{chunkSize:2048}) can do this nicely but...
    
  • File.pipe() never ends because of the issue you've now addressed, so I will test now with this
    
    

If I drop the baudrate of the ESP8266 usart down to 19200, I can make things work fairly reliably. But this speed isn't ideal for more than a proof-of-concept.

The other thing I see happen (sometimes) is long (2 seconds plus) pauses that I assume are garbage collection. Sometimes the streaming of data survives this, sometimes it doesn't and again incoming serial packets from the ESP8266 to get dropped

Is there a way to request the GC to pause and unpause upon request?

My write speed to my SD card is effectively ~650000 bps so the file writing shouldn't be an issue for incoming data at 115200 bps, but something is blocking the serial interrupt and a few packets always get dropped at speeds over 19200 i.e. 38400 or higher.

I'll produce another log for you (attached), or you can try it for yourself: if you drop the baudrate in the test down to 9600, you'll see you get more recv events before all the serial data has been buffered by ipdHandler and the deltatime between each one is ~500ms. Once the idpHandler is finished then the deltatime between each recv event is down to ~14ms

OK I see your confusion: with the instrumentation in ipdHandler, line 16 (above) is commented out ///* so you don't see the packets reported where it is known there is more data to come i.e. ipdHandler (more) as opposed to ipdHandler (empty). This was just to avoid flooding the logging. If you want to see everything as it arrives just remove the first // on line 16 (above) and you'll get the lot.

So if I understand you right: socketserver.c would not use CHUNK to define a fixed buffer length for incoming data but just take whatever netRecv gets from netCallbacks.recv with no maxlen specified, as it will always return everything in sockData[sckt].

If I have this right, then sockData[sckt] will still be buffering around 6-8K (in your log above: 7063 bytes on the first recv) as recv only gets to run every ~500ms while serial data is arriving. Is there a way to make this more frequent, say once every ~100ms to keep the buffering down? Even a 6-8K buffer will stretch available memory.

P.S. the deltaTime in the logging is only meaningful on the second call to recv onwards.

My other crazy plan, that I will ask a few question of @tve to qualify if it is worth the effort, is to to run Espruino on the ESP8266 which implements the SDK's flow control for TCP sockets espconn_recv_hold and espconn_recv_unhold. Then write some javaScript to run on the ESP8266 paired with a module on the Pico that effectively proxies the ESP8266 Wifi module across the serial connection between the two devices, along with a kind of proxy for the netCallbacks on the Pico side paired with the net and sockets modules on the ESP8266 side. Waddayoureckon? Crazy huh! If this can be made to work, we get to use the memory on the ESP8266 for some of the buffering and we get TCP flow control.

Hi, I've spent many hours now setting up the tests and investigating different settings. Note: this code doesn't do anything with the incoming data, just counts the bytes.

The good news is that with 'well matched' settings the pipe/stream will flow all the way from the ipdHandler to the req.on('data',... in your own code. More on what 'well matched' means later.
The bad news is that under default settings (i.e. 115200 baud usart to ESP8266 and a 64 byte CHUNK size in socketserver.c) the netCallbacks.recv never gets the chance to get going: the ipdHandler (or more likely the ser.on("data",... in AT.js) dominates and buffers all the incoming serial data from the ESP8266 into sockData[sckt] before the netCallbacks.recv gets going. The memory pressure doubles (more or less) on the following calls to netCallbacks.recv as it tries to remove a 64 byte chunk from the beginning of sockData[sckt]

    if (sockData[sckt]) {
      /** INSTRUMENTED =>**/ var len = sockData[sckt].length;
      var r;
      if (sockData[sckt].length > maxLen) {
        r = sockData[sckt].substr(0,maxLen);
        sockData[sckt] = sockData[sckt].substr(maxLen);
      } else {

that second .substr(...) is the culprit as it makes a copy of the whole buffer (less 64 (maxlen) bytes).
On a pico running only the attached code, the maximum http message body (file upload) is just a bit less than 20K before an out-of-memory error occurs. I've attached a log generated from my instrumented copy of ESP8266WIFI_0v25 (also attached) where you can see that netCallbacks.recv only gets invoked once or twice by the time ipdHandler has buffered the whole http message.

If you play around with dropping the baud rate down to say 9600 you can see that while the serial data is arriving, netCallbacks.recv gets invoked only every ~500ms.

So what are some 'well matched' settings?
Well I'd been playing around with compiling my own build of Espruino just to play around with the CHUNK size in socketserver.c.
So as an example: a CHUNK size of 1460 matched with a baudrate of 19200 gets the stream flowing nicely (actually I can bring the baudrate upto 24000 with this chunk size) and the maximum size of the sockData[sckt] never gets much larger than the CHUNK size. This isn't exactly a fantastic baudrate, but I can stream in a file of any size (tested upto 300K, but saw that the buffer always stayed small), although I'm just counting the bytes and nothing more 'busy'.

So what next?

• flow control isn't an option as the ESP8266 only does hardware flow control with AT firmware, and the USART2 on the Pico doesn't support hardware flow control
  
• using a bigger CHUNK size and a low baudrate might be an option, memory permitting for a given project
  
• rewriting the ESP8266WIFI_0v25 module in native code in Espruino for better memory efficiency (one and only one copy of the data until it is consumed?) and optimising the balance between receiving serial data and scheduling javaScript execution
  
• if this isn't an option and we have to acknowledge the maximum buffer size that can be achieved with sockData then the memory pressure can be managed by using an index into the buffer for the start of the next chunk, and only removing delivered chunks when the buffer is at least half consumed(?)
  
  

Ok, I'll post my code tomorrow when I get back to my computer. I'm not doing anything time consuming, I'm actually just buffering the data again for now until I solve this problem and can then work more like a pipe/stream.

I take your last point and am inclined to agree.

What I was expecting to see was each of the three bullet-pointed steps happen mostly sequentially for each incoming packet of data, so overall it's like a stream or pipe with only a little buffering to keep things moving.
However what I am observing is that the 3rd bullet-point doesn't start until all the many, available incoming data packets have been buffered in points 1&2. Then when the socket server does start it results in the out of memory condition as the larger part of the buffer then get copied because we're chunking up the buffer in JavaScript (netcallback.recv) and not C/C++ (socketServer.c)

I'll also send the output from my instrumentation so you can see this happening.

Thanks for all the thought provoking input. I'll try to summarise what I've learnt in the past 24 hours.

• @Gordon pointed me in the right direction to the ESP8266WIFI_0v25 module. I'd failed to remember that modules get minified, which explains why I couldn't identify the module myself. So thanks.
  
• @DrAzzy, yes this is a Pico with the Esp8266 connected via the shim.
  
• So I instrumented the ESP8266WIFI_0v25 module and found that the problem lies there or in the AT module: once the serial data starts to arrive from the Esp8266 the ser.on('data',... gets all the priority and all the incoming data is buffered by the ipdHandler in sockData[sckt].
  
• only once all the data is buffered, then the socketServer calls netcallback.recv to start getting chunks to send on to req.on('data',...
  
• this is the point where I got the out of memory error: because JavaScript does expensive string copies for functions like String.slice() and String.substr(); as the netcallback.recv returns a chunk from sockData[sckt] it tries to prune the chunk from the buffer causing it to copy the larger remaining part of the buffer
  
  

So there are plenty of places where you could run out of memory with this behaviour, basically because the code doesn't let you process and discard each incoming chunk of data as it arrives, but buffers it and then chunks it up after it is all buffered.

Ideally the ESP8266WIFI_0v25 module and/or the AT module need to be redesigned to behave as a stream, and not just to emulate a stream by actually buffering everything and then chucking out from the buffer.

I hope all this makes sense.

Great I'll try it out when I've climbed out of the hole I'm down with ESP8266WIFI_0v25 chewing up memory.

I am using a pico with an ESP8266 for wifi.
I'm writing code that buffers the body of in incoming http request and attempts to write it to a file.

For small files (around 1-2K) it works. As soon as I try with a larger file I get the following error, I think its while I'm still buffering the body of the request.

ERROR: Out of Memory!
at line 1 col 106
...tr(0,c),h[a]=h[a].substr(c)):(b=h[a],h[a]="");return b}retur...
                              ^
in function called from system
ERROR: Error processing Serial data handler - removing it.
WARNING: Unable to create string as not enough memory
WARNING: Unable to create string as not enough memory
Execution Interrupted during event processing.
at line 1 col 189
...+1);h[b[0]]+=a.substr(c+1,f);return"+IPD,"+b[0]+","+(b[1]-f)...
         in function called from system
at line 1 col 106
...tr(0,c),h[a]=h[a].substr(c)):(b=h[a],h[a]="");return b}retur...
                            at line 2 col 73
...E3,function(c){d[a]=void 0})
                              ^
in function called from sysExecution Interrupted

I'm at a bit of a loss of how to go about tracking this down as it looks like its somewhere in the ESP8266WIFI_0v25 or the AT library. I could be wrong, but it doesn't look like any of my code.

Any help would be appreciated.

What was Espruino printing that was causing characters to get lost?

Actually it was the string for the espruino favicon you provided in the http server example titled Favourite Icon on WebServer

If you load that code onto the espruino and then do a dump(); then the long string of the favicon variable is not displayed as it is entered (where all non-ascii values are escaped), instead non-ascii values are displayed as 8-bit (unicode UTF-8 I assume) characters.

Going back to a post from a few days ago under the command-line topic
We were talking about tern and WebStorm and JSDoc.

You pointed me at the files on http://www.espruino.com/json, especially the espurino.json tern file.

I have since written a node script that parses this file, and reads the appropriate *.json file for a nominated board (as per the WebIDE tern.js plugin) and produces a comprehensive 'library' stub file along with board pins and devices complete with all corresponding JSDoc annotations. Please see attached for an example for the Pico.

So this is completely generated. There are only 3 rewrite rules in the script, each of a very generic nature, that deal with a few impedance mismatches between tern and JSDoc formats.

This JSDoc library is working very nicely with WebStorm. I'll send some screen shots later.
Hopefully this would be useful to anyone using an editor that requires this format, not only WebStorm.

At the moment the script is probably a bit brittle re: any changes in the *.json files, so I'd like to nurse it a little more before making it available.

Yes I'm on OSX.
The issue I raised on gitHub is also a solution for the this problem.
I got around to forking espruino/Espruino (as dmcnaugh) and have successfully built an image that implements the escape character fix, and I'm not having this issue now. The root cause appears to be that different terminals handle non-printable/non-ascii characters differently, so if you don't send them non-printable/non-ascii characters, then you don't get the problem.

The espruino command line now supports require() as per node . It implements the logic of node's require().resolve according to the rules set out at https://nodejs.org/dist/latest-v5.x/docs/api/modules.html#modules_all_together (with the exceptions of not supporting node binaries *.node and only looking in the modules folder in the current working directory, not recursively following back up to the path to the root)

See this post

That includes files, folders, sub-folders, .json, index.js and package.json support.

This hasn't made its way to the WebIDE and I don't know if it ever will.

Thanks, the code suggested at:

As you've got the EspruinoTools source handy, you could try this: github.com/espruino/espruino-too­ls/issues/6#issuecomment-101716152

solved the problem of sending code to the espruino. Is there a permanent fix in the issues queue? Or is this seen as a problem with node-serialport ?

I tracked down the 'dropped characters' problem when the espruino is sending to the terminal a little further and it looks like nothing is dropped in the transmission but different terminals produce different results. The issue occurs only with very long lines, typical when you dump(); and it is sending the content of a large, multi-line function as a single line of text:

• iTerm (OSX) shows only as many characters as it can to fill one line and then ignores the remainder of the line, and doesn't wrap
  
• the WebIDE wraps at the righthand end of the terminal and keeps showing everything, over multiple lines (best solution in my opinion)
  
• the terminal in WebStorm does - I don't know? - but you don't see the start of the line; the end of the line; consecutive characters from somewhere in the middle of the line; and sometimes you don't see anything at all for a long line. This is why I thought characters were being dropped.
  
  

So I'll try filing a bug with WebStorm but meanwhile I can't find a simple solution to make terminals wrap at the width of the terminal window.

I'm using

>process.env
={
  "VERSION": "1v84.145",
  "BUILD_DATE": "Jan  5 2016",
  "BUILD_TIME": "10:13:59",
  "GIT_COMMIT": "2f9e0124fad90ebcf32746a02ff3fd56e24a35f3",
  "BOARD": "PICO_R1_3",
  ...

A build you pointed me at to improve the RTC accuracy without a crystal (under my gitHub personna: dmcnaugh) - as expected its better but not fantastic - averaging +5 seconds every 10 minutes.

From what I can tell I'm not losing characters transmitting to the Pico but who knows. The same file will load then not load, then load. It will almost always load after a power cycle of the Pico, but then again I can make files that won't event do that. If I change the echo(0); to echo(1); in codeWriter.js -> `writeToEspurino then it looks like its the echoed characters that are sometimes being dropped, but I'm not getting syntax errors so I don't think its the transmitted characters. At some point after about 16Kb (changes) of code has been sent then the Pico just freezes and wont talk anymore until it has been power cycled.

Hmmm, 2 steps forward, 1 step back.
Happy with my mods, I've tried a build on a 'large' project and I'm having serial comms. problems with the espruino-cli dropping characters in both directions.
When I say large, I mean a combined 26Kb of source.

• I have checked it's not the code I am loading: I can capture it with -o and then load this file via the WebIDE successfully
  
• I have forced Espruino.Config.SERIAL_THROTTLE_SEND = true; and verified this is in effect, and it doesn't solve the problem.
  
  

When espruino -v source.js sends the code to the Pico there are dropped characters, even if I leave out the -v the result is the same. If I use espruino in terminal mode and do a dump(); on the full set of code (loaded with the WebIDE) then there are dropped characters in what I see. Again, I don't have these problems with the WebIDE.

Any ideas about what to try next? I currently think its somewhere in the 2 different code paths for serial comms. when you compare the WebIDE with espruino-cli.

OK.
But 2 things:

1. I wrote the localModules code for node so I haven't tested if it is browser compatible
   
2. The serial stuff doesn't stop you manually providing a port using -p -port it only affects the auto-selection of the 'first' port and limits it to the 'first espruino like port (STM32Fxxx)'