Randomly Out of memory errors precedeed by JsVarOverflow

Thank you for the explanation. That helps me to follow along now.

Hybrid may now be the approach: implementing wire-close things in satellite processors in C that brings the signals conditioned into a higher level, and running the application in Javascript in Espruino... You could give a try and pack the low level C stuff onto Espruino as well, but that could interfere with Espruino's (Interpreter) 'terms and conditions'... - For sure a very interesting and real world challenge... beyond just some tinkering. Good luck. I'm strong in the Javascript (and Java) world, and looking forward to benefit from your C experience... ;-)

Well, hybrid developpment is still not on the front line as I hope to make a full espruino project. In fact it is also a unconventionnal approach to learn advanced programming methods in a context unusual to the JavaScript language. That's to the risk of disappointing you, sorry.

Now what remains could be buffer overflows.

You can check with E.getErrorFlags() - it should say if a buffer has overflowed?

what is the reason to use the bitwise OR (|) for the length

It's actually a performance thing. In compiled code dealing with integers is fast, but anything else falls back to Espruino's built-in maths, which must handle strings/arrays/floats/etc so is much slower.

The compiler isn't aware that .length actually returns an integer, and so it uses built-in maths for it and anything to do with it, which makes the whole for loop slow. By forcing it to an int the compiler can then do the whole for loop using ints (which go straight into registers).

Just out curiosity, why not store the NMEA data in JSON format?

@Gordon Thank's, diagnostic is always the most tricky point. I think, for now, that the errors returned will be of interest to my application. More precisely those ones:

1. 'CALLBACK': A callback (setWatch, setInterval, on('data',...)) caused an error and so was removed.
   
2. 'BUFFER_FULL': A buffer for a stream filled up and characters were lost. This can happen to any stream - Serial,HTTP,etc.
   
   

One more question: I am now moving a maximum of computations out of the Serial.on 'real time' event handler. If I understand well the .emit('someevent',function) would allow to postpone some computations and keep the order of the emits. However, what happens if I do something as setTimeot(f,0); That is a 0 timeout call. Is it queued at the end of the events waiting or inserted at the head of the events queue?

@cwilt I 'd rather use some binary floating point format to log time stamped data streams. JSON is verbous and designed to communicate between servers through the web, not to store data on an sd-card in time / space critical situations. This was to be considerated. Right now, I am even considering using the ublox own binary protocol, non nmea protocol. It might be faster, polled, so we get data when we can want, and avoids any float -> string -> float conversions. This should save time, space thus energy.

I have not explored the SD file writing routines to see how well handles JSON. Was just a passing thought.

what is the reason to use the bitwise OR (|) for the length

It's actually a performance thing...

Encountering this 0 | ... I removed the 0 | and first noticed a performance decrease timing 1K invocations... but later I felt not seeing it aymore and that's why I asked. The explanation makes it obvious... may be for compiled one should be able to provide type information for variables... to help the compiler a bit more... may be in the form of jsDoc... For running uncompiled it will (for sure) slow down the process... except something has changed in the upload to remove comments within a stament / function / object

var /*{String}*/ s = "Abc";
var /*{int}*/ len = s./*{int}*/length; // (or s/*{int}*/.length;

Similar approach for function signatures incl. return type.

hat happens if I do something as setTimeout(f,0); That is a 0 timeout call. Is it queued at the end of the events waiting or inserted at the head of the events queue?

Yes, it's executed after the Serial events have been processed - that could change at some point in the future though.

However the 'real-time' handling of Serial events may do what you need anyway. When it calls your on('data' handler, it searches for all the Serial data it can find on the top of the event queue and calls the handler with it.

Suppose you have a stream of data ABCDABCDABCDABCD... and you have a very fast handler that can execute data as it comes in, it'll get called with:

handler("A")
handler("B")
handler("C")
// ...

But if you have a slow handler, it'll get called with more data:

handler("ABCDABCDABCDABCDABCDABCD");
handler("ABCDABCDABCDABCD");
// ...

I guess perhaps by splitting off your handling you could detect whether you had more than one GPS sentence waiting to be processed and could voluntarily drop one rather than filling up the input buffer.

I think the batching is the better idea then try to have handlers that are actually fast enough to do the job. ...batching even to a custome defineable delimiter... or number of bytes received. An application (timeout) triggered getData(callbackData,callbackNoData) - w/ synchronous callbacks - could be an option to check for and get data...

For the SDI-12 protocol I was thinking of having a similar 'buffering middle-layer' that gets pulled data from the app in desired chunks... This allows failing patterns to drop before they even go into the buffer.

Hello,
I must be blind: I had totally forgotten the Testing environment included in the web ide.
So, as you wrote, there are videos explaining how to use it.

Well, gps is all about real time and position...

Yep, it is a great help in monitoring your Espruino... to see what is going on... The challenge is that it 'interferes' timewise, because code is exectuted on an interval bases that consumes cycles to get the data and send it over the Serial/USB/Console to the testing tool.

@Gordon and all others interested:
My problem was indeed an out of memory error, happening due to the exessive size of the gps object created.
The object was very oversize in memory due to using an eval() to transform my dynamically computed strings into functions.
Eval uses the local context which was the local context of the initialising function. Thus eval linked all of its local vars and values, local functions and so on.
This discussion gave me the answer.

How to call eval in a given context? 3 words. Use a closure.

var result = function(str){
  return eval(str);
}.call(context,somestring);

The context object above is a global object created outside of the parseHandlers function.
It could be seen as a pre initialised gps object. Something like
{ serial: Serial4 } is sufficient if provided as a parameter of the function that computes the handling functions.

The joined file is actually the parsing function that creates the gps object with its nmea sentences as methods. You then can call gps.GPGGA(d) were d is the array, comma splitted, resulting of the nmea string received after checksum and dollar having been removed.

Once intialised, you just could delete parseHandlers function which will free the memory space it used: around 650 jsVars... While gps object is less than a 100 jsVars....

So, so...

1 Attachment

• gps_gen13.js

That's great! Thanks for posting up!

Do you think there's anything that could be done to warn users when this is happening or help them detect it? I guess E.getSizeOf could have an option where it drills down into the object, showing hidden properties as well.

Definitly yes,
While the case about "this" is a classic problem to javascript, it becomes crucial on very small platforms such as Espruino. This would have gone unnoticed in a web browser for instance.
Drilling down in objects is a need to diagnostic and i ended with a difficulty to read the trace output.

@asez73, where do you draw the line 'real time'? Batching to simplify in the internal processing does not mean it is not real time for the user. Batching was also meant for just collecting all the received bytes until they make a reasonable gps sentence. This process should not be visible to the application. When look at @Gordon's implementation of the GPS moudle, it has a continuous buffering of received data. When something is ready to put into the buffer it is just appended, and when a sentence is found, it removes that part from the buffer and calls the callback. In other words, the filling of the buffer happens in chunks not necessarily aligned with the calling of the callback. If the buffering happens in pieces smaller than a complete sentence callback is not called on recption/buffering event. If more than one sentence is received, then the code loops around invocation of the callback until no complete sentence is found anymore in the buffer.

Real time, normally, means with a known delay between this instant of the event and the start of the routine in charge of handling the event.
In the particular case of the Espruino interpreter, things are handled by pushing a time stamp and the handler gets it with the pertinent data somehow later.
So, in such a very different context, I would call "realtime" anything that would end up by consumming events fast enough so as not to overflow the event fifo....
I guess this is how Espruino was designed.
Of course, in my particular gps goals, I need to end up all of the computations within a second since the gps gives its data every second, or 1/10 of a second if properly setup.
At this point, your own requirments are the rule.
Do forget that writing to aflash memory takes literally ages per comparison since they are blocking apis.
Even worst, the so called non blocking apis are never so:
Either you use a multi core / processor, which is not Espruino's case and you could delegate blocking tasks to one of them.
Or you have to buffer and delay blocking actions to the risk of not handling whatever happens at that instant.... So i would say that buffering is, per itself, a problem! That's why, I would that consider only buffering summaries, that can wait in a buffer for a very long time (seconds to hours). While i'd prefer to handle the full data stream in "realtime".
There is no universal answer to this!
It will, forever, be a matter of priorities: should I kill my batteries by over discharging them, or let my drone fall on someone's else head? A boat or a car could just safely stop in place and not destroy anything or anyone.

There is no universal answer to this!

That's right, It all depends.

While i'd prefer to handle the full data stream in "realtime".

There are the layers... such as the comm layer byte by byte, bytes by bytes, sentence by sentence, processing sentence by sentence, etc. I think the solution is optimal to have each layer to be as efficient as possible. I see your point that you actually could even start to process a single value from a sentence as soon as available. For certain things that can make sense. On the other hand, to make a "final" decision, you nead 3 things in a 3D world.

I think your approach to tailor first which sentences and then do the same for which values in the remaining sentences, that gets you where you want to be: as realtime as possible, realtime as to the point: as soon as the information is available.

For storing the values? Do you already have a solution? If you can process the pertine data in time, how much time do you have left to store it?... for example: in sequences of deltas? reduced granularity? encoded to use minimal space? Any processing requires additional time before storing. 100ms is not much the get everything done... Your app may become the first multi-PICO solution... or you switch over to the Propeller.

Well, you spotted it : To my opinion and so far identified needs, I would say that the "low level" realtime layer separates the gps data stream from the use of these data.
You could just use these data to display them or produce much more dynamic computations and finally store only the results of these...
Yet, all considered, those computations have to be real time too. As far as no other delaying hardware impose its own pace.
By dynamic computations I mean data values based complexity.
As it happens with the big data world it can just be impossible to do some post treatments either due to the data size (mb) or to the volume of data to get of the storage support per time unit (mb/s) .
Do we pretend that the espruino has to solve any real time need? I would n't

I've now made E.getSizeOf a bit more powerful. On latest git builds (and for 1v81), you can do:

>function a(b) {
:  return function() {
:  };
:}
=function (b) { ... }

>E.getSizeOf(a("Something really big"))
=12

>E.getSizeOf(a("Something really big"), 1)
=[
  {
    "name": "ÿcod",
    "size": 3 },
  {
    "name": "ÿsco",
    "size": 8 }
 ]

>E.getSizeOf(a("Something really big"), 2)
=[
  {
    "name": "ÿcod",
    "size": 3 },
  {
    "name": "ÿsco",
    "size": 8,
    "more": [
      {
"name": "b",
"size": 4 },
      {
"name": "return",
"size": 2 }
     ]
   }
 ]