GPS Module, Extensions for handling other sentences, and u-blox NEO-6M GPS receiver

A five part series

In preparation for a navigation and tracking device with GPS and Display I looked into using the GPS module. The current module is very frugal. It supports just the NMEA $PGGA - essential Fix 3D position - sentence - basically: time, longitude, latitude, and altitude.

Initially I wanted to use my 20+ years old Magellan marine GPS with RS232 serial connectivity, but I dropped it in favor of a more recent and readily available ublox NEO-6M module, the same as used in Espruinos GPS moduel doc - http://www.espruino.com/GPS.

Hardware and software connectivity worked right away out of box. It took some time for the GPS module to pickup the satellites, but everything came together nicely.

From the past I knew that NMEA has also sentences that deliver, bearing, speed, etc... Speed over ground and bearing are part of a the NMEA $PGRMC - essential pvt (position/velocity/time) - sentence.

Looking into the implementation of the GPS module to extend it with a handler for other sentences - handleGPSLine()-methods - was a bit sobering... to say the least. My basic expectation was to find an externally extensible sentence filter with hooks and registration for other line handlers, or at least a way to override the already implemented, default GPS line handler. Unfortunately, the module is built in a way that ,gives no access what so ever for modifications or extensions. ;(

Sure, it is not much code there, but what I liked and noticed by rolling up the history of the modules especially in the forum, that quite some enhancements and validation went into the on.data method to extract and process available - (CR) LF delimited = lines.

Below is some code that is backward compatible and allows even the reuse of the existing default lineHandler for the use in extended mode. (The code is temporarily adjusted to run embedded while elaborating on the extensibility - see respective comments.)

// ---- beg (modified) GPS module code

function defaultLineHandler(line, callback) {
  var tag = line.substr(3,3);
  if (tag=="GGA") {
    var d = line.split(",");
    var dlat = d[2].indexOf(".");
    var dlon = d[4].indexOf(".");
    callback({ tag: tag,
      time : d[1].substr(0,2)+":"+d[1].substr(2,2)+":"+d[1].substr(4,2),
      lat : (parseInt(d[2].substr(0,dlat-2),10)+parseFloat(d[2].substr(dlat-2))/60)*(d[3]=="S"?-1:1),
      lon : (parseInt(d[4].substr(0,dlon-2),10)+parseFloat(d[4].substr(dlon-2))/60)*(d[5]=="W"?-1:1),
      fix : parseInt(d[6],10),
      satellites : parseInt(d[7],10),
      altitude : parseFloat(d[9])
    });
  }
}

var connect = function(serial, handler) { // replace with next line for module
// exports.connect = function(serial, callback) {
  var gps = {line:""}; var lineHandler;
  if (typeof handler == "function") {
    lineHandler = defaultLineHandler;
  } else {
    lineHandler = handler.lineHandler;
    if (handler.includesDefaultLineHandler) {
      handler.defaultLineHandler = defaultLineHandler;
    }
  }
  serial.on('data', function(data) {
    gps.line += data;
    var idx = gps.line.indexOf("\n");
    while (idx>=0) {
      var line = gps.line.substr(0, idx);
      gps.line = gps.line.substr(idx+1);
      lineHandler(line, handler);
      idx = gps.line.indexOf("\n");     
    }
    if (gps.line.length > 80)
      gps.line = gps.line.substr(-80);
  });
  return gps;
}
; var gps = {connect: connect}; // drop line for module

// ---- end (modified) GPS module code

Btw, I added the tag into the line handler's return object. It is useful when having only one callback for multiple sentences.

The basic idea of detecting and operating in the extended mode vs. the basic mode is by passing a handler object vs a handler callback function. Line 23 implements the detection. With this approach backward compatibility is given, and extensibility is opened up.

I adjusted some names to make the code extensions and the multi purpose use of the existing implementation pattern obvious.

The existing internal - not exposed - .handleGPSLine() function is renamed to defaultLineHandler, and is assigned to an internal variable named lineHandler in basic mode - when a in the .connect() a callback function is passed as handler (line 24). This variable is then used in the serial's on-data callback for each detected line (line 37). The function expects - as in the original code - a line and the external, custom callback for the line handler for invocation with the data object as built from the line.

The above setup to retain backward compatibility create constraints/requirements for the handler object in the enhanced mode: the handler object has to have a .lineHandler() method that accepts the same line and callback function arguments as the (existing) .handleGPSLine() - respectively renamed - .defaultLineHandler() function. This handler object's .lineHandler() method is then also assigned to the same internal variable named lineHandler, and corollary, is then also invoked by the serial's on-data callback for each detected line (line 37).

When the handler object includes a true evaluating .includeDefaultLineHandler property, it is complemented with the .defaultLineHandler() function as same-named method and therefore accessible in the handler object's lineHandler method - as the follow up post will show.

part 1 of 5 of: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

part 2 of 5 of: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

All code fragments of previous post and this one can coexist and can therefore be copied and pasted into the Web IDE edit pane, sent at the same time to the board, and individually invoked from the console pane or custom made functions.

The following setup is common to both - basic and enhanced - modes. Again, notice comments for the inline vs. module way of using the code.

// common setup
Serial4.setup(9600,{tx:C10,rx:C11});
// var gps = require("GPS"); // uncomment when using GPS code as module

The code below - example A) - shows the basic mode usage and validates the backward compatibility.

// A) for basic use: gpsHandlerFunction as callback as per ESPRUINO doc 
var gpsHandlerFunction = function(data) { console.log(data); };

For convenience - and coexistence in the Web IDE and sending to board - I added a goXyz...() function to be invoked in the command pane to run the vaious modes A) through C).

The goBasic() function connects the gps and passes the callback handler function .gpsHandlerFunction() as specified in A) above. This function is the same as used in Espruino's module documentation - http://www.espruino.com/GPS. Instead of defining it inline while passing it, it is defined as a variable and used in all scenarios to write to the console the data objects which is created by the various line handlers.

function goBasic() {
  gps.connect(Serial4, gpsHandlerFunction);
}

After sending the code to the board, enter goBasic() into the command pane to use the GPS module in basic mode.

Example B) is kind of a twitter: uses an wrapping line handler to wrap the defaultLineHandler() of the GPS module and is thus reusing it. The .includeDefaultLineHandler property with value true, makes the enhance GPS module to add the defaultLineHandler() function as same named method to the passed handler object.

// B) for enhanced use: gpsHandlerObjectDefaultOnly reusing just same basic function as in A)
var gpsHandlerObjectDefaultOnly = { 
  includesDefaultLineHandler: true,
  lineHandler: function(line, handler) { 
      handler.defaultLineHandler(line,gpsHandlerFunction);
  }
};

Invoke the function below in the command pane to run example B).

function goEnhancedWithDefaultOnly() {
  gps.connect(Serial4, gpsHandlerObjectDefaultOnly);
}

part 2 of 5 of: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

part 3 of 5 of: GPS Module, Extensions, and u-blox NEO-6M

Example C_ shows - finally - extensions with two line handlers in addition the to reused defaultLineHandler() of the GPS module:

1. transforming line handler implementation for the NMEA $GPRMC sentence
   

2. raw line handler - handling all sentences not handled by transforming line handlers

   // C) for enhanced use: gpsHandlerObjectDefaultAndOther reusing basic function as in A and other extensions
   var gpsHandlerObjectDefaultAndOther = { 
   // GGA (default): tag, time, lat[deg], lon[deg], fix, satellites, altitude[m]
   // GSA:
   // GSV (multiples):
   // GLL
   // RMC // NMEA's pvs - position, velocity, and speed (plus other things: true angle, date and time, mag var)
   includesDefaultLineHandler: true,
   handle: true,
   lineHandler: function(line, handler) {
   if (handler.handle) {
     var tag = line.substr(3,3);
     if/*sel*/ (tag=="GGA") {
       handler.defaultLineHandler(line,changeHandler);
     } else if (tag=="RMC") {
       handler.handleRMC(line,gpsHandlerFunction);
     } else {
       gpsHandlerFunction({"raw":line});
     }
   }
   },
   handleRMC: function(line,callback) {
   var d = line.split(",");
   callback({ tag:"RMC",
     timec: d[1].substr(0,6),
     latc: d[3].substr(0,2) + "_" + d[3].substr(2) + "_" + d[4],
     lonc: d[5].substr(0,3) + "_" + d[5].substr(3) + "_" + d[6],
     velo: d[7],
     angl: d[8],
     date: d[9]
   });
   }
   };
   

Use the related goEnhancedWithDefaultAndOther() function to run the example.

function goEnhancedWithDefaultAndOther() {
   gps.connect(Serial4, gpsHandlerObjectDefaultAndOther);
}

The raw handler shows the line 'as is' in the console for the purpose of exploring all the sentences the GPS receiver sends on serial data event, creating the following out put:

>echo(0);
=undefined
>goEnhancedWithDefaultAndOther()
=undefined
{
  "raw": "1\r"
 }
{
  "tag": "GGA",
  "time": "09:11:24",
  "lat": 41.22417566666, "lon": -102.87468133333, "fix": 2, "satellites": 11, "altitude": 70.3 }
{
  "raw": "$GPGSA,A,3,13,51,07,20,46,16,08,10,09,27,23,,2.73,1.11,2.49*08\r"
 }
{
  "raw": "$GPGSV,5,1,17,02,02,309,,03,22,042,,06,09,273,,07,45,265,27*7F\r"
 }
{
  "raw": "$GPGSV,5,2,17,08,65,279,41,09,62,328,22,10,30,309,28,13,57,326,26*7E\r"
 }
{
  "raw": "$GPGSV,5,3,17,16,46,059,31,19,02,135,,20,41,182,34,23,71,061,45*74\r"
 }
{
  "raw": "$GPGSV,5,4,17,27,12,106,18,30,16,250,,32,06,155,,46,42,142,34*77\r"
 }
{
  "raw": "$GPGSV,5,5,17,51,44,156,32*48\r"
 }
{
  "raw": "$GPGLL,4113.45054,N,10252.48088,W,091124.00,A,D*7E\r"
 }
{
  "tag": "RMC",
  "timec": "091120",
  "latc": "41_13.45054_N",
  "lonc": "102_52.48076_W",
  "velo": "0.035",
  "angl": "",
  "date": "270914"
 }
{
  "raw": "$GPVTG,,T,,M,0.035,N,0.066,K,D*20\r"
 }
{
  "tag": "GGA",
.....
...
.

part 3 of 5 of: GPS Module, Extensions, and u-blox NEO-6M

Looking into the implementation of the GPS module to extend it with a handler for other sentences - handleGPSLine()-methods - was a bit sobering... to say the least. My basic expectation was to find an externally extensible sentence filter with hooks and registration for other line handlers, or at least a way to override the already implemented, default GPS line handler. Unfortunately, the module is built in a way that ,gives no access what so ever for modifications or extensions. ;(

Why not extend the GPS module itself to fix this, and put in a pull request?

Modifying the module code would be a much more graceful solution, I'd say.

On my way to there. Question is: how much should it be backward compatible?

how much should it be backward compatible?

It needs to be completely backwards compatible. People do use it as-is and will be upset if their code breaks. What about something like this:

// Normal
require("GPS").connect(Serial1, function(data) {
});
// With Handlers
require("GPS").connect(Serial1, function(data) {
}, function(data, callback) {
  // do stuff
  callback(data);
 return true; // if handled
});

So that way the module code is backwards compatible, and only needs changing to:

exports.connect = function(serial, callback, handlerFn) {
  var gps = {line:""};
  serial.on('data', function(data) {
    gps.line += data;
    var idx = gps.line.indexOf("\n");
    while (idx>=0) {
      var line = gps.line.substr(0, idx);
      gps.line = gps.line.substr(idx+1);
      if (!handlerFn || !handlerFn(line,callback)
        handleGPSLine(line, callback); 
      idx = gps.line.indexOf("\n");     
    }
    if (gps.line.length > 80)
      gps.line = gps.line.substr(-80);
  });
  return gps;
}

Not tested, but it should work.

I'm in very 'violent' agreement with the 100% backward compatibility - nothing is as annoying as trying something out of the box and it does not work... and stops working after a while.

Other comments:

1. Had something like this with third argument in my first - not posted - elaboration.
   
2. Do not like to have to provide 2nd argument when using extended mode.
   
3. Do not like the comparison inside the loop that is done over and over again.
   
4. Do not like the constraint that handlerFn's return value has to evaluate to true.
   
5. Module provides still no means to start/stop/enable/disable line handling.
   
6. I like very much the leanness of the basic GPS module.
   
7. Would like to foster more object oriented approach in building and using modules and components - already done in many places where an object is provided as a parameter and not just a function-object (even though, JS allows to stick 'things' - value and function properties - it, but that is dangerous...).
   
   

Therefore I chose to use different type(s) for 2nd argument. Type is handled only once at connect execution time. The handler object's constructions allows to reuse the existing line handling.

@Gordon, you did not mention the addition of the tag in the existing handleGPSLine() function. Would you consider the code with the addition still compatible?

Furthermore, you did not mention the renaming of the internals. Would that keep the compatibility for the people?

May be a general discussion about initial setup of modules whit very limited, exemplary functionality is worth a discussion. The questions to answer would be:

1. What is the purpose of a module?
   
2. Who is the audience for a module?
   
3. How is extensibility rated?
   
4. How about the module's general 'structure': Lambda or Object oriented - or both?
   
5. How could extensibility - particularly for GPS module - be implemented - override - addition - registration?
   

Well, your method with the object as the second argument seems good, but I'd remove includesDefaultLineHandler and would just always add lineHandler as a field in the gps structure instead...

I don't think renaming internal things would cause problems - they'll probably just be minified anyway. I'd try not to use long variable/function names though (while still trying to keep them informative and readable).

In terms of modules:

• They need to provide a simple way to access the device in question
  
• The audience is 95% of users, with extensibility where it can be added without much overhead. If you know what you're doing you can always just modify the existing module, but if the module is so focussed on extensibility that nobody can understand it, I think it's failed.
  
• Can you give an example of what you'd consider a 'lambda-oriented structure?'
  
  

Just to clear up my thinking here, if given the following choices:

1. Write clear, simple code that is easy to understand and modify, but doesn't do everything.
   
2. Write complicated but extensible code, and then document it extensively so that people can extend it without touching the original code.
   
   

I think the first option is the right one for Espruino.

Espruino straddles a difficult line - it's trying to be easy to use, but it is a microcontroller with very limited storage. I don't think we should waste that storage on functionality that most people won't use when they pull in the module.

About the addition of the tag: Yes, it'd be compatible - but I don't think you should add it. If you're extending the GPS module yourself, why can't you just add it if you need it? Otherwise for everyone else it's just wasting their memory.

part 4 of 5 of: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

@Gordon, had some time to think... and code... under the influence of your input.

The code below is a first attempt.

// GPS2.js

// ---- beg (modified) GPS module code

function ggaHandler(line, callback) {
  var tag = line.substr(3,3);
  if (tag=="GGA") {
    var d = line.split(",");
    var dlat = d[2].indexOf(".");
    var dlon = d[4].indexOf(".");
    callback({ tag: tag,
      time : d[1].substr(0,2)+":"+d[1].substr(2,2)+":"+d[1].substr(4,2),
      lat : (parseInt(d[2].substr(0,dlat-2),10)+parseFloat(d[2].substr(dlat-2))/60)*(d[3]=="S"?-1:1),
      lon : (parseInt(d[4].substr(0,dlon-2),10)+parseFloat(d[4].substr(dlon-2))/60)*(d[5]=="W"?-1:1),
      fix : parseInt(d[6],10),
      satellites : parseInt(d[7],10),
      altitude : parseFloat(d[9])
    });
    return true;
  } else {
    return false;
  }
}

function handle(line,handlers) {
  var idx = handlers.length; while(idx > 0) { 
    idx--; 
    try {
      if (handlers[idx][0](line,handlers[idx][1])) return;
    } catch(x) {
      if (console) console.log("GPS:",x);
    }
  }
}

var connect = function(serial, handling) { // replace with next line for module
// exports.connect = function(serial, handling) {
  var handlers = (typeof handling == "function")
        ? [[null,handling]] : handling;
  if (!handlers[0][0]) handlers[0][0] = ggaHandler; 
  var gps = { line:"", handle:handle, handlers: handlers, enabled: true };
  serial.on('data', function(data) {
    gps.line += data;
    var idx = gps.line.indexOf("\n");
    while (idx>=0) {
      var line = gps.line.substr(0, idx);
      gps.line = gps.line.substr(idx+1);
      if (gps.enabled) gps.handle(line, gps.handlers);
      idx = gps.line.indexOf("\n");
    }
    if (gps.line.length > 80)
      gps.line = gps.line.substr(-80);
  });
  return gps;
}
; var gps = {connect: connect}; // drop line for module


// ---- end (modified) GPS module code


// common setup
console.log("Serial4 > ",Serial4);
console.log("Serial4.setup() > ", Serial4.setup(9600,{tx:C10,rx:C11}));

// var gps = require("GPS"); // uncomment when using GPS code as module
console.log("gps = req(GPS) >",gps);

// (default) callback for gps
var callback = function(data) { 
  console.log("data > ",data);
};

function go() { // connects the gps and gets it handling the lines
  console.log("gps.connect() >", gps = gps.connect(Serial4,callback));
}

function pause() { // pauses the handling of lines
  gps.enabled = false;
}

function resume() { // resumes the handling of lines
  gps.enabled = true;
}

Output in console:

"time": "09:22:57",```

data > {```

"lat": 41.22417449999, "lon": -102.87471283333, "fix": 2, "satellites": 8, "altitude": 74.8 }```

Basically - for the usage of the module - either:

• a callback function accepting a the data object of the module's line handler is passed
  
  

or:

• an array of tuples of lineHandlers and callbacks are passed.
  
  

Basically - for the implementation -

1. Paired / tuples - or chain - of line handlers and callbacks stored as handlers are used (implemented as array of 2-elements-arrays, with lineHandler as first and callback as second element of the 2-elements-arrays). A lineHandler looks at the line and calls the paired up callback with the handled line data. ggaHandler() - current module's handleGPSLine - is an example of a lineHandler. callback is an example of a callback and does the 'stuff' the GPS module user wants to have done for the line by the (paired up) lineHandler.
   
2. For each line from the GPS receiver, the module iterates over this array of handlers in handle() function and stop looping when handler did 'hit'. The handlers, such as ggaHandler() handling GGA tagged lines, return true when handling the line, otherwise return false (line 38).
   
3. The same data structure and algorithms are used for both 100% compatible basic and extended mode: if first tuple's lineHandler is null (evaluates to false), the modules basic ggaHandler() is reused for it (line 40).
   
4. Like line, handle() and handlers[] - and an extra enabled flag - are kept in the object returned by the connect() function
   
   

Btw, I would consider above approach a Lambda or functional approach, where after module load 'no' application nor object oriented object is (yet) available, but merely a bootstrap function, which is not even equally to a 'new' or a constructor. This 'into an object wrapped' function has first to be called in order to get a (useful) object (with state, and now also with function) back. - Btw, what is/was the rational of returning the {line:""} object on .connect()? - The module documentation http://www.espruino.com/Writing+Modules follows more the object-oriented approach, even though - due to the nature of JS - the 'class definition' is a function. For the GPS module I would like to have a similar approach chosen, even if the GPS stays a Singleton / would not have a constructor, so that already with require("GPS") an object is available for custom alteration. I will look into this option in a next phase while - of course - keeping 100% compatibility with current 'usage documentation'.

I still try to find what's right for Espruino. Having now lived many years - and still live - in a world of memory abundance, doing the right thing for Espruino is a welcome challenge. Not that I waste memory or cycles - I actually have the opposite habit - because I started out with having just 16KB (Kilo bytes) for application AND data AND ANY utility code - such as ISAM-create/read/update/delete, input routines, etc., and some of my Espruino-like hardware real world projects had just 256 bytes and 1KB EPROM and 4MHz 8-bit processing capabilities. Since then there is always the frugal guy in the back of my head watching over my spending of both memory and cycles, because even in today's memory and cycle paradise resources are limited... if not technically, then more so economically (currently building software for 15K+ concurrent transactional users - where user AND operator has to be kept happy: first one w/ snappy response times and latter one w/ spending 'no fancy' $s).

@Gordon, some pointers about the implementation of js in Espruino is very welcome here. I provide you an example in more detail at (see http://forum.espruino.com/conversations/255954) from code I'm working on: An array with a lot of small string elements in source code uses obviously more memory than one with few but long strings that later are chopped up into the small ones and stored again in the very same array. The short string are of the form of "keyNMO".

Back to the GPS discussion at hand:

Add the following code to the above one, send code to board, and use command go2() in the command pane to get the gps connected and handling lines (just) with the custom RMC line handler instead of the module's own ggaHandler. For simplicity, same already known callback is used.

var handlers =
[ [ function(line,callback) { // rmcHandler
      var tag = line.substr(3,3);
      if (tag=="RMC") {
        var d = line.split(",");
        callback(
          { timc: d[1].substr(0,6)
          , latc: (parseInt(d[3].substr(0,2),10) + parseFloat(d[3].substr(2))/60) * (d[4]=="S"?-1:1)
          , lonc: (parseInt(d[5].substr(0,3),10) + parseFloat(d[5].substr(3))/60) * (d[4]=="W"?-1:1)
          , velc: parseFloat(d[7])
          , angl: d[8]
          , datc: d[9]
          });
        return true;
      } else {
        return false;
      }
     }
  , callback
  ]
];

function go2() { // connects the gps and gets it handling the lines
  console.log("gps.connect() >", gps = gps.connect(Serial4,handlers));
}

Output in console:

"timc": "103314",```

"angl": "",```

}```

"timc": "103315",```

"angl": "",```

}```

You may notice that the ggaHandler 'is lost' for good... (became inaccessible for good).

The module's valuable ggaHandler can though easily be reused/included in the custom handling. Just provide the first tuple of 'handler duos' with null (no) line handler. On connect, the module's ggaHandler will be pulled and paired up with the callback provided in the first handler duo. To make that obvious / visible, a different calback2 call back is provided.

Add the following code to the above one, send code to board, and use command go2() in the command pane to get the gps connected and handling now two lines, the GGA tagged line with the module's own line handler and callback2(), and the RMC tagged line with the custom line handler and initially established callback().

// special callback for in reuse of GPS's module own ggaHandler
var callback2 = function(data) { 
  console.log("cbk2 > ",data);
};

var handlers2 =
[ [ null // re-using GPS module's ggaHandler...
  , callback2 // ...w/ callback2
  ]
, [ function(line,callback) { // rmcHandler
      var tag = line.substr(3,3);
      if (tag=="RMC") {
        var d = line.split(",");
        callback(
          { timc: d[1].substr(0,6)
          , latc: (parseInt(d[3].substr(0,2),10) + parseFloat(d[3].substr(2))/60) * (d[4]=="S"?-1:1)
          , lonc: (parseInt(d[5].substr(0,3),10) + parseFloat(d[5].substr(3))/60) * (d[4]=="W"?-1:1)
          , velc: parseFloat(d[7])
          , angl: d[8]
          , datc: d[9]
          });
        return true;
      } else {
        return false;
      }
     }
  , callback
  ]
];

function gox() { // connects the gps and gets it handling the lines
  console.log("gps.connect() >", gps = gps.connect(Serial4,handlers2));
}

Output in console:

"timc": "124700",```

"angl": "",```

}```

"time": "12:47:01", "lat": 41.2242155, "lon": -102.87470616666, "fix": 2, "satellites": 9, "altitude": 73.8 }```

I'm not really sure about the overall value of enabled - but I liked the option to to shut up the console... - last but not least for copying console output to this post without having a nervously scrolling away text - or, in the target application - to quiet down the display... ;)

What should be discussed is the value of the try-catch-block. In different context I had issues with the process stopping after some time - less than one hour - and errors showing in the console. Especially with line handlers that do not take into account when the GPS is not yet providing complete data, for example, while still trying to get enough satellites for decently reliable position determination.

What I'm though confident about is:

1. 100% compatibility
   
2. Easy, simple, but quite powerful and flexible extensibility
   
3. Handling can be modified even after being established
   
4. Extensibility of the extension showed in the example, for example, adding the a 3rd array element to name the tuples for addressable change or removal - preferably when disabled
   
5. ...and many things more.
   
   

part 4 of 5 of: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

Re: Strings

A string takes 1 jsvar per 12 characters.

A simple array takes 2 jsvars per elements in the array, plus each of the elements has to be stored.
Since your keys are ~6 chars long, you're taking 3 jsvars per key, vs ~7/12ths of a jsvar for storing them as a big string.

Large simple arrays are very inefficient in Espruino.

@DrAzzy, this explains why I ran out of memory... but how do you explain that after adding even more logical strings (keys) in less physical strings, but then storing them back with the old format - 1 key per element - out of memory condition was avoided? The explanation would be: A string takes 1 jsvar per 12 characters in the source code. Does that sound right?

I cannot explain that. Maybe gordon can. I'd missed the bit about your being able to create the same array with split and not getting the out of memory error - that makes not a bit of sense to me either.

...may be I tapped into memory beyond the safety margins, which is a sure call for disaster... I'll start a new conversation - http://forum.espruino.com/conversations/255954 - for that memory stuff and cross reference to keep this one focused on the GPS with the tint of Extensibility and what is right for Espruino.

part 5 of 5 of: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

PS: If you want to see the raw data in the console all the times, add a line handler and callback tuple with an alway false returning line handler as last to the handlers. This will get you all lines send by the GPS receiver printed in the console.

var handlers3 =
[ [ null // re-using GPS module's ggaHandler...
  , callback2 // ...w/ callback2
  ]
, [ function(line,callback) { // rmcHandler
      var tag = line.substr(3,3);
      if (tag=="RMC") {
        var d = line.split(",");
        callback(
          { timc: d[1].substr(0,6)
          , latc: (parseInt(d[3].substr(0,2),10) + parseFloat(d[3].substr(2))/60) * (d[4]=="S"?-1:1)
          , lonc: (parseInt(d[5].substr(0,3),10) + parseFloat(d[5].substr(3))/60) * (d[4]=="W"?-1:1)
          , velc: parseFloat(d[7])
          , angl: d[8]
          , datc: d[9]
          });
        return true;
      } else {
        return false;
      }
     }
  , callback
  ]
, [ function(line,callback) { callback(line); return false; }
  , function(data) { console.log("raw > ",data); }
  ]
];

Output in the console:

cbk2 > {```

"lat": 41.2242, "lon": -102.87471116666, "fix": 2, "satellites": 9, "altitude": 92.4 }```

raw > $GPGSV,3,1,10,02,31,052,15,05,53,089,37,10,15,042,21,12,42,164,33*79```

raw > $GPGSV,3,3,10,46,38,145,,51,44,156,38*7C```

raw > $GPRMC,131934.00,A,4113.45229,N,10252.48332,W,1.188,204.02,300914,,,D*7A```

"timc": "131934",```

"angl": "204.02",```

}
raw > $GPVTG,204.02,T,,M,1.188,N,2.199,K,D*3F

End of 5 parts series: GPS Module, Extensions, and u-blox NEO-6M GPS receiver

Hopefully I've covered the question of memory usage in the other post...

I'm afraid I don't have that much time at the moment, so can't give you a very in-depth reply...

Returning just {line:...} was actually a tweak that was made when I was trying to debug some issues that someone was having. Originally nothing was returned.

Moving to a more object oriented approach would be better I guess - it would push the memory usage up but it might be worth it.

I'm not convinced about your approach with the array. To me, it is slower and uses more memory in all cases, and users have to understand about the array and structure their code around it (even if it's not what they want to do). Surely it is much better to have them define an array themselves if they want that behaviour.

Same with enabled. Surely it's better to handle that in your own code if you need it? I can understand that a user might want to totally remove the listener from the Serial port, but that's another modification.

What about (not tested):

function GPS(serial,callback) {
  this.serial = serial;
  this.line = "";
  this.start(callback);
}

GPS.prototype.lineHandler = function (line) {
  var tag = line.substr(3,3);
  if (tag=="GGA") {
    var d = line.split(",");
    var dlat = d[2].indexOf(".");
    var dlon = d[4].indexOf(".");
    return {
      time : d[1].substr(0,2)+":"+d[1].substr(2,2)+":"+d[1].substr(4,2),
      lat : (parseInt(d[2].substr(0,dlat-2),10)+parseFloat(d[2].substr(dlat-2))/60)*(d[3]=="S"?-1:1),
      lon : (parseInt(d[4].substr(0,dlon-2),10)+parseFloat(d[4].substr(dlon-2))/60)*(d[5]=="W"?-1:1),
      fix : parseInt(d[6],10),
      satellites : parseInt(d[7],10),
      altitude : parseFloat(d[9])
    };
  }
}

GPS.prototype.start = function(callback) {
 var gps = this;
 this.serial.on('data', function(data) {
    gps.line += data;
    var idx = gps.line.indexOf("\n");
    while (idx>=0) {
      var line = gps.line.substr(0, idx);
      gps.line = gps.line.substr(idx+1);
      var gpsData = gps.lineHandler(line); 
      if (gpsData && callback) callback(gpsData);
      idx = gps.line.indexOf("\n");     
    }
    if (gps.line.length > 80)
      gps.line = gps.line.substr(-80);
  });
}

GPS.prototype.stop = function() {
 this.serial.removeAllListeners('data');
}

exports.connect = function(serial, callback) {
  return new GPS(serial, callback);
}

// then normal:
require("GPS").connect(Serial2, function(data) {
 ...
});
// or advanced
require("GPS").connect(Serial2);
GPS.prototype.lineHander = function(line) {
  // do whatever you want
};
// or advanced while keeping the old handler and callback pattern
require("GPS").connect(Serial2, function(data) {
});
GPS.prototype.oldHander = GPS.prototype.lineHander;
GPS.prototype.lineHander = function(line) {
  if (iCanHandleLine) {
   return { my:Data };
  } else return this.oldHandler(line);
};

I believe this will use a bit more memory, but it has a few advantages:

• It's in the 'suggested' form for writing modules
  
• You can now properly start and stop the GPS module - so no processing is done when a line comes in
  
• Because functions are not defined in the module scope, when you say GPS.prototype.lineHander = ... it will overwrite the old one, freeing its memory.
  
• When you override lineHandler you get full control over everything
  
• You can still 'keep' lineHandler if you want to use it yourself.
  
• You don't have to use the callback pattern for handlers if you don't want to
  
• You can create an 'AdvancedGPS' module pretty easily
  
  

What do you think?

Thanks for sinking your already scarce time resource into this. I'm sure you would like to have satellite brains and body clones working for you and executing all the creative ideas going on in your master brain... ;) - something like this happens in my dreams. Back to - nevertheless - exciting reality:

covered the question of memory usage in the other post...

Gives me enough food for thought to take the next step in adjusting into the architecture and spirit of Espruino as a whole. I'm sure the two links in the addendum post will explain the term root scope. I will resume with the memory conversation when resuming with the related project, which is: http://forum.espruino.com/conversations/127039 - Exploring 2.8" Color TFT Touch LCD by PacMan Game attempt.

What I'm thinking about you suggested 'new' implementation of GPS module - well: I'm going back to school is for sure part of it... other parts are:

I began with list of thoughts below... until paying closer attention to '.start()' method... and got stuck in it.... which toppled almost everything but the first thought to the point of actually being wrong... and to being wrong about code 'annihilates' - or at least - questions most of my (professional) past... therefore: Disregard the list. I will come back, re-think/buke and re-write the list. For now I can say that I 'see' great progress. ;). To have an idea where this all is heading for, take a look at the running application of the GPS module at http://forum.espruino.com/conversations/255759 - DIY Marine GPS using enhanced GPS Module, u-blox NEO-6M GPS receiver, and ILI9341 Module controlled 2.8" Color TFT LCD.

1. Comparison with my code / solution patterns highlights the suggested form of writing modules
   
2. Having .start() and .stop() methods and their way of implementation great in many ways - and with it - the whole module passes object-orientation with flying colors. It now allows also to have multiple GPSs... ;) - which I was thinking about in the previous solution, but considered it a rare option... I'd rather have two complete systems connected to a network for 'high'-availability.
   
3. The .stop() 'disconnects' the handlers completely vs. the approach I chose. I was thinking about something like that but had not studied Serial object enough yet. My solution comforted me by the fact that it keeps up with 'complete' lines in the background makes a resume flawless... which I guess is not the case with suggested .start()/.stop()............................ ooooops.................. see comment in italics above this list.
   

Cool - yes, start/stop would miss out on some lines - I don't think that would cause problems.

My feeling with enable is it's actually not that much more inefficient to implement it in the callback fn (after all, it only gets called once a second, at most 10 times a second).

'root scope' - It's not really the correct term - you'd probably be more used to 'Closure'/'Lambda'. For instance the obvious example:

var a = (function() {
  var z = 42; // <----- here
  return function() { return z; };
})();

Also applies to modules, which are implemented a lot like:

// --- 
var a = require("foo").connect();
// foo.js
var z = 42; // <----- here
String.prototype.hello = 42;
exports.connect = function() { return z; };

// Goes to
var a = (function() {
  var exports = {};
  // start module code ----------------
  var z = 42; // <----- here
  String.prototype.hello = 42;
  exports.connect = function() { return z; };
  // end module code -----------------
  return exports;
}).connect();

So what I'm saying is, if you define z, it's there forever, because it's part of the closure of the connect function. But if you define String.prototype.hello, that can be deleted.

I'm not concerned about missing out on lines. I see more the challenge in getting partial / incomplete lines (GPS sentences) on (re-)start/resume. On the other hand, the very rigid tag extract and check would then just make the partial line to be skipped too. The next one though is pickup for sure, which - with (most) receiver's default configuration - shows up right within the next second (I have seen increasing the sentence (set) frequency on u-blox receivers, which of course requires also to increase serial's speed/baud rate in order to send the - also configurable set of - messages within the configured time period.

Thanks for backfilling the term 'root scope'. I like it also for enforcing 'privacy' / enhanced encapsulation... (as 'private' in Java). It is a neat, JS distinctive thing to be able to establish things in the function/method body definition that then can only be seen from within the 'definition-time-scope' and will 'stay for ever' - or at least as the function/method is not garbage collected . The same happens to all things defined in modules when not exported... because it happens all in require()'s scope... (that's why the current GPS module's handleGPSLine() function is not accessible / lacks/suffers from accessibility - lacking or benefiting from absent accessibility is though always dependent on the design intention).

In GPS module I have seen single method export... and in my embedded code, I did emulate the multiple exports because require() returned that kind of object (see line 56 in post http://forum.espruino.com/comments/11857912/ - part 4 of 5). From your code snipped I conclude that you can have a 'collection' of exports... - or, more precisely - an object (instance) with named properties - or plain js object - POJSO (like POJO as in Java). I hoped for and assumed something like that... and now I see how it is done.

Because in the Espruino and a-like realm most challenges come from connecting to things / devices - and module help with that a great deal: hide the nitty-gritty, always the same looking things and (can also) provide additional convenience (methods). I see though the module technique not limited to that only and can see many other application that would have something else than connect as the initial method to get going. I could also see start and stop as a different scope than connect/disconnect, because start, stop, pause, and resume have a different feel to me... almost as we distinguish also boot/cold-start/warm-boot/start,... similar like sleep, deep sleep, and hibernate... none of these are reset or boot. These various types of processes or sequences/sets of life-cycle steps exist all in their own rights....

Oooops: I have - again - to remind my dreamy mind that with Espruino I deal with (very) limited resources... and thus many of these processes have to collapse into dedicated, simplified ones - just as needed. It's almost going back 50 or 60+ years in computing, when computers were built for a distinctive task... Only later systems, such as, for example, IBM System /360 computer announced early 60' were declared as a general purpose machines - not just to do Operations Research (OR), or Accounting, or... you name it. The key logo / badge for this first IBM general purpose computer was a compass rose: you can go any direction with this thing (see http://www.computerhistory.org/revolution/mainframe-computers/7/161).

Today, with 'cheap' hardware available in masses, we can go back to the future and 'back' to dedication... and there is need for a big warehouse, a power plant, and cooling towers for operating such things. I bet that Espruino has multitude of processing power and with wifi connectivity unlimited storage compared to /360 systems with relative little memory - some models with much less and some with more then Espruino - and 7+MB disk storage... and /360s were called a Main Frames... ;-): Everything has it's season.