SIM900 throws error after resetting and re-connecting.

This could be caused by a bug I found in the SIM900 driver. In the function 'receiverHandler' there is a line of code:
sockData[parms[0]] += line.substr(colon+1,parms[1]);

This needs to be changed to
sockData[parms[0]] += line.substr(colon+3,parms[1]);

I found if the whole response fitted within one handler transfer the orginal code would corrupt the response.

Amazinly doing so.. it seems to works! I just put the max=1 param out. As of today a collection length of >3 broke that.
@Gordon To the FIFO issue.... may it be a wrong canditate. I was noticing, the E.getErrorStats always returns FIFO_full on any request. But it is going to be erased later on.
Anyway the SIM900 driver is of really great help!

>  $http({
:      host: 'api.carriots.com',
:      path: '/streams/',
:      port: '80',
:      method: 'GET',
:      params: {
:        _t: 'str',
:        device: 'Commander@afitterling.afitterling',
:        max: 10
:      }
:    }).then(function(res){ a=res})
{
  "host": "api.carriots.com",
  "path": "/streams/?_t=str&device=Commander@afitterling.afitterling&max=10",
  "port": "80",
  "method": "GET",
  "headers": {
    "Content-Type": "application/json",
    "carriots.apiKey": "xxxxxxxxx",
    "Content-Length": 0 }
 } null
={  }
>
>
OK
>
0, CONNECT OK
>
>
>
>
0, SEND OK
>
+
RECEIVE,0,1380:
HTTP/1.1 200 OK
Dat
e: Fri, 29 Jul 2016 20:55:58 GMT
Content-Type: applicatio
n/vnd.carriots.v2+json; charset=utf-8
Conne
ction: close
Server: Carriots REST API
Ca
che-Control: no-cache
Allow: GET, POST
Access-Con
trol-Allow-Origin: *
Access-Control-Allow-Credentia
ls: true
Access-Control-Expose-Headers: Dat
e,Server,Allow,Connection,Content-Length,Con
tent-Type,Cache-Control
{"total_do
cuments":6,"result":[{"_id":"579bc2385c5d75660953772e","protocol
":"v2","checksum":"","_t":"str","at":1
469825592,"device":"Commander@afitterl
ing.afitterling","data":{"This":"Is a JSO
N example","Carriots":{"text":"Hello World"
,"number":123,"cool":true}},"id_developer":"8c19
07cf2402f422b64069832d07aa47276a6414fd4fdcf558a96f6a9
13c0a14@afitterling.afitterling","created_at":1469825592,"o
wner":"afitterling"},{"_id":"579bc2435c5d75a007537738","protocol":"v2
","checksum":"","_t":"str","at":1469825603,"device":"Commander@afitterling.afitt
erling","data":{"This":"Is a JSON example","Carriots":{"text":"Hello World","number":123,"cool":
true}},"id_developer":"d98688e13aae2861be6dd0bed33ba44f4d9171dfbe90d4ed64f5c091bb624cf4@afitterling.afitterling","crea
ted_at":1469825603,"owner":"afitterling"},{"_id":"579bc24c5c5d755102537814","protocol":"v2","checksum":"","_t":"str","at":1469825612,"device":"Commande
r@afitterling.afitterling","data":{"This":"Is a JSON example","Carriots":{"text":"Hello W
+RECEIVE,0,1339:
orld","number":123,"cool":true}},"id_developer":"6ddc5cab0331a2505419eb9d08afd74e2c31cc35c4
185d6e7de98f89b0655aed@afitterling.afitterling","created_at":1469825612,"owner":"afitterling"},{"_id":"579bc2bc5c5d751b7e537889","protocol":"v2","checksum":"","_t":"str","at":1469825724,"device":"Commander@afitterling.afitterling","data":{"This":"Is a JSON example","Carriots":{"text":"Hello Wor
ld","number":123,"cool":true}},"id_developer":"c1564b155fa18858c1f608f4db64f359e6539d
ec9789d51f14576c2819b6788e@afitterling.afitterling","created_at":1469825724,"owner":
"afitterling"},{"_id":"579bc2c75c5d75d908537723","protocol":"v2","checksum":"","_t":"str","at":1469
825735,"device":"Commander@afitterling.afitterling","data":{"This":"Is a JSON example","Carriots":{"text":"Hello World","n
umber":123,"cool":true}},"id_developer":"0a1e227e3914808240d4b29385fcc426f8dca235cce0295db059e41433a81a8a@afitterling.afitterling","created_at":1469825735,"o
wner":"afitterling"},{"_id":"579bc2d25c5d75f87f537843","protocol":"v2","checksum":"","_t":"str","at":1469825746,"device":"Commander@afitterling.afitterling","data":{"This":"Is a JSON example","Carriots":{"text
":"Hello World","number":123,"cool":true}},"id_developer":"e148352107e64d155ed2f32511afb197717f9d1f0e3e537773d0376ec1d0fec6@afitterling.afitterling","created_at":1469825746,"owner":"afitterling"}]}
>
0
, CLOSED
>a
={ "total_documents": 6,
  "result": [
    {
      "_id": "579bc2385c5d75660953772e",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825592,
      "device": "Commander@afitterling.afitterling",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "8c1907cf2402f422b" ... "rling.afitterling",
      "created_at": 1469825592,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2435c5d75a007537738",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825603,
      "device": "Commander@afitterling.afitterling",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "d98688e13aae2861b" ... "rling.afitterling",
      "created_at": 1469825603,
      "owner": "afitterling"
     },
    {
      "_id": "579bc24c5c5d755102537814",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825612,
      "device": "Commander@afitterling.afitterling",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "6ddc5cab0331a2505" ... "rling.afitterling",
      "created_at": 1469825612,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2bc5c5d751b7e537889",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825724,
      "device": "Commander@afitterling.afitterling",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "c1564b155fa18858c" ... "rling.afitterling",
      "created_at": 1469825724,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2c75c5d75d908537723",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825735,
      "device": "Commander@afitterling.afitterling",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "0a1e227e391480824" ... "rling.afitterling",
      "created_at": 1469825735,
      "owner": "afitterling"
     },
    {
      "_id": "579bc2d25c5d75f87f537843",
      "protocol": "v2",
      "checksum": "",
      "_t": "str",
      "at": 1469825746,
      "device": "Commander@afitterling.afitterling",
      "data": {
"This": "Is a JSON example",
"Carriots": {
  "text": "Hello World",
  "number": 123, "cool": true }
       },
      "id_developer": "e148352107e64d155" ... "rling.afitterling",
      "created_at": 1469825746,
      "owner": "afitterling"
     }
   ]
 }
> 

I have to test this over a long period of time. Thanks @jonreid if this works!
The FIFO issue also sounded plausible to me. It may be an issue. As @Gordon knows his module.

I think @Gordon is right:

"FIFO_FULL",
"LOW_MEMORY"
       ]
     }
   }
 }
Uncaught SyntaxError: Got ID:total_documents expected ','
 at line 3 col 3
{"total_documents":3,"result":[{"_id":"579bc6ad5c5d751b7e537...
  ^
in function called from system

My fix is for responses that are small in size - so you still need that.

If you are getting low memory errors you need to process your data in smaller chunks ie either optimise your json or request it in smaller chunks.

You never want to get into a low memory scenario as things start breaking pretty fast. It is worthwhile monitoring 'process.memory().free' at various parts of your code to highlight issues.

In my project - if I ever get an 'out of memory' error (which you should code to prevent) I reboot otherwise the code operation will be in an unknown state, also use the new watchdog feature.

@sp33c, I asume that the data you published in this forum is test data and therefore, my conclusion may not be the answer to the payload capacity issues... but I still want to mention it.

JSON is already much leaner than XML, but it is still 'fat', especially for repetitive data... and last but not least the lots of double quotes required. JSON is very easy and fast to process, but may be you can afford some CPU cycles to get over the bump. In case 'you know your data' / 'have control over the data', a CSV string is still the most effective to get there.... and furthermore, even having a catalog with ids for values that repeat.

The challenge is now to provide what xml does best and JSON about equally as good: provide the meta data (reference) about the individual data/information elements, and the structure...

The first one 'is easy' and is applicable if you have a number of defined data structures: first or last data item includes the id to the definition of the data structure.

The second one is when the values are sparse and variable in distribution across the overall (complete) structure: use the 1st approach - if useful - to identify the main structure, and a second - or second last - value for a bitmap that identifies which attribute (always same sequence) has value... (saves you from sending commas).

When values are repeated, a dictionary is built, passed as first thing in the payload, and then payload-local references describe the content (Some DBs do compression this way in order to not have to run uncompress algorithms).

Further option is to code the values if the values are known beforehand.

@jonreid yes, i knew about it and my status post requests are publishing the memory free. may i ask how do you reboot? what command?

hm... my code is not that big, actually at any point there shouldn't be a less memory state. i even taking care about intervals and timeout myself. it might due to memory leaking somewhere else.

@allObjects the blownup json is due mainly of infrastructure. carriots is a great iot platform. of course i can write the whole platform myself and being leaner. but i rely on such infrastructure for inter device communication. i like there way. i think the espruino device should be capable of handling such simple things. the productive response data is a bit leaner just carrying two values, cycle (watercycle) and time. also i use the max param to limit the get to one response entry at a time.

carriots doesnt represent a full schema but all you need to be able to handle different devices and logik. one optimization point can be to put the logic into carriots and have not a autonomous device anymore.

the overhead to deal with csv isnt really an advantage.

i am experienced with garbage collection issues, it should theoretically never be the case to run out of memory if constantly destroying references my self carefully. also not json is an isue as after processing the objects are nulled.

i may have to rething the overall model of that piece of hardware.

I updated my version to get rid of self caused memory leaks, I use the immediate return for timeouts. Thanks to the fix to the SIM900 module it is so far quite stable.
Additionally I set the SIM900 baud rate to 9600. Additionally I pull jobs out of resources with the max=1 parameter, means get only as less data as possible.
Memory seems stable so far.
I put the code online: https://github.com/afitterling/espruino-projects2/blob/master/RemoteWateringSystem.js
The cariots module can now handle wrong JSON and will reject the successful call:
https://github.com/afitterling/Espruino-Modules/blob/develop/carriots.js
this is the result of the status push at call 104.

    {
        "stats": {
            "process": {
                "stackEndAddress": 536959420,
                "flash_binary_end": 376280,
                "flash_code_start": 134234112,
                "free": 2875,
                "flash_length": 393216,
                "usage": 2225,
                "total": 5100,
                "flash_start": 134217728,
                "history": 1753
            },
            "errorFlags": [],
            "requests": 104,
            "errors": 0,
            "resets": 1
        }
    }

I just noticed, the SIM900 library is memory leaking, maybe du networkJS.create(netCallbacks).
Upon a repetive call on reconnect old references aren't getting collected with garbage collection, I assume old references are still being kept externally.
Could this issue be due to the networkJS library? Or is it due the SIM900 library only?

per reconnect my code is leaking 200 memory units (a 20bytes?)

I have given my SIM900 module to Gordon so he can sync in the bug fixes I have made. There are a couple of other basic things required for reliable operation ie in netcallbacks/create - there are more unregisterLine required for error cases and you need to remove the CIPSTART exception throw. Also if the SIM800 CIPSTART fails, the driver wont catch that so I added a timeout which then closes the socket and things are cleaned up properly.

At the app level, you need to ensure that the connect/send/receive process is completely finished before calling again otherwise memory will be consumed - this also means catching errors so the process can unwind properly. Use the request.on('error', function(e) ...

If you are in a low coverage area and the SIM800 can falloff the network there are a whole lot more considerations that come into play - you could just choose the approach of if things go wrong, reset and start all over, but I chose to handle all the states the SIM800 can get into and handle them in a more managed fashion.

Hopefully Gordon will have the time to sync in the basics of my changes to benefit everyone.

thanks for the feedback. could you send me your code as well? Do you have any? (info@sp33c.de)
I try to sync with gordon, but I am abroad a couple weeks, propably offline.

Thanks - I'll try and update this, but maybe not this week - things are a bit busy!

For the memory leak, it's actually most likely that the sockets are still open? You should be able to check in gsm.debug() I think and see a list of active sockets.

thank you very much. eventhough i resetted the sockets, the leak was still there. i expect the netcallbacks to leave external references (to the sim900 code) in the networkjs. closures are prone for memory leaks but most websites do not run above 1h, so nobody notices it.

i have not closed the sockets. i declared them to null. this may not have been enough. however this may be part of networkjs. i cannot test further i am abroad, the next 4 weeks. thank you for your kind support so far.