response from GET request is truncated (RAK8211)

Posted on
  • I am working on a RAK8211 module with Espruino 2v01.17

    I have been able to successfully send data using the http module with a "POST" request.
    I am now trying to receive data from a server. To do this I am sending this HTTP request:

    GET /api/v1/devices/configuration HTTP/1.1
    User-Agent: Espruino 2v01.17
    Content-Type: application/json
    Host: serverPath.com

    This is the data I am expecting, and the data that I am actually receiving:

    Expected:

    [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":true,"pressure":true,"temperature­":true,"eventTriggers":true},{"type":"Ac­tions","reboot":false,"dfwUpdate":false,­"findDevice":false}]
    

    Received:

    [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t
    

    As you can see, the response is truncated.

    I have tried this using both the http.get function and using the http.request function, both results in the same output.

    What is interesting is that the content length is 300, but only 168 characters are printed.
    When I do a print(res.available()) it returns 168

    The response from the server has the following headers:

    HTTP/1.1 200
    Server: nginx/1.12.2
    Date: Tue, 12 Feb 2019 06:28:31 GMT
    Content-Type: application/json;charset=UTF-8
    Content-Length: 300
    Connection: close
    X-Content-Type-Options: nosniff
    X-XSS-Protection: 1; mode=block
    Cache-Control: no-cache, no-store, max-age=0, must-revalidate
    Pragma: no-cache
    Expires: 0
    X-Frame-Options: DENY
    

    Why would the available bytes be showing only 168 when the Content-Length is 300 bytes?


    Some more detailed information of things I've tried below:

  • Using http.get():

    http.get(options, function(res) {
        print("available: " + res.available());                     //returns 168
        print("Content-Length: " + res.headers['Content-Length']);  //returns 300
        print("Manual read: " + res.read())
    
        res.on('data', function(data) {
            print("\t Response: code: " + res.statusCode);      
            print("--> " + data);
        });
    
        res.on('close', (data) => print("close: " + data)); //for debugging
        res.on('drain', (data) => print("drain: " + data)); //for debugging
        res.on('error', (e) => print("Error: " + e));           //for debugging
    });
    
    req.on('error', function(e) {
        print("ERROR", e);
    });
    

    Results:

    available: 168
    Content-Length: 300
    Manual Read: [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t
    code: 200
    --> [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t

    Using http.request:

    var req = http.request(options, function(res) {
        print("available: " + res.available());                     //returns 168
        print("Content-Length: " + res.headers['Content-Length']);  //returns 300
        print("read: " + res.read())
    
        res.on('data', function(data) {
          print("\t Response: " + "code: " + res.statusCode);
          print("--> " + data);
        });
    });
    
    req.on('error', function(e) {
        print("error: " + e);
    });
    

    Results:

    available: 168
    Content-Length: 300
    Manual Read: [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t
    code: 200
    --> [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t

    I also tried concatenating a string with each data that is returned:

    tempMsg = "";
    var req = http.request(options, function(res) {
        var l = res.headers['Content-Length'];
        print("l = " + l);
        res.on('data', function(data) {
            tempMsg += data;
            if (tempMsg.length >=1) {
            print("tempMsg: " + tempMsg);
            print("End of Message");
            }
        });
        res.on('close', function(data) {
            print("\t Response: " + "code: " + res.statusCode);
        });
        res.on('error', (e)=> print("error: " + e));    //for debugging
    });
    
    req.on('error', function(e) {
        if (callback) callback(e.code, e.message);
    });
    
    req.end(body);
    

    Results:

    l = 300
    [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t
    tempMsg: [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t
    End of Message
    code: 200

  • Can you try just going with what's done in all the examples? http://www.espruino.com/Internet

    Eg. remove print("Manual read: " + res.read()), concat all the data, and ONLY print it on close:

    http.get(options, function(res) {
        print("available: " + res.available());                     //returns 168
        print("Content-Length: " + res.headers['Content-Length']);  //returns 300
        var content = "";
        res.on('data', function(data) {  content += data; });
        res.on('close', (data) => print("close: " + content));
        res.on('error', (e) => print("Error: " + e));
    });
    

    I believe by doing the read you may be pulling data out that should be going to the data event, and when you are concatenating the data into tempMsg you seem to be printing End of Message after the very first data event.

    But basically:

    The callback for get/request gets called after the header is received - and not when all data has come in. It's then up to you to add a data handler and handle the data as it comes in.

    The reasoning is it's pretty common for HTTP requests to send more data than would fit in the device's RAM, so by leaving the handling of the data up to you, you can decide what is done with it.

  • I forgot to mention that I had tried that one too.
    Unfortunately I got the same result:

    function getHttp(options, callback) {
    
      http.get(options, function(res) {
        print("available: " + res.available()); //returns 168
        print("Content-Length: " + res.headers['Content-Length']);  //returns 300
        var content = "";
    
        res.on('data', (data) => content+=data);
        res.on('close', (data) => print("close: " + content))
        res.on('error', (e) => print("Error: " + e));
    
      }).on('error', function(e) {
        print("ERROR", e);
      });
    }
    

    Results:

    available: 168
    Content-Length : 300
    close: [{"type":"Positioning","accelerometer":t­rue,"trilateration":true,"globalPosition­ing":true},{"type":"EventCadence","smart­Mode":true},{"type":"Environmental","hum­idity":t

    I only added the "manual read" after it failed the first time, just to see if I could force it to read all 300 characters, which didnt work either because theres only 168 bytes available.

  • I turned on debug statements in the lower level code (QuectelBC95.js) and there seems to be an issue there. I'll take a look at that instead.

  • Ok - what's your full code? https://www.espruino.com/RAK8211 says you should be using QuectelM35 for the 8211-G, not QuectelBC95 (in fact there is no BC95 module?)

    Are you definitely using firmware 2v01? I seem to recall there was some issue like this with one of the RAK boards in the past but I was pretty sure it got fixed.

  • Hi Gordon,
    We made our own QuectelBC95.js file, because our device uses a BC95-G module.

    The low level code that we've written in this file is what is causing issues, specifically in the receive handler. So its nothing with Espruino that is the issue. I have figured out a solution now, by using the "+NSONMI" receive handler to output the number of bytes received and then retrieve them at a later time using "AT+NSORF", rather than retrieve them in the receive handler itself.

    Thanks for your help.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

response from GET request is truncated (RAK8211)

Posted by Avatar for Nadnerb @Nadnerb

Actions