compare event based handling of post-request with piping

Project is to support uploading of files and save on SD.
It is tested on ESP32, I don't have hardware to compare with espruino board.
If this is a ESP32 problem only, please move it to ESP32 forum.
Doing the job with piping does not save full upload, some data at the end is missing

function postUpload(req,res){
    var a = url.parse(req.url,true);  //there is a parameter given for location of new file on SD
    var f = E.openFile(a.query.pathname,"w");
    req.pipe(f,{chunkSize:32, end:false,
      complete:function(){
          f.close();
          res.end();
      }
    });
  }

Doing same without piping works fine

function postUpload(req,res){
    var a = url.parse(req.url,true);
    var f = E.openFile(a.query.pathname,"w");
    req.on("data",function(data){f.write(data);});
    req.on("close",function(){
      f.close();
      res.end();
    });
  }

Looks like it could be a bug.

Do you think you could come up with a full (but minimal) example (ideally with a server, and a client that performs the request) that I can run on a Linux build of Espruino?

I can try and track this down a bit more easily then.

Hopefully this helps (and there is no typo in text :-) ).

First see attached files.

• copy testPipe.html to sd
  
• connect sd to your board
  
• copy tesppipeJS.txt to webIDE
  
• change ssid/passwd and pins to your board
  
• send to your board
  
• open http://nameOfServerOnYourBoard/testPipe.html
  
• fill textarea with a lot of chars, more than 1000
  
• click Upload piped
  
• check file on sd in webide console.log(require("fs").readFile("uploaded.html"). In my case only a few lines, up to 25, are found )
  
• drop file on sd require("fs").unlink("uploaded.html")
  
• click Upload eventdriven
  
• do same check as before. In my case the file now holds everything
  
  

BTW, looks like different sd cards result in different size for Upload piped

2 Attachments

• tesppipeJS.txt
• testPipe.html

Which board is this on?

Did you try it on the Linux Espruino build? That would be so much easier for me - especially when it comes to trying to debug this

Sorry, I don't have a linux build. My Ubuntu server is leased in the internet, and my laptop has windows.
It's tested on ESP32, all my Espruino boards are without WiFi :-(

I just looked in to this, and it's a few issues.

• First, you never do res.end() so in both cases you keep the connection to the client open and close is never called
  
• You specified quite a small chunkSize. HTTP shoves data into the stream as it arrives, but because the stream is stuck shifting 32 bytes at a time (which is also really slow for file IO) data piles up and after 512 bytes Espruino starts chucking it away. If you check 'E.getErrorFlags' I bet you'll find a BUFFER_FULL error in there.
  
• Finally there was an issue where pipes no longer picked up the close event from sockets properly
  
  

So:

• git pull
  
• Add res.end() into your handler immediately (Espruino won't close the connection until content-length is received)
  
• Change chunksize to 1024 (it won't really hurt having it big as the data is already in memory)
  
  

And that's it - it should work. The whole pipe overflowing thing is a pain, but I don't see a nice way around it unless the pipe is allowed to ignore the chunksize argument when it sees that it has a bunch of data available for it - but that could cause problems for endpoints that really only do want small amounts of data.

Also, I had to spend ages getting your code running under Linux first. In future it's easy to spin up a VM or even use a raspberry pi.

Oh, hopefully, your kids will recognize that old man at the door ;-)
Being serious now, what was strange in my code, which took away your time ?

Thanks for your feedback. I will do as suggested.
One thing I don't understand, there is an res.end in function uploadStd and another one in function uploadPipe. Are they never called, wrong location, or ??

BTW, do you know about any guideline about setting up a VM for Espruino on Windows 10.

@JumJum
The best thing to do is to use the window 10 64 bit bash on windows. I'm using this for development.
Install Bash on Ubuntu on Windows 10 https://msdn.microsoft.com/da-dk/commandline/wsl/install_guide

git clone  --recursive  git://github.com/espruino/EspruinoBuildTools.git
cd EspruinoBuildTools/esp32/build/Espruino

source scripts/provision.sh LINUX
make clean && make
./espruino

what was strange in my code, which took away your time ?

:) I had to add some print statements to figure out what was happening at first, but the URLs requested all start with / - which means that when running under Linux it's trying to read and create files in the root directory. Also uploadStd and uploadPipe did the opposite of their names ;)

there is an res.end in function uploadStd and another one in function uploadPipe. Are they never called, wrong location, or ??

Wrong location I'm afraid - they are called when the pipe closes, but the pipe only closes when the socket closes (at least in Espruino) - and the socket never closes because res.end() wasn't called!

As @Wilberforce says, using Ubuntu on Windows is pretty nice. With the new Espruino builds you may have to do BOARD=LINUX make though - but I might tweak that soon so it's not needed.

@gordon any chance you could post your code - I would like to see where you put the end handler.

Sure, the code I ended up using was:

var http = require("http");
var page = `<html>
<head>
	<title>test pipe</title>
	<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
</head>
<body>
	  <button id="uploadPipe">Upload piped</button>
	  <button id="uploadStd">Upload eventdriven</button>
	  <button id="getUploaded">Read uploaded</button>
	  <textarea rows="25" cols="100" id="Htmlfile"></textarea>
</body>
<script>
  $("#Htmlfile").val( $("html").html());
  $("#uploadPipe").click(function(){
    $.post("uploadPipe",$("#Htmlfile").val(),function(data){});
  });
  $("#uploadStd").click(function(){
    $.post("uploadStd",$("#Htmlfile").val(),function(data){});
  });
  $("#getUploaded").click(function(){
    $.get("uploaded.html",function(data){
	  $("#Htmlfile").val(data);
	});
  });
</script>
</html>`;
var r;

function httpServer(){
  var me = this;
  var srv;
  function handleGet(req,res){
    res.writeHead(200,{"Content-Type":"text/html"});
    if (req.url!=="/uploaded.html") {
      res.end(page);
      return;
    } 
    var f = E.openFile("uploaded.html","r");
    var data,ext,type;
    if(f !== undefined){
      do{
        data = f.read(32);
        if (data) res.write(data);
      } while(data);
      res.end();
      f.close();
    }
    else{
      res.end(a.pathname + " not found");
    }
  }
  function uploadStd(a,req,res){
    console.log("uploadPipe",a);
    var f = E.openFile("uploaded.html","w");
    req.on("data",function(data){f.write(data);});
    req.on("close",function(){
      console.log();
      f.close();
      res.end("event upload done");
    });

  }
  
  function uploadPipe(a,req,res){
    console.log("uploadStd",a);
    var f = E.openFile("uploaded.html","w");
    req.pipe(f,{chunkSize:512, end:false,
      complete:function(){
          console.log("Complete");
          f.close();
          res.end("pipe upload done");
      }
    });
    res.end();
  }
  function handlePost(req,res){
    var a = url.parse(req.url,true); 
    switch(a.pathname){
      case "/uploadPipe":uploadPipe(a,req,res);break;
      case "/uploadStd":uploadStd(a,req,res);break;
    }
  }
  function handle(req,res){
    if (req.method == 'POST') 
	  handlePost(req,res);
    else if(req.method == 'GET') 
	  handleGet(req,res);
    else req.connection.destroy();
  }
  me.init = function(){
    srv = http.createServer(handle);
    srv.listen(8080);
  };
}
var srv = new httpServer();
srv.init();

So basically I call end immediately, because the connection will still be held open while the data received < Content-Length

Thanks

I'm looking at using the file upload input element with the multi-part encoding, so that files can be uploaded including binary files.

Can you think of a cunning way of decoding the multi-part message?

This? https://github.com/espruino/Espruino/issues/226#issuecomment-181443491

Thanks Gordon. Decoding the post boundarys is quite difficult so I decided to do it the easy way - put the file content as the body and just stream that up.

So this lets the browser do all the hard work instead... It allows you to select a file and upload it using the PUT method... insteading pasting the file contents. It should allow binary files too.

Next steps are to clean up the look, and allow folder creation and folder selection.....

/*
Proof of concept fs upload file ESP32 / Linux
(c) 2017 Wilberforce

*/

function post_file(req,res){
  console.log("post_file",req.url);
  var f = E.openFile(req.url,"w");
  req.pipe(f,{chunkSize:512, end:false,
              complete:function(){
                console.log("Complete");
                f.close();
                res.end("pipe upload done");
              }
             });
  res.end();
}


var html_index=`
<html>
<head>
<title>Uploader</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/js/bootstrap.min.js"></script>
<link href="https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.3.7/css/bootstrap.min.css" rel="stylesheet">
</head>
<body>

<form action="" method="post">
    <input type="file" name="user_file" />
    <button type="submit">Submit</button>
</form>

<script>
var ajaxFileUpload = function (filename,data) {
    var xhr = new XMLHttpRequest();
    xhr.open("PUT", "http://192.168.15.18:88/node_modules/"+filename, true);
    xhr.addEventListener("load", function (e) {
        // file upload is complete
        console.log(xhr.responseText);
    });
    //xhr.send(data);
 xhr.send(data);
};

var form = document.querySelector("form");
//debugger;
form.addEventListener("submit", function (e) {

//debugger;
var input = document.querySelector('input[type="file"]');
var fdata = new FormData();
var file;

    file = input.files[0];
    //fdata.append(file.name, file);

//var file = $('#load-file')[0].files[0];
//debugger;
  var fileReader = new FileReader();
  fileReader.onloadend = function (e) {
    var arrayBuffer = e.target.result;
    var fileType = $('#file-type').val();

  //debugger;
  ajaxFileUpload(file.name,arrayBuffer);

  };
  fileReader.readAsArrayBuffer(file);

    // Prevents the standard submit event
    e.preventDefault();
    return false;
}, false);
</script>

</html>
`

function page_index(req,res) {
  res.writeHead(200);
  res.end(html_index);
  return 200;
}

var http = require("http");
  http.createServer(function (req, res) {
    console.log({header:req.header});
    console.log({method:req.method});
    console.log({url:req.url});
    code=404;
    if(req.url=='/' && req.method == 'GET') code=page_index(req,res);
    
    if(req.url=='/post' && req.method == 'POST') code=post_file(req,res);
    if(req.url=='/put' || req.method == 'PUT') code=post_file(req,res);
    if ( code == 404 ) {
      res.writeHead(404);
      res.end('Not Found');
    }
  }).listen(88); 
// Not found

fs=require("fs");
if ( typeof(fs.readdirSync())==="undefined" ) {
  console.log("Formatting FS");
  E.flashFatFS({format:true});
}
console.log(fs.readdirSync());
console.log(fs.readdirSync('node_modules'));

@Wilberforce
Were you ever able to to upload files to your espruino with the pipe?

I am following your example, but I am not seeing the complete file save in the flash.

File index.htm size is 4752, printed POST data is "cRcv": 4752, but file in flash does not match.

I know that the problem is not in the filesystem, because using "multipart/form-data" on the form post I get the complete file... plus some added strings like this:

------WebKitFormBoundaryZ2Lus84JA68itxnM
Content-Disposition: form-data; name="file"; filename="index.html"
Content-Type: text/html

...html file contents...

------WebKitFormBoundaryZ2Lus84JA68itxnM--

httpSRq: { "type": 1,
  "res": httpSRs: {  },
  "svr": httpSrv: { "type": 1,
    "#onconnect": function (b,d) {if(b.headers.Connection&&0<=b.headers.Connection.indexOf("Upgrade")){var f=b.headers["Sec-WebSocket-Key"];f=btoa(E.toString(require("crypto").SHA1(f+"258EAFA5-E914-47DA-95CA-C5AB0DC85B11")));d.writeHead(101,{Upgrade:"websocket",Connection:"Upgrade","Sec-WebSocket-Accept":f,"Sec-WebSocket-Protocol":b.headers["Sec-WebSocket-Protocol"]});var g=new e(void 0,{masking:!1,connected:!0});g.socket=d;b.on("data",g.parseData.bind(g));b.on("close",function(){clearInterval(g.srvPing);g.srvPing=void 0;g.emit("close")});
g.srvPing=setInterval(function(){g.emit("ping",!0);g.send("ping",137)},g.keepAlive);c.emit("websocket",g)}else a(b,d)},
    "#onwebsocket": function (ws) {print(ws);
        ws.on('message', msg => { print("[WS] " + JSON.stringify(msg)); });
        ws.on('close', evt => { print("[WS] CLOSED"); });},
    "port": 80, "sckt": 4098 },
  "sckt": 4099,
  "headers": {
    "Host": "192.168.25.8",
    "Connection": "keep-alive",
    "Content-Length": "4752",
    "Origin": "http://192.168.25.8",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.53 Safari/537.36",
    "DNT": "1",
    "Accept": "*/*",
    "Referer": "http://192.168.25.8/upload",
    "Accept-Encoding": "gzip, deflate",
    "Accept-Language": "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,it;q=0.6,ja;q=0.5,es;q=0.4"
   },
  "cRcv": 4752,
  "method": "PUT",
  "url": "/upload",
  "hdrs": true,
  "dRcv": ""
 }

>fs.statSync("upload")
={ "size": 3216, "dir": false,
  "mtime": Date: Mon Feb 1 2077 00:00:00 GMT+0000
 }

        if (a.pathname == "/upload") {
            print(req);
            if (req.method == "PUT") {
                try {
                  fs.unlink(a.pathname);
                }
                catch (e) { }
                var f = E.openFile(a.pathname, "w");
                req.pipe(f, {
                    chunkSize: 512, end: false,
                    complete: () => {
                        f.close();
                        res.end();
                    }
                });
            }
            else {
                res.writeHead(200, {'Content-Type':'text/html'});
                res.end(html_index);
            }
        }

I had some code for handling multipart form data here if it helps:

var fileNo = 0;

function onPageRequest(req, res) {
  var d = "";
  var boundary;
  var f;
  req.on('close', function() { 
    if (f) {
      f.close();
      console.log("File uploaded");
    }
  });
  req.on('data', function(data) { 
    d += data;
    if (f) {
      var idx = d.indexOf(boundary);
      /* ensure we always leave some in our buffer
      in case boundary straddles packets */
      if (idx>=0) {
        console.log("Boundary found - starting again");
        f.write(d.substring(0, idx));
        f.close();
        f = undefined;
      } else {
        idx = d.length-boundary.length; 
        f.write(d.substring(0, idx));
      }
      d = d.substring(idx);
    } else {
      var idx = d.indexOf("\r\n\r\n");
      if (idx>=0) {
        var header = d.substring(0,idx).split("\r\n");
        boundary = "\r\n"+header.shift();
        header = header.map(function(x){return x.split(": ");});
        console.log(JSON.stringify(header));  
        // choose a filename
        var fileName = "f"+fileNo+".txt";
        console.log("Writing to "+fileName);
        f = E.openFile(fileName, "w");
        fileNo++;
        d = d.substring(idx+4);
      }
    }
  });
  res.writeHead(200, {'Content-Type': 'text/html'});
  res.write('<html><body><form method="post" enctype="multipart/form-data">');
  res.write('<input type="file" name="file1" id="file"><input type="file" name="file2" id="file"><input type="submit">');
  res.end('</form></body></html>');
}
require('http').createServer(onPageRequest).listen(8080);

Thank you Gordon. I will give it a try.

I saw that the espruino tool is capable of placing a file in the storage area, but it seems that this is only for the original espruinos.

EDIT: It worked perfectly. Had to make 'var boundary = "";' else it would complain about it not having the .length property.

That's strange - boundary.length should only get used if f is set, by which point boundary should be set up as well.