Extending the File Server Example to Load and Run HTML, CSV, and JS(called from script in HTML)

Posted on Wed 13th, April 2016

• #1

This runs on an Espruino board connected to an ESP8266 running the AT commands.
The ESP8266 is on a separate 3.3V supply and connects to Serial port 4 of the Espruino board
Put an HTML and a CSV file on an SD card and insert it into the Espruino board,
Load the following code into the Espruino board via USB.
Open a browser and use the IP address to access the hardware via Wi-Fi.
Example:
http://192.168.1.3:8080/

You should see a directory of the SD card.
If you click on an HTML file, it will load the Webpage instead of displaying the content as text.
If the HTML file loads a .js file on the SD card it should work as well.

From the SD card directory, if you clicl on a .csv file, your browser will launch Excel and download the csv file into a workbook. You can then save the data to your hard drive.

The code follows:

//Use for Espruino board wired to AT ESP8266
//Uses HTTP to access and load a web site and display on console
var serial=Serial4;
var SSID="ssid";
var key= "passcode";
var filename,ext;
serial.setup(115200, { rx: C11, tx : C10 });

var wifi = require("ESP8266WiFi_0v25").connect(serial, function(err) {//5
  if (err) throw err;

  wifi.reset(function(err) {//4
    if (err) throw err;
    console.log("Connecting to WiFi");

    wifi.connect(SSID,key, function(err) {//3
      if (err) throw err;
      console.log("Connected");
      // Now you can do something, like an HTTP request
        var l="";
      serveHTML();
    });//3
  });//4
});//5

function onPageRequest(req, res) { 
  var a = url.parse(req.url, true);
if (req.method=="POST") {
  console.log("Post");
  doPost(req,res);
}else{
  doGet(req,res);
}//endif
}//end on PageRequest

function doPost(req,res){  //Stub function for now
  var info = url.parse(req.url, true);// console.log("res= ",res);
    res.writeHead(200);
    res.end();
 
}//end doPost

function doGet(req,res){
  var a = url.parse(req.url, true);
  filename=a.pathname;
  ext=filename.split(".").pop();
  console.log("Get"+filename);
  if (a.pathname.substr(-1)=="/") { // a slash at the end, list the directory
    res.writeHead(200, {'Content-Type': 'text/html'});
    res.write("<html><body><p>Contents of "+a.pathname+"</p><ul>");
    require("fs").readdir(a.pathname.substr(0,a.pathname.length-1)).map(function(f) { 
      res.write('<li><a href="'+a.pathname+f+'">'+f+'</a></li>');
    });
    res.end("</ul></body></html>");
  } else { // No slash, try and open file
    var f = E.openFile(a.pathname, "r"); //1
    if (f !== undefined) { // File open succeeded - send it!
console.log(ext);

switch (ext){
 case "html":
    res.writeHead(200, {'Content-Type': 'text/HTML'});
    f.pipe(res); // streams the file to the HTTP response
 break;
 case "js":
    res.writeHead(200,{'Content-Type':'text/javascript'});
    f.pipe(res); // streams the file to the HTTP response
 break;
 case "csv":
    res.writeHead(200,{'Content-Type':'text/csv'});
    f.pipe(res); // streams the file to the HTTP response
 break;
 default:
      res.writeHead(200, {'Content-Type': 'text/plain'});
      f.pipe(res); // streams the file to the HTTP response
 break;

}//end switch ext      

  } else { // couldn't open file //1
      // first check if this was a directory             
      if (require("fs").readdir()!==undefined) { 
        // it was a directory - forward us to a page with the '/' on the end
        res.writeHead(301, {'Location': a.pathname+"/", 'Content-Type': 'text/plain'});  
        res.end("Moved");                                
      } else {                                              
        // else not found - send a 404 message
        res.writeHead(404, {'Content-Type': 'text/plain'});
        res.end("404: Page "+a.pathname+" not found");
      }
    }
  }
}


function serveHTML(){
var http=require("http").createServer(onPageRequest).listen(8080);
}//end serveFile

Note the POST function is a stub for now.

These links may be of use in the switch statement
https://en.wikipedia.org/wiki/Media_type
https://www.iana.org/assignments/media-types/media-types.xhtml#application
https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

• #2

ClearMemory041063
Current files are attached.
The html file goes on the SD card to be served by the JS file to the client.
The JS file serves the files and responds to the POST request in the html page.
The req.read, req.on('data') methods don't seem to work.
I stuffed the POST message in the header. On the server side the message from the header has to have octal characters parsed back to the original message.
Any help with getting the POST message to show up will be greatly appreciated.

2 Attachments
- JPpostT8.html
- serveHTMLpost8Useurl.js
• #3

Gordon

Looks good!

About post - I've definitely had req.on('data') working fine, even for handling uploads of multiple files. Is your Espruino firmware up to date?

Do you just get no calls to it at all? Perhaps you could try POSTing from something like curl on your PC, just to make sure it's not something with the way you're POSTing from the web browser?

To be honest for simple on/off stuff I've just been encoding it on the URL (/set?light=on/off) and decoding it with url.parse - I don't think there are huge problems with that as long as POST is used?
• #4

ClearMemory041063

Hi Gordon,
I'm running 1v85 using the WebIDE. The ESP8266 AT reply is as follows.
1v85 Copyright 2016 G.Williams

echo(0);
=undefined
""
"AT+GMR\r\r\nAT version:0.40.0.0(Aug 8 2015 14:45:58)\r\nSDK version:1.3.0\r\nAi-Thinker Technology Co.,Ltd.\r\nBuild:1.3.0.2 Sep 11 2015 11:48:04\r\nOK\r\n"
")m\xE9M\xAAK\xF8"
Done!

One thing I've tried to troubleshoot the POST problem is to use a
serial.on('data') to grab everything coming from the client.
I get the POST header followed by the POST message. The client is sending it all.
A req.read() gets the POST message with varying results. About 1 in 7 tries I get the whole message. The req.on('data') just isn't working and should have fixed the problem. I'm open to suggestions including a bifocal error on my part. If needed I can run some diagnostics and post them.

I set out to modify a program I used on Android using SL4A that reads analog inputs and plots them on a strip chart on the client. It makes use of the Flot graphics. The HTML loads a Flot JS file which is rather large even in minified form.

http://www.flotcharts.org/

I stubbed the Android calls in the HTML program, put the HTML and Flot JS files on the SD card. It runs but takes several minutes to load at 115k Baud. I've yet to try loading the Flot JS file from the client side which should speed up the loading. Otherwise it's create a light version of a strip chart.
• #5

ClearMemory041063
Found the bug
It was in the Get section of the code. I know the light was better in the POST code:)
File Server Example
http://www.espruino.com/http_file_server
```
switch (ext){
 case "html":
    res.writeHead(200, {'Content-Type': 'text/HTML'});
    f.pipe(res); // streams the file to the HTTP response
 break;
```
Notice that in the example fileserver code and my program there is no res.end().
Both produce an obscure error:

Uncaught Error: Expecting a function to call, got Number

By changing the program as follows:
```
case "html":
    res.writeHead(200, {'Content-Type': 'text/HTML'});
    f.pipe(res,{chunkSize:4096,end:true,complete:res.end()}); // streams the file to the HTTP response
 break;
```
The error goes away. I first tried 512 instead of 4096 and only got a partial load of my HTML file. The 4096 is the length of the HTML file. Does the chunkSize parameter need to be the file length?
So now the POST code req.on('data') works.
```
var pdata="";
function onPageRequest(req, res) { 
  var a = url.parse(req.url, true);
  req.on("data", function(d) {pdata+=d;});
```
Pdata= {"name":"LED2","a":true,"cmd":"pobj.a=!pobj.a;digitalWrite(LED2,pobj.a)"}{"name":"LED2","a":0,"cmd":"pobj.a=!pobj.a;digitalWrite(LED2,pobj.a)"}

So there are two questions.
1. Is the chunkSize the file length?
2. The res.end() worked, what about closing the file after the Pipe finishes?
• #6

Gordon
I think:
```
    f.pipe(res,{chunkSize:4096,end:true,complete:res.end()}); 
```
Is the problem, and would explain Uncaught Error: Expecting a function to call, got Number

What happens is by typing res.end() you are executing res.end right then and setting complete to what it returns (which is almost certainly not a function). What you want is:
```
   f.pipe(res,{chunkSize:4096,end:true,complete:function() {res.end()}}); 
```
It sounds like you weren't actually doing that at the time - but maybe you're doing something very similar somewhere else.

Definitely calling res.end() seems like a bad idea to me - that'll try and close the connection, which would mean that it's unlikely any posted data would get read.

Is the chunkSize the file length?

No, this should be the maximum size of a chunk of data that is sent down the pipe each time data is available. (Pipe effectively reads chunkSize bytes from the source and then writes them to the destination whenever data is available)

The res.end() worked, what about closing the file after the Pipe finishes?

Maybe do it when the connection itself closes, on res.on('close', ...)?
• #7

ClearMemory041063
Program is working now, files attached
Thanks for the suggestions.
Checking the length in the header against the length of the data collected in the on('data') event and the proceeding to process the message and reply made it work.
```
var reqq;
var ress;
function doPost(req,res){
var length;
  length=req.headers["Content-Length"];
  reqq=req;ress=res;
  req.on("data", function(d) {pdata+=d;if(pdata.length==length)doPost1(reqq,ress);});
  req.on('close',"console.log(\"Closed\",pdata);");
}
```
2 Attachments
- JPpostT8.html
- serveHTMLpost1l.js
• #8

allObjects

Some serious engineering is going on here... I like it very much. I like it very much and like it very much for Espruino!
• #9

d0773d

Good work! This will definitely come in handy for my project :-)

I would like to implement this on a Pico. Is there anything else I need to impliment along with following the File IO and SD cards example? I would try this out, but I don't have a SD breakout... So I'm only brain storming ATM.
• #10

ClearMemory041063
Taking a break
It seems the WEB IDE doesn't want to load the ESP8266 module tonight for some reason. Other files get sent to the board and run.
Thanks Gordon for the pipe suggested arguments. The weird error occurs even using the original Example f.pipe(res); // streams the file to the HTTP response
I have a Pico and ESP8266 but no shim as yet. Sparkfun has mico SD card breakout boards or you can use an adapter as shown on the Espruino site. I'm curious to find out if this code would work on the Pico as well.

I was working on the GET side of the code with the pipe function, it's arguments and the error message that appears. It seems this didn't fix the POST bug like I thought. Just getting a better handle on how the on('data') function works was part of the learning curve that fixed the problem.

I did manage to edit the HTML file to remove the message from the header and replace it with "CMD". The message gets posted after the header. This change works.
Leaving the field as "" doesn't work. I have considered using this field in a switch statement so that other types of POST message could be developed. Candidates might be delete a file, or write a file...
```
//yhttp.open("POST",sendText, true);
yhttp.open("POST","CMD", true);
```

Post a reply
- Bold
- Italics
- Link
- Image
- List
- Quote
- code
- Preview
Formatting Help

Don't worry about formatting, just type in the text and we'll take care of making sense of it. We will auto-convert links, and if you put asterisks around words we will make them bold.

Tips:

Create headers by underlining text with ==== or ----

To *italicise* text put one asterisk each side of the word

To **bold** text put two asterisks each side of the word

Embed images by entering:
![](https://www.google.co.uk/images/srpr/logo4w.png)
That's the hard one: exclamation, square brackets and then the URL to the image in brackets.

* Create lists by starting lines with asterisks

1. Create numbered lists by starting lines with a number and a dot

> Quote text by starting lines with >

Mention another user by @username

For syntax highlighting, surround the code block with three backticks:

```
Your code goes here
```
Just like Github, a blank line must precede a code block.

If you upload more than 5 files we will display all attachments as thumbnails.

For a full reference visit the Markdown syntax.

Extending the File Server Example to Load and Run HTML, CSV, and JS(called from script in HTML)

Formatting Help

About

Extending the File Server Example to Load and Run HTML, CSV, and JS(called from script in HTML)

Actions

Espruino