Readonly FileSystem module for ESP8266 and maybe else

When I started learning Espruino, I saw that there is a module "Storage" for working with files. But to my surprise, the my firmware set a limit on the use of the first 196Kb.

require("Storage").getStats() returns { totalBytes: 196608, ... }

I have an existing project on C++ with web interface does not fit into this limitation and so I decided to make my own FS with blackjack and w... The ESP8266 has flash memory on board with ~3MB of free space and we use it. This is a simple FS and I am limited to the write and read functions. Generally, files will not be overwritten. These files are supposed to be the content for the web page: css/js frameworks, png/jpg images, etc.

This is alpha version of module. To check if it works, follow these steps:
1) Upload module content in storage. I use IDE for that:
https://github.com/VSVLAD/ESP8266_Espruino_FS/blob/main/FlashFS.min.js

2) Let's upload a test file to our FS. Download script below in attachment. Note:
use require("Flash").getFree() to figure out which areas in flash are empty and can be used. Study this very carefully, and even better study a real dump of your flash memory, because. information in my case is not always true! In case of problems, you will need to reflash using esptools...

3) Read file in pipe very well

// Create some object for pipe
function someObject() {
	this.write = function(buffer){
   		console.log("console: " + buffer);
   };
}

// prepare object
let flash = require("Flash");
let fsv = require("FlashFS.min.js")(flash, 1048576, 2097152);
let f = fsv.openFile("Lorem", "r");
let co = new someObject();

// Pipe is working!
E.pipe(f, co, {chunkSize: 32, complete: function(){ console.log("finished")}});

also work with methods:

let f = fsv.openFile("Lorem", "r");
f.read(11);
f.seek(0);
f.readBytes(11);
f.seek(0);

Note: esp8266 has wdt and if you read many bytes, you need f.pipe or read in small portions, else your board will reset. You'll see message and cause reason in console.

A more interesting option is testing file reading and network transmission:

let flash = require("Flash");
let fsv = require("FlashFS.min.js")(flash, 1048576, 2097152);
let wifi = require("Wifi");
let http = require("http");

function onPageRequest(req, res) {
    var u = url.parse(req.url, true);

    if (u.pathname == "/"){
		let loremFile = fsv.openFile("Lorem", "r");

        res.writeHead(200, {"Content-Type": "text/html"});
		E.pipe(loremFile, res, {"chunkSize": 128, complete: function(){console.log("completed!")}});

    } else {
		res.writeHead(404, {"Content-Type": "text/plain"});
    	res.end("Not found");
	}
}

wifi.connect("SOME_AP", {password: "*****"}, function(err){
    if (!err){
        console.log(wifi.getIP());
    } else {
        console.log(err);
    }
});

http.createServer(onPageRequest).listen(80);

This also works, although the download speed of the file is not very fast. If "chunkSize": 128, then the download takes 4 seconds. Perhaps it can be optimized, I have not figured it out yet...

Thanks for reading :)

1 Attachment

• Lorem.txt

But to my surprise, the my firmware set a limit on the use of the first 196Kb

Storage module is for directly accessible flash. AFAIK for ESP8266 only 1MB is directly mapped to CPU memory so that 196KB is probably some free space below 1MB.

The rest over 1MB needs to use some filesystem and looks like the board file has this commented out here https://github.com/espruino/Espruino/blob/master/boards/ESP8266_4MB.py#L31 unlike ESP32 board.

I think this is FAT filesystem on top of flash which would be OK for your mostly read only use case. Not really sure why it is disabled by default.

BTW, I just tried to build it with those lines uncommented and it does build fine. Also briefly tested in esp8266 4MB board and it seems to work somehow, the key is to use this http://www.espruino.com/Reference#l_E_flashFatFS and format the filesystem. By default it uses 250 4KB sectors near the end of flash as per https://github.com/espruino/Espruino/blob/master/libs/filesystem/fat_sd/flash_diskio.h#L24

Then one can use fs module or http://www.espruino.com/Reference#l_E_openFile

EDIT: build attached

1 Attachment

• espruino_2v15.54_esp8266_4mb.tgz

Thanks for compiling the firmware! FAT FS really works on ESP8266 4Mb. I tested a little bit (on selected range of 2Mb area) on file operations and it work as they should...

Initially, I checked that the E.flashFatFS method did not exist in the "official" firmware version 2v15, so I began to develop my own FS module. If this feature were enabled by default, then it would be possible to use it and not make other decisions.