Best way to stream base64 encoded string to a web client

I would like to serve up a minified single HTML (including CSS, JS, images, etc.) from my Espruino. To minimize space, I'm going to gzip the resulting file and use the Content-Encoding HTTP header when serving it up.

I'm also thinking about inlining this gzip file in my Espruino JS code, and base64 encode it. What is the most memory efficient way to stream this from the Espruino, and not decode the whole string into memory first?

Or is there a better idea to do this? I don't have an SD card storage, just the flash of my ESP8266 board.

New firmwares should have 'Storage' which will allow you to store complete files. It might make more sense to use that. The IDE at espruino.com/ide will work with serial devices and has a storage manager that allows you to upload files from your PC, and if you use one of the 'cutting edge' builds you can use Storage.read(file,offset,length) to stream just a bit of it each request.

Is it possible to upload to a named storage entry with the Espruino CLI? I'm currently using that with save on send to upload my code. A similar thing for this html asset would be nice.

Sat 2020.01.25

@nistvan.86 Not entirely sure about the 'this html asset' part, but would any part of this section of the reference apply?

http://www.espruino.com/Reference#Storag­e

I'm familiar with how to create Storage entries using the interpreter over the serial connection.

My problem is that with the Espruino CLI I can't seem to send other than the main script file going to .bootcde during the upload (this is done by the --config SAVE_ON_SEND=1 option).

I tried to do something like this:

./node_modules/.bin/espruino --board ESP8266_4MB -o test.js --ohex test.hex --storage html:./portal/dist/index.html --config SAVE_ON_SEND=1 ./firmware/dist/app.js

It's not possible to remove the --ohex option, and the test.js output clearly only contains the E.setBootCode part.

Even if it's the only way to do it, I can't find any instructions on how to flash this hex file to the ESP8266 (probably with the esptool.py). But I would rather not do that and use only the Espruino CLI if possible.

I think it's clear now why I'm thinking about bundling the assets to the main script file instead.

Also it's funny that even though I don't connect to any device during the above command, it mentions that I'm using an old firmware.

Espruino Command-line Tool 0.1.30
-----------------------------------

--ohex used - enabling MODULE_AS_FUNCTION
Explicit board JSON supplied: "ESP8266_4MB"
You have pre-1v96 firmware. Upload size is limited by available RAM
Writing output to test.js
Writing hex output to test.hex

Am I using an old Espruino dependency or the board definition triggers this and there's a new ESP8266 name available I'm not aware of?
Mod: yep, if I leave out the --board switch it doesn't print that.

Well, I use a two step approach

• first one js file with one write to flash using Storage for each part like
  
  • web pages
    
  • modules
    
  • config data
    
  • ....
    
• second js file for code
  

Thank you! So handcrafting something is the only way for now.

Is it okay to do something like s.write('html', [<very_long_byte_array>]); or should I split it up to let's say 5kB chunks? Also can I upload the .bootcde script manually in chunks using the Storage API?

I personally split html in logical parts like head and body as needed and use this style to keep easy to read.

s.write('html',`

........

`);

I have no experience with large data amounts you are trying to handle, max was 3k.

Also can I upload the .bootcde script

see E.setBootCode() and Boot process: (v2.0 and later) It looks for files in Storage named .boot0, .boot1, .boot2 and .boot3 and executes them in sequence.

it is possible to include values as vars, which are replaced during upload

var x = 5;
s.write('html',`
        ........
        ${x}
        ........
`);

I've written a small tool which creates a helper script, and can be executed on the board as suggested to create the environment. I'm sending the file in small 32 byte chunks as a byte array.

It goes like:

var s = require("Storage");
s.eraseAll();
s.write("index", "", 0, 15014);
s.write("index", [60, 33, 68, 79, 67, 84, 89, 80, 69, 32, 104, 116, 109, 108, 62, 10, 60, 104, 116, 109, 108, 62, 10, 10, 32, 32, 60, 104, 101, 97, 100, 62], 0);
s.write("index", [10, 32, 32, 32, 32, 60, 116, 105, 116, 108, 101, 62, 81, 55, 82, 70, 32, 77, 81, 84, 84, 32, 67, 111, 110, 102, 105, 103, 117, 114, 97, 116], 32);
// ... goes on

The problem is, that only the first write succeeds, and only the first 32 bytes can be read from the entry. If I manually execute the lines in the interpreter, only the first write returns true, others are not even returning false. Any idea what causing this? I think something fails silently in the native code.
It doesn't seem to be offset related, if I change the second write from 32 to 33 or 31 it does the same.

This works:

>s.write("test", "", 0, 20);
=true
>s.write("test", [0, 1, 2, 3, 4], 5);
=true
>s.write("test", [0, 1, 2, 3, 4], 15); 
=true
>s.read("test");
="\xFF\xFF\xFF\xFF\xFF\0\1\2\3\4\xFF\xFF­\xFF\xFF\xFF\0\1\2\3\4"

Also it seems I have more than enough memory left in storage:

>s.getFree();
=181528

Okay, I've figured it out.
This line causes the issue:
s.write("index", "", 0, 15014);
If I put the length attribute to the first chunk of the actual data it works as intended. I don't know if it's a bug or intentional, but now it works correctly. Strange that with the small example it's not an issue.