• Okay, apologies in advance for putting out a post without clearly reproducible code. But maybe someone has seen this before and can help.

    I am able to write and delete to Bangle JS using the StorageFile api.

    Currently my code displays ERROR on screen when my file has tried to write over an MB and failed. Thereafter it doesn't try to write anymore until the file has been dowloaded and the old file deleted. Following are bits of code that do these things

    Bangle.AppLog = {
                currentFile: null,
                shardCount: 0,
                currentFileName: "",
                error: false,
                init: (filename) => {
                    Bangle.AppLog.currentFileName = filename;
                    try{
                        if(filename != null && filename != ''){
                            Bangle.AppLog.currentFile = require("Storage").open(filename, "a");
                            console.log("File created: " + filename);
                        }
                        else{
                            console.error("Log file name not provided");
                        }
                    }
                    catch(ex){
                        console.log("Failed to create file", ex);
                    }
                },
                write : (sentence) => {
                    if(Bangle.AppLog.error != true){
                        try{
                            Bangle.AppLog.currentFile.write(sentence­);   
                        }   
                        catch (ex){
                            console.log(ex);
                            Bangle.AppLog.currentFile = null;
                            Bangle.AppLog.error = true;
                        }              
                    }
                },
                clearLog: () => {
                    Bangle.AppLog.currentFile = null;
                    require("Storage").open(Bangle.AppLog.cu­rrentFileName, "w").erase();
                    Bangle.AppLog.error = false;
                },
                beginSync: () => {
                    var f = require('Storage').open(Bangle.AppLog.cu­rrentFileName, 'r');
                    var line = '';
                    while (line != null && line.indexOf('\xFF') == -1){
                        line = f.readLine();
                        if(line != null){
                            print(line);
                        }
                        else{
                            break;
                        }
                    }
                    print("<!-- finished sync -->");
                    f=null;
                }
            };
    

    Every so often (every 1 out of 3 tries) after I have deleted the old file by calling Bangle.AppLog.clearLog() from the WebIDE - the app will try to (re)create the file, (it doesn't throw any error when calling open(...)), but fails when trying to write to the file.

    >Bangle.Helper.size("fitclock.log");
    0
    =undefined
    Trying to create new file:fitclock.log
    File created: fitclock.log
    Error: Error: Unable to find or create file
    Trying to create new file:fitclock.log
    File created: fitclock.log
    

    As seen from the above log entries to the console, init() doesn't throw an error but the first write does.

    Unfortunately, once the storage gets into this state, I cannot get Bangle to write to any file with any other name. Rebooting using "Button 1 & 2 press" doesn't help. I have to to Storage.eraseAll() to get back ability to write to file system again. All of them throw the same "Unable to find of create file" error.

    >
    >
    >
    >var f = require("Storage").open("test", "a");
    =StorageFile: {
      name: "test",
      chunk: 1, offset: 0, addr: 0, mode: 97 }
    >f.write("Testing");
    Uncaught Error: Unable to find or create file
     at line 1 col 18
    f.write("Testing");
                     ^
    > 
    

    It is worth noting, even though StorageFile.open(...) did not throw an error, the file doesn't exist or is not recognised in the file WebIDE's file browser (ref to attached image).

    I looked through the entire repository and there is only one instance of the error message Unable to find or create file and it seems to be only present here

    That led me to the open issue... can't tell for sure but these two might be related?

    P.S. Interesting side note... there seems to be very little disk space left for real or so Bangle thinks... So the files probably didn't get deleted and their space reclaimed? That would explain the 1/3 frequency. That certainly explains why things work after eraseAll... Hmm... what am I doing wrong with my clearLog function?

    >require("Storage").getFree();
    =3144
    

    1 Attachment

    • Screen Shot 2020-03-08 at 11.25.09 PM.png
  • Hi - that open issue won't be your problem (it's for FAT on ESP32 - it's E.openFile, not Storage related).

    Does require("Storage").list() show you any files that look like they could have come from test?

    It is worth noting, even though StorageFile.open(...) did not throw an error

    That's expected - it won't actually start writing the file until you write data into it.

    require("Storage").getFree()==3144 is your main problem though. You need at least 4100(ish) bytes to start writing a file with StorageFile.

    Does require("Storage").compact() do anything to free up space?

  • I think I have in the past. I'll double check as soon as it recurs and report back with findings. I did an eraseAll last night to record the night's sleep movement and HR values, but the battery died midway. That reminds me I should merge from upstream again :-)

  • Okay, @Gordon I think compact does the trick. The following code breaks if you comment out the compact() call (line 34 below).

    *** WARNING Does a Storage.eraseAll() ***

    function repro() {
      require("Storage").eraseAll();
      for(var i = 0; i < 5; i++){
        print("Writing test file, attempt " + (i+1));
        var file = require("Storage").open("testfile.log", "a");
        try
        {
          while (true){
            var sentence = "Long string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberish\n";
            file.write(sentence);
          }
        }
        catch(ex){
          console.log(ex);
          console.log("File size after write error: " + file.getLength().toString());
          file = null;
        }
        var f = require('Storage').open("testfile.log", 'r');
        var line = '';
        print("Faking a sync please wait");
        while (line != null && line.indexOf('\xFF') == -1){
            line = f.readLine();
            if(line != null){
                //print(line);
            }
            else{
                break;
            }
        }
        print("<!-- finished sync -->");
        print("Files before Erase " + (i+1) + "\n" + JSON.stringify(require("Storage").list()­));
        print("Storage.getFree Before Erase " + (i+1) + " " + require("Storage").getFree().toString())­;
        require("Storage").open("testfile.log", "w").erase();
        require("Storage").compact();
        print("Storage.getFree After Erase and compact" + (i+1) + " " + require("Storage").getFree().toString())­;
        print("Files after Erase " + (i+1) + "\n" + JSON.stringify(require("Storage").list()­));
      }
    }
    
    repro();
    

    That's strange because I am sure I have compact in my clearLog function now but that doesn't seem to work.
    I will do more testing and report back.

    process.env
    ={
      VERSION: "2v04.398",
      GIT_COMMIT: "292f7d0d",
      BOARD: "BANGLEJS",
      FLASH: 524288, SPIFLASH: 4194304, STORAGE: 4194304, RAM: 65536,
      SERIAL: "a0cf0da6-7b6c234d",
      CONSOLE: "Bluetooth",
      MODULES: "Flash,Storage,hea" ... "tensorflow,locale",
      EXPTR: 536883676 }
    > 
    
    
  • Yup, my code doesn't seem to work (even if I try to delete the file before it has errored out). The watch was not on my wrist so it wasn't trying to record any movement.

    After dowloading the log I wanted to clear the log,

    Bangle.Helper.size("fitclock.log");
    869585
    =undefined
    >require("Storage").getFree();
    =3292892
    >Bangle.AppLog.clearLog();
    =undefined
    >require("Storage").getFree();
    =3292892
    >require("Storage").compact();
    =undefined
    >require("Storage").getFree();
    =3292892
    

    Here's the minified->uploaded->downloaded->beautifi­ed code for clearLog

    Bangle.AppLog = {
                currentFile: null,
                lock: !1,
                shardCount: 0,
                currentFileName: "",
                error: !1,
                diskFull: !1,
                init: e => {
                    Bangle.AppLog.currentFileName = e;
                    try {
                        null != e && "" != e ? (Bangle.AppLog.currentFile = require("Storage").open(e, "a"), console.log("File created: " + e)) : console.error("Log file name not provided")
                    } catch (e) {
                        console.log("Failed to create file", e)
                    }
                },
                write: e => {
                    if (1 != Bangle.AppLog.lock && 1 != Bangle.AppLog.error) try {
                        Bangle.AppLog.currentFile.write(e)
                    } catch (e) {
                        console.log(e), Bangle.AppLog.currentFile = null, Bangle.AppLog.error = !0
                    }
                },
                clearLog: () => {
                    Bangle.AppLog.lock = !0, Bangle.AppLog.currentFile = null, require("Storage").open(Bangle.AppLog.cu­rrentFileName, "w").erase(), require("Storage").compact(), Bangle.AppLog.lock = !1, Bangle.AppLog.error = !1
                },
                beginSync: () => {
                    for (var e = require("Storage").open(Bangle.AppLog.cu­rrentFileName, "r"), t = ""; null != t && -1 == t.indexOf("ÿ") && null != (t = e.readLine());) print(t);
                    print("\x3c!-- finished sync --\x3e"), e = null
                }
            }
    

    I am not sure if my minification using gulp-terser is mangling code in some way. I'll test with direct require("Storage").open('...','w').erase­() from console tomorrow morning.

  • Okay... I used the WebIDE to delete the new file and that didn't help either

    >require("Storage").getFree();
    =2686684
    >require("Storage").open("fitclock.log",­"w").erase();
    =undefined
    >require("Storage").getFree();
    =2686684
    >require("Storage").compact();
    =undefined
    >require("Storage").getFree();
    =2686684
    > 
    

    Next I'll try to set the file handle to null before deleting it.

  • What's in Storage.list()? Is it possible the file is called something other than "fitclock.log"?

  • Okay @Gordon... I have finally got the reason why my App code is failing and sample code is working. I decided to read the compact documentation more closely to realise it fails if there isn't enough RAM. Digging through the code showed me it needs atleast allocated + 1024 to continue with compact. I haven't traced it as far as how much is allocated but when I have my app loaded and I run process.memory() I get this:

    process.memory();
    ={ free: 1036, usage: 1064, total: 2100, history: 7,
      gc: 0, gctime: 3.72314453125, blocksize: 16, "stackEndAddress": 536923400, flash_start: 0,
      "flash_binary_end": 483876, "flash_code_start": 1073741824, flash_length: 524288 }
    

    Even if allocated is 126 or 256, we can see the amount of free memory (1036) isn't enough for compact to continue.

    However, the reason my sample above kept working is because on a Bangle with everything erased, if I run process.memory() I get the following:

    >process.memory();
    ={ free: 2056, usage: 44, total: 2100, history: 7,
      gc: 0, gctime: 2.99072265625, blocksize: 16, "stackEndAddress": 536923400, flash_start: 0,
      "flash_binary_end": 483876, "flash_code_start": 1073741824, flash_length: 524288 }
    > 
    

    This means, require("Storage").compact() is ineffective if you have Bootloader + Default Launcher + Battery Widget + BT Widget + Settings and 1 clock App.

    How we resolve this I am not sure, maybe after erase compact is called without the check for minimum RAM? Will that work? I have to dig deeper, but essentially we have an issue where we are unable to realise the full potential of Bangle and the storage it has.

  • Sorry, didn't see this reply earlier (I didn't refresh the page oops). No, the filename is correct. To verify the glitch, have a Bangle watch with bare minimum apps (Bootloader + Default Launcher + Battery Widget + Morph Clock App). Now run the following code (same as above without the erase all and dummy sync)

    function repro() {
      
      for(var i = 0; i < 1; i++){
        print("Writing test file, attempt " + (i+1));
        var file = require("Storage").open("testfile.log", "a");
        try
        {
          while (true){
            var sentence = "Long string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberishLong string of gibberish\n";
            file.write(sentence);
          }
        }
        catch(ex){
          console.log(ex);
          console.log("File size after write error: " + file.getLength().toString());
          file = null;
        }
        
        print("Files before Erase " + (i+1) + "\n" + JSON.stringify(require("Storage").list()­));
        print("Storage.getFree Before Erase " + (i+1) + " " + require("Storage").getFree().toString())­;
        require("Storage").open("testfile.log", "w").erase();
        require("Storage").compact();
        print("Storage.getFree After Erase and compact " + (i+1) + " " + require("Storage").getFree().toString())­;
        print("Files after Erase " + (i+1) + "\n" + JSON.stringify(require("Storage").list()­));
      }
    }
    repro();
    
    

    You will see getFree before and after erase doesn't make a difference even though the testfile.log file gets removed from the list().

  • process.memory().free shows the amount of memory available in blocks, which are 16 bytes each - so you actually have 16k or more free. But maybe you need to have less than that amount in flash for it to be able to successfully compact.

    Either way this looks like a potential problem. I think Storage assumes that if there's a blank page then that's the end of the written Storage, which causes a problem when you go to erase a big file. I think I'll have to put a bit of work into trying to figure out how to tweak it.

  • I think I'll have to put a bit of work into trying to figure out how to tweak it.

    That's cool, no rush. I'll continue with the eraseAll workaround for now :-)

    I am just glad I was able to narrow it down to be easily reproducible.

  • Ok, there's a bug open for another vaguely related Storage issue here so let's handle stuff there for now: https://github.com/espruino/Espruino/iss­ues/1707

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Occasional StorageFile errors, needing an 'eraseAll'

Posted by Avatar for PiOfThings @PiOfThings

Actions