Cannot download large StorageFiles [Resolved]

Posted on
  • My Python script to download StorageFiles seems to have gone bust completely. It now times out waiting for first byte. I changed the previous timeout from 10 seconds to upto 30 seconds and it still didn't kick off. In the last attempt I got this in the file instead of actual data.

    Execution Interrupted
    at line 2 col 21
      line = f.readLine();
                        ^
    New interpreter error: MEMORY_BUSY
    
    

    The command sent to the Bangle (via Python/Bleak) is

    getFileCommand = b"\x03\x10\
    var f = require('Storage').open('ftclog', 'r');\n\x10\
    var line = '';\n\x10\
    while ((line != null && line != undefined) && (line.indexOf('\xFF') == -1)){\n\x10\
      line = f.readLine();\n\x10\
      print(line);\n\x10\
    }\n\x10\
    "
    

    I also tried using the WebIDE and got similar error. At the time I wasn't sure if it was because I had initiated another connect on a different browser tab.

    getFileCommand = b"\x03\x10\
    var f = require('Storage').open('ftclog', 'r');\n\x10\
    var line = '';\n\x10\
    while ((line != null && line != undefined) && (line.indexOf('\xFF') == -1)){\n\x10\
      line = f.readLine();\n\x10\
      print(line);\n\x10\
    }\n\x10\
    "
    

    In the WebIDE, atleast it spent good 10 minutes downloading the blurb before failing.

    I will try with a longer timeout in the evening in my script.

    Firmware details:

    {
      VERSION: "2v04.375",
      GIT_COMMIT: "9b362663",
      BOARD: "BANGLEJS",
      FLASH: 524288, SPIFLASH: 4194304, STORAGE: 4194304, RAM: 65536,
      SERIAL: "a0cf0da6-7b6c234d",
      CONSOLE: "Bluetooth",
      MODULES: "Flash,Storage,hea" ... "le,graphical_menu",
      EXPTR: 536883676 
    }
    

    P.S. This may not be related to the firmware refactor. I started having this issue on Friday when I was running 2v04.99

    Full Python script on Github

  • Please can you try doing a storage.eraseAll? I know the latest firmware added longer filenames and increased the usable memory area to 4mb and that meant that the storage area needed to be completely wiped.

    Also, can you view the file via the web IDE storage dialog?

  • Thanks @Gordon. The WebIDE also failed with similar error.

    I failed to download the large file and ended up doing an eraseAll. Bangle had nearly frozen when the file was that big.

    I am currently on firmware 4.375 and have turned on HRM and hopefully will get about 10-15 minutes of GPS info as well by the time I get home. It will be interesting to see how big a file I can push, now that the whole of 4Mb is available. I'll probably do an upper limit of an Mb or so.

    I currently have two statuses, one is diskfull and other is error. Funnily enough I had hit Error today not diskfull. Looking forward to redoing that bit using the new length call.

    Sorry for pushing the boundaries and keeping things on the edge for you :-)

  • @Gordon

    Looks like there is an issue with readLine() in StorageFile

    I am getting this error:

    Execution Interrupted
    at line 2 col 21
      line = f.readLine();
                        ^
    New interpreter error: LOW_MEMORY,MEMORY
    

    WebIDE also fails with this error:

    index.js:93593 Uncaught TypeError: Cannot read property 'length' of undefined
        at Object.isASCII (index.js:93593)
        at showViewFileDialog (index.js:131556)
        at index.js:131654
        at index.js:131329
        at nextStep (index.js:93250)
        at onTimeout (index.js:93281)
    

    I get a massive blurb output in the IDE (check image of the debug window attached, got the same the output in the WebIDE console).

    Update: The ftclog log file is '80 shards long'.

    Cheers.


    1 Attachment

    • Screen Shot 2020-03-02 at 9.02.23 PM.png
  • What's the JS code on the watch? Is it possible that you're just never writing any newline characters, so readLine is attempting to read a 1MB long line of text?

  • oh my... 'blushes furiously'... you are right. This one is down to 1D-10-T error :D

  • @Gordon I understand why the readline is failing but why is the read() failing? Is the file too big for length ?

  • Honestly I'm a bit stuck. I've tested here and it works fine.

    All I have from you is 'it's not working' - and I can't even see a reference to any code you've got that is using .read?

    If you did call .read with a bigger length than you have free RAM then yes, it'd likely fail with an out of memory error though...

    Please could you see if you can produce a single JS file that you can upload that'll cause the problem? Maybe start from this and see what you have to do to get it to break.

    f = require('Storage').open("foobar","w");
    f.write("Hell");
    f.write("o World\n");
    f.write("Hello\n");
    f.write("World 2\n");
    
    var f = require('Storage').open('foobar', 'r');
    var line = '';
    while (line !== undefined) {
      line = f.readLine();
      print(line);
    }
    

    Just a note about your code - there's no need to do all those checks in the while loop. Just a single !==undefined will work just fine

  • ... also I notice you've ended your code with \n\x10\ - the 0x10 will mean that the next line you send after the command doesn't output anything on the console, which may make it seem like Espruino is unresponsive.

  • @Gordon Apologies, for not providing code reference.

    My code for readline is now working as expected after I added the \n back to the code. Don't remember how it went missing but my bad.

    I don't have a read in my code. When I was mistakenly writing logs without linebreaks, the WebIDE functionality of downloading a file was also failing.

    My question should have been framed, why was read function used by WebIDE failing to download a large file.

    var i = 10000;
    var file = require("Storage").open("test", "a");
    while (i > 0){
      var sentence = "Long string of gibberish";
      file.write(sentence);
      i--;
    }
    
    file = null;
    

    If you create a file using the above code and try to download the file using WebIDE, the download fails.

    Thank you very much for your stellar support and responses so far. Did not mean to annoy you.

  • No problem - I'm happy to help out and it's good to have someone pushing things. I just need enough information to be able to help out :)

    So... I just copied the 'read file' code from the Web IDE and added it to your example:

    function go() {
      console.log("writing");
      var i = 10000;
      var file = require("Storage").open("test", "w");
      while (i > 0){
        var sentence = "Long string of gibberish ";
        file.write(sentence);
        i--;
      }
      file = null;
    
      console.log("reading");
      var f = require("Storage").open("test","r");
      var d = f.read(384);
      while (d!==undefined) {
        console.log(btoa(d));
        d = f.read(384);
      }
    }
    

    And that appears to work fine too, as does the IDE's 'load file' menu (ish). The IDE file loading itself works fine, but the file takes so long to download that the IDE times out waiting for it all to come in and then the IDE didn't handle that very well. I've just made some tweaks (not published yet) which will at least output a proper error in the case that that happens...

  • Thanks for looking into it @Gordon. I just did a "Clean cache and hard reload" on the WebIDE and tried to download the test file. It gave me the following error in console.

    index.js:93593 Uncaught TypeError: Cannot read property 'length' of undefined
        at Object.isASCII (index.js:93593)
        at showViewFileDialog (index.js:131556)
        at index.js:131654
        at index.js:131329
        at nextStep (index.js:93250)
        at onTimeout (index.js:93281)
    

    I guess this is the timeout you are referring to in #11. I mistook that as an issue with the length call. So all is well now. Marking this one as resolved.


    1 Attachment

    • Screen Shot 2020-03-03 at 4.27.04 PM.png
  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Cannot download large StorageFiles [Resolved]

Posted by Avatar for PiOfThings @PiOfThings

Actions