reading large files with readArrayBuffer

Posted on
  • hi,

    i'm trying to read a large file (200kb).
    (i just tested on the emulator and not the watch.)

    using : let a = require("Storage").readArrayBuffer("test­.map");
    it loads but a.length is equal to 22053.

    if i try let a = require("Storage").read("test.map", 0, 200000);
    it seems to work but i get a string and i'd rather like to turn my arraybuffer into various arrays.

    is there a size limit on readArrayBuffer and is there a reason for it ? do you know any kind of workaround i could use ?

    thanks.

  • unfortunately there is 64KB limit for arrays, may be related to this issue https://github.com/espruino/Espruino/iss­ues/2036

  • as for workaround there is E.toArrayBuffer so maybe it could work on <64K substring so you could access big string in 64KB chunks as array.

    EDIT: I just tested it in RAM, hopefully it will work fine with flash strings too.

    s="".padEnd(65536) // 64K of spaces
    s=s.padEnd(65540,'a') // pad with few 'a' over 64KB
    
    >s.substring(65535)
    =" aaaa"
    >E.toArrayBuffer(s.substring(65535))
    =new Uint8Array([32, 97, 97, 97, 97]).buffer
    >E.toArrayBuffer(s) // this fails due to 64KB limit
    =new Uint8Array([32, 32, 32, 32]).buffer
    
  • ok, thanks for the quick answer. i think i actually may be able to have a workaround with inline c and strings

  • oh i didn't see your workaround. thanks again, i'll give it a try

  • Instead of substrings the best workaround is probably to combine Storage.read with offset,length parameters and wrap the result by E.toArrayBuffer

    As for Inline C it may not work as you expect with storage in SPI flash (Bangle watches), the 200K string is not directly in memory but read over SPI by JS interpreter on demand when iterating over it.
    It might work with Storage in internal flash but there is not much space for big files there.

  • Instead of substrings the best worlaround is probably to combine Storage.read with offset,length parameters and wrap the result by E.toArrayBuffer

    Yep, I'd say try this...

  • ok, i'll give it a try. thanks a lot

  • yeah. it's all good. thanks again.


    1 Attachment

    • gps.png
  • Wow, very cool - a vector map?

  • i extracted my city's ways from openstreetmap. with some clever encoding (btw i laughed at Uint24Array but it's just what i needed) i'm actually below 200k. i'm amazed it's fast enough to display on the watch with little optimisations (1.5 secs to display). i still need to add the street names and port the low mem path algorithm to javascript and we might have a full gps. i'm not still 100% sure the path algorithm will fit the ram though, it's kind of a challenge. well, all the stuff i like.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

reading large files with readArrayBuffer

Posted by Avatar for wagnerf42 @wagnerf42

Actions