Bangle.js 2: Reproducing compaction issues

Hi,

It seems a few of you have had issues with Storage compaction breaking Storage and losing your data (specifically on Bangle.js 2). I've been trying to reproduce this for ages so I can figure out what happens and fix it, but I just can't reliably cause issues.

It's been suggested that a Bangle.factoryReset() followed by installing certain apps would cause issues, but I've tried it here - now installing over 50 apps from favourites - and I can't reproduce it.

Another issue might have been having hugely fragmented storage, and I have come up with this code which can be run in the IDE which should really fragment everything:

var l = require("Storage").list();
function frag() {
  var n = 0|(Math.random()*l.length);
  var fn = l[n];
  console.log("Fragment "+fn);
  var d = require("Storage").read(fn);
  require("Storage").erase(fn);
  require("Storage").write(fn, d);
  
  if (require("Storage").getFree() > 1000000)
    setTimeout(frag,10);
  else
    console.log("Done! Storage almost full");
}
frag();

But running that for ages (with 50 apps installed) and then running:

require("Storage").compact()

(running from the IDE will at least allow us to see if there are any error messages).

This doesn't seem to cause any issues immediately after compaction for me either.

However I did manage to experience something - it's possible that while compact works fine it doesn't clear out the pages of flash storage immediately after the last file, which would cause a "file written with different data" error. I'm going to continue to look but any other input/hints others have would be hugely appreciated.

You can check what's actually in storage at the last (and subsequent pages) using:

// Get address of last file's beginning
var fileName = require("Storage").list().pop();
var addr = peek32(E.getAddressOf(require("Storage").read(fileName)));
print(fileName, addr.toString(16), require("Storage").read(fileName).length+" bytes");
// go back to page containing the header
addr= (addr-32)&~4095
// print page contents - note that the last file may go past the end of what's labelled 'PAGE0'
for (var p=0;p<3;p++) {
  print("PAGE +"+p);
  for (var i=0;i<4096;i+=32) {
    print(addr.toString(16).padStart(8,0), require("Flash").read(32,addr).slice().map(x=>x.toString(16).padStart(2,0)).join(" "));
    addr += 32;
  }
}

I think it might help to slightly extend the contents of the app file after it has been erased then write it back. Otherwise the chances are you might just write it back to the same slot in the storage. This would be a more real world use case.

Otherwise the chances are you might just write it back to the same slot in the storage.

You don't do that. That's why the compact is there. You can't write to same place twice without block erase. So you always write to 'new' erased space. And once you are out of it you start erasing to reclaim dead space and fill it again with valid data next to each other. then you end with one free erased area and continue until it is filled again. Or did I miss your point? Could be.

What I mean is that https://www.espruino.com/Reference#l_Storage_erase just marks dead space which is unusable until you do compact()

I think I got it. I need to come up with a reliable way to reproduce, but if you have a flash page that starts with 0xFF but then doesn't end with it, then a file that fits within that area of 0xFF, compact can get in a situation where it thinks the page is empty but it's not.

It turns out you can reproduce reliably with:

function dump() {
  var addr = 0x60000000;
  for (var p=0;p<2;p++) {
    print("PAGE +"+p);
    for (var i=0;i<4096;i+=32) {
      print(addr.toString(16).padStart(8,0), require("Flash").read(32,addr).slice().map(x=>x.toString(16).padStart(2,0)).join(" "));
      addr += 32;
    }
  }
}

var s = require("Storage");
s.eraseAll();
a = new Uint8Array(5000); a.fill(255);
s.write("test1",a)
s.write("test2","This is my test that I hope will have disappeared")
a = new Uint8Array(4090); a.fill(255);
s.write("test3",a)
s.erase("test1");
s.compact();
// Get address of last file's beginning
var fileName = require("Storage").list().pop();
var addr = peek32(E.getAddressOf(require("Storage").read(fileName)));
print(fileName, "0x"+addr.toString(16), require("Storage").read(fileName).length+" bytes", "end 0x"+(addr+require("Storage").read(fileName).length).toString(16));
// shows data in middle of second page, past the end of the last file
dump(); 
// if you write now you get 'Uncaught Error: File already written with different data' 
for (var i=0;i<20;i++) require("Storage").write("newfile","Hello World "+Math.random());

So it's a tricky one because often right after a compact the storage will look ok, but it's only when you start writing new files to it that soon you'll get an error and everything falls apart.

The absolute latest firmware build should now have this fixed!

Is the fix already in 2v11.45 ?

espruino_2v11.45_banglejs2.zip	2022-01-26 12:47

Yes it is. You can see the builds sorted by git commit hashes here: http://www.espruino.com/binaries/travis/?C=M;O=D

@fanoush - looks like @Gordon has found the issue.

My point was that writing the same file size back after a deletion would not result in fragmentation. The fragmenter stress test app needed to start with a file of say 500 bytes, (maybe packed with spaces on the end), then write it back with a smaller size (say 480 bytes). This then maroons 20 bytes that have to be tracked. Repeat multiple times with different app files that are shrunk and then expanded and you would fragment the storage.

@Gordon, great news, will update my firmware and try it out.

The fragmenter needed to start with a file of say 500 bytes, (maybe packed with spaces on the end), then write it back with a smaller size (say 480 bytes). This then maroons 20 bytes that have to be tracked.

if you write file with 500 bytes, then 'overwrite' it with 480 bytes you end with 500 bytes of (tracked) dead space + 480 bytes of newly allocated space because nothing is reclaimed until you fill everything and/or compact

The fragmenter code was reading into a variable, erasing the storage, then writing the same file back from the variable, thats not the same as an overwrite which I assume (could be wrong) would skip the erase. In such situations I would expect Espruino to find the first available slot for the file which could be the same space it had before. Having said that I am guessing as I have not read the code.

My point was that a stress test needs to try to get close to what a user will do in practice. My use of the Bangle goes like this.

1) factoryRest
2) Install favourites
3) Every few days update my repo and try new apps
4) delete the new apps I dont like
5) Take updates for the apps I use a lot
6) repeat every few days for about a month
7) The run the compactor and lose apps.

In such situations I would expect Espruino to find the first available slot for the file which could be the same space it had before.

maybe re-read post #3 I would otherwise just copy paste it :-) there are no slots to fill again as you cannot erase flash with byte granularity.

nothing is reclaimed until you fill everything and/or compact

ok. quite a simple file system then. I guess a block of bytes is marked as occupied by some sort of header that records the length and that cant be changed until you compact. Not something you want to muck about with if its working.

Yes - it was designed for smaller, more tightly coupled flash devices, and to allow large contiguous areas of flash memory to be allocated (which means SPIFFs/FAT/etc isn't really an option). It's not optimal on the 4MB/8MB flash chips in Bangle.js, but it is very good on reducing flash wear.

Tried out 2.11.253. Done 3 successful compacts, no data loss. Looks good.

Really nice to see how quickly the loader opens up with 20 apps installed.

I can confirm this. Upgraded to 2.11.53 - compact worked totally fine and my BangleJs2 feels extremely fast now - I was already happy before but now it feels 10x faster :D Really great thx a lot!

Hi,
launched compaction yesterday, it took an average pair of minutes but successfully achieved and the Bangle2 seemed more responsive since then.
Thanks !

This was a nasty bug to find. It was a case of smoke without fire. I'm so glad this is fixed.
Just feels better.

hi,
compaction failure happened again to me this morning.
After half an hour, compaction was still saying it would take approx. 1 min.
I pushed the button once and it took me back to the menu and then to the clock. I noticed it wasn’t the one that i set. Clicking again got me a message that ´´launcher’´ wasn’t found.
checking the storage in IDE, most of files/apps are missing…

latest firmware 2v12.40 and bootloader
kickstarter bangle2

Argh, that's a shame. As always - if anyone has a way to reproduce this it'd be great.

Did you ever "Install Default Apps" since moving to 2v12? Because it is actually possible that corruption caused by previous firmwares could have broken things for you.

No, i never used it.
On the other hand, I just tried, after a "remove all Apps" to do "Install favourite apps" to restore my apps, but without success.
I will try the "Install default apps" now

EDIT : successful, I ll now try again to restore the fav apps

Edit : fav apps successfully installed too

Damn.... looks like it s happening again
After having installed a new app (the nice mountainclock) and updated a few ones (messages, LCars, health, and anotherone i cant remember which, all coming from the developping appstore), I launched a compaction action.
It s still compacting since 20minutes now.
Last time it did that, I longpressed the button to reset the watch... and lost almost all the files/apps that were on the watch.
So, trying to be a bit more patient to avoid re-installing and re-setting everything, I m now waiting for (fingers crossed) it finally manages to finish compaction (even if it somehow looks like it s stuck on something)
@Gordon if you believe i could do something to help understand what is going on (with webide for exemple) tell me

EDIT : flatening the battery launched itself (I probably touch the screen while removing the watch from my wrist ?) I stopped it with the appropriate longpress as suggested.
Now the bangle is stuck on the startupscreen. I guess i lost the battle once again. All files must have gone again or corrupted

so far, a few essential files corrupted

edit : and at least 21 apps missing

1 Attachment

My advice would be factory reset. Updrade firmware to 2v13 and test again.
Since the last fix, compact storage has been reliable for me.
I am seeing an off file though - might be totally unlrelated.

1 Attachment

@nicoboss which firmware were you on when this happened?

@Gordon the last one : 2v13.6