Pretokeniser - on by default ?

I am wondering if the pretokeniser should be on by default.
Over months I experienced occassional hangs and thought it was something with my code.

Since setting pretokeniser OFF (months ago) I can no longer experience those hangs (note they could not be reproduced through some sequence of actions, just seemed to be random every now and again).

Of course 1 report does not make a fact. But I have heard other people talk about the pre-tokeniser. So I wonder if it should be switched off by default and labelled (experimental or caution). When you first start out with your Bangle 2 you are not even aware of this setting. So maybe OFF should be the default position ?

Well, the other school of thought may think that it works quite well and if there is a bug it is hard to find it when it is not used and if someone really has some issue it is easy to turn it off.

If there is a bug it should be testable by set of tests that is part of espruino
https://github.com/espruino/Espruino/tre­e/master/tests
provided that there is a batch run with pretokeniser enabled(?) and every tokenizable keyword is there

Also when thinking about it there is another part on server for pretokenizing files before upload so that would need another method for testing.

But anyway, tokenizing is quite deterministic so it should either work or the tokenized source is wrong so it should fail in deterministic way, random bugs are likely caused by something else.

Honestly, I leave the pretokeniser on for everything I do here, and it would appear to work fine for me. Given it's on by default (and I think is enabled for all the 'factory' apps) I'd expect to have got a lot more complaints if something really was broken with it.

As @fanoush mentions you'd hope that if pretokenised code broke something, the brokenness would be very repeatable rather than it just being a one-off.

... but that's not to say there isn't a problem. If you can find a way to reproduce it reliably then please let me know and I'll try and figure out what's wrong.

Slightly offtopic, but I wonder what the status with "minify apps" is. What type of app would benefit from it or in other question: What app should we test this feature on?

The main thing minifying does over pretokenisation is shortening the variable names. It'll reduce memory usage a bit and in apps that use long variable names in tight loops it'll help with speed - I think someone mentioned around 10% speed boost.

The problem we have is the minifier we have that runs nicely in the browser also doesn't appear to really be maintained any more and has issues (specifically it doesn't realise when an app uses a global variable like g and will rename local variables to also be called g). So it's why there are big danger signs by it - realistically if you enable it for all your apps, Bangle.js will break.

Right now it's not something I'd recommend.