Couple of minor WebIDE issues

I encountered a couple of slightly annoying things working with the IDE today and was wondering if these were known issues. I wasn't sure if I should post here on the forum first or open issues on GitHub, but here's a brief description of each:

The first is that viewing the Bangle.js debug log (which seems to be written to storage file log.txt, if I'm not mistaken) is quite fiddly because the IDE insists that it's a binary file when it should be text. The hex dump is not helpful (nor does it help that the window appears too narrow in my browser to display properly even though there would be plenty of space), so I'm forced to save the file and then open it manually in a text editor. Further complicating matters is the fact that the default filename lacks the correct extension (it's log.txt (Storage file) instead of something ending in .txt), and renaming it adds an additional unnecessary step to the process. Ideally, a file ending in .txt should be just that, and it should open as text in one step without a hassle.

The other one is a bit weird, and I'm not sure how to debug it. Occasionally (I haven't been able to discern any particular pattern) when my watch is connected to the IDE and I click the Storage button, the list of storage files fails to appear. Instead, I only get a Send to RAM button and then a dump of strings representing the names of my storage files on the debug pane an instant after the dialog window pops up. Another possible clue is that when this happens there is usually a delay of a couple of seconds, which is longer than it normally takes. If I cancel and try again, it usually works fine, but before long the glitch crops up again.

the IDE insists that it's a binary file when it should be text

Quite likely it is binary (or maybe even UTF-8 now?) = there is some non-ASCII byte there. Often there is a button there to switch to text mode, maybe it should be there always (currently it is not).

. Further complicating matters is the fact that the default filename lacks the correct extension (it's log.txt (Storage file) instead of something ending in .txt)

Which OS or browser it is? On Windows in Chrome it definitely adds just .txt

Instead, I only get a Send to RAM button and then a dump of strings representing the names of my storage files on the debug pane an instant after the dialog window pops up. Another possible clue is that when this happens there is usually a delay of a couple of seconds, which is longer than it normally takes.

It may be out of memory (or low stack?) situation. Maybe some app that use lot of memory/stack is running? Do you see some error or exception message in the 'dump of strings' or series of dots inside of (json) output?

I've seen this on low memory devices (nrf51822 with 16kb ram), the code sent by IDE that does the listing was the most complex thing to run there. While other simple code was running fine (there is not really much RAM available there) listing the storage failed a lot until I reduced total variables for the build => increased space for stack. I think it needs to build a relatively big JSON object (which depends on total number of files there) which needs a lot of stack space.

Quite likely it is binary (or maybe even UTF-8 now?) = there is some non-ASCII byte there. Often there is a button there to switch to text mode, maybe it should be there always (currently it is not).

There are what look like random escape codes in the tracebacks and such, which look like formatting codes for terminal emulators. But they're not really that useful for a written .txt log in a file, given that they confuse programs.

The only button I get is for “decode JS”, but that doesn't seem to do anything in this case. Having a way to force it to treat the file as text would be helpful, though, especially in cases like these where I know it really is text with just a couple of stray rouge bytes that can be safely ignored.

Which OS or browser it is? On Windows in Chrome it definitely adds just .txt

For some reason on Ubuntu and Chromium it wants to add the string “ (StorageFile)” to the end of the filename, which confuses the system file save dialog. I can rename it but it's annoying. That's what displays in the storage list in the UI, too, but I'd argue that the “(StorageFile)” prefix should be dropped in the suggested filename when saving (or else placed just before the extension, or at the very start of the name or something).

It may be out of memory (or low stack?) situation. Maybe some app that use lot of memory/stack is running? Do you see some error or exception message in the 'dump of strings' or series of dots inside of (json) output?

It doesn't appear that there is a memory issue. There are no errors or exceptions, either, just the list of strings. However, I am starting to notice the pattern: It seems to happen consistently if the watch has been idle for a minute or so before I request the list. If the Bluetooth link was just active, it works fine, but after a couple of minutes, the odd delay and strange behavior occur, only for it to work fine if I immediately try again. It sort of makes me wonder if it could have anything to do with power management on the Bluetooth link somehow causing problems.

I happened to catch this in the IDE error log when it happened:

>>> Receiving...
WARNING: No result found for "require('Storage').list().forEach(x=>pr­int(JSON.stringify(x)));" - just got "< <<\r\n\".bootcde\"\r\n\"launch.app.js\"\­r\n\"launch.settings.js\"\r\n\"launch.in­fo\"\r\n\"antonclk.app.js\"\r\n\"antoncl­k.img\"\r\n\"antonclk.info\"\r\n\"about.­app.js\"\r\n\"about.img\"\r\n\"about.inf­o\"\r\n\"widlock.wid.js\"\r\n\"widlock.i­nfo\"\r\n\"widbat.wid.js\"\r\n\"widbat.i­nfo\"\r\n\"welcome.boot.js\"\r\n\"welcom­e.app.js\"\r\n\"welcome.settings.js\"\r\­n\"welcome.img\"\r\n\"welcome.info\"\r\n­\"health.img\"\r\n\"health.settings.js\"­\r\n\"setting.img\"\r\n\"sched.bo"
>>> 
getFileList TypeError: Cannot read properties of undefined (reading 'trim')

Which clearly looks like the data it received is being interrupted somehow (although I then get what looks like an untruncated dump of it in the debug pane, whereas normally when it works correctly nothing at all appears there except a new prompt).

As far as I know, I haven't seen other issues when doing other actions after such inactivity (such as sending/receiving apps with the App Loader, etc.) Just the storage list in the IDE.

I happened to catch this in the IDE error log when it happened:

I just tried to run require('Storage').list().forEach(x=>pri­nt(JSON.stringify(x))); and it produces just list of strings, no JSON, so that is better than I remembered. Either it was changed recently or the nrf51822 had issues even with the .forEach(x=>print(JSON.stringify(x))); giving back simple strings. Anyway it indeed looks like the output is cut. The "sched.bo" string is incomplete and whole list should probably end with > >>. And yes, the connection interval is increased automatically when connection is idle so first packets may be slow until it goes back.

You can verify this is the case by setting it yourself via https://www.espruino.com/Reference#l_NRF­_setConnectionInterval
, the {minInterval:20, maxInterval:100} should still save some power when idle and should be faster. You can even force it to single value so you can try setting it to 500 or more to be pretty slow and see if the issue appears consistently

log file

The issue is it's a dump of the text that would have been output over Bluetooth to a VT100 terminal emulator (including backspace chars and maybe even escape codes). Definitely making the IDE render it as text if it's just backspace would be a big improvement though - I'm not sure how far we could/should go ignoring extra chars though.

storage file listing

Originally we did load all the filenames into memory and output them as JSON, but that changes years ago I think.

What clock/app do you have running? The IDE has a timeout for a response after it sends a query - most likely what's happened is you're running the IDE with your app running and your app is spending a long time doing something - so long that the IDE times out waiting for a response.

It looks like the text does start to be sent, but that maybe by the time it is all through the IDE has timed out... IIRC it has one timeout for when it gets the first response, and a longer one for the total time it waits...

it wants to add the string “ (StorageFile)”

Yes - this is more laziness in implementation I think. The text is added to the filename in the Storage window so you can see that the files are different (as this can be important), but then when saving the original filename should really be used.

Although... I have to juggle a lot of stuff with Espruino - it is literally just me working full time on this - and it's unlikely I'll get time to put in fixes for all this stuff. If you'd be willing to jump in and make some fixes, that'd be great - as with the app loader you can just fork the IDE and host your own copy with GitHub pages for development if that's easier than having a local server

the majority of Storage file handling is done in https://github.com/espruino/EspruinoWebI­DE/blob/master/js/plugins/storage.js and the actual (Storage file) text gets added in
https://github.com/espruino/EspruinoWebI­DE/blob/master/js/plugins/storage.js#L35­8

The codes in the debug log definitely look like terminal escapes, though I'm not sure exactly what formatting they're for. Normally colors and formatting aren't that essential, but perhaps other characters in certain output might be important. If it is just the same output that the left pane in the IDE would display, I wonder if it would make sense to have a button or tab shortcut there for viewing the contents of the debug log using the same VT100 interpretation of escape codes. I'm not sure if the debug log file is an Espruino- or just Bangle-specific implementation, but it seems like a quick-view feature could be a handy debug tool.

The BLE interval makes sense. That would also explain why the response in the console sometimes seemed a little laggy for a moment. Checking the running app is a good point; I should have thought of that but didn't. I've mainly been using Timer Clock as the clock, but I just tried it with the default Anton clock and the first page of Settings and didn't notice any difference in behavior—it still happens the same.

I played with setConnectInterval and do notice that any interval higher than about 100ms seems to consistently reproduce the bad behavior. Lower intervals work okay. So far it seems that this is the only thing that causes the weird behavior.

Yes - this is more laziness in implementation I think. The text is added to the filename in the Storage window so you can see that the files are different (as this can be important), but then when saving the original filename should really be used.
Although... I have to juggle a lot of stuff with Espruino - it is literally just me working full time on this - and it's unlikely I'll get time to put in fixes for all this stuff. If you'd be willing to jump in and make some fixes, that'd be great - as with the app loader you can just fork the IDE and host your own copy with GitHub pages for development if that's easier than having a local server

Understandable. I'm kind of overwhelmed with various ideas myself and it's hard to decide what to do when. :) It looks like the priorities are in the right places, though, as the important underlying functionality seems pretty solid. Most of the things I'm encountering are fairly minor UI/polish type stuff, and that can always be fixed. I may consider taking a stab at some of it at some point.