-
I agree. So there are two options:
- Option 1: do this for "end user" - that we're shooting for the use case where user follows the "the few shots" and gets his custom app to work, with properties:
- code sharing is to provide more examples mostly, not as end artifact, millions of apps created & "published"
- no "settings" in apps, apps are simpler and "just do the job"
- to change the app, the user will continue this 'few shots' approach to change whatever setting he likes
- code sharing is to provide more examples mostly, not as end artifact, millions of apps created & "published"
- Option 2: do this for "developer" - the 'few shot' approach helps the developer to create & package the app for "conventional usage scenarios", props:
- installable packaged app is a final artifact of the flow
- apps contain familiar interfaces, settings, menus, etc.
- when developing app feature, this becomes "enhanced copilot"
- installable packaged app is a final artifact of the flow
I'm for option 1 - although this "end user" is probably still a developer enthusiast but personally for me it is exactly the fun I'm looking for, and it shoots directly into the vast space of unknowns of what the future interface would look like.
- Option 1: do this for "end user" - that we're shooting for the use case where user follows the "the few shots" and gets his custom app to work, with properties:
-
I suggest to experiment with completely different UX
Like,
- the user looks at examples gallery (like short screenshot videos gallery)
- picks and idea of what he wants in his watch e.g.
- I want this watch face but smaller font and red color
- or I want an app that looks like "notes" but lists top items from slashdot RSS instead
- I want the watch face from app X but also show temperature from my BLE coffee mug in bottom with a tiny mug icon, and a one-tap 2 min timer for my toaster, but show that only until 10am, afterwards show steps counter... ....
- I want this watch face but smaller font and red color
- ChatGPT generates app, packages, pushes to watch
- If the user doesn't like the result, they tell to fix, or delete the app
- The user asks to add some functions to existing app on his watch from the list of "ideas" in that gallery in p.1
it's just and idea and still needs a lot of polishing and figuring out the details but I guess high-level properties of this are smth like:
- every app is custom
- apps will have any amount of functions. e.g. I can ask the sleep info app to also show weather forecast for today, and ask it to light up when wake up alarm triggers
- very social/storylike experience with users and chatbots discussing these apps
- amount of "apps" created is beyond listable
- no "settings" or other hard-to-use "menus" on the watch - just ask ChatGPT to change news feed from slashdot to hackernews... or add another app...
- the user looks at examples gallery (like short screenshot videos gallery)
-
Yes that was exactly my thinking. We should also think of compatible method to deploy & publish those apps, as regular DevOps doesn't make sense here as presumably the app is defined by English not code.
-
... and also Ember BLE mug temperature reader that I wrote with ChatGPT
-
So I created the RSS reader app with ChatGPT, not without issues though.
Where it failed was:
- It used incorrect http api instead of Gadgetbridge API, but was able to correct itself when I told to use that.
- It did not know the Gadgetbridge API .resp trick with returned "data" so I had to manually fix
- It completely broke on attempting to write XML parser with RegExp but apparently because RegExp implementation in espruino is too non-standard and I ended up asking it to rewrite the parser wtih string slicing, and it worked.
I believe it would have been doable in one shot if I fed it ahead of time the API nuances with tiny use case examples, specifically the "rough edges" like unsupported RegExp.
I can definitely create a neat "GPTs" demo case where it writes the app in one shot.
Full transcript to the point where I understood it won't fix itself: https://chat.openai.com/share/18d6df03-7af0-41da-a660-f419c1abd686
Final RSS reader app attached.
- It used incorrect http api instead of Gadgetbridge API, but was able to correct itself when I told to use that.
-
This is super cool!
I will see if I can enhance this process and get back
-
I'm not familiar with Open Interpreter, looks cool, but from what I can see in the video it is more about controlling the machine with commands that are "interpreted" by LLM, while we do not have this capacity on the watch. We need to write very small code, then store and make launchable & usable.
It would be interesting to look inside, they seem to have some neat ideas and a lot of experience with feeding just enough context for it to work even on open source models.
Btw, ChatGPT seems to know Bangle.js already and can write code without pre-feeding context.
-
These two actually intersect a lot as there is no fine line between "what calls to call and in what order" and "make computer do the thing you want". And technically, they are the same and called A.I. planning and program synthesis and the difference is probably in the grounding level of the generated code- either AI can write calls with unrolled loops call-by-call, or create higher-order logic to describe the program with flow control primitives. Lifting the logic is computationally expensive though.
Where this is becoming really promising is that Bangle.js provides a confined and simple enough "world model" for the LLM to be able to handle it much more efficiently than the "open-world" approach of generic Python code writing.
- We have JS, which is exhaustively represented in any LLM dataset
- We have the realm of "watch applications" which is much smaller scope than any other application scope (probably smallest consumer electronics scope ever)
- We have a significantly smaller scope of library calls and possible I/O scenarios due to, basically, the nature of the bangle.js project
- All these combined make the total problem space size much, much smaller than "open world" rhetoric from OpenAI et al., given that every scope increase step is a combinatorial explosion.
I believe that if automatic programming will ever happen "in a mass consumer product" it will happen here, in the watch applications, first. Or at least in the first batch of breakthroughs.
I don't believe multi-shot (feedback from interpreter) approach is feasible in general, but might improve the accuracy by some significant percentage points. The problem is in the cost of it - while trying a couple attempts might be OK, running thousand run/feedback shots of LLM is extremely expensive to do just to throw away all the missed results.
The problem of writing larger applications can be solved by fine-tuning the LLM by providing more examples of the code, and/or training a world model using some kind of metaheuristic-guided exploration on a simulator and then mass-training on that simulator in a GAN-like approach to get a corpus of feasible working programs. Still orders of magnitude cheaper to do than open-world problem.
As a product development "feature" (if you allow me to wildly hypothesize), one could collect the feature requests from users that the LLM was not able to code in one-shot approach, and then use those to re-train the LLM model so that these "featureless features" will become gradually available for one-shot implementation by user request.
- We have JS, which is exhaustively represented in any LLM dataset
-
"Point to" - no, still not enough compute/context
But doing smart prompting will definitely do it.
I would approach it like this:
- user writes the description of a program
- in the "first-level" prompt there is a pre-loaded summarized list of applications, so ChatGPT selects applicable programs to use as examples
- same thing with documentation pages, select what is likely to be relevant topic-wise
- in second turn, load the examples and documentation pages into context, preempting by relevancy score
- write code based on example, potentially run simulator to see any useful feedback
- if the software runs on a simulator, package as app, if it does not - show the code to fix manually
Even better results will be with changing current applications - like changing / inventing new watch faces by modifying code
- user writes the description of a program
-
some sort of BSOD watchface
That would be super useful to show it's not "bricked", and also maybe a message "still waiting for firmware" - although it is not actually waiting in that "BANK0 error" state, it requires the long-press reboot to enter the waiting mode.
And yes, I was able to avoid an engineering kit flashing mode involving the third-party flasher app, but having me prepared for such outcome would be very handy once I click the lucrative glowing yellow button while naively expecting the shiny new firmware after that one click. But instead experiencing a setback of half an hour to figure out what just happened.
Specifically, a message "if something goes wrong, long press & repeat, if it does not help, try another computer/browser, if that does not help too, download firmware from X and use app Y on the phone"
-
... and another high-priority feature: override whitelisting, that would fire notification alarm even if "do not disturb" and "quiet" are set on all devices.
Stopword-based whitelisting would also save me from searching for sub-menus that would probably? allow to do this on the "smart"phone GUI side.
-
but your device isn't actually bricked
I am an engineer and I can hypothetically un-solder the chip, connect power and SPI and re-flash. And I've done it in the past.
But if the device fails to boot into UI after a documented regular user procedure, it IS bricking.
I'm still very satisfied with the product, just making sure user voice is heard ;)
-
I'm still processing this, but it is insane
Imagine a watch that has no functions at start.
You describe functions in English, ChatGPT writes code.
I was able to combine ChatGPT with code examples and it wrote me almost-working code to read and display current tea temperature from my Ember smart mug over BLE. For smaller programs it would just write working code directly, potentially with no user interaction needed, today.
Here is how it works:
- In general, I use Web IDE and write a..js files to launch with "App Forge" application, for simplicity. Packaging would add too much overhead for this case
- I open ChatGPT and paste multiple examples of apps with similar functionality (in this case the ble hrm app), including a github page about Ember mug (I could have probably sent just the link)
- I ask ChatGPT to write application using the examples above that would connect to the BLE mug, read the temperature and output on screen
- ChatGPT figures out from examples how to do this, writes code
- I paste the code to Web IDE, save as a.mug.js, then open in on the watch with "App Forge" (I had to do some modifications though)
I believe it can be automated to the point of just asking for specific functionality and it writes code. Maybe it will need emulator feedback to fix errors, and some additional documentation trickery - like asking it to summarize documentation first and paste that summary as initial prompt so it is more aware about custom function call patterns.
This is absolutely stunning. Wow. I have the smartwatch with all imaginable functions at once.
- In general, I use Web IDE and write a..js files to launch with "App Forge" application, for simplicity. Packaging would add too much overhead for this case
-
I am filtering on the watch because I need to specify reaction "level" on the watch by priority, not just throwing away message
- I want prioritization to trigger different reactions - sound, high/low vib, etc.
- For different "keywords" I want different pattern of notifications
- I want to try experimenting with "speaking with vibration motor" - e.g. t,b, p - high-energy short pulse, a ,e, o - different low-energies with longer pulses.. to try to identify which keyword triggered the notification by "vibing (reading with vibration) out the word"
- Maybe even trying to "vibe out" sender name if that was a person sending message
- Maybe experimenting with "congestion control" if too many messages arrive per unit of time by dynamically increasing prioritization level
Publishing and packaging stuff takes more time that I'm ready to afford as of right now but I'll do in near future
- I want prioritization to trigger different reactions - sound, high/low vib, etc.
-
Interestingly, this is exactly the REASON I bought Bangle.js smartwatch - to be able to do advanced filtering of notifications by "stopwords" and sorting them by "priority" by checking whitelist of words and using different styles of notifications - sounds, vibration, how much, etc.
I am literally right now writing some code in "messages" library to do this, so if you're interested - we can work together to make it a feature.
-
I was able to un-brick the device by holding down the button till it enter BT flashing again, and then following same update procedure on a Steam Deck after following instructions at https://www.espruino.com/Quick+Start+BLE#linux
There is also an "Android" based flashing which I did not use but would have probably worked too.
This is a windows problem.
-
Physicist.
-
Just clicked the glowing yellow button to update firmware on brand new v2 watch and immediately bricked
Probably requires immediate action to prevent mass-bricking by disabling the "update" button
Scenario:
Disabled BT on android phone BT device appeared in Chrome/Windows 11 Clicked Pair Clicked firmware update Followed instruction to hold button, release while indicator moving on watch screen Clicked next button ERROR on device screen & web (can't remember which) Device screen says press button to reboot Button pressed, bootloop with error BANK0 INVALIDContext:
Received device yesterday, brand new, installed a couple of apps, paired to both PC and android.
Debug info:
Loading https://www.espruino.com/json/BANGLEJS2.json index.js:92945 Board JSON loaded blockly_espruino.js:114 Blockly.setBoardJSON Object index.js:92945 Firmware >1.43 supports faster writes over USB index.js:92945 Set Slow Write = false ('Throttle Send'='Auto') index.js:92945 FIRMWARE: Current 2v19, Available 2v20 index.js:92945 New Firmware 2v20 available index.js:92945 [notify_info] New Firmware available (2.19 installed, 2v20 available) index.js:92945 Device found {"portName":"Bangle.js c684"} index.js:92945 [success] Connected to Web Bluetooth, Bangle.js c684 index.js:92945 >>> Connected to Web Bluetooth, Bangle.js c684 index.js:92945 Downloading https://www.espruino.com/binaries/espruino_2v20_banglejs2.zip index.js:92945 >>> Downloading binary... index.js:92945 >>> Done. index.js:92945 BT> Disconnected (gattserverdisconnected) index.js:92945 Disconnect callback... {"port":"Web Bluetooth","portName":"Bangle.js c684"} index.js:92950 [notify_warn] Disconnected from Web Bluetooth, Bangle.js c684 (anonymous) @ index.js:92950 index.js:92945 >>> Disconnected from Web Bluetooth, Bangle.js c684 index.js:92945 stepFlashNordicDFU: Object index.js:92945 [success] Initialising... 2index.js:92945 >>> Initialising... index.js:92945 connected to gatt server index.js:92945 found DFU service index.js:92945 found 2 characteristic(s) index.js:92945 [success] Updating application: espruino_2v20_banglejs2_app.bin... 2index.js:92945 >>> Updating application: espruino_2v20_banglejs2_app.bin... index.js:92945 connected to gatt server index.js:92945 found DFU service index.js:92945 found 2 characteristic(s) index.js:92945 found packet characteristic index.js:92945 found control characteristic index.js:92945 enabled control notifications index.js:92945 transferring init index.js:92945 >>> Uploading... index.js:92945 crc32 not found, skipping CRC check index.js:92945 written 146 bytes index.js:92945 transfer complete index.js:92945 transferring firmware index.js:92945 crc32 not found, skipping CRC check index.js:92945 written 1900 bytes index.js:92945 notify: Error: The state of the DFU process does not allow this operationAlso created ticked with more screenshots at https://github.com/espruino/EspruinoWebIDE/issues/292
Don't upderstand where to go from here. It says
SW RESET DFV 2V19 CHECK STORAGE NO NEW FW BANK0 INVALID BTN1 = REBOOTI have experience with soldering, have a couple of RPIs for potential UART'ing but no idea where to go from here...
Happy to be in the community btw and thanks for the great product that I hopefully will get up and running again!
Espruino
Espruino is a JavaScript interpreter for low-power Microcontrollers. This site is both a support community for Espruino and a place to share what you are working on.
As Open Interpreter is just an alternative (and weaker) version of ChatGPT, I would probably see using open interpreter as using a Motorola 6800 when Intel 8086 was already available...