Your experience with Bluetooth HID on the Bangle?

Hello all,

I have been experimenting with using the Bangle as an HID keyboard, based on the Bluetooth / Binary Bluetooth Keyboard apps, but I find it to be extremely unstable to the point of being unusable.

In the best case it works for a minute or so, with some lag between pushing buttons, but more often than not it will either not register buttons at all, or only after a few seconds, sometimes repeating them several times. Also sooner or later I get disconnects when using HID keys. First I thought it was a spotty Bluetooth driver implementation on Linux, but it behaves the same on my Windows work PC.

I am using a Plugable Bluetooth adapter with BCM chipset on a Linux 18.04 machine with BlueZ 5.48. I am sitting right next to the dongle when testing, and when programming the Bangle it seems to be quite stable usually.

So what are your experiences with HID over Bluetooth? Is this behaviour common?

Are you running up to date firmware on Bangle.js? I know early firmwares (before 2v07) did have some issues with Bluetooth LE HID stability, but I was pretty sure those were sorted now.

Yes, I have 2v08 on it.

So, to follow up...after I sorted out my other problem with the storage I thought I'd give BLE HID another try, and whaddayaknow...it works much better now! Virtually like I expected it to.

Great! Thanks for letting me know!

Hi,
I am experiencing exactly the same problems described by @Sebastian in his first post.

This is my setup:

• BangleJS Firmware version 2v08 stable.
  
• PC OS: Xubuntu 20.04.2
  
• PC Kernel version: 5.4.0.66
  
• PC Bluetooth adapter: Intel 7260
  
• BlueZ version: 5.53
  
  

The problems that happen to me more often are keystrokes not registered and keystrokes repeated several times (sometimes indefinitely until the next key press).
I found out that the best way to reproduce both problems is following these steps:

• Assure that the BangleJS is connected to the PC.
  
• Wait at least a minute without doing anything on BangleJS.
  
• Cause a temporary disconnection on the PC (either due to long distance from the BangleJS or forcing a disconnection using the Bluetooth control interface).
  
• Restore the Bluetooth connection.
  
• Send several keystrokes.
  
  

The bug persists until I disable and then reactivate Bluetooth on both the PC and the BangleJS device.

@Sebastian, how did you manage to make HID keyboard work as expected?
@Gordon, can there possibly be a fix for this problem in any cutting edge firmware version?

Hi,

as I hinted in my previous post, I fixed a problem that I had which seemed to influence also the stability of Bluetooth HID by "Installing Default Apps" from the App Loader. I can't say what exactly the problem was, I somehow messed up the storage and maybe some important firmware parts with it, but afterwards it was much more stable.

That said, I still occasionally encounter these same issues (repeated keystrokes or no keystrokes registered at all), like maybe once out of 10 times. It's not the reliability of a wired solution connected via USB for example. I never used a Bluetooth device before, but my assumption is that's just how Bluetooth works and not necessarily a fault of the Bangle.

Repeated/no keystrokes definitely sounds like HID packets not being sent (a keystroke involved one packet for a press, and one for a release - so if the release gets missed that'd cause the problem).

It's interesting that both of you having problems are running Linux though.

It'd be good to know whether on Bangle.js, it's actually the NRF.sendHIDReport command that fails with an exception, or whether the data is accepted by the Bluetooth stack and then silently lost.

Are you using an existing app, or your own? If it's your own, you could put a try {...} catch around the call and display something on the screen if it fails. It might help to track it down

@Franzo- if there is something we can figure out that's actually broken in the Bangle.js firmware then yes, it could be fixed. However if this was an issue on non-Linux devices I'd have thought I'd have had more complaints by now - so that makes me wonder whether it's actually some issue in Linux's handling of BLE HID devices

Thank you for your replies!

@Sebastian: I am also not very versed in Bluetooth devices, but before buying a BangleJS I used a cheap Bluetooth keyboard to achieve what I am trying to do now (i.e. remote control of media playback on PC). That keyboard never gave me these kind of problems, except a reasonable delay on the first keystroke after a reconnection. This is why I found the behaviour of BangleJS quite unexpected.

@Gordon: I am using a slightly modified version of hidkbd app, where I only changed the keys sent with ble_hid_keyboard module and added a reconnection routine which calls NRF.wake() when NRF.getSecurityStatus().connected is false. I implemented this routine some time ago because I thought that the repeated keystrokes could have been a sort of "buffering" effect on the commands sent when the device is disconnected, but actually it did not fix the problem at all.
The issue is reproducible with the hidkbd app too, so I do not think that the problem lies with the changes I made.

I do not know if the problem is only limited to Linux: I am not using native Windows anymore on my PC, but when I did I often experienced some issues when I tried to reconnect, which could only be solved by removing the BangleJS from the trusted devices list and rediscovering it again. Never seen repeated keystrokes, though.

As you suggested, I just tried to add a try {...} catch block around the call to kb.tap(). An exception is indeed caught on the next keypress after a repeated one and it says: Error: BLE HID already sending.
It seems that some sort of communication issue is actually happening...

What Gordon says about packets not being sent makes sense to me - when a keystroke is missed it's the "press" that is not sent, and when it's keystrokes repeating it is the "release" missing. That is also confirmed by the fact that it can usually be stopped by pressing the button again once or twice.

I also have the impression it depends on distance between the Bangle and the BT dongle. As soon as I get 2 or 3 m between the watch and the dongle it seems there is slightly more latency, and the misses/repeats become more frequent. @Franzo: Do you have a lot of distance between your watch and the BT receiver when this happens?

I can try with a Windows machine and see if there's any difference.

Ok, that's interesting then. So I guess maybe what's happened is:

• The Bluetooth HID packet gets sent
  
• Espruino sees this, and calls the callback thinking it has actually got sent
  
• The original HID packet wasn't received so a resend is attempted
  
• The hidkbd code tries to send a second HID packet, but it fails because the original was still being sent
  
  

Does the 'key stuck on' problem happen more often than just the missed key?

If it is the problem above then I might be able to do something about sendHIDReport such that it didn't report the key as sent until it had actually 'gone', but I'm not sure I could do much about the missing initial keypress, as that might just be due to a particularly bad connection and the send just timing out.

Does the 'key stuck on' problem happen more often than just the missed key?

I don't think so - I would say the repeated key is just more noticeable than an initial keypress missed, but I could be wrong.

What exactly happens when an HID report is sent? Is there some sort of handshake or request/response with the receiver to make sure the packet has been received? Or is it just fired out in the hope the paired device receives it? From your comment I seem to understand there is a difference between the callback being triggered and the report actually being sent out / received?

@Sebastian: I also agree on the lost "release" packet theory: a repeated keystroke can almost always stopped pressing the corresponding button again.
Regarding the operating distance, I generally use HID and experience this issue when I am at least 2 m from my PC. Only when I stay at a "safe" distance (< 0.5 m) during the whole duration of hidkbd app operation the issue seems not to be happening.
As already mentioned in my first post, I am pretty sure that the problem lies in the Bluetooth reconnection procedure. If I force a disconnection and wait for reconnection, I can reproduce the bug even at a safe distance. Probably the increase in keystroke misses and repeats rate that you experience during long range operation is the consequence of a reconnection triggered after a momentary lost signal disconnection.

@Gordon: in my use case (media playback control) the 'key stuck on' problem is much more harmful than the missed key. So yes, I would be very grateful even only for a fix for the repeated keystrokes problem!
So, if I got this right from your post, in the BLE HID protocol a sort of ACK is expected after each keystroke. If this is true, maybe even the missing initial keypress could wait until a timeout and report a failure as an exception of kb.tap() call. This exception could be caught from user code such as hidkbd and possibly trigger another kb.tap() call or print an error message on screen. This, combined with my reconnection routine described in post #9, should do the trick!
What do you think? Can this be feasible?

What exactly happens when an HID report is sent? Is there some sort of handshake or request/response with the receiver to make sure the packet has been received? Or is it just fired out in the hope the paired device receives it? From your comment I seem to understand there is a difference between the callback being triggered and the report actually being sent out / received?

My understanding is there's a bluetooth 'notification' fired from the Bangle, and the PC is able to 'accept' that notification. It's entirely likely that Espruino marks the HID packet as sent when the notification is sent, and not when it is accepted.

experience this issue when I am at least 2 m from my PC

Ok, that would definitely back up the lost packet theory then.

Could you add a NRF.on('disconnect', function(reason) { Bangle.buzz() }); to your code? That way at least you'll know if the Bangle is disconnecting.

My concern is that if there really is a disconnect, there's nothing we can do. If a keydown event is sent and then the Bangle disconnects, even if we were able to detect that, it's not going to be able to send a key up event until it reconnects.

I am pretty sure that the problem lies in the Bluetooth reconnection procedure

I didn't mention before but there should be no reason to call NRF.wake() unless you've set up Bluetooth to be off - once the Bangle bluetooth is 'woken' it should stay awake even after a disconnect (until NRF.sleep() is called).

I guess it is possible that if the Bangle gets disconnected in the middle of sending a HID report, the state of the HID module ends up being invalid and then doesn't recover. But if that happened, even after forcing a reconnect you should no longer be able to send any HID reports - and it sounds like that doesn't happen.

the missing initial keypress could wait until a timeout

Yes - if we're in the situation where you call NRF.sendHIDReport and the callback never gets called, that's something we could definitely add.

However it doesn't sound like that happens? It seems like if anything the callback is being called too early?

I have done a Wireshark capture to try to get a better idea of what is actually going on when the issue happens. These are the actions I have performed during the capture:

1. Pair and connect to BangleJS at a safe distance (< 0.5 m).
   
2. Send some keystrokes, always keeping myself at a safe distance.
   
3. Move away from my PC and wait for disconnection.
   
4. Go back to my PC at the same distance as before and wait for reconnection.
   
5. Send again some keystrokes until the issue happens.
   
   

I observed that the keystrokes of step 2 correctly alternate the key down and the key up actions, whereas the ones of step 5 quickly lose the correct sequence and the order is messed up. As we correctly guessed, the issues are caused by missing packets, i.e.:

• "No keystroke" happens when only the key up action is received.
  
• "Repeated keystroke" happens when 2 consecutive key down actions are received.
  
  

@Gordon if you are interested in analyzing the .pcap capture file, I can DM it to you. It should not contain any sensitive information but I'd rather avoid publishing it here...

Ok, thanks! It's good to have a solid way to reproduce - I'm on Linux here so hopefully I'll be able to get it to happen.

I think I'll be fine without the pcap if I can get it to happen here, but you're saying that the Wireshark capture of actual Bluetooth traffic shows the packets are missing, or is it a capture of input events?

the Wireshark capture of actual Bluetooth traffic shows the packets are missing

This one.

In the capture, the HID commands are classified as Bluetooth Attribute Protocol (ATT) packets and each keystroke is a pair of key down and key up actions with corresponding packets.
If I send a space keystroke from my app, Wireshark reports first a packet with an ATT value of 00002c0000000000 for key down, then another packet with a value of 0000000000000000 for key up. The second packet is sent ~15 milliseconds after the first one.

When the issue is happening this sequence is jumbled up because some packets are missing. In addition, for the few successful keystrokes, the delay after which the key up action is sent is much higher (~150 ms).

@Gordon, have you managed to reproduce the problem with your Linux PC?

Thanks - I haven't had a chance to try it yet I'm afraid - I've been pretty busy with other things the last week or so.

However the 150ms delay is interesting. I wonder if the issue is actually that the Bluetooth 'power saving' mode of Bangle.js got activated and then didn't ever get turned off - that would mean it was using a 200ms connection interval and could explain a lot.

Please could you try adding NRF.setConnectionInterval(7.5) to the top of your app? That'll force Espruino to always use the fastest Bluetooth poll interval available.

Please could you try adding NRF.setConnectionInterval(7.5) to the top of your app? That'll force Espruino to always use the fastest Bluetooth poll interval available.

I have done what you asked and I found out something very interesting.

Wireshark reports the connection interval setting as the first message sent by BangleJS to my PC after connection creation. It is a L2CAP protocol Connection Parameter Update Request packet containing:

0000   02 01 2e 10 00 0c 00 05 00 12 03 08 00 06 00 06
0010   00 00 00 90 01

Byte 10 is the Command Identifier and its value is 0x03.

After the connection is established, keystrokes have a delay of [100 - 150] ms for the key up action. In this state the usual issue of repeated and lost keystrokes can happen, especially if I quickly press the button.

However, if I wait for ~40 seconds after connection creation Wireshark captures another Connection Parameter Update Request packet sent by BangleJS with the following contents:

0000   02 01 2e 10 00 0c 00 05 00 12 05 08 00 06 00 06
0010   00 00 00 90 01

The contents are the same except for byte 10, which is now 0x05.

After this packet reception, the delay of key up actions drops to [15 - 30] ms and there are no more repeated or lost keystrokes.

Could it be that the Command Identifier of the first Parameter Update is wrong? I can see from the Wireshark dump that all the requests are accepted with a Connection Parameter Update Response packet sent by my PC, but maybe the Command Identifier 0x03 is actually ignored by the PC BLE controller...

Wow, thanks for digging into this - that's really interesting!

So Bangle.js (and Bluetooth Espruino in general) has a power save mode that moves to a low power mode after ~1 minute of inactivity, and then after some data is sent it requests the 7.5ms mode.

However if NRF.setConnectionInterval(7.5) had taken effect, that mode should be disabled and it shouldn't be sending the 'Connection Parameter Update Request'...

The low power stuff is handled by the DYNAMIC_INTERVAL_ADJUSTMENT ifdefs, and it should get reset with the code here on disconnect https://github.com/espruino/Espruino/blob/master/targets/nrf5x/bluetooth.c#L1148

But... maybe something about the way it disconnects when out of range causes that code not to work?

Some updates on the tests I have been doing during the week.

In my attempt to analyze the Command Identifier byte which is different between the two Parameter Update messages, I simplified the app I am using on BangleJS.
This is its code:

const kb = require("ble_hid_keyboard");

// Force Bluetooth to use the fastest poll interval available
NRF.setConnectionInterval(7.5);
// Enable BLE HID providing an HID report.
NRF.setServices(undefined, { hid : kb.report });

function drawApp() {
  g.clear();
  g.setFont("6x8",2);
  g.setFontAlign(0,0);
  g.drawString("HID test", 120, 120);
}

function logPrint(message) {
  const now = new Date();
  Terminal.println(now.getHours() + ":" + now.getMinutes() + ":" + now.getSeconds() + " - " + message);
}

NRF.on('connect', function(addr) { logPrint("Connected to " + addr); });
NRF.on('disconnect', function(reason) { logPrint("Disconnected: " + reason); });

// Start!
drawApp();

The app configures the minimum connection interval and sets the HID report, using the one provided by ble_hid_keyboard module. Then it does nothing else other than printing to terminal every connection and disconnection event. I also simplified the actions I do to reproduce the problem: now I always leave the BangleJS near my PC and control connections and disconnections to the device using the Blueman interface.

Using this app I am still able to sniff the Bluetooth traffic with Wireshark and see the two Parameter Update messages sent by BangleJS, but I also noticed some other interesting details:

1. The second Parameter Update message is not always sent by the device after a connection, but when it is present its timestamp is always 41 seconds after the first captured packet (i.e. the command triggered by my click on "Connect to device" on PC).
   
2. The second Parameter Update message is not generated when the connection to BangleJS includes the device pairing procedure.
   
3. After the connection is established, a momentary disconnection often (about 50% of my tests) occurs before 41 seconds have passed. The app prints on the BangleJS Terminal: Disconnected: 8, which should be the error code for Connection Timeout. After the reconnection has been completed no further traffic is reported by Wireshark, so no second Parameter Update message is present even in this case.
   
   

The low power stuff is handled by the DYNAMIC_INTERVAL_ADJUSTMENT ifdefs, and it should get reset with the code here on disconnect https://github.com/espruino/Espruino/blo­b/master/targets/nrf5x/bluetooth.c#L1148­

But... maybe something about the way it disconnects when out of range causes that code not to work?

I, too, think than could be the root cause of the problem... The cases I described in this post tend to happen "alternately", for example: if in a Wireshark capture I see a spontaneous disconnection (case 3), I can almost be sure that in the next capture I will see two Parameter Update messages (case 1). It is as if something from the previous connection does not get correctly reset and keeps affecting the next connection.

Thanks! I'll try and take a look at this although I'm a bit busy this week.

The impression I get is the softdevice which Bangle.js uses has a bit of an issue which is that to set the connection interval, you may need an active connection - so if the connection isn't there, you can't set the interval and then it comes up in the old state. I'll have to investigate further