Debugging HID/Keyboard with serial to another Puck?

Hey,

I'm using the HID Keyboard example - https://www.espruino.com/Puck.js+Keyboard plus simple LED on/off with button press/release - to try to send a single keypress to an iPad.

While it sometimes works, once it's cut off it does nothing: the LED still lights when pressed so the code's still working, but there's no effect on the iPad. The Puck is still paired and connected with the iPad.

Now, I've noticed https://github.com/espruino/Espruino/issues/1101 and some other ideas, but what's really tricky is the limitation that the Puck must be disconnected from the IDE to pair it with the iPad: only one connection at a time. It means I have no error messages visible to debug the problem.

My NoName™ FT232 USB-to-serial board is not working for some reason - probably lack of flow control - and the wiring is unwieldy, so I was thinking the other Puck I got from Kickstarter might be an easy way to do it. Failing that, I might rope a spare Pico in to act as a dumb serial converter.

So, I guess my question is, what's a straightforward way to cross-connect the serials of one Puck with another Puck, so the IDE (or at least the command-line errors) can be viewed while still connected as a HID device? Or, any other approaches to try?

Thanks,
Tom

Wow, that's pretty extreme! I'd say try using a Pico if your USB-TTL converter doesn't work - the flashing code for http://www.espruino.com/ESP8266 actually sets a Pico up as a pass-through.

With Puck-Puck there's a bit of a catch-22. The serial port takes power, so won't initialise itself if the Puck is powered on without serial connected (the RX pin pulled high). Obviously if you connect 2 Pucks together neither will initialise, so you'd have to manually enable serial on one first with Serial1.setup.

I actually fixed a whole bunch of HID-related bugs just recently, so first it might be worth trying the attached zip file.

I'm still trying to get it working stably, but the attached build just happens to work fine by the look of it. There are 2 issues that might cause problems that the recent build fixes:

• If you try and send a key while disconnected it generates an error, but that stops you sending any more HID commands
  
• If you disconnect then reconnect, the connecting device expects Puck.js to remember some info about it, but Puck.js didn't - and that can cause some problems.
  

1 Attachment

• espruino_1v91.715_puckjs.zip

Ah, okay... I'm DFU'ing that build now, thanks.

I was thinking exactly of that Pico pass-through code before I wondered if I could use a second puck. I see what you mean about the Catch-22.

Anyway, I'll try this out. FYI, this is how it's being used:

Cool - what's the idea? Is that some kind of rubbery gauntlet thing?

It's actually rigid: Thermomorph / Polymorph, ie. polyester plastic that melts in hot water.

It's so that when I'm relaxing and recovering from back pain, lying down, I can just turn page on iBooks on iPad by sending a "right" keypress. That way I don't have to hold the iPad or reach up to turn page, thus saving my shoulder and upper back muscles from unnecessary tension.

Incidentally, I haven't managed to get the new firmware talking yet. In fact, it's having trouble connecting to the Web IDE. I think my Mac needs a reboot. Still trying.

Nope.. this build seems to just do a full reset (RGB LEDs all to full for about a second, then off, then a short red blink) when I try to connect to it via Web IDE, or pair it with the iPad.

I'm going to DFU it again to make sure it's not my code, but I doubt it.

That's really neat! I'd love to see Puck.js used for some more stuff like this!

I guess you could detect long presses (or maybe even use the magnetometer) and mimic other keypresses with it.

That's odd - maybe it's an issue with saved code (I had to move the saved code location and it might be messing up the pairing).

Can you try this on a working firmware, and then uploading the new one?

var f = require("Flash");
f.erasePage(0x75000);
f.erasePage(0x76000);
f.erasePage(0x77000);

Okay, that did the trick: I DFU'ed to 1.91, did the lines above, and then DFU'ed to 1.91.715. It now pairs more successfully.

However, it hasn't been entirely successful. It took about three goes of code tweaking, re-upload, resets, unpair/pair before the iPad would actually do the "page right" function. It's working okay now, even after an NRF.disconnect(). I'll keep an eye on it and see if it stops working again.

FYI: at least with this firmware, onInit() doesn't seem to work. Once reconnected, dump() yields garbage.

Thanks for checking - I'll make sure I do something about wiping the old saved code in the final version.

Odd about the dump() problems though. I'll take another look - it's all really flaky at the moment for some reason - I've had a lot of problems with the Nordic peer management stuff :(

(Incidentally, I've just written up the project)

After exhausting a CR2032 in about a month, I've been tweaking my page-turner trying to make it more efficient.

I'm wondering about the best approach for saving energy between clicks as well as between sessions. I've got a timeout of ~10 seconds that is reset every time the button is clicked; when the timeout does activate, it performs an NRF.sleep(). Then, when the button is clicked, it does NRF.setServices(...hid...) and NRF.setAdvertising(...).

I've tried doing NRF.wake() in the button's watch function, as well as NRF.disconnect() during the sleep routine in an attempt to cut off the connection so the iPad doesn't keep thinking it's attached to a keyboard (and so hide the on-screen keyboard until I switch off Bluetooth)

However, I get some weird bugs. With NRF.disconnect() reconnecting is unreliable, and sometimes all of the LED light flashes are slowed down, which makes me think there's something else going on.

When NRF.wake() is included, I get:

NRF ERROR 0x8 at ?:0
REBOOTING.

I assume setDeepSleep() is not relevant on the Puck, but what would you see to be the right sequence of events for this application?

It's to be woken by a button click, and doesn't really require discovery when the device is "cold", so sleeping all of the BLE stuff seems reasonable; I'm not sure if an entire shutdown of the softdevice is in order. Disconnecting any HID connections when the device isn't being used is also desirable for the on-screen keyboard reason given above.

My code's attached, with the NRF.wake() and NRF.disconnect() calls commented out for the time being.

Any thoughts?

var hid = require("ble_hid_keyboard");

var long_click = 2000;  // 2 seconds
var double_click = 250; // 0.25 second

// Detect short press ( < 2 seconds)
var is_short = false;
var short_timeout = false;

// Detect fast-click count ( < 0.25 seconds)
var clickcount = 0;
var clearclick_timeout;

// Sleep when not in use.  Needs to be long enough that it doesn't
// interfere with attempts to connect to it via the IDE.  Saying that,
// a triple click will postpone the sleep.
var sleep_timeout;
var sleep_delay = 10000; // 30 seconds
var sleep_long_delay = 600000; // 10 minutes

function setupSleep(is_long) {

  if (undefined !== sleep_timeout)
    clearTimeout(sleep_timeout);

  // Schedule the sleep function to run in the future.  A button click
  // will reset this timer.
  sleep_timeout = setTimeout(sleep, is_long ? sleep_long_delay : sleep_delay);
}

function sleep() {

  // Flicker the LED to indicate sleep.  This is really for debugging
  digitalWrite([LED1,LED2,LED3], 0x101);

  setTimeout(function () {
    // Clear the LED
    digitalWrite([LED1,LED2,LED3], 0);

    // Clear the timeout
    sleep_timeout = undefined;

    //// Disconnect any running HID sessions -- problematic; doesn't reconnect well.
//   NRF.disconnect();

    // And shut down the chip
    NRF.sleep();
  }, 250);
}


// Wake is called for two reasons: woken by a button press; and called by
// init() for a full reset.
function wake() {
  //// This is VERY problematic: big-time crash in 1v92
//  NRF.wake();

  // Set up HID (keyboard) services
  NRF.setServices(undefined, { hid : hid.report });

  // Name it
  NRF.setAdvertising({},{name:"Page-Turn-O-Matic 4000"});
}

// Upon reset of the device
function init() {

  // Clear any current button event handlers
  clearWatch();

  // Just in case, we want to make sure this is off.
  Puck.magOff();

  // Set up BLE
  wake();

  // Set up button event handlers for both rising and falling.  These will
  // also take it out of sleepage.
  setWatch(btnPressed, BTN, { repeat:true, edge:'rising', debounce : 50 });
  setWatch(btnReleased, BTN, { repeat:true, edge:'falling', debounce : 50 });
}

// The primary action, on a single click
function primary() {
  hid.tap(hid.KEY.RIGHT, 0);
}

// The secondary action, on a double click
function secondary() {
  hid.tap(hid.KEY.LEFT, 0);
}

// Triple-click
function tertiary() {
  // Postpone the sleep cycle for a long time (ten minutes) for debug purposes
  setupSleep(true);
}

// The reset function, on a long click (>2s)
function long() {
  NRF.disconnect();
  init();
}

function btnPressed() {

  // When the button's pressed, we clear the LEDs. Any
  // LED activity is on _release_
  digitalWrite([LED1,LED2,LED3], 0);

  // Count clicks in chain
  clickcount ++;

  // Reset execution of short-click chain
  if (undefined !== clearclick_timeout) {
    clearTimeout(clearclick_timeout);
    clearclick_timeout = undefined;
  }

  // Assume it's a short press
  is_short = true;

  // Set a timeout for two seconds to recognise a long press
  short_timeout = setTimeout(function () {
    // It's been two seconds, so...

    // Long press detected
    is_short = false;
    short_timeout = null;

    // Full blast RGB
    digitalWrite([LED1,LED2,LED3], 0x111);

    // and don't do anything until release...

  }, long_click);
}

// Once a chain of repeated rapid clicks is over (ie.
// the 0.25 second threshold has passed)...
function chainEnded() {
  var o = clickcount;
  clickcount = 0;
  clearclick_timeout = undefined;

  // Reset the sleep watchdog
  wake();
  setupSleep();

  switch (o) {
  case 1:
    // Simple click;  GREEN
    digitalWrite([LED1,LED2,LED3], 0b010);
    primary();
    break;

  case 2:
    // Double-click;  RED
    digitalWrite([LED1,LED2,LED3], 0b100);
    secondary();
    break;

  // Triple-click, etc. can be added as additional `case`s.

  case 3:
    // Triple-click;  YELLOW?
    digitalWrite([LED1,LED2,LED3], 0b110);
    tertiary();
    break;

  default:
    // Too many clicks. Ignore.
    break;
  }
}

function btnReleased() {

  // `short_timeout` is there to _deny_ a short
  // click.  If it times out, then it means the button
  // has been held longer than a short click.
  //
  // So, for a short click, if the timeout is still going
  // then clear it:  `short` should remain whatever it is,
  // including `true`.
  if (short_timeout) {
    clearTimeout(short_timeout);
    short_timeout = null;
  }

  if (is_short) {
    // Set a timeout to process short clicks
    clearclick_timeout = setTimeout(chainEnded, 250);
  }
  else {
    // Long press: reset
    digitalWrite([LED1,LED2,LED3], 0x001);
    clearclick_timeout = undefined;
    long();
  }

  // And clear any LEDs after a reasonable period.
  setTimeout(function () {
    digitalWrite([LED1,LED2,LED3], 0);
  }, double_click);
}

// Set the initialisation function
E.on('init', init);

Have you tried NRF.setLowPowerConnection(true)?

Realistically it'll be the actual HID connection to the computer that sucks all the power, rather than the advertising. setLowPowerConnection modifies the connection interval so next time you connect it's only polling twice a second rather than ~50 times a second.

Worrying about the NRF.wake causing a reset - I wonder - is it possible it's being called when you already have a BLE connection? It basically enables advertising, but you can't advertise when connected so I wonder if I've actually accounted for that.

I'd give it a shot with all the wake/sleep/disconnect stuff removed, and just with setLowPowerConnection - I reckon that should make a massive difference.

Thanks, I'll try that. I do want to get disconnect working in some way, as I want it to actively disconnect from the iPad so the iPad doesn't keep thinking it's got an external keyboard; unless I can figure out how to get it to realise that it's not really a keyboard, which I imagine will be nasty -- it's nasty to do in Bluetooth Classic, IIRC.

So, do you think keeping track of the HID connection (the BluetoothRemoteGATTServer via the Promise, presumably?) and issuing a server.disconnect() is the way to go?

(EDIT: Oh, hang on... that won't work... these connections are initiated by the iPad; there's no connect call going on, and we're the server, so there's no BluetoothRemoteGATTServer object. Hmm. If acting as a server rather than a client, is there a way to selectively disconnect a connection, or is it just a case of NRF.disconnect() and let the UART service and anything else be damned? )

Yes - I think you'll need to just do NRF.disconnect() - and I guess if you want to avoid it reconnecting you'll need to use sleep and wake.

If you can reproducably get the 3 lights on with a small big of code it'd be great (even better if it was without HID) - then I should be able to track it down a lot better. I wonder whether it might be related to Bonding again...

I was going to say I've managed to isolate it down to a few lines of code, but ended up with this situation:

=undefined
 _____                 _
|   __|___ ___ ___ _ _|_|___ ___
|   __|_ -| . |  _| | | |   | . |
|_____|___|  _|_| |___|_|_|_|___|
          |_| http://espruino.com
 1v92 Copyright 2016 G.Williams
>dump()
pinMode(D29, "input_pulldown", true);
=undefined
>NRF.setAdvertising({},{name:"Test"});
NRF ERROR 0x8 at ?:0
REBOOTING.
Disconnected

Hmm. I think there's something rather wrong. I've DFU'ed it back to 1v92, and tried various resets and flash resets. This is since I started doing NRF.setLowPowerConnection(true), although I've since done NRF.setLowPowerConnection(false) a few million times.

UPDATE (#3): I've managed to get it (and the other Puck I soft-bricked) back using the five-green-flashes method, E.setBootCode(), although that didn't seem to help originally.

This is all on 1v92 stock on one puck and 1v92.3045 (a fresh build) on the other.

I think the NRF ERROR 0x8 is happening when it keeps reconnecting and then advertising changes while the connection is active, or something like that. I'm not sure if that makes sense.

I've had big problems with setLowPowerConnection, as it messes with the timing, giving me runs of repeated taps.

From my limited understanding of BLE, I can see that the idea of getting the client (the iPad) to stop connecting to the Puck is problematic. The only way I can see this working is if I can shut down BLE, or HID advertisement, and then restart it when a button is pressed to wake it up. setLowPowerConnection alone might save power, but it won't handle the disconnection I need for keyboard access.

I'm still rather unclear on what can and can't be done while connections are open, and so forth. Some commands will queue for the next reset, but things like NRF.setAdvertising, NRF.sleep, NRF.wake might not have the right guards in there, and I think they may be causing the crashes and reboots.

I'm still trying to isolate some code to demonstrate it.

It's really strange about your error 8 - I guess it must have been something in the saved code.

I think I need to come up with a better way of handling the 'save on send' thing - it seems a lot of people are hitting problems when turning it on, uploading, then trying to do things after it's turned off if the existing code is still saved.

It definitely looks like there's some issue with wake, sleep and setAdvertising as you say. sleep and wake should do what you want (stopping advertising) but I bet there are some checks missing in there (maybe calling sleep or wake while connected?). They tend not to be used that much so I haven't focussed on them as much as I should - I'll see if I can find a way to reproduce it here as well.

Ok, I managed to reproduce it by calling NRF.sleep during an active connection, and I've now fixed that - so if you use one of the absolute latest travis builds then it should work for you.

When built, they should be available here: http://www.espruino.com/binaries/travis/9c179b2421900319c045c7eb39362caf3568b921

Thanks… that does seem better as far as the ERROR 0x8 fatal crashes are concerned. I'm still having trouble with a lot of BLE error 13313s, though... looks like my services are getting munged, or something like that.

I'm not really too sure about the order of execution here, and what's necessary in what order; such as whether NRF.setAdvertising() needs to be called after an NRF.wake(), whether NRF.setServices() needs to be called after a NRF.setAdvertising(). A simple test rig seems to work fine, but as soon as I start introducing a) HID and b) NRF.sleep/wake, it all starts going horribly wrong.

I expect this would be easier if I wasn't trying to do HID and UART through Web Bluetooth (which is hinky at the best of times) but finding a USB-to-Serial or rigging a Pico to do the same job and then trying to connect to the Puck non-destructively is a big PITA. I'm also suspicious of connecting to Espruinos via serial because whenever I've tried the different redirection of the console seems to change behaviour subtly.

Anyhoo. I'll soldier on. Thanks

I've never seen an error 13313 before! If you can get something simple that reproduces it then I'll see what I can do to fix it though.

For the order of setAdvertising/setServices and sleep/wake - it shouldn't matter. If they're set up when in sleep, it should just apply when you wake. Internally, if the device is 'awake', setAdvertising actually calls sleep, sets up advertising, then calls wake - so it should be fine either way.

The Error 13313s seem to come when hid.tap() calls are made on a HID connection while also doing NRF.disconnect(), NRF.sleep(), NRF.wake() in various combinations. I'm not entirely sure yet. I think it might be a case of the service characteristics getting lost after an NRF.wake(), but redoing them fails too. I'm seeing these 13313 errors in the Espruino IDE when connected with Web Bluetooth, so there is a connection open with the computer, albeit UART.

BLE_ERROR_GATTS_SYS_ATTR_MISSING composes to 13313. Pertinent? https://devzone.nordicsemi.com/question/6085/strange-error-code-13313-0x3401-returned-by-sd_ble_gatts_hvx/?answer=6106#post-id-6106

The context: the script sets timeouts to make it execute NRF.disconnect(); ... NRF.sleep() after a set time (5 mins) of inactivity; each button press will clearTimeout() and start a new 5 minute timeout. Then, when the button is pressed, an NRF.wake() call is made if necessary, before hid.tap() is called subsequently on release.

When this behaviour works (it has sporadically worked in random hack versions along the way) it works really well - the Puck disconnects and shuts down when idle, and then wakes up and posts a keypress when it's activated with a click (forward) or a double-click (back)

And, if the on-screen keyboard is wanted, you can prematurely disconnect and sleep the Puck with a triple-click. That also acts as a manual "Off" when you remember to do so rather than idling it for five minutes.

A long click does a disconnect without a sleep, allowing Web Bluetooth to see the Puck for long enough to connect to it for maintenance.

Anyway, I wouldn't analyse that too much: part of the problem of getting you a neat little test case is the unpredictability, and the limitation of debugging while also having connects and disconnects, sleep and wake happen at the same time... I'm mainly working on cryptic LED flashes. I am still trying to isolate it though. My weekend's packed, unfortunately.

As far as I understand it, HID works a bit like this:

• Device wakes, starts advertising
  
• PC/Mac sees it some time after
  
• PC/Mac connects
  
• It requests a list of services
  
• It does interesting things with those services and remembers stuff about them
  
• It requests to be notified to some of them
  
  

That all takes a bunch of time though, so even if you get the connected event I guess the HID might not have been initialised enough that a call to the NRF HID function would actually work.

What happens if you leave a reasonable delay (5 seconds?) between connecting and trying to send any HID packets?

Also, when you're getting that error, is your Mac actually showing the device as a HID device? Or does it show it as a normal serial device?

Are you calling setServices or setAdvertising more than once at all? I guess that could mess things up too - if the services change then the CCCD numbers might change, and that could confuse the computer.

I haven't tried having a delay per se., but when it's connected to the IDE and producing 13313 errors, it'll do it whenever the button is pressed, repeatedly for as long as you like.

I have been running setServices and setAdvertising whenever it comes out of sleep with NRF.wake() too... I wasn't sure if doing NRF.sleep() wiped the existing records. Regardless, I was trying to do setAdvertising() to update the battery level, so it would need to be called repeatedly.

To be honest, the lack of an isConnected() and an isSleeping() method is a bit inconvenient. Getting the status of the softdevice would be useful, rather than trying to track it manually.

Hmm. Something's getting stuck, I think.

With this stripped-down code on 1v92.91; after a power cycle and an E.setBootCode("");, an upload and a refresh, I get Error 13313 when clicking. However, if I Remove (ie. unpair and forget) the Puck with Opt-click on the menu bar, it works okay.

var hid = require("ble_hid_keyboard");

var is_connected = false;

var ad_values = {};
var ad_options = { name: "Page-Turn-o-Matic 4000" };

function on_connect () {
  digitalPulse(LED3, true, 100);
  is_connected = true;
}

function on_disconnect() {
  digitalPulse(LED1, true, 100);
  is_connected = false;
}

function btn_released() {
  if (is_connected) {
    digitalPulse(LED2, true, 100);
    hid.tap(hid.KEY.RIGHT, 0);
  }
  else
    digitalPulse(LED1, true, 500);
}



function init () {
  NRF.on('connect', on_connect);
  NRF.on('disconnect', on_disconnect);

  clearWatch();

  setTimeout(function () {
    setWatch(btn_released, BTN, { repeat:true, edge:'falling', debounce : 50 });

    NRF.disconnect();

 //   ad_values[0x180F] = Math.round(Puck.getBatteryPercentage());
    NRF.setAdvertising(ad_values, ad_options);
    NRF.setServices(undefined, { hid : hid.report });
  }, 3000);
}

E.on('init', init);

I'll start putting stuff back in, bit-by-bit, and unpair if/when it goes bad. It's times like these I really wish Puck had a USB port.

:) Yeah, it's not that hard to add a USB-serial converter to it, but doing the wireless programming can be a bit of a pain.

I'd call setServices and setAdvertising immediately on onInit if I were you - and while calling setAdvertising again shouldn't be an issue I'd avoid calling setServices - that'll trigger a softdevice restart which could confuse the Mac.

It sounds a bit like what's happening is the Mac is 'forgetting' that it's a HID device though, so when it connects it doesn't request the HID stuff, and then you get the error when trying to send a keypress. It's possible that now it's working, without the repeated setServices, it'll keep working.

However if you connect again at any point when it's reset, the Mac might see there's no HID and then stop treating it as a HID device - so you'll have to go through that whole remove/pair thing again :(

The delay during init is really so it can do a disconnect -- and hence do the set* calls without queueing -- without interfering with the upload loop. I've found that if onInit immediately disconnects, it can confuse the IDE.

I've run out of batteries for the pucks right now -- I've been away from home this weekend -- but I think the code works fine until sleep and wake get involved. After that, it's all a bit unpredictable. Now I've got a barebones implementation, I'll try to add them back in.