Experimental text input method on Pixl.js/Chording-ish Bluetooth keyboard

Hello all,

I thought I'd share the code for the input method I've just thought of and been playing around with today. (Lockdown creativity!)

I've been wanting a method to enter text "on the go" on my phone, so thought that a Pixl.js was a good way to experiment. The attached code implements a Bluetooth keyboard that works with a Pixl.js with no other hardware.

The idea is to press buttons in sequence then release them. So BTN4 is 'T', BTN3 is 'A' etc. To get other letters you hold down a sequence - 'H' is hold BTN4, hold BTN3, release both. 'P' is hold down three buttons then release all.

This is a bit complex to remember, so I made a graphical pattern for each letter - see the attached picture. This works holding a Pixl "rotated" so that the 'Pixl.js..Espruino' writing is on the right hand side as you hold it, so that 'a' is the top right button, 'e' is the bottom left.

I don't think it's too bad to use; I was writing text reasonably quickly and found I was beginning to memorise the patterns. I've used letter frequencies so (in English) more common letters need fewer presses.

I'd like to expand this to have numbers and other characters, and possibly to work with a capacitative sensor so you can just write out the shapes with a finger. It would be cool to have this sensor woven into fabric, or via a tiny sensor with four quadrants you could wear as a thimble or something.

Future plans:

• Work out how to add more characters - e.g. numbers and capital letters. Theoretically the pattern length can be infinite, or maybe have a combination to switch 'mode'.
  
• Work out how to use 4 cap sense inputs on the Pixl and experiment with conductive thread to make pads on fabric. I've got a couple of I2C cap sensor chips lying around but I suspect the NRF chip can do this.
  
• Work out how to bond the Pixl to the phone so the phone automatically connects to the device - currently I have to manually connect on the phone.
  
  

What do you think?

Colin

The code is below (Indentation has gone a bit wrong but hopefully it's readable)

To explain the code:

• states maintains a mapping of buttons to letter. The starting state is '_'.
  
• Pressing button 1 moves to the 'E' state. If all buttons are released then 'E' is emitted. If more buttons are pressed then the state moves from 'E' to the next state according to the button.
  
• '?' is an error state - e.g. pressing button 1 twice without releasing it isn't possible.
  

• '!' is an unused state so extra things can be added.

      
  // Key state transitions
  var states = {
  //  BTN. 1   2   3   4
  //Single key
  '_': ['E',' ','A','T'],
  // Two keys
  ' ': ['L','?','I','M'],
  'E': ['?','D','C','S'],
  'A': ['U','O','?','R'],
  'T': ['N','W','H','?'],
  // Three keys
  'W': ['F','!','!','!'],
  'D': ['!','!','!','G'],
  'C': ['!','J','!','Y'],
  'H': ['P','!','!','!'],
  'S': ['!','B','X','!'],
  'M': ['V','!','!','!'],
  'I': ['K','!','!','!'],
  'N': ['!','Q','!','!'],
  'R': ['Z','!','!','!'],
  'Z': ['!','CR','!','!'],
  // Error
  '?': ['?','?','?','?']
  };
  
  function keydown(key) {
  if (state=='?') Terminal.println("Error - bad state");
  ++keydowncount;
  state=states[state][key]; // Transition to the new state
  }
  
  function keyup(key) {
  --keydowncount;
  
  if (keydowncount==0) { // All keys released: Emit the key
    emitkey(state);
    state='_'; // Reset state transition
  }
  }
  
  // Do whatever when the key is released
  function emitkey(letter) {
  Terminal.print(letter);
  // CR hack
  if (letter=='CR') Terminal.println("");
    
  if (kb) {
    // CR hack
    if (letter=='CR') {
      kb.tap(kb.KEY.ENTER,0);
    }
    else
      kb.tap(kb.KEY[letter], 0);
  }
  }
  
  // Map a button to the position in the state array
  function wireKey(button,position) {
  setWatch(function() { keydown(position); }, button, { edge: 'rising', repeat: true});
  setWatch(function() { keyup(0); }, button, { edge: 'falling', repeat: true});
  }
  
  function onInit() {
  keydowncount = 0;
  state = '_'; // Starting state
  
  wireKey(BTN1,0);
  wireKey(BTN2,1);
  wireKey(BTN3,2);
  wireKey(BTN4,3);
  
  kb = undefined;
  kb = require("ble_hid_keyboard");
  NRF.setServices(undefined, { hid : kb.report });
  }
  
  

1 Attachment

Tue 2020.04.21

Wow @ColinP what an ingenious way to encode button presses! Bravo!

I like the way frequency of use is taken into consideration to speed input. While I don't have the time to test out on my Pixl until later this weekend, I'm sure it will be a delight!

What would really be cool is to have a small area of display for hinting the movements until enough skill has been learned to do so from memory. Considered that?

'possibly to work with a capacitative sensor'

What about tracing the characters in the air using a Puck connected via BLE?

That's really neat! Thanks! The way you laid out the code is really tidy too.

I guess you could actually just put 4 switches together in a grid and then you could pretty much do everything with your thumb?

I guess you could stick them to something like a Puck.js and you'd have this tiny text input method in your pocket. It'd be really cool.

About reconnects: What exactly causes the phone not to reconnect? Is it after you load up new code, or power cycle the Pixl? Is it ok if you just move away from it and go back to it?

I know that booting up with BTN1 held will clear out the bonding info, but I wonder whether it happens at other times as well.

Thanks for the positive feedback!

Robin - I did actually display the letters on the screen to show which button did which. I wrote a prototype for this using a web browser a while ago - on the computer it did work OK, but on the Pixl I found it easier just to have a paper reference and think of each letter as a shape. Annoyingly I deleted the Pixl version which did draw to the screen!

If you're interested in trying then I think this code should work as a base, but I just wrote it and it might have bugs (e.g. I think it will show the ! unused marker). Changing the console.log to g.drawString in the appropriate place, and calling the showLabels(state) function after key up and key down should do the trick I think.

// Traverse the tree to show characters reachable from a certain state
// as a string
function reachablefrom(state) {
   if (state=='?') return ""; // End recursion
   return reachablefrom(states[state][0])+
     reachablefrom(states[state][1])+
     reachablefrom(states[state][2])+
     reachablefrom(states[state][3]);
}

// Show characters available from each button
function showLabels(state) {
   console.log("KEY1",reachablefrom(states[­state][0]));
   console.log("KEY2",reachablefrom(states[­state][1]));
   console.log("KEY3",reachablefrom(states[­state][2]));
   console.log("KEY4",reachablefrom(states[­state][3]));
}

Gordon - I'm glad you're thinking the same as I am about the Puck making a tiny input device. Funnily enough that's why I bought it a while ago and am only just having time to play with it :-)

I'm actually wondering about some sort of finger movement detection sensor rather than buttons - It feels to me that 'writing out' the shapes by swiping a finger around might actually be quicker than pressing and holding buttons in sequence.

One option might be to use an optical mouse sensor - but they seem a bit harder to get hold of now (That's what I get for throwing out a hoard of old mice...note: never throw out old junk). The APDS-9960 sensor looks like it can do gestures from a distance, but I'm thinking more of touches - faster movements much closer to the device.

A capacitative sensor might work, but I'm not sure having 4 pads and just sensing a touch on each would work, as ideally it would be quite small and a fat finger might just smash all pads at once. I've got hold of an MPR121 board, which looks as though it can read the raw value of each sensor. Doing something clever with the raw values might allow movement detection - I will need to experiment.

Regarding the reconnection on the phone - I'll need to experiment more. I had a bit of a frustrating time when the Pixl stopped accepting any connections and I couldn't reconnect to debug it. I even put the Pixl in the microwave oven (with the oven off!) to see if it made a difference, as I got paranoid that somehow one of my neighbours was connecting to the UART service all the time! And shielded by the microwave there was no connection - so the problem wasn't the Pixl thinking it had a connection when it didn't. It turns out that my Mac was auto-connecting to it as soon as it advertised as an HID device and was staying connected even when the screen was closed (I thought if it was the Mac closing the screen would disconnect, so this sent me down a wild goose chase). Grr. So I will try the phone connection again with my Mac switched off - maybe it was working all along. I haven't yet investigated the bonding and PIN stuff however - I will do this also which might help.

...pretty cool, and even without AI (real I happened at design time).

As with any coding of any sorts including computer (machine) coding - a great deal of practice is required... Having a display at hand is helpful...

Hmm, I wonder whether something like https://www.ebay.co.uk/itm/202275372451 is available as a 2x2 block.

Other option (although still tactile) is to use a little joystick switch?

Hello,

Thanks for the block hint. I've actually got a 12 key version of that somewhere which I could try. The joystick switch also sounds interesting.

I ordered an MPR121 touch sensor board, which has just arrived. I'm currently glueing pieces of tin foil onto a piece of cardboard as an experiment. I'm firstly going to try a chord keyboard to compare, then will see if I can fashion some touch sensors into a square such that they detect swipe movements.

In a stunning example of high quality electronics construction(!) - no amateur construction of sticky tape, Pritt stick and tin foil here - I had a go this afternoon at making a gesture version, so you "write" letters by swiping the patterns above.

There are four pieces of tin foil glued to a piece of cardboard, an MPR121 breakout board and a MDBT42 module at the top. To ensure a quality construction the tin foil is covered with a thin piece of plastic from an old envelope.

It works surprisingly well, and I find the gestures are becoming part of muscle memory.

It would be cool to see if the nRF52 chip could implement four touch sensors directly -without the mpr121-, and then 4 pads could be attached to the cover of a puck. I know there is one cap sense input, but unsure if it can be easily extended to four, and how well they would work compared to the auto-calibration and filtering that the mpr121 chip does.

1 Attachment

The modifications to the above to get it to work with MPR121 - put the following into onInit()

SCL=D15;
SDA=D14;
IRQ=D16;

I2C1.setup({scl:SCL,sda:SDA});

sensor = require("MPR121").connect(I2C1, function() {});

// Function to be called when MPR121 notifies us of change
setWatch(function(e) {
    var value = sensor.touched() & 0x0f; // Filter out other inputs
    if (value==0) { // All buttons up: Emit the key
      emitkey(state);
      state='_';
    }
// Buttons map to MP121 inputs 0,1,2,3 -> bits 1,2,4,8
    if (value==1) keydown(0);
    if (value==2) keydown(1);
    if (value==4) keydown(2);
    if (value==8) keydown(3);

}, IRQ, {edge: 'falling', repeat: true});

That's great! Love the capacitance sensors :)

The nRF52 has this weird configurable hardware thing in it that can be used for stuff like capacitance sense (on multiple channels). See http://www.espruino.com/NRF52LL#hardware­-capacitive-sense-on-two-pins

It's getting pretty hardcore at that point, but you definitely could do hardware capacitance sense directly (however it would hurt battery usage).

Thanks for the capsense info. I saw it before, but couldn't remember where. (You might notice the two 470k resistors connected to the module. That was a previous attempt at sending morse code!)...