Serial Keyboard / Mouse to Websockets/Xdotool Keyboard & Mouse events :)

Hi there !

I am currently working in a project that involves acting as a keyboard+mouse combo & sending data over serial.
Since I only have an original Espruino boardon hand & a soldered HC-05 module ( NOT flashed with a RN42 firmware .. ), I can't:

• set the HC-05 to some hid device ( I could by flashing it with the RN42 firmware, and also be able to choose whether to describe itself as spp or hid by setting the GPIO11 pin high on power up, but not spp+hid combo here )
  
• use the Espruino USBKeyboard or USBMouse since theses don't work for the original Espruino board.
  
  

I ended up writing a little nodejs app to forward & parse serial stuff to either websockets or xdotool:

• using xdotool, the keyboard & mouse events are OS-wide & seems to work flawlessly
  
• using websockets, the client ( running in a web page ) creates the necessary DOM events
  
  

This solution is quite nice for quick tries & until I get myself a board supporting my needs ;)
Also, it's API is based on Gordon's USBMouse & USBKeyboard implementations ( thx for the code btw - I didn't forget to mention the original author in the files hosted on the repo ;p ), so usage code is almost identical :)

It can be found on the following repo: https://github.com/stephaneAG/serialToWebsockets

Last but not least, I'll be looking for an implementation offering Mouse + Keyboard + serial over bluetooth ( and later ble/smart ) & I don't know if I'd be able to do so using, for example, the Puck.js
( as a start, an usb hid for serial+mouse+keyboard 'd be neat )

I now know better how to write hid report descriptors, but I'm not sure of how easy it'd be to have such "composite device" working quickly ;)
On the hid subject, I had some fun writing a quick ( hugely wip ) helper to generate a descriptor's hex representation ( if anyone better knowing the subject is interested: https://github.com/stephaneAG/usbHidDescriptors ) :)

ps: as the author of https://eleccelerator.com/usbdescreqparser/ states it, the official program to generate hid report descriptors is "unlikely anyone's cup of tea", and anyway is only available for windows .. if anyone knows of a web-based implementation, I'l be glad to know :)

Hoping this helps someone else, any advice is hugely welcome :)
Happy coding everyone ;p

Looks neat - thanks!

Puck.js or any of the BLE espruino devices will handle Bluetooth HID devices and serial via some modules that are very similar to the USBKeyboard/etc ones: http://www.espruino.com/Puck.js+Keyboard

However handling composite devices I'm not sure about... That might need some work, but I'd have thought it would be possible!

;p

thanks for the quick answer :)

I actually did have some luck when lastly trying to connect to a Puck.js from Mac OS Xusing a CSR4.0 USB dongle ( no config was necessary :P )

As soon as I get my hands on it , I'll try setting the hid to a composite device & giving you a feedback on this :)

Also I was wondering how would appear a pico acting as a hid device + an HC-05 with original firmware ? ( I am guessing nothing, since the HC doesn't natively act as a "passthrough" device but rather as an spp one ? )
-> on this also, I guess I'll see what are the results I get ( .. )

On another subject, I have a faulty pico that doesn't seems to respond to anything ( and that has red & green leds lit but not pulsating when connected to USB while its button is pressed ): I wonder how to get it back to life ( .. )

So, I'll post updates as soon as I got those :)
++

Also I was wondering how would appear a pico acting as a hid device + an HC-05

Yes, the Pico could be USB HID, but HC-05 is just standard Bluetooth serial

I have a faulty pico that doesn't seems to respond to anything ( and that has red & green leds lit but not pulsating when connected to USB while its button is pressed )

Could start a new thread for that? Are the LEDs lit brightly? Can you make sure the USB contacts are properly clean? Either use contact cleaner, or even a pencil eraser will often work.

Is it possible you powered the Pico via the USB connector from voltage significantly over 5V? Sometimes that can cause the USB voltage detection circuitry to blow up - but even if that's dead, you could still program it over serial.

Hi there !
-> as you suggested, I'll start another thread for half-dead Pico ;)
-> still, the LEDs are indeed led brightly, the contacts are clean, and I put some tape on the backside of the USB connector since part of it was a little scratched ( even if it shouldn't impact ? .. ), and it's very unlikely that I powered this from something other than a laptop USB port at 500mA ..

I had some luck/fun using USBKeyboard.js on an Espruino WiFi, only to find that the mapping is for us/qwerty ;)
-> hence I wrote the following to help toggling between the two: https://github.com/stephaneAG/serialToWebsockets/blob/master/USBKeyboard.js
.. and then realized that I now need to map dot, colon & cie as well ..
.. and also add support for left & right parenthesis as well as '#' ( I plan to use the keyboard feature combined with a TCS34725 color sensor to type rgb or hex color ;P ) ..

In short, the following seems to be useful on a system running Ubuntu:

xinput // to get the ID of our device that acts as a keyboard
setxkbmap -device 13 -layout us // to set its layout

The sad part is:

• it has to be run after each time the keyboard is "connected"
  
• it won't work for other devices / mappings
  
  

So right now, I'm wondering what'd be the most practical approach to get the most common layouts selectable within the Espruino USBKeyboard.js module ..

I guess I'll investigate the subject right after finishing up the po with the TCS34725 ;)

Hi there !

A little update on the topic: I now have a working implementation of a modified version of USBKeyboard.js ( modified to handle FR layout & "high-type" feature* )

https://github.com/stephaneAG/serialToWebsockets/blob/master/USBKeyboard2.js

It could be used as a drop-in replacement for USBKeyboard module, and while it's a little bit bigger, it's quite nice to use ( at least to my taste ;p )

*the main goal, aside from having a layout that maps to my keyboard key mapping, is to be able to 'type' stuff while not taking care of which modifier flag(s) have to be set for some specific character: just '.htype()' & this 'll be handled internally :D

usage examples:

var kb = require("USBKeyboard2");
// use as the original USBKeyboard module
setWatch(function() {
  kb.setModifiers(kb.MODIFY.SHIFT, function() {
    kb.type("HELLO WORLD", function() {
      kb.setModifiers(0, function() {
        kb.tap(kb.KEY.ENTER);
      });
    });
  });
}, BTN, {debounce:100,repeat:true, edge:"rising"});

// enjoy niceties
exports.htype('abcdefghijklmnopqrstuvwxyz');
exports.htype('abcdefghijklmnopqrstuvwxyz'.toUpperCase());
exports.htype('aAbBcCdDeEfFgGhHiIjJkKlLmMnNoOpPqQrRsStTuUvVwWxXyYzZ');
exports.htype('1234567890');

// combine with .tap() .. and do inception ? ^^
exports.htype('exports.htype("lol");', function(){ exports.tap(KEY.ENTER); });

// combine any char & it will auto handle applying the right modifier flag(s)
exports.htype('rgb(0, 255, 128)');
exports.htype('#0FAC3A');
exports.htype('hsl(360, 100%, 50%)');

// not all characters are supported this way, but adding them is quite easy ( ex: for "©" )
exports.htype('Hello World Espruino© USBKeyboard2.js module !');

Support for accents & additional characters is to be added in a near future ;)

Side note: it seems the ™ character isn't supported in the Espruino IDE ? ( it prints "?", which isn't the mapped key ? .. )

Now onto testing "mixed" device ( appearing as a keyboard + mouse combo ) on it & then on puckjs, I hope this 'll be helpful to some folks out there ;P
+

it seems the ™ character isn't supported in the Espruino IDE ?

I'm not sure it's part of the non-unicode (<255) charset? It doesn't seem to be part of http://www.asciitable.com/ - but if it is, you could always just escape the keycodes you need?

Also, Espruino won't handle even characters above char code 128 in the code itself - only in strings

hi there !

it seems not :/ ( also absent from https://theasciicode.com.ar/ascii-control-characters/escape-ascii-code-27.html ) & is Unicode U+2122 ( https://en.wikipedia.org/wiki/Trademark_symbol ).
For my current goal, it won't be a problem since the char did appear outside of espruino's IDE ( ex: an open atom or textedit window will work, as will do a webpage .. ) ;)

well noted for the charcodes above 128, thx for the hint :)