Streaming Pixl.js LCD Content over Web Bluetooth

Posted on
  • A new experiment I have been working on, allowing to easily add remote control (over Web Bluetooth) to existing Pixl.js projects:

    https://medium.com/@urish/mirroring-lcd-­display-over-web-bluetooth-bb9ff2cb5d30

    In action:

    Espruino-mirror In Action

  • That's cool, I like it!

    In other message I ask about reading a 57x57 pixels QRCode from the pixl's screen, the idea is that you read it with the phone and it takes you to a web page where you can configure it, so the UI of the device setup/config is just a matter of a bit of html+js that runs on the browser on the phone, and I'm relieved of having to program it all from scratch to run on the espruino.

    Hey, could you please transfer these QRCode images to the Pixl and try to read them with your phone, please?

    http://forum.espruino.com/comments/14464­888/

  • @urish, you picked yourself a nice, challenging cherry... or may be more like a basket full of all kinds of tasty fruits...

    (forum entry got somehow duplicated while editing it... so I break it apart at this spot).

    NOT btw, I notice the subtile - subversive? - presence of lovely 'fine' art in the prevailing cold, stone hard technology information stream... (there are obviously still cycles available to burn on non-essentials... which make life worth living!)

  • There are some nice challenges in the implementation, I feel... and can think of right of the bat...

    The synchronous and uninterrupted transmission of the screen triggered by the .flip() is a great start. I do though not know about the side effects that keeps you BLOCKED / un-interruptable busy for 40% of the time assuming a .flip() happens (at least?) about every second... It does with the (implied) clock application, when seconds are in use...

    To stay responsive - to remote control commands - I'd separate the sync out from the flip with a snapshot buffer and frequency limiting timer, and also make the loop over the chunks javascript process interruptible. For the actual transmission, I would implement a (multi queue, queue, multi-priority) queued channel / time multiplex system so that each packet can transmit what ever is of highest priority. Load of the queue would slow down the frequency of the display sync events (delay the completion of the current one, up to the point with deferring the start of the next one)... For the screen streaming, it become then like a flip-flip or double-flip / multi-flip pattern when looking from start to finish of one complete screen update.

    A simple setup with only a few things but very interesting and sufficient to study the many issues communication w/ band limitation faces... and visualize what actually is going on.

    Side note: Version 2v00 celebration w/ my 2k-th post... ;-)

  • Wow - that's great!

    Just a few random thoughts...

    • Espruino keeps track of the area of the display that's modified with g.getModified() - so potentially you could send just the area of the screen that changed to make it faster?
    • Espruino v2 has the 'heatshrink' library for LZSS compression which you could use to get the size down. I don't think there's a JS implementation (looks like there may be a TypeScript decompressor), but I guess one could be built with emscripten.

    I'm totally up for increasing the MTU size in Espruino as well - just have to figure out how to do it :)

  • So I made a few tweaks following lots of @Gordon's advice, and I got it to support 50ms frame delay.


    1 Attachment

  • Wow, that's very cool. Do you have the code you used?

    Did you use heatshrink in the end, or was it just only sending the contents that had changed?

  • Lovely! I'd love to see the updated code as well - or if you are up to it, a Pull Request will also be appreciated :)

  • @Gordon its just using heatshrink. More savings to be made if we sent just the changes as well but we'd need two packet types then, one for the initial "full" send, and then another for just updates.

    @urish I will work on a pull request, the code is very bad at the moment. I need to send 1 frame of all 0x00 bytes before the real frames, then 1 frame of 0xFF bytes so that we can identify the "real" frames and make sure they are in order (sometimes they out OOO even though BLE should order them). I'm sure there are many improvements to be made.

  • Awesome. What did you use for the heatshrink decoding in the end? It'd be great to get a link to that in the docs as it seems like a pretty common use case

  • I went with https://www.npmjs.com/package/heatshrink­-ts just had to configure it with the

    WINDOW_BITS, LOOKAHEAD_BITS, INPUT_BUFFER_LENGTH

    parameters that you found for me.

  • Still looking for some more optimisations here as I'm running out of memory.

    @Gordon the docs for https://www.espruino.com/Reference#t_hea­tshrink say it doesn't support streaming, but I know that the C implementation does. Is this something you could look at changing?

  • Not easily - the intention was to support streaming initially but it looked like it was going to be a bunch of extra effort to do :(

    To be honest the display contents are 1kB, so compressed it must be much less than that? Realistically the extra code to support streaming (and memory used to hold compressor state while doing the streaming) would blow through that.

    Have you considered 'save on send' to flash? that should allow you to keep the function code in flash which will save you loads of RAM

  • Yeah I'm already doing Flash save-on-send.

    I don't run out of memory on every loop so I must either have a leak somewhere, or I'm confusing the GC and it isnt cleaning up properly.

  • Post a reply
    • Bold
    • Italics
    • Link
    • Image
    • List
    • Quote
    • code
    • Preview
About

Streaming Pixl.js LCD Content over Web Bluetooth

Posted by Avatar for urish @urish

Actions