App layout library

Hi! As some of you know, I'm looking into launching a Bangle.js 2.0 soon which should resolve a lot of the niggles that folks have had with the original Bangle. One part of that is moving to a sunlight readable full touch LCD which has a different resolution (176x176) to the current Bangle's 240x240.

This presents an issue for pretty much all existing apps as they're all laid out for the 240x240 screen, and while they could be converted ideally we want the same app to be able to run on both devices.

However this feels like a an opportunity. In many cases all we want is a simple layout - sometimes just one line above another - so a library to handle this would solve a lot of issues and would clean up a lot of apps. There's also the possibility of easily handling touch input and maybe even having some layout tools.

Does anyone have any thoughts about this? Can you think of any nice solutions?

I'm imagining a simple thermometer app might look something like this:

var layout = ["vertical",
  {type:"text", font:"6x8", txt:"Temperature"},
  ["horizontal",
    {type:"text", font:"30%", txt:"20.0"},
    {type:"text", font:"6x8", txt:"'C"}
  ],
  {type:"button", txt:"more...", touch:myTouchHandler},
  //{type:"custom", width:16, height:16, render:myRendererer},
  //{type:"image", src: _=>atob("imagebase64code")},
];

Layout.update(layout);
Layout.render(layout);

setInterval(function() {
  layout[1][1].txt = E.getTemperature().toFixed(1);
  Layout.update(layout);
  Layout.render( layout[1]); // re-render just this part
});

But I'm very open to suggestions. We just need to keep things as simple and lightweight as possible.

Ah, I like it, just like the html flexible layout boxes?It's friendly to frontend developers!

I'm not using bangle.js yet (mostly because I'm stuck in a SEA country with unreliable shipping, though plan on getting one shortly -- maybe version 2? :P), but I feel this would be a real break through issue for Espruino. I often write code with display elements on say an OLED screen (128x64), but then after want to redeploy this code on a 160x128 or 240x240 or 320x240 screen -- you get the idea -- and it's a real pain.

So I'm suggesting it deserves some thought as it's a real opportunity. I like Flexbox and something similar might not be a bad first step!

I'm not sure what it would look like, but lines for example could be expressed in terms of percentage of screen (start at 10% and go to 80%), or text could be positioned at 1/3 of the screen or who knows. I guess this would break a little if aspect ratios changed. Some ST7735's are 128x128, vs 160x128. But even just that would make resolution independent display elements easier.

Some ideas...
-hfc

I like that very much. What about position and color? I know about to make it lightweight as possible is a goal, but it should not be too limiting. First thing I notice is that position, length, color follow some rules which are for me a bit too limiting.

More later. Based on experience gained in https://www.espruino.com/ui (UI Framework for microcontrollers) - no visuals yet (my bad) - based on Modular and extensible UI framework and ui elements. with a lot of visuals - work that started 5+ years ago with last updates less than 2 years ago and was developed in an emulator in the browser... - I would say that layout should not go as crazy as that but presented specs are a Byte(! - not just a bit) frugal.

What I also can say that layouts need to be hw oriented... otherwise complexity ends up comparable with a (graphical) Web browser... and I would consider those be the most complex applications ever exist(ed).

PS: @MaBe worked recently thru some layout design and implementation exercise and may spare a cent or two on this... (sorry @MaBe, squeezing even more out of the already time-emtpy 24h day-lemon - :\ ).

Yes, I think this is a good opportunity, not just for Bangle.js but for Espruino in general.

And yes, as much as it'd be nice to follow a web browser API, I think we're going to have to stay pretty low-level to make this perform well. I think we should aim to handle 80% of what people generally want to do, and then if there's anything more fancy it can be handled in code with a type:"custom" handler.

Good point about color - there could definitely be color: fields (or maybe just a config field that contains a function that sets everything up?).

And I think the ability to re-render just one part of the UI is pretty crucial as well, especially for Bangle.js where it's unbuffered. Being able to do the update in a more or less flicker-free way would be great.

This would be very nice, fiddling with UI positions is definitely one of my least favorite activities.

I think we also want align:, and a way to specify which part of the screen to use. a) so you can use it with or without widgets, and b) to test if a layout fits on a smaller screen.
(But default to just using the whole screen, or even check if Bangle.loadwidgets() has been called?)

It would also be nice if you could make one part get all "remaining" space, so you could do something like this:

var layout = ["vertical",
  {type:"text", font:"6x8", txt:"Temperature"},   // centered? at top of screen
  ["horizontal",               // right below "Temperature"
    {type:"text", font:"30%", txt:"20.0"},    // against left screen edge
    {type:"text", font:"6x8", txt:"'C"}       // against right screen edge
  ],
  {type:"custom", size:"fill", render:drawThermometer},//draw in middle of screen
  {type:"button", txt:"more...", touch:myTouchHandler},   // centered? at bottom
];
function drawThermometer(x,y,w,h) {
  // x,y,w,h: available space after other parts are placed
}

Some more questions:

• What happens if you re-render just one "6x8" text, but it now has a different length?
  
• How to handle buttons on a Bangle.js 1.0?
  

think we also want align

Me too,....

align

Yes, absolutely.

a way to specify which part of the screen to use

Yes - initially I was planning on just making it check if widgets were loaded and size accordingly, but hopefully it'd be flexible enough that you could specify a width and a height to fit to.

one part get all "remaining" space

Thanks - yes, that'd be a great idea. Just got to figure out how to handle that!

What happens if you re-render just one "6x8" text, but it now has a different length?

I was thinking Layout.update runs over everything, and if the size/position of anything gets changed, it is marked as 'dirty'. Then you can maybe choose to just render the bits that have changed?

How to handle buttons on a Bangle.js 1.0?

I think we just use BTN1/BTN3 to move focus between them, and BTN2 to select - a bit like we do for showPrompt currently? It'd be one of the benefits of doing stuff this way as well.

What do you all think about something like this: https://www.espruino.com/ide/?emulator&upload&gist=8cad25a87c633e2c614d0eefa52fcd2e

The layout for an app might look something like this:

var layout = { type: "v", content: [
    {type:"txt", font:"6x8", label:"Temperature", col:"#f00", halign:0, pad:4},
    {type:"h", content: [
      {type:"img", src: ()=>atob("CgoCLguH9f2/7+v6/79f56CtAAAD9fw/n8Hx9A=="), valign:0, pad:4},
      {type:"txt", font:"30%", label:"20.0"},
      {type:"txt", font:"6x8", label:"'C", valign:-1}
    ]},
    {type:"txt", font:"4x6", label:"left", halign:-1},
    {type:"txt", font:"4x6", label:"center"/*, halign:0*/},
    {type:"txt", font:"4x6", label:"right", halign:1},
    {type:"custom", _w:64, _h:64, render:myRenderer, fill:true},
    {type:"btn", label:"More...", click:() => print("Hello")},
    {type:"custom", _w:64, _h:64, render:myRenderer, fill:true}
  ]
};

So you can have horizontal/vertical alignment, the ability to fill available space, padding and color. By putting in elements with fill:true you can pad out certain areas too,

Nothing sorted for button handling yet, but so far the code seems like it'd be reasonably fast and compact.

Maybe the thing to do is to look at existing apps and see how easily they can be converted to use it.

A clock might end up looking like this:

var layout = { type: "v", content: [
    {type:"txt", font:"30%", label:"12:00"},
    {type:"txt", font:"6x8", label:"Date"},
  ]
};

function draw() {
  var d = new Date();
  
  layout.content[0].label = require("locale").time(d,1);
  layout.content[1].label = require("locale").date(d);
  Layout.clear(layout);
  Layout.update(layout);
  Layout.render(layout);
}

g.clear();
draw();
setInterval(draw, 60000);

Right now the width and the height of items can't really be defined explicitly (other than for custom). Potentially we could add the ability to define a minimum size, and maybe even allow things like "50%" to make something take up 50% of the available screen height.

Very cool !

Yeah that's great! I can already see myself building some menus with something like this ;) And it is more what I had in mind too -- I didn't mean to suggest we should implement a CSS style layout engine. This really hits the "elegant simplicity" zone.

I like the idea of relative heights/widths (DroidScript does this for Android: 1.0 = full width/height, so use 0.5 for half etc) as it should adjust to other screen sizes automatically. Also if fonts were (somehow) density-agnostic (like device independent pixels) so if I load font_small it's 4x6 on lower resolutions and 6x8 on higher.

I also like the layout manager approach: I'm guessing "v" is vertical here, so top -down is simple, then nesting an "h" inside the first "v" element gives you a grid, and you could have anchors like "ne", "n", "nw" to align to corners, top, bottom (in an abbreviated format)

also it would be nice if the Layout automagically knew to redraw when it's content changed so you wouldn't have to .clear() .update() ... just .render(). That last one may call g.flip() as needed too?

now mapping 12-bit colors down to 3-bit... ouch.

May be a good idea for someone with LOTS of time on their hands to go through the bangle apps and try to lay them out using these rules, see how complex it gets, see where the majority tend to pattern.

I'm in favour as long as the results look nice at the end of the day.
Nothing worse than being forced to build to a butt ugly ui layout.

Also would be nice to keep it a simple as possible with sensible defaults but with the ability to augment for more complex things.

If BTN handling is going to be part of this can we have long press built into the firmware so we can assign a call back for a short press and long press to different functions. I do this through timers at the moment.

Really looking forward to the new watch going on Sale. I'd like to pre-order if I could.

I'm guessing "v" is vertical here, so top -down is simple, then nesting an "h" inside the first "v" element gives you a grid

That's right, yes.

and you could have anchors like "ne", "n", "nw" to align to corners, top, bottom

Not sure I understand this one? We have halign/valign - but maybe ne/n/nw is tidier?

it would be nice if the Layout automagically knew to redraw

Yes, I was wondering about that. Thinking about even the clock example I feel there might be too many edge cases for us to be able to implement it easily.

now mapping 12-bit colors down to 3-bit... ouch.

If you use the #rgb color format it's not too painful - I've been going through some of the existing apps/widgets getting them working already. I've found that because you know it's going to round the values you can generally get something that ok on both. Having said that there's now g.theme (since the new 3 bit screen looks best with black on white, and Bangle is nicer with white on black), and that can mess things up!

go through the bangle apps and try to lay them out using these rules

Absolutely, yes! As a first start I think i'll add this library to BangleApps and then see about modifying one or two apps.

Nothing worse than being forced to build to a butt ugly ui layout.

I'm open to ideas here to make it produce nice results. At the end of the day nobody is forced to use it - the plan is to make it easy to build common app layouts, but also to allow folks to render all or part of the screen themselves if they want to.

can we have long press built into the firmware

Maybe that's part of a bigger problem. I guess in a lot of apps you will want the 3 'hard' buttons to do specific tasks rather than having on-screen buttons.

So really you want:

• On Bangle.js 1.0: 3x labels down the right of the screen in line with the buttons - direct access from the Bangle's buttons
  
• On Bangle.js 2.0: 3x buttons down the right of the screen - touches on-screen to make them work
  
  

And that wants to be built into the layout library somehow. Adding a long-press callback to that should be pretty easy though

it would be nice if the Layout automagically knew to redraw

Yes, I was wondering about that. Thinking about even the clock example I feel there might be too many edge cases for us to be able to implement it easily.

Not to be contrarian, but when using a using a g.flip() approach I always have a "screen redraw" function in my code after doing some updates. The Espruino app code is generally so small that hand crafted screen updates seem like the logical thing to do -- so I'm fine with calling a "layout.update()" function.

-hfc

Just pushed some updates to the Gist, with the ability to have buttons (which work on original Bangle as hard buttons or the new one as 'soft' buttons)

Wondering if the layout render will be smart enough to avoid flicker with the direct screen writes. I often write bits of code to reduce flicker by checking that the text I am about to g.drawString() is not just the same as the previous one I wrote. I usually keep a prevText variable and check before doing the clearRect() and drawString(). Now I know you can use an ArrayBuffer but I found they had a performance impact and that they caused memory fragmentation and then OOM errors when switching between watch faces in a multiclock format.

In terms of BTNs I think I might want to redesign my App to work with a single button. Probably use more swipes etc for some of the cycling through features I have done using BTN1 etc.

Hoping that we will still be free to code the old way if needed and that the layout manager is not the only option. The layout manager will need to be aware of parts of the screen where an ArrayBuffer might be used or a dynamic image (eg a rotating arrow for a compass) constructed from g.fillPoly() etc.

Wondering if the layout render will be smart enough to avoid flicker

It's able to clear just the area that changed and then render the text. It's what most people do and mostly avoids flicker. I guess usage of offscreen buffers could be an option but the idea of this is to be lightweight - not to grow into an all-encompassing UI library.

layout manager is not the only option

No, you can do whatever you want.

eg a rotating arrow for a compass

That's why we have the 'custom' element - so you can render just the bit that you need, but where it is gets handled for you.

Hi I was making a simple clock with the layout library. Here is my code:

require("Font7x11Numeric7Seg").add(Graphics);

var Layout = require("Layout");
var layout = new Layout( 
  {type:"h", c: [
    {type:"txt", font:"7x11Numeric7Seg:2", label:"", id:"time"},
  ]},
  {btns:[
    {label:"", cb: l=>print("Two")},
], lazy:true});


layout.update();
layout.time.x = 170;
layout.time.y = 10;

function draw() {
  var currentDate = new Date();
  var currentHour = currentDate.getHours();
  var currentMinute = currentDate.getMinutes();

  layout.time.label = currentHour+":"+currentMinute;

  layout.render();
}

g.clear();
draw();
setInterval(draw, 5000);

The problem is I am using lazy rendering and the clock does not seem to automatically redraw. It just keeps drawing on top of itself. I am using the Bangle.js 1 emulator. I put a screenshot below of the emulator after I left it on for a while.

1 Attachment

I suspect the issue is that since layout.time.label is initially empty, and you don't call layout.update after setting it, the layout code thinks it still has zero width and thus tries to clear an area with zero width.

Try either setting the label to a placeholder value of "00:00" when you construct the layout or call layout.update() before each render.

@NebbishHacker Thanks! It works now.