Troubleshooting Tensorflow Lite in Bangle

I have created a Tensorflow model that expects a (1, 12, 3) sample as input. I converted it to Tensorflow Lite and print the declaration as in the example provided in a tutorial in this forum:
print("var model=atob(\""+str(base64.b64encode(mode­l_no_quant_tflite))+"\");")
I'm using quantization because it wasn't reducing any further the size, the size of the model I'm using is 4356 bytes .

Here the problem, the command "var tf = require("tensorflow").create(2048, model);" keeps giving an error "Uncaught Error: Model provided is schema version 0 not equal to supported version 3." when I try to load it to the Emulator (loading to the watch also fails, it even disconnects the Bluetooth half loading and the loading fails).

Do you know what could be the cause of the error?

Thank you very much.

1 Attachment

I haven't come across this before, but it seems like it might be related to how the model is being created? Maybe you could post up the code used to make it?

It seems like there are tools to convert TF models to new schemas (https://android.googlesource.com/platfor­m/external/tensorflow/+/2db2230841e851e8­0374b6c5d9e6d9d7f35e0384/tensorflow/cont­rib/lite/schema/upgrade_schema.py) but I'm not sure why yours would be using schema 0 - could your Tensorflow version be an old one?

Hi @Gordon, thank you for your reply.
My Tensorflow version is the latest one tf.version Out[1]: '2.5.0', I'm attaching the model configuration in a screenshot and also the information provided when running the model, I don't see anything strange but I might be missing something?

I've tried the tool that you recommended but I couldn't make it work, it keeps giving errors (also screenshot attached).

I'd appreciate any help. Thanks!

I don't see the code for actually exporting the model there?

It seems that after you've got model_final you need to do something like:

# Convert the model to the TensorFlow Lite format without quantization
converter = tf.lite.TFLiteConverter.from_keras_model­(model_final)
tflite_model = converter.convert()

# Save the model to disk
open("model_final.tflite", "wb").write(tflite_model)

# Convert the model to the TensorFlow Lite format with quantization
converter = tf.lite.TFLiteConverter.from_keras_model­(model_final)
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
tflite_model = converter.convert()

# Save the model to disk
open("model_final_quantized.tflite", "wb").write(tflite_model)

import base64;
print("var model=atob(\""+base64.b64encode(tflite_m­odel)+"\");")

I managed to load it to the Bangle.js, there were two issues:

1. Apparently base64.b64encode(tflite_model) wasn't working in my base64 version, it was giving a corrupted string, I had to create it with base64.b64encode(tflite_model).decode('u­tf-8')
   
2. I had to change the conv1d layers to conv2d layers, otherwise there was a different error.
   
   

The problem is now that the model does not seem to take any values as inputs. I trained the model with inputs in shape (1, 12, 3, 1), so I've prepared the data in the bangle in the same format: the input for one prediction is of shape (1, 12, 3, 1) that is, an array of arrays.
But when I execute the following commands the input has been assigned all to NaN.

**print(input);
tf.getInput().set(input);
print(tf.getInput());**

1 Attachment

The inputs (and outputs) of the Tensorflow implementation on Espruino are just flat arrays... So you want the result of print(input); to look like:

[ 
  -0.9...,
  0.52...,
 ...
 ...
]

then it should work

It works, thank you very much!

Hi @user130273, would you be willing to share your code? I've been able to collect data and train a model using Edge Impulse (if interested, see this tutorial I wrote). This is however using the Bangle gesture commands, but now I'd like to learn how to use raw accelerometer and other data.
Especially this part is unclear: when I've uploaded a trained model (.tfmodel + .tfnames) to the watch, how can I run inference code on it?

Edit: Aargh, it should be something in line with the below code that I've seen many times... Still interested to see your code though.

var model=atob("......");

var tf = require("tensorflow").create(2048, model);
tf.getInput()[0] = x;
tf.invoke();
print(tf.getOutput()[0]);

Hmm, the below code is copied from here and just slightly appended. When running it, Bangle restarts itself, when tf.invoke() is run.
What is amiss here, the code, or me?

Edit 1: While the code resets the physical watch, it runs fine in the emulator, both for Bangle 1 and 2
Edit 2: Tried to reset the watch by installing default apps, still same problem

var model=atob("GAAAAFRGTDMAAA4AGAAEAAgADAAQ­ABQADgAAAAMAAADsCQAAuAUAAKAFAAAEAAAACwAA­AJAFAAB8BQAAJAUAANQEAADMBAAAxAQAALwEAABs­BAAAXAAAAAwAAAAEAAAAePb//7b6//8EAAAAQAAA­AJ4IZD5QVk0+7cXIPkq+Er8xYmi8i2/MPqSmNL4q­Z529BkrJPsPwij08RhC/GISaPuFzVD5EElI+BF7y­vgBVWz4C+///BAAAAAAEAADy1BE+hO4/PpbYQL5l­m62+Of0OvqzJgT6emCU+N++UvGqNxr4E0iC+4KYU­PWwbhb5x7CC9a6WBPqhwhb5h/qw+nsY4vtWAXb6M­jwa8fL+nPfqwjj4yjtU8vxOaPpw0Tj643ZQ+dEUk­PCglib1csyS+V3pqPm3mj75NqYs+sw0xPucVkb4s­MBm+Fy3DPbxymL3qREg+7ZoVvowzlz0K6sm+lNNo­PmBunr2AD6i8St+MvgJSlD4awpw+1JWxvhaUv7xr­opi+lvRdPX9dkr6rZKy+rv/LPngF7L5YDX89OhGd­Pl5g1L5tvj2+4Ol1PQZm273xFcM9Gyq0PnMCuD4k­pDy/kV5svk89SD/TlMY9fCvWvo7gej1gHou/MkCC­vmLRGD++MdG7ugULv/cOhL74yme/w3mIvVy3jT2/­a6k+MgrFvqiPzb1Hgt49c3a5PiUNWL5SGq4+0RRv­PnKPKT6wWhw/F4r3PS0LyrwP4dy+snCYPuusFb6D­8AW/FpNrvuegab4q+Mk+NoaZPiCL4b2HZLg+7/cs­vhoWKLwUMh6+xI8OPnLyqj7oXFk+nJrPPS/Hwz7t­aiC+aPmHvhXdwL4OATa+ui26PfVEVD5ZaKQ+pQq/­vqB3gz0sESU+c3e+vnfT6j05v2i+jyi6vsD2gr4v­EFG9RfnSvtxRFb+YV7G+Q9ZiPsFggL/waZa+nK12­vhExuj5gUCQ/0FmlPeTUkL4MUQk/s0jOvRxroj0I­U+q97BiUvhZEtz5mrL2/yDrKvo1wgL081wy+Ls6Z­viwB4D3uUyw+dGXxvdQz0r6lutq+IKx+PNqohb7t­XNI+OiQrvmyHnL0ySyG+rLOJva95lb5EJ8C9OleU­vrNLyD4pp9K+F0vKPgBZ/T4QSIg+32DAPsQcDj5U­sOY+16FAPlxYM76Z2Tq+ymy9PeuehL7wLRM9JGu6­vliYBD7vTRw+9dhAPmKtpL51poK+W26OPUhirr4e­Akq/DYCePotyzL5wbo2+R4jpPF8/5L6KHq49wh6z­vinn4D7i7ay9vDeqPmJUTT7gQ8q9ah+APsw4vz3k­ggS+iKScvV8iu739oJW+TPKAPa7c47wjmbO+U4um­vWdOmz461hq+jEe5PhHmTb78mcg+yW3Vvhl0Dz4T­+wK+kmdRPpC+Rr+8Hys+dNmzvSBiHTwSc7E+1VVt­PmVFgD7g6iU9Jt69vkL1Lr7+1Gg+umFkvsjsDT2n­VZK9JzjfPhkItD71E20+JnrgPbUmKj2AN6y7/ufB­PtjiRr7N5Ti+LC78PZRozD7YYIU9WODYve5crT6A­Wly9lJ4FPjok1b7qobK+tB7evBUoHr5csxG+W7Ki­PsDasb3In8a+jLqGPSBkVT0zh1i+Dv///wQAAABA­AAAAJdpOvPRygT04MxO+aJ2/Ph26vj4CeZo9sKz7­veJrTD2+Vy8/AAAAAH+BMz6e9MK+rqn4PWjdUr4S­wwo+ljZPuyD7//8k+///KPv//2b///8EAAAAQAAA­AFJaHb8b6Oi9u/r9vgAAAACIXR8/w63IPQAAAABy­Glc/izWfvpB3tD4AAAAAViLXvmQqwD5Sxe2+AAAA­AE71FLyy////BAAAAEAAAAAlH/88PwChvXBShLmT­yIO/H4J2v/Wb3b3fm5E+PEGdORjtnj+wcjy9HCbR­vpgF8T+gFZc4SAQRP8muzr5FmUC+AAAGAAgABAAG­AAAABAAAAAQAAABPOD2+3Pv//w8AAABUT0NPIENv­bnZlcnRlZC4AAQAAABAAAAAMABQABAAIAAwAEAAM­AAAA8AAAAOQAAADYAAAABAAAAAMAAACQAAAASAAA­AAQAAADO////AAAACBgAAAAMAAAABAAAAED8//8B­AAAAAAAAAAMAAAAHAAAACAAAAAkAAAAAAA4AFAAA­AAgADAAHABAADgAAAAAAAAgcAAAAEAAAAAQAAAC6­////AAAAAQEAAAAHAAAAAwAAAAQAAAAFAAAABgAA­AAAADgAWAAAACAAMAAcAEAAOAAAAAAAACCQAAAAY­AAAADAAAAAAABgAIAAcABgAAAAAAAAEBAAAABAAA­AAMAAAABAAAAAgAAAAMAAAABAAAAAAAAAAEAAAAB­AAAACgAAAOwCAACEAgAAJAIAANwBAACYAQAAOAEA­APAAAACsAAAATAAAAAQAAABK/f//OAAAAAEAAAAM­AAAABAAAADz9//8eAAAAc2VxdWVudGlhbC9kZW5z­ZV8yL01hdE11bF9iaWFzAAABAAAAAQAAAI79//9M­AAAAAgAAAAwAAAAEAAAAgP3//zIAAABzZXF1ZW50­aWFsL2RlbnNlXzIvTWF0TXVsL1JlYWRWYXJpYWJs­ZU9wL3RyYW5zcG9zZQAAAgAAAAEAAAAQAAAA6v3/­/zAAAAAEAAAADAAAAAQAAADc/f//FwAAAHNlcXVl­bnRpYWwvZGVuc2VfMS9SZWx1AAIAAAABAAAAEAAA­ACr+//84AAAABwAAAAwAAAAEAAAAHP7//x4AAABz­ZXF1ZW50aWFsL2RlbnNlXzEvTWF0TXVsX2JpYXMA­AAEAAAAQAAAAbv7//0wAAAAIAAAADAAAAAQAAABg­/v//MgAAAHNlcXVlbnRpYWwvZGVuc2VfMS9NYXRN­dWwvUmVhZFZhcmlhYmxlT3AvdHJhbnNwb3NlAAAC­AAAAEAAAABAAAADK/v//MAAAAAoAAAAMAAAABAAA­ALz+//8VAAAAc2VxdWVudGlhbC9kZW5zZS9SZWx1­AAAAAgAAAAEAAAAQAAAACv///zgAAAADAAAADAAA­AAQAAAD8/v//HAAAAHNlcXVlbnRpYWwvZGVuc2Uv­TWF0TXVsX2JpYXMAAAAAAQAAABAAAABO////TAAA­AAkAAAAMAAAABAAAAED///8wAAAAc2VxdWVudGlh­bC9kZW5zZS9NYXRNdWwvUmVhZFZhcmlhYmxlT3Av­dHJhbnNwb3NlAAAAAAIAAAAQAAAAAQAAAKr///9E­AAAABQAAACwAAAAMAAAACAAMAAQACAAIAAAAEAAA­AAQAAAABAAAAAAB/QwEAAAAAAAAACwAAAGRlbnNl­X2lucHV0AAIAAAABAAAAAQAAAAAADgAUAAQAAAAI­AAwAEAAOAAAAKAAAAAYAAAAQAAAACAAAAAQABAAE­AAAACAAAAElkZW50aXR5AAAAAAIAAAABAAAAAQAA­AAEAAAAQAAAAAAAKAAwABwAAAAgACgAAAAAAAAkD­AAAA");

var x = 1;

var tf = require("tensorflow").create(2048, model);
print ("1");
tf.getInput()[0] = x;

print ("2");
tf.invoke();

print ("3");
print(tf.getOutput()[0]);

print ("4");

Ok, updated from a 2 week old firmware (2v10.something) to the latest 2v10.107.
Now it is not crashing anymore!

Hello @ThomasVikström !
Great to hear that is not crushing. The structure looks pretty much like mine, the minor differences are:

1. input x is an Array because my model has more than one input.
   
2. I set the input as tf.getInput().set(x_array); I think this is better when you have more than one input.
   
3. Invoke is the same.
   
4. Output is similar: output = tf.getOutput(); basically instead of selecting the first element of the output I select all the output values because my model has more than one output.
   

Thx for the comments!
As I have a .tfmodel file exported from Edge Impulse, I’d like to use that but I’m not sure how to read it with JS and if it needs conversion before.
The same .tfmodel file works as such with the Bangle gesture functions.

@Gordon I guess you are superbusy with the Kickstarter campaign right now. When you get time, could you please shed some light on how to use a .tfmodel file created by an external system in Javascript.
I.e., like in the code in post #9 above, how should the .tfmodel-file be "introduced" into the code? As the Bangle gesture functions can use it, I guess there should be a way to use it within JS also?

Hi - I think you can just load it from storage and use it as-is, like this?

var model = require("Storage").read("mytfmodel");
var tf = require("tensorflow").create(2048, E.toString(model));
tf.getInput()[0] = x;
tf.invoke();
print(tf.getOutput()[0]);

Please let me know if you have issues though...

var model = require("Storage").read("mytfmodel");

var tf = require("tensorflow").create(2048, model);
tf.getInput()[0] = x;
tf.invoke();
print(tf.getOutput()[0]);

Hi - Thx, this makes sense now once I saw it, am new to JS! However, it produces the below error. I've uploaded the same tfmodel file used with the gesture functions to the watch and named it mytfmodel
Does this perhaps mean that the tfmodel file should be converted to a flat string? In that case, can that be done with JS?

>Uncaught TypeError: Model is not a Flat String/ArrayBuffer
 at line 2 col 50
var tf = require("tensorflow").create(2048, model);
                                                 ^
Uncaught Error: Cannot read property 'getInput' of undefined
 at line 3 col 3
tf.getInput()[0] = x;
  ^
Uncaught Error: Cannot read property 'invoke' of undefined
 at line 4 col 3
tf.invoke();
  ^
Uncaught Error: Cannot read property 'getOutput' of undefined
 at line 5 col 9
print(tf.getOutput()[0]);
        ^
> 

Ahh, sorry, I just corrected the code above - please can you try:

var tf = require("tensorflow").create(2048, E.toString(model)); 

What happens is require("Storage").read("mytfmodel") doesn't actually read the file out of Storage, it returns a special variable type which points to it in external memory - it's so you can for instance write images to the Bangle's screen than are bigger than would fit in RAM.

You just need to force Espruino to load the file into RAM, which you can do with E.toString

Makes sense, at least it does not crash now, instead gives me a value.
Great! I'll try out this way of handling tf-files, if I get a holistic understanding of this, I might create/update a tutorial on it as well.