Better Pedometer - HELP NEEDED! #5201
Replies: 5 comments
-
Posted at 2021-02-03 by d3nd3-o0 Do you want separate recordings for each walking style, 150 paces of each. Or one recording of mixed styles across 150 paces? Posted at 2021-02-03 by @gfwilliams I think probably a few recordings of normal-ish activities - so eg if you're walking then going up stairs and then walking fast that'd good to have in one recording. Posted at 2021-02-03 by cartron here you go :-) walking down/up stairs, walking fast and slow Attachments: Posted at 2021-02-03 by @gfwilliams Great - thanks! Posted at 2021-02-03 by HughB App loaded will give it a spin a few times tonight. log 0 is walking round the house, upstairs and downstairs. Attachments: Posted at 2021-02-04 by @gfwilliams Thanks for this! Yes, it's the oxford pedometer algorithm that had been mentioned previously. However even if that doesn't work out, having a bunch of data should really help to come up with a reliable algorithm. Posted at 2021-02-12 by d3nd3-o0 HOpe it helps! Attachments: Posted at 2021-02-15 by @gfwilliams Thanks! I'll try and get on it this week Posted at 2021-02-17 by user107850 thank you all! you can follow the development at espruino/Espruino#1846 Posted at 2021-02-17 by HughB Thanks for sharing, interesting reading. Looking forward to a more accurate step counter. Posted at 2021-02-28 by Pablo Hi. Here are 150 steps. I will upload more data hiking next week. Thanks! Attachments: Posted at 2021-03-01 by @gfwilliams Thanks! I think we've got a decent amount of data now. There is now a Branch of Bangle.js with some calibrated data in: https://github.com/espruino/Espruino/tree/step_counting But we just have to do some work to stop it automatically lowering the step threshold too much and just randomly counting steps when you're stationary :) Posted at 2021-03-15 by Guillaume_G walking around in my bedroom Attachments: Posted at 2021-04-04 by valvin I'm new bangle.js owner and effectively pedometer needs improvment. It looks gadget bridge and other pedometer widget don't have the same result. Do they calculte step their own way or is directly implement in the firmware? Using the last stable firmware 1.8 I think the main issue is sensitivy. Steps are triggered too quickly. This morning at wake up i did already 1k steps :) Do you need more sample with other kind of use case? Do you need testing on this specific branch? Posted at 2021-04-06 by @gfwilliams Hi - for now I think we're good as far as sample data goes. wedpedom and gadgetbridge should match, but I hope to have an update soon (but it'll probably still be a few weeks off). The issue is that with both old and new step counters, they should filter out detected steps that don't form part of a regular step...step..step pattern - but they don't currently do this. Posted at 2021-04-29 by @gfwilliams Just a note that I have now committed a better step counting algorithm: http://forum.espruino.com/conversations/363082/#comment15962618 Posted at 2021-04-30 by HughB sitting on sofa, stroking dog, looking at watch, 30 seonds Attachments: Posted at 2021-04-30 by HughB sitting at desk, look at watch, stand up 5 seonds, sit down 5 seconds, look at watch, repeat, 30 seconds. Attachments: Posted at 2021-04-30 by HughB sitting on sofa, crossing and uncrossing legs, pause, swival round to look at the window, shuffle about on the sofa, look at watch. Approx 30 seconds. Attachments: Posted at 2021-05-04 by @gfwilliams Great - thanks! Posted at 2021-05-04 by @gfwilliams Just pushed an update that should be a significant improvement here. Looks like some of the original data had a very slow walk - and to detect that the threshold was being pushed low enough that it detected steps when it shouldn't Posted at 2021-05-04 by HughB Hi @gfwilliams - I have updated to 2v09.9 and I am still experiencing counting steps (2-10) when twist wrist to look at the watch whilst stationary. Just want to check I have the version that has your latest checkin. Posted at 2021-05-05 by @gfwilliams Yes, 2v09.9 is the one. You can always check the git hash shown in You're saying that just a single twist would make it count 10 steps? Running the files you'd given through it I think it was counting maybe 5 steps over the whole period for those - so were you doing something very different to what had previously been recorded? At the end of the day, the pedometer code isn't going to be perfect - but I am hoping it will be significantly better than the old code on the watch. Posted at 2021-05-05 by HughB @gfwilliams - appreciate its not going to be perfect (2% error over 1 km would be excellent, 5% would be good.). But its definitely still counting 1000s of steps over the course of a day when steps are not there. Today I have apparantly done 3671 steps but all I have done is been up and down stairs maybe 4 times and the rest of the time sat at a desk. I would expect maybe 250 or 300 steps max for such activity. Its quite easy to reproduce the issue. Just sit look at the watch then twist your wrist through 90 degrees and back varying the speed at which you turn the wrist. The twist used to turn the LCD on should not cause steps. Will send you another accel log of me doing this, maybe another tuning session will get this to be filtered out. Posted at 2021-05-05 by HughB Here you go. This log is wrist twisting through 90 degrees and back, on my right hand. Maybe 30 wrist twists at the rate of 1 per second in this sample. Attachments: Posted at 2021-05-05 by @gfwilliams Is it actually better than the old step counter? If not, I might as well revert it. Thanks for the log - However...
Is that something you normally do? I don't really see how I'm supposed to filter this out if it's at a rate that's the same as walking? I could be wrong but I think if you did this to most step counters, they would count steps as well. I can't really afford to spend too long on this, so I've put my current code with test harness up at https://github.com/gfwilliams/step-count Maybe someone will have a bit more success than me Posted at 2021-05-05 by HughB
Gut feel, I think it is improved, but still quite a way off being able to say we have reliable step counter. I was going to give a good test on a known 1.03mile route I have previously used many times to test step counters - but when I saw I was logging 1000s of steps sat at a desk I thought it would be better to see if that got sorted first. I may try it with the activepedom as well, assuming it is still compatible with that widget. ActivePedom has the X steps in X seconds feature which effectively takes out any counting when sat still.
No. Apologies - maybe that sample was not that great an idea. Maybe I should have done it just 3 times in 30 seconds.
I have an AmizFit Bip and an old FitBit and neither count steps on reaching across a desk or rotating a wrist, maybe the odd time but not consistantly to the point that by the end of the day when my AmizFit would say 1200 steps and the bangle says 8000+ today.
Yeah - I understand, this will be one of many things wanting attention. Maybe its time to have a competition to see if anyone can get a better version. I might have a go myself if its not to hard a jump to get into. I had a quick look at the link. What is the lifcycle of trying to improve this ? EG - do you just add samples until you get better figures at the end or would the C need to be adjusted etc ? How would one actually test this for real on a Bangle ? Is there a way to patch the firmware or would you have to build your own firmware from the source etc and then use the DFU loader etc. I think real world testing is needed here - I know all too well how time consuming that is from walking outside waiting for a GPS fix etc. Having a good enough step counter is an essential base level feature of any smartwatch though. I would rather Bangle became my default smartwatch than have to wear a second one to get an accurate step count etc. Maybe float the idea of a competion and see if there is much enthiusiasm for it ? Posted at 2021-05-06 by @gfwilliams
My point was just that if you repeatedly rotate your wrist once every second, they may well start to count steps. But yes, normal reaching/rotating is something that obviously needs filtering out :)
Yes, totally! Is anyone interested in this? I had been considering it... I guess the thing to do would be for us to ensure we have a really good representative set of data in https://github.com/gfwilliams/step-count and then see who can get lowest. I now have a bunch of what will probably be the 'new' Bangle.js (the SMA Q3) and I could send one of those out to the winner... Or maybe something else?
Realistically it's a matter of changing the C code. It's really pretty basic at the moment.
Right now on the Bangle there's a specific step counter file: https://github.com/espruino/Espruino/blob/master/libs/misc/stepcount.c So you'd just have to copy/paste into there, recompile, and DFU. Realistically the step-count repo above should just use that file directly though, then you could just move the file across when you were happy with it. You actually could just include it with Inline C though, so you could tweak it quickly on real hardware without recompiling. If there's interest I could paste up an example of how to do that? While obviously it helps to do it on real hardware, there's a lot to be said for doing it 'offline' just because you can run over the same data over and over, look at a graph, see exactly what's going on and why it is/isn't working. Posted at 2021-05-08 by HughB
This sounds interesting, tell me more. I was wondering if there would be a another smartwatch hardware that would run Espurino. I will google it. I would love something like the form factor of the AmizFit Bip if that were possible. It has some advantages. The charging connector is not magnetic and interferes less with compass calibration. Its smaller. Battery life is very good. Having said that I curently have that test Bangle you sent me not needing a recharge after 300+ hours. Found this. https://hackaday.io/project/175577-hackable-nrf52840-smart-watch Looks really cool. When are these going on sale in the Bangle Shop, can you place advance orders. Only snag for me is that is another GPS data sheet to grapple with. Nice looking watch though. Posted at 2021-05-08 by HughB
Sounds like a great option. I think having to recompile the firmware might put a few off. If you can write up the basic process loop of what you do to iterate round the process. I could have a go. I've done a lot of C in the past but nothing todo with filters or machine learning etc. I still have a few more enhancements and bug fixes to do to 'kitchen combo' though. Is there a way to simulate this in javascript first, run on the bangle and then translate it to C ? Posted at 2021-05-10 by @gfwilliams Yes, that Hackaday link is basically all the info I have on the Q3 at the moment, and you can build Espruino (although I'm not doing automatic builds at the moment). Basically what we're missing is the heart rate sensor right now (since I have to reverse engineer it all). I'm hoping to get them in the shop 'soon' - hopefully within the next month - but they're still very much beta as I have to decide exactly what to do about making existing apps compatible (the screen/buttons are different). GPS is a pain though - it doesn't seem to be quite as good as the Bangle's, and as you say it's a completely different command set too.
Yes - it's not a very difficult algorithm (there is no machine learning), so this is probably what I should have actually done at the start. Try:
On the graph that's displayed, blue is the actual accelerometer data, yellow is the filtered data, and red is the threshold. The issue is really the filter...
So if using the filter actually turns out to be a good idea, we basically need a way of only passing through steps when the filter is producing values that are all the same (ish) amplitude, and not passing through the outliers. Posted at 2021-05-10 by @fanoush
Yes definitely try googling if you wonder about such things :-) Or even look around this forum. There are plenty of watches running Espruino. Some even run some Bangle apps. Posted at 2021-05-10 by @fanoush
Even the HW was beta, looks like newer ones that some guys received later have small hole and pressure sensor works much better with that. Without hole the pressure inside just goes up with the temperature and does not reflect air pressure very much. Posted at 2021-05-10 by @gfwilliams
The new ones I have have the hole too. I hope they've done something about waterproofing! The software they ship with definitely needs some work. Even when 'off' they are advertising with Bluetooth, so as a result it's a bit tricky trying to use Bluetooth right now (with 100 Posted at 2021-05-10 by @fanoush There is that offline/silent/powersaving mode? you swipe down and one of the icons turns off bluetooth and everything and puts on some simple clock. Also there is power off in the hidden menu. When in settings menu hold button until 4 digit code entry pops up and then 2020 is power off and 2019 is test menu which also has power off menu. Hopefully it is not advertising in that off mode. But anyway doing this with 100 watches must be a pain if they come like this from the factory. Posted at 2021-05-16 by HughB @gfwilliams - Good news with firmware 2v09.9, and step counting. I did a test walk this morning 1.53 miles I did a similar walk yesterday and got very close results but did not write them down before midnight. I appreciate that 1 test result does not make strong evidence - on the other hand I have been routinely testing the step counter widgets (with different settings) across multiple walks since I purchased my Bangle in December 2020. I have never managed to get within 400 steps of the AmizFit bip - so to be within 11 steps is a brilliant result. I have been using the ActivePedom Widget with the following settings: Step Threshold 5: Active Res (ms) 5000. In other words only register steps after 5 steps in 5 seconds. All other settings in ActivePedom left as default. I think if you produced a firmware with this additional threshold built in you would have a pretty accurate step counter. I'd be happy to do some real world testing. Posted at 2021-05-16 by HughB I installed your code to try and undertand how this works. I can see the ringing effect that you point out. As per my post above I think all that is a needed is a two filter approach. This existing filter in your code that detects potential steps. The 2nd filtering stage ensures that 5 steps in 5 seconds are detected before counting them as step events. The step count will always be 5 seconds behind but I think that is ok. I have noticed (external observations as a user only) that FitBits and AmizFit Bips both appear to have a threshold before they will start registering steps. From a pure technical perspective that might not be the best theoretical approach but it appears to be working from a pragmatic point of view. Worth a try and get some feedback from others maybe ? Posted at 2021-05-17 by @fanoush some fitness trackers I tested ignore few steps if the activity does not continue but I think there was no time limit like the 5 seconds you mentioned. So first few steps are delayed until that threshold (e.g. 10 continuous steps) is reached, but then it adds it all and it continues with real number, not delayed Posted at 2021-05-17 by @gfwilliams Just to be clear - you're using a new firmware (with the new step counter code), but then just having that filter for the amount of steps in a set time? Interestingly I had looked at that before, but the lack of filtering meant you could get a bunch of steps very close together. The addition of the filter probably makes doing something simple like that quite a good approach. I guess there is also the option of just adding 5 steps to the step count once we've had 5 steps in a row as well? Posted at 2021-05-23 by HughB Hi @gfwilliams - apologies been out of contact for a few days. Yes - using v209.9 step code and the activepedom widget with the thresholds set to 5 steps in 5 seconds. This seems the most accurate I have seen so far. Just want to check that doing this was / is a valid test case - what do you think ? To be clear I have not written any code. I am assuming that what comes out of the v2.09.9 firmware is the step count code and the ActivePedom is effectively acting as a second filter, this may not be the case though. Thats the tests that I have done. I think it might be enough to go down to 3 steps in 3 seconds and still be enough to get rid of miscounting of slight movement when moving arms, wrist twisting when sat still etc. Posted at 2021-05-24 by @gfwilliams Thanks - this sounds really promising. As far as I can tell, activepedometer really does just work off the 'step' event so acts as a second filter. The other filtering it does shouldn't have much effect, so just the 3 steps in 3 seconds thing sounds very easy to implement. Posted at 2021-05-25 by HughB I can do another 1.5 mile test with those settings later in the week. Will report back. Posted at 2021-05-25 by HughB
I've been thinking about this. I think the best way would be to say, the screen is different size, its got one button, its a different watch but same API, you will have to port your App. Hopefully the original authors of the Apps would do the porting. The thing with a one button watch is that the whole way of interacting with an App will be totally changed. With it being a full touch screen I would expect use of Up, Down, Left, Right swipe would provide a lot more options for navigating round things. I'm definitely up for having a go at the low power GPS again. It was a lot of work but worth it. There were a few comments made about me stepping outside the house a lot to get a signal :) Posted at 2021-05-26 by @gfwilliams Yes, that's definitely the direction I'm headed in! I'm planning on adding a simple UI/layout library that'll handle a lot of the common layouts I'm seeing in apps, and then I'll change a bunch of the most common/useful apps over to use that so that they can work on both Bangles (and maybe even Pixl.js too). Posted at 2021-06-17 by @gfwilliams Quick update here - step counting on 'cutting edge' should be significantly improved now. I added the 'X steps in Y seconds' algorithm then trained it based on the recorded step count data I have, and we're getting far better accuracy than before now. Posted at 2021-06-18 by HughB I'll download and flash the latest firmware. Posted at 2021-06-22 by HughB Hi @gfwilliams - I have flashed 2.09.90. And have done some tests. I just want to check a couple of things. 1) All that is necessary is to uplift to the cutting edge firmware. 2) there is no need to configure anything. Not sure what X steps in Y seconds is though. Is it 5 steps in 5 seconds or 3 steps in 3 seconds. To test it I am just using the widpedom widget and using the getSteps() call. Posted at 2021-06-23 by @gfwilliams
Correct, yes
It's actually 6 steps in 6 seconds now. I updated https://github.com/gfwilliams/step-count so now it uses the exact Espruino step count algorithm, then brute-forced it on all the recorded data we had and that provided the best results. On average over all the 18 data files we have recorded, it's only 1.5 steps out per file - around 1%.
That should be fine. Just don't use Active Pedometer as that applies extra filtering on top of the already filtered data Posted at 2021-06-23 by HughB OK - its not good news I'm afraid. My test strategoy was to wear the Bangle and an Amafit bip from midnight onwards. Sleep with the watches on, go about my normal work day, mostly sat down, a couple of short walks of the dog etc. At 6pm I took both watches off and recorded the step count. Bangle: 9902 and Amazfit Bip: 4479. When I woke up in the morning the Bangle was on about 1200 steps to the Amizfit Bip 200 or so. The idea of testing recorded accelerometer samples against the C code is good, but something is making it not work out in practice. At the end of the day its only the field test result that counts. I like the javascript idea and might try and produce a widget out of the code you did. I have had a look at a couple of papers on step counting out of curiosity so might try out a few things. I'm away for a few weeks in July so progress might be slow. Posted at 2021-06-23 by HughB Trying to understand your code example - have added questions as comments.
Posted at 2021-06-25 by @gfwilliams It's worth looking at the actual C code as it's documented a lot better there (and also it's slightly different now), but here are my comments on the JS:
It's a shame the new step counting hasn't worked out - is it substantially worse than it was before, or only marginally better? I guess making a widget is a good idea, although it won't be great for battery. You could actually run multiple widgets at once I guess and see which ones gave better values. I wouldn't add your widget to the main app store though - the last thing we need is another slightly different pedometer widget to confuse people. If you can get the step counting substantially better then we can look at pulling those changes back into the Bangle.js firmware Posted at 2021-06-25 by HughB Thanks for filling in the gaps in my knowledge. I spent a good couple of hours looking at the code.
Gut feel ...about the same. Thats not bery scientific I know. I have have generally got used to measuring activepedom and then using the latest firmware through activepedom. That felt better. I have previously only tested walks and done comparisons against my AmizFit Bip. OR I have tested sitting still or watching TV to watch out for false step detecting. This was the first time I decided to do a midnight to 6pm test.
I think we need a javascript implementation that people can tweak. Its a lot easier than having to flash the firmware and means you can test multiple iterations on the same firmware. For me to go back to the previous version I will have to reflash older cutting edge firmware. I have started reading a few papers that I have downloaded. A number of people have tested multiple approaches BUT they never show any code that you can look at. One thought I had was about writing to all the authors of papers in the last 5 years and asking if they were aware of Bangle and it they would like to attempt a step counter in javascript. If we provide a basic template App or Widget then it is easier to get started. Might not get much engagement but you never know. There is a substantial amount of time and effort needed for these kind of things as you need to repeat multiple tests many times with different people and different versions of software and algorithms. Another idea might be to crowd fund for a cash prize for someome to write an accurate step counter that will be open source for Bangle. A free Bangle is not going to attract that many to have a go otherwise. I think there might be enough of us wanting a good step counter that we could each chip in £20 to get it done. 20 such donations would make a cash prize of £400. Ref the accelerometer being 12.5Hz - has anyone done any tests to validate that it is regular ? Also the papers I read suggested 20Hz as an ideal sampling freq. Thinking outloud - I reckon I generally walk at 2 steps per second. Thats only 6 samples per step at 12.5Hz, where as 20Hz would give 10 samples per step. Granted you only need to see the crossing of the thresholds. It feels like there is an assumption somewehere that is not correct in practice. Again not very scientific. Is there any accuracy data on the how the stock firmware step counter performs ? Could it be limited by the hardware ? Posted at 2021-06-25 by HughB here's the filter screenshot I got when I tried to enter the values you put in the code. Attachments: Posted at 2021-06-25 by HughB This paper suggests that you get a 10% error rate when using a 10Hz sampling rate, this reduces to 2.5% for a 50Hz sampling rate. In the chart below magnitude is the algorithm Bangle is using. Having said that the problem is still down to the detection of steps when basically sat still. Attachments: Posted at 2021-06-26 by HughB Had a look at the firmware code and translated it to a javascript version.
Posted at 2021-06-28 by @gfwilliams
Well, I did spend time doing that 2 months ago, but then nobody bothered to do anything with it! For developing, absolutely - but in the end we need it baked into the firmware or it'll kill battery life. Your JS version looks good though. I think it's possible that just raising stepCounterThreshold to maybe 1500, lowering STEPCOUNTERHISTORY to 2 and STEPCOUNTERHISTORY_TIME to 40ish might actually have a pretty good effect. The accelerometer regularity should be absolutely fine unless you have an app/clock that's blocking execution for more than around 100ms - in which case you could end up with some duplicated samples. In terms of sample rate - we're currently 200% out - so I don't think that is our main problem. If we increase the sample rate then that drains battery quicker (especially if we're processing in JS) and does cause some issues in other areas, so I'd rather not do it unless it turns out to be absolutely required. You can however tweak it yourself from JS if you want to try: https://github.com/espruino/BangleApps/blob/master/apps/accelrec/app.js#L35-L38 Posted at 2021-06-28 by HughB
Appreciated. It was me that requested it then I had other stuff to do. I'm making use of it now though. I've built the App below. It has allowed me to get a much better understanding of what is going on. The main ISSUE IS THE RINGING produced by the filter. One impulse (eg raising your hand to scratch the back of your neck and lowering it back) can cause a ring that will produce 7 steps. The step counting is fairly accurate - the big issue is the ringing. This is why I think it counts 1200 steps when you sleep for 8 hours. A few shuffles every hour or so adds up over that 8 hour period. Maybe a simpler single stage filter would produce less ringing. Maybe we dont actually need a filter. The act of walking produces a strong enough up/down cycle in the magnitude of the accelerometer to drown out noise etc. Not sure if I understood how the approach to X steps in Y seconds is working. It took a while for me to figure out what is going on. I think what is happening is 1) step comes in 2) did it occur within 75 samples (6 seconds, 6x12.5=75) Yes/No record it as a step in the history, 3) the 5 element step history fills up with steps that could each have occured between 0-X samples. When it fills up you check the OLDEST entry in the step history step that it arrived within 75 samples ago (ie within 6 seconds ago). Hmm - ok - I think I go it. I have done it as a state machine. First step is detected move to STEP_1 state. If the next step is within 1 second move to STEP_225 state otherwise go back to STEP_1 state. When you get to 5 steps each arriving within 1 second of each other then you are in STEPPING state. After that 2 seconds of in activity means go back to STEP_1. When you get to STEPPING state the 5 held back steps are released to be counted. The state machine works fine. The problem is with the filter ringing. To test BTN1 to start. Walk 10 steps across the room, stop. You will over count by 2,3,4 steps maybe. More if you are walking fast / stomping. Here's my test code:
Attached is a test log of me doing 10 steps, stopping and then watching steps 11,12 and 13 count themselves by ringing. Attachments: Posted at 2021-06-28 by HughB
I'm not convinced based on my observations about the ringing. Tuning the thresholds is quite tricky as I briefly experimented with taking out the filter. It would be good to know how much amplication the convolve() function gives. If the input is -128 to +127, what is the expected output range. How do you scale it back to the -128 to +127 range. Another thought I have had is PEAK detection. You then just need 3 points A B C where B > C AND and B > A to detect a PEAK. You would have to take off the clipping though otherwise you get a plateau. The advantage of PEAK detection is that you dont need to worry about setting a threshold for that part. However you do need to determine a band in which you say any magnitude M between -X and +X is considered to be too weak to be walking. I would suggest a good start for that band is -50 to +50. These are the max/min values that I observe for raw_clipped when I lift my hand to rub my hair and drop it back to my side. These are clearly not working values for walking which will be much higher.
I might try PEAK detection and removing the filter once I understand how to scale the output of convolve(). Posted at 2021-06-29 by @gfwilliams Does my code for X steps in Y time actually work though? It's pretty basic - you're just storing how long ago each step occurred. If the second from last step occurred less than 3 seconds ago then it's good. It feels like a state machine is overkill. Yes, the ringing is a pain. I'd originally hoped that the filter would have the effect of significantly amplifying repeated steps (which it does) but the flipside is it tends to make instantaneous peaks oscillate. It's why I though raising the threshold would be a good help, as you have to have really strong movements to get peaks anywhere near as high as you'd get when you start doing repeated movements. If you look at the step count tester I'd posted up at https://github.com/gfwilliams/step-count then it actually produces nice graphs where you can see the behaviour in relation to the threshold. It's entirely possible that a simple high/low pass filter (rather than bandpass) wouldn't ring though - and since we're counting step frequency probably all we care about is cutting out high frequency. Posted at 2021-06-29 by HughB Agree - looked up ringing on wikipedia and the upshot is that ringing is enevitable and complex to get rid of. https://en.wikipedia.org/wiki/Ringing_artifacts BUT - Eureka - I think I have worked out a way to deal with this - but will need to do the experiments.
Will build an App tonight and test it. Posted at 2021-06-29 by @gfwilliams That sounds really promising - thanks! However I do wonder about the ringing issue if we're having to manually account for the extra rings it adds (I bet it'll be different for different people). I guess one benefit of the state machine if you can actually add the missing 10 steps back in once you reach that point. My code doesn't do that (but maybe the ringing at the end makes up for it slightly!) Posted at 2021-06-29 by HughB
I think once you actually get walking the ringing will be more consistant. I think it might also depend on a persons weight and momentum. Once I have a prototype that does not record many steps whilst sleeping I will think up a set of tests. My gold standard is a 1.03 mile circuit I have used many times that FitBit used to consistantly come back with 2000 steps. I would do a test at 2mph 3mph,4pm and jogging. I'm afraid all my distances are in miles - I know what an 8 minute mile feels like to run :) Another test will be a standard working day with odd walks round the house etc, will have to wear another tracker to compare. If this line of attack does not come up with improved accuracy then will have to start looking at how other filter specs perform in terms of ringing. Its interesting to note that I have not seen mention of ringing in the various papers I have read. They just say a low pass filter was used. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2021-06-29 by HughB Here is the test App. I am just loading it through the filemanager or IDE. Only done cursory tests so far - but I beleive this is significantly better than anything done so far. The main improvement is screening out counting steps when you are not on your feet - but I think this is the best we are going to get with this filter. Key Result 1: Walked 0.31mile round the block: Key Result 2: Sat on the sofa, scratch my head, typing for 1 hour - Zero steps. Never been seen before with any of the previous iterrations. Key result 3: Will count 10 steps from standing still - with varying results (sometimes less or more). But for continuous counting its pretty accurate. The key problem this is trying to solve is counting steps when you are not actually walking. Attached is one of the better papers I have found so far. This suggests a LPF with 20Hz cut off. This would not be possible using 12.5Hz as the sampling frequency. There is 6 sample formula for the LPF which looks like it would be simple to implement. Is it worth experimenting with 40Hz sample and a 20Hz cut off LPF ? OR after you have tried this out and others have tested it you could just translate it into the next cutting edge version of the Firmware. As I say - I think this is significantly less susceptable to counting when you are sat down typing etc. Do you have a Bangle with Stock Firmware that could be tested ? I'd like to see if the commercial product was any good on the step counting front.
Attachments: Posted at 2021-06-30 by @gfwilliams Great!
Did you do anything special for that? Or you mean just with your X steps in Y seconds implementation? With the filtering, I believe you're getting a certain amount of filtering from the accelerometer. I believe it takes more than 12.5 samples/sec, but then averages them down. So if you're looking at a 20 Hz low pass filter, maybe it's worth just trying with no filtering whatsoever and seeing what it behaves like? Posted at 2021-06-30 by HughB Yes - so changes were: Overnight sleep test result is good (should not be counting steps when asleep) I have seen Fitbit and AmizFit Bip report 200 steps overnight so thats a good result. I did briefly try no filtering BUT thresholds then all have to be rebalanced and I have not tinkered with the C code to run your automated threshold calculator. I would like to try a simpler LPF if possible. Maybe 5 or 6Hz cut off only. Posted at 2021-06-30 by HughB So all day test result is not so good and it is this test that matters. Amizfit GTS2 902 , Bangle State M/C V2 - 2814 Not done a lot of walking etc, been in the house working at my desk, going up and down stairs and a short walk in the morning for 10 minutes. So going to have to adjust the X steps to 11. But I do suspect that I'm fighting a loosing battle with the ringing. I'm not entirely convinced that a filter is needed if you detect PEAKs or threshold crossing. It should not matter but most of the papers I have read say you should use a LPF. Posted at 2021-07-01 by @gfwilliams Thanks for trying this out!
I seem to recall that earlier you felt you had a lot of success with just the active pedometer on earlier firmwares? In that case, we could look at removing all the filtering code, going with just a standard threshold, and doing the X steps in Y seconds thing? Posted at 2021-07-01 by HughB
I think you are referring to comment #37 a couple of months back on Firmware 2v09.9 which I think had the filter. The test was only a single walk - which I have now established repeatedly will come out quite accurely. You then added the X steps in Y implementation a while later which is what I have tested. Its when the bangle is at rest (when sleeping or sittting / typing) that the false counting happens. But I probably should retest one of the v208 firmwares with the X steps in Y idea. Do you have the C implementation I can look at for the pre-filter step counter ? I grabbed the git repo for Espruino but lib/misc/stepcount.c only goes as far back as April 2021. It must have been somewhere else before that ? earliest commit I can find on stepcount.c is:
Posted at 2021-07-01 by HughB Here's my latest attempt. I came up the idea of keeping a rolling average of the ABS values coming out of the raw a.mag value. Once the values had dropped below an at rest value I force the average to -1. That allows me to check if steps have shutdown before the ringing has stopped. This works well for stopping counting immediately - but its not the right end of the problem to be solved. I tested it overnight and during the day and the counts were over by 2000+ at the end of the day. So bad result. Just logging my experiments / results here in case I need to look them up later.
Posted at 2021-07-01 by HughB This warrants experimentation:
From: In other words: Just need to find the equivalent values for: Worth a try to see if I can use this to gate stepping. Posted at 2021-07-04 by HughB
So this translates to approximately 0.1g (g is 9.8m/s/s) which is not distinguishable from moving your arm up to scratch the back of your neck. So this was a red herring. Posted at 2021-07-05 by HughB @gfwilliams - I think I have cracked the major problems. I am down to the level of improvements that require much tighter testing metholodoly (average of multiple tests, calibration of the device that it is being compared against). The code below is the current best. And I think you could implement this in the firmware without too much disruption to memory requirements or the existing code. NEXT STEP is to implement this in firmware (ideally with X in X being configurable) so that I can continue testing. I have introduced 2 improvements that have been tested over the last 3 days with lots of success.
I have got the counting while the person is sleeping problem down from 1200 steps to 46 steps. 1000 steps in a day is a 10% error over 10K steps. There are multiple tests that have to be done in different scenarios.
Attachments: Posted at 2021-07-05 by HughB Will supply some more test results by the end of the day. BTW - THE FILTER IS NEEDED. It does a good job at normalising the raw input into a sine wave that crosses the boundaries consistantly. Without it the detection of the UP/DOWN thresholds would be a nightmare and so would accurate detection of the period of a steps. The only issue with the filter is the ringing which I have provided some good solutions to take that out of the problem. Posted at 2021-07-05 by HughB Here's my log of test results. This would indicate increasing X steps in X seconds reduces the over counting when doing every day tasks like sitting, sleeping driving. But that this is at the expense of 1% per step accuracy when comparing the counts on 1-2 miles of walking. The sweet spot would be around 7. Key: Attachments: Posted at 2021-07-05 by HughB We need some other testers to join in. Posted at 2021-07-07 by @gfwilliams This looks really promising! Thanks for the log as well - that really helps. Sorry for the lack of replies. I've been a bit busy lately and I'm on holiday for 2 weeks starting next week, so there's a lot of preparation/finishing stuff off.
Yes, it was in Threshold beforehand was 0.99g -> 1.01g so the acceleration had to dip below 0.99 then rise above 1.01 for it to be counted - but that's way less that in the paper you found Posted at 2021-08-04 by HughB Anymore thoughts on this @gfwilliams. I have tried a couple of other iterations but the payoff is between accuracy and not recording steps when not walking. The only major thing I could suggest would be a simpler filter that has much less ring. After that you would have to go to more complex approaches like the muti-threaded oxford counter - but that will use up more precious RAM etc. I think I have gone as far as I can with this. Posted at 2021-08-05 by @gfwilliams Thanks - I've been a bit busy lately I'm afraid. I feel a bit uneasy about the need for things like
Yes, I've been wondering about this. I was thinking a simple low pass filter might do what we need (producing a nice, smooth signal) without introducing anywhere near as much ringing. Did you find anything out about the best frequency cut-off to use? I know I used 2.5Hz, and I could just do that as a low-pass filter (without trying to cut off anything below 1hz). Posted at 2021-08-05 by HughB 3Hz was quoted a lot in the papers I read. I constructed the table below when thinking about this. 2.5 steps / s is fast marching pace where its easier to run that walk. 3Hz would allow for some slow running to have a chance of working. Suggest trying a simple cut off at 3Hz. The 'windowing' concept in the oxford step counter is effectievely the state machine approach which ensures step events arrive within the expected time range for every step. Attachments: Posted at 2021-08-07 by valvin Hey, it looks to have some interesting work here :) Is there some implementation available on which I can do test? I see there was a js app in the thread should I use it or is there a firmware which already implements what does the js app? Thank you Posted at 2021-08-09 by HughB I dont think there is anything new to test yet. But it would be good to gear up for testing something new as real world tests take up a lot of time and require focus to record the results and repeat the same test again etc. The steps_mc35.js on page 3 of this thread is my attempt to improve on the current firmware, on the basis that experimenting in Javascript is a lot easier than compiling the C and flashing a new version of firmware onto your watch. The key issue with the current step counting algorithm is that it counts steps when you are not walking eg when sat at a desk typing, when driving, when sleeping. Its no good waking up with 1200 steps on your step counter when all you have been doing is sleeping in bed for 8 hours. This is due to a couple of problems. 1) the low pass filters keeps oscillating for 5-7 'rings' after the movement has stopped 2) the firmware only enforces 6 steps in 6 seconds but does not enforce the period between each and every step. So example sssss_s (where s is a step and _ is a longish gap) is accepted as long as it took 6 seconds. What is needs is that the gap between each step is checked against the timestamp of the last so that it falls within 0.3s to 1s. steps_mc35.js adds a gating meachanism to shut output of the filter off if the movement has stopped and secondly adds a state machine to check the gap berween each step. You would be more than welcome to test that so that you get a feel for how I was experimenting. Just load the file onto the watch through the IDE and execute it through the FileManager. When testing I normally wear the bangle and another smartwatch like a FitBit or AmizFit Bip on the same wrist. I record step counts at start and end of the test period and the type of activity and compare results. As well as walking 1,2 or 3 miles I check counts when sat at a desk working for a few hours, when driving, when sleeping. The conclusion at the moment is that we need to try a different low pass filter that has less ring. If we can get close to 1 ring when the movement stop that can be compensated for in the state machine. Posted at 2021-08-19 by @gfwilliams I've just pushed some new code to the Espruino, so the cutting edge travis builds will have a new step counter: https://www.espruino.com/binaries/travis/master/ Basically it's a much simpler ~3Hz low-pass filter with virtually no ringing, with the previous counter which handles 4 steps in ~8 seconds. I'd be really interested to see how it performs. Now I don't have a fitbit, but wearing this and a totally standard Q3 watch on my wrist has the Q3 reporting almost exactly half the number of steps. If I actually count what I consider to be a step (one foot hitting the ground) then Bangle.js is now seems pretty much spot on so I can only assume that the Q3 is actually reporting a step as 'one,two'. I've been typing for several minutes now and it's yet to report a single step, so I think that's a good sign too. @hughb how would you feel about running the 'acceleration recorder' over the course of a whole day, and taking some step count figures from other devices over that time period? I feel like that'd give a really good baseline that we could try these algorithms against, as obviously the odd little sections of walking data aren't really cutting it. Posted at 2021-08-19 by HughB I'll flash the 2v10.9 firmware. I'll do 4 basic tests. WALKING - 1m, 2m walks - looking for around 2000 steps counted per mile. I suspect this will work. Counting steps when actually walking is a lot easier than not counting them when doing othre things. The previous version of the firmware with the old filter was quite good at this. SLEEP - how many steps recorded while sleeping - needs to be less than 100. That will give an initial set of results that can be compared with the other results I published to this thread. We can then look at doing the recordings. Will there be enough space on the bangle to record 8 hours of accelerometer data ? Posted at 2021-08-20 by @gfwilliams Thanks! And yes, good point about the storage. I reckon it could be around 7MB of data, so too much for 8 hours, but 4 would be ok :( I guess if we stored data in binary it'd be ok, but right now with the acceleration logger it's not enough... Posted at 2021-08-20 by HughB Early results on 2v10.9 Pedometer .................................... Amiz GTS2 ............ Bangle I will do a SLEEP test tonight. Posted at 2021-08-20 by HughB Looks like the filter is much smaller AND massively improved - no ringing that I can see.
Here is the ringing on the V2.09.90 filter in response to a single arm sideways open and close. Attachments: Posted at 2021-08-20 by HughB Same again but for the new low pass filter. A general observation is that I can definitely observe fast movements of the hand and arm are not resulting in any amplication or much of a filtered output. This is loads more effective than the previous filter. Attachments: Posted at 2021-08-20 by HughB I am happy to record a 4 hour segment of mywork day with a bit of walking around the house etc. Posted at 2021-08-21 by HughB SLEEP test on v2.10.9 firmware - 149 steps.This is a really good result. v2.09.90 was about 1200 steps. 100 steps is 1% of a 10,000 step goal so anything over 100 steps needs to be looked at. The accuracy goal for me is 97% accuracy - thats 300 steps off in a 10K step day. For a step counter to be considered good it needs to be 98% of a calibrated mechanical step counter. That was quoted in one of the papers I looked at. Doing an overall test for today, which will be housework, going to the tip, walking the dog, buzzing about etc. Will update at the end of the day. 3Hrs GENERAL ActivitiesAmizFit GTS2 .... 1984 steps Bangle (2.10.9)....... 6323 steps, massively over. This was 3hrs of washing up, clearing the garage, driving to a chartity shop, unloading, driving to the local tip, unloading, driving home. Posted at 2021-08-21 by HughB 4 steps in ~8 seconds. I would argue this is too long a period for a single step. It would allow steps at the rate of 2 seconds for a single step. Which in reality is not walking. Its slow motion. 1 step per second is about as low as we want to go. Posted at 2021-08-21 by HughB @valvin - new cutting edge firmware out. V2.10.9 . New Pedometer. Worth testing. Posted at 2021-08-22 by valvin Edit: solved but i still have low mem on startup when it displays 'gps time' but disappear after. Thank you, unfortunately i'm unable to upload any firmware on my bangle. Posted at 2021-08-22 by HughB summary of results so far. I attempted to build a javascript version (3.91) using the new filter. But this failed. I noticed steps were not being detected as the 1000 threshold was not being crossed. This is a difference in the way the filter amplifies the input. Rather than tweak threholds I decided to try PEAK detection. This has the advantage of being self adjusting. However it does need a simple gating mechanism otherwise you start finding peeks in the noise signal when the bangle is at rest. When combined with the state machine it works really well. I have tested against the oxford filter but will go back and repeat against the bangle v2.10.9 filter taps and see which one looks best. The C code of the oxford step counter has a much smalled set of filter taps and implements two filter ranges. See code. https://github.com/Oxford-step-counter/C-Step-Counter/blob/master/src/filterStage.c This has:
I figured I could scale the vales down so divided by 256 and tried this as a filter as well.
My latest javascript code is at: Next step for me is to retest another version (3.11) of 3.10 with the same filter taps in the 2.10.9 Bangle Firmware. The key problem is when driving. Even Fitbit had a lot of trouble with driving counting steps for a while. There is a support thread on the Fitbit site with users complaining about counting 1000s of steps which driving. I think it depends on the device and formware as I suspect different devices had different teams write the firmware. My original Charge HR seemed to do ok when driving. AmazFit have the driving problem cracked - so it is doable. I'm happy to try the 4/8 hour recordings but think it would be better just to record actual activities like WALKING, WORKING at a DESK, SLEEPING. My worry would be if you just adjust thresholds to fit an end of day value that you will loose accuracy when counting walking steps. The key problem to be solved I think is screening out counting steps while driving. I did some recordings of my own today, just printed the raw/filtered output to a file in the IDE when I was driving for a few minutes. I could clearly see the impact of going over speed bumps and little bumps in the road. I am wondreing if it will be possible to get an RMS value for driving vs walking etc. Attachments: Posted at 2021-08-23 by @gfwilliams Thanks for all the research! So I guess right now the firmware is a decent improvement over what it was at least. I'd have thought that actually recording specific activities would be absolutely fine - I think the real thing would be to ensure that we have a balanced set of data - so when running tests on the PC I'm not placing too much emphasis on eg, walking, and not enough on sleep or driving. I was surprised about the 4 steps in 8 seconds too - it's just that running through the data I had, that's what appeared to give the best result. Although as we know, the data I have doesn't seem particularly great. With the adjustable threshold - I have tried a variety of ways of doing a threshold, and at least from my experience it seemed that it wasn't worth it. Obviously you may find it's different, but especially now the filter is reasonably flat (eg the amplitude doesn't depend on step frequency) it seemed to me that a flat threshold might work better. Posted at 2021-08-23 by HughB New firmware is better but still way out for none walking activities. With regards to threshold - I will post some charts I did that compare the filter outputs when walking. The old filter had more pronounced amploification and also tended to centre the output wave around zero better. The new filter can have a full since wave above the zero line. This is why I have now coded a PEAK detection method. To do that I store the last 3 consecutive filter readings (Left Middle, Right), If the middle one is greater than the left and right reading then a PEAK has been reached. This seems to be much more adapatable, self tuning and reliable. You can see my code in the github link above. What did you think about using the simpler oxford filter ? I'm testing the firmware one and the oxford one at the moment. Looks like the oxford one is even better. Posted at 2021-08-24 by @gfwilliams Ahh, thanks for the explanation - yes, peak detection sounds good - I guess with the low pass filter we're smoothing the waveform out enough that it makes a lot of sense. The new filter is low-pass, so it's passing the low frequency including any offset - hence it being above zero I think. I'm not against using the Oxford filter either, if it ends up better. I guess ideally I'd like to be able to run it over some data and say for sure that yes, it does do a better job. Posted at 2021-08-24 by HughB Working on 2 sets of test results. 3.10 and 3.11 as same code except 3.10 uses the oxford filter. I'm leaning towards that one as it is looking better. Here is an accellog for 1834 steps walk (as per AmizFit GTS2). Attachments: Posted at 2021-08-24 by HughB last record in above log is: 938968,-1468,217,-8059 Posted at 2021-08-24 by HughB here's why we need PEAK detection not threshold crossing. Look at the steps after 6000ms. They are not crossing the orange threshold line. This was why when I took the filter taps into my App the state machine was not being called. Attachments: Posted at 2021-08-24 by HughB Here's 10 steps with the oxford filter. In my 3.10 version. Attachments: Posted at 2021-08-24 by HughB Moved my latest experimental code to: Posted at 2021-08-24 by HughB
Can I beg you to implemnt the state machine as well in the firmware. It really is the only part of the algorithm that is accepting and rejecting real and false step sequences on every step at the moment. The current F/W implemenation does not do this well at all in my view. Its an essential part of the algorithm. If you look at the results I am getting with the javascript version - the improvement for non step counting is entirely down to the state machine. It may seem a bit more complex BUT I think the results prove its worth. It would potentailly mean you would have to change the STEP event to return X or 1 steps depending on if there had been a full pass through the state machine at the start of a sequence. But its a small price to pay in my opinion for a vastly improved step counter. There is still a lot more tweaking and testing work to be done. But if you move to the same approach I am using in 3.10 we will be aligned. I am considering building a test rig so I can feed recordings into my javascript version so I can do quick tests of different values of X in X and the raw threshold etc. Posted at 2021-08-25 by HughB new recording. Sitting for 1hr with 2-3 getting up and moving to a different room. The AmizFit recorded 58 steps for this. Attachments: Posted at 2021-08-25 by @gfwilliams Thanks! Yes, I'll add it in. I guess from my point of view it seemed like it didn't matter whether it recorded two separate periods of stepping or not as long as it rejected steps that were on their own, but if you're finding that it really does make a difference in step rejection then yes it shouldn't be too hard to add. As far as the test rig goes - yes, you could build something with Node.js on the desktop very easily I imagine, then you could plough through the test data very quickly. However at some point if the approaches are aligned you could move to the test rig I'd already made for the C version? It's not that hard to compile and get working at all - if you had a raspberry Pi I imagine it would 'just work' out of the box. Posted at 2021-08-25 by @gfwilliams Just a quick one - in your tests, is the minimum step threshold actually needed? It seems like the low pass filter will filter out any high frequency so you shouldn't even gets steps that are less than 0.3 sec apart? It's just one less variable that needs tweaking :) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2021-08-25 by HughB Hi @gfwilliams - I recorded a 6hr accelog recording. When I attempted to download I got the following being dumped to the IDE console and the watch hung and stopped updating the watch app. Had to do a reset.
Posted at 2021-08-25 by HughB Assume you are referring to the min time threshold of 333ms for a step in the state machine. Good question. Yes its needed. In theory it should not be needed but we dont have a perfect filter and a perfect sampling rate. I have seen the lower threshold reject false steps when driving and other occasions. I have done a lot of watching of the debug output I print to the console so have a good feel for what is going on now. We could take it out and do test runs through the recorded data to see the impact. At the moment I dont think it is causing any harm or a major source of error. One thing I am wondering if we need a different format for the recorder. The recorder logs the x,y,z componants - but in the code we use the m value which is calculated by the firmware. To eliminate another potential difference / source of error - I think we should record time and m only. Posted at 2021-08-25 by HughB I have a linux setup on my chromebook so I could setup the C version. I will do that once I feel I have got to the end of testing / loggng. I will also setup the Node.js as well. Posted at 2021-08-25 by @gfwilliams Argh, shame about the download. Are you using the online IDE, or the 'native' one? In the past I know we had a problem with timeouts on long downloads, but I thought that was fixed now. What if you download using the app loader itself? Connect to your watch, go to 'my apps' and then click the upload button next to the app. Thanks for the info about the threshold... In terms of the logging - working out magnitude from the data is pretty easy and is also easy for us to test. While it's less data to log (we don't even really need to log the timestamp), part of me was thinking that at some point we may need to look at the orientation of the watch to filter out certain types of unwanted step - and in that case we wouldn't want to have to re-record any data! Posted at 2021-08-25 by @gfwilliams Ok, I've just converted this to C and pushed updated code. Running it with the csv file I gave shows interesting stats though:
So during walking, not much difference in the state machine, but it will hopefully manage to reject more non-walking steps as you say. Posted at 2021-08-25 by HughB Assume line 1 - is original code / filter.
Its the driving that is the weakest part now. I suspect tweaking things to get these to 0 will massively reduce the accuracy of when counting actual steps. Here are the things I think can be tweaked.
Attachments: Posted at 2021-08-25 by HughB Why there need to be a threshold on the raw M inputPlace the watch on a table and you will still see some raw m signal from the accelerometer:
This signal has PEAKs. When using PEAK detection it means that the step state machine will get called from very small movements - EG when sat on a sofa. I have flashed 2.10.13 and am testing against 3.11. 3.11 has this bit of code :
What this code is doing is saying is the raw m value bigger than room noise for more than 3 samples. I'm convinced this concept is needed AND it can also be one of the parameters we use to TUNE the step counter against the accellog recordings. Posted at 2021-08-26 by @gfwilliams Thanks for all that extra data! I just added it to the step-count repo. Thanks, I'd missed the threshold in your code. I've now implemented it but as far as I can see it made no difference whatsoever to the steps recorded using your 3 data files. I guess maybe the threshold needs to be higher. ... but again, this is why it's really good to be able to run the code in a test harness However, I feel that what you've basically got there is a threshold? Before using the peak detector, we had the threshold on the filtered data - and personally I feel like it would be a lot cleaner to do the threshold after the filter. As is, when gate_open becomes true, accFiltered is now suddely>0 and all it takes is for the next sample after that to be lower and you've now got a step recorded. Posted at 2021-08-26 by HughB
Totally agree. Once we have all the necessary logs and agreed algorithm, identified parameters that we tune against the data - this will be the only practical way to run the 100+ tests that might be required to optimise the 2-4 tuning parameters. I have installed Node.js / npm but I have no Node.js experience - so its going to take a while for me to convert my code to a test harness. Going to focus on getting good recordings for a bit longer. Plus track the firmware updates you are doing. SLEEP TEST RESULTS
No this is not the case. Count up to 3 consecutive ABS(raw) values that are over threshold - then open the gate The gate opens up quickly when there is real movement, stays open and closes down quickly when the movement stops. The M of the accelerometer when the watch is at rest/stationary is biased in the 3-8 range.
Ideally I would agree but the amplication of the filter is not linear and very dependant on what gets through. With the 2.9.90 filter any input got heavily amplified. Also with a 12.5Hz sampling frequency we can only have a filter that works up to 6.25Hz - this introduces distortations and The only way we could prove lineararity would be through mass tests and to try and get a set of real world recordings that had a range of increasing raw values in the output. The driving recordings are the most interesting for this. Using the theoretical response is not going to help when in the real world the signal has got low level noise and bumps in the round etc. Posted at 2021-08-26 by @gfwilliams
I'm not entirely sure you got my point about this. You set up If you got rid of JS test harness would be good, but I guess if we have all the data, we already have a working test harness in https://github.com/gfwilliams/step-count that uses the actual C code from Bangle.js, so long-term it feels safer to use than than to maintain two different step counters - one in JS and one in C Posted at 2021-08-26 by HughB
Yes apologies, I realised a few hours later this is what you mean't. Good point, probably wont matter as we have the state machine to get through, but that assumes the state machine has timed out. But I agree its one less false PEAK and logically better.
Yes I agree. The javascript stepcounter app was only to allow quick prototype of ideas etc. Uploading a log of 1 hour working at a desk. 0 Steps recorded on the AmizFit GTS2. Have just cloned the step counter test harness repository and ran it successfully. Done a lot of experiments and arrived at 8 steps in 8, gated entry to the state machine based on a raw threshold of 17 and no threshold at all on the filter output. This works really well. I did a test with the gating removed first and the difference on counting non steps was in the 1000s. Latest test run below. File, Expected, Simulated, Diff, (Orignal) Have done a couple of pull requests to the step-count test harness AND the Espruino repositories. I think we now have a pedometer that works ! Attachments: Posted at 2021-08-27 by @gfwilliams That's great news, thanks for all your work on this! That's really exciting! Posted at 2021-08-27 by HughB Added 2 more logs, updated harness, done pull request. Posted at 2021-08-28 by HughB Ok - so I think in practice the Test Harness results are a bit on the optimistic side. Does the M value come from the chip or is it calculated by the firmware ? gcc -std=c99 main.c -o main Posted at 2021-08-29 by HughB I did some actual realworld tests and came to the conclusion that the test harness results are not matching what you would get in a real world physical test. However the situation is complicated. In order to investigate will need to record the steps according to the AmizFit and also the current version of the step counter running on the bangle. Then compare with what the test harness gives. My final conclusions about what is working are based on real world tests rather than the test harness results. Our users will only every test this way. 8 steps and 17 as the Thresold turned out to be too hard = it reduced the impact on DRIVING but at the expense of the accuracy of real world walking.
Tried v3.13: 6 Steps / Threshold of 14. SLEEP Amiz 0, 3.13 65 More real world test cases required BUT Here is my justification for the pay offs between This is what we ideally want in terms of behaviour from our pedometer: Lets consider a typical day to consist of: ** HOWEVER - I have observed that DRIVING is very non deterministic even with the AmixFit. The My final push on the firmware repro will be to set X=6 and RAW_THRESHOLD=14. Future Work Posted at 2021-08-31 by @gfwilliams Ok, great. I'm not quite sure what you mean by the M value? Acceleration magnitude? You can check in the code but the calculations to get that from x,y,z are done in the Bangle and are super simple. We should really be doing the same stuff in the test harness Posted at 2021-08-31 by HughB
Yes Magnitude.
Yes - need to be the same so we can do predicatble modelling that will match the real world measures when actually walking with the device etc. I was not sure if the accelerometer chip does the calculation for magnitude OR the firmware does it. In the test harness I can see the In32_sqrt() function. Is this the same in the Bangle firmware. Was not sure where to look in the code. Found this in ./libs/banglejs/jswrap_bangle.c
Found this in
Not really sure how this compares to:
Apologies I think * is confusing the code formatter Posted at 2021-08-31 by @gfwilliams I think you're using 4 backticks where you only need 3, which is what messes with the formatter. The micro ECC one isn't related. What happens is:
and The test harness should be the same: https://github.com/gfwilliams/step-count/blob/master/main.c#L77-L87 The int32_sqrt is defined inside stepcount.c: https://github.com/espruino/Espruino/blob/fc2d816c57bd36da89972aa39775ba03c3d591a5/libs/misc/stepcount.c#L156 so it should be identical between calls Posted at 2021-08-31 by HughB Thanks for that. The test harness is basically using the same code that the step counter is using ? So I was scratching my head and realised that my javascript version does
Hence wondering if a.mag comes from the accelerometer chip or its calculated in the firmware ? Posted at 2021-08-31 by HughB I am wondering about widpedom though and the step event.Have now flashed my bangle 1 and 2 with 2v10.27 so should be able to do some real world tests against the actual firmware now rather than the javascript version. Does Widpedom get 6 step events when the firmware passes through the state machine to reach 6 steps ?
During the day there can be 200 passes through the state machine that would result in 6x200 steps. 1200 steps. If the firmware only returns 1 step for these events then at the end of the day widpedom will be 1000 steps short. Have just verified experimentally that widpedom is only counting one step for the first 6 steps of the state machine. I think either a) Bangle.on('step') needs to provide a step size value of 1 or 6. OR b) the firmware will need to send 6 step events when it moves from S_STEP_22N to S_STEPPING state OR c) need a Bangle.getSteps() which will return the number of steps since the last reboot or something like that. Posted at 2021-08-31 by HughB Results of testing javascript version 3.13 will test the Firmware v2.10.27 once its clear how widpedom is handling the passes through the state machine. Attachments: Posted at 2021-09-01 by @gfwilliams Ahh - right. So It should be 1/8192 stepcount uses an integer square root but otherwise the value should be basically identical Posted at 2021-09-01 by @gfwilliams
Ahh - yes, you're spot on there. The Posted at 2021-09-01 by HughB
Fantastic. Will update tonight. Still puzzling over why the test harness results seem different to what I measure in reality but will investigate with the latest widget first as its the only way I can get the step count out of the firmware. Posted at 2021-09-01 by HughB Accellog v0.2 appears to be broken on both b1 and b2 ? What I see is a quick flash of the overwrite YES/NO dialogue then it returns to the accellog menu. Nothing on the console when connected via the IDE. Posted at 2021-09-02 by @gfwilliams Just tried and It works fine here for me... Maybe you can run the IDE at the same time and see if any errors are reported? I know you had issues with the IDE not uploading that font module for you (was that ever fixed?), and I'm wondering whether you're getting the same thing with the App Loader now where it can't find the Posted at 2021-09-02 by HughB On B2 - Deleted through the AppLoader. Checked console log in Chrome Browser, no obvious errros. Attachments: Posted at 2021-09-02 by HughB Getting early test results now against the 2.10.27 firmware that has the full implementation of the step counter in it. Also using the upated widpedom. SLEEP Amiz 8, 2.10.27 89
Added logs for the the DRIVE and WALK and into the test harness. X_STEPS = 6, RAW_THRESHOLD = 14 If you look at the walk results, the simulated value of 3013 its actually only 33 steps off the value Going to leave everyone in peace for a bit as on holiday next week :) Posted at 2021-09-03 by @gfwilliams I just realised something about accellog - maybe try now? I hadn't updated the minified Layout library so the old one was getting used. Thanks for your work on this! I'm on holiday next week too - but this is all looking really promising now! Posted at 2021-09-03 by HughB Just tried accelog. Problem solved. It is just about usable on a B2 through the touch screen. The problem is that the font for the list menu's is so small you need pin point precision with your finger poking to invoke the right option. I think we need at least double the font height for the list menu's. The problem will then be the horizontal space to get the TEXT in. So maybe the list needs to be spread out a bit so that the available touch area to select the option is a bit larger. I guess these will all be discussed over the coming weeks as we gather experience of the B2. Posted at 2021-09-03 by HughB
I think we have reached the point now where we have a credible - but not perfect step counter. Improving it further would require a much more complex algorithm, but we now have a good baseline if someone facies taking it to the next level. Be really interesting to see what others think when 2v11 gets released - assume that is sometime away as you have only just released 2v10 ? Posted at 2021-09-14 by @gfwilliams That's great! I don't have a timeline for 2v11 yet, but I don't plan on leaving it so long this time. I may attempt to get some built-in handling of HRM and pedometer logging done, in which case I'd do a 2v11 release after that.
I've had other complaints about that too, but that is why you don't actually have to tap on the menu item. You just drag your finger up and down to scroll the selected item, and tap to select. I felt that as you point out, if you had to actually tap the item itself, realistically you'd only be able to have maybe 3 menu items on a screen at once, which would make most of the menus very hard to navigate. Maybe Posted at 2021-09-14 by HughB
Yes, done the scrolling bit but assumed that the tap had to be on the actual menu item. Its pretty nerve racking when you want to select VIEW and the option below it is ERASE or DELETE though.
Currently there are 8 lines of menu items but they are hard to see and select. I guess this discussion belongs in a seperate thread. Posted at 2021-09-15 by @gfwilliams It's configurable in that you can install an app the specifies its own I had added a new, larger font for Bangle.js 2 (the one used in the menus), but I guess that could be even larger and maybe more bold?
The thing is they're not hard to select though? You only think they're hard to select because you think you have to press your finger directly on the menu item, but you don't. I've just tried here and even with my large thumb pressed flat on the screen I can easily and repeatably select menu items. I get your point about the text though. Personally I find it ok and I made the new font to ensure it was the same physical size as the Bangle.js 1 menu font which I felt everyone had come to terms with, but a bump in size wouldn't hurt. But yes, this should definitely be in a new discussion. It's actually really hard to get a clear idea of what most people want. Many will find it absolutely fine as-is and would find having less than half the number of menu items on the screen frustrating, so it's just getting a good balance. Posted at 2021-09-15 by HughB Thanks - I can confirm I can use the TAP mechanism and it works well. Posted at 2021-09-26 by HughB Here's the baseline set of test results for the step counter in the current firmware (2.10.27 and beyond). Attachments: Posted at 2021-09-27 by @gfwilliams Great! Looks pretty promising. Shame about the housework one though. Posted at 2021-09-27 by HughB
Here's the baseline results through the test harness. X_STEPS = 6, RAW_THRESHOLD = 14 Now it is possible to improve the housework log to +32% over by increasing the thresholds. X_STEPS = 8, RAW_THRESHOLD = 17 So I think without a radical rethink this is the limit of what can be acheived with this algorithm. Posted at 2021-09-28 by @gfwilliams Yes, seems sensible. There's also nothing to say that the watch you're using for a baseline is actually logging the steps for housework correctly either... Posted at 2021-10-02 by user107850 Hi all, excited to see how things are progressing here. I was curious to know how much of the original windowed peak detection is left in the current implementation? The last version of the Oxford step counter in C was optimised for Bangle actually, and I even added a detection stage to filter out non-movement based on a minimum amplitude. I collected some data myself, 17 files with manually counted reference, and the accuracy was >90%. Gordon, maybe having some sort of competition would be indeed a good idea, but we need to set it up well, which means good data and a simple way to run the code. I could also propose it to students here at the Uni. A bit unrelated, but we could do the same for heart rate. Posted at 2021-10-02 by HughB @user107850 - were you involved with the Oxford Step counter project. I dont think @gfwilliams's attempt to follow the Oxford approach got off the ground. The actual C code for the Bangle step counter is at: If you were involved in the implementing the Oxford step counter I would be really interested to undertsand how the code works as I found it hard to follow after spending many hours looking at it. The event / queue driven structure of the code had me totally confused. I am also interested in what testing the Oxfrod team did to test non stepping activities like driving.
I have collected accelerometer logs for various activties. We have about 20 measured logs and these were calibrated against an AmizFit Bip which appears to be pretty accurate from the various tests I have done. When collecting the logs I wore the Amizfit Bip on the same wrist as the Bangle and recorded the steps recorded by both devices. The output in my above posts are from the test harness and the results are a good match for what gets measured in reality - so as a good simulation against algorithm changes. I would be really interested in talking with you if you know more about the Oxford approach. Posted at 2021-10-02 by user107850 Hi, yes I was involved, I proposed the idea as a BsC project and I supervised students both at Oxford and at Malmo who worked on that. The approach is described in two papers: here (behind paywall) and here. The basic idea is that the signal is converted to magnitude, then peaks are somewhat amplified, identified and selected. You can split the algorithm into a pipeline of stages, with buffers between each stage. Not all stages are mandatory, for example I observed that interpolation and linear filtering do not bring any significative improvement and can be skipped. As far as I remember, Gordon embedded the algorithm into the Espruino firmware at some point, but he complained about the fact that it took too much memory. After that I spent some time trying to remove redundancies and optimising memory a little bit, but I don't know if Gordon included those changes or if he decided to adopt another approach in the end. Re. non-walking activities: the algorithm was meant to be used during walk and was not optimised to distinguish between genuine walk and other activities. To address this, I later added a basic movement detection step which improved things a bit on my short tests, but I have never tested it "in the wild" for a long time. As for the competition, we could separate the algorithm from the Espruino code, prepare the testing dataset and add some wrapping code that runs the algorithm and compares it with the reference. I have something similar already I used myself to test my algorithm: I would happily polish it a little and share it if there's interest. Posted at 2021-10-02 by HughB
Yes - I looked at this code. I took the filter taps from its and scaled them down to work with the Bangle. A lot of the code is around the pipeline and buffers. It would be interesting to see what it looked like without the buffering and pipeline and was just sequential. The approach the Bangle currently takes is very similar
We have some good accelerometer data at: I would be really interested to see how the oxford step counter performs using the logs below: HughB-drive-a3-b136.csv HughB-walk-a10021-b10248.csv - is a 10K step log which can be used to test walking The a value is the steps recorded by the AmizFit Bip. These 4 test cases are enough to assess the accuracy when walking and not walking.
This where all the difficulty lies. I am pretty certain any student can make a step counter that is reasonably accurate when walking. I have seen all sorts of code / papers that claim good accuracy but they never discuss a real world test methodology around not counting steps for non step activities. This is the bit that no one seems to do. As you say some of the other parts of the algorithm are not really contributing much when the problem is just counting steps. It is the screening out of the non step activities that is the important thing otherwise you count 2000+ steps just typing at a computer sitting at a desk for 4 hours.
If you could build a framework around those test cases above then I think we would have what we need. I'd really like to see what the oxford results are for the 3 non stepping cases, if it does this well then we might not need a competition. Posted at 2021-10-03 by user107850 Hello,
Yes, I asked students to do that but they didn't have enough time. I agree that the buffers are not needed.
Peak detectors are somewhat similar. The main issue is adapting to a signal where amplitude can change quickly. There are also other approaches than peak detection, like analysis in frequency, wavelets, or even machine learning, but I don't think they are suitable for an embedded device like a smartwatch.
Yes, I'll set it up and post the link here. I'll try to do it next week. Posted at 2021-10-03 by HughB Regarding ringing, yes we switched to a low pass filter. Its the same one you use in the c code with the coefficients scaled down. Posted at 2021-10-04 by @gfwilliams Hi @user107850, Thanks for your work on this! I did pull in and try your changes (and they were in 'cutting edge' Bangle.js firmwares for a while), but actually I think the most recent changes came after I'd moved to a new algorithm myself, so I didn't end up trying those I'm afraid. I felt like the problem initially was that the original, super-basic step counter actually did a half decent job of step counting for normal walking (it was within 10%). The issue was for everything else. The Oxford step counter was undoubtedly better for walking, but it actually (at least initially) seemed even worse for non-walking activities, and I had the same issue as @hughb understanding what was really going on so I couldn't easily make useful changes (although looking back, your 'pipeline of stages' explanation suddenly makes things look a lot clearer!). Given the reasonably large memory overhead and complexity, I figured I was better off starting from scratch with something I understood as it seemed like the overall algorithm should have been really simple. I did however shamelessly steal a few of the Oxford Step counter ideas though - developing on desktop first with a data-driven approach (https://github.com/gfwilliams/step-count) and using that great filter generating website. As @hughb mentioned, initially I thought that a bandpass filter would help me filter out low frequency elements of the accelerometer data as well as high, but while on the whole it worked well, a big spike of acceleration would create ringing that looked like a bunch of steps. We moved to a low pass filter (then finally to the exact Oxford step counter one!), and @hughb added peak detection code (which then didn't require high pass filter) and a state machine to count steps. I feel like it's actually working pretty well now (although I still wonder whether filtering on X/Y/Z separately and then getting magnitude would help with false steps). However, there is always room for improvement, and getting students at Malmö involved would be amazing if there's interest. I'm happy to do a competition, but last time that was mentioned, there wasn't too much interest :( Apart from step counting there's heart rate as you say, which is potentially very interesting. It seems the manufacturer supplied algorithm (which is a binary blob that we don't really have access to the code for) uses accelerometer data (probably to discount readings) but is able to work off very low res data (like maybe 4 samples/sec) which means vastly improved energy usage since we currently run at 50 sps I believe. There's also a whole area of 'exposure adjustment' where the LED power is scaled in order to get the best readings. We're actually using the manufacturer-provided version which we reverse engineered at the moment on Bangle.js 2, but it just takes forever. There's also sleep tracking which would be really cool (and maybe other options around that). I imagine that's something that could definitely be handled in a data-driven way if we had a few captures of accelerometer data over the course of the night. Posted at 2021-10-04 by user107850 Hello @gfwilliams , thanks for the reply. I understand that it makes sense to try with a more ad-hoc peak detector, especially if you know that the hardware is always the same. Our algorithm can be optimised a lot, starting from removing the buffers, but we didn't have enough time for that. As for the heart rate, I would probably trust the proprietary blob better than anything else. With time, if we manage to get a reliable algorithm, we would ideally get rid of it. Back to the competition idea: I have created a repository where we can test algorithms, see here. I have already populated it with the data I have collected myself plus some of the data collected by @hughb. I have not included those files where the two measurements of steps are too different, because I don't trust either, and for those that I have included, I have computed the average between the two given step counts. @hughb would you be able to send me a pull request with your algorithm? Please check the structure inside the dummy algo. If that doesn't fit with yours, I am happy to change it, just le me know (or submit an issue on Github). With time, we can add more algorithms, or simply improve ours. It's going to be fun (at least for me😀). I would like to keep this repository also for testing the algorithms for heart rate or sleep detection when we get to it, and I will probably share it with students if I manage to involve some. Posted at 2021-10-04 by HughB
Its pretty good, but not perfect. I'd like to get the housework log down a bit. When I have a day at home sat at a desk I expect to see about 600-800 steps for the day, and often see 1500-2000. But where as before it was really obvious where the inaccuracies were I think it might be a while before users start raising issues. Once the new daily logging is in place then I thnk it will be more obvious for people to spot the over counting.
We'd have to do the experiments. Maybe an experiment for the future using the javascript version. Posted at 2021-10-04 by HughB I will have a look at how easy it is to attach the current step counting algorithm into your framework. Note its already in the current firmware at: I'm concerned that the test harness does not have enough non-stepping data and long enough logs. We must have good tests cases for when not walking - otherwise we are just prone to the same mistakes and errors as before. We can easily be fooled in thinking a code tweak has improved things when it fact it has not. A bad selection of test data will waste our time. Any step counter must be measured against stepping and non-stepping scenarios in equal measures. The Accelerometer logs below are all ZERO steps and should be part of the controlled test data in my view. I was not walking when they were logged, I was sitting or driving or typing at a Desk. These and similar logs must form at least 50% of the test data in any test harness. HughB-drive-36min-0.csv It would be useful to keep the same file names so that provenance of the logs can be tracked. Or we should keep a proper register of every log detailed where it came from and and any other measurements that were done against it at the time. I am also concerned that the current 0 step samples (0.csv 1.csv, 2,csv) are only 30 seconds long. This is not long enough to flush out problems. For example the drive-a3-b136 log was a 15 minute log and the AmizFit Bip registered 3 steps where as the Bangle registered 136. I suspect it would be possible to select many different 30 second segments from that log and have a target step counter register 0 steps - thus fooling ourselves. I'm glad you have included HughB-walk-a10021-b10248.csv as 10134.csv as its a really good log. It was recorded recently. I wore both watches on my left wrist whilst walking across the yorkshire dales :) Posted at 2021-10-04 by HughB @user107850 - I ran into build issues. I had to create the build directory, but there is also a clash with time_t. I am on a Linux Debian environment
Posted at 2021-10-04 by HughB @user107850 - I managed to get it to compile with this hack: I dont think you need to worry about rolling over time_t in 1 year of milli seconds for a test harness where the logs are a few hours at most.
But when I ran it I didn't match the results you showed in the README file.
I noticed that the step count was just incrementing file on file.
Now the results look more realistic and the 3058.csv result for the Oxford of 3301 steps EXACTLY matches your results.
Can you check the 2 changes I have made to the test framework code are correct. I will do a pull request for the test files. Posted at 2021-10-05 by user107850 Hi,
Yes, good idea! The only requirement is that the first characters correspond to what we consider the "ground truth".
Yes, if you are sure about the fact that you did not do any step.
The more data we have the better. Actually we should also include running and walking in different conditions, like on soft floor, hard floor, with and without shoes etc. No algorithm is going to be perfect, some may be more precise when walking, others when not. In the end it depends on the use case: runners may prefer better accuracy when running, others may prefer better accuracy when walking etc. As a general aim I think we should try to have something as good as an average fitness tracker. Posted at 2021-10-05 by user107850 Hello, I have added the espruino step counter as well (with minor changes in names). Looks like it has better accuracy than the Oxford one, especially in absence of movement. Well done! We could use this repository to try to improve the algos with further tweaking. Pull requests are very welcome. Posted at 2021-10-05 by HughB Got the same build error again. Did you resolve the time_t definition clash ? I recloned the repository and started from scratch. Still get the build error.
The easiest way to fix this is to comment out the time_t definition in types.h and add include <time.h> at the top of types.h |
Beta Was this translation helpful? Give feedback.
-
Posted at 2021-10-05 by HughB Looks like the test harnesses are in agreement in that the results we get from the Bangle harness are the exactly the same counts as your harness. Brilliant to see these comparisons.
Thank you. Its good to see the confirmation of the approach and also the approach to testing.
Totally agree. I was attemoting to collect up to 4 10K+ step logs when I ran into a problem with the logger. I will get back to collecting logs again soon. Need some 1 hour logs for sleep, driving, working, sitting watching TV - all ZERO step logs. Posted at 2021-10-05 by user107850 I forget how little portable C is!
Yes, but let's not forget collecting steps data under different circumstances (slow walk, fast walk, running, etc). It would be nice if we could involve other users, just to confirm that there are no major differences across devices. Posted at 2021-10-05 by johan_m_o As soon as I have a working watch (I've lost my charging cable for my Bangle.js v1) I can probably start contributing some data. Posted at 2021-10-05 by HughB Thanks @johan_m_o, do you have another device to calculate the step count? We need logs that are a known step count. I use an Amazfit bip or GTs on the same wrist when making recordings. This allows us to check the simulated counted steps against the step count recorded by the other device. We need logs with 1000s of steps, its not really practical to manually count steps. Posted at 2021-10-05 by johan_m_o @hughb I've got an old Fitbit that I can use as a control. Posted at 2021-10-05 by HughB @johan_m_o - thats brilliant. I've found them to be quite accurate for walking and non walking The method I use is:
When you upload the file looks like we have agree'd a filename format like: 1234_Name_Activity_bYYYY.csv I install the active Pedometer Widget and set the display to large digits so I can see what the step count is. I'd be happy to checkin the logs into the various repo's if you are not used to git etc. Posted at 2021-10-05 by HughB @user107850
Thanks - can confirm compiles without error now. Thanks to work on the harness - its great to have 2 algorithms to compare.
Agree. I would like to get to a consistant 2% within control on walking and 200 steps across the day on non stepping activities. 200/10000 steps is 2%. Still a long way from the later. I have a 1 hour train journey tomorrow so going to try an capture a log just sitting on a train - 0 steps.
I've had a few fitness bands / watches over the years. The most common use case for me and most people I know who use them is just monitoring that you are getting your 10,000 steps per day in. For training / running and anything that needs split second and distance accuracy you have to use GPS - I would see the Bangle Run app fulfilling that niche - if its made to work well. 3hz is within slow running pace, approx 12 mins per mile. There is a fundemental limit on what we can do with the Bangle with a 12.5hz sample time and 3hz low pass filter. The effectiveness of the LPF is constrained by the sampling rate. Posted at 2022-01-14 by johan_m_o @hughb You guys still working on this? The other day I was looking through a drawer and found that old Fitbit I mentioned earlier in this thread (a Charge 2). Made me remember the work you guys were doing here... Today I wore it right next to my Bangle.js 2 with the Pedometer widget running. Wore them from 07:00 to 22:00 and here are the results: This was a mixed day of sitting at the computer (a lot), doing housework, walking to the store, playing with the little one, etc. The discrepancy is quite big, so if you're still working on optimising the algorithms I'd be happy to help any way I can. Posted at 2022-01-14 by HughB Hi there @johan_m_o, I have gone as far as I can with the current algorithm. In the early days it was a aweful lot of testing by actually walking. The current approach has weaknesses but I am not sure how to proceed to fix them. The challenge is not so much counting steps but not counting them when you are not walking. Driving and Housework are the area's where the current algorithm over counts. The good news is that we have a test harness for testing new approaches that is fairly accurate in predicting real word results. We can run a number of accelerometer logs of known step counts against the test harness. I was very pleased to see that we beat the oxford step counter both on walking and non walking recordings. I have tried a couple of tweaks but when tested they did not improve things and in some cases made them worse. In comparison to earlier implementations of the steps counter; on Bangle 2 I have hardly seen anyone complain about the step counter, its been several months since Bangle 2 got out there and your post is the first I have seen saying you think its not as accurate as it might be. There have been a couple of posts by people who appear to have bad accelerometers but not other mention. The other thing I would say is that there can be a wide variation of results across step counters even when doing a controlled test like a 30 minute walk around the same route or a 20 minute drive, there can be 1-5 % variation on the same device etc. FitBit in the early days was notorious for counting steps when driving. We have a new generation of Bangle JS programmers now so I am hoping someone will be interested enough to take it to the next level - I hope they can come up with something that performs better than where I was able to get it to AND the test harness results prove the improvements work. Posted at 2022-01-14 by myownself
I just spent the last hour since @johan_m_o posted reading all about step counting algorithms. When I'm actually on the computer with access to a compiler I might try a few things out just to get a grasp of how the test harness works (the link is quite buried, so for people who haven't already read this whole thread twice: https://github.com/dariosalvi78/banglejs-algos-tester.). I was hoping to have a look at this paper, which seemed interesting: https://ieeexplore.ieee.org/abstract/document/9050443 but sadly it is paywalled. Some of the other papers I have looked at use gyroscope readings instead of or in addition to accelerometer, which is something I don't think the data in the test harness has (right?). Posted at 2022-01-15 by HughB I read a variety of papers, most of them did not demonstrate test results for non stepping activities; such as driving, sleeping, sitting at a desk. From experiments I came to the conclusion that anyone can think they have built a step counter that counts steps well when walking but 95% of the problem is coming up with one that does not count when driving, sitting, moving a mouse etc etc. Any paper that does not cover how to differentiate between activities or tests for non step activities probably is suspect. The Oxford step counter looked good on paper, the technical paper looked impressive but the results were actually pretty bad under a balanced test regime. The easiest way to start is to do the work in Javascript, then you can quickly prototype ideas and dont have to compile and flash firmware. If you read this whole thread you will see some sample code I wrote in javascript which Gordon converted to C. I then tweaked the C and tested against various accelerometer logs that I recorded of known step counts. The link you posted above came back 404. The test harness Gordon developed for Bangle is at: Posted at 2022-01-15 by myownself
I had added an extra dot to the URL by mistake, but I was apparrently confused about which of the test harnesses was the test harness (there are two discussed in the forum, and working backwards through the posts the one I came to was the one I found first).
Maybe, but if I'm going to prototype I may as well prototype in any language. As far as I know the test harness(es) only exist in C, so it is write in C or create my own test environment for another language (and if I'm doing that, it may as well be Python). Someone has kindly sent me the paper I linked to, so I will read that later and see what it brings. Posted at 2022-01-15 by HughB
Sure but the point about prototyping in Javascripts is that you can actually run it on the watch as an app and test it out in the real world that your idea's work. Many ideas look like they work on paper but in reality behave less when on the watch. I implemented the current algorithm (or changes to the current algorithm) in javascript and was able to compare against the existing firmware at the same time.
One is a generic harness that works in C that the guy who managed the oxford step counter built. He added the the Bangle code into it. That would be a good place to start BUT you need to test this on much longer walking samples - 10K steps, not just a burst of 150 steps, an approach I feel is flawed as it hides the errors that will build up over a longer period. The 2nd Test Harness is is the Espruino codebase and you can fiddle with the step code in C and rerun the harness against all the samples. If you are really serious about this I could spend sometime writing up how to run the test harnesses but the information is actuall all there. If you get it up and running make some improvements or come up with better code / options that beat the current results, especially in terms of the weak area's like housework, driving, and sleeping which should be lower AND without reducing accuracy on the counting when walking - I will send you my spare Bangle 2. Posted at 2022-01-15 by myownself I have played around for a few hours this evening. I didn't have any success compiling Gordon's test arness, even though the instructions were straight forward. Is there anything obvious to you?
Posted at 2022-01-16 by HughB I after updating my Espruino and step-count repos, I get the same error.
Posted at 2022-01-16 by myownself No problem. I might be done with this for the day, but if I get to it before Gordon responds I know from your message that I can probably just check out an older commit. I've played with an algorithm today that depending on parameters does great on actual steps but not on e.g. your train journey, or does very well on the train journey (not perfect, but not bad) but misses too many real steps. I am hoping a low pass filter will improve matters. Posted at 2022-01-16 by HughB The LPF we used in the existing algorithm is actually quite good. A previous filter suffered from lots of ringing. So one step would generate 4-5 echo steps. The existing algorithm uses the Magnitude of the accelerometer signal. This SQRT(xx + yy + z*z). Many designs use the Magnitude method as it means you dont have to worry about the orientation of the device - eg held flat. One article I read showed a process of dealing with the x,y,z signals seperately, then passing them through an LPF and them making a decision what was going on. For example when driving one of the directions (y) will show acceleration and deceleration due to the car moving forward, another z will be impacted by bumps in the road at random points, x will be impacted when moving the steering wheel. I am not sure how the accelerometer chip knows its orientation (unless it has built in gyros). So for example what if you wore the watch strap around you knuckles. Does that mean that x,y,z swap round. BTW - what device are you using to experiment with algorithms - is it a mobile phone ? Be aware that sampleing frequency of the Bangle accelerometer is about 12.5Hz. I read a paper that suggest you should use 100hz and that accuracy at 20Hz was poor. I am told by Gordon that 100Hz would plave too much load on the Bangle though. Posted at 2022-01-16 by myownself
My current best versions use only the component which has changed most since the previous sample.
I don't believe the accelerometer does know its orientation, and the watch will be getting turned this way and that while walking and especially running. Other algorithms use a separate gyroscope to determine orientation and then use just the Z component, which best corresponds to a running motion. I thought that the Bangle had a gyroscope but looking through the reference I can't find it, so I may have misremembered (possibly because I've also been reading about the Puck?). Edit: It says here it has "device orientation algorithms" https://www.kionix.com/product/KX022-1020
Actually, I'm just using the data in the two test harness git repos. I didn't check all of them, but I assume they're all as 12.5 Hz. I haven't noticed any papers using as high as 100 Hz, but I did see quite a few use up to 50 Hz. I could use a phone for testing I suppose, but from what I've read the difference between a phone in a pocket and a wrist-based device are large. Posted at 2022-01-16 by HughB
Great. If you are experimenting in C and using the test harness then you are up and running. I recorded a lot of those samples on a Bangle 2 using the accellog app. Most of the ones I recorded are called hughb-something. Its good to have fresh minds on this. Posted at 2022-01-17 by myownself I am just uaimg the data, not the actual harness at the moment. Don't be disappointed but I threw together a 5 line test harness in Python for quicker testing. Everything I've done can be implemented in C, though it has been a good decade since I wrote any. Posted at 2022-01-17 by @gfwilliams Ahh, sorry about this - If you pull now I've just fixed it Posted at 2022-01-17 by myownself Thanks. I'm back at work work today, but hopefully I can find some time this evening to try it out. Posted at 2022-01-17 by HughB
If you have written lots of C in the past (before modern tools etc), then you are probably steeped in it to the point that it will all come back very quickly and you will know how to double check a ton of stuff as you write it otherwise you spent the rest of night looking at a core dump. I found the process of translating my javascript code back into changes to the C part fairly simple. One of the nice things about javascripts is that you can write as if it were C, ie the basic block structure. You will see the state machine I wrote was almost a cut and past job for Gordon to slot it into the C code. My only worry would be if you write another test harness in Python - just because you need to prove the test harness produces good results that match the real world. Both of the test harnesses work reasonably well. Posted at 2022-01-17 by myownself
That is fair enough. My main reason for using Python is that I can most quickly prototype in Python and determine what is a route worth going down. The true measure of success will be the official test harness and, ultimately, real world tests. Oh, forgot to say, thanks @gfwilliams - I can compile the harness now and it runs fine too. Posted at 2022-01-20 by myownself @hughb Although I may come back to this at some point, so far your current approach keeps winning out. It is easy to beat it for one or two of the sample files, but it is always at the great expense of another. My best experiments have all come back to a variation on the same theme as your state machine, and none have beat it. Posted at 2022-01-21 by johan_m_o After @gfwilliams changed the threshold for the step counting I'm spending today testing this. Using the same setup as described earlier, with both an old Fitbit Charge 2 and a Bangle.js 2 on the same arm, and after the first few hours it's looking pretty good. Have had the pedometers working for 4 hours, while I've mainly been going about the house and the majority of the time I've been working at the computer . Current step count is 834 vs 839... I'm gonna do some more housework during the day and maybe take a short walk in the afternoon, so it's going to be interesting to see how it develops. Posted at 2022-01-21 by @gfwilliams
That's very promising :) Posted at 2022-01-21 by HughB Wow appreciate the feedback and experiment you have done. Posted at 2022-01-21 by johan_m_o Update... The time is about 22:00 (CET) and I've taken the watches off. On the 14 threshold the result was, after wearing both watches from 07:00 to 22:00 (mixed activities): Today with the 18 threshold, after wearing both watches from 07:00 to 22:00 (mixed activities, but not as active as last time), the result was: Would have liked to see about the same number of steps, but there's definitely a change. Threshold 14 meant that the Bangle would count 16% more steps than the Fitbit, while threshold 18 meant the Bangle counted 9% fewer steps than the Fitbit. While monitoring the count throughout the day it seemed to me that the count while being still and doing housework was more accurate this time around. The big discrepancy didn't really start until I went for a short walk in the afternoon and then it stayed at about 400-600 steps difference the rest of the day. The above observations seem to correlate somewhat with the reasoning made by @gfwilliams here: http://forum.espruino.com/comments/16360753/. Walking became more inaccurate while other activities became more accurate. More testing needed though... Posted at 2022-01-21 by myownself The question is, what is better. Obviously in the case that led to the threshold change, the phantom steps were unacceptable, but missing real steps for real walks is also not great. Several of the articles I read accept that "steps" is really a proxy for physical activity, and in my view housework counts somewhat in that regard too. If I am using the step counter because I'm on a hike or a run, I want accuracy for actual steps. If I'm using it as a monitor of my general lifestyle, I probably want other exercise-like behaviour included, or at least don't mind about it. Posted at 2022-01-21 by HughB
Agree.
Yes. Posted at 2022-01-21 by HughB @johan_m_o Appreciate this is just a first round of testing, a just a rought feedback and more is needed. Mixed activities are not great, they dont make good comparisons. We have to be quite rigid in the testing regime and a range of activties measured seperately AND the activity set needs to be as close a possible the same as the last test. I came up with this list of tests:
Even if you walk exactly the same walk, same route, same distance 10 times it will comes out different each time with the same device because your path will always be slighly different, unless you had painted footprints in the path and walked exactly the same footprints etc. Whats is already known: on Threshold 14 Bangle will over count on none stepping acctivties like driving, housework but be within 98% of a control device like a fitbit or Amazfit Bip. On Threshold 18 - I would expect the Bangle to under count by maybe 2000 steps over a 10K steps walk. I did the tests and have the result somewhere. When I set the threhold at 14 it was because over 10K steps of actual walking I got the most accurate results. I dont want to go on a 10 mile walk and the step counter tell me I have only walked 14000 steps instead of the 20,0000+ it should be saying- as its a pretty worthless step counter in that situation. On the days I am not walking I am less bothered about accuracy. If I know I have been in the house all day apart from taking the dog round the block, done a bit of housework - I dont really care if the counter reads 1000 steps or 1500 steps - I know I simply did not get enough exercise that day and I'm not going to fool myself just because a step counter says I did 200-300 steps more than I actually did. Posted at 2022-01-21 by johan_m_o
Yes, hence "more testing needed though". So far I've not made any attempts at any kind of controlled tests... Currently don't have much time for anything else than the mixed activities rough testing. Will hopefully be able to provide more controlled results in a near future. Posted at 2022-01-21 by myownself
Actually, I think if we can get the timestamps of when the activity changed this could be very useful as an additional comparison.
Running would be another good one. I think you said somewhere in the thread that runners will be more interested in other things, but step detection is important for some of the stats runners are interested in. If we can get the step count right, we can even put in a rougher version of distance if GPS is turned off. I also liked (or hated) your train journey sample @hughb - I take the train far more often than I drive. I suspect that sample would look very, very different with a higher sampling rate. I'd be interested in more train journey samples if anyone happens to be commuting that way at the moment. Posted at 2022-01-21 by myownself I had an idea just before falling asleep last night, I hope I can still remember enough of it to give it a try (when I come to it, it might suddenly turn out to be obvious nonsense). Sadly I've got day job stuff to catch up on first. Posted at 2022-01-21 by HughB I thought I would explain how the current step counter works, where it is weak and where it could be improved. I refer to the diagram below to the various stages. TH: - the raw signal is taken scaled and a gating signal derived from the input. F : The raw signal is passed onto the LOW PASS FILTER with a cut off frequency of 3hz if I remember. This is to take out high frequencies that will not be walking. Steps take beween 300-1300ms. I did a spreadsheet of ranges that is published in this thread. However the sampling frequency is only 12.5Hz and this in itself results in a filter that could be improved and is unable to The filter has an amplification effect on any signal. A small amount of noise will come out of the filter amplified with peaks and troughs that could look like steps - this has been observed with 2 people who have watches that phantomly count steps. P : Peak detection. The undulating signal its checked for a Peak. The primary reason for this is that the input to the filter can not be guaranteed to be a blanced zero average signal. In many cases it is offset in the positive direction. This means that some low points will be above the zero line and will not cross a negative threshold value. Using peak detection avoids this issue. The peak detection is crude BUT it works. It was well tested with recordings and I sometimes walked around with a debug version of the step counter connected to the IDE running on my phone. It maybe does not look elegant or good code but it works in practice. I would not say this is the weakest link. The primary weakness is the implementation of the Threshold Gating signal - I will cover the reasons for saying after describing the G: and S: phases. G: The gate. The gate_open boolen value set by the TH code is used to decide do we send the detected PEAK onto the statemachine or send throw the PEAK away. The idea is that we are not just sending amplified noise to the state machine. It is also necessary because I swapped to PEAK detection shortly after switching to the Oxford filter. The previous filter would result in a more balanced sinewave like signal that would cross the zero line. However the previous BAND-PASS filter also head a 'Ringing' issue that mean't when you stopped moving the signal would echo a further 5 or more cycles that would be counted as steps. Having swapped to PEAK detection it was important to ensure we were not just sending a peak in the noise signal onto the state machine. So returning to the TH Threshold detection.
The code assumes that the raw signal will not be offset and only checks that the raw signal is greater that +TH or less that -TH where TH is the threshold value. What is really needed is a way to detect the average and to offset the raw signal so that its undulations rise and fall around a zero line. Then what is being fed into the LOW PASS filter will be balanced and just be amplified. OR there needs to be two thresolds one for the position range and one for the negative range. EG 14-18 positive but -5 for negative. The gate needs to open and close quickly, we allow 3 samples in a row that are above or below the threshold to set the gate and for this to be reset after 3 samples below the threshold. If we extend the time take to decide if the signal is some movement or noise is too long then we could start miss counting more steps during that period. 3 samples is less than 1 second at a 12.5hz sample frequency.
One approach to improving this would be have a POS_THRESHOLD and NEG_THRESHOLD and to run tests against the test harness to establish the best pair of values for both. But we also need a lot more controlled samples from different people and different watches. Attachments: Posted at 2022-01-21 by myownself @hughb Thanks for the explanation, it confirms my understanding and adds a couple of details I missed. Your spreadsheet of step speeds is at http://forum.espruino.com/comments/16122158/ - I have just realised that the range I was looking for is slightly extended (allows for faster running), so when I get back to this I am going to pay more attention to the effect of that. My own running pace, which I didn't think was that fast, falls outside of your highlighted range. Posted at 2022-01-21 by HughB Its a bit of a compromise with running and a question of how far you go. If you go less that 300ms for a step then its possble that a rythmic vibration will look like you are stepping. For running I think we should rely on the RUN app / GPS for accuracy. I am about to checkin a fix for the RUN app that corrects the distance calculation from the GPS. I think if we improve the more common 'walking' and non walking cases first we can then explore the limits (eg moderate to faster running) and see how it performs. Posted at 2022-01-22 by HughB I'd really like to find time to properly write this up and but I've already spent a huge amount of time on it already and kind of need to pull back a bit. Posted at 2022-01-22 by HughB Real world Test results v2.11.30 Step Implementation - INDICATIVE only. I have only two two tests so these are indicative. Method: I wore 3 watches on my left wrist. AmizFit BIP, Bangle 2 with 2.11.27 and Bangle 2 with 2.11.30. I used the RUN app with GPS and HEART turned off but it makes it much easier to count steps for a test. AMIZ: 5438 10k (scalled up to 10K = 1.84 factor) I have scalled these to 10k to compare what the step count might be at 10K steps. I have used the scaling factor of the AMIZ - this is just for comparison purposes. Really I would always like to test a 10K step stretch but it is not always possible. This shows that 2.11.27 is 100.37% of the AMIZ which is a great result. HOUSEWORK - I did a test doing 1h44 housework BUT the 2.11.27 watch got poked somehow and rebooted so I lost the result as the steps got reset. This is the hazard you get when you try and wear 3 watches on the same wrist :) AMIZ 524 More tests to do. Posted at 2022-01-22 by johan_m_o
After my own very unscientific testing I had the same thought. Wanted to do some more testing before making any conclusions though... Nice to see that your small test showed something similar. Posted at 2022-01-24 by @gfwilliams Thanks for looking into this further...
All I can really go by is the test data, and with that data, 18 clearly came out on top in a number of scenarios. I'll change it back to 17 though and you can see. My big concern here is the reason for the change wasn't because I thought I could do better than @hughb's testing - it was that others users have had issues with the step counter counting phantom steps when their accelerometer had a slight (totally within spec) DC offset. The DC offset could be significantly higher than it was in @Mr_Ploppy's case and still be within spec, so I wanted to be on the high side to make it far less likely anyone encountered the phantom steps. As noted above though the step count really is just a proxy to see how much exercise you've done over the day. My personal feeling is if you're using the step count to see how much activity you've had during the day (which I think most people are) then for many people, walking makes up only a small proportion of the day and it's much more important that the other 80% of the day is more accurate. Posted at 2022-01-24 by jeffmer @hughb Your work with Gordon on the Pedometer is very much appreciated. I thought you might be interested in the results I have been getting using it with a different accelerometer the sc7a20 which is found in a lot of watches such as the P8, ROCK, Magic3 and SN80. The sc7a20 is a very cheap device which sells for pennies and many examples have readings that are well off centre. As a result, I have had to introduce calibration which Gordon has noted is not that simple. You have to gently move the watch into six different positions to get full calibration data - face- up/down, edge - up/down and end - up/down. However, I have only needed to calibate once. I have been comparing the step count during dog walks with that of my iPhone. The results for walking are quite impressive - here are two recent walks which are typical of the results I have been getting: iPhone: 5121 SN80: 5124 I have measured this walk at 2.6 miles with the Bangle GPS recorder. Personally, I am interested in measuring steps as a proxy for distance during walks - enabling step counting at the start of a walk and disabling at the end. I am very impressed with the bangle algorithm. Posted at 2022-01-24 by @gfwilliams I just had a thought about this... Right now, we use the absolute magnitude of the acceleration value. If you were to gently rock the Bangle around its axis, that would not give you any change in the value at all. What if we use the difference in accelerometer values? That is by its very definition going to be centered, and any rotational movement would add to the value you saw. Obviously it needs work and testing, but could be an interesting way forward anyway. Posted at 2022-01-24 by myownself Yes, some of my experiments used the difference and it showed promise. Posted at 2022-01-24 by jeffmer The problem is that if the readings for an axis are off centre you do get a change in the magnitude as you rotate with respect to gravity. So rotating your wrist will give a false acceleration that may look like swinging your arm while walking. It is why the continuous steps were detected by some people in only some orientations. The current algorithm is trying to detect accelerations that are independent with respect to your orientation to the downward force of gravity which is why the iPhone in my pocket agrees with the watch on my wrist. Posted at 2022-01-24 by HughB
Just to be clear are you saying you are using the B2 algorithm on a SN80 and getting these results ? If so that is really encouraging.
Its an ok approximation method providing you work out the stride distance or have a setting for stride distance. You will know it will be different for each person depending on height. I know for me that 2000 steps (on a pavement) is 1 mile. But walking on tracks, grass, mud etc etc, stride length changes a lot. I have not tested how accurate the Recorder calculation of distance is using the GPS trace. It may suffer from the bug I just fixed in the RUN app where the distance between two lat/lon's was not an accurate calculation. The 0.03 of RUN app should be good now. I would definitely recommend you take a look at the RUN App. You can use it with GPS on / off. One option would be to add a stride length and when the GPS is off to estimate PACE and distance based on steps*stride length. Would save a lot of battery. Posted at 2022-01-24 by HughB
@gfwilliams - totally get it. We want these users to have a decent step counter if their accelerometer is in spec. The test harness showed that 15 would have been enough - so jumping to 18 felt a bit too quick. I think 16 would have been as high as I would have gone at the moment. But 17 will probably good too. I'd like to understand the DC offset a bit better. I have a suspcicion that the DC offset may always be positive. In which case maybe thresholds of +17 and -10 might be better. There's more tests that could be done using the test harness to establish that. Plus we need more acceleromert recordings and samples from the phatom step users. We dont just want a step counter that's fine tuned for @hughb's Bangle 2. We need some stats on the variance of these accelerometers and the batches we have. I dont know the hardware stuff well enough to comment. But there were 2 cases complaining about phantom steps out of 2000 watches sent out ? Were they freak accelerometers or are there a lot more that never looked at step counts or didn't report any issues because they were more interested in clocks or GPS or games etc. I'm heartenned by the fact that no one complained about the step counter for so long. I've been expecting a lot more people to complain saying they drove 1 hour to work and count so many hundred steps etc. I'd like to see if we could collectively come up with a better approach for the THRESHOLD / GATE concept that can work out this DC offset and compensate accordingly. Either some sort of dynamic approach or a calibration approach. There are some really smart and experienced people in this forum who can knock things together in an hour that would take me 2 evenings, whilst I'm still playing catch up as I don't really have prior experience with Javascript. Posted at 2022-01-24 by johan_m_o Today's testing on 2v11.31, threshold 17. Took a walk to the store and back, 1.4 km one way. Going there: Walking around the store (about 15 minutes): Walking home: 5 hours spending time at home, playing with the little one, doing housework, etc: Looking pretty good. Sounds like there are some good ideas being thrown around so looking forward to see how things will progress. Posted at 2022-01-24 by jeffmer Yes, it’s the B2 algorithm on a SN80. I programmed the first version of plot track for B1 that Gordon then added to the gps recorder app and it used exactly the equirectangular approximation you have suggested for the Run app. I have just checked and the current version still uses it so the distance should be accurate as I did check its accuracy at the time. Posted at 2022-01-24 by HughB Hi @johan_m_o - good tests. The first one concerns me. If you scale the FitBit result up to 10000 steps - you get a scale factor of 6.72 . Multiply the 1388*6.72 gives 9334. Which indicates 93.34% of the Fitbit result. This to me isn't accurate enough, 98% was my target, over 10K steps. I used a FitBit Charge HR for several years and know they are really accurate, so I trust them for comparisons. Posted at 2022-01-25 by myownself @hughb is the difference between the positive and negative noise just gravity? I am not at the computer but I can't remember where in the current algorithm it is accounted for. Posted at 2022-01-25 by jeffmer I had a go with using a DC filter to remove the bias and here are the results. First. here is the simulated performance of the current Pedometer with
Now here are the results with using a DC Filter and reducing the threshold to 15:
NSAMPLE is a parameter of the filter:
@hughb It appears that the accuracy gets worse on your shorter walks and better on your longer walk with this approach. Posted at 2022-01-25 by HughB Interesting. Whats the results at 14 and 10. 14 was the original baseline. Note we need more logs from @Mr_Ploppy as 60s of holding the Bangle is not enough. I'd really like 15mins driving and 1 mile walking. Can NSAMPLES be 6? What is the impact of that? Posted at 2022-01-25 by jeffmer At threshold 14, the results are still pretty good:
10 still solves Mr Ploppy but I think it's too sensitive.
With NSAMPLE=6 you lose too much low frequency I think and step counting suffers:
So 15 with NSAMPLE=12 worked best, however, I agree more data would be desirable. Posted at 2022-01-26 by Mr_Ploppy Hi there is some extra log files. The 2 train ones should come to very low step counts. The 2 walking ones my pebble had at 600 and 500 steps. Attachments: Posted at 2022-01-26 by jeffmer Thanks, here are the results for those files. First the current algorithm with threshold 17:
and next the algorithm with the DCFilter and threshold 15:
It's marginal but in my view, the DCFilter helps here. Posted at 2022-01-26 by HughB @jeffmer - could you attach the Espruino/libs/misc/stepcount.c file. I'd like to have a look at RAW vs DCfilter output on a graph. Also just easier to see the full context by looking at the whole code etc. Posted at 2022-01-26 by HughB Here's my baseline result from back in september on Firmware v2.11.27
I'd be interested to see this set against RAW_THRESHOLD = 12 with the DC Filter. Posted at 2022-01-27 by jeffmer I have forked a copy of the test harness repository and added my copy of Posted at 2022-01-27 by @halemmerich Today I have tried wearing both my bangles (2v11) on the same arm to compare the step counters. The results were as follows: b77 seems reasonable, albeit probably some percentage points on the low side. Surprised by the massive difference I have recorded some accelerometer data and run it through the test program:
I have walked a counted 150 steps for the first files. The plots while walking seem ok, but the stationary b87 has a lot of "movement" for being very still and having both watches on the same arm. I don't know if it is related, but the internal HRM also seems to be a lot better on b77 than on b87. I have not recorded data for this, but during testing the difference was often > 20bpm. Maybe some hardware issue interfering with both sensors (missing filter cap, bad solder joint somewhere)? Can I collect some other or additional data? Attachments: Posted at 2022-01-27 by HughB whats a b77 and a b87 and whats the difference between these. Posted at 2022-01-28 by @halemmerich Two Bangle.js 2, 77 and 87 are part of the MAC. If I watch the b77 while walking, close to every step gets counted. Posted at 2022-01-28 by @gfwilliams Are we saying that b77 is the new firmware and b87 is 2v11? Posted at 2022-01-28 by jeffmer It looks like they are both using the original threshold of 14 which is why the simulation with 17 is very different. @halemmerich I would be really interested in seeing these with the DCFilter - b77 has a strong positive bias. Happy to run the test harness if you can provide the files or you run it yourself using my version of the harness from here. Posted at 2022-01-28 by @halemmerich Two bangles, both on 2v11, so I assume old threshold. b77 was a lot better than b87, especially while both were stationary. b87 was extremely noisy between 80-180ms, 200-280ms, 450-510ms. I do not know why. Posted at 2022-01-28 by jeffmer Yes, master branch and there is a local copy of Posted at 2022-01-28 by HughB @gfwilliams - its gets a bit confusing when people say 2v11, I never know if they mean what is 2.11.0 or 2.11.x a later build. Maybe the minor release should be 2.M.0 ? Posted at 2022-01-28 by @halemmerich I have had both bangles on 2v11.53 for a few hours now (mostly sitting at desk) and the step counters are still different. jeffmers test harness and stepcount.c yielded:
I have had play with the RAW_THRESHOLD and got down to 7 before the results were reasonable.
I also tried walking 150 steps. b77 counted 80 steps and b87 counted 103. Posted at 2022-01-28 by HughB I really dont recommend using very small step samples. This is the mistake the oxford step counter team made. I would recommend walking at least 1 mile. would it be possible to show the result of HughB-walk-a10021-b10248.csv so there is a known quantity to compare against. Posted at 2022-01-28 by @halemmerich
So that creates too many step counts with this low threshold. Posted at 2022-01-28 by HughB @jeffmer - like your dc filter a lot. This is one of my samples, about 10 seconds in once I am up and walking. I can clearly see any bias being corrected. Very nice. I will run some tests to see where RAW_THRESHOLD could be set. Attachments: Posted at 2022-01-28 by HughB
This is what we are trying to understand. How much variation there is between accelerometers in one device compared to the next. However the point of @jeffmer's dc filter is to correct those differences to produce a balanced signal. Posted at 2022-01-28 by jeffmer That’s a very nice demonstration of it in action. I have now dispensed with calibration as the results I get using the filter are exactly the same as I got with calibrating the accelerometer. I am using a threshold of 15 which gives accurate walking while reducing the count from other activities. Posted at 2022-01-28 by HughB There's not much in it either way between 14 or 15.
I reckon you should push the DC filter into the main branch with RAW_THRESHOLD=15. Hats off to you @jeffmer with that little bit of code. Very elegant and simple. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2022-01-28 by HughB
You need the latest Espruino, and replace Espruino/libs/misc/stepcount.c with Posted at 2022-01-29 by jeffmer I have submitted a PR to add the DC Filter and set threshold 15 as you suggest. Posted at 2022-01-30 by @halemmerich After a longer walk today, the counters on both my bangles are nearly perfectly in sync, 6165 (b77) vs. 6154 (b87). Running jeffmers harness with threshold 15 results in 6145 (b77) and 6159 (b87). Posted at 2022-01-31 by @gfwilliams Just merged - thanks! lets hope that helps out :) Posted at 2022-01-31 by HughB Tried the new Firmware Updater App - seemed to work a treat. I'm now on 2.11.70 so will see if I notice any issues. Posted at 2022-02-01 by johan_m_o I'm trying to do some proper testing (with accel logs) today. So far it looks like 2v11.70 with the DC filter and threshold 15 is quite accurate for walking (counted 1000 steps during a short walk and the Bangle registered 971, while the Fitbit registered 980), but overcounts the steps when just being around the house (haven't saved any logs for that yet, too busy getting the little one ready for school, but during a little over an hour of just getting ready for the day the Fitbit registered 608 steps while the Bangle did 872). Will update with results (and logs) later. Posted at 2022-02-01 by Chasolla This morning, for the first time, I have done no steps overnight instead of the 3000 I'm normally told I've done. I've done my morning walk and the Bangle says 8115 steps and my phone says 8306. No phantom steps while typing or because I'm holding my wrist wrong. This is a brilliant result and I'm hugely grateful to everybody that has contributed to this. Thanks again. I'm a happy Chas Posted at 2022-02-01 by HughB Brilliant, could you confirm that this is on 2.11.70. Posted at 2022-02-01 by HughB Hi @johan_m_o, some measured accelerometer logs against a fit bit would be great. If you could do ones a 1,2,3 miles that would be great. Also good to record what the Bangle counted by noting start and end counts. If you attach the logs here I will check them in. Ref over counting, that is the key challenge. Its a very fine balance between stopping counts during driving, typing and washing up and accurately starting to count when walking. All step counters over count on non step activities to some extent, even Fitbit and Amizfit count steps while sleeping, some more than others, it's a question of how much. The latest change, hats off to @jeffmer has resolved the bias issue between different accelerometers. There will continue be more work to do on the step counter until someone comes up with a new approach that smashes our current results. Posted at 2022-02-01 by Chasolla I was on 2.11.67 which was the bleeding edge when I went to bed last night. I've upgraded to 2.11.70 this morning. Charlie Posted at 2022-02-01 by johan_m_o
Yeah, I'm gonna walk down to the store in moment (about 1.4 km one way). Meanwhile I've been sitting at the computer (not much typing, mostly research) for just over 2 hours (about 125 minutes) and the Fitbit registered 5 steps while the Bangle registered 46. I did actually have the accelerometer logger running (wasn't expecting to sit that long and then I forgot about the logging being active), but the file is too big for me to pull of the watch... The Web IDE just times out while trying to save it. Edit: change of plans... No more time for a walk. Something happened to my watch when trying to pull that large file from it through the Web IDE. All of a sudden I was experiencing severe lag and the watch would lock up. After a full reset it seems better (no lockups), but the "A Configurable Analog Clock" still only updates the seconds hand every 2-3 seconds. I'm pretty sure it properly updated every second before... Although, it's not affecting Anton Clock, so it might be an ACAC thing. Posted at 2022-02-01 by johan_m_o Since my plans for more testing got somewhat stumped and cut short for the day (see above), I'll just attach the accelerator log from my 1000 counted steps walk (so about 1 km), in case anyone's interested. The Bangle registered 971 of those 1000 steps. Attachments: Posted at 2022-02-01 by HughB Nice. 97% not bad. Posted at 2022-02-01 by Alessandro Hello, This is my experience with latest firmware:
I just installed the logger, next time I'll try to record the walk ps: English is not my mother tongue, sorry for mistakes Posted at 2022-02-01 by HughB Thanks for that. We really need measurements against another device like a 'fibit' or 'amazfit bip'. Watching a step counter while walking and manually counting steps is very hard. It also skews the accelerometer readings because the watch is not always in the usual walking orientation with respect to your body etc, arms slightly swinging by your side etc. Posted at 2022-02-01 by johan_m_o The 97% while walking was really good, but the almost 10 time difference between the Fitbit and the Bangle when sitting still at the computer wasn't as good... Although way too small a result to say much. I'll try some other situations in the coming days. Posted at 2022-02-01 by myownself I think recording the steps (not watching the counter) and counting can be good. We don't know which way other devices are skewed, so if you can walk normally and count then the data is useful. I find I can count accurately for 1000 steps, and then I get bored and my mind drifts, but for those 1000 steps I am pretty certain of my counting, and I think only for the first few steps I am paying more attention to the steps than I usually would (and possibly walking unnaturally). Posted at 2022-02-01 by johan_m_o Yeah, I just walk normally and count to 10 over and over and use the fingers on my right hand to count the 10s up to 100 and then I use my left hand to keep track of the 100s. That way I can very accurately count much higher than 1000 without much effort. Posted at 2022-02-01 by myownself Hah, my technique is similar but I only use a hand for the hundreds, not the tens. Posted at 2022-02-01 by johan_m_o I find it a lot easier to keep up with steps when only counting to ten... Posted at 2022-02-02 by HughB Its a known issue and there is plenty of test data already to show the over counting when not stepping. Its not going to get any better without a different approach. Posted at 2022-02-02 by johan_m_o Was threshold 15 really the best approach then, and not a slightly higher one? Posted at 2022-02-02 by Alessandro
I have a Mi Band 5, I can make a comparison with that. Posted at 2022-02-02 by HughB The threshold is finely balanced. Its purpose is to say, if we dont have activity above this threshold then dont look for steps. Setting too low you overcount, too high and you undercount. Setting above 15 reduces 10K step accuracy down to 95% or less which for me is too low. For a step counter to be consider good it should be 98% - I have that in a paper somewhere. 95% accuracy looks like 9500 steps counted when the fitbit would count 10,000. To me thats too inaccurate.
The new DC filter means we have to gather more data. What the DC filter does is correctly compensate for positive bias in the signal (ie it would look like the average eas not 0 but say 5). Now that we are compensating for bias then we need to gather more data. My gut feel for it (and supported by test harness runs) is that somewhere between 14 and 15 is the right value. BUT it is really hard to know what is going on out there. Not everyone with a B2 reads or posts to this forum and some may just not be interested in steps at all. We really need some sort of polling system in the forum 'is the step counter good enough ? ' 0 for poor, 10 for excellent. Posted at 2022-02-02 by Alessandro
Is there a way to change the threshold from the Bangle? Or a way to temporarily disable the steps counter? E.g. before lunch I swept the garden sidewalk and the Bangle (but Mi Band also) counted over 100 steps but I was "standing" in place. Posted at 2022-02-02 by HughB No. Counting steps when making a rythmic forwards and backwards motion is a standard way to fool all step counters. I can make any step counter count steps by holding the watch in my hand and flicking it backwards and forwards about once per second. The amount of gravitation force when moving , arms swinging and head bobbing up and down will be very similar to a sweeping or mowing motion. This is why for really accurate distance measurements the GPS is used in most smart watches for sporting activities etc. Posted at 2022-02-02 by Alessandro
I know it, which is why sometimes I would like to (temporarily) stop the step counter. I'd like press a "pause counter" button, swept the garden sidewalk and then press a "unpause counter" button :-D Is it technically possible to develop a js app that does this? Today I wore both a Bangle2 and a Mi Band 5 on the same arm. Before the treadmill: Bangle2 1683 vs Mi Band 1046 steps. After 2 Km on the treadmill: 3557 vs 3480 so the Bangle counted 1874 vs 2434 steps. (I read that small and short activities are ignored by Mi Band by design, e.g. I walk from study to kitchen, pause, and then from kitchen to study: Bangle counted 10+11 steps vs 0 by Mi Band). I attached the log of the treadmill walk, hope it helps! Attachments: Posted at 2022-02-03 by Mi We know extraction of steps from acc is difficult and best way to improve is test against a set of known results - as you did with stepcount.c So what about the following idea (new or already discussed?): strap another bangle1/2 or puck at ankle. So far I have recorded two such pairs. Both times bangle2 (b2) at wrist and bangle1 (b1) over ankle. Additionaly I had a Mi3 band for comparison. Attachments:
Posted at 2022-02-03 by Mi By the way: doing that experiment I found
Posted at 2022-02-03 by HughB This is great. Do you have the step counts recorded by the Bangle as well? This helps sanity check differences with the harness and actual steps counted on the Bangle. Posted at 2022-02-03 by HughB Have you uploaded 4 separate logs, looks like 2 logs. Posted at 2022-02-03 by Mi
yes, should be 4 different logs: 2 walks with two devices each.
No unfortunatly not. Will do this next time. X_STEPS = 6, RAW_THRESHOLD = 16 Just a quick manual test. Havent understood the auto search yet. You see that bangle1 (ankle) is closer to Mi3 than bangle2 (wrist) - but its not as clear and stable as I would have expected Posted at 2022-02-03 by HughB
I give any device I am going to compare against a test on a circuit that I know is 1.03m. Might be worth doing a 1 mile walk with the Run app v0.05 and checking how many steps are counted and distance is reported by the Mi5 band to see how accurate it is. The idea of using another bangel running 'in development' software does not sound great as both measurements will be subject to version control and known issues. I think we either have to manually count the steps or use a known quantity (another sports watch that we have checked).
Hmm, I think the only truth would be that it was a bangle nearer the ground. Thats does not mean it will reliably count the steps any better. I would kind of expect the results of the B1 at ankle level to be better as the accelerometer swing will be much greater. However since the watch is going to be worn on the wrist tests have to be done with the measurement watch on the same wrist and accelerometer logs need to be done on the wrist as well otherwise they introduce another variable into the tests. Posted at 2022-02-03 by HughB
Yes , you could build a clock where pressing BTN1 would start and stop counting steps if you wanted to. The Run App works like this already. You could copy that as a starting point. Or you could count steps using the Bangle.on('step',x) event BUT only count steps when you had not put your app into pause mode. Posted at 2022-02-03 by HughB
Each watch does this slightly differently. Fitbit HR requires 5 steps before it will count, AmizFit Bip requires 8, Bangle requires 6. Mi may require a lot more, you can establish how many through multiple experiments. If anyone is watching when you do this you look quite odd. I once did this in a farmers field :) How many actual steps is it from your Study to Kitchen ? Posted at 2022-02-03 by Mi
No, that is the idea of a) with the thesis that ankle is easier than wrist -which of cours is to bee seen and needs to be testet. I hope to see a much more robust outcome when varying parameters. b) contains a bit of real ground truth! We have two acc input data streams a,b for which we know for sure that steps(a)-steps(b) =! 0. So even if we do not know the number of steps, we know that the bigger that difference is, the worse the algorithm "steps()" performs. No approximation here. Posted at 2022-02-04 by Alessandro Good to know, thank you! I'm putting this on my to-do list ;-) Posted at 2022-02-04 by Alessandro
I counted 12 steps so the Bangle is quite correct.
Haha! :-D Posted at 2022-02-04 by HughB
Sure -if you want to walk around with a bangle watch strapped to your ankle. The watch is for wearing on the wrist not the ankle. Most people are not going to strap it to their leg etc. The accelerometer traces and thresholds will be completely different for a step counter designed work off a foot to one worn on a wrist. Its completely different approach. We are trying to optimise for the step counter to work when the watch is being worn - just like any other fitness band. Adding accelerometer logs recorded at ground level is introducing an uncalibrated meaure thats going to confuse matters.
We are looking for 95-98% of a good step counter. 20% is far too inaccurate. The fundemental problem isn't counting steps accurately when walking - its how NOT to count steps when NOT walking but still count them accurately when walking. This is where the bulk of our thinking needs to go. Posted at 2022-02-04 by Mi
Yes of course. That interval was meant to exclude degenerate cases. Just means something like "we can assume Fitbit/Mi not worse than 20%".
Yes, I am very aware of that. I seem not to be able to express myself clearly - I am very sorry for that. Attachments: Posted at 2022-02-05 by HughB
Definitely heart rate needs some help right now. Steps are probably good enough for now. 12 months ago steps were in the same position as HRM is in right now. Posted at 2022-02-07 by radswid Hey there, the signal strength of the bluetooth connection between Bangle and phone is measurable, right? What came to my mind yesterday was, when I'm walking, there is a certain "distance pattern" between watch and phone most of the time. If a parameter like "only count steps if bluetooth signal strength shows pattern x" is added, could this prevent the step counting when not walking? Posted at 2022-02-07 by myownself @Radswid it could work, but as you say it only works if the phone is present and connected. If you've always got a phone in your pocket, you could argue that you might as well just use that for step counting (in all but the biggest pockets fewer degrees of freedom and they pretty much all have an accelerometer). Personally my Bangle and phone are not connected unless I am uploading an app, and one of my reasons for getting a watch was to not need my phone to track run distance. Posted at 2022-09-10 by KTibow My Bangle's step counter seems generally inaccurate, how could I help contribute data to the current algorithm? Posted at 2022-09-12 by @gfwilliams Is your firmware up to date? There's a test harness at https://github.com/gfwilliams/step-count and I believe the README links to a forum post explaining how to get the data. However on the whole the current system seems to work pretty well - I think yours is the first complaint since the new code went in a year or so ago - so I'd be a bit concerned that making big changes would make things worse for everyone else Posted at 2022-09-13 by KTibow It's the latest version. Of course it's hard to know what your feet are doing when you're doing complex activities with your hand, but sometimes I was able to get it to register like 5 steps by doing a... "reverse punch"? I put my hand so it faced upwards and brought it down quickly. Anyway I'll take a look at if I can gather any more data. Posted at 2022-09-13 by HughB
One reverse punch or multiple reverse punches? Posted at 2022-09-13 by KTibow They were vertical, and the watch (Bangle.JS 1, not 2) was facing me with the buttons facing away from my hand. Doing that motion usually registered at least 2. Posted at 2022-09-14 by @gfwilliams Honestly, you might expect something a bit like that. It's not magic, so it has to guess - and especially if you're trying to find ways to fool it you will. But the reality is that in normal use it'll be well within 10% of your actual step count, and for most people that's more than good enough as it's just to give you an idea of the amount of exercise you are getting Posted at 2022-09-14 by KTibow Understandable. Anyway how would I contribute training data exactly? Posted at 2022-09-15 by HughB If you hold any smart watch in your hand and rock it backwards and forwards once per second you will fool it into counting steps. You can try this with a fitbit or AmazFit bip etc. The step counter is not trained as such. It uses combination of filters, thresholds and finite state machine. This entire thread will tell you how to make calibrated accelerometer recordings that can then be used. There are also posts that explain how the algorithm works. Posted at 2022-09-15 by KTibow I already saw that you can use an app to record a CSV file. My question is where do these CSV files get fed into? And is there a place where I can make a PR or something to update the algorithm? Posted at 2022-09-17 by HughB I have written a detailed README file at: https://github.com/hughbarney/bangle_bits/blob/master/step-count-docs/README.md Posted at 2022-09-17 by KTibow Thanks. I'll see how I can contribute. Posted at 2022-09-21 by @gfwilliams I did mention above in http://forum.espruino.com/comments/16681689/ but https://github.com/gfwilliams/step-count links to a thread post that I think shows you all you need to do. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2021-02-03 by @gfwilliams
Hi! I'm looking into improving the accuracy of Bangle.js's pedometer by adding this project
However to calibrate it, I need data!
If you're willing to help, please could you install this app:
https://banglejs.com/apps/#accellog
Then run it (
Accel Log
), chooseStart
, and then walk exactly 150 paces in a variety of ways - along a road, bumpy path, up/down stairs, fast, slow, pausing, and so on - then press BTN2 to stop recording when you're done.You can then:
My Apps
look forAcceleration Logger
Hopefully when I get a few examples of different walking styles I can feed it into their tools and it'll calibrate itself!
Beta Was this translation helpful? Give feedback.
All reactions