Unexpected movement data in health DB (hourly vs. daily)

I'm trying to wrap my head around how the movement data tracking works in the health app. The figures in the hourly data make sense, generally ranging from, say, 50 to 800 depending on the amount of movement, but the daily averages for the same metric don't. I'm getting values for each day like 6, 8, 8, 8, 8, 8, etc—definitely not an arithmetical average. That seems fishy to me.

So I'm digging deep into the Health app code and trying to understand how the data is being calculated and written to the underlying DB. The first thing I notice is this part where it writes the daily summaries: I think it's reading byte 2 of the record for movement data, but that seems to be the same byte that it's reading for heart rate data? Other places in the code use byte 3 for movement instead (and 2 for HR), so I wonder if I missed something or if that's actually a bug.

So I tried modifying this code to what I'm guessing it should be, but it doesn't seem to change the result much. Yesterday's movement summary was still just 6. Looking a bit deeper, though, I see the definition of getRecordData(), which bundles the data up into a DB record for writing, and it divides the movement data by 8 before writing. But that gets called twice. I might be reading the code wrong, but it seems like it's doing that when writing normal records (every ten minutes?), but then it gets called again when doing the summary… So I'm wondering if maybe it's accidentally doing the division twice, once for the initial data and then again when averaged for the day (meaning the dailies are now divided by 64 while the hourly is only divided by 8). Which isn't a difference I would expect to see in the user-facing interface when comparing hourly vs. daily data.

If anyone could shed some light on this or confirm/debunk what I'm thinking, that would be helpful.

So just as an experiment, I applied a quick hack to the health app on my watch to avoid the double-division to see what would happen. Toward the end of the day I got a daily summary “movement” value in the Health app of about 49. In the debug console I grabbed the log of hourly health data for that day and manually averaged it to compare:

movCnt = 0; movTot = 0;
require("health").readDay(new Date('2023-07-27'), (data) => {
  movCnt += 1; movTot += data.movement;
})
movTot / movCnt

and finally got a value that agreed with the written summary for that day, 49.41666666666. This is after making both code changes that I mentioned above, having the Health summary code read the correct byte of the record, and also not dividing the already-divided result when calculating and writing the summary.

I don't want to be too quick to assume things since I'm new here and don't know about all of the design decisions that might have been made, but so far this does seem to confirm my above suspicions. If it is a bug, I'm up to seeing if I can try to put together a pull request.

Thanks - it looks like you're right - the daily summary code should be using index 3 for getting the movement, and it's dividing too much as well.

To date nobody has really done that much with the movement data, so it looks like nobody really noticed. If you could do a PR to fix this it'd be awesome!

Oof… I just posted the pull request, but I just realized something else that bothers me. In the Health UI, movement data per hour frequently measures above 255, even though the DB field for it is only uint8, so I think it must be adding together all of the values per hour. My PR generates the daily summary by averaging everything instead, so it still looks a bit low compared to the by-hour graph.

The daily summaries alone should at least still be useful for comparing with themselves, but they wouldn't be appropriate for comparing with the hourly values shown on the graph. I guess in that case one way to make them consistent would be to add up the values per hour, then average the hourly values? That's a bit more complicated. I'm not really sure what the right thing to do here is, given that it doesn't seem well defined what exactly the “movement” metric and its summary means, apart from being a somewhat arbitrary reference.

I think I answered this one on your GitHub PR, so let's discuss on there: https://github.com/espruino/BangleApps/p­ull/2942#issuecomment-1663568321