You are reading a single comment by @DrAzzy and its replies. Click here to read the full conversation.
  • I'm gonna try and fix this tonight.

    I think it can be fixed with judicious use of parenthesis

    Edit: Okay, yeah, I see the problem. It looks like that was written for an unsigned 32-bit integer (the code was ported from arduino), and still has workarounds for handling the large numbers.

    var p = Math.round(B7 < 0x80000000 ? (B7 << 1) / B4 : (B7 / B4) << 1);
    

    In my tests at ambient pressure, B7 is 1.6 billion or so - so when it left-shifts it, it turns into a negative number.

    In at least the current ambient conditions, changing that to

    var p=Math.round((B7/B4) << 1);
    

    seems to work, but I don't think it will work at all pressures - I fear that B7 can get large enough that it could roll over to a negative number on the line before that one (since they test whether it's greater than 0x80000000, which is the point at which it would have already rolled over)

About

Avatar for DrAzzy @DrAzzy started