• it wasn't in the underlying mechanism that creates the floating point equivalent

    Correct, yes.

    Was it a number typo (then) or was there a particular reason that value was chosen?

    I believe at the time it was a badly thought out attempt to ensure that digits further down the number didn't cause rounding - for example 0.5549 -> 0.56. Of course because it works in at least 90% of cases it wasn't immediately apparent that there was an issue.

    how that online calculator provides a slightly different higher mantissa value that does either the near matching Browser or Espruino tests.

    https://www.exploringbinary.com/floatingĀ­-point-converter/ says:

    It is implemented with arbitrary-precision arithmetic, so its conversions are correctly rounded

    I'm not entirely sure on what it means there but on Espruino the stringToFloatWithRadix function uses repeated multiply and add with floats to convert the number, and multiply/divide by 10 is a particularly bad thing to do for accuracy.

    The converter could quite happily store the integer and fractional parts as two arbitrarily long integers and then combine them at the end, and that would probably provide a more accurate number in a lot of cases.

    To confirm, to debug this would require the steps Wilberforce and AkosLukacs #6 #7 #8 #9 point out?

    It's worth playing with WSL anyway - it's really simple to compile Espruino in it, and you could then change the Espruino source to output the binary values of the number at each stage of the arithmetic.


Avatar for Gordon @Gordon started