• Tue 2019.08.27

    Anyone with a running 'C' compiler and debugger able to cross check these values please? Or any ideas to provide other helpful means that will assist in pin pointing what is going on?


    It is puzzling that Espruino and the Chrome browser produce near identical numerals to the right of the D.P. when hand calculated, or using the output value from the @valderman Feb 2012 snippet.

    Value 1234.505

    Espruino - Single precision 32 bit: 0.50499999523
    Chrome - Single precision 32 bit:0.5049999952316284

    The online calculator produces a slightly greater value:

    Single precision 32 bit: 1234.505004882812
    Double precision 64 bit:1234.50500000000010913936421275138854980­46875

    I originally thought this might be that double precision was more accurate, as there are an additional 32 bits that when summed, would cause a higher value. But it can be seen that is not the case using the online calculator. surprisingly, the value is smaller


Avatar for Robin @Robin started