-
• #3
Ok, so summarised the
correct.jsandincorrect.jsoutput the same in browser. Tested in node, they do output the same:> node .\correct.js L145 msband=E40000 L146 midand=C000 L147 lsband=0 > node .\incorrect.js L145 msband=E40000 L146 midand=C000 L147 lsband=0, but different in Espruino, right?
My guess is rounding is somewhat messed up, the interesting part:
>var val = 915E6; =915000000 >var newVal = val * Math.pow(2, 19); =479723519999999.9375 >var newDiv = newVal / 32000000 =14991360 // but doesn't equal to the literal value: >newDiv==(14991360) =false // it's smaller: >newDiv<(14991360) =true >newDiv>(14991360) =falseSo it's not exactly 14991360, but a bit smaller. In your
correctversion you set it to the exact literal value, that's why it behaves differently.// Espruino: >newDiv & 0xff00 =48896 // with literal value >14991360 & 0xff00 =49152If you round it, you get what you expect:
>Math.round(newDiv) == 14991360 =true >Math.round(newDiv) & 0xff00 =49152 // in hex: >(newDiv & 0xff00).toString(16) ="bf00" >(Math.round(newDiv) & 0xff00).toString(16) ="c000"That doesn't mean I have an explanation, probably a different behaviour with bitwise operations on fractions in Espruino.
Maybe a bug? Because after explicit rounding it works the same as in node / browser. -
• #4
Yepp, it's rounding and bitwise operations, simplest repro is
🥁🥁🥁
🥁🥁🥁:
0.99999999999999995 & 1Results:
// Firefox and Chromium: > 0.99999999999999994&1 0 > 0.99999999999999995&1 1 > 0.99999999999999996&1 1 // Espruino: >0.99999999999999994&1 =0 >0.99999999999999995&1 =0 // It's ok just a bit higher: >0.99999999999999996&1 =1Edit: with @Robin's numbers:
// In browser: > var val = 915E6; > var newVal = val * Math.pow(2, 19); > var newDiv = newVal / 32000000 > {val, newVal, newDiv} {val: 915000000, newVal: 479723520000000, newDiv: 14991360} > (newDiv-14991360)*1e9 0 // Exactly 0 in browser// Espruino: ... >newDiv-14991360 =-0 >(newDiv-14991360)*1e9 =-1.86264514923 // not exactly 0! -
Thr 2019.07.25
Thank you @AkosLukacs for the speedy reply. Thanks for testing on different OS and browser. At least I can be assured I'm not going insane. ;-) (yet)
Always amazes me how each of us tackle problems. I'll take time to study the effort you put in. Still not solved though.
After I posted and gave it some thought, I wondered if a floating point issue might be at hand.
But if that be true, then Line L43 L44 in correct.js
var midsav = newDiv; var lsbsav = newDiv;would mean that the assignment of the same value is being corrupted. This would imply that the floating point value, of an identical value, is changing after the first assignment!!!
-
• #6
Nope no magic corruption:) Only different decimal precision
Look at code at line 57-L60:
/*57*/ var newDiv = 0xE4C000; /*58*/ var msband = newDiv & 0xFFFF0000; /*59*/ var midand = newDiv & 0x0000FF00; /*60*/ var lsband = newDiv & 0x000000FF;Going back from line 60-59-58:
Assign 3 variables using the value of a variable callednewDivand some binary operations. What's the value of newDiv at that moment in time? It's exactly0xE4C000, because you just assigned that value at L57.
Anything happened before is irrelevant.To demonstrate it (start with
correct.js):Add
var bla = newDivto L45
Addprint('bla == 14991360:', bla == 14991360)to L66It will print
bla == 14991360: falseso, theblavariable still holds that almost-14991360-but-not-exactly value.
Just typebla-14991360and you get-0. Or type(bla-14991360)*1e9, and you get the same-1.86264514923.
No corruption here, as far as I can tell, only different fraction handling. -
' It's exactly 0xE4C000, because you just assigned that value at L57'
Yes and now you have proved my point. We are masking bits with identical hex values. In either case the answer should be identical, as we are dealing with bits. Thats the reason for performing a mask operation, to detect which bit.
'as far as I can tell, only different fraction handling'
That case is using a floating point value for input. The example provided used the same integer equivalent of the hex notation. I left out the
parseInt()conversion as the browser tests didn't need it.We have the node.js test, the Firefox browser test and my Chrome browser test, that produce correct identical results in these three examples. In this instance on one device using a different mechanism, we get different conflicting results. We shouldn't.
I'm reading up at mozilla . . .
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Bitwise_Operators
"Bitwise operators treat their operands as a sequence of 32 bits (zeroes and ones), rather than as decimal, hexadecimal, or octal numbers."
"The operands of all bitwise operators are converted to signed 32-bit integers in two's complement format, except for zero-fill right shift which results in an unsigned 32-bit integer. "
As these are numbers, it shouldn't matter if it is decimal or in hex notation. The browser tests show this. (a thought - aren't integers converted to floating point under the hood? hmmmm)
I did notice that the mozilla site uses integer constants for their flags, rather than hex notation as I did. Could this be a difference between C/C++ and Javascript?
Testing that . . .
-
• #8
The point is -IMO - there is something wrong with floating point calculation. The bit-mask mismatch is a result of the floating point error.
The result of calculation
915E6 * Math.pow(2,19)in Espruino is not 479723520000000, it's 479723519999999.9375.To demonstrate it, ran this in
- Espruino 'Linux' build on my laptop
- DS-D6 (nRF52832, just sitting in a box, BLE console is really convenient :) )
- Browser console
- node v8.11.2
- F# interactive (.Net, to test on completely different platform)
Results are identical in non-Espruino environments:
> 915E6 * Math.pow(2,19) // The result in Espruino: 479723519999999.9375 // The result in browser / node / F#: 479723520000000Oh, btw just bit-shifting indeed wraps around in browser & node:
> 915E6<<10 657129472 > 915E6<<11 1314258944 > 915E6<<12 -1666449408 > 915E6<<13 962068480 > 915E6<<14 1924136960 > 915E6<<15 -446693376 > 915E6<<16 -893386752 > 915E6<<17 -1786773504 > 915E6<<18 721420288 > 915E6<<19 1442840576 - Espruino 'Linux' build on my laptop
-
Offtopic: Try 0.1+0.2 in nodejs(v10.15.1) and in Espruino (2v04.6).
Results are 0.30000000000000004 for nodejs and 0.3 for Espruino.That's Javascript, and you should always keep in mind how numbers are represented internally.
Ontopic (Espruino 2v04.6)
>Math.pow(2,19) =524288 >915E6*Math.pow(2,19) =479723519999999.9375 >915E6*524288 =479723520000000and one fix is to manually round the pow result:
>915E6*Math.round(Math.pow(2,19)) =479723520000000 //because >524288 === Math.pow(2,19) =false >524288 === Math.round(Math.pow(2,19)) =trueAnother fix is to use 524288 instead of Math.pow(2,19) for this calculation.
And of course that result should be rounded before it is used for binary operations. -
Fri 2019.07.26
Good evening AkosLukacs and maze1980 as I sit down for my morning coffee.
Thanks for cranking out those calcs @AkosLukacs.
'there is something wrong with floating point calculation'
Yes, I agree. I remember removing the parseInt() wrapper as the integer result was not the same as the corresponding one compared to the E notation.
So, how serious is a non compatible F.P. calculation engine or what acceptable error range? I haven't found any specification docs on the site yet.
Thanks @maze1980 for the intermediate steps as a possible work around.I also, plucked all the toString(16) toString(2) parseInt() etc from the multitude of lines created trying to isolate this issue.
The Mozilla docs indicate that for bit manipulation, 2's compliment is used. It is also indicated that only 1's and 0's are used, rather than numbers. So big question, what is the proper way to get from user input to only 1's and 0's?
The original C++ line to convert:
uint64_t frf = ((uint64_t)val << 19) / 32000000;C/C++ uses 64 bits in their shift calculations. But the divide sign bothers me in the above line. Still searching for docs on 2's compliment and using division. Doesn't F.P. get used (yes?) under the hood when dividing? If yes, then the above, although works on Arduino, seems rather pointless. Why go to the trouble of left shifting, maintaining the 1's and 0's only to introduce rounding error with F.P. division? Sadly absolutely no docs or examples with that source. (maybe the author realized that the lower LSB weren't really needed?)
Since we are limited to 32 bits for bitwise calcs in Javascript, maybe an answer might be to somehow mask off the MSB just before left shift into no mans land, and shift that bit into a separate var container nineteen times, thus preserving 1's and 0's up to 64 bits? But, then how to properly handle the division? The author of the Arduino source must have had a reason to left shift nineteen times, rather than just do the calculation in F.P. wouldn't you agree?
Is anyone able to confirm that only integer values, for both operands and flags, (maybe not hex either) should be used in bitwise calcs?
In the mean time until the F.P. issue can be looked at, will just use constants to prove out the initial code conversion, but will need a solution for down the road in a ~month or so.
Time for more research. . . .
EDIT:
I wonder if inline 'C' using E.compiledC will improve things?Just did a lookup for the Math.round() function
https://github.com/espruino/Espruino/blob/master/libs/math/jswrap_math.c#L330
Just looking around and found the comment at L291 for Math.pow() VERY interesting.
Maybe this is why the F.P. calcs are off?
-
Math.pow-> Looks like you have found something:>Math.pow(2,19) == 524288 =falseAFAIK all math operations are floating point operations in javascript & Espruino.
So while(something << 19) / 32000000is perfectly valid 64 bit integer operation in C, and probably reasonably performant since bit-shifting is supported since forever. That just doesn't exist in javascript.
What's the source of those operations? -
'Looks like you have found something'
I can't take the credit for that, @maze1980 did the groundwork in #9. (unless of course you are referring to comment at L291)
'That just doesn't exist in javascript.'
So original assumption correct, what is the best way to supply the operand to be able to perform this stunt in Javascript? All docs I've read so far indicate use of integers. But aren't integers a representation of what is store using F.P.? hmmmmm
'So while (something << 19) / 32000000 is perfectly valid 64 bit integer operation in C'
But doesn't the divide convert from integer to F.P. for the calculation, introducing rounding error as you so kindly pointed out?
'What's the source of those operations?'
Part of a massive Arduino script I want to see if it possible to run using EspruinoWiFi's, so that those guys don't have all the fun!! ;-)
-
Fri 2019.07.26
While I am unable to verify if the methods are acceptable, these authors have solutions:
It appears (x).toString(2); may be acceptable
https://codeburst.io/using-javascript-bitwise-operators-in-real-life-f551a731ff5as is parseInt('100001', 2);
Getting/Setting bits using arrow functions
https://lucasfcosta.com/2018/12/25/bitwise-operations.htmlhttps://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions
ES6 spec
http://www.ecma-international.org/ecma-262/6.0/#sec-literals-numeric-literalsGood beginner article
https://medium.com/techtrument/a-comprehensive-primer-on-binary-computation-and-bitwise-operators-in-javascript-81acf8341f04Division of a 2's complement value appears to be iterative and not a conversion to floating point.
https://www.electrical4u.com/2s-complement-arithmetic/Nicely done visual on 2's complement division
http://www.weigu.lu/tutorials/microcontroller/01_numbers_codes/index.htmlSo, that might explain why division of an extremely large number may follow a bit shift.
Still, we are stuck with just 32bits in Javascript.
From #1 'the max safe integer is 231-1, or 2147483647'
Still stuck. This solution needs a workaround for nineteen left shifts. Original value is one third of max safe integer for bit manipulation.
2147483647
911000000Shifting left will force MSB bits to drop off, as we have way larger than 32 bits. There would be 52 after the shift. See #1 hilight in red bit pattern.
-
In JavaScript there is only "var" to define are a variable, and if it happens to be a number stored/converted to floating point for math operations.
In C you have different data types, there are float and double for floating point numbers, and of course byte, integer, uint64_t etc. A byte is stored using 8 bits. And if you store the number 3 in your byte and divide it by the result is 1, not 1.5. The same is true for uint64_t.
Compiled C should work with uint64_t. I wouldn't go this way and stick with JavaScript for simplicity.
-
Sat 2019.07.27
'I wouldn't go this way and stick with JavaScript for simplicity'
Sort of agree with you but, . . . as @AkosLukacs and yourself have pointed out, and with the note in the link from #10 that I found, there is an issue (fewer identical upper MSB values-see your #9) with the floating point values compared with other means. This severely (in my need) affects accuracy where the LSB bits will be needed. My suggestion for an attempt with inline 'C' might bypass what the Javascript Math library does, which is dependent on floating point. Don't know if the inline 'C' functions might use the identical floating point mechanism at the hardware level. i.e. Math.pow() is the library layer on top of the Espruino Javascript layer on top of inline 'C' asm on top of the physical hardware layer itself.
EDIT:
Note to @Gordon would you briefly describe how the floating point section works to build it's values please, and maybe where in the source they are extracted. Are log tables used? More of a curiosity question as I hadn't realized how complex doing 'simple' math actually is for a micro.https://github.com/espruino/Espruino/blob/master/libs/math/jswrap_math.c#L291
Just thought of a possible hacky solution using the original Arduino script. As my design will require the end user to input one of only 20 to 50 valid values using this C++ function conversion, I could possibly write a small Arduino code file and build a table of those values that it puts out. Then using an array or switch case code block in Javascript, extract the hex bits already converted that way. This would however, make for headaches for future user update requests of the finished deployed module.
Naaaahh, as a purist, I'll wait and see if we can find a Javascript solution.
-
If you provide a link to a datasheet, description and the original source, we may be able to figure out what it wants to do, and help with a re-implementation.
-
A re-implementation from the original C line
uint64_t frf = ((uint64_t)val << 19) / 32000000;can be re-written to C code using floating point numbers
uint64_t frf = (uint64_t)((float)val * 0.016384);and then converted to JavaScript:
var frf = Math.round(val * 0.016384);(0.016384 is equal to Math.pow(2, 19) / 32000000)
-
I meant the original intention of the code, so maybe we can figure out a way to do it without floating point operations and rounds.
Or maybe wouldn't gain anything, if this works well enough :) -
Sat 2019.07.27 midnight
Two topics to follow, Topic 1
Burning the midnight oil here it's 3am, but learned a ton!
Thank you @maze1980 for yet another angle to approach. But thought about the solution on a rather long walk-about.
There are four areas to observe in that process.
See #10 2's compliment is used in original equation for bit shifting - just ones and zeros.
Floating point numbers are made up of a mantissa and characteristic, both with round off error suspectability.
Typecasting introduces yet another form of rounding.
@AkosLukacs pointed out in his worked examples the rounding error after six digits (base 10) of accuracy. #3 and #4
This reveals the limitation of the floating point engine Espruino uses, after around 5-6 digits.
The original equation required pow(2,19) - ~pow(2,~7.?) is approx ~pow(2,~11.?) is approx 0xFFx??? where the error creeps in.
This means we need accuracy better than
0x1xxxor 4096 which requires more accuracy within those 6 digits.So, although the maths actually do work in that example,
- The (float-32) typecast converts a nine digit integer to it's mantissa and characteristic - 32
- Multiplied by a decimal constant, whose absent visible trailing zeros affect its characteristic -32
- Typecasting to a (double int-64) improves accuraccy with both more digits in the mantissa and characteristic -64
- Using the Math library in Javascript is prone to the mis-matched floating point values -32
it's the combination of rounding error that allows it to happen.
It appears that attempting to use floating point to solve a 2's compliment equation is not the way to go as the need is, to shift ones and zeros and not a representation of a number using a mantissa and characteristic. The accuracy is just not there.
And, sadly the kicker:
E.compiledC'Arithmetic with floats, doubles and uint64 may not work since it requires functions that won't be compiled in.'
Fourth link below has a nice table for comparrisonTable of bits
float - 32 bits
Nice visual
http://fabiensanglard.net/floating_point_visually_explained/
Sec 5 Nice pictorial Sec 6 Round off error in mantissa
http://www.cs.yale.edu/homes/aspnes/pinewiki/C(2f)FloatingPoint.html
Type casting in C
-
While
((uint64_t)val << 19) / 32000000does no rounding, but you still have some loss in accuracy unless the number is an integer multiple of "32000000". The final result has only three bytes, as needed for the next steps.I'm too lazy to test the result in JavaScript using a 64 bit math module vs. normal numbers, but in C it is like this:
_#include <stdio.h> //remove the underscore at pos 1, needed for formatting here only int main() { printf("Hello World\n"); __uint64_t a; double b; __uint64_t c; for (long i = 1; i < 0xFFFFFFFF; ++i) { //no errors a = ((__uint64_t)i << 19) / 32000000; b = (__uint64_t)((double)i * 0.016384); c = a - b; if (c != 0) { printf("Err: %d, %d \n", i, c); } } printf("Done\n"); return 0; }If you want to see more than "Hello World", "Done" change
b = (__uint64_t)((double)i * 0.016384);tob = (__uint64_t)((float)i * 0.016384);.It appears that attempting to use floating point to solve a 2's compliment equation is not the way to go as the need is, to shift ones and zeros and not a representation of a number using a mantissa and characteristic.
For me it doesn't look like there's any need to shift ones and zeros. Think about it.
@AkosLukacs:
The source is most likely this file here, found by searching for "<< 19) / 32000000".
For the linked code: A 32 bit frequency value is taken as input, converted to 24 bit (three byte values) with a loss in accuracy of 8 bit. -
Sun 2019.07.28
'I'm too lazy to test the result in JavaScript'
@AkosLukacs perfomed the how proof in #3 and #4
I explained the why in #19
'For me it doesn't look like there's any need to shift ones and zeros'
There isn't any bit masking in that example. It is a division proof using type casting in C.
Will work on the I2C slave example you posted:
Thank you for that suggestion
-
Sun 2019.07.28
'I meant the original intention of the code'
@AkosLukacs, the source wasn't needed as I only requested a snippet of how Arduino masked bits.
Too many over-lapping projects here. Was working on a tutorial for bitwise operators.
Unable to locate a suitable example on this site, was curious how it was done with 'C'. Was sent the snippet in #1, which maze1980 pointed out is being used by others. The formula wasn't even needed as I was just after how to mask bits. But interestingly, the formula did assist us in uncovering why my implementation started throwing errors, as you discovered.It does appear that the source sent me is a kludge copy of that or a similar file. Source that isn't commented is always suspect. Seeing the same no comments there (link #20) also, now puts the kludge chunk sent me in question.
So the formula does appear to have some physics/mathematical relationship for others to have used it.
Bigger fish to fry. Attempting to rework a rather large source is the big show stopper right now. Found this as an alternate means using additional memory:
Then on to having two devices talk back-n-forth, which I started here:
Using a formula to mask bits seems iffy anyway. What if down the road there becomes an EspruinoWiFi Deluxe using a different processor and different/unique floating point engine that is different from the existing one. Any attempt to port on to the new device, would result in errors there also. Then, immediately I'd get the blame for crappy code. A constants table makes more sense.
We'll wait for the manufacturer datasheet as you so pointed out.
Using named indexes would allow the ability to 'screen' a user request.
So, this appears to be the better solution, or an array of constants.
Thank you for your assistance with the detective work. It has been an exciting exchange and a fantastic learning experience.
-
Ah, now I got it. That's why I asked @Robin what is this about. And reading the latest comment, turns out it's just a piece of random code...
Sooooooo, originally Semtech gives users a formula in chapter 4.1.4! you don't need 64 bit integer, or more precise FP, just do the math, and multiply with 0.016384! as @maze1980 already pointed out. And now 915e6 makes sense as well: 915MHz frequency.
Quick aside: a good compiler probably can optimize away the bit shift and division, so the programmer just leaves it there instead of manually calculating / simplifying the formula. And most likely easier to follow the code.
1 Attachment
-
-
Ok, so I'm a bit late to this as I had a long weekend, but it seems like this is basically solved now thanks to @AkosLukacs and @maze1980.
It's unfortunate that Espruino gets very slightly different answers when doing some floating point maths, but if you're dependent on the exact value of a floating point number after multiplying and dividing it, it's probably a sign that the code should be working some other way.
The differences in Espruino vs browser may actually be related to this bug: https://github.com/espruino/Espruino/issues/1573
So the maths itself may well be identical, but the conversion of the actual string into a floating point number (and back again) may not be exactly the same as the desktop browsers do it.
maze1980
AkosLukacs
Gordon-
Post a reply
About
New var creation with right bit shift shows incorrect result using same initializer
Posted by
@Robin
Thr 2019.07.25
This is a rather strange anomally, and not what was originally tackled.
Foundation:
Was in the process of converting this C++ script into Javascript:
C/C++ uses a long int 64 bits, so shifting 19 places will work, and it appears that the uint8_t typecast is being used to strip leading and trailing bytes.
Kept getting incorrect result. Had a hunch that left shifing 19 places might be an issue
Down a rabbit hole, . . . but the wrong one!
After manually creating binary strings
0001101101000100111001010110000000000000000000000000 / 0001111010000100100000000000
determined that the bits were getting sent to no mans land during left shift, as expected.
Verified how Javascript vars work:
"The MAX_SAFE_INTEGER constant has a value of 9007199254740991 (9,007,199,254,740,991 or ~9 quadrillion). The reasoning behind that number is that JavaScript uses double-precision floating-point format numbers as specified in IEEE 754 and can only safely represent numbers between -(253 - 1) and 253 - 1."
"They are 64-bit floating point values, the largest exact integral value is 253-1, or 9007199254740991 . In ES6, this is defined as Number.MAX_SAFE_INTEGER. Note that the bitwise operators and shift operators operate on 32-bit ints, so in that case, the max safe integer is 231-1, or 2147483647."
So the data should fit inside that value, as we are one third of the max value. Realized that as left shifting is nothing more than doubling, tried the same using
Math.pow()Find that rabbit hole.
Process:
Wrote many string representation formatting functions to show the decimal values shown using the WebIDE debugger as both formatted hex and formatted binary to determine if it was in fact a shift issue.
It didn't appear to be, . . . until . . .
Spent two days attempting to understand why a previously working/tested format function all of a sudden started giving incorrect output. In a final act of desperation, started whacking out everything that wasn't needed from the 700+ lines of code. Still had the incorrect output.
Decided to make Html scripting pages to test. They tested fine. So where was the issue?
Found the rabbit hole.
Observation:
I made a copy of the original data in a new var assignment. An Viola! Found what was causing what appeared to be a bit shift issue. Still not quite sure, as the debugger doesn't reveal anything new on the right-hand editor side of the WebIDE. But it appears to be the assignment of an existing var!
The offending lines. They appear to be valid to me. See comment at top of source file for commented out lines. Files contain identical source, just more comments in 'incorrect' version.
Google's closure compiler shows zero errors
And with the two lines commented in:
Hoisting doesn't appear to be an issue, nor does redefining:
"If you re-declare a JavaScript variable, it will not lose its value."
Works in Chrome, either .htm files demonstrate this.
Spending as much time on this as I have, I would really like to understand where in the source this culprit exists. I'll try a few attempts at renaming the var declaration and assignment, to see if that may be part of the issue.
The two lines commented out are indicated in each source file at page top.
4 Attachments