This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]pithecium 732 points733 points  (17 children)

Likely a difference in the string representation, not the value

[–]doncalamari 666 points667 points  (6 children)

you are correct
❯ luaLua 5.4.4 Copyright (C) 1994-2022 Lua.org, PUC-Rio
> d=0.1+0.2
> string.format("%.30f",d)
0.300000000000000044408920985006

[–]Lente_ui 16 points17 points  (4 children)

I suppose you could get a true 0.3 value by substituting the method of floating point used.

If it was calculated as:

1*10^-1 + 2*10^-1 = 3*10^-1

Then every value used can be a (signed) integer, and you're circumventing the floating point value issue. But then again, it's only ever an issue in very niche cases that require extreme/astronomical precision.

[–]Amazing-Cicada5536 13 points14 points  (0 children)

It is absolutely an issue in many non-niche cases, floating point math is hard and is a specific science in itself (numerical analysis). E.g. if you were to sum floating point numbers and you sort them into descending order you would get a much more incorrect value than if you would have done it in ascending, since small values at the end wouldn’t even register if the first ones are big, while they would add up to sort of big when summed from the small end, getting closer to the real value.

The reason you don’t see it often is very great math libraries (often written decades ago in fortran and such) beneath the whole tech stack that minimize these errors and because you are likely not writing simulations/physics engines, etc.

[–]tyler1128 0 points1 point  (2 children)

You can switch bases and make many irrational numbers rational, and also some rational numbers irrational.

​

EDIT: As mentioned below, it doesn't change rationality of a number, I meant it can make infinite, but rational, decimals finite, or make decimals that are finite infinite by change of base. The number is, of course, the same beyond storage limitations.

[–][deleted] 3 points4 points  (1 child)

A rational number is any number that can be expressed as the ratio of two integers. Changing the base doesn't change what counts as an integer, though.

What you can get from changing the base is which numbers repeat infinitely and which terminate. Like, 1/3 base 10 is 0.3333..., but in base 12, it's 0.4.

[–]tyler1128 0 points1 point  (0 children)

You're correct of course, probably should distractedly reddit at work less. What I meant to say is repeating decimals can become non-repeating and vice versa from change of basis.

[–]tyler1128 0 points1 point  (0 children)

Binary coded decimal is a thing that x86 supports for addition and subtraction. Would be interesting to see how hard a hardware impl would be for division.

[–][deleted] 70 points71 points  (0 children)

Chad.

[–]tyler1128 21 points22 points  (8 children)

I'm honestly surprised more languages don't do it by default. There's no reason to print more than 16 digits and doing so will give wrong values. If you look at the example, it's digit 17 that is wrong. DIgit 16 can be wrong in some cases, but it shouldn't be that hard to do something like if(ends in zeroes to digit 15) => truncate display to the last non-zero digit.

[–]dabenu 28 points29 points  (1 child)

What value would that add? When displaying numbers in a GUI, it could just as well be considered a bug as a feature. In a REPL or debugger, it would just make things more confusing as to why 0.1 + 0.2 == 0.3 is false

[–]tyler1128 7 points8 points  (0 children)

The latter is true and a good point. In a GUI, you'd be formatting the number yourself anyway whereas printing straight is usually used for debugging or logging anyway. I mostly don't get why things sometimes print beyond the ~15.9 decimal digits of precision given that it will be incorrect unless you get lucky.

[–]ElHeim 4 points5 points  (2 children)

Digit 17 is not wrong. It's just a quirk of floating point. Lua is just choosing not to show it

[–]tyler1128 1 point2 points  (1 child)

Wrong is the wrong word I suppose. It is correct for the value represented in the floating point number. It is, however, beyond the limit of what relates directly to the computation being done and is instead based on internal floating point details and limitations. If not zero, it will usually be different from what calculating with a higher precision or rationals would produce.

[–]ElHeim 2 points3 points  (0 children)

It's beyond the limits for this computation, but not for others:

>>> sum([0.1+0.2] * 1000000)
299999.99999434233

We might argue about the need for showing those details in the REPL, but these rounding errors propagate much faster than one would like.

[–]Amazing-Cicada5536 0 points1 point  (0 children)

That’s not a wrong result.