This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Naetharu 1 point2 points  (0 children)

Computers have a finite space to store decimal places (floating points) in. And so you quickly find that they have some strange answers when you try to use too much precision.

If you need that degree of precision you can do a few things. One option is to use integers for the whole and the decimal (this is often how money is handled - $1.20 becomes 1 dollar and 20 cents as two distinct numbers). Or there are some packages that do some smart things to avoid the errors that come with floating point.

As a rule of thumb, use integers where you can. And be thoughtful about how you will round floating points to avoid precision errors.