Floating Point Numbers Are Approximations

One of the Newton 2.x OS Q&As
Copyright © 1997 Newton, Inc. All Rights Reserved. Newton, Newton Technology, Newton Works, the Newton, Inc. logo, the Newton Technology logo, the Light Bulb logo and MessagePad are trademarks of Newton, Inc. and may be registered in the U.S.A. and other countries. Windows is a registered trademark of Microsoft Corp. All other trademarks and company names are the intellectual property of their respective owners.


For the most recent version of the Q&As on the World Wide Web, check the URL: http://www.newton-inc.com/dev/techinfo/qa/qa.htm
If you've copied this file locally, click here to go to the main Newton Q&A page.
This document was exported on 7/23/97.

Floating Point Numbers Are Approximations (3/28/97)

Q: The functions Floor and Ceiling seem broken. For instance, Floor(12.2900 * 10000) returns 122899, not 122900. What's going on?

A: This is not a bug in Floor or Ceiling. This happens because of the way floating point numbers are stored, and the limitation is common to many real number representations. In the same way that 1/3 cannot accurately be represented in a finite number of digits in base 10 (it is .3333333333...), likewise 1/10 cannot be exactly represented as a fractional part in base 2. Because number printers typically round to a small number of significant digits, you don't normally notice this. The NTK inspector, for example, displays only 5 significant figures in floating point numbers. However, if you display the number with enough precision, you'll see the representation error, where the real is actually slightly larger or smaller than the intended value.
FormattedNumberStr(0.1, "%.18f") -> "0.100000000000000010"
    FormattedNumberStr(0.3, "%.18f")  ->  "0.299999999999999990"


The functions Floor and Ceiling are strict, and do not attempt to take this error into account. In the example, 12.29 is actually 12.2899999999999990, which multiplied by 10000 is 122,899.999999999990. The largest integer less than this number (Floor) is correctly 122899.

There are usually ways to work around this problem, depending on what you are trying to accomplish. To convert a floating point number to an integer, use RIntToL, which rounds to the nearest integer avoiding the problems caused with round-off error and Floor or Ceiling. RIntToL(x) produces the same result that Floor(Round(x)) would produce.
    RIntToL(12.29*10000)  ->  122900


If you need to format a number for display, use a formatting function such as FormattedNumberStr. These functions typically round to the nearest displayable value. To display 2 decimal digits, use "%.2f":
    FormattedNumberStr(12.29, "%.2f")  ->  "12.29"


If you're working with fixed point numbers such as dollar amounts, consider using integers instead of reals. By representing the value in pennies (or mils, or whatever) you can avoid the imprecision of reals. For example, represent $29.95 as the integer 2995 or 29950, then divide by 100 or 1000 to display the number. If you do this, keep in mind that there is a maximum representable integer value, 0x1FFFFFFF or 536870911, which is sufficient to track over 5 million dollars as pennies, but can't go much over that.

If you really need to find the greatest integer less than a certain number and can't tolerate how Floor deals with round off errors, you'll need to do some extra work keeping track of the precision of the number and the magnitude of the round off error. It's worthwhile to read a good numeric methods reference. Floating point numbers in NewtonScript are represented by IEEE 64-bit reals, which are accurate to around 15 decimal digits. The function NextAfterD provides a handy way to see how 'close together' floating point numbers are.
        FormattedNumberStr(NextAfterD(0.3, kInfinity), "%.18f");
                                                    ->  "0.300000000000000040"