Previous
Previous
 
Next
Next

Controlling Errors When Comparing Floating Point Numbers

A standard restriction on the use of floating point numbers in a computer language is that you cannot expect exact equality in a comparison of two floating point numbers when either number is the result of an arithmetic operation. For example, on some systems, the following statement returns a NO instead of the expected YES.

SHOW .1 + .2 EQ .3

When you deal with decimal data, do not code direct comparisons. Instead, use the ABS or the ROUND function to allow a tolerance for approximate equality. For example, either of the following two statements produce the desired YES.

SHOW ABS((.1 + .2) - .3) LT .00001
SHOW ROUND(.1 + .2) EQ ROUND(.3, .00001)