Previous
Previous
 
Next
Next

Controlling Errors Due to Numerical Precision

Suppose expense is a decimal variable whose value is set by a calculation. When the result of the calculation is 100.000001 and the number of decimal places is two, then the value appears in output as 100.00. However, the output of the following statement returns NO.

SHOW expense EQ 100.00

You can use the ABS or the ROUND function to ignore these slight differences when making comparisons.