# Precision

## Float precision¶

Precision of floats is a tricky issue since floats (*doubles* in other languages) are actually binary rational approximations to real numbers. Whenever you are concerned with precision, set `\P 0`

before doing anything else, so that you can see what’s really going on.

Due to the finite accuracy of the binary representation of floating-point numbers, the last decimal digit of a float is not reliable. This is not peculiar to q.

q)\P 0 q)1%3 0.33333333333333331

Q takes this into account in its implementation of the equality operator `=`

, which should actually be read as “tolerantly equal.” Roughly speaking, this means that the difference is relatively small compared to some acceptable representation error. This makes the following hold:

q)r7:1%7 q)sum 7#r7 0.99999999999999978 q)1.0=sum 7#r7 1b

`0=x-y`

. Thus, we find:
q)0=1.0-sum 7#r7 0b

q)r3:1%3 q)1=r3+r3+r3 1b q)0=1-r3+r3+r3 1b

`r3+r3+r3`

is exactly 1.0. This is part of the IEEE spec, not q, and seems to be related to rounding conventions for binary floating point operations.
The `=`

operator uses tolerant equality semantics. Not all primitives do.

q)96.100000000000009 = 96.099999999999994 1b q)0=96.100000000000009-96.099999999999994 0b q)deltas 96.100000000000009 96.099999999999994 96.100000000000009 -1.4210854715202004e-014 q)differ 96.100000000000009 96.099999999999994 10b q)96.100000000000009 96.099999999999994 ? 96.099999999999994 1 q)group 96.100000000000009 96.099999999999994 96.100000000000009| 0 96.099999999999994| 1

Not transitive

Tolerant equality does not obey transitivity:

q)a:96.099999999999994 q)b:96.10000000001 q)c:96.10000000002 q)a 96.099999999999994 q)b 96.100000000009999 q)c 96.100000000020003 q)a=b 1b q)b=c 1b q)a=c 0b

The moral of this story is that we should think of floats as being “fuzzy” real values and never use them as keys or where precise equality is required – e.g., in `group`

or `?`

.

For those interested in investigating these issues in depth, we recommend the excellent exposition by David Goldberg “What Every Computer Scientist Should Know about Floating Point Arithmetic’.

### Q SIMD sum¶

The l64 builds of q now have a faster SIMD `sum`

implementation using SSE. With the above paragraph in mind, it is easy to see why the results of the older and newer implementation may not match.

Consider the task of calculating the sum of `1e-10*til 10000000`

.

The SIMD code is equivalent to the following (`\P 0`

):

q){x+y}over{x+y}over 0N 8#1e-10*til 10000000 4999.9995000000017

q){x+y}over 1e-10*til 10000000 4999.9994999999635

Worth noting is that the left-to-right order is not in some way “more correct” than others, seeing as even reversing the order of the elements yields different results:

q){x+y}over reverse 1e-10*til 10000000 4999.9995000000026

## Comparison tolerance¶

Comparison tolerance is the precision with which two numbers are determined to be equal. It applies only where one or the other is a finite floating-point number, i.e. types real, float, and datetime (see Dates below). It allows for the fact that such numbers may be approximations to the exact values. For any other numbers, comparisons are done exactly.

Formally, there is a *comparison tolerance* `t`

such that if `x`

or `y`

is a finite floating-point number, then `x=y`

is 1 if the magnitude of `x-y`

does not exceed `t`

times the larger of the magnitudes of `x`

and `y`

. `t`

is set to 2^{-43}, and cannot be changed. In practice, the implementation is an efficient approximation to this test.

Note that a non-zero value cannot equal 0, since for any non-zero `x`

, the magnitude of `x`

is greater than `t`

times the magnitude of `x`

. Thus `0=a-b`

tests for strict equality between `a`

and `b`

.

Comparison tolerance is not transitive, and can cause problems for *find* and `distinct`

. Thus, floats should not be used for database keys.

For example:

q)t:2 xexp -43 / comparison tolerance q)a:1e12 q)a=a-1 / a is not equal to a-1 0b q)t*a / 1 is greater than t*a 0.1136868 q)a:1e13 q)a=a-1 / a equals a-1 1b q)t*a / 1 is less than t*a 1.136868 q)0=a-(a-1) / a is not strictly equal to a-1 0b

### Use¶

Comparison tolerance is used by:

`=`

`<`

`>`

`~`

`differ`

`within`

And prior to V3.0

`floor`

`ceiling`

It is *not* used by other verbs that have tests for equality:

`?`

`distinct`

`except`

`group`

`in`

`inter`

`union`

`xgroup`

sort functions: `asc`

`desc`

`iasc`

`idesc`

`rank`

`xasc`

`xdesc`

### Examples¶

q)a:1f q)b:a-10 xexp -13

`b`

is treated equal to `a`

, i.e. equal to `1`

:
q)a=b 1b q)a~b 1b q)a>b 0b q)floor b 1

`b`

is treated not equal to `a`

:
q)(a,a)?b 2 q)(a,a) except b 1 1f q)distinct a,b 1 0.99999999999989997 q)group a,b 1 | 0 0.99999999999989997| 1 q)iasc a,b 1 0

### Dates¶

The datetime type is based on float, and hence uses comparison tolerance, for example:

q)a:2000.01.02 + sum 1000#1%86400 / add 1000 seconds to a date q)a 2000.01.02T00:16:40.000 q)b:2000.01.02T00:16:40.000 / enter same datetime q)a=b / values are tolerantly equal 1b q)0=a-b / but not strictly equal 0b