Skip to content

A. Built-in Functions

A.0 Overview

The collection of built-in functions in q is rich and powerful. We include here a user guide for those built-ins that were not covered in the main text. For more details, see the Reference section at code.kx.com.

A.1 acos

The atomic acos is the mathematical inverse of cos. For a float argument between -1 and 1, acos returns the float between 0 and π whose cosine is the argument.

q)acos 1
0f
q)acos 1.414213562373095
0n
q)acos -1
3.141593
q)acos 0
1.570796

A.2 all

The aggregate all is (&/), which applies & cumulatively across a numeric list and returns the boolean result.

q)all 100100b
0b
q)all 10 20 30
1b

A.3 and

The function and is the same as & for people who like typing extra characters.

q)1b and 0b
0b
q)42 and 43
42

A.4 any

The aggregate any is (|/), which applies | cumulatively across a numeric list and returns the boolean result.

q)any 100100b
1b
q)any null til 10
0b

A.5 asc

The uniform function asc returns its argument list of comparables sorted in ascending order with the `s# attribute applied. When evaluated on a dictionary, it reorders the key-value pairs (on a copy) so that the values are sorted.

q)asc 3 7 2 8 1 9
`s#1 2 3 7 8 9
q)asc `b`c`a!3 2 1
a| 1
c| 2
b| 3

A.6 asin

The atomic asin is the mathematical inverse of sin. For a float argument between -1 and 1, asin returns the float between –π/2 and π/2 whose sine is the argument.

q)asin 0
0f
q)asin 1.414213562373095%2
0.7853982
q)asin 1
1.570796
q)asin -1
-1.570796

A.7 atan

The atomic atan is the mathematical inverse of tan. For a float argument, atan returns the float between –π/2 and π/2 whose tangent is the argument.

q)atan 0
0f
q)atan 1.414213562373095
0.9553166
q)atan 1
0.7853982

A.8 attr

The function attr returns any attribute of its argument as a symbol. No attributes is the empty symbol

q)attr til 5
`
q)attr asc 30 20 40 10
`s

A.9 avg

The aggregate avg returns the float average of a numeric list.

q)avg 1+til 1000
500.5

The function avg is equivalent to

{sum[x]%count x}

It is possible to apply avg to a nested list provided the sublists conform. In this context, the result conforms to the sublists and the average is calculated recursively on the sublists.

q)avg (1 2;100 200;1000 2000)
367 734f

A.10 avgs

The uniform avgs is (avg\), which computes the running averages of a numeric list.

q)avgs 1+til 10
1 1.5 2 2.5 3 3.5 4 4.5 5 5.5

A.11 bin

The binary bin takes a simple list of items (target) sorted in strictly increasing order as its first parameter. It is atomic in its second parameter (source). Loosely speaking, the result of bin is the position at which source would fall in target, looking from the left. The type of source must strictly match the type of target; no type promotion is performed.

Note

While the items of the first argument bin should be in strictly increasing order for the result to meaningful, this condition is not enforced. The result of bin when the first argument is not strictly increasing is essentially undefined.

More precisely, the result is -1 if source is less than the first item in target. Otherwise, the result is the index of the last item of target that is less than or equal to source; this is the found index if source is in target. If source is greater than the last item in target, the result is the last index of target

Tip

For large sorted lists, the binary search performed by bin can be much faster than the linear search performed by ?.

Here are some examples.

q)L:1.0+til 50000000
q)L bin 25000000.
24999999
q)L?25000000.
24999999
q)\t L bin 25000000.
0
q)\t L?25000000.
33
q)L bin 25000000.5
24999999
62
q)L?25000000.5
50000000
q)\t L bin 25000000.5
0
q)\t L?25000000.5
63
q)L bin 50000000.5
49999999
q)L?50000000.5
50000000

A.12 binr

The binary binr is closely related to bin. Whereas bin looks from the left, binr looks from the right. The result is the index of the first item of the first parameter that is greater than or equal to the second parameter.

q)L:1.0+til 50000000
q)L bin 25000000.5
24999999
q)L binr 25000000.5
25000000

A.13 Roll/Deal ?

The binary operators Roll and Deal generate pseudo-random results.

In the case where the first parameter is a non-negative long and the second parameter is a non-negative numeric value (bound), ? returns a list of pseudo-random numbers with replacement between 0 and bound, not including bound.

q)5?10
2 4 5 4 2
q)5?10
7 8 5 6 4
q)10?100.0
10.24432 86.71096 72.78528 16.27662 68.84756 81.77547 75.20102 10.86824 95.98..
q)10?100.0
64.30982 67.08738 67.89082 41.2317 98.77844 38.67353 72.6781 40.46546 83.5506..

In the case where the first parameter is a negative long and the second parameter is a non-negative long (bound), ? returns a list of pseudo-random numbers without replacement between 0 and bound, not including bound.

q)-5?10
1 8 5 7 0
q)5?10
2 4 5 4 2

Zen Moment

The expression –n?n returns a random permutation of til n.

In a case where the second parameter is a list (source) and the first parameter is a long (count), ? returns a list of count items randomly drawn from source, with replacement for positive count and without replacement for negative count.

q)5?`a`b`c`d`e`f
`b`c`f`f`b
q)5?`a`b`c`d`e`f
`e`e`a`e`e
q)-5?`a`b`c`d`e`f
`c`a`d`f`e
q)-5?`a`b`c`d`e`f
`f`a`e`d`c

In a case where the first parameter is a positive long (count) and the second argument is a symbol of the form `n where n is a positive integer no greater than eight, the result is a random list of count symbols, each comprising exactly n characters. The symbols are distinct when count is negative.

q)10?`1
`c`d`o`m`h`m`n`e`m`m
q)-10?`1
`h`p`j`n`e`o`a`k`l`i
q)-10?`3
`emb`agl`mho`ndm`gmj`egi`gek`hcc`mmb`kbh

A.14 Enum Extend ?

Enum Extend ? has as first parameter the symbolic name of a (presumably unique) list of symbols (target). It is atomic in the second parameter, which is a symbol (source). The result is the enumeration of source over target, where source is appended to target if it is not already contained therein. As a side effect of the function, symbols from source not in target are appended to target.

q)sym:`a`b`c
q)`sym?`a`x`b`y`z
`sym$`a`x`b`y`z
q)sym
`a`b`c`x`y`z

A.15 cor

The binary cor takes two numeric lists of the same count and returns the mathematical correlation between the items of the two lists as a float.

q)23 -11 35 0 cor 42 21 73 39
0.9070229

The function cor is equivalent to

{cov[x;y]%dev[x]*dev y}

A.16 count

The aggregate count returns a long representing the number of items in its atom or list parameter. As with any function defined on lists, it also applies to the value list of a dictionary and hence to tables and keyed tables.

q)count 42
1
q)count til 100
100
q)count `a`b`c`d!10 20 30 40
4
q)count ([] c1:10 20 30; c2:1.1 2.2 3.3)
3
q)count ([k:10 20] c:`one`two)
2

Tip

You cannot use count to determine whether an entity is an atom or list since atoms and singletons both have count 1. Instead test the sign of the type.


q)0>signum type 42
1b
q)0>signum type enlist 42
0b

Do you know why they call it count? Because it loves to count! Nyah, ha, ha, ha, ha. Vun, and two, and tree, and …

A.17 cov

The binary cov takes two numeric lists of the same length and returns a float equal to the mathematical covariance between the items of the two lists. If both arguments are lists, they must have the same count; atoms return 0f.

q)23 -11 35 0 cov 42 21 73 39
411.25

The function cov is equivalent to,

{avg[x*y]-avg[x]*avg y}

A.18 cross

The binary cross takes atoms or lists as arguments and returns their Cartesian product – that is, the set of all pairs drawn from the two arguments.

q)1 2 cross `a`b`c
1 `a
1 `b
1 `c
2 `a
2 `b
2 `c

You can also apply cross to dictionaries and tables.

The function cross is equivalent to,

{raze x,\:/:y}

A.19 cut

The binary function cut can take a list of long (indices) as its first parameter and a list (source) as it right parameter. It returns the list obtained by splitting source at the positions in indices.

If the initial item of indices is not 0, the first sublist split out is dropped.

q)L:10 20 30 40 50 60 70 80 90 100
q)0 2 5 7 cut L
10 20
30 40 50
60 70
80 90 100
q)2 5 7 cut L
30 40 50
60 70
80 90 100

When the first parameter is a non-negative integral atom (width), source is split at indices that are multiples of width.

q)2 cut L
10 20
30 40
50 60
70 80
90 100
q)3 cut L
10 20 30
40 50 60
70 80 90
,100

A.20 cut _

The binary function _ has several forms, depending on the types of its parameters. See also Drop and cut.

When _ is used infix, surrounding it with whitespace will avoid it getting mixed up with variable names, since _ is a valid q name character.

When the first parameter is an integral atom (count) and the second parameters is a list (source), the result is source with count items dropped from the head if count is positive and from the tail when count is negative.

q)2 _ 10 20 30 40 50 60 70 80 90 100
30 40 50 60 70 80 90 100
q)-2 _ 10 20 30 40 50 60 70 80 90 100
10 20 30 40 50 60 70 80

When the first parameter of _ is a list of long (indices) and the second parameter as a list (source) , it returns the list obtained by splitting source at the positions in indices. This is the same behavior as cut for this signature.

If the initial item of indices is not 0, the first sublist split out is dropped.

q)L:10 20 30 40 50 60 70 80 90 100
q)0 2 5 7 _ L
10 20
30 40 50
60 70
80 90 100
q)2 5 7 _ L
30 40 50
60 70
80 90 100

When the second parameter of _ is a dictionary (source) and the first parameter is a list of key values whose type matches source, the result is a dictionary obtained by removing the specified key-value pairs from the target. Since tables and keyed tables are dictionaries, _ applies to them by extension.

q)`a`c _ `a`b`c!10 20 30
b| 20
q)`c1`c2 _ ([] c1:`a`b; c2:10 20; c3:1.1 2.2)
c3
---
1.1
2.2
q)([] k:`a`c) _ ([k:`a`b`c] v:10 20 30)
k| v
-| --
b| 20

The first parameter must be a list, so a single value for the first parameter in this form must be enlisted.

q)(enlist `b) _ `a`b`c!10 20 30
a| 10
c| 30

When the first parameter of _ is a list or a dictionary (source) and the second parameter is an atom representing an index or key, the result is obtained by deleting the specified item from (a copy of) source.

q)101 102 103 104 105 _ 2
101 102 104 105
q)(`a`b`c!10 20 30) _ `b
a| 10
c| 30
q)([] c1:`a`b; c2:10 20; c3:1.1 2.2) _ 1
c1 c2 c3
---------
a 10 1.1
q)([k:101 102 103] c:`one`two`three) _ 102
k | c
---| -----
101| one
103| three

A.21 deltas

The uniform deltas is (-':), which returns the difference of each item in a numeric list with its predecessor.

q)deltas 1 2 3 4 5
1 1 1 1 1
q)deltas 96.25 93.25 58.25 73.25 89.50 84.00 84.25
96.25 -3 -35 15 16.25 -5.5 0.25

If you are looking to bound the absolute difference between successive elements, the second example shows that the initial item of the result will be troublesome. In this case, you can use,

deltas0:{first[x] –': x}

In our example above,

q)deltas0 96.25 93.25 58.25 73.25 89.50 84.00 84.25
0 -3 -35 15 16.25 -5.5 0.25

A.22 desc

The uniform desc returns (a copy of) its argument list of comparables sorted in descending order. When evaluated on a dictionary, it reorders the key-value pairs (on a copy) so that the values are sorted.

q)desc 3 7 2 8 1 9
9 8 7 3 2 1
q)desc `b`c`a!2 3 1
c| 3
b| 2
a| 1

A.23 dev

The aggregate dev returns the float standard deviation of a numeric list.

q)dev 1000?100.
29.40271

The function dev is equivalent to,

{sqrt var x}

A.24 differ

The uniform differ is

not (~':)

It asks if each item in a list is not identical to its predecessor. The item at index 0 in the result is always 1b.

q)differ 0 1 1 2 3 2 2 2 4 1 1 3 4 4 4 4 5
11011100110110001b
q)differ "mississippi"
q)differ (1 2; 1 2; 3 4 5)
101b

One use of differ is to locate runs of repeated items in a list.

q)L:0 1 1 2 3 2 2 2 4 1 1 3 4 4 4 4 5
q)nd|next nd:not differ L
01100111011011110b

A.25 distinct

The function distinct returns the unique items in its list argument, in order of first occurrence. Note that it does not apply the `u# attribute.

q)distinct 1 2 3 2 3 4 6 4 3 5 6
1 2 3 4 6 5

Tip

Following is a test for existence of duplicates. {count[x]=count distinct x}

Since a table is a list of records, distinct effectively removes duplicate rows. All fields’ records must be identical for them to be considered identical.

q)distinct ([]a:1 2 3 2 1; b:`washington`adams`jefferson`adams`washington)
a b
------------
1 washington
2 adams
3 jefferson
q)distinct ([]a:1 2 3 2 1; b:`washington`adams`jefferson`adams`wasington)
_

A.26 enlist

The function enlist returns a list whose items comprise its arguments. The most common use is to create a singleton list from a single argument.

Unlike user-defined functions, the number of arguments to enlist is not restricted to eight.

q)enlist 42
,42
q)count enlist 10 20 30
1
q)enlist[1;2;3;4;5;6;7;8;9;10]
1 2 3 4 5 6 7 8 9 10

A.27 eval

The unary eval evaluates a list that is a valid q parse tree; it is the same code used in the q interpreter. Such a list can be produced using parse on a string containing a valid q expression, or by hand – if you know what you're doing. A full discussion of parse trees is beyond the scope of this tutorial.

q)eval parse "a:6*7"
42
q)a
42
q)eval (+;1;(*;6;7))
43

A.28 except

The binary except takes a list (or dictionary) as its first parameter (target) and an atom or list of items (or keys) of the same type as target for its second parameter (source) and returns those items in target that are not specified in source. The returned items are in the order of their first occurrence in target.

q)1 2 3 4 3 2 except 2
1 3 4 3
q)string[2015.01.01] except "."
"20150101"
q)(`a`b`c`d!10 20 30 40) except `a`d!10 40
20 30
q)([] c1:`a`b`c`d; c2:10 20 30 40) except ([] c1:`a`d; c2:10 40)
c1 c2
-----
b 20
c 30

The result of except is always a list.

A.29 exit

The unary exit takes a long as its parameter (retval) and causes the q process to exit, returning retval to the OS.

There is no prompt for confirmation.

A.30 Fill ^

The binary fill ^ takes a list as its second parameter (target) and an atom of the same type as its first parameter (fillval). It returns (a copy of) target with null values filled with fillval.

q)42^1 2 3 0N 5 0N
1 2 3 42 5 42
q)"_"^"Now is the time"
"Now_is_the_time"
q)`NA^`First`Second``Fourth
`First`Second`NA`Fourth

Fill is atomic in the second parameter.

q)42^(1;0N;(100;200 0N))
1
42
(100;200 42)

Like all functions, fill operates on the values of a dictionary.

q)42^`a`b`c`d!100 0N 200 0
a| 100
b| 42
c| 200
d| 0
q)0^([]c1:1.0 2.0 0n; c2:0N 2 0N)
c1 c2
-----
1 0
2 2
0 0

A.31 Find ?

The binary operator ? Find takes a list (target) as its first parameter and an atom of the same type (source) as its second parameter. It returns the index of the first occurrence of source in target or the count of target if it is not found. It is atomic in the second parameter.

The simplest case is when source is an atom.

q)100 99 98 87 96 98?98
2
q)`one`two`three?`four
3
q)"Now is the time"?"the"
7 8 9

Tip

The first example demonstrates that Find returns only the index of the first occurrence of source. To find the indices of all occurrences you could use,


q){where x=y}[100 99 98 87 96 98;98]
2 5

Find also works on general lists, although if you need this you should probably consider redesigning your program.

q)(1 2; 3 4; `a`b)?`a`b
2
q)((0;1 2);3 4;5 6)?5 6
2

Find can also take a dictionary as its first parameter (target) and a dictionary value as its second parameter *value*. It returns the first key that maps to value. It is atomic in the second parameter.

q)(`a`b`c`d!10 20 30 10)?10
`a
q)(`a`b`c`d!10 20 30 10)?10 30
`a`c

By extension, find applies to tables and keyed tables.

q)([] c1:`a`b`c; c2:10 20 30)?`c1`c2!(`b;20)
1
q)([] c1:`a`b`c; c2:10 20 30)?(`b;20)
1
q)([k:1 2 3] c:100 101 102)?101
k| 2

Zen Moment

Viewing a list or dictionary as a mapping, Find is the inverse mapping.

A.32 fills

The uniform fills is ^\, which fills forward, meaning that non-null items are filled over succeeding null items.

q)fills 1 0N 3 0N 0N 5
1 1 3 3 3 5
q)fills `x``y```z
`x`x`y`y`y`z
q)update fills c2 from ([] `a`b`c`d`e`f; c2:1 0N 3 0N 0N 5)
x c2
----
a 1
b 1
c 3
d 3
e 3
f 5

If you need to fill initial nulls, use the binary form of ^\.

q)fills 0N 0N 3 0N 5
0N 0N 3 3 5
q)0 ^\ 0N 0N 3 0N 5
0 0 3 3 5

A.33 flip

The unary function flip transposes a rectangular list, column dictionary or table (source).

When source is a rectangular list, the items are rearranged, effectively reversing the first two indices in indexing. For example,

q)show m:(1 2 3;10 20 30)
1 2 3
10 20 30
q)flip m
1 10
2 20
3 30

When flip converts a column dictionary to a table and vice versa, no data is actually rearranged. Only column indexing is reversed.

q)dc:`c1`c2!(`a`b`c;10 20 30)
q)t:([] c1:`a`b`c; c2:10 20 30)
q)dc[`c1;2]~t[2;`c1]

A.34 getenv

The unary function getenv takes a symbol argument representing the name of an OS environment variable and returns the value if any of that environment variable as a string.

q)getenv `SHELL
"/bin/bash"

A.35 group

The unary function group takes a list (source) and returns a dictionary in which each distinct item in source is mapped to the indices of its occurrences in source. The keys of the result are in the order of their first appearance in source.

q)group "i miss mississippi"
i| 0 3 8 11 14 17
 | 1 6
m| 2 7
s| 4 5 9 10 12 13
p| 15 16

A.36 gtime

The atomic gtime converts local time to UTC time.

q).z.P
2015.04.12D12:10:20.685653000
q)gtime .z.P
2015.04.12D22:10:24.861692000
q).z.p
2015.04.12D22:10:28.597654000

A.37 iasc

The uniform iasc takes a list or a dictionary (source). Considering source as a mapping, the result of iasc is a list of the indices/keys of source in the order that would sort it ascending. Otherwise put, composing source with the result of iasc sorts in ascending order.

q)L:30 70 20 80 10 90
q)iasc L
4 2 0 1 3 5

q)L iasc L
10 20 30 70 80 90

q)d:`c`a`b!30 10 20
q)iasc d
`a`b`c
q)d iasc d
10 20 30

This is useful when you want to use one list to control the order of others – e.g., manual sort on the serialized columns of a splayed table (see §14.3).

Zen Moment

The composite iasc iasc provides the list of indices that transforms the sorted entity back to the original.

A.38 Identity ::

The unary operator denoted :: is the identity function – i.e., it returns its argument.

q)::[42]
42
q)(::) `Zaphod
`Zaphod

The identity function cannot be used naked with prefix syntax or @. You must either enclose it in parentheses or enclose its argument in square brackets.

A.39 idesc

The unary function idesc takes a list or a dictionary (source). Considering source as a mapping, the result of idesc is a list of the indices/keys of source in the order that would sort it descending. Otherwise put, composing source with the result of idesc sorts in descending order.

q)L:30 70 20 80 10 90
q)idesc L
5 3 1 0 2 4
q)L idesc L
90 80 70 30 20 10
q)d:`c`a`b!30 10 20
q)idesc d
`c`b`a
q)d idesc d
30 20 10

This is useful when you want to use one list to control the order of others – e.g., manual sort on the serialized columns of a splayed table (see §14.2).

Zen Moment

The composite idesc idesc provides the list of indices that transforms the sorted entity back to the original.

A.40 in

The binary function in is atomic in its first parameter (source) and takes an atom or list second parameter (target). It returns a boolean indicating whether source appears in target. The comparison is strict with regard to type.

q)42 in 0 6 7 42 98
1b
q)4 in 42
0b
q)"4" in "42"
1b
q)"cat" in "abcdefg"
110b

A.41 inter

The binary inter returns the items in its first list parameter that occur in its second list parameter.

q)1 2 3 4 inter 3 4 5
3 4

Lists differ from sets in that they are ordered and allow duplicates, so inter is not commutative in general.

q)1 2 3 1 inter 4 1
1 1
q)4 1 inter 1 2 3 1
,1
q)1 2 3 inter 4 3 2
2 3
q)4 3 2 inter 1 2 3
3 2

You can use inter to find common records in tables having identical schemas.

q)t1:([] c1:`a`b`c; c2:10 20 30)
q)t2:([] c1:`c`d; c2:30 40)
q)t1 inter t2
c1 c2
-----
c  30

A.42 inv

The unary inv returns the inverse of a float matrix.

q)m:(1.1 2.1 3.1; 2.3 3.4 4.5; 5.6 7.8 9.8)
q)inv m
-8.165138 16.51376 -5
12.20183 -30.18349 10
-5.045872 14.58716 -5

An integer matrix must be cast to float.

A.43 Join ,

The binary operator Join , appends its second parameter to the first. When both operands are either lists or atoms, the result concatenates the second parameter to the first.

q)1,2 3 4
1 2 3 4

q)1 2 3,4
1 2 3 4

q)1 2,3 4
1 2 3 4

The result is a general list unless all items are of homogeneous type.

q)1 2 3,4.0
1
2
3
4f

When both parameters are dictionaries, the result is the merge of the second parameter into the first using upsert semantics. Otherwise put, assignments in the second parameter prevail over those in the first.

q)(`a`b`c!10 20 30),`c`d!300 400
a| 10
b| 20
c| 300
d| 400

By extension, you can use , on tables and keyed tables with identical schemas.

A.44 key

The key operator vies for the title of most overloaded q operator. Its eponymous action is to return the key of a dictionary, including the special case of the key table portion of a keyed table. It can be applied by value or name.

q)key `a`b`c!10 20 30
_
q)kt:([k:`a`b`c] v:10 20 30)
q)key kt
_
q)key `kt
_

A context – i.e. the content of a namespace – is a dictionary, so key applied to the symbolic name will return the names of its constituents. Observe that contexts below the root all have an initial null name.

q)key `.
_
q).jab.a:42
q)key `.jab
_

To list all contexts, apply key to the null symbol.

q)key `
`q`Q`h`j`o`jab

Given a simple list, key returns the symbolic name of its type.

q)key 10 20 30
`long
q)key "so long"
_

Given an enumerated entity, key returns the name of its enumeration domain. This applies to foreign key columns, where it returns the name of the target keyed table.

q)sym:`c`b`a
q)key `sym$`a`b`a`c`a
_

q)kt:([k:`a`b`c] v:10 20 30)
q)t:([] k:`kt$`a`b`a`b`a; v: 10 20 30 49 59)
q)key t `k
_

Given an I/O handle corresponding to a directory, key returns a list of the symbolic names of entities in that directory, including hidden files.

q)\ls q
"README.txt"
"m32"
"q.k"
"q.q"
"s.k"
"sp.q"
"trade.q"

q)key `:q
`.DS_Store`README.txt`m32`q.k`q.q`s.k`sp.q`trade.q

Tip

An empty directory returns an empty list of type symbol whereas a non-existent directory returns an empty general list.

This provides a simple test for the existence of a directory.

q)\ls q/jab
ls: q/jab: No such file or directory
'os
q)()~key `:q/jab
1b
q)\mkdir q/jab
q)key `:q/jab
`symbol$()

Given an I/O handle corresponding to a file, key returns the symbolic file name if the file exists and the general empty list () if it does not. As above, this provides a test for existence of the file.

q)key `:q/trade.q
`:q/trade.q
q)()~key `:q/quote.q
1b

Last, but not least, key also acts as a synonym of til.

q)key 10
_

A.45 like

The binary like performs pattern matching on its first string parameter (source) according to the pattern in its string second parameter (pattern). It returns a boolean result indicating whether pattern is matched. The pattern is expressed as a mix of regular characters and special formatting characters. The special chars are ?,*, the pair[and], and a ^ that is enclosed in square brackets.

The special char ? represents an arbitrary single character in the pattern.

q)"fan" like "f?n"
1b
q)"fun" like "f?n"
1b
q)"foP" like "f?p"
0b

The special char * represents an arbitrary sequence of characters in the pattern.

As of this writing (Sep 2015), only a single occurrence of * is allowed in the pattern, except that a * can occur at both the beginning and end of the pattern.

q)"how" like "h*"
1b

q)"hercules" like "h*"
1b

q)"wealth" like "*h"
1b

q)"flight" like "*h*"
1b

q)"Jones" like "J?ne*"
1b

q)"Joynes" like "J*ne*"
'nyi

The special character pair [ and [ encloses a sequence of alternatives for a single character match.

q)"flap" like "fl[ao]p"
1b

q)"flip" like "fl[ao]p"
0b

q)"459-0609" like "[0-9][0-9][0-9]-0[0-9][0-9][0-9]"
1b

q)"459-0609" like "[0-9][0-9][0-9]-1[0-9][0-9][0-9]"
0b

The special character ^ is used in conjunction with [ and ] to indicate that the enclosed sequence of characters is disallowed. For example, to test whether a string ends in a numeric character,

q)"M26d" like "*[^0-9]"
1b

q)"Joe999" like "*[^0-9]"
0b

Enclose a special char in square brackets to escape it as a normal char.

q)"a_c" like "a[*]"
0b

q)"a*c" like "a[*]c"
1b

A.46 lower

The atomic lower takes a char, string or symbol argument and returns the result of converting all alpha text to lower case.

q)lower `A
`a

q)lower "a Bc42De"
"a bc42de"

A.47 lsq

The binary matrix function lsq takes float matrix parameters A and B and returns the matrix X that solves the following matrix equation, where · is matrix multiplication.

A = X·B

For example,

q)A:(1.1 2.2 3.3;4.4 5.5 6.6;7.7 8.8 9.9)
q)B:(1.1 2.1 3.1; 2.3 3.4 4.5; 5.6 7.8 9.8)
q)A lsq B
1.211009 -0.1009174 2.993439e-12
-2.119266 2.926606 -3.996803e-12
-5.449541 5.954128 -1.758593e-11

Observe that lsq is equivalent to,

q)A mmu inv B
1.211009 -0.1009174 1.77991e-12
-2.119266 2.926606 -5.81224e-12
-5.449541 5.954128 -1.337952e-11

Integer matrices must be cast to float.

A.48 ltime

The atomic ltime converts UTC time to local time.

q).z.P
2015.04.12D12:12:42.475837000

q)ltime .z.P
2015.04.12D02:12:45.987860000

q).z.p
2015.04.12D22:12:47.859567000

A.49 ltrim

The unary ltrim takes a string parameter and returns the result of removing leading blanks.

q)ltrim " abc "
"abc " "
q)ltrim " "
""

You can also apply ltrim to a non-blank char but it fails for a blank char.

q)ltrim "a"
"a"
q)ltrim " "
k){$[~t&77h>t:@x;.z.s'x;" "=*x;(+/&\" "=x)_x;x]}
'type
_
1b
" "

A.50 mavg

The binary mavg takes first parameter a long (length) and returns the length-wise moving average of its numeric list second parameter (source). At each position in source, it computes the average of the length preceding items, or as many as are available up to length, ignoring internal nulls. It is uniform in its second argument.

In the following example, the first item in the result is the average of itself only; the second result item is the average of the first two source items; all other items reflect the average of the item at the position along with its two predecessors.

q)3 mavg 10 20 30 40 50
10 15 20 30 40f

q)3 mavg 0N 10 20 30 40 50
0n 10 15 20 30 40

For length 1, the result is the source converted to float.

A.51 max

The aggregate max is (|/), which applies | cumulatively across a list of items of underlying numeric type and returns the final result. It is the same as any on a boolean list.

q)max 100100b
1b

q)max 10?2015.01.01
2013.09.09

A.52 maxs

The uniform maxs is (|\), which applies | across a list of comparable items and returns the intermediate results.

q)maxs 1 2 5 4 10
1 2 5 5 10

q)maxs "Beeblebrox"
"Beeelllrrx"

A.53 mcount

The uniform binary mcount takes as first parameter a long (length) and returns a long equal to the length-wise moving count of its list second parameter (source). At each position in source, it counts the number of non-null preceding items up to length. This function is useful in computing other moving quantities, such as moving average, since it correctly reports the number of predecessors at the head of the list and also ignores internal nulls.

q)3 mcount 10 20 30 40 50
1 2 3 3 3i

q)3 mcount 10 20 0N 40 50 60 0N
1 2 2 2 2 3 2i

A.54 md5

The unary function md5 computes the MD5 (Message Digest Algorithm 5) hash of a string as a list of 16 bytes.

q)md5 "Life the Universe and Everything"
0x0e9a5631e4db880d43808504d05348df

A.55 mdev

The uniform binary mdev takes first parameter a long (length) and returns the length-wise moving standard deviation of its numeric list second parameter (source). At each position in source, it computes the standard deviation of the length preceding items, or as many as are available at that position up to length. It excludes internal nulls from the calculation.

q)3 mdev 10 20 30 40 50
0 5 8.164966 8.164966 8.164966

q)3 mdev 10 20 30 0N 50
0 5 8.164966 5 10

A.56 med

The aggregate med returns the float median of a numeric list.

For lists and dictionaries, the result is a float.

q)med 1000?1000.
499.1908

The function med is equivalent to,

{avg x (iasc x)@floor .5*-1 0+count x,:()}

A.57 min

The aggregate min is (&/), which applies & cumulatively across a list of underlying numeric type and returns the final result. It is the same as and for a boolean list.

q)min 100?100
2

q)min 100100b
0b

A.58 mins

The uniform mins is (&\), which computes the cumulative minimums of a list of comparables and returns the intermediate results.

q)mins 10 4 5 1 2
10 4 4 1 1

q)mins "the cat"
"theecaa"

A.59 mmax

The uniform binary mmax takes as first parameter a long (length) and returns the length-wise moving maximum of its second parameter list of comparables (source). At each position in source, it computes the maximum of the length preceding items, or as many as are available at that position up to length.

In the following example, the first item in the result is the max of itself only; the second result item is the max of the first two source items; all other items reflect the max of the item at the position along with its two predecessors.

q)3 mmax 20 10 30 50 40
20 20 30 50 50

q)3 mmax 20 10 30 0N 40
20 20 30 30 40

A.60 mmin

The uniform binary mmin takes first parameter a long (length) and returns the length-wise moving minimum of its second parameter list of comparables (source). At each position in source, it computes the minimum of the length preceding items, or as many as are available at that position up to length.

In the following example, the first item in the result is the min of itself only; the second result item is the min of the first two source items; all other items reflect the min of the item at the position along with its two predecessors.

q)3 mmin 20 10 30 50 40
20 10 10 10 30

For length less than or equal to 0 the result is source.

A.61 mmu

The binary matrix multiplication mmu returns the matrix product of its two float vector or matrix parameters, which must be of the correct shape. It reduces to dot product on vectors. Integer matrices must be cast to float.

q)m1:(1.1 2.2 3.3;4.4 5.5 6.6;7.7 8.8 9.9)
q)m1 mmu flip m1
16.94 38.72 60.5
38.72 93.17 147.62
60.5 147.62 234.74

q)m2:`float$(0 0 1; 0 1 0; 1 0 0)
q)m2 mmu m2
1 0 0
0 1 0
0 0 1

q)1 2 3f mmu 1 2 3f
14f

The $ operator is overloaded to yield matrix multiplication when its parameters are float vectors or matrices.

q)m2 $ m2
1 0 0
0 1 0
0 0 1

A.62 msum

The uniform binary msum takes first parameter a long (length) and returns the length-wise moving sum of its numeric list second parameter (source). At each position in source, it computes the sum of the length preceding items, or as many as are available at that position up to length. It excludes internal nulls from the sum.

In the following example, the first item in the result is the sum of itself only; the second result item is the sum of the first two source items; all other items reflect the sum of the item at the position along with its two predecessors.

q)3 msum 10 20 30 40 50
10 30 60 90 120

A.63 next

The uniform next returns its list parameter shifted one position to the left with a null (or empty) in the last item. Otherwise put, each item in the original is replaced with its successor.

q)next 1 2 3 4 5
2 3 4 5 0N

q)next (1 2; 3 4 5; 6 7)
3 4 5
6 7
`long$()

A.64 null

The atomic function null tests its argument for null value. It is preferred to testing for equality as it works for all types.

q)null 42
0b

q)null 0n
1b

q)null "Now is the time"
000100100010000b

A common idiom combines where with null to obtain the positions of the null items in a list.

q)where null 1 2 3 0N 5 0N
3 5

Because null is atomic, it picks out null values in all fields.

q)null ([] c1:`a``b; c2:0N 20 30)
c1 c2
-----
0 1
1 0
0 0

This can be used to identify null columns.

q)where all null ([] c1:`a`b`c; c2:0n 0n 0n; c3:10 0N 30)
,`c2

A.65 or

The function or is the same as | for people who like typing extra characters.

q)1b or 0b
1b

q)42 or 43
43

A.66 over

The function over is a convenience function that renames the unary form of / for those allergic to k.

q){x+2*y} over 2 3 5 7
32

q)(+) over 2 3 5 7
17

A.67 parse

The unary parse returns the q parse tree for a string representing a q expression. Applying eval to the result of parse effectively evaluates the expression. The composition of eval after parse is essentially the q interpreter. A discussion of q parse trees is beyond the scope of this tutorial.

q)parse "7*4+2"
*
7
(+;4;2)
q)eval parse "7*4+2"
42

It is useful to apply parse to a string containing a query template for the purpose of expressing the query in functional form. The result will often include k code but it is usually recognizable and you can use it in functional form.

The constraint portion of the resulting parse tree will contain an extra level of nesting displayed in the k form ,,.

You must remove one level for the functional form.

q)t:([]c1:`a`b`c; c2:10 20 30)
q)parse "select c2:2*c2 from t where c1=`c"
?
`t
,,(=;`c1;,`c)
0b
(,`c2)!,(*;2;`c2)
q)?[`t; enlist (=;`c1;enlist `c); 0b; (enlist `c2)!enlist (*;2;`c2)]
c2
--
60

A.68 peach

A full discussion of how q handles concurrency and distributed processing is beyond the scope of this text. We provide a description of the essential features in q3.2 as of this writing (Sep 2015) and refer the reader to the KX site for up-to-date details.

Concurrency in q uses two basic constructs: peach and slaves. Slaves can be either threads spawned by the main q process or independent q processes. How many and which type of slaves are declared to the q process at startup. The function peach is a “parallel” version of each that applies a function over a list by distributing the items of the list across slaves. This is q’s implementation of the “map” portion of the distributed map-reduce paradigm, which is much-ballyhooed in other environments.

If q is not started with slaves, peach reverts to each, meaning function application across the list occurs sequentially in the main q thread.

Use the command line parameter -s to start a slave-enabled q session. In classic Arthurian fashion, the interpretation of –s depends on the sign of the integer that follows it. A positive integer n indicates that the q process should start with n slave threads to be used by peach. A negative integer –n means that you will provide n independent slave q processes for peach to use. The physical implementation of work distribution is different in these two cases, but the logical result is the same.

A.68.1 Threads

When q is started with -s n, the q process automatically spawns n slave threads in addition to the main thread at startup. When you subsequently use peach to apply a function – i.e., map it – across a list, the application is automatically distributed across the slave threads for you. The result of the application is the same as if you had used each but if q is running on a machine with multiple cores, you should expect a speedup, depending on what the function does, how slave threads are allocated to cores and the load on your machine. For example, on the author’s laptop that has 4 cores and hyperthreading, we find that the benefit of multiple slaves for evaluating a compute-intensive calculation on a list of 8 items plateaus at 4 slaves.

$ rlwrap q/m32/q -s 2
q)\t {sum exp x?1.0} each 8#1000000
134
q)\t {sum exp x?1.0} peach 8#1000000
72
$ rlwrap q/m32/q -s 4
q)\t {sum exp x?1.0} each 8#1000000
136
q)\t {sum exp x?1.0} peach 8#1000000
46
$ rlwrap q/m32/q -s 8
q)\t {sum exp x?1.0} each 8#1000000
135
q)\t {sum exp x?1.0} peach 8#1000000
43

In order to maintain consistency during concurrent application, the following restriction is placed on function evaluation within slave threads. Code executed in a slave thread cannot update global variables – i.e., a function can only update local variables.

Tip

If you use a timer on the main thread or other tricks to update globals that are read by slave threads, you are cruising for a bruising. While these updates are locked, there is no notion of a transaction with attendant rollback. It is unlikely that your function executing on the timer is atomic at the hardware level. Should it fail mid-flight, you should expect the workspace would be in an inconsistent state with no built-in recovery mechanism.

Now we describe how the slave thread execution is implemented. The items in the list are pre-assigned to slaves in a round-robin fashion. For example, if you peach a function on a list with 9 items in a session with 4 slaves: items 0, 4 and 8 will be executed sequentially on slave 0; items 1, 5 and 9 will be executed sequentially on slave 1; items 2 and 6 will be executed sequentially on slave 2; and items 3 and 7 will be executed sequentially on slave 3. The sequential execution within slaves occurs concurrently across slaves.

The slaves use thread-local heaps and the (partial) results of application within each slave are serialized and copied to the main thread where they are deserialized and assembled into the final result. Since there is overhead associated with serialization/deserialization, we have:

Tip

A function applied with peach should do enough computation to outweigh the serialization overhead. You can estimate the serialization time and space of a q entity with \ts:100 -9!-8!entity

We collect here observations on execution within a slave. See the KX site for more details.

  1. If the list has a single item, peach application occurs in the main q thread only.

  2. If the function applied with peach performs grouping on a symbol list – e.g., a select that groups on a ticker symbol – this will be significantly slower in the slave because the optimized algorithm used in the main thread is not available in the slave.

  3. In q3.*, a socket can be used from the main thread only. As there is no locking around a socket descriptor, a communication handle shared between threads would result in garbage due to message interleaving.

  4. Each slave thread has its own heap, a minimum of 64MB. Executing .Q.gc[] in the main thread triggers garbage collection in the slave threads too. Automatic garbage collection within each thread is only executed for that particular thread, not across all threads.

  5. Symbols are internalized in a single memory area accessible to all threads.

A.68.2 Distributed peach

Starting q with –s –n indicates:

  1. In addition to the main q process, you will instantiate n workers, where a worker is an independent q process with an open port

  2. You will connect the master q process to each worker

  3. You will provide the master process a unique’d list of open handles to the workers

The list of handles to the workers is obtained from the system variable .z.pd upon each invocation of peach. You can set .z.pd either to an integer list of open handles with the `u# attribute applied, or a function that returns the same.

Tip

You must apply the `u# attribute or you will get the error:


'.z.pd - expected unique vector of int handles

In contrast to the situation with slave threads, where the threads are created for you automatically, with distributed peach it is your responsibility to instantiate the separate worker processes and make them available to the main q process. In production environments this may entail meeting security requirements – e.g., Kerberos. You can start the workers externally to your main process or you can start them from within the process using system.

The workers can be on the same machine as the master process or on any machine on a network so long as it is accessible from the master process. Since they are independent q processes that share no data with the main process, the distributed function is not restricted as to what it can do on the worker. This provides a very flexible framework for distributed computing.

Once you have started the worker processes, you must open a connection to each one, keeping track of the open handles.

You cannot use these handles for other messaging, as peach will close any handle that presents a message not in its expected format.

You can assign the unique’d list of the open connections to .z.pd or you can return such a list from a function that you assign to .z.pd. In contrast to the slave thread implementation, where the assignment to slaves is determined in advance, individual computations are automatically distributed to workers that become free. Thus we get a simple form of dynamic load balancing.

Following is an example that starts four external workers on the same machine from within the master process, wires the workers directly into .z.pd and then uses peach to distribute work to them.

~$q -s -4
..
q){system "q -q -p ",string[x]," &"} each 20000+til 4
q).z.pd:`u#hopen each `$"::",/:string 20000+til 4
q)\t {sum exp x?1.0} each 8#1000000
134
q)\t {sum exp x?1.0} peach 8#1000000
46

You can also assign to .z.pd a function that manages the connections. The code below monitors closing connections via .z.pc, which removes the handle of any closed connection from its self-maintained list of active handles. The function assigned to .z.pd checks to see if all expected handles are open and, if not, closes any open handles and then reopens all the specified handles.

q)handles:`u#`int$()
q).z.pd:{
 if[0>=n:neg system "s";
 '"must start q with -s -n"];
 if[n<>count handles; hclose each handles;
 `handles set `u#hopen each 20000+til n];
 handles}
q).z.pc:{`handles set `u#handles except x;}

A.69 prd

The aggregate prd is (*/), which applies * cumulatively across a numeric list and returns the final result.

Note the missing ‘o’ in the name.

q)prd 1+til 5

q)prd 1+til 5
120

q)prd 10?100.
2.667807e+13

It is possible to apply prd to a nested list provided the sublists conform. In this case, the result conforms to the sublists and the product is calculated recursively on the sublists.

q)prd (1 2; 100 200; 1000 2000)
100000 800000

A.70 prds

The uniform prds is (*\), which applies * cumulatively across a list of numeric type and returns the intermediate results.

Note the missing ‘o’ in the name.

q)prds 1+til 5
1 2 6 24 120

A.71 prev

The uniform prev returns a list argument shifted one position to the right with a null (or empty) in the first item. Otherwise put, each item in the original is replaced with its predecessor.

q)prev 1 2 3 4 5
0N 1 2 3 4

q)prev (1 2; 3 4 5; 6 7)
`long$()
1 2
3 4 5

A.72 prior

The function prior is a convenience function that renames the unary form of ': for those allergic to k.

q)(-) prior 10 11 12 13 14
10 1 1 1 1

A.73 rank

The uniform rank returns the sort order of each item in a list of comparables.

q)rank 50 20 30 10 40
4 1 2 0 3

q)rank `x`a`b`z`c
3 0 1 4 2

A.74 ratios

The uniform ratios is (%':), which applies % across a list of numeric type and returns the intermediate results. That is, it returns the ratio of each item with its predecessor. The initial item in the result is 1.0.

q)ratios 10 20 30 40 50
10 2 1.5 1.333333 1.25

If you are looking to bound the relative difference between successive elements, this example shows that the initial item of the result will be troublesome.

In this case, you can use,

ratios1:{first[x] %': x}

For our example above,

q)ratios1 10 20 30 40 50
1 2 1.5 1.333333 1.25

A.75 raze

The unary raze is (,/). It effectively eliminates the top-most level of nesting from a list by concatenating across it.

q)raze (1 2 3;3; 4 5)
1 2 3 3 4 5

Observe that raze only removes the top-most level of nesting but you can use each to reach deeper levels.

q)raze ((1 2;3 4);(5;(6 7;8 9)))
1 2
3 4
5
(6 7;8 9)
q)raze each ((1 2;3 4);(5;(6 7;8 9)))
1 2 3 4
(5;6 7;8 9)a

A.76 reval

Introduced in q3.3, reval is a version of eval that behaves as if the command line option -b had been set. This means that all updates on the server are blocked for the duration of the evaluation. See §A.27 and §13.2.1.

A.77 reverse

The uniform function reverse inverts the item order of its list argument.

q)reverse 1 2 3 4 5
5 4 3 2 1

q)reverse `a`b`c!10 20 30
c| 30
b| 20
a| 10

q)reverse ([] c1:`a`b`c; c2:10 20 30)
c1 c2
-----
c 30
b 20
a 10

q)reverse ([k:`a`b`c] v:10 20 30)
k| v
-| --
c| 30
b| 20
a| 10

For nested entities, the reversal takes place only at the topmost level but you can use each to reach lower levels.

q)reverse (1 2 3;"abc";`Four`Score`and`Seven)
`Four`Score`and`Seven
"abc"
1 2 3

q)reverse each (1 2 3;"abc";`Four`Score`and`Seven)
3 2 1
"cba"
`Seven`and`Score`Four

A.78 rload

The function rload is a convenience function that loads a splayed table into a variable with the same name as the splayed directory. The usual way to do this is with get, whose result can be assigned to an arbitrary variable.

q)`:db/t/ set ([] c1:10 20 30; c2:1.2 2.2 3.3)
`:db/t/
q)rload `:db/t
`t
q)count t
3

q)t1:get `:db/t
q)count t1
3

A.79 rotate

The uniform binary rotate takes as its first parameter a long (length) and returns the result of rotating its second parameter by length positions. The rotation is to the left for positive length and to the right for negative length. For length 0, it returns the source.

q)2 rotate 1 2 3 4 5
3 4 5 1 2

q)22 rotate 1 2 3 4 5
3 4 5 1 2

A.80 rsave

The function rsave is a convenience function that splays a table variable to a directory with the same name. The preferred way to do this is with set, which allows the directory name to be specified.

q)t:([] c1:10 20 30; c2:1.2 2.2 3.3)
q)rsave `:db/t
`:db/t/
q)`:db/t1/ set t
`:db/t1/

A.81 rtrim

The unary rtrim takes a string argument and returns the result of removing trailing blanks.

q)rtrim " abc "
" abc"
q)rtrim " "
""

You can apply rtrim to a non-blank char but it fails for a blank char.

q)rtrim "a"
"a"
q)rtrim " "
k){$[~t&77h>t:@x;.z.s'x;" "=*x;(+/&\" "=x)_x;x]}
'type
_
1b
" "

A.82 scan

The function scan is a convenience function that renames the unary form of \ for those allergic to k.

q){x+2*y} scan 2 3 5 7
2 8 18 32
q)(+) scan 2 3 5 7
2 5 10 17

A.83 scov

The binary scov returns the statistical covariance of its numeric list arguments of the same length.

q)2 3 5 7 scov 4 3 0 2
-2.416667

It is equivalent to,

{cov[x;y]*count[x]%-1+count x}

A.84 sdev

The unary sdev returns the statistical standard deviation of its numeric list argument.

q)sdev 10 343 232 55
155.1322

It is equivalent to,

{sqrt var[x]*count[x]%-1+count x}

A.85 setenv

The binary setenv takes as first parameter a symbol representing the name of an OS environment variable and a string second parameter. It calls the underlying OS to set the named environment variable to the specified string value.

q) `FOO setenv "test"
q)getenv `FOO
_

A.86 sin

The atomic sin takes a float and returns its mathematical sine.

q)sin 0f
0f

q)pi:3.141592653589793
q)sin pi
1.224647e-16

q)pi%2
1.570796

q)sin pi%4
0.7071068

A.87 ss

The binary ss, for "string search", performs similar pattern matching as like against its first string parameter (source), looking for matches to its string second parameter (pattern). However, the result of ss is a list containing the position(s) of the matches of the pattern in source.

q)ss["Now is the time for all good men to come to";"me"]
13 29 38

q)"fun" ss "f?n"
,0

If no matches are found, an empty list of long is returned.

q)ss["ab";"z"]
`long$()

You cannot use * to match with ss.

q)"fun" ss "f*n"
'length

A.88 ssr

The triadic ssr, for "string search and replace", extends the capability of ss with replacement. The result is a string based on the first string parameter (source) in which all occurrences of the second string parameter (pattern) are replaced with the third string argument.

q)ssr["suffering succotash";"s";"th"]
"thuffering thuccotathh"

You cannot use * to match with ssr.

You can use the over iterator with ssr to replace multiple items.

q)(ssr/)["results_%div_%dept.csv"; ("%div";"%dept"); ("banking";"m&a")]
"results_banking_m&a.csv"

A.89 string

The unary string can be applied to any q entity to produce a textual representation. For atoms, lists and functions, the result of string is a list of char that does not contain any q formatting characters. Following are some examples.

q)string 42
"42"

q)string 6*7
"42"

q)string 424224242i
"424224242"

q)string `Zaphod
"Zaphod"

q)string {[a] a*a}
"{[a] a*a}"

The first example demonstrates that string is not atomic, because the result of applying it to an atom is a list of char.

Although string is not atomic, it is pseudo-atomic, in that it recurses through a list.

q)string 42 98
"42"
"98"
q)("42";"98")
"42"
"98"

This accounts for the un-intuitive behavior of string on an actual string.

q)string "Beeblebrox"
,"B"
,"e"
,"e"
,"b"
,"l"
,"e"
,"b"
,"r"
,"o"

Considering a list as a mapping, string acts on the range of the mapping. Viewing a dictionary as a generalized list, we conclude that the action of string on a dictionary should also apply to its range.

q)string 1 2 3!100 101 102
1| "100"
2| "101"
3| "102"

A table is the flip of a column dictionary, so we expect string to operate on the range of the column dictionary.

q)string ([] a:1 2 3; b:`a`b`c)
a    b
---------
,"1" ,"a"
,"2" ,"b"
,"3" ,"c"

Finally, a keyed table is a dictionary, so we expect string to operate on the value table.

q)string ([k:1 2 3] c:100 101 102)
k| c
-| -----
1| "100"
2| "101"
3| "102"

A.90 sublist

The binary sublist retrieves a sub-list of contiguous items from a list. The first parameter is a simple list of two non-negative integers: the first item is the starting index (start); the second item is the number of items to retrieve (count). The second parameter (target) is a list or dictionary.

q)0 3 sublist 10 20 30 40 50
10 20 30
q)0 10 sublist "abcdef"
"abcdef"
q)0 2 sublist `a`b`c!10 20 30
a| 10
b| 20

Tip

The construct 0 n sublist … is preferred to n# … for extracting the first n items from a list since the latter will repeat the initial items if necessary to reach the specified count. This is ignored surprisingly often in practice.

A.91 sum

The aggregate sum is (+/), which applies + cumulatively across a numeric list and returns the final result.

q)sum 1+til 100
5050

q)sum 100?100.
5387.56

It is possible to apply sum to a nested list provided the sublists conform. In this case, the result conforms to the sublists and the sum is calculated recursively on the sublists.

q)sum (1 2;100 200;1000 2000)
1101 2202

A.92 sums

The uniform sums is (+\), which applies + cumulatively across a list of numeric type and returns the intermediate results.

q)sums 1+til 10
1 3 6 10 15 21 28 36 45 55

A.93 sv

The binary sv – "scalar from vector" – has several forms. It is usually written infix since that is easier to read.

The first form takes a char as its first parameter and a list of strings (source) as its second. It returns a string that is the concatenation of the strings in source, separated by the specified char.

q)"," sv ("Now";"is";"the";"time";"")
"Now,is,the,time,"

When sv is used with an empty symbol as its first parameter and a list of symbols as its second (source), the result is a symbol in which the items in source are concatenated with a separating dot. This is useful for q context names.

q)` sv `qalib`stat
`qalib.stat

When sv is used with an empty symbol as its first parameter and a symbol second parameter (source) whose first item is a file handle, the result is a symbol in which the items in source are concatenated with a separating /. This is useful for fully qualified path names.

q)` sv `:`q`tutorial`draft`3
`:/q/tutorial/draft/3

When sv is used with an empty symbol as its first parameter and a list of strings as the second parameter, it concatenates the strings, inserting a new line character after each.

q)` sv ("abc";"de")
"abc\nde\n"

When sv is used with a long first parameter (base) that is greater than 1, together with a second parameter of a simple list of place values expressed in base, the result is a long representing the converted base 10 value.

q)2 sv 101010b
42

q)10 sv 1 2 3 4 2
12342

q)256 sv 0x001092
4242

More precisely, the last version of sv evaluates the polynomial,

(d[n-1]*b exp n-1) + ... +d[0]

where d is the list of digits, n is the count of d, and b is the base.

Thus, we find,

q)10 sv 1 2 3 11 2
12412

q)-10 sv 2 1 5
195

A.94 svar

The unary svar returns the statistical standard variance of its numeric list argument.

q)svar 2 3 5 7
4.916667

It is equivalent to,

{var[x]*count[x]%-1+count x}

Warning

The svar function was added in version 3.2 meaning that finance apps can no longer use a variable name svar for stress value at risk in this and later versions.

A.95 system

The unary system takes a string argument and executes it as a q command, if recognized, or an OS command otherwise. This is convenient to execute q commands programmatically. The normal result of the command becomes the return value of the evaluation.

Do not include the initial \ as you would when executing the command from the q session prompt.

q)system "c"
25 80i
q)system "c 40 400"
q)system "c"
40 400i

q)system "cd /data"
q)system "pwd"
"/data"

A.96 Take/Reshape #

The binary operator # has several forms.

When the first parameter of # is an integer atom count, it returns count items from the atom or list second parameter (source). The items are extracted from the head when source is positive and from the tail when count is negative. When count is zero, the result is an empty list of the same type as the first item in source. When count is greater than the count of source, items are repeatedly drawn from source as described until count items are obtained.

The result of Take is always a list.

q)2#10 20 30 40 50
10 20

q)-3#10 20 30 40 50
30 40 50

q)0#10 20 30 40 50
`long$()

q)3#42
42 42 42

q)10#10 20 30 40 50
10 20 30 40 50 10 20 30 40 50

The idiom 0# atom is a common way to create an empty list of the type of atom.

q)meta ([] a:0#0; b:0#`)
c| t f a
-| -----
a| j
b| s

Since a dictionary is ordered, take also applies to dictionaries and, by extension, to tables and keyed tables.

q)2#`a`b`c!10 20 30
a| 10
b| 20

q)-2#([] c1:`a`b`c; c2:10 20 30)
c1 c2
-----
b 20
c 30

q)2#([k:`a`b`c] v:10 20 30)
k| v
-| --
a| 10
b| 20

Another form of Take has first parameter a list of keys and the second parameter a dictionary. The result is the sub-dictionary for these keys.

q)`a`c#`a`b`c!10 20 30
a| 10
c| 30

Since a table is a column dictionary this form also applies to a table with a list of column names.

q)`c1`c2#([] c1:`a`b`c; c2:10 20 30; c3:1.1 2.2 3.3)
c1 c2
-----
a 10
b 20
c 30

Using an anonymous table is a nifty way to generate a list of keys for takes with a keyed table.

q)([] k:`a`c)#([k:`a`b`c] v:10 20 30)
k| v
-| --
a| 10
c| 30

Another form of Take has first parameter a list (r; c) of two non-negative integers and second parameter a list or atom (source). The result is a nested list with r items, each having c items, drawn sequentially from the beginning of source, repeatedly if necessary. Otherwise put, the result is an r by c array obtained by reshaping source.

q)2 3#10 20 30 40 50
10 20 30
40 50 10

If in the previous form either r or c is null, the result is a nested list obtained by reshaping the second parameter into the specified number of rows or columns. This is generally a ragged array.

q)2 0N#10 20 30 40 50
10 20 30
40 50
q)0N 3#10 20 30 40 50
10 20 30
40 5

A.97 tan

The atomic tan takes a float and returns its mathematical tangent.

q)tan 0
0f

q)pi:3.141592653589793
q)tan pi
-1.224647e-16

q)tan pi%2
1.633124e+16

q)tan pi%4
1f

q)tan pi%8
0.4142136

The function tan is equivalent to

(sin x)%cos x

A.98 til

The unary til takes a non-negative long parameter (n) and returns a list of n consecutive longs starting with 0.

The argument is not included in the result.

q)til 4
0 1 2 3
q)count til 1000000
1000000

The result of til is always a list.

q)til 0
`long$()
q)til 1
,0

To generate other regular numeric sequences, perform vector operations on the result of til.

q)1+til 10
1 2 3 4 5 6 7 8 9 10
q)2*til 10
0 2 4 6 8 10 12 14 16 18
q)1+2*til 10
1 3 5 7 9 11 13 15 17 19
q).5*til 10
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
q)2015.01.01+til 5
2015.01.01 2015.01.02 2015.01.03 2015.01.04 2015.01.05

A.99 trim

The unary trim takes a string argument and returns the result of removing leading and trailing blanks.

q)trim " abc "
"abc"

The function trim is equivalent to,

{ltrim rtrim x}

A.100 ungroup

The unary ungroup flattens tables with nested columns that are the result of a select query that groups without aggregation, or of xgroup. Of course you can also apply it to tables with nested columns of the same form that are otherwise generated.

The action of ungroup expands each key group, resulting in one row for each item in the nested columns. The fields in a result row are drawn from corresponding positions across the nested columns.

We use the distribution example.

q)\l sp.q
+`p`city!(`p$`p1`p2`p3`p4`p5`p6`p1`p2;`london`london`london`london`london`lon..
(`s#+(,`color)!,`s#`blue`green`red)!+(,`qty)!,900 1000 1200
+`s`p`qty!(`s$`s1`s1`s1`s2`s3`s4;`p$`p1`p4`p6`p2`p2`p4;300 200 100 400 200 300)

q)sp
s  p  qty
---------
s1 p1 300
s1 p2 200
s1 p3 400
s1 p4 200
s4 p5 100
s1 p6 100
s2 p1 300
s2 p2 400
s3 p2 200

q)select s, qty by p from sp
p | s               qty
--| -------------------------------
p1| `s$`s1`s2       300 300
p2| `s$`s1`s2`s3`s4 200 400 200 200
p3| `s$,`s1         ,400
p4| `s$`s1`s4       200 300
p5| `s$`s4`s1       100 400
p6| `s$,`s1         ,100

q)ungroup select s, qty by p from sp
p  s  qty
---------
p1 s1 300
p1 s2 300
p2 s1 200
p2 s2 400
p2 s3 200
p2 s4 200
p3 s1 400
p4 s1 200
p4 s4 300
p5 s4 100
p5 s1 400
p6 s1 100

q)`p xgroup sp
p | s               qty
--| -------------------------------
p1| `s$`s1`s2       300 300
p2| `s$`s1`s2`s3`s4 200 400 200 200
p3| `s$,`s1         ,400
p4| `s$`s1`s4       200 300
p5| `s$`s4`s1       100 400
p6| `s$,`s1         ,100

q)ungroup `p xgroup sp
p  s  qty
---------
p1 s1 300
p1 s2 300
p2 s1 200
p2 s2 400
p2 s3 200
p2 s4 200
p3 s1 400
p4 s1 200
p4 s4 300
p5 s4 100
p5 s1 400
p6 s1 100

Tip

Observe that ungroup is not quite the inverse of a grouping operation. Since grouping sorts on the keys, a subsequent ungroup returns the original records sorted by the grouped column(s).

A.101 union

The binary union returns the set theoretic union of its two list parameters. The result has the distinct items of the two parameters in order of their first appearance.

q)1 1 2 3 union 2 2 3 4
1 2 3 4
q)"a good time" union "was had by all"
"a godtimewshbyl"

Since a table is a list of records, union can be used to perform the analogue of SQL UNION for tables having the same schemas.

q)([] c1:`a`b`c; c2:10 20 30) union ([] c1:`c`d; c2:300 400)
c1 c2
------
a  10
b  20
c  30
c  300
d  400

Use uj for tables that do not have matching schemas – see §9.9.7.

A.102 upper

The atomic upper takes a char, string or symbol argument and returns the result of converting any alpha characters to upper case.

q)upper `a
`A

q)upper `a`b
`A`B

q)upper "a Bc42De"
"A BC42DE"

A.103 value

The function value also vies for the title as the most overloaded q operator.

The functions value and get are identical. Conventionally get is used with file I/O.

When value is applied to a symbolic variable name, it returns the associated value.

q)a:42
q)value `a
42

When applied to a dictionary, value returns the values of the dictionary. This includes the special case of the value table of a keyed table.

q)value `a`b`c!10 20 30
10 20 30

q)kt:([k:`a`b`c] v:10 20 30)
q)value kt
_

A common idiom uses value to extract the columns lists from a table.

q)value flip ([] c1:`a`b`c; c2:10 20 30)
a b c
10 20 30

When value is applied to an enumerated value, it de-enumerates it. This is handy when switching between kdb+ databases having different sym domains.

q)sym:()
q)ev:`sym?`a`x`b`y`a`c`b
q)ev
`sym$`a`x`b`y`a`c`b
q)value ev
`a`x`b`y`a`c`b

Given a function, value returns the following list,

(bytecode;parms;locals;(context;globals);constants[0];...;constants[n];defn)

q)f:{[a;b]d::neg c:a*b+42;c+e}
q)value f
0xa0794178430316220b048100028276410004
`a`b
,`c
``d`e
42
"{[a;b]d::neg c:a*b+42;c+e}"q)sym:()

Given an alias (aka “view”), value returns the list

(cached value;parse tree;dependencies;definition)

When the evaluation of the expression in the alias is as yet deferred, the cached value is ::.

q)a:42
q)b::a+1            / evaluation is deferred
q)value `.[`b]      / inspect global context
::
(+;`a;1)
,`a
"a+1"

q)b                 / force evaluation
43
q)value `.[`b]
43
(+;`a;1)
,`a
"a+1"

Given a projection, value returns a list containing the function body followed by the supplied arguments.

q)f:{x+y+z}
q)value f[2;3;]
{x+y+z}
2
3
::
q)value +[2]
_

Given a function that is the result of applying a k iterator, value returns the argument to the iterator, viewing the latter as a higher-order function.

q)value (+/)
+
q)value (+')
+
q)f:{x+y}
q)value (f')
{x+y}

The final and most powerful form of value is that it is the q interpreter itself. It will evaluate a string as if it had been typed at the console.

q)value "6*7"
42
q)value "{x*x} til 10"
0 1 4 9 16 25 36 49 64 81

It will also evaluate a parse tree – i.e., a (potentially) nested list of functions followed by arguments. If the function is specified by name, that name is resolved first.

q)value (*; 6; 7)
42
q)f:{x*y}
q)value (`f; 6; 7)
42

Warning

This use of the value function is a powerful feature that allows q code to be written and executed on the fly. This can expose your program to potential attack unless done very carefully. If abused, it can quickly lead to unmaintainable code. (The spellchecker suggests "unmentionable" instead of "unmaintainable." How did it know?)

A.104 var

The aggregate var takes a numeric list and returns the mathematical variance of the items as a float.

q)var 42 45 37 38
13.66667

The function var is equivalent to

{(avg[x*x]) - (avg[x])*(avg[x])}

A.105 vs

The binary vs – ”vector from scalar” – has several versions. It is usually written infix for ease of reading.

The first form takes a char as its first parameter and a string (source) as its second parameter. It returns a list of strings containing the tokens of source as delimited by the specified char.

q)" " vs "Now is the time "
"Now"
"is"
"the"
"time"
""

You probably want to trim the input in the above example.

When vs is used with an empty symbol as its first parameter and a symbol second parameter (source) containing dots, it returns a simple symbol list obtained by splitting source along the dots.

q)` vs `qalib.stat
`qalib`stat

When vs is used with an empty symbol as its first parameter and a symbol representing a fully qualified file name as the second parameter, it returns a simple list of symbols in which the first item is the path and the second item is the file name. Note that in this usage, vs is not quite the inverse of sv.

q)` vs `qalib.stat
`qalib`stat

When vs is used with a null of binary type as the first parameter and an value of integer type as the second parameter (source), it returns a simple list whose items comprise the digits of the corresponding binary representation of source.

q)0x00 vs 4242
0x0000000000001092

q)10h$0x00 vs 8151631268726338926j
"q is fun"

q)0b vs 42
0000000000000000000000000000000000000000000000000000000000101010b

The 0b version can be used to display the internal representation of special values.

q)0b vs 0W
0111111111111111111111111111111111111111111111111111111111111111b
q)0b vs -0W
1000000000000000000000000000000000000000000000000000000000000001b

A.106 wavg

The binary wavg takes two numeric lists of the same count and returns the float average of the items in the second parameter, weighted by the items of the first.

q)1 2 3 4 wavg 500 400 300 200
300f

The function wavg is equivalent to,

{(sum x*y)%sum x}

It is possible to apply wavg to a nested list provided all sublists of both parameters conform. In this context, the result conforms to the sublists and the weighted average is calculated recursively across the sublists.

q)(1 2;3 4) wavg (500 400; 300 200)
350 266.6667

A.107 where

The unary where has two forms.

The first form returns the indices of 1b in a boolean list, or the keys associated to 1b in a dictionary with boolean values.

q)where 101010b
0 2 4
q)where `a`b`c`d`e`f!101010b
`a`c`e

This is useful when the boolean mapping is the result of a predicate.

q)where ","="First, second, third"
5 13

q)@[L; where L=","; :; "\t"]
"First\t second\t third"

q)where null `a`b`c`d`e`f!1 0n 3 4 0n 6
`b`e

When the argument of where is a list of non-negative long (c), the result is a list obtained by catenating c[i] copies of i, for each item in c.

q)where 2 1 3
0 0 1 2 2 2
q)where 2 1 0 3 2
0 0 1 3 3 3 4 4
q)where 4#1
0 1 2 3

Zen Moment

The second form of where generalizes the form on a boolean list.

A.108 within

The binary within is atomic in its first parameter (source) and takes a second parameter that is a list (b;e) that are comparable. It returns a boolean representing whether source is greater than or equal to b and less than or equal to e. It will do type promotion on arguments.

q)3 within 2 5
1b

q)(til 7)within 2 5
0011110b

q)"c" within "az"
1b

q)2015.03.14 within 2015.02.01 2015
1b

q)`ab within `a`b
1b

q)100 within "aj"
1b

Ensure the items in the second argument are in increasing order or you will get no matches.

A.109 wsum

The binary wsum takes two numeric lists of the same count and returns the float sum of the items of second parameter weighted by the items of the first.

q)1 2 3 4 wsum 500 400 300 200
3000f

The function wsum is equivalent to,

{sum x*y}

It is possible to apply wsum to a nested list provided all sublists of both arguments conform. In this case, the result conforms to the sublists and the weighted sum is calculated recursively across the sublists.

q)(1 2;3 4) wsum (500 400;300 200)
1400 1600

A.110 xbar

The uniform binary xbar takes first parameter a non-negative numeric atom (width) and returns the largest integral multiple of width that is less than or equal to its second parameter. Visually, the result is the left endpoint of the band of width width that contains the second parameter. Since xbar is atomic in the second parameter, this is an effective way to bin the items in a numeric list.

q)1 xbar 0 .5 1 1.5 2 2.5 3 3.5
0 0 1 1 2 2 3 3f

q)1.5 xbar 0 .5 1 1.5 2 2.5 3 3.5
0 0 0 1.5 1.5 1.5 3 3

q)2 xbar 0 .5 1 1.5 2 2.5 3 3.5
0 0 0 0 2 2 2 2f

Since a q month is actually the count of months since the millennium, you can use xbar to determine quarters. Recall that a month is equal to the first day of the month.

q)`date$3 xbar `month$2015.11.19 / beginning of that quarter
2015.10.01
q)`date$3+3 xbar `month$2015.11.19 / beginning of next quarter
_
q)-1+`date$3+3 xbar `month$2015.11.19 / end of that quarter
_

A.111 xprev

The binary xprev takes a long as its first parameter (shift) and shifts the list in its second parameter by shift positions. When shift is 0 or positive, the shift is forward; otherwise it is backward. The beginning – respectively end – of the list is filled with shift null (or empty) items.

q)2 xprev 10 20 30 40 50
0N 0N 10 20 30

q)-2 xprev 10 20 30 40 50
30 40 50 0N 0N

q)2 xprev (1 2 3; 4 5; enlist 6; 7 8)
`long$()
`long$()
1 2 3
4 5

A.112 xrank

The binary xrank has first parameter a positive long (n) and is uniform in its second parameter (source). It returns a list of long containing the n-quantile into which each item of source falls, considering source as a distribution. For example, choosing n to be 4 returns quartiles and 100 yields percentiles.

q)4 xrank 30 10 40 20 90
1 0 2 0 3

q)100 xrank 200?1000.
35 14 30 96 19 68 39 21 91 73 27 79 30 66 27 79 75 8 99 84 84 77 ..
Back to top