6. Functions
6.0 Overview
In this chapter, we cover functions in depth. Before diving in, you may wish to review the Mathematics Refresher in Chapter 0 for the approach and terminology we adopt.
We describe various built-in functions along the way in this tutorial. Sometimes we shall use a built-in function with minimal explanation; simply look it up in Appendix A, which contains specifics and examples of the most common q built-in functions.
6.1 Function Specification
The notion of a function in q corresponds to a (mathematical) map that is specified by an algorithm. A function is a sequence of expressions to be evaluated, having a return value and optional input parameters. Application of a function is the process of evaluating the expressions in sequence, substituting actual argument values for any formal parameters. The result of the final evaluation, should there be one, is the function's output value. A function always returns a value even if it is a nil value that isn't displayed at the console.
Advanced
Because a q function can modify global variables, q is not a pure functional language. A mathematical function can never reach outside its own body and have side effects. You should minimize such behavior for code maintainability.
6.1.1 Function Definition
The syntax of function definition is a matching pair of braces { and } enclosing (optional) parameters and a sequence of expressions. In contrast to statically typed languages, the input parameters and the return value are not explicitly typed. In fact, they don't even need to be declared.
Important
A q function does not have a name, in contrast to many traditional languages. It is what is called an "anonymous function" or a "lambda expression" ("lambda" for short) in those languages. This is a reference to the Lambda Calculus created by Alonzo Church in order to abstract and axiomatize the compositional behavior of mathematical functions.
Info
We will not use the terminology "lambda" in this tutorial since it is superfluous to "function" in q. But you can use it to sound sophisticated.
Following is a full specification of a function that squares its input. In this case, the function simply returns the value of the last expression evaluated. We have added optional whitespace after the parameter declaration for readability.
q){[x] x*x}
Apply this function by following it with an argument enclosed in square brackets. This causes the supplied argument to be passed for all the parameter x in the function body, the expression(s) to be evaluated, and the result is returned as the output value.
q){[x] x*x}[3]
9
Important
Unlike a pure functional programming language, the parameters of a q function are actually local variables and can be modified within the function body. We consider this a bad practice to be avoided for code maintainability.
A function is a first class value - i.e., it is data just like an integer or float. In particular, a function can be assigned to a variable, whereupon it is associated with the name of the variable. This variable can be used in place of the function.
q)f:{[x] x*x}
q)f[3]
9
Tip
To see a sorted list all the function names in the current workspace issue the command \f. In a fresh q session,
q)f:{x*x}
q)g:{x+y}
q)\f
`s#`f`g
6.1.2 Function Notation and Terminology
The formal notation for function definition is,
{[p1;...;pn]e1; ...; en}
The optional p1, ... , pn are formal parameters. The expressions e1, ... , en which presumably involve the parameters, represent the steps of an algorithm that are evaluated in order. Note that while the expressions are sequenced left-to-right, the expressions themselves are always evaluated right-to-left.
For readability, in this text we shall normally insert optional whitespace after the square bracket that closes the parameter list, as well as after each semicolon separator. Other styles may differ.
Important
The semicolons in the body are not terminators; better to think of them as separators. Do not place a semicolon after
the last expression unless you deliberately want to suppress a return value. In fact ; is an operator that evaluates its left operand and discards the result, then evaluates the right operand and returns that value. If you meditate on this, you will realize that this is precisely the behavior of the separators in a function body.
q)a:2*3;a*7
42
q)a
6
q)f:{[x] a:2*x; a*7}
q)f[3]
42
The number of formal input parameters (implicit or explicit) is the function valence or rank. Most common are unary (valence 1) and binary (valence 2).
A nullary function in q is a function that has no meaningful input. This is similar to a function taking void in C. In functional programming this is represented as a function defined on a type called "unit" having a single value, often written (). In q it is written with an empty parameter declaration,
f:{[] ... }
and is applied with no argument - i.e., f[]. In this case the function either returns a constant or has side effects, meaning that it manipulates resources outside its body.
{[] 42} / pure function returns constant 42
{[] a*a} / impure function references global a
The maximum valence permitted as of this writing (July 2025) is 8; specifying more than eight parameters will cause an error.
q)f{[p1; p2; p3; p4; p5; p6; p7; p8; p9] 42}
'params
[0] f{[p1; p2; p3; p4; p5; p6; p7; p8; p9] 42}
^
Tip
You can circumvent this restriction by encapsulating multiple parameters in a list or dictionary.
The output value of a function written in functional programming style is the value of the last expression in its body. This is the recommended style of writing q since it results in linear flow that is easy to follow and debug.
{[x] x*x}
{[x;y] a:x*x; b:y*y; r:a+b; r}
This convention can be supplanted by using a unary overload of : which immediately terminates function evaluation and returns the value of the expression to its right. This style is discouraged.
{[x] :x*x} / unnecessary use of :
{[x] a:1; :x*x; b:3} / dead code
Tip
Q functions should be compact and modular: each function should perform a well-defined unit of work. Due to the power of q operators and built-in functions, many functions are one line. When a function exceeds 20 statements, you should look to refactor it.
Advanced
Arthur Whitney, the creator of q, said that q is not ambivalent. He doesn't mean that q is decisive. Rather, in contrast to k, q operators and built-in functions are not overloaded on valence, meaning that the same function does not have different behavior in a unary and binary form. That said, q operators and built-in functions are (multiply) overloaded on just about anything else, including the type, sign or rank of arguments.
6.1.3 Function Application
The notation for function application parallels that of the formal parameters in its definition. The function is followed by square brackets enclosing arguments separated by semicolons. Application causes the expressions in the function body to be evaluated in sequence, substituting the value of each argument for the corresponding formal parameter.
q){[x] x*x}[3]
9
q)f:{[x] x*x}
q)f[4]
16
q){[x;y] x+y}[3; 4]
7
q)g:{[x;y] x+y}
q)g[3;4]
7
Function application in q is strict, meaning that expressions in arguments are evaluated before substitution is performed.
q)f:{[x] x*x}
q)f[0N!3+1]
4
16
Tip
The valence of a function is also called its rank. Application with too many arguments generates a 'rank error. This doesn't mean your code stinks (even if it does).
q) {[x] x*x}[3;4]
'rank
Apply a nullary function with no arguments or with an arbitrary argument, which is ignored.
q)const42:{[] 42}
q)const42[]
42
q)const42[98.6]
42
q)const42 ()
42
6.1.4 Functions with No Argument or No Return Value
Sometimes a function has no meaningful input value. For example, you call a function that generates a GUID or simply returns a value. In this case the function is used purely for side effects. It is called nullary and is invoked with no argument.
q){42}[]
42
However, things are not quite as they seem. There is an implicit nil argument whose display we can force to stdout.
q){1 .Q.s1 x; 42}[]
::42
Another case of using a function purely for side effects is a function that has has no meaningful return value. For example, it could send off a message asynchronously without waiting for acknowledgement of receipt; or it may update a database or a global table. To emphasize that there is no meaningful return value you can achieve the equivalent of a void return in C. Make the expression after the last semicolon in the function body empty.
q)fvoid:{[x] `a set x;}
q)fvoid 42
q)
Once again things are not as they seem. Even a function with no explicit return value does return a value. By now you will guess that the actual return value is the nil item ::. Again we can display its value with .Q.s1.
q).Q.s1 fvoid 42
"::"
So we see that that behind the scenes, a q function is always a true mathematical function in that it has an input and an output value.
Tip
A common error is to use semicolons in a function body as expression terminators rather than as sequencers. This can happen unintentionally while editing the function body and you inadvertently leave a trailing semicolon. This effectively makes the function return nil since the last expression is empty. This can be quite vexing to debug in a complex code.
6.1.5 We Don't Need No Stinkin' Brackets
There is an alternative notation for application of unary functions (only) that dispenses with brackets. In the case of a function assigned to a variable, separate the name from the argument with whitespace - customarily a single blank.
q){[x] 2*x} 42
84
q)f:{[x] x*x}
q)f 5
25
This convention for function application is called juxtaposition and is the norm in functional programming. While it may be jarring for newcomers, the decrease in superfluous punctuation - i.e., program noise - is worth the effort.
6.1.6 Application by Name
When a function has been assigned to a global variable, it can be applied by name - i.e., use the symbolic name of the variable in place of the variable itself.
q){[x] 2*x} 42
84
q)
q)f:{[x] x*x}
q)f 5
25
q)f:{x*x}
q)f[5]
25
q)`f 5
25
q).my.name.space.f:{2*x}
q)`.my.name.space.f[5]
10
This works because the global variable is implemented as a symbolic-name/function pair in a dictionary. Referring to the variable is actually lookup by name. Consequently application by name does not work for local variables or q system variables. As a matter of style we don't recommend using this unless you have a good reason.
Tip
Do not confuse application by the function name with call-by-name in which parameters are passed by name.
6.1.7 Implicit Parameters
If, like mathematicians, you are satisfied with parameters x, y and z, you can omit the formal parameter declaration and the brackets. Three implicit parameters x, y and (in that order) are automatically defined and available in a function's body when no parameters are explicitly declared. Thus, the following two specifications result in equivalent input-output relations.
{[x] x*x}
{x*x}
As do,
{[x; y] x+y}
{x+y}
{[x; y; z] x+y+z}
{x+y+z}
When using implicit parameters, x is always substituted by the first actual argument, y the second and z the third. The following function f only evaluates partially unless it is called with three parameters.
q)g:{x+z} / likely meant x+y; requires 3 args in call
q)g[1;2] / still waiting for 3rd arg—i.e., is a projection
{x+z}[1;2]
q)g[1;2;3] / 2nd arg is provided and ignored
4
The following is permitted but obtuse.
{[x] x*x}
Tip
Only use the names x, y and z for the first three parameters of a function, either explicit or implicit. Any other use leads down the path to perdition.
q)evil:{[y;z;x] ...}
6.1.8 Anonymous Functions and Lambda Expressions
Many traditional programming languages include the function name as part of its definition. A function without a name is then called an anonymous function. In functional programming, functions are all anonymous and are called lambda expressions or lambdas for short. All q functions are anonymous until assigned to a variable whose name is then associated with the function.
Tip
We do not use the term "lambda" in this text since "function" already suffices in q. Anyone who has studied the lambda calculus will find the casual use of "lambda" pretentious.
An anonymous function can be useful as an in-line macro where it would otherwise be awkward to substitute. For example, you may see,
f{[...] ...; v:{...}[...]; ...}
Some find it more readable to separate this out as a separate function.
Another use for anonymous functions is a rudimentary form of dynamic dispatch, in which functions are placed in a collection (e.g., list, dictionary, table) and then selected dynamically (i.e., at runtime) rather than by an intrinsic name.
q)power:({1}; {x}; {x*x}; {x*x*x})
q)selected:2
q)power[selected][5]
25
6.1.9 The Identity Function ::
When :: is used with function application syntax it is the identity function - i.e., it returns its input as output. This can be useful for sophisticated forms of recursion.
Tip
The naked identity function cannot be used with juxtaposition; it must have brackets or be enclosed in parentheses.
q)::[42]
42
q)::[`a`b`c]
`a`b`c
q):: 42 / error
'
q)(::)42
42
6.1.10 Functions Are Data
The q data entities we have met until now are atoms of various types, lists and dictionaries. For those new to functional programming, it may come as a surprise that functions are also data. A function can be passed around just like an integer or float - e.g., as an item in a list.
q)(1; 98.6; {x*x})
1
98.6
{x*x}
q)f:{x*x}
q)(f; neg)
{x*x}
-:
q)(f; neg)[0]
{x*x}
q)(f; neg)[1; 5] / good q test question
-5
A function can also be used as the input or output value of another function. For example, here is a function that expects a function and a value and applies the function to the value.
q)apply:{x y}
q)sq:{x*x}
q)apply[sq; 5]
25
A function that operates on other functions is called a higher order function. Higher order functions are a very powerful concept in functional programming and play a key role in q. The derivative and indefinite integral are higher-order functions in elementary calculus.
6.1.11 Filter Functions
We come to the next level of pattern matching, filter functions.
Tip
Advanced: Those who know should recall the syntax of table definition
([] c:expr; ...)
in which expr is any (valid) q expression that will be evaluated to create a column list.
In the case of filter functions, the pattern takes the form,
(v:f ... )
Here v is a name pattern for assignment and f is a unary function to be applied to the incoming data prior to assignment. This is powerful so we will start with a simple example that ensures the assigned value is non-negative.
q)(v:abs):42
q)v
42
q)(v:abs):-42
q)v
42
Here is an example that uses a projected operator (see 6.4).
q)(a; b:10+; c:100+):1 2 3
q)a,b,c
1 12 103
Here is an example that uses a filter function to coerce numeric items to float type.
q)(a:"f"$; b:"f"$; c:"f"$; d:"f"$; e:"f"$):(1b; 1h; 1i; 1j; 1f)
q)a,b,c,d,e
1 1 1 1 1f
Recall the discussion of using pattern matching to type check items of a list assignment in 3.12.2. Here we specify symbol type for name and long type for age.
q)(name:`s; age:`j):(`nuba; 11)
q)name,age
`nuba
11
And now we come to the moment the q community has been waiting for: q 4.1 introduces type checking of function arguments. Observe that enclosing parentheses on the pattern aren't necessary here due to the brackets. This capability shifts responsibility for bad data to the caller rather than the callee.
q)showInfo:{[name:`s; age:`j] string[name]," is ",string age}
q)showInfo[`jeffry; 42]
"jeffry is 42"
q)showInfo[`jeffry; 98.6]
'type
[4] showInfo:{[name:`s; age:`j] string[name]," is ",string age}
^
This is truly a game changing feature for our ability to develop robust q libraries.
You can intermix parameters with and without type checking.
q)showInfo:{[name:`s; weight] string[name]," is ",string weight}
q)showInfo[`devi; 10]
"devi is 10"
q)showInfo[`devi; 10.5]
"devi is 10.5"
We can also use pattern filter functions on function arguments. Here is a particularly useful filter pattern applied to a function parameter.
q){x[0]} 42 / body performs list op
'Cannot write to handle 42. OS reports: Bad file descriptor
[1] {x[0]}
^
q){[x:(),] x[0]} 42 / guarantee list
42
How many times has each of us written this idiom in the first line of a function to ensure the supplied argument is a list?
Here is a proactive way to coerce argument type instead of relying on type checking to signal an error.
q)showInfo:{[name:`s; age:`long$] string[name]," is ",string age}
q)showInfo[`jeffry; 98.6]
"jeffry is 99"
6.2 Call by Name
Ordinary function application in q uses call-by-value which is eager, meaning that arguments are expressions that are reduced to values before substitution. All operations are performed on copies and any original input variables are undisturbed. When the size of the input data is large, this copy may be prohibitive. In this case, there is an alternative method of application, call-by-name, in which names of (global) variables are passed instead of their values.
There is no special semantics for call-by-name in q. Instead, the function implementation simply presumes a symbol with the name and handles it appropriately. One example of a function that uses call by name is the built-in function get, which returns the value of the global variable whose symbolic name is passed.
q)a:42
q)get `a
42
Another example is binary set, which does the actual work of global assignment. It expects the name of a global variable and the value to be assigned.
q)`a set 43
`a
q)a
43
The result of a built-in call-by-name function is the input symbol. This enables call-by-name functions to be chained - i.e., composed.
Tip
Do not confuse the return value of call by name with an error message. The former begins with a back-tick whereas the latter begins with tick, usually called a single quote or apostrophe in normal speech. The author confesses that this still occasionally causes a start.
6.3 Local and Global Variables
6.3.1 Defining Local and Global Variables
A variable that is assigned with : inside in a function body is called a local variable. For example, a is a local variable in the following function.
{a:42; a+x}
Some notes on local variables
- A local variable exists only for the duration of an application. There is no notion of a static variable that maintains its value across applications.
- A local variable is not visible outside its immediate scope of definition.
- A local variable cannot be used as an argument to a q function that uses call-by-name.
-
A local variable is not visible within the body of a local function defined in the same scope. Otherwise put,
q does not have lexical scoping.
As of this writing (July 2025), the maximum number of local variables permitted in a function is 110.
A variable assigned outside any function definition is called a global variable. Global variables are visible inside any function body.
q)b:7
q)f:{b*x}
q)f[6]
42
Tip
As of this writing (July 2025), the maximum number of global variables that can be referenced in a function is 110. If this is a problem, redesign your code. Also, the maximum number of literal constants in a program is 239 minus the total number of locals and globals.
6.3.2 Assigning Global Variables within a Function
Although functional programming purists will cringe, you can assign a global within a function body using double colon ::, provided there is no local variable or parameter with the same name.
b:6
q)b:6
q)f:{b::7; x*b}
q)f[6]
42
q)b
7
Tip
If there is a local variable of the same name as the global you intend to assign, double-colon assigns the local, not the global.
b:6
q)b:6
q)f:{b:42; b::x; b}
q)f[98]
98
q)b
6
For this reason, we recommend against using :: for global assignment. Instead, use set which always works and doesn't get hung up on a local-global name collision. As we saw earlier, set uses call-by-name.
q)a:42
q)f:{a:98.6; `a set x; a}
q)f 43
98.6
q)a
43
6.4 Projection
Projection is the result of specifying only some of the parameters in function application, yielding a function of the remaining parameters.
6.4.1 Function Projection
In some use cases, a portion of the arguments to a multivalent function may be fixed while the remaining arguments vary over repeated applications. A common example is where a date is the last parameter and you apply the function for many dates keeping all other inputs fixed. In this situation, you can specify the fixed arguments and leave the others for later. This is called projecting the function onto the remaining arguments. Notationally, projection is a partial application in which some arguments are supplied and the others are omitted - i.e., nothing is in those slots.
For example, a homegrown binary function that adds its arguments can be projected onto the second parameter by providing just the first argument.
q)add:{x+y}
q)add[42;]
{x+y}[42;]
You can see from the console that q views the projection as a partial application that awaits its remaining argument. The projected function is a legitimate unary function, which adds its input to the constant 42.
q)add[42;][3]
45
Since functions are first class, we can assign a projection to a variable and then apply it, none the wiser of its origin.
q)g:add[42;]
q)g[3]
45
Projection is related to currying in functional programming (named after Haskell Curry who did important early work on the theory of combinators). Both provide realizations of the mathematical observation that a function of two parameters is equivalent to a function of one parameter that returns a function of the other. This extends to multiple parameters as well. In q you can simply project out successive parameters to emulate functional programming in which all functions are unary.
q)add3:{x+y+z}
q)add3[2][3][4]
9
Here we can view the result of add3 applied to 2 as a function that when applied to 3 returns a function that when applied to 4 yields add[2;3;4].
Style Recommendation
We recommend that you do not omit trailing semicolons in projections, as this can obscure the intent of code. For example, if you came upon the following while reading q code, how would you know that mystery is actually a binary function ... without the comment line?
q)mystery:f[2]
/ surprise! f is actually f:{x+y+z}
The brackets denoting a function projection in the first parameter are optional, but if you want to apply this projection they are required; otherwise, you will be projecting on a single vector argument.
q){x+y} 42
{x+y}[42]
q){x+y} 42 3
{x+y}[42 3]
q){x+y}[42;] 3
45
Choice of notation is a matter of coding style.
Important
The function body in a projection is captured at the time it is first encountered by the interpreter. Thus if the underlying function is subsequently changed, the projection is not.
q)f:{x-y}
q)g:f[42;]
q)g
{x-y}[42;]
q)g[6]
36
q)f:{x+y}
q)g
{x-y}[42;]
q)g[6]
366
6.4.2 Operator Projection
When used in infix form, a q operator can only be projected by fixing its left operand and this requires parentheses. For example, the projection of * onto its right operand is,
q)(7*) 6
42
Due to the overloading of parentheses in q, an operator cannot be projected by fixing its right argument, since this would lead to notational ambiguity. For example, -42 must be the atom -42 and not a projection of subtraction.
However, every binary operator is a binary function, so you can project it in prefix form.
q)-[;42] 98
56
The whitespace is not necessary in juxtaposition since the brackets make things unambiguous.
q)(7*)6
42
q)-[;42]98
56
Such notation is definitely an acquired taste.
6.4.3 Multiple Projections
Functions having valence rank greater than two can be projected in multiple ways. For example, a function of rank three can be projected onto the second parameter by specifying the first and third. The result is a unary function of the second parameter only.
q){x+y+z}[1;;3]
{x+y+z}[1;;3]
q){x+y+z}[1;;3] 2
6
We could arrive at the same result in two steps.
-
Project onto the second and third parameters by specifying just the first argument. This yields a binary function.
q){x+y+z}[1;;] {x+y+z}[1;;] q){x+y+z}[1;;][2;3] 6 -
Project the resulting binary function onto its first parameter by providing only the second.
q){x+y+z}[1;;][;3] {x+y+z}[1;;][;3] q){x+y+z}[1;;][;3] 2 6
This is equivalent to projecting in the reverse order.
q){x+y+z}[;;3][1;]
{x+y+z}[;;3][1;]
q){x+y+z}[;;3][1;] 2
6
6.5 Everything Is a Map
Here we explore the deep relationship between data structures and functions. While it can be skipped on first reading, that would be like not eating your vegetables when you were a kid.
6.5.1 Similarity of Notation
You have no doubt noticed that the notation for list indexing, dictionary lookup and unary function application are identical. Namely,
q)L:0 1 4 9 16 25 36
q)d:0 1 2 3 4 5 6!0 1 4 9 16 25 36
q)f:{x*x}
q)L[2]
4
q)d[2]
4
q)f[2]
4
This is no accident. In Chapter 3 we saw that a list is a map defined by positional retrieval via item indexing. In Chapter 5 we saw that a dictionary is a map defined by a key-value lookup. A function is a map defined by a sequence of expressions representing an algorithm for computing an output value from the input. The common notation is simply that of a map. It may take a little time to get accustomed to the Zen of q.
Note
In the KX Documentation you will find the term applicable value used to include any entity in q that can be applied like a function. This covers a wide domain: lists, dictionaries, functions, open handles and more. While we like the terminology, we don't find its benefits warrant the considerable effort at introducing it throughout this text. So you will not see it here.
6.5.2 List of Indices, Keys and Arguments
We can use a list of indices to retrieve from a list. Similarly, lookup retrieves a list of values corresponding to a list of keys. And a list of arguments to an atomic function yields a list of outputs. This justifies the common notation.
q)L:10 20 30 40 50
q)I:2 5
q)L I
4 25
q)d:([a:10; b:20; c:30])
q)ks:`a`c
q)d ks
10 30
q)f:{x*x}
q)f I
4 25
In fact, these are all examples of composition of maps.
I L
--------------
0 |-> 2 |-> 30
1 |-> 5 |-> 25
ks d
---------------
0 |-> `a |-> 10
1 |-> `c |-> 30
I f
--------------
0 |-> 2 |-> 4
1 |-> 5 |-> 25
6.5.3 Indexing at Depth
Next we examine the correspondence between indexing at depth and multivalent function evaluation. Notationally, a nested list is a list of lists, but it can also be viewed functionally as multivalent position retrieval. The display of a nested list tells us how: go down and over.
q)L:(1 2 3; 100 200)
q)L
1 2 3
100 200
q)L[1; 0]
100
The first index goes down and the second index does over. A deeper nested list is a positional map whose rank is one greater than the depth of nesting.
Similarly, a dictionary with complex values can also be viewed as a multivalent map.
q)cd:([a:1 2 3; b:10 20 30])
q)cd[`a; 1]
2
In this case, the first parameter locates a key and the second goes into the value list.
6.5.4 Projection and Index Elision
You will have no doubt noticed that the notations of projection for list definition, dictionary definition and function application are consistent since they are all maps.
q)L:(10; 20;)
q)L 30
10 20 30
q)d:([a:10; b:20; c:])
q)d 30
a| 10
b| 20
c| 30
q)f:{x+y+z}
q)f[10;20] 30
60
You will also notice that index elision and function projection are analogous.
q)m:(1 2 3; 10 20 30)
q)m[;2] 1
30
q)f:{x+y}
q)f[;2] 1
3
6.5.5 Out of Bounds Index
Viewing a list as a map on an integer domain also motivates the behavior of item indexing in the case of an out of bounds index. In traditional languages, this would either result in some sort of uncaught error - the infamous indexing off the end of an array - or an exception.
By viewing a list as a function defined on a sub-domain of integers, it is reasonable to extend the domain of the function to all integers by assigning a null output value outside the proper domain. Here null means "missing value." As we have seen previously,
q)10 20 30 40[100]
0N
q)`a`b`c[-1]
`
q)(1.1; 1; `1)[3]
0n
The behavior for dictionaries is analogous.
q)d:([a:10; b:20; c:])
q)d[`x]
0N
6.6 Atomic Functions
The characterization of q as a vector language arises from the fact that most of the built-in operations are atomic. We examine the notion of atomic functions in more detail. The application of an atomic function is characterized by the fact that it effectively recurses into an argument's structure until it gets to atoms and then applies there. It does this without explicit loops or other control flow constructs.
6.6.1 Unary Atomic Functions and "map"
Atomic behavior is easiest to see with a unary function.
q)neg 10
-10
q)neg 10 20 30
-10 -20 -30
q)neg (10 20 30; 40 50)
-10 -20 -30
-40 -50
q)neg ([a:10; b:20; c:30])
a| -10
b| -20
c| -30
q)neg ([a:10 20; b:30 40 50; c:60])
a| -10 -20
b| -30 -40 -50
c| -60
A consequence of this behavior is that the result of a unary atomic function always has the same shape as its argument - i.e., the output conforms to the input.
A special case of this is that applying an atomic function to a list is the same as applying it to each item in the list, which is sometimes described as "atomic functions extend automatically to lists." Similar functionality is achieved with "foreach" in traditional languages and the higher-order function "map" in functional languages.
6.6.2 Binary Atomic Functions and "zip"
Atomic behavior is more interesting for binary functions and binary operators. A binary function can be atomic in either or both arguments. The first case can be thought of as a unary atomic function if you fix the non-atomic argument - i.e. project the function. For example, find ? is atomic in only its second argument since it consumes the entire first argument in its search.
q)10 20 30?10
0
q)10 20 30?10 20 30 40 50
0 1 2 3 3
q)(enlist 10)?10
0
q)10?10 / why?
8 1 8 9 9 2 9 8 8 9
The arithmetic, comparison and relation operators are all atomic in both operands. This leads to four cases for application: atom with atom; atom with list; list with atom; and list with list. In the last case, the two list operands must have the same length.
q)1+10
11
q)1+10 20 30
11 21 31
q)1 2 3+10
11 12 13
q)1 2 3+10 20 30
11 22 33
q)1 2 3+10 20 30 40
'length
Similar behavior is achieved with "zip" in traditional and functional programming.
6.6.3 Creating Atomic-like Functions
A moment's thought will convince you that the composition of atomic functions is atomic. Hence, the easy way to build your own atomic functions is to compose built-in atomic functions. First up: unary.
q)f:{(x*x)+(2*x)-1}
q)f 0
-1
q)f til 10
-1 2 7 14 23 34 47 62 79 98
Now binary.
q)pyth:{sqrt (x*x)+y*y}
q)pyth[1; 1]
1.414214
q)pyth[1; 1 2 3]
1.414214 2.236068 3.162278
q)pyth[1 2 3; 1 2 3]
1.414214 2.828427 4.242641
You might ask how to get this behavior with a function that is not atomic. Excellent question.
6.7 Iterators
Iterators, called adverbs in old-style q-speak, are higher-order functions that modify other functions to repeat their behavior. The earlier terminology derives from thinking of built-in q functions as verbs. We prefer to view functions as data - i.e., nouns - when operated on by other functions.
Proficiency in the use of iterators is one skill that separates q pretenders from q contenders. Otherwise put, expect to be asked about adverbs/iterators on a q test.
Not all functions are atomic, but we still may want to apply them to individual items of a data structure without loopy code. Iterators to the rescue.
6.7.1 Unary Each
We have seen that an atomic function automatically applies to the atoms in a nested list.
q)neg (1 2 3; 4 5)
-1 -2 -3
-4 -5
Clearly a function that aggregates a list into an atom cannot be atomic since it collapses the list structure. In particular, aggregate functions operate only at the top level of a nested list.
q)count 10 20 3
3
q)count (10 20 30; 40 50)
2
Suppose instead that we want to count each top-level item in our nested list. This is precisely what the each iterator does. It modifies a unary function so that it applies to each item in a list rather than to the list as a whole. When used infix, each follows the function after (required) whitespace.
q)count each (10 20 30; 40 50)
3 2
The nature of each as a higher-order function is clearly revealed in its equivalent prefix form. This may look odd at first, but it shows that each takes a function and modifies its behavior on a list.
q)each[count] (10 20 30; 40 50)
3 2
Tip
The higher-order function each is similar to "foreach" in imperative languages and is called map in functional programming languages. It is one half of the famous map-reduce paradigm.
We could say informally that each makes a non-atomic function act atomically; this is almost correct. For deeply nested lists, you need to iterate each to apply a function at deeper levels.
q)(count each) each ((1 2 3; 3 4); (100 200; 300 400 500))
3 2
2 3
q)each[each[count]] ((1 2 3; 3 4); (100 200; 300 400 500))
3 2
2 3
The effect of iterated each is perhaps most clearly observed when applied to a matrix, where successive higher-order applications cause the original function to be applied on another axis.
q)count (1 2 3; 10 20 30)
2
q)count each (1 2 3; 10 20 30)
3 3
q)(count each) each (1 2 3; 10 20 30)
1 1 1
1 1 1
You can also write this form of each with ' in which case you must either parenthesize it or use brackets in the application.
q)count'[(1 2 3; 10 20 30)]
3 3
q)(count') (1 2 3; 10 20 30)
3 3
q)count'(1 2 3; 10 20 30)
''
[0] count'(1 2 3; 10 20 30)
^
Note
Experienced q programmers tend to use this form of Each since it is more economical to write and generalizes to functions of higher valence in the next section. We use both forms through the remainder of this book.
You can use Each with any unary function, although it is redundant on atomic functions.
q)reverse "live"
"evil"
q)reverse ("life"; "the"; "universe"; "and"; "everything")
"everything"
"and"
"universe"
"the"
"life"
q)(reverse') ("life"; "the"; "universe"; "and"; "everything")
"efil"
"eht"
"esrevinu"
"dna"
"gnihtyreve"
Tip
It often arises that you want to convert a vector of length n to an n x 1 matrix. You can do that with enlist each but flip enlist executes faster for large lists.
q)(enlist') 1001 1002 1004 1003
1001
1002
1004
1003
q)flip enlist 1001 1002 1004 1003
1001
1002
1004
1003
6.7.2 Each Parallel
An atomic operator automatically applies to corresponding pairs of items when applied to lists of the same length.
q)1 2 3+10 20 30
11 22 33
The binary join operator , is not atomic since it consumes (copies of) both of its operands in their entirety when it concatenates.
q)"abc","de"
"abcde"
Suppose we want to join list items pair-wise. For example,
q)("abc"; "uv") ??? ("de"; "xyz")
"abcde"
"uvxyz"
The higher order function Each Parallel ' will modify a binary function (operator) to apply its behavior pairwise to corresponding list items of its arguments. Conventionally no whitespace should be between ' and the function it is applied to but this does not generate an error. In some programming languages, Each Parallel is called "zip".
Note
In earlier generations of q this was called "each both." One problem with this terminology is that it is awkward when generalized to more than two arguments.
q)("abc"; "uv"),'("de"; "xyz")
"abcde"
"uvxyz"
There is no difference from the unmodified operator when the arguments are atoms.
q)3,'4
3 4
A function modified by Each Parallel can be used infix, even of the original functional is yours, which cannot be used infix.
q)f:{x,y}
q)1 2 f 30 40
'type
[0] 1 2 f 30 40
^
q)1 2 f' 30 40
1 30
2 40
A function modified by Each Parallel has the usual properties of atomic binary functions. For example, you get a length error if arguments don't match up.
q)("abc"; "uv"),'("de"; "xyz"; "uhoh")
'length
The modified function also extends an atom to match a list. You will see this idiom often. For example,
q)1,'10 20 30
1 10
1 20
1 30
q)1 2 3,'10
1 10
2 10
3 10
q)2#'("abcde"; "fgh"; "ijklm")
"ab"
"fg"
"ij"
Tip
The idiom ,' works fine for making a list of pairs from two simple lists but does not always work as desired with general lists. For example, it loses the enlisting of the singleton in the following.
q)L1:(enlist `a; `b)
q)L2:1 2
q)L1,'L2
`a 1
`b 2
The following idiom is a reliable way to make a list of pairs from a pair of lists. (This is a good q test question).
q)flip (L1; L2)
,`a 1
`b 1
Advanced: A nice example of ,' arises when both operands are tables. Since a table is a list of records, it is possible to apply ,' to tables with the same number of rows. The record dictionaries in each row are upserted, resulting in a sideways join of the tables.
q)t1:([] c1:1 2 3)
q)t2:([] c2:`a`b`c)
q)t1,'t2
c1 c2
-----
1 a
2 b
3 c
If you are so inclined, you can use a binary function modified by Each Parallel in prefix form. This is not common but can simplify things in functional form of queries.
q),'[("abc"; "uv"); ("de"; "xyz")]
"ab"
"fg"
"ij"
Each Parallel also generalizes to functions of arbitrary rank in this form. Here is a simple example.
q){x,":",y,"-",z}'["abc"; "ABC"; "012"]
"a:A-0"
"b:B-1"
"c:C-2"
6.7.3 Each Left \:
An atomic operator extends an atom in the right operand to match a list on the left.
q)1 2 3+10
11 12 13
Each Left \: modifies a binary function so that it applies the second operand with each item of the first operand. For example, to append a given string to each string in a list. We insert unnecessary whitespace around the modified function in the following example to make it more readable.
q)("abc"; "de"; enlist "f") ,\: ">"
"abc>"
"de>"
"f>"
Traditionally there is no space between Each Left and the function it modifies but this causes no error in this case.
6.7.4 Each Right /:
An atomic operator extends an atom in the left operand to match a list on the right.
q)10+1 2 3
11 12 13
Each Right /: modifies a binary operator so that it applies the first operand to each item of the second operand. For example, to prepend a given string to each string in a list. Again we surround the modified function with unnecessary whitespace for readability.
q)"</" ,/: ("abc"; "de"; enlist "f")
"</abc"
"</de"
"</f"
Composing the last two examples makes data xml-ey (say it aloud) without loops.
q)"</",/:("abc"; "de"; enlist "f"),\:">"
"</abc>"
"</de>"
"</f>
6.7.5 Cross Product ,/:\:
To achieve a cross (Cartesian) product of two lists requires that each item on the let be joined with each item on the right. If we compose Each Right with Each Left, we are almost there. There is an extra level of nesting, which we eliminate with the built-in raze.
q)1 2 3,/:\:10 20
1 10 1 20
2 10 2 20
3 10 3 20
q)raze 1 2 3,/:\:10 20
1 10
1 20
2 10
2 20
3 10
3 20
Tip
The built-in operator (cross) computes the cross product.
q)(1 2 3 cross 10 20)~raze 1 2 3,/:\:10 20
1b
Observe that we could alternatively compose Each Left with Each Right and would arrive at the previous result ordered by the second column.
q)raze 1 2 3,\:/:10 20
1 10
2 10
3 10
1 20
2 20
3 20
6.7.6 Over / for Accumulation
The higher-order function Over / is an iterator that provides the principal mechanism for recursion in q. In its simplest form it
modifies a binary function to accumulate results over a list.
Suppose we want to add the first 10 natural numbers starting at 1. In imperative programming this often requires control flow: initialize a result variable, establish a loop counter and then loop while adding into the result until the counter reaches the end of the list. In functional programming, declare that you want to accumulate starting with an initial value of the accumulator. No counters, no tests, no loop. You say "what" to do rather than "how" to do it.
Let's see this in q. For readability, we have included optional whitespace.
q)0 +/ 1 2 3 4 5 6 7 8 9 10
55
/.
Important
There can be no whitespace between the function and / otherwise / will start a comment.
The initial accumulator value is the left operand and the list to accumulate over is the right operand. Under the covers, each item of the list is progressively added into the accumulator, moving across the list until the end is reached; then the accumulated value is returned. In our example, 1 is added into the initial accumulator value of 0 to get 1; then 2 is added to this value to get 3; and so on until the end of the list. Just like old-fashioned imperative code, except you never see it. Here is an instrumented form of accumulating sum. It uses the (0N!) operator to display the accumulator x and the next input y before returning their sum.
q)addi:{0N!(x;y); x+y}
q)0 addi/ 1 2 3 4 5
0 1
1 2
3 3
6 4
10 5
15 / the result
Observe that as a higher-order function Over takes a binary function and returns a related binary function. The operands of the modified function are different and they are no longer symmetric - i.e., the left operand is the initial accumulator value and the right operand is the list to be accumulated.
Tip
In functional programming languages this form of Over is called fold or reduce. Together with Each they comprise the famous map-reduce paradigm.
Often the generality of specifying the initial value of the accumulator is unnecessary. In our summation example, we could initialize the accumulator with the first list item and then accumulate the remaining items. The syntax for this form omits the initial accumulator and can either enclose the modified operator in parentheses or use brackets.
q)(+/) 1 2 3 4 5 6 7 8 9 10
55
q)+/[1 2 3 4 5 6 7 8 9 10]
55
Another way to think of this is by representing the list in general form.
(1; 2; 3; 4; 5; 6; 7; 8; 9; 10)
Then over effectively replaces the semicolons with '+', associated to the left.
A few notes about the alternative form of over.
- The parentheses or brackets are mandatory
- The modified function in this form is unary
- This construct is actually k. Don't tell anyone you are writing k.
The binary form can also be written prefix.
q)+/[0; 1 2 3 4 5 6 7 8 9 10]
55
The basic Over pattern is powerful. Use various binary functions to obtain common aggregate operations.
q)(*/) 1 2 3 4 5 6 7 8 9 10 / product
3628800
q)(|/) 7 8 4 3 10 2 1 9 5 6 / maximum
10
q)(&/) 7 8 4 3 10 2 1 9 5 6 / minimum
1
Common aggregates are given their own names in q.
| Aggregate | Name |
|---|---|
| +/ | sum |
| */ | prd |
| |/ | max |
| &/ | min |
Note
Anyone with information about the location of the missing 'o' in (prd) please contact KX.
Applying , across a list effectively eliminates the top level of nesting by concatenating all the items. If there is only one level of nesting, this flattens the list. The function ,/ goes by the descriptive name (raze).
q)raze (1 2; 3 4 5; 6 7)
1 2 3 4 5 6 7
q)(,/) ((1 2 3; 4 5); (100 200; 300 400 500))
1 2 3
4 5
100 200
300 400 500
You can use your own function with either form of Over and the modified version can even be used infix.
q)f:{2*x+y}
q)100 f/ 1 2 3 4 5 6 7 8 9 10
106472
q)(f/) 1 2 3 4 5 6 7 8 9 10
3560
Finally we note that for the notationally challenged, you can write Over as (over). You must parenthesize it to pass it as an argument.
6.7.7 More Over
The next pattern of recursion we investigate is a declarative version of an imperative for loop. In this version of Over, the left operand is a natural number indicating how many times to perform the iteration and the right operand is the initial value.
Note
Use this version of Over rather than do whenever possible.
Let's compute the nth Fibonacci number - see 1.10 for a more detailed derivation of fib. Again the modified function can be used infix or prefix.
q)fib:{x,sum -2#x}
q)10 fib/ 1 1
1 1 2 3 5 8 13 21 34 55 89 144
q)fib/[10; 1 1]
1 1 2 3 5 8 13 21 34 55 89 144
Here is a nifty way to visualize exactly how q performs this form of iteration.
q)5(`f;)\1
1
(`f;1)
(`f;(`f;1))
(`f;(`f;(`f;1)))
(`f;(`f;(`f;(`f;1))))
(`f;(`f;(`f;(`f;(`f;1)))))
Yet another version of Over runs the recursion until convergence, or until a loop is detected. Our example uses a simple form of Newton-Raphso - see 1.11 for a more detailed exposition for nth roots. We choose the function {x*x-2} in order to compute the square root and select an initial value for which the algorithm converges. The brackets and no whitespace after '/' are mandatory.
q)newtsqrt:{[xn] xn-(-2+xn*xn)%2*xn}
q)newtsqrt/[1.0]
1.414214
How does q determine convergence? At each step the result is compared to that of the previous step. If the two agree within equality tolerance (approximately 10-14 at the time of this writing - see the the KX Documentation for details) the algorithm is considered to have converged and the recursion is complete; otherwise it continues.
Mathematically, it is possible to generate an infinite loop with Newton-Raphson. To wit, we select the function x3 -- 2x + 2 and choose the initial value 0, which causes the algorithm to go into a cycle.
q)newtcycle:{[xn] xn-((xn*xn*xn)+(-2*xn)+2)%-2+3*xn*xn}
q)newtcycle/[0.0]
1f
How does over break the cycle? At each step it compares the result with the initial value. If they match, a cycle has occurred and the recursion is halted; otherwise it continues. Note that we mean "match" literally as in the ~ operator.
Tip
If the values are equal but of different type, no cycle is detected and the loop will continue. In our example, providing the same initial value as a non-float will cause an infinite loop. You will have to kill your console session if you enter the following.
newtcycle/[0] / oops
The final overload of over is equivalent to a "while" loop in imperative programming. It provides a declarative way to specify a test to end the iteration. Thinking functionally, we supply a predicate function that is applied to the result after each step. The iteration continues as long as the result is 1b and stops otherwise.
Note
Use this version of Over rather than while whenever possible.
For example, we stop the computation of the Fibonacci sequence once it exceeds 1000 as follows.
q)fib:{x,sum -2#x}
q)fib/[{1000>last x}; 1 1]
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597
6.7.8 Scan \
The iterator Scan \ is a higher-order function that accumulates just like Over except that the modified function returns all the intermediate accumulations instead of just the final one. Whereas Over produces an aggregate function that collapses a list, Scan produces a function whose output has the same number of items as its input - i.e., a uniform function. Informally, Scan is a "running" version of Over.
We omit optional whitespace in the following so that you can get accustomed to reading "real q."
q)0+\1 2 3 4 5 6 7 8 9 10
1 3 6 10 15 21 28 36 45 55
q)(*\)1 2 3 4 5 6 7 8 9 10
1 2 6 24 120 720 5040 40320 362880 3628800
q)(|\)7 8 4 3 10 2 1 9 5 6
7 8 8 8 10 10 10 10 10 10
q)(&\)7 8 4 3 10 2 1 9 5 6
7 7 4 3 3 2 1 1 1 1
q)f:{2*x+y}
q)100 f\1 2 3 4 5 6 7 8 9 10
202 408 822 1652 3314 6640 13294 26604 53226 106472
q)(f\)1 2 3 4 5 6 7 8 9 10
1 6 18 44 98 208 430 876 1770 3560
All the notational variations of Over are also available with Scan. Switching to Scan can be useful to see intermediate results of an accumulation. For example, we can see how fast Newton-Raphson converges to the square root by setting the float display to unlimited precision and running Scan.
q)newtsqrt\[1.0]
q)\P 0
q)newtsqrt\[1.0]
1 1.5 1.4166666666666667 1.4142156862745099 1.4142135623746899 1.4142135623730951
6.7.9 Each Prior ':
The iterator Each Prior ': provides a declarative way to perform a binary operation on each item of a list with its predecessor. During traversal of the list, the current item is the left operand x and the previous item is the right operand y.
Note
In earlier generations of q this was sometimes called "each previous."
Since the initial item of the list does not have a predecessor, we can provide one in the left operand of the modified operator. One choice is the initial value of the input. For example, to calculate the deltas of list of prices.
q)100 -': 100 99 101 102 101
0 -1 2 1 -1
As with the other iterators, there is a unary form of Each Prior (as it is now known). It may come as a surprise that the unary form does not choose the initial item of the input list as the initial predecessor. Rather, it returns the initial item of the input on (-) and (%).
q)(-':)100 99 101 102 101
100 -1 2 1 -1
q)deltas 100 99 101 102 101
100 -1 2 1 -1
q)(%':)100 99 101 102 101
100 0.99 1.020202 1.009901 0.9901961
q)ratios 100 99 101 102 101
100 0.99 1.020202 1.009901 0.9901961
The motivation is that the initial predecessor is implicitly chosen as the identity for the binary operator (assuming such exists). This has the advantage of preserving desirable invariants.
q)sums deltas 100 99 101 102 101
100 99 101 102 101
q)deltas sums 100 99 101 102 101
100 99 101 102 101
On the other hand, if you are looking for anomalies in absolute differences - e.g., price changes - you will get a nasty surprise.
q)deltas 100 99 101 102 101
100 -1 2 1 -1
It is easy enough to make our own version for this case.
q)deltas0:{first[x] -': x}
q)deltas0 100 99 101 102 101
0 -1 2 1 -1
A powerful idiom with Each Prior uses match ~ to test if successive items are identical. It is often more useful to know where they do not match; the latter is built in as differ.
q)(~':) 1 1 1 2 2 3 4 5 5 5 6 6
011010001101b
q)not (~':) 1 1 1 2 2 3 4 5 5 5 6 6
100101110010b
q)differ 1 1 1 2 2 3 4 5 5 5 6 6
100101110010b
Tip
Observe that the 1b values in the result of differ are the start points for runs of identical items, so where yields their
indices. Then use cut to split them out.
q)L:1 1 1 2 2 3 4 5 5 5 6 6
q)where differ L
0 3 5 6 7 10
q)(where differ L) cut L
1 1 1
2 2
,3
,4
5 5 5
6 6
Example: Let's do some q.
q)runs:(where differ L) cut L / store runs
q)cts:count each runs / store count of each run
q)runs where cts=max cts / find the runs of maximum length
1 1 1
5 5 5
While some q coders don't like the style, we point out that the above can be written in a single line of q using intra-line assignments. This style is perfectly acceptable provided you don't use the intra-line variables outside this line.
q)runs where cts=max cts:count each runs:(where differ L) cut L
1 1 1
5 5 5
Combine comparison and the correct initial value with the above technique to find runs of increasing or decreasing values.
q)L:9 8 7 11 10 12 13
q)(where -0W>':L) cut L
9 8 7
11 10
,12
,13
q)(where 0W<':L) cut L
,9
,8
7 11
10 12 13
This technique is the raison d'être for integer infinities.
6.7.10 A Vegan Example of Iterators in Action
In §1.1.12 we saw the use of basic iteration-modified functions to calculate FIFO allocation of buys to a sell. We recommend that you (re)read that section now since it serves as the basis for the following. There we saw how to allocate a sell FIFO across multiple buys with nary a loop in sight.
q)buys:2 1 4 3 5 4f
q)sell:12f
q)deltas sell&sums buys
2 1 4 3 2 0f
Now fasten your seatbelts as we switch to warp drive. We show a meaty example with no animals injured by loops.
In real-world FIFO allocation problems, we actually want to allocate buys FIFO not just to a single sell, but to a sequence of sells. You say, surely this must require a loop. Please don’t call me Shirley. And no loopy code either.
We take buys as before but now we have a list sells, which are to be allocated FIFO from buys.
q)buys:2 1 4 3 5 4f
q)sells:2 4 3 2
q)allocations
2 0 0 0 0 0
0 1 3 0 0 0
0 0 1 2 0 0
0 0 0 1 1 0
The idea is to extend the allocation of buys across multiple sells by considering both the cumulative amounts to be allocated as well as the cumulative amounts available for allocation.
q)sums[buys]
2 3 7 10 15 19f
q)sums[sells]
2 6 9 11
q)2&sums[buys]
2 2 2 2 2 2f
q)6&sums[buys]
2 3 6 6 6 6f
q)9&sums[buys]
2 3 7 9 9 9f
q)11&sums[buys]
2 3 7 10 11 11f
Contemplate this koan and you will realize that each line includes the allocations to all the buys preceding it. From this we will be able to unwrap cumulatively along both the buy and sell axes to get the incremental allocations.
Our first task is to produce the above result as a list of lists.
2 2 2 2 2 2
2 3 6 6 6 6
2 3 7 9 9 9
2 3 7 10 11 11
Iterators to the rescue! Our first task requires an iterator that applies a binary function and a given right operand to each item of a list on the left. That is Each Left with the funky notation (:). We use it to accomplish in a single operation the four individual (&) operations above.
q)sums[sells] &\: sums[buys]
2 2 2 2 2 2
2 3 6 6 6 6
2 3 7 9 9 9
2 3 7 10 11 11
Now we apply (deltas) with its internal iteration to unwind the allocation in the vertical direction.
q)deltas sums[sells]&\:sums[buys]
2 2 2 2 2 2
0 1 4 4 4 4
0 0 1 3 3 3
0 0 0 1 2 2
The iterator we need is Each. In the context of our allocation problem, we realize that (deltas each) is just the ticket to unwind the remaining cumulative allocation within each row.
q)deltas each deltas sums[sells] &\: sums[buys]
2 0 0 0 0 0
0 1 3 0 0 0
0 0 1 2 0 0
0 0 0 1 1 0
Voila! The solution to our allocation problem in a single line of q. The power of compositional iterators is breathtaking.
6.8 General Application
We have seen that the syntactic forms of list indexing, key lookup and function application can use either square brackets or juxtaposition.
q)L:(1 2;3 4 5; enlist 6)
q)L[0]
1 2
q)L 1
3 4 5
q)L[0 2]
1 2
,6
q)L[1][2]
5
q)L[1; 2]
5
q)d:([a:1 2; b:3 4 5; c:enlist 6])
q)d[`a]
1 2
q)d[`a`c]
1 2
,6
q)d[`a][1]
2
q)d[`a; 1]
2
q)f:{x*x}
q)f[0]
0
q)f 1
1
q)f[0 2]
0 4
q)g:{x+y}
q)g[1][2]
3
q)g[1;2]
3
It turns out that both forms are syntactic sugar for how q really treats application. You won't be surprised that application is actually a higher-order function that takes a function and value(s) and evaluates the function at the value(s). Thorough understanding of the general application is another test that separates the q pretenders from the contenders. The simpler forms of general application are usually written infix.
6.8.1 Amend At
Basic application comprises retrieving a list item by index, looking up a dictionary value by key, or evaluating a unary function. These are all instances of application of a univalent mathematical mapping. These are to be distinguished from indexing at depth, which is discussed in the next section.
The higher-order function Amend At @ is the true form of basic unary application in q. It applies a unary mapping to an argument. As with all built-in functions, it can be written infix or prefix.
q)10 20 30 40@1
20
q)L:10 20 30 40
q)L@1
20
q)@[L; 1]
20
q)count@L
4
q)@[count; L]
4
q){x*x}@L
100 400 900 1600
q)d:([a:10; b:20; c:30])
q)d@`a
10
q)@[d;`b]
20
Applying a nullary function with @ is best done using the nil argument :: but any argument suffices since it is ignored.
q)f:{6*7}
q)f[]
42
q)@[f; ::]
42
q)f@(::)
42
q)f@43
42
For contrast with vector application in the next section, we collect previously discussed facts about ordinary application with a non-atom argument.
For lists, dictionaries and atomic functions, Amend At yields an output that conforms to the shape of the input.
An aggregate function collapses the top level of nesting from the input.
A uniform function has output the same length as an input list.
Advanced: Amend At @ also applies to tables and keyed tables since they are lists and dictionaries, respectively. Since a table is a list of records, using @ retrieves records; for dictionaries and keyed tables it looks up keys.
q)t:([] c1:1 2 3; c2:`a`b`c)
q)t@1
c1| 2
c2| `b
q)d:([a:10; b:20; c:30])
q)d@`b
20
q)kt:([k:`a`b`c] v:1.1 2.2 3.3)
q)kt@`c
v| 3.3
6.8.2 Amend .
Indexing a list at depth, retrieving a nested value from a dictionary and evaluating a function with multiple parameters are all instances of applying a multivalent mapping. The higher-order function Amend . is the true form of multivalent application in q. It applies a multivalent mapping to a list of arguments and can be written infix or prefix.
Note
The right argument of . must be a list.
Tip
Separate Amend . from its operands by whitespace if they are names or literal constants, as proximity to dots used in
name-spacing and in decimal numbers can lead to confusion.
Here we show the various forms above subsumed under ..
q)L:(10 20 30; 40 50)
q)L[1][0]
40
q)L[1; 0]
40
q)L . 1 0
40
q)d:([a:10 20 30; b:40 50; c:enlist 60])
q)d[`b][0]
40
q)d[`b; 0]
40
q)d . (`b; 0)
40
q)g:{x+y}
q)g[1; 2]
3
q)g . 1 2
3
Observe that Amend effectively allows us to apply a multivalent function to a single list of arguments instead of multiple individual arguments. Mathematically this means that we are thinking of the function as defined on a vector instead of the vector components.
Amend provides function application syntax that is conveniently independent of the valence of the function. This is useful for situations in which a function and its arguments are generated dynamically – e.g., dynamic dispatch - and the valence is not known in advance.
q)g:{x+y}
q)g . 1 2
3
q)h:{x+y+z}
q)h . 1 2 3
6
Tip
You can apply a unary function with . too. The insight is that a function of a scalar is inter-convertible with a function of a singleton vector. You realize this correspondence in q by enlisting the argument and using . for application instead of @.
q)f:{x*x}
q)f@5
25
q)f . enlist 5
25
q)f . enlist 1 2 3
1 4 9
We conclude that Amend . can apply a mapping of any valence and that @ is actually unnecessary. However, most q programmers do use @ where applicable (pun intended).
To specify an elided index with . application use the nil item.
q)m:(1 2 3;4 5 6)
q)m[0;]
1 2 3
q)m . (0; ::)
1 2 3
q)m . (::; 1)
2 5
Evaluating a nullary function with . should properly use the enlisted nil item although any enlisted value will suffice since it is ignored.
q)f:{6*7}
q)f . enlist (::)
42
Zen Enlightenment: Let's climb the philosophical mountain. All data structures in q are composed from lists and
dictionaries. Because q views both lists and dictionaries as mathematical mappings, retrieval from each is function application.
Retrieval from un-nested lists and dictionaries is unary application with Amend At @. Retrieval from nested entities is multivalent
application with Amend .. In light of the previous observation that a unary mapping can be viewed as a mapping on a singleton vector, Amend . provides the mechanism for retrieval from an arbitrary data structure. Looking back from the summit we realize: it is . all the way down in q.
Following are some examples of complex data structures and indexing at depth. See if you can predict them before entering at the console. (They make good q test questions)
q)L:10 20 30
q)L . enlist 1
20
q)m:(10 20 30; 100 200 300)
q)m . 0 1
20
q)ds:(([a:10; b:20; c:30]); ([x:100; y:200]))
q)ds . (0; `b)
20
q)mix:(10 20 30; ([a:1; b:20; c:300 400]))
q)mix . (1; `c; 1)
400
q)dc:([c1:1 2 3; c2:`a`b`c])
q)dc . (`c2; 1)
`b
q)t:([]c1:1 2 3;c2:`a`b`c)
q)t . (1; `c2)
`b
q)kt:([k:`a`b`c] v:1.1 2.2 3.3)
q)kt . `b`v
2.2
Tip
Note that indexing at depth no longer fails for accessing items inside a nested table column as was the case in earlier versions of q.
t:([] c1:`a`b`c; c2:(1 2; 100 200 400; enlist 1000))
q)t . (1; `c2)
100 200 400
q)t . (1; `c2; 1)
200
q)t . (1; `c2; 1)
200
6.8.3 General Form of Amend At and Amend
Indexing a list at depth, retrieving a nested value from a dictionary and evaluating a function with multiple parameters are all instances of Amend. The uses of @ and . heretofore presented are actually special cases of even more general higher-order functions of higher valence. Since these must be written prefix, we begin by writing in prefix some simple examples of Amend from the previous section. We use lists for simplicity but other data structures are also applicable (ouch).
q)L:10 20 30 40 50
q)@[L; 1]
20
q)@[L; 0 2]
10 30
q)m:(10 20 30; 100 200 300)
q).[m; 0 2]
30
Each application above can be viewed as retrieval from the data structure along a subdomain of its definition. From this perspective, the more general forms will apply an arbitrary operation along a subdomain. Otherwise said, they modify a substructure within the original structure. This is a very powerful technique that is not present in traditional programming languages.
6.8.3.1 General Amend At @ with Unary Functions
Let's begin with the @ retrieval examples in the previous section.
q)L:10 20 30 40 50
q)@[L; 0 2]
10 30
Instead of simply retrieving along the subdomain, we can supply a function to be applied along the subdomain.
q)@[L; 1; neg]
10 -20 30 40 50
q)@[L; 0 2; neg]
-10 20 -30 40 50
Effectively we reach inside (a copy of) the list, apply (neg) along the specified subdomain, and return the modified copy. Contrast this with normal application on the subdomain, which returns only the modified items.
q)neg L@0 1
-10 -20
The significance of enhanced Amend At is that it returns the entire modified list, which allows the result to be chained (i.e., composed) into further operations. If you don't think this is a big deal, think again.
The syntax for general application of a unary atomic function on a list is,
@[L;I;f]
where L is the list and I is a list of indices into L. Viewing L as a mapping, I is a top-level subdomain of L. In fact, this form generalizes to any data structure viewed as a mapping. For example, given a dictionary and a list of keys,
q)d:([a:10; b:20; c:30])
q)ks:`a`c
q)@[d; ks; neg]
a| -10
b| 20
c| -3
Important
For compound data structures such as nested lists, tables and keyed tables, Amend At @ occurs along a subdomain at the
top level. We'll see how to reach into the structure in the next section.
q)m:(10 20 30; 100 200 300; 1000 2000 3000)
q)@[m; 0 2; neg]
-10 -20 -30
100 200 300
-1000 -2000 -3000
All the examples we have shown thus far work on copies of the input data structure. It is also possible to modify the original data structure in place. Although q does not have references, Amend At provides pass-by-name using the symbolic name. We demonstrate with a simple example. In the first application, L is unchanged; in the second it is modified.
q)L:10 20 30 40
q)@[L; 0; neg]
-10 20 30 40
q)L
10 20 30 40
q)@[`L; 0 ; neg]
`L
q)L
-10 20 30 40
Observe that the result of an application of pass-by-name is the symbolic name passed. This allows chaining of applications that modify in place.
6.8.3.2 General Amend At @ with Binary Operators
In the previous section we saw how to apply a unary function within the top level of a data structure. In the same way, we can apply a binary atomic operator using @ by providing the operator together with an operand to apply along the subdomain. Clearly the shape of supplied operand must conform to the specified subdomain. The exception is applying an atom, in which case the it is automatically extended to match the subdomain.
q)L:10 20 30 40
q)@[L; 0 1; +; 100 200]
110 220 30 40
q)@[L; 0 1; +; 100]
110 120 30 40
q)d:([a:10; b:20; c:30])
q)@[d; `a`b; +; 100 200]
a| 110
b| 220
c| 30
q)@[d; `a`b; +; 100]
a| 110
b| 120
c| 30
The general form of Amend At @ for a binary atomic function is,
@[L; I; g; v]
The notation is suggestive of lists, but L can be any data structure, viewed as a mapping. I is a top-level subdomain of L; g is a binary function; and v is an atom or list conforming to I.
We contrast application along a subdomain with normal application on the separate substructure.
q)L:10 20 30 40
q)@[L; 0 1; +; 100 200]
110 220 30 40
q)100 200+L@0 1
110 220
Note
As in the unary case, Amend At @ along a subdomain of compound data structures (e.g., nested lists, tables and keyed tables) occurs at the top level.
q)m:(10 20 30; 100 200 300; 1000 2000 3000)
q)@[m; 0 2; +; 1 2]
11 21 31
100 200 300
1002 2002 3002
Tip
A useful case of binary application uses the assignment operator : to modify values along the subdomain.
q)L:10 20 30 40
q)@[L; 0 2; :; 42 43]
42 20 43 40
As in the unary case, to apply to the original in place instead of to a copy, use pass-by-name.
q)L:10 20 30 40
q)@[`L; 0 2; :; 42 43]
`L
q)L
42 20 43 40
6.8.3.3 General Amend . for Unary Functions
We have seen that Amend At @ applies a function on a vector along the top level of a data structure. Now we consider indexing at depth. Although the construct is valid at any depth, there is a case to be made not to go beyond depth two in q (data normalization with tables).
We begin by restating simple examples of indexing at depth in prefix notation. We use moderately nested lists and dictionaries for simplicity.
q)m:(10 20 30; 100 200 300)
q).[m; 0 1]
20
q)d:([a:10 20 30; b:40 50; c:enlist 60])
q).[d; (`a; 1)]
20
In contrast with Amend At @, a vector argument to Amend . reaches down into the data structure and picks out a single point in the domain. Here we target that point with a unary function.
q).[m; 0 1; neg]
10 -20 30
100 200 300
q).[d; (`a; 1); neg]
a| 10 -20 30
b| 40 50
c| ,60
As before, these applications work on a copy of the data. To modify the original in place, . also provides pass-by-name.
q)m
10 20 30
100 200 300
q).[`m; 0 1; neg]
`m
q)m
10 -20 30
100 200 300
q).[d; (`a; 1); neg]
a| 10 -20 30
b| 40 50
c| ,60
q)d
a| 10 20 30
b| 40 50
c| ,60
q).[`d; (`a; 1); neg]
`d
q)d
a| 10 -20 30
b| 40 50
c| ,60
We previously saw that we elide an index in simple Amend . retrieval by placing the nil item :: in that slot. The same holds for general application, where it means apply the function along all indices at that level. Continuing from the results of the previous examples,
q).[m; (0; ::); neg]
-10 20 -30
100 200 300
q)d:([a:10 20 30; b:40 50; c:enlist 60])
q).[d; (`a; ::); neg]
a| -10 -20 -30
b| 40 50
c| ,60
q).[d; (::; 0); neg]
a| -10 20 30
b| -40 50
c| ,-60
The general form of Amend . for unary functions is,
.[L; I; f]
Here L is a data structure, I is an in-depth subdomain of L and f is a unary atomic function.
6.8.3.4 General Amend . for Binary Functions
By now you can predict how general Amend . with a binary function works. The form of functional . for a binary function is,
.[L;I;g;v]
Here L is a data structure, I is an in-depth subdomain of L, g is a binary function and v is an atom (for g atomic) or a vector that conforms to I. To apply in place use pass-by-name.
q)m:(10 20 30; 100 200 300)
q).[m; 0 1; +; 1]
10 21 30
100 200 300
q).[m; (::; 1); +; 1 2]
10 21 30
100 202 300
q)m
10 20 30
100 200 300
q).[`m; (::; 1); +; 1]
`m
q)m
10 21 30
100 201 300
q)d:([a:10 20 30; b:40 50; c:enlist 60])
q).[d; (`a; 1); +; 1]
a| 10 21 30
b| 40 50
c| ,60
q).[d; (`a; ::); +; 1]
a| 11 21 31
b| 40 50
c| ,60
q).[d; (::; 0); +; 1]
a| 11 20 30
b| 41 50
c| ,61
q)d
a| 10 20 30
b| 40 50
c| ,60
q).[`d; (::; 0); :; 42]
`d
q)d
a| 42 20 30
b| 42 50
c| ,42
6.8.4 Cross Sections with Amend .
We have seen that Amend . interprets a vector argument as indexing at depth into a nested data structure. Viewing the data structure as multidimensional, the vector index specifies a single point. It is also possible to specify a cross section of the index space with a multidimensional index. This is the most general form of Amend ..
We limit our examples to two dimensions for ease of reading. Here we index the cross section formed by the first two rows and the last two columns of a 3x3 matrix.
q)show m:(1 2 3; 10 20 30; 100 200 300)
1 2 3
10 20 30
100 200 300
q)m[0 1; 1 2]
2 3
20 30
q)m . (0 1; 1 2)
2 3
20 30
q).[m; (0 1; 1 2)]
2 3
20 30
Using the last form, we can apply a unary or binary operator along the cross section subdomain.
q).[m; (0 1; 1 2); neg]
1 -2 -3
10 -20 -30
q).[m; (0 1; 1 2); +; 1000]
1 1002 1003
10 1020 1030
100 200 300
q).[m; (0 1; 1 2); :; 42]
1 42 42
10 42 42
100 200 300
We can address all items in a cross-sectional index by placing the nil item in that slot.
q).[m; (::; 1 2); neg]
1 -2 -3
10 -20 -30
100 -200 -300
q).[m; (0 1; ::); neg]
-1 -2 -3
-10 -20 -30
100 200 300
q).[m; (::; ::); neg]
-1 -2 -3
-10 -20 -30
-100 -200 -300
Note
All the examples in this section can also be used with call-by-name to modify the original in place rather than a copy.
6.9 Composition
Normally we write the explicit composition of functions with left-of-right syntax. For example, if you wanted Visual Basic style loopy code with explicit indices you could write,
q)til[count["Visual Basic"]]
0 1 2 3 4 5 6 7 8 9 10 11
If we want to use this composition frequently it would be convenient to define it as a (higher order) function itself.
6.9.1 Composition of Unary Functions
By now we know the interpreter sees the left-of-right composition as syntactic sugar for Amend At @.
q)til[count["Visual Basic"]]
0 1 2 3 4 5 6 7 8 9 10 11
q)til count "Visual Basic"
0 1 2 3 4 5 6 7 8 9 10 11
q)til@count@"Visual Basic"
0 1 2 3 4 5 6 7 8 9 10 11
If we look at the last expression carefully we can interpret @ here as a higher order function that composes unary functions. As such, it turns out that only the right-most '@' is necessary.
q)tc:til@count@
q)tc "Visual Basic"
0 1 2 3 4 5 6 7 8 9 10 11
q)tc:til count@
q)tc "Visual Basic"
0 1 2 3 4 5 6 7 8 9 10 11
What a clever way to disguise your VB code since til count is normally a dead giveaway.
More generally, we can compose any number of unary functions - yours or q's - as follows.
f1@f2@…fn@
or
f1 f2 …fn@
q)neg@first@reverse 10 20 30 40 50
-50
q)sq:{x*x}
q)sq first reverse@ 10 20 30 40 50
2500
6.9.2 Composition of Unary with Multivalent Function
We can also compose a multivalent function with a unary function. As you might expect this uses '. Because ' has so many overloads we are limited in how we can write the composition; in particular, it cannot be written infix here and must be prefix.
'[f; g]
Here g is a multivalent function and f is unary. The resulting function has the same valence as g.
q)f:{x*x}
q)g3:{x+y+z}
q)'[f;g3][1; 2; 3]
36
q)'[neg;g3] . 1 2 3
-6
q)h:('[f;g3]) / parentheses necessary
q)h[1;2;3]
36
Now we can compose a sequence of unary functions with a multivalent function in two steps using the two composition forms.
q)h:('[;])[(10*)@neg@f@; g3] / parentheses necessary
q)h[1;2;3]
-360
q)h:('[;])[(10*) neg f@; g3]
q)h[1;2;3]
-360
Alternately we can compose the sequence of unary functions with a multivalent function without first precomposing the unary functions. Fasten your seat belts because the ride is gong to get bumpy. We use the ('[;]) overload of ' and we use Over to iterate this over the list of functions to be composed.
q)'[;]/[(10*; neg; f; g3)][1;2;3]
-360
q)h:('[;])/[(10*; neg; f; g3)] / parentheses necessary
q)h[1;2;3]
-360
q)h . 1 2 3
-360