System commands

\a tables \s number of secondary threads \b views \S random seed \B pending views \t timer \c console size \T timeout \cd change directory \ts time and space \C HTTP size \u reload user password file \d directory \v variables \e error trap clients \w workspace \E TLS server mode \W week offset \f functions \x expunge \g garbage collection mode \z date parsing \l load file or directory \1 & \2 redirect \o offset from UTC \_ hide q code \p listening port \ terminate \P precision \ toggle q/k \r replication primary \\ quit \r rename

System commands control the q environment. They have the form:

\cmd [p]

for some command cmd, and optional parameter list p.

Commands with optional parameters that set values, will show the current values if the parameters are omitted.

Some system commands have equivalent command-line parameters.

An alternative method to executing system commands is to use the system keyword. This executes a string representation of a system command and returns its result.

\a (tables)

List tables

\a
\a ns

Lists tables in namespace ns – defaults to current namespace.

q)\a
`symbol$()
q)aa:bb:23
q)\a
`symbol$()
q)tt:([]dd:12 34)
q)\a
,`tt
q).nn.vv:([]uu:12 45)
q)\a
,`tt
q)\a .n
'.n
q)\a .nn
,`vv
q)\d .nn
q.nn)\a
,`vv
q.nn)vv
uu
--
12
45
q.nn)

\b (views)

List dependencies

\b
\b ns

Lists dependencies (views) in namespace ns – defaults to current namespace.

q)a::x+y
q)b::x+1
q)\b
`s#`a`b

.z.b
Views

\B (pending views)

List pending dependencies

\B
\B ns

Lists pending dependencies (views) in namespace ns, i.e. dependencies not yet referenced, or not referenced after their referents have changed. Defaults to current namespace.

q)a::x+1          / a depends on x
q)\B              / the dependency is pending
,`a
q)x:10
q)\B              / still pending after x is defined
,`a
q)a               / use a
11
q)\B              / no longer pending
`symbol$()

\c (console size)

Console maximum rows and columns

\c 
\c size

Where size is a pair of integers: rows and columns, these values determine when q truncates output with ... The values are coerced to the range [10,2000].

The default values are as set by environment variables LINES and COLUMNS. If the environment variables are undefined, the defaults are

V4.0 or less   25 80
V4.1+          dimensions of the command-shell window

Environment variables LINES and COLUMNS

See Bash documentation for shopt parameter checkwinsize to make sure they’re reset as needed.

q)\c
45 160
q)\c 5 5
q)\c
10 10
q)til each 20+til 10
0 1 2 3..
0 1 2 3..
0 1 2 3..
0 1 2 3..
0 1 2 3..
0 1 2 3..
0 1 2 3..
..

-c command-line option

\C (HTTP size)

HTTP display maximum rows and columns

\C 
\C size

Where size is a pair of integers: rows and columns, the values determine when q truncates output with ... The default is 36 2000; values are coerced to the range [10,2000].

-C command-line option

\cd (change directory)

Current directory

\cd
\cd fp

Where fp is a filepath, sets the current directory. Creates the directory if it does not exist.

q)\cd
"/home/guest/q"
q)\cd /home/guest/dev
q)\cd
"/home/guest/dev"
q)\pwd
"/home/guest/dev"

\d (directory)

Current namespace

\d
\d ns

Where ns is the name of a namespace, shows or sets the current namespace, also known as directory or context. The namespace can be empty, and a new namespace is created when an object is defined in it. The q session prompt indicates the current namespace.

q)\d                  / default namespace
`.
q)\d .o               / change to .o
q.o)\f
`Cols`Columns`FG`Fkey`Gkey`Key`Special..
q.o)\d .              / return to default
q)key`                / lists namespaces other than .z
`q`Q`o`h
q)\d .s               / change to non-existent namespace
q.s)key`              / not yet created
`q`Q`o`h
q.s)a:1               / create object, also creates namespace
q.s)key`
`q`Q`o`h`s

Q for Mortals §12.7 Working in a Context

\e (error trap clients)

Error trapping

\e
\e mode

Governs error trapping for client requests. The default mode is 0 (off).

mode	behavior
0	When a client request has an error, by default the server clears the stack. Appropriate for production use as it enables the server to continue processing other client requests.
1	The server suspends on an error, and does not process other requests until the stack is cleared. Appropriate for development: enables debugging on the server.
2	Dumps stack to stderr for untrapped errors during request from a remote. (Since V3.5 2016.10.03)

Command-line option -e

\E (TLS server mode)

\E

Displays TLS server mode as an int:

0i   plain
1i   plain and TLS
2i   TLS only

Command-line option -E to set the mode

\f (functions)

List functions

\f
\f ns

Where ns is the name of a namespace, lists functions in it; defaults to current namespace.

q)f:g:h:{x+2*y}
q)\f
`f`g`h
q)\f .h
`cd`code`data`eb`ec`ed`es`estr`fram`ha`hb`hc`he`hn`hp`hr`ht`hta`htac`htc`html`http`hu`hu..
q){x where x like"ht??"}system"f .h"
`htac`html`http

\g (garbage collection mode)

\g            / current garbage-collection mode
\g mode       / set garbage-collection mode

Show or set garbage-collection mode. The default mode is 0 (deferred). Setting the garbage-collection mode will automatically call .Q.gc[] after setting the provided value.

Q manages its own thread-local heap. Objects in q use reference counting. As soon as there are no references to an object, its memory is eligable to be returned to the heap.

0 (deferred)

Returns memory to the thread-local heap. Will subsequently return memory to the OS when either .Q.gc[] is called or an allocation fails, hence has a performance advantage, but can be more difficult to dimension or manage memory requirements.

1 (immediate)

As memory is returned to the thread-local heap, if the object is ≥64MB then the memory is returned to the OS instead. This has an associated performance overhead. As per defered mode, memory used by the heap may be subsequently returned to the OS when either .Q.gc[] is called or an allocation fails.

When q is denied additional address space from the OS, it invokes .Q.gc[] and retries the request to the OS. If the subsequent attempt fail, the request exits with 'wsfull.

Notes on the allocator

Q’s allocator bins objects in power-of-two size categories, from 16b (e.g. an atom) to 64MB.

In this example, various vectors of longs (8 bytes per long) are created of different sizes using til. The memory used for the operation is shown via \ts. Note that more bytes are reported that only the pure vector size due to other house keeping, for example the type information.

q)\ts til 800     / 800*8=6400, needs a 2^13=8192 byte slab (too big for a 2^12=4096 byte slab)
0 8368
q)\ts til 1000    / 1000*8=8000, needs a 2^13=8192 byte slab (memory same as smaller vector above)
0 8368
q)\ts til 1200    / 1200*8=9600, cant fit in a 2^13=8192 bytes slab, needs 2^14=16384 byte slab
0 16560

If there is already a slab in the object category’s freelist, it is reused. If there are no available slabs, a larger slab is recursively split in two until the needed category size is reached. If there are no free slabs available, a new 64MB slab is requested from the system. When an object is de-allocated, its memory slab is returned to the corresponding category’s freelist.

Allocations larger than 64MB are requested from the OS directly, and this is what -g 1 causes to be immediately returned.

Note that larger allocations do not cause any fragmentation and in case of -g 1 always immediately return.

It is the smaller allocations (<64MB) that typically represent the bulk of a process allocation workload that can cause the heap to become fragmented.

There are two primary cases of heap fragmentation:

split slab

Suppose that at some point q needed a 32MB allocation. It requested a new 64MB slab from the OS, split it in half, used and freed the object, and returned the two 32MB slabs to the freelist. Now if q needs to allocate 64MB, it will have to make another request to the OS. When .Q.gc is called (or an allocation fails), it would attempt to coalesce these two 32MB slabs together back into one 64MB, which would allow it to be returned to the OS (or reused for larger allocations, if the resulting slab is <64MB).

leftover objects

If most of the objects allocated from a 64MB slab are freed but one remains, the slab still cannot be returned to the OS (or coalesced).

The following example shows freeing an object ≥64MB in deferred mode, while inspecting memory usage via .Q.w[]:

q).Q.w[]`used`heap    / original memory used and memory reserved by kdb+ at time of test
371552 67108864
q)a:til 10000000      / need memory ≥64MB to store value
q).Q.w[]`used`heap    / heap (memory reserved by kdb+) has grown, and used memory grown from the heap has grown
134589328 201326592
q)a:1                 / variable assigned different value, old value no longer used
q).Q.w[]`used`heap    / heap (memory reserved by kdb+) hasn't reduced as it is kept for future use, used memory has reduced
371616 201326592
q)a:til 10000000      / need memory ≥64MB to store value again
q).Q.w[]`used`heap    / heap memory (no increase) as memory used has been taken from the available heap
134589328 201326592

The same example will differ when using immediate mode, by returning memory to the OS (as the object free'd is greater than 64MB):

q).Q.w[]`used`heap    / original memory used and memory reserved by kdb+ at time of test
371648 67108864
q)a:til 10000000      / need memory ≥64MB to store value
q).Q.w[]`used`heap    / heap (memory reserved by kdb+) has grown, and used memory from the heap has grown
134589424 201326592
q)a:1                 / variable assigned different value, old value no longer used
q).Q.w[]`used`heap    / heap (memory reserved by kdb+) has reduced, it has been returned to OS
371712 67108864
q)a:til 10000000      / need memory ≥64MB to store value again
q).Q.w[]`used`heap    / heap memory has increased (requested from OS) as memory used is more than whats available to use in heap
134589328 201326592

Immediate mode will not return the memory to the OS when several objects less than 64MB each are freed, even though their sum may be more than 64MB. In this situation, immediate and deferred mode operate identically by adding the freed memory to the heap for future use.

The following examples shows this effect when running in immediate mode. No memory is returned to the OS on freeing the objects, and only when .Q.gc[] is run is the memory coalesced and freed.

q).Q.w[]`used`heap               / original memory used and memory reserved by kdb+ at time of test
371648 67108864
q)v:`a`b`c`d`e`f`g`h`i`j         / create a list of 10 variable names to use
q){set[x;til 1000000]} each v    / create a global variable using each of the names in v, each containing 1000000 longs
q).Q.w[]`used`heap               / heap (memory reserved by kdb+) has grown, and used memory from the heap has grown
84258096 134217728
q)![`.;();0b;v]                  / delete all the variables and their contents
q).Q.w[]`used`heap               / used memory has been reduced, but none of the heap memory has returned to the OS
371824 134217728
q).Q.gc[]                        / running garbage collection freed over 64MB
67108864

Command-line option -g (garbage collection mode), Command-line parameter -w (workspace memory limit), System command \w (memory stats and workspace memory limit)
Q for Mortals §13.1.10 Garbage Collection \g

\l (load file or directory)

\l name
\l .

Where name is the name of a

• q script, executes the script
• serialized object, deserializes it into memory as variable name
• directory of a splayed table, maps the table to variable name, without loading any columns into memory
• directory and the value of one of the permitted partition types, the most recent partition directory is inspected for splayed directories and each such directory mapped into memory with the name of the splayed directory
• directory containing a kdb+ database, recursively loads whatever it finds there: serialized objects, scripts, splayed tables, etc.

Current directory When a directory is opened, it becomes the current directory.

Reload current directory You can reload the current database with \l .. This will ignore scripts and reload only data.

Never mind the dollars If a file or directory under the path being loaded has a dollar-sign suffix then it is ignored. e.g. db/tickdata/myfile$ and db/tickdata/mydir$ would be ignored on \l db/tickdata or on \l . if db/tickdata is the current directory.

q)\l sp.q            / load sp.q script
...
q)\a                 / tables defined in sp.q
`p`s`sp
q)\l db/tickdata     / load the data found in db/tickdata
q)\a                 / with tables quote and trade
`p`quote`s`sp`trade

If logging is enabled, the command checkpoints the .qdb file and empties the log file.

Operating systems may create hidden files, such as DS_Store, that block \l on a directory.

load, .Q.l (load)
Logging
Q for Mortals §10.3 Scripts, §13.2.6 Logging -l and -L

\o (offset from UTC)

\o
\o n

Show or set the local time offset, as integer n hours from UTC, or as minutes if abs[n]>23. The initial value of 0N means the machine’s offset is used.

q)\o
0N
q).z.p                        / UTC
2010.05.31D23:45:52.086467000
q).z.P                        / local time is UTC + 8
2010.06.01D07:45:53.830469000
q)\o -5                       / set local time as UTC - 5
q).z.P
2010.05.31D18:45:58.470468000
q)\o 390                      / set local time as UTC + 6:30
q).z.P
2010.06.01D06:16:06.603981000

This corresponds to the -o command line parameter.

Command-line option -o

\p (listening port)

Show or set listening port

\p [rp,][hostname:][portnumber|servicename]

See Listening port for detail.

hopen
-p command-line option
Multithreaded input mode, Changes in 3.5
Socket sharding with kdb+ and Linux

\P (precision)

\P
\P n

Show or set display precision for floating-point numbers, i.e. the number of digits shown.

The default value of n is 7 and possible values are integers in the range [0,17]. A value of 0 means use maximum precision. This is used when exporting to CSV files.

q)\P                       / default
7i
q)reciprocal 7             / 7 digits shown
0.1428571
q)123456789                / integers shown in full
123456789
q)123456789f               / floats shown to 7 significant digits
1.234568e+08

q)\P 3
q)1%3
0.333

q)\P 10
q)1%3
0.3333333333

Use .Q.fmt and .q.f to format numbers to given width and precision

q).Q.fmt[8;6]a            / format to width 8, 6 decimal places
"0.142857"
q).Q.f[2;]each 9.996 34.3445 7817047037.90  / format to 2 decimal places
"10.00"
"34.34"
"7817047037.90"

.Q.f (precision format), .Q.fmt (precision format with length)
Precision, -P command-line option, -27! precision format with IEEE754 rounding
What Every Computer Scientist Should Know About Floating-Point Arithmetic

\r (replication primary)

\r

This should not be executed manually otherwise it can disrupt replication. It is executed automatically by the replicating process on the primary process, and returns the log file name and log file count.

-r command-line option

\r (rename)

\r src dst

Rename file src to dst.

It is equivalent to the Unix mv command, or the windows move command (except that it will not rename to a different disk drive).

\s (number of secondary threads)

\s
\s N

Show or , where N is an integer, set the number of secondary threads available for parallel processing, within the limit set by the -s command-line option. N is an integer.

Since V3.5 2017.05.02, secondary threads can be adjusted dynamically up to the maximum specified on the command line. A negative N indicates processes should be used, instead of threads.

q)0N!("current secondary threads";system"s");system"s 4";0N!("current,max secondary threads";system"s";system"s 0N"); / q -s 8
("current secondary threads";0i)
("current,max secondary threads";4i;8i)
q)system"s 0" / disable secondary threads
q)system"s 0N" / show max secondary threads
8i

N    parallel processing uses
------------------------------------
>0   N threads
<0   processes with handles in .z.pd

For processes:

• peach or ': will call .z.pd for a list of handles to the processes, which must have been started previously
• the absolute value of -N in the command line is ignored

-s command-line option, Parallel processing

\S (random seed)

\S
\S n

Where n is

• omitted: display the last value to which the random seed was initialized
• 0N: display the current value of the random seed (since V3.6)
• non-zero integer: re-initialize the seed to n

Note that \S displays the last value to which the seed was initialized: it is not updated as the random-number generator (rng) is used.

q)\S                       / default
-314159i
q)5?10
8 1 9 5 4
q)5?10
6 6 1 8 5
q)\S -314159               / restore default seed
q)5?10                     / same random numbers generated
8 1 9 5 4
q)\S                       / seed is not updated
-314159
q)x:system "S 0N"          / current value of seed
q)r:10?10
q)system "S ",string x     / re-initialize seed
q)r~10?10
1b

Allows user to save and restore state of the rng. (Since V3.6 2017.09.26.)

q)x:system"S 0N";r:10?10;system"S ",string x;r~10?10
1b

Thread-local

Since V3.1 2013.08.19 random-number generation (rng) is thread-local. \S 1234 sets the seed for the rng for the main thread only. The rng in a secondary thread is assigned a seed based on the secondary thread number.

In multithreaded input mode, the seed is based on the socket descriptor.

Instances started on ports 20000 through 20099 (secondary threads, used with e.g. q -s -4 have the main thread’s default seed based on the port number.

\t (timer)

\t         / show timer interval
\t N       / set timer interval
\t exp     / time expression
\t:n exp   / time n repetitions of expression

This command has two different uses, according to the parameter. If the parameter is omitted, it shows the number of milliseconds between timer ticks: 0 means the timer is off.

N (integer)

Set the number of milliseconds between timer ticks. If 0, the timer is disabled, otherwise the timer is enabled and the first tick given. On each tick, the function assigned to .z.ts is executed.

This usage corresponds to the -t command-line option

[:n] e (expression)

A q expression e (other than a single integer) is executed and the execution time shown in milliseconds. Since V3.0 2011.11.22, if n is specified, e is executed n times.

q)/Show or set timer ticks
q)\t                           / default off
0
q).z.ts:{show`second$.z.N}
q)\t 1000                      / tick each second
q)13:12:52
13:12:53
13:12:54
\t 0                           / turn off

q)/Time an expression
q)\t log til 100000            / milliseconds for log of first 100000 numbers
3
q)\t:100 log til 100000        / timing for 100 repetitions
186

Actual timer tick frequency

The actual timer tick frequency is determined by the timing granularity supported by the underlying operating system. This can be considerably different from a millisecond.

\T (timeout)

\T
\T n

Show or set the client execution timeout, as n (integer) number of seconds a client call will execute before timing out. The default is 0: no timeout. Note this is in seconds, not milliseconds like \t.

-T command-line option

\ts (time and space)

\ts exp
\ts:n exp

Executes the expression exp and shows the execution time in milliseconds and the space used in bytes. (Since 3.1 2014.02.07)

q)\ts log til 100000
7 2621568

q)\ts:10000 log til 1000           /same as \ts do[10000; log til 1000]
329 24672

\u (reload user password file)

\u

When q is invoked with the -u parameter specifying a user password file, then \u will reload the password file. This allows updates to the password file while the server is running.

-u command-line option

\v (variables)

\v
\v ns

Lists the variables in namespace ns; defaults to current namespace.

q)a:1+b:2
q)\v
`a`b
q)\v .h
`HOME`br`c0`c1`logo`sa`sb`sc`tx`ty
q){x where x like"????"}system"v .h"
`HOME`logo

To expunge a from the default namespace

delete a from `.

Q for Mortals §12.5 Expunging from a Context

\w (workspace)

\w          / current memory usage
\w 0|1      / internalized symbols
\w n        / set workspace memory limit

With no parameter, returns current memory usage, as a list of 6 long integers.

0   number of bytes from the heap that are currently in use
1   heap size in bytes
2   maximum heap size so far
3   limit on thread heap size, from -w command-line option or \w system command
4   mapped bytes
5   physical memory

q)\w
168144 67108864 67108864 0 0 8589934592

\w 0 and \w 1 return a pair of longs:

0   number of internalized symbols
1   corresponding memory usage

q)\w 0
577 25436

The utility .Q.w formats all this information.

Run-time increase Since 2017.11.06, \w allows the workspace limit to be increased at run-time, if it was initialized via the -w command-line option. For example \w 128 sets the limit to 128MB if the -w command line option was specified with a smaller value. The operation will return the current setting in bytes.

If the system tries to allocate more memory than allowed, it signals -w abort and terminates with exit code 1.

Specifying too large a number will fall back to the same behavior as \w 0 or \w 1.

q)\w
339168 67108864 67108864 104857600 0 8589934592
q)\w 0
651 28009
q)\w 128
134217728
q)\w 1000000000
1048576000000000
q)\w 1000000000000
651 28009

If the workspace limit has not been set by the command-line option -w, an error is signalled.

q)\w 3
'-w init via cmd line

Domain-local Since V4.0 2020.03.17 returns information for the current memory domain only.

q)value each ("\\d .m";"\\w";"\\d .";"\\w")
::
353968 67108864 67108864 0 0 8589934592
::
354032 67108864 67108864 0 0 8589934592

-w workspace command-line option, \g (garbage-collection mode)
.m namespace (DAX-enabled filesystems)

\W (week offset)

\W
\W n

Show or set the start-of-week offset n, where 0 is Saturday. The default is 2, i.e Monday.

-W command-line option

\x (expunge)

\x .z.p*

By default, callbacks like .z.po are not defined in the session. After they have been assigned, you can restore the default using \x to delete the definition that was made.

q).z.pi                       / default has no user defined function
'.z.pi
q).z.pi:{">",.Q.s value x}    / assign function
q)2+3
>5
q)\x .z.pi                    / restore default

Works only for .z.p* variables defined in k before q.k is loaded

For example, as .z.ph is defined in q.k, there is no default for it to be reset to.

\z (date parsing)

\z
\z 0|1

Show or set the format for "D"$ date parsing. B is 0 for mm/dd/yyyy and 1 for dd/mm/yyyy.

q)\z
0
q)"D"$"06/01/2010"
2010.06.01
q)\z 1
q)"D"$"06/01/2010"
2010.01.06

-z command-line option

\1 & \2 (redirect)

\1 filename
\2 filename

\1 and \2 let you redirect stdout and stderr to files from within the q session. The files and intermediate directories are created if necessary.

~/q$ rm -f t1.txt t2.txt
~/q$ l64/q
KDB+ 4.0 2021.04.26 Copyright (C) 1993-2021 Kx Systems
...

q)\1 t1.txt              / stdout
q)\2 t2.txt              / stderr
til 10
2 + "hello"
\\

~/q$ cat t1.txt          / entry in stdout
0 1 2 3 4 5 6 7 8 9
~/q$ cat t2.txt          / entry in stderr
q)q)'type

On macOS and Linux \1 /dev/stdin returns output to the default.

\_ (hide q code)

\_               / show client write access
\_ scriptname    / make runtime script

This command has two different uses depending on whether a parameter is given.

If no parameter, then \_ checks if client write-access is blocked.

q)\_
0b

-b command-line option

If a parameter is given, it should be a scriptname and \_ f.q makes a runtime script f.q_. The q code loaded from a runtime script cannot be viewed or serialized.

q)`:t1.q 0:enlist "a:123;f:{x+2*y}"
q)\_ t1.q               / create locked script
`t1.q_
q)\l t1.q_              / can be loaded as usual
q)a                     / definitions are correct
123
q)f[10;1 2 3]
12 14 16
q)f                     / q code is not displayed
locked
q)-8!f                  / or serialized
'type
  [0]  -8!f
         ^
q)read0`:t1.q
"a:123;f:{x+2*y}"
q)read0`:t1.q_          / file contents are scrambled
"'\374E\331\207'\262\355"
"S\014%\210\0273\245"

Protecting proprietary code

\ (terminate)

At the debugger’s q)) prompt clears one level from the execution stack and (eventually) returns to the interactive session.

q)f:{g[]}
q)g:{'`xyz}
q)f[]
{g[]}
'xyz
@
{'`xyz}
::
q))\
q)

Debugging

Without a suspension, \ toggles in an out of the k interpreter.

If there is a suspension, this exits one level of the suspension. Otherwise, it toggles between q and k mode. (To switch languages from inside a suspension, type "\".)

q){1+x}"hello"
{1+x}
'type
+
1
"hello"
q))\                         / clear suspension (only one level)
q)\                          / toggle to k mode

\ (toggle q/k)

In the interactive session \ toggles between the q and k interpreters.

q)\
  \
  !5                  / this is k
0 1 2 3 4
  \
q)

The k programming language is exposed infrastructure.

Exposed infrastructure

\\ (quit)

\\

• In the interactive session type \\ at the prompt to quit the session.
• Inside a function, use value"\\\\" or exit 0 for the same result.

exit, value, .z.exit

Final comments

The text following \\ and white space is ignored by q. This is often useful in scripts where \\ can be followed by comments or usage examples.

Interrupt and terminate

Ctl-c signals an interrupt to the interpreter.

Some operations are coded so tightly the interrupt might not be registered.

Ctl-z will kill the q session. Nothing in memory is saved.

OS commands

If an expression begins with \ but is not recognized as a system command, then it is executed as an OS command.

Typos can get passed to the OS

q)\ls                 / usual ls command
"help.q"
"k4.lic"
"l64"
"odbc.k"
"profile.q"
"q.k"
..

---