Skip to content

get, set

Read or set the value of a variable or a KDB-X data file

get

Read or memory-map a variable or KDB-X data file

get x     get[x]

Where x is

  • the name of a global variable as a symbol atom
  • a file or folder named as a symbol atom or vector

returns its value.

Signals an error if the file is not a KDB-X data file. For example, get signals a parse error on a text file.

Use get to map columns of splayed databases in and out of memory during queries, or to read q log files.

q)a:42
q)get `a
42

q)trade:([]time:09:30:00.000 09:30:01.000;sym:`IBM`MSFT;price:101.5 52.25;size:100 200)
q)`:NewTrade set trade                  / save trade data to file
`:NewTrade
q)t:get`:NewTrade                       / t is a copy of the table
q)t~trade
1b
q)`:SNewTrade/ set .Q.en[`:.;trade]     / save splayed table
`:SNewTrade/
q)s:get`:SNewTrade/                     / s has columns mapped on demand
q)-3!s                                  / a reference to the table on disk
"+`time`sym`price`size!`:SNewTrade/"
q)get `:SNewTrade/sym                   / a single column of the splayed table
`sym$`IBM`MSFT

value is a synonym for get

By convention, value is used for other purposes, but the two keywords are interchangeable.

q)value "2+3"
5
q)get "2+3"
5
q)a:1 2 3
q)get `a
1 2 3
q)get `q`w`e!(1 2;3 4;5 6)
1 2
3 4
5 6
q)get (+;1;2)
3

eval, value

set

Assign a value to a global variable
Persist an object as a file or directory

nam set y                 set[nam;y]                /set global var nam
file set y                set[file;y]               /serialize y to file
dir set t                 set[dir;t]                /splay t to dir
(file;lbs;alg;lvl) set y  set[(file;lbs;alg;lvl);y] /write y to file, compressed and/or encrypted
(dir;lbs;alg;lvl) set t   set[(dir;lbs;alg;lvl);t]  /splay t to dir, compressed and/or encrypted
(dir;dic) set t           set[(dir;dic);t]          /splay t to dir, compressed and/or encrypted

Where

alg   integer atom     compression/encryption algorithm
dic   dictionary       compression/encryption specifications
dir   filesymbol       directory in the filesystem
file  filesymbol       file in the filesystem
lbs   integer atom     logical block size
lvl   integer atom     compression level
nam   symbol atom      valid q name
t     table
y     (any)            any q object

Compression parameters alg, lbs, and lvl

Encryption parameters alg and lbs
Compression/Encryption specification dictionary

Examples:

q)`a set 42                         / set global variable
`a
q)a
42

q)`:a set 42                        / serialize object to file
`:a

q)t:([]tim:100?23:59;qty:100?1000)  / splay table
q)`:tbl/ set t
`:tbl/

q)(`:zfile;17;2;6) set t            / serialize compressed
`:zfile

q)(`:ztbl/;17;2;6) set t            / splay table compressed
`:ztbl/

To encrypt, use algorithm 16. Encryption needs the OpenSSL libraries and a master key loaded with -36!; refer to Data at rest encryption.

q)(`:etbl/;17;16;6) set t           / splay table encrypted
`:etbl/

When writing an anymap, set detects consecutive top-level objects that are the same object in memory (matching addresses), and writes each of them only once to save space. In the following example, a1 has 6,000 items but only 3 distinct objects, so its data file (a1#) is the same size as that of a0.

q)a:("hi";"there";"world")
q)`:a0 set a
`:a0
q)`:a1 set a@where 1000 2000 3000
`:a1
q)(hcount`$":a0#")=hcount`$":a1#"
1b

Empty vectors without attributes are also deduplicated automatically when writing an anymap, including enum vectors when the enum name is sym.

Writing nested data read from an anymap (type 77) file, as happens when you compress a file, needs little extra memory. Here, copying 50 MB of strings uses less than 10 MB:

q)`:a set 500000 100#"abc"
`:a
q)1e7>last system"ts `:b set get`:a"   / bytes used
1b

You can also use 1: to write anymaps; that page describes the differences.

Splayed table

To splay a table t to directory dir:

  • dir must be a filesymbol that ends with a /
  • t must have no primary keys
  • Columns of t must be vectors or compound lists
  • Symbol columns in t must be fully enumerated

Splayed tables

Format

set saves the data in a binary tag-value format that keeps the structure of the data as well as its values.

q)`:data/foo set 10 20 30
`:data/foo
q)read0 `:data/foo
"\376 \007\000\000\000\000\000\000\000\000\000\000\000\000\000"
"\000\000\000\000\000\000\000\024\000\000\000\000\000\000\000\036\000\000\000..

Setting variables in the KX namespaces can result in undesired and confusing behavior.

These are .h, .j, .Q, .q, .z, and any other namespaces with single-character names.

Compression/Encryption

The following forms write y or splay t with compression or encryption:

(fil;lbs;alg;lvl) set y   / write y to fil, compressed and/or encrypted
(dir;lbs;alg;lvl) set t   / splay t to dir, compressed and/or encrypted

Arguments lbs, alg, and lvl are compression parameters, encryption parameters, or both.

Splay table t to directory ztbl/ with gzip compression:

q)(`:ztbl/;17;2;6) set t
`:ztbl/

The following form takes a dictionary of specifications:

(dir;dic) set t            / splay t to dir, compressed

The keys of dic are either column names of t or the null symbol `. The value of each entry is an integer vector: lbs, alg, and lvl.

To set compression or encryption for unspecified columns, add an entry for the null symbol, as in the following example, or set .z.zd. The example uses algorithm 5 (zstd) for the unspecified columns, which needs the zstd library to be installed.

q)m1:1000000
q)t:([]a:m1?10;b:m1?10;c:m1?10;d:m1?10)

q)/ Specify compression for cols a, b and defaults for others
q)show dic:``a`b!(17 5 3;17 2 6;17 2 6)
 | 17 5 3
a| 17 2 6
b| 17 2 6
q)(`:ztbl/;dic) set t               / splay table compressed
`:ztbl/

Compression and performance

Compression can make set faster or slower. The performance impact depends mainly on the data characteristics and the storage speed.


File system

File compression

Data at rest encryption (DARE)