get, set¶
Read or set the value of a variable or a KDB-X data file
get¶
Read or memory-map a variable or KDB-X data file
get x get[x]
Where x is
- the name of a global variable as a symbol atom
- a file or folder named as a symbol atom or vector
returns its value.
Signals an error if the file is not a KDB-X data file. For example, get signals a parse error on a text file.
Use get to map columns of splayed databases in and out of memory during queries, or to read q log files.
q)a:42
q)get `a
42
q)trade:([]time:09:30:00.000 09:30:01.000;sym:`IBM`MSFT;price:101.5 52.25;size:100 200)
q)`:NewTrade set trade / save trade data to file
`:NewTrade
q)t:get`:NewTrade / t is a copy of the table
q)t~trade
1b
q)`:SNewTrade/ set .Q.en[`:.;trade] / save splayed table
`:SNewTrade/
q)s:get`:SNewTrade/ / s has columns mapped on demand
q)-3!s / a reference to the table on disk
"+`time`sym`price`size!`:SNewTrade/"
q)get `:SNewTrade/sym / a single column of the splayed table
`sym$`IBM`MSFT
value is a synonym for get
By convention, value is used for other purposes, but the two keywords are interchangeable.
q)value "2+3"
5
q)get "2+3"
5
q)a:1 2 3
q)get `a
1 2 3
q)get `q`w`e!(1 2;3 4;5 6)
1 2
3 4
5 6
q)get (+;1;2)
3
set¶
Assign a value to a global variable
Persist an object as a file or directory
nam set y set[nam;y] /set global var nam
file set y set[file;y] /serialize y to file
dir set t set[dir;t] /splay t to dir
(file;lbs;alg;lvl) set y set[(file;lbs;alg;lvl);y] /write y to file, compressed and/or encrypted
(dir;lbs;alg;lvl) set t set[(dir;lbs;alg;lvl);t] /splay t to dir, compressed and/or encrypted
(dir;dic) set t set[(dir;dic);t] /splay t to dir, compressed and/or encrypted
Where
alg integer atom compression/encryption algorithm
dic dictionary compression/encryption specifications
dir filesymbol directory in the filesystem
file filesymbol file in the filesystem
lbs integer atom logical block size
lvl integer atom compression level
nam symbol atom valid q name
t table
y (any) any q object
Compression parameters alg, lbs, and lvl
Encryption parameters alg and lbs
Compression/Encryption specification dictionary
Examples:
q)`a set 42 / set global variable
`a
q)a
42
q)`:a set 42 / serialize object to file
`:a
q)t:([]tim:100?23:59;qty:100?1000) / splay table
q)`:tbl/ set t
`:tbl/
q)(`:zfile;17;2;6) set t / serialize compressed
`:zfile
q)(`:ztbl/;17;2;6) set t / splay table compressed
`:ztbl/
To encrypt, use algorithm 16. Encryption needs the OpenSSL libraries and a master key loaded with -36!; refer to Data at rest encryption.
q)(`:etbl/;17;16;6) set t / splay table encrypted
`:etbl/
When writing an anymap, set detects consecutive top-level objects that are the same object in memory (matching addresses), and writes each of them only once to save space.
In the following example, a1 has 6,000 items but only 3 distinct objects, so its data file (a1#) is the same size as that of a0.
q)a:("hi";"there";"world")
q)`:a0 set a
`:a0
q)`:a1 set a@where 1000 2000 3000
`:a1
q)(hcount`$":a0#")=hcount`$":a1#"
1b
Empty vectors without attributes are also deduplicated automatically when writing an anymap, including enum vectors when the enum name is sym.
Writing nested data read from an anymap (type 77) file, as happens when you compress a file, needs little extra memory. Here, copying 50 MB of strings uses less than 10 MB:
q)`:a set 500000 100#"abc"
`:a
q)1e7>last system"ts `:b set get`:a" / bytes used
1b
You can also use 1: to write anymaps; that page describes the differences.
Splayed table¶
To splay a table t to directory dir:
dirmust be a filesymbol that ends with a/tmust have no primary keys- Columns of
tmust be vectors or compound lists - Symbol columns in
tmust be fully enumerated
Format¶
set saves the data in a binary tag-value format that keeps the structure of the data as well as its values.
q)`:data/foo set 10 20 30
`:data/foo
q)read0 `:data/foo
"\376 \007\000\000\000\000\000\000\000\000\000\000\000\000\000"
"\000\000\000\000\000\000\000\024\000\000\000\000\000\000\000\036\000\000\000..
Setting variables in the KX namespaces can result in undesired and confusing behavior.
These are .h, .j, .Q, .q, .z, and any other namespaces with single-character names.
Compression/Encryption¶
The following forms write y or splay t with compression or encryption:
(fil;lbs;alg;lvl) set y / write y to fil, compressed and/or encrypted
(dir;lbs;alg;lvl) set t / splay t to dir, compressed and/or encrypted
Arguments lbs, alg, and lvl are compression parameters, encryption parameters, or both.
Splay table t to directory ztbl/ with gzip compression:
q)(`:ztbl/;17;2;6) set t
`:ztbl/
The following form takes a dictionary of specifications:
(dir;dic) set t / splay t to dir, compressed
The keys of dic are either column names of t or the null symbol `. The value of each entry is an integer vector: lbs, alg, and lvl.
To set compression or encryption for unspecified columns, add an entry for the null symbol, as in the following example, or set .z.zd.
The example uses algorithm 5 (zstd) for the unspecified columns, which needs the zstd library to be installed.
q)m1:1000000
q)t:([]a:m1?10;b:m1?10;c:m1?10;d:m1?10)
q)/ Specify compression for cols a, b and defaults for others
q)show dic:``a`b!(17 5 3;17 2 6;17 2 6)
| 17 5 3
a| 17 2 6
b| 17 2 6
q)(`:ztbl/;dic) set t / splay table compressed
`:ztbl/
Compression and performance
Compression can make set faster or slower. The performance impact depends mainly on the data characteristics and the storage speed.