Skip to content

Working with KDB-X files

This page explains how to persist a q value in q's own on-disk format: writing and reading it with save, load, set and get, appending to it with hopen, and when a single file is the right choice.

Alongside the text and binary formats, q has a native on-disk format. It adds a small header recording the data type, attributes, and length, which is what lets q map a file back into memory as a typed object rather than reparse it — and a directory of such files is a KDB-X database.

Any q object can be saved to disk — a vector, a dictionary, a nested list, a function, a table. When we do it with a table, we often call the result a flat table, to distinguish it from the splayed and partitioned forms. Every keyword below writes it, so the type, length, and any attributes are preserved along with the data.

The file header

The header is 16 bytes, and its size does not depend on what the file holds. Save a million unsorted longs, and a million sorted longs, and compare the size:

q)a: til 1000000
q)save `a
`:a
q)b: `s#til 1000000
q)save `b
`:b

hcount shows both files are 16 bytes larger than the eight million bytes of data:

q)hcount `:a
8000016
q)hcount `:b
8000016

Those 16 bytes are what a raw binary write omits. Writing the same vector with 1: gives exactly the data and nothing else:

q)`:raw 1: b
`:raw
q)hcount `:raw
8000000

This is why the attribute survives a save but not a 1: write:

q)attr get `:a
`
q)attr get `:b
`s

The header is what makes a file readable

get needs the header to know what it is reading. A file written with 1: carries no type information, so get cannot interpret it at all — see writing bytes.

save and load

Keywords save and load let us serialize and write any q object to a file of the same name, by default in the working directory. That includes tables, making this the simplest way to persist one.

q)cities:([]city:`Tokyo`Delhi`Shanghai;pop:37435191 29399141 26317104)

q)key `:.                       / nothing in working directory
`symbol$()
q)save `cities
`:cities
q)key `:.                       / file in working directory
,`cities

q)delete cities from `.         / delete from memory
`.
q)cities
'cities
  [0]  cities
       ^

q)load `cities                  / load from filesystem
`cities
q)cities
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

Because save takes the file name from the variable name, you cannot choose what the file is called. To control the file name and location as well, use set.

Not only tables

save works on any global, so the same two keywords persist a vector or a dictionary:

q)v: til 5
q)save `v
`:v
q)get `:v
0 1 2 3 4

q)d: `a`b!1 2
q)save `d
`:d
q)get `:d
a| 1
b| 2

This matters more than it looks: objects like these sitting in a database root are loaded as variables and are queryable alongside the tables. See a database is more than tables.

Tip

By specifying the extension (for example .csv, .xls), you can also export the table in another format.

q)save `cities.csv
`:cities.csv
q)read0 `:cities.csv
"city,pop"
"Tokyo,37435191"
"Delhi,29399141"
"Shanghai,26317104"

No extension means the KDB-X format instead. The recognized extensions, and the rest of q's text and binary file handling, are covered in how to work with files.

While this is satisfactory for many needs, it lacks further customizable features. For more options for writing and reading, you need the keywords used to define save and load:

set and get

Keywords set and get differ from save and load:

  • set is a binary; its left argument says where in the filesystem to write
  • get returns the object's value rather than the name of the variable it has been assigned to

Notice the similarity of reading a value from memory to reading it from the filesystem.

q)get `:cities                      / from filesystem
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

q)get `cities                       / from memory
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

q)`:foo/bar/bigcities set cities
`:foo/bar/bigcities
q)get `:foo/bar/bigcities
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

As that last example shows, set creates any missing directories in the path, and overwrites an existing target rather than complaining.

value has an overload that does the same job as get, so value `:file and get `:file are interchangeable on a data file.

A trailing slash changes what you get

A trailing slash on the handle writes a directory of column files — a splayed table — rather than a single file:

q)`:file set cities      / one file
`:file
q)`:dir/ set cities      / a directory, one file per column
`:dir/

The slash matters on reading too, where it selects deferred rather than immediate memory mapping.

Loading a directory of files

Using load on a directory of individual object files gives a dictionary keyed by file name:

q)`:t/a set 1 2 3
`:t/a
q)`:t/b set 4 5 6
`:t/b
q)load `:t
`t
q)t
a| 1 2 3
b| 4 5 6

Instead, rload reads a directory as a splayed table, using its .d file to recover the columns; it's the counterpart to rsave:

q)`:t/ set ([] a: 1 2 3; b: 4 5 6)
`:t/
q)rload `t
`t
q)t
a b
---
1 4
2 5
3 6

To load a whole database rather than one object, use \l.

Appending with hopen

Opening a KDB-X data file with hopen appends typed values. The file stays a valid KDB-X file, and get reads back the extended list:

q)`:L set 10 20 30
`:L
q)h:hopen `:L
q)h[42]
7i
q)h 100 200
7i
q)hclose h
q)get `:L
10 20 30 42 100 200

This is the difference from opening a file that has no KDB-X header, where the same handle appends raw bytes and get cannot read the result at all. Which behavior you get depends on the file, not on how you open it.

Use cases

Serialization as a flat table suits a table that is

  • small relative to memory
  • frequently read
  • has most of its columns required by most queries

A table too large for that should be splayed, and one that grows over time partitioned.

Mapped lists

A mapped list (anymap, type 77h) is a nested list stored so that its inner vectors stay mapped on disk instead of being copied to the heap. This type superseded the legacy mapped nested types (77h+t) by dropping the requirement that all elements share a type:

q)a: get`:a set (1 2 3;"cde")
q)b: get`:b set ("abc";"def")
q)77 77h~(type a;type b)
1b

Elements can be of any type, including lists, dictionaries, and tables:

q)a: get`:a set ((1 2;3 4);`time`price`vol!(.z.p;1.;100i);([]a:1 2;b:("ab";"cd")))
q)77 0h~(type a;type first a)
1b

Writing `:a 1: x rather than set lets mapped lists nest inside one another. Whatever the depth, every vector in the structure stays mapped, so q can work with them without copying to the heap — note that used does not move when a column is extracted:

q)a: get`:a 1: ((1 2;3 4);([]time:1000?.z.p;price:1000?100.);([]time:1000?.z.p;price:1000?200))
q)77 77h~(type a;type first a)
1b
q).Q.w[]`used`mmap
336736 40432
q)p: exec price from a[1]
q).Q.w[]`used`mmap
336736 40432

Saving compound data creates companion files beside the main one:

  • file# holds the underlying storage, and is always written. It stays mapped as long as any mapped object within it is still referenced.
  • file## holds the enumeration domain, and appears only when the data contains symbols. q enumerates symbol vectors and atoms against it automatically, and de-enumerates — and so always copies — on access.
q)`:file set ((`a`b;`b`c);0 1)   / the symbols cause file## to be written too
`:file

This is also how a nested column of a splayed table is stored.

Compressed mapped lists hold memory

Anything decompressed out of a file# stays in memory until the last reference to that file is gone. Watch out for this when combining mapped lists with compression.

Immediate and deferred memory mapping

When the target is a directory, a trailing slash on the handle decides how q maps it:

  • Immediate (no trailing slash): the directory's contents are memory-mapped at load time.
  • Deferred (trailing slash): nothing is mapped at load time; files are mapped and unmapped as they are accessed.

To see the difference, splay a table to disk:

q)`:/tmp/data/tab/ set ([] 100000?100; 100000?1000f)
`:/tmp/data/tab/

Three tools show what happens:

  • .Q.w for memory statistics, of which mmap is the mapped total.
  • .Q.s1 for the object's structure as a string.
  • \t to time a repeated query; higher is slower.

Loading without the trailing slash maps everything up front, and queries then run against memory:

q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q)\t:100 select from t
0
q)\t:100 select from t
0
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q).Q.s1 t
"+`x`x1!(81 96 32 46 99 88 61 54 31 6 25 49 61 76 30 10 37 90 92..."

With the trailing slash, mmap stays at zero and each query pays to map the columns it touches. .Q.s1 shows why: the table holds the path, not the data:

q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab/
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q)\t:100 select from t
11
q)\t:100 select from t
10
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q).Q.s1 t
"+`x`x1!`:/tmp/data/tab/"

Deferred mode therefore holds less memory over the life of the session, at the cost of mapping on every access. Note that used barely moves in either case: neither mode copies file contents onto the heap.

The trailing slash is significant on write too

It decides whether you get a single file or a directory of column files — see set and get and splayed tables.

Next steps