Skip to content

The KDB-X file format

This page explains the native KDB-X on-disk format: what its header records, how nested data is stored so it can stay mapped, and how a single file or directory is mapped into memory when you read it.

The KDB-X file format is the native on-disk format, and the one every keyword in this page writes unless you ask for something else. It adds a small header recording the data type, attributes, and length, and this is what lets q map a file back into memory as a typed object rather than reparse it.

A directory of such files is a KDB-X database. This page is about the format and how it maps. For the keywords that write it, see serializing as an object, splayed tables, and partitioned tables, and for loading a whole database at once, loading a database.

The file header

The header is 16 bytes, and its size does not depend on what the file holds. Save a million unsorted longs, and a million sorted longs, and compare the size:

q)a: til 1000000
q)save `a
`:a
q)b: `s#til 1000000
q)save `b
`:b

hcount shows both files are 16 bytes larger than the eight million bytes of data:

q)hcount `:a
8000016
q)hcount `:b
8000016

Those 16 bytes are what a raw binary write omits. Writing the same vector with 1: gives exactly the data and nothing else:

q)`:raw 1: b
`:raw
q)hcount `:raw
8000000

This is why the attribute survives a save but not a 1: write:

q)attr get `:a
`
q)attr get `:b
`s

The header is what makes a file readable

get needs the header to know what it is reading. A file written with 1: carries no type information, so get cannot interpret it at all — see writing bytes.

Mapped lists

A mapped list (anymap, type 77h) is a nested list stored so that its inner vectors stay mapped on disk instead of being copied to the heap. This type superseded the legacy mapped nested types (77h+t) by dropping the requirement that all elements share a type:

q)a: get`:a set (1 2 3;"cde")
q)b: get`:b set ("abc";"def")
q)77 77h~(type a;type b)
1b

Elements can be of any type, including lists, dictionaries, and tables:

q)a: get`:a set ((1 2;3 4);`time`price`vol!(.z.p;1.;100i);([]a:1 2;b:("ab";"cd")))
q)77 0h~(type a;type first a)
1b

Writing `:a 1: x rather than set lets mapped lists nest inside one another. Whatever the depth, every vector in the structure stays mapped, so q can work with them without copying to the heap — note that used does not move when a column is extracted:

q)a: get`:a 1: ((1 2;3 4);([]time:1000?.z.p;price:1000?100.);([]time:1000?.z.p;price:1000?200))
q)77 77h~(type a;type first a)
1b
q).Q.w[]`used`mmap
336736 40432
q)p: exec price from a[1]
q).Q.w[]`used`mmap
336736 40432

Saving compound data creates companion files beside the main one:

  • file# holds the underlying storage, and is always written. It stays mapped as long as any mapped object within it is still referenced.
  • file## holds the enumeration domain, and appears only when the data contains symbols. q enumerates symbol vectors and atoms against it automatically, and de-enumerates — and so always copies — on access.
q)`:file set ((`a`b;`b`c);0 1)   / the symbols cause file## to be written too
`:file

This is also how a nested column of a splayed table is stored.

Compressed mapped lists hold memory

Anything decompressed out of a file# stays in memory until the last reference to that file is gone. Watch out for this when combining mapped lists with compression.

Immediate and deferred memory mapping

When the target is a directory, a trailing slash on the handle decides how q maps it:

  • Immediate (no trailing slash): the directory's contents are memory-mapped at load time.
  • Deferred (trailing slash): nothing is mapped at load time; files are mapped and unmapped as they are accessed.

To see the difference, splay a table to disk:

q)`:/tmp/data/tab/ set ([] 100000?100; 100000?1000f)
`:/tmp/data/tab/

Three tools show what happens:

  • .Q.w for memory statistics, of which mmap is the mapped total.
  • .Q.s1 for the object's structure as a string.
  • \t to time a repeated query; higher is slower.

Loading without the trailing slash maps everything up front, and queries then run against memory:

q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q)\t:100 select from t
0
q)\t:100 select from t
0
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q).Q.s1 t
"+`x`x1!(81 96 32 46 99 88 61 54 31 6 25 49 61 76 30 10 37 90 92..."

With the trailing slash, mmap stays at zero and each query pays to map the columns it touches. .Q.s1 shows why: the table holds the path, not the data:

q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab/
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q)\t:100 select from t
11
q)\t:100 select from t
10
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q).Q.s1 t
"+`x`x1!`:/tmp/data/tab/"

Deferred mode therefore holds less memory over the life of the session, at the cost of mapping on every access. Note that used barely moves in either case: neither mode copies file contents onto the heap.

The trailing slash is significant on write too

It decides whether you get a single file or a directory of column files — see serializing as an object and splayed tables.

Next steps