Skip to content

Working with q files

This page explains how to persist a q value in q's own on-disk format: writing and reading it with save, load, set, and get. It also covers appending to a file with hopen, and when a single file is the right choice.

Alongside the text and binary formats, q has a native on-disk format. It adds a small header recording the data type, attributes, and length, which is what lets q map a file back into memory as a typed object rather than parse it again. A directory of such files is a kdb database.

You can save any q object to disk — a vector, a dictionary, a nested list, a function, a table. A table saved this way is called a flat-file table (also called basic or non-splayed), to distinguish it from the splayed and partitioned forms. Every keyword below writes it, so each one preserves the type, length, and any attributes along with the data.

The file header

The header is 16 bytes for most types, and its size does not depend on the vector's length or attribute. Save a million longs:

q)a: til 1000000
q)save `a
`:a

hcount shows the file is 16 bytes larger than the 8 million bytes of data:

q)hcount `:a
8000016

Those 16 bytes are what a raw binary write omits. Writing the same vector with 1: gives exactly the data and nothing else:

q)`:raw 1: a
`:raw
q)hcount `:raw
8000000
Enumerated columns get a bigger header

An enumerated vector's header also has to record the name of the domain it points to, so it grows to 4096 bytes regardless of the domain name's length:

q)sym:`a`b`c
q)s:`sym$1000000?sym
q)save `s
`:s
q)`:raw 1: s
`:raw
q)hcount `:s
8004096
q)hcount `:raw
8000000

The header is what makes a file readable

get needs the header to know what it is reading. A file written with 1: carries no type information, so get cannot interpret it at all — see writing bytes.

save and load

Keywords save and load let you serialize and write any q object to a file of the same name, by default in the working directory. That includes tables, making this the shortest way to persist one.

q)cities:([]city:`Tokyo`Delhi`Shanghai;pop:37435191 29399141 26317104)

q)key `:.                       / nothing in working directory
`symbol$()
q)save `cities
`:cities
q)key `:.                       / file in working directory
,`cities

q)delete cities from `.         / delete from memory
`.
q)cities
'cities
  [0]  cities
       ^

q)load `cities                  / load from filesystem
`cities
q)cities
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

Because save takes the file name from the variable name, you cannot choose what the file is called. To control the file name and location as well, use set.

Save vectors and dictionaries

save works on any global, so the same keyword persists a vector or a dictionary just as it does a table, and get reads either back:

q)v: til 5
q)save `v
`:v
q)get `:v
0 1 2 3 4

q)d: `a`b!1 2
q)save `d
`:d
q)get `:d
a| 1
b| 2

q loads objects like these from a database root as variables, and you can query them alongside the tables. See a database is more than tables.

Tip

By specifying the extension (for example .csv, .xls), you can also export the table in another format.

q)save `cities.csv
`:cities.csv
q)read0 `:cities.csv
"city,pop"
"Tokyo,37435191"
"Delhi,29399141"
"Shanghai,26317104"

No extension means the q format instead. How to work with files covers the recognized extensions, and the rest of q's text and binary file handling.

save and load cannot set the file name, location, or on-disk layout. For those options, you need the keywords used to define them.

set and get

Keywords set and get differ from save and load:

  • set is a binary function; its left argument says where in the filesystem to write.
  • get returns the object's value rather than the name of the variable it has been assigned to.

Notice the similarity of reading a value from memory to reading it from the filesystem.

q)get `:cities                      / from filesystem
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

q)get `cities                       / from memory
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

q)`:foo/bar/bigcities set cities
`:foo/bar/bigcities
q)get `:foo/bar/bigcities
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

As that last example shows, set creates any missing directories in the path, and overwrites an existing target rather than complaining.

value has an overload that does the same job as get, so value `:file and get `:file are interchangeable on a data file.

A trailing slash changes what you get

A trailing slash on the handle writes a directory of column files — a splayed table — rather than a single file:

q)`:file set cities      / one file
`:file
q)`:dir/ set cities      / a directory, one file per column
`:dir/

The slash matters on reading too, where it selects deferred rather than immediate memory mapping.

Loading a directory of files

load on a directory of individual object files gives a dictionary keyed by file name:

q)`:t/a set 1 2 3
`:t/a
q)`:t/b set 4 5 6
`:t/b
q)load `:t
`t
q)t
a| 1 2 3
b| 4 5 6

Instead, rload reads a directory as a splayed table, using its .d file to recover the columns; it's the counterpart to rsave:

q)`:t/ set ([] a: 1 2 3; b: 4 5 6)
`:t/
q)rload `t
`t
q)t
a b
---
1 4
2 5
3 6

To load a whole database rather than one object, use \l.

Appending with hopen

Open a q data file with hopen and the handle appends typed values. The file stays a valid q file, and get reads back the extended list:

q)`:L set 10 20 30
`:L
q)h:hopen `:L
q)h[42]
7i
q)h 100 200
7i
q)hclose h
q)get `:L
10 20 30 42 100 200

This is the difference from opening a file that has no q header, where the same handle appends raw bytes and get cannot read the result at all. Which behavior you get depends on the file, not on how you open it.

Use cases

Serialization as a flat-file table suits a table that is:

  • Small relative to memory
  • Read frequently
  • Mostly read in full, rather than a few columns at a time

Splay a table too large for that, and partition one that grows over time. See splayed tables and partitioned tables.

Memory mapping and the heap

Data in a q session lives in one of two places: on the heap, or memory-mapped from a file. The difference decides how much memory a session uses, how fast it starts, and how it behaves when data is larger than RAM.

  • The heap is memory that q allocates for itself. Everything you create in a session — til 10000000, the result of a query, a table you build — is on the heap, as is anything q reads from a file by copying it. Heap data counts toward used in .Q.w, and it all has to fit in physical memory.
  • A memory map asks the operating system to present a file's bytes as if they were memory. q copies nothing when it maps a file. The operating system reads a page from disk only when something touches it, keeps recently used pages in its cache, and can discard them under memory pressure and read them again later. Mapped data counts toward mmap in .Q.w, not used.

Save a vector of ten million longs, then compare creating it with mapping it:

q)w: {`used`mmap#.Q.w[]}
q)w[]
used| 469280
mmap| 0
q)v: til 10000000        / on the heap
q)w[]
used| 134687008
mmap| 0
q)`:v set v
`:v
q)delete v from `.       / free the heap copy
`.
q)w[]
used| 469280
mmap| 0
q)m: get `:v             / map the file
q)w[]
used| 470864
mmap| 80000016
q)sum m                  / reads the pages it needs
49999995000000
q)w[]
used| 470864
mmap| 80000016

The mapped vector behaves like any other long vector, but used barely moves: 80MB of data is addressable, and none of it was copied to the heap. Deleting the last reference to it releases the mapping:

q)delete m from `.
`.
q)w[]
used| 470864
mmap| 0

What is mapped, and what is not

q can map data only when the bytes in the file are exactly what it needs in memory. That limits mapping to:

Mapped Copied to the heap
Vectors of a fixed-width type, such as longs and floats Symbol vectors, which q must intern in the symbol pool
The columns of splayed and partitioned tables Dictionaries and flat-file tables
Mapped lists, including nested columns Results of queries, whichever table they come from

Loading a database applies the same rules to every object in its root.

Compare the two

Heap Memory-mapped
Reported by .Q.w as used mmap
Cost to load Reads and copies the whole object Almost nothing — pages load as you touch them
Limit Physical memory Address space and disk, with the operating system's cache holding what is in use
First access Immediate May wait for the read from disk, so it depends on the storage
Changes Stay in the session Stay in the session. q does not write them back to the file
Suits Small, hot data, and anything you modify Large, read-mostly data

Queries copy what they select to the heap

A query on a mapped table reads only the columns it needs, but the result is an ordinary table on the heap. A select that returns most of a large table needs memory for that result, whatever the table's size on disk.

A modified mapped vector is a private copy

Amending a mapped vector changes it for the session only. The file keeps its original bytes, so get on it returns the original values. To change the data on disk, write it again with set.

Mapped lists

A mapped list (anymap, type 77h) is a nested list stored so that its inner vectors stay mapped on disk instead of being copied to the heap — see memory mapping and the heap. This type supersedes the legacy mapped nested types (77h+t) by dropping the requirement that all elements share a type:

q)a: get`:a set (1 2 3;"cde")
q)b: get`:b set ("abc";"def")
q)77 77h~(type a;type b)
1b

Elements can be of any type, including lists, dictionaries, and tables:

q)a: get`:a set ((1 2;3 4);`time`price`vol!(.z.p;1.;100i);([]a:1 2;b:("ab";"cd")))
q)77 0h~(type a;type first a)
1b

Write `:a 1: x rather than set to let mapped lists nest inside one another. Whatever the depth, all vectors in the structure stay mapped, so q can work with them without copying to the heap — note that used does not move when you extract a column:

q)a: get`:a 1: ((1 2;3 4);([]time:1000?.z.p;price:1000?100.);([]time:1000?.z.p;price:1000?200))
q)77 77h~(type a;type first a)
1b
q).Q.w[]`used`mmap
336736 40432
q)p: exec price from a[1]
q).Q.w[]`used`mmap
336736 40432

Saving compound data creates companion files beside the main one:

  • file# holds the flat, concatenated bytes of every vector nested inside the structure — the actual payload, as opposed to the shape recorded in file itself — and q always writes it. It stays mapped as long as anything still references a mapped object inside it.
  • file## holds the enumeration domain, and appears only when the data contains symbols. q enumerates symbol vectors and atoms against it automatically. On access it de-enumerates them, and so always copies.
q)`:file set ((`a`b;`b`c);0 1)   / the symbols cause file## to be written too
`:file

q stores a nested column of a splayed table the same way.

A column of same-shaped dictionaries is not a nested column

q collapses a list of dictionaries that all share the same keys into a table. Writing such a column hits set's table overload rather than its mapped-list handling, so it lands on disk as a flat-file table, not as an anymap column with #/## companions.

Immediate and deferred memory mapping

When the target is a directory, a trailing slash on the handle decides how q maps it:

  • Immediate (no trailing slash): q memory-maps the directory's contents at load time.
  • Deferred (trailing slash): q maps nothing at load time, and maps and unmaps each file as you access it.

To see the difference, splay a table to disk:

q)`:/tmp/data/tab/ set ([] 100000?100; 100000?1000f)
`:/tmp/data/tab/

Three tools show what happens:

  • .Q.w for memory statistics, of which mmap is the mapped total.
  • .Q.s1 for the object's structure as a string.
  • \t to time a repeated query; higher is slower.

Load without the trailing slash and q maps everything up front, so queries then run against memory:

q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q)\t:100 select from t
0
q)\t:100 select from t
0
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q).Q.s1 t
"+`x`x1!(81 96 32 46 99 88 61 54 31 6 25 49 61 76 30 10 37 90 92..."

With the trailing slash, mmap stays at zero and each query pays to map the columns it touches. .Q.s1 shows why: the table holds the path, not the data:

q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab/
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q)\t:100 select from t
11
q)\t:100 select from t
10
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q).Q.s1 t
"+`x`x1!`:/tmp/data/tab/"

Deferred mode therefore holds less memory over the life of the session, at the cost of mapping on every access. Note that used barely moves in either case: neither mode copies file contents onto the heap.

The trailing slash is significant on write too

It decides whether you get a single file or a directory of column files — see set and get and splayed tables.

Next steps