# Working with q files

_This page explains how to persist a q value in q's own on-disk format: writing and reading it with `save`, `load`, `set`, and `get`. It also covers appending to a file with `hopen`, and when a single file is the right choice._

Alongside the [text](text-files.md) and [binary](binary-files.md) formats, q has a _native on-disk format_. It adds a small header recording the data type, attributes, and length, which is what lets q [map a file back into memory](#memory-mapping-and-the-heap) as a typed object rather than parse it again. A directory of such files is a kdb database.

You can save any q object to disk — a vector, a dictionary, a nested list, a function, a table. A table saved this way is called a _flat-file table_ (also called basic or non-splayed), to distinguish it from the [splayed](../../interact_with_databases/splayed-tables.md) and [partitioned](../../interact_with_databases/partition.md) forms. Every keyword below writes it, so each one preserves the type, length, and any [attributes](../../../ref/set-attribute.md) along with the data.

## The file header

The header is 16 bytes for most types, and its size does not depend on the vector's length or attribute. Save a million longs:
```q
q)a: til 1000000
q)save `a
`:a
```

[`hcount`](../../../ref/hcount.md) shows the file is 16 bytes larger than the 8 million bytes of data:
```q
q)hcount `:a
8000016
```

Those 16 bytes are what a [raw binary write](binary-files.md#writing-bytes) omits. Writing the same vector with `1:` gives exactly the data and nothing else:
```q
q)`:raw 1: a
`:raw
q)hcount `:raw
8000000
```

??? note "Enumerated columns get a bigger header"

    An enumerated vector's header also has to record the name of the domain it points to, so it grows to 4096 bytes regardless of the domain name's length:
    ```q
    q)sym:`a`b`c
    q)s:`sym$1000000?sym
    q)save `s
    `:s
    q)`:raw 1: s
    `:raw
    q)hcount `:s
    8004096
    q)hcount `:raw
    8000000
    ```

> **Note — The header is what makes a file readable**
>
>
> [`get`](../../../ref/get.md#get) needs the header to know what it is reading. A file written with `1:` carries no type information, so `get` cannot interpret it at all — see [writing bytes](binary-files.md#writing-bytes).
>
## `save` and `load`

Keywords [`save`](../../../ref/save.md) and [`load`](../../../ref/load.md) let you serialize and write any q object to a file of the same name, by default in the working directory. That includes tables, making this the shortest way to persist one.
```q
q)cities:([]city:`Tokyo`Delhi`Shanghai;pop:37435191 29399141 26317104)

q)key `:.                       / nothing in working directory
`symbol$()
q)save `cities
`:cities
q)key `:.                       / file in working directory
,`cities

q)delete cities from `.         / delete from memory
`.
q)cities
'cities
  [0]  cities
       ^

q)load `cities                  / load from filesystem
`cities
q)cities
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104
```

Because `save` takes the file name from the variable name, you cannot choose what the file is called. To control the file name and location as well, use [`set`](#set-and-get).

### Save vectors and dictionaries

`save` works on any global, so the same keyword persists a vector or a dictionary just as it does a table, and `get` reads either back:
```q
q)v: til 5
q)save `v
`:v
q)get `:v
0 1 2 3 4

q)d: `a`b!1 2
q)save `d
`:d
q)get `:d
a| 1
b| 2
```

q loads objects like these from a database root as variables, and you can query them alongside the tables. See [a database is more than tables](../../interact_with_databases/overview.md#a-database-is-more-than-tables).

> **Tip**
>
> By specifying the extension (for example `.csv`, `.xls`), you can also export the table in another format.
>
> ```q
> q)save `cities.csv
> `:cities.csv
> q)read0 `:cities.csv
> "city,pop"
> "Tokyo,37435191"
> "Delhi,29399141"
> "Shanghai,26317104"
> ```
>
> No extension means the q format instead. [How to work with files](overview.md) covers the recognized extensions, and the rest of q's text and binary file handling.
>
`save` and `load` cannot set the file name, location, or on-disk layout. For those options, you need the keywords used to define them.

## `set` and `get`

Keywords [`set` and `get`](../../../ref/get.md) differ from `save` and `load`:

- `set` is a binary function; its left argument says where in the filesystem to write.
- `get` returns the object's value rather than the name of the variable it has been assigned to.

Notice the similarity of reading a value from memory to reading it from the filesystem.
```q
q)get `:cities                      / from filesystem
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

q)get `cities                       / from memory
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104

q)`:foo/bar/bigcities set cities
`:foo/bar/bigcities
q)get `:foo/bar/bigcities
city     pop
-----------------
Tokyo    37435191
Delhi    29399141
Shanghai 26317104
```

As that last example shows, `set` creates any missing directories in the path, and overwrites an existing target rather than complaining.

[`value`](../../../ref/value.md) has an overload that does the same job as `get`, so `` value `:file `` and `` get `:file `` are interchangeable on a data file.

> **Warning — A trailing slash changes what you get**
>
>
> A trailing slash on the handle writes a _directory of column files_ — a [splayed table](../../interact_with_databases/splayed-tables.md) — rather than a single file:
>
> ```q
> q)`:file set cities      / one file
> `:file
> q)`:dir/ set cities      / a directory, one file per column
> `:dir/
> ```
>
> The slash matters on reading too, where it selects [deferred rather than immediate memory mapping](#immediate-and-deferred-memory-mapping).
>
## Loading a directory of files

`load` on a directory of individual object files gives a dictionary keyed by file name:
```q
q)`:t/a set 1 2 3
`:t/a
q)`:t/b set 4 5 6
`:t/b
q)load `:t
`t
q)t
a| 1 2 3
b| 4 5 6
```

Instead, [`rload`](../../../ref/load.md#rload) reads a directory as a splayed table, using its `.d` file to recover the columns; it's the counterpart to [`rsave`](../../interact_with_databases/splayed-tables.md#rsave-and-rload):
```q
q)`:t/ set ([] a: 1 2 3; b: 4 5 6)
`:t/
q)rload `t
`t
q)t
a b
---
1 4
2 5
3 6
```

To load a whole database rather than one object, use [`\l`](../../interact_with_databases/loading-db.md#loading-with-l).

## Appending with `hopen`

Open a q data file with [`hopen`](../../../ref/hopen.md#files) and the handle appends typed values. The file stays a valid q file, and `get` reads back the extended list:
```q
q)`:L set 10 20 30
`:L
q)h:hopen `:L
q)h[42]
7i
q)h 100 200
7i
q)hclose h
q)get `:L
10 20 30 42 100 200
```

This is the difference from opening a file that has no q header, where the same handle appends [raw bytes](binary-files.md#appending-bytes-with-hopen) and `get` cannot read the result at all. Which behavior you get depends on the file, not on how you open it.

## Use cases

Serialization as a flat-file table suits a table that is:

- Small relative to memory
- Read frequently
- Mostly read in full, rather than a few columns at a time

Splay a table too large for that, and partition one that grows over time. See [splayed tables](../../interact_with_databases/splayed-tables.md) and [partitioned tables](../../interact_with_databases/partition.md).

## Memory mapping and the heap

Data in a q session lives in one of two places: on the heap, or memory-mapped from a file. The difference decides how much memory a session uses, how fast it starts, and how it behaves when data is larger than RAM.

- **The heap** is memory that q allocates for itself. Everything you create in a session — `til 10000000`, the result of a query, a table you build — is on the heap, as is anything q reads from a file by copying it. Heap data counts toward `used` in [`.Q.w`](../../../ref/dotq.md#w-memory-stats), and it all has to fit in physical memory.
- **A memory map** asks the operating system to present a file's bytes as if they were memory. q copies nothing when it maps a file. The operating system reads a page from disk only when something touches it, keeps recently used pages in its cache, and can discard them under memory pressure and read them again later. Mapped data counts toward `mmap` in `.Q.w`, not `used`.

Save a vector of ten million longs, then compare creating it with mapping it:
```q
q)w: {`used`mmap#.Q.w[]}
q)w[]
used| 469280
mmap| 0
q)v: til 10000000        / on the heap
q)w[]
used| 134687008
mmap| 0
q)`:v set v
`:v
q)delete v from `.       / free the heap copy
`.
q)w[]
used| 469280
mmap| 0
q)m: get `:v             / map the file
q)w[]
used| 470864
mmap| 80000016
q)sum m                  / reads the pages it needs
49999995000000
q)w[]
used| 470864
mmap| 80000016
```

The mapped vector behaves like any other long vector, but `used` barely moves: 80MB of data is addressable, and none of it was copied to the heap. Deleting the last reference to it releases the mapping:
```q
q)delete m from `.
`.
q)w[]
used| 470864
mmap| 0
```

### What is mapped, and what is not

q can map data only when the bytes in the file are exactly what it needs in memory. That limits mapping to:

| Mapped | Copied to the heap |
| --- | --- |
| Vectors of a fixed-width type, such as longs and floats | Symbol vectors, which q must intern in the symbol pool |
| The columns of [splayed](../../interact_with_databases/splayed-tables.md) and [partitioned](../../interact_with_databases/partition.md) tables | Dictionaries and flat-file tables |
| [Mapped lists](#mapped-lists), including nested columns | Results of queries, whichever table they come from |

[Loading a database](../../interact_with_databases/loading-db.md#what-l-maps) applies the same rules to every object in its root.

### Compare the two

| | Heap | Memory-mapped |
| --- | --- | --- |
| Reported by `.Q.w` as | `used` | `mmap` |
| Cost to load | Reads and copies the whole object | Almost nothing — pages load as you touch them |
| Limit | Physical memory | Address space and disk, with the operating system's cache holding what is in use |
| First access | Immediate | May wait for the read from disk, so it depends on the storage |
| Changes | Stay in the session | Stay in the session. q does not write them back to the file |
| Suits | Small, hot data, and anything you modify | Large, read-mostly data |

> **Warning — Queries copy what they select to the heap**
>
>
> A query on a mapped table reads only the columns it needs, but the result is an ordinary table on the heap. A `select` that returns most of a large table needs memory for that result, whatever the table's size on disk.
>
> **Note — A modified mapped vector is a private copy**
>
>
> Amending a mapped vector changes it for the session only. The file keeps its original bytes, so `get` on it returns the original values. To change the data on disk, write it again with [`set`](#set-and-get).
>
## Mapped lists

A _mapped list_ (`anymap`, type `77h`) is a nested list stored so that its inner vectors stay mapped on disk instead of being copied to the heap — see [memory mapping and the heap](#memory-mapping-and-the-heap). This type supersedes the legacy mapped nested types (`77h+t`) by dropping the requirement that all elements share a type:
```q
q)a: get`:a set (1 2 3;"cde")
q)b: get`:b set ("abc";"def")
q)77 77h~(type a;type b)
1b
```

Elements can be of any type, including lists, dictionaries, and tables:
```q
q)a: get`:a set ((1 2;3 4);`time`price`vol!(.z.p;1.;100i);([]a:1 2;b:("ab";"cd")))
q)77 0h~(type a;type first a)
1b
```

Write `` `:a 1: x `` rather than `set` to let mapped lists nest inside one another. Whatever the depth, all vectors in the structure stay mapped, so q can work with them without copying to the heap — note that `used` does not move when you extract a column:
```q
q)a: get`:a 1: ((1 2;3 4);([]time:1000?.z.p;price:1000?100.);([]time:1000?.z.p;price:1000?200))
q)77 77h~(type a;type first a)
1b
q).Q.w[]`used`mmap
336736 40432
q)p: exec price from a[1]
q).Q.w[]`used`mmap
336736 40432
```

Saving compound data creates companion files beside the main one:

- `file#` holds the flat, concatenated bytes of every vector nested inside the structure — the actual payload, as opposed to the shape recorded in `file` itself — and q always writes it. It stays mapped as long as anything still references a mapped object inside it.
- `file##` holds the enumeration domain, and appears only when the data contains symbols. q enumerates symbol vectors and atoms against it automatically. On access it de-enumerates them, and so always copies.
```q
q)`:file set ((`a`b;`b`c);0 1)   / the symbols cause file## to be written too
`:file
```

q stores a nested column of a [splayed table](../../interact_with_databases/splayed-tables.md#nested-columns) the same way.

> **Note — A column of same-shaped dictionaries is not a nested column**
>
>
> q collapses a list of dictionaries that all share the same keys into a table. Writing such a column hits `set`'s table overload rather than its mapped-list handling, so it lands on disk as a flat-file table, not as an `anymap` column with `#`/`##` companions.
>
## Immediate and deferred memory mapping

When the target is a directory, a trailing slash on the handle decides how q [maps](#memory-mapping-and-the-heap) it:

- **Immediate** (no trailing slash): q memory-maps the directory's contents at load time.
- **Deferred** (trailing slash): q maps nothing at load time, and maps and unmaps each file as you access it.

To see the difference, splay a table to disk:
```q
q)`:/tmp/data/tab/ set ([] 100000?100; 100000?1000f)
`:/tmp/data/tab/
```

Three tools show what happens:

- [`.Q.w`](../../../ref/dotq.md#w-memory-stats) for memory statistics, of which `mmap` is the mapped total.
- [`.Q.s1`](../../../ref/dotq.md#s1-string-representation) for the object's structure as a string.
- [`\t`](../../../ref/syscmds.md#t-timer) to time a repeated query; higher is slower.

Load without the trailing slash and q maps everything up front, so queries then run against memory:
```q
q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q)\t:100 select from t
0
q)\t:100 select from t
0
q)`used`mmap#.Q.w[]
used| 373360
mmap| 1600032
q).Q.s1 t
"+`x`x1!(81 96 32 46 99 88 61 54 31 6 25 49 61 76 30 10 37 90 92..."
```

With the trailing slash, `mmap` stays at zero and each query pays to map the columns it touches. `.Q.s1` shows why: the table holds the path, not the data:
```q
q)`used`mmap#.Q.w[]
used| 371616
mmap| 0
q)t:get`:/tmp/data/tab/
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q)\t:100 select from t
11
q)\t:100 select from t
10
q)`used`mmap#.Q.w[]
used| 373344
mmap| 0
q).Q.s1 t
"+`x`x1!`:/tmp/data/tab/"
```

Deferred mode therefore holds less memory over the life of the session, at the cost of mapping on every access. Note that `used` barely moves in either case: neither mode copies file contents onto the heap.

> **Note — The trailing slash is significant on write too**
>
>
> It decides whether you get a single file or a directory of column files — see [`set` and `get`](#set-and-get) and [splayed tables](../../interact_with_databases/splayed-tables.md).
>
## Next steps

- [Database overview](../../interact_with_databases/overview.md) — what a database holds, and the flat, splayed, partitioned, and segmented forms a table can take. You persist and query vectors, dictionaries, and nested lists the same way.
- [Splayed tables](../../interact_with_databases/splayed-tables.md), [partitioned tables](../../interact_with_databases/partition.md), and [segmented databases](../../interact_with_databases/segment.md) — a directory of column files, and the forms that scale beyond one.
- [Loading a database](../../interact_with_databases/loading-db.md) — loading a whole database at once, and what it costs in memory.
- [Compression](../../interact_with_databases/file-compression.md) and [encryption at rest](../../interact_with_databases/dare.md) — both configured on the same `set` call.
- [Text files](text-files.md) and [binary files](binary-files.md) — reading and writing the non-native formats.
