Skip to content

Atomicity and integrity

This page explains what happens when a write to a database is interrupted, at the level of a single table and of a whole partition, and how to publish a write so that readers only ever see a complete one.

A KDB-X database is a tree of ordinary files and directories, and q writes them one at a time. Nothing in the format provides a transaction, so there is no point at which a half-finished write is hidden from readers or rolled back. Whether a reader sees a loud error or silently wrong data depends on exactly where the write stopped.

A single write is not atomic

Neither set nor upsert is atomic. If a write to a splayed table fails partway — a permission problem, a full disk, a killed process — what is left on disk is neither the old table nor the new one.

set writes the .d file first, then each column in the order .d lists them (not alphabetical order). So a failure leaves .d promising columns that were never written:

q)t:([] alpha:1 2 3; zeta:10 20 30; mid:1.5 2.5 3.5)
q)\mkdir -p t2/zeta      / put something in the way of column zeta
q)`:t2/ set t
't2/zeta. OS reports: Is a directory

The directory now looks like a table, and even reads back as one, because get trusts .d:

q)get `:t2/.d
`alpha`zeta`mid
q)get `:t2/
+`alpha`zeta`mid!`:t2/
q)get `:t2/alpha
1 2 3

but the column that was never reached does not exist, so any query touching it fails:

q)get `:t2/mid
't2/mid. OS reports: No such file or directory
q)select from get `:t2/
't2/mid. OS reports: No such file or directory

A partial upsert is worse, because nothing complains. Appending to a splayed table extends each column file in turn, so a failure partway leaves the columns at different lengths:

q)`:t3/ set t
`:t3/
q)\chmod 444 t3/zeta     / make column zeta unwritable
q)`:t3/ upsert ([] alpha:10 11; zeta:100 110; mid:9.5 8.5)
't3/zeta. OS reports: Permission denied
q){(x;count get x)} each `:t3/alpha`:t3/zeta`:t3/mid
`:t3/alpha 5
`:t3/zeta  3
`:t3/mid   3

The row count of a splayed table is the length of its first column, so the table now claims five rows and pads the short columns with nulls:

q)count get `:t3/
5
q)get `:t3/
alpha zeta mid
--------------
1     10   1.5
2     20   2.5
3     30   3.5
10
11

A partial append is silent

The two extra rows read back without an error, with nulls where the columns ran out. Nothing distinguishes them from genuine null data. This is why a write that was interrupted — including a process killed mid-upsert — should be treated as having corrupted the table rather than as having partly succeeded.

Publishing a write atomically

The remedy in every case is the same shape: write the new data where no reader is looking, then move it into place in a single operation. Only that last step is visible, so a reader sees either the old data or the new, never a half-written directory.

Somewhere no reader is looking means a name the loader skips — a directory whose name ends in $ is ignored, so it can sit inside the database while it is being built. A $ cannot appear in a `: literal, where it would parse as the cast operator, so build the handle with hsym:

q)(hsym `$"db/new$/trade/") set ([] id:1 2 3; px:1 2 3.0)
`:db/new$/trade/

When nothing is there yet

mv is all you need. Moving a directory within one filesystem is a rename(2), and that is atomic:

mv 'db/new$/trade' db/trade

Replacing a table that already exists

Here mv stops working, and it fails quietly. rename(2) refuses to replace a directory that is not empty:

$ python3 -c "import os; os.rename('db/new$/trade','db/trade')"
OSError 66 Directory not empty

mv does not report that as a failure. Faced with an existing directory it moves the source inside the target, reports success, and publishes nothing:

$ mv -f 'db/new$/trade' db/trade
$ echo $?
0
$ ls db/trade/
id  px  trade

q)\l db
q)count trade
2

Still the old two-row version, with the new one stranded in a subdirectory beneath it.

So the target has to be cleared first, which takes two moves — one to park the old version, one to publish the new:

mv db/trade 'db/trade.prev$'      # park the old version
mv 'db/new$/trade' db/trade       # publish the new one

Each move is atomic and the old version survives for rollback, so this is enough for many purposes. But two atomic steps are not an atomic step. Between them the table does not exist, and a reader loading the database in that window sees a database without it:

q)\l db
q)tables[]
,`other

In a partitioned database the same gap is worse, for the reasons below: a table missing from the most recent partition vanishes from the database altogether, and one missing partition directory makes queries fail for every date.

The window is two metadata operations wide, so it is brief — but it is real, and a process that dies between them leaves the database missing a table until someone intervenes. Where that is not acceptable, the swap has to be a single operation.

Park the old version under a $ name too

trade.prev is not skipped by the loader. Once the new table is in place, trade cannot be both a table and a namespace, and the load fails:

q)\l db
'trade.prev

Ending the name in $ keeps the parked copy out of the loader's way.

A single-operation swap, using a symlink

A directory cannot be replaced atomically, but a file can: rename(2) over an existing file is atomic. So make the published name a symlink, and swap the link rather than the directory.

The versions live inside the database, in a $-suffixed container, and the name in the database root becomes a link to one of them:

db/
├── versions$/
│   ├── trade.v1/
│   └── trade.v2/
├── other/
└── trade -> versions$/trade.v2

Build the first version, link it in, and q reads straight through the link:

q)(hsym `$"db/versions$/trade.v1/") set ([] id:1 2; px:1 2.0)
`:db/versions$/trade.v1/
q)\ln -s 'versions$/trade.v1' db/trade
()
q).Q.lo[`:db;0;0]
q)trade
id px
-----
1  1
2  2

Writing the next version touches nothing that is being read, so it can take as long as it likes and fail as often as it likes. Only the final ln is visible to a reader:

q)(hsym `$"db/versions$/trade.v2/") set ([] id:1 2 3; px:1 2 3.0)
`:db/versions$/trade.v2/
q)\ln -sfn 'versions$/trade.v2' db/trade
()
q).Q.lo[`:db;0;0]
q)trade
id px
-----
1  1
2  2
3  3

The price is that the live data no longer lives where the table appears to be. versions$ is not scratch space — it is where the production table actually is, and the database root holds only a pointer to it. That can be confusing to whoever inherits the database, and it has practical consequences: a backup must follow the links (rsync -L, tar -h) or cover the versions directory too, du on the root under-reports, and old versions need a retention policy rather than accumulating forever.

A few further details decide whether this works:

  • Do not use mv to replace the link. mv follows a symlink that points at a directory, so instead of replacing the link it moves the new link inside the old version:

    $ ln -s 'versions$/trade.v1' db/trade.new
$ mv -f db/trade.new db/trade      # does NOT replace db/trade
$ ls 'db/versions$/trade.v2/'
id  px  trade.new

  • ln -sfn is not quite atomic. It removes the old link and creates the new one, leaving a brief window in which the name does not exist. For a genuinely atomic swap, create the new link under a temporary name and rename(2) it over the old one — os.replace in Python, or mv -T on Linux where coreutils provides it:

    ln -s 'versions$/trade.v1' db/.trade.tmp
python3 -c "import os; os.replace('db/.trade.tmp','db/trade')"

  • Keep the old version until nothing is reading it. A process that has already mapped trade.v1 keeps that mapping regardless of where the symlink now points, and goes on serving the old data until it reloads. That is usually what you want — queries already running finish against a consistent snapshot — but it means the old directory cannot be deleted immediately.

  • The versions can live on other storage. Because a link can point anywhere, the versions need not share a filesystem with the database, whereas mv between filesystems degrades to copy-then-delete and is not atomic at all.

On Linux there is one genuine atomic directory swap that avoids symlinks altogether — renameat2 with the RENAME_EXCHANGE flag, which exchanges two directories in a single operation, keeping the old version and leaving no window. It is not portable, needs filesystem support, and mv --exchange only exists in coreutils 9.5 and later, so reaching it usually means a few lines of ctypes.

A partition is not atomic either

A partition usually holds more than one table — a day of market data might be a trade directory and a quote directory side by side — and they are written one after another. Between the two writes the partition exists but is incomplete, and that is visible to every reader.

It is worse than a missing directory, because q decides which tables the database has from the most recent partition. A table absent from that partition does not simply lack a day of data: it does not exist at all.

pdb/
├── 2026.01.01/
│   ├── quote/
│   └── trade/
└── 2026.01.02/
    └── trade/      (quote not written yet)

q)\l pdb
q)tables[]
,`trade
q)select count i by date from quote
'quote

quote is gone from the whole database, including the day that was complete. The error names the table, not the partition, which makes the cause easy to misread.

The mirror case is a table missing from an earlier partition. The table then exists, because the last partition has it, but any query that spans the gap fails on the missing directory:

q)select count i by date from quote
'/…/pdb/2026.01.01/quote. OS reports: No such file or directory

This is what .Q.chk repairs

.Q.chk fills in the missing table directories as empty tables, after which a query returns only the dates that hold data. Run it after a load that went wrong, and see database maintenance for the difference between repairing the disk with .Q.chk and patching the session with .Q.bv.

The fix is the same shape as for a single table: write where no reader is looking, and publish with a symlink. Because a $-suffixed directory is only skipped in the database root, the staging area goes there rather than in the partition.

Build the new table under versions$, and nothing about the database has changed yet — the load still sees only trade:

q)(hsym `$"pdb/versions$/2026.01.02/quote/") set ([] id:3 4; bid:3 4.0)
`:pdb/versions$/2026.01.02/quote/
q)\l pdb
q)tables[]
,`trade

Adding the link is what publishes it, and the table appears complete:

q)\ln -s '../versions$/2026.01.02/quote' pdb/2026.01.02/quote
()
q)\l pdb
q)tables[]
`s#`quote`trade
q)select count i by date from quote
date      | x
----------| -
2026.01.01| 2
2026.01.02| 2

For a whole new partition, link the partition rather than each table, so that every table in it appears in one operation. The link lives in the database root, so its target is relative to that:

ln -s 'versions$/2026.01.02' pdb/2026.01.02

A partition directory may itself be a symlink — q resolves it like any other path, and the partition name still has to look like a partition value.

The same trick works for a whole database

Nothing here is specific to one table or one partition. Symlinking a whole database root lets you assemble a rebuilt database out of the way and publish it in one step.

Next steps