Skip to content

Atomicity, integrity, and durability

This page explains what happens when a write to a database is interrupted, at the level of a single table and of a whole partition; how to publish a write so that readers only ever see a complete one; and what it additionally takes for that publish to survive a crash.

A KDB-X database is a tree of ordinary files and directories, and q writes them one at a time. Nothing in the format provides a transaction, so there is no point at which a half-finished write is hidden from readers or rolled back. Whether a reader sees a loud error or silently wrong data depends on exactly where the write stopped.

A single write is not atomic

Neither set nor upsert is atomic. If a write to a splayed table fails partway — a permission problem, a full disk, a killed process — what is left on disk is neither the old table nor the new one.

set writes the .d file first, then each column in the order .d lists them (not alphabetical order). So a failure leaves .d promising columns that were never written:

q)t:([] alpha:1 2 3; zeta:10 20 30; mid:1.5 2.5 3.5)
q)\mkdir -p t2/zeta      / put something in the way of column zeta
q)`:t2/ set t
't2/zeta. OS reports: Is a directory

The directory now looks like a table, and even reads back as one, because get trusts .d:

q)get `:t2/.d
`alpha`zeta`mid
q)get `:t2/
+`alpha`zeta`mid!`:t2/
q)get `:t2/alpha
1 2 3

but the column that was never reached doesn't exist, so any query touching it fails:

q)get `:t2/mid
't2/mid. OS reports: No such file or directory
q)select from get `:t2/
't2/mid. OS reports: No such file or directory

A partial upsert is worse, because nothing complains. Appending to a splayed table extends each column file in turn, so a failure partway leaves the columns at different lengths:

q)`:t3/ set t
`:t3/
q)\chmod 444 t3/zeta     / make column zeta unwritable
q)`:t3/ upsert ([] alpha:10 11; zeta:100 110; mid:9.5 8.5)
't3/zeta. OS reports: Permission denied
q){(x;count get x)} each `:t3/alpha`:t3/zeta`:t3/mid
`:t3/alpha 5
`:t3/zeta  3
`:t3/mid   3

The row count of a splayed table is the length of its first column, so the table now claims five rows and pads the short columns with nulls:

q)count get `:t3/
5
q)get `:t3/
alpha zeta mid
--------------
1     10   1.5
2     20   2.5
3     30   3.5
10
11

A partial append is silent

The two extra rows read back without an error, with nulls where the columns ran out. Nothing distinguishes them from genuine null data. This is why a write that was interrupted — including a process killed mid-upsert — counts as having corrupted the table, not as having partly succeeded.

Publishing a write atomically

The remedy in every case is the same shape: write the new data where no reader is looking, then move it into place in a single operation. Only that last step is visible, so a reader sees either the old data or the new, never a half-written directory.

Somewhere no reader is looking means a name the loader skips — a directory whose name ends in $ is ignored, so it can sit inside the database while it is being built. A $ cannot appear in a `: literal, where it would parse as the cast operator, so build the handle with hsym:

q)(hsym `$"db/new$/trade/") set ([] id:1 2 3; px:1 2 3.0)
`:db/new$/trade/

When nothing is there yet

mv is all you need. Moving a directory within one filesystem is a rename(2), and that is atomic:

mv 'db/new$/trade' db/trade

Replacing a table that already exists

Here mv stops working, and it fails quietly. rename(2) refuses to replace a directory that is not empty. mv does not report that as a failure. Faced with an existing directory it moves the source inside the target, reports success, and publishes nothing:

$ mv -f 'db/new$/trade' db/trade
$ echo $?
0
$ ls db/trade/
id  px  trade

q)\l db
q)count trade
2

Still the old two-row version, with the new one stranded in a subdirectory beneath it.

So you have to clear the target first, which takes two moves — one to park the old version, one to publish the new:

mv db/trade 'db/trade.prev$'      # park the old version
mv 'db/new$/trade' db/trade       # publish the new one

Each move is atomic and the old version survives for rollback, so this is enough for many purposes. But two atomic steps are not an atomic step. Between them the table does not exist, and a reader loading the database in that window sees a database without it:

q)\l db
q)tables[]
,`other

In a partitioned database the same gap is worse, for the reasons below: a table missing from the most recent partition vanishes from the database altogether, and one missing partition directory makes queries fail for every date.

The window is two metadata operations wide, so it is brief — but it is real. A process that dies between them leaves the database missing a table until someone intervenes. Where that is not acceptable, the swap has to be a single operation.

Park the old version under a $ name too

The loader doesn't skip trade.prev. Once the new table is in place, trade cannot be both a table and a namespace, and the load fails:

q)\l db
'trade.prev

Ending the name in $ keeps the parked copy out of the loader's way.

A single-operation swap, using a symlink

A directory cannot be replaced atomically, but a file can: rename(2) over an existing file is atomic. So make the published name a symlink, and swap the link rather than the directory.

The versions live inside the database, in a $-suffixed container, and the name in the database root becomes a link to one of them:

db/
├── versions$/
│   ├── trade.v1/
│   └── trade.v2/
├── other/
└── trade -> versions$/trade.v2

Build the first version, link it in, and q reads straight through the link:

q)(hsym `$"db/versions$/trade.v1/") set ([] id:1 2; px:1 2.0)
`:db/versions$/trade.v1/
q)\ln -s 'versions$/trade.v1' db/trade
()
q).Q.lo[`:db;0;0]
q)trade
id px
-----
1  1
2  2

Writing the next version touches nothing that a reader is reading, so it can take as long as it likes and fail as often as it likes. Only the final ln is visible to a reader:

q)(hsym `$"db/versions$/trade.v2/") set ([] id:1 2 3; px:1 2 3.0)
`:db/versions$/trade.v2/
q)\ln -sfn 'versions$/trade.v2' db/trade
()
q).Q.lo[`:db;0;0]
q)trade
id px
-----
1  1
2  2
3  3

The price is that the live data no longer lives where the table appears to be. versions$ is not scratch space — it is where the production table actually is, and the database root holds only a pointer to it.

That can be confusing to whoever inherits the database. It also has practical consequences: a backup must follow the links (rsync -L, tar -h) or cover the versions directory too, du on the root under-reports, and old versions need a retention policy rather than accumulating forever.

A few further details decide whether this works:

  • Do not use mv to replace the link. mv follows a symlink that points at a directory, so instead of replacing the link it moves the new link inside the old version:

    $ ln -s 'versions$/trade.v1' db/trade.new
$ mv -f db/trade.new db/trade      # does NOT replace db/trade
$ ls 'db/versions$/trade.v2/'
id  px  trade.new

  • ln -sfn is not quite atomic. It removes the old link and creates the new one, leaving a brief window in which the name does not exist. For a genuinely atomic swap, create the new link under a temporary name and rename(2) it over the old one — os.replace in Python, or mv -T on Linux where coreutils provides it:

    ln -s 'versions$/trade.v1' db/.trade.tmp
python3 -c "import os; os.replace('db/.trade.tmp','db/trade')"

  • Keep the old version until nothing is reading it. A process that has already mapped trade.v1 keeps that mapping regardless of where the symlink now points, and goes on serving the old data until it reloads. That is usually what you want — queries already running finish against a consistent snapshot — but it means you cannot delete the old directory immediately.

  • The versions can live on other storage. Because a link can point anywhere, the versions need not share a filesystem with the database, whereas mv between filesystems degrades to copy-then-delete and is not atomic at all.

On Linux there is one genuine atomic directory swap that avoids symlinks altogether — renameat2 with the RENAME_EXCHANGE flag, which exchanges two directories in a single operation, keeping the old version and leaving no window. It is not portable, needs filesystem support, and mv --exchange only exists in coreutils 9.5 and later, so reaching it usually means a few lines of ctypes.

A partition is not atomic either

A partition usually holds more than one table — a day of market data might be a trade directory and a quote directory side by side — and q writes them one after another. Between the two writes the partition exists but is incomplete, and that is visible to every reader.

It is worse than a missing directory, because q decides which tables the database has from the most recent partition. A table absent from that partition does not simply lack a day of data: it does not exist at all.

pdb/
├── 2026.01.01/
│   ├── quote/
│   └── trade/
└── 2026.01.02/
    └── trade/      (quote not written yet)

q)\l pdb
q)tables[]
,`trade
q)select count i by date from quote
'quote

quote is gone from the whole database, including the day that was complete. The error names the table, not the partition, which can lead a reader to misread the cause.

The mirror case is a table missing from an earlier partition. The table then exists, because the last partition has it, but any query that spans the gap fails on the missing directory:

q)select count i by date from quote
'/…/pdb/2026.01.01/quote. OS reports: No such file or directory

This is what .Q.chk repairs

.Q.chk fills in the missing table directories as empty tables, after which a query returns only the dates that hold data. Run it after a load that went wrong, and see database maintenance for the difference between repairing the disk with .Q.chk and patching the session with .Q.bv.

The fix is the same shape as for a single table: write where no reader is looking, and publish with a symlink. Because the loader only skips a $-suffixed directory in the database root, the staging area goes there rather than in the partition.

Build the new table under versions$, and nothing about the database has changed yet — the load still sees only trade:

q)(hsym `$"pdb/versions$/2026.01.02/quote/") set ([] id:3 4; bid:3 4.0)
`:pdb/versions$/2026.01.02/quote/
q)\l pdb
q)tables[]
,`trade

Adding the link is what publishes it, and the table appears complete:

q)\ln -s '../versions$/2026.01.02/quote' pdb/2026.01.02/quote
()
q)\l pdb
q)tables[]
`s#`quote`trade
q)select count i by date from quote
date      | x
----------| -
2026.01.01| 2
2026.01.02| 2

For a whole new partition, link the partition rather than each table, so that every table in it appears in one operation. The link lives in the database root, so its target is relative to that:

ln -s 'versions$/2026.01.02' pdb/2026.01.02

A partition directory may itself be a symlink — q resolves it like any other path, and the partition name still has to look like a partition value.

The same trick works for a whole database

Nothing here is specific to one table or one partition. Symlinking a whole database root lets you assemble a rebuilt database out of the way and publish it in one step.

Atomic is not durable

Everything above makes the visible switch — the rename(2) behind a mv or an ln -sfn — atomic: a concurrent reader never sees a half-written table. That says nothing about what survives a crash. The OS and the storage hardware are both free to reorder when writes actually reach stable storage, and nothing forces them to honor the order your script issued them in.

write returning success only means the kernel accepted the bytes into its page cache. Whether — and when — those bytes reach the disk is up to the kernel's writeback and the device's own write cache, neither of which promises to flush in program order. So the staged column files and the rename that publishes them are not necessarily written to stable storage in that order, even though your script performed the writes first and the rename after. If a hardware failure or power loss lands between them, the machine can come back up with the new symlink already durable, pointing at column files that are not — truncated, zero-filled, or simply absent. That is the same corrupt state described earlier, reached by a crash rather than a q error, and a reader has no way to tell the two apart.

fsync closes that gap by acting as a commit: it blocks until its target has actually reached stable storage, giving you a point after which the data is guaranteed to survive a crash. Two calls are needed, at two different points, because a file and a directory entry are different kinds of durable state:

  • Sync the staged data before flipping the link. Once set or upsert returns, fdatasync(2) — from the shell, sync -d — forces a file's bytes to disk without also waiting on metadata like its mtime, which nothing here depends on.
  • Sync the directory after flipping the link. A rename or ln is a write to the directory's metadata, not to any file's data, so fdatasync on the file doesn't cover it — you need a full fsync(2) (plain sync from the shell) on the directory that holds the new entry.
# after `set`/`upsert` into the staging area has returned
sync -d 'pdb/versions$/2026.01.02/quote'/*

# only now publish it...
ln -s '../versions$/2026.01.02/quote' pdb/2026.01.02/quote

# ...and make the publish itself durable
sync pdb/2026.01.02

Skip the first sync and a crash can leave the link durable while the data behind it is not. Skip the second and a crash can silently undo the publish — the data is safe on disk, but the directory entry pointing at it never was, so the reboot sees the old version again.

fsync only helps if the hardware keeps its word

This guarantee runs only as deep as the storage stack's willingness to honor it. A device with a volatile write cache that reports completion before data is actually persisted — common on consumer SSDs and some virtualized or network-backed disks — can make fsync return successfully for data that a power loss then still erases. There is no software fix for that from q's side; it is a property of the disk you chose.

q has no built-in fsync

There's no keyword for this — call it from the OS, as above, before performing the symlink flip described earlier in this page. How aggressively to sync is a real trade-off: syncing every column file adds latency to every write-down, so weigh it against how much a given version can afford to lose — the final publish of a partition matters more than an intermediate scratch write into versions$ that a rebuild would redo anyway.

Next steps