Encrypting data at rest

This page explains how to encrypt KDB-X data on disk: generating and loading a master key, encrypting writes, reading the result back, what encryption does and does not cover, and the operational limits to plan around.

Data-at-rest encryption (DARE), also called transparent disk encryption, protects the contents of database files from anyone who can read the files but does not hold the key. Encryption is applied when a file is written and undone transparently when it is read, so no query changes — the only thing a session needs is the master key.

It uses the same three-parameter mechanism as compression, with the algorithm code selecting AES rather than a compression algorithm. That means a single write can compress and encrypt together.

Prerequisites

Encryption needs a CPU with the AES-NI instruction set and the OpenSSL shared libraries. Neither is optional in practice: without AES-NI everything involving encrypted data is dramatically slower, and without OpenSSL q cannot encrypt at all.

Check for AES-NI support

LinuxmacOSWindows

grep -m1 -o aes /proc/cpuinfo

sysctl -a | grep machdep.cpu.features | grep AES

# Windows has no native command to check AES-NI.
# Download Coreinfo from Microsoft Sysinternals, then run:
./coreinfo.exe -f | Select-String "AES"

Check OpenSSL from inside q

The openssl command-line version is a starting point, but it is not the test that matters:

openssl version
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

What matters is whether the q process can load the OpenSSL shared libraries at runtime, which is a separate question from what the CLI reports. Ask q directly — -26! reports the linkage it actually has:

q)(-26!)[]
SSLEAY_VERSION   | OpenSSL 3.0.7 1 Nov 2022
SSL_CERT_FILE    | /etc/pki/tls/server-crt.pem
SSL_CA_CERT_FILE | /etc/pki/tls/cacert.pem
SSL_CA_CERT_PATH | /etc/pki/tls
SSL_KEY_FILE     | /etc/pki/tls/server-key.pem
SSL_CIPHER_LIST  | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| YES

If q cannot load them, this call and every attempt to load a key (-36!) fail, naming the library it looked for:

q)(-26!)[]
'Could not initialize openssl. Error was dlopen(libssl.1.1.dylib, ...)
q)-36!(`:testkek.key;"MySecurePassword!")
'Encryption lib unavailable. dlopen(libcrypto.1.1.dylib, ...)

The library named in the error is the one to install, and it is not always the newest one: which OpenSSL version a build wants depends on the build. Take the name from the error rather than assuming that whatever openssl version reports will do.

Generate the master key

The master key is a 256-bit AES key that encrypts the per-file Data Encryption Keys (DEKs). Generate it with OpenSSL:

openssl rand 32 | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key -pass pass:"MySecurePassword!"

What each option does

• rand 32 — generates 32 random bytes
• aes-256-cbc — AES cipher with a 256-bit key
• -md SHA256 — SHA-256 for key derivation
• -salt — random salt, to defeat dictionary attacks
• -pbkdf2 — Password-Based Key Derivation Function 2
• -iter 50000 — 50,000 iterations, to slow brute-force attempts
• -pass ... — avoids an interactive prompt; do not use this form in production, where it would put the password in your shell history

The iteration count must be exactly 50000

KDB-X expects -iter 50000. Any other value is rejected when you load the key, with an error that points at the password rather than the real cause:

q)-36!(`:bad.key;"pw")
'Invalid password for bad.key

The strength of the whole scheme rests on the password and on where you keep the key file. Choose a high-entropy password — over 80 bits is the published recommendation — and then:

• store the key file outside the database directory
• back up the key file and its password separately and securely
• restrict file permissions to the KDB-X process owner

Load the master key

A process can decrypt and encrypt only after -36! has loaded the key. Keep the password out of the script by passing it through the environment:

Linux / macOSWindows

export KDB_MASTER_KEY_PW="MySecurePassword!"
q

$env:KDB_MASTER_KEY_PW = "MySecurePassword!"
q

q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)

Called with (::) instead of a key, -36! reports whether a key is already loaded — which works even in a process with no OpenSSL, and so is the cheapest way for a start-up script to check:

q)-36!(::)          / before loading
0b
q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)
q)-36!(::)          / after
1b

A third, boolean argument locks or unlocks the loaded key.

Where and when it can be called

Loading the key takes around 500ms, and it may be called only from the main thread and only under handle 0 — that is, from the process's own start-up script or console, never on behalf of a client connection. Plan for it in start-up, not on demand.

Errors from -36!

Error	Meaning
Encryption lib unavailable	failed to load the OpenSSL libraries
Invalid password	wrong password, or the iteration count is not 50000
Main thread only	called from a secondary thread
PKCS5_PBKDF2_HMAC	the library invocation failed
Restricted	not called under handle 0
Unrecognized key format	the master-key file is not in a recognized format

Encrypt data

Encryption is requested through the same (blockSize; algorithm; level) triple as compression. The algorithm code selects it:

Code	Meaning	Notes
16	AES-256-CBC, no compression	The usual choice where compression adds nothing
18	gzip then AES-256-CBC	2 + 16: compression code plus the encryption flag

16 is additive: adding it to a compression algorithm code asks for both, in one pass, with no need to compress and then encrypt separately.

Compressing before encrypting can leak information

Because the compressed size depends on the plaintext, an attacker who can observe file sizes may be able to infer something about the contents — the class of attack behind CRIME and BREACH. Whether that matters is a question about your threat model; if it does, encrypt without compressing.

Encrypt every write with .z.zd

Set .z.zd and every subsequent write is encrypted:

q).z.zd: 17 16 0       / 128KB blocks, AES-256-CBC, no compression
q)`:secure_table set ([] time: 10?.z.t; sym: 10?`3; price: 10?100f)
`:secure_table

Encrypt a single write

To encrypt one file rather than all of them, pass the parameters in the left argument of set:

q)t: ([] a: 1 2 3; b: 4 5 6)
q)(`:test.enc;17;16;0) set t
`:test.enc
q)-21!`:test.enc
compressedLength  | 168
uncompressedLength| 81
algorithm         | 16i
logicalBlockSize  | 17i
zipLevel          | 0i

Note the two lengths in that example: the encrypted file is larger than the 81 bytes it holds. Encryption adds a fixed header and pads to the cipher block size, so on a tiny file that overhead dominates. It is negligible on files of a realistic size — see measure the overhead.

Compress and encrypt in one pass

Algorithm 18 applies gzip and then AES, with the level applying to the compression stage:

q)(`:both.enc;17;18;6) set 1000000?10
`:both.enc
q)-21!`:both.enc
compressedLength  | 745648
uncompressedLength| 8000016
algorithm         | 18i
logicalBlockSize  | 17i
zipLevel          | 6i

Read encrypted data

Reading takes no parameters — each file records the algorithm that produced it. What it does need is the key. Without it, the file cannot be read at all:

q)get `:test.enc
'test.enc. no key loaded for encrypted file test.enc
  [0]  get `:test.enc
       ^

Load the key and the same file reads normally:

q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)
q)get `:test.enc
a b
---
1 4
2 5
3 6

Verify that a file is encrypted

-21! reports the algorithm actually used, so algorithm 16 confirms the file is encrypted:

q)(-21!`:secure_table)`algorithm
16i

The first eight bytes of the file carry a signature that distinguishes the two cases:

q)first system "head -c 8 secure_table"
"kxzippEd"

Signature	Meaning
kxzippEd	encrypted (capital E)
kxzipped	compressed but not encrypted

What is not encrypted

Encryption covers file contents. It does not cover the shape of the database, which stays readable to anyone who can list the directory:

• directory names — and therefore table and partition names
• file names — and therefore column names in a splayed table
• the .d file — the column list and its order

So a reader without the key still learns your schema, your table names, and how much data each partition holds. Encryption also provides confidentiality rather than integrity: beyond an HMAC-SHA256 over the metadata block, it does not prove that a file has not been tampered with.

If the directory structure itself is sensitive, DARE alone is not enough — combine it with filesystem or full-disk encryption.

Operational limits

Encrypted enumeration domains lock during append

Encryption and compression differ here, and the difference matters. An encrypted enumeration domain can be appended to — .Q.en interns new symbols into an encrypted sym file without complaint, where a compressed one refuses outright.

The catch is concurrency: the file is locked for writing during the append, and it must not be read while that is happening. Where one process writes down while others query, serialize the writes and keep readers off the domain while it grows.

Rotating the password is not rotating the key

The procedure below re-encrypts the master-key file under a new password. The encryption key itself, and therefore every encrypted file on disk, is unchanged. Rotating the actual key means re-encrypting all of your data, which is a much larger exercise — plan it separately, and do not treat password rotation as a substitute.

Rotate the master-key password

Decrypt the key file with the old password, re-encrypt it with the new one, and replace it:

Linux / macOSWindows

OLD_PW='MySecurePassword!'
NEW_PW='NewRotatedPassword2026!'

# Decrypt -> pipe -> encrypt
openssl aes-256-cbc -md SHA256 -d -iter 50000 -in testkek.key -pass pass:"$OLD_PW" | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key.new -pass pass:"$NEW_PW"

mv testkek.key.new testkek.key
export KDB_MASTER_KEY_PW="$NEW_PW"

$OLD_PW = 'MySecurePassword!'
$NEW_PW = 'NewRotatedPassword2026!'

# Decrypt -> pipe -> encrypt
openssl aes-256-cbc -md SHA256 -d -iter 50000 -in testkek.key -pass pass:"$OLD_PW" | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key.new -pass pass:"$NEW_PW"

Move-Item -Path "testkek.key.new" -Destination "testkek.key" -Force
$env:KDB_MASTER_KEY_PW = $NEW_PW

Because this only rewrites the key file, it is fast and carries no risk to the data — which is exactly why it should be routine.

Measure the overhead

Encryption costs space and time, and both are worth measuring rather than assuming.

On space, the published overhead is under 2% of typical database file sizes; -21! gives the figure for a given file.

On time, the dominant factor is whether AES-NI is being used: one published benchmark reports roughly a 400% difference between hardware acceleration enabled and disabled. You can reproduce that comparison on your own hardware. Create ebench.q:

/ load master key
-36!(`:testkek.key; getenv`KDB_MASTER_KEY_PW)

/ write an encrypted table
(`:etest;20;16;0) set 100000000?10000

/ time the read
system "ts max get`:etest"

Run it normally, then again with AES-NI disabled for OpenSSL:

Linux / macOSWindows

q ebench.q

OPENSSL_ia32cap="~0x200000200000000" q ebench.q

q ebench.q

$env:OPENSSL_ia32cap = "~0x200000200000000"
q ebench.q
$env:OPENSSL_ia32cap = $null

As with compression, use your own data, your own queries, and production-equivalent hardware: the overhead that matters is the one your workload actually pays.

Next steps

• Combine encryption with compression, which shares the same three parameters.
• See what the encrypted files themselves contain in the KDB-X file format.
• Read the Data At Rest Encryption white paper for the threat model, the comparison with full-disk encryption and PCI-DSS compliance, file locking for encrypted enumeration domains, and the cryptographic internals — AES-256-CBC, PBKDF2, HMAC-SHA256, and encrypt-then-MAC.