# Encrypting data at rest

_This page explains how to encrypt KDB-X data on disk: generating and loading a master key, encrypting writes, reading the result back, what encryption does and does not cover, and the operational limits to plan around._

**Data-at-rest encryption** (DARE), also called transparent disk encryption, protects the contents of database files from anyone who can read the files but does not hold the key. Encryption is applied when a file is **written** and undone transparently when it is **read**, so no query changes — the only thing a session needs is the master key.

It uses the same three-parameter mechanism as [compression](file-compression.md#compression-parameters), with the algorithm code selecting AES rather than a compression algorithm. That means a single write can compress and encrypt together.

## Prerequisites

Encryption needs a CPU with the **AES-NI** instruction set and the **OpenSSL** shared libraries. Neither is optional in practice: without AES-NI everything involving encrypted data is dramatically slower, and without OpenSSL q can't encrypt at all.

### Check for AES-NI support

=== "Linux"
    ```bash
    grep -m1 -o aes /proc/cpuinfo
    ```

=== "macOS"
    ```bash
    sysctl -a | grep machdep.cpu.features | grep AES
    ```

=== "Windows"
    ```powershell
    # Windows has no native command to check AES-NI.
    # Download Coreinfo from Microsoft Sysinternals, then run:
    ./coreinfo.exe -f | Select-String "AES"
    ```

### Check OpenSSL from inside q

The `openssl` command-line version is a starting point, but it is **not** the test that matters:
```bash
openssl version
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
```

What matters is whether the q process can load the OpenSSL _shared libraries_ at runtime, which is a separate question from what the CLI reports. Ask q directly — [`-26!`](../../ref/internal.md#-26x-ssl) reports the linkage it actually has:
```q
q)(-26!)[]
SSLEAY_VERSION   | OpenSSL 3.0.7 1 Nov 2022
SSL_CERT_FILE    | /etc/pki/tls/server-crt.pem
SSL_CA_CERT_FILE | /etc/pki/tls/cacert.pem
SSL_CA_CERT_PATH | /etc/pki/tls
SSL_KEY_FILE     | /etc/pki/tls/server-key.pem
SSL_CIPHER_LIST  | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| YES
```

If q cannot load them, this call and every attempt to load a key ([`-36!`](../../ref/internal.md#-36-load-master-key)) fail, naming the library it looked for:
```q
q)(-26!)[]
'Could not initialize openssl. Error was dlopen(libssl.1.1.dylib, ...)
q)-36!(`:testkek.key;"MySecurePassword!")
'Encryption lib unavailable. dlopen(libcrypto.1.1.dylib, ...)
```

The library named in the error is the one to install, and it's not always the newest one: which OpenSSL version a build wants depends on the build. Take the name from the error rather than assuming that whatever `openssl version` reports is enough.

## Generate the master key

The master key is a 256-bit AES key that encrypts the per-file Data Encryption Keys (DEKs). Generate it with OpenSSL:
```bash
openssl rand 32 | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key -pass pass:"MySecurePassword!"
```

??? info "What each option does"

    - **`rand 32`** — generates 32 random bytes
    - **`aes-256-cbc`** — AES cipher with a 256-bit key
    - **`-md SHA256`** — SHA-256 for key derivation
    - **`-salt`** — random salt, to defeat dictionary attacks
    - **`-pbkdf2`** — Password-Based Key Derivation Function 2
    - **`-iter 50000`** — 50,000 iterations, to slow brute-force attempts
    - **`-pass ...`** — avoids an interactive prompt; do not use this form in production, where it puts the password in your shell history

> **Warning — The iteration count must be exactly 50000**
>
>
> KDB-X expects `-iter 50000`. Any other value is rejected when you load the key, with an error that points at the password rather than the real cause:
>
> ```q
> q)-36!(`:bad.key;"pw")
> 'Invalid password for bad.key
> ```
>
The strength of the whole scheme rests on the password and on where you keep the key file. Choose a high-entropy password — over 80 bits is the published recommendation — and then:

- store the key file **outside** the database directory
- back up the key file and its password separately and securely
- restrict file permissions to the KDB-X process owner

## Load the master key

A process can decrypt and encrypt only after [`-36!`](../../ref/internal.md#-36-load-master-key) has loaded the key. Keep the password out of the script by passing it through the environment:

=== "Linux / macOS"
    ```bash
    export KDB_MASTER_KEY_PW="MySecurePassword!"
    q
    ```

=== "Windows"
    ```powershell
    $env:KDB_MASTER_KEY_PW = "MySecurePassword!"
    q
    ```
```q
q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)
```

Called with `(::)` instead of a key, `-36!` reports whether a key is already loaded — which works even in a process with no OpenSSL, and so is the cheapest way for a startup script to check:
```q
q)-36!(::)          / before loading
0b
q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)
q)-36!(::)          / after
1b
```

A third, boolean argument locks or unlocks the loaded key.

> **Note — Where and when it can be called**
>
>
> Loading the key takes around 500ms. It may be called only from the **main thread** and only under **handle 0** — that is, from the process's own startup script or console, never on behalf of a client connection. Plan for it in startup, not on demand.
>
??? info "Errors from `-36!`"

    | Error                        | Meaning                                             |
    |------------------------------|-----------------------------------------------------|
    | `Encryption lib unavailable` | failed to load the OpenSSL libraries                |
    | `Invalid password`           | wrong password, or the iteration count is not 50000 |
    | `Main thread only`           | called from a secondary thread                      |
    | `PKCS5_PBKDF2_HMAC`          | the library invocation failed                       |
    | `Restricted`                 | not called under handle 0                           |
    | `Unrecognized key format`    | the master-key file is not in a recognized format   |

## Encrypt data

Encryption is requested through the same `(blockSize; algorithm; level)` triple as [compression](file-compression.md#compression-parameters). The algorithm code selects it:

| Code | Meaning                     | Notes                                                 |
|------|-----------------------------|-------------------------------------------------------|
| `16` | AES-256-CBC, no compression | The usual choice where compression adds nothing       |
| `18` | `gzip` then AES-256-CBC     | `2` + `16`: compression code plus the encryption flag |

`16` is additive: adding it to a compression algorithm code asks for both, in one pass, with no need to compress and then encrypt separately.

> **Warning — Compressing before encrypting can leak information**
>
>
> Because the compressed size depends on the plaintext, an attacker who can observe file sizes might be able to infer something about the contents — the class of attack behind CRIME and BREACH. Whether that matters is a question about your threat model; if it does, encrypt without compressing.
>
### Encrypt every write with `.z.zd`

Set [`.z.zd`](../../ref/dotz.md#zzd-compressionencryption-defaults) and every subsequent write is encrypted:
```q
q).z.zd: 17 16 0       / 128KB blocks, AES-256-CBC, no compression
q)`:secure_table set ([] time: 10?.z.t; sym: 10?`3; price: 10?100f)
`:secure_table
```

### Encrypt a single write

To encrypt one file rather than all of them, pass the parameters in the left argument of [`set`](../../ref/get.md#compressionencryption):
```q
q)t: ([] a: 1 2 3; b: 4 5 6)
q)(`:test.enc;17;16;0) set t
`:test.enc
q)-21!`:test.enc
compressedLength  | 168
uncompressedLength| 81
algorithm         | 16i
logicalBlockSize  | 17i
zipLevel          | 0i
```

Note the two lengths in that example: the encrypted file is _larger_ than the 81 bytes it holds. Encryption adds a fixed header and pads to the cipher block size, so on a tiny file that overhead dominates. It is negligible on files of a realistic size — see [measure the overhead](#measure-the-overhead).

### Compress and encrypt in one pass

Algorithm `18` applies gzip and then AES, with the level applying to the compression stage:
```q
q)(`:both.enc;17;18;6) set 1000000?10
`:both.enc
q)-21!`:both.enc
compressedLength  | 745648
uncompressedLength| 8000016
algorithm         | 18i
logicalBlockSize  | 17i
zipLevel          | 6i
```

## Read encrypted data

Reading takes no parameters — each file records the algorithm that produced it. What it does need is the key. Without it, the file cannot be read at all:
```q
q)get `:test.enc
'test.enc. no key loaded for encrypted file test.enc
  [0]  get `:test.enc
       ^
```

Load the key and the same file reads normally:
```q
q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)
q)get `:test.enc
a b
---
1 4
2 5
3 6
```

## Verify that a file is encrypted

[`-21!`](../../ref/internal.md#-21x-compressionencryption-stats) reports the algorithm actually used, so algorithm `16` confirms the file is encrypted:
```q
q)(-21!`:secure_table)`algorithm
16i
```

The first eight bytes of the file carry a signature that distinguishes the two cases:
```q
q)first system "head -c 8 secure_table"
"kxzippEd"
```

| Signature  | Meaning                          |
|------------|----------------------------------|
| `kxzippEd` | encrypted (capital `E`)          |
| `kxzipped` | compressed but **not** encrypted |

## What is not encrypted

Encryption covers file _contents_. It does not cover the shape of the database, which stays readable to anyone who can list the directory:

- **directory names** — and therefore table and partition names
- **file names** — and therefore column names in a splayed table
- **the `.d` file** — the column list and its order

So a reader without the key still learns your schema, your table names, and how much data each partition holds. Encryption also provides **confidentiality rather than integrity**: beyond an HMAC-SHA256 over the metadata block, it does not prove that a file has not been tampered with.

If the directory structure itself is sensitive, DARE alone is not enough — combine it with filesystem or full-disk encryption.

## Operational limits

> **Warning — Encrypted enumeration domains lock during append**
>
>
> Encryption and compression differ here, and the difference matters. An encrypted enumeration domain **can** be appended to — [`.Q.en`](../../ref/dotq.md#en-enumerate-varchar-cols) interns new symbols into an encrypted `sym` file without complaint, where a [compressed one refuses outright](file-compression.md#compress-an-existing-database).
>
> The catch is concurrency: the file is **locked for writing** during the append, and it must **not be read** while that is happening. Where one process writes down while others query, serialize the writes and keep readers off the domain while it grows.
>
> **Warning — Rotating the password is not rotating the key**
>
>
> The procedure below re-encrypts the **master-key file** under a new password. The encryption key itself, and therefore every encrypted file on disk, is unchanged. Rotating the actual key means re-encrypting all of your data, which is a much larger exercise — plan it separately, and do not treat password rotation as a substitute.
>
## Rotate the master-key password

Decrypt the key file with the old password, re-encrypt it with the new one, and replace it:

=== "Linux / macOS"
    ```bash
    OLD_PW='MySecurePassword!'
    NEW_PW='NewRotatedPassword2026!'

    # Decrypt -> pipe -> encrypt
    openssl aes-256-cbc -md SHA256 -d -iter 50000 -in testkek.key -pass pass:"$OLD_PW" | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key.new -pass pass:"$NEW_PW"

    mv testkek.key.new testkek.key
    export KDB_MASTER_KEY_PW="$NEW_PW"
    ```

=== "Windows"
    ```powershell
    $OLD_PW = 'MySecurePassword!'
    $NEW_PW = 'NewRotatedPassword2026!'

    # Decrypt -> pipe -> encrypt
    openssl aes-256-cbc -md SHA256 -d -iter 50000 -in testkek.key -pass pass:"$OLD_PW" | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key.new -pass pass:"$NEW_PW"

    Move-Item -Path "testkek.key.new" -Destination "testkek.key" -Force
    $env:KDB_MASTER_KEY_PW = $NEW_PW
    ```

Because this only rewrites the key file, it is fast and carries no risk to the data — which is exactly why it can be routine.

## Measure the overhead

Encryption costs space and time, and both are worth measuring rather than assuming.

On **space**, the published overhead is under 2% of typical database file sizes; [`-21!`](#verify-that-a-file-is-encrypted) gives the figure for a given file.

On **time**, the dominant factor is whether AES-NI is being used: one published benchmark reports roughly a 400% difference between hardware acceleration enabled and disabled. You can reproduce that comparison on your own hardware. Create `ebench.q`:
```q
/ load master key
-36!(`:testkek.key; getenv`KDB_MASTER_KEY_PW)

/ write an encrypted table
(`:etest;20;16;0) set 100000000?10000

/ time the read
system "ts max get`:etest"
```

Run it normally, then again with AES-NI disabled for OpenSSL:

=== "Linux / macOS"
    ```bash
    q ebench.q

    OPENSSL_ia32cap="~0x200000200000000" q ebench.q
    ```

=== "Windows"
    ```powershell
    q ebench.q

    $env:OPENSSL_ia32cap = "~0x200000200000000"
    q ebench.q
    $env:OPENSSL_ia32cap = $null
    ```

As with [compression](file-compression.md#measure-the-impact-on-your-own-hardware), use your own data, your own queries, and production-equivalent hardware: the overhead that matters is the one your workload actually pays.

## Next steps

- Combine encryption with [compression](file-compression.md), which shares the same three parameters.
- See what the encrypted files themselves contain in [the KDB-X file format](../basics/work_with_files/kdb-files.md).
- Read the [Data At Rest Encryption white paper](https://code.kx.com/q/kb/dare/) for the threat model, the comparison with full-disk encryption and PCI-DSS compliance, file locking for encrypted enumeration domains, and the cryptographic internals — AES-256-CBC, PBKDF2, HMAC-SHA256, and encrypt-then-MAC.
