Send Feedback
Skip to content

Encrypt Data at Rest

Use this guide to set up, configure, and verify Transparent Disk Encryption (TDE) for KDB-X.

Overview

Check system requirements

First, verify that your environment supports these essential requirements:

  • Hardware: A CPU with the AES-NI instruction set
  • Software: OpenSSL 1.1.1 or later

Check for AES-NI support

Run the command for your OS to check for aes support:

grep -m1 -o aes /proc/cpuinfo

sysctl -a | grep machdep.cpu.features | grep AES

# Windows has no native command to check AES-NI.
# Download Coreinfo from Microsoft Sysinternals, then run:
./coreinfo.exe -f | Select-String "AES"

Performance impact

Hardware acceleration speeds up encryption. Without the AES-NI instruction set, database operations on encrypted data are significantly slower.

Check OpenSSL version

Next, ensure you have the correct software. You need OpenSSL 1.1.1+ to generate secure keys.

# Check installed OpenSSL version
openssl version
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
Verify KDB-X OpenSSL linkage

You can also check the version KDB-X uses from within the process:

q)// Check the OpenSSL version KDB-X is using
q)(-26!)[]
SSLEAY_VERSION   | OpenSSL 3.0.13 30 Jan 2024
SSL_CERT_FILE    | /usr/lib/ssl/server-crt.pem
SSL_CA_CERT_FILE | /usr/lib/ssl/cacert.pem
SSL_CA_CERT_PATH | /usr/lib/ssl
SSL_KEY_FILE     | /usr/lib/ssl/server-key.pem
SSL_CIPHER_LIST  | ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RS..
SSL_VERIFY_CLIENT| NO
SSL_VERIFY_SERVER| YES

Generate the master key

With requirements met, you need a master key to encrypt data. This master 256-bit AES key encrypts the unique Data Encryption Keys (DEKs) for each file.

  1. Use OpenSSL to generate a secure key:

    openssl rand 32 | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key -pass pass:"MySecurePassword!"
    Command breakdown
    • rand 32: Generates 32 random bytes
    • aes-256-cbc: Uses the AES cipher with a 256-bit key
    • -md SHA256: Uses SHA-256 for key derivation
    • -salt: Adds a random salt to protect against dictionary attacks
    • -pbkdf2: Uses the newer Password-Based Key Derivation Function 2
    • -iter 50000: Performs 50,000 iterations to slow down brute-force attacks
    • -pass ...: Automates password entry (Avoid this in production command history)

  2. Protect the key file: The testkek.key file is critical. Therefore, you must:

    • Save it outside your database directory
    • Back up the key and its password securely
    • Restrict file permissions to the KDB-X process owner

Start KDB-X with the master key

Once the key is generated, initialize KDB-X with encryption support by loading the master key at startup.

  1. Set the environment variable: Store the password in an environment variable to avoid hardcoding it.

    export KDB_MASTER_KEY_PW="MySecurePassword!"
q

    $env:KDB_MASTER_KEY_PW = "MySecurePassword!"
q

  2. Load the key: Then, use the -36! function to read and decrypt the key file.

    q)// Load master key using the environment variable
q)-36!(`:testkek.key; getenv `KDB_MASTER_KEY_PW)
q)show $[-36!(::);"Master key loaded"; "Failed to load master key"]
"Master key loaded"

Set encryption defaults

The process can now decrypt data. However, to encrypt new data automatically, you must configure the .z.zd handler.

Set .z.zd to (blockSize; algorithm; compressionLevel):

q)// Block size 17 (128kb), Algo 16 (AES256CBC), Level 0 (No compression)
q).z.zd:17 16 0;
Supported Algorithms
ID Description Usage
16 AES-256-CBC Encryption only. Best for data where compression adds overhead
18 Gzip + AES-256-CBC Compression then encryption. Warning: Risk of side-channel attacks (CRIME/BREACH)
1 IPC No encryption

Verify data encryption

After configuring the defaults, verify your configuration by writing and inspecting a test file.

Write a simple table to disk:

q)// Enable encryption
q).z.zd:17 16 0;
q)// Save an encrypted table
q)`:secure_table set([]time:10?.z.t;sym:10?`3;price:10?100f);
`:secure_table
q)// 1. Verify file signature
q)header:first system "head -c 8 secure_table";
q)show signature:$[header like "kxzippEd*";"Matched";"Failed"];
"Matched"
q)// 2. Verify compression stats
q)stats:-21!`:secure_table;
q)show algorithm:$[stats[`algorithm]=16i;"AES-256 Encrypted";"Unencrypted"];
"AES-256 Encrypted"
File signatures
Signature Status Description
kxzippEd Encrypted File is encrypted
kxzipped Unencrypted File is compressed but not encrypted

Rotate the master key

In addition to initial setup, rotate your master key regularly. This re-encrypts the key file, not your data, so it is safe and fast.

Decrypt the key with the old password, pipe it to a new encryption command, and replace the file:

OLD_PW='MySecurePassword!'
NEW_PW='NewRotatedPassword2026!'

# Decrypt -> Pipe -> Encrypt
openssl aes-256-cbc -md SHA256 -d -iter 50000 -in testkek.key -pass pass:"$OLD_PW" | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key.new -pass pass:"$NEW_PW"

# Replace key file
mv testkek.key.new testkek.key

# Update session variable
export KDB_MASTER_KEY_PW="$NEW_PW"

echo "Key rotation complete."

$OLD_PW = 'MySecurePassword!'
$NEW_PW = 'NewRotatedPassword2026!'

# Decrypt -> Pipe -> Encrypt
openssl aes-256-cbc -md SHA256 -d -iter 50000 -in testkek.key -pass pass:"$OLD_PW" | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out testkek.key.new -pass pass:"$NEW_PW"

# Replace key file
Move-Item -Path "testkek.key.new" -Destination "testkek.key" -Force

# Update session variable
$env:KDB_MASTER_KEY_PW = $NEW_PW

Write-Host "Key rotation complete."

Measure performance

Finally, be aware that encryption adds overhead. Measure the impact on your hardware with a benchmark.

  1. Create ebench.q:

    / Load master key
-36!(`:testkek.key;getenv`KDB_MASTER_KEY_PW)

/ Write an encrypted table
(`:etest;20;16;0) set 100000000?10000

/ Time the read operation
system "ts max get`:etest"

  2. Run with acceleration (AES-NI): 

    q ebench.q

  3. Run without acceleration to see the difference:

# Disable AES-NI for OpenSSL
OPENSSL_ia32cap="~0x200000200000000" q ebench.q

# Disable AES-NI for OpenSSL
$env:OPENSSL_ia32cap = "~0x200000200000000"
q ebench.q
$env:OPENSSL_ia32cap = $null

Review security risks

Visible metadata

The file content is encrypted, but file system metadata remains visible:

  • Directory names: Table names
  • Filenames: Column names (in splayed directories)
  • .d files: Column ordering files

Explore advanced encryption topics

This guide covers standard encryption setup. For deeper technical details and architectural considerations, read the Data At Rest Encryption white paper, which covers:

  • TDE vs. Full Disk Encryption (FDE): Comparison of security models and PCI-DSS compliance
  • File locking: Managing concurrency for encrypted enumeration domains
  • Compression risks: Understanding side-channel attacks when combining compression and encryption
  • Cryptographic internals: Technical details on AES256CBC, PBKDF2, and HMAC-SHA256 implementation

Summary

In this guide, you learned how to:

  • Verify requirements: Ensure CPU support (AES-NI) and OpenSSL versions
  • Generate keys: Create secure master keys using OpenSSL
  • Enable encryption: Start KDB-X with a master key and configure defaults
  • Verify protection: Confirm file signatures and encryption status
  • Rotate keys: Securely update master keys without re-encrypting data
  • Measure performance: Benchmark the encryption overhead