Skip to content

Data at rest encryption

Encryption of data is essential for organizations to meet compliance, regulatory and internal information security requirements.

kdb Insights Enterprise supports data at rest encryption (DARE), which encrypts data inside your databases using the kdb+ DARE feature. This uses Transparent Disk Encryption (TDE).

DARE is enabled by default for new kdb Insights Enterprise databases.

For upgrades to kdb Insights Enterprise using the kdb Insights Enterprise command line interface (CLI), the default behaviour is to maintain the existing databases with no change. Any new databases created after upgrading have DARE enabled by default.

Databases deployed with assemblies

Databases deployed with the deprecated kxi assembly command don't support DARE. If you upgrade with databases deployed from assemblies and want to encrypt them using DARE, you must convert them into packages before attempting to encrypt them.

DARE is not enabled by default for converted databases.

To encrypt a database deployed with the kxi assembly command:

  1. Convert the database to use packages by following the instructions in Converting assemblies to packages.

  2. Follow the instructions in the Enable DARE on an existing database section to enable DARE for your converted database.

The current status of DARE on your installation is visible in the Database Settings page in the UI (user interface).

Prerequisites

When you install or upgrade to version 1.11 or later of kdb Insights Enterprise, you must use version 1.11 or later of the kxi CLI. Any upgrades or installations performed with version 1.11 or later has all of the DARE prerequisites included and ready to use. Refer to Upgrade with CLI for more information.

You must have an encryption secret containing the database encryption key. Run kxi install setup to generate the kxi-encryption-secret.

Considerations

Availability

When you upgrade an existing deployment to kdb Insights Enterprise version 1.11 or later, any existing databases with data stored remain unencrypted. You can enable DARE on these databases after the upgrade. If you choose to encrypt an existing database using the toggle in the UI, that database becomes unavailable for ingest/query during encryption. You can check the status of encryption in the UI.

Resource usage and performance

Enabling DARE causes an increase in query times and CPU usage, usually a low single digit percentage increase. The precise figure depends on the hardware you use. For more information, refer to the performance section of the q and kdb+ DARE documentation.

During encryption, kdb Insights Enterprise creates a second copy of the database and performs the encryption on that copy. If encryption isn't successful or is interrupted, the copied version is deleted and the original, unencrypted database remains active. If encryption succeeds, the original database is deleted and the new encrypted version is used.

Storage space required for encryption

During encryption, you need to have at least twice as much storage space available as the current database size to accommodate the copy. This requirement only exists during the encryption process since the extra copy is deleted once encryption is complete.

Use DARE for new deployments

DARE is enabled by default on new deployments in version 1.11 or later of kdb Insights Enterprise. The kxi-encryption-secret is created automatically when you run the kxi install setup command, as in the following example:

$ kxi install setup
kdb Insights Enterprise Configuration Setup

Running in namespace insights on the cluster arn:aws:eks:us-east-2:xxxxxxxxxxx:cluster/xxxxxx

Checking ingress configuration...
Please enter the hostname for the installation: hostname.insights.kx.com
Using Let's Encrypt TLS certificates

Checking license details...
...
<skip to encryption prompt>
...
Checking database encryption configuration...
Secret kxi-encryption-secret successfully created

Configuration saved, kdb Insights Enterprise is now ready for install

Helm values file for installation saved in values.yaml

You can check that kdb Insights Enterprise has created an encryption secret successfully by using the following command:

$ kubectl get secret kxi-encryption-secret
NAME                    TYPE     DATA   AGE
kxi-encryption-secret   Opaque   2      2m20s

The encryption secret is named kxi-encryption-secret. If a secret already exists, kdb Insights Enterprise uses the existing secret instead of creating a new one.

The encryption block in your values.yaml should be similar to the following example:

global:
  caIssuer:
    name: kxi-certificate
    secretName: kxi-certificate
  encryption:
    auth:
      existingSecret: kxi-encryption-secret

Use DARE in an upgraded installation

Upgrading to a kdb Insights Enterprise version that has the DARE feature requires a kxi-encryption-secret to be present. Before you run the kxi install upgrade command, run either the kxi install setup or kxi install create-encryption-key command.

If you run kxi install upgrade without an encryption key, the pre-upgrade validation checks fail and the upgrade is prevented.

When you upgrade an existing installation of kdb Insights Enterprise to a version that includes the DARE feature, existing databases aren't automatically encrypted. New databases you create in that installation have DARE enabled by default.

Enable DARE on an existing database

To enable DARE on an unencrypted database after an upgrade, follow the steps in the relevant tab below:

  1. Teardown the database 

    kxi package teardown my-database

  2. Edit the package to include encryption settings in the deployment config. It may be necessary to create it first using kxi package add --to . deployment-config

    $schema=https://code.kx.com/insights/1.11/enterprise/packaging/schemas/package.json#/$defs/DeploymentConfig
uuid: <redacted>
name: deployment_config
attach: true
dbSettings:
encryption:
    encryptAll: true
    auth:
    existingSecret: kxi-encryption-secret

  3. Create a new package version and redeploy

    kxi package packit my-database --version 0.0.2 --tag
kxi package push my-database/0.0.2 --force --hostname ${INSIGHTS_HOSTNAME}
kxi package deploy my-database/0.0.2 --hostname ${INSIGHTS_HOSTNAME}

To use the UI to enable DARE, refer to the Encryption section of the Database Settings documentation.

Encryption or decryption begins immediately on deployment

When you redeploy a database to encrypt or decrypt a databases, the process starts immediately. The database is unavailable for ingest or query until the encryption or decryption process has completed successfully.

Estimated completion time

You can find an estimate of how long your database will take to finish the encryption or decryption process either through an API call in the CLI or finding a progress bar in the UI:

Use the following command to access the status API:

curl https://insights.kxi-dev.kx.com/servicegateway/database/my-database/status

Below is an example of a response from the above command:

{
    "state": "normal",
    "encryption": "encrypting",
    "progress": {
        "cmdCurrent": "idb_sl",
        "cmdIndex": 2,
        "cmdTotal": 4,
        "subCurrent": "op=apply.encryptCol tn=table1 a=.d",
        "subIndex": 4,
        "subTotal": 4,
        "completeETA": "0D00:20:00.006728252"
    },
    "memory": {
        "usage": 168960000,
        "limit": 4294967296
    }
}

You can find the estimated time remaining for encrypting or decrypting the database in the progress.completeETA field of the status API response. This ETA is continuously updated as each part of the database is encrypted.

In the UI, you can find a progress indicator at the top of the Database Settings page. For more information, refer to the Encryption section of the Database Settings documentation.

If the ETA is greater than the time your database is allowed to be unavailable, then you can interrupt the encryption process by tearing down the database and resetting the encryption. You can then find another suitable timeframe to allow the process to be completed.

Interrupt or cancel encryption

If you interrupt the encryption process, existing progress on database encryption is undone. You must complete the encryption in a single, uninterrupted operation.

Decrypt an encrypted database

To decrypt a database, go to the Database Settings page in the UI and toggle the Encryption setting to Off. Then, redeploy the database to begin the decryption process.

To decrypt an encrypted database, follow either of the below sets of steps:

  1. Teardown the database 

    kxi package teardown my-database

  2. Edit the package and set the encryptAll: value to false:

    $schema=https://code.kx.com/insights/1.11/enterprise/packaging/schemas/package.json#/$defs/DeploymentConfig
uuid: <redacted>
name: deployment_config
attach: true
dbSettings:
encryption:
    encryptAll: false
    auth:
    existingSecret: kxi-encryption-secret

  3. Create a new package version and redeploy

    kxi package packit my-database --version 0.0.2 --tag
kxi package push my-database/0.0.2 --force --hostname ${INSIGHTS_HOSTNAME}
kxi package deploy my-database/0.0.2 --hostname ${INSIGHTS_HOSTNAME}

To use the UI to decrypt a database, refer to the Encryption section of the Database Settings documentation.

Encryption secret

The encryption secret contains the encryption key used to encrypt and decrypt the database, and the password you use to access the encryption key. This section describes interactions with both the encryption key and password that the secret contains.

Change the password

Changing password requires the CLI

You must use the CLI to change your encryption password - this functionality isn't supported via the UI.

If you need to change the password for an encryption key, you must get the existing secret and then replace it with a new secret containing the updated password.

Follow the below steps to change your password safely and securely:

  1. Download your existing encryption key and get your existing password:

    secret=$(kubectl get secret kxi-encryption-secret -o json)
echo "$secret"|jq -r '.["data"]["key"]'|base64 -d >database.key
echo "$secret"|jq -r '.["data"]["password"]'|base64 -d

  2. Change the password:

    # Change password for master key:
# Prompt for existing password
key=`openssl aes-256-cbc -md SHA256 -d -iter 50000 -in kxi-encryption.key`  
# Prompt for a new password
echo $key | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out kxi-encryption-new.key 
# Remove the raw key from the environment
unset key

  3. Back up the existing encryption secret:

    kubectl get secret kxi-encryption-secret --namespace=insights -o yaml | sed 's/name: .*/name: kxi-encryption-secret-backup/' | kubectl apply -f -

  4. Delete the existing secret containing the old password:

    kubectl delete secret kxi-encryption-secret --namespace=insights

  5. Create the new secret with the new password:

    kubectl create secret generic kxi-encryption-secret --from-file=key=kxi-encryption-new.key --from-literal=password=<password entered above>

  6. Redeploy the database:

    # Push the package
kxi package push <package-name-and-version>
# Deploy the package
kxi package deploy <package-name>

After your redeploy the database, it detects and uses the new key, including the new password.

Change the key

If you need to change both the password and the encryption key, then you need to decrypt any encrypted databases first. To find out how to decrypt a database, refer to the Decrypt a database section.

Once the database has been successfully decrypted, you can update the encryption secret using the following command:

kubectl create secret generic kxi-encryption-secret --from-file=key=kxi-encryption-new.key --from-literal=password=<password entered above>

You can then re-enable encryption using the UI. Once redeployed, the database uses the new encryption secret.

Create a key manually

To create an encryption key manually, run the following commands before you run kxi install setup:

openssl rand 32 | openssl aes-256-cbc -md SHA256 -salt -pbkdf2 -iter 50000 -out kxi-encryption.key # Prompts for the new password

# Create secret
kubectl create secret generic kxi-encryption-secret --from-file=key=kxi-encryption.key --from-literal=password=<password entered above>

Alternatively, to create a DARE key without going through the setup process, you can run kxi install create-encryption-key.

Backup encryption keys

You must backup your encryption key if you create one manually. If you lose the key and haven't created a backup, you cannot unencrypt your database. Follow the instructions in the next section on this page to back up your encryption key.

Back up an encryption key

It is the responsibility of the system administrator to make sure that the encryption key created during installation, kxi-encryption-secret, is backed up in another location. To create a backup in the same namespace, run the following commands:

kubectl get secret kxi-encryption-secret --namespace=insights -o yaml | sed 's/name: .*/name: kxi-encryption-secret-backup/' | kubectl apply -f -

Further reading