Migrate Existing kdb+ Data (Initial Import)

This page explains how to migrate an existing kdb+ historical database (HDB) into the KDB-X DB Service using the initial import process.

Batch ingest vs initial import

Use initial import when migrating a partitioned kdb+ HDB. Use batch ingest (via the file import API or kdb database API) for ongoing backfills and scheduled loads. Initial import uses symbolic links to avoid copying data, making it much faster than re-ingesting through the API for large datasets.

Prerequisites

The kubectl commands on this page assume $NAMESPACE is set to the namespace you are deploying into, as in the quickstart.

Before starting an initial import:

• Clustered only. Initial import is a feature of the clustered DB Service. For single-node, use the kdb database import API instead.
• Back up your data. The import process modifies the sym file and partition structure. Take a full backup before proceeding.
• Schema compatibility. Table schemas in the source HDB must be compatible with the target DB Service table definitions. Use the create-schema.q script (see Step 2) to generate a compatible assembly YAML schema from your existing HDB.
• Partitioned, splayed, and basic tables are supported. Data in object storage is not transformed and stays in standard kdb+ format.

Process overview

Initial import stages your existing HDB on a persistent volume, then has the storage manager (SM) register it in place on first startup rather than re-ingesting it. The steps that follow work through that process in order:

1. Create and configure a Kubernetes PVC to hold the HDB data
2. Generate an assembly schema from your existing HDB using create-schema.q
3. Create a copy pod, and copy the sym file and partitions to the PVC
4. Configure the assembly YAML with initialImport: true
5. Deploy with Helm — SM detects and registers the staged data on startup
6. Verify the migrated data is queryable

Step 1: Create the PVC

Create a Kubernetes PVC to hold the HDB data. The PVC must be created before deploying the DB Service so the SM can mount it on startup.

hdb-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-db-hdb    # must be <assembly-name>-hdb
spec:
  storageClassName: standard    # adjust to your cluster's storage class
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi

kubectl apply -f hdb-pvc.yaml --namespace "$NAMESPACE"

Check available storage classes on your cluster:

kubectl get storageclass

Step 2: Generate a schema

The create-schema.q script reads an existing kdb+ HDB and generates a compatible assembly YAML schema section. Copy the script from Insights schema creation into a file called create-schema.q, then run it against your source HDB:

q create-schema.q -db /path/to/source-hdb -out schema.yaml -fmt yaml

Options:

Option	Description
-db <path>	Path to the source kdb+ database. Defaults to the current directory.
-out <path>	Output path. Defaults to stdout.
-fmt yaml	Output format. Use yaml to generate the assembly schema section used by the DB Service.

Review the output

The script cannot determine all configuration values automatically. Output contains @EDITME@ placeholders where values must be filled in manually. Review and edit the generated schema before using it in your assembly.

Step 3: Copy data to the PVC

Create a copy pod to mount the PVC and copy your source data into it.

Create the copy pod

copy-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: import-copy-pod
spec:
  securityContext:
    fsGroup: 2000
  containers:
    - name: copy-container
      image: amazonlinux:2
      stdin: true
      command: ["/bin/bash", "-c"]
      args:
        - |
          yum install -y rsync
          mkdir -p /data/db/hdb/data/
          while true; do sleep 30; done;
      volumeMounts:
        - mountPath: /data/db/hdb
          name: hdb-storage
  volumes:
    - name: hdb-storage
      persistentVolumeClaim:
        claimName: my-db-hdb

kubectl apply -f copy-pod.yaml --namespace "$NAMESPACE"
kubectl get pod import-copy-pod --namespace "$NAMESPACE"   # wait for Running

Copy the sym file

The sym file must be copied to the data/ subdirectory of the HDB mount root. This is the location SM expects:

kubectl cp sym import-copy-pod:/data/db/hdb/data/ --namespace "$NAMESPACE"

Copy partitions

For efficient transfer with retry logic, use krsync.sh — a wrapper script that uses rsync over kubectl exec. Create the file krsync.sh with the following content:

#!/bin/bash
if [ -z "$RSYNC_RUNNING" ]; then
    if [ "$#" -ne 3 ]; then
        echo "Usage: $0 <local_directory> <pod_name> <target_directory_in_pod>"
        exit 1
    fi
    local_dir=$1
    pod=$2
    target_dir=$3
    max_retries=5
    retry_count=0
    success=0
    export RSYNC_RUNNING=true
    while [ $retry_count -lt $max_retries ]; do
        echo "Attempt $(($retry_count + 1)) of $max_retries..."
        rsync -av --progress --stats -e "bash $0" "$local_dir" "$pod:$target_dir"
        if [ $? -eq 0 ]; then
            echo "Rsync successful on attempt $(($retry_count + 1))"
            success=1
            break
        fi
        retry_count=$(($retry_count + 1))
        sleep 10
    done
    if [ $success -eq 0 ]; then
        echo "Rsync failed after $max_retries attempts"
        exit 1
    fi
    exit 0
fi
pod=$1
shift
kubectl exec -i $pod -- "$@"

chmod +x krsync.sh

# Copy each date partition
bash krsync.sh /path/to/source-hdb/2026.01.01 import-copy-pod /data/db/hdb/data/
bash krsync.sh /path/to/source-hdb/2026.01.02 import-copy-pod /data/db/hdb/data/
# Repeat for all partitions, or use a loop

For object storage, upload partitions directly to your bucket while preserving the directory structure (sym, 2026.01.01/tablename/, etc.).

Step 4: Configure the assembly

Add the generated schema to your assembly YAML. Set initialImport: true on the SM — this tells the SM to look for existing data in the HDB mount's data/ directory on startup:

name: my-db
labels:
  region: amer

tables:
  # Paste generated schema from Step 2 here
  trade:
    type: partitioned
    prtnCol: realTime
    sortColsDisk: [sym]
    sortColsOrd: [sym]
    columns:
      - name: realTime
        type: timestamp
      - name: sym
        type: symbol
        attrDisk: parted
        attrOrd: parted
      - name: price
        type: float
      - name: size
        type: long

bus:
  stream:
    protocol: rt
    topic: stream

mounts:
  rdb:
    type: stream
    partition: none
    baseURI: none
  idb:
    type: local
    partition: ordinal
    baseURI: file:///data/db/idb
  hdb:
    type: local
    partition: date
    baseURI: file:///data/db/hdb

elements:
  dap:
    instances:
      rdb:
        mountName: rdb
      idb:
        mountName: idb
      hdb:
        mountName: hdb
  sm:
    source: stream
    initialImport: true    # tells SM to register existing data on startup
    tiers:
      - name: rdb
        mount: rdb
      - name: idb
        mount: idb
        schedule:
          freq: 00:10:00
      - name: hdb
        mount: hdb
        schedule:
          freq: 1D00:00:00
          snap: 01:35:00

Step 5: Deploy with Helm

Deploy by following the Clustered steps in Start the DB Service. The single-node steps in that section do not apply, as initial import is clustered-only. Two things differ for an initial import:

• Deploy with the assembly you configured in Step 4, including initialImport: true, rather than an unmodified assembly from the reference architecture.
• The namespace and the <assembly-name>-hdb PVC already exist from Step 1, so the SM finds the staged data when it mounts the PVC on startup.

SM validates the staged data against the schema on startup. If validation fails, SM logs details and terminates. Resolve the reported issues and restart SM.

Remove initialImport after first startup

Once SM has successfully started and registered the data, remove initialImport: true from the assembly. It is redundant after the first run.

Step 6: Verify

Once all pods are running, confirm the historical data is accessible:

qREST

session:dbs.createSession["your-gateway-host:8080"]
session.querySimple[`table`startTS`endTS`limit!(`trade; 2026.01.01D; 2026.01.02D; 5)]

curl -X POST "http://your-gateway-host:8080/api/v0/query/simple" \
  -H "Content-Type: application/json" \
  -d '{"table": "trade", "startTS": "2026.01.01D", "endTS": "2026.01.02D", "limit": 5}'

Troubleshooting

SM terminates on startup.

Check SM logs for validation errors:

kubectl logs -l component=storage-manager --namespace "$NAMESPACE"

Common causes: sym file not in <baseURI>/data/, schema mismatch between assembly and source HDB, or missing partitions.

No data returned after deployment.

Confirm startTS/endTS cover the date range of the staged partitions and that the sym file was copied before the partition directories.

For additional troubleshooting, see Troubleshooting.

Next steps

• Manage tables (clustered)
• Storage tiers
• Query data
• Backup and restore