Recover Data from Logs

This guide shows how to restore your database. You will learn to replay logs, check for errors, and fix corrupt files.

Overview

• Understand data recovery: Learn how log replay works
• Prepare the environment: Install modules and create scripts
• Generate and inspect log files: Check log integrity
• Recover data: Replay logs to restore data
• Filter data during recovery: Replay selected data
• Repair corrupt logs: Fix damaged files
• Explore advanced recovery techniques: Read the white paper

Understand data recovery

The Tickerplant (TP) records every update to a log file. Because the Real-Time Database (RDB) runs in memory, it loses data when it restarts. To restore this data, replay the log file. This process reads the file and re-inserts the data into the database.

graph LR
    TP[Tickerplant] -->|Writes Updates| L[(Log File)]
    TP -->|Publishes Updates| RDB[RDB]
    L -.->|Replays on Start| RDB

Figure 1: Log Replay Architecture

Log file structure

A log file consists of a list of messages, each following this structure:

(`upd; table_name; data)

Log message structure

Component	Description
Function (upd)	The function name. You must define this function (typically upd:insert) before you replay the log.
Table	The target table name (for example, `trade).
Data	The data to insert.

Prepare the environment

Set up the environment for this guide. These steps prepare the necessary modules and directories for the examples and are not part of standard log replay.

Run setup commands:

Install the datagen module and create the necessary directories.

Linux / macOSWindows

# Install datagen module to ~/.kx/mod
export QPATH="$QPATH:$HOME/.kx/mod"
mkdir -p ~/.kx/mod/kx/
# Check if directory exists before cloning to avoid errors
if [ ! -d "$HOME/.kx/mod/kx/datagen" ]; then
    git clone https://github.com/KxSystems/datagen ~/.kx/mod/kx/datagen
fi

# Create project directory
mkdir -p ~/kdb_recovery/tplogs
cd ~/kdb_recovery

# Install datagen module to ~/.kx/mod
$env:QPATH = "$env:QPATH;$HOME\.kx\mod"
New-Item -ItemType Directory -Force -Path "$HOME\.kx\mod\kx"
if (-not (Test-Path "$HOME\.kx\mod\kx\datagen")) {
    git clone https://github.com/KxSystems/datagen "$HOME\.kx\mod\kx\datagen"
}

# Create project directory
New-Item -ItemType Directory -Force -Path "$HOME\kdb_recovery\tplogs"
Set-Location "$HOME\kdb_recovery"

Generate and inspect log files

To practice recovery, you need a log file. In production, the Tickerplant generates these automatically, but for this guide you will create one manually.

Generate a sample log

Create the log file, write some data to it, and save it.

q)// Define helpers
q)([getInMemoryTables]): use `kx.datagen.capmkts
q)(sampletrade; samplequote):2#getInMemoryTables[]

q)// 1. Create the log file path and initialize it
q)logPath:`$":tplogs/sym",string .z.d
q)logPath set ()
`:tplogs/sym2026.01.01

q)// 2. Open the file handle
q)h:hopen logPath

q)// 3. Write messages (upd; table; data)
q)h enlist (`upd;`trade;value first sampletrade);
q)h enlist (`upd;`quote;value first samplequote);
q)h enlist (`upd;`trade;value first sampletrade);

q)// 4. Save changes by closing the handle
q)hclose h

Verify log integrity

Check the file for errors before you replay it. Use the -11! function with the -2 argument. This scans the log but does not execute the messages.

• Valid: Returns the number of valid chunks (messages)
• Corrupt: Stops at the first error

Run the check. You should see 3 valid chunks.

q)-11!(-2;logPath)
3

Recover data

To restore data, replay the log. This process reads the file and executes each message to populate the tables.

In your q session:

q)// 1. Initialize the schema
q)(trade; quote): 0#'(sampletrade; samplequote)

q)// 2. Define the upd function
q)//    The log calls `upd`. We define it to insert data.
q)upd:insert

q)// 3. Replay the log file
q)-11!logPath
3

q)// 4. Verify recovery
q)count trade
2
q)count quote
1

Filter data during recovery

To restore only a specific table, define a custom upd function that filters incoming data.

In your q session:

q)// 1. Prepare: Clear the tables
q)(trade; quote): 0#'(sampletrade; samplequote)

q)// 2. Define a filter
q)//    If the table is `trade, insert the data. Otherwise, do nothing.
q)upd:{[t;x] if[t=`trade; t insert x]}

q)// 3. Replay the log
q)-11!logPath
3

q)// 4. Verify results
q)count trade
2
q)count quote
0

q)// 5. Restore the default upd handler for future steps
q)upd:insert

Repair corrupt logs

Log files can become corrupt due to issues like disk space exhaustion. Standard replay stops at the first error, potentially discarding valid data. To recover the remaining valid data, use a custom upd function to trap errors and save valid messages.

Create a corrupt log

First, create a broken log file. Write a valid message, then an invalid one.

q)// 1. Create a bad message (symbol where float expected)
q)badData:value first sampletrade
q)badData[2]:`BAD_PRICE

q)// 2. Create a new log file
q)badLog:`$":tplogs/sym",string[.z.d],"_corrupt"
q)badLog set ()
`:tplogs/sym2026.01.01_corrupt

q)// 3. Write valid and invalid messages
q)h:hopen badLog
q)h enlist (`upd;`trade;value first sampletrade); // Valid
q)h enlist (`upd;`trade;badData);                 // Invalid
q)h enlist (`upd;`quote;value first samplequote); // Valid
q)hclose h

q)// 4. Verify that replay fails with a type error
q)-11!badLog
'type

Error Explanation

The 'type error happens because the data is invalid (symbol instead of float). Replay stops here.

Recover valid data

Recover the valid data by defining a safe upd function with a try-catch block.

How the error trap works

The .[function; args; error_handler] block handles errors safely:

1. Try: It attempts to insert the data AND write it to the log
2. Catch: If it fails (like our type error), it catches the error
3. Handle: It runs the error handler, saving the bad message to badEntries

In your q session:

q)// 1. Prepare: Clear tables and open a new clean log file
q)(trade; quote): 0#'(sampletrade; samplequote)
q)cleanLog:`$":tplogs/sym",string[.z.d],"_clean"
q)cleanLog set ()
`:tplogs/sym2026.01.01_clean
q)hNew:hopen cleanLog
q)badEntries:()

q)// 2. Define a safe 'upd' that traps errors
q)//    If successful: insert to memory AND write to new log
q)//    If failed:     catch error and save message to badEntries
q)upd:{[t;x] .[{insert[y;z]; x enlist (`upd;y;z)};(hNew;t;x);{[t;x;e] badEntries,::enlist (`upd;t;x)}[t;x]]}

q)// 3. Replay the corrupt log (should complete successfully)
q)-11!badLog
3

q)// 4. Close the new log file
q)hclose hNew

q)// 5. Inspect the trapped errors
q)count badEntries
1
q)show badEntries
(`upd;`trade;(`TXN;09:30:00.383;`BAD_PRICE;37;0b;"F";`))

Verify the repair

Check your work. Validate both the database and the new log file.

q)// 1. Memory: Expect 1 trade, 1 quote (the bad trade was skipped)
q)t!count each value each t:tables[]
quote| 1
trade| 1

q)// 2. Disk: New log should have 2 valid messages
q)-11!(-2;cleanLog)
2

Explore advanced recovery techniques

This guide covers the basics. For production systems, read the data recovery white paper. It covers:

• Custom Replay: Control playback with .z.ps
• Partial Replay: Stream chunks with -11!
• High Availability: Hot-warm redundancy
• Zero-Loss Recovery: Synchronous logging with -L

Summary

You have learned the core KDB-X recovery workflows:

• Prepare: Set up the environment
• Inspect: Check logs with -11!
• Recover: Replay logs using upd
• Filter: Load specific tables
• Repair: Save valid data from broken logs
• Verify: Confirm your data is correct