# Recover Data from Logs

_This guide shows how to restore your database. You will learn to replay logs, check for errors, and fix corrupt files._

## Overview

-   **[Understand data recovery](#understand-data-recovery)**: Learn how log replay works
-   **[Prepare the environment](#prepare-the-environment)**: Install modules and create scripts
-   **[Generate and inspect log files](#generate-and-inspect-log-files)**: Check log integrity
-   **[Recover data](#recover-data)**: Replay logs to restore data
-   **[Filter data during recovery](#filter-data-during-recovery)**: Replay selected data
-   **[Repair corrupt logs](#repair-corrupt-logs)**: Fix damaged files
-   **[Explore advanced recovery techniques](#explore-advanced-recovery-techniques)**: Read the white paper

## Understand data recovery

The [Tickerplant](architecture.md#tickerplant-tp) (TP) records every update to a log file. Because the Real-Time Database (RDB) runs in memory, it loses data when it restarts. To restore this data, **replay** the log file. This process reads the file and re-inserts the data into the database.
```mermaid
graph LR
    TP[Tickerplant] -->|Writes Updates| L[(Log File)]
    TP -->|Publishes Updates| RDB[RDB]
    L -.->|Replays on Start| RDB
```
_<center>Figure 1: Log Replay Architecture</center>_

### Log file structure

A log file consists of a list of messages, each following this structure:
```q
(`upd; table_name; data)
```

> **Info — Log message structure**
>
> | **Component** | **Description** |
> | :--- | :--- |
> | **Function** ([`upd`](kdb-tick.md)) | The function name. You must define this function (typically `upd:insert`) before you replay the log. |
> | **Table** | The target table name (for example, `` `trade ``). |
> | **Data** | The data to insert. |
>
## Prepare the environment

Set up the environment for this guide. These steps prepare the necessary modules and directories for the examples and are not part of standard log replay.

**Run setup commands**:

Install the `datagen` module and create the necessary directories.

=== "Linux / macOS"
    ```bash
    # Install datagen module to ~/.kx/mod
    export QPATH="$QPATH:$HOME/.kx/mod"
    mkdir -p ~/.kx/mod/kx/
    # Check if directory exists before cloning to avoid errors
    if [ ! -d "$HOME/.kx/mod/kx/datagen" ]; then
        git clone https://github.com/KxSystems/datagen ~/.kx/mod/kx/datagen
    fi

    # Create project directory
    mkdir -p ~/kdb_recovery/tplogs
    cd ~/kdb_recovery
    ```

=== "Windows"
    ```powershell
    # Install datagen module to ~/.kx/mod
    $env:QPATH = "$env:QPATH;$HOME\.kx\mod"
    New-Item -ItemType Directory -Force -Path "$HOME\.kx\mod\kx"
    if (-not (Test-Path "$HOME\.kx\mod\kx\datagen")) {
        git clone https://github.com/KxSystems/datagen "$HOME\.kx\mod\kx\datagen"
    }

    # Create project directory
    New-Item -ItemType Directory -Force -Path "$HOME\kdb_recovery\tplogs"
    Set-Location "$HOME\kdb_recovery"
    ```

## Generate and inspect log files

To practice recovery, you need a log file. In production, the Tickerplant generates these automatically, but for this guide you will create one manually.

### Generate a sample log

Create the log file, write some data to it, and save it.
```q
q)// Define helpers
q)([getInMemoryTables]): use `kx.datagen.capmkts
q)(sampletrade; samplequote):2#getInMemoryTables[]

q)// 1. Create the log file path and initialize it
q)logPath:`$":tplogs/sym",string .z.d
q)logPath set ()
`:tplogs/sym2026.01.01

q)// 2. Open the file handle
q)h:hopen logPath

q)// 3. Write messages (upd; table; data)
q)h enlist (`upd;`trade;value first sampletrade);
q)h enlist (`upd;`quote;value first samplequote);
q)h enlist (`upd;`trade;value first sampletrade);

q)// 4. Save changes by closing the handle
q)hclose h
```

### Verify log integrity

Check the file for errors before you replay it. Use the [`-11!`](../../ref/internal.md#-11-streaming-execute) function with the `-2` argument. This scans the log but does not execute the messages.

-   **Valid**: Returns the number of valid chunks (messages)
-   **Corrupt**: Stops at the first error

Run the check. You should see `3` valid chunks.
```q
q)-11!(-2;logPath)
3
```

## Recover data

To restore data, **replay** the log. This process reads the file and executes each message to populate the tables.

In your q session:
```q
q)// 1. Initialize the schema
q)(trade; quote): 0#'(sampletrade; samplequote)

q)// 2. Define the upd function
q)//    The log calls `upd`. We define it to insert data.
q)upd:insert

q)// 3. Replay the log file
q)-11!logPath
3

q)// 4. Verify recovery
q)count trade
2
q)count quote
1
```

## Filter data during recovery

To restore only a specific table, define a custom `upd` function that filters incoming data.

In your q session:
```q
q)// 1. Prepare: Clear the tables
q)(trade; quote): 0#'(sampletrade; samplequote)

q)// 2. Define a filter
q)//    If the table is `trade, insert the data. Otherwise, do nothing.
q)upd:{[t;x] if[t=`trade; t insert x]}

q)// 3. Replay the log
q)-11!logPath
3

q)// 4. Verify results
q)count trade
2
q)count quote
0

q)// 5. Restore the default upd handler for future steps
q)upd:insert
```

## Repair corrupt logs

Log files can become corrupt due to issues like disk space exhaustion. Standard replay stops at the first error, potentially discarding valid data. To recover the remaining valid data, use a custom [`upd`](kdb-tick.md) function to trap errors and save valid messages.

### Create a corrupt log

First, create a broken log file. Write a valid message, then an invalid one.
```q
q)// 1. Create a bad message (symbol where float expected)
q)badData:value first sampletrade
q)badData[2]:`BAD_PRICE

q)// 2. Create a new log file
q)badLog:`$":tplogs/sym",string[.z.d],"_corrupt"
q)badLog set ()
`:tplogs/sym2026.01.01_corrupt

q)// 3. Write valid and invalid messages
q)h:hopen badLog
q)h enlist (`upd;`trade;value first sampletrade); // Valid
q)h enlist (`upd;`trade;badData);                 // Invalid
q)h enlist (`upd;`quote;value first samplequote); // Valid
q)hclose h

q)// 4. Verify that replay fails with a type error
q)-11!badLog
'type
```

??? info "Error Explanation"
    The [`'type`](../../ref/errors.md#type) error happens because the data is invalid (symbol instead of float). Replay stops here.

### Recover valid data

Recover the valid data by defining a safe `upd` function with a try-catch block.

??? info "How the error trap works"
    The [`.[function; args; error_handler]`](../../ref/apply.md#trap) block handles errors safely:

    1.  **Try**: It attempts to insert the data AND write it to the log
    2.  **Catch**: If it fails (like our `type` error), it catches the error
    3.  **Handle**: It runs the error handler, saving the bad message to `badEntries`

In your q session:
```q
q)// 1. Prepare: Clear tables and open a new clean log file
q)(trade; quote): 0#'(sampletrade; samplequote)
q)cleanLog:`$":tplogs/sym",string[.z.d],"_clean"
q)cleanLog set ()
`:tplogs/sym2026.01.01_clean
q)hNew:hopen cleanLog
q)badEntries:()

q)// 2. Define a safe 'upd' that traps errors
q)//    If successful: insert to memory AND write to new log
q)//    If failed:     catch error and save message to badEntries
q)upd:{[t;x] .[{insert[y;z]; x enlist (`upd;y;z)};(hNew;t;x);{[t;x;e] badEntries,::enlist (`upd;t;x)}[t;x]]}

q)// 3. Replay the corrupt log (should complete successfully)
q)-11!badLog
3

q)// 4. Close the new log file
q)hclose hNew

q)// 5. Inspect the trapped errors
q)count badEntries
1
q)show badEntries
(`upd;`trade;(`TXN;09:30:00.383;`BAD_PRICE;37;0b;"F";`))
```

### Verify the repair

Check your work. Validate both the database and the new log file.
```q
q)// 1. Memory: Expect 1 trade, 1 quote (the bad trade was skipped)
q)t!count each value each t:tables[]
quote| 1
trade| 1

q)// 2. Disk: New log should have 2 valid messages
q)-11!(-2;cleanLog)
2
```

## Explore advanced recovery techniques

This guide covers the basics. For production systems, read the **[data recovery white paper](https://code.kx.com/q/wp/data-recovery/)**. It covers:

-   **Custom Replay**: Control playback with [`.z.ps`](../../ref/dotz.md#zps-set)
-   **Partial Replay**: Stream chunks with [`-11!`](../../ref/internal.md#-11-streaming-execute)
-   **High Availability**: Hot-warm redundancy
-   **Zero-Loss Recovery**: Synchronous logging with [`-L`](../../ref/cmdline.md#-l-log-sync)

## Summary

You have learned the core KDB-X recovery workflows:

-   **Prepare**: Set up the environment
-   **Inspect**: Check logs with `-11!`
-   **Recover**: Replay logs using [`upd`](kdb-tick.md)
-   **Filter**: Load specific tables
-   **Repair**: Save valid data from broken logs
-   **Verify**: Confirm your data is correct
