Skip to content

Recover your data

This page explains the KX Sensors database backup locations and schedule, how to safely recover IDB and HDB data from backups, and how to create and restore full database snapshots.

This chapter covers the various KX Sensors entities that should be backed up.

Overview

In general, kdb+ databases are stored as a series of files and directories on disk. This makes handling databases extremely easy because database files can be manipulated as operating system files. You can back up a kdb+ database by using any standard file-system backup utility.

This section outlines the various KX Sensors entities that should be backed up. The method of backup and recovery is up to the customer. However, key considerations for any backup are resource contention of the server as well as the timing of any backup procedures. For example, given that the contents of HDB are altered at least once a day during the EOD procedure, if a daily backup of HDB takes more than 24 hours, this may result in unexpected issues.

The following table outlines recommended locations to be backed up as well as their recommended schedule:

Directories Contents Recommended schedule
$KXS_ROOT KX Sensors installation and configuration files N/A
${KXS_ROOT}/db/hdb Historical database. If there are multiple storage tiers, they should also be backed up as well After EOD completes
${KXS_ROOT}/db/idb This directory contains the intra-day data, full copy of master data and the sym file After EOI completes
${KXS_ROOT}/logs Log files Back up daily but avoid during EOD or log cutover

Recover your data

Recovery from a backup should be carefully done to ensure that the directory structures within IDB and HDB are intact. When recovering, the version of the IDB and HDB directories must be from the same date to avoid database mismatches. Depending on the method of file copy, symlinks can accidentally resolve to their targets when copied, resulting in a broken database with duplicate data and no symlinks. Additionally, as symlinks contain absolute paths to their respective targets, the recovered location must have the exact same paths as when the backup was taken.

Database snapshot creation and restoration

This chapter describes how to create a snapshot of an existing KXS database and how to restore it to a fresh installation. The procedure is intended for development and testing purposes and not production environments.

Create a database snapshot

Note

It's recommended to avoid taking the database snapshot during an EOD. Doing so during an EOD is harmless, but results in larger snapshots as transient files will be included.

  1. Stop ingestion to the system.

  2. From a maintenance console, manually invoke an EOI to ensure all in-flight data is written to disk.

    kxsctl mc issueEOI

q).db.initPurview[]
q).db.requestWD`rdb

  3. Stop the system using kxsctl stop system.

  4. Manually update the pos column in the IdbStatus table to 0N within your database from a maintenance console.

    Note

    This step is appropriate for restoring the snapshot to new installation. This step is not appropriate if the intent is to leave the source system in place and operational after taking the snapshot. In that case, updating the pos columns should be done to a copy of the database from a raw q session.

    [app@kxsd01 ~]$ kxsctl mc maint
q){p set update pos:0N from get p:` sv .cfgp.idbDir,`data,x}each`IdbStatus.0`IdbStatus.1

  5. Copy (or move if disk space is limited) the database directory to a backup location on your filesystem.

The snapshot is complete and can be restored to a fresh installation. See steps in the following section.

Note

This snapshot must be moved/copied back to the exact original file path to ensure all sym links within the database are restored correctly. Make note of this location to ensure your system parameters for the IDB/HDB point to the same location in the fresh installation. As long at the database roots remain the same, the snapshot can be used on other servers as well.

Restore a database snapshot

  1. Perform an installation using the KXS installer.

    Note

    If an environment already exists on the server, uninstall it first.

  2. Copy the snapshot from the backup location to the configured database directory.

    Note

    The installer automatically creates empty directories db/hdb and db/idb; delete these before restoring the snapshot.

    [app@kxsd01 kxs]$ rm -rf /data/app/kxs/db
[app@kxsd01 kxs]$ cp -r /data/app/backup/db /data/app/kxs/db

  3. Start a MAINT console and run the following commands to initialize the discovery registry. These objects are typically initialized by DBW the first time a system is being started or during a v2 to v3 upgrade; we run them manually here. These commands should only be run in this context.

    kxsctl mc initDisc

q).disc.putSys[`haState;::;.schema.getTable0`HaState;0]; / HA system state
q).disc.putSys[`purview;::;.schema.getTable0`Purview;0]; / Database purview
q).disc.putSys[`loci;::;.schema.getTable0`Loci;0]; / Data presence loci

At this point, the snapshot has been successfully restored and the system is ready to start using kxsctl start system.

Next steps