IDB files¶
This page describes the IDB directory, its contents including sym files and intraday partitions, and how it is flushed during EOD.
Note
Depending on your use case, your KX Sensors implementation may not employ the intraday database (IDB).
You specify the location of your IDB directory via the system parameter idbDir. This directory stores key database components such as the sym file used by all symbol type data, full copies of master referential data, and partitions for data ingested today that cannot be held in memory. Transient on-disk data buffers are also written to this directory.
This directory will grow in size throughout the day and will be flushed during the EOD procedure as the DBW organizes and moves data to the HDB directory to start the new day. Any non-partitioned data, such as master data or MRU data will be persisted through this process.
This directory must have enough disk space to support the entire daily data volume plus some extra contingency space to account for fluctuation in volumes if static volumes cannot be predicted.
This directory contains absolute path symlinks to other locations within this directory and thus cannot be simply copied to another mount. The IDB root directory structure (idbroot) "->" signifies a symlink relationship (i.e. link->target).
idbroot/
│
┌───┴─────────┬─────────────────┐
│ │ └─ data/
└─ current/ → └─ idb.1/ │
│ │
├─ sym → ├─ sym
├─ kxsIdbStatus → ├─ kxsIdbStatus.0
│ ├─ journalDate1/
├─ 0/ │ ├─ 0/
│ ├─ pTable1/ → │ │ ├─ pTable1/
│ ├─ ... │ │ ├─ ...
│ └─ pTableN/ → │ │ └─ pTableN/
├─ 1/ │ ├─ 1/
│ ├─ pTable1/ → │ │ ├─ pTable1/
│ ├─ ... │ │ ├─ ...
│ └─ pTableN/ → │ │ └─ pTableN/
├─ ... │ └─ ...
│ ├─ ...
│ └─ journalDateN/
├─ k/ ├─ 1/
│ ├─ pTable1/ → │ ├─ pTable1/
│ ├─ ... │ ├─ ...
│ └─ pTableN/ → │ └─ pTableN/
├─ ... ├─ ...
└─ n/ └─ m/
├─ pTable1/ → ├─ pTable1/
├─ ... ├─ ...
└─ pTableN/ → ├─ pTableN/
│
├─ pTable1.sl/
├─ ...
├─ pTableN.sl/
│
├─ pTable1.ss/
├─ ...
├─ pTableN.ss/
│
├─ sTable1.ss/
├─ ...
├─ sTableN.ss/
│
├─ bTable1.ss
├─ ...
├─ bTableN.ss
│
└─ ssz
The current symlink always points to the active loadable kdb+ database located at either idb.0 or idb.1 (alternates throughout the day during EOI), which contains the symlinks for partitioned tables pTable1..N. In IDB this also includes the MRU tables, the sym file symlink and the kxsIdbStatus table (a symlink to the alternating tables data/kxsIdbStatus.0|1).
The data directory contains the actual data for the partitioned and MRU tables. The last integer partition in the last journal date directory there also contains the snapshots (with .ss extension) for the partitioned (pTable0..N), splayed (sTable0..N) and basic (bTable0..N) tables, sequential logs (with .sl extension) for the partitioned tables and the snapshot completion marker (ssz).
Next steps¶
- HDB files — where this data lands once it is organized and moved out during the EOD procedure.
- Delete historical partitions — the exceptions that apply when deleting partitions from IDB rather than HDB.
- Recover your data — back up and restore this directory alongside HDB.