HDB files

This page describes the HDB directory structure, its disk space requirements, and how daily partitions are added during the EOD procedure.

You specify the location of your HDB directory in the system parameter hdbDir. All time series data ingested into the system will eventually be organized and stored in this directory and a daily partition is added during the EOD procedure. Master data, such as referential lookup tables, will not be stored in this directory. It is common to see the disk usage of this directory fluctuate during the EOD procedure as some of the reliability and recoverability measures of this procedure include transient duplication of some data partitions.

This directory must have enough disk space to support the required number of days of data.

Like the IDB directory, this directory contains absolute path symlinks to locations within this directory and thus cannot be simply copied to another mount. The HDB root directory structure (hdbroot) "->" signifies a symlink relationship (i.e. link->target).

hdbroot/
    │
┌───┴──────────┬─────────────────┐
│              │                 └─ data/
└─ current/ →  └─ hdb.1/             │
                   │                 │
idbroot/data/sym ← ├─ sym            │
                   ├─ HdbStatus    → ├─ HdbStatus.0
                   ├─ dateN/         ├─ dateN/
                   │   ├─ pTable1/ → │   ├─ pTable1.1/
                   │   ├─ ...        │   ├─ ...
                   │   └─ pTableN/ → │   └─ pTableN.0/
                   ├─ dateN-1/       ├─ dateN-1/
                   │   ├─ pTable1/ → │   ├─ pTable1.0/
                   │   ├─ ...        │   ├─ ...
                   │   └─ pTableN/ → │   └─ pTableN.0/
                   ├─ ...            └─ ...
                   │
                   │                hdbtier2/
                   │                 │
                   ├─ ...            └─ ...
                   │
                   ├─ ...           ...
                   │
                   │                hdbtierN/
                   │                 │
                   ├─ ...            ├─ ...
                   ├─ date2/         ├─ date2/
                   │   ├─ pTable1/ → │   ├─ pTable1.0/
                   │   ├─ ...        │   ├─ ...
                   │   └─ pTableN/ → │   └─ pTableN.1/
                   └─ date1/         └─ date1/
                       ├─ pTable1/ →     ├─ pTable1.1/
                       ├─ ...            ├─ ...
                       └─ pTableN/ →     └─ pTableN.0/

The current symlink always points to the active loadable kdb+ database located at either hdb.0 or hdb.1 (alternates during EOD), which contains the symlinks for partitioned tables pTable1..N, the sym file symlink (points to the sym file in IDB directory) and the HdbStatus table (a symlink to the alternating tables data/HdbStatus.0|1). The data directory and configured HDB tiers contain the actual data for the partitioned tables.

If you enable row-level retention, each table directory that DBM has pruned also holds a hidden .retention marker file recording the retention policy applied to it. The marker is written and committed with the data, and isn't visible to readers of the table.

Next steps

• IDB files — the intraday counterpart to this directory; the sym file in HDB is itself a symlink into IDB.
• Delete historical partitions — remove partitions directly from this directory structure.
• Compression — configure tiering and compression for the data stored here.
• Row-level retention — prune rows within aging partitions in this directory structure.