Storage Tiers in KDB-X DB Service¶

This page explains how the DB Service organizes data across storage tiers, how data moves between them, and how tiering affects table configuration and queries.

The DB Service stores data across multiple tiers, managed by the Storage Manager (SM). Each tier represents a different stage in the data lifecycle — from in-memory real-time data through to on-disk historical data and, optionally, object storage.

Understanding storage tiers is important for configuring tables, writing queries, and planning storage capacity.

Tier overview¶

Tier	Name	Storage	Purpose
RDB	Real-time database	In-memory	Holds the most recent data, continuously updated from the ingest stream
IDB	Intraday database	On-disk (local)	Persisted snapshot of intraday data written down from RDB at regular intervals
HDB	Historical database	On-disk (local or object storage)	Long-term historical data written down at end of day
Object storage	—	Cloud object store (S3, Azure Blob, GCS)	Optional remote tier for HDB, enabling cost-effective long-term retention

Partition column¶

The prtnCol in your table definition determines how data is partitioned across tiers. Only timestamp columns can be used as prtnCol.

For query performance, filters on the partition column are highly recommended — they allow the HDB to read only the relevant date partitions. The startTS and endTS query parameters are evaluated against the partition column, which is what allows the gateway to route a query to the right tiers and partitions. See Partitioned tables for how to set it.

RDB — real-time (in-memory)¶

The RDB holds the current day's streaming data in memory. Data arrives via Reliable Transport and is immediately available for query.

Data is held entirely in memory — fast to read, no disk I/O.
Memory usage grows throughout the day and is reclaimed at end-of-interval (EOI) or end-of-day (EOD) when data is written down to the IDB or HDB.
Queries that span recent data always include the RDB DAP.

The amount of data retained in the RDB is controlled by the pctMemThreshold setting. When memory usage exceeds this threshold, an early write-down is triggered. You can set pctMemThreshold in the assembly, and it applies to both the Data Access Processes (DAPs) and the Storage Manager.

IDB — intraday (write-down buffer)¶

The IDB is an intermediate on-disk tier that captures intraday write-downs from the RDB. It provides data durability without requiring a full end-of-day cycle.

Data is written down from the RDB to the IDB at end-of-interval (EOI), which typically runs on a configured schedule (for example, every hour).
Between EOIs, the IDB holds the most recent write-down snapshot.
Queries against recent-but-not-current data are served from the IDB DAP.
Intraday write-down data is rolled into the HDB and cleared from the IDB at end-of-day (EOD).

The IDB reduces RDB memory pressure and provides a recovery point if the system restarts mid-day.

HDB — historical (on-disk)¶

The HDB is the long-term storage tier for date-partitioned data. It holds all data older than the current day.

Data is written to the HDB at EOD, partitioned by date.
Each date partition is stored as a directory on disk (or in object storage).
HDB data is loaded by DAPs using memory-mapped files — fast random access without loading the full dataset into memory.
Queries that include historical date ranges are served from the HDB DAP.

Object storage¶

Object storage is an optional remote tier for the HDB. When configured, date partitions are written to a cloud storage bucket instead of (or in addition to) local disk.

Supported providers: AWS S3, Azure Blob Storage, Google Cloud Storage.
Object storage is significantly cheaper than local disk for large historical datasets.
Read performance is lower than local disk. For latency-sensitive historical queries, local HDB is preferred.
Direct Parquet reads from object storage, without first ingesting data through the DB Service pipeline, are not yet supported.

Object storage is configured in the assembly YAML, by setting the store URI of a Storage Manager tier to the bucket location. For configuration details, see the Insights storage configuration reference.

How data flows between tiers¶

graph TD
    %% Nodes
    RT["Streaming ingest (RT)"]
    BATCH["Batch ingest (import APIs)"]
    RDB["RDB — in-memory"]
    IDB["IDB — on-disk"]
    HDB[("HDB — on-disk or object storage")]

    %% Data flow
    RT -->|continuous| RDB
    RDB -->|EOI write-down| IDB
    IDB -->|EOD write-down| HDB
    BATCH -->|non-partitioned tables| IDB
    BATCH -->|partitioned tables| HDB

Streaming and batch ingest take different paths. Streamed updates land in the RDB and move down through the tiers. Batch imports bypass the RDB, and where the data lands depends on the table type:

Partitioned tables — data is written into the HDB, in the HDB tier appropriate to its partition.
Non-partitioned tables (basic, splayed, and splayed_mem) — data is stored in the IDB.

End-of-interval (EOI) runs on a configurable schedule. The RDB data is written to the IDB, and the RDB is reset for the next interval. Queries are fully available throughout — the DB Service maintains availability during write-down processing.

End-of-day (EOD) consolidates the intraday write-down data into a new HDB date partition at the end of the trading day (or at a configured time). After EOD, that data is cleared from the IDB and a new intraday cycle begins.

The Storage Manager handles EOI and EOD automatically. Manual EOD triggers are available for operators when needed — see manual EOD trigger.

How queries span tiers¶

The service gateway routes queries across tiers automatically based on the startTS and endTS parameters, which are evaluated against the table's partition column:

Recent data (today) → RDB
Same-day data written to disk → IDB
Historical data (previous days) → HDB

Results from each tier are aggregated before being returned to the client. For structured and SQL queries this is fully transparent — you query the same endpoint regardless of which tier your data is on.

q queries behave differently. A query that selects directly from a table runs against each tier independently, so it can return only that tier's data and miss rows held elsewhere. Use .kxi.selectTable to span all tiers correctly, including late data, and to hide internal metadata columns.

Tier configuration¶

Storage tier settings are configured differently depending on deployment type:

Single-node: tier configuration is not exposed. It is abstracted away, and only the most basic settings are available in a single-node deployment. If you need to control tiering, use the assembly YAML.
Clustered: tiers are configured in the assembly YAML — storage locations under mounts, and the tier list, schedules, and retention under elements.sm.tiers. For the full reference, see the Insights storage configuration reference.