Send Feedback
Skip to content

Architecture of KDB-X Systems

This page explains what a KDB-X tick-based architecture is and what it includes.

A KDB-X tick-based architecture can be used to capture, process and analyse vast amounts of real-time and historical data.

The following diagram illustrates the components that are often found in a vanilla KDB-X tick setup:

architecture

Data flows from the feedhandler into the tickerplant, is published to intraday consumers, and is persisted to disk for historical analysis.

Components

Data feed

This is a source of real-time data; for example, financial quotes and trades from Bloomberg or Refinitiv, or readings from a network of sensors.

Feedhandler

Parses data from the data feed to a format that can be ingested by KDB-X.

Multiple feedhandlers can be used to gather data from a number of different sources and feed it to the KDB-X system for storage and analysis.

KX’s Fusion interfaces connect KDB-X to a range of other technologies, such as R, Apache Kafka, Java, Python and C.

Tickerplant (TP)

A kdb+ process acting as a TP (tickerplant) captures the initial data feed, writes it to the log file, and publishes these messages to any registered subscribers.

  • Aims for zero-latency.

  • It can also ingest data in batch mode.

  • Manages subscriptions: adds and removes subscribers, and sends subscriber table definitions.

  • Handles end-of-day (EOD) processing.

tick.q represents a tickerplant and is provided as a starting point for most environments.

Best practices for tickerplants

  • Tickerplants should be lightweight, not capturing data and using very little memory.
  • For the best resilience, and to avoid core resource competition, run them on their own cores.

TP Log

This is the file to which the tickerplant logs the q messages it receives from the feedhandler. It is used for recovery: if the RDB has to restart, the log file is replayed to return to the current state.

Best practices for log files

Store the file on a fast local disk to minimize publication delay and I/O waits.

Real-time database (RDB)

A KDB-X process acting as a RDB (real-time database) subscribes to messages from the tickerplant, stores them in memory, and allows this data to be queried intraday.

At startup, the RDB sends a message to the tickerplant and receives a reply containing the data schema, the location of the log file, and the number of lines to read from the log file. It then receives subsequent updates from the TP as they are published.

At end of day, it usually writes intraday data to the Historical Database and sends it a new EOD message.

r.q represents a tickerplant and is provided as a starting point for most environments.

Best practices for real-time databases

  • RDBs queried intraday should exploit attributes in their tables. For example, a trade table might be marked as sorted by time (`s#time) and grouped by sym (`g#sym).

  • RDBs require RAM as they are storing the intraday messages. Calculate how much RAM your RDB needs for a given table:

    (Expected max # of messages) * schema cost * flexibility ratio

    Where:

    • schema cost: for a given row, a sum of the datatype size.
    • flexibility ratio: 1.5 is a common value.

  • Intraday writedown solutions

Real-time engine/subscriber (RTE/RTS)

A KDB-X process acting as a RTE (real-time engine) subscribes to the intraday messages and typically performs some additional function on receipt of new data – for example, calculating an order book or maintaining a subtable with the latest price for each instrument. A RTE is sometimes referred to as a RTS (real-time subscriber).

Best practices for real-time subscribers

  • Write streaming analytics to compute the required results, rather than timed computations.
  • Ensure analytics can deal with multiple messages, so there are no dependencies here if the tickerplant runs in batch mode.
  • Check analytic run time versus expected TP publish intervals to ensure you don’t bottleneck. In general, look to the most busy and stressful market day for this, and add additional scaling factors. For example, if my TP publishes a message ~every 30ms, my analytic should take less than 30ms to run. To allow for message throughput to double in the TP, the analytic should run in <15ms.
  • Order Book: a KDB-X intraday storage and access methodology

Historical database (HDB)

A KDB-X process acting as a HDB (historical database) provides a queryable data store of historical data; for example, for creating customer reports on order execution times, or sensor failure analyses.

Large tables are usually stored on disk partitioned by date, with each column stored as its own file.

The dates are referred to as partitions and this on-disk structure contributes to the high performance of KDB-X.

Best practices for historical databases

  • Attributes are key. Partition tables on disk on the most-queried column.
  • If the first two columns are time and sym, sorting on time within sym partitions is assumed and provides a performance boost.
  • Can add grouping attribute for other highly-queried columns.

  • When creating the database schema, consider the symbol versus string type choice very carefully:

    • symbol type: use symbols for columns with highly repeating data that are queried most frequently, for example, sym, exchange, side.
    • string type: any highly variable data, for example, order ID.

  • Database sizing follows the same formula as the RDB sizing.

  • Consider using compression for older data, or less-queried columns, to reduce on-disk size. Typically compression sees ⅕ the space usage. When compressing databases, choose compression algorithm and blocksizes through performance comparisons on typical queries.

Compression in KDB-X

Example HDB script

A q script named hdb.q that can be used by KDB-X to create a HDB process:

/q tick/hdb.q sym -p 5012
if[1>count .z.x;show"Supply directory of historical database";exit 0];
hdb:.z.x 0
/Mount the Historical Date Partitioned Database
@[{system"l ",x};hdb;{show "Error message - ",x;exit 0}]
Usage 
q hdb.q SRC [-p 5012]

Parameter Name Description Default
SRC The directory used by the RDB to which it saves previous values at end-of-day <none>
-p listening port for client communication, for example, a RDB instructing the HDB to reload its DB or client queries against the HDB data <none>

Standard KDB-X command line options may also be passed

A HDB is empty until the RDB saves its first set of tables at end-of-day.

Gateway

The entry point into the KDB-X system. Responsible for routing incoming queries to the appropriate processes, and returning their results.

Gateways do not store data; they coordinate access to other processes.

Can connect both the real-time and historical data to allow users to query across both. In some cases, a gateway combines the result of a series of queries to different processes.

Best practices for gateways