Skip to content

Historical database (HDB)

This page explains how to serve a loaded KDB-X database to other users as an HDB process, and what limits a single process.

Loading a database in a single session is for that session's own use — a q prompt, a script, or a Python process using KDB-X Python. There is no server, and most analysis never needs one.

The step to take this into production is to keep such a process running and let other people query it. A q process that has loaded a database and serves queries over IPC is conventionally called an HDB (historical database) when the data it serves is historical. It is the same \l and the same mapping; what changes is that the process outlives the question and answers on behalf of others. There is still no database server to install — an HDB is just q, started against a database directory.

Beyond the data, an HDB almost always loads a script of its own:

q hdb.q -db /tmp/testdb -p 5010 -s 4

where -db names the database root and hdb.q loads it and defines everything the process is expected to expose:

system "l ",first .Q.opt[.z.x]`db

/ business logic clients call
vwap:{[t;s] exec size wavg price from t where sym=s}
/ constants and reference data
tickSize:0.01

That script is code: it belongs in version control and goes through review and release like any other code. The database directory beside it is data, and is not versioned the same way. Keeping the two apart is what lets one database be served by several processes running different releases of the logic, and lets the logic be rolled forward without touching a byte of data.

One query at a time

By default a q process is single-threaded and handles incoming messages sequentially, so an HDB serves one query at a time. While it is executing, other clients' requests wait in its input queue: one expensive or badly bounded query delays everybody connected to that process.

Secondary threads (-s) do not change this. They parallelize work within one query — across partitions and columns — so that query finishes sooner. But the process still takes one request at a time.

Multithreaded input mode does not solve this

q has a multithreaded input queue mode that gives each connection its own thread, but it is designed for serving static in-memory data (typically a real-time database, or RDB) and explicitly not for data from disk. Queries running in it cannot update globals.

Handling concurrent queries

Concurrency for an HDB is solved by running more than one HDB against the same database, with something in front to spread the work across them — socket sharding, a kdb+ gateway, or a dedicated protocol-aware load balancer. See handling concurrent queries.

Query timeout

A single query can keep a q process busy for a very long time — an unbounded select over every partition, an accidental cross join, a scan with no constraint on the partitioning column — and because an HDB serves one query at a time, that one query blocks every other user connected to it.

The -T command-line option puts a ceiling on it: -T n aborts any client call still running after n seconds with 'stop and leaves the process free to take the next request, so a runaway query costs one client an error instead of stalling everybody. The default is 0, meaning no limit, and \T changes it in a running process.

Note that it applies only to calls arriving over IPC — work typed at the console or run from a startup script is never interrupted, so testing -T in a local session appears to do nothing.

Next steps