Skip to content

How to choose hardware for KDB-X

This page explains how storage, memory, CPU, and the network affect KDB-X, and what to measure when you size a host.

Choose hardware for the target and representative workload, not from one headline specification. A selective query depends on storage latency; a column scan depends on sustained throughput; a warm aggregation often depends on memory bandwidth; and serial ingestion or request handling depends on one core. The system is limited by whichever resource reaches capacity first.

Use nano and the NYSE TAQ benchmarks to compare candidate hosts, then run your own workload before committing to a configuration.

Storage

Historical queries whose working set is not in the page cache often wait for storage. Measure more than peak sequential bandwidth:

  • Latency determines how quickly the first block arrives and dominates queries that touch small ranges in many files.
  • Sustained throughput determines the speed of large column scans and bulk writes.
  • IOPS and queue behavior matter when secondary threads or concurrent queries issue many requests.
  • Write endurance and durability matter for logging, ingestion, and end-of-day processing even when the query tier is read-heavy.

Common storage choices have different tradeoffs:

  • Local NVMe or SSD provides low latency, high throughput, and internal parallelism. It is a common hot-tier choice. Confirm sustained rather than burst performance, and replicate or back up data if the device is local or ephemeral.
  • Hard disk drives (HDDs) have high seek latency but useful sequential throughput and low cost per terabyte. They suit colder data that is read in long contiguous ranges.
  • Network storage, including NFS and parallel file systems, provides shared capacity. The storage controllers, network path, mount options, and competing clients all become part of query performance. Benchmark from the KDB-X host, not only from the storage appliance.
  • Object storage has high request latency but large capacity and aggregate throughput when requests run in parallel. It suits infrequently queried data. KDB-X accesses it through the object storage module.

Most deployments combine tiers. Move data according to its measured access frequency with storage tiering. A segmented database spreads partitions across locations; LVM or RAID striping spreads blocks below KDB-X.

Page cache

KDB-X memory-maps on-disk columns and the operating system brings the referenced pages into memory. Recently read pages remain in the page cache until the operating system needs the RAM. A warm query can therefore run at memory speed even though the table is larger than the q heap.

Size RAM for the active working set when warm-query latency matters. Do not equate the virtual size of mapped files with resident RAM: mmap in .Q.w reports mapped bytes, while operating-system resident and page-cache metrics show which pages occupy physical memory. Compare cold and warm reads explicitly.

Memory

Memory affects KDB-X through capacity, bandwidth, and locality.

Capacity

Allow RAM for all of the following, not only the raw table columns:

  • in-memory tables, reference data, and cached results;
  • intermediate vectors, grouping and sorting indexes, query results, and IPC serialization;
  • concurrent queries and the independent heaps of secondary threads;
  • the symbol table, process code, libraries, and the operating system; and
  • the page cache for frequently queried on-disk columns.

q's allocator uses power-of-two size classes, so heap consumption can exceed the sum of column widths. Measure peak resident memory with realistic concurrency and leave headroom for data growth and unusually broad queries.

The -w option limits the q heap across threads in memory domain 0, so a runaway heap allocation can fail rather than consume all available memory. It is not a limit on total host memory: mapped pages, the symbol table, other memory domains, other processes, and operating-system caches also consume RAM. Use cgroups or equivalent host controls when a total process limit is required.

Bandwidth

Many q primitives perform little computation for each byte they move. On large vectors, memory bandwidth rather than CPU arithmetic becomes the limit. Aggregate bandwidth depends on memory speed, the number of populated channels, and the CPU architecture. Populate channels symmetrically and benchmark the actual configuration. The multithreaded primitives page shows why more threads provide little gain after bandwidth is saturated.

Locality

On a multi-socket host, each socket has local memory. A core reaches memory attached to another socket more slowly, so thread and memory placement can make latency variable. KDB-X is not NUMA-aware. Prefer a working set that fits on one NUMA node, or measure a deliberate cross-node policy; see NUMA.

Heap return policy

KDB-X reuses released heap blocks. In the default deferred mode it generally returns them to the operating system only after .Q.gc or allocation pressure. Immediate garbage collection with -g 1 returns released blocks of at least 64 MB sooner but adds system-call overhead. Choose from measured latency and memory behavior rather than using garbage collection to compensate for oversized intermediates.

CPU

Different KDB-X components need different CPU properties:

  • Single-core performance controls code on the main thread, including serial query work, message handling, and many ingestion paths. A tickerplant or latency-sensitive RDB often values a fast core more than a large core count.
  • Physical core count helps secondary threads, concurrent database replicas, and independent q processes. More cores help only while storage and memory bandwidth can feed them.
  • Cache capacity and latency help small hot tables, indexes, and repeated work. Keeping the working set compact improves cache use.
  • Instruction throughput affects compiled q primitives and compute-heavy analytics. Measure the whole expression; large vector operations are often limited by memory instead.

.z.c returns the number of physical cores visible to q. It does not tell you how many are available after cgroup, virtual-machine, license, or other workload limits. Check .Q.lim and the host controls as well.

Network

The network is on the critical path for:

  • Remote storage, where every storage request crosses the network.
  • IPC, where q serializes a request or result on one process and deserializes it on another.
  • Secondary processes, where a negative -s distributes peach work through handles returned by .z.pd.
  • Replication and streaming, where packet loss, receiver speed, and backpressure can limit sustained ingestion.

Measure latency, sustained bandwidth, retransmissions, and end-to-end serialization cost. To reduce network work:

  • Send a function call to the data and return an aggregate instead of fetching raw rows to the client.
  • Select only required rows and columns, and paginate or cap interactive results.
  • Batch small updates and requests, while preserving the latency target.
  • Prefer binary qIPC to text formats such as JSON when the client supports it.
  • Use a Unix domain socket for processes on the same host.
  • Let q compress eligible remote IPC messages, but measure CPU as well as bytes saved.

Do not size a fast network only by its link rate. A single connection, remote-storage protocol, client deserializer, or small-message workload can reach a different limit first.

Shared and virtual infrastructure

In a virtual machine, container, or cloud service, the advertised CPU and storage type might not equal sustained dedicated capacity. Check for CPU steal, throttling, burst-credit exhaustion, network-volume limits, and noisy neighbors. Pin or reserve resources for latency-sensitive processes where the platform supports it, and benchmark for long enough to exhaust burst behavior.

Local instance storage can be fast but ephemeral. Treat persistence, replication, recovery time, and the performance during recovery as part of the hardware design.

Next steps