Skip to content

How to run KDB-X in containers and virtual machines

This page explains how container and virtual-machine limits change KDB-X performance, and how to configure a container or guest so that q gets the CPU, memory, storage, and network capacity it appears to have.

A container shares the host kernel. Its limits come from Linux control groups (cgroups) and namespaces, and several of them are invisible to q: q can report the host's CPUs and memory while running inside a much smaller limit. A virtual machine has its own kernel, but its CPUs, memory, and devices are shared with other guests. In both cases, measure inside the environment that runs in production, with its real limits, and apply the operating-system and hardware guidance to the resources that are actually available.

The measurements on this page come from Docker with cgroup v2 on a 2-socket, 64-core host running Linux 5.14 and the kdbx-q:5.0.20261015-rocky9-r1 image. Treat them as examples of the effect, not as expected values.

Test with the official image

For testing, use the official kdbx-q image from the KX downloads portal rather than building your own KDB-X image. Pin an exact tag so that a benchmark can be reproduced.

CPU

A container can be limited to a set of CPUs (a cpuset) or to a CPU quota (a share of time across all CPUs). q sees only the first:

Limit Docker Kubernetes .z.c reports
None — — All host CPUs
CPU set --cpuset-cpus=0-3 Static CPU manager policy, Guaranteed pod with whole CPUs The CPUs in the set
CPU quota --cpus=4 resources.limits.cpu All host CPUs

.z.c counts the logical CPUs that q can run on when it starts. With a quota, it still reports every host CPU, so a -s value or thread pool sized from .z.c or nproc can run many more threads than the quota allows.

Under a quota, the kernel lets the container's threads run until the quota for the current period, 100 ms by default, is used, then stops all of them until the next period. This appears as pauses rather than as a lower average speed. Check throttling inside the container:

cat /sys/fs/cgroup/cpu.max      # quota and period in microseconds; "max" means no quota
grep throttled /sys/fs/cgroup/cpu.stat
  • Use a CPU set for a latency-sensitive process, and keep it on the NUMA node of its memory with --cpuset-mems. The image doesn't include numactl; set placement with the container runtime or orchestrator.
  • Where a quota is unavoidable, set -s to the quota, not to .z.c, and confirm that nr_throttled doesn't rise under load.
  • A process in a CPU set still shares those cores with other containers unless the platform reserves them.

Memory

A container memory limit, --memory in Docker or resources.limits.memory in Kubernetes, sets the cgroup's memory.max. q doesn't see it.

Heap

When the q heap exceeds the limit, the kernel kills the process. Set the -w workspace limit below the container limit so that q fails in a way that can be logged and diagnosed. Leave room outside the heap for the symbol table, libraries, other memory domains, and the page cache that queries need.

Page cache

The cgroup limit also covers page cache that the container's processes read or write. A memory-mapped HDB column needs its pages in cache to be read at memory speed, so a limit smaller than the working set forces repeated disk reads. There is no error and no out-of-memory kill; queries are only slower.

In a test, a container wrote a 3.2 GB column and summed it four times:

Container limit Page cache held Sum of mapped column
--memory=8g 3.2 GB 44 ms
--memory=1g 0.65 GB 5.4–5.8 s

Watch the cgroup's own counters rather than host-wide free:

grep -E '^(anon|file) ' /sys/fs/cgroup/memory.stat   # heap and page cache, in bytes
cat /sys/fs/cgroup/memory.events                     # a rising "max" means reclaim at the limit

Page cache is charged to the cgroup that first reads or writes the page. A file already cached by the host or another container can therefore read quickly in a test and slowly in production. Test with cold data, written or first read by the container under test.

Size the container limit for the q heap plus the page cache of the data it should query warm. Otherwise a tight limit turns warm queries into cold ones, however much free memory the host has.

Storage

Keep databases, logs, and other written data on a volume or bind mount, not in the container's writable layer. The writable layer, a bind mount, and a named volume perform the same when they use the same physical storage, but the writable layer has two drawbacks: it is removed with the container, and modifying a file that exists in the image first copies the whole file.

A volume or bind mount uses the host file system and device directly, so the file-system, disk-combining, and read-ahead guidance applies on the host. For network or cloud volumes, also check the platform's throughput and IOPS limits for the volume type.

Kernel settings and resource limits

A container shares the host kernel, but some settings are per network namespace. They take the namespace's own value, which can differ from the host's, and can be set per container. Others are host-wide, and a container can neither change them nor, for some, read them:

Setting Scope Set with
net.ipv4.tcp_rmem, net.ipv4.tcp_wmem Per network namespace docker run --sysctl or the pod's securityContext.sysctls
net.core.somaxconn Per network namespace --sysctl
net.unix.max_dgram_qlen Per network namespace; was 10 in a container and 512 on the host --sysctl
net.core.rmem_max, net.core.wmem_max, net.core.rmem_default, net.core.wmem_default, net.core.netdev_max_backlog Host-wide; not visible in the container Host sysctl
vm.max_map_count, vm.swappiness, transparent huge pages Host-wide; visible but read-only Host sysctl or /sys

So network buffer tuning splits in two: set the TCP buffer limits on each container, and the net.core limits on the host. A container on the host network (--network host) uses the host's values. Kubernetes allows only some namespaced sysctls by default; you must allow others on the node.

Check the resource limits inside the container as well:

  • Open files: q inherits the runtime's limit and doesn't raise it. In a container, this limit can be significantly lower than on the host. A partitioned database with .Q.MAP or many compressed files can need more; raise it with --ulimit nofile=65536:65536 or the runtime's default limits.
  • Memory maps: vm.max_map_count is the host's value and applies to each process. Raise it on the host when q in a container maps many files.
  • Shared memory: Docker gives /dev/shm 64 MB by default. Raise it with --shm-size if q or a library uses it.

Network

The container's network mode changes both latency and how q decides whether to compress. In a test, a q client sent 800 MB of 8 MB asynchronous messages, with compression disabled, and measured 10,000 synchronous round trips of a small message:

Path Throughput Round trip
Host network, localhost 4.0 GB/s 37 µs
Host network, Unix domain socket 3.6 GB/s 31 µs
Bridge, to the container's IP address 4.0 GB/s 37 µs
Bridge, to a published port (-p) on localhost 4.0 GB/s 440–580 µs

On this host the bridge itself added no measurable cost. Connecting through a published port on localhost went through Docker's user-space proxy and increased the round trip more than tenfold. For latency-sensitive connections between containers, connect to the container's address on a user-defined network, or use the host network.

q's Unix domain sockets are abstract sockets, which belong to a network namespace. Processes can use them across containers only when the containers share a network namespace, such as the containers of one Kubernetes pod, or containers started with --network host or --network container:NAME.

IPC compression

q compresses eligible IPC messages to a peer it sees as remote. A container on a bridge or overlay network sees the host and other containers as remote, so q compresses in both directions although the data never leaves the host. Between q processes on the same host, compression typically isn't worth it: compressing takes longer than the transfer it saves.

From KDB-X 5.0.20260706, disable compression per handle with the two-argument form of .Q.ipc. Each side compresses only what it sends, so set it on both the client and the server:

h:hopen `:tp:5010
.Q.ipc[h;([z:0b])]                      / client: do not compress what it sends
.z.po:{.Q.ipc[.z.w;([z:0b])]}           / server: do not compress replies or published updates

Keep compression for connections that do cross a slower network. Measure CPU at both ends as well as throughput; see IPC compression for the trade-off.

Virtual machines

A virtual machine runs its own kernel, so the operating-system guidance applies inside the guest. The hypervisor decides how well the guest's resources match the hardware:

  • CPU steal: a vCPU waits while the host runs other work. Check the st column of top or vmstat. Sustained steal means the guest doesn't have the capacity it appears to have; use dedicated or pinned vCPUs for latency-sensitive processes.
  • Topology: a large guest can span host sockets. Check whether the guest's lscpu and numactl --hardware show the host's NUMA and SMT layout. If they show one node, q can't be placed by node and memory latency varies; ask for a guest that fits one host node, or one with a virtual NUMA layout that matches the host.
  • Memory: ballooning, overcommitment, and host swapping make guest memory slower than its size suggests. Reserve the guest's memory for a production q host. Huge pages in the guest help only if the host backs them with huge pages.
  • Storage and network: paravirtual devices add CPU work per I/O and per packet. Measure the device with the q workload; for high packet rates or low latency, compare device pass-through or SR-IOV where the platform offers it.
  • Clock: check that the guest uses a stable clock source (cat /sys/devices/system/clocksource/clocksource0/current_clocksource), and synchronize clocks when measuring latency across hosts.

Cloud instances add burst credits and per-instance network and volume limits. See shared and virtual infrastructure.

Next steps