Send Feedback
Skip to content

Scale Real-Time Architecture

Use chained processes to throttle downstream feeds and isolate analytic queries from your critical data capture system.

Overview

Understand the scaled architecture

In a standard KDB-X setup, a Tickerplant (TP) logs data to disk and sends it to an in-memory Realtime Database (RDB). While this works well for basic needs, high loads can degrade performance.

To address this, you can chain components to distribute the workload.

graph TD
    %% Nodes
    TP[Primary Tickerplant :5010]
    Log[TP Log File]
    WQ[Write-Only RDB :5012]
    HDB[(HDB Disk)]
    CTP[Chained Tickerplant :5110]
    CRDB[Chained RDB :5111]
    Dashboard((Dashboard))
    Users((Users))

    %% Core Data Flow
    TP -->|Log| Log
    TP -->|Zero-Latency| WQ
    TP -->|Zero-Latency| CTP
    WQ -->|Write| HDB

    %% Distribution
    CTP -->|Throttled| CRDB
    CTP -->|Throttled| Dashboard

    %% Access
    Users -->|Query| CRDB

Figure 1: Chained tickerplant architecture

Primary tickerplants typically use zero-latency mode and send every update immediately. Consequently, this places unnecessary strain on clients that don't need every single update (for example, a dashboard that updates once a second).

A chained tickerplant solves this issue by acting as a buffer between the primary TP and the clients. Specifically, it collects updates and sends them in batches (for example, every 1 second).

Key differences

  • No Logging: A chained tickerplant only publishes updates to clients, leaving the primary TP to maintain the log file
  • Throttling: It groups updates and sends them on a timer

Prepare the environment

Start by creating a directory for your project files and downloading the necessary scripts.

The following table lists the required files:

File What it does
tick/u.q Handles the Pub/Sub protocol
tick.q The main tickerplant script
sym.q Defines your table structure
chainedtick.q Republishes data from the primary TP
chainedr.q A read-only database for queries
w.q Saves data to disk to save memory

Create directories

Create the tick and hdb folders.

mkdir -p tick hdb

New-Item -ItemType Directory -Force -Path "tick", "hdb"

Check ports

Make sure ports 5010, 5110, and 5111 are open.

lsof -i :5010,5110,5111

Get-NetTCPConnection -LocalPort 5010, 5110, 5111 -ErrorAction SilentlyContinue

Port conflicts

If a port is busy, stop the other process or pick a new port.

Create the schema

Create tick/sym.q to define the trade table schema.

echo 'trade:([] time:`timespan$(); sym:`symbol$(); price:`float$(); size:`long$())' > tick/sym.q

Set-Content -Path "tick\sym.q" -Value 'trade:([] time:`timespan$(); sym:`symbol$(); price:`float$(); size:`long$())'

Download scripts

Get the necessary scripts from GitHub.

Pub/Sub library (tick/u.q)

curl -L -o tick/u.q https://github.com/KxSystems/kdb-tick/raw/master/tick/u.q

Invoke-WebRequest -Uri "https://github.com/KxSystems/kdb-tick/raw/master/tick/u.q" -OutFile "tick\u.q"

Tickerplant (tick.q)

curl -L -O https://github.com/KxSystems/kdb-tick/raw/master/tick.q

Invoke-WebRequest -Uri "https://github.com/KxSystems/kdb-tick/raw/master/tick.q" -OutFile "tick.q"

Chained Tickerplant (chainedtick.q)

curl -L -O https://github.com/KxSystems/kdb/raw/master/tick/chainedtick.q

Invoke-WebRequest -Uri "https://github.com/KxSystems/kdb/raw/master/tick/chainedtick.q" -OutFile "chainedtick.q"

Chained RDB (chainedr.q)

curl -L -O https://github.com/KxSystems/kdb/raw/master/tick/chainedr.q

Invoke-WebRequest -Uri "https://github.com/KxSystems/kdb/raw/master/tick/chainedr.q" -OutFile "chainedr.q"

Write-only RDB (w.q)

This script saves data to disk.

curl -L -O https://github.com/simongarland/tick/raw/master/w.q && sed -i -e 's/system "cd ",1_-10_string first reverse y;//' -e 's|:../tmp.|:./tmp.|' w.q
Command details
  • sed -i -e ...: Modifies the downloaded w.q script in-place
  • s/system "cd ...;//: Removes a command that attempts to change the directory based on the usage string, which can cause path issues in this setup
  • s|:../tmp.|:./tmp.|': Updates the temporary log directory path to use the current tick directory structure instead of a parent directory
Invoke-WebRequest -Uri "https://github.com/simongarland/tick/raw/master/w.q" -OutFile "w.q"; (Get-Content w.q) -replace 'system "cd ",1_-10_string first reverse y;', '' -replace ':../tmp.', ':./tmp.' | Set-Content w.q
Command details
  • (Get-Content ...): Reads the downloaded file
  • -replace ...: Removes the directory change command that is incompatible with this folder structure and updates the temporary log directory path
  • | Set-Content ...: Saves the modified content back to w.q

Launch the chain

Begin by starting the primary tickerplant on port 5010 to capture all data.

Infrastructure tip

Run it in the background so it keeps running if you close the terminal.

Start the process

Run the following command to start the tickerplant:

nohup q tick.q sym . -p 5010 </dev/null >tp.log 2>&1 &
Command details
  • nohup ... &: The nohup command (no hangup) combined with the trailing ampersand (&) runs the process in the background and prevents it from terminating when you log out
  • q tick.q sym . -p 5010: Starts the KDB-X process running tick.q with the sym schema defined in tick/sym.q, logging to the current directory (.), on port 5010
  • </dev/null: Redirects standard input from /dev/null to prevent the background process from waiting for interaction
  • >tp.log: Redirects standard output to the file tp.log
  • 2>&1: Redirects standard error to the same location as standard output so that both are logged to tp.log
Start-Process -FilePath q -ArgumentList "tick.q","sym",".","-p","5010" -RedirectStandardOutput "tp.log" -RedirectStandardError "tp.log" -PassThru
Command details
  • Start-Process: Starts a separate process used to run the Tickerplant in the background
  • -FilePath q: Specifies the executable (q) to run
  • -ArgumentList: Passes arguments to the q process: the script (tick.q), schema (sym), log directory (.), and port (-p 5010)
  • -RedirectStandardOutput / -RedirectStandardError: Redirects both stdout and stderr to tp.log
  • -PassThru: Returns the process object so you can see the Process ID (PID)

Deploy the chained tickerplant

Start the chained tickerplant. It reads from the primary TP (:5010) and shares data on port 5110.

Use the -t flag to set a 1-second timer (1000 ms). This groups updates together.

nohup q chainedtick.q :5010 -p 5110 -t 1000 </dev/null >chain.log 2>&1 &

Start-Process -FilePath q -ArgumentList "chainedtick.q",":5010","-p","5110","-t","1000" -RedirectStandardOutput "chain.log" -RedirectStandardError "chain.log" -PassThru

Logs

The chained TP does not make a log file. It trusts the primary TP to save data.

Isolate queries with a chained RDB

A chained RDB serves as a read-only copy of the database and, unlike the primary RDB, does not write data to disk.

By connecting to the chained TP (:5110), it protects the primary system from performance issues caused by slow user queries.

Why use it?

  • Safety: Bad queries won't crash data collection
  • Efficiency: It uses less CPU when connected to a throttled feed
  • Flexibility: It can subscribe to just the data it needs

RAM Usage

This database lives in memory. Make sure your server has enough RAM for a second copy of the data.

nohup q chainedr.q :5110 -p 5111 </dev/null >rdb.log 2>&1 &

Start-Process -FilePath q -ArgumentList "chainedr.q",":5110","-p","5111" -RedirectStandardOutput "rdb.log" -RedirectStandardError "rdb.log" -PassThru

Optimize memory with a write-only RDB

Since the standard RDB maintains all data in RAM, it can become resource-intensive.

To optimize RAM usage, a write-only RDB (w.q) stores only a small buffer (default 100,000 rows) in memory before writing it to disk.

Keep On Exit

Use the -koe (Keep On Exit) flag to save data if the process stops.

nohup q w.q :5010 ./hdb -koe </dev/null >w.log 2>&1 &

Start-Process -FilePath q -ArgumentList "w.q",":5010","./hdb","-koe" -RedirectStandardOutput "w.log" -RedirectStandardError "w.log" -PassThru

Do not query

Do not query this process. It only has recent data. Query the HDB or chained RDB instead.

Verify the deployment

Check that data flows through the system.

  1. Feed data to the Primary TP (5010)
  2. Check the Chained RDB (5111) to see the data
  3. Check the Write-Only RDB to ensure it saves to disk
q)h:hopen 5010

q)// 1. Connect to Primary TP and Feed Data
q)h".u.upd[`trade;(`AAPL;150.0;100)]"
q)h".u.upd[`trade;(`GOOG;2800.0;50)]"

q)// 2. Verify Chained RDB (Port 5111)
q)crdb:hopen 5111
q)count crdb"trade" 
2

q)// 3. Verify Write-Only Logic (w.q buffers until MAXROWS=100000)
q)// Trigger a flush by sending bulk data
q)h".u.upd[`trade;(100000#`MSFT;100000?300.0;100000?10)]"

q)// Exit q
q)\\

Check your disk for the data folder (tmp...). This is where the write-only RDB saves data when a flush occurs.

ls -d tmp.*

You should see a directory like tmp.1234.2026.01.01.

Get-ChildItem -Path "tmp.*" -Directory

Configure components

Chained tickerplant (chainedtick.q)

Flag Meaning Default
-t Send updates every N milliseconds 0 (Instant)
-p Port for clients to connect 0 (Off)

Write-only RDB (w.q)

Setting Meaning
MAXROWS Rows to keep in RAM before writing to disk
-koe Keep temporary data if the process exits

Summary

You have successfully deployed a scaled real-time architecture:

  • Deployed a chained tickerplant to buffer high-frequency data and throttle updates
  • Connected a chained RDB to isolate query access and protect the primary capture path
  • Configured a write-only RDB to capture data to disk intraday, reducing RAM usage
  • Verified data flow from the primary tickerplant through the entire chain