Send Feedback
Skip to content

Ingesting and Processing Live Data Feeds

Welcome to this example, where we'll explore how to handle live data feeds in KDB-X by setting up a simple streaming workflow.

One of the strengths of KDB-X is its ability to process real-time data efficiently, making it ideal for use cases such as financial market data, IoT sensor streams, and telemetry data. This example will walk you through the basics of setting up a client-server architecture for ingesting and processing live data feeds in-memory.

By the end of this example, you’ll understand how to establish a real-time data pipeline in KDB-X, append incoming data to a table dynamically, and simulate a continuous data stream.

Here, we'll cover:

  • Setting up a server process to receive incoming data
  • Establishing a client process to simulate live data streaming
  • Using IPC (Interprocess Communication) for client-server communication
  • Append incoming data dynamically

1. Install q

Before we start q it must first be installed. If you have yet to install q you can follow the installation guide. Once you have q installed and have been able to start your first q process you can continue with this example.

2. Setup the Server

A server in KDB-X is simply a process that listens for incoming data from other processes. This is useful in real-world scenarios where multiple sources (e.g., market data providers, sensors, trading engines) send real-time data to a centralized system.

First, let's setup a server process that listens for incoming data and stores it in an in-memory table. Run the following command in a terminal to launch a process specifying the port 1234 with -p.

q -p 1234

By setting a port we have exposed this process so other clients can send data to it, the server is now listening for incoming connections on port 1234.

Next, we can create an empty in-memory table named t, which will store our real-time data.

t:([]time:`timestamp$();sym:`$();price:`float$();size:`long$())

Here's a breakdown of what's happening:

  • We define a new table with table notation ([] col1:<values>; col2:<values>: ....) with empty values ():
  • time - Timestamp of the trade
  • sym - Ticker symbol
  • price - Trade price
  • size - Trade volume

Now the server and table are ready to receive data!

3. Setup the Client

The client acts as a data source that continuously sends new trade events to the server. In real-world scenarios, this could be a market data feed, an IoT sensor, or a trading algorithm.

Open a new q session (client process) and establish a connection to the server using hopen:

h:hopen 1234

In the above:

  • hopen 1234 creates a connection handle (h) to communicate with the server.
  • If the server is not running, this will return an error.

The client is now connected to the server and ready to send data!

4. Send Data from the Client

In KDB-X, a client can send commands to a server using IPC. One way to add data to a table is by upserting new records into the table stored on the server.

Let's define a function f to generate trade data. Defining the following function on the client process.

genData:{[n] ([]time:n#.z.P;sym:n?`A`B`C;price:10+n?10f;size:50*1+n?20)}

In the above:

  • {[] ... } is the syntax of function definition where we define parameter names within [], in our case this is n, the number of rows to create
  • We define a table with table notation ([] col1:<values>; col2:<values>: ....) that matches teh table schema on our server process
  • time is populated with n timestamps at the current time using .z.P retrieved with the # operator
  • sym is populated with random symbols selected from a list using ?
  • price and trade size are randomly generated

Let's test this works by passing 2 as the parameter n and we see that 2 rows of data are been generated.

genData[2]

Great, now we are ready to send this to our server process using the handle h. Let's define a function to do this on the client process:

sendData:{[n] neg[h](upsert;`t;genData[n])}

In the above:

  • neg[h] Sends the commands and data to the server process. The use of neg specifies that the data is to be sent asynchronously meaning that we do not wait for a result to be returned to proceed.
  • (x;y;z) Round brackets and semicolon allows us to specify a list where:
  • upsert Appends new data to a table
  • t is the name of the table on the server process to append data to
  • genData[n] generates n rows of data

Let's run this for 5 rows of data and check the table on the server:

// Client
sendData 5

// Server
select from t

We should see that 4 rows of data have been sent from the client to the server process!

5. Simulate a Continuous Data Feed

In real-world applications, data may arrive continuously rather than in large batches over the course of a day. To simulate this instead of manually sending data, we can automate the process using a timer that simulates an incoming data feed.

q provides a built-in timer function .z.ts, which will execute a given set of commands at a set time interval. We can define the following on the client process

// Client
.z.ts:{sendData 5+rand 5;} / define the function to be executed
\t 1000                    / Execute .z.ts every 1000ms (1 second)

In the above:

  • .z.ts calls sendData, which sends 5-9 new trades to the server
  • Using \t we can trigger this function to run every 1000 milliseconds (1 second)

We can run count on the table t on our server process to see this in action, run it a few times to see the number increasing.

// Server
count t

You should now see a growing number of trades on a per-second basis.

At this point the server continuously receives new trade data just like a real-time market data feed!

6. Cleanup

Long-running timers or open connections can cause unintended issues, such as:

  • Performance slowdowns (if .z.ts keeps running in the background)
  • Connection leaks (if clients don’t properly close hopen connections)

It is good practice therefore to stop the timer on the client and close the connection when you are finished with this example:

// Client
\t 0
hclose h
  • \t 0 disables .z.ts, stopping the automatic data generation
  • hclose closes the connection so the client can no longer communicate with the server process.

7. Beyond the Basics: Scaling Real-Time Data Processing

The example we’ve built here is designed to demonstrate just how powerful and elegant q/KDB-X is, even in its simplest form. In real life a professional KDB-X trading system would most likely make use of tick architecture, which:

  • Logs all incoming data to disk for durability.
  • Sends updates to multiple subscribers (real-time databases, analytics engines)
  • Manages high-frequency streams efficiently

To learn more about real-world KDB-X architecture , check out these resources:

  • Free KX Course: KDB-X Architecture – A structured introduction to how KDB-X handles real-time and historical data.
  • GitHub: kdb-tick – The official repository for tick.q, a production-grade framework for real-time data ingestion, storage, and analytics.

This is how real-world trading systems ingest, process, and analyze billions of data points every day—and now you have the foundation to build your own! 🚀

Next Steps

Now that you have built your first real-time example you might be interested in the following: