Interprocess communication (IPC) in KDB-X

This page demonstrates how to connect KDB-X processes, exchange data and commands using synchronous and asynchronous messaging, and build robust, distributed systems.

Overview

Learn the fundamentals and advanced techniques of KDB-X IPC. This guide covers the following topics:

• Core client-server interaction: Establishing, managing, and closing connections
• Sending requests: Using synchronous (request-response) and asynchronous (fire-and-forget) patterns
• Handling incoming messages: Customizing server behavior with .z namespace callbacks
• Advanced messaging patterns: Implementing deferred responses, message flushing, and broadcasting
• Security and management: Securing and monitoring your IPC connections
• IPC protocol details: Understanding the low-level details for troubleshooting

KDB-X processes communicate with each other and external applications using a simple, high-performance TCP/IP-based protocol. This capability is built directly into the q language, enabling you to create distributed systems where processes can exchange data and commands efficiently.

Other IPC mechanisms

This document focuses on TCP/IP sockets. KDB-X also supports other IPC mechanisms, such as Unix Domain Sockets (UDS) and named pipes, which use the familiar pattern of opening a handle with hopen, sending requests, and closing the connection with hclose.

Core client-server interaction

The fundamental IPC workflow involves a server process that listens for incoming connections and a client process that connects to the server to send requests.

Establishing and closing connections

Before processes can communicate, you must establish a connection. The server must be listening on a port, and the client must open a connection to that port.

Start a listening process (server)

To make a KDB-X process act as a server, instruct it to listen on a specified port, either at launch or at runtime.

• At startup (command line):

  # Start a q process listening on port 5001
  q -p 5001
  

• At runtime (system command):

  // Set the listening port to 5001 from within the q session
  q)\p 5001
  

The operating system's file descriptor limit determines the maximum number of simultaneous connections. If the limit is reached, the server rejects new connections with a 'conn error.

Connect to a listening process (client)

A client process connects to a server using the hopen function, which takes a target host and port. hopen returns an integer handle representing the established connection.

// Connect to a process listening on localhost, port 5001
q)h: hopen `::5001

// The handle 'h' is an integer (OS file descriptor)
q)h
3i

Close a connection

To release system resources, close the connection from either the client or server side using hclose when it's no longer needed.

q)hclose h

Explicitly close connections

KDB-X does not automatically close connections when their handle goes out of scope in your code. You must explicitly call hclose to terminate a connection and free up resources on both the client and server.

Sending requests: Sync vs. async

Once a connection is established, the client can send messages to the server using the connection handle. A message can be either a string to be evaluated or a list representing a function call.

• String format: The server parses and evaluates the string.

  q)h"2+2"
  4
  

• List format: The server executes the function in the first item of the list with the remaining items as arguments. This is the preferred method for passing data and functions.

  q)h(+;2;2)
  4
  

Using the list format, you can pass local data or even entire functions from the client to the server for evaluation.

// Define a local function on the client
q)clientFn:{4+x}

// Define a variable on the client
q)v:10

// Pass the client's function and variable to the server for execution
q)h(clientFn;v)
14

// Pass an anonymous function (lambda) directly to the server
q)h({x*y};5;10)
50

KDB-X supports two primary modes for sending messages: synchronous (request-response) and asynchronous (fire-and-forget).

Synchronous requests (request-response)

A synchronous request sends a message and blocks until the server processes it and returns a response. This is the default behavior and is ideal for queries that require an immediate result.

By default, the server executes the incoming message using value and automatically sends the result back to the client.

// Send a sync request and wait for the result
q)h"sum 1 2 3"
6

Executing remote vs. local functions

You can control whether a function defined on the client or a function of the same name on the server is executed.

• To execute a function defined on the server, pass its name as a symbol (for example, `add).
• To execute a function defined on the client, pass the function value directly (for example, add).

Example:

1. On the server process, define add:

   // Server's 'add' function sums its arguments
   q)add:{x+y}
   

2. On the client process, connect and define a different add:

   q)h: hopen `::5001
   // Client's 'add' function multiplies its arguments
   q)add:{x*y}
   
   // Call the SERVER's function by passing its name as a symbol
   q)h(`add;10;5)
   15
   
   // Pass the CLIENT's function object to the server for execution
   q)h(add;10;5)
   50
   

Avoid nested synchronous requests

Do not nest synchronous requests. For example, if a client sends a synchronous request to a server, and that server then sends another synchronous request to a third process, the responses might arrive out of order, causing unpredictable behavior.

One-shot requests

For a single, infrequent query, you can send a synchronous request without explicitly establishing a persistent connection first. However, this approach is less efficient than reusing an existing connection for multiple messages, because each request incurs the overhead of setting up a new connection.

// Shorthand to open a connection, send a query, get a response, and close it
q)`::5001 "1+1"
2

Asynchronous requests (fire-and-forget)

An asynchronous request sends a message and immediately returns without waiting for a response. This method is ideal for high-throughput scenarios such as logging or data publishing, as well as for commands that do not require a result.

To send an asynchronous message, apply it to a negative connection handle (for example, use the neg operator on the handle).

// Asynchronously instruct the server to create a variable 'a'
q)neg[h]"a:10"

// Asynchronously instruct the server to print "hello from client" to its console
q)neg[h]("0N!";"hello from client")

Because the client does not wait for a response, asynchronous messaging is critical in systems like tickerplants, where blocking for a slow subscriber is unacceptable.

Performance optimization

To improve performance, consider increasing the TCP send/receive buffer sizes on your operating system.

Handling incoming messages (server-side)

On the server side, you can customize how incoming messages are handled by defining callback functions in the .z namespace.

Callback	Trigger	Default behavior
.z.po	A new client connection is opened.	Does nothing.
.z.pc	A client connection is closed.	Does nothing.
.z.pg	A synchronous (get) message is received.	Executes value x on the message.
.z.ps	An asynchronous (set) message is received.	Executes value x on the message.
.z.pw	A new client attempts to authenticate.	Validates user credentials.

Override these functions to add custom logic, such as logging, permission checks, or specialized message routing.

Example: Logging incoming requests and connections

// Log connection open events, showing handle, user, and IP
.z.po: {0N!(`ConnectionOpened; `handle`.z.w; `user`.z.u; `ip`.z.a)}

// Log connection close events, showing the handle that closed
.z.pc: {0N!(`ConnectionClosed; `handle`x)}

// Log each sync request before executing it
.z.pg: {0N!(`SyncRequest; .z.w; .z.u; x); value x}

// To revert to default behavior, delete your custom definitions
q)\x .z.po .z.pc .z.pg

Advanced messaging patterns

Beyond simple synchronous and asynchronous calls, KDB-X supports more complex communication patterns to help manage data flow and build sophisticated distributed systems.

Managing asynchronous communication flow

Blocking for an async response

Although asynchronous messages do not block the sender, the receiving process can be programmed to send a response back. The original sender can then block and wait specifically for that response.

To block and wait for the next incoming message on handle h, call h[].

// On the client:
// 1. Send an asynchronous request
q)neg[h] (`someAsyncFunction; 1; 2);

// 2. Block and wait for a message to arrive on the same handle
//    This assumes the server is programmed to send a message back on this handle
q)response: h[];

Deferred sync

This pattern combines asynchronous and synchronous behavior. The client sends an asynchronous request, performs other work, and then blocks only when it needs the final result.

1. Server setup:

   q)\p 5000
   q)add: {x+y+z}
   
   // This function receives an async request, computes a result,
   // and sends it back to the original client handle (.z.w) asynchronously
   q)proc: {neg[.z.w] (add . x)}
   

2. Client execution:

   q)h: hopen `::5000
   
   // 1. Send an async request to run 'proc' on the server
   //    Execution continues immediately on the client
   q)neg[h](`proc; 1 2 3);
   
   // ... client can perform other tasks here ...
   
   // 2. Now, block and wait for the response on handle 'h'
   q)res:h[];
   q)res
   6
   

Flushing the message queue

When you send async messages, KDB-X buffers them in user space before writing them to the socket. To force all queued messages for a handle to be sent over the network, perform an async flush.

// Flush all pending async messages on handle h
q)neg[h][]
// or equivalently:
q)neg[h](::)

You can inspect the size of the outbound queue for all handles using .z.W or for a specific handle with -38!x.

Flushing and confirmation

Sending any synchronous message on the same handle (for example, h"") will also flush the asynchronous queue. Because messages are processed in order, receiving a response to that message confirms that the remote end has processed all prior asynchronous messages.

Broadcasting a message

To send the same message to multiple clients, use the async broadcast (-25!) function. This approach is more efficient than sending the message to each client in a loop, because the message is serialized only once.

// Handles for three connected clients
q)h1:3; h2:4; h3:5;

// Serialize 'msg' once and send it to handles h1, h2, and h3
q)msg:(`someFunction; `arg1; `arg2);
q)-25!((h1;h2;h3); msg)

Security and management

Properly securing and monitoring IPC connections is essential for production systems.

Authentication and authorization

KDB-X provides hooks for implementing custom security logic.

Authentication

You can enable basic user/password authentication by starting the KDB-X process with a user credentials file (using the -u or -U flag). When a client connects, the server validates the provided credentials against this file.

For more advanced authentication (for example, LDAP or Kerberos), you can override the .z.pw callback. This function receives the username and password and must return a boolean indicating whether the connection is allowed.

Authorization

After a user is authenticated, you can implement finer-grained permissions by customizing the .z.pg (sync) and .z.ps (async) callbacks. This allows you to restrict access to certain functions or data based on the user (.z.u) or their connection handle (.z.w).

Example: Whitelist of allowed functions for sync calls

// On the server:
q)\p 5000

// Define a list of functions that clients are allowed to call
q)allowedFns:(`func1;`func2;`func3;+;-)

// Define a validation function that signals an error if the call is disallowed
q)checkFn:{if[not x in allowedFns;'(.Q.s1 x)," not allowed"]}

// Recursively validate the parse tree
// If x is a general list (0h), check if the first element is a function call
// Then recursively validate any nested lists
q)validatePT:{if[0h=type x;if[(not 0h=type first x)&1=count first x;checkFn first x];.z.s each x where 0h=type each x]}

// Override .z.pg to check the request before executing it
// If the request is a string (10h), parse it first
q).z.pg:{if[10h=type x;x:parse x];validatePT x;eval x}

A client that attempts to call a disallowed function receives an error:

// On the client:
q)h: hopen `::5000

// This call is allowed
q)h"1+1"
2

// This call is blocked by our custom .z.pg
q)h"1*1"
'* not allowed
  [0]  h"1*1"
       ^

Performance considerations

For high-volume data feeds, such as tickerplants that use .z.ps for data publication, consider bypassing these authorization checks for trusted handles to avoid performance overhead.

Monitoring and error handling

Tracking connections

You can inspect active connections on a server:

• .z.H returns a list of all active socket handles
• -38!x provides a detailed dictionary of information about all open sockets

For custom connection tracking, use the .z.po (open) and .z.pc (close) callbacks to maintain a state table of connected clients.

Interrupting requests

A long-running synchronous query on the client can be interrupted by sending an interrupt signal (for example, using kill -s INT <PID>) to the client process. The client then receives a 'rcv handle error, and the handle becomes invalid.

// Client sends a long-running query that blocks
q)h"system\"sleep 30\""

// --- User interrupts the client process from another terminal ---

// Client console shows the error:
'rcv handle: 4. OS reports: Interrupted system call
  [0]  h"system\"sleep 30\""
       ^

// The handle is now broken and cannot be used
q)h"1+1"
'Cannot write to handle 4. OS reports: Bad file descriptor
  [0]  h"1+1"
       ^

Summary

In this guide, you:

• Learned how to establish and manage client-server connections using hopen and hclose
• Sent both synchronous (request-response) and asynchronous (fire-and-forget) messages
• Customized server behavior for connection and message handling by overriding .z namespace callbacks
• Implemented advanced patterns like deferred synchronization, message flushing, and broadcasting
• Secured and monitored IPC connections using authentication and authorization hooks

IPC protocol details

For advanced users

This information is provided for debugging and troubleshooting purposes only. You do not need to understand the low-level protocol for typical KDB-X development.

The raw byte representation of a serialized KDB-X object can be viewed with -8!object.

Handshake

1. A client opens a socket to the server
2. The client sends a null-terminated ASCII string in the format "username:password\C", where \C is a single capability byte that indicates the client's features (for example, support for compression, UUIDs, large messages)
3. If the server rejects the credentials, it closes the connection
4. If the server accepts, it responds with a single byte representing the common capability level of both client and server

Capability bytes:

Byte	Features supported
0	No compression, timestamp, timespan, or UUID (V2.5)
1--2	Compression, timestamp, timespan (V2.6--2.8)
3	Compression, timestamp, timespan, UUID (V3.0)
5	Support for messages > 2GB; vector counts must be ≤ 2 billion
6	Support for messages > 2GB; vector counts can be > 2 billion

Client compatibility

Java and C# clients have array length limits that make capabilities 5 and 6 incompatible with their standard object models. Ensure you are using a compatible capability level when interfacing with these clients.

Compression

An outgoing message is automatically compressed if all of the following conditions are met:

• The uncompressed data is larger than 2000 bytes
• The connection is not to localhost or 127.0.0.1
• The connection is not a Unix Domain Socket (UDS)
• The compressed data is less than half the size of the uncompressed data

The compression algorithm is a variant of Lempel-Ziv. A reference implementation is available in the official KDB-X Java API.