Skip to content

Interprocess communication basics

This page explains the fundamentals of interprocess communication (IPC) in q: opening and closing connections, and sending synchronous requests.

q processes communicate with each other, and with external applications, over a high-performance TCP/IP protocol. The protocol is built into the language, so you can build distributed systems in which processes exchange data and commands without any additional software.

Other IPC mechanisms

This page covers TCP/IP sockets. q supports other mechanisms too, such as Unix domain sockets (UDS). They all follow the same pattern: open a handle with hopen, send requests, and close the connection with hclose.

Open a connection

An IPC exchange has two sides: a server that listens on a port, and a client that connects to that port to send requests.

To make a q process act as a server, tell it to listen on a port, either at startup or at runtime.

The operating system's file descriptor limit caps the number of simultaneous connections. Once a server reaches that limit, it rejects new connections with a 'conn error.

A client connects with hopen, which takes a target host and port and returns an integer handle for the connection:

q)h: hopen 5001   / connect to a process listening on localhost, port 5001
q)h               / the handle is an integer, an OS file descriptor
3i

A bare integer such as 5001 is shorthand for the local communication handle `::5001 — the host is omitted, so it always targets the local machine. Use the full communication handle syntax to reach a remote host, a Unix domain socket, or a TLS-encrypted port:

q)hopen `:10.43.23.198:5010          / TCP, by IP address
q)hopen `:mydb.us.com:5010           / TCP, by hostname
q)hopen `:mydb.us.com:5010:user:pass / TCP, with credentials
q)hopen `:unix://5010                / Unix domain socket, localhost
q)hopen `:tcps://mydb.us.com:5010    / SSL/TLS, by hostname

Prefer UDS between processes on the same host

When two q processes run on the same host, connect over the Unix domain socket (`:unix://port) rather than TCP (`:port, `::port, or a bare integer). A UDS connection bypasses the network stack, so it uses fewer system resources and gives you lower latency and higher throughput than TCP over loopback.

Send requests

Once connected, the client sends messages on the handle. A message is either a string for the server to evaluate, or a list representing a function call.

  • A string. The server parses and evaluates it:

    q)h"2+2"
4

  • A list. The server applies the first item to the remaining items as arguments. Prefer this form for passing data and functions:

    q)h(+;2;2)
4

The list form lets you send local data, and even local functions, to the server:

q)clientFn: {4+x}   / a function defined on the client
q)v: 10             / a variable defined on the client
q)h(clientFn; v)    / send both to the server to run there
14
q)h({x*y}; 5; 10)   / send an anonymous function
50

q offers two ways to send a message: synchronously, as a request and a response, and asynchronously, where the sender continues without waiting. Asynchronous messaging has enough depth to warrant its own page.

Synchronous requests

A synchronous request blocks until the server has processed the message and returned a result. This is the default, and it suits a query whose answer you need before you can continue.

By default the server evaluates the incoming message with value and sends the result back:

q)h"sum 1 2 3"
6

Run remote or local functions

You choose whether the server runs its own function or one the client sends:

  • To run a function defined on the server, pass its name as a symbol, such as `add.

  • To run a function defined on the client, pass the function itself, such as add.

  • On the server, define add:

    q)add: {x+y}   / the server's 'add' sums its arguments

  • On the client, connect and define a different add:

    q)h: hopen `::5001
q)add: {x*y}       / the client's 'add' multiplies its arguments
q)h(`add; 10; 5)   / the symbol names the server's function
15
q)h(add; 10; 5)    / the value sends the client's function to the server
50

Avoid nested synchronous requests

Don't nest synchronous requests. If a client sends a synchronous request to a server, and that server sends another synchronous request to a third process, responses can arrive out of order and the behavior becomes unpredictable.

One-shot requests

For a single, infrequent query, you can send a synchronous request without opening a persistent connection first. Each one-shot request pays the cost of setting up and tearing down a connection, so reuse a handle from hopen when you send more than a few messages.

q)`::5001 "1+1"   / open, send, receive, and close, in one expression
2

A one-shot request is also the only outgoing socket call permitted from a secondary thread — inside peach, or from a connection thread in multi-threaded input mode. q restricts a persistent handle from hopen, and the requests you send on it, to the main thread; using one from a secondary thread signals a 'nosocket error.

Close a connection

q doesn't close a connection when its handle goes out of scope in your code. Release the resources explicitly with hclose, from either the client or the server side:

q)hclose h

An unclosed handle holds resources on both sides

A connection that neither side closes keeps a file descriptor open on the client and on the server. A long-lived process that opens handles in a loop without closing them reaches the file descriptor limit and then fails with 'conn.

Server-side callbacks

To customize how a server handles incoming messages, define callbacks in the .z namespace:

Callback Trigger Default behavior
.z.po A client connection opens Does nothing
.z.pc A client connection closes Does nothing
.z.pg A synchronous (get) message arrives Runs value x on the message
.z.ps An asynchronous (set) message arrives Runs value x on the message
.z.pw A client attempts to authenticate Validates the user's credentials

Override these to add logging, permission checks, or custom message routing. The definitions below log each new connection and each synchronous request, then hand the request on to value:

q).z.po: {0N!(`connectionOpened; .z.w; .z.u; .z.a)}   / handle, user, IP address
q).z.pg: {0N!(`syncRequest; .z.w; .z.u; x); value x}  / log, then run
q)\x .z.po .z.pg                                      / revert to the defaults

Read .z.w inside the callback

.z.w gives the handle of the connection being served, so it's meaningful only while a callback is running. Reading it at the console reports 0i, the handle of the console itself.

Next steps