Interprocess communication¶
Simple, powerful, fast
\p -p listen to port hopen hclose open/close connection .z handle message (callbacks)
A kdb+ process can communicate with other processes through TCP/IP, which is baked into the q language.
Fusion interfaces
Clients for kdb+
This page discusses TCP/IP sockets, but there are other types of IPC, that use the familiar open/request/close paradigm. All use hopen
to connect.
Connection handles, File system
Listen to a port¶
To start a kdb+ process listening on a port, use the command \p port
q)\p 5001
or start the q process with the -p
port command line parameter:
$ q -p 5001
This process is now awaiting incoming connections via TCP/IP. To stop the process listening on a port, instruct it to listen on port 0:
q)\p 0
Another kdb+ process can connect to this process with hopen
:
q)h:hopen `::5001
q)h /h is the socket (an OS file descriptor)
3i
Message format¶
Where h
is the socket the message may be a string or parse tree.
q)h "2+2" /string
4
q)h (+;2;2) /parse tree
4
Use the parse tree format to pass local functions and data to the receiver.
q)h"f:{2+x}" /Set function f on receiver
q)f:{4+x} /Set local function f
q)h(`f;2) /Receiver definition of f called
4
q)h(f;2) /Local f passed to receiver for evaluation
6
q)v:10
q)h(`f;v) /Passing variable as argument
12
q)h({x+y};2;3) /Extend the list to pass more variables
5
Send messages¶
Send messages from the client to the server using the connection handle returned from hopen
.
There are three message types: async, sync, and response.
Async message (set)¶
Serializes and puts a message on the output queue for handle h
, and does not block client. A negative handle signifies async.
q)neg[h]"a:10" / on the remote instance, sets the variable a to 10
To ensure an async message is sent immediately, flush the pending outgoing queue for handle h
with
q)neg[h][]
which blocks until pending outgoing messages on handle h
have been written to the socket.
To ensure an async message has been processed by the remote, follow with a sync chaser, e.g.
q)h"";
You may consider increasing the size of TCP send/receive buffers on your system to reduce the amount of blocking whilst trying to write into a socket.
Sync request (get)¶
Sends any pending outgoing (async) messages on h
, sends the sync request message, and processes any pending incoming messages on h
until a response (or error) message is received.
q)h"2+2" / this is sent to the remote process for calculation
4
A useful shorthand for a one-shot get is:
q)`::5001 "1+1"
2
Nesting sync requests is not recommended: response messages may be out of request order.
Response message (get response)¶
Sent automatically by the listening process on completing a sync (get) request.
Handle messages¶
Message handlers on the server are defined in the .z
namespace. Their default values can be overridden. The default handler for both sync and async messages is value
:
.z.pg:value / port get - for sync messages
.z.ps:value / port set - for async messages
These can be made a little more interesting by inserting some debug info.
Dump the handle, IP address, username, timestamp and incoming request to stdout, execute the request and return:
.z.pg:{0N!(.z.w;.z.a;.z.u;.z.p;x);value x}
To detect when a connection opens, simply override the port open handler, .z.po
:
/ dump the port open handle to stdout
.z.po:{0N!(`portOpen;x);}
To detect when a connection is closed from the remote end, override the port close handler, .z.pc
:
/ dump the handle that has just been closed to stdout
.z.pc:{0N!(`portClosed;x);}
.z
namespace
Using .z
for more resources, including contributed code for tracing and monitoring
Block, queue, flush¶
To block until any async message is received on handle h
r:h[] / store message in r
Messages can be queued for sending to a remote process through using async messaging. Kdb+ will queue the serialized message in user space, later writing it to the socket as the remote end drains the message queue. One can see how many messages are queued on a handle and their sizes as a dictionary through the command variable .z.W
.
Sometimes it is useful to send a large number of aysnc messages, but then to block until they have all been sent. This can be achieved through using async flush – invoked as neg[h][]
or neg[h](::)
. If you need confirmation that the remote end has received and processed the async messages, chase them with a sync request, e.g. h""
– the remote end will process the messages on a socket in the order that they are sent.
Users¶
Access control and authentication is supported through using the -U
command-line option to specify a file of users and passwords, and .z.pw
for further integration with enterprise standards such as LDAP. Access control is possible through overriding the message handlers and inspecting the incoming requests for function calls, and validating whether the user is allowed to call such functions.
Protocol¶
The protocol is extremely simple, as is the message format.
One can see what a TCP/IP message looks like by using -8!object
, which generates the byte vector for the serialization of the object.
This information is provided for debugging and troubleshooting only.
Handshake¶
After a client has opened a socket to the server, it sends a null-terminated ASCII string "username:password\N"
where \N
is a single byte (0…3) which represents the client’s capability with respect to compression, timestamp|timespan and UUID, e.g. "myname:mypassword\3"
.
(Since 2012.05.29.)
- If the server rejects the credentials, it closes the connection immediately.
- If the server accepts the credentials, it sends a single-byte response which represents the common capability.
Kdb+ recognizes these capability bytes:
byte | effect |
---|---|
0 | (V2.5) no compression, no timestamp, no timespan, no UUID |
1..2 | (V2.6-2.8) compression, timestamp, timespan |
3 | (V3.0) compression, timestamp, timespan, UUID |
4 | reserved |
5 | support msgs >2GB; vectors must each have a count ≤ 2 billion |
6 | support msgs >2GB and vectors may each have a count > 2 billion |
Java and C# have array length limits which make capabilities 5 and 6 inviable with their current object models.
Compression¶
For releases since 2012.05.29, kdb+ and the C-API will compress an outgoing message if
- Uncompressed serialized data has a length greater than 2000 bytes
- Connection is not
localhost
- Size of compressed data is less than ½ the size of uncompressed data
The compression/decompression algorithms are proprietary and implemented as the compress
and uncompress
methods in c.java
. The message validator does not validate the integrity of compressed messages.
HTTP server supports gzip compression via Content-Encoding: gzip
for responses to form?…
-style requests.
The response payload must be 2,000+ chars and the client must indicate support via Accept-Encoding: gzip
in the HTTP header.
(Since V4.0 2020.03.17.)
The HTTP client supports gzip content, and .Q.hg
, .Q.hp
, and .Q.hmb
indicate this in the request via the HTTP header Accept-Encoding: gzip
.
(Since V4.0 2020.03.17.)
Enumerations are automatically converted to values before sending through IPC.
hopen
, hclose
,
hsym
.h
namespace for markup
.z
namespace for callback functions
.Q.addr
(IP address),
.Q.hg
(HTTP get),
.Q.host
(hostname),
.Q.hp
(HTTP post)
Connection handles
Serialization examples
WebSockets
Kdb+ and WebSockets
Q for Mortals
§11.6 Interprocess Communication