Interprocess communication

Simple, powerful and fast.

Not just sockets

This page discusses TCP/IP sockets, but there are other types of IPC, that use the familiar open/request/close paradigm. All use hopen to connect.

To start a kdb+ process listening on a port, use the command \p port:

q)\p 5001

or start the q process with the -p port command line parameter:

$ q -p 5001

This process is now awaiting incoming connections via TCP/IP. To stop the process listening on a port, instruct it to listen on port 0:

q)\p 0

Another kdb+ process can connect to this process with hopen:

q)h:hopen `::5001
q)h  /h is the socket (an OS file descriptor)
3i

Messages

Messages can now be sent from the client to the server using the handle returned from hopen.

There are 3 message types: async, sync, and response.

Async message (set)

serializes and puts a message on the output queue for handle h, and does not block client. A negative handle signifies async.

q)neg[h]"a:10" / on the remote instance, sets the variable a to 10

To ensure an async message is sent immediately, flush the pending outgoing queue for handle h with

q)neg[h][] 

which blocks until pending outgoing messages on handle h have been written to the socket.

To ensure an async message has been processed by the remote, follow with a sync chaser, e.g.

q)h"";

You may consider increasing the size of TCP send/receive buffers on your system to reduce the amount of blocking whilst trying to write into a socket.

Sync request (get)

Sends any pending outgoing (async) messages on h, sends the sync request message, and processes any pending incoming messages on h until a response (or error) message is received.

q)h"2+2" / this is sent to the remote process for calculation
4

A useful shorthand for a single-shot get is:

q)`::5001 "1+1"
2

Nesting sync requests is not recommended since in such scenarios response messages may be out of request order.

Response message (get response)

Sent automatically by the listening process on completing a sync (get) request.

.z

There are message handlers on the server that can be overridden. The default handler for both sync and async handlers is value:

.z.pg:value / port get - for sync messages
.z.ps:value / port set - for async messages

These can be made a little more interesting by inserting some debug info.

Dump the handle, IP address, username, timestamp and incoming request to stdout, execute the request and return:

.z.pg:{0N!(.z.w;.z.a;.z.u;.z.p;x);value x}

To detect when a connection opens, simply override the port open handler, .z.po:

.z.po:{0N!(`portOpen;x);} / dump the port open handle to stdout

To detect when a connection is closed from the remote end, override the port close handler, .z.pc:

.z.pc:{0N!(`portClosed;x);} / dump the handle that has just been closed to stdout

Knowledge Base: Using .z for more resources on .z including contributed code for tracing and monitoring

Block, queue, flush

To block until any message is received on handle h

r:h[] / store message in r

Messages can be queued for sending to a remote process through using async messaging. Kdb+ will queue the serialized message in user space, later writing it to the socket as the remote end drains the message queue. One can see how many messages are queued on a handle and their sizes as a dictionary through the command variable .z.W.

Sometimes it is useful to send a large number of aysnc messages, but then to block until they have all been sent. This can be achieved through using async flush – invoked as neg[h][] or neg[h](::). If you need confirmation that the remote end has received and processed the async messages, chase them with a sync request, e.g. h"" – the remote end will process the messages on a socket in the order that they are sent.

Users

Access control and authentication is supported through using the -U command-line option to specify a file of users and passwords, and .z.pw for further integration with enterprise standards such as LDAP. Access control is possible through overriding the message handlers and inspecting the incoming requests for function calls, and validating whether the user is allowed to call such functions.

Protocol

The protocol is extremely simple, as is the message format.

One can see what a TCP/IP message looks like by using -8!object, which generates the byte vector for the serialization of the object.

This information is provided for debugging and troubleshooting only.

Handshake

After a client has opened a socket to the server, it sends a null-terminated ASCII string "username:password\N" where \N is a single byte (0…3) which represents the client’s capability with respect to compression, timestamp|timespan and UUID, e.g. "myname:mypassword\3". (Since 2012.05.29.)

  • If the server rejects the credentials, it closes the connection immediately.
  • If the server accepts the credentials, it sends a single-byte response which represents the common capability.

Kdb+ recognizes these capability bytes:

byte effect
0 (V2.5) no compression, no timestamp, no timespan, no UUID
1..2 (V2.6-2.8) compression, timestamp, timespan
3 (V3.0) compression, timestamp, timespan, UUID
4 reserved
5 support msgs >2GB; vectors must each have a count ≤ 2 billion
6 support msgs >2GB and vectors may each have a count > 2 billion

Java and C#

Java and C# have array length limits which make caps 5 and 6 inviable with their current object models.

Compression

For releases since 2012.05.29, kdb+ and the C-API will compress an outgoing message if

  • Uncompressed serialized data has a length greater than 2000 bytes
  • Connection is not localhost
  • Size of compressed data is less than ½ the size of uncompressed data

The compression/decompression algorithms are proprietary and implemented as the z and u methods in c.java. The message validator does not validate the integrity of compressed messages.

Serialization examples

Integer of value 1

q)-8!1
0x010000000d000000fa01000000

We can break this down as

  • 0x01: architecture used for encoding the message, big endian (0) or little endian (1)
  • 00: message type (0 – async, 1 – sync, 2 – response)
  • 0000
  • 0d000000: msg length (13)
  • fa: type of item following (-6, meaning a 4-byte integer follows)
  • 01000000: the 4-byte int value (1)

Integer vector

q)-8!enlist 1
0x010000001200000006000100000001000000
  • 0x01: little endian
  • 000000
  • 12000000: message length
  • 06: type (int vector)
  • 00: attributes (00 – none, 01 – s, 02 – u, 03 – p, 04 – g)
  • 01000000: vector length (1)
  • 01000000: the item, a 4 byte integer (1)

Byte vector

q)-8!`byte$til 5
0x01000000130000000400050000000001020304
  • 0x01: little endian
  • 000000
  • 13000000: message length
  • 04: type (byte vector)
  • 00: attributes
  • 05000000: vector length (5)
  • 0001020304: the 5 bytes

General list

q)-8!`byte$enlist til 5
0x01000000190000000000010000000400050000000001020304
  • 0x01: little endian
  • 000000
  • 19000000: message length
  • 00: type (list)
  • 00: attributes
  • 01000000: list length (1)
  • 04: type (byte vector)
  • 00: attributes
  • 05000000: vector length (5)
  • 0001020304: the 5 bytes

Dictionary with atom values

q)-8!`a`b!2 3
0x0100000021000000630b0002000000610062000600020000000200000003000000
  • 0x01: little endian
  • 000000
  • 21000000: message length
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 02000000: vector length
  • 6100: null terminated symbol (`a)
  • 6200: null terminated symbol (`b)
  • 06: type (6 – integer vector)
  • 00: attributes
  • 02000000: vector length
  • 02000000: 1st item (2)
  • 03000000: 2nd item (3)

Sorted dictionary with atom values

q)-8!`s#`a`b!2 3
0x01000000210000007f0b0102000000610062000600020000000200000003000000
  • 0x01: little endian
  • 000000
  • 21000000: message length
  • 7f: type (127 – sorted dict)
  • 0b: type (11 – symbol vector)
  • 01: attributes (`s#)
  • 02000000: vector length
  • 6100: null terminated symbol (`a)
  • 6200: null terminated symbol (`b)
  • 06: type (6 – integer vector)
  • 00: attributes
  • 02000000: vector length
  • 02000000: 1st item (2)
  • 03000000: 2nd item (3)

Dictionary with vector values

q)-8!`a`b!enlist each 2 3
0x010000002d000000630b0002000000610062000000020000000600010000000200000006000100000003000000
  • 0x01: little endian
  • 000000
  • 2d000000: message length
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 02000000: vector length (2)
  • 6100: null terminated symbol (`a)
  • 6200: null terminated symbol (`b)
  • 00: type (0 – list)
  • 00: attributes
  • 02000000: list length (2)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 02000000: 1st item (2)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 03000000: 1st item (3)

Table

Note the relation to the previous example.

q)-8!'(flip`a`b!enlist each 2 3;([]a:enlist 2;b:enlist 3))
0x010000002f0000006200630b0002000000610062000000020000000600010000000200000006000100000003000000
0x010000002f0000006200630b0002000000610062000000020000000600010000000200000006000100000003000000
  • 0x01: little endian
  • 000000
  • 2f000000: message length
  • 62: type (98 – table)
  • 00: attributes
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 02000000: vector length (2)
  • 6100: null terminated symbol (`a)
  • 6200: null terminated symbol (`b)
  • 00: type (0 – list)
  • 00: attributes
  • 02000000: list length (2)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 02000000: 1st item (2)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 03000000: 1st item (3)

Sorted table

Note the relation to the previous example.

q)-8!`s#([]a:enlist 2;b:enlist 3)
0x010000002f0000006201630b0002000000610062000000020000000603010000000200000006000100000003000000
  • 0x01: little endian
  • 000000
  • 2f000000: message length
  • 62: type (98 – table)
  • 01: attributes (`s#)
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 02000000: vector length (2)
  • 6100: null terminated symbol (`a)
  • 6200: null terminated symbol (`b)
  • 00: type (0 – list)
  • 00: attributes
  • 02000000: list length (2)
  • 06: type (6 – int vector)
  • 03: attributes (`p#)
  • 01000000: vector length (1)
  • 02000000: 1st item (2)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 03000000: 1st item (3)

Keyed table

q)-8!([a:enlist 2]b:enlist 3)
0x010000003f000000636200630b00010000006100000001000000060001000000020000006200630b0001000000620000000100000006000100000003000000
  • 0x01: little endian
  • 000000
  • 3f000000: message length
  • 63: type (99 – dict)
  • 62: type (98 – table)
  • 00: attributes
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 6100: null terminated symbol (`a)
  • 00: type (0 – list)
  • 00: attributes
  • 01000000: vector length (1)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 02000000: 1st item (2)
  • 62: type (98 – table)
  • 00: attributes
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 6200: null terminated symbol (`b)
  • 00: type (0 – list)
  • 00: attributes
  • 01000000: vector length (1)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 03000000: 1st item (3)

Sorted keyed table

Note the relation to the previous example.

q)-8!`s#([a:enlist 2]b:enlist 3)
0x010000003f0000007f6201630b00010000006100000001000000060001000000020000006200630b0001000000620000000100000006000100000003000000
  • 0x01: little endian
  • 000000
  • 3f000000: message length
  • 7f: type (127 – sorted dict)
  • 62: type (98 – table)
  • 01: attributes (`s#)
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 6100: null terminated symbol (`a)
  • 00: type (0 – list)
  • 00: attributes
  • 01000000: vector length (1)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 02000000: 1st item (2)
  • 62: type (98 – table)
  • 00: attributes
  • 63: type (99 – dict)
  • 0b: type (11 – symbol vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 6200: null terminated symbol (`b)
  • 00: type (0 – list)
  • 00: attributes
  • 01000000: vector length (1)
  • 06: type (6 – int vector)
  • 00: attributes
  • 01000000: vector length (1)
  • 03000000: 1st item (3)

Function

q)-8!{x+y}
0x010000001500000064000a00050000007b782b797d
  • 0x01: little endian
  • 000000
  • 15000000: message length
  • 64: type (100 – lamda)
  • 00: null terminated context (root)
  • 0a: type (10 – char vector)
  • 00: attributes
  • 05000000: vector length
  • 7b782b797d: {x+y}

Function in a non-root context

q)\d .d
q.d)test:{x+y}
q.d)-8!test
0x01000000160000006464000a00050000007b782b797d
  • 0x01: little endian
  • 000000
  • 16000000: message length
  • 64: type (100 – lambda)
  • 6400: null terminated context (.d)
  • 0a: type (10 – char vector)
  • 00: attributes
  • 05000000: length (5)
  • 7b782b797d: {x+y}

Notes

  1. KxSystems/kdb/c/kx/c.java, KxSystems/kdb/c/c.cs etc., are simply (de)serializers for these structures.

  2. Enumerations are automatically converted to values before sending through IPC.

.h namespace for markup
.z namespace for callback functions
.Q.addr (IP address), .Q.hg (HTTP get), .Q.host (hostname), .Q.hp (HTTP post)