Cookbook/IPCInANutshell

From Kx Wiki
Jump to: navigation, search

The wiki is moving to a new format and this page is no longer maintained. You can find the new page at code.kx.com/q/cookbook/ipc.

The wiki will remain in place until the migration is complete. If you prefer the wiki to the new format, please tell the Librarian why.


kdb+ Inter-Process Communication (IPC)

Simple, powerful and fast.
This page uses TCP/IP sockets, but there are other types of IPC that all use the familiar open/request/close paradigm and all use hopen to connect to kdb+

To start a kdb+ process listening on a port, use the command \p port:

q)\p 5001

or start the kdb+ process with -p port command line paramteter:
q -p 5001
This process is now awaiting incoming connections via tc/ip. To stop the process listening on a port, instruct it to listen on port 0:

q)\p 0

Another kdb+ process can connect to this process with hopen:

q)h:hopen `::5001    /see hopen for more examples of `: 
q)h                  /h is the socket (an OS file descriptor)
3i

Messages can now be sent from the client to the server using the handle returned from hopen.

There are 3 message types

0 async message (set)

serializes and puts message on output queue for handle h, and does not block client. A negative handle signifies async

q)neg[h]"a:10" / on the remote instance, sets the variable a to 10

To ensure an async message is sent immediately, flush the pending outgoing queue for handle h with

neg[h][] / blocks until pending outgoing messages on handle h have been written into the socket

To ensure an async message has been processed by the remote, follow with a sync chaser, e.g.

h"";

You may consider increasing the size of tcp send/receive buffers on your system to reduce the amount of blocking whilst trying to write into a socket.

For more on asynchronous IPC, see Cookbook/Callbacks

1 sync request (get)

sends any pending outgoing (async) messages on h, sends the sync request message, and processes any pending incoming messages on h until a response (or error) message is received.

q)h"2+2" / this is sent to the remote process for calculation
4

A useful shorthand for a single-shot get is:

q)`::5001 "1+1"   
2

Nesting sync requests is not recommended since in such scenarios response messages may be out of request order.

2 response message (get response)

sent automatically by the listening process on completing a sync (get) request.

.z

There are message handlers on the server that can be overridden. The default handler for both sync and async handlers is "value":

.z.pg:value / port get - for sync messages
.z.ps:value / port set - for async messages

These can be made a little more interesting by inserting some debug info:

.z.pg:{0N!(.z.w;.z.a;.z.u;.z.p;x);value x} / dump the handle, ip address, username, timestamp and incoming request to stdout, execute the request and return

To detect when a connection opens, simply override the port open handler, .z.po

.z.po:{0N!(`portOpen;x);} / dump the port open handle to stdout

To detect when a connection is closed from the remote end, override the port close handler, .z.pc

.z.pc:{0N!(`portClosed;x);} / dump the handle that has just been closed to stdout

For more resources on .z including contributed code for tracing and monitoring, see Contrib/UsingDotz

Block, Queue, Flush

To block until any message is received on handle h

r:h[] / store message in r


Messages can be queued for sending to a remote process through using async messaging. kdb+ will queue the serialized message in user space, later writing it into the socket as the remote end drains the message queue. One can see how many messages are queued on a handle and their sizes as a dictionary through the command variable .z.W.

Sometimes it is useful to send a large number of aysnc messages, but then to block until they have all been sent. This can be achieved through using async flush - invoked as neg[h][] or neg[h](::). If you need confirmation that the remote end has received and processed the async messages, chase them with a sync request, e.g. h"" - the remote end will process the messages on a socket in the order that they are sent.

Users

Access control and Authentication is supported through using the -u command line option to specify a file of users/passwords, and .z.pw for further integration with enterprise standards such as LDAP. Access control is possible through overriding the message handlers and inspecting the incoming requests for function calls, and validating whether the user is allowed to call such functions.

IPC in other languages

Personal tools
Namespaces
Variants
Actions
Navigation
Print/export
Toolbox