J client for kdb+

JQ is a socket interface to kdb+ from J, for example to provide a GUI for a kdb+ server. JQ was written in J602 for V2.4, and has been tested on w32, l32 and l64.

kxcontrib/cburke/jk

Sample session

Start a q session, listening on a given port. Load J, then:

load 'qclient.ijs'           NB. load JQ client application

Create a client instance, defining cover functions in z:

   1 conew 'cqclient'
+-+
|1|
+-+
   connect 'robot';5001;'elmo'  NB. connect to robot on port 5001 with user elmo

   cmd 'trade:([]time:();sym:();price:();size:())' NB. create trade database

Send a command string to insert a record:

   cmd '`trade insert(09:30:00;`a;10.75;100)'

   get 'trade' NB. get database
+-----+----+------+-----+
|`time|`sym|`price|`size|
+-----+----+------+-----+
|34200|`a  |10.75 |100  |
+-----+----+------+-----+

Get database in extended format. Each item is a pair: Qdatatype;value. The display shows that trade is the flip (type 98) of a dict (type 99). The dict is a pair: varchar, object list.

   getx 'trade'
+--+------------------------------------------------------------------------------+
|98|+--+-------------------------------------------------------------------------+|
|  ||99|+----------------------------+------------------------------------------+||
|  ||  ||+--+-----------------------+|+-+--------------------------------------+|||
|  ||  |||11|`time `sym `price `size|||0|+----------+-------+---------+-------+||||
|  ||  ||+--+-----------------------+|| ||+--+-----+|+--+--+|+-+-----+|+-+---+|||||
|  ||  ||                            || |||18|34200|||11|`a|||9|10.75|||6|100||||||
|  ||  ||                            || ||+--+-----+|+--+--+|+-+-----+|+-+---+|||||
|  ||  ||                            || |+----------+-------+---------+-------+||||
|  ||  ||                            |+-+--------------------------------------+|||
|  ||  |+----------------------------+------------------------------------------+||
|  |+--+-------------------------------------------------------------------------+|
+--+------------------------------------------------------------------------------+

Insert records using the insert function. The result is the record number:

   execr 'insert';(s:<'trade');<('09:31:12';(s:<'a');11.01;200)
1
   execr 'insert';(s:<'trade');<('09:31:15';(s:<'bc');10.80;150)
2

The next example has a numeric time field, and integer price field. Extended format ensures the correct datatypes are sent:

   execr 'insert';(s:<'trade');<((,:18;34301);(s:<'bc');(,:9;22);500)
3

The utility col, defined below, formats cells into columns:

   col=: ,.@:>L:0

   col getf 'trade'
+--------+----+------+-----+
|`time   |`sym|`price|`size|
+--------+----+------+-----+
|09:30:00|`a  |10.75 |100  |
|09:31:12|`a  |11.01 |200  |
|09:31:15|`bc | 10.8 |150  |
|09:31:41|`bc |   22 |500  |
+--------+----+------+-----+

The next result is displayed as a pair – key columns and data columns.

   col cmdr 'select sum size by sym from trade'
+------+-------+
|+----+|+-----+|
||`sym|||`size||
|+----+|+-----+|
||`a  |||300  ||
||`bc |||650  ||
|+----+|+-----+|
+------+-------+

Send command string to load the sp.q database:

cmd '\l sp.q'

Query the sp.q database:

   col cmdr 'select sum qty by p.color from sp'
+--------+------+
|+------+|+----+|
||`color|||`qty||
|+------+|+----+|
||`blue ||| 900||
||`green|||1000||
||`red  |||1200||
|+------+|+----+|
+--------+------+

   col get 's'
+-----+------------------------+
|+---+|+------+-------+-------+|
||`s |||`name |`status|`city  ||
|+---+|+------+-------+-------+|
||`s1|||`smith|20     |`london||
||`s2|||`jones|10     |`paris ||
||`s3|||`blake|30     |`paris ||
||`s4|||`clark|20     |`london||
||`s5|||`adams|30     |`athens||
|+---+|+------+-------+-------+|
+-----+------------------------+

Load client

The client is in script qclient.ijs. This populates locale cqclient.

load 'qclient.ijs'

Methods

methods	role
connect close	open connection, close connection
cmd cmdr cmdrf cmdrx	send command string
exec execr execrf execrx	execute q function
get getf getx	get value
set	set value

All methods return two items:

• returncode (0=success)
• result, or error message for non-zero returncode.

The non-zero returncodes are

• 1: error signalled in client
• 2: error signalled in q server

The get and set methods are convenience functions that call corresponding cmd or exec methods.

The cmd family methods send character strings to be executed by the q value function, as if they were entered on the terminal. The exec family methods send a function name and parameters in q internal format. This is more efficient than cmd for large amounts of data.

Methods differ in the type of response returned from q. For example, the cmd method has four variants:

method	effect
cmd	no response
cmdr	with response
cmdrf	with formatted response
cmdrx	with extended response

The response variants are described in section Datatypes below.

When used under program control, create an instance of cqclient and use the instance handle when calling methods. Your application should check the return code of each command, for example:

q=: conew 'cqclient'
connect__q 'robot';5001;'elmo'
cmd__q '\l sp.q'
cmdr__q 'select sum qty by p.color from sp'

Interactive use

For interactive use, you can define cover verbs of the same name in z, by calling conew with a left argument of 1. For a zero returncode, the result is the method result, otherwise the result is the returncode and error message.

The examples in the rest of this manual use these cover verbs. The above example showing session output is:

   1 conew 'cqclient'     NB. result is the instance locale
+-+
|1|
+-+
   connect 'robot';5001;'elmo'
   cmd '\l sp.q'
   cmdr 'select sum qty by p.color from sp'
+-------------------+---------------+
|+-----------------+|+-------------+|
||`color           |||`qty         ||
|+-----------------+|+-------------+|
||`blue `green `red|||900 1000 1200||
|+-----------------+|+-------------+|
+-------------------+---------------+

Method syntax

connect

This connects to a q session. Parameters are:

• host: host name or IP address
• port: port number
• user[:password]: user name, with optional password

close

Closes the session and erases the instance locale. Same as calling codestroy in the instance locale. The argument is ignored.

cmd

The argument is a text string to be executed in q. No response is expected from kdb+ (an asynchronous message). The method result simply indicates whether the string was sent successfully.

For example:

cmd 'val: sum key 100'

cmdr cmdrf cmdrx

As cmd, except that the response from q is returned (a synchronous message).

exec

Sends a function name and parameters to be executed in q. As with cmd, no response is expected from q, and the method result simply indicates whether the message was sent successfully.

For example, load the trade table, and add a record using the insert function:

cmd '\l trade.q'

exec 'insert';(s:<'trade');<('09:31:15';(s:<'bc');10.80;150)

execr execrf execrx

As exec, except that the response from q is returned.

get getf getx

Identical to cmdr, cmdrf, cmdrx.

set

Binary with form:

'name' set value

Calls the set function, and checks the name was set in q.

Datatypes

There is no one-to-one mapping between q and J datatypes.

All q datatypes are mapped in some way to J datatypes. Three such mappings are supported: raw, formatted and extended. Corresponding methods are named, for example, cmdr, cmdrf, cmdrx. Essentially:

raw

converts q to J equivalents suitable for further processing. In particular, all temporal types are converted to numbers.

formatted

as raw, except that temporal types are formatted.

extended

as raw, but each mapping is a 1×2 array: Qdatatype;value. This provides an exact mapping between q and J.

J datatypes that are in the range of q mappings can be mapped back to q. Other datatypes such as complex numbers or rank-3 arrays, cannot be sent to q.

The next sections discuss primitive datatypes (e.g. boolean, byte etc), complex datatypes (flip, dict, keyed table), and object lists.

Primitive datatypes

The following is a summary:

 Qname     Qexample             Jraw       Jfmt      Jnull
---------------------------------------------------------
boolean   1b                   boolean    as Jraw   n/a
byte      0xff                 character  as Jraw   n/a
short     23h                  integer    as Jraw   <.-2^31
int       23i                  integer    as Jraw   <.-2^31
long      23                   extended   as Jraw   -2^56x
real      2.3e                 floating   as Jraw   __
float     2.3                  floating   as Jraw   __
char      "a"                  character  as Jraw   ''
varchar   `ab                  symbol     as Jraw   `
month     2003.03m             integer    as Q      <.-2^31 | nnnn.nnm
date      2003.03.23           integer    as Q      ditto | numbers to "n"
datetime  2003.03.23T08:31:53  integer    as Q           ditto
minute    08:31                integer    as Q           ditto
second    08:31:53             integer    as Q           ditto
time      09:10:35.000         integer    as Q           ditto

Raw invertible datatypes are: boolean, int, long, float, char, varchar. Formatted invertible datatypes are all except: byte, short, real.

Q atoms, lists and empties are mapped into J atoms, lists and empties. The only exception is the formatted values for temporal types – in this case, q atoms are mapped to J character strings, and q lists to J character tables.

Q long is mapped into J extended integer in the range ± 2^56x.

Use extended notation when mapping back non-invertible datatypes, for example, the following sets a list of short:

'xx' set ,:5;2 3 5 7

Enumerations are mapped into J lists.

Nulls

Q short null is mapped into <.-2^31, and not into <.-2^15 (the short null value in q).

Q real and float nulls are mapped into J negative infinity.

Formatted temporal nulls have "n" in each numeric position.

Example of nulls:

   get 'trade[99]'
+------+-----------+
|`time |_2147483648|
+------+-----------+
|`sym  |`          |
+------+-----------+
|`price|__         |
+------+-----------+
|`size |_2147483648|
+------+-----------+

Complex datatypes

Flips and dicts are mapped to J boxed tables. Examples:

   cmd 'd:`x`y`z!(("two";"three";"five";"seven");2 3 5 7;`pi`hk`de`hk)'

   get 'd'                        NB. dict
+--+----------------------+
|`x|+---+-----+----+-----+|
|  ||two|three|five|seven||
|  |+---+-----+----+-----+|
+--+----------------------+
|`y|2 3 5 7               |
+--+----------------------+
|`z|`pi `hk `de `hk       |
+--+----------------------+

   get 'key d'                    NB. dict keys
`x `y `z

   get 'value d'                  NB. dict values
+----------------------+-------+---------------+
|+---+-----+----+-----+|2 3 5 7|`pi `hk `de `hk|
||two|three|five|seven||       |               |
|+---+-----+----+-----+|       |               |
+----------------------+-------+---------------+

   cmd 'f:flip d'                 NB. flip
   get 'f'
+----------------------+-------+---------------+
|`x                    |`y     |`z             |
+----------------------+-------+---------------+
|+---+-----+----+-----+|2 3 5 7|`pi `hk `de `hk|
||two|three|five|seven||       |               |
|+---+-----+----+-----+|       |               |
+----------------------+-------+---------------+

Key tables are mapped to a 2-item list: primary key columns;data columns. Example:

   cmd '`y xkey `f'                NB. convert to keyed table
   get 'f'
+---------+----------------------------------------+
|+-------+|+----------------------+---------------+|
||`y     |||`x                    |`z             ||
|+-------+|+----------------------+---------------+|
||2 3 5 7|||+---+-----+----+-----+|`pi `hk `de `hk||
|+-------+|||two|three|five|seven||               ||
|         ||+---+-----+----+-----+|               ||
|         |+----------------------+---------------+|
+---------+----------------------------------------+

      get 'key f'                  NB. key table keys
+-------+
|`y     |
+-------+
|2 3 5 7|
+-------+

      get 'value f'                NB. key table data
+----------------------+---------------+
|`x                    |`z             |
+----------------------+---------------+
|+---+-----+----+-----+|`pi `hk `de `hk|
||two|three|five|seven||               |
|+---+-----+----+-----+|               |
+----------------------+---------------+

Object lists

Object lists are mapped to J boxed lists. For example:

   cmdr '(`one;(2 3 5;"seven");01100b)'
+----+-------------+---------+
|`one|+-----+-----+|0 1 1 0 0|
|    ||2 3 5|seven||         |
|    |+-----+-----+|         |
+----+-------------+---------+

Temporal numbers

Day zero is 2000 1 1.

Dates are integer day numbers, months are integer month numbers. For example:

   get '1999.01.01,2000.01.01,2003.03.23'
_365 0 1177

   get '1999.01m,2000.01m,2003.03m'
_12 0 38

Minutes, second, time are all integers. For example:

   get '00:00,01:00,08:31'
0 60 511

   get '00:00:00.123,08:31:24.000'
123 30684000

Datetime is a float: day number + seconds as a fraction of day. For example:

   0j6 ":  get '2003.03.23T08:31:24'
1177.355139

   (60 #. 8 31 24) % 86400
0.355139

Message format

Interprocess communication for the internal layout of messages

Initial connection

Messages are plain text, zero terminated. Once a connection is established, the client sends user name and optional password, and the server responds with "c" if successful, or "e" if error.

Subsequent communications

The message header has 8 bytes:

byte |
-----|----------------------------------------------------------- 0 | endian: 0=big endian (e.g. Sun); 1=little endian (e.g PC) 1 | type: 0=async; 1=sync; 2=response from Q
2, 3 | 0 (unused)
4-7 | overall message length

The rest of the message is the data.

Atoms

Atoms have a single byte giving the type (e.g. int is 250 = 256-6), followed by the data. The data size is fixed except for varchar which is zero-terminated. For example, using the toQ keyword in the J client:

   a. i. toQ 123456
250 64 226 1 0

   256 #. |. }. a. i. toQ 123456
123456

Lists

Lists have a byte giving the type (e.g. int is 6), followed by 0, followed by the length of the list in 4 bytes. The data size is the length times the fixed size for an atom, except for varchar, for which individual atoms are zero-terminated. For example:

   a. i. toQ 123456 + i.3
6 0 3 0 0 0 64 226 1 0 65 226 1 0 66 226 1 0

   256 #. _4 |.\ 6 }. a. i. toQ 123456 + i.3
123456 123457 123458

   a. i. toQ s: 'two';'three'
11 0 2 0 0 0 116 119 111 0 116 104 114 101 101 0

Dict, flip, keyed table

Dict has a byte giving the type (99), followed by two lists. For example, the dictionary:

`x`y!(`a;10)

has representation:

99 11 0 2 0 0 0 120 0 121 0 0 0 2 0 0 0 245 97 0 250 10 0 0 0

Therefore it is a list of varchar, followed by an object list consisting of an atom varchar and an atom integer.

A flip has a two-byte header, type 98 followed by 0, followed by the representation of its dictionary contents. For example, the flip:

flip(`x`y!(`a`b;10 11))

has representation:

98 0 99 11 0 2 0 0 0 120 0 121 0 0 0 2 0 0 0 11 0 2 ...

A keyed table is like a dict, except that the two lists are two flips, the key columns and the data columns. For example, if y is set as a key field in the above flip, the representation is:

99 98 0 99 11 0 1 0 0 0 121 0 0 0 1 0 0 0 6 0 2 0 0 0 ...

Object lists

Object lists have a header of length 6, followed by each data item.

The header is two 0 bytes, followed by the length of the object (i.e. number of top level items). For example:

   a. i. toQ 10;(s:<'papa');'0'
0 0 3 0 0 0 250 10 0 0 0 245 112 97 112 97 0 246 48

Asynchronous messages

These are sent as a character string, and evaluated by the q value function.

Synchronous messages

These are sent as an object list. The first item is the function name as a text string, and the remaining items are the parameter list. For example, the J client set method sends an object list in the form:

"set";`abc;1 2 3

Error messages

Error messages from q have a type byte of 128, followed by a zero-terminated text string.