11. I/O¶
11.0 Overview¶
I/O in q is one of the most powerful and succinct features of the language. The names and behavior of the functions are idiosyncratic but the economy of expression is unrivaled.
I/O is realized via handles, which are symbolic names of resources such as files or machines on a network. One-and-done operations can be performed directly on the symbolic handle – e.g., you can read a file into memory in a single operation. For continuing operations, you open the symbolic handle to obtain an open handle. The open handle is a function that is applied to perform operations. When you have completed the desired operations, you close the open handle to free any allocated resources.
11.1 Binary Data¶
In q, files come in two flavors: text and binary. Routines to process text data have ‘0’ in their names, whereas routines to process binary data have ‘1’. A text file is considered to be a list of strings – i.e., a list of char lists – and a binary file is a list of byte lists. While all text files can also be processed as binary data, not all binary data represents text. As mentioned above, file operations use handles.
11.1.1 File Handles¶
A file handle is a symbol that represents the name of a directory or file on persistent storage. A symbolic file handle starts with a colon :
and has the form,
`:
[path]name
where the bracketed expression represents an optional path and name is a file or directory name. The combination should be recognized as valid by the underlying operating system.
Some q operations require that you append a trailing slash /
to indicate that you mean a directory. We will point these out.
It is generally easier to work with paths and names as strings so that blanks and other special characters can be handled easily. While `$
converts a string to a symbol, it can be awkward to include the leading :
required in the symbolic handle. The keyword hsym
, which inserts a leading colon into a symbol, serves this purpose.
q)hsym `$"/data/file name.csv"
`:/data/file name.csv
Note that q always represents separators in paths by the forward slash /
, even when running on Windows. If you run q on Windows, you can type either /
or \
but q will always display /
in its response.
Tip
To make life easier when you are generating paths dynamically, hsym
is idempotent, meaning that it will accept its own output and pass it through.
q)hsym hsym `$"/data/file name.csv"
_
11.1.2 hcount
and hdel
¶
The first one-and-done operation that works directly on a symbolic file handle is hcount
, which returns a long representing the size of the file in bytes as reported by the OS.
q)hcount `:/data/solong.txt
35
The next one-and-done is hdel
, which instructs the OS to remove the file specified by its symbolic handle operand.
q)hdel `:/data/solong.txt
`:/data/solong.txt
Some notes.
- The return value of the symbolic file handle itself indicates that the deletion was successful. It should not be confused with an error message, which starts with a tick rather than a backtick.
- You will get an error message if the file does not exist or if the delete cannot be performed.
- You will not be prompted for confirmation. Back up any files that are important.
11.1.3 Serializing and Deserializing q Entities¶
Every q entity can be serialized and persisted to storage. Unlike traditional languages, where you must instantiate serializers and writers, things are simple and direct in q. This is because q data is self-describing, so that its internal representation can be written out as a sequence of bytes and then read directly back into memory. This is as close to the Star Trek transporter as we are likely to get.
The magic is done by (an overload of) the binary set
, whose left operand is a file handle and right operand is the entity to be written. The result is the symbolic handle of the written file. The file is automatically closed once the write is complete.
q)`:/data/a set 42
`:/data/a
q)`:/data/L set 10 20 30
_
q)`:/data/t set ([] c1:`a`b`c; c2:10 20 30)
_
The behavior of set
is to create the file if it does not exist and overwrite it if it does. It will also create the directory path if it does not exist.
A serialized q data file can be read using (an overload of) the unary get
, whose argument is a symbolic file handle and whose result is the q entity contained in the data file.
q)get `:/data/a
42
q)get `:/data/L
_
q)get `:/data/t
_
An equivalent way to read a data file is with (an overload of) value
.
q)value `:/data/t
_
Alternatively, you can use the command \l
to load a data file into memory and assign it to a variable with the same name as the file. Here you do not use a file handle; rather, specify the path to the file without any decoration. In a fresh q session,
q)t
't
q)\l /data/t
`t
q)t
_
11.1.4 Binary Data Files¶
As with traditional languages, for continuing operations on a q data file, you open the file, perform the operation(s) and then close it. Unlike traditional languages, opening a symbolic handle returns a function, called an open handle, that is used to perform operations.
As mentioned previously, q files come in two flavors, binary and text. Serialized q data persisted with set
is written in binary form with a header at the beginning of the file. You can read it as raw binary data to inspect its internals.
Open a data file handle with hopen
, whose result is a function called the open handle. This function should be stored in a variable, traditionally h
, which is functionally applied to data to write it to the file. We will explain the result of applying the open handle shortly. We begin with a file containing serialized q data and show how to append to it.
q)`:/data/L set 10 20 30
`:/data/L
q)h:hopen `:/data/L
q)h[42]
3i
q)h 100 200
3i
Always apply hclose
to the open handle to close it and flush any data that might be buffered.
Failure to do so may cause your program to run out of file handles unnecessarily.
We verify that the appends have been made.
q)hclose h
q)get `:/data/L
10 20 30 42 100 200
We can also create a new file and write raw binary data to it.
q)h:hopen `:/data/raw
q)h[42]
3i
q)h 10 20 30
3i
q)hclose h
Now, what is the deal with the 3i
return value of applying the open handle?
q)h:hopen `:/data/raw
q)h 43
3i
In fact, the return value is the value of the open handle itself.
q)h
3i
Surely, you say, we can’t use an int as a function to write data. But you would be wrong.
q)h:hopen `:/data/new
q)h
3i
q)3i[100 200 300]
3i
q)hclose 3i
q)get `:/data/new
_
The last expression above signals an error.
get
requires a file to be initialized byset
. But appending data to a file initialized byset
is neither reliable nor recommended. Ed.
Tip
Apparently q assigns an int to each open file and keeps track of which int values are valid handles. This accounts for the cryptic error message when you attempt to use variables with simple list notation.
q)a:42
q)b:43
q)a b
': Bad file descriptor
11.1.5 Writing and Reading Binary¶
Apply read1
on a file handle to read any file into q as a list of bytes. For example, we can read the previously serialized value L
as bytes.
q)read1 `:/data/L set 10 20 30
0xfe2007000000000003000000000000000a0000000000000014000000000000001e..
This shows the internal representation of the serialized q entity. How cool is that?
If you want to write raw binary data, as opposed to the internal representation of a q entity containing the data, use the infelicitously named 1:
. It takes a symbolic file handle as its left argument and a simple byte list as its right argument. Bytes in the right operand are essentially streamed to the file.
q)`:/data/answer.bin 1: 0x06072a
`:/data/answer.bin
q)read1 `:/data/answer.bin
0x06072a
11.1.6 Using Apply Amend¶
Fundamentalists can use Apply Amend in place of set
to serialize q entities to files. To write the file, or overwrite an existing file, use assign :
.
q).[`:/data/raw; (); :; 1001 1002 1003]
`:/data/raw
q)get `:/data/raw
1001 1002 1003
To append to an existing file use ,
.
q).[`:/data/raw; (); ,; 42]
`:/data/raw
q)get `:/data/raw
1001 1002 1003 42
11.2 Save and Load on Tables¶
We have already seen that it is easy to write and read tables to/from persistent storage.
q)`:/data/t set ([] c1:`a`b`c; c2:10 20 30; c3:1.1 2.2 3.3)
`:/data/t
q)get `:/data/t
_
The save
and load
functions make this even easier.
In its simplest form, save
serializes a table in a global variable to a binary file having the same name as the variable. It overwrites an existing file.
q)t:([] c1:`a`b`c; c2:10 20 30; c3:1.1 2.2 3.3)
q)save `:/data/t
`:/data/t
q)get `:/data/t
_
This is equivalent to using set
above with the table name as file name.
As you might expect, load
is the inverse of save
meaning that it reads a serialized table from a file into a variable with the same name as the file. It creates the variable in the workspace or overwrites it if it already exists.
In a fresh q session after t
has been saved as above,
q)t / t doesn't exist
't
q)load `:/data/t
`t
q)t / now it does
_
You can also use save
to write a table to a text file. You determine the format of the text with the file extension in the file handle.
All the following versions of save
can also be performed with the more general 0:
– see §11.5.
Save the table with .txt
extension to obtain tab-delimited records. There is no corresponding load
but you can parse the text file – see §11.5.1.
q)save `:data/t.txt
`:data/t.txt
The resulting file is
c1\tc2\tc3
a\t10\t1.1
b\t20\t2.2
c\t30\t3.3
Save the table with .csv
extension to obtain comma-separated values. There is no corresponding load
but you can parse the CSV file – see §11.5.2.
q)save `:data/t.csv
`:data/t.csv
The resulting file is
c1,c2,c3
a,10,1.1
b,20,2.2
c,30,3.3
Save the table with .xml
extension to obtain XML records. There is no direct way to read XML into q although libraries have been contributed – see code.kx.com.
q)save `:data/t.xml
`:data/t.xml
The resulting file is
<R>
<r><c1>a</c1><c2>10</c2><c3>1.1</c3></r>
<r><c1>b</c1><c2>20</c2><c3>2.2</c3></r>
<r><c1>c</c1><c2>30</c2><c3>3.3</c3></r>
</R>
Save the table with .xls
extension obtain an Excel spreadsheet. This file can be loaded by Excel work-alikes.
q)save `:data/t.xls
`:data/t.xls
11.3 Splayed Tables¶
We have already seen how to persist a table to a file using set
. There are no restrictions on the types of columns in the table or the file name in this scenario.
q)`:/data/t set ([] c1:`a`b`c; c2:10 20 30; c3:1.1 2.2 3.3)
`:/data/t
q)get `:/data/t
_
This creates a single file, as the OS verifies.
>ls -l /data/t
-rw-r--r-- 1 jeffry wheel 98 Mar 6 08:22 /data/t
For larger tables that may not fit into memory on all machines, you can ask q to serialize each column of the table to its own file in a specified directory. A table persisted in this form is called a splayed table. The advantage is that when querying a splayed table, only the columns referred to in the query will be loaded into memory. This is a substantial memory win for a table having many columns.
It is worthwhile looking up the origin of the English word “splay”. Also, please don’t spay your tables.
To splay a table, use set
and specify a directory as the target location indicated by a trailing slash /
in the left operand.
q)`:/data/tsplay/ set ([] c1:10 20 30; c2:1.1 2.2 3.3)
`:/data/tsplay/
List the directory in the OS and you will see a directory tsplay
that contains three files, one file for each column in the original table, as well as a hidden .d
file.
>ls -l -d /data/tsplay
drwxr-xr-x 5 jeffry wheel 170 Mar 6 08:36 /data/tsplay
>ls -l -a /data/tsplay
total 24
drwxr-xr-x 5 jeffry wheel 170 Mar 6 08:36 .
drwxr-xr-x 9 jeffry wheel 306 Mar 6 08:36 ..
-rw-r--r-- 1 jeffry wheel 14 Mar 6 08:36 .d
-rw-r--r-- 1 jeffry wheel 40 Mar 6 08:36 c1
-rw-r--r-- 1 jeffry wheel 40 Mar 6 08:36 c2
Nearly all the metadata regarding the splayed table can be read from the file system – i.e., the name of table from directory and names of the columns from the files. The one missing bit is the order of the columns, which is stored as a serialized list in the hidden .d
file.
q)get hsym `$"/data/tsplay/.d"
`c1`c2
Important
There are restrictions on tables that can be splayed.
- All columns must be simple or compound lists. The latter means a list of simple lists of uniform type. An arbitrary general list column cannot be splayed.
- Symbol columns must be enumerated.
Thus the following succeed.
q)`:/data/tok/ set ([] c1:2000.01.01+til 3; c2:1 2 3)
`:/data/tok/
q)`:/data/tok/ set ([] c1:1 2 3; c2:(1.1 2.2; enlist 3.3; 4.4 5.5))
`:/data/tok/
And the following fail.
q)`:/data/toops/ set ([] c1:1 2 3; c2:(1;`1;"a"))
k){$[@x;.[x;();:;y];-19!((,y),x)]}
'type
q)`:/data/toops/ set ([] c1:`a`b`c; c2:10 20 30)
k){$[@x;.[x;();:;y];-19!((,y),x)]}
'type
The first set
above works in later versions of kdb+. [Ed.]
The convention for enumerating symbols in splayed tables is to enumerate all symbol columns in all tables over the domain sym
and store the resulting sym list in the root directory – i.e., one level above the directory holding the splayed table. You can do this manually but practically no one does.
q)`:/db/tsplay/ set ([] `sym?c1:`a`b`c; c2:10 20 30)
`:/db/tsplay/
q)sym
`a`b`c
q)`:/db/sym set sym
`:/db/sym
Normally folks use one of the .Q
utilities, in spite of the official KX admonition not to use them. For example, here we use .Q.en
.
q)`:/db/tsplay/ set .Q.en[`:/db; ([] c1:`a`b`c; c2:10 20 30)]
`:/db/tsplay/
Only unofficially documented, .Q.en
prepares a qualified table for splaying by enumerating all its symbol columns. The first argument is the symbolic file handle of the root directory for the persistent residence of the enumeration domain sym
(no choice in the name). The second argument is a table. See §14.5.2 for more detail on its behavior.
Update: .Q
is now documented at code.kx.com. Ed.
11.4 Text Data¶
We have seen that q views a record in a binary data file as a list of bytes. Similarly, a record in a text file is viewed as a list of char – i.e., a string. Thus reading a text file results in a list of strings and you pass a list of strings to write to a text file.
11.4.1 Reading and Writing Text Files¶
Read a text file with the unary read0
that takes a symbolic file handle argument. The result is a list of strings, one for each line in the file. For the file /data/solong.txt
with content,
So long
and thanks
for all the fish
we find,
q)read0 `:/data/solong.txt
"So long"
"and thanks"
"for all the fish"
You can see the underlying binary values of the text by using read1
or casting the result of read0
to bytes.
q)read1 `:/data/solong.txt
_
q)"x"$read0 `:/data/solong.txt
0x4c696665
0x54686520556e697665727365
0x416e642045766572797468696e67
Or you can read the data as binary and cast the result to char. Observe that the data is a simple list of char so the newline character does not cause line breaks in the console display.
q)"c"$read1 `:/data/solong.txt
"Life\nThe Universe\nAnd Everything\n"
To write string as text, use the (infelicitously named) binary 0:
, which takes a file handle in the left operand and a list of strings in the right operand. It creates the directory path if necessary and overwrites the file if it already exists.
q)`:/data/solong.txt 0: ("Life"; "The Universe"; "And Everything")
`:/data/solong.txt
q)read0 `:/data/solong.txt
_
11.4.2 Using hopen
and hclose
¶
Just as with a binary data file, a symbolic text file handle can be opened with hopen
. The result is again an int that is conventionally stored in the variable h
and is used with function application syntax to write data. The difference is that instead of using plain h
to write binary data, you use neg[h]
to write strings as text. Seriously.
q)h:hopen `:/data/new.txt
q)neg[h] enlist "This"
-3i
q)neg[h] ("and"; "that")
-3i
q)hclose h
q)read0 `:/data/new.txt
_
Observe that you apply hclose
to h
, not to neg[h]
.
If the file already exists, opening with hopen
and applying the open handle will append rather than overwrite.
q)h:hopen `:/data/new.txt
q)neg[h] ("and"; "more")
-3i
q)hclose h
q)read0 `:/data/new.txt
_
11.4.3 Preparing Text¶
We saw the built-in functions for saving tables as text files in §11.2. When you need to control the filename, you can write the table yourself with 0:
, but then you must prepare the table columns as formatted text. A separate overload of 0:
is available for this purpose. A confusing naming convention, to say the least.
In this use, 0:
has as left operand a char delimiter and as right operand a table or list of columns. Observe the use of the pre-defined constant csv
, which is simply ","
.
q)t:([] c1:`a`b`c; c2:1 2 3)
q)"\t" 0: t
"c1\tc2"
"a\t1"
"b\t2"
"c\t3"
q)"|" 0: t
_
q)csv
","
q)csv 0: t
_
q)`:/data/t.csv 0: csv 0: t
_
In the last snippet we applied 0:
with two different meanings: to prepare and then write text. We hope you’ve grown fond of this name, since §11.5 will introduce yet another version of 0:
for parsing text records.
11.5 Parsing Records¶
Binary forms of 0:
and 1:
parse individual fields according to data type from text or binary records. Field parsing is based on the following field types.
0 | 1 | Type | Width(1) | *Format(0) * |
---|---|---|---|---|
B | b | boolean | 1 | [1tTyY] |
X | x | byte | 1 | |
H | h | short | 2 | [0-9a-fA-F][0-9a-fA-F] |
I | i | int | 4 | |
J | j | long | 8 | |
E | e | real | 4 | |
F | f | float | 8 | |
C | c | char | 1 | |
S | s | symbol | n | |
P | p | timestamp | 8 | date?timespan |
M | m | month | 4 | [yy]yy[?]mm |
D | d | date | 4 | [yy]yy[?]mm[?]dd or [m]m/[d]d/[yy]yy |
Z | z | datetime | 8 | date?time |
N | n | timespan | 8 | hh[:]mm[:]ss[[.]ddddddddd] |
U | u | minute | 4 | hh[:]mm |
V | v | second | 4 | hh[:]mm[:]ss |
T | t | time | 4 | hh[:]mm[:]ss[[.]ddd] |
blank | skip | |||
* | literal chars |
The column labeled ‘0’ contains the (upper case) field type char for text data. The (lower case) char in column ‘1’ is for binary data. The column labeled ‘Width(1)’ contains the number of bytes that will be parsed for a binary read. The column labeled ‘Format(0)’ displays the format(s) that are accepted in a text read.
The parsed records are returned in column form rather than row form to make it easy to associate a list of symbol names with !
and then flip into a table.
11.5.1 Fixed-Width Records¶
The binary form of 0:
and 1:
for reading fixed length files is,
(
Lt;
Lw) 0:
f
(
Lt;
Lw) 1:
f
The left operand is a nested list containing two items: Lt is a simple list of char containing one letter per field; Lw is a simple list of int containing one integer width per field. The sum of the field widths in Lw should equal the width of the record. The result of the function is a list of lists, one list arising from each field.
We demonstrate 0:
here since it is more commonly used; 1:
works analogously. The simplest form of the right operand f is a symbolic file handle. For example, suppose we have a file with records of the form,
1001 98.000ABCDEF1234Garbage2015.01.01
1002 42.001GHUJKL0123Garbage2015.01.02
1003 44.123nopqrs9876Garbage2015.01.03
We could parse the records of the file with,
q)("JFS D";4 8 10 7 10) 0: `:/data/Fixed.txt
1001 1002 1003
98 42.001 44.123
ABCDEF1234 GHUJKL0123 nopqrs9876
2015.01.01 2015.01.02 2015.01.03
This reads a text file containing fixed length records of width 39. The first field is a long occupying 4 positions; the second field is a float occupying 8 positions; the third field consists of a symbol occupying 10 positions; the fourth slot of 6 positions is ignored; the fifth field is a date occupying 10 positions.
You might think that the widths are superfluous, but they are not. The actual data width can be narrower than the normal size due to small values, as in our case of the long field. Or you may need to specify a width larger than that required by the corresponding data type due to whitespace in the fields, as in the case of our float field.
Observe how easy it is to make a table from the result.
q)flip `c1`c2`c3`c4!("JFS D";4 8 10 7 10) 0: `:/data/Fixed.txt
c1 c2 c3 c4
---------------------------------
1001 98 ABCDEF1234 2015.01.01
1002 42.001 GHUJKL0123 2015.01.02
1003 44.123 nopqrs9876 2015.01.03
Also note that it is possible to parse a list of strings using the same format, since they represent text records in memory.
q)fixed: read0 `:/data/Fixed.txt
q)("JFS D";4 8 10 7 10) 0: fixed
_
The more general form for the right operand f is,
(
hfile;
i;
n)
where hfile is a symbolic file handle, i is the offset into the file to begin reading and n is the number of bytes to read. This is useful for sampling a file or for large files that cannot be read into memory in a single gulp.
A read operation should begin and end on record boundaries or you will get meaningless results.
In our trivial example, the following reads just the second and third records,
q)("JFS D";4 8 10 7 10) 0: (`:/data/Fixed.txt; 40; 80)
_
11.5.2 Variable Length Records¶
The binary form of 0:
and 1:
for reading variable length, delimited files is
(
Lt;
D) 0:
f
(
Lt;
D) 1:
f
The left operand is a list comprising two lists. Lt is a simple list of char containing one type letter per corresponding field. D is either a char representing the delimiting character or an enlisted char.
Specify D as a delimiter char when the first record of the file does not contain column names. In this case, the result of the parse is a list of column lists, each of which contains items of type specified by Lt. The simplest form of the right operand f is a symbolic file handle.
For example, say we have a comma-separated file /data/Simple.csv
having records
1001,DBT12345678,98.6
1002,EQT98765432,24.75
1004,CCR00000001,121.23
Parsing with a delimiter char ","
results in a list of column lists. As with parsing fixed format records, it is easy to make the result into a table.
q)("JSF"; ",") 0: read0 `:/data/Simple.csv
1001 1002 1004
DBT12345678 EQT98765432 CCR00000001
98.6 24.7 121.23
q)flip `c1`c2`c3!("JSF"; ",") 0: read0 `:/data/Simple.csv
_
Observe that it is possible to retrieve the second field as a string instead of a symbol using "*"
as the data type specifier,
q)("J*F"; ",") 0: read0 `:/data/Simple.csv
1001 1002 1004
"DBT12345678" "EQT98765432" "CCR00000001"
98.6 24.7 121.23
Specify D as an enlisted char when the first record contains a separated list of names. Subsequent records are read as data specified by the types in Lt. The result is a table in which the column names are taken from the first record.
Say we have a comma-separated file /data/Titles.csv
having records,
id,ticker,price
1001,DBT12345678,98.6
1002,EQT98765432,24.7
1004,CCR00000001,121.23
Reading with an enlisted ","
delimiter results in a table.
q)("JSF"; enlist ",") 0: `:/data/Titles.csv
id ticker price
-----------------------
1001 DBT12345678 98.6
1002 EQT98765432 24.7
1004 CCR00000001 121.23
11.5.3 Key-Value Records¶
The operator 0:
can also be used to process text representing key-value pairs. In this situation, the left operand is a three-character string Pf that specifies the pair format. The first char of Pf can be "S" to indicate the key is a string or "I" to indicate the key is an integer. The second char indicates the key-value separator. The third char indicates the pair delimiter.
The following examples illustrate various combinations in Pf.
q)"S=;" 0: "one=1;two=2;three=3"
one two three
,"1" ,"2" ,"3"
q)"S:/" 0: "one:1/two:2/three:3"
_
q)"I=;" 0: "1=one;2=two;3=three"
_
Again it is easy to make the result into a table.
q)flip `k`v!"I=;" 0: "1=one;2=two;3=three"
k v
---------
1 "one"
2 "two"
3 "three"
11.6 Interprocess Communication¶
The ease with which a q process can communicate with another q process residing on the network is one of the most impressive features of q. We shall cover all the basics of interprocess communication (IPC) so that you can follow the section on callbacks in Chapter 1 – Q Shock and Awe.
We shall use the following terminology. The process that initiates the communication is called the client, while the process receiving and processing requests is the server. The server process can be on the same machine, the same network, a different network or on the Internet, so long as it is accessible. The communication can be synchronous (wait for a result to be returned) or asynchronous (don’t wait and no result returned).
The only way to learn IPC is to do it, and the easiest way to do this is to set up two processes on the same machine. We recommend you use the machine running your q sessions for this tutorial, provided it will allow a port to be opened. In what follows, we shall assume that a server q process has been started on a machine with an open port.
>q -p 5042
q)
The client process is a separate q process running on the same machine.
>q
q)
11.6.1 Communication Handle¶
Symbolic communication handles look similar to file handles but they specify resources on the network. A communication handle has the form,
`:
[server]:
port
Here the bracketed expression represents an optional server machine identifier and port is a port number. An omitted server specification, or one of the form localhost
, refers to the machine on which the originating q session lives. The following both refer to port 5042 on the same machine as the q session in which they are entered.
q)`::5042
_
q)`:localhost:5042
_
You can refer to a machine on the network by name. For example, on the author’s laptop the following is equivalent to the two previous network handles.
q)`:aerowing:5042
_
You can use the IP address of a machine.
q)`:198.162.0.2:5042
_
Finally, you can also use a URL.
q)`:www.myurl.com:5042
_
11.6.2 Opening a Connection Handle¶
As with a file handle, apply hopen
to a communication handle to obtain an open connection handle that is used as a function. As before, the value is an int that is traditionally stored in the variable h
. Also as with file I/O, the behavior of this function differs between using the original positive handle or its negation.
Let’s see how this works with our two sessions. (You did start them, didn’t you?). Remember, the session that opened port 5042 is the server; the other session is the client. In the client session, open a handle to the server and store it in h
, then apply h
to the string as shown. Finally close the connection handle.
q)h:hopen `::5042
q)h "a:6*7"
q)h "a"
42
q)hclose h
Whitespace between h
and the quoted string is optional, as this is simply prefix syntax. We include it for readability.
As you have no doubt realized, the application of h
sent the string to the server to be evaluated. On the server, we see,
q)a
42
How cool is that?
11.6.3 Remote Execution¶
We have seen that when you open a connection to a q process, you have the full capability of that process available remotely. Apply the connection handle to any q expression in a string and it will be evaluated on the server. As you contemplate the IPC Zen, a dark cloud passes over your tranquility. You realize that, by default, the server is wide open.
Allowing quoted q strings to be executed on a server makes the server susceptible to all manner of breaches.
Good practice does not permit this on a production server. You can mitigate this by having your server process accept only requests whose first item is a symbol (see below), which you should verify is the name of a function you have decided to expose.
An alternative format for remote execution is to apply the connection handler to a list of the form
(
f;
arg1;
arg2;...)
Here f is a client-side expression that evaluates to a map that will be applied on the server. It can be:
- The value of, or variable associated to, a map on the client
- The symbolic name of a map on the server.
We use the term map here to be any q expression that can be evaluated as function application – e.g., a list on an index, a dictionary on a key or a function on an argument. Most commonly f is a function
The remaining items arg1, arg2, … are optional values sent along to the server for the evaluation. These are arguments when f
is a function, indices when it is a list, or keys when it is a dictionary.
Application of the connection handle to such a list sends the list to the server where it is evaluated. Any result is sent back to the client, where it is presented as the result of the connection handle application. By simply applying the naked handle, this sequence of steps is synchronous, meaning that execution of the q session on the client blocks until the result of the server evaluation is returned.
Our examples will cover the case when f
is of function type since that is most common. We first consider the first case when f
is a map on the client side. In this situation the function (list, dictionary, etc.) is actually transported to the server along with the supplied arguments, where it is applied.
On the client in our two-session setup:
q)h:hopen`::5042 / client
q)h ({x*y}; 6; 7)
42
q)f:{x*y}
q)h (f; 6; 7)
42
Before you get too enamored of this form, we point out the limitations that disqualify it from production use. First, global variables referred to in the transported function will need to be present remotely in the exact contexts in effect when the function was defined. This can be avoided by restricting f
to be a pure function that does not refer to any global entities. More damning is:
Allowing a function to be sent to the server for remote execution is as dangerous as sending quoted q strings
The function can access resources on the server and instigate an attack. Good practice does not permit this in production environments.
The remaining format for remote execution can be made safe for production environments. The function to be executed remotely must already be defined on the server and you pass its name and arguments via the connection handle.
On the server,
q)g:{x*y} / server
On the client,
q)h (`g; 6; 7) / client
42
Now consider the case when the remote function performs an operation on a table and returns the result. This is the q analogue of a remote stored procedure. For example, suppose t
and f
are defined on the server as,
q)t:([] c1:`a`b`c; c2:1 2 3) / server
q)f:{[x] select c2 from t where c1=x}
Now “call” the function f
remotely from the client.
q)h (`f; `b) / client
c2
--
2
The difference from SQL stored procedures is that the remote procedure can be any q function on the server, making the full power of q available remotely.
11.6.4 Synchronous and Asynchronous Messages¶
The IPC in the previous sections was synchronous, meaning that upon application of the connection handle, the client process blocks, waiting for a result from the server before proceeding. The value returned from the server becomes the return value of the open handle application.
Under the covers, IPC is implemented as messages passed over an open connection between q processes. When the positive open handle is applied to an argument, the message passing is synchronous, meaning that the following steps occur in sequence.
- The client sends a message containing the argument(s) of the handle application to the server and waits for a return message.
- The server receives the message, interprets it as the appropriate function application and obtains the result.
- The server sends a message containing the result back to the client.
- The client receives the result and resumes execution from the point it left off.
When a client sends multiple messages to a server in synchronous message passing, the next message is not sent until the result of the previous message is received. Consequently the messages always arrive at the server in the order in which they are sent. Also, the results from the server arrive back at the client in the order in which the original messages were sent.
It is also possible to perform asynchronous IPC in q. In this case the message is sent to the server and execution on the client continues immediately. In particular, there is no return value from the server. This is useful to initiate a task on the server when you don’t care about the result. For example, you could initiate a long running operation, or you could send a message that the server will route to other processes.
Use the negation of the open connection handle to send an asynchronous message to the server. Let’s define an instrumented function on the server to demonstrate what is happening.
q)sq:{0N!x*x} / server
Now invoke sq
asynchronously from the client
q)neg[h] (`sq; 5) / client
q)
You will observe 25 displayed on the server console. Also, the client session returns immediately with no return value. The expression on the console actually has a nil value ::
that is suppressed by the console display.
When sending asynchronous messages, always send an empty “chaser” message immediately before applying hclose
to the open handle
If you do not do this, buffered messages may not be sent when the connection is closed.
In order to convince ourselves that the client actually does return immediately without waiting for a return from the server, we wrap the client expression in a function. Observe that the client continues with the next statement.
q){neg[h] (`sq.; 5); 42}[] / client
42
Because a q session is single threaded by default, the server will process messages in the order in which they are received. However, in asynchronous messaging there is no guarantee that the messages arrive at the server in the order in which they are sent. It can be difficult to observe indeterminancy in simple examples, but you must assume that it will occur in practice.
11.6.5 Processing Messages¶
Assuming that you have passed the server either a function from the client side or the name of a function on the server side, the appropriate function is evaluated on the server. During evaluation, the communication handle of the remote process is available in the system variable .z.w
( “who” called). For an asynchronous call, this can be used to send messages back to the server during the function application on the server.
Both the client and the server have connection handles when a connection between them is opened
However, these handles are assigned independently and their int values are not equal in general.
Here is a simple example showing how to use .z.w
to send a message back to the client. On the server, we define a function that displays its received parameter and then asynchronously calls mycallback
with the passed argument incremented.
q)f:{show "Received ",string x; neg[.z.w] (`mycallback; x+1)}
On the client we define mycallback
to display its parameter on the console. Then we make an asynchronous call to the function f
on the server with an argument of 42.
q)mycallback:{show "Returned ",string x;}
q)neg[h] (`f; 42)
q)"Returned 43"
The result is that "Received 42"
is displayed on the server console and "Returned 43"
is displayed on the client console. Congratulations! We have just invented callbacks in q.
When performing asynchronous messaging, always use neg[.z.w]
to ensure that all messages are asynchronous
Otherwise you will get a deadlock as each process waits for the other.
You can override the default behavior of message processing in q by assigning your own handler(s) to the appropriate system variables. Assign your function to the variable .z.pg
to trap and process synchronous messages and to .z.ps
for asynchronous messages. The names end in ‘g’ and ‘s’ because synchronous processing has "get" semantics and asynchronous processing has "set" semantics.
In the following we set the asynchronous handler to a trivial function, essentially ignoring asynchronous calls.
On the server,
q).z.ps:{show "ignore"} / server
On the client send an asynchronous message.
q)neg[h] "6*7" / client
This results in "ignore"
being displayed on the server console.
Now we set the synchronous handler to a function that only accepts “safe” remote calls by function name. It then performs a protected evaluation on the function with the arguments passed, thus ensuring that a failed application does not hang the server.
On the server,
q).z.pg:{$[-11h=type first x; .[value first x; 1_x; ::]; `unsupported]}
Now send synchronous messages from the client.
q)h (`sq; 5) / client
25
q)h (`sq; `5)
"type"
q)h "6*7"
`unsupported
q)h ({x*y};6;7)
`unsupported
You can also specify handlers to be called upon connection open and close by assigning functions to the system variables .z.po
and .z.pc
, respectively. The connection handle of the sending process is passed as the lone argument to the functions assigned to .z.po
and to .z.pc
.
Here is a simple example that tracks connections and allows client processes to register callbacks with the server. Start a fresh q session on the server and open port 5042. Create a keyed table called Registry
and define a function that can be invoked remotely to register a callback. Attach a handler to .z.po
that initializes a dummy entry in Registry
for the connection being opened and attach a handler to .z.pc
to remove the record when a connection is closed.
q)Registry:([zw:`int$()] callback:`symbol$())
q)register:{[cb] `Registry upsert (.z.w; cb);}
q).z.po:{`Registry upsert (x; `unregistered);}
q).z.pc:{delete from `Registry where zw=x;}
Start a fresh q session on the client and connect to the server.
q)h:hopen`::5042 / client
We check that an item has been entered into Registry
on the server.
q)Registry / server
zw| callback
--| ------------
6 | unregistered
Next we register the name of a callback function from the client. Note the asynchronous message.
q)neg[h] (`register; `mycallback) / client
Again we check Registry
on the server and observe that our callback name has indeed been registered.
q)Registry / server
zw| callback
--| ----------
6 | mycallback
Finally, we close the connection on the client.
q)hclose h / client
And observe that the client has been automatically unregistered.
q).z.pg:{show x 0; show x 1; ; string value 1_x 0}
zw| callback
--| --------
11.6.6 Remote Queries¶
In this section, we demonstrate how to execute q-sql queries against a remote server. First, we splay a table to stand for a time-series database. We use the mktrades
script that we created in §9.3.1 to create a trades table with 1,000,000 rows and then splay it to disk.
q)trade:mktrades[`aapl`goog`ibm; 1000000]
q)(`:/db/trade/) set .Q.en[`:/db;]
_
Now start a fresh server process (the server), open a port, say 5042, and map the splayed trade table into memory. Check that the mapping succeeded by running a query.
q)\p 5042 / server
q)\l /db
q)select from trade where dt=2015.01.01,sym=`ibm
dt tm sym qty px
---------------------------------------
2015.01.01 00:00:01.796 ibm 7080 218.74
2015.01.01 00:00:10.581 ibm 3250 206.88
..
Leave the server process running and start another fresh process (the client), open a connection to the server and send the same query to the server for remote execution.
q)h:hopen`::5042 / client
q)h "select from trade where dt=2015.01.01,sym=`ibm"
dt tm sym qty px
---------------------------------------
2015.01.01 00:00:01.796 ibm 7080 218.74
2015.01.01 00:00:10.581 ibm 3250 206.88
..
We have already pointed out that allowing remote execution of arbitrary strings is bad practice because it exposes the server to injection attack. So here is a simplistic example of a “safe” function that can be used as a stored procedure. It takes a symbolic table name, a list of symbolic column names for the result and a date range for the where phrase. Enter on the server:
q)extract:{[tn;cnms;dtrng] ?[tn;enlist (within;`dt; dtrng);0b;cnms!cnms]}
Now on the client we (synchronously) call the stored procedure by name with appropriate arguments.
q)h (`extract;`trade;`dt`tm`sym`qty`px;2015.01.01 2015.01.02)
dt tm sym qty px
----------------------------------------
2015.01.01 00:00:01.194 aapl 6770 94.62
2015.01.01 00:00:01.796 ibm 7080 218.74
..
In an actual application you would validate the input parameters and wrap the core evaluation in protected evaluation to trap unanticipated errors. You would also want to implement an entitlements system based on LDAP.
11.7 HTTP and Web sockets¶
11.7.1 HTTP Connections¶
When you open a port in a q session, by default that session serves HTTP requests. To demonstrate this, start a q session and open a port, say 5042. Then bring up a relatively recent browser on the same machine (the author uses Chrome) and enter the following URL
http://localhost:5042/?6%2A7
You should see 42 in the browser page display.
You can trap HTTP GET and POST traffic by assigning functions to the system variables .z.ph
and .z.pp
respectively. The default handler for .z.ph
is to evaluate the content of the first item of the passed argument.
There is no default handler for .z.pp
.
Here is a simple example that duplicates the default GET processing and shows the two items of its list argument. Define the following handler on the server process opened previously. It displays the two items of the input list then executes the first after removing the leading ?
and then returns the result as a string.
q).z.ph:{show x 0; show x 1; ; string value 1_x 0} / server
Now enter the following from a browser on the same machine.
http://localhost:5042/?6*7
The server will display,
q)"?6*7"
Host | "localhost:5042"
Connection | "keep-alive"
Cache-Control | "max-age=0"
Accept | "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,\*/\*;q=0.8"
User-Agent | "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36"
Accept-Encoding | "gzip, deflate, sdch"
Accept-Language | "en-US,en;q=0.8"
And the browser page displays “42”.
11.7.2 Basic WebSockets¶
WebSockets is a network protocol that upgrades an initial HTTP handshake into a TCP/IP socket connection. It was initially used to enhance communication capability between browsers and web servers but it can be used for general client-server applications. Once the WebSocket connection is established, either the client or server can message the other; in particular, this provides the capability for the server to push data to the client.
As of this writing (Sep 2015) q implements only asynchronous messaging in WebSockets.
In this section we show the basic mechanism for establishing a WebSocket connection between a browser and a q process acting as the server. We use Chrome for the examples but recent versions of Internet Explorer are now WebSockets-capable and should work similarly.
In the examples of this section we assume basic familiarity with HTML5 and JavaScript.
We begin with an extremely simple HTML page with a button that, when clicked, displays the answer to life, the universe and everything. Save the following as a text file sample0.html
in a location accessible to your browser.
<!doctype html>
<html>
<head>
<script>
function sayN(n) {
document.getElementById('answer').textContent = n;
}
</script>
</head>
<body>
<h1 style='font-size:200px' id='answer'></h1>
<button onclick='sayN(42)'>get the answer</button>
</body>
</html>
In our case we saved the file to /pages/sample0.html
on the local drive, so we enter the following URL in the browser:
file:///pages/sample0.html
You should see a page with a single button labeled “get the answer”. Click the button and you will see the answer in a very large font.
Now we enhance this basic page to connect to a q process via WebSockets and retrieve the answer from q. Save the following script as sample1.html
. We explain it below.
For simplicity in the example, we have placed a copy of c.js
in the pages directory. You should modify this to reflect its location in your installation.
<!doctype html>
<html>
<head>
<script src="c.js"></script>
<script>
var serverurl = "//localhost:5042/",
c = connect(),
ws;
function connect() {
if ("WebSocket" in window) {
ws = new WebSocket("ws:" + serverurl);
ws.binaryType="arraybuffer";
ws.onopen=function(e){
ws.send(serialize({ payload: "What is the meaning of life?" }));
};
ws.onclose=function(e){
};
ws.onmessage=function(e){
sayN(deserialize(e.data));
};
ws.onerror=function(e) {window.alert("WS Error") };
} else alert("WebSockets not supported on your browser.");
}
function sayN(n) {
document.getElementById('answer').textContent = n;
}
</script>
This script first declares the script c.js
, which is required for using q WebSockets.
The script then defines JavaScript variables
serverurl
to hold the URL of our q servicec
to hold the connection object returned by theconnect
functionws
to hold a WebSocket object.
The function connect()
is where the WebSocket action happens.
- It first tests to see if
WebSocket
is in the window, meaning that the browser supports WebSockets. If so, it makes the connection to the server; otherwise it displays an error alert. - The first step in the connection is to create a WebSocket object by connecting to the specified server URL, and storing the result in
ws
. - Then set the
binaryType
field inws
to the value needed by the q sockets code.
Now we assign handlers for the main WebSockets events.
- The open handler serializes (into q form) a JavaScript object with a
payload
field and then sends it to the server. Consequently when a connection is opened, we immediately ask the server the meaning of life. - The close handler is empty.
- The message handler deserializes the data field of the parameter
e
and applies thesayN
function to display the result on the page. - The error handler displays an alert page with the error message.
The sayN
function locates the answer
field on the page and places the text of its argument there. Finally, the script defines a simple HTML element answer.
In contrast, the server side q code is blissfully short. Start a fresh q session, open port 5042 and set the WebSockets handler .z.ws
to a function that will be invoked to handle WebSockets messages.
q)\p 5042
q).z.ws:{0N!-9!x; neg[.z.w] -8!42}
The handler first deserializes its parameter and displays it to the console for debugging, at which point we have no further use for it in this example. Then it serializes the answer to the question asked by the browser and asynchronously sends it back to the browser. That’s all there is to it!
Now point the browser to
file:///pages/sample1.html
and you will see the answer displayed on the page. At this point you are equipped to follow §1.19 in Q Shock and Awe.
11.7.3 Pushing Data to the Browser¶
In ordinary Web applications, the browser initiates interaction with the server. It sends a request to a specific URL on the server and the server replies with the requested page or data. Each such interaction is self-contained and is synchronous in that the browser waits for the server response.
In WebSockets the browser initiates the connection, but once the WebSocket request for protocol upgrade is successful, the browser – i.e., client – and the server are on equal footing. Either side can send messages. Moreover, in the current q implementation of WebSockets all interaction is asynchronous. Given that most current browsers and the default q session are both single-threaded, you don’t have to worry about races and deadlocks but you do have to set up callbacks.
In this section we demonstrate how the q server can push data to the browser, beginning with the browser script. Actually this script is a simplification of sample1.html
in that we remove the initial call to the server upon open; everything else remains the same. The key point is that the onmessage
handler will be called every time data is received, resulting in the data being displayed on the screen. Save the following as sample2.html
.
<!doctype html>
<html>
<head>
<script src="c.js"></script>
<script>
var serverurl = "//localhost:4242/",
c = connect(),
ws;
function connect() {
if ("WebSocket" in window) {
ws = new WebSocket("ws:" + serverurl);
ws.binaryType="arraybuffer";
ws.onopen=function(e){
};
ws.onclose=function(e){
};
ws.onmessage=function(e){
sayN(deserialize(e.data));
};
ws.onerror=function(e) {window.alert("WS Error") };
} else alert("WebSockets not supported on your browser.");
}
function toQ(x) { ws.send(serialize({ payload: x })); }
function sayN(n) {
document.getElementById('answer').textContent = n;
}
</script>
</head>
<body>
<h1 style='font-size:200px' id='answer'></h1>
</body>
</html>
And now for the q side. You can enter the following in the console of a fresh q session; or you can save it as a script and load it with \l
.
q)\p 4242
q)answer:42
q).z.po:{`requestor set x; system "t 1000";}
q).z.ts:{neg[requestor] -8!answer;; answer+:1;}
Here is what’s happening in the q code.
- First we open the port and initialize the
answer
variable. - Then we set the connection open handler to store the client
.z.w
value of its parameter into the global requestor and start the system timer firing every 1000 milliseconds. Note that this only happens after the browser initiates a connection. - Finally, we set the timer handler to send an asynchronous message containing the serialized value of answer and then increment answer.
Change since q3.2
“The .z.wo
and .z.wc
message handlers were introduced in kdb+ version 3.3 (2014.11.26) to be evaluated whenever a WebSocket connection is opened (.z.wo
) or closed (.z.wc
). Prior to this version, .z.pc
and .z.po
provide an alternative solution however, these handle the opening and closing of all connections over a port and don’t distinguish WebSocket connections.”
— Whitepaper Kdb+ and WebSockets
See Reference: System and callbacks
Now point the browser to
file:///pages/sample2.html
and you will see the answer ticking every second on the page.