Skip to content

Files (words)

Files (numbers)

csv

Syntax: csv (constant)

A synonym for "," for use in preparing text for CSV files, or reading them.

 .h.cd, .h.td

dsave

Syntax: x dsave y

Saves global tables to disk as splayed, enumerated, indexed kdb+ tables. (Since V3.2 2014.05.07.)

Where

  • x is the save path as a symbol or symbol list
  • y is a list of table names

returns the list of table names, having saved the table/s as a side effect.

The first column of each table saved has the `p attribute applied to it. If the path is a list, the first item is the HDB root (where the sym file, if any, will be stored), while the remaining ites are a path within the HDB (e.g. a partition).

dsave provides roughly the same functionality as the combination of .Q.en and set or .Q.dpft, but in a simpler form.

q)t:flip`sym`price`size!100?'(-10?`3;1.0;10)
q)q:flip`sym`bid`ask`bsize`asize!900?'(distinct t`sym;1.0;1.0;10;10)
q)meta t
c    | t f a
-----| -----
sym  | s    
price| f    
size | j    
q)meta q    
c    | t f a
-----| -----
sym  | s    
bid  | f    
ask  | f    
bsize| j    
asize| j    
q)type each flip t
sym  | 11
price| 9
size | 7
q)type each flip q
sym  | 11
bid  | 9
ask  | 9
bsize| 7
asize| 7
q)`:/tmp/db1 dsave`sym xasc'`t`q
`t`q
q)\l /tmp/db1
q)meta t
c    | t f a
-----| -----
sym  | s   p
price| f    
size | j    
q)meta q
c    | t f a
-----| -----
sym  | s   p
bid  | f    
ask  | f    
bsize| j    
asize| j    
q)type each flip t
sym  | 20
price| 9
size | 7
q)type each flip q
sym  | 20
bid  | 9
ask  | 9
bsize| 7
asize| 7
q)

In the following, the left argument is a list, of which the second item is a partition name.

q)t:flip`sym`price`size!100?'(-10?`3;1.0;10)
q)q:flip`sym`bid`ask`bsize`asize!900?'(distinct t`sym;1.0;1.0;10;10)
q)meta t
c    | t f a
-----| -----
sym  | s    
price| f    
size | j    
q)meta q
c    | t f a
-----| -----
sym  | s    
bid  | f    
ask  | f    
bsize| j    
asize| j    
q)type each flip t
sym  | 11
price| 9
size | 7
q)type each flip q
sym  | 11
bid  | 9
ask  | 9
bsize| 7
asize| 7
q)`:/tmp/db2`2015.01.01 dsave`sym xasc'`t`q
`t`q
q)\l /tmp/db2
q)meta t
c    | t f a
-----| -----
date | d    
sym  | s   p
price| f    
size | j    
q)meta q
c    | t f a
-----| -----
date | d    
sym  | s   p
bid  | f    
ask  | f    
bsize| j    
asize| j    

set, .Q.en, .Q.dpft, .Q.hdpf

get

Syntax: get x

Reads or memory maps kdb+ data file x. A type error is signalled if the file is not a kdb+ data file.

Used to map columns of databases in and out of memory when querying splayed databases, and can be used to read q log files etc.

q)\l trade.q
q)`:NewTrade set trade                  / save trade data to file
`:NewTrade
q)t:get`:NewTrade                       / t is a copy of the table
q)`:SNewTrade/ set .Q.en[`:.;trade]     / save splayed table
`:SNewTrade/
q)s:get`:SNewTrade/                     / s has columns mapped on demand

get and value

get has several other uses. However, the function value is a synonym for get and by convention is used for other purposes. But the two are completely interchangeable.


q)value "2+3"
5
q)get "2+3"
5

FIXME: describe other uses.

hclose

Syntax: hclose x

Closes file or process handle x.

q)h:hopen `::5001
q)h"til 5"
0 1 2 3 4
q)hclose h
q)h"til 5"
': Bad file descriptor

hcount

Syntax: hcount x

Returns as a long integer the size in bytes of file x.

q)hcount`:c:/q/test.txt
42j

On a compressed file returns the size of the original uncompressed file.

hdel

Syntax: hdel x

Where x is a file or folder symbol, deletes it.

q)hdel`:test.txt   / delete test.txt in current working directory
`:test.txt
q)hdel`:test.txt   / should generate an error
'test.txt: No such file or directory

hdel can delete folders only if empty


q)hdel`:mydir
'​mydir​. OS reports: Directory not empty
  [0]  hdel`:​mydir​

Delete a folder and its contents

To delete a folder and its contents, recursively list the contents and delete in reverse order:


​/diR gets recursive dir listing​
q)diR:{$[11h=type d:key x;raze x,.z.s each` sv/:x,/:d;d]}
​/hide power behind nuke​
q)​nuke:hdel​ ​each​ ​​desc diR​@​ / desc sort!​
​q)nuke`:mydir

For a general visitor pattern with hdel

​q)visitNode:{if[11h=type d:key y;.z.s[x]each` sv/:y,/:d;];x y}
q)nuke:visitNode[hdel]

hopen

Syntax: hopen x

Where x is one of

  • a process handle
  • a 2-item list of a process handle and a timeout in milliseconds
  • a filename

opens a file or a process handle, and returns a positive integer handle.

A process handle has the form:

TCP
`:host:port[:user:password]
host can be a hostname or IP address; omitted, it denotes the localhost
Unix domain socket
`:unix://port[:user:password]
(Since V3.4.) Unix domain sockets can have significantly lower latency and higher throughput than a localhost TCP connection
SSL/TLS
`:tcps://host:port[:user:password]
Cookbook: SSL/TLS

User and password are required if the server session has been started with the -u or -U command line options, and are passed to .z.pw for (optional) additional processing.

The optional timeout applies to the initial connection, not subsequent use of it.

q)h:hopen `:10.43.23.198:5010                    / IP address
q)h:hopen `:mydb.us.com:5010                     / hostname
q)h:hopen `::5010                                / localhost
q)h:hopen 5010                                   / localhost
q)h:hopen `:unix://5010                          / localhost, Unix domain socket
q)h:hopen `:tcps://mydb.us.com:5010              / SSL/TLS with hostname
q)h:hopen (`:mydb.us.com:5010:elmo:sesame;10000) / full arg list, 10s timeout

To send messages to the remote process:

q)h"2+2"          / synchronous (GET)   
4
q)(neg h)"a:2"    / asynchronous (SET)

If only one synchronous query/request is to be run, then the single-shot synchronous request can be used to open a connection, send the query, get the results, then close the connection. It is more efficient to keep a connection open if there is an opportunity to re-use it for other queries.

q)`:mydb.us.com:5010:elmo:sesame "1+1"
2

Cookbook: Client-server, .Q.Xf (create file)

File handles

A file handle is used for writing to a file. The hopen argument is a symbol filename:


q)hdat:hopen `:f.dat             / data file (bytes)
q)htxt:hopen `:c:/q/test.txt     / text file
To append to these files, the syntax is the same as for IPC:

q)r:hdat 0x2324
q)r:htxt "some text\n"
q)r:htxt ` sv("asdf";"qwer")

Fifo/named pipes

V3.4 Unix builds have support for reading from a Fifo/named pipe, where the hopen argument has the form `:fifo://filename.

Cookbook: Named pipes, Client-server, SSL/TLS

hsym

Syntax: hsym x

Converts symbol x into a file name, or valid hostname, or IP address. Since V3.1, x can be a symbol list.

q)hsym`c:/q/test.txt
`:c:/q/test.txt
q)hsym`10.43.23.197
`:10.43.23.197

key

key for

  • contents of a directory
  • existence of a file

load

Syntax: load x

Loads binary data from the filesystem and returns the name of the table loaded.

q)t:([]x: 1 2 3; y: 10 20 30)
q)save`t             / save to a binary file (same as `:t set t)
`:t
q)delete t from `.   / delete t
`.
q)t                  / not found
't

q)load`t             / load from a binary file (same as t:get `:t)
`t
q)t
x y
----
1 10
2 20
3 30

If x is a directory, a dictionary of that name is created and all data files are loaded into that dictionary, keyed by file name.

q)\l sp.q
q)\mkdir -p cb
q)`:cb/p set p
`:cb/p
q)`:cb/s set s
`:cb/s
q)`:cb/sp set sp
`:cb/sp
q)load `cb
`cb
q)key cb
`p`s`sp
q)cb `s
s | name  status city
--| -------------------
s1| smith 20     london
s2| jones 10     paris
s3| blake 30     paris
s4| clark 20     london
s5| adams 30     athens

.Q.dsftg (load process save), .Q.fps (streaming algorithm), .Q.fs (streaming algorithm), .Q.fsn (streaming algorithm)

read0

Syntax: read0 x

Returns data from text file or handle. Where x is

  • a filename, returns the lines of the file as a list of strings. Lines are assumed delimited by either LF or CRLF, and the delimiters are removed.
q)`:test.txt 0:("hello";"goodbye")  / write some text to a file
q)read0`:test.txt
("hello";"goodbye")

q)/ read 500000 lines, chunks of (up to) 100000 at a time
q)d:raze{read0(`:/tmp/data;x;100000)}each 100000*til 5
q)rl:{1">> ";read0 0}
q)rl`
>> xiskso
"xiskso"
  • a list of the form (file; offset; length), returns bytes from file.
q)`:foo 0: enlist "hello world"
q)read0 (`:foo;6;5)
"world"

Starting with V3.4 2016.05.31 read0 allows user to specify how many bytes to read from a Fifo.

q)h:hopen`$":fifo:///etc/redhat-release"
q)read0(h;8)
"Red Hat "
q)read0(h;8)
"Enterpri"

read1

Syntax: read1 x

Returns bytes from a file or named pipe.

Where x has the form (file; offset; length), returns corresponding bytes from file. Where only file is supplied, the content of the entire file is returned.

q)`:test.txt 0:("hello";"goodbye")      / write some text to a file
q)read1`:test.txt                       / read in as bytes
0x68656c6c6f0a676f6f646279650a
q)"c"$read1`:test.txt                   / convert from bytes to char
"hello\ngoodbye\n"

q)/ read 500000 lines, chunks of (up to) 100000 at a time
q)d:raze{read1(`:/tmp/data;x;100000)}each 100000*til 5 

In V3.4 you can specify how many bytes to read from fifo.

q)h:hopen`$":fifo:///etc/redhat-release"
q)"c"$read1(h;8)
"Red Hat "
q)"c"$read1(h;8)
"Enterpri"

rload

Syntax: rload x

Load a splayed table: where x is the table name as a symbol, the table is read from a directory of the same name. rload is the converse of rsave.

The usual, and more general, way of doing this is to use get, which allows a table to be defined with a different name than the source directory.

q)\l sp.q
q)rsave `sp           / save splayed table
`:sp/
q)delete sp from `.
`.
q)sp
'sp
q)rload `sp           / load splayed table
`sp
q)3#sp
s  p  qty
---------
s1 p1 300
s1 p2 200
s1 p3 400
q)sp:get `:sp/        / equivalent to rload `sp

.Q.v (get splayed table)

rsave

Syntax: rsave x

Save a table, splayed to a directory of the same name: where x is the table name as a symbol. The table must be fully enumerated and not keyed.

The usual and more general way of doing this is to use set , which allows the target directory to be given.

q)\l sp.q
q)rsave `sp           / save splayed table
`:sp/
q)\ls sp
,"p"
"qty"
,"s"
q)`:sp/ set sp        / equivalent to rsave `sp
`:sp/

.Q.dpft (save table)

save

Syntax: save x

Save global data to file: where x is the filename as a symbol, returns the filename itself. The file shortname (ignoring path and extension) must match the name of a global table. The format used depends on the file extension:

  • no extension – saved as binary
  • .csv – comma-separated values
  • .txt – plain text
  • .xls – Excel spreadsheet
  • .xml – Extensible Markup Language (XML)
q)t:([]x:2 3 5;y:`ibm`amd`intel;z:"npn")
q)save `t            / binary
`:t
q)read0 `:t
"\377\001b\000c\013\000\003\000\000\000x\000y\000z\000\000\..
"\000\003\000\000\000npn"
q)save `t.csv        / CSV
`:t.csv
q)read0 `:t.csv
"x,y,z"
"2,ibm,n"
"3,amd,p"
"5,intel,n"
q)save `t.txt        / text
`:t.txt
q)read0 `:t.txt      / columns are tab separated
"x\ty\tz"
"2\tibm\tn"
"3\tamd\tp"
"5\tintel\tn"
q)save `t.xls        / Excel
`:t.xls
q)read0 `:t.xls
"<?xml version=\"1.0\"?><?mso-application progid=\"Excel.Sheet\"?>"
"<Workbook xmlns=\"urn:schemas-microsoft-com:office:spreadsheet\" x...
q)save `t.xml       / XML
`:t.xml
q)read0 `:t.xml    / tab separated
"<R>"
"<r><x>2</x><y>ibm</y><z>n</z></r>"
"<r><x>3</x><y>amd</y><z>p</z></r>"
"<r><x>5</x><y>intel</y><z>n</z></r>"
"</R>"

You can specify a path for the file:

q)save `$"/tmp/t"
`:/tmp/t

Saving local data

To save local data you can do explicitly what save is doing under the covers.


q)`:t set t /save in binary format as a single file
q)/ save in binary format as a splayed table 
q)/ (1 file/column, symbols enumerated against the sym file in current dir)
q)`:t/ set .Q.en[`:.;t] 
q)`:t.csv 0:.h.tx[`csv;t] / save in csv format
q)`:t.txt 0:.h.tx[`txt;t] / save in txt format
q)`:t.xml 0:.h.tx[`xml;t] / save in xml format
q)`:t.xls 0:.h.tx[`xls;t] / save in xls format

.Q.Xf (create file)

set

Syntax: x set y

Assigns the value of y to variable name or filename x

q)`a set 1 2 3            / set name a
`a
q)a
1 2 3

q)a:`t
q)a set 1 2 3             / set name t (indirect assignment)
`t
q)t
1 2 3

q)a:"t"
q)a set 1 2 3             / fails, as name must be a symbol
:["type"]

If x is a filename, the values are written to file:

q)`:work.dat set 1 2 3    / write values to file
`:work.dat
q)get `:work.dat
1 2 3

Write a table to a single file:

q)\l sp.q
q)`:mytable.dat set sp
`:mytable.dat
q)get `:mytable.dat
s  p  qty
---------
s1 p1 300
s1 p2 200
s1 p3 400
..

To save a table splayed across a directory, x must be a path (i.e. ends with a /), and the table must be fully enumerated, with no primary keys:

q)`:mydata/ set sp
`:mydata/
q)\ls mydata
,"p"
"qty"
,"s"
q)get `:mydata
s  p  qty
---------
s1 p1 300
s1 p2 200
s1 p3 400
..

Watch out

Avoid setting variables in the .q namespace, as undesired and confusing behaviour can result.