Skip to content

Working with binary files

This page explains how to read and write raw binary data in q: whole files and fixed-width records, appending over several writes, and compressing as you write.

Reading bytes

read1 reads a file as a byte vector. The following example writes and then reads back a 40-byte file of ten 4-byte integers:

q)`:integers.dat 1: raze reverse each 0x0 vs/:`int$til 10
`:integers.dat
q)read1 `:integers.dat
0x00000000010000000200000003000000040000000500000006000000070000000800000009000000

You can use an offset and a length to read a subsection — here from byte 4 for 36 bytes:

q)read1 (`:integers.dat;4;36)
0x010000000200000003000000040000000500000006000000070000000800000009000000

Reading text as bytes

Casting the bytes from read1 to characters gives the file as one string with its line breaks preserved, which is what you want when the breaks themselves matter. For the test.txt used earlier:

q)"c"$read1 `:test.txt
"hello\nworld\n"

Compare with read0, which splits on the line breaks and discards them.

Fixed-width binary records

1: reads binary data as fixed-width records, given the field types, widths, and endianness. For more detail on options, see column types and widths. It repeats the record definition until the file is consumed.

Reading the 10-integer file above as records of two 4-byte integers yields two columns of five:

q)("ii";4 4)1:`:integers.dat
0 2 4 6 8
1 3 5 7 9

An offset and a length apply here too. Starting at byte 4 and reading 8 bytes gives a single record:

q)("ii";4 4)1:(`:integers.dat;4;8)
1
2

This is how you skip a header or footer, where the layout differs from the records that follow.

Binary load is multithreaded

Like the text loaders, 1: reads in parallel when q runs with secondary threads.

Writing bytes

1: with a file handle on the left writes raw bytes. The following example writes the five ASCII bytes that spell hello — 0x68 is h, 0x65 is e, and so on:

q)`:file.txt 1: 0x68656c6c6f
`:file.txt

Any vector is written as its binary representation. This creates a 400-byte file of a hundred 4-byte integers:

q)`:ints.dat 1:"i"$til 100
`:ints.dat

1: writes values only, not metadata

A raw binary write records the bytes of the values and nothing else, so attributes are lost. The KDB-X format is what preserves them.

Writing a plain list and a sorted list produces byte-identical files, as md5 hashes of the results confirm:

q)a: til 10
q)b: `s#til 10
q)`:a 1: a
`:a
q)`:b 1: b
`:b
q)md5 "c"$read1 `:a
0xcdff2051b710c66f3373e26c901ee505
q)md5 "c"$read1 `:b
0xcdff2051b710c66f3373e26c901ee505

Contrast with save, which keeps the attribute — the 16 bytes of file header are the difference.

Appending bytes with hopen

hopen opens a binary file for appending in the same way it opens a text file. Apply the handle to bytes or characters, and hclose it when done:

q)h:hopen `:test.dat
q)h "a"
6i
q)h "bc"
6i
q)h 0x21
6i
q)hclose h

Each write echoes the handle itself, not a byte count. test.dat now holds abc! with no line breaks, 0x21 being the ASCII code for !. The file carries no type information, so get cannot read it as KDB-X data:

q)get `:test.dat
'test.dat
  [0]  get `:test.dat
       ^

Opening a file that does carry a KDB-X header behaves differently: the same handle then appends typed values and the file stays readable — see appending with hopen. Which behavior you get depends on the file, not on how you open it.

Compressing a binary file

Extend the left argument of 1: with compression parameters to compress as you write. -21! reports what was achieved:

q)(`:file;17;2;9) 1:"i"$til 1000000
`:file
q)-21!`:file
compressedLength  | 1383683
uncompressedLength| 4000000
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 9i

Set .z.zd to apply the same parameters to every write, so you don't need to repeat them:

q).z.zd:(17;2;9)
q)`:file 1:"i"$til 1000000
`:file
q)-21!`:file
compressedLength  | 1383683
uncompressedLength| 4000000
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 9i
q)`:another_file 1:til 10000
`:another_file
q)-21!`:another_file
compressedLength  | 15194
uncompressedLength| 80000
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 9i

You don't need parameters to read as a compressed file records the algorithm it used.

Next steps