Skip to content

Compressing data on disk

This page explains how to compress KDB-X data on disk: the three compression parameters, how to write and read compressed files, how to convert a database you already have, which data compresses well, and how to choose an algorithm for your own workload.

Compression is applied when a file is written, and undone transparently when it is read. No query changes, and nothing needs to know a file is compressed: q records the algorithm in the file itself and decompresses on access.

You are spending CPU to save disk. Whether that also costs you query time or buys you query time depends mostly on how fast your storage is — on slow storage, reading fewer bytes can more than pay for decompressing them, while on fast storage there is no I/O saving left to win. That makes it a decision to measure rather than assume.

Compression parameters

Compression is specified by three integers, given in this order:

(logicalBlockSize; algorithm; compressionLevel)

logicalBlockSize

The exponent of a power of two, between 12 (4KB) and 20 (1MB), giving the size of the chunks that are compressed independently.

Blocks are compressed and decompressed as units, so the block size sets the smallest amount of data q must decompress to retrieve even a single value: with a block size of 17, reading one element of a column decompresses 128KB. That read amplification is the cost to weigh against whatever the block size gains you in ratio — and since both depend on the data and the access pattern, it is worth measuring on your own data rather than assuming.

The valid minimum depends on the platform that reads the file

A block size below the memory page size is rejected, and the page size varies: 4KB on AMD64, 8KB on SPARC, 16KB on Apple Silicon, and a 64KB allocation granularity on Windows. Use a value valid on every platform that reads the files directly — the minimum across all of them — because the file is rejected at read time on a platform that cannot map it, not at write time.

On Apple Silicon, for instance, 12 is too small:

q)(`:f;12;2;6) set 1000?10
'bad blockSize 4096 for f

Values above 20 are rejected everywhere.

algorithm

Code Algorithm Levels Introduced
0 None 0
1 q IPC 0
2 gzip 0–9
3 snappy 0 V3.4
4 lz4hc 0–16 V3.6
5 zstd -7–22 V4.1

Compression libraries

Every algorithm except 0 and 1 needs an external shared library, which q loads on demand. A missing library is reported when you try to use it:

q)(`:f;17;5;1) set 1000?10
'zstd libs required to compress f. dlopen(libzstd.1.dylib, ...)
Algorithm Linux macOS Windows
gzip libz.so.1 libz.dylib (pre-installed) zlibwapi.dll
snappy libsnappy.so.1 libsnappy.dylib snappy.dll
lz4hc liblz4.so.1 liblz4.dylib liblz4.dll
zstd libzstd.so.1 libzstd.1.dylib libzstd.dll

On macOS the libraries other than gzip come from Homebrew or MacPorts. A 32-bit q needs 32-bit libraries and a 64-bit q needs 64-bit ones.

Building zlib for Windows

The out-of-the-box build files produce a DLL with the cdecl calling convention, which doesn't work in KDB-X. Define the ZLIB_WINAPI preprocessor symbol to get the required stdcall convention. For example, with MinGW:

>make -f win32/Makefile.gcc --eval 'kx-dummy:;@echo $(CFLAGS)'
-O3 -Wall
>make -f win32/Makefile.gcc CFLAGS="-O3 -Wall -DZLIB_WINAPI" SHARED_MODE=1 zlib1.dll

compressionLevel

An algorithm-specific integer trading ratio against compression time; higher is usually smaller and slower to write. Out-of-range values are rejected.

For lz4hc, level 0 means the algorithm's own default, and anything above 16 behaves as 16.

Do not reach for the maximum: the ratio flattens out well before the top of most ranges while write time keeps climbing, and the level makes almost no difference to how fast the data reads back. See choose an algorithm for the level recommended for each.

Write a compressed file

Put the three parameters in the left argument of set, after the file handle:

q)(`:zipped;17;2;6) set 1000000?10
`:zipped

The data is unchanged by the round trip:

q)`:plain set 1000000?10
`:plain
q)(`:zipped;17;2;6) set get `:plain
`:zipped
q)get[`:plain] ~ get `:zipped
1b

Put source and target on different drives

Compressing an existing file reads and writes at the same time. On one physical drive that causes seek contention; on two it doesn't.

Compress a splayed table column by column

Pass a dictionary instead of a single parameter list to vary the settings across the columns of a splayed table. The empty symbol key supplies the default for any column not named:

q)m1: 1000000
q)t: ([] a: m1?10; b: m1?10; c: m1?10; d: m1?10)
q)dict: ``a`b!(17 1 0;17 2 6;17 2 9)
q)(`:compressed_multi/;dict) set t
`:compressed_multi/

Columns a and b are named, so they get gzip at levels 6 and 9; c and d fall back to the default of q IPC compression:

q)-21!`:compressed_multi/b
compressedLength  | 690776
uncompressedLength| 8000016
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 9i
q)-21!`:compressed_multi/c
compressedLength  | 1916149
uncompressedLength| 8000016
algorithm         | 1i
logicalBlockSize  | 17i
zipLevel          | 0i

This is worth doing because algorithms suit different data patterns, and one database can mix several. Reading needs no special handling: each file records what produced it.

Compress every write with .z.zd

Set .z.zd to apply the same three parameters to every subsequent write, rather than repeating them:

q).z.zd: 17 2 6
q)`:auto set 1000000?10
`:auto
q)-21!`:auto
compressedLength  | 744022
uncompressedLength| 8000016
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 6i

It applies only to files written with no extension, so exports are untouched:

q)`:auto.csv set 1000000?10
`:auto.csv
q)-21!`:auto.csv      / empty: not compressed

Two ways to turn it off again — by default .z.zd is undefined and files are written uncompressed:

q).z.zd: 3#0          / zeroes
q)\x .z.zd            / or expunge the variable

Read a compressed file

Reading takes no parameters and no special syntax: every q operation that reads a file reads a compressed one the same way. Only the blocks a query actually touches are decompressed, so a query constrained to a few partitions or columns pays only for those, and the decompressed data is cached for the duration of that operation. It is not retained afterwards, which is why a repeated query keeps paying to decompress even when the compressed bytes are already in the operating system's page cache.

To make an uncompressed copy, read and write it back with no compression parameters:

q)`:plain_again set get `:zipped
`:plain_again

Inspect what a file uses

-21! returns a dictionary of compression statistics, and an empty dictionary for a file that is not compressed — which makes it the test for whether a file is compressed at all:

q)-21!`:zipped
compressedLength  | 744022
uncompressedLength| 8000016
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 6i
q)0=count -21!`:plain
1b

hcount reports the uncompressed length, not the size on disk, so it is not the figure to use for a disk-space estimate:

q)hcount `:zipped
8000016

Append to a compressed file

upsert appends to a compressed file, or to a compressed splayed table, and the file stays compressed:

q)(`:zt;17;2;6) set 100000?10
`:zt
q)`:zt upsert 100000?10
`:zt
q)-21!`:zt
compressedLength  | 148936
uncompressedLength| 1600016
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 6i

Appending to a file with an attribute rewrites it

If the file carries an attribute — for example p# on a symbol column — q reads and rewrites the whole file on append, rather than adding to the end.

Compress an existing database

A database written uncompressed can be converted in place, one column file at a time, by reading each file and writing it back with compression parameters. Given an uncompressed partitioned database under :db:

q)0=count -21!`:db/2026.04.01/trade/price      / not compressed yet
1b

Compress each column file of each table directory, leaving the .d file alone:

q)zipCol:{[par;c] (c, par) set get c;}
q)zipDir:{[par;d] zipCol[par] each d .Q.dd/:key[d] except `.d; d}
q)zipDir[17 2 6] each `:db/2026.04.01/trade`:db/2026.04.02/trade
`:db/2026.04.01/trade`:db/2026.04.02/trade

The columns are compressed, and the table loads and queries exactly as before:

q)-21!`:db/2026.04.01/trade/price
compressedLength  | 1163678
uncompressedLength| 1600016
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 6i
q)\l db
q)select cnt:count i, avgPrice:avg price by date from trade
date      | cnt    avgPrice
----------| ---------------
2026.04.01| 200000 49.89157
2026.04.02| 200000 49.93976

Alternatively set .z.zd and resave the data, which compresses as it writes and avoids the intermediate uncompressed file. That is the better route when you are rebuilding a database anyway, or writing down a new partition for the first time.

Do not compress the sym file

The enum domain — the sym file at the database root — can be compressed, and a database with a compressed sym still loads and queries correctly. But it can no longer be extended, so the next .Q.en that meets a new symbol fails:

q).Q.en[`:.; ([] sym:`BRANDNEW; v:1)]
'no append to zipped enums: ./sym

A whole-database conversion that sweeps up the root therefore breaks the next writedown. Exclude sym, and any other enum domain, from compression.

What compresses well

Compression exploits redundancy, so the ratio is a property of the data, not of the algorithm alone. Three things dominate.

How few distinct values there are. Low cardinality compresses well; values drawn from the full range of the type do not.

Whether equal values sit together. Sorting a column before writing it can change the ratio by orders of magnitude, which is one more reason to apply the parted attribute and sort on the column you filter by. The white paper quantifies the effect on generated data, comparing a two-column table unsorted, sorted on one column, and sorted on both.

How much of the type's width is really used. A boolean uses one byte to hold one bit. A float holding prices to two decimal places leaves most of its mantissa unused. Both reclaim that space; a full-range long or a random float has nothing to give back.

Because the ratio follows the data, it varies enormously between the columns of a single table. The FSI case study measured every column of the NYSE TAQ quote table — 1.78 billion rows, 180GB uncompressed — at several levels of every algorithm, and publishes the result per column as a percentage of the original size. Two of its columns show the range:

  • Symbol is low-cardinality, heavily repeating, and sorted, and effectively disappears: a fraction of one percent under every algorithm.
  • Sequence_Number is a monotonically increasing integer with repeats. gzip takes it to around 41%, while lz4 cannot compress it at all — 100.0%, meaning not one byte saved at any level.

That second row is the important one. On a column like that the choice of algorithm is the difference between a real saving and none whatsoever, which is the strongest argument for compressing column by column rather than applying one setting to a whole database. Consult the case study's table for the column shapes that resemble your own.

A symbol column in a splayed table is stored as its enumeration — an integer vector — so it is the cardinality and ordering of that enumeration, not the symbols themselves, that decide how the column compresses.

Leave poorly-compressing files uncompressed

Because q reads compressed and uncompressed files interchangeably, compression is a per-file decision. There is no benefit in compressing a column that barely shrinks: you pay decompression on every read for almost no space saved. The case study also advises leaving columns that carry the parted attribute — typically sym — uncompressed, since they are the columns queries use to find data.

Choose an algorithm

Three properties matter, and no algorithm is best at all three:

  • Compression ratio — how much disk you save.
  • Compression speed — bounds your ingestion rate and writedown window.
  • Decompression speed — paid by every query that reads the data.

The FSI compression case study compares all of them at many levels on a day of NYSE TAQ data — a 1.78-billion-row quote table and a 76-million-row trade table — across fast NVMe and slow NFS storage. It is the most thorough published comparison, and the guidance below follows it.

On ratio, gzip and zstd deliver the best overall results. zstd beats lz4 and snappy by nearly 2x but is only marginally better than gzip, and q IPC is the worst of the five — it does not even compress every column.

On level, the case study's recurring finding is that high levels rarely pay for themselves: the ratio flattens out while compression time keeps climbing. zstd at level 22 was two orders of magnitude slower to write than uncompressed on some columns for little gain over level 10, and gzip levels 6 to 9 differ barely at all. The counterpart is that level hardly affects decompression speed for any algorithm, so a higher level costs you at write time and buys nothing back at read time beyond the smaller file.

Putting those together:

  • zstd (5) — the ratio leader. Level 10 is the sweet spot for most columns; above it the gain is negligible and the write cost is not. Level 1 is the fastest compressor of any algorithm, so it is the choice when write throughput matters more than size.
  • gzip (2) — ratios close to zstd, and distinctly better than zstd on monotonic integer columns such as sequence numbers, where level 5 is the recommended setting. Level 1 does poorly on low-entropy columns.
  • lz4 / lz4hc (4) — trades disk space for speed. Levels 5 or 6 when query speed is the priority and storage saving only the second concern. It cannot compress monotonic sequence numbers at all.
  • snappy (3) — similar to lz4 on ratio, and the choice when you want migration into the compressed tier to be fast.
  • q IPC (1) — needs no external library, which is its only real advantage; it has the worst ratio of the five.

A tiered strategy

Rather than one setting everywhere, the case study recommends matching compression to how often data is read:

Tier Data Recommendation
Hot Last few weeks, on fast storage No compression — maximal ingestion rate and fastest queries
Second High volume, queried less often For speed, snappy or lz4 at level 5–6. For size, zstd 10 for most columns and gzip 5 for sequence-number-like columns. Leave parted columns such as sym uncompressed.
Cold High volume, rarely read, on cheaper or slower storage Compress; if the second tier used lz4 or snappy, recompress with zstd to reclaim more space.

The same logic applies within a table as well as across one: a heavily queried table might have columns that are almost never read, and those column files can be moved to a slower tier — with symbolic links, so the table still loads as one.

Measure the impact on your own hardware

Published figures are a starting point, not an answer, because the direction of the effect depends on your storage. The case study measured the same queries on both fast and slow storage and found compression cutting in opposite directions:

  • On fast NVMe, compression made large reads 4–5.5x slower, and cached, CPU-bound aggregations as much as 20x slower. There was no I/O saving left to win, so the decompression cost was all that remained.
  • On slow NFS, the same large query ran in 0.2–0.4x the uncompressed time — a speed-up of 2.5–5x — because reading fewer bytes more than paid for decompressing them.

The lesson is that compression is a storage-speed decision before it is a CPU one: the slower the storage relative to the CPU, the more compression gives back. Test before committing a database to a setting.

When you do:

  • Use your own data and queries. Compression ratios follow the data patterns above, and query effects follow your access pattern. Synthetic data misleads you on both.
  • Test on production-equivalent hardware and storage. The gap between NVMe and network storage changes the conclusion, not just the numbers.

  • Flush the page cache between runs, or you measure the cache rather than the storage:

    sync; echo 3 | sudo tee /proc/sys/vm/drop_caches    # Linux
purge                                               # macOS

  • Measure both directions. Time the writedown as well as the queries; a level that halves your storage bill but doubles your writedown window might not fit the schedule.

  • Compare per column. -21! gives the ratio actually achieved for each file, so you can stop compressing the columns that gain nothing.

For the full methodology, and results across block sizes, data types, sort orders, and query shapes, see the compression white paper and the FSI compression case study.

Cautions and limits

One file, one thread

Do not read from or write to the same compressed file from more than one thread at a time. Different compressed files may be accessed concurrently, one thread each — which is what lets a segmented database decompress in parallel across segments.

Never compress log files

Streaming compression holds the last block in memory and writes it only when the handle is closed. If the process dies, a compressed log is missing its end metadata and cannot be replayed — which defeats the purpose of having it.

Some lz4 versions are broken

lz4-1.7.5 fails to compress and lz4-1.8.0 can hang the process. KDB-X needs at least lz4-r129; lz4-1.8.3 is known good, and the latest stable release is the safer choice.

Nested columns compress their companion files automatically

Compressing a nested column also compresses its # and ## companion files. Do not compress those explicitly.

Use set, not the gzip command

External tools produce a different container that q cannot read with random access. Compression has to come from set (or .z.zd) to be usable.

Resource management

File descriptors. Each compressed file uses two descriptors, so a process reading many of them might need a higher limit than an uncompressed database needs (ulimit -n).

Virtual memory. When q reads a compressed vector it reserves enough address space for the whole uncompressed content, however little the query touches, because decompressed pages have no backing file to be evicted to. Reserving address space is not the same as allocating physical memory — but the OS must believe it can swap the data out, so a compressed database needs swap sized for the worst case. wsfull with apparently free memory usually means a ulimit -v cap. On Linux, overcommit_memory and overcommit_ratio govern how much address space the kernel hands out.

Kernel tuning. On Linux, vm.dirty_background_ratio and vm.dirty_ratio can be worth tuning; good values depend on how many compressed files you hold open and whether access is sequential or random.

Next steps