# Compressing data on disk _This page explains how to compress KDB-X data on disk: the three compression parameters, how to write and read compressed files, how to convert a database you already have, which data compresses well, and how to choose an algorithm for your own workload._ Compression is applied when a file is **written**, and undone transparently when it is **read**. No query changes, and nothing needs to know a file is compressed: q records the algorithm in the file itself and decompresses on access. You are spending CPU to save disk. Whether that also costs you query time or _buys_ you query time depends mostly on how fast your storage is — on slow storage, reading fewer bytes can more than pay for decompressing them, while on fast storage there is no I/O saving left to win. That makes it a decision to [measure](#measure-the-impact-on-your-own-hardware) rather than assume. ## Compression parameters Compression is specified by three integers, given in this order: ```txt (logicalBlockSize; algorithm; compressionLevel) ``` ### `logicalBlockSize` The exponent of a power of two, between 12 (4KB) and 20 (1MB), giving the size of the chunks that are compressed independently. Blocks are compressed and decompressed as units, so the block size sets the smallest amount of data q must decompress to retrieve even a single value: with a block size of 17, reading one element of a column decompresses 128KB. That read amplification is the cost to weigh against whatever the block size gains you in ratio — and since both depend on the data and the access pattern, it is worth [measuring on your own data](#measure-the-impact-on-your-own-hardware) rather than assuming. > **Warning — The valid minimum depends on the platform that reads the file** > > > A block size below the memory page size is rejected, and the page size varies: 4KB on AMD64, 8KB on SPARC, 16KB on Apple Silicon, and a 64KB allocation granularity on Windows. Use a value valid on **every** platform that reads the files directly — the minimum across all of them — because the file is rejected at read time on a platform that cannot map it, not at write time. > > On Apple Silicon, for instance, 12 is too small: > > ```q > q)(`:f;12;2;6) set 1000?10 > 'bad blockSize 4096 for f > ``` > > Values above 20 are rejected everywhere. > ### `algorithm` | Code | Algorithm | Levels | Introduced | |------|-----------|-----------|------------| | `0` | None | `0` | | | `1` | q IPC | `0` | | | `2` | `gzip` | `0`–`9` | | | `3` | `snappy` | `0` | V3.4 | | `4` | `lz4hc` | `0`–`16` | V3.6 | | `5` | `zstd` | `-7`–`22` | V4.1 | #### Compression libraries Every algorithm except `0` and `1` needs an external shared library, which q loads on demand. A missing library is reported when you try to use it: ```q q)(`:f;17;5;1) set 1000?10 'zstd libs required to compress f. dlopen(libzstd.1.dylib, ...) ``` | Algorithm | Linux | macOS | Windows | |-----------|------------------|------------------------------|----------------| | `gzip` | `libz.so.1` | `libz.dylib` (pre-installed) | `zlibwapi.dll` | | `snappy` | `libsnappy.so.1` | `libsnappy.dylib` | `snappy.dll` | | `lz4hc` | `liblz4.so.1` | `liblz4.dylib` | `liblz4.dll` | | `zstd` | `libzstd.so.1` | `libzstd.1.dylib` | `libzstd.dll` | On macOS the libraries other than `gzip` come from [Homebrew](https://brew.sh/) or [MacPorts](https://www.macports.org/). A 32-bit q needs 32-bit libraries and a 64-bit q needs 64-bit ones. ??? note "Building zlib for Windows" The out-of-the-box build files produce a DLL with the `cdecl` calling convention, which doesn't work in KDB-X. Define the `ZLIB_WINAPI` preprocessor symbol to get the required `stdcall` convention. For example, with MinGW: ```cmd >make -f win32/Makefile.gcc --eval 'kx-dummy:;@echo $(CFLAGS)' -O3 -Wall >make -f win32/Makefile.gcc CFLAGS="-O3 -Wall -DZLIB_WINAPI" SHARED_MODE=1 zlib1.dll ``` ### `compressionLevel` An algorithm-specific integer trading ratio against compression time; higher is usually smaller and slower to write. Out-of-range values are rejected. For `lz4hc`, level `0` means the algorithm's own default, and anything above 16 behaves as 16. Do not reach for the maximum: the ratio flattens out well before the top of most ranges while write time keeps climbing, and the level makes almost no difference to how fast the data reads back. See [choose an algorithm](#choose-an-algorithm) for the level recommended for each. ## Write a compressed file Put the three parameters in the left argument of [`set`](../../ref/get.md#set), after the file handle: ```q q)(`:zipped;17;2;6) set 1000000?10 `:zipped ``` The data is unchanged by the round trip: ```q q)`:plain set 1000000?10 `:plain q)(`:zipped;17;2;6) set get `:plain `:zipped q)get[`:plain] ~ get `:zipped 1b ``` > **Tip — Put source and target on different drives** > > > Compressing an existing file reads and writes at the same time. On one physical drive that causes seek contention; on two it doesn't. > ### Compress a splayed table column by column Pass a dictionary instead of a single parameter list to vary the settings across the columns of a [splayed table](splayed-tables.md). The empty symbol key supplies the default for any column not named: ```q q)m1: 1000000 q)t: ([] a: m1?10; b: m1?10; c: m1?10; d: m1?10) q)dict: ``a`b!(17 1 0;17 2 6;17 2 9) q)(`:compressed_multi/;dict) set t `:compressed_multi/ ``` Columns `a` and `b` are named, so they get gzip at levels 6 and 9; `c` and `d` fall back to the default of q IPC compression: ```q q)-21!`:compressed_multi/b compressedLength | 690776 uncompressedLength| 8000016 algorithm | 2i logicalBlockSize | 17i zipLevel | 9i q)-21!`:compressed_multi/c compressedLength | 1916149 uncompressedLength| 8000016 algorithm | 1i logicalBlockSize | 17i zipLevel | 0i ``` This is worth doing because algorithms suit different data patterns, and one database can mix several. Reading needs no special handling: each file records what produced it. ### Compress every write with `.z.zd` Set [`.z.zd`](../../ref/dotz.md#zzd-compressionencryption-defaults) to apply the same three parameters to every subsequent write, rather than repeating them: ```q q).z.zd: 17 2 6 q)`:auto set 1000000?10 `:auto q)-21!`:auto compressedLength | 744022 uncompressedLength| 8000016 algorithm | 2i logicalBlockSize | 17i zipLevel | 6i ``` It applies only to files written with no extension, so exports are untouched: ```q q)`:auto.csv set 1000000?10 `:auto.csv q)-21!`:auto.csv / empty: not compressed ``` Two ways to turn it off again — by default `.z.zd` is undefined and files are written uncompressed: ```q q).z.zd: 3#0 / zeroes q)\x .z.zd / or expunge the variable ``` ## Read a compressed file Reading takes no parameters and no special syntax: every q operation that reads a file reads a compressed one the same way. Only the blocks a query actually touches are decompressed, so a query constrained to a few partitions or columns pays only for those, and the decompressed data is cached for the duration of that operation. It is not retained afterwards, which is why a repeated query keeps paying to decompress even when the compressed bytes are already in the operating system's page cache. To make an uncompressed copy, read and write it back with no compression parameters: ```q q)`:plain_again set get `:zipped `:plain_again ``` ### Inspect what a file uses [`-21!`](../../ref/internal.md#-21x-compressionencryption-stats) returns a dictionary of compression statistics, and an **empty** dictionary for a file that is not compressed — which makes it the test for whether a file is compressed at all: ```q q)-21!`:zipped compressedLength | 744022 uncompressedLength| 8000016 algorithm | 2i logicalBlockSize | 17i zipLevel | 6i q)0=count -21!`:plain 1b ``` [`hcount`](../../ref/hcount.md) reports the **uncompressed** length, not the size on disk, so it is not the figure to use for a disk-space estimate: ```q q)hcount `:zipped 8000016 ``` ## Append to a compressed file [`upsert`](../../ref/upsert.md) appends to a compressed file, or to a compressed splayed table, and the file stays compressed: ```q q)(`:zt;17;2;6) set 100000?10 `:zt q)`:zt upsert 100000?10 `:zt q)-21!`:zt compressedLength | 148936 uncompressedLength| 1600016 algorithm | 2i logicalBlockSize | 17i zipLevel | 6i ``` > **Note — Appending to a file with an attribute rewrites it** > > > If the file carries an attribute — for example [`p#`](../../ref/set-attribute.md#grouped-and-parted) on a symbol column — q reads and rewrites the whole file on append, rather than adding to the end. > ## Compress an existing database A database written uncompressed can be converted in place, one column file at a time, by reading each file and writing it back with compression parameters. Given an uncompressed partitioned database under `:db`: ```q q)0=count -21!`:db/2026.04.01/trade/price / not compressed yet 1b ``` Compress each column file of each table directory, leaving the `.d` file alone: ```q q)zipCol:{[par;c] (c, par) set get c;} q)zipDir:{[par;d] zipCol[par] each d .Q.dd/:key[d] except `.d; d} q)zipDir[17 2 6] each `:db/2026.04.01/trade`:db/2026.04.02/trade `:db/2026.04.01/trade`:db/2026.04.02/trade ``` The columns are compressed, and the table loads and queries exactly as before: ```q q)-21!`:db/2026.04.01/trade/price compressedLength | 1163678 uncompressedLength| 1600016 algorithm | 2i logicalBlockSize | 17i zipLevel | 6i q)\l db q)select cnt:count i, avgPrice:avg price by date from trade date | cnt avgPrice ----------| --------------- 2026.04.01| 200000 49.89157 2026.04.02| 200000 49.93976 ``` Alternatively set [`.z.zd`](#compress-every-write-with-zzd) and resave the data, which compresses as it writes and avoids the intermediate uncompressed file. That is the better route when you are rebuilding a database anyway, or writing down a new partition for the first time. > **Danger — Do not compress the sym file** > > > The enum domain — the `sym` file at the database root — *can* be compressed, and a database with a compressed `sym` still loads and queries correctly. But it can no longer be **extended**, so the next [`.Q.en`](../../ref/dotq.md#en-enumerate-varchar-cols) that meets a new symbol fails: > > ```q > q).Q.en[`:.; ([] sym: enlist`BRANDNEW; v: 1)] > 'no append to zipped enums: ./sym > ``` > > This restriction lifts if the file is also [encrypted](dare.md): a `sym` file that is both compressed and encrypted can still be extended by `.Q.en`. > > A whole-database conversion that sweeps up the root therefore breaks the next writedown. Exclude `sym`, and any other enum domain, from compression. > ## What compresses well Compression exploits redundancy, so the ratio is a property of the data, not of the algorithm alone. Three things dominate. **How few distinct values there are.** Low cardinality compresses well; values drawn from the full range of the type do not. **Whether equal values sit together.** Sorting a column before writing it can change the ratio by orders of magnitude, which is one more reason to apply the [parted attribute](../../ref/set-attribute.md#grouped-and-parted) and sort on the column you filter by. The [white paper](https://code.kx.com/q/wp/compress/) quantifies the effect on generated data, comparing a two-column table unsorted, sorted on one column, and sorted on both. **How much of the type's width is really used.** A boolean uses one byte to hold one bit. A float holding prices to two decimal places leaves most of its mantissa unused. Both reclaim that space; a full-range long or a random float has nothing to give back. Because the ratio follows the data, it varies enormously between the columns of a single table. The [FSI case study](https://code.kx.com/q/kb/compression/fsicasestudy/) measured every column of the NYSE TAQ quote table — 1.78 billion rows, 180GB uncompressed — at several levels of every algorithm, and publishes the result per column as a percentage of the original size. Two of its columns show the range: - `Symbol` is low-cardinality, heavily repeating, and sorted, and effectively disappears: a fraction of one percent under every algorithm. - `Sequence_Number` is a monotonically increasing integer with repeats. `gzip` takes it to around 41%, while `lz4` cannot compress it **at all** — 100.0%, meaning not one byte saved at any level. That second row is the important one. On a column like that the choice of algorithm is the difference between a real saving and none whatsoever, which is the strongest argument for [compressing column by column](#compress-a-splayed-table-column-by-column) rather than applying one setting to a whole database. Consult the case study's table for the column shapes that resemble your own. A symbol column in a splayed table is stored as its enumeration — an integer vector — so it is the cardinality and ordering of that enumeration, not the symbols themselves, that decide how the column compresses. > **Tip — Leave poorly-compressing files uncompressed** > > > Because q reads compressed and uncompressed files interchangeably, compression is a per-file decision. There is no benefit in compressing a column that barely shrinks: you pay decompression on every read for almost no space saved. The case study also advises leaving columns that carry the [parted attribute](../../ref/set-attribute.md#grouped-and-parted) — typically `sym` — uncompressed, since they are the columns queries use to find data. > ## Choose an algorithm Three properties matter, and no algorithm is best at all three: - **Compression ratio** — how much disk you save. - **Compression speed** — bounds your ingestion rate and writedown window. - **Decompression speed** — paid by every query that reads the data. The [FSI compression case study](https://code.kx.com/q/kb/compression/fsicasestudy/) compares all of them at many levels on a day of NYSE TAQ data — a 1.78-billion-row quote table and a 76-million-row trade table — across fast NVMe and slow NFS storage. It is the most thorough published comparison, and the guidance below follows it. On **ratio**, `gzip` and `zstd` deliver the best overall results. `zstd` beats `lz4` and `snappy` by nearly 2x but is only marginally better than `gzip`, and q IPC is the worst of the five — it does not even compress every column. On **level**, the case study's recurring finding is that high levels rarely pay for themselves: the ratio flattens out while compression time keeps climbing. `zstd` at level 22 was two orders of magnitude slower to write than uncompressed on some columns for little gain over level 10, and `gzip` levels 6 to 9 differ barely at all. The counterpart is that **level hardly affects decompression speed** for any algorithm, so a higher level costs you at write time and buys nothing back at read time beyond the smaller file. Putting those together: - **`zstd` (`5`)** — the ratio leader. **Level 10** is the sweet spot for most columns; above it the gain is negligible and the write cost is not. Level 1 is the fastest compressor of any algorithm, so it is the choice when write throughput matters more than size. - **`gzip` (`2`)** — ratios close to `zstd`, and distinctly better than `zstd` on monotonic integer columns such as sequence numbers, where **level 5** is the recommended setting. Level 1 does poorly on low-entropy columns. - **`lz4` / `lz4hc` (`4`)** — trades disk space for speed. Levels **5 or 6** when query speed is the priority and storage saving only the second concern. It cannot compress monotonic sequence numbers at all. - **`snappy` (`3`)** — similar to `lz4` on ratio, and the choice when you want migration _into_ the compressed tier to be fast. - **q IPC (`1`)** — needs no external library, which is its only real advantage; it has the worst ratio of the five. ### A tiered strategy Rather than one setting everywhere, the case study recommends matching compression to how often data is read: | Tier | Data | Recommendation | |--------|--------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Hot | Last few weeks, on fast storage | **No compression** — maximal ingestion rate and fastest queries | | Second | High volume, queried less often | For speed, `snappy` or `lz4` at level 5–6. For size, `zstd` 10 for most columns and `gzip` 5 for sequence-number-like columns. Leave parted columns such as `sym` uncompressed. | | Cold | High volume, rarely read, on cheaper or slower storage | Compress; if the second tier used `lz4` or `snappy`, recompress with `zstd` to reclaim more space. | The same logic applies within a table as well as across one: a heavily queried table might have columns that are almost never read, and those column files can be moved to a slower tier — with symbolic links, so the table still loads as one. ## Measure the impact on your own hardware Published figures are a starting point, not an answer, because the direction of the effect depends on your storage. The case study measured the same queries on both fast and slow storage and found compression cutting in opposite directions: - On **fast NVMe**, compression made large reads **4–5.5x slower**, and cached, CPU-bound aggregations as much as **20x slower**. There was no I/O saving left to win, so the decompression cost was all that remained. - On **slow NFS**, the same large query ran in **0.2–0.4x** the uncompressed time — a speed-up of 2.5–5x — because reading fewer bytes more than paid for decompressing them. The lesson is that compression is a storage-speed decision before it is a CPU one: the slower the storage relative to the CPU, the more compression gives back. Test before committing a database to a setting. When you do: - **Use your own data and queries.** Compression ratios follow the data patterns above, and query effects follow your access pattern. Synthetic data misleads you on both. - **Test on production-equivalent hardware and storage.** The gap between NVMe and network storage changes the conclusion, not just the numbers. - **Flush the page cache between runs**, or you measure the cache rather than the storage: ```bash sync; echo 3 | sudo tee /proc/sys/vm/drop_caches # Linux purge # macOS ``` - **Measure both directions.** Time the writedown as well as the queries; a level that halves your storage bill but doubles your writedown window might not fit the schedule. - **Compare per column.** [`-21!`](#inspect-what-a-file-uses) gives the ratio actually achieved for each file, so you can stop compressing the columns that gain nothing. For the full methodology, and results across block sizes, data types, sort orders, and query shapes, see the [compression white paper](https://code.kx.com/q/wp/compress/) and the [FSI compression case study](https://code.kx.com/q/kb/compression/fsicasestudy/). ## Cautions and limits > **Warning — One file, one thread** > > > Do not read from or write to the same compressed file from more than one thread at a time. Different compressed files may be accessed concurrently, one thread each — which is what lets a [segmented database](segment.md) decompress in parallel across segments. > > **Warning — Never compress log files** > > > Streaming compression holds the last block in memory and writes it only when the handle is closed. If the process dies, a compressed log is missing its end metadata and cannot be replayed — which defeats the purpose of having it. > > **Warning — Some `lz4` versions are broken** > > > `lz4-1.7.5` fails to compress and `lz4-1.8.0` can hang the process. KDB-X needs at least `lz4-r129`; `lz4-1.8.3` is known good, and the latest stable release is the safer choice. > > **Note — Nested columns compress their companion files automatically** > > > Compressing a nested column also compresses its `#` and `##` [companion files](../basics/work_with_files/kdb-files.md#mapped-lists). Do not compress those explicitly. > > **Note — Use `set`, not the `gzip` command** > > > External tools produce a different container that q cannot read with random access. Compression has to come from `set` (or `.z.zd`) to be usable. > ## Resource management **File descriptors.** Each compressed file uses two descriptors, so a process reading many of them might need a higher limit than an uncompressed database needs (`ulimit -n`). **Virtual memory.** When q reads a compressed vector it reserves enough address space for the whole uncompressed content, however little the query touches, because decompressed pages have no backing file to be evicted to. Reserving address space is not the same as allocating physical memory — but the OS must believe it can swap the data out, so a compressed database needs swap sized for the worst case. `wsfull` with apparently free memory usually means a `ulimit -v` cap. On Linux, `overcommit_memory` and `overcommit_ratio` govern how much address space the kernel hands out. **Kernel tuning.** On Linux, `vm.dirty_background_ratio` and `vm.dirty_ratio` can be worth tuning; good values depend on how many compressed files you hold open and whether access is sequential or random. ## Next steps - Understand what is being compressed in [the KDB-X file format](../basics/work_with_files/kdb-files.md). - See how compressed files are mapped and held in memory in [loading a database](loading-db.md#what-l-maps). - Combine compression with [encryption at rest](dare.md), which takes the same three parameters. - Read the [FSI compression case study](https://code.kx.com/q/kb/compression/fsicasestudy/).