Parquet for q developers¶

This page compares Parquet with the kdb+ partitioned database format, for q developers who are new to Parquet.

Both formats store tables column by column, so a query reads only the columns it references. The kdb+ format is proprietary and optimized for speed in q. Parquet is an open format with wide adoption, and it's the de facto standard for columnar data, so many engines and tools can read the same files.

The formats also differ in how they lay out files, partition data, encode values, represent symbols, append new rows, overwrite existing values, and avoid reading data that a query doesn't need.

This page compares the storage formats, not particular Parquet libraries. Parquet features are often optional, and reader support varies substantially.

The following table summarizes the main differences:

Aspect	kdb+ partitioned database	Parquet
Table storage	In each partition, a directory with one file for each column and a `.d` file for the column order. A database with symbol columns also has an enumeration (`sym`) file	A dataset typically has one or more self-contained files. Each file holds all columns for a slice of rows
Schema	Implied by the `.d` file and the column files in each partition	Stored in the file footer, so each file describes itself
Partition levels	One level, named after the partition value, such as `2026.10.26`	Not part of the file format. Dataset conventions such as Hive partitioning allow nested levels, such as `date=2026-10-26/exch=N`
Partition column	Limited to `date`, `month`, `year`, or `int`	Determined by the dataset convention and reader. Hive layouts commonly partition by one or more scalar columns
Minimizing I/O	Attributes (indexes): `s#, `u#, `p#, and `g#	Optional statistics, page indexes, dictionary encoding, and Bloom filters. Reader support varies
Append	Rows can be appended to a splayed table	A dataset is normally extended with a new file. Extending an existing file requires replacing its footer and isn't widely supported
Overwrite	Eligible values can be overwritten in place; otherwise, only the affected column file is rewritten	A value can't be overwritten in place. The affected Parquet file must be replaced
Schema evolution	Applied to every partition, for example with the DBmaint module	One schema for each file, reconciled by many readers; renaming or casting a column rewrites the files
Memory mapping	Uncompressed column files can be mapped and used directly in q's native representation	A reader can memory-map a Parquet file for I/O, but encoded or compressed pages must be decoded into an in-memory representation
Compression	Optional, set for each column file	Set for each column chunk and usually enabled
Nulls	In-band null values, such as `0N`	Stored separately from the values
Symbols	Enumerated against a `sym` file	Strings, often dictionary-encoded within each column chunk
Timestamps	Nanoseconds from January 1, 2000, with no timezone metadata	Milliseconds, microseconds, or nanoseconds from the Unix epoch, with a flag that distinguishes UTC-adjusted instants from local date-times
Nested and complex types	A cell can hold any q object, such as a list, a dictionary, or a table. Nested data can stay memory-mapped	Nested schema groups, including `LIST` and `MAP`. Readers reassemble nested values in memory
Linked columns	Supported on disk: a column of row indexes into another table	No format-level equivalent. Related data is typically joined from another table or denormalized into the same table

Table storage¶

In a kdb+ database, a table on disk is a splayed table: a directory with one file for each column, and a .d file that lists the column order. Symbol columns are enumerated against a sym file in the parent directory:

sym
trade/
├── .d
├── sym
├── time
├── price
└── size

A Parquet file holds a whole table, or a slice of rows from it:

trade-0.parquet
trade-1.parquet

Each file is self-contained. It starts with the magic number PAR1, and ends with a footer that holds the schema, the location of every column chunk, and any available column-chunk statistics. A reader reads the footer first. Between the magic number and the footer, the file splits the table in two directions:

Row groups. A row group is a horizontal slice of the table: a contiguous range of rows.
Column chunks. Inside a row group, each column is stored as a column chunk, a vertical slice of that row group. Each column chunk is split into pages, which hold the encoded and compressed values.

A kdb+ splayed table splits only vertically, into column files. A Parquet file splits horizontally into row groups first, and then vertically into column chunks.

Data structures beyond tables

A Parquet file holds a single table. A kdb+ database can also persist standalone q data structures, such as lists and dictionaries. A well-chosen structure can replace a join, and so make a query shorter and easier to read.

For example, the capmkts module of datagen can write a partitioned trade table together with a dictionary, exnames, that maps each exchange code to the exchange name:

q)dg:use`kx.datagen.capmkts
q)dg.buildPersistedDB["db"]
q)\l db
q)3#exnames
A| "NYSE American"
B| "NASDAQ OMX BX"
C| "NYSE National"

Indexing the dictionary with the ex column translates the codes inside the by clause, so the query needs no join:

q)5#asc select sum size by exch:exnames ex from trade
exch                     | size
-------------------------| ------
"New York Stock Exchange"| 190131
"NASDAQ OMX PSX"         | 204774
"NYSE National"          | 207829
"Cboe EDGA Exchange"     | 216944
"NASDAQ OMX BX"          | 227621

In a Parquet dataset, the typical normalized representation puts the exchange names in a second table, which a query joins to trade. Alternatively, the exchange name can be denormalized into every trade row. Denormalization avoids the join, but duplicates the reference data across files and potentially across tables. Changing an exchange name then requires updating every copy, and missed updates can leave the dataset inconsistent.

Partitioning¶

A kdb+ partitioned database has one directory for each partition value, with a splayed table in each. The partition value, such as a date, is the directory name, and it appears in queries as a virtual column. The sym file at the root holds the symbol enumeration for all partitions:

db/
├── sym
├── 2026.10.26/
│   └── trade/
│       ├── .d
│       ├── sym
│       └── ...
└── 2026.10.27/
    └── trade/
        └── ...

The Parquet format itself has no notion of partitions. Writers such as Apache Spark use Hive partitioning: they split a dataset into many files and arrange them in directories named key=value, one level for each partition column:

trade/
├── date=2026-10-26/
│   ├── part-00000.parquet
│   └── part-00001.parquet
└── date=2026-10-27/
    └── part-00000.parquet

As in kdb+, the partition values come from the directory names, not from the files. Unlike kdb+, Hive partitioning can nest several partition columns, such as year=2026/month=10/, and a partition can hold several files. The types inferred from directory names, and even whether partition discovery happens automatically, depend on the reader.

Symbols and strings¶

In a kdb+ database, a symbol column stores integer indexes into the sym file. See Managing sym files.

Parquet has no symbol type and no shared enumeration across files. Text columns are strings, and writers often use dictionary encoding within each column chunk to store repeated values compactly. The kx.pq module reads them as strings (type C), not symbols:

q).pq:use`kx.pq
q)t:.pq.pq`:trade.parquet
q)select from t where i<2
sym   time                          price  size stop cond  ex
---------------------------------------------------------------
"CUZ" 2026.10.26D09:30:00.011627745 29.06  65   0    ,0x42 ,"L"
"PRU" 2026.10.26D09:30:00.153776659 103.02 20   0    ,0x42 ,"J"

Types and encoding¶

The column files of a kdb+ database store values in q's in-memory layout. q can map an uncompressed column file and use it directly, with no decoding.

Parquet stores each column chunk as pages that are encoded, for example with dictionary or run-length encoding, and usually compressed. A reader decodes the pages into its own in-memory representation when a query needs them.

Other type differences to keep in mind:

Timestamps. The epoch, precision, and timezone semantics differ. See Timestamps.
Nulls. q represents a null as a special value of the column type, such as 0N for a long. Parquet stores nulls separately from the values, so any value of the type is valid data.

Timestamps¶

A q timestamp is a count of nanoseconds from January 1, 2000. It has no timezone metadata: the value alone doesn't say whether it represents UTC, local time in a particular timezone, or an unspecified wall-clock time.

A Parquet TIMESTAMP is a count from the Unix epoch, January 1, 1970, in milliseconds, microseconds, or nanoseconds. Its isAdjustedToUTC flag defines how to interpret that count:

true: an instant. The value is normalized to UTC and identifies one point on the timeline. The original timezone isn't stored, so a reader can display the instant in UTC or convert it to a chosen timezone, but can't recover the writer's original timezone.
false: a local date-time. The value represents fields such as 2026-10-28 09:30:00 without identifying a timezone or a unique instant. Interpreting it as UTC changes its meaning.

Converting either form to a q timestamp necessarily drops the distinction, because q has no corresponding timezone flag. A reader or application must therefore choose and document a policy: preserve the UTC instant, preserve the local clock fields, or apply a known timezone. This is especially important around daylight-saving transitions, when a local time can be ambiguous or nonexistent.

Nested and complex types¶

Parquet represents nested data with schema groups. It defines standard annotations for LIST and MAP; an unannotated group can represent a struct or record. Parquet stores nested data as flat leaf columns, with definition and repetition levels that record how the values nest. A reader reassembles the nested values from these leaf columns into memory.

In q, a single table cell can hold any q object, such as a list, a dictionary, or even a table. A splayed table stores a nested column in a column file and a # companion file that holds the flat bytes of every nested vector. A ## file holds the enumeration domain when the data contains symbols:

q)quotes: ([]sym:`:sym?`AAPL`MSFT;bids:(270.1 270.0 269.9;541.2 541.1);book:(`lvl`px!(1 2;270.1 270.0);`lvl`px!(1 2;541.2 541.1)))
q)quotes
sym  bids            book
----------------------------------------------
AAPL 270.1 270 269.9 `lvl`px!(1 2;270.1 270)
MSFT 541.2 541.1     `lvl`px!(1 2;541.2 541.1)
q)`:quotes/ set quotes
`:quotes/
q)key`:quotes
`s#`.d`bids`bids#`book`book#`book##`sym
q)type get`:quotes/bids
77h

Type 77h is a mapped list (anymap). Its inner vectors stay mapped on disk, so q can use them without copying them to the heap. Popular Parquet readers typically decode and reassemble nested values in memory, even when they memory-map the file, so they have no equivalent to this.

Compression¶

Both formats support compression, and each supports several algorithms:

Format	Compression	Algorithms
kdb+	Optional, set for each file	`gzip`, `snappy`, `zstd`, `lz4`, `qIPC`
Parquet	A codec is set for each column chunk; uncompressed chunks are allowed	`uncompressed`, `snappy`, `gzip`, `zstd`, `lz4_raw`, `brotli`, and legacy `lzo` and `lz4`

For kdb+, see Compressing data on disk. For Parquet, see Compression.

Uncompressed kdb+ data¶

Compression is optional in kdb+. Without it, q memory-maps the column files. When a query reads a block, the operating system loads it into the page cache. The next query that needs the same block gets it straight from the page cache, with no extra work, so repeated queries are fast.

Compressed data¶

A compressed block can't be used in place, so q must decompress it into memory before it can use it. Decompression costs CPU time, on top of the memory for the decompressed data. q doesn't keep the decompressed data after the query. The second query can't reuse the work of the first one, and decompresses the same data again. This is true even when the compressed bytes are in the page cache.

Most Parquet files are compressed, so a reader must decompress a page before it can use the values. Reading the file through the page cache, or memory-mapping it, can avoid reading the compressed bytes from storage again, but doesn't make the encoded page directly usable. Whether a reader caches the decoded values between queries is an implementation choice. In this respect, Parquet is more similar to compressed kdb+ data than to uncompressed kdb+ data.

Block size and page size¶

Both formats compress data in independent units. In kdb+, the unit is the logical block size, a power of two from 4 KB to 1 MB that you set when you write the file. In Parquet, the unit is the page, and the writer sets the target page size.

The two settings capture the same concept: the amount of data that is compressed and decompressed together. A larger unit usually gives a better compression ratio. The trade-off is read amplification. Even if a query needs a single value, q or a Parquet reader must decompress the entire block or page that holds it. Decompressing a large block takes CPU time and memory, so it has a negative impact on query performance. A smaller unit decompresses less data to find a value, but compresses less well.

Uncompressed kdb+ data doesn't have this negative impact. q maps the column file and reads only the values that the query needs, without decompressing anything.

Minimizing I/O¶

Both formats include structures that reduce the amount of data a query reads. In database terminology, kdb+ attributes are indexes. Parquet has no indexes. It relies on metadata that describes the data instead.

kdb+ attributes¶

Attributes tell q how a column is organized, so that q can choose a faster algorithm:

`s# (sorted) lets q use binary search.
`u# (unique) and `g# (grouped) build hash indexes.
`p# (parted) records where each run of identical values starts, typically on sym in a partition.

Attributes help q find matching rows inside the data it reads. At a higher level, a query that filters on the partition column, such as date, reads only the matching partitions.

Parquet metadata¶

Parquet has no attributes or database indexes. Instead, it can store structures that let a reader decide which parts it can skip:

Statistics. Optional minimum and maximum values, and a count of nulls, for each column chunk.
Page indexes. Optional minimum and maximum values for individual pages, together with their locations, so a reader can skip pages without first reading their headers.
Bloom filters. An optional probabilistic structure for each column chunk. A reader can check whether a value might be in the chunk, which suits equality filters on columns with many distinct values, such as IDs.
Dictionary pages. If every data page in a column chunk uses dictionary encoding, the dictionary lists every distinct value in the chunk. Writers can fall back to another encoding, in which case the dictionary alone can't exclude the values in those pages.
Sort order. Writers can record the columns that each row group is sorted on.

These structures can tell a reader which row groups or pages it can skip. They are optional, and support for using them varies substantially between readers. Sort-order metadata is most useful together with statistics or a page index; by itself, it doesn't give the value boundaries needed for pruning. Unlike `s# or `g#, these structures don't help find rows inside data that the reader does read.

Append data¶

kdb+¶

Appending rows to a kdb+ table needs little memory and little code. upsert adds the new rows to the end of each column file of a splayed table, so q doesn't need to read the existing data into memory:

q)`:trade/ set .Q.en[`:.;] ([]sym:`AAPL`MSFT;price:270.1 541.2;size:100 200)
`:trade/
q)`:trade/ upsert .Q.en[`:.;] ([]sym:`GOOG`AMZN;price:310.5 226.4;size:50 75)
`:trade/
q)select from get`:trade/
sym  price size
---------------
AAPL 270.1 100
MSFT 541.2 200
GOOG 310.5 50
AMZN 226.4 75

The only caveat is that upsert isn't atomic, because a splayed table has a file for each column. See Atomicity, integrity, and durability.

Parquet¶

The footer at the end of a Parquet file records the schema and the metadata for every row group and column chunk. A writer can't simply add bytes after the footer, because the footer length and PAR1 magic bytes must remain at the end of the file.

To extend an existing file, a writer has two broad options:

Append the rows to the last row group. The writer must read the last row group into memory, add the rows, and write it again. This is slow and needs a lot of memory.
Add a new row group. Preserve the existing column chunks, replace the old footer with the new column chunks, and then write a new footer:
1. Read and parse the footer. Read the end of the file to find the existing schema and the metadata of the row groups.
2. Serialize the new rows. Encode the new rows into column chunks, and write them sequentially starting where the old footer began.
3. Reconstruct the footer. Build a new footer that includes the metadata for both the old row groups and the new row group.
4. Write the new footer. Write the new footer at the end of the file, followed by the length of the footer and the PAR1 magic bytes.

This operation needs explicit support from the writer and is vulnerable to interruption if performed in place. Common libraries such as PyArrow don't reopen a completed file for append. Instead, writers normally add a new file to the dataset. Appending to the dataset is therefore straightforward even though appending to an existing file isn't.

Writing a new file for every small batch leads to the known small file problem. A dataset with many small files suffers from:

Slower queries
Higher storage-access costs
Worse compression
Maintenance overhead

Example implication: stream a large file into a database

A common use of appending is loading a symbol-sorted file that is too large for memory, in batches. See Stream into a database. In kdb+, .Q.fs reads the file in chunks, and the loader appends each chunk to a splayed table on disk and then releases it.

With kdb+, you can choose any batch size that matches your memory budget. The batch size doesn't change the stored result, because the rows are added at the end of the column files.

With Parquet, the batch size sets an upper limit on the size of the row groups, because a writer such as PyArrow writes each batch as one or more row groups. The batch size forces a row group to end, but it doesn't stop the writer from ending one earlier. If the writer also ends a row group when the symbol changes, each row group holds a single symbol. Row group pruning then works well, because the minimum and maximum of the sym column exclude every row group of the other symbols.

A symbol with more rows than the batch size still spans several row groups. A query for that instrument reads all of them. This is less efficient than a single row group for the symbol, because the reader checks the metadata of more column chunks, and reads and decodes more, smaller chunks. In kdb+, the rows of a symbol in a symbol-sorted input stay contiguous in each column file, whatever batch size the loader uses, and the parted attribute (`p#) on sym records where each symbol starts.

Overwrite existing data¶

kdb+¶

To fix a value in a kdb+ table, use amend at on the column file. Amend at indexes directly into the file and overwrites the value in place, without rewriting the file. The following example corrects the second price of the splayed trade table. Pass the index and the new value as lists:

q)@[`:trade/price;enlist 1;:;enlist 545.0]
`:trade/price
q)select from get`:trade/
sym  price size
---------------
AAPL 270.1 100
MSFT 545   200
GOOG 310.5 50
AMZN 226.4 75

Amend at on disk works only on column files that are uncompressed, have no attribute, and aren't enumerated. To overwrite a complete row, or a value in a compressed or enumerated column such as sym, read the column file, amend it in memory, and write it back with set. Each column is a separate file, so these approaches load into memory only the columns that you change. To fix one column of a wide table, q loads that column and no others.

Parquet¶

A Parquet file has no way to overwrite a value in place, because a cell usually has no independent, fixed byte location that you can overwrite. A value lives inside an encoded and usually compressed page, and the footer records the size and statistics of that page.

For an occasional correction, read the affected file into a table, change the value, and write a replacement file. The following example changes the price in the third row. Writing to a temporary file and then renaming it means that readers see either the old file or the new one:

import os
import pyarrow as pa
import pyarrow.compute as pc
import pyarrow.parquet as pq

table = pq.read_table('trade.parquet')
mask = pa.array([i == 2 for i in range(table.num_rows)])
price = pc.replace_with_mask(table['price'], mask, pa.array([412.0]))
table = table.set_column(table.schema.get_field_index('price'), 'price', price)
pq.write_table(table, 'trade.parquet.tmp')
os.replace('trade.parquet.tmp', 'trade.parquet')

This approach loads every column of the file into memory, even though you change a single value in one column, because a replacement file must contain all the columns. For a large file, or a wide table with many columns, this can need a lot of memory, and can even be impossible. To reduce the memory, a writer can copy the file one row group at a time, but it still reads and writes the entire file.

Rewriting a file takes time in proportion to its size, so this approach doesn't suit frequent updates.

Table formats over Parquet: Apache Iceberg and Delta Lake

Parquet defines files, not a transactional table. Apache Iceberg and Delta Lake add a table layer that records which Parquet files make up the current version, together with schema, partition, and history metadata. A writer creates new or replacement data files, or separate delete information, and then commits a new table version atomically, so readers see either the old version or the new one. This enables concurrent writes, updates, deletes, schema evolution, rollback, and time travel. It doesn't make Parquet cells mutable: updates can still rewrite data files or add delete data that readers must merge.

Apache Iceberg. Iceberg represents each table version as a snapshot, using metadata files and manifests to track its data files. A catalog identifies the table's current metadata. Readers and writers must understand Iceberg and use the catalog; modifying the Parquet files directly bypasses the table metadata. Frequent commits accumulate snapshots, manifests, small data files, and files left behind by failed writes. Production tables therefore need scheduled maintenance: expire snapshots, remove orphan files, compact small data files, and sometimes rewrite manifests. Keeping more snapshots improves rollback and time travel but increases metadata and storage costs; expiring them removes that history.

Delta Lake. Delta Lake stores a versioned transaction log in _delta_log beside the Parquet files. Delta-aware readers reconstruct a snapshot from that log and its checkpoints, and writers use optimistic concurrency to commit a new version. Direct file changes bypass the log and can make the table inconsistent. Delta tables commonly need OPTIMIZE to compact small files and VACUUM to remove files no longer referenced by retained versions. Vacuuming reduces storage but also limits time travel, and an unsafe retention period can interfere with long-running readers or writers. Enabled table features also set protocol requirements, so older or less capable engines may no longer be able to read or write the table; some protocol upgrades are irreversible. Delta transactions are scoped to one table and don't provide foreign keys.

Both formats replace simple file management with metadata, compatible processing engines, retention policies, and recurring maintenance jobs. Those costs are often worthwhile for large, shared tables that need reliable updates, but can be unnecessary for an append-only dataset or occasional corrections.

Schema evolution¶

Schemas change over time. You add a column, rename one, or change its type. The two formats handle this differently.

kdb+¶

Every partition of a kdb+ table must have the same columns, in the same order, with the same types. A schema change must therefore reach every partition, or queries that span the change fail. The DBmaint module applies a change across every partition in one call. The following example adds an ex column and renames size to qty in a database with two partitions:

q)dbm:use`kx.dbmaint
q)dbm.addCol[`:db;`trade;`ex;" "]
q)dbm.renameCol[`:db;`trade;`size;`qty]
q)dbm.listCols[`:db;`trade]
`sym`price`qty`ex
q)\l db
q)select from trade
date       sym  price qty ex
----------------------------
2026.10.26 AAPL 270.1 100
2026.10.26 MSFT 541.2 200
2026.10.27 AAPL 271   150
2026.10.27 MSFT 545   250

Each column is a separate file, so these changes are cheap. Adding a column writes one new file for each partition. Renaming a column renames a file and updates the .d file, without touching the data. Casting a column rewrites only that column's files. For more, see Database maintenance.

Parquet¶

Each Parquet file stores its schema in its footer, so files written at different times can have different schemas. You don't need to change a dataset all at once. To add a column, you write new files with the extra column. Many readers can reconcile the schemas of the files by column name, and treat a column that is missing from a file as nulls. Some, such as Apache Spark, do this only when you enable schema merging, with the mergeSchema option.

Other changes are more expensive. The column names and types are part of each file, so renaming or casting a column means rewriting every file that contains it, including the columns that don't change.

Linked columns¶

A kdb+ database can store a relationship between two tables on disk, as a link column. For each row of a child table, a link column holds the row index of the matching row in a parent table. A query follows the link with dot notation, so it needs no join.

For example, the capmkts module of datagen can write a partitioned trade table with a link column, master, that points into a splayed master table of instrument reference data:

q)([buildPersistedDB]): use `kx.datagen.capmkts
q)buildPersistedDB["ldb"; ([linked:1b; mastertype:`splayed])]
q)\l ldb
q)meta trade
c     | t f      a
------| ----------
date  | d
sym   | s        p
time  | n
price | f
size  | j
stop  | b
cond  | c
ex    | s
master| j master

The query reads the description column of master through the link. The data is random, so your values differ:

q)3#select date,sym,price,master.description from trade where date=min date
date       sym  price  description
-----------------------------------
2026.09.28 AAPL 255.47 "Apple Inc."
2026.09.28 AAPL 255.47 "Apple Inc."
2026.09.28 AAPL 255.18 "Apple Inc."

Parquet has no format-level equivalent. The typical normalized design stores the related data in another table and joins the tables at query time. A join is slower than following a predefined relation, because it must find the matching rows on every query. A link column already holds the row index of the match.

The alternative is to denormalize the related columns into each child row. This avoids the join, but duplicates the reference data across files and possibly across several tables. Updating the reference data then requires finding and rewriting every copy, and partial updates can leave inconsistent values in the dataset.

Next steps¶

Follow the Quickstart to generate a Parquet file and query it.
Work through the Examples for row groups, pruning, and partitioning across files.
See the Parquet file format documentation for the specification.