Parquet Reference

This page is the reference for the Parquet module API, supported types, compression, and encodings.

Import the module before calling any of its functions:

q).pq:use`kx.pq

For concepts, see the Overview. For worked examples, see Examples. For details of the Parquet file format, see the Apache Parquet documentation.

Functions

pq

Maps a Parquet file and returns a virtual table.

.pq.pq path

Name	Type	Description	Required
path	file symbol	Path to the Parquet file	Yes

The virtual table has a hidden virtual column, RG__, holding the row group number. It supports select and exec queries.

q)t:.pq.pq`:types.parquet

The file is read through the operating system page cache, using mmap.

op

Opens a Parquet file, reads the metadata, and returns a data structure.

.pq.op path

Name	Type	Description	Required
path	file symbol	Path to the Parquet file	Yes

Subject to change

op and rd are low-level functions. The structure that op returns is subject to change.

rd

Returns a column chunk from the structure that op returns.

.pq.rd[data;row;col]

Name	Type	Description	Required
data	structure	The structure returned by op	Yes
row	long	Row group index	Yes
col	long	Column index	Yes

q)f:.pq.op`:rowgrp.parquet
q).pq.rd[f;0;1]
1 2 3h

See Read column chunks.

Virtual table API

The virtual table module is a submodule of pq. Load it with a multi-part module name:

q).pq.t:use`kx.pq.t

tt

Wraps a kdb+ table in a virtual table. This allows combining Parquet and kdb+ tables as partitions in a virtual table.

.pq.t.tt tbl

Name	Type	Description	Required
tbl	table	A kdb+ table	Yes

mkP

Creates a partitioned virtual table from a key table and a list of virtual tables.

.pq.t.mkP map

Name	Type	Description	Required
map	dictionary	A key table mapped to a list of virtual tables, one for each row of the key table	Yes

The columns of the key table are virtual columns. Filtering on them avoids reading the partitions that don't match. See Partition across multiple files.

Types

The following Parquet logical types are supported, and map to the q types shown.

Parquet type	q type
string	char[]
binary	byte[]
bool	boolean
uuid	guid
int8	short
int16	short
int32	int
int64	long
uint8	byte
uint16	int
uint32	long
uint64	long
date	date
timestamp	timestamp
interval	timespan
duration	long
null	short

Columns of physical type FIXED_LEN_BYTE_ARRAY with a length (width) of n map to the following q types:

Width n	q type
2	short
4	int
8	long
16	guid
any other	byte[]

Compression and encoding

The following byte-stream compressors are supported:

• gzip
• snappy
• brotli
• lz4
• zstd

The following encodings are supported:

• run-length
• dictionary
• delta (DELTA_BINARY_PACKED)