Skip to content

Parquet in KDB-X

This page explains what the Parquet module is and the concepts you need to query Parquet files from KDB-X.

Apache Parquet is a columnar storage format designed for efficient storage and retrieval. It's an open format that many engines support.

The kx.pq module reads Parquet files from q. It presents each file as a virtual table that you query with select and exec.

Capabilities

Key concepts

Virtual tables

pq returns a virtual table for a Parquet file. Virtual tables support select and exec queries. The built-in meta function returns the usual column information for them.

Performance

The module is built with performance in mind. It uses:

  • Pruning. The module uses the metadata in the Parquet file to skip data that can't match a where clause, such as row groups within a file. When you partition across files, it also skips whole files. For a demonstration, see Filter on row groups.
  • Vector operations. The columnar layout of Parquet lets the module hand whole columns to q's vector operations.

Partitioning across files

The virtual table API combines several Parquet files into one virtual table, using virtual partition columns. Filtering on those columns avoids reading the files that don't match, in the same way as for partitioned kdb+ tables.

Planned improvements

Future releases plan to add support for:

  • Object storage, such as S3, including caching.
  • Nested types, such as interval in pandas, or MAP, STRUCT, and UNION.
  • Nested (non-atomic) virtual columns, such as strings.
  • hive and iceberg layouts.

Next steps

  • Follow the Quickstart to load the module and run a first query.
  • Work through the Examples for row groups, pruning, and partitioning across files.
  • See the Reference for the API, supported types, and compression.
  • Read Parquet for q developers to see how Parquet differs from a kdb+ partitioned database.