# Parquet Reference

_This page is the reference for the Parquet module API, supported types, compression, and encodings._

Import the module before calling any of its functions:
```q
q).pq:use`kx.pq
```

For concepts, see the [Overview](overview.md). For worked examples, see [Examples](examples.md). For details of the Parquet file format, see the [Apache Parquet](https://parquet.apache.org/docs/) documentation.

## Functions

### `pq`

Maps a Parquet file and returns a virtual table.
```txt
.pq.pq path
```

| **Name** | **Type** | **Description** | **Required** |
|---|---|---|---|
| `path` | file symbol | Path to the Parquet file | Yes |

The virtual table has a hidden virtual column, `RG__`, holding the row group number. It supports `select` and `exec` queries.
```q
q)t:.pq.pq`:types.parquet
```

The file is read through the operating system page cache, using `mmap`.

### `op`

Opens a Parquet file, reads the metadata, and returns a data structure.
```txt
.pq.op path
```

| **Name** | **Type** | **Description** | **Required** |
|---|---|---|---|
| `path` | file symbol | Path to the Parquet file | Yes |

> **Warning — Subject to change**
>
> `op` and `rd` are low-level functions. The structure that `op` returns is subject to change.
>
### `rd`

Returns a column chunk from the structure that `op` returns.
```txt
.pq.rd[data;row;col]
```

| **Name** | **Type** | **Description** | **Required** |
|---|---|---|---|
| `data` | structure | The structure returned by [`op`](#op) | Yes |
| `row` | long | Row group index | Yes |
| `col` | long | Column index | Yes |
```q
q)f:.pq.op`:rowgrp.parquet
q).pq.rd[f;0;1]
1 2 3h
```

See [Read column chunks](examples.md#read-column-chunks).

## Virtual table API

The virtual table module is a submodule of `pq`. Load it with a multi-part module name:
```q
q).pq.t:use`kx.pq.t
```

### `tt`

Wraps a kdb+ table in a virtual table. This allows combining Parquet and kdb+ tables as partitions in a virtual table.
```txt
.pq.t.tt tbl
```

| **Name** | **Type** | **Description** | **Required** |
|---|---|---|---|
| `tbl` | table | A kdb+ table | Yes |

### `mkP`

Creates a partitioned virtual table from a key table and a list of virtual tables.
```txt
.pq.t.mkP map
```

| **Name** | **Type** | **Description** | **Required** |
|---|---|---|---|
| `map` | dictionary | A key table mapped to a list of virtual tables, one for each row of the key table | Yes |

The columns of the key table are virtual columns. Filtering on them avoids reading the partitions that don't match. See [Partition across multiple files](examples.md#partition-across-multiple-files).

## Types

The following Parquet [logical types](https://parquet.apache.org/docs/file-format/types/logicaltypes/) are supported, and map to the q types shown.

| **Parquet type** | **q type** |
|---|---|
| `string` | `char[]` |
| `binary` | `byte[]` |
| `bool` | `boolean` |
| `uuid` | `guid` |
| `int8` | `short` |
| `int16` | `short` |
| `int32` | `int` |
| `int64` | `long` |
| `uint8` | `byte` |
| `uint16` | `int` |
| `uint32` | `long` |
| `uint64` | `long` |
| `date` | `date` |
| `timestamp` | `timestamp` |
| `interval` | `timespan` |
| `duration` | `long` |
| `null` | `short` |

Columns of [physical type](https://parquet.apache.org/docs/file-format/types/) `FIXED_LEN_BYTE_ARRAY` with a length (width) of _n_ map to the following q types:

| **Width _n_** | **q type** |
|---|---|
| 2 | `short` |
| 4 | `int` |
| 8 | `long` |
| 16 | `guid` |
| any other | `byte[]` |

## Compression and encoding

The following [byte-stream compressors](https://parquet.apache.org/docs/file-format/data-pages/compression/) are supported:

- gzip
- snappy
- brotli
- lz4
- zstd

The following [encodings](https://parquet.apache.org/docs/file-format/data-pages/encodings/) are supported:

- run-length
- dictionary
- delta (`DELTA_BINARY_PACKED`)
