Parquet Quickstart¶
This quickstart generates a sample trade file in Parquet format, loads the Parquet module, and runs an aggregation query on the file.
Prerequisites¶
Before you begin, you need:
- A working KDB-X installation and a valid license that includes the
pqfeature - The
datagenmodule, which generates the sample data - Python 3 with
pykx,pandas, andpyarrowinstalled
Create a sample file¶
The capmkts module of datagen generates capital markets data. Use PyKX to generate a table of trades, and pandas to write it to a Parquet file:
import pandas as pd
import pykx as kx
capmkts = kx.use('kx.datagen.capmkts')
df = capmkts.getInMemoryTables({'tbls': 'trade'})[0].pd()
df['time'] = pd.Timestamp.today().normalize() + df['time']
df.to_parquet('trade.parquet')
The first table that getInMemoryTables returns is trade. The generated time column holds the time of day, so the script adds today's date to turn it into a full timestamp. The file holds a few thousand trades. The data is random, so your values differ from the results shown on this page.
Want to work with real data?
Use the TAQ module to load NYSE Trade and Quote (TAQ) data.
Load the module¶
Import the module and assign it to a name:
q).pq:use`kx.pq
Open the file¶
pq maps the Parquet file and returns a virtual table. It doesn't load the data into memory:
q)t:.pq.pq`:trade.parquet
q)select from t where i<3
sym time price size stop cond ex
----------------------------------------------------------------
"CUZ" 2026.10.26D09:30:00.011627745 29.06 65 0 ,0x42 ,"L"
"PRU" 2026.10.26D09:30:00.153776659 103.02 20 0 ,0x42 ,"J"
"TSLA" 2026.10.26D09:30:00.167902241 463.29 27 0 ,0x45 ,"V"
Aggregate with group by¶
Use a select with aggregation and a by clause to summarize the AAPL trades for each hour of the day:
q)select trades:count i,volume:sum size,vwap:size wavg price,high:max price,low:min price by hour:time.hh from t where sym like"AAPL"
hour| trades volume vwap high low
----| ------------------------------------
9 | 6 400 257.4315 258.57 255.6
10 | 6 363 262.8368 263.9 260.3
11 | 12 483 267.7276 269.23 264.89
12 | 9 447 271.2061 273.16 269.52
13 | 6 271 275.1221 276.29 274.21
14 | 9 357 278.1386 279.98 276.61
15 | 9 467 283.6481 285.48 280.51
The query filters the rows, buckets them by the hour of time, and computes the trade count, volume, volume-weighted average price (VWAP), high, and low for each hour. It reads only the columns it references.
The same select syntax works on Parquet files as on any kdb+ table, so you can filter, derive columns, and aggregate without leaving q.
Inspect the schema¶
Use meta to see the column types the module inferred:
q)meta t
c | t f a
-----| -----
sym | C
time | p
price| f
size | j
stop | b
cond | X
ex | C
See Types for how Parquet types map to q types.