Database: tables in the filesystem

Tables are important first-class entities in q. While smaller q tables can be held in memory, we will need to persist them if they are large enough or if we want to keep them after the process terminates. We might also want to do operations on these persisted tables.

This document lists the options for saving tables to the filesystem in KDB-X.

How we serialize a table depends on its size and how we need to use it.

Serialization	Representation	Recommended use case
flat table	single binary file	small and most queries use most columns
splayed table	directory of column files	up to 100 million rows
partitioned table	table partitioned by e.g. date, with a splayed table for each date	more than 100 million records; or growing steadily
segmented database	partitioned tables distributed across disks	tables larger than disks; or you need to parallelize access

Flat table

q will serialize and file any object as a single binary file – the simplest way to persist a table.

A database with tables trades and quotes:

db/
├── quotes
└── trades

We can also export the table in other formats (for example, .csv, .xls) if needed with save.

Splayed table

A table is splayed by storing each of its columns as a single file. The table is stored as a directory.

db/
├── quotes/
|   ├── time
|   ├── sym
|   └── price
└── trades/
|   ├── time
|   ├── sym
|   ├── price
|   └── vol
|   └── sym
└── sym

With a splayed table, each column file is only read into memory when a query requires it.

Consider splaying a table if most queries on a reasonably sized table do not need all the columns.

Partitioned table

The records of a partitioned table are distributed across multiple partition directories within its root directory. The table is partitioned by the values of a single column. Each partition contains records that have the same value in the partitioning column. With time series data, this is most commonly a date or time.

db/
├── 2020.10.03/
│   ├── quotes/
│   │   ├── price
│   │   ├── sym
│   │   └── time
│   └── trades/
│       ├── price
│       ├── sym
│       ├── time
│       └── vol
├── 2020.10.05/
│   ├── quotes/
│   │   ├── price
│   │   ├── sym
│   │   └── time
│   └── trades/
│       ├── price
│       ├── sym
│       ├── time
│       └── vol
└── sym

The partition directory is named for its partition value and contains a splayed table with just the records that have that value.

Consider partitioning a table if any of the following conditions apply:

It grows over time.
It contains more than 100 million records.
It includes columns that exceed the maximum object size allowed in memory.

Segmented database

The root directory of a segmented database contains only two files:

par.txt: a text file listing the paths to the segments
the sym file for enumerated symbol columns

Segments are stored outside the root, usually on various volumes. Each segment contains a partitioned table.

DISK 0             DISK 1                     DISK 2
db/                db/                       db/
├── par.txt        ├── 2020.10.03/           ├── 2020.10.04/
└── sym            │   ├── quotes/           │   ├── quotes/
                   │   │   ├── .d            │   │   ├── .d
                   │   │   ├── price         │   │   ├── price
                   │   │   ├── sym           │   │   ├── sym
                   │   │   └── time          │   │   └── time
                   │   └── trades/           │   └── trades/
                   │       ├── .d            │       ├── .d
                   │       ├── price         │       ├── price
                   │       ├── sym           │       ├── sym
                   │       ├── time          │       ├── time
                   │       └── vol           │       └── vol
                   ├── 2020.10.05/           ├── 2020.10.06/
                   │   ├── quotes/           │   ├── quotes/
                   ..                        ..

Consider segmenting a table across multiple storage devices if any of the following conditions apply:

The table exceeds the capacity of a single storage device.
You need to parallelize access to the table.
You want to partition the table by a non-integer datatype, such as a symbol.

Dividing a table across storage devices allows you to:

Store very large tables that exceed single-device capacity.
Parallelize queries across partitions.
Isolate updates to specific partitions, avoiding rewrites of the entire table.

Mix of table types

A KDB-X database typically contains many tables of different types. Flat, splayed and partitioned tables may reside nicely next to each other. Flat and splayed tables are in the root directory together with other KDB-X objects (like lists and dictionaries).

Before proceeding, generate sample data using the Datagen module.

q)([getInMemoryTables; buildPersistedDB]): use `kx.datagen.capmkts
q)buildPersistedDB["/tmp/testdb"; ([tbls: `trade`quote`daily; start: 2026.04.10; end: 2026.04.11])]

Review the file structure:

/tmp/testdb
├── 2026.04.10
│   ├── quote
│   │   ├── .d
│   │   ├── asize
│   │   ├── ask
│   │   ├── bid
│   │   ├── bsize
│   │   ├── ex
│   │   ├── mode
│   │   ├── sym
│   │   └── time
│   └── trade
│       ├── .d
│       ├── cond
│       ├── ex
│       ├── price
│       ├── size
│       ├── stop
│       ├── sym
│       └── time
├── daily
│   ├── .d
│   ├── close
│   ├── date
│   ├── high
│   ├── low
│   ├── open
│   ├── price
│   ├── size
│   └── sym
├── exnames
├── master
└── sym

Where:

exnames stores a dictionary
master is a flat table
sym is the symbol list
daily is a splayed table
trade and quote are partitioned tables

Loading the database (for example, with \l) copies all files in the root directory into memory. Splayed and partitioned tables are memory-mapped.

Serializing objects (including flat tables) Splayed tables Partitioned tables Segmented databases

Q for Mortals §14. Introduction to KDB-X