Partitioning tables across directories
A partitioned table is a splayed table that is further decomposed by grouping records having common values along a column of special type. The allowable special column types have underlying integer values: date, month, year and long.
db ├── 2020.10.04 │ ├── quotes │ │ ├── .d │ │ ├── price │ │ ├── sym │ │ └── time │ └── trades │ ├── .d │ ├── price │ ├── sym │ ├── time │ └── vol ├── 2020.10.06 │ ├── quotes .. └── sym
Partition data correctly: data for a particular date must reside in the partition for that date.
For partitioned databases, q caches the count for a table, and this count cannot be updated from within a reval expression or from a secondary thread.
noupdate errors on queries on partitioned tables, put
count table in your startup script.
Partition a table if either
- it has over 100 million records
- it has a column that cannot fit in memory
- it grows
- many queries can be limited to a range of values of one column