Skip to content

Taq module

This page explains what the Taq module is and when to use it.

The Taq module parses NYSE Trade and Quote (TAQ) data in pipe-separated values (PSV) files and turns them into kdb+ data. You can write the data to a date-partitioned database on disk or load it into in-memory tables. You can also stream it to a kdb+ tick tickerplant (TP) or to the KX Database Service (DB Service) through Reliable Transport (RT).

The module is written in q. It replaces the legacy k-based scripts with multithreaded parsing and parallel column writes. Batch processing and filtering by symbol give you control over memory use when you work with large datasets.

Key features

  • Three destinations: Persist data to disk with parseToDisk, load it into memory with parseToMemory, or stream it with streamToPubSub.
  • Stream to kdb+ tick or DB Service: Publish data to a TP, or to DB Service through an RT publisher, as if it arrived from the exchange. Use the starttime and endtime options to replay a slice of the trading day.
  • Schemas for integration: Get the master, trade, and quote schemas as q tables, DB Service column definitions, or YAML assembly definitions.
  • Memory management: Set batchsize to process large files in batches and avoid exhausting RAM.
  • Customizable filtering: Filter by the first letter of the symbol and exclude test symbols.
  • Compression support: Compress data while writing it, using .z.zd settings.
  • Parallel ingestion: Use secondary threads (-s) to parse PSV files and write columns to disk at the same time.

Generate data on disk or in memory

Use parseToDisk to create a partitioned database that you load with q /path/to/db. Use parseToMemory to skip the disk and load the trade, quote, and master tables, plus the exnames dictionary, into your session. The in-memory tables are sorted by time and carry a grouped attribute on sym, which matches a typical RDB. You can change the sort columns and the attribute with the sortcols and symattr options.

taq: use `kx.taq
(trade; quote; master; exnames): taq.parseToMemory["/tmp/nysetaqpsv"; 2025.10.02]

Stream to kdb+ tick or DB Service

streamToPubSub publishes master first. It then sends trade and quote rows in batches, paced to match the timing of the original events. The input PSV files must be sorted by time. The module includes the sort.sh script for this.

  • kdb+ tick: Pass a connection handle to a TP. The module sends each batch asynchronously.
  • DB Service: Pass an RT publisher function. Create the master, trade, and quote tables in DB Service before you publish.

Use streamToPubSub for demos, not stress testing.

Typical use cases

Benchmarking

Use the module to benchmark kdb+ across environments:

  • Hardware validation: Test disk I/O and CPU scaling by measuring ingestion speed. Measure query times on real data.
  • Optimization tuning: Compare kdb+ settings, such as compression algorithms (for example, LZ4 and Zstd) and logical block sizes. Weigh disk space against ingestion speed and query latency.
  • Memory and disk sizing: Choose a dataset size, from tiny to full, to match the memory and disk you want to test.

Demos

Real data often makes a better demo than generated data. Real TAQ data has realistic symbol counts, volumes, and distributions, so queries and dashboards show results your audience recognizes.

  • Quick start: Load a small slice into memory with parseToMemory and run queries immediately.
  • Live feed: Stream data to a TP or DB Service with streamToPubSub to show a live market feed, without a market data subscription.
  • Replay a moment: Stream a window of the day, such as the closing auction, to show how your application handles a burst of activity.