Datagen Module¶
The KDB-X Datagen module generates reproducible synthetic data for domains such as Capital Markets and DevOps. You can generate the data in memory, persist it to disk, or stream it in real time.
Key features¶
- Realistic schemas: Generate tables that follow real-world schema patterns, such as
trade,quote, andnbbo. - Configurable output: Control data volume, date range, and domain-specific parameters.
- Multiple delivery modes: Create in-memory tables, a date-partitioned or segmented database, or a real-time feed.
- High volume: Vectorized q implementations support large datasets.
Delivery modes¶
| Mode | Use it to |
|---|---|
| In-memory tables | Prototype queries and learn q, q-SQL, and SQL. |
| Persisted database | Test queries, storage layout, and performance against historical data. |
| Real-time stream | Feed a tickerplant (TP) or Reliable Transport (RT) process to demo or test streaming pipelines, including the KX Database Service. |
The real-time stream replays generated data at its original inter-event timing, in configurable batch sizes. It targets demos rather than stress testing.
Example¶
Load the Capital Markets generator, create the in-memory tables, and query them:
([getInMemoryTables]): use `kx.datagen.capmkts
(trade; quote; nbbo; master; exnames): getInMemoryTables[]
select sum size by exnames ex from trade
Further information¶
For parameters, and examples, see the Datagen documentation on GitHub.