Skip to content

Sample assembly configuration

name: Tick data sample
labels:
  region: canada

tables:
  trade:
    description: Trade data
    type: partitioned
    shards: 11
    blockSize: 10000
    prtnCol: realTime
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMemory: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Real timestamp
        type: timestamp
      - name: price
        description: Trade price
        type: float
      - name: size
        description: Trade size
        type: long

  quote:
    description: Quote data
    type: partitioned
    shards: 11
    blockSize: 10000
    prtnCol: realTime
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMemory: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Real timestamp
        type: timestamp
      - name: bid
        description: Bid price
        type: float
      - name: ask
        description: Ask price
        type: float
      - name: bidSize
        description: Big size
        type: long
      - name: askSize
        description: Ask size
        type: long

bus:
  stream:
    protocol: rt
    topic: dataStream

mounts:
  rdb:
    type: stream
    baseURI: file:///mnt/foo/i/
    partition: none
  idb:
    type: local
    baseURI: file:///data/db/idb
    partition: ordinal
  hdb:
    type: local
    baseURI: file:///data/db/hdb
    partition: date

elements:
  sm:
    description: Storage manager
    source: stream # bus configuration
    tiers:
      - name: mem
        mount: rdb
      - name: intra
        mount: idb
        schedule:
          freq: 0D00:10:00 # every 10 minutes
      - name: hist1
        mount: hdb
        schedule:
          freq: 1D00:00:00 # every day
          snap:   01:35:00 # at 1:35 AM
        retain:
          time: 2 days
          rows: 200000
      - name: hist2
        mount: hdb
        store: file:///data/db/hdbtier2/
        retain:
          time: 5 weeks
          size: 2 TB
          rows: 10000000
      - name: hist3
        mount: hdb
        store: s3://hdbtier3/
        retain:
          time: 3 months
          size: 1 PB
          rows: 20000000

  dap:
    smEndpoints: sm:10001
    gwEndpoints: rc:5050
    instances:
      RDB:
        mountName: rdb
      IDB:
        mountName: idb
      HDB:
        mountName: hdb

The above sample assembly has the following configuration:

  • Uses tick database schema.

  • Uses KX Insights Reliable Transport as the message bus

  • Defines 3 mount mappings: rdb (realtime), idb (intraday), and hdb (historical), where idb and hdb map to disk volumes.

  • Writes down the data spread across the following 5 tiers:

name mount (type-partition) storage time range
mem rdb (stream-none) 10 minutes
intra idb (local-ordinal) $baseURI/current 1 day
hist1 hdb (local-date) $baseURI/data 2 days
hist2 hdb (local-date) /data/db/hdbtier2 5 weeks
hist3 hdb (local-date) s3:/hdbtier2 3 months
  • Has 3 Data-Access instances mapping to rdb, idb, and hdb mounts respectively