Sample assembly configuration
name: Tick data sample
labels:
region: canada
tables:
trade:
description: Trade data
type: partitioned
shards: 11
blockSize: 10000
prtnCol: realTime
columns:
- name: time
description: Time
type: timespan
- name: sym
description: Symbol name
type: symbol
attrMemory: grouped
attrDisk: parted
attrOrd: parted
- name: realTime
description: Real timestamp
type: timestamp
- name: price
description: Trade price
type: float
- name: size
description: Trade size
type: long
quote:
description: Quote data
type: partitioned
shards: 11
blockSize: 10000
prtnCol: realTime
columns:
- name: time
description: Time
type: timespan
- name: sym
description: Symbol name
type: symbol
attrMemory: grouped
attrDisk: parted
attrOrd: parted
- name: realTime
description: Real timestamp
type: timestamp
- name: bid
description: Bid price
type: float
- name: ask
description: Ask price
type: float
- name: bidSize
description: Big size
type: long
- name: askSize
description: Ask size
type: long
bus:
stream:
protocol: rt
topic: dataStream
mounts:
rdb:
type: stream
baseURI: file:///mnt/foo/i/
partition: none
idb:
type: local
baseURI: file:///data/db/idb
partition: ordinal
hdb:
type: local
baseURI: file:///data/db/hdb
partition: date
elements:
sm:
description: Storage manager
source: stream # bus configuration
tiers:
- name: mem
mount: rdb
- name: intra
mount: idb
schedule:
freq: 0D00:10:00 # every 10 minutes
- name: hist1
mount: hdb
schedule:
freq: 1D00:00:00 # every day
snap: 01:35:00 # at 1:35 AM
retain:
time: 2 days
rows: 200000
- name: hist2
mount: hdb
store: file:///data/db/hdbtier2/
retain:
time: 5 weeks
size: 2 TB
rows: 10000000
- name: hist3
mount: hdb
store: s3://hdbtier3/
retain:
time: 3 months
size: 1 PB
rows: 20000000
dap:
smEndpoints: sm:10001
gwEndpoints: rc:5050
instances:
RDB:
mountName: rdb
IDB:
mountName: idb
HDB:
mountName: hdb
The above sample assembly has the following configuration:
-
Uses tick database schema.
-
Uses KX Insights Reliable Transport as the message bus
-
Defines 3 mount mappings:
rdb
(realtime),idb
(intraday), andhdb
(historical), whereidb
andhdb
map to disk volumes. -
Writes down the data spread across the following 5 tiers:
name | mount (type-partition) | storage | time range |
---|---|---|---|
mem |
rdb (stream-none) |
10 minutes | |
intra |
idb (local-ordinal) |
$baseURI/current |
1 day |
hist1 |
hdb (local-date) |
$baseURI/data |
2 days |
hist2 |
hdb (local-date) |
/data/db/hdbtier2 |
5 weeks |
hist3 |
hdb (local-date) |
s3:/hdbtier2 |
3 months |
- Has 3 Data-Access instances mapping to
rdb
,idb
, andhdb
mounts respectively