Storage tiers
Tiers describe the locality, segmentation format, and rollover configuration of each storage tier. Storage tiers are used to migrate data over time from fast, expensive storage to slower, less-expensive storage. Depending on your use case, this configuration can be tuned to either have more data in memory for faster query performance, or have more data on disk to reduce costs.
Tier configuration
Refer to tier configuration for the full list of configuration options.
Tier types
Tiers differ in performance and cost. As data ages, its value decreases and it can be transitioned to slower, less-expensive storage. Tiers for a database are split into a single stream tier, a single ordinal tier and one or more date tiers. The type of the tier is determined based on the configuration of the tier's mount.
sm:
tiers:
- name: rdb
mount: rdb
- name: idb
mount: idb
schedule:
freq: 0D00:10:00 # every 10 minutes
- name: hdb
mount: hdb
schedule:
freq: 1D00:00:00 # every day
snap: 01:35:00 # at 1:35 AM
retain:
time: 2 days
Stream tier
A stream tier is responsible for in-memory data that is received between write-down events. This tier is used to hold the most recent data available in memory for the fastest possible query results. This tier is typically referred to as the real-time database or RDB. This tier has no persisted data, so it must have enough memory capacity to hold data for the interval between writes to disk. This interval has a default of 10 minutes. There can only be one stream tier in a tier configuration and it must always be the first tier.
A stream tier is configured by using a stream mount with no partitioning scheme.
mounts:
rdb:
type: stream
partition: none
elements:
sm:
tiers:
- name: rdb
mount: rdb
Ordinal tier
An ordinal tier is responsible for data written to disk after migrating out of the stream tier. This tier holds data that has been written periodically throughout the day after flushing the data from the stream tier. As a result, it is typically referred to as the intra-day database or IDB. Data in this tier is stored on disk in a partitioned table that is sequenced in order of the intervals written throughout the day. There can only be one ordinal based tier in a tier configuration and it must always follow the stream tier.
An ordinal tier is configured using a local
mount with an ordinal
partitioning scheme. The frequency for how often data is flushed from the stream tier and written to disk here is configurable, and has a default of 10 minutes.
mounts:
rdb:
type: stream
partition: none
idb:
type: local
partition: ordinal
sm:
tiers:
- name: rdb
mount: rdb
- name: idb
mount: idb
schedule:
freq: 0D00:10:00 # interval every 10 minutes
Date tier
A tier configuration can have one or more date tiers following an ordinal
tier. Date tiers hold historical data partitioned by date, and you can choose to have one or many of them. Date tiers can be configured to hold data for a set duration before either migrating it to the next tier, or discarding it. This is based on your desired configuration and data retention requirements. Data is migrated exactly once per day from the ordinal
tier to the first date
tier, and between tiers. A snap
value can be set to configure what time of day the migration happens relative to midnight. Note: when configuration is changed, data is moved between tiers to reach the desired distribution. This means that data might be moved in the "reverse direction", if retain time of e.g. the first tier is increased, data will be moved from the second tier to the first. Data in the object store are not subject to relocation.
A date tier is configured by using a local
mount with a date
partitioning scheme. Multiple date tiers can be added sequentially with slower, less-expensive disks as data ages. Note that there can only be one mount
added with date
partitioning. Too add a second tier with date
partitioning(see hdb1b
in example below):
- a new tier should be added to the elements.sm.tiers
section with unique name
- the mount should be set to the name of the date
partitioned mount in mounts
section
- the store
set to the location where the next tier of data should be stored(see store)
Date tiers in object storage
For data that is very old and/or infrequently queried, a date object storage tier can be added. This is configured using a normal date mount, with extra tier configuration to point to the object storage bucket. See object storage tier below for details.
mounts:
rdb:
type: stream
partition: none
baseURI: none
idb:
type: local
partition: ordinal
baseURI: file:///data/db/idb
hdb:
type: local
partition: date
baseURI: file:///data_ssd/db/hdb
elements:
sm:
source: stream
tiers:
- name: rdb
mount: rdb
- name: idb
mount: idb
schedule:
freq: 0D00:10:00 # every 10 minutes
- name: hdb1a
mount: hdb
schedule:
freq: 1D00:00:00 # every day
snap: 01:35:00 # at 1:35 AM
retain:
time: 2 days
- name: hdb1b
mount: hdb
retain: 2 weeks
store: file:///data_hdd/db/hdb
Object storage tier
For data that needs to be preserved on very inexpensive storage, it can be migrated to an object storage tier. Object storage tiers offer the benefit of being endlessly scalable and highly available. An object storage tier is a final destination for data in the tier lifecycle.
Object storage tiers vs mounting an object storage database
An object storage tier is a read-write location and is configured differently than a read-only database. See querying object storage for an example of how to query an existing object storage database.
An object storage tier is a date tier with extra details about the object storage location. To configure an object storage tier, add a store
field with the remote data location. To improve query performance, a cache index called an inventory file can be configured.
In the example below, data flows from the normal hdb
disk to the odb
object storage tier after being on disk for 2 weeks. Both the hdb
and the odb
tier use the same hdb
mount, but the odb
tier overrides the store
location to point to an AWS S3 location.
mounts:
rdb:
type: stream
partition: none
baseURI: none
idb:
type: local
partition: ordinal
baseURI: file:///data/db/idb
hdb:
type: local
partition: date
baseURI: file:///data/db/hdb
elements:
sm:
source: stream
tiers:
- name: rdb
mount: rdb
- name: idb
mount: idb
- name: hdb
mount: hdb
retain:
time: 2 weeks
- name: odb
mount: hdb
store: s3://kxi-example-data/db
inventory:
enabled: true
location: inventory/test-db-inventory.tgz
Data must persist to disk before migrating to object storage
Before data can be migrated to an object storage tier, it must be persisted in a date tier for at least one full day. This implies that the minimum retention period for the HDB tier is 1 day.
To learn more about configuring object storage tiers, see the guide on object storage configuration.