Persisting to Object Storage
The KX Insights Platform can persist ingested data to object storage.
Persisting to object storage has major cost saving implications, and should be considered for older datasets.
For an example on reading data that has already been persisted to object storage, see querying from existing object storage.
Immutable data
Data in object storage cannot be modified. Primarily data that is meant for long term storage should go here. If the data is modifed/deleted externally, you should restart any HDB pods, and drop cache.
Prerequisites
You will need to configure environment variables for object storage.
You will need to provide credentials or utilize service accounts to access your private buckets.
Authentication
For more information on service accounts see automatic registration and environment variables.
To configure environment variables in an assembly, you may include them underneath the sm.env, dbm.env, eoi.env, eod.env, and dap.instances.*.env components as a list.
To set the environment vars for all pods, you may set spec.env.
For example, to set trace logging for objstor and an AWS_REGION:
spec:
env:
- name: AWS_REGION
value: us-east-2
- name: KX_TRACE_S3
value: "1"Performance considerations
Table layout
The layout of your table should be considered before uploading to object storage.
Neglecting to apply attributes, as well as not sorting symbols or partitioning by date can result in any query becoming a linear scan that pulls back the whole dataset.
Attributes and table layouts are specified in assembly YAML, and configured in the spec.table section of the assembly
Deployment
To persist data to cloud storage, you must deploy an assembly with an HDB tier that has a storage key.
This setting will create a segmented database where the HDB loads on-disk and cloud storage data together seamlessly.
Data in object storage is immutable, and cannot be refactored after it has been written. For this reason, date based partitions bound for object storage can only be written down once per day.
For example, this sm tiering configuration will:
- store data in memory for 10 minutes
- keep the last 2 days of data on disk
- and keep the remainder in object storage
sm:
tiers:
- name: streaming
mount: rdb
- name: interval
mount: idb
schedule:
freq: 00:10:00
snap: 00:00:00
- name: ondisk
mount: hdb
schedule:
freq: 1D00:00:00
snap: 01:35:00
retain:
time: 2 Days
- name: s3
mount: hdb
store: s3://examplebucket/dbMounts
You may see other KX Insights Platform examples referring to object type mounts. Those mounts are for reading from existing cloud storage.
For writing your own database to storage, no object mounts are involved, only HDB tier settings to make your HDB a segmented database.
For a guide on reading from an existing object storage database see querying object storage
Examples
Deploying using explicit credentials
All examples assume your data is published into a topic called "south".
The name of the example bucket is s3://examplebucket, and we want to save our database to a root folder called db.
Our example uses the AWS_REGION: us-east-2.
Stream processor and table schema sections have been omitted.
First create a new secret with your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
kubectl create secret generic aws-access-secret\
--from-literal=AWS_ACCESS_KEY_ID=${MY_KEY}\
--from-literal=AWS_SECRET_ACCESS_KEY=${MY_SECRET}Now create your assembly, setting sm.tiers to write to an S3 store, and environment variables for storage.
spec:
env:
- name: AWS_REGION
value: us-east-2
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: aws-access-secret
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: aws-access-secret
key: AWS_SECRET_ACCESS_KEY
mounts:
rdb:
type: stream
baseURI: none
partition: none
dependency:
- idb
idb:
type: local
baseURI: file:///data/db/idb
partition: ordinal
hdb:
type: local
baseURI: file:///data/db/hdb
partition: date
dependency:
- idb
sm:
source: south
tiers:
- name: streaming
mount: rdb
- name: interval
mount: idb
schedule:
freq: 00:10:00
snap: 00:00:00
- name: ondisk
mount: hdb
schedule:
freq: 1D00:00:00
snap: 01:35:00
retain:
time: 2 Days
- name: s3
mount: hdb
store: s3://examplebucket/db
dap:
instances:
idb:
mountName: idb
hdb:
mountName: hdb
rdb:
tableLoad: empty
mountName: rdb
source: southThe name of the example storage container is ms://mycontainer, and we want to save our database to a folder within the container called db.
Our example uses the AZURE_STORAGE_ACCOUNT: iamanexample. This means we will write our database to a folder called 'db', inside the container 'mycontainer' for the storage account iamanexample.
Stream processor and table schema sections have been omitted.
First create a new secret with your AZURE_STORAGE_SHARED_KEY.
kubectl create secret generic azure-storage-secret --from-literal=AZURE_STORAGE_SHARED_KEY=${MY_KEY}sm.tiers to write to an Azure store container, and environment variables for storage.
spec:
env:
- name: AZURE_STORAGE_ACCOUNT
value: mystorageaccount
- name: AZURE_STORAGE_SHARED_KEY
valueFrom:
secretKeyRef:
name: azure-storage-secret
key: AZURE_STORAGE_SHARED_KEY
mounts:
rdb:
type: stream
baseURI: none
partition: none
dependency:
- idb
idb:
type: local
baseURI: file:///data/db/idb
partition: ordinal
hdb:
type: local
baseURI: file:///data/db/hdb
partition: date
dependency:
- idb
sm:
source: south
tiers:
- name: streaming
mount: rdb
- name: interval
mount: idb
schedule:
freq: 00:10:00
snap: 00:00:00
- name: ondisk
mount: hdb
schedule:
freq: 1D00:00:00
snap: 01:35:00
retain:
time: 2 Days
- name: s3
mount: hdb
store: ms://mycontainer/db
dap:
instances:
idb:
mountName: idb
hdb:
mountName: hdb
rdb:
tableLoad: empty
mountName: rdb
source: southApply the assembly with kubectl apply -f object-tier.yml.