Skip to content

Object storage

Authenticate with cloud credentials via Kurl and get native access to cloud object storage

Example: a single bucket on each of the clouds, using the prefixes:

aws     `:s3://
gcp     `:gs://
azure   `:ms://

Start kdb+ with the object store library loaded.

KDB+cloud 4.0 2021.01.15 Copyright (C) 1993-2020 Kx Systems
l64/ 8(16)core 32005MB ...

Delve deeper into the buckets


and get the contents of the file.


Other read operations work as if the file were on block storage.

q)hcount `:gs://kxinsights-marketplace-data/zd1726/db/2018.09.04/trade/sym

compressedLength  | 579442
uncompressedLength| 276955208
algorithm         | 2i
logicalBlockSize  | 17i
zipLevel          | 6i


To mount an HDB directly from object storage, we set up a partition file locally as

$ more db/par.txt 

with the HDB sym file local too. Any number of paths can be specified in par.txt, mixing both cloud and block storage paths. There should be no tailing / after the object store path!

The local directory tree should look like

├── par.txt
└── sym

The sym file

The sym file in this db dir should be the enum domain, i.e. a symbol list. The partitions listed in par.txt should not contain this enum domain in their root.


The HDB can then be mounted via the usual

q)\l db

Since kdb+ changes the working directory to db, the following will not work

q)\l s3://kxinsights-marketplace-data/db

because the S3 storage is not presented as a POSIX filesystem. Hence the only way presently to load a database is by par.txt.

Splayed tables resident on object storage can be mapped and queried directly:

q)select from t
time         sym cond ex price size  stop
07:17:49.434 A   T    P  67.56 40    0
09:30:00.329 A   I    Z  67.23 1     0
09:30:00.329 A   I    Z  67.23 1     0
09:30:01.004 A   O    N  67.34 13674 0

or the shorter

q)select from `:s3://kxtaq/data/zd1726/db/2018.09.04/trade/
time         sym cond ex price size  stop
07:17:49.434 A   T    P  67.56 40    0
09:30:00.329 A   I    Z  67.23 1     0
09:30:00.329 A   I    Z  67.23 1     0
09:30:01.004 A   O    N  67.34 13674 0


All keys in a bucket are cached in memory, along with each object’s size. The first time a bucket is read from, all keys will be retrieved. To trigger a reload of this metadata, use a path of _ after the bucket, indicating ‘drop’. e.g.


Read only

The objstor library allows read-only access to object storage. The objects should be created using the cloud vendor’s standard CLI tooling to copy data from block storage to the cloud, e.g.

aws s3 cp "/path/to/file.txt" s3://kxinsights-marketplace-data/ --recursive
gsutil cp -r "/path/to/file.txt" gs://kxinsights-marketplace-data/
azcopy cp "/path/to/file.txt" "https://[account][container]/[path/to/blob]"

Environment variables

Although the Kurl module can detect most of the necessary components for credentials from the environment, the following additional environment variables are required.


If a region is not selected, then us-east-1 is used by default.

The URIs requested to the cloud are printed to STDERR if the following environment variable is set. e.g.

export KX_TRACE_S3=1

The DNS prefix for your storage account; e.g. for the name would be mystorage. The list of your storage accounts can be displayed using the Azure CLI tool az:

az storage account list | jq -r '.[] | .name'

A unique, user-assigned ID that can be used as the request header x-goog-project-id in Google APIs. It is used as a request header that specifies which project you are working on. It may be any valid project number or name. This request header tells Cloud Storage which project to create a bucket in or which project to list buckets for. Examples:


The list of your projects can be displayed using the Google CLI tool, Gcloud, via

gcloud projects list


Cloud object storage is high-latency and low bandwidth, and will perform significantly worse than block storage.


Due to the high latency of cloud storage, offers to cache the results of requests on a local high-performance disk, the path to which should be specified in the environment variable KX_S3_CACHE_PATH, e.g.

export KX_S3_CACHE_PATH=/myfastssd/kxs3cache

and kdb+ will then create the cached files under a subdir as $KX_S3_CACHE_PATH/objects.

Shared cache

This cache area on disk is designed to be shared by multiple kdb+ processes, and will only ever populate it.

Managing eviction from this cache folder is done by the kxreaper process.

Bear in mind that cloud vendors charge for object storage as a combination of volume stored, per retrieval request, and volume egress. Using the built-in compression and the cache can help to reduce these costs.

Secondary threads

The way to achieve concurrency with these high-latency queries is with secondary threads, through the command line option -s. It is expected that the larger the number of secondary threads, irrespective of CPU core count, the better the performance of object storage. Conversely the performance of cached data appears to be better if the secondary-thread count matches the CPU core count. A balance is to be found. We expect in future to improve the thread usage for these requests.

The impact of threading is seen mainly in two use cases:

Kdb+ V4.0 has an optimization that columns used in a query are mapped in parallel at the start of a select, so the number of secondary threads should be at least the number of columns selected. Assuming peach is not already being used in an outer function then select performance is improved as it will mmaps columns in parallel.

q)// -s 8
q)\t select from quote where date=2018.09.07

q)// -s 0
q)\t select from quote where date=2018.09.07

Multithreaded prims will show improved performance when running against long vectors, triggering concurrent requests to object store.

q)// -s 8
q)\t select max bid from quote where date=2018.09.07

q)// -s 0
q)\t select max bid from quote where date=2018.09.07


Due to the cost of storage, possible egress costs, high-latency and low bandwidth, we recommend storing data on cloud object storage using compression.