Send Feedback
Skip to content

Object Store Reference

This page documents URIs, environment variables, APIs, and configuration for the objstor module.

URI schema

Object storage is accessed using URI-style prefixes that identify the provider:

  • :s3:// — Amazon S3 buckets
  • :ms:// — Microsoft Azure Blob Storage
  • :gs:// — Google Cloud Storage

The prefix tells KDB-X which cloud provider and authentication method to use.

Note

S3-compatible services (such as MinIO) are also supported by combining the :s3:// prefix with a custom endpoint.

.ojbstor.init

This API will initialize values based on the environment variables set. If no parameter it passed, it'll attempt to register with each of the 3 cloud vendors: AWS, Azure and GCP

Input parameters

Name Type Description Required Default
vendor list Options include aws, azr, gcp, none or leave empty to register with 3 cloud vendors No All

As a prerequisite, you must have the appropriate envrionment variables defined.

To register with AWS, you can define the environment variables below, or populate your credentials file. Typically this is ~/.aws/credentials

export AWS_ACCESS_KEY_ID="… "
export AWS_SECRET_ACCESS_KEY="…"
export AWS_SESSION_TOKEN="… "

export AZURE_STORAGE_SHARED_KEY=".."
export AZURE_STORAGE_ACCOUNT=".."

gcloud init
export GCP_TOKEN=$(gcloud auth print-access-token)

Examples

.objstor.init[]; // register with AWS, Azure and GCP
.objstor.init`aws;
.objstor.init`none;

Environment variables

KX_TRACE_OBJSTR

The URIs requested to the cloud are printed to STDERR if the following environment variable is set. e.g.

export KX_TRACE_OBJSTR=1
KX_OBJSTR_INVENTORY_FILE

Location of optional inventory file

Although the Kurl module can detect most of the necessary components for credentials from the environment, the following additional environment variables are required.

AWS_REGION

If a region is not selected, then us-east-1 is used by default.

KX_S3_USE_PATH_REQUEST_STYLE

Alters the URL that is constructed for requests to S3 compatible storage.

By default, virtual host style is used. Virtual host style uses the bucket name to construct the hostname from which data is requested. The virtual host style uses the https://[bucket-name].s3.[region].amazonaws.com/[key-name] format for S3. Setting KX_S3_USE_PATH_REQUEST_STYLE=1 changes to path style, using the format https://s3.[region].amazonaws.com/[bucket-name]/[key-name] where all requests go to the same host. This is often used in S3-compatible object storage that are alternatives to Amazon S3 (for example, MinIO).

KX_S3_ENDPOINT

Changes the endpoint URL used for requests to S3 compatible storage, for example KX_S3_ENDPOINT=https://play.min.io:9000. When used with KX_S3_USE_PATH_REQUEST_STYLE=1, requests use the format [KX_S3_ENDPOINT]/[bucket-name]/[key-name]. This is often used in S3-compatible object storage that are alternatives to Amazon S3 (for example, MinIO).

AZURE_STORAGE_ACCOUNT

The DNS prefix for your storage account; e.g. for mystorage.blob.core.windows.net the name would be mystorage. The list of your storage accounts can be displayed using the Azure CLI tool az:

az storage account list | jq -r '.[] | .name'
GCLOUD_PROJECT_ID

A unique, user-assigned ID that can be used as the request header x-goog-project-id in Google APIs. It is used as a request header that specifies which project you are working on. It may be any valid project number or name. This request header tells Cloud Storage which project to create a bucket in or which project to list buckets for. Examples:

000111222333
my-project-name
example.com:my-google-apps-for-work-project-name

The list of your projects can be displayed using the Google CLI tool, Gcloud, via

gcloud projects list

Next steps

  • Refer to Examples for working objstor examples