# Object Storage Module

_This page explains what the Object Storage module is and when to use it._

The Object Storage (`objstor`) module allows KDB-X processes to read data directly from cloud storage as if they were local files.

It provides a compute–storage separation, enabling scalable query access to large datasets without keeping them on block storage.

## Use cases

Common use cases for the Object Storage module include:

- Storing historical HDB data in cloud storage while retaining query access.  
- Reducing storage costs by offloading infrequently accessed data.  
- Disaster recovery and archival.  
- Hybrid deployments: combining block storage with cloud object storage.  

## Cloud providers

KX support major clouds providers like:

- Amazon Web Services (AWS)
- Microsoft Azure
- Google Cloud Provider (GCP)

### S3-compatible stores

In addition to the major cloud providers, the module supports **S3-compatible object stores** such as MinIO.  
These can be used for testing locally or in hybrid deployments.

Refer to the [examples](examples.md) for a complete MinIO walkthrough.

## Key concepts

Understanding these concepts helps you work effectively with object storage in KDB-X.

- **URI prefixes**: `:s3://`, `:ms://`, `:gs://`  
- **Authentication**: Integrated with **Kurl** (or inline credentials in simple cases)  
- **Inventory files**: JSON describing bucket contents for faster metadata loading  
- **Hybrid HDBs**: combine cloud and block storage paths in `par.txt`

Refer to the [quickstart](quickstart.md) and [reference](reference.md) pages for more details.

## Read only

The `objstor` module is **read-only**. The objects must be created using the cloud vendor’s standard CLI tooling to copy data from block storage to the cloud. For example:

- **Amazon Web Services (AWS)**
  ```
  aws s3 cp "/path/to/file.txt" s3://kxs-prd-cxt-twg-roinsightsdemo/kxinsights-marketplace-data/ --recursive
  ```

- **Microsoft Azure**
  ```
  azcopy cp "/path/to/file.txt" "https://[account].blob.core.windows.net/[container]/[path/to/blob]"
  ```

- **Google Cloud Platform (GCP)**
  ```
  gsutil cp -r "/path/to/file.txt" gs://kxinsights-marketplace-data/
  ```

## Metadata in object storage

Metadata is important because queries can't run until metadata is loaded.

When querying data in cloud object storage, **metadata** refers to the list of objects (keys) within a bucket, along with their sizes. 

This metadata must be loaded before queries can run, since it tells KDB-X which partitions and files are available.

### How metadata is loaded

- On **first access**, the module retrieves and caches the list of all keys under the requested directory in the bucket.

- This cache remains in memory to avoid repeated lookups.

- To force a refresh of the metadata, append `/_` to the bucket path:
  ```q
  / AWS
  q)key`:s3://mybucket/_

  / Azure
  q)key`:ms://mycontainer/_

  / GCP
  q)key`:gs://mybucket/_
  ```

## Performance

Compared to block storage, cloud object storage has higher latency and lower bandwidth. This means queries can be slower, especially when scanning large buckets. KDB-X provides several features like caching, secondary threads, compression, and inventory files to help mitigate these challenges.

Refer to the [performance considerations](quickstart.md#performance-considerations) for more details.

## Next steps

- Refer to the [quick start guide](quickstart.md) to get started with object storage.
- Check the [Object Storage Release Notes](./release-notes/changelog.md) for version history and fixes.
