Skip to content

Running Storage Manager

The Storage Manager service (SM) is a stateful application, which calln be run in Docker using Docker-Compose, or in Kuberenetes as a StatefulSet.

SM uses the kxi-sm-single image. The following images are optional:

purpose images
SM discoverable by DAP processes kxi-sidecar
discovery_proxy
kxi-eureka-discovery
Collect monitoring statistics kxi-sidecar
prom/prometheus

The examples provided here are for running with

Prerequisites

Kdb+ license

A valid kdb+ license is required to run Storage Manager (SM). In the examples in this page, we provide it by mapping the /opt/kx/lic container path to a host mount which contains the license.

Message bus

SM requires a message bus compatible with Reliable Transport. The examples below assume a tickerplant server running on the host machine using the following command:

q tick.q sym /tmp/tplogs -p 5010

Configuration

SM is configured through a set of environment variables. The requirements vary between containers (described below), but at minimum, the name (KXI_NAME). service class (KXI_SC), and the path to the data-pipeline assembly (KXI_ASSEMBLY_FILE) assembly configuration are required.

The assembly file contains the following required information:

  • database schema
  • message bus
  • database mount and tier configuration

The examples below use this assembly configuration:

name: Example assembly
description: Sample assembly file

# Database schema
tables:
  trade:
    description: Trade data
    type: partitioned
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    sortColsDisk: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: price
        description: Trade price
        type: float
      - name: size
        description: Trade size
        type: long

  quote:
    description: Quote data
    type: partitioned
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    softColsDisk: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: bid
        description: Bid price
        type: float
      - name: ask
        description: Ask price
        type: float
      - name: bidSize
        description: Big size
        type: long
      - name: askSize
        description: Ask size
        type: long

# Message bus configuration
bus:
  stream:
    protocol: custom
    nodes: ${host_ip}:5010
    topic: dataStream

# Database mount configuration
mounts:
  rdb:
    type: stream
    baseURI: file:///mnt/foo/i/
    partition: none
  idb:
    type: local
    baseURI: file:///data/db/idb
    partition: ordinal
  hdb:
    type: local
    baseURI: file:///data/db/hdb
    partition: date

# Service configuration
elements:

  # SM configuration (also used by EOI, EOD, and DBM)
  sm:
    description: Storage manager
    source: messages
    # Tier configuration
    tiers:
      - name: stream
        mount: rdb
      - name: idb
        mount: idb
        schedule:
          freq: 0D00:10:00 # every 10 minutes
      - name: hdb1
        mount: hdb
        schedule:
          freq: 1D00:00:00 # every day
          snap:   01:35:00 # at 1:35 AM
        retain:
          time: 2 days
          rows: 200000
      - name: hdb2
        mount: hdb
        store: file:///data/db/hdbtier2
        retain:
          time: 5 weeks
          size: 2 TB
          rows: 10000000
      - name: hdb3
        mount: hdb
        store: file:///data/db/hdbtier3
        retain:
          time: 3 months
          size: 1 PB
          rows: 20000000
    disableDiscovery: true  # Disables registering with discovery

The variable ${host_ip} represents the address of the message bus server. For the examples on this page, replace the variable with the public address of the host machine of the tickerplant server.

The first run

The first run of SM is important, because it is when it determines whether it needs to create a virgin on-disk database, or imports a pre-existing standard Kdb+ partitioned database to SM's format. See initial-import.md for more details.

To restore the state before the first SM run

Shut down SM and delete the contents of the /data mount volume.

Running with minimal configuration

To run with the minimal set of images and configuration; i.e. neither automatic discovery, nor performance metric collection is required

The directory structure:

├── db/
├── docker-compose.yaml
└── cfg/
   └── assembly_sample.yaml

The cfg sub-directory contains the data-pipeline assembly configuration file. It is accessed by the containers through the /cfg bind mount.

cfg/assembly_sample.yaml is the assembly configuration file.

The db sub-directory is where the database is written. It is accessed by the containers through the /data bind mount. This directory requires read-write access by other users (chmod o+rw db).

docker-compose.yaml:

version: "3.0"
services:
  sm:
    image: "registry.dl.kx.com/kxi-sm-single:1.1.0"
    environment:
      - KXI_NAME=sm
      - KXI_SC=SM
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_RT_LIB=/opt/kx/app/cfg/rt_tick_client_lib.q
      - KXI_PORT=10001
      - KXI_LOG_FORMAT=text
    ports:
      - 10001:10001
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

networks:
  kx:

Environment variables

SM requires certain environment variables in the containers.

variable required description
KXI_NAME yes Process name
KXI_SC yes Service class
KXI_PORT yes Port
KXI_ASSEMBLY_FILE yes Data-pipeline assembly configuration file
KXI_RT_LIB yes1 Path to Reliable-Transport client-side q module
KXI_SM_SMADDR yes SM container’s address for inter-container communication
KXI_SM_EOIADDR yes EOI container’s address for inter-container communication
KXI_RT_SM_LOG_PATH Specifies the path to the logs for the SM process. (e.g., "/logs/rt/sm")
KXI_RT_EOI_LOG_PATH Specifies the path to the logs for the EOI process. (e.g., "/logs/rt/eoi")
KXI_SM_EOI_THREADS Thread count for the EOI process. (e.g., "8")
KXI_SM_EOD_THREADS Thread count for the EOD process. (e.g., "8")
KXI_SM_DBM_THREADS Thread count for the DBM process. (e.g., "8")
AWS_ACCESS_KEY_ID yes2 AWS access key
AWS_SECRET_ACCESS_KEY yes2 AWS secret key associated with the access key
AWS_REGION yes2 AWS region
AZURE_STORAGE_ACCOUNT yes3 Azure storage account name
AZURE_STORAGE_SHARED_KEY yes3 Azure storage key

In addition, the following environment variables apply to both the sidecar and SM images.

variable required container description
KXI_CONFIG_FILE yes sidecar discovery configuration file
KXI_LOG_FORMAT ALL log message format
KXI_LOG_DEST ALL log endpoints
KXI_LOG_LEVELS ALL component routing
KXI_LOG_CONFIG ALL alternative logging configuration: replaces KXI_LOG_FORMAT, KXI_LOG_DEST, and KXI_LOG_LEVELS

Volumes

SM requires the following volumes mounted:

volume description
/data database volume
/tplogs tickerplant log location†
/cfg volume containing configuration files
/opt/kx/lic kdb+ license file

† SM receives the path to tickerplant log as a result of a call to the tickerplant server; so for SM to access the same location, the mounted volume must match that seen by the tickerplant server.

Running

To run SM:

docker compose up

Running SM with discovery and monitoring integration

To run SM with discovery and monitoring integration, modify cfg/assembly_sample.yaml by changing

    disableDiscovery: true # Disables registering with discovery

to

    disableDiscovery: false # Disables registering with discovery

Then modify docker-compose.yaml by adding services for the remaining images:

  sm-sidecar:
    image: registry.dl.kx.com/kxi-sidecar:0.13.1
    command: -p 8080
    ports:
      - 11001:8080
    environment:
      - KXI_CONFIG_FILE=/cfg/sm-sidecar.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  prom:
    image: prom/prometheus
    command: --config.file=/cfg/prometheus.yml
    ports:
      - 9090:9090
    volumes:
      - ./cfg:/cfg
    networks:
      - kx

  proxy:
    image: registry.dl.kx.com/kxi-discovery-proxy:0.13.1
    command: -p 5000
    ports:
      - 5000:5000
    environment:
      - KXI_CONFIG_FILE=/cfg/proxy.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  eureka:
    image: registry.dl.kx.com/kxi-eureka-discovery:1.0.6
    ports:
      - 9000:8761
    networks:
      - kx

The directory structure is now:

├── docker-compose.yaml
├── db/
└── cfg/
   ├── assembly_sample.yaml
   ├── prometheus.yml
   ├── proxy.json
   ├── qlog.json
   └── sm-sidecar.json

We added the following configuration files:

cfg/proxy.json

The configuration file for the Service Discovery proxy:

{
    "discovery":
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

cfg/qlog.json

The QLog configuration file. This is an alternative (and recommended) way to configure logging for the container.

{
  "endpoints": [ "fd://stdout", "fd://stderr" ],
  "formatMode": "text",
  "routings": {
    "ALL": "INFO",
    "qlog": "DEBUG"
  }
}

cfg/prometheus.yml

Prometheus configuration:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'sm-monitoring'
    static_configs:
      - targets: ['sm:10001']
  - job_name: 'eoi-monitoring'
    static_configs:
      - targets: ['eoi:10002']

Monitoring

cfg/sm-sidecar.json

The configuration file for the sidecar:

{
    "connection": ":sm:10001",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:5000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 30
    },

    "metrics":
    {
      "enabled": "true",
      "frequency": 5,
        "handler":
        {
            "pc": false,
            "pg": false,
            "ph": false,
            "po": false,
            "pp": false,
            "ps": false,
            "ts": false,
            "wc": false,
            "wo": false,
            "ws": false
        }
    }
}

Note that it contains both Discovery and Monitoring configurations.

See "kxi-sidecar" page for more details.

Running

As in the first example, run from the command shell.

docker compose up

  1. Required when using a message bus of type custom

  2. Required when using object storage on AWS. 

  3. Required when using object storage on AZURE.