Skip to content

Running Storage Manager

In its runnable form, the Storage Manager microservice (SM) comprises the Docker images: kxi-sm, kxi-sm-eoi, kxi-sm-eod, and kxi-sm-dbm. Docker Compose is typically used to collectively run the corresponding containers; within the same network and sharing the same volume (the database volume).

Other Docker images are optional:

purpose images
SM discoverable by DAP processes kxi_sidecar
discovery_proxy
kxi-eureka-discovery
Collect monitoring statistics kxi_sidecar
prom/prometheus

The examples are for running with

Prerequisites

Kdb+ license

A valid kdb+ license is required to run Storage Manager (SM). In the examples in this page, we provide it by mapping the /opt/kx/lic container path to a host mount which contains the license.

Message bus

SM requires a message bus compatible with Reliable Transport. The examples below assume a tickerplant server running on the host machine using the following command:

q tick.q sym /tmp/tplogs -p 5010

Assembly configuration

SM requires the path to the data-pipeline assembly configuration. The examples below assume this assembly configuration:

name: Sample
description: Sample assembly file

# Database schema
tables:
  trade:
    description: Trade data
    type: partitioned
    shards: 11
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: price
        description: Trade price
        type: float
      - name: size
        description: Trade size
        type: long

  quote:
    description: Quote data
    type: partitioned
    shards: 11
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: bid
        description: Bid price
        type: float
      - name: ask
        description: Ask price
        type: float
      - name: bidSize
        description: Big size
        type: long
      - name: askSize
        description: Ask size
        type: long

# Message bus configuration
bus:
  stream:
    protocol: custom
    nodes: ${host_ip}:5010
    topic: dataStream

# Database mount configuration
mounts:
  rdb:
    type: stream
    uri: file://mnt/foo/i/
    partition: none
  idb:
    type: local
    uri: file://data/db/idb/current/
    partition: ordinal
  hdb:
    type: local
    uri: file://data/db/hdb/current/
    partition: date

# Service configuration
elements:

  # SM configuration (also used by EOI, EOD, and DBM)
  sm:
    description: Storage manager
    source: messages
    # Tier configuration
    tiers:
      - name: stream
        mount: rdb
      - name: idb
        mount: idb
        store: file://data/db/idb/data/
        schedule:
          freq: 0D00:10:00 # every 10 minutes
      - name: hdb1
        mount: hdb
        store: file://data/db/hdb/data/
        schedule:
          freq: 1D00:00:00 # every day
          snap:   01:35:00 # at 1:35 AM
        retain:
          time: 2 days
          rows: 200000
      - name: hdb2
        mount: hdb
        store: file://data/db/hdbtier2/
        retain:
          time: 5 weeks
          size: 2 TB
          rows: 10000000
      - name: hdb3
        mount: hdb
        store: file://data/db/hdbtier3/
        retain:
          time: 3 months
          size: 1 PB
          rows: 20000000
    disableDiscovery: true  # Disables registering with discovery

The variable ${host_ip} represents the address of the message bus server. For the examples on this page, replace the variable with the public address of the host machine of the tickerplant server.

The first run

The first run of SM is important, because it is when it determines whether it needs to create a virgin on-disk database, or imports a pre-existing standard Kdb+ partitioned database to SM's format. See initial-import.md for more details.

To restore the state before the first SM run

Shut down SM and delete the contents of the /data mount volume.

Running with minimal configuration

To run with the minimal set of images and configuration; i.e. neither automatic discovery, nor performance metric collection is required

The directory structure:

├── db/
├── docker-compose.yaml
└── cfg/
   └── assembly_sample.yaml

The cfg sub-directory contains the data-pipeline assembly configuration file. It is accessed by the containers through the /cfg bind mount.

cfg/assembly_sample.yaml is the assembly configuration file.

The db sub-directory is where the database is written. It is accessed by the containers through the /data bind mount. This directory requires read-write access by other users (chmod o+rw db).

docker-compose.yaml:

version: "3.0"
services:
  sm:
    image: "gcr.io/cloudpak/kxi-sm:0.8.0"
    environment:
      - KXI_NAME=sm
      - KXI_SC=SM
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_RT_LIB=/opt/kx/app/cfg/rt_tick_client_lib.q
      - KXI_PORT=10001
      - KXI_LOG_FORMAT=text
    ports:
      - 10001:10001
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

  eoi:
    image: "gcr.io/cloudpak/kxi-sm-eoi:0.8.0"
    environment:
      - KXI_NAME=eoi
      - KXI_SC=EOI
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_RT_LIB=/opt/kx/app/cfg/rt_tick_client_lib.q
      - KXI_PORT=10002
      - KXI_LOG_FORMAT=text
      - KXI_SM_SMADDR=sm:10001
    ports:
      - 10002:10002
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

  eod:
    image: "gcr.io/cloudpak/kxi-sm-eod:0.8.0"
    environment:
      - KXI_NAME=eod
      - KXI_SC=EOD
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_PORT=10003
      - KXI_LOG_FORMAT=text
      - KXI_SM_SMADDR=sm:10001
      - KXI_SM_EOIADDR=eoi:10002
    ports:
      - 10003:10003
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

  dbm:
    image: "gcr.io/cloudpak/kxi-sm-dbm:0.8.0"
    environment:
      - KXI_NAME=dbm
      - KXI_SC=DBM
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_PORT=10004
      - KXI_LOG_FORMAT=text
      - KXI_SM_SMADDR=sm:10001
    ports:
      - 10004:10004
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

networks:
  kx:

Environment variables

SM requires certain environment variables in the containers.

variable reqd containers description
KXI_NAME yes SM, EOI, EOD, DBM process name
KXI_SC yes SM, EOI, EOD, DBM service class
KXI_PORT yes SM, EOI, EOD, DBM port
KXI_ASSEMBLY_FILE yes SM, EOI, EOD, DBM data-pipeline assembly configuration file
KXI_RT_LIB yes SM, EOI path to Reliable-Transport client-side q module
KXI_SM_SMADDR yes EOI, EOD, DBM SM container’s address for inter-container communication
KXI_SM_EOIADDR yes EOD EOI container’s address for inter-container communication
KXI_LOG_FORMAT ALL log message format
KXI_LOG_DEST ALL log endpoints
KXI_LOG_LEVELS ALL component routing
KXI_LOG_CONFIG ALL alternative logging configuration: replaces KXI_LOG_FORMAT, KXI_LOG_DEST, and KXI_LOG_LEVELS
KXI_CONFIG_FILE yes sidecar discovery configuration file
AWS_ACCESS_KEY_ID yes1 SM, EOI, EOD, DBM AWS access key
AWS_SECRET_ACCESS_KEY yes1 SM, EOI, EOD, DBM AWS secret key associated with the access key
AWS_REGION yes1 SM, EOI, EOD, DBM AWS region
AZURE_STORAGE_ACCOUNT yes2 SM, EOI, EOD, DBM Azure storage account name
AZURE_STORAGE_SHARED_KEY yes2 SM, EOI, EOD, DBM Azure storage key

Volumes

SM requires the following volumes mounted:

volume containers description
/data SM, EOI, EOD, DBM database volume
/tplogs SM, EOI tickerplant log location†
/cfg SM, EOI, EOD, DBM volume containing configuration files
/opt/kx/lic ALL kdb+ license file

† SM receives the path to tickerplant log as a result of a call to the tickerplant server; so for SM to access the same location, the mounted volume must match that seen by the tickerplant server.

Running

To run SM:

docker compose up

Running SM with discovery and monitoring integration

To run SM with discovery and monitoring integration, modify cfg/assembly_sample.yaml by changing

    disableDiscovery: true # Disables registering with discovery

to

    disableDiscovery: false # Disables registering with discovery

Then modify docker-compose.yaml by adding services for the remaining images:

  sm-sidecar:
    image: gcr.io/cloudpak/kxi_sidecar:0.1.1
    command: -p 8080
    ports:
      - 11001:8080
    environment:
      - KXI_CONFIG_FILE=/cfg/sm-sidecar.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  prom:
    image: prom/prometheus
    command: --config.file=/cfg/prometheus.yml
    ports:
      - 9090:9090
    volumes:
      - ./cfg:/cfg
    networks:
      - kx

  proxy:
    image: gcr.io/cloudpak/discovery_proxy:0.1.0
    command: -p 5000
    ports:
      - 5000:5000
    environment:
      - KXI_CONFIG_FILE=/cfg/proxy.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  eureka:
    image: gcr.io/cloudpak/kxi-eureka-discovery:0.1.0
    ports:
      - 9000:8761
    networks:
      - kx

The directory structure is now:

├── docker-compose.yaml
├── db/
└── cfg/
   ├── assembly_sample.yaml
   ├── prometheus.yml
   ├── proxy.json
   ├── qlog.json
   └── sm-sidecar.json

We added the following configuration files:

cfg/proxy.json

The configuration file for the Service Discovery proxy:

{
    "discovery":
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

cfg/qlog.json

The QLog configuration file. This is an alternative (and recommended) way to configure logging for the container.

{
  "endpoints": [ "fd://stdout", "fd://stderr" ],
  "formatMode": "text",
  "routings": {
    "ALL": "INFO",
    "qlog": "DEBUG"
  }
}

cfg/prometheus.yml

Prometheus configuration:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'sm-monitoring'
    static_configs:
      - targets: ['sm:10001']
  - job_name: 'eoi-monitoring'
    static_configs:
      - targets: ['eoi:10002']

Monitoring

cfg/sm-sidecar.json

The configuration file for the sidecar:

{
    "connection": ":sm:10001",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:5000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 30
    },

    "metrics":
    {
      "enabled": "true",
      "frequency": 5,
        "handler":
        {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
        }
    }
}

Note that it contains both Discovery and Monitoring configurations.

See "kxi-sidecar" page for more details.

Running

As in the first example, run from the command shell.

docker compose up

  1. Required when using object storage on AWS. 

  2. Required when using object storage on AZURE.