Running Storage Manager

The Storage Manager service (SM) is a stateful application, which calln be run in Docker using Docker-Compose, or in Kuberenetes as a StatefulSet.

SM uses the kxi-sm-single image. The following images are optional:

purpose	images
SM discoverable by DAP processes	kxi-sidecar discovery_proxy kxi-eureka-discovery
Collect monitoring statistics	kxi-sidecar prom/prometheus

The examples provided here are for running with

minimal configuration: neither Service Discovery nor monitoring required
full discovery and monitoring integration

Prerequisites

Kdb+ license

A valid kdb+ license is required to run Storage Manager (SM). In the examples in this page, we provide it by mapping the /opt/kx/lic container path to a host mount which contains the license.

Message bus

SM requires a message bus compatible with Reliable Transport. The examples below assume a tickerplant server running on the host machine using the following command:

q tick.q sym /tmp/tplogs -p 5010

Configuration

SM is configured through a set of environment variables. The requirements vary between containers (described below), but at minimum, the name (KXI_NAME). service class (KXI_SC), and the path to the data-pipeline assembly (KXI_ASSEMBLY_FILE) assembly configuration are required.

The assembly file contains the following required information:

database schema
message bus
database mount and tier configuration

The examples below use this assembly configuration:

name: Example assembly
description: Sample assembly file

# Database schema
tables:
  trade:
    description: Trade data
    type: partitioned
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    sortColsDisk: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: price
        description: Trade price
        type: float
      - name: size
        description: Trade size
        type: long

  quote:
    description: Quote data
    type: partitioned
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    softColsDisk: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: bid
        description: Bid price
        type: float
      - name: ask
        description: Ask price
        type: float
      - name: bidSize
        description: Big size
        type: long
      - name: askSize
        description: Ask size
        type: long

# Message bus configuration
bus:
  stream:
    protocol: custom
    nodes: ${host_ip}:5010
    topic: dataStream

# Database mount configuration
mounts:
  rdb:
    type: stream
    baseURI: file:///mnt/foo/i/
    partition: none
  idb:
    type: local
    baseURI: file:///data/db/idb
    partition: ordinal
  hdb:
    type: local
    baseURI: file:///data/db/hdb
    partition: date

# Service configuration
elements:

  # SM configuration (also used by EOI, EOD, and DBM)
  sm:
    description: Storage manager
    source: messages
    # Tier configuration
    tiers:
      - name: stream
        mount: rdb
      - name: idb
        mount: idb
        schedule:
          freq: 0D00:10:00 # every 10 minutes
      - name: hdb1
        mount: hdb
        schedule:
          freq: 1D00:00:00 # every day
          snap:   01:35:00 # at 1:35 AM
        retain:
          time: 2 days
          rows: 200000
      - name: hdb2
        mount: hdb
        store: file:///data/db/hdbtier2
        retain:
          time: 5 weeks
          size: 2 TB
          rows: 10000000
      - name: hdb3
        mount: hdb
        store: file:///data/db/hdbtier3
        retain:
          time: 3 months
          size: 1 PB
          rows: 20000000
    disableDiscovery: true  # Disables registering with discovery

The variable ${host_ip} represents the address of the message bus server. For the examples on this page, replace the variable with the public address of the host machine of the tickerplant server.

The first run

The first run of SM is important, because it is when it determines whether it needs to create a virgin on-disk database, or imports a pre-existing standard Kdb+ partitioned database to SM's format. See initial-import.md for more details.

To restore the state before the first SM run

Shut down SM and delete the contents of the /data mount volume.

Running with minimal configuration

To run with the minimal set of images and configuration; i.e. neither automatic discovery, nor performance metric collection is required

The directory structure:

├── db/
├── docker-compose.yaml
└── cfg/
   └── assembly_sample.yaml

The cfg sub-directory contains the data-pipeline assembly configuration file. It is accessed by the containers through the /cfg bind mount.

cfg/assembly_sample.yaml is the assembly configuration file.

The db sub-directory is where the database is written. It is accessed by the containers through the /data bind mount. This directory requires read-write access by other users (chmod o+rw db).

docker-compose.yaml:

version: "3.0"
services:
  sm:
    image: "registry.dl.kx.com/kxi-sm-single:1.1.0"
    environment:
      - KXI_NAME=sm
      - KXI_SC=SM
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_RT_LIB=/opt/kx/app/cfg/rt_tick_client_lib.q
      - KXI_PORT=10001
      - KXI_LOG_FORMAT=text
    ports:
      - 10001:10001
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

networks:
  kx:

Environment variables

SM requires certain environment variables in the containers.

variable	required	description
`KXI_NAME`	yes	Process name
`KXI_SC`	yes	Service class
`KXI_PORT`	yes	Port
`KXI_ASSEMBLY_FILE`	yes	Data-pipeline assembly configuration file
`KXI_RT_LIB`	yes¹	Path to Reliable-Transport client-side q module
`KXI_SM_SMADDR`	yes	SM container’s address for inter-container communication
`KXI_SM_EOIADDR`	yes	EOI container’s address for inter-container communication
`KXI_RT_SM_LOG_PATH`		Specifies the path to the logs for the SM process. (e.g., `"/logs/rt/sm"`)
`KXI_RT_EOI_LOG_PATH`		Specifies the path to the logs for the EOI process. (e.g., `"/logs/rt/eoi"`)
`KXI_SM_EOI_THREADS`		Thread count for the EOI process. (e.g., `"8"`)
`KXI_SM_EOD_THREADS`		Thread count for the EOD process. (e.g., `"8"`)
`KXI_SM_DBM_THREADS`		Thread count for the DBM process. (e.g., `"8"`)
`AWS_ACCESS_KEY_ID`	yes²	AWS access key
`AWS_SECRET_ACCESS_KEY`	yes²	AWS secret key associated with the access key
`AWS_REGION`	yes²	AWS region
`AZURE_STORAGE_ACCOUNT`	yes³	Azure storage account name
`AZURE_STORAGE_SHARED_KEY`	yes³	Azure storage key

In addition, the following environment variables apply to both the sidecar and SM images.

variable	required	container	description
`KXI_CONFIG_FILE`	yes	sidecar	discovery configuration file
`KXI_LOG_FORMAT`		ALL	log message format
`KXI_LOG_DEST`		ALL	log endpoints
`KXI_LOG_LEVELS`		ALL	component routing
`KXI_LOG_CONFIG`		ALL	alternative logging configuration: replaces `KXI_LOG_FORMAT`, `KXI_LOG_DEST`, and `KXI_LOG_LEVELS`

Volumes

SM requires the following volumes mounted:

volume	description
/data	database volume
/tplogs	tickerplant log location†
/cfg	volume containing configuration files
/opt/kx/lic	kdb+ license file

† SM receives the path to tickerplant log as a result of a call to the tickerplant server; so for SM to access the same location, the mounted volume must match that seen by the tickerplant server.

Running

To run SM:

docker compose up

Running SM with discovery and monitoring integration

To run SM with discovery and monitoring integration, modify cfg/assembly_sample.yaml by changing

    disableDiscovery: true # Disables registering with discovery

to

    disableDiscovery: false # Disables registering with discovery

Then modify docker-compose.yaml by adding services for the remaining images:

  sm-sidecar:
    image: registry.dl.kx.com/kxi-sidecar:0.13.1
    command: -p 8080
    ports:
      - 11001:8080
    environment:
      - KXI_CONFIG_FILE=/cfg/sm-sidecar.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  prom:
    image: prom/prometheus
    command: --config.file=/cfg/prometheus.yml
    ports:
      - 9090:9090
    volumes:
      - ./cfg:/cfg
    networks:
      - kx

  proxy:
    image: registry.dl.kx.com/kxi-discovery-proxy:0.13.1
    command: -p 5000
    ports:
      - 5000:5000
    environment:
      - KXI_CONFIG_FILE=/cfg/proxy.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  eureka:
    image: registry.dl.kx.com/kxi-eureka-discovery:1.0.6
    ports:
      - 9000:8761
    networks:
      - kx

The directory structure is now:

├── docker-compose.yaml
├── db/
└── cfg/
   ├── assembly_sample.yaml
   ├── prometheus.yml
   ├── proxy.json
   ├── qlog.json
   └── sm-sidecar.json

We added the following configuration files:

`cfg/proxy.json`

The configuration file for the Service Discovery proxy:

{
    "discovery":
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

`cfg/qlog.json`

The QLog configuration file. This is an alternative (and recommended) way to configure logging for the container.

{
  "endpoints": [ "fd://stdout", "fd://stderr" ],
  "formatMode": "text",
  "routings": {
    "ALL": "INFO",
    "qlog": "DEBUG"
  }
}

`cfg/prometheus.yml`

Prometheus configuration:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'sm-monitoring'
    static_configs:
      - targets: ['sm:10001']
  - job_name: 'eoi-monitoring'
    static_configs:
      - targets: ['eoi:10002']

Monitoring

cfg/sm-sidecar.json

The configuration file for the sidecar:

{
    "connection": ":sm:10001",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:5000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 30
    },

    "metrics":
    {
      "enabled": "true",
      "frequency": 5,
        "handler":
        {
            "pc": false,
            "pg": false,
            "ph": false,
            "po": false,
            "pp": false,
            "ps": false,
            "ts": false,
            "wc": false,
            "wo": false,
            "ws": false
        }
    }
}

Note that it contains both Discovery and Monitoring configurations.

See "kxi-sidecar" page for more details.

Running

As in the first example, run from the command shell.

docker compose up

Required when using a message bus of type custom. ↩
Required when using object storage on AWS. ↩↩↩
Required when using object storage on AZURE. ↩↩