Skip to content

Running Storage Manager

The Storage Manager service (SM) is a stateful application, which can be run in Docker using Docker-Compose, or in Kuberenetes as a StatefulSet.

SM uses the kxi-sm-single image. The following images are optional:

purpose images
SM discoverable by DAP processes kxi-sidecar
discovery_proxy
kxi-eureka-discovery
Collect monitoring statistics kxi-sidecar
prom/prometheus

The examples provided here are for running with

Prerequisites

Kdb+ license

A valid kdb+ license is required to run Storage Manager (SM). In the examples in this page, we provide it by mapping the /opt/kx/lic container path to a host mount which contains the license.

Message bus

SM requires a message bus compatible with Reliable Transport. The examples below assume a tickerplant server running on the host machine using the following command:

q tick.q sym /tmp/tplogs -p 5010

Configuration

SM is configured through a set of environment variables and an assembly file. The requirements vary between containers (described below), but at minimum, the name (KXI_NAME). service class (KXI_SC), and the path to the data-pipeline assembly (KXI_ASSEMBLY_FILE) assembly configuration are required.

The assembly file contains the following required information:

  • database schema
  • message bus
  • database mount and tier configuration
  • service configuration, including SM

See the deployment example for a complete assembly example including SM.

The first run

The first run of SM is important, because it is when it determines whether it needs to create an initial on-disk database, or import a pre-existing standard kdb+ partitioned database to SM's format. See initial-import.md for more details.

To restore the state before the first SM run

Shut down SM and delete the contents of the /data mount volume.

Running with minimal configuration

See the deployment example for a minimal deployment of SM. Note, that deployment does also include Data Access Processes and the Service Gateway as a complete example, but those services can be omitted.

Environment variables

SM requires certain environment variables in the containers.

variable required description
KXI_NAME yes Process name
KXI_SC yes Service class
KXI_PORT yes Port
KXI_ASSEMBLY_FILE yes Data-pipeline assembly configuration file
KXI_RT_LIB yes1 Path to Reliable-Transport client-side q module
KXI_SM_SMADDR yes SM container’s address for inter-container communication
KXI_SM_EOIADDR yes EOI container’s address for inter-container communication
KXI_RT_SM_LOG_PATH Specifies the path to the logs for the SM process. (e.g., "/logs/rt/sm")
KXI_RT_EOI_LOG_PATH Specifies the path to the logs for the EOI process. (e.g., "/logs/rt/eoi")
KXI_SM_EOI_THREADS Thread count for the EOI process. (e.g., "8")
KXI_SM_EOD_THREADS Thread count for the EOD process. (e.g., "8")
KXI_SM_DBM_THREADS Thread count for the DBM process. (e.g., "8")
AWS_ACCESS_KEY_ID yes2 AWS access key
AWS_SECRET_ACCESS_KEY yes2 AWS secret key associated with the access key
AWS_REGION yes2 AWS region
AZURE_STORAGE_ACCOUNT yes3 Azure storage account name
AZURE_STORAGE_SHARED_KEY yes3 Azure storage key

In addition, the following environment variables apply to both the sidecar and SM images.

variable required container description
KXI_CONFIG_FILE yes sidecar discovery configuration file
KXI_LOG_FORMAT ALL log message format
KXI_LOG_DEST ALL log endpoints
KXI_LOG_LEVELS ALL component routing
KXI_LOG_CONFIG ALL alternative logging configuration: replaces KXI_LOG_FORMAT, KXI_LOG_DEST, and KXI_LOG_LEVELS

Volumes

SM requires the following volumes mounted:

volume description
/data database volume
/tplogs tickerplant log location†
/cfg volume containing configuration files
/opt/kx/lic kdb+ license file

† SM receives the path to tickerplant log as a result of a call to the tickerplant server; so for SM to access the same location, the mounted volume must match that seen by the tickerplant server.

Running SM with discovery and monitoring integration

To run SM with discovery and monitoring integration, modify cfg/assembly_sample.yaml by changing

    disableDiscovery: true # Disables registering with discovery

to

    disableDiscovery: false # Disables registering with discovery

Then modify docker-compose.yaml by adding services for the remaining images:

  sm-sidecar:
    image: registry.dl.kx.com/kxi-sidecar:0.13.1
    command: -p 8080
    ports:
      - 11001:8080
    environment:
      - KXI_CONFIG_FILE=/cfg/sm-sidecar.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  prom:
    image: prom/prometheus
    command: --config.file=/cfg/prometheus.yml
    ports:
      - 9090:9090
    volumes:
      - ./cfg:/cfg
    networks:
      - kx

  proxy:
    image: registry.dl.kx.com/kxi-discovery-proxy:0.13.1
    command: -p 5000
    ports:
      - 5000:5000
    environment:
      - KXI_CONFIG_FILE=/cfg/proxy.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  eureka:
    image: registry.dl.kx.com/kxi-eureka-discovery:1.0.6
    ports:
      - 9000:8761
    networks:
      - kx

The directory structure is now:

├── docker-compose.yaml
├── db/
└── cfg/
   ├── assembly_sample.yaml
   ├── prometheus.yml
   ├── proxy.json
   ├── qlog.json
   └── sm-sidecar.json

We added the following configuration files:

cfg/proxy.json

The configuration file for the Service Discovery proxy:

{
    "discovery":
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

cfg/qlog.json

The QLog configuration file. This is an alternative (and recommended) way to configure logging for the container.

{
  "endpoints": [ "fd://stdout", "fd://stderr" ],
  "formatMode": "text",
  "routings": {
    "ALL": "INFO",
    "qlog": "DEBUG"
  }
}

cfg/prometheus.yml

Prometheus configuration:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'sm-monitoring'
    static_configs:
      - targets: ['sm:10001']
  - job_name: 'eoi-monitoring'
    static_configs:
      - targets: ['eoi:10002']

Monitoring

cfg/sm-sidecar.json

The configuration file for the sidecar:

{
    "connection": ":sm:10001",
    "frequencySecs": 5,
    "discovery":
    {
        "registry": ":proxy:5000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 30
    },

    "metrics":
    {
      "enabled": "true",
      "frequency": 5,
        "handler":
        {
            "pc": false,
            "pg": false,
            "ph": false,
            "po": false,
            "pp": false,
            "ps": false,
            "ts": false,
            "wc": false,
            "wo": false,
            "ws": false
        }
    }
}

Note that it contains both Discovery and Monitoring configurations.

See "kxi-sidecar" page for more details.

Running

As in the first example, run from the command shell.

docker compose up

  1. Required when using a message bus of type custom

  2. Required when using object storage on AWS. 

  3. Required when using object storage on AZURE.