Skip to content

Running

In its runnable form, Storage Manager microservice (SM) comprises the following set of Docker images: kxi-sm, kxi-sm-eoi, kxi-sm-eod, and kxi-sm-dbm. Docker compose is typically used to collectively run the corresponding containers; within the same network and sharing the same volume (the database volume).

There are other Docker images whose usage is optional, and depends on:

  • Whether it is desired for SM to be discoverable by DAP processes, in which case the following images are required: kxi_sidecar, discovery_proxy, and kxi-eureka-discovery.

  • Whether it is desired to collect monitoring statistics, in which case the following images are required: kxi_sidecar, and prom/prometheus.

This document shows 2 examples, for the following 2 scenarios: - Running with minimal configuration (neither discovery, nor monitoring is required). - Running with full discovery and monitoring integration.

Prerequisites

KDB License

A valid KDB+ license is required to run SM. In the examples in this page, we provide it by mapping the /opt/kx/lic container path to a host mount which contains the license.

Message Bus

SM requires a message bus compatible with Reliable Transport. See rt.md for more information, and example describing how to employ a Kdb+ ticker plant. The examples below assume there is a tickerplant server running on the host machine using the following command:

q tick.q sym /tmp/tplogs -p 5010

Assembly Configuration

SM requires the path to the data-pipeline assembly configuration. See Configuration.md for more details. The examples below assume an assembly configuration with the following content:

name: Sample
description: Sample assembly file

# Database schema
tables:
  trade:
    description: Trade data
    type: partitioned
    shards: 11
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: price
        description: Trade price
        type: float
      - name: size
        description: Trade size
        type: long

  quote:
    description: Quote data
    type: partitioned
    shards: 11
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Partitioning timestamp column
        type: timestamp
      - name: bid
        description: Bid price
        type: float
      - name: ask
        description: Ask price
        type: float
      - name: bidSize
        description: Big size
        type: long
      - name: askSize
        description: Ask size
        type: long

# Message bus configuration
bus:
  stream:
    protocol: custom
    nodes: ${host_ip}:5010
    topic: dataStream

# Database mount configuration
mounts:
  rdb:
    type: stream
    uri: file://mnt/foo/i/
    partition: none
  idb:
    type: local
    uri: file://data/db/idb/current/
    partition: ordinal
  hdb:
    type: local
    uri: file://data/db/hdb/current/
    partition: date

# Service configuration
elements:

  # SM configuration (also used by EOI, EOD, and DBM)
  sm:
    description: Storage manager
    source: messages
    # Tier configuration
    tiers:
      - name: stream
        mount: rdb
      - name: idb
        mount: idb
        store: file://data/db/idb/data/
        schedule:
          freq: 0D00:10:00 # every 10 minutes
      - name: hdb1
        mount: hdb
        store: file://data/db/hdb/data/
        schedule:
          freq: 1D00:00:00 # every day
          snap:   01:35:00 # at 1:35 AM
        retain:
          time: 2 days
          rows: 200000
      - name: hdb2
        mount: hdb
        store: file://data/db/hdbtier2/
        retain:
          time: 5 weeks
          size: 2 TB
          rows: 10000000
      - name: hdb3
        mount: hdb
        store: file://data/db/hdbtier3/
        retain:
          time: 3 months
          size: 1 PB
          rows: 20000000
    disableDiscovery: true  # Disables registering with discovery

Note the variable ${host_ip} represents the address of the message bus server. For the examples on this page, replace the variable with the public address of the host machine (where tickerplant server was run).

The first run

The first run of SM is important, because it is when SM determines whether it needs to create a virgin on-disk database (based on the schema defined in the assembly file), or converts a pre-existing standard Kdb+ partitioned database (that matches the schema defined in the assembly file). If the desire is to start with an existing Kdb+ partitioned database, then it must: - Be a standard KDB+ partitioned database. - Be date-partitioned. This is a temporary limitation that will be removed in a future release. - Data is rooted at location pointed to by the first HDB-based tier, that is the first tier whose mount has type:local, and partition:date. - Data location (mentioned above) is not at or under /data/db/hdb directory. This is a temporary limitation that will be removed in a future release.

It is always recommended to have a backup copy of the to-be-converted data, but it is currently essential as the rollback logic (if something goes wrong during conversion) is not yet implemented (will be in a future version).

To get back to the state before the first SM run: bring SM down, then delete the contents of /data mount volume.

Running SM with minimal configuration

This example assumes that we want to run with the minimal set of images and configuration (neither automatic discovery, nor performance metric collection is required).

The directory structure looks like the following:

.
├── cfg
│   └── assembly_sample.yaml
├── db
└── docker-compose.yaml

cfg sub-directory contains the data-pipeline assembly configuration file. It is accessed by the containers through the /cfg bind mount.

cfg/assembly_sample.yaml is the assembly configuration file (shown above).

db sub-directory is where the database is written. It is accessed by the containers through the /data bind mount. Note that this directory requires read-write access by other users (chmod o+rw db).

The docker-compose.yaml file has the following content:

version: "3.0"
services:
  sm:
    image: "gcr.io/cloudpak/kxi-sm:0.8.0"
    environment:
      - KXI_NAME=sm
      - KXI_SC=SM
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_RT_LIB=/opt/kx/app/cfg/rt_tick_client_lib.q
      - KXI_PORT=10001
      - KXI_LOG_FORMAT=text
    ports: 
      - 10001:10001
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

  eoi:
    image: "gcr.io/cloudpak/kxi-sm-eoi:0.8.0"
    environment:
      - KXI_NAME=eoi
      - KXI_SC=EOI
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_RT_LIB=/opt/kx/app/cfg/rt_tick_client_lib.q
      - KXI_PORT=10002
      - KXI_LOG_FORMAT=text
      - KXI_SM_SMADDR=sm:10001
    ports:
      - 10002:10002
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

  eod:
    image: "gcr.io/cloudpak/kxi-sm-eod:0.8.0"
    environment:
      - KXI_NAME=eod
      - KXI_SC=EOD
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_PORT=10003
      - KXI_LOG_FORMAT=text
      - KXI_SM_SMADDR=sm:10001
      - KXI_SM_EOIADDR=eoi:10002
    ports:
      - 10003:10003
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

  dbm:
    image: "gcr.io/cloudpak/kxi-sm-dbm:0.8.0"
    environment:
      - KXI_NAME=dbm
      - KXI_SC=DBM
      - KXI_ASSEMBLY_FILE=/cfg/assembly_sample.yaml
      - KXI_PORT=10004
      - KXI_LOG_FORMAT=text
      - KXI_SM_SMADDR=sm:10001
    ports:
      - 10004:10004
    volumes:
      - ./db:/data
      - /tmp/tplogs:/tmp/tplogs
      - ./cfg:/cfg
      - ~/.qp.licenses:/opt/kx/lic
    networks:
      - kx
    stdin_open: true
    tty: true

networks:
  kx:

Environment variables

SM requires certain environment variables to be defined in the containers. The following table shows both required and optional variables, along with their applicable containers:

variable required containers description
KXI_NAME Yes SM, EOI, EOD, DBM Process name.
KXI_SC Yes SM, EOI, EOD, DBM Service class.
KXI_PORT Yes SM, EOI, EOD, DBM Port.
KXI_ASSEMBLY_FILE Yes SM, EOI, EOD, DBM Data-pipeline assembly configuration file.
KXI_RT_LIB Yes SM, EOI path to Reliable-Transport client-side q module.
KXI_SM_SMADDR Yes EOI, EOD, DBM SM container's address for inter-container communication.
KXI_SM_EOIADDR Yes EOD EOI container's address for inter-container communication.
KXI_LOG_FORMAT No ALL Log message format (see "Logging" page).
KXI_LOG_DEST No ALL Log endpoints (see "Logging" page).
KXI_LOG_LEVELS No ALL Component routing (see "Logging" page).
KXI_LOG_CONFIG No ALL Alternative logging configuration. This replaces KXI_LOG_FORMAT, KXI_LOG_DEST, and KXI_LOG_LEVELS. (see "Logging" page).
KXI_CONFIG_FILE Yes sidecar Discovery configuration file (see "Discovery" page).

Volumes

SM requires the following volumes to be mounted: | volume | containers | description | |-------------------|-------------------------------|-------------------------------------------------------------------| | /data | SM, EOI, EOD, DBM | Database volume. | | /tplogs | SM, EOI | tickerplant log location. (see note below). | | /cfg | SM, EOI, EOD, DBM | Volume containing configuration files. | | /opt/kx/lic | ALL | Kdb+ license file. |

Note: SM receives the path to tickerplant log as a result of a call to the tickerplant server, thus in order for SM to access the same location; the mounted volume must match that as seen by the tickerplant server.

Running

SM is run using the following command:

docker compose up

Running SM with discovery and monitoring integration

This example assumes that we want to run SM with discovery and monitoring integration.

Modify cfg/assembly_sample.yaml by changing:

    disableDiscovery: true # Disables registering with discovery

to

    disableDiscovery: false # Disables registering with discovery

Then modify docker-compose.yaml by adding services for the remaining images:

  sm-sidecar:
    image: gcr.io/cloudpak/kxi_sidecar:0.1.1
    command: -p 8080
    ports:
      - 11001:8080
    environment:
      - KXI_CONFIG_FILE=/cfg/sm-sidecar.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  prom:
    image: prom/prometheus
    command: --config.file=/cfg/prometheus.yml
    ports:
      - 9090:9090
    volumes:
      - ./cfg:/cfg
    networks:
      - kx

  proxy:
    image: gcr.io/cloudpak/discovery_proxy:0.1.0
    command: -p 5000
    ports:
      - 5000:5000
    environment:
      - KXI_CONFIG_FILE=/cfg/proxy.json
      - KXI_LOG_CONFIG=/cfg/qlog.json
    volumes:
      - ~/.qp.licenses:/opt/kx/lic
      - ./cfg:/cfg
    networks:
      - kx
    stdin_open: true
    tty: true

  eureka:
    image: gcr.io/cloudpak/kxi-eureka-discovery:0.1.0
    ports:
      - 9000:8761
    networks:
      - kx

The directory structure now looks like the following:

.
├── cfg
│   ├── assembly_sample.yaml
│   ├── prometheus.yml
│   ├── proxy.json
│   ├── qlog.json
│   └── sm-sidecar.json
├── db
└── docker-compose.yaml

We added the following configuration files:

cfg/proxy.json

The configuration file for discovery proxy, with the following content:

{
    "discovery": 
    {
        "registry": ":eureka:8761",
        "adaptor": "discEurekaAdaptor.q"
    }
}

See "Discovery" for more details.

cfg/qlog.json

The qlog configuration file. Note that this is an alternative (and recommended) way of specifying logging configuration for the container. The file has the following content:

{
  "endpoints": [ "fd://stdout", "fd://stderr" ],
  "formatMode": "text",
  "routings": {
    "ALL": "INFO",
    "qlog": "DEBUG"
  }
}

See "Logging" for more details.

cfg/prometheus.yml

The configuration file used by Prometheus. The file has the following content:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'sm-monitoring'
    static_configs:
      - targets: ['sm:10001']
  - job_name: 'eoi-monitoring'
    static_configs:
      - targets: ['eoi:10002']

See "Monitoring" for more details.

cfg/sm-sidecar.json

The configuration file for the sidecar. The file has the following content:

{
    "connection": ":sm:10001",
    "frequencySecs": 5,
    "discovery": 
    {
        "registry": ":proxy:5000",
        "adaptor": "discEurekaAdaptor.q",
        "heartbeatSecs": 30,
        "leaseExpirySecs": 30
    },

    "metrics":
    {
        "enabled": "true",
        "frequency": 5,
        "handler":
        {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
        }
    }
}

Note how it contains both discovery and monitoring configuration.

See "kxi-sidecar" page for more details.

Running

Same as first example, we run using the following command:

docker compose up