Skip to content

Docker metrics deployment

This page provides an example of a Docker deployment using the kxi-sidecar to provide Prometheus metric capture. This example builds on the basic Docker example.

Prerequisites

Pulling the images requires a login with:

docker login portal.dl.kx.com -u <username> -p <password>

Directory structure

Before running this example, the following directory structure should be created:

db/     # Empty directory where the database will be stored on disk
cfg/    # Directory for configuration files
    assembly.yaml
    da/
        config.json
        rdb-config.json
        idb-config.json
        hdb-config.json
    sm/
        config.json
kdb-tick/   # Clone of kdb-tick for a tickerplant service
    logs/
    tick/{r.q, u.q, sym.q}
    tick.q
.env
docker-compose-da.yaml
docker-compose-sm.yaml
docker-compose-tp.yaml

Write permissions

The db/ and kdb-tick/logs directory on the host must allow write permission by the SM and TP containers who run as the user "nobody".

  • kdb-tick/ is cloned from the KX Github.
  • The sym.q schema must include the _prtnEnd and _reload tables, as in:
kdb-tick/tick/sym.q
// internal tables
// with `time` and `sym` columns added by RT client for compatibility
(`$"_prtnEnd")set ([] time:"n"$(); sym:`$(); signal:`$(); endTS:"p"$(); opts:());
(`$"_reload")set ([] time:"n"$(); sym:`$(); mount:`$(); params:(); asm:`$())
(`$"_heartbeats")set ([] time:"n"$(); sym:`$(); foo:"j"$())
(`$"_batchIngest")set ([] time:"n"$(); sym: `$(); batchUpdType: `$(); session:`$(); address:`$(); callback:(); merge:"b"$(); datacheck:"b"$());
(`$"_batchDelete")set ([] time:"n"$(); sym: `$(); batchUpdType: `$(); session:`$(); address:`$(); callback:(); endTS:"p"$(); filter:(); startTS:"p"$(); table:`$());
(`$"_schemaChange")set ([] time:"n"$(); sym: `$(); batchUpdType: `$(); session:`$(); address:`$(); callback:(); changes:());

trade:([] time:"n"$(); sym:`$(); realTime:"p"$(); price:"f"$(); size:"j"$())
quote:([] time:"n"$(); sym:`$(); realTime:"p"$();
          bid:"f"$(); ask:"f"$(); bidSize:"j"$(); askSize:"j"$())

Run

To run, execute the following:

docker-compose \
    -f docker-compose-tp.yaml \
    -f docker-compose-da.yaml \
    -f docker-compose-sm.yaml \
    -f docker-compose-sg.yaml \
    up

Environment

Each of the Docker Compose files below use a .env file specifying the images and licenses to be used. Below, the RELEASE, QCE_RELEASE and SIDECAR_RELEASE environment variables are configured to point to the latest releases of kdb Insights Microservices and kdb Insights respectively. Additionally a license must be provided to run this example.

License

This example requires a license to run. See microservice prerequisites for details about getting a license.

.env
# Images
kxi_sg_gw=$REGISTRY/kxi-sg-gw:$RELEASE
kxi_sg_rc=$REGISTRY/kxi-sg-rc:$RELEASE
kxi_sg_agg=$REGISTRY/kxi-sg-agg:$RELEASE
kxi_sm_single=$REGISTRY/kxi-sm-single:$RELEASE
kxi_da_single=$REGISTRY/kxi-da-single:$RELEASE
kxi_sidecar=$REGISTRY/kxi-sidecar:$SIDECAR_RELEASE
kxi_q=$REGISTRY/qce:$QCE_RELEASE

# Paths
local_dir="."
mnt_dir="/mnt"
shared_dir="/mnt/shared"
cfg_dir="/mnt/cfg"
db_dir="/mnt/data/db"
logs_dir="/mnt/data/logs"

# Network
network_name=kx

Assembly file

The Assembly file is the main business configuration for the database. Table schemas and logical process configuration are defined here. See assembly configuration for more information.

cfg/assembly.yaml
name: fin-example
description: Data access assembly configuration
labels:
  region: New York
  assetClass: stocks

tables:
  _heartbeats:
    type: splayed_mem
    columns:
      - name: time
        type: timespan
      - name: sym
        type: symbol
      - name: foo
        type: float
  trade:
    description: Trade data
    type: partitioned
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    sortColsDisk: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Real timestamp
        type: timestamp
      - name: price
        description: Trade price
        type: float
      - name: size
        description: Trade size
        type: long

  quote:
    description: Quote data
    type: partitioned
    blockSize: 10000
    prtnCol: realTime
    sortColsOrd: sym
    sortColsDisk: sym
    columns:
      - name: time
        description: Time
        type: timespan
      - name: sym
        description: Symbol name
        type: symbol
        attrMem: grouped
        attrDisk: parted
        attrOrd: parted
      - name: realTime
        description: Real timestamp
        type: timestamp
      - name: bid
        description: Bid price
        type: float
      - name: ask
        description: Ask price
        type: float
      - name: bidSize
        description: Bid size
        type: long
      - name: askSize
        description: Ask size
        type: long

bus:
  stream:
    protocol: tp
    nodes: tp:5010
    topic: dataStream

mounts:
  rdb:
    type: stream
    baseURI: file://stream
    partition: none
  idb:
    type: local
    baseURI: file:///mnt/data/db/idb
    partition: ordinal
  hdb:
    type: local
    baseURI: file:///mnt/data/db/hdb
    partition: date

elements:
  dap:
    gwAssembly: gw-assembly
    smEndpoints: sm:20001
    instances:
      dap:
        mountList: [rdb, idb, hdb]

  sm:
    description: Storage manager
    source: stream
    tiers:
      - name: stream
        mount: rdb
      - name: idb
        mount: idb
        schedule:
          freq: 0D00:10:00 # every 10 minutes
      - name: hdb
        mount: hdb
        schedule:
          freq: 1D00:00:00 # every day
    disableDiscovery: true  # Disables registering with discovery

Docker compose

Each service of the database is deployed here as a separate Docker Compose file for clarity. Each of these could be combined into a single Docker Compose file instead.

Service Gateway

The Service Gateway configures the three containers that make up the gateway.

docker-compose-sg.yaml
networks:
  kx:
    name: ${network_name}

services:
  sgrc:
    image: ${kxi_sg_rc}
    environment:
      - KXI_NAME=sg_rc
      - KXI_PORT=5050
      - KXI_LOG_FORMAT=text
      - KXI_LOG_LEVELS=default:info
      - KDB_LICENSE_B64
    networks: [kx]
    volumes:
      - ${local_dir}:${mnt_dir}

  sgagg:
    image: ${kxi_sg_agg}
    environment:
      - KXI_NAME=sg_agg
      - KXI_PORT=5060
      - KXI_SG_RC_ADDR=sgrc:5050
      - KXI_LOG_FORMAT=text
      - KXI_LOG_LEVELS=default:info
      - KDB_LICENSE_B64
    deploy: # Optional: deploy multiple replicas.
      mode: replicated
      replicas: 1
    networks: [kx]
    volumes:
      - ${local_dir}:${mnt_dir}

  sggw:
    image: ${kxi_sg_gw}
    environment:
      - GATEWAY_QIPC_PORT=5040
      - GATEWAY_HTTP_PORT=8080
      - KXI_SG_RC_ADDR=sgrc:5050
      - KXI_LOG_FORMAT=text
      - KXI_LOG_LEVELS=default:info
    networks: [kx]
    deploy: # Optional: deploy multiple replicas.
      mode: replicated
      replicas: 1
    volumes:
      - ${local_dir}:${mnt_dir}

Data Access Processes

A set of Data Access Processes are configured, each of which connect to the Resource Coordinator of the Service Gateway launched above.

docker-compose-da.yaml
networks:
  kx:
    name: ${network_name}

services:
  dap:
    image: ${kxi_da_single}
    command: -p 5080
    environment:
      - KXI_NAME=dap
      - KXI_SC=dap
      - KXI_PORT=5080
      - KXI_ASSEMBLY_FILE=${cfg_dir}/assembly.yaml
      - KXI_SG_RC_ADDR=sgrc:5050
      - KXI_CONFIG_FILE=${cfg_dir}/da/config.json
      - KXI_LOG_CONFIG=${cfg_dir}/qlog.json
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    networks: [kx]
  rdb-sidecar:
    image: ${kxi_sidecar}
    environment:
      - KXI_CONFIG_FILE=${cfg_dir}/da/rdb-config.json
      - KXI_LOG_CONFIG=${cfg_dir}/qlog.json
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    command: -p 8080
    networks: [kx]
  idb-sidecar:
    image: ${kxi_sidecar}
    environment:
      - KXI_CONFIG_FILE=${cfg_dir}/da/idb-config.json
      - KXI_LOG_CONFIG=${cfg_dir}/qlog.json
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    command: -p 8080
    networks: [kx]
  hdb-sidecar:
    image: ${kxi_sidecar}
    environment:
      - KXI_CONFIG_FILE=${cfg_dir}/da/hdb-config.json
      - KXI_LOG_CONFIG=${cfg_dir}/qlog.json
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    command: -p 8080
    networks: [kx]

Storage Manager

The Storage Manager is configured as a single container, allowing connections by Data Access Processes configured above.

docker-compose-sm.yaml
networks:
  kx:
    name: ${network_name}

services:
  sm:
    image: ${kxi_sm_single}
    command: -p 20001
    environment:
      - KXI_NAME=sm
      - KXI_SC=SM
      - KXI_ASSEMBLY_FILE=${cfg_dir}/assembly.yaml
      - KXI_CONFIG_FILE=${cfg_dir}/sm/config.json
      - KXI_LOG_CONFIG=${cfg_dir}/qlog.json
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    networks: [kx]
  sm-sidecar:
    image: ${kxi_sidecar}
    environment:
      - KXI_CONFIG_FILE=${cfg_dir}/sm/config.json
      - KXI_LOG_CONFIG=${cfg_dir}/qlog.json
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    command: -p 8080
    networks: [kx]

Tickerplant

A standard tickerplant is put in front of the Storage Manager and Data Access Process to provide durable data ingestion.

Note

Within kdb Insights Enterprise, the transport used is kdb Insights Reliable Transport rather than a tickerplant. This allows for fault-tolerance and durable messaging despite potentially unreliable network connections. When using a standard tickerplant instead, the interface must adhere to the API expected by RT.

docker-compose-tp.yaml
networks:
  kx:
    name: ${network_name}

services:
  tp:
    image: ${kxi_q}
    command: tick.q sym ${logs_dir} -p 5010
    working_dir: ${shared_dir}/kdb-tick
    environment:
      - KDB_LICENSE_B64
    volumes:
      - ${local_dir}:${mnt_dir}
    networks: [kx]

Metrics configuration

Metrics configuration is stored in a config.json file for each of the respective processes. This configuration tells the sidecar where to scrape metrics from and what metrics to include. See the configurable metrics table for a list of available metrics.

Data Access Processes

cfg/da/config.json
{
    "connection": ":dap:5080",
    "frequencySecs": 5,
    "metrics":
    {
        "enabled":"true",
        "frequency": 5,
        "handler": {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
          }
    }
}
cfg/da/rdb-config.json
{
    "connection": ":dap:5081",
    "frequencySecs": 5,
    "metrics":
    {
        "enabled":"true",
        "frequency": 5,
        "handler": {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
          }
    }
}
cfg/da/idb-config.json
{
    "connection": ":dap:5082",
    "frequencySecs": 5,
    "metrics":
    {
        "enabled":"true",
        "frequency": 5,
        "handler": {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
          }
    }
}
cfg/da/hdb-config.json
{
    "connection": ":dap:5083",
    "frequencySecs": 5,
    "metrics":
    {
        "enabled":"true",
        "frequency": 5,
        "handler": {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
          }
    }
}

Storage Manager

cfg/sm/config.json
{
    "connection": ":sm:20001",
    "frequencySecs": 5,
    "metrics":
    {
        "enabled":"true",
        "frequency": 5,
        "handler": {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
          }
    }
}

Prometheus configuration

An additional Prometheus client has can be launched using the provided prometheus.yaml to scrape the metrics from the sidecars.

cfg/prometheus.yml
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'monitoring'
    static_configs:
      - targets: ['dap-sidecar:8080','rdb-sidecar:8080','idb-sidecar:8080','hdb-sidecar:8080','sm-sidecar:8080']