Running Storage Manager
The Storage Manager service (SM) is a stateful application, which can be run in Docker using Docker-Compose, or in Kuberenetes as a StatefulSet.
SM uses the kxi-sm-single
image. The following images are optional:
purpose | images |
---|---|
SM discoverable by DAP processes | kxi-sidecar discovery_proxy kxi-eureka-discovery |
Collect monitoring statistics | kxi-sidecar prom/prometheus |
The examples provided here are for running with
- minimal configuration: neither Service Discovery nor monitoring required
- full discovery and monitoring integration
Prerequisites
Kdb+ license
A valid kdb+ license is required to run Storage Manager (SM). In the examples in this page, we provide it by mapping the /opt/kx/lic
container path to a host mount which contains the license.
Message bus
SM requires a message bus compatible with Reliable Transport. The examples below assume a tickerplant server running on the host machine using the following command:
q tick.q sym /tmp/tplogs -p 5010
Configuration
SM is configured through a set of environment variables and an assembly file. The requirements vary between containers (described below), but at minimum, the name (KXI_NAME). service class (KXI_SC), and the path to the data-pipeline assembly (KXI_ASSEMBLY_FILE) assembly configuration are required.
The assembly file contains the following required information:
- database schema
- message bus
- database mount and tier configuration
- service configuration, including SM
See the deployment example for a complete assembly example including SM.
The first run
The first run of SM is important, because it is when it determines whether it needs to create an initial on-disk database, or import a pre-existing standard kdb+ partitioned database to SM's format. See initial-import.md for more details.
To restore the state before the first SM run
Shut down SM and delete the contents of the /data
mount volume.
Running with minimal configuration
See the deployment example for a minimal deployment of SM. Note, that deployment does also include Data Access Processes and the Service Gateway as a complete example, but those services can be omitted.
Environment variables
SM requires certain environment variables in the containers.
variable | required | description |
---|---|---|
KXI_NAME |
yes | Process name |
KXI_SC |
yes | Service class |
KXI_PORT |
yes | Port |
KXI_ASSEMBLY_FILE |
yes | Data-pipeline assembly configuration file |
KXI_RT_LIB |
yes1 | Path to Reliable-Transport client-side q module |
KXI_SM_SMADDR |
yes | SM container’s address for inter-container communication |
KXI_SM_EOIADDR |
yes | EOI container’s address for inter-container communication |
KXI_RT_SM_LOG_PATH |
Specifies the path to the logs for the SM process. (e.g., "/logs/rt/sm" ) |
|
KXI_RT_EOI_LOG_PATH |
Specifies the path to the logs for the EOI process. (e.g., "/logs/rt/eoi" ) |
|
KXI_SM_EOI_THREADS |
Thread count for the EOI process. (e.g., "8" ) |
|
KXI_SM_EOD_THREADS |
Thread count for the EOD process. (e.g., "8" ) |
|
KXI_SM_DBM_THREADS |
Thread count for the DBM process. (e.g., "8" ) |
|
AWS_ACCESS_KEY_ID |
yes2 | AWS access key |
AWS_SECRET_ACCESS_KEY |
yes2 | AWS secret key associated with the access key |
AWS_REGION |
yes2 | AWS region |
AZURE_STORAGE_ACCOUNT |
yes3 | Azure storage account name |
AZURE_STORAGE_SHARED_KEY |
yes3 | Azure storage key |
In addition, the following environment variables apply to both the sidecar and SM images.
variable | required | container | description |
---|---|---|---|
KXI_CONFIG_FILE |
yes | sidecar | discovery configuration file |
KXI_LOG_FORMAT |
ALL | log message format | |
KXI_LOG_DEST |
ALL | log endpoints | |
KXI_LOG_LEVELS |
ALL | component routing | |
KXI_LOG_CONFIG |
ALL | alternative logging configuration: replaces KXI_LOG_FORMAT , KXI_LOG_DEST , and KXI_LOG_LEVELS |
Volumes
SM requires the following volumes mounted:
volume | description |
---|---|
/data | database volume |
/tplogs | tickerplant log location† |
/cfg | volume containing configuration files |
/opt/kx/lic | kdb+ license file |
† SM receives the path to tickerplant log as a result of a call to the tickerplant server; so for SM to access the same location, the mounted volume must match that seen by the tickerplant server.
Running SM with discovery and monitoring integration
To run SM with discovery and monitoring integration, modify cfg/assembly_sample.yaml
by changing
disableDiscovery: true # Disables registering with discovery
to
disableDiscovery: false # Disables registering with discovery
Then modify docker-compose.yaml
by adding services for the remaining images:
sm-sidecar:
image: registry.dl.kx.com/kxi-sidecar:0.13.1
command: -p 8080
ports:
- 11001:8080
environment:
- KXI_CONFIG_FILE=/cfg/sm-sidecar.json
- KXI_LOG_CONFIG=/cfg/qlog.json
volumes:
- ~/.qp.licenses:/opt/kx/lic
- ./cfg:/cfg
networks:
- kx
stdin_open: true
tty: true
prom:
image: prom/prometheus
command: --config.file=/cfg/prometheus.yml
ports:
- 9090:9090
volumes:
- ./cfg:/cfg
networks:
- kx
proxy:
image: registry.dl.kx.com/kxi-discovery-proxy:0.13.1
command: -p 5000
ports:
- 5000:5000
environment:
- KXI_CONFIG_FILE=/cfg/proxy.json
- KXI_LOG_CONFIG=/cfg/qlog.json
volumes:
- ~/.qp.licenses:/opt/kx/lic
- ./cfg:/cfg
networks:
- kx
stdin_open: true
tty: true
eureka:
image: registry.dl.kx.com/kxi-eureka-discovery:1.0.6
ports:
- 9000:8761
networks:
- kx
The directory structure is now:
├── docker-compose.yaml
├── db/
└── cfg/
├── assembly_sample.yaml
├── prometheus.yml
├── proxy.json
├── qlog.json
└── sm-sidecar.json
We added the following configuration files:
cfg/proxy.json
The configuration file for the Service Discovery proxy:
{
"discovery":
{
"registry": ":eureka:8761",
"adaptor": "discEurekaAdaptor.q"
}
}
cfg/qlog.json
The QLog configuration file. This is an alternative (and recommended) way to configure logging for the container.
{
"endpoints": [ "fd://stdout", "fd://stderr" ],
"formatMode": "text",
"routings": {
"ALL": "INFO",
"qlog": "DEBUG"
}
}
cfg/prometheus.yml
Prometheus configuration:
global:
scrape_interval: 15s # By default, scrape targets every 15 seconds.
evaluation_interval: 15s # Evaluate rules every 15 seconds.
scrape_configs:
- job_name: 'sm-monitoring'
static_configs:
- targets: ['sm:10001']
- job_name: 'eoi-monitoring'
static_configs:
- targets: ['eoi:10002']
cfg/sm-sidecar.json
The configuration file for the sidecar:
{
"connection": ":sm:10001",
"frequencySecs": 5,
"discovery":
{
"registry": ":proxy:5000",
"adaptor": "discEurekaAdaptor.q",
"heartbeatSecs": 30,
"leaseExpirySecs": 30
},
"metrics":
{
"enabled": "true",
"frequency": 5,
"handler":
{
"pc": false,
"pg": false,
"ph": false,
"po": false,
"pp": false,
"ps": false,
"ts": false,
"wc": false,
"wo": false,
"ws": false
}
}
}
Note that it contains both Discovery and Monitoring configurations.
See "kxi-sidecar" page for more details.
Running
As in the first example, run from the command shell.
docker compose up