Docker reference deployment

A reference deployment using Docker Compose is provided below.

Pre-requisites

Pulling the images requires a login with:

docker login registry.dl.kx.com -u <username> -p <password>

Directory structure

Before running this example, the following directory structure should be created:

db/     # Empty directory where the database will be stored on disk
cfg/    # Directory for configuration files
    assembly.yaml
    rt_tick_client_lib.q
src/    # Directory for custom API code
    agg.q
    da.q
kdb-tick/   # Clone of kdb-tick for a tickerplant service
    sym/
    tick/{r.q, u.q, sym.q}
    tick.q
.env
docker-compose-sg.yaml
docker-compose-da.yaml
docker-compose-sm.yaml
docker-compose-tp.yaml

Write permissions

The db/ and kdb-tick/sym directory on the host must allow write permission by the SM and TP containers who run as the user "nobody".

The sources for agg.q and da.q for this example are provided here.
kdb-tick/ is cloned from the KX github.
The sym.q schema must include the _prtnEnd and _reload tables, as in:

File: `kdb-tick/tick/sym.q`

SYMCONFIG

Run

To run, execute the following:

docker-compose \
    -f docker-compose-tp.yaml \
    -f docker-compose-da.yaml \
    -f docker-compose-sm.yaml \
    -f docker-compose-sg.yaml \
    up

Environment

Each of the Docker Compose files below use a .env file specifying the images and licenses to be used.

File: `.env`

# Images and license
KDB_LICENSE_B64="[MASKED]"
RELEASE="1.2.0"
kxi_sg_gw=registry.dl.kx.com/kxi-sg-gw:$RELEASE
kxi_sg_rc=registry.dl.kx.com/kxi-sg-rc:$RELEASE
kxi_sg_agg=registry.dl.kx.com/kxi-sg-agg:$RELEASE
kxi_sm_single=registry.dl.kx.com/kxi-da-single:$RELEASE
kxi_da_single=registry.dl.kx.com/kxi-da-single:$RELEASE
kxi_q=registry.dl.kx.com/qce:3.1.0

# Paths
custom_da_dir="$PWD/src/da"
custom_agg_dir="$PWD/src/agg"
cfg_dir="$PWD/cfg"
mnt_dir="$PWD/db"
tp_dir="$PWD/kdb-tick/sym"
custom_dir="./custom"

Assembly file

The Assembly file is the main business configuration for the database. Table schemas and logical process configuration are defined here. See assembly configuration for more information.

File: `cfg/assembly.yaml`

ASSEMBLY

Docker compose

Each subservice of the database is deployed here as a separate Docker Compose file for clarity. Each of these could be combined into a single Docker Compose file instead.

Service Gateway

The Service Gateway configures the three containers that make up the gateway.

File: `docker-compose-sg.yaml`

SGCONFIG

Data Access Processes

A set of data access processes are configured, each of which connect to the Resource Coordinator of the Service Gateway launched above.

File: `docker-compose-da.yaml`

DACONFIG

Storage Manager

The Storage Manager is configured as a single container, allowing connections by Data Access Processes configured above.

File: `docker-compose-sm.yaml`

SMCONFIG

Tickerplant

A standard tickerplant is put infront of the Storage Manager and Data Access Process to provide durable data ingestion.

Note

Within KX Insights Platform, the transport used is KX Insights Reliable Transport rather than a tickerplant. This allows for fault-tolerance and durable messaging despite potentially unreliable network connections. When using a standard tickerplant instead, the interface must adhere to the API expected by RT.

File: `docker-compose-tp.yaml`

TPCONFIG

Custom API code

Custom code is entirely optional, but allows creation of custom APIs to reside within Data Access Processes and Aggregators. These APIs are then registered with the Service Gateway to provide unified access across tiers to clients.

File: `src/da/custom.q`

DACODE

File: `src/agg/custom.q`

AGGCODE

RT client library

To connect to the tickerplant, both Data Access Processes and the Storage Manager are configured with a custom script defining connection details and providing the necessary .rt.* APIs expected by the services.

File: `cfg/rt_tick_client_lib.q`

RTCLIENTLIB

Docker reference deployment

Pre-requisites

Directory structure

File: kdb-tick/tick/sym.q

Run

Environment

File: .env