Skip to content

Docker reference deployment

A reference deployment using Docker Compose is provided below.

Pre-requisites

Pulling the images requires a login with:

docker login registry.dl.kx.com -u <username> -p <password>

Directory structure

Before running this example, the following directory structure should be created:

db/     # Empty directory where the database will be stored on disk
cfg/    # Directory for configuration files
    assembly.yaml
    rt_tick_client_lib.q
src/    # Directory for custom API code
    agg.q
    da.q
kdb-tick/   # Clone of kdb-tick for a tickerplant service
    sym/
    tick/{r.q, u.q, sym.q}
    tick.q
.env
docker-compose-sg.yaml
docker-compose-da.yaml
docker-compose-sm.yaml
docker-compose-tp.yaml

Write permissions

The db/ and kdb-tick/sym directory on the host must allow write permission by the SM and TP containers who run as the user "nobody".

  • The sources for agg.q and da.q for this example are provided here.
  • kdb-tick/ is cloned from the KX github.
  • The sym.q schema must include the _prtnEnd and _reload tables, as in:

```q title="kdb-tick/tick/sym.q" SYMCONFIG


## Run

To run, execute the following:

```bash
docker-compose \
    -f docker-compose-tp.yaml \
    -f docker-compose-da.yaml \
    -f docker-compose-sm.yaml \
    -f docker-compose-sg.yaml \
    up

Environment

Each of the Docker Compose files below use a .env file specifying the images and licenses to be used.

```bash title=".env"

Images and license

KDB_LICENSE_B64="[MASKED]" kxi_sg_gw=registry.dl.kx.com/kxi-sg-gw:1.2.0 kxi_sg_rc=registry.dl.kx.com/kxi-sg-rc:1.2.0 kxi_sg_agg=registry.dl.kx.com/kxi-sg-agg:1.2.0 kxi_sm_single=registry.dl.kx.com/kxi-da-single:1.2.0 kxi_da_single=registry.dl.kx.com/kxi-da-single:1.2.0 kxi_q=registry.dl.kx.com/qce:3.1.0

Paths

custom_da_dir="\(PWD/src/da" custom_agg_dir="\)PWD/src/agg" cfg_dir="\(PWD/cfg" mnt_dir="\)PWD/db" tp_dir="$PWD/kdb-tick/sym" custom_dir="./custom"



## Assembly file

The *Assembly file* is the main business configuration for the database. Table schemas and logical process configuration are defined here. See [assembly configuration](./configuration_dap.html#assembly) for more information.

```yaml title="cfg/assembly.yaml"
ASSEMBLY

Docker compose

Each subservice of the database is deployed here as a separate Docker Compose file for clarity. Each of these could be combined into a single Docker Compose file instead.

Service Gateway

The Service Gateway configures the three containers that make up the gateway.

```yaml title="docker-compose-sg.yaml" SGCONFIG


### Data Access Processes

A set of data access processes are configured, each of which connect to the Resource Coordinator of the Service Gateway launched above.

```yaml title="docker-compose-da.yaml"
DACONFIG

Storage Manager

The Storage Manager is configured as a single container, allowing connections by Data Access Processes configured above.

```yaml title="docker-compose-sm.yaml" SMCONFIG


### Tickerplant

A standard tickerplant is put infront of the Storage Manager and Data Access Process to provide durable data ingestion.

!!! note Reliable Transport

    Within *KX Insights Platform*, the transport used is *KX Insights Reliable Transport* rather than a tickerplant. This allows for fault-tolerance and durable messaging despite potentially unreliable network connections. When using a standard tickerplant instead, the interface must adhere to the API expected by RT.

``` yaml title="docker-compose-tp.yaml"
TPCONFIG

Custom API code

Custom code is entirely optional, but allows creation of custom APIs to reside within Data Access Processes and Aggregators. These APIs are then registered with the Service Gateway to provide unified access across tiers to clients.

```q title="src/da/custom.q" DACODE


``` q title="src/agg/custom.q"
AGGCODE

RT client library

To connect to the tickerplant, both Data Access Processes and the Storage Manager are configured with a custom script defining connection details and providing the necessary .rt.* APIs expected by the services.

q title="cfg/rt_tick_client_lib.q" RTCLIENTLIB