Skip to content

Database

Databases are deployed using a set of two helm charts:

  • query-base Chart: Deploy this chart once. It includes the gateway, resource coordinator, stream-processor coordinator, and aggregator components.
  • db-base Chart: Deploy this chart for each database instance. It includes data-access processes, the storage manager, reliable transport, and optionally stream processor workers and controllers.

Helm setup

Add the KX Helm chart repository to your helm install:

helm repo add --username <username> --password <password> \
    kxi-repo https://nexus.dl.kx.com/repository/kx-insights-charts/
"kxi-repo" has been added to your repositories

Update your Helm repository local chart cache:

helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kxi-repo" chart repository
Update Complete. ⎈Happy Helming!⎈

Add a secret to your cluster to allow access to the image:

kubectl create secret docker-registry kx-repo-access \
    --docker-username=<username> \
    --docker-password=<password> \
    --docker-server=portal.dl.kx.com

URLs

For up to date URLs to use, please refer to the links shown in the Enterprise prerequisites

Quickstart

Set up the base components and two databases called db-one and db-two.

Each database includes one table: trade in db-one and quote in db-two. Both tables publish generated data on a timer.

This example uses minimal resource limits for demonstration purposes.

First, create a global.yml file to define communal values such as image.repository overrides and licensing. For example, set the log level to DEBUG for all components and override the license:

global.yml
global:
  logging:
    routings:
      DEFAULT: DEBUG
  license:
    asFile: true
    secretName: kxi-license
    type: k4
  resources:
    limits:
      cpu: 100m
      memory: 2048Mi
    requests:
      cpu: 100m
      memory: 1024Mi

Create query-base.yml. This file is initially blank but illustrates how you can override each component's settings:

query-base.yml
# Uncomment to set individual values
#sg-gateway: {}
#resource-coordinator: {}
#aggregator:{}

Create db-one.yml to define the schema for the first database and specify the pipeline configuration.

Here, the pipeline generates data on a timer:

db-one.yml
global:
  assembly:
    tables:
      trade:
        description: Trade data
        type: partitioned
        shards: 11
        blockSize: 10000
        prtnCol: realTime
        primaryKeys: [sym, realTime]
        sortColsOrd: sym
        sortColsDisk: sym
        columns:
          - name: time
            description: Time span
            type: timespan
          - name: sym
            description: Symbol name
            type: symbol
            attrMem: grouped
            attrDisk: parted
            attrOrd: parted
          - name: realTime
            description: Time stamp
            type: timestamp
          - name: price
            description: Trade price
            type: float
          - name: size
            description: Trade size
            type: long

db-pipeline:
  spec: |-
    N:25
    trade: ([]sym:`g#"i"$();realTime:"p"$();time:"n"$();price:"f"$();size:"j"$());
    pubFn:{
      t:update price:N?100f, sym:N?`3, realTime:.z.p, size:N?10 from N#trade;
      show meta t;
      pub t;
      };
    // Add N rows every minute;
    .qsp.onStart {
      .tm.add[`forcePub;(`pubFn;::);60000;0]
      };
    .qsp.run
        .qsp.read.fromCallback[`pub]
        .qsp.write.toRT[`trade]

# Uncomment to set individual values
#kxi-rt: {}
#da: {}
#sm: {}

Create db-two.yml, which is similar to db-one.yml but differs in schema, as it defines the quote table:

db-two.yml
global:
  assembly:
    tables:
      quote:
        description: Quote table
        type: partitioned
        shards: 11
        blockSize: 10000
        prtnCol: realTime
        primaryKeys: [sym, realTime]
        sortColsOrd: sym
        sortColsDisk: sym
        columns:
          - name: time
            description: Time span
            type: timespan
          - name: sym
            description: Symbol name
            type: symbol
            attrMem: grouped
            attrDisk: parted
            attrOrd: parted
          - name: realTime
            description: Time stamp
            type: timestamp
          - name: ask
            description: quote price
            type: float

db-pipeline:
  spec: |-
    N:25
    quote: ([]sym:`g#"i"$();realTime:"p"$();time:"n"$();ask:"f"$());
    pubFn:{
      t:update ask:N?100f, sym:N?`3, realTime:.z.p from N#quote;
      pub t;
      };
    // Add N rows every minute;
    .qsp.onStart {
      .tm.add[`forcePub;(`pubFn;::);60000;0]
      };
    .qsp.run
        .qsp.read.fromCallback[`pub]
        .qsp.write.toRT[`quote]

Deploy each chart using helm:

helm install base kxi-repo/query-base -f global.yml -f query-base.yml
helm install db-one kxi-repo/db-base -f global.yml -f db-one.yml
helm install db-two kxi-repo/db-base -f global.yml -f db-two.yml

Port-forward the service-gateway:

kubectl get pods --no-headers=true | awk '/sg-gateway/{print $1}'| cut -f 1 -d ' '| xargs bash -c '</dev/tty kubectl port-forward "$@" 5050:5050' dummy

Make a request to get table data for quote where the ask is greater than or equal to (>=) 50:

curl -X POST --header "Content-Type: application/json"\
    --header "Accepted: application/json"\
    --data '{"table":"quote","filter":[[">=","ask","50"]]}'\
    "http://localhost:8080/data"

Configuration

  • The query-base chart configuration is specified under the keys global, resource-coordinator, aggregator, and sg-gateway.
  • The db-base chart configuration is specified under the keys global, da, sm, db-pipeline, and kxi-rt.

Global values

These values can be set globally or overridden at the component level. They apply to both query-base and db-base components.

For example, to set HELLO_WORLD=true on all components and override it to HELLO_WORLD=false on the aggregator:

global:
  env:
    HELLO_WORLD: "true"
aggregator:
  env:
    HELLO_WORLD: "false"
option default description

Common and Global values

Not all common values are displayed here. Values used by all KX Helm charts, such as fullnameOverride and service, are omitted.

query-base

Deploying the query-base chart configures the base query routing components, which are reused by multiple db-base charts.

When installing the query-base chart, you can provide options under global, resource-coordinator, aggregator, and sg-gateway.

Aggregator

Specify values under aggregator:

.e.g:

aggregator:
   affinity: soft
   image:
     pullPolicy: Always

Resource Coordinator

Specify values under resouce-coordinator:

.e.g:

resouce-coordinator:
   affinity: soft
   image:
     pullPolicy: Always

Gateway

Specify values under sg-gateway:

.e.g:

sg-gateway:
   affinity: soft
   image:
     pullPolicy: Always

db-base

Deploying the db-base chart creates a database with a supplied schema, the subscription topic defaults to the chart name.

When installing the query base chart, you can provide options under global, da, sm, kxi-rt, and db-pipeline.

Global and assembly values

These values can be set globally or overridden at each component level. Setting options at the global level applies to all components.

option default description
assembly {} Assembly is required as a global key. At a minimum, specify assembly.tables.
assembly.name Defaults to chart name The name of the assembly.
assembly.tables {} An object table names to schemas
assembly.labels Defaults to database:chart name An object label names to values
assembly.mounts {} An object of mount names to mounts
assembly.bus {} An object of bus configuration
assembly.elements Defaults to object of dap and sm An object of dap configuration and sm configuration.
assembly.labels._merge true By default, merge assembly.labels with default labels.
assembly.mounts._merge true By default, merge assembly.mounts with default mounts.
assembly.bus._merge true By default, merge assembly.bus with default bus.
assembly.elements_merge true By default, merge assembly.elements with default elements.
checkpointPath /sp/checkpoints Directory to write SP checkpoints to.
specPath /opt/kx/spec.q File path where spec.q is mounted.
assemblyPath /opt/kx/assembly.yml File path to assembly.yml.
rtLogPath /logs/rt Directory path to RT log folder.
hdbPath /data/db/hdb Directory path to HDB folder.
idbPath /data/db/idb Directory path to IDB folder.
checkpointStorage 20Gi Disk size for SP checkpoints mount.
rtLogStorage 20Gi Disk size for RT Log storage.
hdbStorage 20Gi Disk size for HDB.
idbStorage 20Gi Disk size for IDB.
volumes {} Default volumes for the statefulset.
volumeMounts {} Default volumeMounts for the statefulset.
volumeClaimTemplates {} Default volumeClaimTemplates for the statefulset.

Helm Template

Run the following command to generate example volumes:

helm template mydb kxi-repo/db-base --set global.assembly=dummy --set db-pipeline.spec=dummy

!!! note _merge By default, when overriding assembly.mounts, assembly.labels, assembly.bus, and assembly.elements, they are merged with the default values. However, If you set _merge: false then the value will override instead of merging.

The merger is a [deep merge operation](https://helm.sh/docs/chart_template_guide/function_list/#merge-mustmerge).

Input with `_merge` true (by default):
```yaml
assembly:
  name: hello
  lables:
     myLabel: "world"
```

Output merged:
```yaml
assembly:
  name: hello
  lables:
     database: "hello"
     myLabel: "world"
```

Input with `_merge` false:
```yaml
assembly:
  name: hello
  lables:
     _merge: false
     myLabel: "world"
```

Output overriden:
```yaml
assembly:
  name: hello
  lables:
     myLabel: "world"
```

DA

Specify values under da:

option default description
createAssembly false Indicates whether this component chart should create the assembly configmap. Only one of the DA or SM should do this. Typically, you do not need to change this.

SM

Specify values under sm:

option default description
createAssembly true Indicates whether this component chart should create the assembly configmap. Only one of the DA or SM should do this. Typically, you do not need to change this.

db-pipeline

Specify values under db-pipeline:

option default description
enabled true If set to false, do not deploy the stream processor.
spec {} Inline contents of spec.q for the stream processor worker.
createSpec true If set to false, deploy the SP without creating a configmap, pods will wait until one is created independently.
sp-controller.name spctl Name prefix for the SP Controller.
sp-controller.port 5000 Port number for the SP Controller.
sp-controller.image {} Refer to global values.
sp-worker.name worker Name prefix for the SP Worker.
sp-worker.port 6000 Port number for the SP Worker.
sp-worker.image {} Refer to global values.

kxi-rt

Specify values under kxi-rt:

option default description
enabled true If set to false, do not deploy RT.
stream destination The name of the stream topic prefix.