Database

Databases are deployed using a set of two helm charts:

query-base Chart: Deploy this chart once. It includes the gateway, resource coordinator, stream-processor coordinator, and aggregator components.
db-base Chart: Deploy this chart for each database instance. It includes data-access processes, the storage manager, reliable transport, and optionally stream processor workers and controllers.

Helm setup

Add the KX Helm chart repository to your helm install:

helm repo add --username <username> --password <password> \
    kxi-repo https://nexus.dl.kx.com/repository/kx-insights-charts/

"kxi-repo" has been added to your repositories

Update your Helm repository local chart cache:

helm repo update

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kxi-repo" chart repository
Update Complete. ⎈Happy Helming!⎈

Add a secret to your cluster to allow access to the image:

kubectl create secret docker-registry kx-repo-access \
    --docker-username=<username> \
    --docker-password=<password> \
    --docker-server=portal.dl.kx.com

URLs

For up to date URLs to use, please refer to the links shown in the Enterprise prerequisites

Quickstart

Set up the base components and two databases called db-one and db-two.

Each database includes one table: trade in db-one and quote in db-two. Both tables publish generated data on a timer.

This example uses minimal resource limits for demonstration purposes.

First, create a global.yml file to define communal values such as image.repository overrides and licensing. For example, set the log level to DEBUG for all components and override the license:

global.yml

global:
  logging:
    routings:
      DEFAULT: DEBUG
  license:
    asFile: true
    secretName: kxi-license
    type: k4
  resources:
    limits:
      cpu: 100m
      memory: 2048Mi
    requests:
      cpu: 100m
      memory: 1024Mi

Create query-base.yml. This file is initially blank but illustrates how you can override each component's settings:

query-base.yml

# Uncomment to set individual values
#sg-gateway: {}
#resource-coordinator: {}
#aggregator:{}

Create db-one.yml to define the schema for the first database and specify the pipeline configuration.

Here, the pipeline generates data on a timer:

db-one.yml

global:
  assembly:
    tables:
      trade:
        description: Trade data
        type: partitioned
        shards: 11
        blockSize: 10000
        prtnCol: realTime
        primaryKeys: [sym, realTime]
        sortColsOrd: sym
        sortColsDisk: sym
        columns:
          - name: time
            description: Time span
            type: timespan
          - name: sym
            description: Symbol name
            type: symbol
            attrMem: grouped
            attrDisk: parted
            attrOrd: parted
          - name: realTime
            description: Time stamp
            type: timestamp
          - name: price
            description: Trade price
            type: float
          - name: size
            description: Trade size
            type: long

db-pipeline:
  spec: |-
    N:25
    trade: ([]sym:`g#"i"$();realTime:"p"$();time:"n"$();price:"f"$();size:"j"$());
    pubFn:{
      t:update price:N?100f, sym:N?`3, realTime:.z.p, size:N?10 from N#trade;
      show meta t;
      pub t;
      };
    // Add N rows every minute;
    .qsp.onStart {
      .tm.add[`forcePub;(`pubFn;::);60000;0]
      };
    .qsp.run
        .qsp.read.fromCallback[`pub]
        .qsp.write.toRT[`trade]

# Uncomment to set individual values
#kxi-rt: {}
#da: {}
#sm: {}

Create db-two.yml, which is similar to db-one.yml but differs in schema, as it defines the quote table:

db-two.yml

global:
  assembly:
    tables:
      quote:
        description: Quote table
        type: partitioned
        shards: 11
        blockSize: 10000
        prtnCol: realTime
        primaryKeys: [sym, realTime]
        sortColsOrd: sym
        sortColsDisk: sym
        columns:
          - name: time
            description: Time span
            type: timespan
          - name: sym
            description: Symbol name
            type: symbol
            attrMem: grouped
            attrDisk: parted
            attrOrd: parted
          - name: realTime
            description: Time stamp
            type: timestamp
          - name: ask
            description: quote price
            type: float

db-pipeline:
  spec: |-
    N:25
    quote: ([]sym:`g#"i"$();realTime:"p"$();time:"n"$();ask:"f"$());
    pubFn:{
      t:update ask:N?100f, sym:N?`3, realTime:.z.p from N#quote;
      pub t;
      };
    // Add N rows every minute;
    .qsp.onStart {
      .tm.add[`forcePub;(`pubFn;::);60000;0]
      };
    .qsp.run
        .qsp.read.fromCallback[`pub]
        .qsp.write.toRT[`quote]

Deploy each chart using helm:

helm install base kxi-repo/query-base -f global.yml -f query-base.yml
helm install db-one kxi-repo/db-base -f global.yml -f db-one.yml
helm install db-two kxi-repo/db-base -f global.yml -f db-two.yml

Port-forward the service-gateway:

kubectl get pods --no-headers=true | awk '/sg-gateway/{print $1}'| cut -f 1 -d ' '| xargs bash -c '</dev/tty kubectl port-forward "$@" 5050:5050' dummy

Make a request to get table data for quote where the ask is greater than or equal to (>=) 50:

curl -X POST --header "Content-Type: application/json"\
    --header "Accepted: application/json"\
    --data '{"table":"quote","filter":[[">=","ask","50"]]}'\
    "http://localhost:8080/data"

Configuration

The query-base chart configuration is specified under the keys global, resource-coordinator, aggregator, and sg-gateway.
The db-base chart configuration is specified under the keys global, da, sm, db-pipeline, and kxi-rt.

Global values

These values can be set globally or overridden at the component level. They apply to both query-base and db-base components.

For example, to set HELLO_WORLD=true on all components and override it to HELLO_WORLD=false on the aggregator:

global:
  env:
    HELLO_WORLD: "true"
aggregator:
  env:
    HELLO_WORLD: "false"

option	default	description

Common and Global values

Not all common values are displayed here. Values used by all KX Helm charts, such as fullnameOverride and service, are omitted.

query-base

Deploying the query-base chart configures the base query routing components, which are reused by multiple db-base charts.

When installing the query-base chart, you can provide options under global, resource-coordinator, aggregator, and sg-gateway.

Aggregator

Specify values under aggregator:

.e.g:

aggregator:
   affinity: soft
   image:
     pullPolicy: Always

Resource Coordinator

Specify values under resouce-coordinator:

.e.g:

resouce-coordinator:
   affinity: soft
   image:
     pullPolicy: Always

Gateway

Specify values under sg-gateway:

.e.g:

sg-gateway:
   affinity: soft
   image:
     pullPolicy: Always

db-base

Deploying the db-base chart creates a database with a supplied schema, the subscription topic defaults to the chart name.

When installing the query base chart, you can provide options under global, da, sm, kxi-rt, and db-pipeline.

Global and assembly values

These values can be set globally or overridden at each component level. Setting options at the global level applies to all components.

option	default	description
`assembly`	{}	Assembly is required as a global key. At a minimum, specify `assembly.tables`.
`assembly.name`	Defaults to chart name	The name of the assembly.
`assembly.tables`	{}	An object table names to schemas
`assembly.labels`	Defaults to `database:chart name`	An object label names to values
`assembly.mounts`	{}	An object of mount names to mounts
`assembly.bus`	{}	An object of bus configuration
`assembly.elements`	Defaults to object of `dap` and `sm`	An object of `dap` configuration and `sm` configuration.
`assembly.labels._merge`	true	By default, merge `assembly.labels` with default labels.
`assembly.mounts._merge`	true	By default, merge `assembly.mounts` with default mounts.
`assembly.bus._merge`	true	By default, merge `assembly.bus` with default bus.
`assembly.elements_merge`	true	By default, merge `assembly.elements` with default elements.
`checkpointPath`	`/sp/checkpoints`	Directory to write SP checkpoints to.
`specPath`	`/opt/kx/spec.q`	File path where `spec.q` is mounted.
`assemblyPath`	`/opt/kx/assembly.yml`	File path to `assembly.yml`.
`rtLogPath`	`/logs/rt`	Directory path to RT log folder.
`hdbPath`	`/data/db/hdb`	Directory path to HDB folder.
`idbPath`	`/data/db/idb`	Directory path to IDB folder.
`checkpointStorage`	20Gi	Disk size for SP checkpoints mount.
`rtLogStorage`	20Gi	Disk size for RT Log storage.
`hdbStorage`	20Gi	Disk size for HDB.
`idbStorage`	20Gi	Disk size for IDB.
`volumes`	{}	Default `volumes` for the statefulset.
`volumeMounts`	{}	Default `volumeMounts` for the statefulset.
`volumeClaimTemplates`	{}	Default `volumeClaimTemplates` for the statefulset.

Helm Template

Run the following command to generate example volumes:

helm template mydb kxi-repo/db-base --set global.assembly=dummy --set db-pipeline.spec=dummy

!!! note _merge By default, when overriding assembly.mounts, assembly.labels, assembly.bus, and assembly.elements, they are merged with the default values. However, If you set _merge: false then the value will override instead of merging.

The merger is a [deep merge operation](https://helm.sh/docs/chart_template_guide/function_list/#merge-mustmerge).

Input with `_merge` true (by default):
```yaml
assembly:
  name: hello
  lables:
     myLabel: "world"
```

Output merged:
```yaml
assembly:
  name: hello
  lables:
     database: "hello"
     myLabel: "world"
```

Input with `_merge` false:
```yaml
assembly:
  name: hello
  lables:
     _merge: false
     myLabel: "world"
```

Output overriden:
```yaml
assembly:
  name: hello
  lables:
     myLabel: "world"
```

DA

Specify values under da:

option	default	description
`createAssembly`	`false`	Indicates whether this component chart should create the assembly `configmap`. Only one of the DA or SM should do this. Typically, you do not need to change this.

SM

Specify values under sm:

option	default	description
`createAssembly`	`true`	Indicates whether this component chart should create the assembly `configmap`. Only one of the DA or SM should do this. Typically, you do not need to change this.

db-pipeline

Specify values under db-pipeline:

option	default	description
`enabled`	`true`	If set to `false`, do not deploy the stream processor.
`spec`	{}	Inline contents of `spec.q` for the stream processor worker.
`createSpec`	`true`	If set to `false`, deploy the SP without creating a `configmap`, pods will wait until one is created independently.
`sp-controller.name`	`spctl`	Name prefix for the SP Controller.
`sp-controller.port`	5000	Port number for the SP Controller.
`sp-controller.image`	{}	Refer to global values.
`sp-worker.name`	`worker`	Name prefix for the SP Worker.
`sp-worker.port`	6000	Port number for the SP Worker.
`sp-worker.image`	{}	Refer to global values.

kxi-rt

Specify values under kxi-rt:

option	default	description
`enabled`	`true`	If set to `false`, do not deploy RT.
`stream`	`destination`	The name of the stream topic prefix.