Database
Databases are deployed using a set of two helm charts:
query-base
Chart: Deploy this chart once. It includes the gateway, resource coordinator, stream-processor coordinator, and aggregator components.db-base
Chart: Deploy this chart for each database instance. It includes data-access processes, the storage manager, reliable transport, and optionally stream processor workers and controllers.
Helm setup
Add the KX Helm chart repository to your helm install:
helm repo add --username <username> --password <password> \
kxi-repo https://nexus.dl.kx.com/repository/kx-insights-charts/
"kxi-repo" has been added to your repositories
Update your Helm repository local chart cache:
helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kxi-repo" chart repository
Update Complete. ⎈Happy Helming!⎈
Add a secret to your cluster to allow access to the image:
kubectl create secret docker-registry kx-repo-access \
--docker-username=<username> \
--docker-password=<password> \
--docker-server=portal.dl.kx.com
URLs
For up to date URLs to use, please refer to the links shown in the Enterprise prerequisites
Quickstart
Set up the base components and two databases called db-one
and db-two
.
Each database includes one table: trade
in db-one
and quote
in db-two
. Both tables publish generated data on a timer.
This example uses minimal resource limits for demonstration purposes.
First, create a global.yml
file to define communal values such as image.repository
overrides and licensing
.
For example, set the log level to DEBUG
for all components and override the license:
global:
logging:
routings:
DEFAULT: DEBUG
license:
asFile: true
secretName: kxi-license
type: k4
resources:
limits:
cpu: 100m
memory: 2048Mi
requests:
cpu: 100m
memory: 1024Mi
Create query-base.yml
. This file is initially blank but illustrates how you can override each component's settings:
# Uncomment to set individual values
#sg-gateway: {}
#resource-coordinator: {}
#aggregator:{}
Create db-one.yml
to define the schema for the first database and specify the pipeline configuration.
Here, the pipeline generates data on a timer:
global:
assembly:
tables:
trade:
description: Trade data
type: partitioned
shards: 11
blockSize: 10000
prtnCol: realTime
primaryKeys: [sym, realTime]
sortColsOrd: sym
sortColsDisk: sym
columns:
- name: time
description: Time span
type: timespan
- name: sym
description: Symbol name
type: symbol
attrMem: grouped
attrDisk: parted
attrOrd: parted
- name: realTime
description: Time stamp
type: timestamp
- name: price
description: Trade price
type: float
- name: size
description: Trade size
type: long
db-pipeline:
spec: |-
N:25
trade: ([]sym:`g#"i"$();realTime:"p"$();time:"n"$();price:"f"$();size:"j"$());
pubFn:{
t:update price:N?100f, sym:N?`3, realTime:.z.p, size:N?10 from N#trade;
show meta t;
pub t;
};
// Add N rows every minute;
.qsp.onStart {
.tm.add[`forcePub;(`pubFn;::);60000;0]
};
.qsp.run
.qsp.read.fromCallback[`pub]
.qsp.write.toRT[`trade]
# Uncomment to set individual values
#kxi-rt: {}
#da: {}
#sm: {}
Create db-two.yml
, which is similar to db-one.yml
but differs in schema, as it defines the quote
table:
global:
assembly:
tables:
quote:
description: Quote table
type: partitioned
shards: 11
blockSize: 10000
prtnCol: realTime
primaryKeys: [sym, realTime]
sortColsOrd: sym
sortColsDisk: sym
columns:
- name: time
description: Time span
type: timespan
- name: sym
description: Symbol name
type: symbol
attrMem: grouped
attrDisk: parted
attrOrd: parted
- name: realTime
description: Time stamp
type: timestamp
- name: ask
description: quote price
type: float
db-pipeline:
spec: |-
N:25
quote: ([]sym:`g#"i"$();realTime:"p"$();time:"n"$();ask:"f"$());
pubFn:{
t:update ask:N?100f, sym:N?`3, realTime:.z.p from N#quote;
pub t;
};
// Add N rows every minute;
.qsp.onStart {
.tm.add[`forcePub;(`pubFn;::);60000;0]
};
.qsp.run
.qsp.read.fromCallback[`pub]
.qsp.write.toRT[`quote]
Deploy each chart using helm
:
helm install base kxi-repo/query-base -f global.yml -f query-base.yml
helm install db-one kxi-repo/db-base -f global.yml -f db-one.yml
helm install db-two kxi-repo/db-base -f global.yml -f db-two.yml
Port-forward the service-gateway
:
kubectl get pods --no-headers=true | awk '/sg-gateway/{print $1}'| cut -f 1 -d ' '| xargs bash -c '</dev/tty kubectl port-forward "$@" 5050:5050' dummy
Make a request to get table data for quote
where the ask
is greater than or equal to (>=
) 50:
curl -X POST --header "Content-Type: application/json"\
--header "Accepted: application/json"\
--data '{"table":"quote","filter":[[">=","ask","50"]]}'\
"http://localhost:8080/data"
Configuration
- The
query-base
chart configuration is specified under the keysglobal
,resource-coordinator
,aggregator
, andsg-gateway
. - The
db-base
chart configuration is specified under the keysglobal
,da
,sm
,db-pipeline
, andkxi-rt
.
Global values
These values can be set globally or overridden at the component level. They apply to both query-base
and db-base
components.
For example, to set HELLO_WORLD=true
on all components and override it to HELLO_WORLD=false
on the aggregator:
global:
env:
HELLO_WORLD: "true"
aggregator:
env:
HELLO_WORLD: "false"
option | default | description |
---|---|---|
Common and Global values
Not all common values are displayed here. Values used by all KX Helm charts, such as fullnameOverride
and service
, are omitted.
query-base
Deploying the query-base
chart configures the base query routing components, which are reused by multiple db-base
charts.
When installing the query-base
chart, you can provide options under global
, resource-coordinator
, aggregator
, and sg-gateway
.
Aggregator
Specify values under aggregator
:
.e.g:
aggregator:
affinity: soft
image:
pullPolicy: Always
Resource Coordinator
Specify values under resouce-coordinator
:
.e.g:
resouce-coordinator:
affinity: soft
image:
pullPolicy: Always
Gateway
Specify values under sg-gateway
:
.e.g:
sg-gateway:
affinity: soft
image:
pullPolicy: Always
db-base
Deploying the db-base
chart creates a database with a supplied schema, the subscription topic defaults to the chart name.
When installing the query base chart, you can provide options under global
, da
, sm
, kxi-rt
, and db-pipeline
.
Global and assembly values
These values can be set globally or overridden at each component level. Setting options at the global level applies to all components.
option | default | description |
---|---|---|
assembly |
{} | Assembly is required as a global key. At a minimum, specify assembly.tables . |
assembly.name |
Defaults to chart name | The name of the assembly. |
assembly.tables |
{} | An object table names to schemas |
assembly.labels |
Defaults to database:chart name |
An object label names to values |
assembly.mounts |
{} | An object of mount names to mounts |
assembly.bus |
{} | An object of bus configuration |
assembly.elements |
Defaults to object of dap and sm |
An object of dap configuration and sm configuration. |
assembly.labels._merge |
true | By default, merge assembly.labels with default labels. |
assembly.mounts._merge |
true | By default, merge assembly.mounts with default mounts. |
assembly.bus._merge |
true | By default, merge assembly.bus with default bus. |
assembly.elements_merge |
true | By default, merge assembly.elements with default elements. |
checkpointPath |
/sp/checkpoints |
Directory to write SP checkpoints to. |
specPath |
/opt/kx/spec.q |
File path where spec.q is mounted. |
assemblyPath |
/opt/kx/assembly.yml |
File path to assembly.yml . |
rtLogPath |
/logs/rt |
Directory path to RT log folder. |
hdbPath |
/data/db/hdb |
Directory path to HDB folder. |
idbPath |
/data/db/idb |
Directory path to IDB folder. |
checkpointStorage |
20Gi | Disk size for SP checkpoints mount. |
rtLogStorage |
20Gi | Disk size for RT Log storage. |
hdbStorage |
20Gi | Disk size for HDB. |
idbStorage |
20Gi | Disk size for IDB. |
volumes |
{} | Default volumes for the statefulset. |
volumeMounts |
{} | Default volumeMounts for the statefulset. |
volumeClaimTemplates |
{} | Default volumeClaimTemplates for the statefulset. |
Helm Template
Run the following command to generate example volumes:
helm template mydb kxi-repo/db-base --set global.assembly=dummy --set db-pipeline.spec=dummy
!!! note _merge
By default, when overriding assembly.mounts
, assembly.labels
, assembly.bus
, and assembly.elements
, they are merged with the default values.
However, If you set _merge: false
then the value will override instead of merging.
The merger is a [deep merge operation](https://helm.sh/docs/chart_template_guide/function_list/#merge-mustmerge).
Input with `_merge` true (by default):
```yaml
assembly:
name: hello
lables:
myLabel: "world"
```
Output merged:
```yaml
assembly:
name: hello
lables:
database: "hello"
myLabel: "world"
```
Input with `_merge` false:
```yaml
assembly:
name: hello
lables:
_merge: false
myLabel: "world"
```
Output overriden:
```yaml
assembly:
name: hello
lables:
myLabel: "world"
```
DA
Specify values under da
:
option | default | description |
---|---|---|
createAssembly |
false |
Indicates whether this component chart should create the assembly configmap . Only one of the DA or SM should do this. Typically, you do not need to change this. |
SM
Specify values under sm
:
option | default | description |
---|---|---|
createAssembly |
true |
Indicates whether this component chart should create the assembly configmap . Only one of the DA or SM should do this. Typically, you do not need to change this. |
db-pipeline
Specify values under db-pipeline
:
option | default | description |
---|---|---|
enabled |
true |
If set to false , do not deploy the stream processor. |
spec |
{} | Inline contents of spec.q for the stream processor worker. |
createSpec |
true |
If set to false , deploy the SP without creating a configmap , pods will wait until one is created independently. |
sp-controller.name |
spctl |
Name prefix for the SP Controller. |
sp-controller.port |
5000 | Port number for the SP Controller. |
sp-controller.image |
{} | Refer to global values. |
sp-worker.name |
worker |
Name prefix for the SP Worker. |
sp-worker.port |
6000 | Port number for the SP Worker. |
sp-worker.image |
{} | Refer to global values. |
kxi-rt
Specify values under kxi-rt
:
option | default | description |
---|---|---|
enabled |
true |
If set to false , do not deploy RT. |
stream |
destination |
The name of the stream topic prefix. |