Manage Deployment Components
This page provides details of how to manage the deployment components in a package.
The deployment components within a package specify what needs to be deployed, detailing the creation and orchestration of processes.
Database components
The database fields in a package, define the schema, storage manager, stream, and data access processes.
A package only deploys one database component.
Although multiple database definitions are allowed in a package, only one is deployed.
Calling the package add command removes an existing component if it shares the same type and name as the new component.
If you run the following command, the original copy of the mypkg database is overwritten:
kxi package add database --to mypkg
# some editing of the files
kxi package add database --to mypkg
Database fields
kxi package field info Database
Fields for Database
╭─────────────────────┬──────────────────────────┬───────────────────────────────────────┬───────────────────────────────────────╮
│ field │ required │ type │ description │
├─────────────────────┼──────────────────────────┼───────────────────────────────────────┼───────────────────────────────────────┤
│ name │ False │ string │ DB Name │
│ shards │ False │ List of Shard │ Shards in DB │
│ tables │ False │ TableList │ Schemas in DB │
╰─────────────────────┴──────────────────────────┴───────────────────────────────────────┴───────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Shard fields
kxi package field info Shard
Fields for Shard
╭─────────────────────────┬─────────────────────┬─────────────────────────┬──────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├─────────────────────────┼─────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────┤
│ name │ False │ string │ Name identifier of shard │
│ labels │ False │ dict │ Shard labels │
│ dbSettings │ False │ DbSettings │ Database settings │
│ sm │ False │ Sm │ Storage Manager Object │
│ daps │ False │ Dap │ Data Access object │
│ sequencers │ False │ dict │ Messsage bus into system │
│ mounts │ False │ dict │ PVs used in package │
╰─────────────────────────┴─────────────────────┴─────────────────────────┴──────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Schema fields
kxi package field info Table
Fields for Table
╭──────────────┬──────────┬────────────────────┬─────────────────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────┼──────────┼────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ columns │ True │ List of Column │ A List of column schemas │
│ type │ True │ Type3 │ One of {splayed, splayed_mem, partitioned, basic}. Required │
│ blockSize │ False │ integer │ Number of rows to keep in-memory before write to disk │
│ description │ False │ string │ A string describing the purpose of this table │
│ isSharded │ False │ boolean │ Specifies if a reference data table is distributed across multiple assemblies │
│ name │ False │ string │ Table name │
│ oldName │ False │ string │ Old table name, for automated renaming │
│ primaryKeys │ False │ List of PrimaryKey │ Names of primary key columns │
│ prtnCol │ False │ string │ A timestamp column name used for data partitioning │
│ sortColsDisk │ False │ List of PrimaryKey │ Names of columns to sort on when stored on disk │
│ sortColsMem │ False │ List of PrimaryKey │ Names of columns to sort on when stored in memory │
│ sortColsOrd │ False │ List of PrimaryKey │ Names of columns to sort on when stored on disk with an ordinal partition │
│ │ │ │ scheme │
╰──────────────┴──────────┴────────────────────┴─────────────────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Stream fields
kxi package field info Sequencer
Fields for Sequencer
╭──────────────────────────┬──────────┬─────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────────────────┼──────────┼─────────────────────┼────────────────────────────────────────────────────────────────────┤
│ external │ True │ boolean │ Externally facing Sequencer │
│ annotations │ False │ Annotations │ Annotations to apply to resources created from Sequencer │
│ args │ False │ List of string │ Command line arguments to be set on element container │
│ env │ False │ List of EnvItem │ Environment Variables to be appended to the element container │
│ externalNodePort │ False │ boolean │ Use Node Port Type for Externally facing Sequencer service │
│ image │ False │ Image │ Sequencer image details │
│ k8sPolicy │ False │ K8sPolicy │ │
│ maxDiskUsagePercent │ False │ integer │ RT Log - Maximum Percentage of disk before Archiving │
│ maxLogSize │ False │ string │ RT Log Maximum size │
│ name │ False │ string │ Sequencer instance name │
│ retentionDuration │ False │ integer │ RT Log retention period in minutes │
│ sideCarConfig │ False │ SideCarConfig │ Side Car Config Map Details │
│ sideCarOverrides │ False │ SideCarOverrides │ Override Enabling Side Car for Sequencer instance │
│ size │ False │ Size │ Size is the replicaCount of the Sequencer StatefulSet │
│ staticIPs │ False │ List of string │ Optional list of IPs to append to the External RT LoadBalancers │
│ topicConfig │ False │ TopicConfig │ Sequencer Topic Prefix │
│ topicConfigDir │ False │ string │ Location of RT 'pull' directory │
│ useInternalLBAnnotations │ False │ boolean │ Apply Service Annotations to set LoadBalancer Service as internal │
│ │ │ │ only │
│ volume │ False │ Volume │ RT Sequencer directory paths │
│ volumeMounts │ False │ List of VolumeMount │ Mount volumes external to Assembly as defined within spec.Volumes │
╰──────────────────────────┴──────────┴─────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Pipeline components
Pipelines are used to enrich and stream data. You can add multiple pipelines to your package. Refer to the pipeline examples for details.
Pipeline fields
kxi package field info Pipeline
Fields for Pipeline
╭────────────────────────────┬──────────┬────────────────────────────┬───────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────────────────────┼──────────┼────────────────────────────┼───────────────────────────────────────────────────────────┤
│ base │ False │ Base │ Image base │
│ config │ False │ dict │ Additional configuration to be applied to SP Pipeline │
│ │ │ │ Assembly element │
│ configMaps │ False │ List of ConfigMap │ Pre-configured Kubernetes config maps to inject into │
│ │ │ │ pipeline │
│ controller │ False │ Controller │ Configure Pipeline Controller │
│ destination │ False │ string │ Sequencer Bus to publish to │
│ │ │ │ 'destination' is used to look up sequencers defined │
│ │ │ │ within the Assembly 'elements.sequencer' or 'bus' fields. │
│ │ │ │ If 'destination' is not located, it is assumed to be an │
│ │ │ │ existing Sequencer name, external to the Assembly. │
│ env │ False │ List of EnvItem │ Environment Variables to be appended to the element │
│ │ │ │ container │
│ group │ False │ string │ Groups a pipeline into a set of replicas that have a │
│ │ │ │ matching group id │
│ id │ False │ string │ SP Pipeline ID │
│ imagePullSecrets │ False │ List of ImagePullSecret │ Pre-configured Kubernetes imagePullSecrets to inject into │
│ │ │ │ pipeline │
│ maxWorkers │ False │ integer │ Maximum worker instances │
│ metrics │ False │ Metrics │ Configure pipeline metrics │
│ minWorkers │ False │ integer │ Minimum worker instances │
│ monitoring │ False │ boolean │ Enable monitoring on Pipeline pods │
│ name │ False │ string │ SP Pipeline name │
│ protectedExecution │ False │ boolean │ Enable Protected Execution │
│ replicaAffinityTopologyKey │ False │ ReplicaAffinityTopologyKey │ The key of node labels. If two Nodes are labelled with │
│ │ │ │ this key and have identical │
│ │ │ │ values for that label, the scheduler treats both Nodes as │
│ │ │ │ being in the same topology. │
│ │ │ │ Used for affinity and anti-affinity rules related to │
│ │ │ │ replicas. │
│ replicas │ False │ integer │ Number of pipeline replicas │
│ secrets │ False │ List of ImagePullSecret │ Pre-configured Kubernetes secrets to inject into pipeline │
│ source │ False │ string │ Sequencer Bus to subscribe to. │
│ │ │ │ 'source' is used to look up sequencers defined within the │
│ │ │ │ Assembly 'elements.sequencer' or 'bus' fields. │
│ │ │ │ If 'source' is not located, it is assumed to be an │
│ │ │ │ existing Sequencer name, external to the Assembly. │
│ spec │ False │ string │ │
│ type │ False │ Type │ "graph" or "spec" pipeline deployment │
│ unsafeMode │ False │ boolean │ Enable unsafe mode, disables determinism │
│ volumes │ False │ List of Volume1 │ List of volumes to attach to Pipeline │
│ worker │ False │ Worker │ Configure Pipeline Worker │
╰────────────────────────────┴──────────┴────────────────────────────┴───────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Router components
Query routing is managed by two primary components, the Service Gateway and the Resource Coordinator. Routers are used to define the query resources when deploying your package.
For more information on how to configure a router, refer to Routing.
Router fields
kxi package field info Router
Fields for Router
╭────────────┬────────────────┬─────────────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────┼────────────────┼─────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ Name of the router │
│ rc │ False │ Rc │ Resource Coordinator object │
│ agg │ False │ Agg │ Aggregator object │
│ qe │ False │ QueryEnvironment │ Configure Query Environments for Assembly │
╰────────────┴────────────────┴─────────────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Configuration components
Deployment configurations are used to define global parameters for use by all the components when deploying your package. For example, the license information and any environment variables shared by all components.
Deployment config fields
kxi package field info DeploymentConfig
Fields for DeploymentConfig
╭───────────────────┬───────────┬───────────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├───────────────────┼───────────┼───────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ Name of the deployment_config object │
│ attach │ False │ boolean │ Enable tty and stdin for each process in the kdb+ insights cluster │
│ env │ False │ List of EnvItem │ Env vars for every process in the kdb+ insights cluster │
│ imagePullPolicy │ False │ string │ Image pull secret policy │
│ imagePullSecrets │ False │ List of ImagePullSecret1 │ List of image registry secrets │
│ license │ False │ License │ License Object to be shared for all CRs │
│ qlog │ False │ Qlog │ Assembly logging configuration │
╰───────────────────┴───────────┴───────────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
A package only deploys one deployment configuration.
Although multiple deployment configuration definitions are allowed in a package, only one is deployed.
Remove and copy components
The examples below show how to remove and copy components.
kxi package rm --help
Usage: kxi package rm [OPTIONS] COMMAND [ARGS]...
Remove an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Package> Package from which the entity will be removed. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ table Remove a table from the specified package │
│ database Remove a database from the specified package │
│ pipeline Remove a pipeline from the specified package. │
│ dep Remove a pipeline from the specified package. │
│ router Remove a router from the specified package. │
│ entrypoint Remove entrypoint from the specified package. │
│ deployment-config Remove deployment_config form the specified package │
│ patch Remove deployment_config form the specified package │
│ stream Remove stream from the specified package │
│ agg Remove aggregator from the specified package │
│ rc Remove resource_coordinator from the specified package │
│ shard Remove a shard from the specified package │
│ view Remove a View from the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯
kxi package copy --help
Usage: kxi package copy [OPTIONS] COMMAND [ARGS]...
Copy an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Src-Package> Package from which the entity will be copied. │
│ --to -t <Target-Package> Package to which the entity will be │
│ copied;[default] `from`. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ For local development ──────────────────────────────────────────────────────╮
│ pipeline Copy a pipeline from the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯