Manage Deployment Components
This page provides details of how to manage the deployment components in a package.
The deployment components within a package specify what needs to be deployed, detailing the creation and orchestration of processes.
Database components
The database fields in a package, define the schema, storage manager, stream, and data access processes.
A package only deploys one database component.
Although multiple database definitions are allowed in a package, only one is deployed.
Calling the package add command removes an existing component if it shares the same type and name as the new component.
If you run the following command, the original copy of the mypkg database is overwritten:
kxi package add --to mypkg database
# some editing of the files
kxi package add --to mypkg database
Database fields
The following command shows information on the database fields:
kxi package field info Database
Database
╭──────────────────┬──────────────┬────────────────────────┬─────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────────┼──────────────┼────────────────────────┼─────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ DB Name ( Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$ ) │
│ description │ False │ string │ DB Description │
│ shards │ False │ List of Shard │ Shards in DB │
│ tables │ False │ TableList │ Schemas in DB │
│ qe │ False │ QueryEnvironment │ Query Environment │
╰──────────────────┴──────────────┴────────────────────────┴─────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Shard fields
The following command shows information on the shard fields:
kxi package field info Shard
Shard
╭─────────────────────────┬─────────────────────┬─────────────────────────┬──────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├─────────────────────────┼─────────────────────┼─────────────────────────┼──────────────────────────────────────────────────────┤
│ name │ False │ string │ Name identifier of shard │
│ labels │ False │ dict │ Shard labels │
│ dbSettings │ False │ DbSettings │ Database settings │
│ sm │ False │ Sm │ Storage Manager Object │
│ daps │ False │ Dap │ Data Access object │
│ sequencers │ False │ dict │ Message bus into system │
│ mounts │ False │ dict │ PVs used in package │
╰─────────────────────────┴─────────────────────┴─────────────────────────┴──────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
This resource does not have resource request/limits set. For details, refer to database configuration.
Object resources
Use the kxi package field info command to understand the fields associated with a Shard object. Its resources are contained within a k8sPolicy object. For example, try running kxi package field info Shard. This displays an Sm, Dap, and Sequencer, all of which contain a k8sPolicy object.
By default, resources such as CPU and Memory are not configured in the package. You can choose to set these by modifying the shard.yaml file, located in the shard directory of the package. Most editors support object spec (included at the top of file) to provide autocomplete suggestions for the fields in the shard.yaml file.
Schema fields
The following command shows information on the schema fields:
kxi package field info Table
Table
╭──────────────┬──────────┬────────────────────┬─────────────────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────┼──────────┼────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤
│ columns │ True │ List of Column │ A list of the columns in the table. │
│ │ │ │ The order of this list of columns is the order they will be organized as. │
│ type │ True │ Type3 │ The type of the table is how the table is managed on disk. For time based │
│ │ │ │ tables, use partitioned │
│ │ │ │ One of {splayed, splayed_mem, partitioned, basic}. Required │
│ blockSize │ False │ integer │ This value indicates when data should be written to disk. │
│ │ │ │ After this many records are received, data is written to disk. Writing more │
│ │ │ │ frequently increases disk IO but will use less memory. │
│ │ │ │ If omitted, the default value is 12 million records. │
│ delCol │ False │ string │ The name of the 'delete column'. │
│ │ │ │ This field can only be used for keyed non-partitioned tables. │
│ │ │ │ It must refer to a boolean-typed column of the table. Any rows with the delete │
│ │ │ │ column set to 1b are deleted, i.e. such rows will not show up in query results │
│ │ │ │ and not use storage either. │
│ │ │ │ Deleted keys can be re-introduced with new updates with the delete column set │
│ │ │ │ to 0b. ( Pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ ) │
│ description │ False │ string │ A textual description for the table. This can be used to provide an overview │
│ │ │ │ of the data collected in the current table. │
│ isSharded │ False │ boolean │ Specifies if this table is split across multiple packages. │
│ │ │ │ When using this property, it must be consistent for all instances of this │
│ │ │ │ table in all packages. │
│ name │ False │ string │ Table name ( Pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ ) │
│ oldName │ False │ string │ The name of a previous version of this table. │
│ │ │ │ This field can be used when changing the name of a table within a schema ( │
│ │ │ │ Pattern: ^[a-zA-Z][a-zA-Z0-9_]*$ ) │
│ primaryKeys │ False │ List of PrimaryKey │ Names of columns to use as primary keys for this table. │
│ │ │ │ Primary keys are used for indicating unique data within a table. │
│ │ │ │ When provided, the table is keyed by these columns and any updates that have │
│ │ │ │ matching keys will update records with matching keys. │
│ prtnCol │ False │ string │ The name of the column within the table to use to partition the content. │
│ │ │ │ The column type it points to must be a timestamp. │
│ │ │ │ This value is required if the table type is set to partitioned ( Pattern: │
│ │ │ │ ^[a-zA-Z][a-zA-Z0-9_]*$ ) │
│ sortColsDisk │ False │ List of PrimaryKey │ A list of columns to use for sorting columns in a normal disk tier. │
│ │ │ │ Settings this value will sort data as it is migrated into disk partitions. │
│ │ │ │ If you want data to also have the sorted attribute, set attrDisk to sorted. │
│ sortColsMem │ False │ List of PrimaryKey │ A list of columns to use for sorting columns in a memory tier. │
│ │ │ │ Setting this value will sort data as it arrives in memory. │
│ │ │ │ If you want data to also have the sorted attribute, set attrMem to sorted │
│ sortColsOrd │ False │ List of PrimaryKey │ A list of columns to use for sorting columns in an ordinal partitioned tier. │
│ │ │ │ Setting this value will sort data as it migrates into ordinal partitions. │
│ │ │ │ If you want data to also have the sorted attribute, set attrOrd to sorted │
╰──────────────┴──────────┴────────────────────┴─────────────────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Stream fields
The following command shows information on the stream fields:
kxi package field info Sequencer
Sequencer
╭──────────────────────────┬──────────┬─────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├──────────────────────────┼──────────┼─────────────────────┼────────────────────────────────────────────────────────────────────┤
│ external │ True │ boolean │ External facing Sequencer, setting true enables External IP │
│ annotations │ False │ Annotations │ Annotations to apply to resources created from Sequencer │
│ args │ False │ List of string │ Args defines the arguments to be passed to the container's │
│ │ │ │ entrypoint. │
│ │ │ │ See │
│ │ │ │ https://kubernetes.io/docs/tasks/inject-data-application/define-c… │
│ env │ False │ List of EnvItem │ Env provides a list of environment variable to inject into the │
│ │ │ │ container. │
│ │ │ │ See │
│ │ │ │ https://kubernetes.io/docs/tasks/inject-data-application/define-e… │
│ externalNodePort │ False │ boolean │ Use Node Port Type for Externally facing Sequencer service │
│ image │ False │ Image │ Sequencer image details │
│ k8sPolicy │ False │ K8sPolicy │ Kubernetes specific configurations for element Pods │
│ maxDiskUsagePercent │ False │ integer │ Maximum percentage of the available disk space that will be used │
│ │ │ │ by RT. │
│ │ │ │ When this percentage is exceeded rolled merged log files are │
│ │ │ │ garbage collected, oldest first. │
│ maxLogSize │ False │ string │ Maximum size of all log files in RT. │
│ │ │ │ Rolled merged log files which push the total size beyond this │
│ │ │ │ limit are garbage collected, oldest first. ( Pattern: │
│ │ │ │ ^([+-]?[0-9.]+)([eEinukmgtpKMGTP]*[-+]?[0-9]*)$ ) │
│ name │ False │ string │ Sequencer instance name ( Pattern: ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ │
│ │ │ │ ) │
│ retentionDuration │ False │ integer │ Retention period for merged RT stream log files in minutes. │
│ │ │ │ Rolled merged log files which contain messages older than this │
│ │ │ │ (based on the message timestamp) are garbage collected │
│ sideCarConfig │ False │ SideCarConfig │ Sidecar Config Map Details │
│ sideCarOverrides │ False │ SideCarOverrides │ Override Enabling Sidecar for Sequencer instance │
│ size │ False │ Size │ Size defines the desired number of pods. │
│ │ │ │ This is a pointer to distinguish between explicit zero and not │
│ │ │ │ specified. │
│ staticIPs │ False │ List of string │ Optional list of IPs to append to the External RT LoadBalancers │
│ topicConfig │ False │ TopicConfig │ Sequencer Topic Prefix │
│ topicConfigDir │ False │ string │ Location of RT 'pull' directory ( Pattern: │
│ │ │ │ ^\/+([a-zA-Z0-9_.-]+\/*)+$ ) │
│ useInternalLBAnnotations │ False │ boolean │ Apply Service Annotations to set LoadBalancer Service as internal │
│ │ │ │ only │
│ volume │ False │ Volume │ RT Sequencer directory paths │
│ volumeMounts │ False │ List of VolumeMount │ Mount volumes external to Assembly as defined within spec.Volumes │
╰──────────────────────────┴──────────┴─────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Pipeline components
Pipelines are used to enrich and stream data. You can add multiple pipelines to your package. Refer to the pipeline examples for details.
Pipeline fields
The following command shows information on the pipeline fields:
kxi package field info Pipeline
Pipeline
╭────────────────────────────┬──────────┬────────────────────────────┬───────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────────────────────┼──────────┼────────────────────────────┼───────────────────────────────────────────────────────────┤
│ base │ False │ Base │ Base image to deploy │
│ config │ False │ dict │ Additional configuration to be applied to SP Pipeline │
│ │ │ │ Assembly element │
│ configMaps │ False │ List of ConfigMap │ Pre-configured Kubernetes config maps to inject into │
│ │ │ │ pipeline │
│ controller │ False │ Controller │ Configure Pipeline Controller │
│ destination │ False │ string │ Sequencer Bus to publish to │
│ │ │ │ 'destination' is used to look up sequencers defined │
│ │ │ │ within the Assembly 'elements.sequencer' or 'bus' fields. │
│ │ │ │ If 'destination' is not located, it is assumed to be an │
│ │ │ │ existing Sequencer name, external to the Assembly. ( │
│ │ │ │ Pattern: ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ env │ False │ List of EnvItem │ Env provides a list of environment variable to inject │
│ │ │ │ into the container. │
│ │ │ │ See │
│ │ │ │ https://kubernetes.io/docs/tasks/inject-data-application… │
│ group │ False │ string │ Groups a pipeline into a set of replicas that have a │
│ │ │ │ matching group id ( Pattern: │
│ │ │ │ ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ id │ False │ string │ SP Pipeline ID │
│ imagePullSecrets │ False │ List of ImagePullSecret │ Pre-configured Kubernetes imagePullSecrets to inject into │
│ │ │ │ pipeline │
│ maxWorkers │ False │ integer │ Maximum worker instances │
│ metrics │ False │ Metrics │ Configure pipeline metrics │
│ minWorkers │ False │ integer │ Minimum worker instances │
│ mode │ False │ Mode │ Pipeline mode, one of default or debug │
│ monitoring │ False │ boolean │ Enable monitoring on Pipeline pods │
│ name │ False │ string │ Name of the pipeline deployment ( Pattern: │
│ │ │ │ ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ protectedExecution │ False │ boolean │ Enable Protected Execution │
│ replicaAffinityTopologyKey │ False │ ReplicaAffinityTopologyKey │ Used for affinity and anti-affinity rules related to │
│ │ │ │ replicas │
│ replicas │ False │ integer │ Number of pipeline replicas │
│ secrets │ False │ List of ImagePullSecret │ Pre-configured Kubernetes secrets to inject into pipeline │
│ source │ False │ string │ Sequencer Bus to subscribe to. │
│ │ │ │ 'source' is used to look up sequencers defined within the │
│ │ │ │ Assembly 'elements.sequencer' or 'bus' fields. │
│ │ │ │ If 'source' is not located, it is assumed to be an │
│ │ │ │ existing Sequencer name, external to the Assembly. ( │
│ │ │ │ Pattern: ^[a-z0-9]+[a-z0-9-]*[a-z0-9]+$ ) │
│ spec │ False │ string │ │
│ type │ False │ Type │ "graph" or "spec" pipeline deployment │
│ unsafeMode │ False │ boolean │ Enable unsafe mode, disables determinism │
│ volumes │ False │ List of Volume1 │ List of volumes to attach to Pipeline │
│ worker │ False │ Worker │ Configure Pipeline Worker │
╰────────────────────────────┴──────────┴────────────────────────────┴───────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
This resource does not have resource request/limits set. For details, refer to database configuration.
Object resources
Use the kxi package field info command to understand the fields associated with a Pipeline object. Its resources are contained within a k8sPolicy object. For example, try running kxi package field info Pipeline. This displays both a Worker and a Controller, each of which contains a k8sPolicy object.
By default, resources such as CPU and Memory are not configured in the package. You can choose to set these by modifying the pipe1.yaml file, located in the pipelines directory of the package. Most editors support object spec (included at the top of file) to provide autocomplete suggestions for the fields in the pipe1.yaml file.
Router components
Query routing is managed by two primary components, the Service Gateway and the Resource Coordinator. Routers are used to define the query resources when deploying your package.
For more information on how to configure a router, refer to Routing.
Router fields
The following command shows information on the router fields:
kxi package field info Router
Router
╭────────────┬────────────────┬─────────────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├────────────┼────────────────┼─────────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ Name of the router │
│ rc │ False │ Rc │ Resource Coordinator object │
│ agg │ False │ Agg │ Aggregator object │
│ qe │ False │ QueryEnvironment │ Configure Query Environments for Assembly │
╰────────────┴────────────────┴─────────────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
Configuration components
Deployment configurations are used to define global parameters for use by all the components when deploying your package. For example, the license information and any environment variables shared by all components.
Deployment config fields
The following command shows information on the deployment configuration fields:
kxi package field info DeploymentConfig
DeploymentConfig
╭───────────────────┬───────────┬───────────────────────────┬────────────────────────────────────────────────────────────────────╮
│ field │ required │ type │ description │
├───────────────────┼───────────┼───────────────────────────┼────────────────────────────────────────────────────────────────────┤
│ name │ False │ string │ Name of the deployment_config object │
│ attach │ False │ boolean │ Enable tty and stdin for each process in the kdb+ insights cluster │
│ env │ False │ List of EnvItem │ Env vars for every process in the kdb+ insights cluster │
│ imagePullPolicy │ False │ string │ Image pull secret policy │
│ imagePullSecrets │ False │ List of ImagePullSecret1 │ List of image registry secrets │
│ license │ False │ License │ License Object to be shared for all CRs │
│ qlog │ False │ Qlog │ Assembly logging configuration │
│ bus │ False │ dict │ Messsage bus into system │
╰───────────────────┴───────────┴───────────────────────────┴────────────────────────────────────────────────────────────────────╯
To get more info on a field use: `kxi package field info type` (only works with bold words)
A package only deploys one deployment configuration.
Although multiple deployment configuration definitions are allowed in a package, only one is deployed.
Add components
The example below demonstrates how to add components.
kxi package add --help
Usage: kxi package add [OPTIONS] COMMAND [ARGS]...
Add an entity to the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --to <Package> Package, to which, the entity will be added. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ agg Add aggregator to the specified package │
│ database Add a database to the specified package. │
│ dep Add a package dependency to the specified package. │
│ deployment-config Add deployment_config to the specified package │
│ entrypoint Add an entrypoint to the specified package. │
│ patch Add patch to the specified package │
│ pipeline Add a pipeline to the specified package. │
│ rc Add resource coordinator to the specified package │
│ router Add a router to the specified package. │
│ shard Same as `add database`. [deprecated] │
│ stream Add a stream to the specified database. │
│ table Add a table to the specified package. │
│ view Add a view to the specified package. │
│ workload Adds a workload to the package's workload.yaml. │
╰──────────────────────────────────────────────────────────────────────────────╯
If you want to add a specific component, like a pipeline, run the command as shown in the following example.
kxi package add pipeline --help
Usage: kxi package add pipeline [OPTIONS]
Add a pipeline to the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --name TEXT Name of the pipeline to be added to the │
│ package. │
│ --replicas INTEGER Number of replica pipelines to create. │
│ --spec-file TEXT Source file for pipeline to use │
│ --base [q|py|q-ml|py-ml] Base image to deploy, one of q, py, q-ml or │
│ py-ml │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
For detailed Pipeline info, run:
kxi package field info Pipeline
Remove and copy components
The examples below show how to remove and copy components.
kxi package rm --help
Usage: kxi package rm [OPTIONS] COMMAND [ARGS]...
Remove an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Package> Package from which the entity will be removed. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ agg Remove aggregator from the specified package │
│ database Remove a database from the specified package │
│ dep Remove a dependency from the specified package. │
│ deployment-config Remove deployment_config from the specified package │
│ entrypoint Remove entrypoint from the specified package. │
│ patch Remove patch from the specified package │
│ pipeline Remove a pipeline from the specified package. │
│ rc Remove resource_coordinator from the specified package │
│ router Remove a router from the specified package. │
│ shard Use `rm database` instead. [deprecated] │
│ stream Remove stream from the specified package │
│ table Remove a table from the specified package │
│ view Remove a view from the specified package. │
│ workload Removes a workload from the package's workload.yaml. │
╰──────────────────────────────────────────────────────────────────────────────╯
kxi package copy --help
Usage: kxi package copy [OPTIONS] COMMAND [ARGS]...
Copy an entity from the specified package.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ --from -f <Src-Package> Package from which the entity will be copied. │
│ --to -t <Target-Package> Package to which the entity will be │
│ copied;[default] `from`. │
│ --help Show this message and exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────╮
│ pipeline Copy a pipeline from the specified package. │
╰──────────────────────────────────────────────────────────────────────────────╯