Skip to content

Pipeline deploy options

When saving or deploying a pipeline, a number of options are available to tune the pipeline. The deploy dialog provides controls for changing pipeline resources, environment variables an tuning pipeline settings.

Deploy dialog

This document breaks down the settings available for each of the deployment options.

Runtime

item description
Error Mode Select between Off, Suspend or Dump Stack. Don't use Suspend in a production environment. When error mode is off, errors will be logged to the console but processing will continue. If error mode is set to suspend, it will halt the pipeline when an error is encountered. This mode is only recommended for debugging. If error mode is set to dump stack, the pipeline will halt and dump the stack in the event of an error.
Protected Execution Enabled for greater granularity in error reporting. When enabled, operating performance may be impacted.
Log Format Define debug reporting format; select between JSON or Text formats. It is recommended to leave this as JSON for more structured logs.
Log Level Select between Trace, Debug, Info, Warning, Error or Fatal. Lower log levels include higher ones. For example, the default level is Info which includes Warning, Error and Fatal.

Resources

Resources are the amount of computational resources that this pipeline will be allowed to consume. When allocating resources, it is important to remember that only the worker is responsible for processing data. To increase throughput, increase resources on the worker. The controller resources will only need to be modified when a high degree of parallelism is used.

Controller

CPU

item description
Minimum CPU Minimum amount of CPU for pipeline; 1 for one core, 0.5 is half time for one core (>=0.01 CPU).
Maximum CPU Maximum amount of CPU for pipeline; must be <= 4 CPU.

Memory

item description
Minimum Memory Minimum memory to allocate to pipeline and always be available; >= 1 MB.
Maximum Memory Maximum memory to allocate to pipeline; once reached it will return out-of-memory error; <= 2000 MB.

Worker

CPU

item description
Minimum CPU Minimum amount of CPU for pipeline; 1 for one core, 0.5 is half time for one core (>=0.01 CPU).
Maximum CPU Maximum amount of CPU for pipeline.

Memory

item description
Minimum Memory Minimum memory to allocate to pipeline and always be available; >= 1 MB.
Maximum Memory Maximum memory to allocate to pipeline; once reached it will return out-of-memory error.

Config

item description
Minimum number of workers Define the minimum number of workers that can be created to run the pipeline; >= 1.
Maximum number of workers Define the maximum number of workers that can be created to run the pipeline; <= 10.
Maximum number of worker threads Maximum number of worker threads; value between 1 and 16.

Persistence

Controller

Kubernetes persistence configuration.

item description
Disabled Enabled by default, click to disable.
Storage Class Kubernetes storage class name; e.g. standard.
Storage Size Size volume allocated to each controller; defaults to 20Gi.
Checkpoint Frequency Check frequency in milliseconds, defaults to 5,000.

Storage Class

Ensure Storage Class is set to a valid class. This is defined in the cluster and may be different across cloud providers i.e. GCP: standard, AWS: gp2, Azure: default. This information can be retrieved by running:

kubectl get storageclass

Worker

Kubernetes persistence configuration.

item description
Disabled Enabled by default, click to disable.
Storage Class Kubernetes storage class name; e.g. standard.
Storage Size Size volume allocated to each Worker; defaults to 20Gi.
Checkpoint Frequency Check frequency in milliseconds, defaults to 5,000.

Kubernetes

item description
Label Add a label and value to created resources. This is a Kubernetes label and can be used to identify resources associated with this pipeline.
Image Pull Secrets Add Kubernetes secret reference name; for example, kx-access. Secrets can be used for mounting in passwords, TLS certificates or just protected data.

Environment Variables

Environment variables can be used as overrides to customize the behavior of the Stream Processor. See the list of environment variables for available options.

Advanced

Worker Image
A registry URL that points to a custom Stream Processor worker image. This can be used for extending the core set of functionality of the Stream Processor with custom code or libraries.
Controller Image
A registry URL that points to a custom Stream Processor controller image.