Pipeline Settings
When creating a pipeline, a number of options are available to tune the pipeline. The settings tab provides controls for changing pipeline resources, environment variables and tuning pipeline settings.
This document breaks down the settings available.
Runtime
| Setting | Description |
|---|---|
| Protected Execution | Enabled for greater granularity in error reporting. When enabled, operating performance may be impacted. |
| Log Format | Define debug reporting format; select between JSON or Text formats. It is recommended to leave this as JSON for more structured logs. |
| Log Level | Select between Trace, Debug, Info, Warning, Error or Fatal. Lower log levels include higher ones. For example, the default level is Info which includes Warning, Error and Fatal. |
Resources
Resources are the amount of computational resources that this pipeline will be allowed to consume. When allocating resources, it is important to remember that only the worker is responsible for processing data. To increase throughput, increase resources on the worker. The controller resources will only need to be modified when a high degree of parallelism is used.
Controller
CPU
| Setting | Description |
|---|---|
| Minimum CPU | Minimum amount of CPU for the controller; 1 CPU or 1000 mCPU for one core, 0.5 CPU or 500 mCPU is half time for one core. Must be >= 0.1 CPU. |
| Maximum CPU | Maximum amount of CPU for the controller; must be <= 4 CPU. |
Memory
| Setting | Description |
|---|---|
| Minimum Memory | Minimum memory to allocate to the controller and always be available; >= 50 MiB. |
| Maximum Memory | Maximum memory to allocate to the controller; once reached it will return out-of-memory error; <= 10 GiB. |
Worker
CPU
| Setting | Description |
|---|---|
| Minimum CPU | Minimum amount of CPU for for the worker; 1 CPU or 1000 mCPU for one core, 0.5 CPU or 500 mCPU is half time for one core. Must be >= 0.1 CPU. |
| Maximum CPU | Maximum amount of CPU for for the worker. |
Memory
| Setting | Description |
|---|---|
| Minimum Memory | Minimum memory to allocate to the worker and always be available; >= 100 MiB. |
| Maximum Memory | Maximum memory to allocate to the worker; once reached it will return out-of-memory error. Must be <= 500GiB. |
Config
| Setting | Description |
|---|---|
| Minimum number of workers | Define the minimum number of workers that can be created to run the pipeline; >= 1. |
| Maximum number of workers | Define the maximum number of workers that can be created to run the pipeline; <= 10. |
| Maximum number of worker threads | Maximum number of worker threads; value between 1 and 16. |
Metrics
Customize or disable pipeline metrics.
Record counting
Pipeline record counting allows pipelines to calculate an average input and output data rate, which is a good indication of how much data the pipeline is ingesting and outputting. Disabling record counting increases pipeline performance by a small amount.
| Setting | Description |
|---|---|
| Readers and Writers | Enable record counting for all readers and writers, allowing input and output data rates to be calculated. This is enabled by default for all new pipelines created in the web interface. When this is set the input and output data rates are displayed on the Pipelines index. |
| Disabled | Disable record counting. This increases performance but doesn't allow input and output data rates to be calculated. |
Persistence
Controller
Kubernetes persistence configuration.
| Setting | Description |
|---|---|
| Disabled | Enabled by default, click to disable. |
| Storage Class | Kubernetes storage class name; e.g. standard. |
| Storage Size | Size volume allocated to each controller; defaults to 20Gi. |
| Checkpoint Frequency | Check frequency in milliseconds, defaults to 5,000. |
Storage Class
Ensure Storage Class is set to a valid class. This is defined in the cluster and may be different across cloud providers i.e. GCP: standard, AWS: gp2, Azure: default. This information can be retrieved by running:
kubectl get storageclass
Worker
Kubernetes persistence configuration.
| Setting | Description |
|---|---|
| Disabled | Enabled by default, click to disable. |
| Storage Class | Kubernetes storage class name; e.g. standard. |
| Storage Size | Size volume allocated to each Worker; defaults to 20Gi. |
| Checkpoint Frequency | Check frequency in milliseconds, defaults to 5,000. |
Kubernetes
| Setting | Description |
|---|---|
| Label | Add a label and value to created resources. This is a Kubernetes label and can be used to identify resources associated with this pipeline. |
| Image Pull Secrets | Add Kubernetes secret reference name; for example, kx-access. Secrets can be used for mounting in passwords, TLS certificates or just protected data. |
Environment Variables
Environment variables can be used as overrides to customize the behavior of the Stream Processor. See the list of environment variables for available options.
Advanced
| Setting | Description |
|---|---|
| Replicas | The number of pipeline replicas to deploy. The default value of 1 means only a single pipeline is deployed with no replicas. |
| Worker Image | A registry URL that points to a custom Stream Processor worker image. This can be used to extend the core set of the Stream Processor's functionalities with custom code or libraries. |
| Controller Image | A registry URL that points to a custom Stream Processor controller image. |
| Unsafe Mode | This enables unsafe mode, which disables determinism and may lead to duplicate or lost data in failover scenarios. The options are: - Automatic: Inherits the system default (default setting). - Disabled: Enables determinism (safe mode). - Enabled: Disables determinism (unsafe mode). |
Test Timeout
This setting defines the maximum amount of time a test is allowed to run before it automatically times out. It ensures that tests do not hang indefinitely if expected data does not arrive in time.
A timeout is especially useful when the timing of incoming data is unpredictable, such as when the pipeline involves multiple readers. Setting a timeout improves the overall testing experience and removes the need to either use a full deployment for testing or stub out data readers with a callback function.
This timeout applies to both quick and full tests, with an additional 20-second startup buffer for full tests.
| Setting | Description |
|---|---|
| Test Timeout | Enter the timeout duration in seconds. The default value is 10 seconds. |