Pipeline Operators Overview
This section outlines the operators available in the Pipeline user interface.
Pipelines can be built using the user interface or using the q or Python coding interfaces. For more details on the coding interfaces, see the q and Python APIs.
The terms operator and nodes are used interchangeably when referring to pipeline components.
Operators
A pipeline operator lets you apply an operation to data in your pipeline.
-
Readers allow you to connect external data sources and bring data into the pipeline.
-
Writers are sinks within the pipeline and let data flow to a destination.
-
All other operators operate on data within the pipeline by either transforming it, grouping it, merging it or filtering it.
Operators can be added to pipelines through the web interface or using APIs. Both methods are detailed below, along with an overview of the functionality provided by each operator category.
| Category | Description | Web Interface Docs | API Docs |
|---|---|---|---|
| Readers | Ingest data from an external or internal source. | Reader nodes | Reader APIs |
| Writers | Send data to a kdb Insights database or to an external destination | Writer nodes | Writer APIs |
| Functions | Transform, filter or merge data using q or Python functions. | Function nodes | Function APIs |
| Decoders | De-serialize data from a data structure into kdb+ format. | Decoder nodes | Decoder APIs |
| Encoders | Serialize data from kdb+ into a data structure. | Encoder nodes | Encoder APIs |
| String | Perform string transformation operations on data. | String nodes | String APIs |
| Stats | Compute various statistics using your data. | Stats nodes | Stats APIs |
| Transform | Apply a no-code common data transformation your data. | Transform nodes | Transform APIs |
| Windows | Group data based on a size or temporal metric for aggregating. | Windows nodes | Windows APIs |
| Machine Learning | Create models and predictions from your data. | Machine Learning nodes | Machine Learning APIs |
Building a pipeline
To learn about more about building a pipeline, see building a pipeline.
To learn more about building pipelines using the Stream Processor APIs read the Stream Processor documentation.
Versioning
kdb Insights Enterprise supports multiple versions of certain operators. When multiple-versions are available a dropdown is displayed in the top-right corner of the node configuration window. The following operators have multiple versions: