Skip to content

Pipeline operators

This section outlines the available operators within the pipeline user interface. Pipelines can be built using the user interface or using the q or Python coding interfaces. For more details on the coding interfaces, see either the q API or the Python API

Operators

An pipeline operator lets you apply an operation to data in your pipeline. Readers allow you to connect external data sources and bring data into the pipeline. Writers are sinks within the pipeline and let data flow to a destination. All other operators operate on data within the pipeline by either transforming it, grouping it, merging it or filtering it. Operators are split based on their category and are enumerated below.

Building a pipeline

To learn about more about building a pipeline, see building a pipeline.

  • Readers - Ingest data from an external or internal source.
  • Writers - Send data to an Insights database or to an external destination.
  • Functions - Transform, filter or merge data using q or Python functions.
  • Decoders - Deserialize data from a data structure into kdb+ format.
  • Encoders - Serialize data from kdb+ into a data structure.
  • String - perform string transformation operations on data.
  • Transform - Apply a no-code common data transformation your data.
  • Windows - Group data based on a size or temporal metric for aggregating.
  • Machine Learning - Create models and predictions from your data.