Encoders
This page explains how to set up encoder operators for kdb Insights Enterprise pipelines using the Web Interface.
Encoding allows data to be converted into a format that can be passed to an external system either by writing the content to a static file or by streaming the data to a different system.
See APIs for more details
Both q and Python interfaces can be used to build pipelines programmatically. See the q and Python APIs for details.
The pipeline builder uses a drag-and-drop interface to link together operations within a pipeline. For details on how to wire together a transformation, see the building a pipeline guide.
Arrow
(Beta Feature) The Arrow operator encodes kdb Arrow data.
Beta Features
Beta feature are included for early feedback and for specific use cases. They are intended to work but have not been marked ready for production use. To learn more and enable beta features, see enabling beta features.
See APIs for more details
q and Python APIs: .qsp.encode.arrow
Optional Parameters:
| name | description | default |
|---|---|---|
| Payload Type | Indicates the message payload that will be passed in the stream. This is used as an optimization if the data shape is known ahead of time. Otherwise, leave set as 'Automatic' | Automatic |
CSV
This operator converts data into CSV format.
See APIs for more details
q and Python APIs: CSV
Required Parameters:
| name | description | default |
|---|---|---|
| Delimiter | Field separator for the records in the encoded data | , |
Optional Parameters:
| name | description | default |
|---|---|---|
| Header | Indicates whether encoded data should start with a header row. Options are Never, First Row, or Always. |
First Row |
JSON
The JSON operator serializes data into JSON format.
See APIs for more details
q and Python APIs: JSON
Optional Parameters:
| name | description | default |
|---|---|---|
| Split | By default, batches are encoded as single JSON objects. Split encodes each value in a given batch separately. When the input is a table, this encodes each row as its own JSON object. | No |
Protocol Buffers
The Protocol Buffers operator serializes data into Protocol Buffers.
See APIs for more details
q and Python API: Protocol Buffer
Required Parameters:
| name | description | default |
|---|---|---|
| Message Name | The name of the Protocol Buffer message type to decode | |
| Message Definition | A .proto definition containing the expected schema of the data to decode. This definition must include a definition of the Message Name referenced above. |
Optional Parameters:
| name | description | default |
|---|---|---|
| Payload Type | Indicates the message payload that will be passed in the stream. This is used as an optimization if the data shape is known ahead of time. Otherwise, leave set as Automatic. |
Automatic |