Skip to content

Encoders

Encode a kdb data into an external data format

Encoding allows data to be converted into a format that can be passed to an external system either by writing the content to a static file or by streaming the data to a different system.

See APIs for more details

A q interface can be used to build pipelines programatically. See the q API for API details.

A Python interface is included along side the q interface and can be used if PyKX is enabled. See the Python API for API details.

The pipeline builder uses a drag-and-drop interface to link together operations within a pipeline. For details on how to wire together a transformation, see the building a pipeline guide.

Arrow

(Beta Feature) Encodes kdb Arrow data

Beta Features

Beta feature are included for early feedback and for specific use cases. They are intended to work but have not been marked ready for production use. To learn more and enable beta features, see enabling beta features.

Arrow encoder node properties

See APIs for more details

q API: .qsp.encode.arrow Python API: kxi.sp.encode.arrow

Optional Parameters:

name description default
Payload Type Indicates the message payload that will be passed in the stream. This is used as an optimization if the data shape is known ahead of time. Otherwise, leave set as 'Automatic' Automatic

CSV

Convert data into CSV format

CSV encoder node properties

See APIs for more details

q API: .qsp.encode.csv •  Python API: kxi.sp.encode.csv

Required Parameters:

name description default
Delimiter Field separator for the records in the encoded data ,

Optional Parameters:

name description default
Header Indicates whether encoded data should start with a header row, either None, First or Always. First

JSON

Serialize data into JSON format

JSON encoder node properties

See APIs for more details

q API: .qsp.encode.json •  Python API: kxi.sp.encode.json

Optional Parameters:

name description default
Split By default, batches are encoded as single JSON objects. Split encodes each value in a given batch separately. When the input is a table, this encodes each row as its own JSON object. False

Protocol Buffers

Serialize data into Protocol Buffers

Protocol buffers encoder node properties

See APIs for more details

q API: .qsp.encode.protobuf •  Python API: kxi.sp.encode.protobuf

Required Parameters:

name description default
Message Name The name of the Protocol Buffer message type to decode None
Message Definition A .proto definition containing the expected schema of the data to decode. This definition must include a definition of the Message Name referenced above None

Optional Parameters:

name description default
Payload Type Indicates the message payload that will be passed in the stream. This is used as an optimization if the data shape is known ahead of time. Otherwise, leave set as 'Automatic' Automatic