Encoders

Encode a kdb data into an external data format

Encoding allows data to be converted into a format that can be passed to an external system either by writing the content to a static file or by streaming the data to a different system.

See APIs for more details

A q interface can be used to build pipelines programmatically. See the q API for API details.

A Python interface is included along side the q interface and can be used if PyKX is enabled. See the Python API for API details.

The pipeline builder uses a drag-and-drop interface to link together operations within a pipeline. For details on how to wire together a transformation, see the building a pipeline guide.

Arrow

(Beta Feature) Encodes kdb Arrow data

Beta Features

Beta feature are included for early feedback and for specific use cases. They are intended to work but have not been marked ready for production use. To learn more and enable beta features, see enabling beta features.

See APIs for more details

q API: .qsp.encode.arrow • Python API: kxi.sp.encode.arrow

Optional Parameters:

name	description	default
Payload Type	Indicates the message payload that will be passed in the stream. This is used as an optimization if the data shape is known ahead of time. Otherwise, leave set as 'Automatic'	Automatic

CSV

Convert data into CSV format

See APIs for more details

q API: .qsp.encode.csv •  Python API: kxi.sp.encode.csv

Required Parameters:

name	description	default
Delimiter	Field separator for the records in the encoded data	,

Optional Parameters:

name	description	default
Header	Indicates whether encoded data should start with a header row. Options are Never, First Row, or Always.	First Row

JSON

Serialize data into JSON format

See APIs for more details

q API: .qsp.encode.json •  Python API: kxi.sp.encode.json

Optional Parameters:

name	description	default
Split	By default, batches are encoded as single JSON objects. Split encodes each value in a given batch separately. When the input is a table, this encodes each row as its own JSON object.	No

Protocol Buffers

Serialize data into Protocol Buffers

See APIs for more details

q API: .qsp.encode.protobuf •  Python API: kxi.sp.encode.protobuf

Required Parameters:

name	description	default
Message Name	The name of the Protocol Buffer message type to decode
Message Definition	A .proto definition containing the expected schema of the data to decode. This definition must include a definition of the Message Name referenced above.

Optional Parameters:

name	description	default
Payload Type	Indicates the message payload that will be passed in the stream. This is used as an optimization if the data shape is known ahead of time. Otherwise, leave set as Automatic.	Automatic