Encoders
Stream Processor encoders.
kxi.sp.encode.__all__ = ['Encoder', 'PayloadType', 'arrow', 'csv', 'json', 'protobuf']
module-attribute
kxi.sp.encode.Encoder
Bases: Operator
kxi.sp.encode.PayloadType
Bases: AutoNameEnum
Enum to specify the payload type for a protobuf encoding.
kxi.sp.encode.PayloadType.table = auto()
class-attribute
table.
kxi.sp.encode.PayloadType.dict = auto()
class-attribute
dictionary.
kxi.sp.encode.PayloadType.array = auto()
class-attribute
array.
kxi.sp.encode.PayloadType.arrays = auto()
class-attribute
arrays.
kxi.sp.encode.PayloadType.auto = auto()
class-attribute
auto.
kxi.sp.encode.ArrowPayloadType
Bases: AutoNameEnum
Enum to specify the payload type for Arrow encoding.
kxi.sp.encode.ArrowPayloadType.table = auto()
class-attribute
table.
kxi.sp.encode.ArrowPayloadType.arrays = auto()
class-attribute
arrays.
kxi.sp.encode.ArrowPayloadType.auto = auto()
class-attribute
auto.
kxi.sp.encode.CSVHeader
Bases: AutoNameEnum
Enum for csv header options.
These enum values can be provided as enum member objects (e.g. CSVHeader.always), or as
strings matching the names of the members (e.g. 'always').
kxi.sp.encode.CSVHeader.first = auto()
class-attribute
Only first batch starts with a header row.
kxi.sp.encode.CSVHeader.none = auto()
class-attribute
Encoded data never starts with a header row.
kxi.sp.encode.CSVHeader.always = auto()
class-attribute
Encoded data always starts with a header row.
kxi.sp.encode.__dir__
kxi.sp.encode.arrow
Encodes data as Arrow.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
schema |
Optional[Union[kx.Table, Dict[str, Union[str, int, kx.CharAtom]]]]
|
A table from which the schema is extracted and used when serializing each batch of data. If not provided, schema is inferred at runtime for each batch. |
None
|
match_cols |
Union[bool, kx.BooleanAtom]
|
When set to True and a schema is provided, tables passed in have their columns rearranged to match the column ordering of the schema. No reordering is done otherwise. |
True
|
payload_type |
ArrowPayloadType
|
A symbol indicating the message payload that will be encoded (one of auto, table, or arrays). |
ArrowPayloadType.auto
|
Returns:
| Type | Description |
|---|---|
Encoder
|
An |
kxi.sp.encode.csv
Encodes data as CSV.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
delimiter |
Union[str, bytes, kx.CharAtom]
|
A field separator for the records in the encoded data, defaults to comma. |
kx.q('","')
|
header |
CSVHeader
|
Whether encoded data starts with a header row. |
CSVHeader.first
|
Returns:
| Type | Description |
|---|---|
Encoder
|
A |
kxi.sp.encode.json
Encodes data as JSON.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
split |
bool
|
Whether a batch should be encoded as a single JSON object, or split into a separate JSON object for each value in the batch. When the input is a table, this encodes each row as a JSON object with the column names as keys mapping to the corresponding row values. |
False
|
Returns:
| Type | Description |
|---|---|
Encoder
|
A |
kxi.sp.encode.protobuf
Encodes data as per protobuf.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
message |
str
|
The name of the Protocol Buffer message type to encode. |
required |
path |
Optional[str]
|
The path to a |
None
|
format |
Optional[str]
|
A string definition of the Protocol Buffer message format to encode. |
None
|
payload_type |
Optional[PayloadType]
|
A |
None
|
Returns:
| Type | Description |
|---|---|
Encoder
|
A |