Encoders
Stream Processor encoders.
kxi.sp.encode.__all__ = ['Encoder', 'PayloadType', 'arrow', 'csv', 'json', 'protobuf']
module-attribute
kxi.sp.encode.Encoder
Bases: Operator
kxi.sp.encode.PayloadType
Bases: AutoNameEnum
Enum to specify the payload type for a protobuf encoding.
kxi.sp.encode.PayloadType.table = auto()
class-attribute
table.
kxi.sp.encode.PayloadType.dict = auto()
class-attribute
dictionary.
kxi.sp.encode.PayloadType.array = auto()
class-attribute
array.
kxi.sp.encode.PayloadType.arrays = auto()
class-attribute
arrays.
kxi.sp.encode.PayloadType.auto = auto()
class-attribute
auto.
kxi.sp.encode.ArrowPayloadType
Bases: AutoNameEnum
Enum to specify the payload type for Arrow encoding.
kxi.sp.encode.ArrowPayloadType.table = auto()
class-attribute
table.
kxi.sp.encode.ArrowPayloadType.arrays = auto()
class-attribute
arrays.
kxi.sp.encode.ArrowPayloadType.auto = auto()
class-attribute
auto.
kxi.sp.encode.CSVHeader
Bases: AutoNameEnum
Enum for csv
header options.
These enum values can be provided as enum member objects (e.g. CSVHeader.always
), or as
strings matching the names of the members (e.g. 'always'
).
kxi.sp.encode.CSVHeader.first = auto()
class-attribute
Only first batch starts with a header row.
kxi.sp.encode.CSVHeader.none = auto()
class-attribute
Encoded data never starts with a header row.
kxi.sp.encode.CSVHeader.always = auto()
class-attribute
Encoded data always starts with a header row.
kxi.sp.encode.__dir__
kxi.sp.encode.arrow
Encodes data as Arrow.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
schema |
Optional[Union[kx.Table, Dict[str, Union[str, int, kx.CharAtom]]]]
|
A table from which the schema is extracted and used when serializing each batch of data. If not provided, schema is inferred at runtime for each batch. |
None
|
match_cols |
Union[bool, kx.BooleanAtom]
|
When set to True and a schema is provided, tables passed in have their columns rearranged to match the column ordering of the schema. No reordering is done otherwise. |
True
|
payload_type |
ArrowPayloadType
|
A symbol indicating the message payload that will be encoded (one of auto, table, or arrays). |
ArrowPayloadType.auto
|
Returns:
Type | Description |
---|---|
Encoder
|
An |
kxi.sp.encode.csv
Encodes data as CSV.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
delimiter |
Union[str, bytes, kx.CharAtom]
|
A field separator for the records in the encoded data, defaults to comma. |
kx.q('","')
|
header |
CSVHeader
|
Whether encoded data starts with a header row. |
CSVHeader.first
|
Returns:
Type | Description |
---|---|
Encoder
|
A |
kxi.sp.encode.json
Encodes data as JSON.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
split |
bool
|
Whether a batch should be encoded as a single JSON object, or split into a separate JSON object for each value in the batch. When the input is a table, this encodes each row as a JSON object with the column names as keys mapping to the corresponding row values. |
False
|
Returns:
Type | Description |
---|---|
Encoder
|
A |
kxi.sp.encode.protobuf
Encodes data as per protobuf.
Parameters:
Name | Type | Description | Default |
---|---|---|---|
message |
str
|
The name of the Protocol Buffer message type to encode. |
required |
path |
Optional[str]
|
The path to a |
None
|
format |
Optional[str]
|
A string definition of the Protocol Buffer message format to encode. |
None
|
payload_type |
Optional[PayloadType]
|
A |
None
|
Returns:
Type | Description |
---|---|
Encoder
|
A |