Function reference

These functions are exposed within the .arrowkdb namespace, allowing users to convert data between the Arrow/Parquet and kdb+.

KxSystems/arrowkdb

.arrowkdb Arrow/Parquet interface Datatype constructors dt.na Create a NULL datatype dt.boolean Create a boolean datatype dt.int8 Create an int8 datatype dt.int16 Create an int16 datatype dt.int32 Create an int32 datatype dt.int64 Create an int64 datatype dt.uint8 Create an uint8 datatype dt.uint16 Create an uint16 datatype dt.uint32 Create an uint32 datatype dt.uint64 Create an uint64 datatype dt.float16 Create a float16 (represented as uint16_t) datatype dt.float32 Create a float32 datatype dt.float64 Create a float64 datatype dt.time32 Create a 32-bit time (units since midnight with specified granularity) datatype dt.time64 Create a 64-bit time (units since midnight with specified granularity) datatype dt.timestamp Create a 64-bit timestamp (units since UNIX epoch with specified granularity) datatype dt.date32 Create a 32-bit date (days since UNIX epoch) datatype dt.date64 Create a 64-bit date (milliseconds since UNIX epoch) datatype dt.month_interval Create a 32-bit interval (described as a number of months, similar to YEAR_MONTH in SQL) datatype dt.day_time_interval Create a 64-bit interval (described as a number of days and milliseconds, similar to DAY_TIME in SQL) datatype dt.duration Create a 64-bit duration (measured in units of specified granularity) datatype dt.binary Create a variable length bytes datatype dt.utf8 Create a UTF8 variable length string datatype dt.large_binary Create a large (64-bit offsets) variable length bytes datatype dt.large_utf8 Create a large (64-bit offsets) UTF8 variable length string datatype dt.fixed_size_binary Create a fixed width bytes datatype dt.decimal128 Create a 128-bit integer (with precision and scale in twos complement) datatype dt.list Create a list datatype, specified in terms of its child datatype dt.large_list Create a large (64-bit offsets) list datatype, specified in terms of its child datatype dt.fixed_size_list Create a fixed size list datatype, specified in terms of its child datatype dt.map Create a map datatype, specified in terms of its key and item child datatypes dt.struct Create a struct datatype, specified in terms of the field identifiers of its children dt.sparse_union Create a sparse union datatype, specified in terms of the field identifiers of its children dt.dense_union Create a dense union datatype, specified in terms of the field identifiers of its children dt.dictionary Create a dictionary datatype specified in terms of its value and index datatypes, similar to pandas categorical dt.inferDatatype Infer and construct a datatype from a kdb+ list

Datatype inspection dt.datatypeName Return the base name of a datatype, ignoring any parameters or child datatypes/fields dt.getTimeUnit Return the TimeUnit of a time32/time64/timestamp/duration datatype dt.getByteWidth Return the byte_width of a fixed_size_binary datatype dt.getListSize Returns the list_size of a fixed_size_list datatype dt.getPrecisionScale Return the precision and scale of a decimal128 datatype dt.getListDatatype Return the child datatype identifier of a list/large_list/fixed_size_list datatype dt.getMapDatatypes Return the key and item child datatype identifiers of a map datatype dt.getDictionaryDatatypes Return the value and index child datatype identifiers of a dictionary datatype dt.getChildFields Return the list of child field identifiers of a struct/spare_union/dense_union datatype

Datatype management dt.printDatatype Display user readable information for a datatype, including parameters and nested child datatypes dt.listDatatypes Return the list of identifiers for all datatypes held in the DatatypeStore dt.removeDatatype Remove a datatype from the DatatypeStore dt.equalDatatypes Check if two datatypes are logically equal, including parameters and nested child datatypes

Field Constructor fd.field Create a field instance from its name and datatype

Field Inspection fd.fieldName Return the name of a field fd.fieldDatatype Return the datatype of a field

Field management fd.printField Display user readable information for a field, including name and datatype fd.listFields Return the list of identifiers for all fields held in the FieldStore fd.removeField Remove a field from the FieldStore fd.equalFields Check if two fields are logically equal, including names and datatypes

Schema constructors sc.schema Create a schema instance from a list of field identifiers sc.inferSchema Infer and construct a schema based on a kdb+ table

Schema inspection sc.schemaFields Return the list of field identifiers used by a schema

Schema management sc.printSchema Display user readable information for a schema, including its fields and their order sc.listSchemas Return the list of identifiers for all schemas held in the SchemaStore sc.removeSchema Remove a schema from the SchemaStore sc.equalSchemas Check if two schemas are logically equal, including their fields and the fields' order

Array data ar.prettyPrintArray Convert a kdb+ list to an Arrow array and pretty print the array ar.prettyPrintArrayFromList Convert a kdb+ list to an Arrow array and pretty print the array, inferring the datatype from the kdb+ list type

Table data tb.prettyPrintTable Convert a kdb+ mixed list of array data to an Arrow table and pretty print the table tb.prettyPrintTableFromTable Convert a kdb+ table to an Arrow table and pretty print the table, inferring the schema from the kdb+ table structure

Parquet files pq.writeParquet Convert a kdb+ mixed list of array data to an Arrow table and write to a Parquet file pq.writeParquetFromTable Convert a kdb+ table to an Arrow table and write to a Parquet file, inferring the schema from the kdb+ table structure pq.readParquetSchema Read the schema from a Parquet file pq.readParquetData Read an Arrow table from a Parquet file and convert to a kdb+ mixed list of array data pq.readParquetColumn Read a single column from a Parquet file and convert to a kdb+ list pq.readParquetToTable Read an Arrow table from a Parquet file and convert to a kdb+ table

Arrow IPC files ipc.writeArrow Convert a kdb+ mixed list of array data to an Arrow table and write to an Arrow file ipc.writeArrowFromTable Convert a kdb+ table to an Arrow table and write to an Arrow file, inferring the schema from the kdb+ table structure ipc.readArrowSchema Read the schema from an Arrow file ipc.readArrowData Read an Arrow table from an Arrow file and convert to a kdb+ mixed list of array data ipc.readArrowToTable Read an Arrow table from an Arrow file and convert to a kdb+ table

Arrow IPC streams ipc.serializeArrow Convert a kdb+ mixed list of array data to an Arrow table and serialize to an Arrow stream ipc.serializeArrowFromTable Convert a kdb+ table to an Arrow table and serialize to an Arrow stream, inferring the schema from the kdb+ table structure ipc.parseArrowSchema Parse the schema from an Arrow stream ipc.parseArrowData Parse an Arrow table from an Arrow stream and convert to a kdb+ mixed list of array data ipc.parseArrowToTable Parse an Arrow table from an Arrow file and convert to a kdb+ table

Utilities util.buildInfo Return build information regarding the in use Arrow library

Datatype constructors

`dt.na`

Create a NULL datatype

.arrowkdb.dt.na[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.na[]]
null
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.na[];(();();())]
3 nulls

`dt.boolean`

Create a boolean datatype

.arrowkdb.dt.boolean[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.boolean[]]
bool
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.boolean[];(010b)]
[
  false,
  true,
  false
]

`dt.int8`

Create an int8 datatype

.arrowkdb.dt.int8[]

kdb+ type 10h can be written to an int8 array

The is supported on the writing path only. Reading from an int8 array returns a 4h list

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.int8[]]
int8
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.int8[];(0x102030)]
[
  16,
  32,
  48
]

`dt.int16`

Create an int16 datatype

.arrowkdb.dt.int16[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.int16[]]
int16
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.int16[];(11 22 33h)]
[
  11,
  22,
  33
]

`dt.int32`

Create an int32 datatype

.arrowkdb.dt.int32[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.int32[]]
int32
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.int32[];(11 22 33i)]
[
  11,
  22,
  33
]

`dt.int64`

Create an int64 datatype

.arrowkdb.dt.int64[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.int64[]]
int64
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.int64[];(11 22 33j)]
[
  11,
  22,
  33
]

`dt.uint8`

Create an uint8 datatype

.arrowkdb.dt.uint8[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.uint8[]]
uint8
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.uint8[];(0x102030)]
[
  16,
  32,
  48
]

`dt.uint16`

Create an uint16 datatype

.arrowkdb.dt.uint16[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.uint16[]]
uint16
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.uint16[];(11 22 33h)]
[
  11,
  22,
  33
]

`dt.uint32`

Create an uint32 datatype

.arrowkdb.dt.uint32[]

Returns the datatype identifier

uint32 datatype is supported by Parquet v2.0 only, being changed to int64 otherwise

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.uint32[]]
uint32
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.uint32[];(11 22 33i)]
[
  11,
  22,
  33
]

`dt.uint64`

Create an uint64 datatype

.arrowkdb.dt.uint64[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.uint64[]]
uint64
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.uint64[];(11 22 33j)]
[
  11,
  22,
  33
]

`dt.float16`

Create a float16 (represented as uint16_t) datatype

.arrowkdb.dt.float16[]

Returns the datatype identifier

float16 datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.float16[]]
halffloat
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.float16[];(11 22 33h)]
[
  11,
  22,
  33
]

`dt.float32`

Create a float32 datatype

.arrowkdb.dt.float32[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.float32[]]
float
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.float32[];(1.1 2.2 3.3e)]
[
  1.1,
  2.2,
  3.3
]

`dt.float64`

Create a float64 datatype

.arrowkdb.dt.float64[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.float64[]]
double
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.float64[];(1.1 2.2 3.3f)]
[
  1.1,
  2.2,
  3.3
]

`dt.time32`

Create a 32-bit time (units since midnight with specified granularity) datatype

.arrowkdb.dt.time32[time_unit]

Where time_unit is the time unit string: SECOND or MILLI

returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.time32[`MILLI]]
time32[ms]
q).arrowkdb.dt.getTimeUnit[.arrowkdb.dt.time32[`MILLI]]
`MILLI
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.time32[`MILLI];(01:00:00.100 02:00:00.200 03:00:00.300)]
[
  01:00:00.100,
  02:00:00.200,
  03:00:00.300
]

`dt.time64`

Create a 64-bit time (units since midnight with specified granularity) datatype

.arrowkdb.dt.time64[time_unit]

Where time_unit is the time unit string: MICRO or NANO

returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.time64[`NANO]]
time64[ns]
q).arrowkdb.dt.getTimeUnit[.arrowkdb.dt.time64[`NANO]]
`NANO
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.time64[`NANO];(0D01:00:00.100000001 0D02:00:00.200000002 0D03:00:00.300000003)]
[
  01:00:00.100000001,
  02:00:00.200000002,
  03:00:00.300000003
]

`dt.timestamp`

Create a 64-bit timestamp (units since UNIX epoch with specified granularity) datatype

.arrowkdb.dt.timestamp[time_unit]

Where time_unit is the time unit string: SECOND, MILLI, MICRO or NANO

returns the datatype identifier

timestamp(nano) datatype is supported by Parquet v2.0 only, being mapped to timestamp(milli) otherwise

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.timestamp[`NANO]]
timestamp[ns]
q).arrowkdb.dt.getTimeUnit[.arrowkdb.dt.timestamp[`NANO]]
`NANO
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.timestamp[`NANO];(2001.01.01D00:00:00.100000001 2002.02.02D00:00:00.200000002 2003.03.03D00:00:00.300000003)]
[
  2001-01-01 00:00:00.100000001,
  2002-02-02 00:00:00.200000002,
  2003-03-03 00:00:00.300000003
]

`dt.date32`

Create a 32-bit date (days since UNIX epoch) datatype

.arrowkdb.dt.date32[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.date32[]]
date32[day]
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.date32[];(2001.01.01 2002.02.02 2003.03.03)]
[
  2001-01-01,
  2002-02-02,
  2003-03-03
]

`dt.date64`

Create a 64-bit date (milliseconds since UNIX epoch) datatype

.arrowkdb.dt.date64[]

Returns the datatype identifier

date64 datatype is changed to date32(days) by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.date64[]]
date64[ms]
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.date64[];(2001.01.01D00:00:00.000000000 2002.02.02D00:00:00.000000000 2003.03.03D00:00:00.000000000)]
[
  2001-01-01,
  2002-02-02,
  2003-03-03
]

`dt.month_interval`

Create a 32-bit interval (described as a number of months, similar to YEAR_MONTH in SQL) datatype

.arrowkdb.dt.month_interval[]

Returns the datatype identifier

month_interval datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.month_interval[]]
month_interval
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.month_interval[];(2001.01m,2002.02m,2003.03m)]
[
  12,
  25,
  38
]

`dt.day_time_interval`

Create a 64-bit interval (described as a number of days and milliseconds, similar to DAY_TIME in SQL) datatype

.arrowkdb.dt.day_time_interval[]

Returns the datatype identifier

day_time_interval datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.day_time_interval[]]
day_time_interval
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.day_time_interval[];(0D01:00:00.100000000 0D02:00:00.200000000 0D03:00:00.300000000)]
[
  0d3600100ms,
  0d7200200ms,
  0d10800300ms
]

`dt.duration`

Create a 64-bit duration (measured in units of specified granularity) datatype

.arrowkdb.dt.duration[time_unit]

Where time_unit is the time unit string: SECOND, MILLI, MICRO or NANO

returns the datatype identifier

duration datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.duration[`NANO]]
duration[ns]
q).arrowkdb.dt.getTimeUnit[.arrowkdb.dt.duration[`NANO]]
`NANO
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.duration[`NANO];(0D01:00:00.100000000 0D02:00:00.200000000 0D03:00:00.300000000)]
[
  3600100000000,
  7200200000000,
  10800300000000
]

`dt.binary`

Create a variable length bytes datatype

.arrowkdb.dt.binary[]

Returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.binary[]]
binary
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.binary[];(enlist 0x11;0x2222;0x333333)]
[
  11,
  2222,
  333333
]

`dt.utf8`

Create a UTF8 variable length string datatype

.arrowkdb.dt.utf8[]

Returns the datatype identifier

kdb+ type 11h can be written to an utf8 array

The is supported on the writing path only. Reading from an utf8 array returns a mixed list of 10h

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.utf8[]]
string
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.utf8[];(enlist "a";"bb";"ccc")]
[
  "a",
  "bb",
  "ccc"
]

`dt.large_binary`

Create a large (64-bit offsets) variable length bytes datatype

.arrowkdb.dt.large_binary[]

Returns the datatype identifier

large_binary datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.large_binary[]]
large_binary
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.large_binary[];(enlist 0x11;0x2222;0x333333)]
[
  11,
  2222,
  333333
]

`dt.large_utf8`

Create a large (64-bit offsets) UTF8 variable length string datatype

.arrowkdb.dt.large_utf8[]

Returns the datatype identifier

large_utf8 datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.large_utf8[]]
large_string
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.large_utf8[];(enlist "a";"bb";"ccc")]
[
  "a",
  "bb",
  "ccc"
]

`dt.fixed_size_binary`

Create a fixed width bytes datatype

.arrowkdb.dt.fixed_size_binary[byte_width]

Where byte_width is the int32 fixed size byte width (each value in the array occupies the same number of bytes).

returns the datatype identifier

kdb+ type 2h can be written to a fixed_size_binary(16) array

The is supported on the writing path only. Reading from a fixed_size_binary array returns a mixed list of 4h

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.fixed_size_binary[2i]]
fixed_size_binary[2]
q).arrowkdb.dt.getByteWidth[.arrowkdb.dt.fixed_size_binary[2i]]
2i
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.fixed_size_binary[2i];(0x1111;0x2222;0x3333)]
[
  1111,
  2222,
  3333
]

`dt.decimal128`

Create a 128-bit integer (with precision and scale in twos complement) datatype

.arrowkdb.dt.decimal128[precision;scale]

Where:

precision is the int32 precision width
scale is the int32 scaling factor

returns the datatype identifier

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.decimal128[38i;2i]]
decimal(38, 2)
q).arrowkdb.dt.getPrecisionScale[.arrowkdb.dt.decimal128[38i;2i]]
38
2
q).arrowkdb.ar.prettyPrintArray[.arrowkdb.dt.decimal128[38i;2i];(0x00000000000000000000000000000000; 0x01000000000000000000000000000000; 0x00000000000000000000000000000080)]
[
  0.00,
  0.01,
  -1701411834604692317316873037158841057.28
]
q) // With little endian twos complement the decimal128 values are 0, minimum positive, maximum negative

`dt.list`

Create a list datatype, specified in terms of its child datatype

.arrowkdb.dt.list[child_datatype_id]

Where child_datatype_id is the identifier of the list’s child datatype

returns the datatype identifier

q)list_datatype:.arrowkdb.dt.list[.arrowkdb.dt.int64[]]
q).arrowkdb.dt.printDatatype[list_datatype]
list<item: int64>
q).arrowkdb.dt.printDatatype[.arrowkdb.dt.getListDatatype[list_datatype]]
int64
q).arrowkdb.ar.prettyPrintArray[list_datatype;((enlist 1);(2 2);(3 3 3))]
[
  [
    1
  ],
  [
    2,
    2
  ],
  [
    3,
    3,
    3
  ]
]

`dt.large_list`

Create a large (64-bit offsets) list datatype, specified in terms of its child datatype

.arrowkdb.dt.large_list[child_datatype_id]

Where child_datatype_id is the identifier of the list’s child datatype

returns the datatype identifier

q)list_datatype:.arrowkdb.dt.large_list[.arrowkdb.dt.int64[]]
q).arrowkdb.dt.printDatatype[list_datatype]
large_list<item: int64>
q).arrowkdb.dt.printDatatype[.arrowkdb.dt.getListDatatype[list_datatype]]
int64
q).arrowkdb.ar.prettyPrintArray[list_datatype;((enlist 1);(2 2);(3 3 3))]
[
  [
    1
  ],
  [
    2,
    2
  ],
  [
    3,
    3,
    3
  ]
]

`dt.fixed_size_list`

Create a fixed size list datatype, specified in terms of its child datatype

.arrowkdb.dt.fixed_size_list[child_datatype_id;list_size]

Where:

child_datatype_id is the identifier of the list’s child datatype
list_size is the int32 fixed size of each of the child lists

returns the datatype identifier

fixed_size_list datatype is changed to list by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q)list_datatype:.arrowkdb.dt.fixed_size_list[.arrowkdb.dt.int64[];2i]
q).arrowkdb.dt.printDatatype[list_datatype]
fixed_size_list<item: int64>[2]
q).arrowkdb.dt.printDatatype[.arrowkdb.dt.getListDatatype[list_datatype]]
int64
q).arrowkdb.dt.getListSize[list_datatype]
2i
q).arrowkdb.ar.prettyPrintArray[list_datatype;((1 1);(2 2);(3 3))]
[
  [
    1,
    1
  ],
  [
    2,
    2
  ],
  [
    3,
    3
  ]
]

`dt.map`

Create a map datatype, specified in terms of its key and item child datatypes

.arrowkdb.dt.map[key_datatype_id;item_datatype_id]

Where:

key_datatype_id is the identifier of the map key child datatype
item_datatype_id is the identifier of the map item child datatype

returns the datatype identifier

q)map_datatype:.arrowkdb.dt.map[.arrowkdb.dt.int64[];.arrowkdb.dt.float64[]]
q).arrowkdb.dt.printDatatype[map_datatype]
map<int64, double>
q).arrowkdb.dt.printDatatype each .arrowkdb.dt.getMapDatatypes[map_datatype]
int64
double
::
::
q).arrowkdb.ar.prettyPrintArray[map_datatype;((enlist 1)!(enlist 1f);(2 2)!(2 2f);(3 3 3)!(3 3 3f))]
[
  keys:
  [
    1
  ]
  values:
  [
    1
  ],
  keys:
  [
    2,
    2
  ]
  values:
  [
    2,
    2
  ],
  keys:
  [
    3,
    3,
    3
  ]
  values:
  [
    3,
    3,
    3
  ]
]

`dt.struct`

Create a struct datatype, specified in terms of the field identifiers of its children

.arrowkdb.dt.struct[field_ids]

Where field_ids is the list of field identifiers of the struct’s children

returns the datatype identifier

q)field_one:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)field_two:.arrowkdb.fd.field[`utf8_field;.arrowkdb.dt.utf8[]]
q)struct_datatype:.arrowkdb.dt.struct[field_one,field_two]
q).arrowkdb.dt.printDatatype[struct_datatype]
struct<int_field: int64 not null, utf8_field: string not null>
q).arrowkdb.fd.fieldName each .arrowkdb.dt.getChildFields[struct_datatype]
`int_field`utf8_field
q).arrowkdb.dt.printDatatype each .arrowkdb.fd.fieldDatatype each .arrowkdb.dt.getChildFields[struct_datatype]
int64
string
::
::
q).arrowkdb.ar.prettyPrintArray[struct_datatype;((1 2 3);("aa";"bb";"cc"))]
-- is_valid: all not null
-- child 0 type: int64
  [
    1,
    2,
    3
  ]
-- child 1 type: string
  [
    "aa",
    "bb",
    "cc"
  ]
q) // By slicing across the lists the logical struct values are: (1,"aa"); (2,"bb"); (3,"cc")

`dt.sparse_union`

Create a sparse union datatype, specified in terms of the field identifiers of its children

.arrowkdb.dt.sparse_union[field_ids]

Where field_ids is the list of field identifiers of the union’s children

returns the datatype identifier

An arrow union array is similar to a struct array except that it has an additional type_id array which identifies the live field in each union value set.

sparse_union datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q)field_one:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)field_two:.arrowkdb.fd.field[`utf8_field;.arrowkdb.dt.utf8[]]
q)union_datatype:.arrowkdb.dt.sparse_union[field_one,field_two]
q).arrowkdb.dt.printDatatype[union_datatype]
sparse_union<int_field: int64 not null=0, utf8_field: string not null=1>
q).arrowkdb.fd.fieldName each .arrowkdb.dt.getChildFields[union_datatype]
`int_field`utf8_field
q).arrowkdb.dt.printDatatype each .arrowkdb.fd.fieldDatatype each .arrowkdb.dt.getChildFields[union_datatype]
int64
string
::
::
q).arrowkdb.ar.prettyPrintArray[union_datatype;((1 0 1h);(1 2 3);("aa";"bb";"cc"))]
-- is_valid: all not null
-- type_ids:   [
    1,
    0,
    1
  ]
-- child 0 type: int64
  [
    1,
    2,
    3
  ]
-- child 1 type: string
  [
    "aa",
    "bb",
    "cc"
  ]
q) // Looking up the type_id array the logical union values are: "aa", 2, "cc"

`dt.dense_union`

Create a dense union datatype, specified in terms of the field identifiers of its children

.arrowkdb.dt.dense_union[field_ids]

Where field_ids is the list of field identifiers of the union’s children

returns the datatype identifier

An arrow union array is similar to a struct array except that it has an additional type_id array which identifies the live field in each union value set.

dense_union datatype is not supported by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q)field_one:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)field_two:.arrowkdb.fd.field[`utf8_field;.arrowkdb.dt.utf8[]]
q)union_datatype:.arrowkdb.dt.dense_union[field_one,field_two]
q).arrowkdb.dt.printDatatype[union_datatype]
dense_union<int_field: int64 not null=0, utf8_field: string not null=1>
q).arrowkdb.fd.fieldName each .arrowkdb.dt.getChildFields[union_datatype]
`int_field`utf8_field
q).arrowkdb.dt.printDatatype each .arrowkdb.fd.fieldDatatype each .arrowkdb.dt.getChildFields[union_datatype]
int64
string
::
::
q).arrowkdb.ar.prettyPrintArray[union_datatype;((1 0 1h);(1 2 3);("aa";"bb";"cc"))]
-- is_valid: all not null
-- type_ids:   [
    1,
    0,
    1
  ]
-- value_offsets:   [
    0,
    0,
    0
  ]
-- child 0 type: int64
  [
    1,
    2,
    3
  ]
-- child 1 type: string
  [
    "aa",
    "bb",
    "cc"
  ]
q) // Looking up the type_id array the logical union values are: "aa", 2, "cc"

`dt.dictionary`

Create a dictionary datatype specified in terms of its value and index datatypes, similar to pandas categorical

.arrowkdb.dt.dictionary[value_datatype_id;index_datatype_id]

Where:

value_datatype_id is the identifier of the dictionary value datatype, must be a scalar type
index_datatype_id is the identifier of the dictionary index datatype, must be a signed int type

returns the datatype identifier

Only the categorical interpretation of a dictionary datatype array is saved by Parquet

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q)dict_datatype:.arrowkdb.dt.dictionary[.arrowkdb.dt.utf8[];.arrowkdb.dt.int64[]]
q).arrowkdb.dt.printDatatype[dict_datatype]
dictionary<values=string, indices=int64, ordered=0>
q).arrowkdb.dt.printDatatype each .arrowkdb.dt.getDictionaryDatatypes[dict_datatype]
string
int64
::
::
q).arrowkdb.ar.prettyPrintArray[dict_datatype;(("aa";"bb";"cc");(2 0 1 0 0))]

-- dictionary:
  [
    "aa",
    "bb",
    "cc"
  ]
-- indices:
  [
    2,
    0,
    1,
    0,
    0
  ]
q) // The categorical interpretation of the dictionary (looking up the values set at each index) would be: "cc", "aa", "bb", "aa", "aa"

`dt.inferDatatype`

Infer and construct a datatype from a kdb+ list

.arrowkdb.dt.inferDatatype[list]

Where list is a kdb+ list

returns the datatype identifier

The kdb+ list type is mapped to an Arrow datatype as described here.

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.inferDatatype[(1 2 3j)]]
int64
q).arrowkdb.dt.printDatatype[.arrowkdb.dt.inferDatatype[("aa";"bb";"cc")]]
string

Datatype inspection

`dt.datatypeName`

Return the base name of a datatype, ignoring any parameters or child datatypes/fields

.arrowkdb.dt.datatypeName[datatype_id]

Where datatype_id is the identifier of the datatype

returns a symbol containing the base name of the datatype

q).arrowkdb.dt.datatypeName[.arrowkdb.dt.int64[]]
`int64
q).arrowkdb.dt.datatypeName[.arrowkdb.dt.fixed_size_binary[4i]]
`fixed_size_binary

`dt.getTimeUnit`

Return the TimeUnit of a time32/time64/timestamp/duration datatype

.arrowkdb.dt.getTimeUnit[datatype_id]

Where datatype_id is the identifier of the datatype

returns a symbol containing the time unit string: SECOND/MILLI/MICRO/NANO

q).arrowkdb.dt.getTimeUnit[.arrowkdb.dt.timestamp[`NANO]]
`NANO

`dt.getByteWidth`

Return the byte_width of a fixed_size_binary datatype

.arrowkdb.dt.getByteWidth[datatype_id]

Where datatype_id is the identifier of the datatype

returns the int32 byte width

q).arrowkdb.dt.getByteWidth[.arrowkdb.dt.fixed_size_binary[4i]]
4i

`dt.getListSize`

Returns the list_size of a fixed_size_list datatype

.arrowkdb.dt.getListSize[datatype_id]

Where datatype_id is the identifier of the datatype

returns the int32 list size

q).arrowkdb.dt.getListSize[.arrowkdb.dt.fixed_size_list[.arrowkdb.dt.int64[];4i]]
4i

`dt.getPrecisionScale`

Return the precision and scale of a decimal128 datatype

.arrowkdb.dt.getPrecisionScale[datatype_id]

Where datatype_id is the identifier of the datatype

returns the int32 precision and scale

q).arrowkdb.dt.getPrecisionScale[.arrowkdb.dt.decimal128[38i;2i]]
38
2

`dt.getListDatatype`

Return the child datatype identifier of a list/large_list/fixed_size_list datatype

.arrowkdb.dt.getListDatatype[datatype_id]

Where datatype_id is the identifier of the datatype

returns the list’s child datatype identifier

q)list_datatype:.arrowkdb.dt.list[.arrowkdb.dt.int64[]]
q).arrowkdb.dt.printDatatype[.arrowkdb.dt.getListDatatype[list_datatype]]
int64

`dt.getMapDatatypes`

Return the key and item child datatype identifiers of a map datatype

.arrowkdb.dt.getMapDatatypes[datatype_id]

Where datatype_id is the identifier of the datatype

returns the map’s key and item child datatype identifiers

q)map_datatype:.arrowkdb.dt.map[.arrowkdb.dt.int64[];.arrowkdb.dt.float64[]]
q).arrowkdb.dt.printDatatype each .arrowkdb.dt.getMapDatatypes[map_datatype]
int64
double
::
::

`dt.getDictionaryDatatypes`

Return the value and index child datatype identifiers of a dictionary datatype

.arrowkdb.dt.getDictionaryDatatypes[datatype_id]

Where datatype_id is the identifier of the datatype

returns the dictionary’s value and index child datatype identifiers

q)dict_datatype:.arrowkdb.dt.dictionary[.arrowkdb.dt.utf8[];.arrowkdb.dt.int64[]]
q).arrowkdb.dt.printDatatype each .arrowkdb.dt.getDictionaryDatatypes[dict_datatype]
string
int64
::
::

`dt.getChildFields`

Return the list of child field identifiers of a struct/spare_union/dense_union datatype

.arrowkdb.dt.getChildFields[datatype_id]

Where datatype_id is the identifier of the datatype

returns the list of child field identifiers

q)field_one:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)field_two:.arrowkdb.fd.field[`utf8_field;.arrowkdb.dt.utf8[]]
q)struct_datatype:.arrowkdb.dt.struct[field_one,field_two]
q).arrowkdb.fd.printField each .arrowkdb.dt.getChildFields[struct_datatype]
int_field: int64 not null
utf8_field: string not null
::
::

Datatype management

`dt.printDatatype`

Display user-readable information for a datatype, including parameters and nested child datatypes

.arrowkdb.dt.printDatatype[datatype_id]

Where datatype_id is the identifier of the datatype,

prints datatype information to stdout
returns generic null

For debugging use only

The information is generated by the arrow::DataType::ToString() functionality and displayed on stdout to preserve formatting and indentation.

q).arrowkdb.dt.printDatatype[.arrowkdb.dt.fixed_size_list[.arrowkdb.dt.int64[];4i]]
fixed_size_list<item: int64>[4]

`dt.listDatatypes`

Return the list of identifiers for all datatypes held in the DatatypeStore

.arrowkdb.dt.listDatatypes[]

Returns list of datatype identifiers

q).arrowkdb.dt.int64[]
1i
q).arrowkdb.dt.float64[]
2i
q).arrowkdb.dt.printDatatype each .arrowkdb.dt.listDatatypes[]
int64
double
::
::

`dt.removeDatatype`

Remove a datatype from the DatatypeStore

.arrowkdb.dt.removeDatatype[datatype_id]

Where datatype_id is the identifier of the datatype

returns generic null on success

q).arrowkdb.dt.int64[]
1i
q).arrowkdb.dt.float64[]
2i
q).arrowkdb.dt.listDatatypes[]
1 2i
q).arrowkdb.dt.removeDatatype[1i]
q).arrowkdb.dt.listDatatypes[]
,2i

`dt.equalDatatypes`

Check if two datatypes are logically equal, including parameters and nested child datatypes

.arrowkdb.dt.equalDatatypes[first_datatype_id;second_datatype_id]

Where:

first_datatype_id is the identifier of the first datatype
second_datatype_id is the identifier of the second datatype

returns boolean result

Internally the DatatypeStore uses the equalDatatypes functionality to prevent a new datatype identifier being created when an equal datatype is already present in the DatatypeStore, returning the existing datatype identifier instead.

q).arrowkdb.dt.equalDatatypes[.arrowkdb.dt.int64[];.arrowkdb.dt.int64[]]
1b
q).arrowkdb.dt.equalDatatypes[.arrowkdb.dt.int64[];.arrowkdb.dt.float64[]]
0b
q).arrowkdb.dt.equalDatatypes[.arrowkdb.dt.fixed_size_binary[4i];.arrowkdb.dt.fixed_size_binary[4i]]
1b
q).arrowkdb.dt.equalDatatypes[.arrowkdb.dt.fixed_size_binary[2i];.arrowkdb.dt.fixed_size_binary[4i]]
0b
q).arrowkdb.dt.equalDatatypes[.arrowkdb.dt.list[.arrowkdb.dt.int64[]];.arrowkdb.dt.list[.arrowkdb.dt.int64[]]]
1b
q).arrowkdb.dt.equalDatatypes[.arrowkdb.dt.list[.arrowkdb.dt.int64[]];.arrowkdb.dt.list[.arrowkdb.dt.float64[]]]
0b

Field constructor

`fd.field`

Create a field instance from its name and datatype

.arrowkdb.fd.field[field_name;datatype_id]

Where:

field_name is a symbol containing the field’s name
datatype_id is the identifier of the field’s datatype

returns the field identifier

q).arrowkdb.fd.printField[.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]]
int_field: int64 not null

Field inspection

`fd.fieldName`

Name of a field

.arrowkdb.fd.fieldName[field_id]

Where field_id is the field identifier

returns a symbol containing the field’s name

q)field:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q).arrowkdb.fd.fieldName[field]
`int_field

`fd.fieldDatatype`

Datatype of a field

.arrowkdb.fd.fieldDatatype[field_id]

Where field_id is the field identifier

returns the datatype identifier

q)field:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q).arrowkdb.dt.printDatatype[.arrowkdb.fd.fieldDatatype[field]]
int64

Field management

`fd.printField`

Display user readable information for a field, including name and datatype

.arrowkdb.fd.printField[field_id]

Where field_id is the identifier of the field,

prints field information to stdout
returns generic null

For debugging use only

The information is generated by the arrow::Field::ToString() functionality and displayed on stdout to preserve formatting and indentation.

q).arrowkdb.fd.printField[.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]]
int_field: int64 not null

`fd.listFields`

List of identifiers for all fields held in the FieldStore

.arrowkdb.fd.listFields[]

Returns list of field identifiers

q).arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
1i
q).arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
2i
q).arrowkdb.fd.printField each .arrowkdb.fd.listFields[]
int_field: int64 not null
float_field: double not null
::
::

`fd.removeField`

Remove a field from the FieldStore

.arrowkdb.fd.removeField[field_id]

Where field_id is the identifier of the field

returns generic null on success

q).arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
1i
q).arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
2i
q).arrowkdb.fd.listFields[]
1 2i
q).arrowkdb.fd.removeField[1i]
q).arrowkdb.fd.listFields[]
,2i

`fd.equalFields`

Check if two fields are logically equal, including names and datatypes

.arrowkdb.fd.equalDatatypes[first_field_id;second_field_id]

Where:

first_field_id is the identifier of the first field
second_field_id is the identifier of the second field

returns boolean result

Internally the FieldStore uses the equalFields functionality to prevent a new field identifier being created when an equal field is already present in the FieldStore, returning the existing field identifier instead.

q)int_dt:.arrowkdb.dt.int64[]
q)float_dt:.arrowkdb.dt.float64[]
q).arrowkdb.fd.equalFields[.arrowkdb.fd.field[`f1;int_dt];.arrowkdb.fd.field[`f1;int_dt]]
1b
q).arrowkdb.fd.equalFields[.arrowkdb.fd.field[`f1;int_dt];.arrowkdb.fd.field[`f2;int_dt]]
0b
q).arrowkdb.fd.equalFields[.arrowkdb.fd.field[`f1;int_dt];.arrowkdb.fd.field[`f1;float_dt]]
0b

Schema constructors

`sc.schema`

Create a schema instance from a list of field identifiers

.arrowkdb.sc.schema[field_ids]

Where fields_ids is a list of field identifiers

returns the schema identifier

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q).arrowkdb.sc.printSchema[.arrowkdb.sc.schema[(f1,f2)]]
int_field: int64 not null
float_field: double not null

`sc.inferSchema`

Infer and construct a schema based on a kdb+ table

.arrowkdb.sc.inferSchema[table]

Where table is a kdb+ table or dictionary

returns the schema identifier

Inferred schemas only support a subset of the Arrow datatypes and is considerably less flexible than creating them with the datatype/field/schema constructors

Each column in the table is mapped to a field in the schema. The column name is used as the field name and the column’s kdb+ type is mapped to an Arrow datatype as as described here.

q)schema_from_table:.arrowkdb.sc.inferSchema[([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))]
q).arrowkdb.sc.printSchema[schema_from_table]
int_field: int64
float_field: double
str_field: string

Schema inspection

`sc.schemaFields`

Return the list of field identifiers used by a schema

.arrowkdb.sc.schemaFields[schema_id]

Where schema_id is the schema identifier

returns list of field identifiers used by the schema

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)schema:.arrowkdb.sc.schema[(f1,f2)]
q).arrowkdb.fd.printField each .arrowkdb.sc.schemaFields[schema]
int_field: int64 not null
float_field: double not null
::
::

Schema management

`sc.printSchema`

Display user readable information for a schema, including its fields and their order

.arrowkdb.sc.printSchema[schema_id]

Where schema_id is the identifier of the schema,

prints schema information to stdout
returns generic null

For debugging use only

The information is generated by the arrow::Schema::ToString() functionality and displayed on stdout to preserve formatting and indentation.

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q).arrowkdb.sc.printSchema[schema]
int_field: int64 not null
float_field: double not null
str_field: string not null

`sc.listSchemas`

Return the list of identifiers for all schemas held in the SchemaStore

.arrowkdb.sc.listSchemas[]

Returns list of schema identifiers

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q).arrowkdb.sc.schema[(f1,f2)]
1i
q).arrowkdb.sc.schema[(f2,f1)]
2i
q).arrowkdb.sc.listSchemas[]
1 2i

`sc.removeSchema`

Remove a schema from the SchemaStore

.arrowkdb.sc.removeSchema[schema_id]

Where schema_id is the identifier of the schema

returns generic null on success

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q).arrowkdb.sc.schema[(f1,f2)]
1i
q).arrowkdb.sc.schema[(f2,f1)]
2i
q).arrowkdb.sc.listSchemas[]
1 2i
q).arrowkdb.sc.removeSchema[1i]
q).arrowkdb.sc.listSchemas[]
,2i

`sc.equalSchemas`

Check if two schemas are logically equal, including their fields and the fields' order

.arrowkdb.sc.equalSchemas[first_schema_id;second_schema_id]

Where:

first_schema_id is the identifier of the first schema
second_schema_id is the identifier of the second schema

returns boolean result

Internally the SchemaStore uses the equalSchemas functionality to prevent a new schema identifier being created when an equal schema is already present in the SchemaStore, returning the existing schema identifier instead.

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q).arrowkdb.sc.schema[(f1,f2)]
1i
q).arrowkdb.sc.schema[(f2,f1)]
2i
q).arrowkdb.sc.equalSchemas[.arrowkdb.sc.schema[(f1,f2)];.arrowkdb.sc.schema[(f1,f2)]]
1b
q).arrowkdb.sc.equalSchemas[.arrowkdb.sc.schema[(f1,f2)];.arrowkdb.sc.schema[(f1,f1)]]
0b
q).arrowkdb.sc.equalSchemas[.arrowkdb.sc.schema[(f1,f2)];.arrowkdb.sc.schema[(f2,f1)]]
0b

Array data

`ar.prettyPrintArray`

Convert a kdb+ list to an Arrow array and pretty print the array

.arrowkdb.ar.prettyPrintArray[datatype_id;list]

Where:

datatype_id is the datatype identifier of the array
list is the kdb+ list data to be displayed

the function

prints array contents to stdout
returns generic null

For debugging use only

The information is generated by the arrow::PrettyPrint() functionality and displayed on stdout to preserve formatting and indentation.

q)int_datatype:.arrowkdb.dt.int64[]
q).arrowkdb.ar.prettyPrintArray[int_datatype;(1 2 3j)]
[
  1,
  2,
  3
]

`ar.prettyPrintArrayFromList`

Convert a kdb+ list to an Arrow array and pretty print the array, inferring the datatype from the kdb+ list type

.arrowkdb.ar.prettyPrintArrayFromList[list]

Where list is the kdb+ list data to be displayed

the function

prints array contents to stdout
returns generic null

The kdb+ list type is mapped to an Arrow datatype as described here.

For debugging use only

The information is generated by the arrow::PrettyPrint() functionality and displayed on stdout to preserve formatting and indentation.

q).arrowkdb.ar.prettyPrintArrayFromList[(1 2 3j)]
[
  1,
  2,
  3
]

Table data

`tb.prettyPrintTable`

Convert a kdb+ mixed list of array data to an Arrow table and pretty print the table

.arrowkdb.tb.prettyPrintTable[schema_id;array_data]

Where:

schema_id is the schema identifier of the table
array_data is a mixed list of array data

the function

prints table contents to stdout
returns generic null

The mixed list of Arrow array data should be ordered in schema field number and each list item representing one of the arrays must be structured according to the field’s datatype.

For debugging use only

The information is generated by the arrow::Table::ToString() functionality and displayed on stdout to preserve formatting and indentation.

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q).arrowkdb.tb.prettyPrintTable[schema;((1 2 3j);(4 5 6f);("aa";"bb";"cc"))]
int_field: int64 not null
float_field: double not null
str_field: string not null
----
int_field:
  [
    [
      1,
      2,
      3
    ]
  ]
float_field:
  [
    [
      4,
      5,
      6
    ]
  ]
str_field:
  [
    [
      "aa",
      "bb",
      "cc"
    ]
  ]

`tb.prettyPrintTableFromTable`

Convert a kdb+ table to an Arrow table and pretty print the table, inferring the schema from the kdb+ table structure

.arrowkdb.tb.prettyPrintTableFromTable[table]

Where table is a kdb+ table

the function

prints table contents to stdout
returns generic null

Each column in the table is mapped to a field in the schema. The column name is used as the field name and the column’s kdb+ type is mapped to an Arrow datatype as as described here.

Inferred schemas only support a subset of the Arrow datatypes and is considerably less flexible than creating them with the datatype/field/schema constructors

Each column in the table is mapped to a field in the schema. The column name is used as the field name and the column’s kdb+ type is mapped to an Arrow datatype as as described here.

For debugging use only

The information is generated by the arrow::Table::ToString() functionality and displayed on stdout to preserve formatting and indentation.

q).arrowkdb.tb.prettyPrintTableFromTable[([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))]
int_field: int64
float_field: double
str_field: string
----
int_field:
  [
    [
      1,
      2,
      3
    ]
  ]
float_field:
  [
    [
      4,
      5,
      6
    ]
  ]
str_field:
  [
    [
      "aa",
      "bb",
      "cc"
    ]
  ]

Parquet files

`pq.writeParquet`

Convert a kdb+ mixed list of array data to an Arrow table and write to a Parquet file

.arrowkdb.pq.writeParquet[parquet_file;schema_id;array_data;options]

Where:

parquet_file is a string containing the Parquet file name
schema_id is the schema identifier to use for the table
array_data is a mixed list of array data
options is a dictionary of symbol options to long/symbol values (pass :: to use defaults)

returns generic null on success

The mixed list of Arrow array data should be ordered in schema field number and each list item representing one of the arrays must be structured according to the field’s datatype.

Supported options:

PARQUET_CHUNK_SIZE - Controls the approximate size of encoded data pages within a column chunk (long, default: 1MB)
PARQUET_VERSION - Select the Parquet format version, either V1.0 or V2.0. V2.0 is more fully featured but may be incompatible with older Parquet implementations (symbol, default V1.0)

The Parquet format is compressed and designed for for maximum space efficiency which may cause a performance overhead compared to Arrow. Parquet is also less fully featured than Arrow which can result in schema limitations

The Parquet file format is less fully featured compared to Arrow and consequently the Arrow/Parquet file writer currently does not support some datatypes or represents them using a different datatype as described here

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.pq.writeParquet["file.parquet";schema;array_data;::]
q)read_data:.arrowkdb.pq.readParquetData["file.parquet";::]
q)array_data~read_data
1b

`pq.writeParquetFromTable`

Convert a kdb+ table to an Arrow table and write to a Parquet file, inferring the schema from the kdb+ table structure

.arrowkdb.pq.writeParquetFromTable[parquet_file;table;options]

Where:

parquet_file is a string containing the Parquet file name
table is a kdb+ table
options is a dictionary of symbol options to long/symbol values (pass :: to use defaults)

returns generic null on success

Supported options:

PARQUET_CHUNK_SIZE - Controls the approximate size of encoded data pages within a column chunk (long, default: 1MB)
PARQUET_VERSION - Select the Parquet format version, either V1.0 or V2.0. V2.0 is more fully featured but may be incompatible with older Parquet implementations (symbol, default V1.0)

Inferred schemas only support a subset of the Arrow datatypes and is considerably less flexible than creating them with the datatype/field/schema constructors

Each column in the table is mapped to a field in the schema. The column name is used as the field name and the column’s kdb+ type is mapped to an Arrow datatype as as described here.

q)table:([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))
q).arrowkdb.pq.writeParquetFromTable["file.parquet";table;::]
q)read_table:.arrowkdb.pq.readParquetToTable["file.parquet";::]
q)read_table~table
1b

`pq.readParquetSchema`

Read the schema from a Parquet file

.arrowkdb.pq.readParquetSchema[parquet_file]

Where parquet_file is a string containing the Parquet file name

returns the schema identifier

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.pq.writeParquet["file.parquet";schema;array_data;::]
q).arrowkdb.sc.equalSchemas[schema;.arrowkdb.pq.readParquetSchema["file.parquet"]]
1b

`pq.readParquetData`

Read an Arrow table from a Parquet file and convert to a kdb+ mixed list of array data

.arrowkdb.pq.readParquetData[parquet_file;options]

Where:

parquet_file is a string containing the Parquet file name
options is a dictionary of symbol options to long values (pass :: to use defaults)

returns the array data

Supported options:

PARQUET_MULTITHREADED_READ - Flag indicating whether the Parquet reader should run in multithreaded mode. This can improve performance by processing multiple columns in parallel (long, default: 0)
USE_MMAP - Flag indicating whether the Parquet file should be memory mapped in. This can improve performance on systems which support mmap (long, default: 0)

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.pq.writeParquet["file.parquet";schema;array_data;::]
q)read_data:.arrowkdb.pq.readParquetData["file.parquet";::]
q)array_data~read_data
1b

`pq.readParquetColumn`

Read a single column from a Parquet file and convert to a kdb+ list

.arrowkdb.pq.readParquetColumn[parquet_file;column_index]

Where:

parquet_file is a string containing the Parquet file name
column_index is the index of the column to read, relative to the schema field order

returns the array’s data

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.pq.writeParquet["file.parquet";schema;array_data;::]
q)col1:.arrowkdb.pq.readParquetColumn["file.parquet";1i]
q)col1~array_data[1]
1b

`pq.readParquetToTable`

Read an Arrow table from a Parquet file and convert to a kdb+ table

.arrowkdb.pq.readParquetToTable[parquet_file;options]

Where:

parquet_file is a string containing the Parquet file name
options is a dictionary of symbol options to long values (pass :: to use defaults)

returns the kdb+ table

Each schema field name is used as the column name and the Arrow array data is used as the column data.

Supported options:

PARQUET_MULTITHREADED_READ - Flag indicating whether the Parquet reader should run in multithreaded mode. This can improve performance by processing multiple columns in parallel (long, default: 0)
USE_MMAP - Flag indicating whether the Parquet file should be memory mapped in. This can improve performance on systems which support mmap (long, default: 0)

q)table:([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))
q).arrowkdb.pq.writeParquetFromTable["file.parquet";table;::]
q)read_table:.arrowkdb.pq.readParquetToTable["file.parquet";::]
q)read_table~table
1b

Arrow IPC files

`ipc.writeArrow`

Convert a kdb+ mixed list of array data to an Arrow table and write to an Arrow file

.arrowkdb.ipc.writeArrow[arrow_file;schema_id;array_data]

Where:

arrow_file is a string containing the Arrow file name
schema_id is the schema identifier to use for the table
array_data is a mixed list of array data

returns generic null on success

The mixed list of Arrow array data should be ordered in schema field number and each list item representing one of the arrays must be structured according to the field’s datatype.

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.ipc.writeArrow["file.arrow";schema;array_data]
q)read_data:.arrowkdb.ipc.readArrowData["file.arrow";::]
q)read_data~array_data
1b

`ipc.writeArrowFromTable`

Convert a kdb+ table to an Arrow table and write to an Arrow file, inferring the schema from the kdb+ table structure

.arrowkdb.ipc.writeArrowFromTable[arrow_file;table]

Where:

arrow_file is a string containing the Arrow file name
table is a kdb+ table

returns generic null on success

Inferred schemas only support a subset of the Arrow datatypes and is considerably less flexible than creating them with the datatype/field/schema constructors

Each column in the table is mapped to a field in the schema. The column name is used as the field name and the column’s kdb+ type is mapped to an Arrow datatype as as described here.

q)table:([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))
q).arrowkdb.ipc.writeArrowFromTable["file.arrow";table]
q)read_table:.arrowkdb.ipc.readArrowToTable["file.arrow";::]
q)read_table~table
1b

`ipc.readArrowSchema`

Read the schema from an Arrow file

.arrowkdb.ipc.readArrowSchema[arrow_file]

Where arrow_file is a string containing the Arrow file name

returns the schema identifier

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.ipc.writeArrow["file.arrow";schema;array_data]
q).arrowkdb.sc.equalSchemas[schema;.arrowkdb.ipc.readArrowSchema["file.arrow"]]
1b

`ipc.readArrowData`

Read an Arrow table from an Arrow file and convert to a kdb+ mixed list of array data

.arrowkdb.ipc.readArrowData[arrow_file;options]

Where:

arrow_file is a string containing the Arrow file name
options is a dictionary of symbol options to long values (pass :: to use defaults)

returns the array data

Supported options:

USE_MMAP - Flag indicating whether the Arrow file should be memory mapped in. This can improve performance on systems which support mmap (long, default: 0)

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q).arrowkdb.ipc.writeArrow["file.arrow";schema;array_data]
q)read_data:.arrowkdb.ipc.readArrowData["file.arrow";::]
q)read_data~array_data
1b

`ipc.readArrowToTable`

Read an Arrow table from an Arrow file and convert to a kdb+ table

.arrowkdb.ipc.readArrowToTable[arrow_file;options]

Where:

arrow_file is a string containing the Arrow file name
options is a dictionary of symbol options to long values (pass :: to use defaults)

returns the kdb+ table

Each schema field name is used as the column name and the Arrow array data is used as the column data.

Supported options:

USE_MMAP - Flag indicating whether the Arrow file should be memory mapped in. This can improve performance on systems which support mmap (long, default: 0)

q)table:([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))
q).arrowkdb.ipc.writeArrowFromTable["file.arrow";table]
q)read_table:.arrowkdb.ipc.readArrowToTable["file.arrow";::]
q)read_table~table
1b

Arrow IPC streams

`ipc.serializeArrow`

Convert a kdb+ mixed list of array data to an Arrow table and serialize to an Arrow stream

.arrowkdb.ipc.serializeArrow[schema_id;array_data]

Where:

schema_id is the schema identifier to use for the table
array_data is a mixed list of array data

returns a byte list containing the serialized stream data

The mixed list of Arrow array data should be ordered in schema field number and each list item representing one of the arrays must be structured according to the field’s datatype.

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q)serialized:.arrowkdb.ipc.serializeArrow[schema;array_data]
q)read_data:.arrowkdb.ipc.parseArrowData[serialized]
q)read_data~array_data
1b

`ipc.serializeArrowFromTable`

Convert a kdb+ table to an Arrow table and serialize to an Arrow stream, inferring the schema from the kdb+ table structure

.arrowkdb.ipc.serializeArrowFromTable[table]

Where table is a kdb+ table

returns a byte list containing the serialized stream data

Inferred schemas only support a subset of the Arrow datatypes and is considerably less flexible than creating them with the datatype/field/schema constructors

Each column in the table is mapped to a field in the schema. The column name is used as the field name and the column’s kdb+ type is mapped to an Arrow datatype as as described here.

q)table:([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))
q)serialized:.arrowkdb.ipc.serializeArrowFromTable[table]
q)new_table:.arrowkdb.ipc.parseArrowToTable[serialized]
q)new_table~table
1b

`ipc.parseArrowSchema`

Parse the schema from an Arrow stream

.arrowkdb.ipc.parseArrowSchema[serialized]

Where serialized is a byte list containing the serialized stream data

returns the schema identifier

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q)serialized:.arrowkdb.ipc.serializeArrow[schema;array_data]
q).arrowkdb.sc.equalSchemas[schema;.arrowkdb.ipc.parseArrowSchema[serialized]]
1b

`ipc.parseArrowData`

Parse an Arrow table from an Arrow stream and convert to a kdb+ mixed list of array data

.arrowkdb.ipc.parseArrowData[serialized]

Where serialized is a byte list containing the serialized stream data

returns the array data

q)f1:.arrowkdb.fd.field[`int_field;.arrowkdb.dt.int64[]]
q)f2:.arrowkdb.fd.field[`float_field;.arrowkdb.dt.float64[]]
q)f3:.arrowkdb.fd.field[`str_field;.arrowkdb.dt.utf8[]]
q)schema:.arrowkdb.sc.schema[(f1,f2,f3)]
q)array_data:((1 2 3j);(4 5 6f);("aa";"bb";"cc"))
q)serialized:.arrowkdb.ipc.serializeArrow[schema;array_data]
q)read_data:.arrowkdb.ipc.parseArrowData[serialized]
q)read_data~array_data
1b

`ipc.parseArrowToTable`

Parse an Arrow table from an Arrow file and convert to a kdb+ table

.arrowkdb.ipc.parseArrowToTable[serialized]

Where serialized is a byte list containing the serialized stream data

returns the kdb+ table

Each schema field name is used as the column name and the Arrow array data is used as the column data.

q)table:([] int_field:(1 2 3); float_field:(4 5 6f); str_field:("aa";"bb";"cc"))
q)serialized:.arrowkdb.ipc.serializeArrowFromTable[table]
q)new_table:.arrowkdb.ipc.parseArrowToTable[serialized]
q)new_table~table
1b

Utilities

`util.buildInfo`

Return build information regarding the in use Arrow library

.arrowkdb.util.buildInfo[]

Returns a dictionary detailing various Arrow build info including: Arrow version, shared object version, git description and compiler used.

q).arrowkdb.util.buildInfo[]
version         | 3000000i
version_string  | `3.0.0-SNAPSHOT
full_so_version | `300.0.0
compiler_id     | `MSVC
compiler_version| `19.26.28806.0
compiler_flags  | `/DWIN32 /D_WINDOWS  /GR /EHsc /D_SILENCE_TR1_NAMESPACE_DEP..
git_id          | `c8c2110cd7d01d2f4420079c450997ef5fa89029
git_description | `apache-arrow-2.0.0-194-gc8c2110cd
package_kind    | `