Protobuf/Protocol Buffers function reference¶
Functions exposed in the .protobufkdb
namespace allow you to generate and parse Protobuf messages
.protobufkdb. Protobuf/Protocol Buffers interface
Library information version Return the libprotobuf version as an integer versionStr Return the libprotobuf version as a string
Import schema addProtoImportPath Add a path from which to import proto schema files importProtoFile Import a proto schema file listImportedMessageTypes List successfully imported message schemas
Inspect schema displayMessageSchema Display the schema definition of the message getMessageFields Get the list of message fields
Serialize/parse using list serializeArrayFromList Serialize from a kdb+ mixed list object to a string serializeArrayArenaFromList Serialize from a kdb+ mixed list object to a string, using a Google Arena for the intermediate message parseArrayToList Parse from a string to a kdb+ mixed list object parseArrayArenaToList Parse from a string to a kdb+ mixed list object, using a Google Arena for the intermediate message
Serialize/parse using dictionary serializeArrayFromDict Serialize from a kdb+ dictionary to a string serializeArrayArenaFromDict Serialize from a kdb+ dictionary to a string, using a Google Arena for the intermediate message parseArrayToDict Parse from a string to a kdb+ dictionary parseArrayArenaToDict Parse from a string to a kdb+ dictionary, using a Google Arena for the intermediate message
Save/load using list saveMessageFromList Serialize from a kdb+ mixed list object to a file loadMessageToList Parse from a file to a kdb+ mixed list object
Save/load using dictionary saveMessageFromDict Serialize from a kdb+ dictionary to a file loadMessageToDict Parse from a file to a kdb+ dictionary
Debugging parseArrayDebug Parse from a string and display debugging loadMessageDebug Parse from a file and display debugging
Message types
Where a function takes a message_type
parameter to specify the name of the message to be be processed, the interface first looks for that message type in the compiled messages.
If that fails it then searches the imported message definitions. Only if the message type is not found in either is an error returned.
addProtoImportPath
¶
Add an import path
.protobufkdb.addProtoImportPath[import_path]
Where import_path
is a path (absolute or relative) as a string
- adds it as a path to be searched when dynamically importing
.proto
file definitions - returns generic null
Establishes virtual file system mappings such that the import locations appear under the current directory, similar to the PATH
environment variable. Can be called more than once to specify multiple import locations.
Where your .proto
file definition imports other .proto
files (including recursively), these must also be available on the import paths.
Importing Google's .proto files
The regular .proto
files provided by Google are available in the install package (either when downloaded or built from source) under the proto
subdirectory.
For examples kdb_type_specifier.proto
imports google/protobuf/descriptor.proto
which is available in the proto
subdirectory.
// successful execution
q).protobufkdb.addProtoImportPath["../proto"]
// incorrect data format provided as input
q).protobufkdb.addProtoImportPath[enlist 1f]
'"Specify import path"
displayMessageSchema
¶
Display the Proto schema of a specified message
.protobufkdb.displayMessageSchema[message_type]
Where message_type
is the name of a message type as a string,
- prints schema format to stdout
- returns generic null
message_type
must match a message name in its .proto
definition.
For use only in debugging
The schema is generated by the libprotobuf DebugString()
functionality and displayed on stdout to preserve formatting and indentation.
q).protobufkdb.displayMessageSchema[`ScalarExampleDynamic]
message ScalarExampleDynamic {
int32 scalar_int32 = 1;
double scalar_double = 2;
string scalar_string = 3;
}
getMessageFields
¶
Get the list of message fields
.protobufkdb.getMessageFields[message_type]
Where message_type
is a message type (string or symbol) matching a message name in the .proto
definition
returns the symbol list of message fields in the order they are declared in the message definition.
q).protobufkdb.getMessageFields[`ScalarExampleDynamic]
`scalar_int32`scalar_double`scalar_string
importProtoFile
¶
Import a .proto
file definition
.protobufkdb.importProtoFile[filename]
Where filename
is the name of a .proto
file as a string
- dynamically imports the file definition into the interface
- returns generic null
Success allows the message types defined in the file to be parsed and serialized by the interface.
Signalled errors contain errors and warnings which occurred in parsing the file.
The filename may not include a path.
Define directory search locations beforehand with .protobufkdb.addProtoImportPath
.
// successful function invocation
q).protobufkdb.importProtoFile["examples_dynamic.proto"]
// proto file does not exist
q).protobufkdb.importProtoFile["not_a_file.proto"]
'Error: not_a_file.proto:-1:0: File not found.
[0] .protobufkdb.importProtoFile["not_a_file.proto"]
^
listImportedMessageTypes
¶
List imported message types
.protobufkdb.listImportedMessageTypes[]
Returns successfully imported message types as a symbol list.
The list does not contain message types compiled into the interface.
q).protobufkdb.listImportedMessageTypes[]
`ScalarExampleDynamic`RepeatedExampleDynamic`SubMessageExampleDynamic`MapExam..
loadMessageDebug
¶
Parse from a Protobuf message file and display the debugging
.protobufkdb.loadMessageDebug[message_type;file_name]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionfile_name
is the name of a file (string or symbol)
the function
- prints debugging information to stdout
- returns generic null
For use only in debugging
The debugging is generated by the libprotobuf DebugString()
functionality and displayed on stdout to preserve formatting and indentation.
q)data
1 2
10 20
"s1" "s2"
q).protobufkdb.saveMessageFromList[`RepeatedExampleDynamic;`trivial_message_file;data]
q).protobufkdb.loadMessageDebug[`RepeatedExampleDynamic;`trivial_message_file]
rep_int32: 1
rep_int32: 2
rep_double: 10
rep_double: 20
rep_string: "s1"
rep_string: "s2"
loadMessageToDict
¶
Parse from a Protobuf message file to a kdb+ dictionary
.protobufkdb.loadMessageToDict[message_type;file_name]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionfile_name
is the name of a file (string or symbol)
returns a dictionary from field name symbols to field values parsed from file_name
.
q)data
rep_int32 | 1 2
rep_double| 10 20
rep_string| "s1" "s2"
q).protobufkdb.saveMessageFromDict[`RepeatedExampleDynamic;`trivial_message_file;data]
q).protobufkdb.loadMessageToDict[`RepeatedExampleDynamic;`trivial_message_file]
rep_int32 | 1 2
rep_double| 10 20
rep_string| "s1" "s2"
loadMessageToList
¶
Parse from a Protobuf message file to a kdb+ mixed list object
.protobufkdb.loadMessageToList[message_type;file_name]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionfile_name
is the name of a file (string or symbol)
returns a kdb+ mixed list object parsed from file_name
.
q)data
1 2
10 20
"s1" "s2"
q).protobufkdb.saveMessageFromList[`RepeatedExampleDynamic;`trivial_message_file;data]
q).protobufkdb.loadMessageToList[`RepeatedExampleDynamic;`trivial_message_file]
1 2
10 20
"s1" "s2"
parseArrayArenaToDict
¶
Parse from a Protobuf message string to a kdb+ dictionary, using a Google Arena for the intermediate message
.protobufkdb.parseArrayArenaToDict[message_type;char_array]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionchar_array
is the serialized Protobuf message (string)
the function
- parses the string as a Protobuf message, which it creates on a Google Arena
- returns the message as a kdb+ dictionary from field name symbols to field values
Identical to parseArrayToDict
except the intermediate Protobuf message is created on a Google Arena
Can improve memory allocation performance for large messages with deep repeated fields/map.
q)fields:.protobufkdb.getMessageFields[`RepeatedExampleDynamic]
q)data:fields!(1 2i;10 20f;("s1";"s2"))
q)array:.protobufkdb.serializeArrayFromDict[`RepeatedExampleDynamic;data]
q)array
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
q).protobufkdb.parseArrayArenaToDict[`RepeatedExampleDynamic;array]
rep_int32 | 1 2
rep_double| 10 20
rep_string| "s1" "s2"
parseArrayArenaToList
¶
Parse from a Protobuf message string to a kdb+ mixed list object, using a Google Arena for the intermediate message
.protobufkdb.parseArrayArenaToList[message_type;char_array]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionchar_array
is the serialized Protobuf message (string)
the function
- parses the string as a Protobuf message, which it creates on a Google Arena
- returns the message as a kdb+ mixed list object
Identical to parseArrayToList
except the intermediate Protobuf message is created on a Google Arena
Can improve memory allocation performance for large messages with deep repeated fields/map.
q)data:(1 2i;10 20f;("s1";"s2"))
q)array:.protobufkdb.serializeArrayFromList[`RepeatedExampleDynamic;data]
q)array
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
q).protobufkdb.parseArrayArenaToList[`RepeatedExampleDynamic;array]
1 2
10 20
"s1" "s2"
parseArrayDebug
¶
Parse from a Protobuf message string and display the debugging
.protobufkdb.parseArrayDebug[message_type;char_array]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionchar_array
is the serialized Protobuf message (string)
the function
- prints debugging information to stdout
- returns generic null
For use only in debugging
The debugging is generated by the libprotobuf DebugString()
functionality and displayed on stdout to preserve formatting and indentation.
q)data:(1 2i;10 20f;("s1";"s2"))
q)array:.protobufkdb.serializeArrayFromList[`RepeatedExampleDynamic;data]
q).protobufkdb.parseArrayDebug[`RepeatedExampleDynamic;array]
rep_int32: 1
rep_int32: 2
rep_double: 10
rep_double: 20
rep_string: "s1"
rep_string: "s2"
parseArrayToDict
¶
Parse from a Protobuf message string to a kdb+ dictionary
.protobufkdb.parseArrayToDict[message_type;char_array]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionchar_array
is the serialized Protobuf message (string)
returns the message as a kdb+ dictionary from field name symbols to field values.
q)fields:.protobufkdb.getMessageFields[`RepeatedExampleDynamic]
q)data:fields!(1 2i;10 20f;("s1";"s2"))
q)array:.protobufkdb.serializeArrayFromDict[`RepeatedExampleDynamic;data]
q)array
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
q).protobufkdb.parseArrayToDict[`RepeatedExampleDynamic;array]
rep_int32 | 1 2
rep_double| 10 20
rep_string| "s1" "s2"
parseArrayToList
¶
Parse from a Protobuf message string to a kdb+ mixed list object
.protobufkdb.parseArrayToList[message_type;char_array]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionchar_array
is the serialized Protobuf message (string)
returns the message as a kdb+ mixed list object.
q)data:(1 2i;10 20f;("s1";"s2"))
q)array:.protobufkdb.serializeArrayFromList[`RepeatedExampleDynamic;data]
q)array
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
q).protobufkdb.parseArrayToList[`RepeatedExampleDynamic;array]
1 2
10 20
"s1" "s2"
saveMessageFromDict
¶
Serialize from a kdb+ dictionary to a Protobuf message file
.protobufkdb.saveMessageFromDict[message_type;file_name;msg_in]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionfile_name
is the name of a file (string or symbol)msg_in
is a kdb+ dictionary from field name symbols to field values
the function
- converts
msg_in
to a Protobuf message ofmessage_type
, serializes it, and writes it tofile_name
- returns generic null
q)data
rep_int32 | 1 2
rep_double| 10 20
rep_string| "s1" "s2"
q).protobufkdb.saveMessageFromDict[`RepeatedExampleDynamic;`trivial_message_file;data]
saveMessageFromList
¶
Serialize from a kdb+ mixed list object to a Protobuf message file
.protobufkdb.saveMessageFromList[message_type;file_name;msg_in]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionfile_name
is the name of a file (string or symbol)msg_in
is a kdb+ mixed list object
the function
- converts
msg_in
to a Protobuf message ofmessage_type
, serializes it, and writes it tofile_name
- returns generic null
q)data
1 2
10 20
"s1" "s2"
q).protobufkdb.saveMessageFromList[`RepeatedExampleDynamic;`trivial_message_file;data]
serializeArrayArenaFromDict
¶
Serialize from a kdb+ dictionary to a Protobuf message string, using a Google Arena for the intermediate message
.protobufkdb.serializeArrayArenaFromDict[message_type;msg_in]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionmsg_in
is a kdb+ dictionary from field name symbols to field values
the function
- serializes
msg_in
as a Protobuf message and creates it on a Google Arena - returns the serialized Protobuf message as a string
Identical to serializeArrayFromDict
except the intermediate Protobuf message is created on a Google Arena
Can improve memory allocation performance for large messages with deep repeated fields/map.
q)fields:.protobufkdb.getMessageFields[`RepeatedExampleDynamic]
q)data:fields!(1 2i;10 20f;("s1";"s2"))
q).protobufkdb.serializeArrayArenaFromDict[`RepeatedExampleDynamic;data]
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
serializeArrayArenaFromList
¶
Serialize from a kdb+ mixed list object to a Protobuf message string, using a Google Arena for the intermediate message
.protobufkdb.serializeArrayArenaFromList[message_type;msg_in]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionmsg_in
is a kdb+ mixed list object
the function
- serializes
msg_in
as a Protobuf message and creates it on a Google Arena - returns the serialized Protobuf message as a string
Identical to serializeArrayFromList
except the Protobuf message is created on a Google Arena
Can improve memory allocation performance for large messages with deep repeated fields/map.
q)data:(1 2i;10 20f;("s1";"s2"))
q).protobufkdb.serializeArrayArenaFromList[`RepeatedExampleDynamic;data]
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
serializeArrayFromDict
¶
Serialize from a kdb+ dictionary to a Protobuf message string
.protobufkdb.serializeArrayFromDict[message_type; msg_in]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionmsg_in
is a kdb+ dictionary from field name symbols to field values
returns the serialized Protobuf message as a string.
q)fields:.protobufkdb.getMessageFields[`RepeatedExampleDynamic]
q)data:fields!(1 2i;10 20f;("s1";"s2"))
q).protobufkdb.serializeArrayFromDict[`RepeatedExampleDynamic;data]
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
serializeArrayFromList
¶
Serialize from a kdb+ mixed list object to a Protobuf message string
.protobufkdb.serializeArrayFromList[message_type; msg_in]
Where
message_type
is a message type (string or symbol) matching a message name in the.proto
definitionmsg_in
is a kdb+ mixed list object
returns the serialized Protobuf message as a string.
q)data:(1 2i;10 20f;("s1";"s2"))
q).protobufkdb.serializeArrayFromList[`RepeatedExampleDynamic;data]
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
version
¶
Library version (integer) used by the interface
.protobufkdb.version[]
returns the version of the libprotobuf
shared object used by the interface, as an integer.
q).protobufkdb.version[]
3012003i
versionStr
¶
Library version (string) used by the interface
.protobufkdb.versionStr[]
returns the version of the libprotobuf
shared object used by the interface, as a string.
q).protobufkdb.versionStr[]
"libprotobuf v3.12.3"