Protobuf/Protocol Buffers function reference

KxSystems/protobufkdb

Functions exposed in the .protobufkdb namespace allow you to generate and parse Protobuf messages.

.protobufkdb   Protobuf/Protocol Buffers interface
Library Information
  version                   Return the libprotobuf version as an integer
  versionStr                Return the libprotobuf version as a string
Import Schema
  addProtoImportPath        Add a path from which to import proto/schema files
  importProtoFile           Import a file
  listImportedMessageTypes  List successfully imported message schemas
Inspect Schema
  displayMessageSchema      Display the schema definition of the message
Serialize / Deserialize
  parseArray                Deserialize encoded Protobuf string
  parseArrayArena           Deserialize encoded Protobuf string, creating an
                            intermediate Protobuf message on Google Arena
  serializeArray            Serialize kdb+ object
  serializeArrayArena       Serialize kdb+ object, creating an
                            intermediate Protobuf message on Google Arena
Save / Load
  loadMessage               Load a file and deserialize it to a kdb+ object
  saveMessage               Serialize kdb+ object with a message, save it to file

Message types

Where a function takes a message_type parameter to specify the name of the message to be be processed, the interface first looks for that message type in the compiled messages.

If that fails it then searches the imported message definitions. Only if the message type is not found in either is an error returned.

addProtoImportPath

Add an import path

.protobufkdb.addProtoImportPath[import_path]

Where import_path is a path (absolute or relative) as a string

1. adds it as a path to be searched when dynamically importing .proto file definitions
2. returns generic null

Can be called more than once to specify multiple import locations.

// successful execution
q).protobufkdb.addProtoImportPath["../proto"]

// incorrect data format provided as input
q).protobufkdb.addProtoImportPath[enlist 1f]
'"Specify import path"

displayMessageSchema

Display the Proto schema of a specified message

.protobufkdb.displayMessageSchema[message_type]

Where message_type is the name of a message type as a string,

1. prints schema format to stdout
2. returns generic null

message_type must match a message name in its .proto definition.

For use only in debugging

The schema is generated by the libprotobuf DebugString() functionality and displayed on stdout to preserve formatting and indentation.

q).protobufkdb.displayMessageSchema[`ScalarExampleDynamic]
message ScalarExampleDynamic {
  int32 scalar_int32 = 1;
  double scalar_double = 2;
  string scalar_string = 3;
}

importProtoFile

Import a .proto file definition

.protobufkdb.importProtoFile[filename]

Where filename is the name of a .proto file as a string

1. dynamically imports the file definition into the interface
2. returns generic null

Success allows the message types defined in the file to be parsed and serialized by the interface.

Signalled errors contain errors and warnings which occurred in parsing the file.

The filename may not include a path.

Define directory search locations beforehand with .protobufkdb.addProtoImportPath.

// successful function invocation
q).protobufkdb.importProtoFile["examples_dynamic.proto"]

// proto file does not exist
q).protobufkdb.importProtoFile["not_a_file.proto"]
'Import error: File not found.
File: 'not_a_file.proto', line: -1, column: 0

  [0]  .protobufkdb.importProtoFile["not_a_file.proto"]
       ^

listImportedMessageTypes

List imported message types

.protobufkdb.listImportedMessageTypes[]

Returns successfully imported message types as a symbol list.

The list does not contain message types compiled into the interface.

q).protobufkdb.listImportedMessageTypes[]
`ScalarExampleDynamic`RepeatedExampleDynamic`SubMessageExampleDynamic`MapExam..

loadMessage

.protobufkdb.loadMessage[message_type;file_name]

Where:

• message_type is a message type (string or symbol) matching a message name in the .proto definition
• file_name is the name of a file (string or symbol)

returns a kdb+ object parsed from file_name.

q)data
1  2
10 20
s1 s2
q).protobufkdb.saveMessage[`RepeatedExampleDynamic;`trivial_message_file;data]
q).protobufkdb.loadMessage[`RepeatedExampleDynamic;`trivial_message_file]
1  2
10 20
s1 s2

parseArray

Parse a proto-serialized string to a kdb+ object

.protobufkdb.parseArray[message_type;char_array]

Where

• message_type is a message type (string or symbol) matching a message name in the .proto definition
• char_array is the serialized Protobuf message (string)

returns the message as a kdb+ object.

q)data:(1 2i;10 20f;`s1`s2)
q)array:.protobufkdb.serializeArray[`RepeatedExampleDynamic;data]
q)array
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
q).protobufkdb.parseArray[`RepeatedExampleDynamic;array]
1  2
10 20
s1 s2

parseArrayArena

Parse a proto-serialized string to a kdb+ object, saving intermediate message

.protobufkdb.parseArrayArena[message_type;char_array]

Where:

• message_type is a message type (string or symbol) matching a message name in the .proto definition
• char_array is the serialized Protobuf message (string)

the function

1. parses the string as a Protbuf message, which it creates on a Google Arena
2. returns the message as a kdb+ object

Identical to parseArray except the intermediate Protobuf message is created on a Google Arena

Can improve memory allocation performance for large messages with deep repeated fields/map.

Google Arenas

q)data:(1 2i;10 20f;`s1`s2)
q)array:.protobufkdb.serializeArray[`RepeatedExampleDynamic;data]
q)array
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..
q).protobufkdb.parseArrayArena[`RepeatedExampleDynamic;array]
1  2
10 20
s1 s2

saveMessage

Write a kdb+ object to file as a Protobuf message

.protobufkdb.saveMessage[message_type;file_name;msg_in]

Where

• message_type is a message type (string or symbol) matching a message name in the .proto definition
• file_name is the name of a file (string or symbol)
• msg_in is a kdb+ object

the function

1. converts msg_in to a Protobuf message of message_type, serializes it, and writes it to file_name
2. returns generic null

q)data
1  2
10 20
s1 s2
q).protobufkdb.saveMessage[`RepeatedExampleDynamic;`trivial_message_file;data]

serializeArray

Serialize a kdb+ object as a Protobuf message

.protobufkdb.serializeArray[message_type; msg_in]

Where:

• message_type is a message type (string or symbol) matching a message name in the .proto definition
• msg_in is a kdb+ object

returns the serialized Protobuf message as a string.

q)data:(1 2i;10 20f;`s1`s2)
q).protobufkdb.serializeArray[`RepeatedExampleDynamic;data]
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..

serializeArrayArena

Serialize a kdb+ object as a Protobuf message, saving it to a Google Arena

.protobufkdb.serializeArrayArena[message_type;msg_in]

Where:

• message_type is a message type (string or symbol) matching a message name in the .proto definition
• msg_in is a kdb+ object

the function

1. serializes msg_in as a Protobuf message and creates it on a Google Arena
2. returns the serialized Protobuf message as a string

Identical to serializeArray except the Protobuf message is created on a Google Arena

Can improve memory allocation performance for large messages with deep repeated fields/map.

Google Arenas

q)data:(1 2i;10 20f;`s1`s2)
q).protobufkdb.serializeArrayArena[`RepeatedExampleDynamic;data]
"\n\002\001\002\022\020\000\000\000\000\000\000$@\000\000\000\000\000\0004@\0..

version

Library version (integer) used by the interface

.protobufkdb.version[]

returns the version of the libprotobuf shared object used by the interface, as an integer.

q).protobufkdb.version[]
3012003i

versionStr

Library version (string) used by the interface

.protobufkdb.versionStr[]

returns the version of the libprotobuf shared object used by the interface, as a string.

q).protobufkdb.versionStr[]
"libprotobuf v3.12.3"