Skip to content

Decoders

.qsp.decode csv decode CSV data into tables json decode JSON data protobuf decode Protocol Buffer messages

.qsp.decode.csv

Parse CSV data to a table

.qsp.decode.csv[schema]
.qsp.decode.csv[schema; delimiter]
.qsp.decode.csv[schema; delimiter; .qsp.use (!) . flip (
    (`header   ; header);
    (`exclude  ; exclude))]

Parameters:

name type description default
schema dict or string A table with the desired output schema; or a list of types to support as type characters. Required
delimiter character Field separator for the records in the encoded data. ","

options:

name type description default
header symbol Whether encoded data starts with a header row, either none, first or always. first
exclude symbol[] or int[] A list of columns to exclude from the output. ()

For all common arguments, refer to configuring operators

This operator decodes CSV data from strings or bytes with delimiter-separated values into tables or matrices of data in the given schema.

Decode from a CSV file:

// Generate a random table of data and store it in an inventory file.
n: 10000
t: ([]
  date: n?.z.d + neg til 10;
  price: (n?1000)div 100;
  item: n?`3;
  description: {rand[100]?.Q.an}each til n;
  quantity: n?10000)

`:/tmp/inventory.csv 0: csv 0: t

// Read and parse the data from a file
schema: ([] date: `date$(); price:`int$();
  item:`symbol$(); description: (); quantity:`long$());

.qsp.run
  .qsp.read.fromFile["/tmp/inventory.csv"]
  .qsp.decode.csv[schema]
  .qsp.write.toConsole[]
                             | date       price item description                      quantity
-----------------------------| ---------------------------------------------------------------
2021.07.16D19:45:03.929480200| 2021.07.16 7     ehm  "3qupWqmNh6y8TeTzJlW49NlRzv0_0"  2659
2021.07.16D19:45:03.929480200| 2021.07.14 2     iif  "_eB_lq"                         8257
2021.07.16D19:45:03.929480200| 2021.07.12 7     eod  "GhUgGe3PH9Ie2NOw"               3907
2021.07.16D19:45:03.929480200| 2021.07.11 0     goj  "Dvmemf3H2P"                     6100
2021.07.16D19:45:03.929480200| 2021.07.09 1     bpm  "GbSjldDmUprmfiBa0UI8I"          367
..

Decode from a CSV file using a dictionary schema:

// Generate a random table of data and store it in an inventory file.
n: 10000
t: ([]
  date: n?.z.d + neg til 10;
  price: (n?1000)div 100;
  item: n?`3;
  description: {rand[100]?.Q.an}each til n;
  quantity: n?10000)

`:/tmp/inventory.csv 0: csv 0: t

// Read and parse the data from a file
schema: `date`price`item`description`quantity!"dis*j";

.qsp.run
  .qsp.read.fromFile["/tmp/inventory.csv"]
  .qsp.decode.csv[schema]
  .qsp.write.toConsole[]
                             | date       price item description                      quantity
-----------------------------| ---------------------------------------------------------------
2021.07.16D19:45:03.929480200| 2021.07.16 7     ehm  "3qupWqmNh6y8TeTzJlW49NlRzv0_0"  2659
2021.07.16D19:45:03.929480200| 2021.07.14 2     iif  "_eB_lq"                         8257
2021.07.16D19:45:03.929480200| 2021.07.12 7     eod  "GhUgGe3PH9Ie2NOw"               3907
2021.07.16D19:45:03.929480200| 2021.07.11 0     goj  "Dvmemf3H2P"                     6100
2021.07.16D19:45:03.929480200| 2021.07.09 1     bpm  "GbSjldDmUprmfiBa0UI8I"          367
..

.qsp.decode.json

Parse JSON data

.qsp.decode.json[]
.qsp.decode.json[.qsp.use enlist[`decodeEach]!enlist decodeEach]

options:

name type description default
decodeEach boolean By default messages passed to the decoder are treated as a single JSON object. Setting decodeEach to true indicates that parsing should be done on each value of a message. 0b

For all common arguments, refer to configuring operators

This operator parses JSON strings to native q types, usually either a dictionary or a table.

Decode JSON from a file:

// Generate a random table of data and write it as JSON data
n: 10000;
t: ([]
  date: n?.z.d + neg til 10;
  price: (n?1000)div 100;
  item: n?`3;
  description: {rand[100]?.Q.an}each til n;
  quantity: n?10000);

`:/tmp/inventory.json 0: enlist .j.j t;

.qsp.run
  .qsp.read.fromFile["/tmp/inventory.json"]
  .qsp.decode.json[]
  .qsp.write.toConsole[];
                             | date         price item  description                                        quantity
-----------------------------| ------------------------------------------------------------------------------------
2021.10.05D19:40:04.536274000| "2021-10-01" 8     "eke" "PlND7JnZejE5j8aKJxSmqLTJycOsxkgTgqz2dB6mH3Q"      5963
2021.10.05D19:40:04.536274000| "2021-10-05" 0     "ldc" "ngctTMTD5PkkTSTOZ_3pwgy2vISuvnJYy"                3057
2021.10.05D19:40:04.536274000| "2021-10-05" 7     "ikb" "nFBU7"                                            8986
2021.10.05D19:40:04.536274000| "2021-09-28" 9     "lhp" "JH9NSxL7UNBGRZ49MYDX9qu_BUYmZoGu11G_GSV"          9488
2021.10.05D19:40:04.536274000| "2021-10-05" 3     "eoi" "E0hp_zZUBAfKERSPvdz_UZnKX07iBe2sd9TgH4mJmFtsLyap" 1301
..

Decode a stream of JSON:

.qsp.run
  .qsp.read.fromCallback[`publish]
  .qsp.decode.json[.qsp.use``decodeEach!11b]
  .qsp.write.toConsole[];

publish .j.j each ([] date: .z.d + til 10; price: 10?100f)
                             | date         price
-----------------------------| ---------------------
2021.10.05D19:47:01.576948900| "2021-10-05" 22.56381
2021.10.05D19:47:01.576948900| "2021-10-06" 51.2789
2021.10.05D19:47:01.576948900| "2021-10-07" 34.48978
2021.10.05D19:47:01.576948900| "2021-10-08" 69.06853
2021.10.05D19:47:01.576948900| "2021-10-09" 71.53166
..

.qsp.decode.protobuf

Parse Protocol Buffer messages to a dictionary or list

.qsp.decode.protobuf[message]
.qsp.decode.protobuf[message; .qsp.use (!) . flip (
    (`file         ; file);
    (`format       ; format);
    (`includeFields; includeFields))]

Parameters:

name type description default
message string or symbol The name of the Protocol Buffer message type to decode. Required

options:

name type description default
file symbol The path to a .proto file containing the message type definition. Either format or file must be provided. `
format string A string definition of the Protocol Buffer message format to decode. ""
includeFields boolean Indicates if the output should include field names or just values. 1b

For all common arguments, refer to configuring operators

This operator decodes protobuf-encoded messages of the chosen message type, given a protobuf schema containing that message type. The protobuf schema can be provided either as a file using the file parameter, or as a string using the format option. Decoded messages are outputted as either a dictionary or list depending on the value of the includeFields option.

Import paths

To import your .proto file, the folder containing the .proto file is added as an import path. This means the folder will be scanned when importing future .proto files, so it is important that you avoid having .proto files with the same filename present in import paths you use.

Decode protobuf messages using a Person.proto file: 

// Person.proto
syntax="proto3";
message Person {
  string name = 1;
  int32 id = 2;
  string email = 3;
}

.qsp.run
    .qsp.read.fromCallback[`publish]
    .qsp.decode.protobuf[`Person;"Person.proto"]
    .qsp.write.toConsole[];

// The bytes listed below are an example encoded Protocol Buffer payload
publish 0x0a046e616d6510651a0f656d61696c40656d61696c2e636f6d;
2021.11.09D19:05:29.933331149 | name | "name"
2021.11.09D19:05:29.933331149 | id   | 101i
2021.11.09D19:05:29.933331149 | email| "email@email.com"

Decode protobuf messages using format into lists: 

format: "syntax=\"proto3\";
 message Person {
    string name = 1;
    int32 id = 2;
    string email = 3;
 }";

.qsp.run
    .qsp.read.fromCallback[`publish]
    .qsp.decode.protobuf[.qsp.use `message`format`includeFields!(`Person;format;0b)]
    .qsp.write.toConsole[];

// The bytes listed below are an example encoded Protocol Buffer payload
publish 0x0a046e616d6510651a0f656d61696c40656d61696c2e636f6d;
2021.11.09D19:11:40.422536637 | "name"
2021.11.09D19:11:40.422536637 | 101i
2021.11.09D19:11:40.422536637 | "email@email.com"