Skip to content

Data

Build an Assembly in the UI from available schemas, databases, pipelines and streams.

Screenshot

Set up

  1. Name the Assembly to create

  2. Define a schema, and add to the Assembly.

  3. Build streams, and add to the Assembly.

  4. Create pipelines, and add to the Assembly.

  5. Configure the database, and add to the Assembly.

  6. Assign label name(s) and associated value(s).

  7. Submit

Assembly Name

Assembly name must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character

Sample Expression Assembly

An Assembly to build a table generated from a kdb expression. The table will have four columns; one for time, two for categories and one for data.

Schema

  • Open a schema document.

  • Set the the table name to exptable and the schema name to expschema.

  • Add columns

name type
time Timestamp
cnt Integer
  • For Table Properties remove the primary key and set the following properties
Primary Keys: (blank)
Description: (blank)
Type: partitioned
Partiton column: date
Block size: (blank)
Partitions: (blank)
Shards: (blank)
Timestamp column: date
Real-time Sort: date
Interval Sort: date
Historical Sort: date

Advanced properties

Type, Block size, Partitions and Shards are Advanced properties

Screenshot

  • Click Submit

Streams

Set up two streams documents: North Stream and South Stream. Configure each before submitting:

North Stream

stream name : expdatabase-ingress
sub topic: data
external facing: true

Screenshot

South Stream

stream name : expdatabase
sub topic:
external facing: false

Pipelines

Next, build two pipelines. One to ingest the data generated by the kdb expression to write to the North Stream. The second to read from the North Stream to write to the South Stream. The table will be read from the South Stream and is the source for the assembly.

Expression pipeline

First pipeline to build:

  • Click-and-drag into the workspace an Expression Reader. Add the following to the editor, then Apply
([] time:200?(reverse .z.p-1+til 10); cnt:200?10)
  • Click-and-drag into the pipeline workspace KX Insights Stream Writer and set the fields to:
table: exptable
deduplicate stream: true

Screenshot

  • Apply the properties.

  • Connect the nodes, Save the pipeline as exppipein. Note, Kx Inisghts Stream nodes have a requirement to assign the Assembly Integration with the name of the publish topic.

Pipeline name: exppipein
Assembly Publish Topic: expdatabase-ingress

Screenshot

North to South stream pipeline

Second pipeline to build.

  • Drag out a KX Insights Stream Reader and leave the Table property blank:

table:
- Apply the properties.

  • Drag out a KX Insights Stream Writer and set the fields to:
table: 
deduplicate stream: true

  • Apply the properties.

  • Connect these nodes, Save the pipeline name as exppipeout and assign the Assembly Integration with the name of the publish topic.

Pipeline name: exppipeout
Assembly Subscription Topic: expdatabase-ingress
Assembly Publish Topic: expdatabase

Pipeline Name

Pipeline names should be lowercase

Database

Create a Time Series database to set up the rdb, idb and hdb.

  • Create a database document and select Time Series.

  • Name the database expdatabase and Submit - this will make available additional configuration options.

  • InAdvanced mode, update the following sections - tab between each section for details:

No change required.

  • mount name: rdb
  • source: expdatabase

Screenshot

  • source: expdatabase

Screenshot

  • Other database parameters, e.g. Mount, use pre-configured values.

  • Submit the expdatabase database.

Configure Assembly

With all components created, next configure the assembly.

Name: expassembly
Schema Configuration: expschema
Database Configuration: expdatabase
Pipelines: Select the two created pipelines (exppipein and exppipeout)
Streams: Select the two created streams (expdatabase-ingress and expdatabase)

  • Set a label to label: name, value: expdatabase

    Screenshot

  • Save

  • Deploy

The assembly should change from RUNNING state (hollow circle) to READY state (green circle).

Query Data

  • Once the assembly has entered the ready state (a green solid circle) open an Explore document.

  • From the Assembly dropdown menu, select your expassembly, and choose hdb if querying against Q.

  • Query your data with:

SELECT * FROM exptable

exptable

  • Define the Output variable; this can be exptable

  • Click Get Data