Skip to content

Data

Build an Assembly in the UI from available schemas, databases, pipelines and streams.

Screenshot

Set up

  1. Name the Assembly to create

  2. Define a schema, and add to the Assembly.

  3. Build streams, and add to the Assembly.

  4. Create pipelines, and add to the Assembly.

  5. Configure the database, and add to the Assembly.

  6. Assign label name(s) and associated value(s).

  7. Submit

Assembly Name

Assembly name must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character

Sample Expression Assembly

An Assembly to build a table generated from a kdb expression. The table will have two columns, count and time.

Schema

  • Open a schema document.

  • Set the the table name to exptable and the schema name to expschema.

  • Add the columns: cnt (int) & time (timestamp).

  • Expand the extra options on cnt and ensure On-Disk Attribute (Temporal Partitioning) and On-Disk Attribute (Ordinal Partitioning) are set to Parted (In-Memory Attribute can be left as Parted).

    Screenshot

  • Expand time and ensure the three attributes are set to none.

    Screenshot

  • For Table Properties remove the primary key and set the following properties

    type: partitioned
    block size: 10000
    partiton column: time
    timestamp column: (none)
    On-Disk Column Sort: (blank)
    In-Memory Column sort: (blank)
    On-Disk Column Sort: (blank)

    Screenshot

  • Remove the other properties and click submit

Streams

Set up two streams documents, North Stream and South Stream. Configure each as outlined below, before sumbitting:

North Stream

stream name : expnorth
external facing: false
sub topic: expassembly
topicConfig:
topic Prefix: rt-

Screenshot

South Stream

stream name : expsouth
external facing: false
sub topic:
topicConfig:
topic Prefix: rt-

Pipelines

Next, build two pipelines. One to ingest the data generated by the kdb expression to write to the North Stream. The second to read from the North Stream to write to the South Stream. The table will be read from the South Stream, and is the source for the Assembly. The two pipelines are:

Expression pipeline

First pipeline to build:

  • Click-and-drag into the pipeline workspace an Expression Reader and add the following to the editor

    ([] time:200?(reverse .z.p-1+til 10); cnt:200?10)
  • Click-and-drag into the pipeline workspace KX Insights Stream Writer and set the fields to:

    assembly: expassembly
    stream: expnorth
    table: exptable
    deduplicate stream: true
    stream prefix: rt

    Screenshot

  • Connect the nodes, Save As the pipeline as exppipein, then complete Save using the same pipeline name.

    Screenshot

North to South stream pipeline

Second pipeline to build:

  • Drag out a KX Insights Stream Reader and set the fields to:

    assembly: expassembly
    stream: expnorth
    table: 
    stream prefix: rt
  • Drag out a KX Insights Stream Writer and set the fields to:

    assembly: expassembly
    stream: expsouth
    table: 
    deduplicate stream: true
    stream prefix: rt
  • Connect these nodes, Save As the pipeline name as exppipeout, then complete Save using the same pipeline name.

Database

Create a Custom database to set up the rdb, idb and hdb.

  • Create a database document and select Custom.

  • Name the database expdatabase.

  • Select Advanced mode and update the following sections, tab between each section for details:

No change required.

    mount name: rdb
    source: expsouth

Screenshot

    source: expsouth

Screenshot

  • Other database parameters, e.g. Mount, use pre-configured values.

  • Submit the expdatabase database.

Configure Assembly

With all components created, next configure the assembly.

Name: expassembly
Schema Configuration: expschema
Database Configuration: expdatabase
Pipelines: Select the two created pipelines (exppipein and exppipeout)
Streams: Select the two created streams (expnorth and expsouth)
  • Set a label to label: type, value: basic

    Screenshot

  • Save

  • Deploy.

The assembly should change from RUNNING state to READY state.

Query Data

  • Once the assembly has entered the ready state (a green solid circle) open an Explore document.

  • Ensure kdb is the selected Editor language

    Screenshot

  • From the Assembly dropdown menu select your expassembly and choose hdb

  • Click Run