Data
Build an Assembly in the UI from available schemas, databases, pipelines and streams.
Set up
-
Name the Assembly to create
-
Define a schema, and add to the Assembly.
-
Build streams, and add to the Assembly.
-
Create pipelines, and add to the Assembly.
-
Configure the database, and add to the Assembly.
-
Assign label name(s) and associated value(s).
-
Submit
Assembly Name
Assembly name must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character
Sample Expression Assembly
An Assembly to build a table generated from a kdb expression. The table will have two columns, count and time.
Schema
-
Open a schema document.
-
Set the the table name to
exptable
and the schema name toexpschema
. -
Add the columns: cnt (int) & time (timestamp).
-
Expand the extra options on cnt and ensure
On-Disk Attribute (Temporal Partitioning)
andOn-Disk Attribute (Ordinal Partitioning)
are set toParted
(In-Memory Attribute
can be left asParted
). -
Expand time and ensure the three attributes are set to
none
. -
For
Table Properties
remove the primary key and set the following propertiestype: partitioned block size: 10000 partiton column: time timestamp column: (none) On-Disk Column Sort: (blank) In-Memory Column sort: (blank) On-Disk Column Sort: (blank)
-
Remove the other properties and click
submit
Streams
Set up two streams documents, North Stream and South Stream. Configure each as outlined below, before sumbitting:
North Stream
stream name : expnorth
external facing: false
sub topic: expassembly
topicConfig:
topic Prefix: rt-
South Stream
stream name : expsouth
external facing: false
sub topic:
topicConfig:
topic Prefix: rt-
Pipelines
Next, build two pipelines. One to ingest the data generated by the kdb expression to write to the North Stream. The second to read from the North Stream to write to the South Stream. The table will be read from the South Stream, and is the source for the Assembly. The two pipelines are:
Expression pipeline
First pipeline to build:
-
Click-and-drag into the pipeline workspace an Expression Reader and add the following to the editor
([] time:200?(reverse .z.p-1+til 10); cnt:200?10)
-
Click-and-drag into the pipeline workspace KX Insights Stream Writer and set the fields to:
assembly: expassembly stream: expnorth table: exptable deduplicate stream: true stream prefix: rt
-
Connect the nodes, Save As the pipeline as
exppipein
, then complete Save using the same pipeline name.
North to South stream pipeline
Second pipeline to build:
-
Drag out a KX Insights Stream Reader and set the fields to:
assembly: expassembly stream: expnorth table: stream prefix: rt
-
Drag out a KX Insights Stream Writer and set the fields to:
assembly: expassembly stream: expsouth table: deduplicate stream: true stream prefix: rt
-
Connect these nodes, Save As the pipeline name as
exppipeout
, then complete Save using the same pipeline name.
Database
Create a Custom database to set up the rdb, idb and hdb.
-
Create a database document and select
Custom
. -
Name the database
expdatabase
. -
Select
Advanced
mode and update the following sections, tab between each section for details:
No change required.
mount name: rdb
source: expsouth
source: expsouth
-
Other database parameters, e.g. Mount, use pre-configured values.
-
Submit the
expdatabase
database.
Configure Assembly
With all components created, next configure the assembly.
Name: expassembly
Schema Configuration: expschema
Database Configuration: expdatabase
Pipelines: Select the two created pipelines (exppipein and exppipeout)
Streams: Select the two created streams (expnorth and expsouth)
-
Set a label to
label: type, value: basic
-
Save
- Deploy.
The assembly should change from RUNNING state to READY state.
Query Data
-
Once the assembly has entered the ready state (a green solid circle) open an Explore document.
-
Ensure
kdb
is the selected Editor language -
From the Assembly dropdown menu select your
expassembly
and choosehdb
-
Click Run