Realtime feed
One of the ways to populate data in the warehouse is through a feed. In a normal system there will be a real feed, usually from an external source; a sensor, market adaptor etc. This data is fed through the TP to the other subscribers. For the purposes of this guide, a dummy feed will be created in place of a real feed of monitoring data.
Create feed process
The feed will be created as a service using the DS_RTE
template.
- Right-click on the workspace tree and select New > Analytic from the context menu.
- Give it a name
.mon.initFeed
and add to theKxWarehouse
package and hit Save. Leave the analytic parameters and code as the defaults for now, these will be configured later. - Now create a new service class by right-clicking and selecting New > Service Class
- Name the process
mon_feed
, set the template toDS_RTE
, and click Add.
- Add a description.
- Configure the Service Parameters.
parameter | value |
---|---|
messagingServer | DS_MESSAGING_SERVER:DS |
publishChannel | |
publishTableList | monCPU, monMem, monDisk |
subscriptionChannel | NO_TABLES |
initialStateFunct | .mon.initFeed |
- In the Schemas subtab, associate the
monCore
group. - Save the service class.
Writing feed code
In order to create the feed, code needs to be developed to generate and publish the feed data. To do this;
- Run the
mon_feed
service class - Right-click the running service in the Process Library Status Viewer
- Hit Connect to open an Analyst IDE
- In Analyst, a scratchpad should be opened by default, which will allow you to type and execute code.
- Copy and paste the following code. This uses a random walk logic to generate mock monitoring data.
HOST:`$"server_",/:("A","B","C","D");
BASE:HOST!20 35 50 65;
MV:HOST!4?rand 0.5f;
n:5;
CBase:MBase:DBase:();
.mon.createUsage:{[b]
base:BASE,b;
usage:HOST!{[host] sums?[n?1.<MV host;neg MV host; MV host]} each HOST;
(base HOST) + usage
}
t: .z.p + 00:00:02 * til n;
ts:raze 4#enlist t;
hosts:raze flip n#enlist HOST;
CPU:.mon.createUsage[CBase];
CBase::last each CPU;
CUsage:raze {max(0;min(x;100))} each CPU;
MemV:.mon.createUsage[MBase];
MBase::last each MemV;
MVUsage:raze {max(0;min(x;100))} each MemV;
MPUsage:raze {min(x;100)} each (first 1?10f) + MVUsage;
Disk:.mon.createUsage[DBase];
DBase::last each Disk;
DUsage:raze {max(0;min(x;100))} each Disk;
CPUData:([] time:ts; sym:hosts; usage:CUsage);
memData:([] time:ts; sym:hosts; virtual:MVUsage; physical:MPUsage);
diskData:([] time:ts; sym:hosts; usage:DUsage);
- Highlight the lines you want to execute, right-click, and select Display from the context menu. The keyboard shortcut
ctrl + d
can be used also.
- Execute the lines to create tables individually and see data generated. The result of the operation will appear in the Console at the bottom of the screen.
- Once the code to generate the tables is working correctly, use the
.dm.pub
function to publish it to subscribers.
- To test if this dummy data was published across processes, connect to the
kxw_rdb
and entermonCPU
in the Scratchpad. Display the results and check to see if there is data populated. Repeat this for the other tables;monMem
,monDisk
Deploying feed code
-
Previously we've manually generated and published data from the
mon_feed
process. To automate this, create an instruction and write a function that generates and publishes data on a timer. Then load the instruction at startup. -
Right-click Analytics Libary and select New Instruction.
-
Name the instruction
.mon.feedLogic
. -
In the Instruction Details subtab, give it a description. For example, Logic to generate and publish random data
-
Copy and paste the code from the appendix to the Content subtab of your
.mon.feedLogic
instruction and save.
- Next, in the
.mon.initFeed
analytic, copy and paste the code below to the Content subtab.
// load instruction
.al.loadinstruction[`.mon.feedLogic]
- The
.al.loadinstruction
loads an instruction into a process, like loading a q script.
Check feed
After creating the feed code, restart the service to check it's working correctly.
- To shutdown, right-click
mon_feed
in the Process Library Status Viewer - Select Stop > Shut Down Q Process.
-
Run the
mon_feed
service class again and Connect again. -
Look in the tree on the left, and open the (.mon) module and Functions folder. Find the genData analytic.
-
The data creating analytics in .mon.feedLogic is called in the initial state (.mon.initFeed), so data should already be generating in three tables.
-
To check that the feed is working correctly end to end, connect to the
kxw_rdb
service and validate that there's data populating in the tables. Once this is confirmed, the feed part of the system is working correctly.
- An alternative way of looking at the tables is to inspect them as shown below. The Visual Inspector allows users to inspect data values in a table view as well as explore their data graphically.