Enabling Refinery capture

You can capture Monitoring data within a Refinery installation to allow historical querying and analysis of the data.

The pipeline for the Monitoring capture is Monitoring. Ensure that enable-monitoring is to true in any pipeline that is required to be monitored including the DefaultEntrypoint Gateway. See pipeline for more details on supported parameters.

The data is published into Refinery via the Monitoring feed handler (Monitoring.0.monFh.0). By default, the host/port connection information for this process is automatically queried from Delta Control.

Available tables

Once enabled, all the Monitoring data (from MonitoringState) is transformed by the feed handler and captured into a set of tables for query and persistence.

Table name	Description	Source metric
SystemCpu	usage of server CPUs, individually and total	system.cpu
SystemMem	server RAM usage across all processes	system.mem
SystemFileSystem	filesystem disk usage	system.fs
SystemMetric	generic other system level metrics	(see below)
SystemNetInterface	network Interface statistics	system.nic
SystemNetProtocol	detailed TCP and UDP network statistics	system.netproto
ProcessCpu	process CPU usage	process.cpu
ProcessMem	process RAM usage	process.mem
ProcessInfo	detailed process configuration (priority, nice etc)	process.process
ProcessPing	process response times	process.ping
ProcessMemKdb	process RAM usage as tracked internally by kdb+	mem.kdbUsageBytes
ProcessConnection	connection open and closes to kdb+ processes	(see below)
ProcessMetric	Generic other process level metrics	(see below)
QueryDispatch	QR query dispatch status	qr.queryDispatch
QueryResponse	QR query response status	qr.queryResult
GwQueryStatus	GW query result status	gw.queryStatus

The following tables contain multiple source metrics:

table name          source metrics
--------------------------------------------
ProcessMetric       ipc.pendingHandleBytes
                    process.queryTimeout
                    process.exit
                    process.userAuthFailure
                    hdb.availableDates
                    hdb.latestDateRowCounts
                    pdb.rollover
                    tp.updDayCount
                    consumer.updDayCount
                    rdb.eodFlush
                    eb.primaryState
                    failover.process
                    failover.stack
                    failover.routing

SystemMetric        system.kernel
                    system.timeSync
                    system.sys
                    system.load
                    system.process

ProcessConnection   process.connectionOpen
                    process.connectionClosed

Two further tables provide the status of the Monitoring daemon and the monitored processes.

table name           description                                       
-----------------------------------------------------------------------
MonitoringMetrics    The metrics available (with send type) per process
MonitoringProcesses  The processes monitored and their connection state

All tables have a time column.

Process IDs are stored for all monitored processes in the MonitoringProcesses table (pid column). This information can map process names to PIDs to link all available data.

Delta messaging server

By default, the capture pipeline uses the emea_tr_a Delta Messaging server for data transfer between the processes. Ensure that the messagingServer parameter is updated for all processes in the DAAS_STACK_global_launch_monitoringCapture workflow if a different Messaging server is in use.

Manually defining a target feed handler

To override the target host/port for the Monitoring feed handler, set the following configuration:

Configuration	Description
.mon.cfg.refCaptureFhSource	Set to config to use the manually defined host and port
.mon.cfg.refCaptureHost	The hostname of the server running the Refinery Monitoring feed handler
.mon.cfg.refCapturePort	The port of the Refinery Monitoring feed handler