Install the Monitoring components

Installation prerequisites for all Monitoring components

  • A default KX Refinery install already present
  • All components are installed using RPM. RPM requires root access to perform package installation; therefore root access is required.
  • The group kx-refinery and user kx-refinery need to be created on the server where installation is taking place. See Example drop-in unit for reference on how to add this group and user if required.

List of Monitoring components to install

  • Monitoring daemon
  • Monitoring Alerting API
  • Monitoring Web API
  • Monitoring Elasticsearch Uploader API
  • Monitoring System Information Generator

Installing Monitoring daemon

The Refinery Monitoring daemon provides an endpoint for all processes within Refinery to send monitoring information (metrics) to.

Initial install using RPM

The daemon is installed via RPM and managed with systemd. To install the Daemon with RPM:

rpm -iv kx-refinery-monitoring-daemon-*version*-*release*.oem.x86_64.rpm

To upgrade the daemon with RPM, use -U:

rpm -Uv kx-refinery-monitoring-daemon-*version*-*release*.oem.x86_64.rpm

Once installed, the following configuration is required:

  • System runtime configuration
  • Application configuration

System runtime configuration

The daemon is managed by systemd. The installed service file can be found at /etc/systemd/system/kx-refinery-monitoring-daemon@.service.

Note that the Monitoring components are configured as template units and therefore require an instance value after the @. At install, one instance is supported: default. The systemd configuration for the daemon sets the following by default:

  • Run-as user (kx-refinery)
  • Run-as group (kx-refinery)
  • Bind to port (10000)

Do not update this file directly as it will be overwritten when the application is updated.

To change any entry in the service file, use the systemd drop-in units.

Example drop-in unit

Application configuration

The Monitoring daemon configuration is stored in /etc/kx-refinery-monitoring-daemon/ with a folder per instance. The primary configuration file is mon.config.q, with additional configuration in mon.config-extra.q. All configuration stored here is protected by RPM from being overwritten by application upgrades. The following configuration must be completed prior to application start in mon.config.q:

.mon.cfg.dcHost   hostname of the server running the Delta Control process
.mon.cfg.dcPort   port of the Delta Control process
.mon.cfg.dcUser   username to log into the Delta Control process
.mon.cfg.dcPass   password to log into the Delta Control process
.mon.cfg.refUser  username to log into the KX Refinery processes to be monitored
.mon.cfg.refPass  password to log into the KX Refinery processes to be monitored

To find the port of the Delta Control process, view the environmental variable DELTACONTROL_PORT within the KX Install's delta.profile file

The latest template version of the configuration can always be found in /opt/kx-refinery-monitoring-daemon/current/config/default. All post-installation steps can also be found in the installation folder at /opt/kx-refinery-monitoring-daemon/current/POST-INSTALL.txt.

Start up the daemon

Following successful install of the daemon through RPM and configuration above, the daemon can be run by following the steps in Running Components. A sample command for this would be:

systemctl start kx-refinery-monitoring-daemon@default

Installing Monitoring Alerting API

The Monitoring Alerting API allows you to send alerts via Slack or email.

Initial install Using RPM

The Alerting API is installed via RPM and managed with systemd. The Alerting API has a dependency on the package mutt and cannot be installed without it. To install mutt run the following on the server:

yum install mutt

To install the Alerting API with RPM:

rpm -iv kx-refinery-monitoring-api-alrt-*version*-*release*.oem.x86_64.rpm

To upgrade the Alerting API with RPM, use -U:

rpm -Uv kx-refinery-monitoring-api-alrt-*version*-*release*.oem.x86_64.rpm

Once installed, the following configuration is required:

  • System runtime configuration
  • Application configuration

System runtime configuration

The Alerting API is managed by systemd. The installed service file can be found at /etc/systemd/system/kx-refinery-monitoring-api-alrt@.service.

Note that the Monitoring components are configured as template units and therefore require an instance value after the "@". At install, one instance is supported "default".The systemd configuration for the Alerting API sets the following by default:

  • Run-as user (kx-refinery)
  • Run-as group (kx-refinery)
  • Bind to port (10003)

Do not update this file directly as it will be overwritten when the application is updated.

To change any entry in the service file, use the systemd drop-in units.

Example drop-in unit

Application configuration

The Alerting API configuration is stored in /etc/kx-refinery-monitoring-api-alrt/ with a folder per instance. The primary configuration file is alrt.config.q, with additional configuration in alrt.config-extra.q.

All configuration stored here is protected by RPM from being overwritten by application upgrades.

The following mandatory configuration must be completed prior to application start in alrt.config.q:

.alrt.cfg.notificationModes: Enable one of the supported notification modes
.sub.cfg.conn.host: The hostname of the server running Kx Refinery Monitoring daemon (generally "localhost")
.sub.cfg.conn.port: The port of the KX Refinery Monitoring daemon (default is 10000)
.sub.cfg.conn.user: The username to log into the KX Refinery Monitoring daemon, if enabled
.sub.cfg.conn.pass: The password to log into the KX Refinery Monitoring daemon, if enabled
.alrt.cfg.notificationModes: The type of alerting to fire when a threshold is breached. Multiple notification modes are supported. `email or `slack or `email`slack for both
.alrt.email.cfg.sendTo: If e-mail notifications are enabled, specify the recipient list.
.alrt.slack.cfg.webhookUrl: If Slack notifications are enabled, specify the target URL.

Create and define alerts for the Alerting component

The latest template version of the configuration can always be found in /opt/kx-refinery-monitoring-api-alrt/current/config/default. All post-installation steps can also be found in the installation folder at /opt/kx-refinery-monitoring-api-alrt/current/POST-INSTALL.txt.

Start up the Alerting API

Following successful install of the Alerting API through RPM and configuration above, you can run the Alerting API by following the steps in Running Components. A sample command for this would be:

systemctl start kx-refinery-monitoring-api-alrt@default

Confirm that the alerting API is running

To confirm that the alerting API is running as expected and a guide on how to further use it, refer to Alerts

Testing email and Slack alerts

Email

Email alerts are sent via the mutt command, which is a dependency of the RPM at install time. However, this must then be configured with the local email server.

To test an email run:

echo "Test <b>HTML</b> e-mail" | mutt -e 'set content_type=text/html' \
   -s "Test HTML e-mail" *your-email-address*

If this succeeds, the Alerting component will be able to send email alerts.

Slack

Slack alerts are sent via a HTTP POST request to a Slack webhook. If the server can connect directly to the Internet, no further configuration is required.

If Internet access is via a proxy, ensure that the proxy allows access to the Slack webhook URL (https://hooks.slack.com) and that the following environment variables are set in the systemd template file:

Environment="HTTP_PROXY=*proxy-server*"
Environment="HTTPS_PROXY=*proxy-server*"
Environment="http_proxy=*proxy-server*"
Environment="https_proxy=*proxy-server*"

To test a Slack alert:

wget -X POST --header='Content-Type:application/json' \
   --post-data '{"text":"Test","username":"user"}' *slack-hook-url*

Installing Monitoring Web API

The Monitoring Web API provides a HTTP JSON and WebSocket API on top of the Monitoring data published by the daemon.

Initial install using RPM

The Web API is installed via RPM and managed with systemd. To install the Web API with RPM:

rpm -iv kx-refinery-monitoring-api-web-*version*-*release*.oem.x86_64.rpm

To upgrade the Web API with RPM, use -U:

rpm -Uv kx-refinery-monitoring-api-web-*version*-*release*.oem.x86_64.rpm

Once installed, the following configuration is required:

  • System runtime configuration
  • Application configuration

The following optional configuration is also available:

  • SSL/TLS configuration

System runtime configuration

The Web API is managed by systemd. The installed service file can be found at /etc/systemd/system/kx-refinery-monitoring-api-web@.service.

Note that the Monitoring components are configured as template units and therefore require an instance value after the "@". At install, one instance is supported "default".The systemd configuration for the Web API sets the following by default:

  • Run-as user (kx-refinery)
  • Run-as group (kx-refinery)
  • Bind to port (10001)

Do not update this file directly as it will be overwritten when the application is updated.

To change any entry in the service file, use the systemd drop-in units.

Example drop-in unit

Application configuration

The Web API configuration is stored in /etc/kx-refinery-monitoring-api-web/ with a folder per instance. The primary configuration file is web.config.q, with additional configuration in web.config-extra.q. All configuration stored here is protected by RPM from being overwritten by application upgrades.

The following mandatory configuration must be completed prior to application start in web.config.q:

.sub.cfg.conn.host: The hostname of the server running Kx Refinery Monitoring daemon (generally "localhost")
.sub.cfg.conn.port: The port of the KX Refinery Monitoring daemon (default is 10000)
.sub.cfg.conn.user: The username to log into the KX Refinery Monitoring daemon, if enabled
.sub.cfg.conn.pass: The password to log into the KX Refinery Monitoring daemon, if enabled

The latest template version of the configuration can always be found in /opt/kx-refinery-monitoring-api-web/current/config/default. All post-installation steps can also be found in the installation folder at /opt/kx-refinery-monitoring-api-web/current/POST-INSTALL.txt.

By default the monitoring API will allow all inbound connections. To lock down access:

1. Enable user authentication by setting '.boot.cfg.enableUserAuthentication' to 1b (true) in web.config.q
2. Add username and passwords to /etc/kx-refinery-monitoring-api-web/*instance*/user-passwd-access

SSL/TLS Configuration

If you require HTTPS access to the Web API, the following environment variables must be set within the Delta Control delta.profile:

KX_SSL_CERT_FILE:    The path to the server certificate file
KX_SSL_CA_CERT_FILE: The path to the certificate file containing the chain to the root certificate
KX_SSL_KEY_FILE:     The path to the key file for the server

With this configuration, the Web API will respond to HTTP and HTTPS requests.

Start up the Web API

Following successful install of the Web API through RPM and configuration above, you can run the Web API by following the steps in Running Components. A sample command for this would be:

systemctl start kx-refinery-monitoring-api-web@default

Confirm that the Web API is running

To confirm that the Web API is running as expected and a guide on how to further use it, refer to Accessing Monitoring Data

Installing Monitoring Elasticsearch Uploader API

The Monitoring Elasticsearch Uploader API uploads all Monitoring data into Elasticsearch for capturing and analysis with Kibana.

Initial install Using RPM

The Elasticsearch Uploader API is installed via RPM and managed with systemd. To install the Elasticsearch Uploader API with RPM:

rpm -iv kx-refinery-monitoring-api-es-*version*-*release*.oem.x86_64.rpm

To upgrade the Elasticsearch Uploader API with RPM, use -U:

rpm -Uv kx-refinery-monitoring-api-es-*version*-*release*.oem.x86_64.rpm

Once installed, the following configuration is required:

  • System runtime configuration
  • Application configuration

System runtime configuration

The Elasticsearch Uploader API is managed by systemd. The installed service file can be found at /etc/systemd/system/kx-refinery-monitoring-api-es@.service.

Note that the Monitoring components are configured as template units and therefore require an instance value after the "@". At install, one instance is supported "default".The systemd configuration for the Web API sets the following by default:

  • Run-as user (kx-refinery)
  • Run-as group (kx-refinery)

The process will bind to a random free port when it starts up.

Do not update this file directly as it will be overwritten when the application is updated.

To change any entry in the service file, use the systemd drop-in units.

Example drop-in unit

Application configuration

The Elasticsearch Uploader API configuration is stored in /etc/kx-refinery-monitoring-api-es/ with a folder per instance. The primary configuration file is esp.config.q, with additional configuration in esp.config-extra.q. All configuration stored here is protected by RPM from being overwritten by application upgrades.

The following mandatory configuration must be completed prior to application start in esp.config.q:

.esp.cfg.es.url:    The URL of the Elasticsearch instance to push to. This must be http:// or https://
.sub.cfg.conn.host: The hostname of the server running Kx Refinery Monitoring daemon (generally "localhost")
.sub.cfg.conn.port: The port of the KX Refinery Monitoring daemon
.sub.cfg.conn.user: The username to log into the KX Refinery Monitoring daemon, if enabled
.sub.cfg.conn.pass: The password to log into the KX Refinery Monitoring daemon, if enabled

The latest template version of the configuration can always be found in /opt/kx-refinery-monitoring-api-es/current/config/default. All post-installation steps can also be found in the installation folder at /opt/kx-refinery-monitoring-api-es/current/POST-INSTALL.txt.

By default the Elasticsearch API will allow all inbound connections. To lock down access:

1. Enable user authentication by setting '.boot.cfg.enableUserAuthentication' to 1b (true) in esp.config.q
2. Add username and passwords to /etc/kx-refinery-monitoring-api-es/*instance*/user-passwd-access

Start up the Elasticsearch API

Following successful install of the Elasticsearch API through RPM and configuration above, you can run the Elasticsearch API by following the steps in Running Components. A sample command for this would be:

systemctl start kx-refinery-monitoring-api-es@default

Confirm that the Elasticsearch API is running

To confirm that the Elasticsearch API is running as expected and a guide on how to further use it, refer to Accessing Monitoring Data

Installing Monitoring System Information Generator

The Monitoring System Information Generator publishes system-level and process-level metrics gathered directly from the operating system itself (similar to SIGAR).

Initial install using RPM

The System Information Generator is installed via RPM and managed with systemd. To install the System Information Generator with RPM:

rpm -iv kx-refinery-monitoring-gen-sysi-*version*-*release*.oem.x86_64.rpm

To upgrade the System Information Generator with RPM, use -U:

rpm -Uv kx-refinery-monitoring-gen-sysi-*version*-*release*.oem.x86_64.rpm

Once installed, the following configuration is required:

  • System runtime configuration
  • Application configuration

System runtime configuration

The System Information Generator is managed by systemd. The installed service file can be found at /etc/systemd/system/kx-refinery-monitoring-gen-sysi@.service.

Note that the Monitoring components are configured as template units and therefore require an instance value after the "@". At install, one instance is supported "default".The systemd configuration for the Web API sets the following by default:

  • Run-as user (kx-refinery)
  • Run-as group (kx-refinery)
  • Bind to port (10002)

Do not update this file directly as it will be overwritten when the application is updated.

To change any entry in the service file, use the systemd drop-in units.

Example drop-in unit

This application requires access to the '/proc' filesystem.

If the current kernel version >= 3.3, it is recommended to modify the '/proc' mount to allow the 'kx-refinery' group full access to this filesystem. See "Mount options" at http://man7.org/linux/man-pages/man5/proc.5.html for more information

If the current kernel version < 3.3, you should override the user and run the application as 'root'.

Application configuration

The Elasticsearch Uploader API configuration is stored in /etc/kx-refinery-monitoring-gen-sysi/ with a folder per instance. The primary configuration file is sysi.config.q, with additional configuration in sysi.config-extra.q.

All configuration stored here is protected by RPM from being overwritten by application upgrades. The only configuration required is in the Monitoring daemon if the port of the Generator is changed.

The latest template version of the configuration can always be found in /opt/kx-refinery-monitoring-api-web/current/config/default. All post-installation steps can also be found in the installation folder at /opt/kx-refinery-monitoring-api-web/current/POST-INSTALL.txt.

By default the System Information Generator will allow all inbound connections. To lock down access:

1. Enable user authentication by setting '.boot.cfg.enableUserAuthentication' to 1b (true) in sysi.config.q
2. Add username and passwords to /etc/kx-refinery-monitoring-gen-sysi/*instance*/user-passwd-access

Start up the System Information Generator

Following successful install of the System Information Generator through RPM and configuration above, you can run the System Information Generator by following the steps in Running Components. A sample command for this would be:

systemctl start kx-refinery-monitoring-gen-sysi@default

Confirm that the System Information Generator is running

To confirm that the System Information Generator is running as expected and a guide on how to further use it, refer to Monitoring Dashboards

Alternative installation - install the Monitoring components through .tar.gz files

An alternative method for installation is to install the Monitoring components through the .tar.gz files builds from Nexus. Below is a walkthrough of how this installation procedure would work for the KX Refinery daemon. The same methods can be performed to install any of the Monitoring components. In our example, we are working with the package kx-refinery-monitoring-daemon-1.0.0.tar.gz

Unzip the .tar.gz installation package

tar -zxvf kx-refinery-monitoring-daemon-1.0.0.tar.gz

Update the config/default/mon.config.q folder to update the daemon config file, identical to the step "Application Configuration" for the daemon found above: Application Configuration

Update the bin/start-kx-refinery-monitoring-daemon.sh file to contain the following variables:

export KX_REF_MON_PORT=10000
export KX_REF_MON_HOME="/home/PACKAGESDIR/monitoring/kx-refinery-monitoring-daemon"
export KX_REF_MON_DATA="/home/DELTAINSTALLDIR/KxRefineryMonitoring/daemon/default"

Run the following script from the bin folder to perform the installation:

./start-kx-refinery-monitoring-daemon.sh --background --instance=default