Deploy the Monitoring Stack Using the CLI

This page guides you through deploying the monitoring stack using the kdb Insights CLI.

About kxi monitoring

The kxi monitoring command installs and configures the following:

Prometheus for metric scraping and storage
- Configure service monitors for service discovery
  
  Prometheus uses a pull-based model for collecting metrics from applications and services. Applications and services expose HTTP(S) endpoints that return metrics in Prometheus format, and Prometheus periodically scrapes these endpoints based on its configuration.
  
  The Prometheus operator includes a Custom Resource Definition for defining ServiceMonitor resources. You use a ServiceMonitor to specify which applications Prometheus should scrape within a Kubernetes cluster.. The controller actions the ServiceMonitors you define and automatically builds the required Prometheus configuration.
  
  Within the ServiceMonitor, you define Kubernetes labels that the Operator uses to identify the target Kubernetes Services, which in turn select the Pods to be monitored.
  
  Refer to the ServiceMonitor documentation for more details.
Grafana for analysis and visualization
- Predefined dashboards for visualization
  
  The kube-prometheus stack deploys a set of built-in dashboards to monitor the state of the Kubernetes cluster. In addition, kdb Insights Enterprise provides predefined dashboards for monitoring platform components. Grafana organizes these dashboards into folders that include the namespace in which kdb Insights Enterprise is deployed.
- Predefined alerts
  
  kdb Insights Enterprise provides a set of pre-configured alerts to help you monitor and maintain the health of kdb Insights Enterprise.
Grafana Loki for log storage is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. Loki is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream. Loki is a complex system with many configuration options. Read more about Loki here.
- Fluent Bit for log collection. kdb Insights Enterprise works with many logging stacks through Fluent Bit, which is a very fast, lightweight, and highly scalable logging and metrics processor and forwarder that you can configure to forward logs to Loki. Refer to the Fluent Bit documentation for more information.
- Kubernetes event exporter for event persistence allows exporting the often missed Kubernetes events to various outputs so that they can be used for observability or alerting purposes. You can configure it to send Kubernetes events to Loki.

Prerequisites

Before you begin, ensure the following are in place:

The latest version of kdb Insights CLI, installed and configured to point to your kdb Insights Enterprise deployment.
The latest KX Management Service and Insights On K8s module package. To ensure you have the latest versions, run an upgrade using the kxi install upgrade command, which prompts you to upgrade if necessary.

Deploy on Kubernetes

The kxi monitoring command does not currently support on-premises installations.

Deployment flow

To deploy the monitoring stack, you run a single CLI command. During execution, the CLI prompts you for configuration values and uses the provided credentials to provision storage and deploy the required components.

Run the installation command

To deploy the monitoring stack using the CLI, run the following command:

kxi monitoring install [OPTIONS] INSIGHTS_CLUSTER_NAME {s3|azure|gcs} CREDENTIALS_PATH

This command:

Deploys the monitoring stack into your Kubernetes cluster
Configures object storage for metrics and logs
Installs and configures Prometheus, Grafana, and Loki

The command requires the following options:

Option	Description
insights_cluster_name	Name of the Kubernetes cluster running Insights.
provider	Object storage provider to configure. Available options: `azure`, `s3`, or `gcs`.
credentials_path	Path to the credentials JSON file matching the provider. Credentials are required for creating the buckets for the metric and log stores. Click on a tab below to see credentials for each cloud provider.

After the command starts, kdb Insights Enterprise prompts you for any remaining required values.

Configuration arguments

The following arguments control how the monitoring stack is installed and configured:

Arguments	Required	Default	Details
namespace		`insights`	Namespace where kdb Insights Enterprise is installed
monitoring-namespace		`monitoring`	Namespace where the monitoring stack will be installed
install-grafana		false	Install Grafana with Prometheus
grafana-admin-user	Yes if `install-grafana` is set		User name of Grafana admin, required if `--install-grafana` is set
grafana-admin-password	Yes if `install-grafana` is set		Password of Grafana admin, required if `--install-grafana` is set
log-retention-period		15	Loki retention period for logs in days
metrics-retention-period		15	Prometheus retention period for metrics in days
metrics-retention-size		45	Prometheus retention for metrics in GB

Provide storage credentials

Select the tab that matches the object storage provider specified in the command. Each provider requires a credentials JSON file with the fields shown below.

Note

These credentials allow the CLI to create and configure the required buckets or containers for metrics and logs.

Microsoft Azure Storage AWS S3 Google Cloud Storage

The credentials are as follows:

{
"mimirStorageAccountName": "value",
"mimirStorageAccountKey": "value",
"lokiStorageAccountName": "value",
"lokiStorageAccountKey": "value"
}

Key	Required	Description
`mimirStorageAccountName`	Yes	The name of the Azure Storage Account for Mimir metrics storage.
`mimirStorageAccountKey`	Yes	The access key to the Mimir Storage Account.
`lokiStorageAccountName`	Yes	The name of the Azure Storage Account for Loki log storage.
`lokiStorageAccountKey`	Yes	The access key to the Loki Storage Account.

The credentials are as follows:

{
"mimirStorageAccessKey": "value",
"mimirStorageSecretKey": "value",
"mimirStorageEndpoint": "value",
"mimirStorageBucket": "value",
"mimirStorageRegion": "value",
"lokiStorageAccessKey": "value",
"lokiStorageSecretKey": "value",
"lokiStorageEndpoint": "value",
"lokiStorageRegion": "value",
"lokiAdminContainerName": "value",
"lokiChunksContainerName": "value",
"lokiRulerContainerName": "value"
}

Key	Required	Description
`mimirStorageAccessKey`	Yes	The S3 access key ID for Mimir metrics storage. Usually 20 char long.
`mimirStorageSecretKey`	Yes	The S3 secret key for Mimir metrics storage. Usually 40 char long.
`mimirStorageEndpoint`	Yes	The URL of the S3 provider for Mimir. For AWS S3 use: `https://s3.__region__.amazonaws.com`.
`mimirStorageBucket`	Yes	The S3 bucket name for Mimir metrics storage.
`mimirStorageRegion`	Yes	The AWS region for Mimir storage.
`lokiStorageAccessKey`	Yes	The S3 access key ID for Loki log storage. Usually 20 char long.
`lokiStorageSecretKey`	Yes	The S3 secret key for Loki log storage. Usually 40 char long.
`lokiStorageEndpoint`	Yes	The URL of the S3 provider for Loki. For AWS S3 use: `https://s3.__region__.amazonaws.com`.
`lokiStorageRegion`	Yes	The AWS region for Loki storage.
`lokiAdminContainerName`	Yes	The S3 container/bucket name for Loki admin operations.
`lokiChunksContainerName`	Yes	The S3 container/bucket name for Loki chunks storage.
`lokiRulerContainerName`	Yes	The S3 container/bucket name for Loki ruler storage.

AWS hosted bucket region

If you are unable to access the AWS hosted bucket with administrator permissions to identify its region, use the following command to retrieve it:

curl -sI https://<bucketname>.s3.amazonaws.com | grep bucket-region

The credentials are as follows:

{
"mimirStorageBucket": "value",
"lokiAdminContainerName": "value",
"lokiChunksContainerName": "value",
"lokiRulerContainerName": "value",
"mimirServiceAccount": {
  "type": "value",
  "project_id": "value",
  "private_key_id": "value",
  "private_key": "value",
  "client_email": "value",
  "client_id": "value",
  "auth_uri": "value",
  "token_uri": "value",
  "auth_provider_x509_cert_url": "value",
  "client_x509_cert_url": "value",
  "universe_domain": "value"
},
"lokiServiceAccount": {
  "type": "value",
  "project_id": "value",
  "private_key_id": "value",
  "private_key": "value",
  "client_email": "value",
  "client_id": "value",
  "auth_uri": "value",
  "token_uri": "value",
  "auth_provider_x509_cert_url": "value",
  "client_x509_cert_url": "value",
  "universe_domain": "value"
}
}

Key	Required	Description
`mimirStorageBucket`	Yes	The GCS bucket name for Mimir metrics storage.
`lokiAdminContainerName`	Yes	The GCS container name for Loki admin operations.
`lokiChunksContainerName`	Yes	The GCS container name for Loki chunks storage.
`lokiRulerContainerName`	Yes	The GCS container name for Loki ruler storage.
`mimirServiceAccount`	Yes	Service account configuration for Mimir GCS access.
`lokiServiceAccount`	Yes	Service account configuration for Loki GCS access.

Service Account Fields:

Field	Description
`type`	The Google Cloud service account type.
`project_id`	The Google Cloud Project ID.
`private_key_id`	The Google Cloud private key ID.
`private_key`	The Google Cloud private key.
`client_email`	The Google Cloud client email.
`client_id`	The Google Cloud client ID.
`auth_uri`	The Google Cloud authorization URI.
`token_uri`	The Google Cloud token URI.
`auth_provider_x509_cert_url`	The Google Cloud auth provider X.509 cert URL.
`client_x509_cert_url`	The Google Cloud client X.509 cert URL.
`universe_domain`	The Google Cloud universe domain.

Google Cloud only provides short lived tokens

A service account needs to be created and exported from keycloak. Refer to managing service accounts for details on creating a service account. The export is a JSON containing the values seen in the credential sections.

Access Grafana

Once the deployment completes successfully, access Grafana at:

https://${INSIGHTS_HOST}/grafana

Log in using the Grafana administrator credentials you provided during installation.

Uninstalling on Kubernetes

To remove the monitoring stack, run the following command:

kxi monitoring uninstall

Arguments	Required	Default	Details
namespace		`insights`	Namespace where kdb Insights Enterprise is installed
monitoring-namespace		`monitoring`	Namespace where the monitoring stack will be installed

Next steps

Learn more about the kdb Insights Grafana Dashboard Reference