Observability

This section we will show you how to configure the application observability; logging and metrics.

Configure logging

You can configure the log level of your application at installation time using the values below. kdb Insights Enterprise uses log components as a way to identify the origin of a log message. The routings object allows you to set the log level for different components with DEFAULT applying to anything that's not set explicitly.

global:
  logging:
    routings:
      DEFAULT: INFO
      InfoService: DEBUG

The example above sets all components to INFO with the exception of the InfoService component, which is set to DEBUG.

See details here on logging stacks to deploy.

Configure metrics

kdb Insights Enterprise generates metrics and includes pre-packaged alerts and monitoring dashboards to allow you to check system performance and quickly identify potential issues. You can monitor data flows, component errors, and a variety of other metrics.

By default these are disabled for kdb Insights Enterprise since the application does not deploy with the Prometheus stack. See details here on how to deploy the Prometheus stack.

To enable metrics, alerts and dashboards for your deployment, update your Helm values file and set global.metrics.enabled to true.

This is created as part of the install setup.

global:
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      interval: 30s
      additionalLabels:
        release: kx-prom

Release names must match

The global.metrics.serviceMonitor.additionalLabels.release parameter must match the Helm release name $RELEASE_NAME from the previous section when Prometheus was installed.

When you have updated the values file, run kxi install run --filepath values.yaml to apply the change.

This deploys Prometheus Service Monitor resources, which scrape kdb Insights Enterprise components and publish metrics to the backend.

Enable Prometheus scraping

There are two methods of enabling Prometheus scraping:

Prometheus ServiceMonitors
pod annotations

ServiceMonitors

The global.metrics.serviceMonitor object configures the use of the Prometheus ServiceMonitor object within the application. With this mode enabled, the application will create a set of ServiceMonitor resources, which Prometheus uses to scrape metrics.

variable	type	example	default
`global.metrics.serviceMonitor.enabled`	`bool`	`true`	`false`
`global.metrics.serviceMonitor.interval`	`int`	`30s`	`10s`
`global.metrics.serviceMonitor.additionalLabels.release`	`string`	`kx-prom`	`""`

enabled - enables the ServiceMonitor mode
interval - metrics polling interval
additionalLabels.release - this is a label to match your Prometheus deployment name.

Pod annotations

The other method of enabling the scaping metrics is to use pod annotations instead of using ServiceMonitors. This is useful in situations where the Prometheus operator cannot be used, or another approach is preferred.

The values file below shows the required configuration.

global:
  metrics:
    enabled: true
    useAnnotations: true
    serviceMonitor:
      enabled: false

Setting global.metrics.useAnnotations to true will add Prometheus annotations to the application pods. These annotations will be prefixed with prometheus.io.

Disabling alerts and dashboards

To scrape metrics but disable both the install of alerts and dashboards, include the following in the metrics section of your Helm values file:

kxi-monitoring:
  enabled: false

To disable alerts but enable dashboards, include the following in the metrics section of your Helm values file:

kxi-monitoring:
  metrics:
    alerts: false

To disable dashboards but enable alerts, include the following in the metrics section of your Helm values file:

kxi-monitoring:
  grafana:
    dashboards: false