Monitoring

The kdb Insights Database can be configured to report metrics about data ingested and queries serviced, which can be reported to a monitoring endpoint. The actual scraping of these metrics is done by the kdb Insights sidecar container which uses the kxi-sidecar image, and can be be set to report to an event monitoring and alerting application such as Prometheus.

Query Metrics

Component	Name	Type	Description
SG	kxi_sg_ipc_requests_total	counter	Total QIPC requests
SG	kxi_sg_ipc_responses_total	counter	Total QIPC responses
SG	kxi_sg_http_requests_total	counter	Total HTTP requests
SG	kxi_sg_http_responses_total	counter	Total HTTP responses
SG	kxi_sg_pending	gauge	Number of pending queries (Both HTTP/IPC)
SG	kxi_sg_connected_aggregators	gauge	Number of connected aggregators
SG	kxi_sg_connected_coordinators	gauge	Number of connected coordinators
SG	kxi_sg_connected_clients	gauge	Number of connected q clients
RC	kxi_rc_reqs_total	counter	Service requests received
RC	kxi_rc_queue_length	gauge	Length of the outstanding request queue
RC	kxi_rc_connected_daps	gauge	Number of connected target DAPs
RC	kxi_rc_connected_aggs	gauge	Number of connected Aggs
RC	kxi_rc_retry_count	counter	Total number of request retry attempts
RC	kxi_rc_req_complete_time	histogram	Histogram of request completion times
Agg	kxi_agg_fn_time	histogram	Histogram of duration of aggregation functions
Agg	kxi_agg_errors	counter	Number of errors from aggregation functions
Agg	kxi_agg_timeouts	counter	Number of timeouts for requests for this agg
Agg	kxi_agg_partials_received	counter	Number of partial responses received
Agg	kxi_agg_requests_held	counter	Number of requests in progress
Agg	kxi_agg_http_json_reqs	counter	Number of HTTP JSON requests
Agg	kxi_agg_http_octet_reqs	counter	Number of HTTP octet stream requests
Agg	kxi_agg_ipc_reqs	counter	Number of IPC requests
DA	kxi_da_purview_start	gauge	Start timestamp of DA purview
DA	kxi_da_purview_end	gauge	End timestamp of DA purview
DA	kxi_da_records_after_purge	gauge	Total records remaining after a purge
DA	kxi_da_stream_msgs	counter	Number of inbound messages received
DA	kxi_da_stream_records	counter	Number of inbound records received
DA	kxi_da_stream_pos	counter	Current RT stream position
DA	kxi_da_requests	counter	Count of requests received in interval
DA	kxi_da_failed_requests	counter	Count of failed requests received in interval
DA	kxi_da_request_time	summary	Duration of requests received in interval
SM	kxi_sm_clients	gauge	Currently connected clients
SM	kxi_sm_stream_records	gauge	Number of records read from RT stream
SM	kxi_sm_msgs	gauge	Number of messages read from RT stream
SM	kxi_sm_eoi_requests_pending	gauge	EOI requests awaiting completion
SM	kxi_sm_eod_requests_pending	gauge	EOD requests awaiting completion
SM	kxi_sm_eoi_count	counter	Number of completed End of Interval runs
SM	kxi_sm_eod_count	counter	Number of completed End of Day runs
SM	kxi_sm_eoi_duration_seconds	gauge	Duration of the most recent End of Interval
SM	kxi_sm_eod_duration_seconds	gauge	Duration of the most recent End of Day
SM	kxi_sm_eoi_stream_pos	gauge	Current RT stream position
SM	kxi_sm_eoi_records	gauge	Number of records written during EOI
SM	kxi_sm_hdb_date_records	gauge	Number of total records in latest EOD partition
SM	kxi_sm_hdb_size	gauge	Size of HDB (in MB)
SM	kxi_sm_hdb_partitions	gauge	Number of partitions in HDB

Configuration

A Docker Compose example of how to set up kxi-sidecar and Prometheus in an environment is detailed below. First in the Docker Compose file, the sidecar and the Prometheus processes need to added.

  rdb: # Data Access Process RDB
    image: kxi-da
    command: -p 5080
    environment:
      - KXI_NAME=rdb
      - KXI_PORT=5080
      - KXI_SC=RDB
      - KXI_ASSEMBLY_FILE=/opt/kx/cfg/assembly/assembly.yml
    networks:
      - kx

  rdb-sidecar: # Sidecar for data access process named RDB
    image: kxi_sidecar:0.9.0 # Can be pulled from kdb Insights repo
    command: -p 8080
    environment:
      - KXI_CONFIG_FILE=/opt/kx/cfg/docker/rdb.json
    networks:
      - kx
    volumes:
      - ./config:/etc/kx/cfg # Make rdb.json available to container

prometheus: # Prometheus monitoring
  image: prometheus
  command: --config.file=/etc/prometheus/prometheus.yml
  ports:
   - "8080:8080"
  networks:
   - kx
  volumes:
   - ./config:/etc/prometheus # Make prometheus configuration available

An example rdb.json file is shown below. In it the connection field points to the main DAP container and is set to scrape the container every 5 seconds through metrics.frequency.

{
    "connection": ":rdb_1:5080",
    "frequencySecs": 5,
    "metrics":
    {
        "enabled":"true",
        "frequency": 5,
        "handler": {
            "pc": true,
            "pg": true,
            "ph": true,
            "po": true,
            "pp": true,
            "ps": true,
            "ts": true,
            "wc": true,
            "wo": true,
            "ws": true
          }
    }
}

An example prometheus.yml set to scrape every 15 seconds and set to actively scrape from the sidecar process is shown below.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

scrape_configs:
  - job_name: 'rdb-monitoring'
    static_configs:
      - targets: ['rdb-sidecar_1:8080'] # Point to RDB's sidecar

Storage Metrics

When integrating with the monitoring sidecar, the following metrics will be available.

component	type	name	description
SM	gauge	kxi_sm_clients	Currently connected clients
SM	gauge	kxi_sm_stream_records	Number of records read from RT stream
SM	gauge	kxi_sm_msgs	Number of messages read from RT stream
SM	gauge	kxi_sm_eoi_requests_pending	EOI requests awaiting completion
SM	gauge	kxi_sm_eod_requests_pending	EOD requests awaiting completion
SM	counter	kxi_sm_eoi_count	Number of completed End of Interval runs
SM	counter	kxi_sm_eod_count	Number of completed End of Day runs
SM	gauge	kxi_sm_eoi_duration_seconds	Duration of the most recent End of Interval
SM	gauge	kxi_sm_eod_duration_seconds	Duration of the most recent End of Day
SM	gauge	kxi_sm_eoi_stream_pos	Current RT stream position
SM	gauge	kxi_sm_eoi_records	Number of records written during EOI
SM	gauge	kxi_sm_hdb_date_records	Number of total records in latest EOD partition
SM	gauge	kxi_sm_hdb_size	Size of HDB (in MB)
SM	gauge	kxi_sm_hdb_partitions	Number of partitions in HDB