Monitoring
Service Gateway Metrics
To configure Service Gateway metrics the following environment variables must be configured:
| Name | Description |
|---|---|
KXI_SG_METRICS_ENABLED |
Boolean - set to true when enabling Service Gateway metrics. |
KXI_SG_METRICS_ENDPOINT |
Endpoint for users to access metrics. For example: /metrics |
KXI_SG_METRICS_PORT |
Port for metrics process and endpoint. |
Environment variables are configured differently depending on the deployment method. Refer to the examples below for configuring using Docker or Kubernetes.
Ensure that metrics are enabled globally when configuring the Service Gateway.
global:
metrics:
enabled: true
In Docker, environment variables are supplied under an environment key for the target service as a list of key-value pairs.
services:
sg:
environment:
- KXI_SG_METRICS_ENABLED=true
- KXI_SG_METRICS_ENDPOINT=/metrics
- KXI_SG_METRICS_PORT=8081
In a Kubernetes deployment, environment variables are passed to components using a values file. For the Service Gateway, these values are supplied under an sg-gateway key.
kdb Insights Enterprise
In an enterprise deployment, values are nested under either a service-gateway or qe-gateway key in the values file. Refer to the enterprise deployment configuration for more information.
service-gateway:
sg-gateway:
env:
- name: KXI_SG_METRICS_ENABLED
value: "true"
- name: KXI_SG_METRICS_ENDPOINT
value: "/metrics"
- name: KXI_SG_METRICS_PORT
value: "8081"
The following list provides the metrics that are reported to the KXI_SG_METRICS_ENDPOINT when enabled.
| Component | Name | Type | Description |
|---|---|---|---|
| SG | kxi_sg_ipc_requests_total |
counter | Total QIPC requests |
| SG | kxi_sg_ipc_responses_total |
counter | Total QIPC responses |
| SG | kxi_sg_http_requests_total |
counter | Total HTTP requests |
| SG | kxi_sg_http_responses_total |
counter | Total HTTP responses |
| SG | kxi_sg_pending |
gauge | Number of pending queries (Both HTTP/IPC) |
| SG | kxi_sg_connected_aggregators |
gauge | Number of connected aggregators |
| SG | kxi_sg_connected_coordinators |
gauge | Number of connected coordinators |
| SG | kxi_sg_connected_clients |
gauge | Number of connected q clients |
Example Service Gateway metrics request:
curl http://localhost:8081/metrics
# HELP kxi_sg_ipc_requests_total Total qipc requests.
# TYPE kxi_sg_ipc_requests_total counter
kxi_sg_ipc_requests_total 4.0
# HELP kxi_sg_pending Number of pending queries (Both HTTP/ipc).
# TYPE kxi_sg_pending gauge
kxi_sg_pending 0.0
# HELP kxi_sg_ipc_responses_total Total qipc responses.
# TYPE kxi_sg_ipc_responses_total counter
kxi_sg_ipc_responses_total 4.0
# HELP kxi_sg_connected_coordinators Number of connected resource-coordinators and outbound connections.
# TYPE kxi_sg_connected_coordinators gauge
kxi_sg_connected_coordinators 1.0
# HELP kxi_sg_connected_clients Number of connected q clients.
# TYPE kxi_sg_connected_clients gauge
kxi_sg_connected_clients 0.0
# HELP kxi_sg_http_requests_total Total HTTP requests.
# TYPE kxi_sg_http_requests_total counter
kxi_sg_http_requests_total 3.0
# HELP kxi_sg_http_responses_total Total HTTP responses.
# TYPE kxi_sg_http_responses_total counter
kxi_sg_http_responses_total 3.0
# HELP kxi_sg_connected_aggregators Number of connected aggregators.
# TYPE kxi_sg_connected_aggregators gauge
kxi_sg_connected_aggregators 1.0
# HELP kxi_sg_http_requests_created Total HTTP requests.
# TYPE kxi_sg_http_requests_created gauge
kxi_sg_http_requests_created 1.729547023193E9
# HELP kxi_sg_http_responses_created Total HTTP responses.
# TYPE kxi_sg_http_responses_created gauge
kxi_sg_http_responses_created 1.729547023193E9
# HELP kxi_sg_ipc_requests_created Total qipc requests.
# TYPE kxi_sg_ipc_requests_created gauge
kxi_sg_ipc_requests_created 1.729547023192E9
# HELP kxi_sg_ipc_responses_created Total qipc responses.
# TYPE kxi_sg_ipc_responses_created gauge
kxi_sg_ipc_responses_created 1.729547023193E9
The kdb Insights Database can be configured to report metrics about data ingested and queries serviced. These metrics can be reported to a monitoring endpoint. The kdb Insights sidecar container, which uses the kxi-sidecar image, scrapes the query and storage metrics and can report them to an event monitoring and alerting application, such as Prometheus.
Docker example
Refer to the Docker metrics example for a complete deployment example for metrics.
Query Metrics
| Component | Name | Type | Description |
|---|---|---|---|
| RC | kxi_rc_reqs_total |
counter | Service requests received |
| RC | kxi_rc_queue_length |
gauge | Length of the outstanding request queue |
| RC | kxi_rc_connected_daps |
gauge | Number of connected target DAPs |
| RC | kxi_rc_connected_aggs |
gauge | Number of connected Aggs |
| RC | kxi_rc_retry_count |
counter | Total number of request retry attempts |
| RC | kxi_rc_req_complete_time |
histogram | Histogram of request completion times |
| Agg | kxi_agg_fn_time |
histogram | Histogram of duration of aggregation functions |
| Agg | kxi_agg_errors |
counter | Number of errors from aggregation functions |
| Agg | kxi_agg_timeouts |
counter | Number of timeouts for requests for this agg |
| Agg | kxi_agg_partials_received |
counter | Number of partial responses received |
| Agg | kxi_agg_requests_held |
counter | Number of requests in progress |
| Agg | kxi_agg_http_json_reqs |
counter | Number of HTTP JSON requests |
| Agg | kxi_agg_http_octet_reqs |
counter | Number of HTTP octet stream requests |
| Agg | kxi_agg_ipc_reqs |
counter | Number of IPC requests |
| DA | kxi_da_pending_reloads |
gauge | Number of expected reloads pending |
| DA | kxi_da_purview_start |
gauge | Start timestamp of DA purview |
| DA | kxi_da_purview_end |
gauge | End timestamp of DA purview |
| DA | kxi_da_records_after_purge |
gauge | Total records remaining after a purge |
| DA | kxi_da_stream_msgs |
counter | Number of inbound messages received |
| DA | kxi_da_stream_records |
counter | Number of inbound records received |
| DA | kxi_da_stream_pos |
counter | Current RT stream position |
| DA | kxi_da_requests |
counter | Count of requests received in interval |
| DA | kxi_da_failed_requests |
counter | Count of failed requests received in interval |
| DA | kxi_da_request_time |
histogram | Duration of requests in milliseconds received in interval. Buckets can be set with KXI_REQUEST_METRIC_BUCKETS environment var. Default "50 100 500 1000 2000 10000" |
Storage Metrics
When integrating with the monitoring sidecar, the following metrics will be available.
| component | type | name | description |
|---|---|---|---|
| SM | gauge | kxi_sm_clients |
Currently connected clients |
| SM | gauge | kxi_sm_stream_records |
Number of records read from RT stream |
| SM | gauge | kxi_sm_msgs |
Number of messages read from RT stream |
| SM | gauge | kxi_sm_eoi_requests_pending |
EOI requests awaiting completion |
| SM | gauge | kxi_sm_eod_requests_pending |
EOD requests awaiting completion |
| SM | counter | kxi_sm_eoi_count |
Number of completed End of Interval runs |
| SM | counter | kxi_sm_eod_count |
Number of completed End of Day runs |
| SM | gauge | kxi_sm_eoi_duration_seconds |
Duration of the most recent End of Interval |
| SM | gauge | kxi_sm_eod_duration_seconds |
Duration of the most recent End of Day |
| SM | gauge | kxi_sm_eoi_stream_pos |
Current RT stream position |
| SM | gauge | kxi_sm_eoi_records |
Number of records written during EOI |
| SM | gauge | kxi_sm_hdb_date_records |
Number of total records in latest HDB partition |
| SM | gauge | kxi_sm_hdb_size |
Size of HDB (in MB) |
| SM | gauge | kxi_sm_hdb_partitions |
Number of partitions in HDB |
| SM | gauge | kxi_sm_hdb_stream_pos |
Stream position at the end of the latest HDB partition |