Skip to content

Routing configuration

Query routing is managed by two primary components, the Service Gateway, and the Resource Coordinator. Queries are broken into partials and distributed across query instances and collected into a final result in the Aggregator

process diagram

Unless otherwise specified, the arrows in the diagram represent asynchronous q IPC communication between processes. Query flow is as follows.

arrow description
1 Client makes API request to a Service Gateway replica. This can be sync or asynchronous.
2 The Service Gateway forwards the request to the Resource Coordinator.
3 The Resource Coordinator sends partial requests to each DAP relevant to the query based on purview.
4 DAPs forward their responses to a single Aggregator for aggregation.
5 The Aggregator sends the response to the same Service Gateway the client connected to.
6 The Service Gateway sends response back to client.

These components route queries across one or more assemblies and are installed as part of a base deployment. These components are generally configured using using environment variables or a values file if using a Kubernetes based deployment. In both environment, environment variables can be used to change configuration.

Environment Variables

Configuration can be supplied to the Service Gateway or the Resource Coordinator using environment variables. Environment variables are configured differently depending on the method of deployment. In all cases, the variables are always string values.

In Docker, environment variables are supplied using under an environment key for the target service as a list of key-value pairs.

      - KXI_NAME=sg

In a Kubernetes deployment, environment variables are passed to compnents using a values file. For the Service Gateway, these values are supplied under an sg-gateway key. For the Resource Coordinator, these are supplied under a resource-coordaintor key.

kdb Insights Enterprise

When running in an enterprise deployment, values are nested under either a service-gateway or qe-gateway key in the values file. See enterprise deployment configuration for more details.

      KXI_SG_TIMEOUT: "50000"
      KXI_SG_TIMEOUT: "50000"

Service Gateway

name description
GATEWAY_QIPC_PORT Gateway port for QIPC traffic.
GATEWAY_HTTP_PORT Gateway port for HTTP traffic.
SM_CONTAINER_NAMES Comma separated list of SM container names the GW should connect to. You should not need to touch this unless you've customized your SM container names. (default:"sm").
DISCOVERY_PROXY Discovery proxy address (not required if not using discovery).
KXI_SG_RC_ADDR host:port of RC process to connect to.
KXI_SG_TIMEOUT Default request timeout in milliseconds (default: 60000).
KXI_SG_TIMEOUT_MARGIN Controls the amount of time to wait for the RC to complete a request in milliseconds. The service gateway waits this many milliseconds after a request has timeout out for the RC to issue a timeout request. (default: 30000)
KXI_RC_SERVICE_NAME The key that the gateway looks for to find Resource Coordinators within the discovery registry. (default: KXI-SG-RC).
KXI_SG_RC_LABEL_SELECTOR Allow the gateway to find Resource Coordinators by label selector. Required for Kubernetes-based discovery.
KXI_SG_DA_LABEL_SELECTOR Allow the gateway to find DAPs pods by label selector.
KXI_SG_SM_LABEL_SELECTOR Allow the gateway to find Storage Manager pods by label selector.
KXI_AUTH_DISABLED Set KXI_AUTH_DISABLED=0 to use the gateway with Keycloak.
KXI_OIDC_JSON_PATH OIDC JSON used for mapping Keycloak authentication endpoints.

Resource Coordinator

name description
KXI_NAME Process name.
KXI_SG_RC_ADDR host:port of RC process to connect to.
KXI_GC_FREQ Frequency in milliseconds to run garbage collect in a timer (default: 600000, set to 0 to disable).
KXI_SG_TIMEOUT Default request timeout in milliseconds (default: 60000).
KXI_SG_DISC Multi-RC discovery mode.
KXI_SG_CONN_TIMEOUT Timeout on connection open.
KXI_SG_MAX_RETRY Maximum number of retries the RC is willing to do for retry-able errors.
KXI_ALLOWED_SBX_APIS Comma-delimited list of sandbox APIs to allow in non-sandbox RCs (e.g. .kxi.sql,.kxi.qsql).
KXI_SG_REQ_DEL_FREQ Time in milliseconds to run the request table delete routine (default: 10000).
KXI_SG_QUEUE_DEL_FREQ Time in milliseconds to run the queue table delete routine (default: 10000).
KXI_SAPI_HB_FREQ Time in milliseconds to run the heartbeat to connected processes (default: 30000).
KXI_SAPI_HB_TOL Number of heartbeat intervals a process can miss before being disconnected (default: 2).
KXI_SG_MULTI_RC_TO_MAX_WAIT On a timeout of a multi-RC request, number of 10 second iterations to wait for timeout details from all RCs (default: 1).
KXI_ENABLE_FLUSH Set to true to enable async flush on messages from RC to DA/Agg and from Agg to GW (default false).