Routing configuration
Query routing is managed by two primary components, the Service Gateway, and the Resource Coordinator. Queries are broken into partials and distributed across query instances and collected into a final result in the Aggregator.
Unless otherwise specified, the arrows in the diagram represent asynchronous q IPC communication between processes. Query flow is as follows.
arrow | description |
---|---|
1 |
Client makes API request to a Service Gateway replica. This can be synchronous or asynchronous. |
2 |
The Service Gateway forwards the request to the Resource Coordinator. |
3 |
The Resource Coordinator sends partial requests to each DAP relevant to the query based on purview. |
4 |
DAPs forward their responses to a single Aggregator for aggregation. |
5 |
The Aggregator sends the response to the same Service Gateway the client connected to. |
6 |
The Service Gateway sends response back to client. |
These components route queries across one or more assemblies and are installed as part of a base deployment. These components are generally configured using environment variables or a values file if using a Kubernetes based deployment. In both environment, environment variables can be used to change configuration.
Environment Variables
Configuration can be supplied to the Service Gateway or the Resource Coordinator using environment variables. Environment variables are configured differently depending on the method of deployment. In all cases, the variables are always string values.
In Docker, environment variables are supplied under an environment
key for the target service as a list of key-value pairs.
services:
sg:
environment:
- KXI_NAME=sg
In a Kubernetes deployment, environment variables are passed to components using a values file. For the Service Gateway, these values are supplied under an sg-gateway
key. For the Resource Coordinator, these are supplied under a resource-coordaintor
key.
kdb Insights Enterprise
When running in an enterprise deployment, values are nested under either a service-gateway
or qe-gateway
key in the values file. See enterprise deployment configuration for more details.
service-gateway:
sg-gateway:
env:
KXI_SG_TIMEOUT: "50000"
resource-coordinator:
env:
KXI_SG_TIMEOUT: "50000"
Service Gateway
name | description |
---|---|
GATEWAY_QIPC_PORT |
Gateway port for QIPC traffic. |
GATEWAY_QIPC_EXT_PORT |
Gateway port for external QIPC traffic. |
GATEWAY_HTTP_PORT |
Gateway port for HTTP traffic. |
SM_CONTAINER_NAMES |
Comma separated list of SM containers to connect to when using Kubernetes-based discovery. You should not need to touch this unless you've customized your SM container names. (default:"sm" ). |
DISCOVERY_PROXY |
Discovery proxy address (not required if not using discovery). |
KXI_SG_RC_ADDR |
host:port of RC process to connect to. |
KXI_SG_TIMEOUT |
Default request timeout in milliseconds (default: 60000 ). |
KXI_SG_TIMEOUT_MARGIN |
Controls the amount of time to wait for the RC to complete a request in milliseconds. The service gateway waits this many milliseconds after a request has timeout out for the RC to issue a timeout request. (default: 30000 ) |
KXI_RC_SERVICE_NAME |
The key that the Service Gateway looks for to find Resource Coordinators within the discovery registry. (default: KXI-SG-RC ). |
KXI_RC_LABEL_SELECTOR |
Allow the gateway to find Resource Coordinators by label selector. Required for Kubernetes-based discovery. |
KXI_DA_LABEL_SELECTOR |
Allow the gateway to find data-access pods by label selector. |
KXI_SM_LABEL_SELECTOR |
Allow the gateway to find storage-manager pods by label selector. |
KXI_EXCLUDE_CONTAINER_NAMES |
Comma separated container sub-strings to avoid connecting to when using Kubernetes-based discovery. Defaults to istio,sidecar . This setting is overridden by KXI_RC_CONTAINER_NAME or KXI_DA_CONTAINER_NAME . |
KXI_RC_CONTAINER_NAME |
Comma separated Resource Coordinator exact container names to connect to when using Kubernetes-based discovery. Overrides KXI_EXCLUDE_CONTAINER_NAMES . |
KXI_DA_CONTAINER_NAME |
Comma separated Data Access container exact container names to connect to when using Kubernetes-based discovery. Overrides KXI_EXCLUDE_CONTAINER_NAMES . |
KXI_SM_CONTAINER_NAME |
Specifies the Storage Manager container names to connect when using Kubernetes-based discovery. If unspecified, we use the list of values from SM_CONTAINER_NAMES . |
KXI_SG_BUFFER_INITIAL |
Initial size in bytes to allocate per connected aggregator (default: 5MB as 5000000 ). |
KXI_SG_BUFFER_RETAIN |
Maximum size in bytes to hold on to per connected aggregator (default: 5MB as 5000000 ). |
KXI_SG_USE_SSL |
Decide if GATEWAY_QIPC_EXT_PORT should use TLS. Default 1. |
KXI_AUTH_DISABLED |
Set KXI_AUTH_DISABLED=0 to use the gateway with Keycloak. |
KXI_OIDC_JSON_PATH |
OIDC JSON used for mapping Keycloak authentication endpoints. |
KXI_SCOPE_AFFINITY |
Scope affinity. Either "hard" or "soft", see scope (default: "hard"). |
KXI_ASSEMBLY_KEY |
Specifies the Kubernetes label key that identifies a pod's assembly (default: "insights.kx.com/app" ). |
USE_SP_WATCHER |
(Kubernetes only) Set to 1 if the GW should configure an SP watcher to automatically discover SP workers for web sockets. |
SP_SELECTOR_OVERRIDE |
If USE_SP_WATCHER=1 , this sets the label selectors the GW will use to find SP workers: default is sp.kx.com/role=worker,sp.kx.com/pipeline=%s . %s will be replaced by the pipeline instance ID. |
SP_SERVICE_PORT_NAME |
If USE_SP_WATCHER=1 , sets the port name to expect to look underneath to get the container port. |
SP_STREAM_HOST |
If not using an SP watcher, hostname for the SP worker |
SP_STREAM_PORT |
If not using an SP watcher, port for the SP worker. |
Streaming message buffers
For streaming messages back to clients, the gateway will stream up to KXI_SG_BUFFER_INITIAL
per write operation.
Total payloads can be any size, and this setting controls the chunk sizes. The default is based empherically on a "moderate" network speed of 4Mb/s.
For configurations with higher performing network speeds, this setting should be increased.
Set KXI_SG_STREAM_THRESHOLD
on the resource-coordinator
and aggregator
to control when they will decide to use streaming. Also defaults to 5MB in bytes.
For non-streaming messages, messages can be up to 2GB, and KXI_SG_BUFFER_RETAIN
is how much heap space in bytes maximum will be shrunk back down after a message is sent.
This allows tuning the application so "average" messages do not pay an overhead of memory re-allocation.
KXI_SG_BUFFER_INITIAL
is the how much space is reserved without any messages having come across, and is the minimum size in bytes that will be retained.
HTTP request size limit
The maximum size of an HTTP request to the gateway is 10MB.
Resource Coordinator
name | description |
---|---|
KXI_NAME |
Process name. |
KXI_PORT |
Port. |
KXI_GC_FREQ |
Frequency in milliseconds to run garbage collect in a timer (default: 600000 , set to 0 to disable). |
KXI_SG_TIMEOUT |
Default request timeout in milliseconds (default: 60000 ). |
KXI_DISC |
Multi-RC discovery mode: "kubernetes" or "" . |
KXI_RC_LABEL_SELECTOR |
Label selector to discover other RCs via Kubernetes-based discovery (default: app.kubernetes.io/name=resource-coordinator ). |
KXI_RC_CONTAINER_NAME |
RC container name for Kubernetes-based discovery (default: resource-coordinator ). |
KXI_SG_CONN_TIMEOUT |
Timeout on connection open. |
KXI_SG_MAX_RETRY |
Maximum number of retries the RC is willing to do for retry-able errors. |
KXI_ALLOWED_SBX_APIS |
Comma-delimited list of sandbox APIs to allow in non-sandbox RCs (e.g. .kxi.sql,.kxi.qsql ). |
KXI_SG_REQ_DEL_FREQ |
Time in milliseconds to run the request table delete routine (default: 10000 ). |
KXI_SG_QUEUE_DEL_FREQ |
Time in milliseconds to run the queue table delete routine (default: 10000 ). |
KXI_SAPI_HB_FREQ |
Time in milliseconds to run the heartbeat to connected processes (default: 30000 ). |
KXI_SAPI_HB_TOL |
Number of heartbeat intervals a process can miss before being disconnected (default: 2 ). |
KXI_SG_MULTI_RC_TO_MAX_WAIT |
On a timeout of a multi-RC request, number of 10 second iterations to wait for timeout details from all RCs (default: 1). |
KXI_ENABLE_FLUSH |
Set to true to enable async flush on messages from RC to DA/Agg and from Agg to GW (default false ). |
KXI_SG_STREAM_THRESHOLD |
Threshold in bytes for the RC to use streaming APIs of the Service Gateway (default: 5MB as 5000000 ). |
ALLOW_OLD_LABEL_STYLE |
Allows queries and APIs to continue to use the old label style prior to 1.5. See upgrading to 1.5 for details. (default: "true") |
KXI_RC_STS_SIZE |
Number of RCs in the stateful set when using Kubernetes-based discovery with DAPs and Aggs. See ordinal connections. |
KXI_ASSEMBLY_NAME |
Set in order for the RC to self-identify as an assembly RC. See routing. |
KXI_WARN_FREQ |
Frequency, in milliseconds, at which the RC issues a warning message if no DAPs or no aggregators are connected (default: 60000 , i.e. 1 minute). |
KXI_WARN_THRESHOLD |
If no DAPs or aggregators are connected, the RC begins generating error messages after this many warnings (default: 10). |