Query scaling

For large volumes of queries or for complex, long-running queries, it may be necessary to increase the amount of resources servicing queries to maintain desired performance. There are different options available to tune, depending on the profile and scale of the queries. Adding more threads can be used to speed up individual queries. Adding more replicas can increase parallelism when there is a high number of concurrent requests. If the number of replicas becomes very high, scaling your routing layer should be considered. Adding replicas is also a good practice for increasing query resiliency by adding redundant processes.

Threads

Adding additional threads to a DAP allows for query filtering and aggregation to be performed in parallel within a single process. DAP process make use of additional threads when they are available.

DAPs are kdb+ processes which are configured to have secondary threads using the -s flag provided to the process.

DockerKubernetes

In a Docker compose deployment of a DAP, add the -s flag to the command field to indicate the number of secondary threads. For best results, this number should not exceed the number of available cores.

services:
  db:
    image: ${kxi_da}
    command: ["-s", "2"]

In a Kubernetes deployment, add the -s flag to the DAP container arguments. For best results, this number should not exceed the number of allocated cores. For example, if your CPU resource request is 2, the number of threads should match.

apiVersion: v1
kind: Pod
spec:
  containers:
  - name: dap
    image: ${kxi_da}
    args: ["-s", "2"]"
    resources:
      requests:
        cpu: 2

Replicas

Data Access Processes are individual kdb+ processes, which, while fast in executing queries, only have a single main thread. To increase query throughput and the number of concurrent queries the system can manage, adding additional DAP replicas will give the gateway a greater number of targets with which to serve the queries in parallel. The gateway makes use of all replicas available when load-balancing query work by default.

DockerKubernetes

In a Docker compose deployment, add replicas indicate the number of DAP instances.

services:
  db:
    image: ${kxi_da}
    deploy:
      replicas: 2

In a Kubernetes deployment, consider using a ReplicaSet and setting the number of replicas to indicate the number of DAP instances. For best results, consider adding an Anti-affinity rule so that the replicas are distributed across independent nodes.

apiVersion: apps/v1
kind: ReplicaSet
metadata:
spec:
  # modify replicas according to your case
  replicas: 3
  template:
    spec:
      containers:
      - name: da
        image: ${kxi_da}

Routing

As the number of DAP replicas increases, it will become necessary to increase the number of Resource Coordinators and Service Gateways that are servicing queries.

Kubernetes only

Automatic RC discovery is available using Kubernetes label discovery. This process must be done manually in Docker.

RCs connect to each other so they can enlist each other for help when the RC receiving the request does not contain the required DAPs to be able to complete the request on its own. RCs can only connect to each other using Kubernetes labels. The RC pods require Kubernetes RBAC permissions for the "get" and "list" verbs of the "pods" resource. The following is a set of example Kubernetes configuration snippets to enable label based discovery.

PodService AccountRoleRole Binding

This is an example of a Kubernetes ReplicaSet for an RC container.

Selector Label

The required label for discovery is app.kubernetes.io/name being set to resource-coordinator. This is required for RCs to find each other.

replica-set.yaml

apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: insights-resource-coordinator
  labels:
    app.kubernetes.io/name: resource-coordinator
spec:
  serviceAccountName: insights-resource-coordinator-service-account
  replicas: 3
  template:
    spec:
      containers:
        - name: resource-coordinator
          ports:
            - containerPort: 5050
              protocol: TCP

Container order

The RC container must be the last container in the RC pod. Discovery relies on this for determining which container to address within the pod.

This is an example Kubernetes ServiceAccount for mapping the required role properties to the deployed workload.

service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: insights-resource-coordinator-service-account

This is an example Kubernetes Role which allows the RC to list deployed services and discover other RCs.

role.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
    name: insights-resource-coordinator-role
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

This is an example Kubernetes RoleBinding that associates the role with the deployed RC workload.

Role Binding

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: insights-resource-coordinator-role-binding
subjects:
- kind: ServiceAccount
  name: insights-resource-coordinator-service-account
  apiGroup: ""
roleRef:
  kind: Role
  name: insights-resource-coordinator-role
  apiGroup: ""

To take advantage of multiple RCs, DAPs should be configured to be associated with a specific RC. The ordinal of the RC replica can be used for associating DAPs with RCs. RCs will balance the requests across available DAPs. DAP configuration needs to be configured with rcEndpoints in an assembly file or KXI_SG_RC_ADDR as an environment variable to point to the correct DAP instance.