Skip to content

KX Insights Overprovisioning

Overview

Scaling nodes in Kubernetes clusters can be slow due to the time taken for new nodes to be provisioned. This time taken can include waiting on rook-ceph to become available along with mounting volumes. A cluster may need to scale if all nodes in the cluster are running at their full capacity, or if there is some burst processing or HA needed.

To reduce the time needed to wait for another node to be provisioned, we can overprovision the cluster by scheduling low priority pods with resource requests that cause the cluster to scale. For example, sizing the resource request of the overprovisioning pod to match size of a node, will scale an extra node.

The KX Overprovisioning chart is responsible for initializing lower-priority pods which should request the majority resources of a node on a cluster. The pods are used for the reservation of resources.

If an over request of resources is triggered, the overprovisioning pods would be evicted to free up resources for pending pods. They would then trigger a new node to be scheduled to started.

Pod Priority

Pod Priority is a Kubernetes features that allow you to assign priorities to pods. Priority indicates the importance of a Pod relative to other Pods. When a cluster is low on memory/cpu resources, lower-priority pods are removed/evicted by the scheduler. This is done in order to make space for higher-priority pods waiting to be scheduled.

Priority class value

The priority class value used by the chart should be set to a low value. The default is deliberately set to a large negative number to ensure it will be evicted by other pods. Ensure that this value is lower than the other pods in your application.

# Specify details of the priority class create
priorityclass:
  name: "kxi-low-priority"
  value: -200000

ReplicaCount and Resources

To use the chart, you must provide the following resource values. These should align with the maximum resources of a node within their cluster.

variable type default description
replicaCount int 1 Number of nodes to provision via overprovisioning pod replicas
resources.requests.cpu string Requested CPU for the Overprovisioning
resources.requests.memory string Requested memory for the Overprovisioning

The example yaml below, defines how a user would overprovision 2 additional nodes.The overprovisioning pods are requesting 8 CPUs and 64GB of memory. This is assuming that resources defined on each of the nodes are slightly larger, therefor the overprovisioning pods are requesting the majority resources.

replicaCount: 2
resources:
  requests:
    cpu: 8
    memory: 64Gi

Prerequisites

In order to run the chart you will need access to the KX Insights Nexus repository and an associated image pull secret for your cluster. If you've already installed Insights, you can re-use the same secret.

Confirm the repo has been added

$ helm repo ls
NAME                    URL
kx-insights             https://nexus.dl.kx.com/repository/kx-insights-charts/

Confirm the image pull secret exists

$ kubectl get secrets
NAME                                            TYPE                                  DATA   AGE
kxi-nexus-pull-secret                           kubernetes.io/dockerconfigjson        1      7d20h

Otherwise the easiest way to setup the prerequisites is to use the CLI. It is advised to follow and use the kxi tools to setup. The below command will setup the necessary secrets needed to install the chart.

kxi install setup

If not using the CLI, and have not already added the repo and secrets. They can be manually installed as follows

Add the KX Helm repo

helm repo add --username <username> --password <password> kx-insights https://nexus.dl.kx.com/repository/kx-insights-charts

Setup the image pull secret

An image pull secret is required in order to pull images from a private Docker registry. Using your credentials for the Insights Nexus registry, you can create a secret for pulling these images.

kubectl create secret docker-registry kxi-nexus-pull-secret \
    --docker-username=<username> \
    --docker-password=<password> \
    --docker-server=registry.dl.kx.com

Create values file

global:
  imagePullSecrets:
    - name: kxi-nexus-pull-secret

replicaCount: 1
resources:
  requests:
    cpu: 8
    memory: 64Gi

Installing the chart

helm allows the user to view available versions of a chart with the command

helm search repo kx-insights/kxi-overprovisioning --versions

The chart can then be installed with the command below and using the values.yaml with the provided values above.

helm install kxi-overprovisioning kx-insights/kxi-overprovisioning  --version=<version> -f values.yaml