Skip to content

Pod autoscaling

The Horizontal Pod Autoscaler (HPA) automatically scales Kubernetes workloads, based on configured metrics, such as CPU or memory

Horizontal Pod Autoscaler

Horizontal pod autoscaling

Horizontal pod autoscaling is not the same as

The charts currently support an HPA configuration for the realtime DBs. The charts can be configured with thresholds, which, when exceeded, will cause the Kubernetes cluster to scale up the number of replicas in the RDBs.

To use HPA, you must also configure resource requests in both the main component container, and the metrics sidecar, if enabled. See the container resources for details on how these are configured.

Example HPA configuration in charts/kx/examples/gcp-hpa.yaml

Once deployed, you can describe the HPA resource:

$ kubectl describe hpa kx-rdb
Name:                                                  kx-rdb
Namespace:                                             default
Labels:                                                app.kubernetes.io/instance=kx
                                                       app.kubernetes.io/managed-by=Helm
                                                       app.kubernetes.io/name=rdb
                                                       app.kubernetes.io/version=4
                                                       helm.sh/chart=rdb-0.1.2
Annotations:                                           meta.helm.sh/release-name: kx
                                                       meta.helm.sh/release-namespace: acmck
CreationTimestamp:                                     Thu, 10 Dec 2020 17:10:48 +0000
Reference:                                             StatefulSet/kx-rdb
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  60% (580m) / 80%
Min replicas:                                          1
Max replicas:                                          5
StatefulSet pods:                                      1 current / 1 desired
Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  True    TooFewReplicas    the desired replica count is less than the minimum replica count

In the above example, we see we have not yet exceeded a threshold of 80% average CPU utilization, so the HPA has not increased the replica count.

If we then generate additional load on the RDB (simulating queries), and describe the HPA resource again, we see the threshold has been exceeded, and the HPA automatically creates a new pod kx-rdb-1.

$ kubectl describe hpa kx-rdb
Name:                                                  kx-rdb
Namespace:                                             acmck
Labels:                                                app.kubernetes.io/instance=kx
                                                       app.kubernetes.io/managed-by=Helm
                                                       app.kubernetes.io/name=rdb
                                                       app.kubernetes.io/version=4
                                                       helm.sh/chart=rdb-0.1.2
Annotations:                                           meta.helm.sh/release-name: kx
                                                       meta.helm.sh/release-namespace: acmck
CreationTimestamp:                                     Thu, 10 Dec 2020 17:10:48 +0000
Reference:                                             StatefulSet/kx-rdb
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  89% (943m) / 80%
Min replicas:                                          1
Max replicas:                                          5
StatefulSet pods:                                      1 current / 2 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    SucceededRescale    the HPA controller was able to update the target scale to 2
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age               From                       Message
  ----    ------             ----              ----                       -------
  Normal  SuccessfulRescale  2s (x2 over 16h)  horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target

The RDB will then replay its tickerplant logs, currently available via the shared volume.

Notice the HPA will also ‘scale down’ the replica set, back to the configured min replicas once the threshold drops below limit.