KEDA : Event-driven Autoscaling

On Kubernetes, you can automatically scale your Deployments using the Horizontal Pod Autoscaler (HPA).
HPA requires the presence of the metric-servers pod (previously called heapster) on your cluster.
HPAs are based on the CPU and memory consumption of the pods of your deployment in order to increase or decrease its number of replicas.

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: hello-world
  namespace: default
spec:
  maxReplicas: 10
  minReplicas: 1
  scaleTargetRef:
    apiVersion: extensions/v1beta1
    kind: Deployment
    name: hello-world
  targetCPUUtilizationPercentage: 60

ex. HorizontalPodAutoscaler object with CPU utilization target

Now suppose you are developing an application that is consuming messages from RabbitMQ and you notice that your application cannot cope with thousands of messages on a specific queue.
That would be great if your stateless application would automatically scale based on the number of messages.

KEDA has been specifically design for this purpose.

Installation

1/ Add helm repository

helm repo add kedacore https://kedacore.github.io/charts \
&& helm repo update

2/ Install Keda (helm v.3)

kubectl create namespace keda
helm install keda kedacore/keda --version 2.4 --namespace keda

Use case example : RabbitMQ

Once KEDA is installed, imagine you want to scale your consumer’s pods based on the number of messages in your queue.

1/ Custom Resources

KEDA relies on 2 custom resources:

• ScaledObject: The ScaledObject maps an event source to the deployment that you want to scale.
• TriggerAuthentication: If required, this resource contains the authentication configuration needed for monitoring the event source.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: rabbitmq-consumer
  namespace: default
spec:
  scaleTargetRef:
    name: rabbitmq-consumer
  pollingInterval: 5 # Optional. Default: 30 seconds
  cooldownPeriod: 30 # Optional. Default: 300 seconds
  minReplicaCount: 1  # Optional. Default: 0
  maxReplicaCount: 30 # Optional. Default: 100
  triggers:
    - type: rabbitmq
      metadata:
        queueName: hello
        queueLength: "5"
      authenticationRef:
        name: rabbitmq-consumer-trigger
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: rabbitmq-consumer-trigger
  namespace: default
spec:
  secretTargetRef:
    - parameter: host
      name: rabbitmq-consumer-secret
      key: RabbitMqHost

The ScaledObject and the deployment referenced in scaleTargetRef.name need to be in the same namespace.
In this example this will be our microservice responsable for consuming the messages (rabbitmq-consumer).

Note: When setting maxReplicaCount parameter make sure to take into account the scaling strategy at the cluster level.

2/ The consumer

apiVersion: v1
kind: Secret
metadata:
  name: rabbitmq-consumer-secret
data:
  RabbitMqHost: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx 
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rabbitmq-consumer
  namespace: default
  labels:
    app: rabbitmq-consumer
spec:
  selector:
    matchLabels:
      app: rabbitmq-consumer
  template:
    metadata:
      labels:
        app: rabbitmq-consumer
    spec:
      containers:
        - name: rabbitmq-consumer
          image: jeffhollan/rabbitmq-client:dev
          imagePullPolicy: Always
          command:
            - receive
          args:
            - "amqp://user:PASSWORD@rabbitmq.default.svc.cluster.local:5672"

3/ The publisher

apiVersion: batch/v1
kind: Job
metadata:
  name: rabbitmq-publish
spec:
  template:
    spec:
      containers:
      - name: rabbitmq-client
        image: jeffhollan/rabbitmq-client:dev
        imagePullPolicy: Always
        command: ["send",  "amqp://user:PASSWORD@rabbitmq.default.svc.cluster.local:5672", "300"]
      restartPolicy: Never
  backoffLimit: 4

This batch job will publish 300 messages in the hello queue.
KEDA will help the HPA and more consumer pods will get created till the queue is empty.
Once the job is done, the consumer will go back to his inital value(minReplicaCount).

kubectl get hpa
NAME                         REFERENCE                      TARGETS     MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-rabbitmq-consumer   Deployment/rabbitmq-consumer   0/5 (avg)   1         30        1          11m

Final thoughts

KEDA is really a nice addition to enhance the Kubernetes HPA.
It supports a variety of Scalers (Azure Service Bus, Apache Kafka, Redis Lists…).
It has also scaling support for Azure Functions running in Kubernetes.

References

• KEDA official documentation
• Sample Go RabbitMQ