Configure GPU autoscaling for Kubernetes

GPU autoscaling in a Kubernetes cluster combines four components: NVIDIA GPU Operator for resource management, KEDA for event-driven pod scaling, Prometheus for metrics collection, and Gcore Cluster Autoscaler for node provisioning. When GPU utilization exceeds the configured threshold, KEDA creates additional pods — Cluster Autoscaler then provisions new GPU Bare Metal nodes to meet the demand.

Gcore Managed Kubernetes includes native autoscaling based on CPU and memory only. To scale workloads based on GPU utilization, KEDA must be manually installed and configured.

Prepare the cluster

Create a Kubernetes cluster with a GPU Bare Metal worker pool and verify that nodes are ready before installing GPU software.

Step 1. Create a Kubernetes cluster

Add a new Managed Kubernetes cluster with a Bare Metal worker pool:

1. In the Gcore Customer Portal, navigate to Cloud > Kubernetes.

2. Click Create Cluster.

3. Select a region — the location of the data center where the cluster will be deployed.

4. Select the Kubernetes cluster version: v1.28.6 or higher.

5. Configure Bare Metal pools:

• Pool name: Enter a unique and descriptive name for the resource pool.
• Minimum nodes and Maximum nodes: Specify how many nodes can be allocated to the pool during traffic fluctuations. The maximum number of nodes must be greater than the minimum to allow the cluster to scale up in response to increased demand.
• Type: Select GPU Bare metal instances.
• Infrastructure: Select the needed GPU-optimized flavor. At least one node with GPU support must be added to the worker pool.

6. (Optional) Add labels to include additional node metadata. Add taints to define which Kubernetes pods can be scheduled on these nodes.

7. Disable the Autohealing nodes toggle.

8. (Optional) Enable the Public IPv4 address option to assign public IPv4 addresses to cluster nodes.

9. To add more pools, click Add pool and configure each according to the previous steps.

10. Complete the remaining cluster settings starting from step 5 in the cluster creation guide.

Step 2. Verify the Kubernetes configuration

1. In the Customer Portal, navigate to Cloud > Kubernetes.

2. Find the cluster created in the previous step and click its name to open it.

3. Check the pool status: it should be Running. If the status shows Scaling up, wait until all resources are allocated and the cluster is ready to use.

4. Download the .config file by clicking Kubernetes config in the top-right corner of the screen.

5. Export Kubernetes configuration locally:

export KUBECONFIG=/path/to/your/k8sConfig.yml

6. Verify that the node is ready:

kubectl get nodes

The expected output looks similar to this:

NAME                STATUS   ROLES    AGE     VERSION 
ed-b16-82-160-248   Ready    <none>   3h26m   v1.28.6  

Install dependencies

Install the GPU Operator to expose GPU resources, then KEDA and Prometheus to drive utilization-based scaling decisions.

Step 3. Install GPU Operator

Use Helm to install the GPU Operator — the official Helm docs cover installation for all platforms.

1. Add the NVIDIA Helm repository and update it:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia && helm repo update

2. Install the GPU Operator. Use version v23.9.1 for Kubernetes 1.28.x or v23.9.2 for Kubernetes 1.29.x:

helm install gpu-operator nvidia/gpu-operator --version v23.9.1 --wait \
    -n gpu-operator --create-namespace \
    --set driver.enabled=false \
    --set operator.defaultRuntime=crio \
    --set operator.logging.level=debug

Once installation completes, verify that GPUs are visible to Kubernetes before continuing.

Step 4. Verify GPU allocation

1. Run kubectl describe node <node-name> and check the Allocatable section. The nvidia.com/gpu value must match the GPU count specified in the Bare Metal flavor.

Allocatable:
  cpu:                192
  ephemeral-storage:  850152999143
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             2113338172Ki
  nvidia.com/gpu:     8
  pods:               110

2. Check that all GPU Operator pods have READY status:

kubectl get pods -n gpu-operator

The expected output looks similar to this:

NAME                                                         READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-f4knk                                  1/1     Running     0          3m9s
gpu-operator-f7ffcf7f8-sf8m7                                 1/1     Running     0          3m34s
gpu-operator-node-feature-discovery-gc-7cc7ccfff8-77g45      1/1     Running     0          3m34s
gpu-operator-node-feature-discovery-master-d8597d549-dnxz5   1/1     Running     0          3m34s
gpu-operator-node-feature-discovery-worker-rlf6p             1/1     Running     0          3m34s
nvidia-container-toolkit-daemonset-rl9p2                     1/1     Running     0          3m1s
nvidia-cuda-validator-78g72                                  0/1     Completed   0          2m32s
nvidia-dcgm-exporter-nv9bk                                   1/1     Running     0          3m9s
nvidia-device-plugin-daemonset-4l5fl                         1/1     Running     0          3m9s
nvidia-mig-manager-rqnsq                                     1/1     Running     0          26s
nvidia-operator-validator-n9rpz                              1/1     Running     0          3m9s

3. Run a test GPU application using the NVIDIA CUDA vectorAdd example. Deploy the example workload and confirm it completes successfully before continuing.

Step 5. Install KEDA

With GPU resources confirmed, install KEDA — it reads Prometheus metrics to determine when to create additional pods.

1. Add the KEDA Helm repository:

helm repo add kedacore https://kedacore.github.io/charts

2. Update the repository:

helm repo update

3. Install KEDA into its own namespace:

helm install keda kedacore/keda --namespace keda --create-namespace

Step 6. Install kube-prometheus-stack

KEDA relies on Prometheus metrics to trigger GPU-based scaling decisions. Install the kube-prometheus-stack to collect and expose those metrics.

1. Add the Prometheus Helm repository, update it, and save the default values:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm inspect values prometheus-community/kube-prometheus-stack > values.yaml

2. Create the values-overrides.yaml file with the following content:

serviceMonitorSelectorNilUsesHelmValues: false

additionalScrapeConfigs:
- job_name: gpu-metrics
  scrape_interval: 1s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - gpu-operator
  relabel_configs:
  - source_labels: [__meta_kubernetes_endpoints_name]
    action: drop
    regex: .*-node-feature-discovery-master
  - source_labels: [__meta_kubernetes_pod_node_name]
    action: replace
    target_label: kubernetes_node

3. Install kube-prometheus-stack:

helm install prometheus-community/kube-prometheus-stack \
   --create-namespace --namespace prometheus \
   --generate-name \
   -f values.yaml -f values-overrides.yaml

The values-overrides.yaml overrides the default Prometheus configuration to define a scrape job that collects DCGM metrics from the gpu-operator namespace.

Configure autoscaling

With metrics collection in place, create the workload and the ScaledObject that tells KEDA how to scale it.

Step 7. Deploy the workload and ScaledObject

Deploy the GPU workload and the KEDA ScaledObject that controls its scaling. Apply both manifests to the cluster.

Workload Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-workload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gpu
  template:
    metadata:
      labels:
        app: gpu
    spec:
      containers:
        - name: gpu-workload
          image: registry.example.com/gpu-app:latest
          command: ["/usr/bin/your-command"]
          args: ["--argument-1", "--argument-2"]
          resources:
            limits:
              nvidia.com/gpu: 8

ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: gpu-workload-scaledobject
spec:
  scaleTargetRef:
    name: gpu-workload
  minReplicaCount: 1
  maxReplicaCount: 16
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus-operated.prometheus:9090
      metricName: gpu-utilization
      threshold: "60"
      query: sum(avg_over_time(DCGM_FI_DEV_GPU_UTIL{pod=~"gpu-workload.*"}[30s]))

The DCGM_FI_DEV_GPU_UTIL metric measures real-time GPU core utilization and is collected by the nvidia-dcgm-exporter installed in Step 3. Any metric from the DCGM exporter can be used in the query field instead.

The {pod=~"gpu-workload.*"} selector in the query filters metrics by pod name prefix. Update gpu-workload to match the actual workload name.

Parameters

Replace the example values in both manifests:

Parameter	Description
gpu-workload	Unique name for the workload. Use the same value in metadata.name (Deployment), spec.containers[].name, and spec.scaleTargetRef.name (ScaledObject).
registry.example.com/gpu-app:latest	Container image URL.
8	Number of GPUs to allocate per container.
1 / 16	Minimum and maximum number of replicas.
60	GPU utilization percentage at which KEDA triggers scaling.
gpu-utilization	Unique identifier for the metrics series.

Verify scaling

Deploy the workload and confirm that KEDA, Cluster Autoscaler, and new GPU nodes respond correctly to GPU utilization changes.

Step 8. Verify autoscaling

The autoscaling sequence has three phases: KEDA monitors GPU utilization and scales pods via the HPA, Cluster Autoscaler provisions new nodes when GPU capacity runs out, then new nodes join the cluster.

1. Monitor HPA in real time:

kubectl get hpa -w

The expected output looks similar to this:

NAME                               REFERENCE                TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-gpu-workload-scaledobject Deployment/gpu-workload  <unknown>/60 (avg)   1         16        0          0s
keda-hpa-gpu-workload-scaledobject Deployment/gpu-workload  0/60 (avg)           1         16        1          16s
keda-hpa-gpu-workload-scaledobject Deployment/gpu-workload  23/60 (avg)          1         16        1          76s
....
keda-hpa-gpu-workload-scaledobject Deployment/gpu-workload  67750m/60 (avg)      1         16        8          10m

The system created eight pods to match the eight GPUs available on the server.

2. When pods exceed GPU capacity, Cluster Autoscaler requests new nodes. Run kubectl get events -w to observe provisioning events — node provisioning takes up to 20–25 minutes.

....
0s    Warning   FailedScheduling   pod/gpu-workload-...-dnzns   0/1 nodes are available: 1 Insufficient nvidia.com/gpu. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod..

....
kube-system   69s   Normal   ScaledUpGroup   configmap/cluster-autoscaler-status   Scale-up: setting group MachineDeployment/.../gpu-workload-machine-deployment size to 2 instead of 1 (max: 2)
kube-system   69s   Normal   ScaledUpGroup   configmap/cluster-autoscaler-status   Scale-up: group MachineDeployment/.../gpu-workload-machine-deployment size set to 2 instead of 1 (max: 2)

3. Once the new node is ready, check pod status:

kubectl get pods

The expected output looks similar to this:

NAME                                   READY   STATUS    RESTARTS   AGE
apiserver-bridge-j7jk6                 1/1     Running   0          141m
gpu-workload-9f866ff47-57zmg           1/1     Running   0          11m
gpu-workload-9f866ff47-5wxxb           1/1     Running   0          15m
gpu-workload-9f866ff47-8gkm8           1/1     Running   0          14m
gpu-workload-9f866ff47-9t4bv           1/1     Running   0          7m57s
gpu-workload-9f866ff47-dnzns           0/1     Pending   0          6m57s
gpu-workload-9f866ff47-fb5cs           0/1     Pending   0          6m57s
gpu-workload-9f866ff47-qthzr           1/1     Running   0          10m
gpu-workload-9f866ff47-rxm2c           1/1     Running   0          17m
gpu-workload-9f866ff47-tc8pq           1/1     Running   0          8m58s
gpu-workload-9f866ff47-zbt8d           1/1     Running   0          12m

Eight pods are running, and several have a Pending status while the new node initializes. After it is ready, the system scales up to the maxReplicaCount value.

After provisioning, it might take up to five minutes for NVIDIA GPU resources to become available in the cluster.

4. Check that the new node has joined the cluster:

kubectl get nodes

NAME                STATUS   ROLES    AGE    VERSION
ed-b16-78-161-160   Ready    <none>   166m   v1.28.6
ed-b16-78-161-169   Ready    <none>   12m    v1.28.6

5. Verify the final HPA status:

kubectl get hpa

The expected output looks similar to this:

NAME                               REFERENCE                TARGETS           MINPODS   MAXPODS   REPLICAS   AGE
keda-hpa-gpu-workload-scaledobject Deployment/gpu-workload  71463m/60 (avg)   1         16        16         41m

When the utilization metric crosses the configured threshold — 60% in the example above — KEDA creates new pods up to the maxReplicaCount limit. If all GPUs are in use and no free resources are available, Kubernetes keeps the pods in Pending state, and Cluster Autoscaler provisions new nodes. Once new nodes are ready and GPUs are initialized (which might take 5 to 10 minutes), the pending pods start running.