Device Scheduling - GPU Share With HAMi

Introduction

GPU is an indispensable device for today’s large AI model training and inference, and can provide powerful computing power support for AI applications. An NVIDIA A100 80 GB GPU can provide up to 249 times the inference performance on the BERT-LARGE task compared to a CPU. However, behind such powerful computing power is an expensive price. An NVIDIA A100 40GB model GPU chip is priced at around $13,000, while an A100 80GB model GPU chip is priced at around $15,000. We have observed that some inference tasks often use less than a full GPU, but only use, for example, 50% of the computing power or gpu memory. Therefore, sharing a GPU with multiple Pods can significantly improve GPU resource utilization.

HAMi is a Cloud Native Computing Foundation sandbox project which provides the ability to share Heterogeneous AI devices among tasks. Koordinator takes advantage of HAMi’s GPU isolation capabilities on the node side to provide an end-to-end GPU sharing solution. It includes two key components:

• HAMi-core is the in-container gpu resource controller, it operates by Hijacking the API-call between CUDA-Runtime(libcudart.so) and CUDA-Driver(libcuda.so). The output from building HAMi-core is libvgpu.so.
• vGPUmonitor is a vGPU monitoring service provided by HAMi. HAMi-core writes vGPU monitoring data to the /usr/local/vgpu/containers directory, and vGPUmonitor parses the data in this directory to export Prometheus metrics.

Setup

Prerequisite

• Kubernetes >= 1.18
• Koordinator >= 1.7

Installation

Please make sure Koordinator components are correctly installed in your cluster. If not, please refer to Installation.

Runtime Requirements

The scheduled GPU devices are bound to the container requires support from the runtime environment. Currently, there are two solutions to achieve this:

Configurations

1. DeviceScheduling is Enabled by default. You can use it without any modification on the koord-scheduler config.
2. The GPUEnvInject FeatureFG need to be enabled in koordlet. Add -feature-gates=GPUEnvInject=true to the args parameters.
3. 从 Koordinator v1.7 开始，koord-manager 的 EnableSyncGPUSharedResource 默认设置为 false。在 GPU 共享场景下，需要将该值设置为 true。可以通过在 koord-manager 的 args 参数中添加 -feature-gates=EnableSyncGPUSharedResource=true 来启用。

Install hami-daemon with helms

hami-daemon can be simply installed by helm v3.5+, which is a simple command-line tool, and you can get it from here.

1. # Firstly add koordinator charts repository if you haven't do this.
2. $ helm repo add koordinator-sh https://koordinator-sh.github.io/charts/

4. # [Optional]
5. $ helm repo update

7. # Install the latest version.
8. $ helm install hami-daemon koordinator-sh/hami-daemon --version 0.1.0

Options: chart parameters

Note that installing this chart directly means it will use the default template values for hami-daemon.

You may have to set your specific configurations if it is deployed into a production cluster.

The following table lists the configurable parameters of the chart and their default values.

Specify each parameter using the --set key=value[,key=value] argument to helm install or helm upgrade.

Use GPU Share With HAMi

1. Create a Pod to apply for a GPU card with 50% computing power and 50% gpu memory, and specify the need for hami-core isolation through the Pod Label koordinator.sh/gpu-isolation-provider=HAMi-core. Starting from version 1.7.0, the scheduler will enforce a mandatory check: for any Pod requiring GPU isolation capabilities, its target node must have the hami-core component installed. Please ensure that all nodes in the cluster with hami-core installed are labeled with koordinator.sh/gpu-isolation-provider=HAMi-core.

1. apiVersion: v1
2. kind: Pod
3. metadata:
4. name: pod-example
5. namespace: default
6. labels:
7. koordinator.sh/gpu-isolation-provider: HAMi-core
8. spec:
9. schedulerName: koord-scheduler
10. containers:
11. - command:
12. - sleep
13. - 365d
14. image: nvidia/cuda:11.8.0-base-ubuntu22.04
15. imagePullPolicy: IfNotPresent
16. name: curlimage
17. resources:
18. limits:
19. cpu: 4
20. memory: 2Gi
21. koordinator.sh/gpu.shared: 1
22. koordinator.sh/gpu-core: 50
23. koordinator.sh/gpu-memory-ratio: 50
24. requests:
25. cpu: 4
26. memory: 2Gi
27. koordinator.sh/gpu.shared: 1
28. koordinator.sh/gpu-core: 50
29. koordinator.sh/gpu-memory-ratio: 50
30. terminationMessagePath: /dev/termination-log
31. terminationMessagePolicy: File
32. restartPolicy: Always

1. $ kubectl get pod -n default pod-example -o yaml

1. apiVersion: v1
2. kind: Pod
3. metadata:
4. annotations:
5. scheduling.koordinator.sh/device-allocated: '{"gpu":[{"minor":1,"resources":{"koordinator.sh/gpu-core":"50","koordinator.sh/gpu-memory":"11520Mi","koordinator.sh/gpu-memory-ratio":"50"}}]}'
6. name: pod-example
7. namespace: default
8. labels:
9. koordinator.sh/gpu-isolation-provider: HAMi-core
10. ...

You can find the concrete device allocate result through annotation scheduling.koordinator.sh/device-allocated.

1. Enter the pod and you can see that the upper limit of the gpu memory seen by the program inside the pod is the value shown in the allocation result above.

1. $ kubectl exec -it -n default pod-example bash

View vGPU monitoring

If the vgpuMonitor is enabled during the installation of hami-daemon and a default Prometheus is present in the cluster, then vGPU-related metrics can be queried from Prometheus.

For example, accessing port 9394 (curl {pod-ip}:9394/metrics) of a hami-daemon pod allows you to retrieve relevant metrics, as shown below:

FAQ

• Drivers above 570 need v2.6.0 to support
• HAMi-Core 和 MPS 冲突