Device Scheduling - Basics

We provide a fine-grained mechanism for managing GPUs and other devices such as RDMA and FPGA, defines a set of APIs to describe device information on nodes, including GPU, RDMA, and FPGA, and a new set of resource names to flexibly support users to apply at a finer granularity GPU resources. This mechanism is the basis for subsequent other GPU scheduling capabilities such as GPU Share, GPU Overcommitment, etc.

Introduction

GPU devices have very strong computing power, but are expensive. How to make better use of GPU equipment, give full play to the value of GPU and reduce costs is a problem that needs to be solved. In the existing GPU allocation mechanism of the K8s community, the GPU is allocated by the kubelet, and it is a complete device allocation. This method is simple and reliable, but similar to the CPU and memory, the GPU will also be wasted. Therefore, some users expect to use only a portion of the GPU’s resources and share the rest with other workloads to save costs. Moreover, GPU has particularities. For example, the NVLink and oversold scenarios supported by NVIDIA GPU mentioned below both require a central decision through the scheduler to obtain globally optimal allocation results.

Setup

Prerequisite

• Kubernetes >= 1.18
• Koordinator >= 0.71

Installation

Please make sure Koordinator components are correctly installed in your cluster. If not, please refer to Installation.

Runtime Requirements

The scheduled GPU devices are bound to the container requires support from the runtime environment. Currently, there are two solutions to achieve this:

Configurations

DeviceScheduling is Enabled by default. You can use it without any modification on the koord-scheduler config.

Use DeviceScheduling

Quick Start

1.check device crd:

1. $ kubectl get device host04 -o yaml

1. apiVersion: scheduling.koordinator.sh/v1alpha1
2. kind: Device
3. metadata:
4. creationTimestamp: "2022-10-08T09:26:42Z"
5. generation: 1
6. managedFields:
7. - apiVersion: scheduling.koordinator.sh/v1alpha1
8. fieldsType: FieldsV1
9. fieldsV1:
10. f:metadata:
11. f:ownerReferences: {}
12. f:spec:
13. .: {}
14. f:devices: {}
15. f:status: {}
16. manager: koordlet
17. operation: Update
18. time: "2022-10-08T09:26:42Z"
19. name: host04
20. ownerReferences:
21. - apiVersion: v1
22. blockOwnerDeletion: true
23. controller: true
24. kind: Node
25. name: host04
26. uid: 09c4f912-6026-467a-85d2-6b2147c9557e
27. resourceVersion: "39011943"
28. selfLink: /apis/scheduling.koordinator.sh/v1alpha1/devices/host04
29. uid: 5a498e1f-1357-4518-b74c-cab251d6c18c
30. spec:
31. devices:
32. - health: true
33. id: GPU-04cea5cd-966f-7116-1d58-1ac34421541b
34. minor: 0
35. resources:
36. kubernetes.io/gpu-core: "100"
37. kubernetes.io/gpu-memory: 16Gi
38. kubernetes.io/gpu-memory-ratio: "100"
39. type: gpu
40. - health: true
41. id: GPU-3680858f-1753-371e-3c1a-7d8127fc7113
42. minor: 1
43. resources:
44. kubernetes.io/gpu-core: "100"
45. kubernetes.io/gpu-memory: 16Gi
46. kubernetes.io/gpu-memory-ratio: "100"
47. type: gpu
48. status: {}

We can find this node has two gpu cards, we can find the detail info of each gpu card here.

2.check node allocatable resource:

1. $ kubectl get node host04 -o yaml

1. apiVersion: v1
2. kind: Node
3. metadata:
4. annotations:
5. flannel.alpha.coreos.com/backend-data: '{"VtepMAC":"5a:69:48:10:29:25"}'
6. creationTimestamp: "2022-08-29T09:12:55Z"
7. labels:
8. beta.kubernetes.io/os: linux
9. status:
10. addresses:
11. - address: 10.15.0.37
12. type: InternalIP
13. - address: host04
14. type: Hostname
15. allocatable:
16. cpu: "6"
17. ephemeral-storage: "200681483926"
18. kubernetes.io/gpu: "200"
19. kubernetes.io/gpu-core: "200"
20. kubernetes.io/gpu-memory: 32Gi
21. kubernetes.io/gpu-memory-ratio: "200"
22. memory: 59274552Ki
23. nvidia.com/gpu: "2"
24. pods: "220"
25. capacity:
26. cpu: "8"
27. kubernetes.io/gpu: "200"
28. kubernetes.io/gpu-core: "200"
29. kubernetes.io/gpu-memory: 32Gi
30. kubernetes.io/gpu-memory-ratio: "200"
31. memory: 61678904Ki
32. nvidia.com/gpu: "2"
33. pods: "220"

We can find the node allocatable resource has merged each gpu card resource.

3.apply pod:

1. apiVersion: v1
2. kind: Pod
3. metadata:
4. name: pod-example
5. namespace: default
6. spec:
7. schedulerName: koord-scheduler
8. containers:
9. - command:
10. - sleep
11. - 365d
12. image: busybox
13. imagePullPolicy: IfNotPresent
14. name: curlimage
15. resources:
16. limits:
17. cpu: 40m
18. memory: 40Mi
19. requests:
20. cpu: 40m
21. memory: 40Mi
22. kubernetes.io/gpu: "100"
23. terminationMessagePath: /dev/termination-log
24. terminationMessagePolicy: File
25. restartPolicy: Always

1. $ kubectl get pod -n default pod-example -o yaml

1. apiVersion: v1
2. kind: Pod
3. metadata:
4. annotations:
5. scheduling.koordinator.sh/device-allocated: '{"gpu":[{"minor":0,"resources":{"kubernetes.io/gpu-core":"100","kubernetes.io/gpu-memory":"12508288Ki","kubernetes.io/gpu-memory-ratio":"100"}}]}'
6. creationTimestamp: "2022-10-08T09:33:07Z"
7. name: pod-example
8. namespace: default
9. resourceVersion: "39015044"
10. selfLink: /api/v1/namespaces/xlf/pods/gpu-pod7
11. uid: 6bf1ac3c-0c9f-472a-8b86-de350bbfa795
12. spec:
13. containers:
14. - command:
15. - sleep
16. - 365d
17. image: busybox
18. imagePullPolicy: IfNotPresent
19. name: curlimage
20. resources:
21. limits:
22. cpu: "1"
23. kubernetes.io/gpu: "100"
24. memory: 256Mi
25. requests:
26. cpu: "1"
27. kubernetes.io/gpu: "100"
28. memory: 256Mi
29. status:
30. conditions:
31. ...
32. hostIP: 10.0.0.149
33. phase: Running
34. podIP: 10.244.2.45
35. podIPs:
36. - ip: 10.244.2.45
37. qosClass: Guaranteed
38. startTime: "2022-10-08T09:33:07Z"

You can find the concrete device allocate result through annotation scheduling.koordinator.sh/device-allocated.

4.more apply protocol:

1. apiVersion: v1
2. kind: Pod
3. ...
4. spec:
5. ...
6. resources:
7. requests:
8. cpu: 40m
9. memory: 40Mi
10. nvidia.com/gpu: "100"

1. apiVersion: v1
2. kind: Pod
3. ...
4. spec:
5. ...
6. resources:
7. requests:
8. cpu: 40m
9. memory: 40Mi
10. kubernetes.io/gpu-core: "100"
11. kubernetes.io/gpu-memory-ratio: "100"

1. apiVersion: v1
2. kind: Pod
3. ...
4. spec:
5. ...
6. resources:
7. requests:
8. cpu: 40m
9. memory: 40Mi
10. kubernetes.io/gpu-core: "100"
11. kubernetes.io/gpu-memory: "16Mi"

4.device resource debug api:

1. $ kubectl -n koordinator-system get lease koord-scheduler --no-headers | awk '{print $2}' | cut -d'_' -f1 | xargs -I {} kubectl -n koordinator-system get pod {} -o wide --no-headers | awk '{print $6}'
2. 10.244.0.64

4. $ curl 10.244.0.64:10251/apis/v1/plugins/DeviceShare/nodeDeviceSummaries
5. $ curl 10.244.0.64:10251/apis/v1/plugins/DeviceShare/nodeDeviceSummaries/host04

1. {
2. "allocateSet": {
3. "gpu": {
4. "xlf/gpu-pod7": {
5. "0": {
6. "kubernetes.io/gpu-core": "100",
7. "kubernetes.io/gpu-memory": "12508288Ki",
8. "kubernetes.io/gpu-memory-ratio": "100"
9. }
10. }
11. }
12. },
13. "deviceFree": {
14. "kubernetes.io/gpu-core": "0",
15. "kubernetes.io/gpu-memory": "0",
16. "kubernetes.io/gpu-memory-ratio": "0"
17. },
18. "deviceFreeDetail": {
19. "gpu": {
20. "0": {
21. "kubernetes.io/gpu-core": "0",
22. "kubernetes.io/gpu-memory": "0",
23. "kubernetes.io/gpu-memory-ratio": "0"
24. }
25. }
26. },
27. "deviceTotal": {
28. "kubernetes.io/gpu-core": "100",
29. "kubernetes.io/gpu-memory": "12508288Ki",
30. "kubernetes.io/gpu-memory-ratio": "100"
31. },
32. "deviceTotalDetail": {
33. "gpu": {
34. "0": {
35. "kubernetes.io/gpu-core": "100",
36. "kubernetes.io/gpu-memory": "12508288Ki",
37. "kubernetes.io/gpu-memory-ratio": "100"
38. }
39. }
40. },
41. "deviceUsed": {
42. "kubernetes.io/gpu-core": "100",
43. "kubernetes.io/gpu-memory": "12508288Ki",
44. "kubernetes.io/gpu-memory-ratio": "100"
45. },
46. "deviceUsedDetail": {
47. "gpu": {
48. "0": {
49. "kubernetes.io/gpu-core": "100",
50. "kubernetes.io/gpu-memory": "12508288Ki",
51. "kubernetes.io/gpu-memory-ratio": "100"
52. }
53. }
54. }
55. }