增强的 NodeResourceFit

介绍

原生的 Kubernetes NodeResourcesFit 插件只能为所有资源采用一种策略，例如 MostRequestedPriority 和 LeastRequestedPriority。然而，在实际工业应用中，这种设计并不适用于某些场景。例如：

  1. 在 AI 场景中，申请 GPU 的 Pod 优先占用整台 GPU 机器，以防止 GPU 碎片化。
  2. 申请 CPU 和内存的 Pod 会被优先分配到非 GPU 机器上，以防止 GPU 机器上的 CPU 和内存被过度消耗，从而导致需要 GPU 的实际任务因非 GPU 资源不足而待处理。

因此，希望这两种策略都能提供以满足这一业务需求。Koordinator 提供 NodeResourcesFitPlus 和 ScarceResourceAvoidance 插件来支持这两个场景的需求，从而：

  • 不同类型的资源可以配置不同的策略，并通过权重的形式进行优先级排序。
  • 防止未申请稀缺资源的 Pod 被调度到具有稀缺资源的节点上。

插件逻辑介绍

NodeResourcesFitPlus

NodeResourcesFitPlus 插件的调度器参数可以配置如下：

1. resources:
2. nvidia.com/gpu:
3. type: MostAllocated
4. weight: 2
5. cpu:
6. type: LeastAllocated
7. weight: 1
8. memory:
9. type: LeastAllocated
10. weight: 1

这两种策略的评分计算方法和效果如下： image

最终的节点评分可以按如下方式计算：

1. finalScoreNode = [(weight1 * resource1) + (weight2 * resource2) + … + (weightN* resourceN)] /(weight1+weight2+ … +weightN)

ScarceResourceAvoidance

ScarceResourceAvoidance 插件的调度器参数可以按如下方式配置：

1. resources:
2. - nvidia.com/gpu
3. - scarceResource1
4. - scarceResource2

从配置中获取稀缺资源类型后，插件将按如下方式对节点和 Pod 的匹配性进行评分：

  1. 获取 Pod 请求的资源类型集合以及节点可以分配的资源类型集合。
  2. ResourceTypesUnused = ResourceTypesTotal - ResourceTypesRequested
  3. ScarceResourceTypesUnused = intersection(ResourceTypesUnused, ScarceResourceTypes)

未使用的稀缺资源类型越多，评分越低。

例子

SchedulerConfiguration 样例

我们可以按如下方式配置 schedulerConfiguration：

1. profiles:
2. - pluginConfig:
3. - args:
4. apiVersion: kubescheduler.config.k8s.io/v1
5. kind: NodeResourcesFitPlusArgs
6. resources:
7. nvidia.com/gpu:
8. type: MostAllocated
9. weight: 2
10. cpu:
11. type: LeastAllocated
12. weight: 1
13. memory:
14. type: LeastAllocated
15. weight: 1
16. name: NodeResourcesFitPlus
17. - args:
18. apiVersion: kubescheduler.config.k8s.io/v1
19. kind: ScarceResourceAvoidanceArgs
20. resources:
21. - nvidia.com/gpu
22. name: ScarceResourceAvoidance
23. plugins:
24. score:
25. enabled:
26. - name: NodeResourcesFitPlus
27. weight: 2
28. - name: ScarceResourceAvoidance
29. weight: 2
30. disabled:
31. - name: "*"
32. schedulerName: koord-scheduler

GPU 应用和节点打分

1. apiVersion: apps/v1
2. kind: Deployment
3. metadata:
4. name: test-scheduler1
5. spec:
6. replicas: 1
7. selector:
8. matchLabels:
9. app: test-scheduler1
10. template:
11. metadata:
12. labels:
13. app: test-scheduler1
14. spec:
15. schedulerName: koord-scheduler
16. containers:
17. - image: dockerpull.com/nginx
18. imagePullPolicy: IfNotPresent
19. name: nginx
20. ports:
21. - containerPort: 80
22. resources:
23. requests:
24. cpu: "100"
25. memory: "100"
26. nvidia.com/gpu: 2
27. limits:
28. cpu: "100"
29. memory: "100"
30. nvidia.com/gpu: 2

节点分配率信息：

1. node1: cpu 58%, memory 21%, nvidia.com/gpu 6 (total 8)

3. node2: cpu 22%, memory 5%, nvidia.com/gpu 0 (total 8)

结果：优先选择 GPU 机器并 Binpack GPU 资源

查看打分 => Top10 scores for pod

1. | # | Pod | Node | Score | NodeResourcesFitPlus | ScarceResourceAvoidance |
2. | 0 | test-scheduler1-xxx | node1 | 358 | 158 | 200 |
3. | 1 | test-scheduler1-xxx | node2 | 296 | 96 | 200 |

CPU 应用和节点打分

1. apiVersion: apps/v1
2. kind: Deployment
3. metadata:
4. name: test-scheduler1
5. spec:
6. replicas: 1
7. selector:
8. matchLabels:
9. app: test-scheduler1
10. template:
11. metadata:
12. labels:
13. app: test-scheduler1
14. spec:
15. schedulerName: koord-scheduler
16. containers:
17. - image: dockerpull.com/nginx
18. imagePullPolicy: IfNotPresent
19. name: nginx
20. ports:
21. - containerPort: 80
22. resources:
23. requests:
24. cpu: "100"
25. memory: "100"
26. limits:
27. cpu: "100"
28. memory: "100"

节点分配率信息：

1. node1: cpu 22%, memory 10%, nvidia.com/gpu 6 (total 8)
2. node2: cpu 40%, memory 50%
3. node3: cpu 30%, memory 20%

结果：优先选择 GPU 机器并 Binpack GPU 资源

查看打分 => Top10 scores for pod

1. | # | Pod | Node | Score | NodeResourcesFitPlus | ScarceResourceAvoidance |
2. | 0 | test-scheduler1-xxx | node1 | 310 | 144 | 166 |
3. | 1 | test-scheduler1-xxx | node2 | 262 | 62 | 200 |
4. | 2 | test-scheduler1-xxx | node3 | 326 | 126 | 200 |

与原生 NodeResourceFit 兼容

MostAllocated

原生配置：

1. apiVersion: v1
2. kind: ConfigMap
3. metadata:
4. name: scheduler-config
5. namespace: kube-system
6. data:
7. scheduler-config.yaml: |
8. apiVersion: kubescheduler.config.k8s.io/v1
9. kind: KubeSchedulerConfiguration
10. profiles:
11. - schedulerName: koord-scheduler
12. pluginConfig:
13. - args:
14. scoringStrategy:
15. resources:
16. - name: cpu
17. weight: 2
18. - name: memory
19. weight: 1
20. type: MostAllocated
21. name: NodeResourcesFit
22. plugins:
23. score:
24. enabled:
25. - name: "NodeResourcesFit"

NodeResourcesFitPlus 配置:

1. apiVersion: v1
2. kind: ConfigMap
3. metadata:
4. name: scheduler-config
5. namespace: kube-system
6. data:
7. scheduler-config.yaml: |
8. apiVersion: kubescheduler.config.k8s.io/v1
9. kind: KubeSchedulerConfiguration
10. profiles:
11. - schedulerName: koord-scheduler
12. pluginConfig:
13. - args:
14. apiVersion: kubescheduler.config.k8s.io/v1
15. kind: NodeResourcesFitPlusArgs
16. resources:
17. cpu:
18. type: MostAllocated
19. weight: 2
20. memory:
21. type: MostAllocated
22. weight: 1
23. name: NodeResourcesFitPlus
24. plugins:
25. score:
26. enabled:
27. - name: "NodeResourcesFitPlus"

LeastAllocated

原生配置:

1. apiVersion: v1
2. kind: ConfigMap
3. metadata:
4. name: scheduler-config
5. namespace: kube-system
6. data:
7. scheduler-config.yaml: |
8. apiVersion: kubescheduler.config.k8s.io/v1
9. kind: KubeSchedulerConfiguration
10. profiles:
11. - schedulerName: koord-scheduler
12. pluginConfig:
13. - args:
14. scoringStrategy:
15. resources:
16. - name: cpu
17. weight: 2
18. - name: memory
19. weight: 1
20. type: LeastAllocated
21. name: NodeResourcesFit
22. plugins:
23. score:
24. enabled:
25. - name: "NodeResourcesFit"

NodeResourcesFitPlus 配置:

1. apiVersion: v1
2. kind: ConfigMap
3. metadata:
4. name: scheduler-config
5. namespace: kube-system
6. data:
7. scheduler-config.yaml: |
8. apiVersion: kubescheduler.config.k8s.io/v1
9. kind: KubeSchedulerConfiguration
10. profiles:
11. - schedulerName: koord-scheduler
12. pluginConfig:
13. - args:
14. apiVersion: kubescheduler.config.k8s.io/v1
15. kind: NodeResourcesFitPlus
16. resources:
17. cpu:
18. type: LeastAllocated
19. weight: 2
20. memory:
21. type: LeastAllocated
22. weight: 1
23. name: NodeResourcesFitPlus
24. plugins:
25. score:
26. enabled:
27. - name: "NodeResourcesFitPlus"