设备调度 - 寒武纪 MLU
- 背景
- 使用方法
  - 前置条件
  - 使用

设备调度 - 寒武纪 MLU

背景

当前 Koordinator 支持寒武纪卡在 K8s 的使用，基于 koord-device-daemon 和 koordlet 组件上报异构 GPU 资源，将异构卡信息汇总到 Device 中供调度器进行拓扑调度。

使用方法

前置条件

昇腾卡的使用需要提前安装配置如下组件

寒武纪 Driver
寒武纪 Cambricon Device Plugin (开启虚拟化参数配置)

1. args:
2. - --mode=dynamic-smlu # device plugin mode: default, dynamic-smlu, env-share, mim, topology-aware
3. - --virtualization-num=1 # virtualization number for each MLU, used only in env-share mode, set to 110 to support multi cards per container in env-share mode
4. - --mlulink-policy=best-effort # MLULink topology policy: best-effort, guaranteed or restricted, used only in topology-aware mode
5. - --cnmon-path=/usr/bin/cnmon # host machine cnmon path, must be absolute path. comment out this line to avoid mounting cnmon
6. - --log-level=info # log level: trace/debug/info/warn/error/fatal/panic" default:"info"
7. - --min-dsmlu-unit=256 # minimum unit for dsmu, used only in dynamic-smlu mode" default:"0" env:"MIN-DSMLU-UNIT"

Koordinator 相关组件(koordinator >= v1.7.0)
需要在 chart 的 scheduler.featureGates 参数中加入 DevicePluginAdaption=true 以启用该功能所需的特性门控

使用

确认寒武纪卡已成功被 Device 识别，示例如下，通过 kubectl get device <node-name> -o yaml 查看 Device 资源。注意：
node.koordinator.sh/gpu-vendor 对应的标签值为 cambricon

1. apiVersion: scheduling.koordinator.sh/v1alpha1
2. kind: Device
3. metadata:
4. labels:
5. node.koordinator.sh/gpu-model: MLU-370 # The model of the Cambricon card.
6. node.koordinator.sh/gpu-vendor: cambricon # The vendor of the card.
7. name: node-1
8. spec:
9. devices:
10. - health: true
11. id: 89033010-2354-0000-0000-000000000000 # The UUID of the Cambricon card (mocked value).
12. minor: 0 # The minor number of the Cambricon card.
13. resources:
14. koordinator.sh/gpu-core: "100" # The total compute capacity of the Cambricon card in percentage.
15. koordinator.sh/gpu-memory: 24Gi # The total GPU memory of the Cambricon card.
16. koordinator.sh/gpu-memory-ratio: "100" # The total GPU memory ratio in percentage.
17. topology:
18. busID: 0000:47:00.0 # The PCI bus ID.
19. nodeID: 3 # The NUMA node ID.
20. pcieID: pci0000:3a # The PCI root ID.
21. socketID: -1
22. type: gpu
23. conditions:
24. - lastTransitionTime: "2025-04-25T10:00:00Z"
25. message: device is healthy
26. reason: DeviceHealthy
27. status: "True"
28. type: Healthy
29. status: {}

确认寒武纪的节点上已正确注册寒武纪卡资源信息，示例如下，通过 kubectl get node <node-name> -o yaml 查看 node.status.allocatable/node.status.capacity 资源。

1. apiVersion: v1
2. kind: Node
3. metadata:
4. name: node-1
5. ...
6. status:
7. allocatable:
8. ...
9. koordinator.sh/gpu.shared: "100" # Report by koordinator
10. koordinator.sh/gpu-core: "100" # Report by koordinator
11. koordinator.sh/gpu-memory: 24Gi # Report by koordinator
12. koordinator.sh/gpu-memory-ratio: "100" # Report by koordinator
13. cambricon.com/mlu.smlu.vcore: "100" # Report by cambricon device plugin
14. cambricon.com/mlu.smlu.vmemory: "96" # Report by cambricon device plugin
15. capacity:
16. ...
17. koordinator.sh/gpu.shared: "100" # Report by koordinator
18. koordinator.sh/gpu-core: "100" # Report by koordinator
19. koordinator.sh/gpu-memory: 24Gi # Report by koordinator
20. koordinator.sh/gpu-memory-ratio: "100" # Report by koordinator
21. cambricon.com/mlu.smlu.vcore: "100" # Report by cambricon device plugin
22. cambricon.com/mlu.smlu.vmemory: "96" # Report by cambricon device plugin

创建 Pod 申请使用寒武纪卡，样例如下，示例中申请了虚拟卡场景。寒武纪不支持配置多卡虚拟化。
注意
此处镜像使用了乌班图基础镜像，用户可根据实际需求选择合适的镜像
resources.limits 和 resources.requests 中均需要额外配置资源
- 虚拟卡场景，以 MLU-370 使用虚拟化为例。
- koordinator.sh/gpu.shared
  - 使用虚拟卡填写 1 即可。
- koordinator.sh/gpu-core
  - 用多少算力，填写多少即可，百分比表示。
- cambricon.com/mlu.smlu.vcore
  - 用多少算力，填写多少即可，百分比表示。
- cambricon.com/mlu.smlu.vmemory
  - 用多少显存，比如 1GB，需要换算，填写分片的数量。以每个分片为 256Mi 为例，分片大小在 Cambricon Device Plugin 的 min-dsmlu-unit中配置。 slice_count = 1GB / 256Mi 。

1. apiVersion: v1
2. kind: Pod
3. metadata:
4. labels:
5. app: demo-sleep
6. name: test-cambricon-partial
7. namespace: default
8. spec:
9. containers:
10. - command:
11. - sleep
12. - infinity
13. image: ubuntu:18.04
14. imagePullPolicy: IfNotPresent
15. name: demo-sleep
16. resources:
17. limits:
18. cpu: "32"
19. memory: 64Gi
20. koordinator.sh/gpu.shared: "1"
21. koordinator.sh/gpu-memory: "1Gi"
22. koordinator.sh/gpu-core: "10"
23. cambricon.com/mlu.smlu.vcore: "10" # percentage
24. cambricon.com/mlu.smlu.vmemory: "4" # slice counts, actual allocated memory: 4*256Mi=1Gi
25. requests:
26. cpu: "32"
27. memory: 64Gi
28. koordinator.sh/gpu.shared: "1"
29. koordinator.sh/gpu-memory: "1Gi"
30. koordinator.sh/gpu-core: "10"
31. cambricon.com/mlu.smlu.vcore: "10"
32. cambricon.com/mlu.smlu.vmemory: "4"

进入容器内部(kubectl exec -it {pod-name} --bash)，在容器内部执行 ls /dev/cambricon* 命令查看卡挂载情况。如果能够正常输出，表示卡已经成功分配到 Pod 中。