GPU & RDMA Joint Allocation

Introduction

In the AI model training scenario, collective communication between GPUs is involved, which may occur across nodes. To improve network communication speed, NICs that support the RDMA protocol are widely used in AI training scenarios.

Since v1.5.0, Koordinator has implemented joint scheduling capabilities for GPU and RDMA. In version v1.6.0, Koordinator provides an end-to-end solution for this. The overall architecture is as follows:

1. Koordlet detects the GPUs and RDMA-capable NICs in nodes and reports related information to the Device CR.
2. Koord-Manager syncs resources from the Device CR to node.status.allocatable.
3. Koord-Scheduler allocates GPUs and RDMA-capable NICs for pods according to device topology and annotates the pods with the allocation results.
4. Multus-CNI accesses the Koordlet PodResources Proxy to obtain the RDMA devices allocated to the pod and attaches the corresponding NICs to the pod’s network namespace.
5. Koordlet provides NRI plugins that can mount devices into containers.

Due to the numerous components involved and the complexity of the environment, we provide this best practice guide. In this guide, we will demonstrate how to deploy Koordinator, Multus-CNI, and SRIOV-CNI step by step, as well as how to use NCCL programs to confirm that our system is indeed working.

The basic validation logic is as follows:

1. the user will submit two Pods requesting GPU and RDMA resources, respectively.
2. Koordinator will allocate these two Pods to designated nodes, ensuring that the GPUs and RDMA assigned to the Pods are located under the same PCIe switch.
3. We will then verify the RDMA connectivity between the Pods and ultimately use MPI to validate that GDR can run successfully.

Finally, from the data plane perspective, the device topology possessed by the Pods and the network association between the Pods appear as follows:

Prerequisite

• Kuberenetes >= 1.28
• Koordinator >= 1.6
• Containerd >= 1.7
• Multus-CNI >= 4.0
• SRIOV-CNI >= 2.0

Environment Setting

Cluster And Nodes

Our cluster has two worker nodes and one master node. The details version info is as follwoing table.

The device information and network connectivity information for the two worker nodes are pictured as follows:

Details about GPUs And NICs

Every worker has 4 Tesla P40 GPUs.

1. root@k8s-node1:~/ss/koo/script# nvidia-smi
2. Wed Nov 27 16:21:46 2024
3. +-----------------------------------------------------------------------------------------+
4. | NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
5. |-----------------------------------------+------------------------+----------------------+
6. | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
7. | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
8. |                                         |                        |               MIG M. |
9. |=========================================+========================+======================|
10. |   0  Tesla P40                      Off |   00000000:02:00.0 Off |                    0 |
11. | N/A   21C    P8             12W /  250W |       0MiB /  23040MiB |      0%      Default |
12. |                                         |                        |                  N/A |
13. +-----------------------------------------+------------------------+----------------------+
14. |   1  Tesla P40                      Off |   00000000:03:00.0 Off |                    0 |
15. | N/A   26C    P8             10W /  250W |       0MiB /  23040MiB |      0%      Default |
16. |                                         |                        |                  N/A |
17. +-----------------------------------------+------------------------+----------------------+
18. |   2  Tesla P40                      Off |   00000000:82:00.0 Off |                    0 |
19. | N/A   23C    P8             10W /  250W |       0MiB /  23040MiB |      0%      Default |
20. |                                         |                        |                  N/A |
21. +-----------------------------------------+------------------------+----------------------+
22. |   3  Tesla P40                      Off |   00000000:83:00.0 Off |                    0 |
23. | N/A   18C    P8              8W /  250W |       0MiB /  23040MiB |      0%      Default |
24. |                                         |                        |                  N/A |
25. +-----------------------------------------+------------------------+----------------------+

27. +-----------------------------------------------------------------------------------------+
28. | Processes:                                                                              |
29. |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
30. |        ID   ID                                                               Usage      |
31. |=========================================================================================|
32. |  No running processes found                                                             |
33. +-----------------------------------------------------------------------------------------+

1. root@k8s-node2:~# nvidia-smi
2. Wed Nov 27 16:22:16 2024
3. +-----------------------------------------------------------------------------------------+
4. | NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4     |
5. |-----------------------------------------+------------------------+----------------------+
6. | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
7. | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
8. |                                         |                        |               MIG M. |
9. |=========================================+========================+======================|
10. |   0  Tesla P40                      Off |   00000000:02:00.0 Off |                    0 |
11. | N/A   31C    P8             10W /  250W |       0MiB /  23040MiB |      0%      Default |
12. |                                         |                        |                  N/A |
13. +-----------------------------------------+------------------------+----------------------+
14. |   1  Tesla P40                      Off |   00000000:03:00.0 Off |                    0 |
15. | N/A   31C    P8             10W /  250W |       0MiB /  23040MiB |      0%      Default |
16. |                                         |                        |                  N/A |
17. +-----------------------------------------+------------------------+----------------------+
18. |   2  Tesla P40                      Off |   00000000:82:00.0 Off |                    0 |
19. | N/A   37C    P8             10W /  250W |       0MiB /  23040MiB |      0%      Default |
20. |                                         |                        |                  N/A |
21. +-----------------------------------------+------------------------+----------------------+
22. |   3  Tesla P40                      Off |   00000000:83:00.0 Off |                    0 |
23. | N/A   30C    P8             10W /  250W |       0MiB /  23040MiB |      0%      Default |
24. |                                         |                        |                  N/A |
25. +-----------------------------------------+------------------------+----------------------+

27. +-----------------------------------------------------------------------------------------+
28. | Processes:                                                                              |
29. |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
30. |        ID   ID                                                               Usage      |
31. |=========================================================================================|
32. |  No running processes found                                                             |
33. +-----------------------------------------------------------------------------------------+

In addition to GPU, we also need to setup rdma environment on the worker nodes in advance.

1. Plan the physical NIC for the test

Log in to node1 and create VF based on the Mellanox CX5 network adapter. Since the host already has two nics, three cx5 nics will appear if VF is successfully created.

Create instruction is as follows:

1. echo '1' > /sys/class/net/ens11f0np0/device/sriov_numvfs

The host runs the following command: “lspci |grep Mell”. If [ConnectX-5 Virtual Function] is displayed, VF is created successfully.

1. root@k8s-node1:/data/cc/code/koordinator# lspci |grep Mell 2. 01:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3. 01:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 4. 01:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function] //VF

If you run ibstat, mlx5_2 in the output is VF：

1. CA 'mlx5_0' 2. CA type: MT4119 3. Number of ports: 1 4. Firmware version: 16.35.4030 5. Hardware version: 0 6. Node GUID: 0x1070fd0300a4487a 7. System image GUID: 0x1070fd0300a4487a 8. Port 1: 9. State: Active 10. Physical state: LinkUp 11. Rate: 25 12. Base lid: 0 13. LMC: 0 14. SM lid: 0 15. Capability mask: 0x00010000 16. Port GUID: 0x1270fdfffea4487a 17. Link layer: Ethernet 18. CA 'mlx5_1' 19. CA type: MT4119 20. Number of ports: 1 21. Firmware version: 16.35.4030 22. Hardware version: 0 23. Node GUID: 0x1070fd0300a4487b 24. System image GUID: 0x1070fd0300a4487a 25. Port 1: 26. State: Down 27. Physical state: Disabled 28. Rate: 25 29. Base lid: 0 30. LMC: 0 31. SM lid: 0 32. Capability mask: 0x00010000 33. Port GUID: 0x1270fdfffea4487b 34. Link layer: Ethernet 35. CA 'mlx5_2' //VF 36. CA type: MT4120 37. Number of ports: 1 38. Firmware version: 16.35.4030 39. Hardware version: 0 40. Node GUID: 0x0000000000000000 41. System image GUID: 0x1070fd0300a4487a 42. Port 1: 43. State: Active 44. Physical state: LinkUp 45. Rate: 25 46. Base lid: 0 47. LMC: 0 48. SM lid: 0 49. Capability mask: 0x00010000 50. Port GUID: 0x0000000000000000 51. Link layer: Ethernet 3. Create a VF on node2

Log in to node2 and create VF based on the Mellanox CX5 network adapter. The host already has two cx5 nics. If the VF is created successfully, three cx5 nics are displayed.

Create instruction is as follows:

1. echo '1' > / sys/class/net/ens11f0np0 / device/sriov_numvfs

The host runs the following command: “lspci |grep Mell”. If [ConnectX-5 Virtual Function] is displayed, VF is created successfully.

1. root@k8s-node3:~# lspci |grep Mell 2. d2:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 3. d2:00.1 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5] 4. d2:01.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]//VF

If you run ibstat, mlx5_2 in the output is VF

1. CA 'mlx5_0' 2. CA type: MT4119 3. Number of ports: 1 4. Firmware version: 16.32.1010 5. Hardware version: 0 6. Node GUID: 0x1070fd0300a4486a 7. System image GUID: 0x1070fd0300a4486a 8. Port 1: 9. State: Down 10. Physical state: Disabled 11. Rate: 40 12. Base lid: 0 13. LMC: 0 14. SM lid: 0 15. Capability mask: 0x00010000 16. Port GUID: 0x0000000000000000 17. Link layer: Ethernet 18. CA 'mlx5_1' 19. CA type: MT4119 20. Number of ports: 1 21. Firmware version: 16.32.1010 22. Hardware version: 0 23. Node GUID: 0x1070fd0300a4486b 24. System image GUID: 0x1070fd0300a4486a 25. Port 1: 26. State: Down 27. Physical state: Disabled 28. Rate: 25 29. Base lid: 0 30. LMC: 0 31. SM lid: 0 32. Capability mask: 0x00010000 33. Port GUID: 0x0000000000000000 34. Link layer: Ethernet 35. CA 'mlx5_2' //VF 36. CA type: MT4119 37. Number of ports: 1 38. Firmware version: 16.35.3006 39. Hardware version: 0 40. Node GUID: 0x1070fd0300a44882 41. System image GUID: 0x1070fd0300a44882 42. Port 1: 43. State: Down 44. Physical state: Disabled 45. Rate: 40 46. Base lid: 0 47. LMC: 0 48. SM lid: 0 49. Capability mask: 0x00010000 50. Port GUID: 0x0000000000000000 51. Link layer: Ethernet

Deploy Koordinator, Multus-CNI and SRIOV-CNI

Deploy Koordinator

1. helm repo add koordinator-sh https://koordinator-sh.github.io/charts/
2. helm repo update
3. helm install koordinator koordinator-sh/koordinator --version 1.6.0

The modified yaml feature parameters of koordlet componet are as follows:

1. - -feature-gates=Accelerators=true,GPUEnvInject=true,RDMADeviceInject=true,RDMADevices=true,PodResourcesProxy=true

Deploy Multus

1. To use latest features try command below which applies a daemonset and installs thick Multus using kubectl:

1. kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml

1. root@k8s-master:~# kubectl get po -n kube-system |grep multus 2. kube-multus-ds-7ddbh 1/1 Running 0 38h 3. kube-multus-ds-cgvqq 1/1 Running 0 38h 4. kube-multus-ds-lc6nv 1/1 Running 0 38h

This indicates that your systems is ready to use Multus CNI. Supported from Multus-CNI release 4.0+. 2. Modify the daemonset to adapt to koordinator

1. kubectl edit ds kube-multus-ds -n kube-system

1. apiVersion: apps/v1 2. kind: DaemonSet 3. metadata: 4. name: kube-multus-ds 5. namespace: kube-system 6. labels: 7. tier: node 8. app: multus 9. name: multus 10. spec: 11. selector: 12. matchLabels: 13. name: multus 14. updateStrategy: 15. type: RollingUpdate 16. template: 17. metadata: 18. labels: 19. tier: node 20. app: multus 21. name: multus 22. spec: 23. containers: 24. - name: kube-multus 25. volumeMounts: 26. ... 27. - name: host-var-lib-kubelet 28. mountPath: /var/lib/kubelet/pod-resources 29. mountPropagation: HostToContainer 30. ... 31. volumes: 32. ... 33. - name: host-var-lib-kubelet 34. hostPath: 35. path: /var/run/koordlet/pod-resources 36. ...

Plan NAD for Nodes

Multus CNI relys on NetworkAttachmentDefinition configuration to allocate ip and configure network, we as cluster admin need to plan NAD configuration file in advance.

1. the NAD of ens11f0np0 on node1 is as following:

1. apiVersion: "k8s.cni.cncf.io/v1"
2. kind: NetworkAttachmentDefinition
3. metadata:
4. name: sriov-attach-k8s-node1-ens11f0np0-kubeflow-conf
5. namespace: kubeflow
6. annotations:
7. k8s.v1.cni.cncf.io/resourceName: koordinator.sh/rdma
8. spec:
9. config: '{
10. "cniVersion": "0.3.1",
11. "name": "sriov-attach",
12. "type": "sriov",
13. "capabilities": {
14. "mac": true,
15. "ipam": true
16. },
17. "master": "ens11f0np0",
18. "mode": "passthrough",
19. "ipam": {
20. "type": "host-local",
21. "subnet": "10.20.12.0/24",
22. "rangeStart": "10.20.12.121", //Plan the IP address range of the Pod
23. "rangeEnd": "10.20.12.121"
24. }
25. }'

1. the NAD of ens3f0np0 on node2 is as following:

1. apiVersion: "k8s.cni.cncf.io/v1"
2. kind: NetworkAttachmentDefinition
3. metadata:
4. name: sriov-attach-k8s-node2-ens3f0np0-kubeflow-conf
5. namespace: kubeflow
6. annotations:
7. k8s.v1.cni.cncf.io/resourceName: koordinator.sh/rdma
8. spec:
9. config: '{
10. "cniVersion": "0.3.1",
11. "name": "sriov-attach",
12. "type": "sriov",
13. "capabilities": {
14. "mac": true,
15. "ipam": true
16. },
17. "master": "ens3f0np0",
18. "mode": "passthrough",
19. "ipam": {
20. "type": "host-local",
21. "subnet": "10.20.12.0/24",
22. "rangeStart": "10.20.12.134",//Plan the IP address range of the Pod
23. "rangeEnd": "10.20.12.134"
24. }
25. }'

1. create Namespace on k8s cluster

1. kubectl create ns kubeflow 2. Run the following command to deploy the nad

1. kubectl apply -f sriov-attach-k8s-node1-ens11f0np0-kubeflow-conf.yaml 2. kubectl apply -f sriov-attach-k8s-node2-ens3f0np0-kubeflow-conf.yaml

Deploy SRIOV-CNI

See the SR-IOV CNI repository for build and installation instructions. Supported from SR-IOV CNI release 2.0+.

Deploy Pods and Check Allocation Result

Deploy Application Pods

Note: This experiment requires two pods, so you need to write yaml files corresponding to two Pods. Expect one Pod directed to node1 and one Pod directed to node2.

1. Label Nodes: In order to facilitate testing, Pod directional scheduling is required to a node, and Node needs to be labeled. Specific instructions are as follows

1. kubectl label nodes k8s-node1 koo=node1 2. kubectl label nodes k8s-node2 koo=node2 2. Deploy Pods: Run the following command to deploy the Pod

1. kubectl apply -f pod01.yaml 2. kubectl apply -f pod02.yaml

1. apiVersion: v1 2. kind: Pod 3. metadata: 4. name: pod-vf01 5. namespace: kubeflow 6. annotations: 7. # this NAD is already written previously 8. k8s.v1.cni.cncf.io/networks: sriov-attach-k8s-node1-ens11f0np0-kubeflow-conf 9. scheduling.koordinator.sh/device-joint-allocate: |- 10. { 11. "deviceTypes": ["gpu","rdma"] 12. } 13. scheduling.koordinator.sh/device-allocate-hint: |- 14. { 15. "rdma": { 16. "vfSelector": {} //apply VF 17. } 18. } 19. labels: 20. selector-type: pod 21. spec: 22. nodeSelector: 23. koo: node1 //Directional scheduling to 1 node 24. schedulerName: koord-scheduler //Uses the koordlet scheduling plug-in 25. containers: 26. - name: container-vf 27. image: nvcr.io/nvidia/pytorch:24.04-py3 28. securityContext: 29. capabilities: 30. add: [ "IPC_LOCK" ] 31. imagePullPolicy: IfNotPresent 32. command: [ "/bin/bash", "-c", "--" ] 33. args: [ "while true; do sleep 300000; done;" ] 34. volumeMounts: 35. - mountPath: /dev/shm 36. name: shm 37. resources: 38. requests: 39. koordinator.sh/gpu: 100//apply a GPU 40. koordinator.sh/rdma: 100//apply a VF 41. limits: 42. koordinator.sh/gpu: 100 43. koordinator.sh/rdma: 100 44. volumes: 45. - name: shm 46. emptyDir: 47. medium: Memory 48. sizeLimit: "10Gi"

1. apiVersion: v1 2. kind: Pod 3. metadata: 4. name: pod-vf02 5. namespace: kubeflow 6. annotations: 7. k8s.v1.cni.cncf.io/networks: sriov-attach-k8s-node3-enp210s0f1np1-kubeflow-conf 8. scheduling.koordinator.sh/device-joint-allocate: |- 9. { 10. "deviceTypes": ["gpu","rdma"] 11. } 12. scheduling.koordinator.sh/device-allocate-hint: |- 13. { 14. "rdma": { 15. "vfSelector": {} 16. } 17. } 18. labels: 19. selector-type: pod 20. spec: 21. nodeSelector: 22. koo: node2 23. schedulerName: koord-scheduler 24. containers: 25. - name: container-vf 26. image: nvcr.io/nvidia/pytorch:24.04-py3 27. securityContext: 28. capabilities: 29. add: [ "IPC_LOCK" ] 30. imagePullPolicy: IfNotPresent 31. command: [ "/bin/bash", "-c", "--" ] 32. args: [ "while true; do sleep 300000; done;" ] 33. volumeMounts: 34. - mountPath: /dev/shm 35. name: shm 36. resources: 37. requests: 38. koordinator.sh/gpu: 100 39. koordinator.sh/rdma: 100 40. limits: 41. koordinator.sh/gpu: 100 42. koordinator.sh/rdma: 100 43. volumes: 44. - name: shm 45. emptyDir: 46. medium: Memory 47. sizeLimit: "10Gi" 3. Check pod running status

1. root@k8s-master:~/ss/koo/rdma/sriov# kubectl get po -n kubeflow -owide 2. NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 3. pod-vf01 1/1 Running 0 103m 10.244.1.10 k8s-node1 <none> <none> 4. pod-vf02 1/1 Running 0 10h 10.244.2.18 k8s-node2 <none> <none>

If the status of the pod is running, the pod is successfully created and running.

Check Device Allocation Result

1. We extract the allocation information of pod-vf01 through the following command

1. kubectl get pod pod-vf01 -n kubeflow -oyaml

1. scheduling.koordinator.sh/device-allocated: '{"gpu":[{"minor":0,"resources":{"koordinator.sh/gpu-core":"100","koordinator.sh/gpu-memory":"23040Mi","koordinator.sh/gpu-memory-ratio":"100"}}],"rdma":[{"minor":0,"resources":{"koordinator.sh/rdma":"1"},"extension":{"vfs":[{"minor":-1,"busID":"0000:01:00.2"}]}}]}' 2. ...... 3. dnsPolicy: ClusterFirst 4. enableServiceLinks: true 5. nodeName: k8s-node1 //It has been scheduled to node 1 6. nodeSelector: 7. koo: node1 8. preemptionPolicy: PreemptLowerPriority 9. priority: 0 10. restartPolicy: Always 11. schedulerName: koord-scheduler 2. Enter the container and run the command “nvidia-smi” and Check the GPU allocation result

``` 1. root@pod-vf01:/home# nvidia-smi 2. Fri Nov 22 06:55:59 2024 3. +-----------------------------------------------------------------------------------------+ 4. | NVIDIA-SMI 550.127.05 Driver Version: 550.127.05 CUDA Version: 12.4 | 5. |-----------------------------------------+------------------------+----------------------+ 6. | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 7. | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 8. | | | MIG M. | 9. |=========================================+========================+======================| 10. | 0 Tesla P40 Off | 00000000:02:00.0 Off | 0 | 11. | N/A 24C P8 10W / 250W | 0MiB / 23040MiB | 0% Default | 12. | | | N/A | 13. +-----------------------------------------+------------------------+----------------------+

1. +-----------------------------------------------------------------------------------------+
2. | Processes: |
3. | GPU GI CI PID Type Process name GPU Memory |
4. | ID ID Usage |
5. |=========================================================================================|
6. | No running processes found |
7. +-----------------------------------------------------------------------------------------+ ```
8. Check whether the Pod named pod-vf01 device assignment results meet affinity

1. kubectl get devices.scheduling.koordinator.sh k8s-node1 -oyaml

1. apiVersion: scheduling.koordinator.sh/v1alpha1 2. kind: Device 3. metadata: 4. ..... 5. spec: 6. devices: 7. - health: true 8. id: GPU-989aa251-1dfe-5bbc-7c12-46e817b1de9a 9. minor: 0 //The GPU to which pod-vf01 is assigned is GPU 0, and the corresponding PCIE is pci0000:00 10. resources: 11. koordinator.sh/gpu-core: "100" 12. koordinator.sh/gpu-memory: 23040Mi 13. koordinator.sh/gpu-memory-ratio: "100" 14. topology: 15. busID: "0000:02:00.0" 16. nodeID: 0 17. pcieID: pci0000:00 18. socketID: -1 19. type: gpu 20. - health: true 21. id: "0000:01:00.0" 22. minor: 0 23. resources: 24. koordinator.sh/rdma: "100" 25. topology: 26. busID: "0000:01:00.0" 27. nodeID: 0 28. pcieID: pci0000:00 29. socketID: -1 30. type: rdma 31. vfGroups: 32. - vfs: 33. - busID: "0000:01:00.2"//pod-vf01 is assigned to this vf device, and the corresponding PCIE is pci0000:00 34. minor: -1 35. - health: true 36. id: GPU-e8a40bd0-e484-2d1b-cad9-75b043139b0c 37. minor: 1 38. resources: 39. koordinator.sh/gpu-core: "100" 40. koordinator.sh/gpu-memory: 23040Mi 41. koordinator.sh/gpu-memory-ratio: "100" 42. topology: 43. busID: "0000:03:00.0" 44. nodeID: 0 45. pcieID: pci0000:00 46. socketID: -1 47. type: gpu 48. - health: true 49. id: "0000:01:00.1" 50. minor: 1 51. resources: 52. koordinator.sh/rdma: "100" 53. topology: 54. busID: "0000:01:00.1" 55. nodeID: 0 56. pcieID: pci0000:00 57. socketID: -1 58. type: rdma 59. - health: true 60. id: GPU-5293b3a7-2bbb-e135-c6ab-c548b5c5b0a6 61. minor: 2 62. resources: 63. koordinator.sh/gpu-core: "100" 64. koordinator.sh/gpu-memory: 23040Mi 65. koordinator.sh/gpu-memory-ratio: "100" 66. topology: 67. busID: 0000:82:00.0 68. nodeID: 0 69. pcieID: pci0000:80 70. socketID: -1 71. type: gpu 72. - health: true 73. id: "0000:05:00.0" 74. minor: 2 75. resources: 76. koordinator.sh/rdma: "100" 77. topology: 78. busID: "0000:05:00.0" 79. nodeID: 0 80. pcieID: pci0000:00 81. socketID: -1 82. type: rdma 83. - health: true 84. id: GPU-d60a283a-a846-eaa7-f551-c0c4f6f4402a 85. minor: 3 86. resources: 87. koordinator.sh/gpu-core: "100" 88. koordinator.sh/gpu-memory: 23040Mi 89. koordinator.sh/gpu-memory-ratio: "100" 90. topology: 91. busID: 0000:83:00.0 92. nodeID: 0 93. pcieID: pci0000:80 94. socketID: -1 95. type: gpu 96. status: {}

According to the topology information, pod-vf01 is assigned to the vf device busID: “0000:01:00.2”, and the corresponding PCIE is pci0000:00. The GPU to which pod-vf01 is assigned is GPU 0, and the corresponding PCIE is pci0000:00. Because PCIE is the same, the GPU and NIC meet the expected topology affinity. 4. In the same way, check whether the device assignment result of pod-vf02 meets affinity

At this point, one GPU and one RDMA device applied by the two Pods are successfully allocated, and the topology affinity is met.

Check RDMA Connectivity

Check Network Connectivity

1. Enter the pod and install basic network tools

1. kubectl exec -it pod-vf01 -n kubeflow -- bash

1. apt-get update 2. apt-get install -y net-tools 3. apt install -y iputils-ping 4. apt-get install infiniband-diags -y 5. apt-get install -y kmod 6. apt-get install -y perftest 7. apt-get install -y ethtool 8. ...... 2. Check the IP address assignment.

``` 1. eth0: flags=4163 mtu 1450 2. inet 10.244.1.10 netmask 255.255.255.0 broadcast 10.244.1.255 3. inet6 fe80::e4c7:a3ff:fe4c:9d15 prefixlen 64 scopeid 0x20 4. ether e6:c7:a3:4c:9d:15 txqueuelen 0 (Ethernet) 5. RX packets 17129 bytes 57434980 (57.4 MB) 6. RX errors 0 dropped 244 overruns 0 frame 0 7. TX packets 13383 bytes 1019323 (1.0 MB) 8. TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

1. lo: flags=73 mtu 65536
2. inet 127.0.0.1 netmask 255.0.0.0
3. inet6 ::1 prefixlen 128 scopeid 0x10
4. loop txqueuelen 1000 (Local Loopback)
5. RX packets 487 bytes 211446 (211.4 KB)
6. RX errors 0 dropped 0 overruns 0 frame 0
7. TX packets 487 bytes 211446 (211.4 KB)

8. TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

9. net1: flags=4163 mtu 1500

10. inet 10.20.12.121 netmask 255.255.255.0 broadcast 10.20.12.255
11. inet6 fe80::6ce7:bfff:fee0:9382 prefixlen 64 scopeid 0x20
12. ether 6e:e7:bf:e0:93:82 txqueuelen 1000 (Ethernet)
13. RX packets 477 bytes 86270 (86.2 KB)
14. RX errors 0 dropped 0 overruns 0 frame 0
15. TX packets 327 bytes 47335 (47.3 KB)
16. TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ```

The net1 network port name here is the network port name assigned by multus-cni to pod, and the address is the address segment we configured in the previous nad named sriov-attach-k8s-node1-ens11f0np0-kubeflow-conf: 10.20.12.121. 3. Same as pod-vf01, the IP address of pod-vf02 is as following:

``` 1. eth0: flags=4163 mtu 1450 2. inet 10.244.2.21 netmask 255.255.255.0 broadcast 10.244.2.255 3. inet6 fe80::f45c:90ff:fe3a:67a2 prefixlen 64 scopeid 0x20 4. ether f6:5c:90:3a:67:a2 txqueuelen 0 (Ethernet) 5. RX packets 21690 bytes 65555332 (65.5 MB) 6. RX errors 0 dropped 1310 overruns 0 frame 0 7. TX packets 15612 bytes 1218973 (1.2 MB) 8. TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

1. lo: flags=73 mtu 65536
2. inet 127.0.0.1 netmask 255.0.0.0
3. inet6 ::1 prefixlen 128 scopeid 0x10
4. loop txqueuelen 1000 (Local Loopback)
5. RX packets 794 bytes 277124 (277.1 KB)
6. RX errors 0 dropped 0 overruns 0 frame 0
7. TX packets 794 bytes 277124 (277.1 KB)

8. TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

9. net1: flags=4163 mtu 1500

10. inet 10.20.12.134 netmask 255.255.255.0 broadcast 10.20.12.255
11. inet6 fe80::ac97:a4ff:fe72:d1f1 prefixlen 64 scopeid 0x20
12. ether ae:97:a4:72:d1:f1 txqueuelen 1000 (Ethernet)
13. RX packets 492 bytes 110501 (110.5 KB)
14. RX errors 0 dropped 0 overruns 0 frame 0
15. TX packets 318 bytes 42371 (42.3 KB)
16. TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 ```

The net1 network port name here is the network port name assigned by multus-cni to pod, and the address is the address segment we configured in the previous nad named sriov-attach-k8s-node2-ens3f0np0-kubeflow-conf: 10.20.12.134. 4. Ping pod-vf02’s net1 network port inside pod-vf01:

1. root@pod-vf01:/workspace# ping 10.20.12.134 2. PING 10.20.12.134 (10.20.12.134) 56(84) bytes of data. 3. 64 bytes from 10.20.12.134: icmp_seq=1 ttl=64 time=0.293 ms 4. 64 bytes from 10.20.12.134: icmp_seq=2 ttl=64 time=0.212 ms 5. 64 bytes from 10.20.12.134: icmp_seq=3 ttl=64 time=0.216 ms 6. 64 bytes from 10.20.12.134: icmp_seq=4 ttl=64 time=0.221 ms

The results show that the two Pods can communicate with each other, but ping is not enough to prove that the VF ports assigned by the two cx5 can communicate. You need to perform further tests on the specified vf port.

Check RDMA Connectivity

1. Check the mounting information of vf devices inside the pod, using POD-VF01 as an example (pod-vf02 refer to pod-vf01 for the same reason, no special explanation is provided here).

1. root@pod-vf01:/workspace# ibstat 2. CA 'mlx5_0' 3. CA type: MT4119 4. Number of ports: 1 5. Firmware version: 16.35.4030 6. Hardware version: 0 7. Node GUID: 0x1070fd0300a4487a 8. System image GUID: 0x1070fd0300a4487a 9. Port 1: 10. State: Active 11. Physical state: LinkUp 12. Rate: 25 13. Base lid: 0 14. LMC: 0 15. SM lid: 0 16. Capability mask: 0x00010000 17. Port GUID: 0x0000000000000000 18. Link layer: Ethernet 19. CA 'mlx5_1' 20. CA type: MT4119 21. Number of ports: 1 22. Firmware version: 16.35.4030 23. Hardware version: 0 24. Node GUID: 0x1070fd0300a4487b 25. System image GUID: 0x1070fd0300a4487a 26. Port 1: 27. State: Down 28. Physical state: Disabled 29. Rate: 25 30. Base lid: 0 31. LMC: 0 32. SM lid: 0 33. Capability mask: 0x00010000 34. Port GUID: 0x0000000000000000 35. Link layer: Ethernet 36. CA 'mlx5_2'//VF 37. CA type: MT4120 38. Number of ports: 1 39. Firmware version: 16.35.4030 40. Hardware version: 0 41. Node GUID: 0x0000000000000000 42. System image GUID: 0x1070fd0300a4487a 43. Port 1: 44. State: Active 45. Physical state: LinkUp 46. Rate: 25 47. Base lid: 0 48. LMC: 0 49. SM lid: 0 50. Capability mask: 0x00010000 51. Port GUID: 0x6ce7bffffee09382 52. Link layer: Ethernet

You can see three network ports: mlx5_0 (Up), mlx5_1 (Down), and mlx5_2 (Up). In fact, the VF we apply for comes from the mlx5_2 virtualized by the physical network adapter mlx5_0. That is, mlx5_2 is a virtual network interface, which is derived from mlx5_0. mlx5_1 is unavailable in the Down state. The pod should actually only use the mlx5_2 virtual VF communication inside. Similarly, the VF port used by pod-vf02 is mlx5_2. So let’s do a test. 2. Enter the Pod-vf01 container, use the mlx5_2 (VF) port to enable the ib_write listening service.

``` 1. root@pod-vf01:/workspace# ib_write_bw -d mlx5_2 -F

1. ---

2. Waiting for client to connect... *

3. ---

   ```
4. Enter the Pod-vf02 container, use the mlx5_2 (VF) port to enable the ib_write service connected to pod-vf01, and run the following command

1. root@pod-vf02:/workspace# ib_write_bw -d mlx5_2 10.20.12.121 2. --------------------------------------------------------------------------------------- 3. RDMA_Write BW Test 4. Dual-port : OFF Device : mlx5_2 5. Number of qps : 1 Transport type : IB 6. Connection type : RC Using SRQ : OFF 7. PCIe relax order: ON 8. ibv_wr* API : ON 9. TX depth : 128 10. CQ Moderation : 1 11. Mtu : 1024[B] 12. Link type : Ethernet 13. GID index : 3 14. Max inline data : 0[B] 15. rdma_cm QPs : OFF 16. Data ex. method : Ethernet 17. --------------------------------------------------------------------------------------- 18. local address: LID 0000 QPN 0x03ad PSN 0x17d925 RKey 0x029300 VAddr 0x0073f17a0af000 19. GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:20:12:134 20. remote address: LID 0000 QPN 0x00e1 PSN 0x146e34 RKey 0x021400 VAddr 0x007bc5c59c3000 21. GID: 00:00:00:00:00:00:00:00:00:00:255:255:10:20:12:121 22. --------------------------------------------------------------------------------------- 23. #bytes #iterations BW peak[MB/sec] BW average[MB/sec] MsgRate[Mpps] 24. Conflicting CPU frequency values detected: 800.000000 != 2000.000000. CPU Frequency is not max. 25. 65536 5000 2758.40 2758.38 0.044134 26. ---------------------------------------------------------------------------------------

1. bytes: The size of data transmitted each time is 65536 bytes. 2. iterations: 5000 iterations are performed. 3. BW peak[MB/sec] : The peak bandwidth is 2758.40 MB/s. 4. BW average[MB/sec] : The average bandwidth is 2758.38 MB/s. 5. MsgRate[Mpps] : The message rate (messages per second) is 0.044134 Mpps. 4. In the preceding result, ibv_wr API:ON indicates that ibv_wr API is used to perform RDMA operations. Transport type: IB: indicates InfiniBand. Note: The IB nic device of the RDMA protocol is used for network communication, which meets expectations.

Next, we test GPU communication, that is, we used GPU collection communication library NCCL to carry out NCCL communication test on VF network ports of two cx5.

Check GPUDirect RDMA

Install the NCCL Library and Compile

Enter pod-vf01 and pod-vf02 respectively, and install nccl and compile, taking pod-vf01 as an example:

1. Enter pod-vf01

1. kc exec -it pod-vf07 -n kubeflow -- bash 2. Enter directory /home

1. cd /home/ 3. Download code

1. git clone https://github.com/NVIDIA/nccl-tests.git 4. Enter directory /home/nccl-tests

1. cd /home/nccl-tests 5. Compile

1. make MPI=1 MPI_HOME=/usr/local/mpi

Setup Mutual Trust between Pods

Install openssh

1. Enter into pod-vf01

1. kc exec -it pod-vf01 -n kubeflow -- bash 2. apt-update

1. apt update 3. install openssh-server

1. apt install vim openssh-server openssh-client -y 4. Repeat the above steps on pod-vf02

Generate RSA Key

1. Write the contents of the file /root/.ssh/id_rsa.pub to /root/.ssh/authorized_keys in all containers: Copy by hand, one per line

1. ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCVRX69XvcjVlF6a1wqxMMh4ZHDNSzEGwPm7qJdsCkO1JPUpCI+2h44NzRtKBFMf1kfw3d6fOqTh/mVhuhBFTmsQVHaGjj8tffkVzieSJ3RAQYFHKvv4ZPvcN3bsbiqbjE9Syq0JLDahZy1sfTygI0ax6p0uJVAVr03bKy31WVAVi2R6f2Hc6QB5tsHVOzIBK7hCehhNe0wfPW8q0vVK8y36DBLwZC92DLPn77x27c8zT87K2nIuDiVGGkKAu3Fkk6utYswPijlZIW6OjMY1Orx8400eo77wZSybCfZJc25Fr9C14l53db7BV4x1vOcy1teGh8OkOJXwtDo6okQpOJhpuG25FlIpFEgQJZPFkYHOFB+q783+o8vAFd7g3xouS2ARlNnqsO7jB8ZvMTaa89NyKlQKWI3ObVkqjqYvRXlZ/gDhRG2Z5QSV/eVhsY3Dx5IMVPobz4R3rV3/n5QIUXRnMebEAxdfM+VeX+0P11yjPOrYyti7D+p1rYB+3Yf5/0= 2. ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCZRkemmpzBFIl8CQ3lb8uzzMs5H9f7Mo8eHm/IVYRR8FF6X1Gh+z8c88q1fdMgfa9vup2JbRywUeHS2LY9+I3Ln2MK6VB568LjRGJFaGK2vrEcBnaQgPKa9W1xXX+k+93CcAgjECw92nVVKCkfALLUyZEEqmw9Va5iV74cPM7le7VBQOfbOWfogweYuwE7FwRHrFDbueyc9GX1BvzOscSFn/V2YEuQzKOkZQHmcX+OAeV/TepZVKzYzt5mN0Q0P7UWmgn2CD+a4IFjQjXxbPw1zDP+wYmD6jIADks2GNHJu8huCK4IMJQzesMOWoch+2kkK80b0UvAQjTUMwMr2t6CPgOQafEygOr623clROYSSycTQ09ikt9g6SO31UZ4idNcoRcYqomDUs3+pceorer9adLHXM8MmRyRl6wEhCufJ4p4hYhwkL0rLCpBQ011NCP0hzoxUlQyVMnW13ztaKazX65ibunelGdpxJVeI++ldHDD6I3ZdhyP9Yiw767ka2k= 2. To generate a rsa key, run the ssh-keygen -t rsa command and press Enter

1. ssh-keygen -t rsa

Start sshd

Execute the following commands inside each pod.

1. mkdir -p /var/run/sshd && /usr/sbin/sshd -p 20024

Verify SSH Connectivity

Note The ip address of Pod-vf02 is 10.244.2.21. To access pod-vf01, run ssh root@10.244.2.21 -p 20024

1. root@pod-vf01:/home# ssh root@10.244.2.21 -p 20024
2. Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 6.5.0-41-generic x86_64)

4. * Documentation:  https://help.ubuntu.com
5. * Management:     https://landscape.canonical.com
6. * Support:        https://ubuntu.com/pro

8. This system has been minimized by removing packages and content that are
9. not required on a system that users do not log into.

11. To restore this content, you can run the 'unminimize' command.
12. Last login: Fri Nov 22 06:51:03 2024 from 10.244.2.1
13. root@pod-vf02:~#

If it is displayed that you can directly jump to the inside of another pod-vf02 container, it means that the no-secret setting is successful!

Double Machine IB Communication

We use the following command to perform GPU communication by RDMA between two Pods on different nodes.

1. mpirun --allow-run-as-root -H 10.244.1.10:1,10.244.2.21:1 -mca plm_rsh_args "-p 20024" -x NCCL_IB_DISABLE=0 -x NCCL_DEBUG=INFO -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_IB_HCA==mlx5_2 -x UCX_NET_DEVICES=eth0 -x NCCL_NET_GDR_READ=1 ./build/all_reduce_perf -b 2M -e 2G -f 2 -g 1 -n 100 -w 5

3. # -x NCCL_IB_HCA==mlx5_2: the name of the VF NIC device；
4. # -H 10.244.1.10:1,10.244.2.21:1  the IP addresses of the two containers, where :1 indicates the number of GPUs.

Inside either container, the personal test executes the following command inside the pod-vf02 container:

``` 1. root@pod-vf02:/home/nccl-tests# mpirun --allow-run-as-root -H 10.244.1.10:1,10.244.2.21:1 -mca plm_rsh_args "-p 20024" -x NCCL_IB_DISABLE=0 -x NCCL_DEBUG=INFO -x NCCL_SOCKET_IFNAME=eth0 -x NCCL_IB_HCA==mlx5_2 -x UCX_NET_DEVICES=eth0 -x NCCL_NET_GDR_READ=1 ./build/all_reduce_perf -b 2M -e 2G -f 2 -g 1 -n 100 -w 5 2. # nThread 1 nGpus 1 minBytes 2097152 maxBytes 2147483648 step: 2(factor) warmup iters: 5 iters: 100 agg iters: 1 validation: 1 graph: 0 3. ............... 4. NCCL version 2.21.5+cuda12.4 5. pod-vf07:15718:15718 [0] NCCL INFO cudaDriverVersion 12040 6. pod-vf07:15718:15718 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 7. pod-vf07:15718:15718 [0] NCCL INFO Bootstrap : Using eth0:10.244.1.10<0> 8. pod-vf08:12090:12099 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so 9. pod-vf08:12090:12099 [0] NCCL INFO P2P plugin IBext_v8 10. pod-vf08:12090:12099 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 11. pod-vf08:12090:12099 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth0:10.244.2.21<0> 12. pod-vf08:12090:12099 [0] NCCL INFO Using non-device net plugin version 0 13. pod-vf08:12090:12099 [0] NCCL INFO Using network IBext_v8 14. pod-vf07:15718:15726 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so 15. pod-vf07:15718:15726 [0] NCCL INFO P2P plugin IBext_v8 16. pod-vf07:15718:15726 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 17. pod-vf07:15718:15726 [0] NCCL INFO NET/IB : Using [0]mlx5_2:1/RoCE [RO]; OOB eth0:10.244.1.10<0> 18. pod-vf07:15718:15726 [0] NCCL INFO Using non-device net plugin version 0 19. pod-vf07:15718:15726 [0] NCCL INFO Using network IBext_v8 20. ..............

1. pod-vf02:12090:12099 [0] NCCL INFO ncclCommInitRank comm 0x5e303a52bd70 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 57000 commId 0xadcb40d61cc1bc4b - Init COMPLETE

3. out-of-place in-place

4. size count type redop root time algbw busbw #wrong time algbw busbw #wrong

5. (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s)

6. 2097152 524288 float sum -1 880.2 2.38 2.38 0 877.1 2.39 2.39 0
7. 4194304 1048576 float sum -1 1735.3 2.42 2.42 0 1737.9 2.41 2.41 0
8. 8388608 2097152 float sum -1 3444.5 2.44 2.44 0 3440.1 2.44 2.44 0
9. 16777216 4194304 float sum -1 6828.2 2.46 2.46 0 6857.6 2.45 2.45 0
10. 335544

（本文超出站点单文档体积上限，已截断）