volcano-vgpu-device-plugin - GitLite

KO | EN

gitlite — search

#javascript #golang #deep-learning #linux #docker #windows #awesome #cli #vue #security #pytorch #mcp #openai #open-source

volcano-vgpu-device-plugin

(★ 166)

Device-plugin for volcano vgpu which support hard resource isolation

파일 탐색기

최종 버전 다운로드 (.zip)

- - dev-image-build.yaml
  - helm-release.yml
  - release-image-build.yml
- - main.go
  - plugin-manager.go
  - root.go
- - feedback.go
  - main.go
  - metrics.go
  - validation.go
- - volcano-vgpu-device-plugin.yml
- config.md
- design.md
- example.png
- hard_limit.jpg
- vgpu-on-volcano.pdf
- Dockerfile.ubuntu20.04
- gpu-share.yml
- vgpu-case01.yml
- vgpu-case02.yml
- vgpu-case03.yml
- vgpu-deployment.yaml
- - ld.so.preload
- - api.go
  - api_mock.go
  - cdi.go
  - imex.go
  - null.go
  - options.go
- - config.go
- - api.go
  - cgo_helpers.go
  - consts.go
  - cuda.go
  - device.go
  - result.go
- - dependencies.go
- - kubeclient.go
  - node.go
- - imex.go
- - version.go
- - empty.go
  - imex.go
  - imex_test.go
  - labeler.go
  - labels.go
  - list.go
  - machine-type.go
  - mig-strategy.go
  - mig-strategy_test.go
  - nvml.go
  - nvml_test.go
  - output.go
  - output_test.go
  - resource.go
  - resource_test.go
  - strategy.go
  - timestamp.go
  - vgpu.go
- - mig-dp.go
  - mig.go
- - daemon.go
  - device.go
  - device_test.go
  - log-tailer.go
  - manager.go
  - options.go
  - root.go
- - api.go
  - factory.go
  - mps.go
  - options.go
  - register.go
  - server.go
  - server_test.go
- - cuda-device.go
  - cuda-lib.go
  - device_mock.go
  - factory.go
  - fallback.go
  - fallback_test.go
  - manager_mock.go
  - null.go
  - nvml-device.go
  - nvml-lib.go
  - nvml-mig-device.go
  - sysfs-device.go
  - sysfs-lib.go
  - types.go
- - allocate.go
  - device_map.go
  - device_map_test.go
  - devices.go
  - health.go
  - health_test.go
  - helper.go
  - nvml_devices.go
  - nvml_manager.go
  - rm.go
  - rm_mock.go
  - rm_test.go
  - tegra_devices.go
  - tegra_manager.go
  - wsl_devices.go
- - types.go
  - util.go
- - pciutil.go
  - pciutil_test.go
  - vgpu.go
  - vgpu_test.go
- - watchers.go
.dockerignore
.gitignore
.gitmodules
go.mod
go.sum
libvgpu
LICENSE
Makefile
Makefile.def
OWNERS
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Volcano vgpu device plugin for Kubernetes

[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin?ref=badge_shield)
[![docker pulls](https://img.shields.io/docker/pulls/projecthami/volcano-vgpu-device-plugin.svg)](https://hub.docker.com/r/projecthami/volcano-vgpu-device-plugin)

**Note**:

Volcano vgpu device-plugin can provide device-sharing mechanism for NVIDIA devices managed by volcano.

This is based on [Nvidia Device Plugin](https://github.com/NVIDIA/k8s-device-plugin), it uses [HAMi-core](https://github.com/Project-HAMi/HAMi-core) to support hard isolation of GPU card.

And collaborate with volcano, it is possible to enable GPU sharing.

## Table of Contents

- [About](#about)
- [Prerequisites](#prerequisites)
- [Quick Start](#quick-start)
  - [Preparing your GPU Nodes](#preparing-your-gpu-nodes)
  - [Enabling vGPU Support in Kubernetes](#enabling-gpu-support-in-kubernetes)
  - [Running vGPU Jobs](#running-vgpu-jobs)
- [Issues and Contributing](#issues-and-contributing)

## Version Compatibility Matrix

The following table shows the compatibility matrix between Volcano-vgpu and Volcano versions. Please ensure you are using compatible versions.

| Volcano-vgpu | Volcano |
|--------------|---------|
| v1.12.0 and below | v1.15.0 and below |
| master       | master  |

## About

The Volcano device plugin for Kubernetes is a Daemonset that allows you to automatically:
- Expose the number of GPUs on each node of your cluster
- Keep track of the health of your GPUs
- Run GPU enabled containers in your Kubernetes cluster.
- Provide device-sharing mechanism for GPU tasks as the figure below.
- Enforce hard resource limit in container.
- Support dynamic-mig, for more details, see [config](doc/config.md)

<img src="./doc/example.png" width = "600" />

## Prerequisites

The list of prerequisites for running the Volcano device plugin is described below:
* NVIDIA drivers > 440
* nvidia-docker version > 2.0 (see how to [install](https://github.com/NVIDIA/nvidia-docker) and it's [prerequisites](https://github.com/nvidia/nvidia-docker/wiki/Installation-\(version-2.0\)#prerequisites))
* docker configured with nvidia as the [default runtime](https://github.com/NVIDIA/nvidia-docker/wiki/Advanced-topics#default-runtime).
* Kubernetes version >= 1.16
* Volcano verison >= 1.9

## Quick Start

### Preparing your GPU Nodes

The following steps need to be executed on all your GPU nodes.
This README assumes that the NVIDIA drivers and nvidia-docker have been installed.

Note that you need to install the nvidia-docker2 package and not the nvidia-container-toolkit.
This is because the new `--gpus` options hasn't reached kubernetes yet. Example:
```bash
# Add the package repositories
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -fsSL https://nvidia.github.io/nvidia-docker/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-docker-keyring.gpg
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-docker-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-docker.list

$ sudo apt-get update && sudo apt-get install -y nvidia-docker2
$ sudo systemctl restart docker
```

You will need to enable the nvidia runtime as your default runtime on your node.
We will be editing the docker daemon config file which is usually present at `/etc/docker/daemon.json`:
```json
{
    "default-runtime": "nvidia",
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
```
> *if `runtimes` is not already present, head to the install page of [nvidia-docker](https://github.com/NVIDIA/nvidia-docker)*

### Configuration

You need to enable vgpu in volcano-scheduler configMap:

```shell script
kubectl edit cm -n volcano-system volcano-scheduler-configmap
```

For volcano v1.9+,, use the following configMap 
```yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: volcano-scheduler-configmap
  namespace: volcano-system
data:
  volcano-scheduler.conf: |
    actions: "enqueue, allocate, backfill"
    tiers:
    - plugins:
      - name: priority
      - name: gang
      - name: conformance
    - plugins:
      - name: drf
      - name: deviceshare
        arguments:
          deviceshare.VGPUEnable: true # enable vgpu
          deviceshare.SchedulePolicy: binpack  # scheduling policy. binpack / spread
      - name: predicates
      - name: proportion
      - name: nodeorder
      - name: binpack
```

### Sharing Mode

Volcano-vgpu supports two types of device-sharing: `HAMi-core` and `dynamia-mig`, A node can either using `HAMi-core`, or `Dynamic-mig`. Heterogeneous is supported(a part of node using HAMi-core, the other using Dynamic-mig)

A brief introduction about these two modes:

HAMi-core is a user-layer resource isolator provided by HAMi community, works on all types of GPU.

Dynamic-mig is a hardware resource isolator, works on Ampere arch or later GPU.

The table below shows the summary:
| Mode        | Isolation        | MIG GPU Required | Annotation | Core/Memory Control | Recommended For            |
| ----------- | ---------------- | ---------------- | ---------- | ------------------- | -------------------------- |
| HAMI-core   | Software (VCUDA) | No               | No         | Yes                 | General workloads          |
| Dynamic MIG | Hardware         | Yes              | Yes        | MIG-controlled      | Performance-sensitive jobs |

You can set the sharing mode and customize your installation by adjusting the [configs](doc/config.md)

### Enabling GPU Support in Kubernetes

Once you have enabled this option on *all* the GPU nodes you wish to use,
you can then enable GPU support in your cluster by deploying the following Daemonset:

#### Configure and install with Helm

Add the volcano-vgpu-device-plugin Helm repository:

```
helm repo add volcano-vgpu-device-plugin https://project-hami.github.io/volcano-vgpu-device-plugin
helm repo update
```

Install the chart:
```
helm install volcano-vgpu-device-plugin volcano-vgpu-device-plugin/volcano-vgpu-device-plugin
```

If you want to enable CDI, you can use the following command to install.
```
helm install volcano-vgpu-device-plugin volcano-vgpu-device-plugin/volcano-vgpu-device-plugin \
    --set cdi.enabled=true
```

#### Install with yaml

##### Normal Mode

```
$ kubectl apply -f deployments/static/volcano-vgpu-device-plugin.yml
```
##### CDI Mode
###### Modify deployments/static/volcano-vgpu-device-plugin.yml
1. Add the following environment variables to the `env` section of the `volcano-device-plugin` container.
```
- name: DEVICE_LIST_STRATEGY
  value: "cdi-annotations"
- name: NVIDIA_DRIVER_ROOT
  value: /
- name: NVIDIA_CDI_HOOK_PATH
  value: /usr/bin/nvidia-ctk
- name: GDRCOPY_ENABLED
  value: "false"
- name: GDS_ENABLED
  value: "false"
- name: MOFED_ENABLED
  value: "false"
```
2. Add the following configuration to the `volumeMounts` section of the `volcano-device-plugin` container.
```
- name: cdi-root
  mountPath: /var/run/cdi
- name: driver-root
  mountPath: /driver-root
  readOnly: true
- name: usrbin
  mountPath: /usrbin
  readOnly: true
```
3. Add the following configuration to the `volumes` section.
```
- name: driver-root
  hostPath:
    path: /
    type: Directory
- name: cdi-root
  hostPath:
    path: /var/run/cdi
    type: DirectoryOrCreate
- name: usrbin
  hostPath:
    path: /usr/bin
    type: Directory
```
###### Deploy
```
$ kubectl apply -f deployments/static/volcano-vgpu-device-plugin.yml
```

### Verify environment is ready

Check the node status, it is ok if `volcano.sh/vgpu-number` is included in the allocatable resources.

```shell script
$ kubectl get node {node name} -oyaml
...
  capacity:
    volcano.sh/vgpu-memory: "89424"
    volcano.sh/vgpu-number: "10"   # vGPU resource
```

### Running VGPU Jobs

VGPU can be requested by both set "volcano.sh/vgpu-number" , "volcano.sh/vgpu-cores" and "volcano.sh/vgpu-memory" in resource.limit

```shell script
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod1
  annotations:
    volcano.sh/vgpu-mode: "hami-core" # (Optional, 'hami-core' or 'mig')
spec:
  schedulerName: volcano
  containers:
    - name: cuda-container
      image: nvidia/cuda:9.0-devel
      command: ["sleep"]
      args: ["100000"]
      resources:
        limits:
          volcano.sh/vgpu-number: 2 # requesting 2 gpu cards
          volcano.sh/vgpu-memory: 3000 # (optinal)each vGPU uses 3G device memory
          volcano.sh/vgpu-cores: 50 # (optional)each vGPU uses 50% core  
EOF
```

You can validate device memory using nvidia-smi inside container:

![img](https://raw.githubusercontent.com/Project-HAMi/volcano-vgpu-device-plugin/main/doc/hard_limit.jpg)

> **WARNING:** *if you don't request GPUs when using the device plugin with NVIDIA images all
> the GPUs on the machine will be exposed inside your container.
> The number of vgpu used by a container can not exceed the number of gpus on that node.*
> You can specify the mode of this task by assigning `volcano.sh/vgpu-mode` annotations, If not, both modes are possible.

### Monitor

volcano-scheduler-metrics records every GPU usage and limitation, visit the following address to get these metrics.

```
curl {volcano scheduler cluster ip}:8080/metrics
```

You can also collect the **GPU utilization**, **GPU memory usage**, **pods' GPU memory limitations** and **pods' GPU memory usage** metrics on nodes by visiting the following addresses:

```
curl {volcano device plugin pod ip}:9394/metrics
```

Example:
```
# HELP Device_last_kernel_of_container Container device last kernel description
# TYPE Device_last_kernel_of_container gauge
Device_last_kernel_of_container{ctrname="cuda-container",deviceuuid="GPU-xxxx",podname="hami-device",podnamespace="default",vdeviceid="0",zone="vGPU"} 0
# HELP Device_memory_desc_of_container Container device meory description
# TYPE Device_memory_desc_of_container counter
Device_memory_desc_of_container{context="331350016",ctrname="cuda-container",data="1778516992",deviceuuid="GPU-xxxx",module="0",offset="0",podname="hami-device",podnamespace="default",vdeviceid="0",zone="vGPU"} 2.109867008e+09
# HELP Device_utilization_desc_of_container Container device utilization description
# TYPE Device_utilization_desc_of_container gauge
Device_utilization_desc_of_container{ctrname="cuda-container",deviceuuid="GPU-xxxx",podname="hami-device",podnamespace="default",vdeviceid="0",zone="vGPU"} 99
# HELP HostCoreUtilization GPU core utilization
# TYPE HostCoreUtilization gauge
HostCoreUtilization{deviceidx="0",deviceuuid="GPU-xxxx",zone="vGPU"} 0
HostCoreUtilization{deviceidx="1",deviceuuid="GPU-xxxx",zone="vGPU"} 100
# HELP HostGPUMemoryUsage GPU device memory usage
# TYPE HostGPUMemoryUsage gauge
HostGPUMemoryUsage{deviceidx="0",deviceuuid="GPU-xxxx",zone="vGPU"} 5.6366661632e+10
HostGPUMemoryUsage{deviceidx="1",deviceuuid="GPU-xxxx",zone="vGPU"} 5.8484457472e+10
# HELP vGPU_device_memory_limit_in_bytes vGPU device limit
# TYPE vGPU_device_memory_limit_in_bytes gauge
vGPU_device_memory_limit_in_bytes{ctrname="cuda-container",deviceuuid="GPU-xxxx",podname="hami-device",podnamespace="default",vdeviceid="0",zone="vGPU"} 3.145728e+09
# HELP vGPU_device_memory_usage_in_bytes vGPU device usage
# TYPE vGPU_device_memory_usage_in_bytes gauge
vGPU_device_memory_usage_in_bytes{ctrname="cuda-container",deviceuuid="GPU-xxxx",podname="hami-device",podnamespace="default",vdeviceid="0",zone="vGPU"} 2.109867008e+09
```

# Issues and Contributing
[Checkout the Contributing document!](CONTRIBUTING.md)

* You can report a bug by [filing a new issue](https://github.com/Project-HAMi/volcano-vgpu-device-plugin)
* You can contribute by opening a [pull request](https://help.github.com/articles/using-pull-requests/)

## Upgrading Kubernetes with the device plugin

Upgrading Kubernetes when you have a device plugin deployed doesn't require you to do any,
particular changes to your workflow.
The API is versioned and is pretty stable (though it is not guaranteed to be non breaking),
upgrading kubernetes won't require you to deploy a different version of the device plugin and you will
see GPUs re-registering themselves after you node comes back online.

Upgrading the device plugin is a more complex task. It is recommended to drain GPU tasks as
we cannot guarantee that GPU tasks will survive a rolling upgrade.
However we make best efforts to preserve GPU tasks during an upgrade.

## License
[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2FProject-HAMi%2Fvolcano-vgpu-device-plugin?ref=badge_large)