nvidia-docker2 在 Kubernetes 上实践

nvida-docker2 可以帮助我们将旧的加速计算应用程序容器化，将特定的 GPU 资源分配给容器，并可以轻松地跨不同的环境共享应用程序、协同工作和测试应用程序。今天带来的分享是有关 nvidia-docker2 在大规模 Kubernetes 集群上的实践记录。本文首发于 OpsDev.cn，转载已获取作者授权。

PS：丰富的一线技术、多元化的表现形式，尽在“HULK一线技术杂谈”，点关注哦！

# If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers

docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f

sudo yum remove nvidia-docker

# Add the package repositories

distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.repo | \

  sudo tee /etc/yum.repos.d/nvidia-docker.repo

# Install nvidia-docker2 and reload the Docker daemon configuration

sudo yum install -y nvidia-docker2

sudo pkill -SIGHUP dockerd

# Test nvidia-smi with the latest official CUDA image

docker run --runtime=nvidia --rm nvidia/cuda:9.0-base nvidia-smi

{

    "default-runtime":"nvidia",

    "runtimes": {

        "nvidia": {

          "path": "/usr/bin/nvidia-container-runtime",

          "runtimeArgs": []

        }

    },

}

systemctl restart docker

Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.

apiVersion: extensions/v1beta1

kind: DaemonSet

metadata:

  name: nvidia-device-plugin-daemonset

  namespace: kube-system

spec:

  template:

    metadata:

      # Mark this pod as a critical add-on; when enabled, the critical add-on scheduler

      # reserves resources for critical add-on pods so that they can be rescheduled after

      # a failure.  This annotation works in tandem with the toleration below.

      annotations:

        scheduler.alpha.kubernetes.io/critical-pod: ""

      labels:

        name: nvidia-device-plugin-ds

    spec:

      tolerations:

      # Allow this pod to be rescheduled while the node is in "critical add-ons only" mode.

      # This, along with the annotation above marks this pod as a critical add-on.

      - key: CriticalAddonsOnly

        operator: Exists

      containers:

      - image: nvidia/k8s-device-plugin:1.9

        name: nvidia-device-plugin-ctr

        securityContext:

          allowPrivilegeEscalation: false

          capabilities:

            drop: ["ALL"]

        volumeMounts:

          - name: device-plugin

            mountPath: /var/lib/kubelet/device-plugins

      volumes:

        - name: device-plugin

          hostPath:

            path: /var/lib/kubelet/device-plugins

kubectl create -f nvidia-docker-plugin.yml

apiVersion: v1

kind: Pod

metadata:

  name: cuda-vector-add

spec:

  restartPolicy: OnFailure

  containers:

    - name: cuda-vector-add

      image: "k8s.gcr.io/cuda-vector-add:v0.1"

      resources:

        limits:

          nvidia.com/gpu: 1

  nodeSelector:

    accelerator: nvidia-tesla-k80 # or nvidia-tesla-k80 etc.

kubectl create -f nvidia-docker2-gpu-pod.yml