Is there any way to add MIG Tesla A100 GPU device to LXD container?

LXD
1

I have a test setup with Tesla A100 GPU and I created several MIG devices on it. Now each MIG device has been provided a device ID.

So for example my new MIG device ID is: 0:0 (GPUDeviceIndex:MIGDeviceIndex) and I pass this to the LXD container with the following command:
lxc config device add test gpu0 gpu id=0:0

But it fails to detect this ID and shows this error:
Error: Failed to start device "gpu0": Failed to detect requested GPU device

As per the documentation provided by Nvidia here: https://docs.nvidia.com/datacenter/tesla/mig-user-guide/index.html#running-containers , there are two ways to access MIG devices inside containers and I have tried both of them. But none seem to work.

Any help would be appreciated.

2

@monstermunchkin do you know if this can be achieved? Thanks

3

@genesis96839 Could you please provide the output of lxc query /1.0/resources | jq '.gpu'? My guess is that id=0:0 is incorrect as this is usually just an integer.

4

@monstermunchkin here’s the output for lxc query /1.0/resources | jq '.gpu':

{
  "cards": [
    {
      "driver": "nvidia",
      "driver_version": "460.32.03",
      "drm": {
        "card_device": "226:0",
        "card_name": "card0",
        "id": 0,
        "render_device": "226:128",
        "render_name": "renderD128"
      },
      "numa_node": 0,
      "nvidia": {
        "architecture": "8.0",
        "brand": "Tesla",
        "card_device": "195:0",
        "card_name": "nvidia0",
        "cuda_version": "11.2",
        "model": "A100-SXM4-40GB",
        "nvrm_version": "460.32.03",
        "uuid": "GPU-c2051740-eac9-9c65-0bc0-5c1ec5c67197"
      },
      "pci_address": "0000:00:04.0",
      "product_id": "20b0",
      "vendor": "NVIDIA Corporation",
      "vendor_id": "10de"
    }
  ],
  "total": 1
}

Also the output of nvidia-smi with MIG devices:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-SXM4-40GB      Off  | 00000000:00:04.0 Off |                   On |
| N/A   34C    P0    49W / 400W |     11MiB / 40536MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  0    1   0   0  |      5MiB / 20096MiB | 42      0 |  3   0    2    0    0 |
|                  |      0MiB / 32767MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+
|  0    2   0   1  |      5MiB / 20096MiB | 42      0 |  3   0    2    0    0 |
|                  |      0MiB / 32767MiB |           |                       |
+------------------+----------------------+-----------+-----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
5

LXD only detects the main GPU here, so it won’t work until we add support for this.

@stgraber is there any reason we can’t support MIG?

6

Currently the reason is that we don’t have hardware to test it.
I’m waiting for NVIDIA to send me something but it’s been taking a while…

7

Ohk. So any chances you will get hands on this in near future? Or if there could be any other workaround for running MIG instances with LXD?

8

Nvidia tells me I should be receiving one soon but no exact ETA.
Without having some time with one of those cards it’s hard to guess what would be needed to support this feature.

It will also depend on whether libnvidia-container supports it already.

1 Like
9

Thanks. I can understand. Would love to know how it works out once you get your hands on it.

10

Fedex just dropped it off now, so just need to find a suitable system here and then some time to poke at it :slight_smile:

11

Haha that was pretty quick! Awesome. Looking forward to it. :slight_smile:

12

Yeah, sadly the only compatible system I have around is an arm64 box and NVIDIA doesn’t seem to have vGPU drivers for that, so I’m waiting for more hardware to arrive over the coming days, hoping I can arrange a test system here which will be compatible with this card (none of our Xeon based systems seem to like it somehow…).

13

Oh ok. The last I got my hands on a physical system was with AMD EPYC series CPU with A100s. But recently when I tried LXD for MIG on the A100s, it was on a VM with Intel Xeons. But not sure which generation was it. Let me see if I can get the name of that Xeon CPU.

14

Looks like the system in question had some issue with 4G memory mappings for GPUs.

Instead I now have the A100 in a consumer grade machine I quickly assembled, Ryzen 6300 on a Asus Pro-WS-X570-ACE. It’s properly detected, nvidia-smi looks happy and I’m just waiting for access to the vGPU drivers to get to validate all the features.

Well I’m also waiting for a bunch of extra fans and fan controller as I’ve got nowhere near enough air flow in that test system to keep both that A100 and the Mellanox Connect-X 6 dual-100Gb card cool :wink:

2 Likes
15

Seems like a nice setup :slight_smile:

Awesome! Looking forward to the updates on MIG instances.

Oh wow. I never got my hands on a Mellanox card. Specially the 100GB cards.

I believe there would be hardly any latency when using such powerful interconnects for HPC clustering. I am pretty interested in clustering aspect as I am working on a remote node clustering solution and such speedy interconnects are only a dream in that case lol.

16

@stgraber any updates on the A100 MIG instance setup?

17

Not quite yet. I have the fancy drivers now at least but I’ve been quite busy with other things so far.

18

Ohk. Sure… I’m looking forward to it

19

I’ve fixed our support for mdev+SR-IOV on the A100 for the VM side of things.
I’ve got a few minor UX fixes I want to sort out for that side of the story before I investigate MIG for containers.

1 Like
20

@genesis96839

root@argos:~# lxc config show c1
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu focal amd64 (20210331_07:42)
  image.os: Ubuntu
  image.release: focal
  image.serial: "20210331_07:42"
  image.type: squashfs
  image.variant: default
  nvidia.runtime: "true"
  volatile.base_image: 9cc48d3d856ca6be1c424874d59f733188d17e7be10bd1b860bcfbdd97b6cac2
  volatile.eth0.host_name: veth536fe939
  volatile.eth0.hwaddr: 00:16:3e:df:47:bb
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 48f2cdef-b806-46e8-8ace-e634c1fa3536
devices:
  a100-0-1:
    gputype: mig
    mig.compute: "1"
    mig.gpu: "0"
    pci: "0000:09:00.0"
    type: gpu
  a100-0-3:
    gputype: mig
    mig.compute: "3"
    mig.gpu: "0"
    pci: "0000:09:00.0"
    type: gpu
ephemeral: false
profiles:
- default
stateful: false
description: ""
root@argos:~# lxc exec c1 bash
root@c1:~# nvidia-smi 
Thu Apr  1 02:10:25 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.69       Driver Version: 460.69       CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  A100-PCIE-40GB      On   | 00000000:09:00.0 Off |                  Off |
| N/A   84C    P0    72W / 250W |                  N/A |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  0    0   1   0  |      0MiB / 40536MiB | 14    N/A |  7   0    5    1    1 |
|                  |      1MiB / 65536MiB |           |                       |
+------------------+                      +-----------+-----------------------+
|  0    0   3   1  |                      | 14    N/A |  7   0    5    1    1 |
|                  |                      |           |                       |
+------------------+----------------------+-----------+-----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
root@c1:~#

Does that look good to you?

2 Likes