K3d inside LXD container with multi-GPU support - Permissions configuration issue

LXD
, ,
1

Hi Team!

I have an 8 GPU server out of which I want to divide 4 GPUs into 2 separate LXD containers - Container A and Container B.

I also have security nesting and privilege mode set on both as true.

So far so good, as I can see the GPUs added in specific containers.

Now, I want to run kubernetes in Container A. I tried multiple guides online and fixed up on K3D as a simple setup for scheduling jobs on multiple GPUs in that container.

The issue I am facing is this:
In order to run K3D with docker containers in the LXD container, I need to add the following in apparmor profile:

raw.lxc: |-
    lxc.cgroup.devices.allow = a
    lxc.cap.drop =

After that I rebooted and k3d cluster started working. But the issue then became:
All the 8 GPUs became visible in the Container A. But my use case required only 4 GPUs out of 8.

How can I restrict to the required number of GPUs while adding enough permissions for the k3d cluster to work properly?

Please suggest what should I do. Appreciate a response on this. Thanks

2

Hi @stgraber would appreciate your thoughts on this issue.

3

Hi Team! @tomp @stgraber could you please suggest how I can resolve this issue?

4

I’m doing pretty much the same thing and can try to help. But first, how did you get to where you are now? I see plenty of guides for installing k3s onto LXC, but K3D is for docker? How did you install K3d with LXC in place of docker?