Cannot Use USB dev Inside Container Anymore

Hi,

Running LXD/LXC via SNAP on Ubuntu 22.04.1 LTS HOST.

$ snap list
lxd     5.7-c62733b  23889  latest/stable  canonical✓  -

$ lxd --version
5.7

$ lxc --version
5.7

 - Storage ZFS over LV
 - Added the following device to a ubuntu 22.04 instance

$ lxc config device show usb01 
ledger:
  gid: "1000"
  type: usb
  uid: "1000"
  vendorid: 2c97

I have a command that calls this USB device and returns OK as expected on HOST.
That same command on Ubuntu(1000)@GUEST says “please check that your device is connected”.

I have succeeded once in the past but I was on Ubuntu 20.04 by that time and maybe a previous version of LXC.

Please help me to debug.

Thanks in advance

Please show output of sudo lsusb on the host and the output of lxc config show <instance> --expanded?

Hi @tomp

I’ll provide with the sudo lsusb as soon as I get home (so that I can plug back the device).

In the mean time, here is the result of the config show

$ lxc config show usb01 --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (release) (20221101.1)
  image.label: release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20221101.1"
  image.type: squashfs
  image.version: "22.04"
  volatile.base_image: 75320c2df0d6f64dd6900b34614abe330940508089db3c9889f1fb8dbabb533b
  volatile.cloud-init.instance-id: 111641fe-85c2-4867-8064-38caa6bbd8d0
  volatile.eth0.host_name: vethdddf76d9
  volatile.eth0.hwaddr: 00:16:3e:96:cb:4d
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.last_state.ready: "false"
  volatile.uuid: b5e66982-8029-4def-b4a9-02d4bad8935a
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  ledger:
    gid: "1000"
    type: usb
    uid: "1000"
    vendorid: 2c97
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Also I was wondering (and maybe you can see that in the previous result) how I should manage the sub{u,g}id thing ?

On HOST,
I launch instances with “user” (not with sudo).
LXC seems to run on HOST with “root” account.
Also, there is an udev rule that says that the owner of this dev is “user” (rule provided by the dev editor).

On GUEST,
I run the operations with “ubuntu” (inside guest, id shows 1000:1000).

Should there be a customization of the subuid/subgid declared on HOST ?

Currently it says

$ cat /etc/subgid
root:100000:1000000000
user:100000:1000000000
$ cat /etc/subuid
root:100000:1000000000
user:100000:1000000000

Seems I’ve modified this already once but cannot remember why and if it was actually needed.
Anyway, if you have a good documentation regarding this subuid subgid I’ll take it :slight_smile:

Thanks for your feedback anyway,

What is the device?

One LEDGER secure key

Sorry, interrupting the post but isnt it necessary to define the productid on the container config show? Or it doesnt matter to define.
Regards.

I’d say it is not mandatory

or

Humm, looks like not mandatory.
I’m wondering, did you try with container or vm? If you havent tried with vm, can you?
Regards.

I don’t know anything about vm. Sorry.
You’re telling me it’s not possible with containers ?

The procedure is the same with container just at the end specify -vm parameter. After creating vm, you can proceed the same procedure once you did for the container before.
Regards.

Hi @Tom_Jabber

In addition to @tomp questions, I think it would be great if you try to run your test command under strace. Like this:
$ strace -f your_usb_device_test_command
and then provide its output. It allows to quickly understand why it fails in the container. Possibly it may be not existing device node in /dev, or permission issues (apparmor, device cgroup, etc…)

Sure ! Here you are:

As I can see from your strace:

[pid  2683] newfstatat(AT_FDCWD, "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/manufacturer", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid  2683] openat(AT_FDCWD, "/sys/devices/pci0000:00/0000:00:14.0/usb1/1-3/manufacturer", O_RDONLY|O_NOCTTY|O_CLOEXEC) = 19
[pid  2683] newfstatat(19, "", {st_mode=S_IFREG|0444, st_size=4096, ...}, AT_EMPTY_PATH) = 0
[pid  2683] read(19, "Ledger\n", 4104)  = 7

so, it looks like device metadata is visible from sysfs.

The error that you can see is printed from there:

And the error is not directly connected with the device, but with the connection problem to some thing which is called “crypto provider”. I’m not familiar with this particular piece of software, but as we can see from strace:

[pid  2683] mprotect(0x3e03000, 245760, PROT_READ|PROT_EXEC) = 0
[pid  2683] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 19
[pid  2683] connect(19, {sa_family=AF_INET, sin_port=htons(21325), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)

your application tries to reach TCP server at 127.0.0.1:21325 (most likely it’s trezord) and fails. That leads to an error that we can see. So, I think we need to shift attention from your command (cardano-hw-cli) to this daemon (trezord) and find logs for it.

Yes I’ve seen this connect to localhost try and indeed trezor might be involved in some case, but not in mine.
I’m gonna get the strace out of the OK case and compare.

In the meantime maybe you could help me understand the needfor udev rules and whether I have to implement them as well on container or not ?

Udevadm monitor doesn’t output the same when on host and when on guest

In the meantime maybe you could help me understand the needfor udev rules and whether I have to implement them as well on container or not ?

sure, please show your udev/udevadm logs.

I’m gonna get the strace out of the OK case and compare.

Sounds like a good plan!

My udev rule is:

# On HOST -- mandatory so that the command works fine
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2c97", ATTRS{idProduct}=="1011|1015", OWNER="user", TAG+="uaccess", TAG+="udev-acl"

$ udevadm monitor

KERNEL[268638.019208] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[268638.037707] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
KERNEL[268638.051431] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
KERNEL[268638.051447] add      /class/usbmisc (class)
KERNEL[268638.051532] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/usbmisc/hiddev0 (usbmisc)
KERNEL[268638.051557] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010/hidraw/hidraw0 (hidraw)
KERNEL[268638.051581] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
KERNEL[268638.051605] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268638.052780] add      /class/usbmisc (class)
KERNEL[268638.056085] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
KERNEL[268638.056125] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [268638.057381] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [268638.058484] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268638.058544] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
UDEV  [268638.059708] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
UDEV  [268638.060408] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/usbmisc/hiddev0 (usbmisc)
UDEV  [268638.062470] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010/hidraw/hidraw0 (hidraw)
UDEV  [268638.063075] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
UDEV  [268638.063683] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268638.064468] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[268639.929446] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/usbmisc/hiddev0 (usbmisc)
KERNEL[268639.929561] remove   /class/usbmisc (class)
KERNEL[268639.929641] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010/hidraw/hidraw0 (hidraw)
KERNEL[268639.929727] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
KERNEL[268639.929836] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
KERNEL[268639.929910] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
KERNEL[268639.929991] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
KERNEL[268639.930215] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
KERNEL[268639.931976] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[268639.932451] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [268639.932577] remove   /class/usbmisc (class)
UDEV  [268639.934376] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/usbmisc/hiddev0 (usbmisc)
UDEV  [268639.934581] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010/hidraw/hidraw0 (hidraw)
UDEV  [268639.935335] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
UDEV  [268639.935397] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
UDEV  [268639.935972] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1011.0010 (hid)
UDEV  [268639.936708] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268639.937168] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268639.937891] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [268639.939037] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[268640.463933] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[268640.497929] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
KERNEL[268640.513259] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011 (hid)
KERNEL[268640.513274] add      /class/usbmisc (class)
KERNEL[268640.513413] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/usbmisc/hiddev0 (usbmisc)
KERNEL[268640.513562] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011/hidraw/hidraw0 (hidraw)
KERNEL[268640.513585] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011 (hid)
KERNEL[268640.513603] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268640.513885] add      /class/usbmisc (class)
KERNEL[268640.518528] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
KERNEL[268640.536309] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012 (hid)
KERNEL[268640.536432] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/usbmisc/hiddev1 (usbmisc)
KERNEL[268640.536470] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012/hidraw/hidraw1 (hidraw)
KERNEL[268640.536494] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012 (hid)
KERNEL[268640.536515] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
KERNEL[268640.541569] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.2 (usb)
KERNEL[268640.541614] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [268640.542455] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [268640.543482] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.2 (usb)
UDEV  [268640.543540] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268640.543822] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
UDEV  [268640.544499] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011 (hid)
UDEV  [268640.545001] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012 (hid)
UDEV  [268640.545029] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/usbmisc/hiddev0 (usbmisc)
UDEV  [268640.545287] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/usbmisc/hiddev1 (usbmisc)
UDEV  [268640.546673] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011/hidraw/hidraw0 (hidraw)
UDEV  [268640.547043] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012/hidraw/hidraw1 (hidraw)
UDEV  [268640.547263] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011 (hid)
UDEV  [268640.547612] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012 (hid)
UDEV  [268640.547878] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0 (usb)
UDEV  [268640.548194] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1 (usb)
UDEV  [268640.548863] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)

strace OK on HOST output here:

Let’s compare.

FAILED case:

[pid  2683] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 19
[pid  2683] connect(19, {sa_family=AF_INET, sin_port=htons(21325), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid  2683] mprotect(0x3e03000, 245760, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
[pid  2683] mprotect(0x3e03000, 245760, PROT_READ|PROT_EXEC) = 0
[pid  2683] epoll_ctl(13, EPOLL_CTL_ADD, 19, {events=EPOLLOUT, data={u32=19, u64=19}}) = 0
[pid  2683] epoll_wait(13, [{events=EPOLLIN, data={u32=16, u64=16}}, {events=EPOLLOUT|EPOLLERR|EPOLLHUP, data={u32=19, u64=19}}], 1024, 0) = 2
[pid  2683] read(16, "\1\0\0\0\0\0\0\0", 1024) = 8
[pid  2683] mmap(NULL, 10737479680, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x7f0cbeff0000
[pid  2683] munmap(0x7f0f3eff0000, 61440) = 0
[pid  2683] mprotect(0x7f0d3eff0000, 655360, PROT_READ|PROT_WRITE) = 0
[pid  2683] getsockopt(19, SOL_SOCKET, SO_ERROR, [ECONNREFUSED], [4]) = 0
[pid  2683] epoll_ctl(13, EPOLL_CTL_DEL, 19, 0x7ffcf95c8404) = 0

SUCCESSFUL case:

[pid 4164659] connect(21, {sa_family=AF_INET, sin_port=htons(21325), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
[pid 4164659] mprotect(0x4bc3000, 245760, PROT_READ|PROT_WRITE|PROT_EXEC) = 0
[pid 4164659] mprotect(0x4bc3000, 245760, PROT_READ|PROT_EXEC) = 0
[pid 4164659] epoll_ctl(13, EPOLL_CTL_ADD, 21, {events=EPOLLOUT, data={u32=21, u64=21}}) = 0
[pid 4164659] epoll_wait(13, [{events=EPOLLIN, data={u32=16, u64=16}}, {events=EPOLLOUT|EPOLLERR|EPOLLHUP, data={u32=21, u64=21}}], 1024, 0) = 2
[pid 4164659] read(16, "\2\0\0\0\0\0\0\0", 1024) = 8
[pid 4164659] write(18, "Ledger app version 4.1.2\n", 25) = 25

They using O_NONBLOCK socket, then epoll to catch the moment when connection is established. So, on failed case we’ve got ECONNREFUSED. In successful case we’ve successful connection.

I think we you need somehow to get straces of the trezord, so we can determine why it fails.

But I don’t use any trezord.
I use ledger.
Trezor is another device.

Anyway
I wonder do I have to implement udev on both sides or on only host / guest.
Whats at stake for lxc device config ?

Also I’ve seen in OK case, call to usb/serial.
Is it allowed to do so on guest as well ?

Plus a lot of mentions of hidraw/hidraw0,
Which doesn’t seem to show on guest case,
Why ?

But I don’t use any trezord.

understood. But anyway, you have some (possibly it is named not as “trezord”) server running at 127.0.0.1: 21325. And you’ve got a failure because you haven’t this server running inside your container. I think you just need to take a look on netstat -tnlp on your host and determine which process is using TCP port 21325 and then we can take a look on ls -la /proc/<pid>/fd and see which files/device nodes it’s using.

Unfortunately, Linux kernel doesn’t providing us with something like “device namespaces”, so, usb devices will trigger events on host, but not in the container. So, you need to manually determine which device node is used by your software (that’s the way on which I’m trying to guide you (-; ) and then create this node (or pass-through it to the container).

I don’t know what to tell you. I have no such server.
Checked with netstat just to be sure and I confirm nothing.
Maybe thats a server that same binary pops up earlier in the process ?

However,
Here is udevadm monitor in guest:

KERNEL[293823.638176] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293823.640549] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293824.168109] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293824.168682] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293853.209568] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293853.240996] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293853.244145] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293853.245050] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293855.326714] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293855.327649] unbind   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293855.355775] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293855.356332] remove   /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293855.841844] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293855.913849] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
KERNEL[293855.917406] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)
UDEV  [293855.918300] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3 (usb)

As you can its a whole different thing.
Its missing a lot of lines.
Like this kind

UDEV  [268640.546673] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011/hidraw/hidraw0 (hidraw)
UDEV  [268640.547043] add      /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.1/0003:2C97:1015.0012/hidraw/hidraw1 (hidraw)
UDEV  [268640.547263] bind     /devices/pci0000:00/0000:00:14.0/usb1/1-3/1-3:1.0/0003:2C97:1015.0011 (hid)