How to properly and fully pass USB modem into the container?

,
1

Greetings,

Goal: to run a fully featured OpenWRT within LXC with USB 5G modem as the uplink
Env: an up-to-date latest Proxmox (Debian 12)
The issue: I am unable to properly/completely pass USB modem to the container.

I am aware this is a solely LXC-related forum, but I had very little luck finding any resources to help me solve this issue.

My current Proxmox LXC container config is as follows:

arch: amd64
cores: 2
features: mknod=1
hostname: openwrt
memory: 1024
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:0E:D1:B1,type=veth
ostype: unmanaged
rootfs: local-lvm:vm-666-disk-0,size=8G
swap: 0
tty: 1
unprivileged: 1
lxc.idmap: u 0 100000 65536
lxc.idmap: g 0 100000 65536
lxc.mount.auto: proc:mixed sys:rw cgroup:mixed
lxc.cgroup2.devices.allow: c 180:* rwm
lxc.cgroup2.devices.allow: c 188:* rwm
lxc.mount.entry: /dev/bus/usb dev/bus/usb none bind,optional,create=dir
lxc.mount.entry: /dev/ttyUSB0 dev/ttyUSB0 none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB1 dev/ttyUSB1 none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB2 dev/ttyUSB2 none bind,optional,create=file
lxc.mount.entry: /dev/ttyUSB3 dev/ttyUSB3 none bind,optional,create=file
lxc.mount.entry: /dev/cdc-wdm0 dev/cdc-wdm0 none bind,optional,create=file
lxc.mount.entry: /sys/devices/pci0000:00/0000:00:13.0/ sys/devices/pci0000:00/0000:00:13.0/ none bind,optional,create=dir
lxc.net.1.name: wwan0
lxc.net.1.type: phys
lxc.net.1.link: wwan0

I am aware it’s a mixed bag of different options, but I kept adding to the configuration, seeing whether anything works. This config requires me also to add o+rw to abovementioned /dev entries (to be replaced with an appropriate udev rule once I have any results)

I’ve also tried using ArchLinux container with pretty much identical setup - while the devices are visible within the container, I can talk to the modem (and the ISP) using qmicli but ModemManager says that there are no modems detected.

Please advise. What am I missing? It feels I’m very close to the soluion, but I lack expertise with LXC and its configurations.

Resources I used so far:

https://lists.linuxcontainers.org/pipermail/lxc-users/2020-September/015337.html
https://forum.turris.cz/t/lxc-container-and-attach-usb-devices/4412/18
https://gist.github.com/crundberg/a77b22de856e92a7e14c81f40e7a74bd
https://forum.proxmox.com/threads/p...containers-in-proxmox-7-2.109981/#post-601753
https://gist.github.com/dragonfire1119/f3acd42414abfff762d2dad5e50f7cc6
https://forum.proxmox.com/threads/h...rsion-of-openwrt-and-run-it-on-proxmox.64786/
https://openwrt.org/docs/guide-user/virtualization/lxc#upgrading_to_the_latest_release
2

Hi!
Indeed this forum is for LXC and Incus.

If you were to use Incus, then

  1. There are already available images for OpenWRT for Incus. Several architectures.
  2. You would use USB passthrough to move the USB modem to the container, i.e. incus config device add myopenwrt myusbmodem usb vendorid=1234 productid=5678
$ incus image list images:openwrt -c lad
+---------------------------------+--------------+-----------------------------------------+
|              ALIAS              | ARCHITECTURE |               DESCRIPTION               |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/21.02 (3 more)          | x86_64       | Openwrt 21.02 amd64 (20240219_11:57)    |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/21.02/arm64 (1 more)    | aarch64      | Openwrt 21.02 arm64 (20240219_12:01)    |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/22.03 (3 more)          | x86_64       | Openwrt 22.03 amd64 (20240219_11:57)    |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/22.03/arm64 (1 more)    | aarch64      | Openwrt 22.03 arm64 (20240219_12:01)    |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/23.05 (3 more)          | x86_64       | Openwrt 23.05 amd64 (20240219_11:57)    |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/23.05/arm64 (1 more)    | aarch64      | Openwrt 23.05 arm64 (20240219_12:00)    |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/snapshot (3 more)       | x86_64       | Openwrt snapshot amd64 (20240219_11:57) |
+---------------------------------+--------------+-----------------------------------------+
| openwrt/snapshot/arm64 (1 more) | aarch64      | Openwrt snapshot arm64 (20240219_12:01) |
+---------------------------------+--------------+-----------------------------------------+
$
3

I’m aware this is an LXC forum, thus I’m looking for an advice regarding the LXC component of my whole stack - I assume running the incus component is not mandatory for getting an answer on this forum? :wink:

So the question would be - what are necessary devices/options/components/configurations/capabilities passed to an LXC container in order for it to be able create and operate on network interfaces internally?

What would be the process of passing an entire USB device to the container without incus, in this case?

4

Yeah, so it’s going to mostly depend on the modem used.
For low bandwidth type stuff, I suspect you basically just need access to:

  • /dev/ttyUSBX device for the modem
  • /dev/ppp device for PPP to work

For that, you need both lxc.mount.entry and the matching lxc.cgroup2.devices.allow to allow interacting with the device.

But that’s really for the basic 3G/4G type modems, higher throughput modems likely behave very differently and use different device types. Your config above suggests some PCI stuff, custom /dev devices and network interfaces, I don’t have experience with that and can’t comment on how well any of this is likely to work inside of a container.

In general, passing /dev/bus/usb or /sys/devices/pci… type stuff to a container isn’t going to really work, at least not in a useful way. What the container can make use of are the resulting devices generated by the kernel from those raw USB and PCI devices, so that’s your /dev/ttyUSBX type devices, traditional network interfaces, other /dev/ char/block devices, …

I’d recommend getting a very good idea of how things work outside of a container first, what process needs to be run, what devices and interfaces does that process interact with, … then once you understand that, pass those to the container.

For this kind of stuff, strace is usually your friend.

5

It’s an USB 5G modem (4Gbit capable, ISP network is capable of ~2Gbit from my own tests), it exposes /dev/ttyUSB* and /dev/cdc-wdm0. I have passed both of these to the test containers already. For ArchLinux container - I can talk to all the devices manually and using qmicli, but ModemManager says there’s no modems available. For OpenWRT container I can talk to the device manually using uqmi, I can talk to the ISP as well, but OpenWRT is unable to create interfaces like below:

Fri Feb 16 17:38:12 2024 daemon.notice netifd: Interface 'T_Mobile' is setting up now
Fri Feb 16 17:38:12 2024 daemon.notice netifd: T_Mobile (925): The interface could not be found.
Fri Feb 16 17:38:12 2024 daemon.notice netifd: T_Mobile (1007): Stopping network T_Mobile
Fri Feb 16 17:38:12 2024 daemon.notice netifd: T_Mobile (1007): Command failed: ubus call network.interface notify_proto { "action": 0, "link-up": false, "keep": false, "interface": "T_Mobile" } (Permission denied)
Fri Feb 16 17:38:12 2024 daemon.notice netifd: Interface 'T_Mobile' is now down

It feels it’s something related to the networking ability within the container, but I’m way too inexperienced with LXC. I have tried following this [lxc-users] Any experiences with modemmanager in unprivilegied container ? but with no luck.

6

Okay, NetworkManager is most likely because it relies on the udev database or something which isn’t likely to have a proper device entry inside of a container, so that’s not too surprising.

The permission denied you’re getting on openwrt is more interesting and it’d be interesting to make use of strace to understand exactly what call is getting that Permission denied so we can get an idea of whether it’s something that’s easily fixable or not (as in, needs some extra device or permission fixes as opposed to kernel development work).