Fresh installation of 6.11 can't create networks

Incus
1

Hi!

I’ve formatted and reinstalled an nvidia jetson orin nano (I needed to get off the sdcard installation and move to an nvme installation). During the time it took me to do that (no thanks to nvidia) incus 6.11 was released! I’ve just installed it, but when trying to run incus admin init, it errored out during the network creation phase:

root@nub:/etc/apt/keyrings# incus list
If this is your first time running Incus on this machine, you should also run: incus admin init
To start your first container, try: incus launch images:ubuntu/22.04
Or for a virtual machine: incus launch images:ubuntu/22.04 --vm

+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+
root@nub:/etc/apt/keyrings# incus admin init
Would you like to use clustering? (yes/no) [default=no]: no
Do you want to configure a new storage pool? (yes/no) [default=yes]: yes
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir, lvm, lvmcluster) [default=dir]:
Where should this storage pool store its data? [default=/var/lib/incus/storage-pools/default]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=incusbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 10.0.5.1/24
Would you like to NAT IPv4 traffic on your bridge? [default=yes]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: none
Would you like the server to be available over the network? (yes/no) [default=no]: yes
Address to bind to (not including port) [default=all]:
Port to bind to [default=8443]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "init" preseed to be printed? (yes/no) [default=no]:
Error: Failed to create local member network "incusbr0" in project "default": Failed to setup firewall: Failed adding outbound NAT rules for network "incusbr0" (inet): Failed apply nftables config: Failed to run: nft -f -: exit status 1 (/dev/stdin:4:7-9: Error: No such file or directory; did you mean chain ‘pstrt.incusbr0’ in table inet ‘incus’?
	type nat hook postrouting priority 100; policy accept;
	     ^^^
/dev/stdin:3:7-20: Error: Could not process rule: No such file or directory
chain pstrt.incusbr0 {
      ^^^^^^^^^^^^^^)

I tried to run the network creation command by hand afterwards to see if I could get any more details, but that also errored out:

root@nub:/etc/apt/keyrings# incus network create incusbr0 --type bridge
Error: Failed to setup firewall: Failed adding outbound NAT rules for network "incusbr0" (inet): Failed apply nftables config: Failed to run: nft -f -: exit status 1 (/dev/stdin:4:7-9: Error: No such file or directory; did you mean chain ‘pstrt.incusbr0’ in table inet ‘incus’?
	type nat hook postrouting priority 100; policy accept;
	     ^^^
/dev/stdin:3:7-20: Error: Could not process rule: No such file or directory
chain pstrt.incusbr0 {
      ^^^^^^^^^^^^^^
/dev/stdin:3:7-20: Error: Could not process rule: No such file or directory
chain pstrt.incusbr0 {
      ^^^^^^^^^^^^^^)

Here’s some more networking output that might be helpful:

root@nub:/etc/apt/keyrings# uname -a
Linux nub 5.15.148-tegra #1 SMP PREEMPT Tue Jan 7 17:14:38 PST 2025 aarch64 aarch64 aarch64 GNU/Linux
root@nub:/etc/apt/keyrings# cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
root@nub:/etc/apt/keyrings# nft list ruleset
table ip nat {
}
table ip filter {
}
root@nub:/etc/apt/keyrings# iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

root@nub:/etc/apt/keyrings# ufw status
Status: inactive

I’m not sure how to proceed at this point - though it appears that perhaps there might be a typo or some kind of syntax error maybe in the scripting under the hood that creates the bridge?

Happy to take any instruction here to see what happens!

2

The Ubuntu kernel on the NVIDIA jetson is missing some critical kernel modules required for basic NAT to function.

You either need to switch to a generic Ubuntu kernel which would fix that, or get the Canonical kernel team to include the missing modules.

3

The first thing I did when I signed up for deepseek is to ask for instructions of how to install incus on nvidia orin. But I haven’t gotten a chance to try it out myself. Since I am not able to upload the .pdf file, I am copy-pasting whatever that deepseek provided.

NOTE: PLEASE USE IT WITH YOUR OWN DISCRETION

25th Jan 2025

Steps to Run Incus on NVIDIA Jetson Orin Nano

  1. Prepare the Jetson Orin Nano
    • Flash the Device: Ensure your Jetson Orin Nano is running a compatible Linux distribution (e.g., NVIDIA JetPack SDK based on Ubuntu 20.04 or 22.04).
    • Update the System:
      sudo apt update
      sudo apt upgrade -y
      sudo reboot

  1. Install Incus Dependencies
    Incus requires certain dependencies to function properly. Install them using the following commands:
    sudo apt install -y snapd curl uidmap

  2. Install Incus
    Incus is distributed as a snap package, which simplifies installation on Ubuntu-based systems.
    • Install Snap (if not already installed): sudo apt install snapd
    • Install Incus:
    sudo snap install incus
    • Add Incus to Your PATH: export PATH=$PATH:/snap/bin

  3. Initialize Incus
    After installation, you need to initialize Incus:
    sudo incus init
    This will guide you through the setup process, including configuring storage, networking, and other options.

  4. Verify Incus Installation
    Check if Incus is running correctly:
    incus list
    If everything is set up properly, this command should return an empty list (no containers or VMs running yet).

  5. Configure Incus for ARM Architecture
    Since the Jetson Orin Nano uses an ARM64 architecture, you may need to ensure that the container images you use are compatible with ARM. Incus supports multi-architecture images, but you should verify compatibility.
    • Pull an ARM-Compatible Image:
    incus launch images:ubuntu/22.04 my-container
    Replace ubuntu/22.04 with any other ARM-compatible image if needed.

  6. (Optional) Enable GPU Acceleration
    If you want to leverage the NVIDIA GPU for GPU-accelerated workloads in containers, you’ll need to install the NVIDIA Container Toolkit.
    • Install NVIDIA Container Toolkit:

      distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
          && curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo
      apt-key add - \
          && curl -s -L
      https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-
      container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-
      toolkit.list
      sudo apt update
      sudo apt install -y nvidia-container-toolkit

• Configure Incus to Use NVIDIA GPU:
Edit the Incus configuration to enable GPU passthrough:
incus config set my-container nvidia.runtime true

  1. Test Your Setup
    • Launch a Container:
    incus launch images:ubuntu/22.04 my-container
    • Access the Container:
    incus exec my-container -- /bin/bash
    • Verify GPU Access (if enabled): Inside the container, run:
    nvidia-smi
    This should display GPU information if the NVIDIA Container Toolkit is configured correctly.

  2. Manage Incus
    • Start/Stop Containers:

      incus start my-container
      incus stop my-container

• Delete a Container:
incus delete my-container
• MonitorResources:
incus info

  1. Troubleshooting
    • ARM Compatibility Issues: Ensure you are using ARM-compatible images. Check the image
    repository for ARM support.
    • GPU Passthrough Issues: Verify that the NVIDIA Container Toolkit is installed and configured correctly.
    • Networking Issues: Use incus network list to check network configurations.
4

I made attempts to compile a fresh kernel and include the required modules, but ran into trouble and now the thing wont boot. sadly, with these things its not as easy as just going into uefi and copying a few files around. I’ve had to order an external nvme drive adapter so that I can mount the thing from another machine and edit stuff that way. if I get it working, I’ll post how I did it!

it seems that the choices here are

  1. no access to the nvidia gpus, but a good kernel with networking built in
  2. the ‘tegra’ kernel, with all the cool toys, but no networking

seems pretty silly for nvidia to ship a kernel for a ‘development board’ with like, basic networking removed.

5

I am still fussing around with trying to compile a custom kernel for the jetson orin nano, and now running into issues with modules that don’t want to cooperate.

is it possible to get a list of what nftables kernel modules incus actually needs, so i can install JUST THOSE? I suspect this will make things a lot easier for me

thank you in advance!

6

I think i figured it out - it only wants ip_nf_raw.
i was able to insert the prebuilt kernel module from the GitHub - jetsonhacks/jetson-orin-kernel-builder: Build the Linux kernel and modules on board the Jetson AGX Orin, Orin Nano or Orin NX repository and then i was able to get past the nftable errors.

now i am trying to completely remove incus and do a clean installation from scratch, because the first few attempts I made were unsuccessful. Is there an easy way to completely remove incus? im trying apt-get --purge remove incus incus-ui-canonical but its leaving behind tons of stuff and running systemd things and i keep stepping on myself. now when i run any incus command it just sorta hangs, i cant even run incus admin init!

7

looks like i spoke too soon… I’ve got a bunch of kernel modules inserted, but since i cant tell which ones i need, im sorta shooting in the dark:

root@nub:~# lsmod | grep -E 'nf_tables|nft_nat|nft_masq|nf_nat|nf_conntrack'
nf_nat                 45056  2 xt_nat,iptable_nat
nf_conntrack          122880  2 nf_nat,xt_nat
nf_defrag_ipv6         24576  1 nf_conntrack
nf_defrag_ipv4         16384  1 nf_conntrack
nf_tables             217088  0
libcrc32c              16384  3 nf_conntrack,nf_nat,nf_tables
nfnetlink              20480  2 nf_tables
root@nub:~# incus network create incusbr0 --type bridge
Error: Failed to setup firewall: Failed adding outbound NAT rules for network "incusbr0" (inet): Failed apply nftables config: Failed to run: nft -f -: exit status 1 (/dev/stdin:4:7-9: Error: No such file or directory; did you mean chain ‘pstrt.incusbr0’ in table inet ‘incus’?
	type nat hook postrouting priority 100; policy accept;
	     ^^^
/dev/stdin:3:7-20: Error: Could not process rule: No such file or directory
chain pstrt.incusbr0 {
      ^^^^^^^^^^^^^^
/dev/stdin:3:7-20: Error: Could not process rule: No such file or directory
chain pstrt.incusbr0 {
      ^^^^^^^^^^^^^^)
root@nub:~#
8

You’re missing nft_chain_nat