Containers with Ubuntu 12.04.5 LTS are not getting IPv4’s

Since a few weeks one of our servers that is running LXD 6.1-efad198 has a problem with IPv4 addresses. They are not longer configured. IPv6 still works.

And if I have the exact same container(s) on another LXD servers are works fine.

I have tried multiple settings/etc but nothing helped. The firewall/ufw rules are setup as described in the docs. I even tried with disabled ufw and apparmor.

Tried it with a fresh ubuntu:12.04 image. That container does not get an IP too. But with ubuntu:24.04 it does! So something is not playing nice with older version of Ubuntu / Linux.

So then I found out that there is a new fork of lxd: incus.

I hoped that migrating to incus from lxd fixed my IP issue. It did not :frowning:

This is a tshark dump of the lxdbr0 bridge:

root@oev4 ~ # tshark -i lxdbr0 -n
Running as user “root” and group “root”. This could be dangerous.
Capturing on ‘lxdbr0’
** (tshark:2551528) 13:07:20.775129 [Main MESSAGE] – Capture started.
** (tshark:2551528) 13:07:20.775165 [Main MESSAGE] – File: “/tmp/wireshark_lxdbr0YQTRS2.pcapng”

1 0.000000000 fe80::216:3eff:feb7:7cf2 → ff02::16     ICMPv6 90 Multicast Listener Report Message v2
2 0.965008841           :: → ff02::16     ICMPv6 90 Multicast Listener Report Message v2
3 1.291009249           :: → ff02::16     ICMPv6 90 Multicast Listener Report Message v2
4 1.528012677           :: → ff02::16     ICMPv6 90 Multicast Listener Report Message v2
5 1.995019271           :: → ff02::1:ffb7:7cf2 ICMPv6 86 Neighbor Solicitation for fe80::216:3eff:feb7:7cf2
6 2.507016954           :: → ff02::16     ICMPv6 90 Multicast Listener Report Message v2
7 3.019024095 fe80::216:3eff:feb7:7cf2 → ff02::16     ICMPv6 110 Multicast Listener Report Message v2
8 3.019072867 fe80::216:3eff:feb7:7cf2 → ff02::2      ICMPv6 70 Router Solicitation from 00:16:3e:b7:7c:f2
9 3.019480723 fe80::216:3eff:feb8:849e → fe80::216:3eff:feb7:7cf2 ICMPv6 142 Router Advertisement from 00:16:3e:b8:84:9e

10 3.163210162 :: → ff02::1:ffb7:7cf2 ICMPv6 86 Neighbor Solicitation for fd42:749:f51c:e43:216:3eff:feb7:7cf2
11 3.851019116 fe80::216:3eff:feb7:7cf2 → ff02::16 ICMPv6 110 Multicast Listener Report Message v2
12 8.075031988 fe80::216:3eff:feb8:849e → fe80::216:3eff:feb7:7cf2 ICMPv6 86 Neighbor Solicitation for fe80::216:3eff:feb7:7cf2 from 00:16:3e:b8:84:9e
13 8.075077453 fe80::216:3eff:feb7:7cf2 → fe80::216:3eff:feb8:849e ICMPv6 78 Neighbor Advertisement fe80::216:3eff:feb7:7cf2 (sol)
14 13.195997432 fe80::216:3eff:feb7:7cf2 → fe80::216:3eff:feb8:849e ICMPv6 86 Neighbor Solicitation for fe80::216:3eff:feb8:849e from 00:16:3e:b7:7c:f2
15 13.196062444 fe80::216:3eff:feb8:849e → fe80::216:3eff:feb7:7cf2 ICMPv6 78 Neighbor Advertisement fe80::216:3eff:feb8:849e (rtr, sol)
16 19.550697561 00:16:3e:39:1c:e7 → ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.4.1? Tell 10.14.13.1
17 20.555010655 00:16:3e:39:1c:e7 → ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.4.1? Tell 10.14.13.1
18 21.579006380 00:16:3e:39:1c:e7 → ff:ff:ff:ff:ff:ff ARP 42 Who has 10.0.4.1? Tell 10.14.13.1

Hi!

Can you summarize which images get an IP address per Linux version of the host, and which ones do not?

Some old runtimes have an issue with newer hosts.

I cannot find the Ubuntu 12.04.4 LTS image here, Index of /releases/12.04.4/release

Can you check whether the same issue is present with Ubuntu 14.04.4 LTS from here, Ubuntu 14.04.5 LTS (Trusty Tahr) so that we can try to reproduce?

yes ubuntu 14.04.5 is working correclty. So it gets an IP adres.
Retried 12.04 from this link: Ubuntu Cloud Images - Ubuntu Server 12.04 LTS (Precise Pangolin)
That does not get an IP.

That directory is empty, Ubuntu Cloud Images - Ubuntu Server 12.04 LTS (Precise Pangolin)

How did you manually import the images into Incus? I am in mid-writing a post on how to import those images in Incus as I think it’s a somewhat involved task. If it’s too easy, I’ll switch to something else.

edit: the page Ubuntu 12.04.5 LTS (Precise Pangolin) takes long to load but in the end it loads, showing the images. This one, is empty, Index of /releases/precise/release

Its not empty but seems not in the structure that incus needs.

I installed it in lxd previously and migrated that to incus.

How can I reinstall it from ubuntu cloud with the link you found?

Regarding the core issue that you are facing, I think it is related to a recent discussion (within last two months) on this forum where old images where using some old style of cgroups vs cgroupsv2 (or AppArmor, or something like that) and this would not allow the instance to get an IPv4 address. Normally when I read the posts on this forum, I click to add a bookmark of such interesting discussions. I just searched my bookmarks in this forum and I could not find that discussion.

Let’s see how to manually add the old Ubuntu 12.04 image as a container in Incus.

We need two files found at Ubuntu 12.04.5 LTS (Precise Pangolin) Note that this site is HTTP and it may stall on some browsers. It’s the -lxd.tar.gz file with metadata and the -root.tar.xz with the rootfs / runtime.
Then, we import to Incus, and finally launch such a container.

wget http://cloud-images-archive.ubuntu.com/releases/precise/release-20170502/ubuntu-12.04-server-cloudimg-amd64-root.tar.xz
wget http://cloud-images-archive.ubuntu.com/releases/precise/release-20170502/ubuntu-12.04-server-cloudimg-amd64-lxd.tar.xz
incus image import ubuntu-12.04-server-cloudimg-amd64-lxd.tar.xz ubuntu-12.04-server-cloudimg-amd64-root.tar.xz --alias ubuntu-12.04-server
incus launch local:ubuntu-12.04-server myinstance-1204

Ok, we got this running. Does it get an IPv4 address?

$ incus list myinstance-1204 -c ns4t
+-----------------+---------+------+-----------+
|      NAME       |  STATE  | IPV4 |   TYPE    |
+-----------------+---------+------+-----------+
| myinstance-1204 | RUNNING |      | CONTAINER |
+-----------------+---------+------+-----------+
$ incus shell myinstance-1204
root@myinstance-1204:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 12.04.5 LTS
Release:	12.04
Codename:	precise
root@myinstance-1204:~# dhclient eth0
RTNETLINK answers: Operation not permitted
RTNETLINK answers: Operation not permitted
root@myinstance-1204:~# 

Now, we can reproduce the issue!

I can then just set the network settings manually (using ifconfig, route and add a nameserver in /etc/resolv.conf), and networking works!

$ incus shell myinstance-1204
root@myinstance-1204:~# ifconfig eth0 10.10.10.199
sudo: unable to resolve host myinstance-1204
root@myinstance-1204:~# route add -net default gw 10.10.10.1
root@myinstance-1204:~# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 10.10.10.1
search incus
root@myinstance-1204:~# host www.google.com
www.google.com has address 172.217.20.68
www.google.com has IPv6 address 2a00:1450:4017:812::2004
root@myinstance-1204:~# 

Super! So for now I will use hard coded IP addresses.
Hope you can fix it in incus soon so that the dhcp way works again.

Using incus 6.0.0 I am able to hack around this issue with my legacy Ubuntu 12.04 containers as described above, thanks so much!

However, the settings are lost once a container is restarted. I can live with static addresses, but would love to the settings become permanent. What would be the best way to achieve that?

Can later versions of incus deal with these 12 year old DHCP requests? :smile:

Incus is also an educational tool, that helps learn about networks and what not.
Let’s learn!

First, we create a network incusbr1, so that it is cleaner when we sudo tshark -r incusbr1. No packets from other managed interfaces.

$ incus network create incusbr1 ipv4.address=10.10.12.1/24 ipv4.nat=true ipv6.address=none ipv6.nat=false
Network incusbr1 created
$ incus network show incusbr1
config:
  ipv4.address: 10.10.12.1/24
  ipv4.nat: "true"
  ipv6.address: none
  ipv6.nat: "false"
description: ""
name: incusbr1
type: bridge
used_by: []
managed: true
status: Created
locations:
- none
project: default
$ 

Then, we create a fresh new Ubuntu 12.04 instance. It assumes that your Ubuntu 12.04 image is stored locally, and is called ubuntu-from-canonical-12.04.

$ incus create ubuntu-from-canonical-12.04 myinstance-1204 --network incusbr1
Creating myinstance-1204
$ 

Then, on a separate terminal, we run (and keep running) the following.

$ sudo tshark -i incusbr1
Running as user "root" and group "root". This could be dangerous.
Capturing on 'incusbr1'
 ** (tshark:88801) 14:36:26.891459 [Main MESSAGE] -- Capture started.
 ** (tshark:88801) 14:36:26.891522 [Main MESSAGE] -- File: "/tmp/wireshark_incusbr17Y81U2.pcapng"

Subsequently, on the initial terminal window, we start the container myinstance-1204. This instance did its network initialization, and we switch terminal window to harvest the results.

$ incus start myinstance-1204
$ incus list myinstance-1204 -c ns4
+-----------------+---------+------+
|      NAME       |  STATE  | IPV4 |
+-----------------+---------+------+
| myinstance-1204 | RUNNING |      |
+-----------------+---------+------+
$ sudo tshark -i incusbr1
Running as user "root" and group "root". This could be dangerous.
Capturing on 'incusbr1'
 ** (tshark:88801) 14:36:26.891459 [Main MESSAGE] -- Capture started.
 ** (tshark:88801) 14:36:26.891522 [Main MESSAGE] -- File: "/tmp/wireshark_incusbr17Y81U2.pcapng"
    1 0.000000000           :: → ff02::16     ICMPv6 90 Multicast Listener Report Message v2
    2 0.302373430      0.0.0.0 → 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x97c8946
    3 0.302517451   10.10.12.1 → 10.10.12.111 DHCP 358 DHCP Offer    - Transaction ID 0x97c8946
    4 0.302579237      0.0.0.0 → 255.255.255.255 DHCP 342 DHCP Request  - Transaction ID 0x97c8946
    5 0.304688165   10.10.12.1 → 10.10.12.111 DHCP 358 DHCP ACK      - Transaction ID 0x97c8946

Wow, what’s with that DHCP ACK? Did it…? Let’s see. Yep, it did!

$ incus exec myinstance-1204 -- cat /var/lib/dhcp/dhclient.eth0.leases
lease {
  interface "eth0";
  fixed-address 10.10.12.145;
  option subnet-mask 255.255.255.0;
  option routers 10.10.12.1;
  option dhcp-lease-time 3600;
  option dhcp-message-type 5;
  option domain-name-servers 10.10.12.1;
  option dhcp-server-identifier 10.10.12.1;
  option dhcp-renewal-time 1800;
  option broadcast-address 10.10.12.255;
  option dhcp-rebinding-time 3150;
  option host-name "myinstance-1204";
  option domain-name "incus";
  renew 3 2024/09/25 12:25:47;
  rebind 3 2024/09/25 12:48:52;
  expire 3 2024/09/25 12:56:22;
}
$ 

But did the instance set the interface’s IP address? Nope.

$ incus exec myinstance-1204 -- ifconfig
eth0      Link encap:Ethernet  HWaddr 00:16:3e:0a:9e:d6  
          inet6 addr: fe80::216:1eff:fe1a:c0ed/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:842 (842.0 B)  TX bytes:1480 (1.4 KB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

$

Verdict? Something is wrong inside the Ubuntu 12.04 runtime and the instance does not set the IPv4 address properly. It gets the DHCP lease but does not set it.
Probably this would require a bit more investigation and the problem in the Ubuntu 12.04 runtime can be identified. We are so close to figuring this out. Anyone can take this up and reach the end of the investigation.

For now though, you can add some script in the instance to manually ifconfig the IPv4 allocation.

For reference, the previous discussion about legacy containers not getting IPV4 via DHCP — the technique is described for LXD but can be applied to incus just the same and for me works just fine for networking of certain legacy containers. The issue with Ubuntu 12.04 in this thread is independent of this.

1 Like

Thanks for this! In summary, modern versions of Linux use nftables:

$ incus info | grep firewall:
  firewall: nftables
$

If the runtime is too old, then it is configured to expect xtables. This, by itself, should not cause a problem. However, the linked post says that the old runtime finds UDP checksums in the DHCP packets and refuses to use them.

In my tests however, I get the following error, RTNETLINK answers: Operation not permitted.

root@myinstance-1204:~# dhclient3 -v -e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1 eth0
Internet Systems Consortium DHCP Client 4.1-ESV-R4
Copyright 2004-2011 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

RTNETLINK answers: Operation not permitted
Listening on LPF/eth0/00:16:3e:0a:9e:d6
Sending on   LPF/eth0/00:16:3e:0a:9e:d6
Sending on   Socket/fallback
DHCPREQUEST of 10.10.12.145 on eth0 to 255.255.255.255 port 67
DHCPACK of 10.10.12.145 from 10.10.12.1
RTNETLINK answers: Operation not permitted
bound to 10.10.12.145 -- renewal in 1783 seconds.
root@myinstance-1204:~#

I posted this problem with the lxd team too. Someone found a solution/work-around. The problem seems to be that Apparmor blocks dhclient.

dhclient3 eth0
RTNETLINK answers: Operation not permitted
Sep 05 12:01:09  kernel: audit: type=1400 audit(1725534069.603:228): apparmor="DENIED" operation="capable" class="cap" namespace="root//lxd-c1_<var-lib-lxd>" profile="/sbin/dhclient" pid=28122 comm="ip" capability=12  capname="net_admin"

See:

The work-around is to disable Apparmor for dhclient:

$ lxc exec c1 -- ln -sf /etc/apparmor.d/sbin.dhclient /etc/apparmor.d/disable/

For me that solved this issue. Even keeps working after rebooting the container (or the server).

Thanks for this.

Let’s convert it into an Incus solution. In this case I just add the capability net_cap that is missing from the AppArmor profile and restart the container.

$ incus launch ubuntu-from-canonical-12.04 myinstance-1204
Launching myinstance-1204
$ incus list myinstance-1204 -c ns4
+-----------------+---------+------+
|      NAME       |  STATE  | IPV4 |
+-----------------+---------+------+
| myinstance-1204 | RUNNING |      |
+-----------------+---------+------+
$ incus exec myinstance-1204 -- /bin/sed -i '/capability net_raw,/a  capability net_cap,' /etc/apparmor.d/sbin.dhclient
$ incus restart myinstance-1204
$ incus list myinstance-1204 -c ns4
+-----------------+---------+---------------------+
|      NAME       |  STATE  |        IPV4         |
+-----------------+---------+---------------------+
| myinstance-1204 | RUNNING | 10.10.10.117 (eth0) |
+-----------------+---------+---------------------+
$

Okay cool. But the lxc solution worked for me too on incus :slight_smile:

On the one had you have the nuclear route that disables all of AppArmor in the container, and on the other you just give a single extra capability to the container, the NET_CAP capability.

Thanks thats a beter solution