The bug is not that it does not get a DHCP on boot. It is that it never gets renewed. I think the topic is wrong. That’s why I was showing the incus dhcp client for OCI seems to use 100% CPU.
Are all cases above about the non-renewal of the DHCP lease? If not, the thread should be split or a new thread needs to appear.
I remember reading about this a few months back. Are we able to set up a system (create a VM in Incus, in the VM install Incus, somehow set the DHCP lease renewal to 2-3 minutes, launch a container, boom!) that demonstrates the issue? Does it happen all the time?
Not that i know of, but it was working on the older release of incus (6.9)
nft list ruleset
table inet nixos-fw {
set temp-ports {
type inet_proto . inet_service
flags interval
comment "Temporarily opened ports"
}
chain rpfilter {
type filter hook prerouting priority mangle + 10; policy drop;
meta nfproto ipv4 udp sport . udp dport { 68 . 67, 67 . 68 } accept comment "DHCPv4 client/server"
fib saddr . mark . iif oif exists accept
jump rpfilter-allow
}
chain rpfilter-allow {
}
chain input {
type filter hook input priority filter; policy drop;
iifname { "lo", "incusbr0" } accept comment "trusted interfaces"
ct state vmap { invalid : drop, established : accept, related : accept, new : jump input-allow, untracked : jump input-allow }
tcp flags & (fin | syn | rst | ack) == syn log prefix "refused connection: " level info
}
chain input-allow {
tcp dport 22 accept
meta l4proto . th dport @temp-ports accept
icmp type echo-request accept comment "allow ping"
icmpv6 type != { nd-redirect, 139 } accept comment "Accept all ICMPv6 messages except redirects and node information queries (type 139). See RFC 4890, section 4.4."
ip6 daddr fe80::/64 udp dport 546 accept comment "DHCPv6 client"
}
}
table inet incus {
chain pstrt.incusbr0 {
type nat hook postrouting priority srcnat; policy accept;
ip saddr 10.0.100.0/24 ip daddr != 10.0.100.0/24 masquerade
ip6 saddr fd42:3c0f:f032:91e3::/64 ip6 daddr != fd42:3c0f:f032:91e3::/64 masquerade
}
chain fwd.incusbr0 {
type filter hook forward priority filter; policy accept;
ip version 4 oifname "incusbr0" accept
ip version 4 iifname "incusbr0" accept
ip6 version 6 oifname "incusbr0" accept
ip6 version 6 iifname "incusbr0" accept
}
chain in.incusbr0 {
type filter hook input priority filter; policy accept;
iifname "incusbr0" tcp dport 53 accept
iifname "incusbr0" udp dport 53 accept
iifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
iifname "incusbr0" udp dport 67 accept
iifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, nd-router-solicit, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
iifname "incusbr0" udp dport 547 accept
}
chain out.incusbr0 {
type filter hook output priority filter; policy accept;
oifname "incusbr0" tcp sport 53 accept
oifname "incusbr0" udp sport 53 accept
oifname "incusbr0" icmp type { destination-unreachable, time-exceeded, parameter-problem } accept
oifname "incusbr0" udp sport 67 accept
oifname "incusbr0" icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert, mld2-listener-report } accept
oifname "incusbr0" udp sport 547 accept
}
}
I think you’re having a different issue then we do.
Your log has “no matching response packet received” with 100% CPU
We have “Giving up on DHCP, couldn’t bring up interface” with, at least for me, no CPU usage
VM debian/13 with incus installed seems to work, its getting an ip and renewals
So it seems this is a NixOS issue? I was also running on NixOS when I had this issue, I did not test any other distro.
As a simple workaround/hack for this I simply reverted OCI improvements by stgraber · Pull Request #1873 · lxc/incus · GitHub with a patch:
{
incus
}:
incus.overrideAttrs (finalAttrs: previousAttrs: {
pname = previousAttrs.pname + "-patched";
patches =
previousAttrs.patches
++ [
./revert.patch
];
})
From 25d0666bd5415a186d1fc6dcce3804b1193ec514 Mon Sep 17 00:00:00 2001
From: Saturn745 <codeberg.sw1mx@slmail.me>
Date: Mon, 12 May 2025 18:17:22 -0700
Subject: [PATCH 1/3] Revert "incusd/main_forknet: Handle missing DNS in DHCP
response"
This reverts commit ef4063ef20d63a2b3151e6a29ef5b0a6a0755f9b.
---
cmd/incusd/main_forknet.go | 52 ++++++++++++++++++--------------------
1 file changed, 25 insertions(+), 27 deletions(-)
diff --git a/cmd/incusd/main_forknet.go b/cmd/incusd/main_forknet.go
index acad9f25e..9d0d60f74 100644
--- a/cmd/incusd/main_forknet.go
+++ b/cmd/incusd/main_forknet.go
@@ -351,43 +351,41 @@ func (c *cmdForknet) RunDHCP(cmd *cobra.Command, args []string) error {
return nil
}
- if lease.Offer.YourIPAddr == nil || lease.Offer.YourIPAddr.Equal(net.IPv4zero) || lease.Offer.SubnetMask() == nil || len(lease.Offer.Router()) != 1 {
- logger.Error("Giving up on DHCP, lease didn't contain required fields")
+ if lease.Offer.YourIPAddr == nil || lease.Offer.YourIPAddr.Equal(net.IPv4zero) || lease.Offer.SubnetMask() == nil || len(lease.Offer.Router()) != 1 || len(lease.Offer.DNS()) < 1 {
+ fmt.Fprintf(os.Stderr, "Giving up on DHCP, lease for %q didn't contain required fields\n", iface)
return nil
}
- if len(lease.Offer.DNS()) > 0 {
- // DNS configuration.
- f, err := os.Create(filepath.Join(args[0], "resolv.conf"))
+ // DNS configuration.
+ f, err := os.Create(filepath.Join(args[0], "resolv.conf"))
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "Giving up on DHCP, couldn't prepare resolv.conf: %v\n", err)
+ return nil
+ }
+
+ defer f.Close()
+
+ for _, nameserver := range lease.Offer.DNS() {
+ _, err = f.Write([]byte(fmt.Sprintf("nameserver %s\n", nameserver)))
if err != nil {
logger.WithError(err).Error("Giving up on DHCP, couldn't create resolv.conf")
return nil
}
+ }
- defer f.Close()
-
- for _, nameserver := range lease.Offer.DNS() {
- _, err = fmt.Fprintf(f, "nameserver %s\n", nameserver)
- if err != nil {
- logger.WithError(err).Error("Giving up on DHCP, couldn't prepare resolv.conf")
- return nil
- }
- }
-
- if lease.Offer.DomainName() != "" {
- _, err = fmt.Fprintf(f, "domain %s\n", lease.Offer.DomainName())
- if err != nil {
- logger.WithError(err).Error("Giving up on DHCP, couldn't prepare resolv.conf")
- return nil
- }
+ if lease.Offer.DomainName() != "" {
+ _, err = f.Write([]byte(fmt.Sprintf("domain %s\n", lease.Offer.DomainName())))
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "Giving up on DHCP, couldn't prepare resolv.conf: %v\n", err)
+ return nil
}
+ }
- if lease.Offer.DomainSearch() != nil && len(lease.Offer.DomainSearch().Labels) > 0 {
- _, err = fmt.Fprintf(f, "search %s\n", strings.Join(lease.Offer.DomainSearch().Labels, ", "))
- if err != nil {
- logger.WithError(err).Error("Giving up on DHCP, couldn't prepare resolv.conf")
- return nil
- }
+ if lease.Offer.DomainSearch() != nil && len(lease.Offer.DomainSearch().Labels) > 0 {
+ _, err = f.Write([]byte(fmt.Sprintf("search %s\n", strings.Join(lease.Offer.DomainSearch().Labels, ", "))))
+ if err != nil {
+ fmt.Fprintf(os.Stderr, "Giving up on DHCP, couldn't prepare resolv.conf: %v\n", err)
+ return nil
}
}
--
2.49.0
From cb354b231dad8d9fe10f20631199763dacef278f Mon Sep 17 00:00:00 2001
From: Saturn745 <codeberg.sw1mx@slmail.me>
Date: Mon, 12 May 2025 18:17:51 -0700
Subject: [PATCH 2/3] Revert "incusd/main_forknet: Handle missing DNS in DHCP
response"
This reverts commit ef4063ef20d63a2b3151e6a29ef5b0a6a0755f9b.
---
cmd/incusd/main_forknet.go | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/cmd/incusd/main_forknet.go b/cmd/incusd/main_forknet.go
index 9d0d60f74..1fabb6437 100644
--- a/cmd/incusd/main_forknet.go
+++ b/cmd/incusd/main_forknet.go
@@ -368,7 +368,7 @@ func (c *cmdForknet) RunDHCP(cmd *cobra.Command, args []string) error {
for _, nameserver := range lease.Offer.DNS() {
_, err = f.Write([]byte(fmt.Sprintf("nameserver %s\n", nameserver)))
if err != nil {
- logger.WithError(err).Error("Giving up on DHCP, couldn't create resolv.conf")
+ fmt.Fprintf(os.Stderr, "Giving up on DHCP, couldn't prepare resolv.conf: %v\n", err)
return nil
}
}
--
2.49.0
From b729c6b26a907f39616eed1a1d0e96170328e416 Mon Sep 17 00:00:00 2001
From: Saturn745 <codeberg.sw1mx@slmail.me>
Date: Mon, 12 May 2025 18:19:37 -0700
Subject: [PATCH 3/3] Revert "Merge pull request #1873 from stgraber/oci"
This reverts commit 16118c2c396a3d9f7c14f52f14f387d9c661cf5d, reversing
changes made to a7edf9d4a2095b7798394f0465c5541de96d5767.
---
.../server/instance/drivers/driver_lxc.go | 42 ++++---------------
1 file changed, 9 insertions(+), 33 deletions(-)
diff --git a/internal/server/instance/drivers/driver_lxc.go b/internal/server/instance/drivers/driver_lxc.go
index 910c80001..53fd22302 100644
--- a/internal/server/instance/drivers/driver_lxc.go
+++ b/internal/server/instance/drivers/driver_lxc.go
@@ -2487,11 +2487,17 @@ ff02::2 ip6-allrouters
return "", nil, err
}
- f, err := os.OpenFile(filepath.Join(d.Path(), "network", "resolv.conf"), os.O_RDWR|os.O_CREATE, 0o644)
+ f, err := os.Create(filepath.Join(d.Path(), "network", "resolv.conf"))
if err != nil {
return "", nil, err
}
+ err = f.Chmod(0o644)
+ if err != nil {
+ f.Close()
+ return "", nil, err
+ }
+
f.Close()
err = lxcSetConfigItem(cc, "lxc.mount.entry", fmt.Sprintf("%s etc/resolv.conf none bind,create=file", filepath.Join(d.Path(), "network", "resolv.conf")))
@@ -2798,38 +2804,8 @@ func (d *lxc) Start(stateful bool) error {
name := project.Instance(d.Project().Name, d.name)
- // Setup minimal environment for forkstart.
- envDict := map[string]string{
- "container": "lxc",
- }
-
- for k, v := range d.expandedConfig {
- if strings.HasPrefix(k, "environment.") {
- envDict[strings.TrimPrefix(k, "environment.")] = v
- }
- }
-
- for _, keepEnv := range []string{"LD_LIBRARY_PATH", "INCUS_DIR", "INCUS_SOCKET"} {
- if os.Getenv(keepEnv) != "" {
- envDict[keepEnv] = os.Getenv(keepEnv)
- }
- }
-
- _, ok := envDict["PATH"]
- if !ok {
- envDict["PATH"] = os.Getenv("PATH")
- }
-
- env := make([]string, 0, len(envDict))
- for k, v := range envDict {
- env = append(env, fmt.Sprintf("%s=%s", k, v))
- }
-
- // Start the LXC container.
- _, _, err = subprocess.RunCommandSplit(
- context.TODO(),
- env,
- nil,
+ // Start the LXC container
+ _, err = subprocess.RunCommand(
d.state.OS.ExecPath,
"forkstart",
name,
--
2.49.0
I am still unsure which commit exactly in that PR caused it and not able to debug further at this time.
2 Likes
@Saturn745 huge thanks. I spent couple hours debugging yesterday what went wrong and adding udhcpc services etc to all my containers before finding this thread and realizing that problem is in Incus (or NixOS?) itself. I didn’t have time to get into the code, but your patching approach worked for me too.
I recently ran into this after updating Incus on my Ubuntu server machines. My Open WebUI app container is not pulling an IPv4.
Hi, I don’t know if this is relevant to anyone with this issue, but just in case … this week I had a problem with DHCP while experimenting. All of a sudden DHCP just didn’t appear to work, or if it did it worked “eventually” maybe after 30 seconds to a minute.
Turns out I enabled STP one of my bridges. Container startups seems to put the bridge into ‘looking’ mode which delayed or dropped the initial DHCP requests. Disabling STP fixed the problem immediately.
STP was/is not enabled on my end. Might be something else 
Thanks a lot for finding this, I was close to tearing my hair out over this issue.
I have reduced the necessary patch to the following, which is enough to get things working again, and also made it work for 6.14:
diff --git a/internal/server/instance/drivers/driver_lxc.go b/internal/server/instance/drivers/driver_lxc.go
index 8b974f5a7..0d0ec498c 100644
--- a/internal/server/instance/drivers/driver_lxc.go
+++ b/internal/server/instance/drivers/driver_lxc.go
@@ -2817,39 +2817,8 @@ func (d *lxc) Start(stateful bool) error {
name := project.Instance(d.Project().Name, d.name)
- // Setup minimal environment for forkstart.
- envDict := map[string]string{
- "container": "lxc",
- }
-
- for k, v := range d.expandedConfig {
- after, ok := strings.CutPrefix(k, "environment.")
- if ok {
- envDict[after] = v
- }
- }
-
- for _, keepEnv := range []string{"LD_LIBRARY_PATH", "INCUS_DIR", "INCUS_SOCKET"} {
- if os.Getenv(keepEnv) != "" {
- envDict[keepEnv] = os.Getenv(keepEnv)
- }
- }
-
- _, ok := envDict["PATH"]
- if !ok {
- envDict["PATH"] = os.Getenv("PATH")
- }
-
- env := make([]string, 0, len(envDict))
- for k, v := range envDict {
- env = append(env, fmt.Sprintf("%s=%s", k, v))
- }
-
// Start the LXC container.
- _, _, err = subprocess.RunCommandSplit(
- context.TODO(),
- env,
- nil,
+ _, err = subprocess.RunCommand(
d.state.OS.ExecPath,
"forkstart",
name,
It had to be the section about modifying the environment, NixOS is particularly itchy when it comes to this.
I am sure it could be reduced further, pretty sure it’s the modifying of the PATH variable, but my server is very underspecced and rebuild-testing this takes about 30 minutes, and this is good enough for me.
1 Like
Hm okay that would make sense. I can try looking into an actual fix however I don’t have a lot of time at the moment. Perhaps the patch should be added to Nixpkgs? Since it seems to be an issue with all NixOS installs?