Incus UI doesn't correctly handle network creation in clusters

oddjobz · June 29, 2025, 8:34pm

Hi, I have a two-node cluster that just updated itself … I noticed when I started getting errors in my backup (!) … it would seem that if I create a new Network (called say “public”) in the GUI, it creates a network in an ERRORED state. No errors in the log file I can see.

Not only that, the GUI then failes to delete the network when asked.

Dropping to the CLI it also shows the network in the errored state, but allows me to delete it. If I then do;

incus network create public --target=rpi1
incus network create public --target=rad
incus network create public

It works quite happily.

I’ve repeated a couple of times with different network names, same issue.

stgraber · June 30, 2025, 2:35am

@presztak can you take a look?

It’s about creation of non-OVN networks in a cluster environment. It looks like it’s not attempting to do the per-target creation and just goes straight to creating the global item triggering some kind of failure.

oddjobz · June 30, 2025, 3:59am

It’s about creation of non-OVN networks in a cluster environment. It looks like it’s not attempting to do the per-target creation and just goes straight to creating the global item triggering some kind of failure.

Yes, I’m no longer using OVN anywhere. Regular networks, default settings.

Piotr_Resztak · June 30, 2025, 4:06pm

Sure!

Piotr_Resztak · July 3, 2025, 12:40pm

@oddjobz I wasn’t able to reproduce the issue on my end, it doesn’t seem to be a general UI problem. That said, could you please provide a bit more detail?

It would be helpful if you could open your browser’s Developer Tools and check the Console tab for any errors. Also, take a look at the Network tab, in particular, check the following requests:

/1.0/networks?project=<project>&target=rpi1
/1.0/networks?project=<project>&target=rad
/1.0/networks?project=<project>

Do any of these requests fail or return something other than a 2xx status? If so, could you share the response details?

oddjobz · July 3, 2025, 12:50pm

Ok, so trying it again now, I get this in the JS console;

index-CfMUciCt.js:93 
 GET https://rpi1:8443/1.0/networks/t?project=default 404 (Not Found)
index-CfMUciCt.js:93 
 GET https://rpi1:8443/1.0/networks/te?project=default 404 (Not Found)
index-CfMUciCt.js:93 
 GET https://rpi1:8443/1.0/networks/tes?project=default 404 (Not Found)
index-CfMUciCt.js:93 
 GET https://rpi1:8443/1.0/networks/test?project=default 404 (Not Found)
index-CfMUciCt.js:93 
 GET https://rpi1:8443/1.0/networks/test?project=default 404 (Not Found)
index-CfMUciCt.js:93 
 GET https://rpi1:8443/1.0/networks/test?project=default 404 (Not Found)
useNetworks-4mRAJCTt.js:1 
 POST https://rpi1:8443/1.0/networks?project=default net::ERR_NETWORK_CHANGED
useNetworks-4mRAJCTt.js:1 
 GET https://rpi1:8443/1.0/networks/test?project=default net::ERR_NETWORK_CHANGED
useNetworks-4mRAJCTt.js:1 
 Uncaught (in promise) TypeError: Failed to fetch
    at f (useNetworks-4mRAJCTt.js:1:692)
    at useNetworks-4mRAJCTt.js:1:1769

Leaves me with a green spinner on the button, i.e. doesn’t complete.
If I reload the GUI I see;

Connections
test
General
Type
Bridge
Description
-
IPv4 address
10.99.223.1/24
IPv6 address
fd42:7156:3344:41f9::1/64
ACLs
-
RX
0 B (0 packets)
TX
0 B (0 packets)
Status
Errored

In the GUI. Created yaml looks like this;

project: default
name: test
description: ''
type: bridge
config:
  ipv4.address: 10.99.223.1/24
  ipv4.nat: 'true'
  ipv6.address: fd42:7156:3344:41f9::1/64
  ipv6.nat: 'true'

CLI looks like this;

# incus network ls test
+------+--------+---------+----------------+---------------------------+-------------+---------+---------+
| NAME |  TYPE  | MANAGED |      IPV4      |           IPV6            | DESCRIPTION | USED BY |  STATE  |
+------+--------+---------+----------------+---------------------------+-------------+---------+---------+
| test | bridge | YES     | 10.99.223.1/24 | fd42:7156:3344:41f9::1/64 |             | 0       | ERRORED |
+------+--------+---------+----------------+---------------------------+-------------+---------+---------+

Nothing in incus.log.

Piotr_Resztak · July 8, 2025, 12:04pm

@oddjobz What’s strange is that I don’t see a request with the target parameter from you. But even so, even if such a request wasn’t sent, you should still get:
Error: Network not pending on any node (use --target <node> first)
or, in case the request was sent to only one server:
Error: Network not defined on nodes: <node1>, <node2>

so, I’m still having trouble reproducing this error. Could you also provide the version of the UI you’re using?

oddjobz · July 8, 2025, 12:14pm

Hi, sure, version in the bottom of the screen;

Version 6.14-ui-0.17.1

So just to update, this issue is coming from my original cluster that inadvertently upgraded to 6.14. I have a “new” production cluster, all machines installed on 6.14 from scratch which seems to work Ok.

Here’s what happens on the old UI:

I save, and it seems to go ok, but then when I look at the networks;

Piotr_Resztak · July 8, 2025, 12:15pm

I also see that you’re receiving net::ERR_NETWORK_CHANGED, which can be caused by a broken DNS configuration. Can you try connecting using the IP address?

oddjobz · July 8, 2025, 12:23pm

So in this particular instance the URL I’m using is;

https://rpi1:8443/ui/settings

Where “rpi1” is an /etc/hosts entry. I just moved to the address;

https://192.168.1.204:8443/ui/project

And deleted the network created above. Had to do it twice, first attempt I got a red error box on the screen and this in the console;

DELETE https://192.168.1.204:8443/1.0/networks/dummy?project=default net::ERR_NETWORK_CHANGED
(anonymous) @ useNetworks-4mRAJCTt.js:1
T @ useNetworks-4mRAJCTt.js:1
o @ NetworkDetail-DS4aaMmw.js:1
D @ index-CfMUciCt.js:76
...

Second attempt worked.
Tried to create network again;

POST https://192.168.1.204:8443/1.0/networks?project=default net::ERR_NETWORK_CHANGED

Same thing, shows errored state in networks.

stgraber · July 9, 2025, 3:37am

Can you try pinging 192.168.1.204 during this?
Trying to figure out why the network connection would suddenly fail during that request.

The error here is really a network error thrown by the web browser, so not something that can particularly easily be handled by javascript…

oddjobz · July 9, 2025, 8:45am

Mm, Ok, so a constant ping from the machine running chrome during the process shows no packet loss …

Two issue here;

This browser error is unlikely to be the reason the network create (and delete) is failing
Although this “looks” like a network issue, again it seems unlikely

The network create is being performed server-side, so any communication with the browser should be irrelevant in this context other than to update the browser. So whereas there is a problem with the communication, this looks to be a slightly different issue.

Most of the references to this error indicate a physical networking issue, however the browser in question is used ‘heavily’ for maybe 12 hours per day performing many tasks, so there is no generic problem with it. The incus server is a production internet facing server running 4 live containers … generally speaking it seems to have no operational problems.

The Incus UI on the box generally works fine. It “only” errors in this instance when trying to create or delete a network. When you do a deep-dive into the error it seems it can also be created by the server “aggressively closing the socket”. Given the process is failing to perform server-side, an aggressive socket close would seem to be a potential side-effect of the problem.

The first thing that I wonder about is that the server is generating a network in an error’d state (for whatever reason) yet has produced no log entries in the incus log file. Is there an easy way to up the log-level on the server? (or is there somewhere else I should be looking for logs)