Bridge device not properly accounting outgoing traffic?

1

I have a raspberry host which has a couple of incus containers running. The network configuration uses a bridge type. For one specific container there significant network traffic, both incoming and outgoing. However the bridge device does not see the incoming traffic.

In the host this are the statistics for the physical device:

3: enx207bd297e6dc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
link/ether 20:7b:d2:97:e6:dc brd ff:ff:ff:ff:ff:ff
RX:   bytes  packets errors dropped  missed   mcast
35360303111 25100489      0       1       0       0
TX:   bytes  packets errors dropped carrier collsns
8537229375  9765756      0       0       0       0

That is roughly 35GB of received data and 8GB of transmitted data. All that traffic is comming from a single container, which has a simple http service.

One would expect the bridge device used by the container to have similar statistics:

9: veth155a6555@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP mode DEFAULT group default qlen 1000
link/ether 9e:6c:9e:3c:77:08 brd ff:ff:ff:ff:ff:ff link-netnsid 1
RX:   bytes  packets errors dropped  missed   mcast
279924702  4170438      0       0       0       0
TX:   bytes  packets errors dropped carrier collsns
35577086390 23629517      0       0       0       0

In that case the transmitted bytes from the host bridge corresponds roughly to the received bytes of the physical device. However the received bytes of the bridge device (279MB) is far from the transmitted bytes of the physical device (8.2GB).

Inside the container the bridge device shows the same statistics as in the host, only inverted, obviously:

8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 10:66:6a:08:1e:4f brd ff:ff:ff:ff:ff:ff link-netnsid 0
RX:   bytes  packets errors dropped  missed   mcast
35577086390 23629517      0       0       0       0
TX:   bytes  packets errors dropped carrier collsns
279924702  4170438      0       0       0       0

How is then possible to do proper account of the container network resources?

This is using Debian 13 as the OS for the host and the container. incus is installed from the repo:
ii incus-base 6.0.4-2+deb13u11 arm64 Powerful system container and virtual machine manager - daemon (container-only)
ii incus-client 6.0.4-2+deb13u11 arm64 Powerful system container and virtual machine manager - client

2

What are the stats for incusbr0?
Also, was the container ever restarted?

3

This are the stats for incusbr0:

5: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 10:66:6a:c4:87:68 brd ff:ff:ff:ff:ff:ff
RX:   bytes  packets errors dropped  missed   mcast
226077446  4203020      0       0       0     435
TX:   bytes  packets errors dropped carrier collsns
35608266589 23671219      0      14       0       0

The container was recently restarted.

4

I have made more tests and I see that outgoing scp traffic is accounted for. But the http outgoing traffic is not. I did not realized before, but it seems that the outgoing traffic is actually accounted for in the lo0 device inside the container:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
RX:  bytes packets errors dropped  missed   mcast
875904181   56650      0       0       0       0
TX:  bytes packets errors dropped carrier collsns
875904181   56650      0       0       0       0

I wonder whether the port proxy is the responsible:

devices:
  port-8000:
  connect: tcp:127.0.0.1:8000
  listen: tcp:0.0.0.0:8000
 type: proxy

Since this is mapping 127.0.0.1 that could be the explanation, right?

5

So the port proxy bypasses completely the bridge?

6

Yep, that kind of proxy has the traffic terminate on the host and then get forwarded cross-namespace by Incus, it never hits eth0 in the container or the veth on the host.

7

Thanks for the explanation! Would be a way to configure it differently so that the port is still offered to the outside but accounted for in the container network device?

8

You can use the nat mode on the proxy.

9

I tried with this:


incus config device add comaps port-8001 proxy listen=tcp:0.0.0.0:8001 connect=tcp:0.0.0.0:8001 nat=true
Error: Invalid devices: Device validation failed for "port-8001": Cannot listen on wildcard address "0.0.0.0" when in nat mode

but failed. I tried also in an experimental machine with incus 7.0.1 and it worked, but I don´t know how to do it in incus 6.0

10

Yeah, it’s something we’ve improved in recent releases.

For Incus 6.0.x, you’d need to set an ipv4.address on the eth0 interface for that container, likely setting it to its current IPv4 address. Once you do that, you can change the connect address to tcp:IPV4-ADDRESS:8001 and nat=true should then be happy.

11

That kind of worked!
I also needed to set listen=tcp:x.x.x.x:8000 where x.x.x.x. is the IPv4 address of the incus host.

Thanks for the support! Maybe a good reason to start using incus 7 LTS :slight_smile: