I have a specific scenario I’m trying to implement, and I haven’t found any solution on this forum.
I hope you guys may be able to help me.
I currently have a Lubuntu 22.04 host with 2 Network Cards:
First card (ens33) is acting as a management interface (to manage the Host and have access to the internet)
Second card (ens34) is acting as a promiscuous interface (port-mirroring configured on switch to send traffic on that interface).
if I run tcpdump -i ens34 on the host machine, I am able to see the captured packets.
Now my first test was to be able to capture packets inside of an LXD container, so I ran this command: lxc config device add centos-worker1 eth1 nic name=eth1 nictype=physical parent=ens34
with this, when I ran tcpdump -i eth1 inside of centos-worker1 container, I was once again able to capture packets.
However, as you already know, using nictype=physical makes ens34 interface from the host vanish, and this interface will only be accessible by centos-worker1 container under the name of eth1.
This is where I’m currently stuck, because my second test consists of using ens34 somewhere else. (A virtualbox VM running on the Host machine in parallel to LXD.)
I tried replacing nictype=physical to nictype=macvlan, but when I run tcpdump -i eth1 in container, I am unable to get captured packets.
i tried creating a bridge on the host ip link add ens35 type bridge and linking it to ens34 using ip link set ens34 master ens35 and then pushing a copy of ens35 in centos-worker1 by replacing nictype to bridged for parent=ens35 but tcpdump still has no traffic on eth1 (even though both ens34 and ens35 have traffic on host)
Is there a way where I can make my LXD container capture packets while at the same time keep my parent interface in my host machine so that I can also use it to capture traffic in parallel?
I’m guessing LXD doesn’t have these capabilities yet?
because when I look at other virtualization environment, we usually can assign the same interface on multiple containers/VMs like virtualbox, vmware workstation, ESXi, etc …
I honestly love LXD, it’s easy to configure, it’s amazing performance wise, we can execute remote commands for intercommunication between the nodes. So I’d really love it and appreciate it if the functionality motioned in this topic is implemented.
I read some of the other posts, and they mentioned firewall rules, so since I’m using Lubuntu, I added rules using ufw.
I added the 3 rules mentioned for lxdbr0 and applied it to ens35:
ufw allow in on ens35
ufw route allow in on ens35
ufw route allow out on ens35
still no luck. This time around, instead of having an empty tcpdump, I received a few packets, but they were all 169.254.x.x .
I received around 70 packets like these above, but I sent about 40 000 packets to that interface.
I don’t think these 70 packets were linked in any sort to my 40 000 packets. I think they were just some intercommunication between the bridge probably. On the host however, I was able to view these 40 000 packets with tcpdump.
basically what I am asking for, in simple terms, is an alternative to nictype=physical.
Something that would give me the same amount of info as if I had nictype=physical for an interface acting as a sniffer, but with the flexibility of creating that interface in multiple LXD containers.
By being able to create it in multiple LXD containers would also mean that interface would still exist in host, meaning I would also be able to use it for other purposes.
Is there really nothing LXD can offer for this functionality?
Have I really hit a wall where I must change my virtual environment completely and give up on LXD?
That’s not what you asked for exactly but there is an easy way around this. Here @home I have a container gw-home using a nictype=physical device called uplink:
root@c2d:~# lxc config device show gw-home
dmz:
name: dmz
nictype: bridged
parent: dmz
type: nic
r2r:
name: r2r
nictype: bridged
parent: r2r
type: nic
uplink:
hwaddr: 5a:61:9e:ed:d0:62
name: uplink
nictype: physical
parent: uplink
type: nic
In the host’s context, I can easily run a tcpdump capture from within the network namespace of gw-home. Here’s how:
root@c2d:~# PID="$(lxc info gw-home | awk '/^PID: / {print $2}')"
root@c2d:~# nsenter -t "$PID" -n tcpdump -ni uplink -w /tmp/uplink.pcap
tcpdump: listening on uplink, link-type EN10MB (Ethernet), snapshot length 262144 bytes
^C25 packets captured
25 packets received by filter
0 packets dropped by kernel
hmmm, that’s an interesting alternative! thank you @sdeziel
I am happy that there is a way to tcpdump an interface that has vanished from the host thanks to nsenter.
If I understand correctly, the difference between your command and lxc exec centos-worker1 -- tcpdump -i eth1 is that nsenter is executed purely from the host using the lxc node as a process? Or is it virtually the same thing?
But yeah, like you mentioned, this is not the exact concept I was looking for, but it gave me some inspiration!
Is there a way to do the opposite maybe? Keep the interface in host but forward the content of sniffer to an LXD interface?
Your strategy gave me the idea of creating pcap files that I would share to my LXD nodes. And my nodes would read the files instead of the interface, though this will not be real time.
I never tested it this way, so I’m not sure if it would work well but I’ll give it a shot. I’ll see with my boss if it’s an acceptable approach.
Meanwhile, I’ll keep this topic open, in case another better solution pops-up.
Because one thing is certain, is that I need the interface on the host to bridge it with VirtualBox VMs.
The lxc exec one really has tcpdump running from inside the container so when using -w, it will write in the container’s filesystem.
The nsenter alternative means that tcpdump runs from the host but as part of the container’s network namespace so it can see the interface but still write to the host’s filesystem.
I have a feeling that with macvlan you should be able to give a macvlan interface to the container while still being able to capture on the parent interface that’d remain in the host. If I find the time, I’ll try to test that out here.
yeah it is true that macvlan keeps the parent interface in the host, which is great. However, when I tried to tcpdump from within the container, the interface wasn’t reading anything at all.
Thank you for taking the time of testing this approach. I’ll be waiting for your feedback if you manage to find a way to tcpdump from within the container with a macvlan.
Yes macvlan will only receive frames for the specific instance’s MAC address.
I’ve not tried it, but potentially you could setup an unmanaged bridge with the external interface connected to it, and then connect an instance to that bridge (using bridged NIC with parent setting), and then perhaps there is a way to enable gratuitous mode or port mirroring on that switch?
thanks for the info,
I tried creating an unmanaged bridge, but the tcpdump was also returning empty.
As in my example above, I named my unmanaged bridge ens35 (sorry for my bad naming, it was just a small test, I should have named it br0).
to manage it temporarily, I executed these commands:
ip link add ens35 type bridge
ip link set ens34 down
ip link set ens34 master ens35
ip link set ens34 up && ip link set ens34 promisc on
ip link set ens35 up && ip link set ens35 promisc on
I then linked it to my container via nictype=bridged.
When I ran tcpdump in the container, I couldn’t see external traffic.
I also read reply you made in another topic and I tried implementing it here by adding rules to my unmanaged bridge:
ufw allow in on ens35
ufw route allow in on ens35
ufw route allow out on ens35
that also didn’t change a thing in the tcpdump result. Maybe I created my unmanaged bridge wrong?
As for the port mirroring, there’s a network team that already configured the port mirroring correctly on the switch and tcpdump is able to see traffic from each office laptop.
However for my development environment, I’m currently just using a second laptop, I attached an ethernet cable on both laptops and created a mini LAN network. I’m using my second laptop to push packets (from a random pcap file I found) to my main laptop. I can see these packets via tcpdump.
ok @tomp, I retried it now, recreating a br0 (without firewall rules) and I sent 246 000 packets to LXD container via interface nictype=bridged and I only managed to read with tcpdump 5 packets marked as ARP. After packet replay finished, I also received 2 IPV6 packets which was my container communicating with host via bridge.
after allowing firewall rule, now I got a bunch of 169.254.X.X packets being received by container, but not the actual packets sent from my second laptop
cat /etc/netplan/01-network-manager-all.yaml
# Let NetworkManager manage all devices on this system
network:
version: 2
renderer: NetworkManager
ethernets:
ens34:
addresses: []
bridges:
br0:
interfaces: [ens34]
addresses: []
I ran tcpdump on promisc interface on host:
tcpdump -ni ens34
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens34, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
88366 packets captured
88366 packets received by filter
0 packets dropped by kernel
51 packets dropped by interface
I then ran tcpdump on bridge interface on host:
tcpdump -ni br0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
...
87244 packets captured
87244 packets received by filter
0 packets dropped by kernel
I added bridge on lxd node: lxc config device add centos-worker1 eth1 nic name=eth1 parent=br0 nictype=bridged
I ran tcpdump inside of container:
tcpdump -ni eth1
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
...
22 packets captured
27 packets received by filter
0 packets dropped by kernel
as you can see, for about the same duration length, the container only captured 27 packets, and these packets aren’t related to what the interface was actually capturing. It was random packets unrelated to the packets I was replaying
so does this mean that current LXD project cannot capture packets on multiple containers using same parent interface while keeping parent interface on host for additional packet capture?
Do we close this case by concluding that it’s not feasible on current LXD version?
and if so, are you guys thinking of adding this functionality in future releases of LXD?