LXD using IPVLAN for public ip alias

LXD
1

Hi, apology if my question sound like story telling.
i have public ip alias on my eth0, and i wish to use it for a container without ding nat or whatever. and i figured ipvlan is the way to go.
i followed this page:
https://www.gitmemory.com/issue/lxc/lxd/5830/500324610
but unfortunately the container dose not get an ip. so i have to modify netplan as follow:

network:
  version: 2
  renderer: networkd
  ethernets:
    eth0:
      dhcp4: no
      addresses:
        - PUBLIC_IP/24
      gateway4: HOST_IP
      nameservers:
          addresses: [8.8.8.8, 1.1.1.1]

then the container gets the ip alias wanted for it to have.
using HOST_IP as gate is because my ISP provider suggested i do so.
i have internet connectivity. doing curl ifconfig.me result on HOST_IP rathre than PUBLIC_IP / or alias.
i could be doing it wrong. thus. i do appreciate a functional example so i can follow.
Thanks in advance.

2

So the ipvlan NIC type will set the IP and gateway settings before the container starts (like routed NIC type too).

Unfortunately the netplan system inside Ubuntu containers (at least) will wipe the configuration and then try to use DHCP, which won’t work.

I’ve also not been able to figure out how to get netplan to create a default “device” route, as it insists that a “via” setting is specified.

Anyway, I’ve managed to coax it into working by adding a fake route to 169.254.0.1 which is how we do it with the routed NIC type.

If you replace your netplan config with something like this then it works.

network:
    version: 2
    ethernets:
      eth0:
        addresses:
          - 192.168.1.200/32
        nameservers:
          addresses:
          - 8.8.8.8
        routes:
          - to: 0.0.0.0/0
            via: 169.254.0.1
            on-link: true

Also, please note the subnet size specified on the IP address is /32 as it is only a single IP and is not bridged to the wider LAN. Using anything other than /32 (or /128 or IPv6 addresses) is likely to have unwanted results.

3

yup. i just did that. 1 thing tho, the ip alias is assigned to the container. but traffic in and out seems to be through host ip.
quick curl ifconfig.me shows host ip.

4

Have you checked outbound NAT rules on your host?

5

Yes, there nothing for the mentioned ip alias or the host.

6

Please can you show me the iptables (or nftables) NAT rules, also please can you show a tcpdump running on the host’s external interface showing the packets leaving with the incorrect source IP.

Thanks
Tom

7

offcourse,

sudo iptables -L -v -n -t nat
Chain PREROUTING (policy ACCEPT 967 packets, 104K bytes)
 pkts bytes target     prot opt in     out     source               destination         
  985  104K PREROUTING_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  985  104K PREROUTING_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  985  104K PREROUTING_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1996 packets, 443K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2034  450K OUTPUT_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POSTROUTING (policy ACCEPT 1996 packets, 443K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2034  450K POSTROUTING_direct  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 2034  450K POSTROUTING_ZONES_SOURCE  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 2034  450K POSTROUTING_ZONES  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 POST_public  all  --  *      vlan0   0.0.0.0/0            0.0.0.0/0           [goto] 
 1937  433K POST_public  all  --  *      br0     0.0.0.0/0            0.0.0.0/0           [goto] 
    1    40 POST_public  all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           [goto] 
   93 16374 POST_public  all  --  *      +       0.0.0.0/0            0.0.0.0/0           [goto] 

Chain POSTROUTING_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POST_external (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    1   108 POST_external_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    1   108 POST_external_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    1   108 POST_external_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POST_external_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1   108 MASQUERADE  all  --  *      !lo     0.0.0.0/0            0.0.0.0/0           

Chain POST_external_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POST_external_log (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POST_public (4 references)
 pkts bytes target     prot opt in     out     source               destination         
 2033  449K POST_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 2033  449K POST_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
 2033  449K POST_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain POST_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POST_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain POST_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PREROUTING_ZONES (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 PRE_public  all  --  vlan0  *       0.0.0.0/0            0.0.0.0/0           [goto] 
  967  104K PRE_public  all  --  br0    *       0.0.0.0/0            0.0.0.0/0           [goto] 
   18   576 PRE_public  all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           [goto] 
    0     0 PRE_public  all  --  +      *       0.0.0.0/0            0.0.0.0/0           [goto] 

Chain PREROUTING_ZONES_SOURCE (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PREROUTING_direct (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PRE_external (0 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 PRE_external_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 PRE_external_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 PRE_external_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain PRE_external_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PRE_external_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PRE_external_log (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PRE_public (4 references)
 pkts bytes target     prot opt in     out     source               destination         
  985  104K PRE_public_log  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  985  104K PRE_public_deny  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
  985  104K PRE_public_allow  all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain PRE_public_allow (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PRE_public_deny (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain PRE_public_log (1 references)
 pkts bytes target     prot opt in     out     source               destination

am not using nftables. just firewalld, and iptables.
about:
please can you show a tcpdump running on the host’s external interface showing the packets leaving with the incorrect source IP.
apology, have no idea how to do that.

8

This line is the issue ^, it says to NAT all outbound traffic that doesn’t come from the local loopback interface to the host’s IP.

9

Brilliant :wink:
now a container running on ipvlan, runs nginx dose it require proxy in order to allow inbound?
without proxy its not working.

10

No you shouldn’t need a proxy device as the container should have a public IP. If ping is working to/from the public IP to/from external destinations then you have connectivity, and its likely a firewall rule blocking inbound traffic somewhere.

11

ping already works, as the ip is an alias on the host side.
am digging more …

12

Can you show output of:

ip a on host
ip r on host

ip a in container
ip r in container

Thanks

13 
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 5000
    link/ether MAC_ADD brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether MAC_ADD brd ff:ff:ff:ff:ff:ff
    inet HOST_IP/26 brd HOST_BRD scope global br0
       valid_lft forever preferred_lft forever
    inet ALIAS_1/29 brd ALIAS_1_BRD scope global br0
       valid_lft forever preferred_lft forever
    inet ALIAS_2/29 brd ALIAS_2_BRD scope global secondary br0
4: testing: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether e2:02:33:e2:23:19 brd ff:ff:ff:ff:ff:ff
    inet 11.11.11.1/24 scope global testing
       valid_lft forever preferred_lft forever
    inet6 fe80::e002:33ff:fee2:2319/64 scope link 
       valid_lft forever preferred_lft forever
ip r
default via HOST_BRD dev br0 
11.11.11.0/24 dev testing proto kernel scope link src 11.11.11.1 
HOST_BRD/26 dev br0 proto kernel scope link src HOST_IP 
----------------------------------------------------------------------------------------------------
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
16: eth0@if3: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether MAC_AD brd ff:ff:ff:ff:ff:ff
    inet IP_ALIAS/32 brd 255.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 IPV6/64 scope link 
       valid_lft forever preferred_lft forever
 ip r
default dev eth0

EDITE:
after reboot i can see this on host. in addition to the other results.
ALIAS_1 dev lo scope link

14

Is IP_ALIAS the same as ALIAS_1 or ALIAS_2? I.e is the IP alias you have inside the container also bound the host’s br0 interface?

15

IP_ALIAS is same as IP_ALIAS_1 .

by bound? not sure there. do i need to make the host ip as default gateway?

16

You shouldn’t have the public IP that you’re adding to the container existing on the host at all.

Otherwise there will be confusion as to where the packets should go when destined for that IP, should it go to the host or the container?

17

It should go to the container, i imagined i needed to add it on the host at first then make it available on the container. am going to remove it from host and try.
please check this edit:

18

That is normal, it is a static route of the public IP alias to the local loopback interface, it is added by LXD when using ipvlan, its only purpose is to activate the proxy ARP on the host so that your host responds to ARP requests for your public IP alias that you’re passing through into the container.

However it is important that the IP itself is not bound (setup) on the host.

19

Very informative.
I just removed from the host. / removed host bond for ALIAS_1 . still can not ping it at all now. or visit port 80.

20

Restart the container, and then gather output of following please:

On host:

ip a
ip r
ip neigh show proxy

And then inside the container;

ip a
ip r

Thanks
Tom