3.19 and Routed networking mode configuration example needed

bmullan · January 24, 2020, 1:07am

I am trying to configure the IP Routed NIC to use with some containers.

The only reference I could find with any example was in the Routed networking mode section:

So I stop my existing containers and per the example in the above URL…

I used the command:

$ lxc config device add cn1 eth0 nic nictype=routed ipv4.address=10.0.2.21
Device eth0 added to cn1
$ lxc start cn1
$ lxc list cn1

And the lxc list cn1 and it shows CN1 now has ip address 10.0.2.21.

However, I still have an t1-br bridge that CN1 was originally built to run under but now that t1-br Bridge takes on the IP address of 10.0.2.1

So I am befuddled and lost as to how to

The LXD container MUST already have been created before the IP NIC is applied to the container CN1.
If so then the LXD bridge (in my case t1-br) is still alive.

At least Release Note example commands lend me to think so as it only lists:

$ lxc start c1
$ lxc list cn1

So what does the LXD “profile” need to look like to use the IP NIC instead of a bridge ??

Is there a complete example to take a system that has been running LXD containers with the traditional lxdbr0 bridge through all that needs to be done to re-configure those existing containers to use the new Routed NIC ?

tomp · January 24, 2020, 8:57am

Hi @bmullan

So routed network mode works as follows:

It uses liblxc under the hood to create a veth pair of interfaces, and moves one side of the veth pair into the container.
It then preconfigures the static IP addresses you have specified onto the interface.
Next it sets up default routes in the container ponting to 169.254.0.1 for IPv4 and fe80::1 for IPv6. These are link-local addresses and serve only to get packets to/from the container to the host.
On the host, static routes are added for the container’s IP pointing to the host-side of the veth pair.
Additionally the link-local gateway IPs are created on every host-side veth pair.

This means you should avoid having any start up services inside the container that remove IPs from the interfaces and attempt to do DHCP. You should also be aware that if you are using static IPs from an existing DHCP range then unlike ‘bridged’ mode, these do not create static reservations in the parent bridged’s local DHCP server and so you should ensure you’ve configured the DHCP range to not overlap with the static IPs assigned.

Note: The routed network mode does not require you to specify a parent option. This means that the IPs you specify for the container do not have to be part of any subnet on the host, and you can instead choose to propagate these routes using a routing daemon.

However you can optionally specify a parent option, and in this case proxy ARP and proxy NDP entries are added to that parent interface “advertising” those IPs at layer2 to the parent interface. If those IPs are in the same subnet as the parent interface’s network, this then acts as a kind of ‘bridge’ allowing those containers to appear on the parent’s interfaces network. However you do not need to use any bridge, and all the containers will appear to be using the host’s parent interface’s MAC address.

I’m not sure what you mean about the IP for the bridged, 10.0.2.1, as routed mode will not alter the bridge IP, especially one it is not a parent of. They key point here is that routed mode doesn’t need any bridges to operate.

So if I start with a container config like so:

devices:
  eth0:
    ipv4.address: 10.138.198.132
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic

With the lxdbr0 interface having an IP of 10.138.198.1.

Then I can change the container’s nictype to “routed”:

devices:
  eth0:
    ipv4.address: 10.138.198.132
    name: eth0
    nictype: routed
    parent: lxdbr0
    type: nic

Starting the container then shows on the host:

ip r
10.138.198.0/24 dev lxdbr0 proto kernel scope link src 10.138.198.1 linkdown 
10.138.198.132 dev veth0ff2a75d scope link

You can also see the container’s IP being ‘advertised’ via proxy ARP to the parent interface.

ip neigh show proxy
169.254.0.1 dev veth0ff2a75d  proxy
10.138.198.132 dev lxdbr0  proxy

And in the container you can see the default link-local routes created:

lxc exec c1 ip r
default via 169.254.0.1 dev eth0 
169.254.0.1 dev eth0 scope link

Note: In this example, if I removed the “parent: lxdbr0” part from my container, then it would prevent other containers connected to the bridged from communicating with my container. This is because without the parent option the container is not advertised at layer 2 onto the lxdbr0 interface.

However the container would still be able to communicate with the host via its default routes.

tomp · January 24, 2020, 8:58am

There’s also some example usage on the PR that added the feature:

tomp · January 24, 2020, 9:18am

So routed mode doesn’t need a bridge to operate, so you could remove the t1-br and the LXD managed network entirely if no other containers are using it.

bmullan · January 24, 2020, 4:47pm

@tomp

Tom

Thank you so much. I’ve been wanting to try the Route NIC feature but there wasn’t enough info in one spot to answer the various questions.

This is great and thanks for taking the time to put it together. I’m sure a lot of others will appreciate this as well.

Brian

bmullan · January 29, 2020, 1:19am

@tomp

Your statement:

“So if I start a container config like so”

Do you have an example for this using

$ lxc network edit rnic

Where rnic was previously created by

$ lxc network create rnic

And if I wanted a profile for this:

$ lxc profile copy default pr-rnic

What changes would need to be made to that profile?

$ lxc profile edit pr-rnic

tomp · January 29, 2020, 9:15am

Because the routed nic type doesn’t require an LXD managed network, I would do the following:

Assuming that:

Interface enp3s0 is a physical port connected to a network 192.168.1.0/24.
The default gateway on the physical network is 192.168.1.1.
An existing default profile has a bridged NIC connected to lxdbr0.

Copy the profile and remove the bridged NIC from new profile:

lxc profile copy default rnic
lxc profile device remove rnic eth0

Add a partially configured routed NIC to the profile, this can optionally have the parent specified or not.

lxc profile device add rnic eth0 nic nictype=routed parent=enp3s0

Now create a container from rnic profile, note I only init the container, I don’t launch as need to add IPs to it (the container will start without IPs though).

lxc init ubuntu:18.04 c1 -p rnic
lxc config device override c1 eth0 ipv4.address=192.168.1.200
lxc start c1
ping 192.168.1.200
lxc exec c1 ping 192.168.1.1

Finally the managed LXD network isn’t needed anymore (unless other containers are using it) so:

lxc network delete lxdbr0

bmullan · January 29, 2020, 12:00pm

@tomp
Thank you. That explains alot of what I was seeing as I was missing some steps.

tomp · March 23, 2020, 10:52am

I use this configuration in LXD and netplan which works fine:

lxc config device add c1routed eth0 nic nictype=routed parent=enp3s0 ipv4.address=192.168.1.200

Note: The parent option is important if you are wanting to make your container appear to be on the host’s external network at the layer 2 rather than relying on the ISP routing traffic for your IPs to your host directly. You haven’t provided you LXD container config so I can’t tell at this stage.

Then in netplan:

network:
    version: 2
    ethernets:
        eth0:
          addresses:
            - 192.168.1.200/32
          nameservers:
            addresses: [8.8.8.8]
          routes:
            - to: 0.0.0.0/0
              via: 169.254.0.1
              on-link: true

Can you ping 8.8.8.8 from your container (you say “ping works fine” but don’t state where you are able to ping to).

If you are able to ping externally from your container, then the routed configuration is working and the most likely issue is a firewall on your host (that is preventing all routed traffic except ICMP ping) or on your wider network.

MSTY · March 23, 2020, 4:41pm

I am very sorry for the confusion. @tomp was kind enough to offer a very quick and comprehensive answer to my question, yet I discovered I had to delete this very question, because it included some network information I did not want to share. So this is a redacted version. If you are reading this thread, this is the question that tomp answered to in his comment above. I will study his answer and provide feedback in another comment, as soon as I find the time.

For me, this doesn’t happen. No routes are created inside the container. I used the commands you list below:

lxc profile copy default rnic
lxc profile device remove rnic eth0
lxc profile device add rnic eth0 nic nictype=routed parent=ens3
lxc init ubuntu:18.04 c1 -p rnic
lxc config device override c1 eth0 ipv4.address=[MY-PUBLIC-IP]
lxc start c1
lxc exec c1 ip r

The last command turns up nothing. No routes are created inside the container. Can I set them up manually? Because when I use this as my /etc/netplan/50-cloud-init.yaml below, I can ping in and out, but nameserver resolution doesn’t work.

network:
    version: 2
    ethernets:
        eth0:
          addresses: [MY-PUBLIC-IP]
          nameservers:
            addresses: [8.8.8.8]
          routes:
            - to: 0.0.0.0/0
              via: 169.254.0.1
              on-link: true

And ip r only comes up like this:

default via 169.254.0.1 dev eth0 proto static onlink

The second line, beginning with the IP, is missing.

I use Bionic and the LXD Snap, currently at 3.22.

This Netplan gives me the same result:

network:
    version: 2
    ethernets:
        eth0:
          addresses: [MY-PUBLIC-IP]
          gateway4: 169.254.0.1
          nameservers:
            addresses: [8.8.8.8]

I can ping fine, but name resolution doesn’t work. And “ip r” only produces this line, nothing, more:

default via 169.254.0.1 dev eth0

When I change /etc/netplan/50-cloud-init.yaml inside the container to the following, I still don’t have dns, but some changes:

network:
    version: 2
    renderer: networkd
    ethernets:
        eth0:
          addresses: [My-Public-IP/32]
          dhcp4: no
          nameservers:
            addresses: [8.8.8.8]
          gateway4: 169.254.0.1
          routes:
            - to: 169.254.0.1/32
              via: 169.254.0.1
              scope: link

“ip r” now results in this:

default via 169.254.0.1 dev eth0 
default via 169.254.0.1 dev eth0 proto static 
169.254.0.1 dev eth0 scope link

But still no name resolution, even though it looks fine to me:

Link 73 (eth0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 8.8.8.8

journalctl says:

Using degraded feature set (UDP) for DNS server 8.8.8.8.

or

Using degraded feature set (TCP) for DNS server 8.8.8.8.

Btw. this is a KVM guest rented from Netcup, where I purchased additional IPv4, which Netcup itself says should be added like this: https://www.netcup-wiki.de/wiki/Zus%C3%A4tzliche_IP_Adresse_konfigurieren

MSTY · March 24, 2020, 9:19pm

I am impressed. Your guessed right. It was the firewall. Everything is working fine, when I disable the ufw firewall on the lxd host machine. Now I just have to figure out how to configure the ufw firewall. But this might be useful to others, since I don’t remember changing anything from the default Ubuntu Bionic configuration except for the prerouting rules.

Darwin · April 7, 2020, 2:40am

Hi Everyone,

I’m trying to configure an existing container over Bridge mode to Routed mode. I followed the instruction above to provide to my containers Public IPs.

I have in my server configured bonding and bridge as follow. but I want to remove br1 and connect directly to bond-wan and apply routed networking to assign Public IPs to my containers.

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s5:
      dhcp4: no
      dhcp6: no

    enp0s6: 
      dhcp4: no
      dhcp6: no

      bonds:
        bond-wan:
          interfaces: [enp0s5, enp0s6]
          dhcp4: false
          dhcp6: false
          parameters:
            mode: active-backup
            mii-monitor-interval: 1
            gratuitious-arp: 5
            primary: enp0s5

      bridges:
        br1:
          interfaces: [bond-wan]
          dhcp4: false   
          dhcp6: false
          addresses: [200.119.xx.xx/xx]
          gateway4: 200.119.xx.xx
          nameservers:
            addresses: [8.8.8.8, 8.8.4.4]

First of all I modified my network configuration in the Host side to remove br1

bonds:
    bond-wan:
      interfaces: [enp0s5, enp0s6]
      addresses: 
        - 200.119.xx.xx/xx
      gateway4: 200.119.xx.xx
      nameservers:
        addresses: [8.8.8.8]
      parameters:
        mode: active-backup
        mii-monitor-interval: 1
        gratuitious-arp: 5
        primary: enp0s5

created a new profile like this

root@copark:~# lxc profile copy default routed
root@copark:~# lxc profile device remove routed eth0
root@copark:~# lxc profile device add routed eth0 nic nictype=routed parent=bond-wan

root@copark:~# lxc profile show routed
config: {}
description: Default LXD profile
devices:
  eth0:
    nictype: routed
    parent: bond-wan
    type: nic
  root:
    path: /
    pool: lxdpool
    type: disk
name: routed
used_by:
- /1.0/instances/container

Then I stoped the container and applied the new profile
root@copark:~# lxc stop container
root@copark:~# lxc profile apply container default,routed
Profiles default,routed applied to container
Within my container I configured a new Public IP with netplan

network:
    version: 2
    ethernets:
        eth0:
          addresses:
            - 200.119.xx.xx/xx
          nameservers:
            addresses: [8.8.8.8]
          routes:
            - to: 0.0.0.0/0
              via: 169.254.0.1
              on-link: true

I can see the configuration of my public ip in the container with this command

root@copark:~# lxc list
+-----------+---------+----------------------+------+-----------+-----------+
|   NAME    |  STATE  |         IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+-----------+---------+----------------------+------+-----------+-----------+
| container | RUNNING | 200.119.xx.xx (eth0) |      | CONTAINER | 0         |
+-----------+---------+----------------------+------+-----------+-----------+

Unfortunately I have not been able to successfully configure the container to surf on internet neither I haven’t been able to make a ping from the container to the host and vice versa.

Please advice.

tomp · April 7, 2020, 11:12am

Hi @Darwin

One of the key principles of the routed NIC type is that LXD will setup static routes and proxy ARP entries on the host to “route” traffic from the parent network to the container’s interface.

In order to do that LXD needs to know what IPs you are assigning to your container.

At the moment you have not told LXD this information, which is why it isn’t working.

You can check this by running ip r on the LXD host and looking for a static route for your container’s IP.

In order to tell LXD what IPs you will be using inside your container you need either:

Add a manually configured NIC device to each container, overriding the profile’s settings, e.g.

lxc profile apply container default,routed
lxc config device override <container> eth0 ipv4.address=x.x.x.x.x

Or you can use a profile per container and store the container’s IPs in the profile config. To see an example of this see @simos guide here How to get LXD containers get IP from the LAN with routed network

Darwin · April 7, 2020, 8:35pm

Hi Thomas,

Thank you for your quick response. You are right, I already made that change and it works fine but now I have a new issue.

I can surf on internet and communicate with the host but from outside (internet) I can not connect to my container.

Do I have to make something on my IPtables or do I have to add a new route (route add)?

When I run networkctl in the host side, it doesn’t show me the Public IP that I configured, instead it shows 169.254.0.1. Do you think that is the issue i’m experiencing when I try to connect from internet?

root@copark:~# networkctl status
●        State: routable
       Address: 200.119.xx.xx on bond-wan
                169.254.0.1 on veth337b7a01  <--- It doesn't show me the Public IP that I configured
                fe80::10exxxxxxxxxxxx on bond-wan
                fe80::fc8xxxxxxxxxxxxx on veth9d2e7234
       Gateway: 200.119.xx.xx (Telco) on bond-wan
           DNS: 8.8.8.8

On the other hand in my LAN network I have another containers with private IPs that can’t communicate each others. What can I do in both cases?

This is for my LAN network

root@test:~# lxc list
+----------+---------+-----------------------+------+-----------+-----------+
|   NAME   |  STATE  |         IPV4          | IPV6 |   TYPE    | SNAPSHOTS |
+----------+---------+-----------------------+------+-----------+-----------+
| Matrix   | RUNNING | 192.168.15.220 (eth0) |      | CONTAINER | 0         |
+----------+---------+-----------------------+------+-----------+-----------+
| ownCloud | RUNNING | 192.168.15.230 (eth0) |      | CONTAINER | 0         |
+----------+---------+-----------------------+------+-----------+-----------+

root@test:~# ip r
default via 192.168.15.1 dev bond-lan proto static 
192.168.15.0/24 dev bond-lan proto kernel scope link src 192.168.15.200 
192.168.15.220 dev veth02923e10 scope link 
192.168.15.230 dev veth2b75367c scope link

root@test:~# networkctl status
●        State: routable
       Address: 192.168.15.200 on bond-lan
                169.254.0.1 on veth02923e10. <-- Again it doesn't shoe me the IP (192.168.15.220)
                169.254.0.1 on veth2b75367c
                fe80::10ef:68ff:fe21:7c77 on bond-lan
                fe80::fc83:62ff:fe80:9de8 on veth9d2e7234
                fe80::fc3e:abff:fee8:9509 on veth337b7a01
       Gateway: 192.168.15.1 on bond-wan
           DNS: 192.168.15.1

Ping from one container to another

root@Matrix:~# ping 192.168.15.230
PING 192.168.15.230 (192.168.15.230) 56(84) bytes of data.
From 192.168.15.220 icmp_seq=1 Destination Host Unreachable
From 192.168.15.220 icmp_seq=2 Destination Host Unreachable
From 192.168.15.220 icmp_seq=3 Destination Host Unreachable

Ping from my laptop to Matrix container

darwin@Darwins-MBP ~ % ping 192.168.15.220
PING 192.168.15.220 (192.168.15.220): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3

tomp · April 8, 2020, 8:19am

Lets take each case separately as otherwise I may get confused which hosts we are talking about.

As I understand it you have a host with an external IP of 200.119.xx.xx on bond-wan and you’re trying to setup a container with another public IP assigned by your ISP.

Please can you show me the output of the follow items:

lxc config show <container> --expanded
ip a on host and in container (if you are redacting IPs, be sure not to redact last octet).
ip r on host and in container (if you are redacting IPs, be sure not to redact last octet).
ip neigh show proxy on host
iptables-save on host
Examples of ping tests from host to container and container to host working
Examples of ping tests you are running to try and access the container’s IP externally that are not working

You should expect to see as the IP address of 169.254.0.1 on the host-side veth interface, as this is used as a link-local next hop address. So that is normal.

Darwin · April 8, 2020, 7:44pm

Hi Thomas,

Yes, this is exactly that I want to do. I have a pool of Public IPs one for host and the others for each container. I’m sending you the result of all commands.

1.lxc config show --expanded

root@copark:~# lxc config show container --expanded  
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu bionic amd64 (20200402_08:37)
  image.os: Ubuntu
  image.release: bionic
  image.serial: "20200402_08:37"
  image.type: squashfs
  volatile.base_image: b2eb08cbfada3ea1301a9ee973fba91aef8a588fee56fb70d084b6306cb741a0
  volatile.eth0.host_name: veth2bb85d9a
  volatile.eth0.hwaddr: 00:16:3e:f6:56:14
  volatile.eth0.last_state.created: "false"
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    ipv4.address: 200.119.xx.xx
    nictype: routed
    parent: bond-wan
    type: nic
  root:
    path: /
    pool: lxdpool
    type: disk
ephemeral: false
profiles:
- default
- routed
stateful: false
description: ""

             ==================== // ====================

ip a on host and in container (if you are redacting IPs, be sure not to redact last octet).

root@copark:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,SLAVE> mtu 1500 qdisc fq_codel master bond-wan state DOWN group default qlen 1000
    link/ether 12:ef:68:21:7c:77 brd ff:ff:ff:ff:ff:ff
3: enp0s6: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond-wan state UP group default qlen 1000
    link/ether 12:ef:68:21:7c:77 brd ff:ff:ff:ff:ff:ff
4: enp0s7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond-lan state UP group default qlen 1000
    link/ether a2:e4:54:74:fe:a9 brd ff:ff:ff:ff:ff:ff
5: enp0s8: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc fq_codel master bond-lan state UP group default qlen 1000
    link/ether a2:e4:54:74:fe:a9 brd ff:ff:ff:ff:ff:ff
6: bond-wan: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 12:ef:68:21:7c:77 brd ff:ff:ff:ff:ff:ff
    inet 200.119.xx.xx/xx brd 200.119.xx.xx scope global bond-wan
       valid_lft forever preferred_lft forever
    inet6 fe80::10ef:68ff:fe21:7c77/64 scope link 
       valid_lft forever preferred_lft forever
9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e6:89:f1:6e:4c:f6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 brd 192.168.10.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::e489:f1ff:fe6e:4cf6/64 scope link 
       valid_lft forever preferred_lft forever
10: bond-lan: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether a2:e4:54:74:fe:a9 brd ff:ff:ff:ff:ff:ff
22: veth2bb85d9a@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:0c:5b:f1:93:c2 brd ff:ff:ff:ff:ff:ff link-netnsid 2
    inet 169.254.0.1/32 scope global veth2bb85d9a
       valid_lft forever preferred_lft forever
    inet6 fe80::fc0c:5bff:fef1:93c2/64 scope link 
       valid_lft forever preferred_lft forever
36: vethcc508384@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:97:36:13:0b:b8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.0.1/32 scope global vethcc508384
       valid_lft forever preferred_lft forever
    inet6 fe80::fc97:36ff:fe13:bb8/64 scope link 
       valid_lft forever preferred_lft forever
42: veth1a2bf28d@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether ca:e7:61:ec:db:de brd ff:ff:ff:ff:ff:ff link-netnsid 3
43: veth3dc65222@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:f2:c6:0c:ae:d1 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet 169.254.0.1/32 scope global veth3dc65222
       valid_lft forever preferred_lft forever
    inet6 fe80::fcf2:c6ff:fe0c:aed1/64 scope link 
       valid_lft forever preferred_lft forever

root@container:~# ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
        inet6 ::1/128 scope host 
           valid_lft forever preferred_lft forever
    2: eth0@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
        link/ether 9e:1c:0c:08:64:fd brd ff:ff:ff:ff:ff:ff link-netnsid 0
        inet 192.168.15.150/24 brd 192.168.15.255 scope global eth0
           valid_lft forever preferred_lft forever
        inet6 fe80::9c1c:cff:fe08:64fd/64 scope link 
           valid_lft forever preferred_lft forever

              ==================== // ====================

ip r on host and in container (if you are redacting IPs, be sure not to redact last octet).

root@copark:~# ip r
default via 200.119.xx.xx dev bond-wan proto static 
192.168.10.0/24 dev br0 proto kernel scope link src 192.168.10.1 
192.168.10.204 dev veth3dc65222 scope link 
192.168.10.207 dev vethcc508384 scope link 
200.119.xx.xx/24 dev bond-wan proto kernel scope link src 200.119.xx.xx
200.119.xx.xx dev veth2bb85d9a scope link

root@container:~# ip r
default via 169.254.0.1 dev eth0 proto static onlink 
200.119.xx.xx/xx dev eth0 proto kernel scope link src 200.119.xx.xx

              ==================== // ====================

ip neigh show proxy on host

root@copark:~# ip neigh show proxy 
169.254.0.1 dev veth3dc65222  proxy
169.254.0.1 dev vethcc508384  proxy
169.254.0.1 dev veth2bb85d9a  proxy
192.168.10.204 dev bond-lan  proxy
192.168.10.207 dev bond-lan  proxy
200.119.xx.xx dev bond-wan  proxy

              ==================== // ====================

iptables-save on host

root@copark:~# iptables-save
# Generated by iptables-save v1.6.1 on Wed Apr  8 12:23:39 2020
*raw
:PREROUTING ACCEPT [695:121142]
:OUTPUT ACCEPT [193:38233]
COMMIT
# Completed on Wed Apr  8 12:23:39 2020
# Generated by iptables-save v1.6.1 on Wed Apr  8 12:23:39 2020
*nat
:PREROUTING ACCEPT [388:93802]
:INPUT ACCEPT [60:6406]
:OUTPUT ACCEPT [3:252]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o bond-wan -m conntrack --ctstate NEW -j SNAT --to-source 200.119.xx.xx
COMMIT
# Completed on Wed Apr  8 12:23:39 2020
# Generated by iptables-save v1.6.1 on Wed Apr  8 12:23:39 2020
*mangle
:PREROUTING ACCEPT [698:121298]
:INPUT ACCEPT [370:33902]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [196:39013]
:POSTROUTING ACCEPT [192:38213]
-A PREROUTING -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0x1fff --ctmask 0x1fff
-A INPUT -m conntrack --ctstate NEW -j CONNMARK --save-mark --nfmask 0x1fff --ctmask 0x1fff
-A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j CONNMARK --restore-mark --nfmask 0x1fff --ctmask 0x1fff
-A POSTROUTING -m conntrack --ctstate NEW -j CONNMARK --save-mark --nfmask 0x1fff --ctmask 0x1fff
COMMIT
# Completed on Wed Apr  8 12:23:39 2020
# Generated by iptables-save v1.6.1 on Wed Apr  8 12:23:39 2020
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT DROP [0:0]
:in_bridge - [0:0]
:in_bridge2wan - [0:0]
:in_lan - [0:0]
:in_lan2wan - [0:0]
:in_wan - [0:0]
:out_bridge - [0:0]
:out_bridge2wan - [0:0]
:out_lan - [0:0]
:out_lan2wan - [0:0]
:out_wan - [0:0]
-A INPUT -i lo -j ACCEPT
-A INPUT -i bond-wan -j in_wan
-A INPUT -i bond-lan -j in_lan
-A INPUT -i br0 -j in_bridge
-A INPUT -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A INPUT -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A INPUT -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A INPUT -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A INPUT -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A INPUT -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID INPUT:"
-A INPUT -m conntrack --ctstate INVALID -j DROP
-A INPUT -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A INPUT -m limit --limit 1/sec -j LOG --log-prefix "IN-unknown:"
-A INPUT -j DROP
-A FORWARD -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A FORWARD -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A FORWARD -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A FORWARD -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A FORWARD -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A FORWARD -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID FORWARD:"
-A FORWARD -m conntrack --ctstate INVALID -j DROP
-A FORWARD -i br0 -o bond-wan -j in_bridge2wan
-A FORWARD -i bond-wan -o br0 -j out_bridge2wan
-A FORWARD -i bond-lan -o bond-wan -j in_lan2wan
-A FORWARD -i bond-wan -o bond-lan -j out_lan2wan
-A FORWARD -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A FORWARD -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A FORWARD -m limit --limit 1/sec -j LOG --log-prefix "PASS-unknown:"
-A FORWARD -j DROP
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -o bond-wan -j out_wan
-A OUTPUT -o bond-lan -j out_lan
-A OUTPUT -o br0 -j out_bridge
-A OUTPUT -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A OUTPUT -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A OUTPUT -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A OUTPUT -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A OUTPUT -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A OUTPUT -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID OUTPUT:"
-A OUTPUT -m conntrack --ctstate INVALID -j DROP
-A OUTPUT -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A OUTPUT -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A OUTPUT -m limit --limit 1/sec -j LOG --log-prefix "OUT-unknown:"
-A OUTPUT -j DROP
-A in_bridge -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A in_bridge -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_bridge -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_bridge -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_bridge -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_bridge -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_bridge -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A in_bridge -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A in_bridge -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID in_bridge:"
-A in_bridge -m conntrack --ctstate INVALID -j DROP
-A in_bridge -m limit --limit 1/sec -j LOG --log-prefix "IN-bridge:"
-A in_bridge -j DROP
-A in_bridge2wan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A in_bridge2wan -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_lan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A in_lan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_lan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_lan -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_lan -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_lan -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_lan -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A in_lan -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A in_lan -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID in_lan:"
-A in_lan -m conntrack --ctstate INVALID -j DROP
-A in_lan -m limit --limit 1/sec -j LOG --log-prefix "IN-lan:"
-A in_lan -j DROP
-A in_lan2wan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A in_lan2wan -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A in_lan2wan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_lan2wan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_wan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A in_wan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_wan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A in_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A in_wan -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_wan -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_wan -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A in_wan -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A in_wan -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A in_wan -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID in_wan:"
-A in_wan -m conntrack --ctstate INVALID -j DROP
-A in_wan -m limit --limit 1/sec -j LOG --log-prefix "IN-wan:"
-A in_wan -j DROP
-A out_bridge -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A out_bridge -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A out_bridge -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_bridge -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_bridge -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_bridge -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_bridge -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_bridge -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_bridge -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A out_bridge -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A out_bridge -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID out_bridge:"
-A out_bridge -m conntrack --ctstate INVALID -j DROP
-A out_bridge -m limit --limit 1/sec -j LOG --log-prefix "OUT-bridge:"
-A out_bridge -j DROP
-A out_bridge2wan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A out_bridge2wan -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_bridge2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_lan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A out_lan -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A out_lan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_lan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_lan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_lan -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_lan -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_lan -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_lan -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A out_lan -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A out_lan -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID out_lan:"
-A out_lan -m conntrack --ctstate INVALID -j DROP
-A out_lan -m limit --limit 1/sec -j LOG --log-prefix "OUT-lan:"
-A out_lan -j DROP
-A out_lan2wan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A out_lan2wan -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A out_lan2wan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_lan2wan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_lan2wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_wan -p icmp -m conntrack --ctstate RELATED -j ACCEPT
-A out_wan -p tcp -m conntrack --ctstate RELATED -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG RST,ACK -j ACCEPT
-A out_wan -m conntrack --ctstate ESTABLISHED -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_wan -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper ftp -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper irc -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper sip -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper pptp -j ACCEPT
-A out_wan -m conntrack --ctstate RELATED,ESTABLISHED -m helper --helper proto_gre -j ACCEPT
-A out_wan -p tcp -m tcp --tcp-flags FIN,ACK FIN,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_wan -p tcp -m tcp --tcp-flags RST,ACK RST,ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_wan -p tcp -m tcp --tcp-flags ACK ACK -m conntrack --ctstate INVALID,NEW -j DROP
-A out_wan -p tcp -m tcp --tcp-flags RST RST -m conntrack --ctstate INVALID,NEW -j DROP
-A out_wan -p icmp -m icmp --icmp-type 3 -m conntrack --ctstate INVALID,NEW -j DROP
-A out_wan -m conntrack --ctstate INVALID -m limit --limit 1/sec -j LOG --log-prefix "BLOCKED INVALID out_wan:"
-A out_wan -m conntrack --ctstate INVALID -j DROP
-A out_wan -m limit --limit 1/sec -j LOG --log-prefix "OUT-wan:"
-A out_wan -j DROP
COMMIT
# Completed on Wed Apr  8 12:23:39 2020

              ==================== // ====================

Examples of ping tests from host to container and container to host working

Ping from Hots to continer

root@copark:~# ping -c4 200.119.xx.xx
PING 200.119.xx.xx (200.119.xx.xx) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
--- 200.119.xx.xx ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3071ms

Ping from container to host

root@container:~# ping -c4 200.119.xx.xx
PING 200.119.xx.xx (200.119.xx.xx) 56(84) bytes of data.

--- 200.119.xx.xx ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3054ms

              ==================== // ====================

Examples of ping tests you are running to try and access the container’s IP externally that are not working

Make ping from a remote laptop to the container

darwin@Darwins-MBP ~ % ping -c4 200.119.xx.xx
PING 200.119.xx.xx (200.119.xx.xx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

--- 200.119.xx.xx ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

tomp · April 9, 2020, 5:44am

Thanks for that.

I would say it looks like your firewall is blocking the traffic.

If you can, I would suggest disabling the firewall temporarily, to check it works without it, at least then you know where to focus your energy.

Focus first on getting host <-> container traffic working, before looking at external hosts.

The “sendmsg: Operation not permitted” strongly indicates a local firewall issue in my experience.

One last point, I’m confused why your container has the IP address 192.168.15.150 but you’ve assigned it an IP of 200.119.xx.xx in LXD, something doesn’t look right there either. You’ll want to either disable netplan from configuring eth0, and let LXD do it, or configure the same IP statically in netplan.

Darwin · April 9, 2020, 4:28pm

Hi Thomas thanks for checking my configuration.

I have been fixing everything inside the container and disabling the firewall on the host. I have good news related to have access from host to container and the container can surf on internet.

You are right, the problem was the firewall. Now I have a new issue. from outside (internet) I can’t reach the container as I showed you on test number 7

root@copark:~# lxc config show container --expanded  
    architecture: x86_64
    config:
      image.architecture: amd64
      image.description: Ubuntu bionic amd64 (20200402_08:37)
      image.os: Ubuntu
      image.release: bionic
      image.serial: "20200402_08:37"
      image.type: squashfs
      volatile.base_image: b2eb08cbfada3ea1301a9ee973fba91aef8a588fee56fb70d084b6306cb741a0
      volatile.eth0.host_name: vethcee4967d
      volatile.eth0.hwaddr: 00:16:3e:f6:56:14
      volatile.eth0.last_state.created: "false"
      volatile.eth0.name: eth0
      volatile.idmap.base: "0"
      volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
      volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
      volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
      volatile.last_state.power: RUNNING
    devices:
      eth0:
        ipv4.address: 200.119.xxx.xxx
        nictype: routed
        parent: bond-wan
        type: nic
      root:
        path: /
        pool: lxdpool
        type: disk
    ephemeral: false
    profiles:
    - default
    - routed
    stateful: false
    description: ""

root@container:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether be:c9:c7:a5:25:ae brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 200.119.xxx.xxx/x brd 200.119.xxx.xxx scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::bcc9:c7ff:fea5:25ae/64 scope link 
       valid_lft forever preferred_lft forever

root@copark:~# ip r
default via 200.119.xxx.xxx dev bond-wan proto static 
192.168.10.0/24 dev br0 proto kernel scope link src 192.168.10.1 
200.119.xxx.xxx/xx dev bond-wan proto kernel scope link src 200.119.xxx.xxx
200.119.xxx.xxx dev vethcee4967d scope link 

root@container:~# ip r
default via 169.254.0.1 dev eth0 proto static onlink 
200.119.xxx.xxx/xx dev eth0 proto kernel scope link src 200.119.xxx.xxx

root@copark:~# ip neigh show proxy
169.254.0.1 dev vethcee4967d  proxy
200.119.xxx.xxx dev bond-wan  proxy

root@copark:~# iptables-save
# Generated by iptables-save v1.6.1 on Thu Apr  9 09:04:49 2020
*raw
:PREROUTING ACCEPT [2206:413622]
:OUTPUT ACCEPT [470:82938]
COMMIT
# Completed on Thu Apr  9 09:04:49 2020
# Generated by iptables-save v1.6.1 on Thu Apr  9 09:04:49 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Thu Apr  9 09:04:49 2020
# Generated by iptables-save v1.6.1 on Thu Apr  9 09:04:49 2020
*mangle
:PREROUTING ACCEPT [2206:413622]
:INPUT ACCEPT [974:90771]
:FORWARD ACCEPT [10:818]
:OUTPUT ACCEPT [471:83214]
:POSTROUTING ACCEPT [481:84032]
COMMIT
# Completed on Thu Apr  9 09:04:49 2020
# Generated by iptables-save v1.6.1 on Thu Apr  9 09:04:49 2020
*filter
:INPUT ACCEPT [978:90979]
:FORWARD ACCEPT [10:818]
:OUTPUT ACCEPT [475:84302]
COMMIT
# Completed on Thu Apr  9 09:04:49 2020

root@copark:~# ping -c4 200.119.xxx.xxx
PING 200.119.xxx.xxx (200.119.xxx.xxx) 56(84) bytes of data.
64 bytes from 200.119.xxx.xxx: icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=2 ttl=64 time=0.063 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=3 ttl=64 time=0.241 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=4 ttl=64 time=0.093 ms

--- 200.119.xxx.xxx ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3068ms
rtt min/avg/max/mdev = 0.031/0.107/0.241/0.080 ms


root@container:~# ping -c4 200.119.xxx.xxx
PING 200.119.xxx.xxx (200.119.xxx.xxx) 56(84) bytes of data.
64 bytes from 200.119.xxx.xxx: icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=2 ttl=64 time=0.159 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=3 ttl=64 time=0.100 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=4 ttl=64 time=0.110 ms

--- 200.119.xxx.xxx ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3072ms
rtt min/avg/max/mdev = 0.059/0.107/0.159/0.035 ms


root@container:~# ping -c4 google.com
PING google.com (172.217.11.174) 56(84) bytes of data.
64 bytes from lax28s15-in-f14.1e100.net (172.217.11.174): icmp_seq=1 ttl=51 time=6.65 ms
64 bytes from lax28s15-in-f14.1e100.net (172.217.11.174): icmp_seq=2 ttl=51 time=9.15 ms
64 bytes from lax28s15-in-f14.1e100.net (172.217.11.174): icmp_seq=3 ttl=51 time=7.90 ms
64 bytes from lax28s15-in-f14.1e100.net (172.217.11.174): icmp_seq=4 ttl=51 time=8.87 ms

--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 6.655/8.146/9.150/0.978 ms

Ping from outside (internet) to my host

darwin@Darwins-MBP ~ % ping -c4 200.119.xxx.xxx
PING 200.119.xxx.xxx (200.119.xxx.xxx): 56 data bytes
64 bytes from 200.119.xxx.xxx: icmp_seq=0 ttl=64 time=0.285 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=1 ttl=64 time=0.449 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=2 ttl=64 time=0.311 ms
64 bytes from 200.119.xxx.xxx: icmp_seq=3 ttl=64 time=0.318 ms

--- 200.119.xxx.xxx ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.285/0.341/0.449/0.064 ms

Ping from outside (internet) to my container

darwin@Darwins-MBP ~ % ping -c4 200.119.xxx.xxx
PING 200.119.xxx.xxx (200.119.xxx.xxx): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2

--- 200.119.xxx.xxx ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

tomp · April 9, 2020, 4:35pm

Thanks good to hear.

Can you private message me the actual IPs and subnet masks you are using for the public IPs on the host and container please.

I’d also like to see the output of tcpdump -l -nn -i bond-wan host 200.119.xxx.xxx (the container’s IP) whilst you are trying to ping it externally.

tomp · April 9, 2020, 4:37pm

And the output of:

sysctl net.ipv4.conf.bond-wan.forwarding