LXD cluster and lxdfan isolation

Hi,

I am trying to configure an LXD cluster with multiples isolated fan networks.
I didn’t understand how I should do to get it done.

This is my set up :

$ lxc cluster list
+------+----------------------------+----------+--------+-------------------+
| NAME |            URL             | DATABASE | STATE  |      MESSAGE      |
+------+----------------------------+----------+--------+-------------------+
| lxd1 | https://192.168.56.11:8443 | YES      | ONLINE | fully operational |
+------+----------------------------+----------+--------+-------------------+
| lxd2 | https://192.168.56.12:8443 | YES      | ONLINE | fully operational |
+------+----------------------------+----------+--------+-------------------+
| lxd3 | https://192.168.56.13:8443 | YES      | ONLINE | fully operational |
+------+----------------------------+----------+--------+-------------------+

$ lxc network list
+---------+----------+---------+-------------+---------+---------+
|  NAME   |   TYPE   | MANAGED | DESCRIPTION | USED BY |  STATE  |
+---------+----------+---------+-------------+---------+---------+
| enp0s3  | physical | NO      |             | 0       |         |
+---------+----------+---------+-------------+---------+---------+
| enp0s8  | physical | NO      |             | 0       |         |
+---------+----------+---------+-------------+---------+---------+
| lxdfan0 | bridge   | YES     |             | 3       | CREATED |
+---------+----------+---------+-------------+---------+---------+
| lxdfan1 | bridge   | YES     |             | 2       | PENDING |
+---------+----------+---------+-------------+---------+---------+

$ lxc network show lxdfan0
config:
  bridge.mode: fan
  fan.overlay_subnet: 240.1.0.0/16
  fan.underlay_subnet: 192.168.56.0/24
description: ""
name: lxdfan0
type: bridge
used_by:
- /1.0/containers/c1
- /1.0/containers/c2
- /1.0/containers/c3
managed: true
status: Created
locations:
- lxd1
- lxd2
- lxd3

$ lxc network show lxdfan1
config:
  bridge.mode: fan
  fan.overlay_subnet: 240.2.0.0/16
  fan.underlay_subnet: 192.168.56.0/24
description: ""
name: lxdfan1
type: bridge
used_by:
- /1.0/containers/d1
- /1.0/containers/d2
managed: true
status: Pending
locations:
- lxd2

$ lxc list
+------+---------+---------------------+------+------------+-----------+----------+
| NAME |  STATE  |        IPV4         | IPV6 |    TYPE    | SNAPSHOTS | LOCATION |
+------+---------+---------------------+------+------------+-----------+----------+
| c1   | RUNNING | 240.1.11.155 (eth0) |      | PERSISTENT |           | lxd1     |
+------+---------+---------------------+------+------------+-----------+----------+
| c2   | RUNNING | 240.1.12.88 (eth0)  |      | PERSISTENT |           | lxd2     |
+------+---------+---------------------+------+------------+-----------+----------+
| c3   | RUNNING | 240.1.13.187 (eth0) |      | PERSISTENT |           | lxd3     |
+------+---------+---------------------+------+------------+-----------+----------+
| d1   | RUNNING | 240.2.11.165 (eth0) |      | PERSISTENT |           | lxd1     |
+------+---------+---------------------+------+------------+-----------+----------+
| d2   | RUNNING | 240.2.12.160 (eth0) |      | PERSISTENT |           | lxd2     |
+------+---------+---------------------+------+------------+-----------+----------+
  • c1 can ping cX and dX (dX should be forbidden)
  • d1 can ping dX and cX (cX should be forbidden)

So I checked iptables rules, traffic is authorized between lxdfan networks:

Chain INPUT (policy ACCEPT 102K packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  lxdfan1 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdfan1 */
    4   277 ACCEPT     udp  --  lxdfan1 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdfan1 */
    1   312 ACCEPT     udp  --  lxdfan1 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdfan1 */
    0     0 ACCEPT     tcp  --  lxdfan0 *       0.0.0.0/0            0.0.0.0/0            tcp dpt:53 /* generated for LXD network lxdfan0 */
    3   205 ACCEPT     udp  --  lxdfan0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:53 /* generated for LXD network lxdfan0 */
    2   624 ACCEPT     udp  --  lxdfan0 *       0.0.0.0/0            0.0.0.0/0            udp dpt:67 /* generated for LXD network lxdfan0 */

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  242 23364 ACCEPT     all  --  *      lxdfan1  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdfan1 */
  242 20731 ACCEPT     all  --  lxdfan1 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdfan1 */
   12  4044 ACCEPT     all  --  *      lxdfan0  0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdfan0 */
   12  1411 ACCEPT     all  --  lxdfan0 *       0.0.0.0/0            0.0.0.0/0            /* generated for LXD network lxdfan0 */

Chain OUTPUT (policy ACCEPT 93036 packets, 13M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     tcp  --  *      lxdfan1  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdfan1 */
    3   237 ACCEPT     udp  --  *      lxdfan1  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdfan1 */
    1   329 ACCEPT     udp  --  *      lxdfan1  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdfan1 */
    0     0 ACCEPT     tcp  --  *      lxdfan0  0.0.0.0/0            0.0.0.0/0            tcp spt:53 /* generated for LXD network lxdfan0 */
    3   237 ACCEPT     udp  --  *      lxdfan0  0.0.0.0/0            0.0.0.0/0            udp spt:53 /* generated for LXD network lxdfan0 */
    2   658 ACCEPT     udp  --  *      lxdfan0  0.0.0.0/0            0.0.0.0/0            udp spt:67 /* generated for LXD network lxdfan0 */

So, I am a bit confused because I really don’t know what’s the best way do to that.
If you have any ideas…

Cheers

LXD doesn’t directly let you manage complex firewalling, that said, you can turn off LXD’s own firewalling management with the ipv4.firewall option which would then let you take control of this using your host system’s firewall.

Thank you Stéphane,

I tried to set ipv4.firewall on my lxdfan networks but it’s forbidden with mode.fan enabled
Could you confirm me what’s the best way to isolate lxdfan on a lxd cluster ?

I managed to drop the traffic between my interfaces but it seems to be complicate if I want to add more networks…

$ sudo iptables -I FORWARD -i lxdfan1 -o lxdfan0 -j DROP
$ sudo iptables -I FORWARD -i lxdfan0 -o lxdfan1 -j DROP