Hi all,
Currently I am trying to deploy openstack to an lxd cloud using juju.
My setup consists of 3 lxd vms with parameters specified in deployment guide.
One of the steps creates mysql innodb cluster on container nodes created within vms created by lxd in host machines of the cluster.
I have set up a fan network with vxlan tunneling and custom overlay and underlay network:
config:
boot.autostart: "true"
limits.cpu: "6"
limits.memory: 12GB
security.nesting: "true"
security.secureboot: "false"
user.network-config: |
version: 1
config:
- type: physical
name: enp5s0
subnets:
- type: dhcp
- type: physical
name: enp6s0
subnets:
- type: dhcp
- type: nameserver
address: 1.1.1.1
description: LXD profile for juju controller based on VLAN99 pxe boot
devices:
eth0:
name: fan-240
nictype: bridged
parent: lxdfan0
type: nic
eth1:
name: eth1
nictype: bridged
parent: testbr0
type: nic
eth2:
name: eth2
nictype: macvlan
parent: VLAN10
type: nic
eth3:
name: eth3
nictype: macvlan
parent: VLAN999
type: nic
root:
path: /
pool: remote-lvm
size: 55GB
type: disk
name: juju-testing
mother@infra2:~$ lxc network show testbr0
config:
ipv4.address: 10.187.24.1/24
ipv4.nat: "true"
ipv6.address: none
ipv6.nat: "true"
description: ""
name: testbr0
type: bridge
config:
bridge.mode: fan
fan.overlay_subnet: 240.0.0.0/16
fan.underlay_subnet: 10.10.11.0/24
ipv4.nat: "true"
description: ""
name: lxdfan0
type: bridge
the problem is that the containers within my vms cannot communicate across vms(on the other hand vms can ping containers from another vm)
How can I further trhoubleshoot the container-container communication?
I came up with following test:
model: testing
machines:
"0":
juju-status:
current: started
since: 11 Feb 2023 08:18:58Z
version: 3.1-rc2
hostname: juju-4e4e14-0
dns-name: 10.187.24.121
ip-addresses:
- 10.187.24.121
- 240.0.12.38
instance-id: juju-4e4e14-0
machine-status:
current: running
message: Running
since: 11 Feb 2023 08:17:00Z
modification-status:
current: applied
since: 11 Feb 2023 08:16:27Z
base:
name: ubuntu
channel: "22.04"
network-interfaces:
enp5s0:
ip-addresses:
- 240.0.12.38
mac-address: 00:16:3e:4a:d0:50
space: alpha
is-up: true
enp6s0:
ip-addresses:
- 10.187.24.121
mac-address: 00:16:3e:d0:b6:44
gateway: 10.187.24.1
space: alpha
is-up: true
lxdbr0:
ip-addresses:
- 10.83.139.1
mac-address: 00:16:3e:4c:e5:af
is-up: true
containers:
0/lxd/0:
juju-status:
current: started
since: 11 Feb 2023 08:24:48Z
version: 3.1-rc2
hostname: juju-4e4e14-0-lxd-0
dns-name: 10.83.139.174
ip-addresses:
- 10.83.139.174
instance-id: juju-4e4e14-0-lxd-0
machine-status:
current: running
message: Container started
since: 11 Feb 2023 08:23:53Z
modification-status:
current: applied
since: 11 Feb 2023 08:23:53Z
base:
name: ubuntu
channel: "22.04"
network-interfaces:
eth0:
ip-addresses:
- 10.83.139.174
mac-address: 00:16:3e:86:16:95
gateway: 10.83.139.1
is-up: true
constraints: arch=amd64 spaces=
constraints: arch=amd64 root-disk=51200M root-disk-source=local-lvm virt-type=virtual-machine
hardware: arch=amd64 cores=0 mem=0M virt-type=virtual-machine
juju model-config | egrep 'fan-config|container-networking-metho
d'
container-networking-method model local
fan-config model 10.10.11.0/24=240.0.0.0/16
The second scenario assumes setting container-networking-method to fan
container-networking-method model fan
fan-config model 10.10.11.0/24=240.0.0.0/16
In former case containers just cannot talk with each other:
root@juju-client:~/openstack# j ssh 0 #connect to 0 machine
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-1025-kvm x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Sat Feb 11 08:33:19 UTC 2023
System load: 0.2783203125 Users logged in: 0
Usage of /: 11.8% of 48.27GB IPv4 address for enp5s0: 240.0.12.38
Memory usage: 9% IPv4 address for enp6s0: 10.187.24.121
Swap usage: 0% IPv4 address for lxdbr0: 10.83.139.1
Processes: 166
0 updates can be applied immediately.
*** System restart required ***
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ubuntu@juju-4e4e14-0:~$ sudo -i
root@juju-4e4e14-0:~# lxc ls
To start your first container, try: lxc launch ubuntu:22.04
Or for a virtual machine: lxc launch ubuntu:22.04 --vm
+---------------------+---------+----------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------------+---------+----------------------+------+-----------+-----------+
| juju-4e4e14-0-lxd-0 | RUNNING | 10.83.139.174 (eth0) | | CONTAINER | 0 |
+---------------------+---------+----------------------+------+-----------+-----------+
root@juju-4e4e14-0:~# lxc shell juju-4e4e14-0-lxd-0
root@juju-4e4e14-0-lxd-0:~# ping 10.79.209.219
PING 10.79.209.219 (10.79.209.219) 56(84) bytes of data.
^C
--- 10.79.209.219 ping statistics ---
35 packets transmitted, 0 received, 100% packet loss, time 34826ms
root@juju-4e4e14-0-lxd-0:~# ip r
default via 10.83.139.1 dev eth0 proto dhcp src 10.83.139.174 metric 100
10.83.139.0/24 dev eth0 proto kernel scope link src 10.83.139.174 metric 100
10.83.139.1 dev eth0 proto dhcp scope link src 10.83.139.174 metric 100
and in the latter I get the information that there are no FAN devices in alpha space
@stgraber I can recall that you elaborated on how the fan works in an lxd cluster
in the Help with DNS on LXD 3.12 cluster with FAN network - #15 by Yosu_Cadilla
Has anything improved/changed since then?
Any help appreciated
Thanks