[solved] Migrating to a different network + breaking fan network (single node cluster) by mistake

killua · March 31, 2024, 10:13am

Hi all! I had the brilliant idea to move a lxd server before migrating it to incus to a different network / infra (from 192.168.88.102 to 10.65.0.99) while half asleep. I started to play around with a surrogate lxd instance and dabbled around the “lxd sql global” commands to see what happens under the hood. I managed to be careless and do some of that without noticing on the still in production server until it was too late. I’m now half way there with reviewing the sqlite3 dump and the interface having 192.168.88.102/24 manually added.

2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether d4:5d:df:1a:22:1a brd ff:ff:ff:ff:ff:ff
    inet 10.65.0.99/24 metric 100 brd 10.65.0.255 scope global dynamic enp0s31f6
       valid_lft 319sec preferred_lft 319sec
    inet 192.168.88.102/24 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::d65d:dfff:fe1a:221a/64 scope link
       valid_lft forever preferred_lft forever

On restarting lxd like this, i get

Mar 31 09:51:24 kai02 lxd.daemon[710150]: Device "lxdfan0" does not exist.
Mar 31 09:51:24 kai02 resolvectl[710154]: Failed to resolve interface "lxdfan0": No such device

Before doing more damage, I’m stopping by to say hi and ask for help. pointers on how to un**** my deeds and finish the migration to 10.65.0.99 will be greatly appreciated

killua · March 31, 2024, 11:22am

ar 31 11:14:52 kai02 systemd[1]: Starting Service for snap application lxd.daemon...
Mar 31 11:15:12 kai02 lxd.daemon[14626]: => Preparing the system (27049)
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Loading snap configuration
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Setting up mntns symlink (mnt:[4026532476])
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Setting up kmod wrapper
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Preparing /boot
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Preparing a clean copy of /run
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Preparing /run/bin
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Preparing a clean copy of /etc
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Preparing a clean copy of /usr/share/misc
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Setting up ceph configuration
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Setting up LVM configuration
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Setting up OVN configuration
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Rotating logs
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Setting up ZFS (2.1)
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Escaping the systemd cgroups
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ====> Detected cgroup V2
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Escaping the systemd process resource limits
Mar 31 11:15:12 kai02 lxd.daemon[14626]: ==> Exposing LXD documentation
Mar 31 11:15:13 kai02 lxd.daemon[8949]: Closed liblxcfs.so
Mar 31 11:15:13 kai02 lxd.daemon[8949]: Running destructor lxcfs_exit
Mar 31 11:15:13 kai02 lxd.daemon[8949]: Running constructor lxcfs_init to reload liblxcfs
Mar 31 11:15:13 kai02 lxd.daemon[8949]: mount namespace: 6
Mar 31 11:15:13 kai02 lxd.daemon[8949]: hierarchies:
Mar 31 11:15:13 kai02 lxd.daemon[8949]:   0: fd:   8: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Mar 31 11:15:13 kai02 lxd.daemon[8949]: Kernel supports pidfds
Mar 31 11:15:13 kai02 lxd.daemon[8949]: Kernel does not support swap accounting
Mar 31 11:15:13 kai02 lxd.daemon[8949]: api_extensions:
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - cgroups
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - sys_cpu_online
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_cpuinfo
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_diskstats
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_loadavg
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_meminfo
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_stat
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_swaps
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_uptime
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - proc_slabinfo
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - shared_pidns
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - cpuview_daemon
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - loadavg_daemon
Mar 31 11:15:13 kai02 lxd.daemon[8949]: - pidfds
Mar 31 11:15:13 kai02 lxd.daemon[8949]: Reloaded LXCFS
Mar 31 11:15:13 kai02 lxd.daemon[14626]: => Re-using existing LXCFS
Mar 31 11:15:13 kai02 lxd.daemon[14626]: ==> Reloading LXCFS
Mar 31 11:15:13 kai02 lxd.daemon[14626]: => Starting LXD
Mar 31 11:15:13 kai02 lxd.daemon[14792]: time="2024-03-31T11:15:13Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use p>
Mar 31 11:15:14 kai02 lxd.daemon[14792]: time="2024-03-31T11:15:14Z" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to update node >
Mar 31 11:15:14 kai02 lxd.daemon[14792]: time="2024-03-31T11:15:14Z" level=warning msg="Could not handover member's responsibilities" err="Failed to transfer leadership: No online voter found"
Mar 31 11:15:14 kai02 lxd.daemon[14792]: Error: Failed to initialize global database: failed to ensure schema: failed to update node version info: updated 0 rows instead of 1
Mar 31 11:15:15 kai02 lxd.daemon[14626]: Killed
Mar 31 11:15:15 kai02 lxd.daemon[14626]: => LXD failed to start
Mar 31 11:15:15 kai02 systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Mar 31 11:15:27 kai02 lxd.daemon[14839]: Device "lxdfan0" does not exist.
Mar 31 11:15:27 kai02 resolvectl[14843]: Failed to resolve interface "lxdfan0": No such device
Mar 31 11:15:27 kai02 resolvectl[14846]: Failed to resolve interface "lxdfan0": No such device
Mar 31 11:15:27 kai02 systemd[1]: snap.lxd.daemon.service: Control process exited, code=exited, status=1/FAILURE
Mar 31 11:15:27 kai02 systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Mar 31 11:15:27 kai02 systemd[1]: Failed to start Service for snap application lxd.daemon.
Mar 31 11:15:28 kai02 systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 6.
Mar 31 11:15:28 kai02 systemd[1]: Stopped Service for snap application lxd.daemon.
Mar 31 11:15:28 kai02 systemd[1]: Starting Service for snap application lxd.daemon...
Mar 31 11:15:48 kai02 lxd.daemon[15380]: => Preparing the system (27049)
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Loading snap configuration
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Setting up mntns symlink (mnt:[4026532476])
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Setting up kmod wrapper
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Preparing /boot
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Preparing a clean copy of /run
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Preparing /run/bin
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Preparing a clean copy of /etc
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Preparing a clean copy of /usr/share/misc
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Setting up ceph configuration
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Setting up LVM configuration
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Setting up OVN configuration
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Rotating logs
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Setting up ZFS (2.1)
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Escaping the systemd cgroups
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ====> Detected cgroup V2
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Escaping the systemd process resource limits
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Exposing LXD documentation
Mar 31 11:15:48 kai02 lxd.daemon[8949]: Closed liblxcfs.so
Mar 31 11:15:48 kai02 lxd.daemon[8949]: Running destructor lxcfs_exit
Mar 31 11:15:48 kai02 lxd.daemon[8949]: Running constructor lxcfs_init to reload liblxcfs
Mar 31 11:15:48 kai02 lxd.daemon[8949]: mount namespace: 6
Mar 31 11:15:48 kai02 lxd.daemon[8949]: hierarchies:
Mar 31 11:15:48 kai02 lxd.daemon[8949]:   0: fd:   8: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Mar 31 11:15:48 kai02 lxd.daemon[8949]: Kernel supports pidfds
Mar 31 11:15:48 kai02 lxd.daemon[8949]: Kernel does not support swap accounting
Mar 31 11:15:48 kai02 lxd.daemon[8949]: api_extensions:
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - cgroups
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - sys_cpu_online
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_cpuinfo
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_diskstats
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_loadavg
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_meminfo
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_stat
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_swaps
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_uptime
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - proc_slabinfo
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - shared_pidns
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - cpuview_daemon
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - loadavg_daemon
Mar 31 11:15:48 kai02 lxd.daemon[8949]: - pidfds
Mar 31 11:15:48 kai02 lxd.daemon[8949]: Reloaded LXCFS
Mar 31 11:15:48 kai02 lxd.daemon[15380]: => Re-using existing LXCFS
Mar 31 11:15:48 kai02 lxd.daemon[15380]: ==> Reloading LXCFS
Mar 31 11:15:48 kai02 lxd.daemon[15380]: => Starting LXD
Mar 31 11:15:48 kai02 lxd.daemon[15543]: time="2024-03-31T11:15:48Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use p>
Mar 31 11:15:49 kai02 lxd.daemon[15543]: time="2024-03-31T11:15:49Z" level=error msg="Failed to start the daemon" err="Failed to initialize global database: failed to ensure schema: failed to update node >
Mar 31 11:15:49 kai02 lxd.daemon[15543]: time="2024-03-31T11:15:49Z" level=warning msg="Could not handover member's responsibilities" err="Failed to transfer leadership: No online voter found"
Mar 31 11:15:49 kai02 lxd.daemon[15543]: Error: Failed to initialize global database: failed to ensure schema: failed to update node version info: updated 0 rows instead of 1

killua · March 31, 2024, 11:46am

Couldn’t resist poking around more. There was a typo on the local db too, so i fixed that and now lxd starts, just the fan network fails to pop to existence

# sqlite3 /var/snap/lxd/common/lxd/database/local.db "select * from config"
2|cluster.https_address|10.65.0.99:9443
4|core.https_address|10.65.0.99:8443
# sqlite3 /var/snap/lxd/common/lxd/database/local.db "select * from raft_nodes"
1|10.65.0.99:9443|0|kai02

killua · March 31, 2024, 12:19pm

Progressing:

~$ lxc network list
+-------------+----------+---------+------+------+-------------+---------+---------+
|    NAME     |   TYPE   | MANAGED | IPV4 | IPV6 | DESCRIPTION | USED BY |  STATE  |
+-------------+----------+---------+------+------+-------------+---------+---------+
| cni-podman1 | bridge   | NO      |      |      |             | 0       |         |
+-------------+----------+---------+------+------+-------------+---------+---------+
| cni-podman2 | bridge   | NO      |      |      |             | 0       |         |
+-------------+----------+---------+------+------+-------------+---------+---------+
| enp0s31f6   | physical | NO      |      |      |             | 0       |         |
+-------------+----------+---------+------+------+-------------+---------+---------+
| lxdfan0     | bridge   | YES     |      |      |             | 23      | CREATED |
+-------------+----------+---------+------+------+-------------+---------+---------+
| wlp1s0      | physical | NO      |      |      |             | 0       |         |
+-------------+----------+---------+------+------+-------------+---------+---------+

after lxc network edit lxdfan0 where I set the

config:
  bridge.mode: fan
  fan.underlay_subnet: 10.65.0.0/24
  ipv4.nat: "true"

some caontainers went up.

killua · March 31, 2024, 12:30pm

to make the remaining containers go up, i needed to review

$ lxc config show kai02-gitea
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu bionic amd64 (20181101_07:42)
  image.os: Ubuntu
  image.release: bionic
  image.serial: "20181101_07:42"
  volatile.base_image: 72f74ccbb968fca13a680b22d00bbc64636bbc65a870dec79d7c711f7f8b9d85
  volatile.eth0.hwaddr: 00:16:3e:e3:e8:34
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: b9eedd83-f8ac-48e2-999f-af0400a2e10a
devices:
  gitea-ssh:
    bind: host
    connect: tcp:127.0.0.1:2222
    listen: tcp:192.168.88.102:2222
    type: proxy
ephemeral: false
profiles:
- default
stateful: false
description: ""

and use lxc config edit kai02-gitea to change listen: tcp:192.168.88.102:2222 to the new ip, listen: tcp:10.65.0.99:2222 … this seems to conclude the whole problem,

system · April 30, 2024, 12:31pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.