Initially I thought that it may be some kind of overlapping of ip, since including 4 in command lxc ls -c 4n causes the output to look this way, but that is not possible, is it?
mother@infra2:~$ lxc exec juju-5b99f0-0 -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth1@if98 UP 240.11.0.131/8 metric 100 fe80::216:3eff:fe54:d0f9/64
eth2@if12 UP fe80::216:3eff:fe89:2cd3/64
mother@infra2:~$ lxc exec maas-ha-1 -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth1@if101 UP 240.11.0.179/8 metric 100 fe80::216:3eff:febb:5d64/64
eth2@if12 UP 10.10.10.101/24 fe80::216:3eff:fe31:7028/64
eth3@if11 UP 10.10.99.101/24 fe80::216:3eff:fe51:bc86/64
mother@infra2:~$ lxc exec maas-ha-db-1 -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth1@if105 UP 240.11.0.231/8 metric 100 fe80::216:3eff:fe0f:c3cc/64
eth2@if12 UP 10.10.10.105/24 fe80::216:3eff:fee2:f97a/64
mother@infra2:~$ lxc exec maas-ha-db-2 -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth1@if100 UP 240.12.0.37/8 metric 100 fe80::216:3eff:fea0:c689/64
eth2@if12 UP 10.10.10.106/24 fe80::216:3eff:fe66:6053/64
mother@infra2:~$ lxc exec maas-proxy-1 -- ip -br a
lo UNKNOWN 127.0.0.1/8 ::1/128
eth1@if108 UP 240.11.0.177/8 metric 100 fe80::216:3eff:fe6d:a14a/64
eth2@if12 UP 10.10.10.104/24 fe80::216:3eff:feae:63d5/64
but why does it behave like this?
The only possble explanation may be me having run lxc network unset lxdfan0 ipv6.address ; lxc network unset lxdfan0 ipv6.nat
But since ipv6 cannot be handled with in fan mode I doubt it had any effectâŚ
Any ideas whatâs different about them? Or if you run the command repeatedly do they show? Also have you tried adding a filter for the name of one of the missing instances (while still keeping the 4 option) and see if it appears?
I also noticed a very strange thing, namely some new problems with maas installation on these containers, as the name suggests they are used for maas installation.
The problem with listing tends to coincide with the maas initialisation error
ERROR: Failed to retrieve storage information: Failed to find â/dev/disk/by-path/pci-0000:00:1f.2-ata-1â: lstat /dev/sr0: no such file or directory
@tomp thank you very much, I jest realized I assigned wrong pool to my containers, but unfortunately it did not solve anything All continers use remote pool now but lxc ls still shows no valid output.
By the way I just found that the local zfs pool still has some images after removing the provious containers, is it possible to somehow prune zfs pool?
Yesterday I faced a problem with microceph(ceph rbd wan not able to unmount the instances located on 2 of 3 cluster members) and due to lack of time I decided to simply restart those 2 cluster members.
It seems the restart has resolved the issue for now but it is quite worrying to be honestâŚ
@tomp do you have any suspects of this? Could it be that one member was somehow causing the DB not to refresh?
1.3.2023 UPDATE
No I am sure that one of cluster nodes is causing the problem.
Today I observed another thing, namely, the 3 containers that have maas installed have tendency to freeze completely, what I mean by that is the fact that it is not possible neither to lxc shell into them or exec anything, the stop freezes. The only thing that works is to force stop and then start:
An example would be:
architecture: x86_64
config:
boot.autostart: "true"
image.architecture: amd64
image.description: Ubuntu jammy amd64 (20230228_07:42)
image.os: Ubuntu
image.release: jammy
image.serial: "20230228_07:42"
image.type: squashfs
image.variant: cloud
limits.cpu: "2"
limits.memory: 4GB
linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
user.network-config: |-
version: 1
config:
- type: physical
name: eth1
subnets:
- type: static
ipv4: true
address: 10.10.10.101/24
netmask: 255.255.255.0
gateway: 10.10.10.1
control: auto
- type: physical
name: eth2
subnets:
- type: dhcp
- type: physical
name: eth3
subnets:
- type: static
ipv4: true
address: 10.10.99.101/24
netmask: 255.255.255.0
control: auto
- type: nameserver
address: 1.1.1.1
user.user-data: |
#cloud-config
packages:
- snapd
- ssh
- net-tools
- jq
- nano
- traceroute
- screen
- iptables
users:
- default
volatile.base_image: d5843acf2be2772f5fc532fa2537d0e2082ea11c131ad5bb857a389c99be7e10
volatile.cloud-init.instance-id: f274c794-733b-4052-af31-a73bc194963c
volatile.eth1.host_name: mac975e636d
volatile.eth1.hwaddr: 00:16:3e:9b:ca:32
volatile.eth1.last_state.created: "false"
volatile.eth2.host_name: veth95b95baa
volatile.eth2.hwaddr: 00:16:3e:6a:bf:ff
volatile.eth3.host_name: macd9fbbb35
volatile.eth3.hwaddr: 00:16:3e:2b:1c:a3
volatile.eth3.last_state.created: "false"
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.uuid: d7bb74c2-f6b4-413e-83d9-95cb68f59c35
devices:
eth1:
name: eth1
nictype: macvlan
parent: VLAN10
type: nic
eth2:
name: eth2
network: lxdfan0
type: nic
eth3:
name: eth3
nictype: macvlan
parent: VLAN99
type: nic
root:
path: /
pool: remote
size: 25GB
type: disk
ephemeral: false
profiles:
- container-vlan10-maas
stateful: false
CONSOLE
[52192.524693] cloud-init[204]: Cloud-init v. 22.4.2-0ubuntu0~22.04.1 running 'init' at Wed, 01 Mar 2023 11:47:57 +0000. Up 17.73 seconds.
[52192.524889] cloud-init[204]: ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
[52192.525031] cloud-init[204]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
[52192.525174] cloud-init[204]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
[52192.525320] cloud-init[204]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
[52192.525459] cloud-init[204]: ci-info: | eth1 | True | 10.10.10.102 | 255.255.255.0 | global | 00:16:3e:97:c2:36 |
[52192.525584] cloud-init[204]: ci-info: | eth1 | True | fe80::216:3eff:fe97:c236/64 | . | link | 00:16:3e:97:c2:36 |
[52192.525704] cloud-init[204]: ci-info: | eth2 | True | 240.12.0.183 | 255.0.0.0 | global | 00:16:3e:cb:4a:19 |
[52192.525823] cloud-init[204]: ci-info: | eth2 | True | fe80::216:3eff:fecb:4a19/64 | . | link | 00:16:3e:cb:4a:19 |
[52192.525944] cloud-init[204]: ci-info: | eth3 | True | 10.10.99.102 | 255.255.255.0 | global | 00:16:3e:71:e2:84 |
[52192.526083] cloud-init[204]: ci-info: | eth3 | True | fe80::216:3eff:fe71:e284/64 | . | link | 00:16:3e:71:e2:84 |
[52192.526206] cloud-init[204]: ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | host | . |
[52192.526326] cloud-init[204]: ci-info: | lo | True | ::1/128 | . | host | . |
[52192.526445] cloud-init[204]: ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
[52192.526566] cloud-init[204]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
[52192.526696] cloud-init[204]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+
[52192.526818] cloud-init[204]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
[52192.526936] cloud-init[204]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+
[52192.527056] cloud-init[204]: ci-info: | 0 | 0.0.0.0 | 10.10.10.1 | 0.0.0.0 | eth1 | UG |
[52192.527177] cloud-init[204]: ci-info: | 1 | 0.0.0.0 | 240.12.0.1 | 0.0.0.0 | eth2 | UG |
[52192.527310] cloud-init[204]: ci-info: | 2 | 10.10.10.0 | 0.0.0.0 | 255.255.255.0 | eth1 | U |
[ OK ] Started Ubuntu Advantage Timer for running repeated jobs.
[52192.527951] cloud-init[204]: ci-info: | 3 | 10.10.99.0 | 0.0.0.0 | 255.255.255.0 | eth3 | U |
[52192.528080] cloud-init[204]: ci-info: | 4 | 240.0.0.0 | 0.0.0.0 | 255.0.0.0 | eth2 | U |
[52192.528202] cloud-init[204]: ci-info: | 5 | 240.12.0.1 | 0.0.0.0 | 255.255.255.255 | eth2 | UH |
[52192.528324] cloud-init[204]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+
[52192.528462] cloud-init[204]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[52192.528585] cloud-init[204]: ci-info: +-------+-------------+---------+-----------+-------+
[52192.528754] cloud-init[204]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[52192.528877] cloud-init[204]: ci-info: +-------+-------------+---------+-----------+-------+
[52192.528997] cloud-init[204]: ci-info: | 0 | fe80::/64 | :: | eth1 | U |
[52192.529117] cloud-init[204]: ci-info: | 1 | fe80::/64 | :: | eth2 | U |
[52192.529239] cloud-init[204]: ci-info: | 2 | fe80::/64 | :: | eth3 | U |
[52192.529358] cloud-init[204]: ci-info: | 4 | local | :: | eth3 | U |
[52192.529497] cloud-init[204]: ci-info: | 5 | local | :: | eth1 | U |
[52192.529618] cloud-init[204]: ci-info: | 6 | local | :: | eth2 | U |
[52192.529737] cloud-init[204]: ci-info: | 7 | multicast | :: | eth1 | U |
[52192.529871] cloud-init[204]: ci-info: | 8 | multicast | :: | eth2 | U |
[52192.529992] cloud-init[204]: ci-info: | 9 | multicast | :: | eth3 | U |
[52192.530112] cloud-init[204]: ci-info: +-------+-------------+---------+-----------+-------+
[ OK ] Reached target Timer Units.
[ OK ] Listening on cloud-init hotplug hook socket.
[ OK ] Listening on D-Bus System Message Bus Socket.
Starting Socket activation for snappy daemon...
[ OK ] Listening on Socket activation for snappy daemon.
[ OK ] Reached target Socket Units.
[ OK ] Reached target Basic System.
[ OK ] Started Regular background program processing daemon.
[ OK ] Started D-Bus System Message Bus.
[ OK ] Started Save initial kernel messages after boot.
Starting Remove Stale Online ext4 Metadata Check Snapshots...
Starting Dispatcher daemon for systemd-networkd...
Starting System Logging Service...
[ OK ] Started Service for snap application maas.supervisor.
[ OK ] Reached target Preparation for Logins.
Starting Snap Daemon...
Starting OpenBSD Secure Shell server...
Starting User Login Management...
Starting Permit User Sessions...
[ OK ] Finished Remove Stale Online ext4 Metadata Check Snapshots.
[ OK ] Finished Permit User Sessions.
[ OK ] Started Console Getty.
[ OK ] Created slice Slice /system/getty.
[ OK ] Reached target Login Prompts.
[ OK ] Started System Logging Service.
[ OK ] Started OpenBSD Secure Shell server.
[ OK ] Started User Login Management.
[ OK ] Started Unattended Upgrades Shutdown.
Starting Hostname Service...
[ OK ] Started Hostname Service.
Starting Authorization Manager...
[ OK ] Started Dispatcher daemon for systemd-networkd.
[ OK ] Started Authorization Manager.
Ubuntu 22.04.2 LTS maas-ha-2 console
The difference between these 3 containers that are freezeing and others is lack of: