2nd System upgraded from Ubuntu 20.04 w/working LXD to Ubuntu 22.04 - LXD again not working

A couple weeks ago I posted about my attempt to upgrade a laptop running Ubuntu 20.04 with a working LXD to Ubuntu 22.04 and LXD.

Painfully, I ended doing a complete reinstall of that system in order to get LXD to where Launching a 22.04 container would actually start the container up.

Now I have a much larger Ubuntu 20.04 Desktop that I upgraded to Ubuntu 22.04 and again LXD is not working correctly.

Below you can see:

  • launching a LXC “xenial” Container(CN1) does start the container but it gets NO IP4 address.
  • launching a LXC “jammy” Container (CN2) … the Container gets created but does not start!
  • launching a LXC “alpine” Container (CN3) creates and starts the container correctly?

$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:
Do you want to configure a new storage pool? (yes/no) [default=yes]:
Name of the new storage pool [default=default]:
Name of the storage backend to use (dir, lvm, zfs, ceph, btrfs) [default=zfs]: btrfs
Create a new BTRFS pool? (yes/no) [default=yes]:
Would you like to use an existing empty block device (e.g. a disk or partition)? (yes/no) [default=no]: yes
Path to the existing block device: /dev/sdc1
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to create a new local network bridge? (yes/no) [default=yes]:
What should the new bridge be called? [default=lxdbr0]:
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]:
Would you like the LXD server to be available over the network? (yes/no) [default=no]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML “lxd init” preseed to be printed? (yes/no) [default=no]:

$ lxc ls
±-----±------±-----±-----±-----±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-----±------±-----±-----±-----±----------+

$ lxc launch ubuntu:xenial cn1
Creating cn1
Starting cn1

$ lxc ls
±-----±--------±-----±----------------------------------------------±----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-----±--------±-----±----------------------------------------------±----------±----------+
| cn1 | RUNNING | | fd42:30d8:8bed:c58f:216:3eff:fe6f:d413 (eth0) | CONTAINER | 0 |
±-----±--------±-----±----------------------------------------------±----------±----------+

$ lxc launch ubuntu:jammy cn2
Creating cn2
Starting cn2

$ lxc launch images:alpine/edge cn3
Creating cn3
Starting cn3

$ lxc ls
±-----±--------±----------------------±----------------------------------------------±----------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±-----±--------±----------------------±----------------------------------------------±----------±----------+
| cn1 | RUNNING | | fd42:30d8:8bed:c58f:216:3eff:fe6f:d413 (eth0) | CONTAINER | 0 |
±-----±--------±----------------------±----------------------------------------------±----------±----------+
| cn2 | STOPPED | | | CONTAINER | 0 |
±-----±--------±----------------------±----------------------------------------------±----------±----------+
| cn3 | RUNNING | 10.245.137.108 (eth0) | fd42:30d8:8bed:c58f:216:3eff:fe20:8838 (eth0) | CONTAINER | 0 |
±-----±--------±----------------------±----------------------------------------------±----------±----------+


So any suggestions as to what I can check to fix this somehow. I really do not want to have to reinstall this large Desktop system and restore from backup as it would take all night?

Any info needed?

  • Ubuntu 22.04 LTS
  • LXD v5.0.0
  • 64GB ram
  • 3 - 4 TB disks
  • Linux 5.15.0-27-generic #28-Ubuntu SMP

Default Profile:

config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
network: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
name: default
used_by:

  • /1.0/instances/cn1
  • /1.0/instances/cn2
  • /1.0/instances/cn3

Thanks for any ideas as I’m out of them myself

Brian

Its also really odd to me that:

  • A “clean install” of 22.04 onto a server then installing LXD works fine.
  • But “upgrading (do-release-upgrade -d)” an existing 20.04 system to 22.04 is causing these problems even though the upgrade itself succeeds?

@tomp
@stgraber
Okay…

I think found the problem and it looks like the Ubuntu Xenial & Jammy Containers can’t start due to a problem with mounting CGROUP at /sys/fs/cgroup/systemd

But the Alpine Container starts just fine !

-----------------------------------------------------------------------------------------------------------------

$ lxc console --show-log cn1

Console log:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!] Failed to mount API filesystems, freezing.
Freezing execution.

-----------------------------------------------------------------------------------------------------------------

$ lxc console --show-log cn2

Console log:

Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!] Failed to mount API filesystems.
Exiting PID 1…

-----------------------------------------------------------------------------------------------------------------

$ lxc console --show-log cn3

Console log:

OpenRC 0.44.10 is starting up Linux 5.15.0-27-generic (x86_64) [LXC]

  • /proc is already mounted
  • Mounting /run … * /run/openrc: creating directory
  • /run/lock: creating directory
  • /run/lock: correcting owner
  • Caching service dependencies … [ ok ]
  • Mounting local filesystems … [ ok ]
  • Creating user login records … [ ok ]
  • Cleaning /tmp directory … [ ok ]
  • Starting busybox syslog … [ ok ]
  • Starting busybox crond … [ ok ]
  • Starting networking … * lo … [ ok ]
  • eth0 …udhcpc: started, v1.35.0
    udhcpc: broadcasting discover
    udhcpc: broadcasting select for 10.245.137.108, server 10.245.137.1
    udhcpc: lease of 10.245.137.108 obtained from 10.245.137.1, lease time 3600
    [ ok ]

Welcome to Alpine Linux 3.12
Kernel 5.15.0-27-generic on an x86_64 (/dev/console)

@tomp @stgraber

I saw others have had this same CGROUP2 problem previously…
example: Containers Fail To Start Ubuntu 21.10 - #9 by RossMadness

Stephane’s proposed workaround is:
Not too sure what’s going on in your case, we usually don’t see quite that much breakage because of cgroup2.

In any case, booting your host system with systemd.unified_cgroup_hierarchy=false passed to the kernel should take care of this.

Brian

@tomp @stgraber
Could this be the bug/core problem with CGROUP2

https://www.mail-archive.com/ubuntu-bugs@lists.ubuntu.com/msg6024379.html

Either on the Host side or with the Ubuntu “images” ??

FYI… my “Host” is running both cgroup1 and cgroup2:

$ grep cgroup /proc/filesystems
nodev cgroup
nodev cgroup2

@tomp @stgraber

Okay… per all the above clues I finally fixed this on my Ubuntu 22.04 system (upgraded from 20.04).

It was a CGROUP2 bug/problem so I disabled “cgroup2” use on the HOST and now things work and all the containers startup Ok.

Work-around Solution was:

add the following string to the GRUB_CMDLINE_LINUX line in /etc/default/grub and then run sudo update-grub.

systemd.unified_cgroup_hierarchy=0

It still begs the question why clean Installs of Ubuntu 22.04 LTS work correctly but “Upgrades” from 20.04LTS to 22.04LTS break something with CGROUP2 ?

1 Like

Glad you got it working. Yes running older distros on a Cgroupv2 only system doesn’t work, see Error: The image used by this instance requires a CGroupV1 host system when using clustering - #2 by tomp

Tom
These weren’t “older” versions.

It was 22.04 and 20.04

Strange, seems like the cgroup setup isn’t correct on upgrade by systemd. Might be worth opening an issue with Ubuntu about this.

Yeah that’s what bothered me

A clean install of 22.04 and everything works including LXD

Doing an “upgrade” from 20.04 to 22.04 and LXD gets broken until I remove CGROUP 2

Yep sounds like an issue with how systemd is setting up cgroups in the upgrade from Focal to Jammy.

Filed a bug:

[Bug 1971571] [NEW] ubuntu 22.04 cgroup2 works for clean install but upgrade to 22.04 causes cgroup2 problems

1 Like

Hi All, I’ve run afoul of this issue on my laptop and I went to add GRUB_CMDLINE_LINUX=“systemd.unified_cgroup_hierarchy=0” to my /etc/default/grub However, the file didn’t exist and even creating it and adding that line with a following update-grub hasn’t worked. Is there a way to verify this setting after it it set?

Much Appreciated.

What OS are you running?

Ubuntu 22.04

Same as me

That file exists by default on fresh installations, and is the correct place to put it.
Did you upgrade or do a fresh install of Ubuntu?

Upgrade from 20.04

Sounds like a bug report to Ubuntu upstream could be in order as there seems to be some issues around upgrading from previous versions. Is there a /etc/default/grub.bak file?