jsnjack
(Yauhen Shulitski)
April 22, 2022, 3:04pm
1
Hi,
Today after 2pm none of our lxd containers are getting ipv4 addresses. I tried to create also a generic centos / rocky / fedora / ubuntu container and none of them getting ipv4 addresses either:
lxc launch images:centos/8-Stream
However if I use old image, everything works fine:
lxc launch podman_base zz
lxc list
+-------------------+---------+----------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------------+---------+----------------------+------+-----------+-----------+
| outgoing-duckling | RUNNING | | | CONTAINER | 0 |
+-------------------+---------+----------------------+------+-----------+-----------+
| zz | RUNNING | 192.168.2.106 (eth0) | | CONTAINER | 0 |
+-------------------+---------+----------------------+------+-----------+-----------+
The issue is happening in lxd versions 4.0.9 - 5.0.0 and on fedora 35, centos 8 stream hosts
tomp
(Thomas Parrott)
April 22, 2022, 3:06pm
2
Hi,
Sorry about this, we are aware, please see
opened 02:31PM - 22 Apr 22 UTC
I had little luck figuring out where to specifically to report this so I am tryi… ng here. I realize this is not an LXD issue per se and probably doesn't belong here. So please let me know if there is a better place for this report.
Nb: I am speaking of the /default images here, not the /cloud images.
Anyway, a week or so ago our deployments of Debian started failing. It turned out that networking was no longer being managed with ifupdown in Debian images but SystemdNetworkd. Despite not finding anything refering to this online I assumed this was just an upstream change in Debian.
However, just today suddenly all the OS's we support (besides Ubuntu which still works as it always has) namely Debian 11, Almalinux 8 and Centos 8 all failed as we can no longer configure networking in these containers.
We detect whether a container uses "centos style" networking, ifupdown, SystemdNetworkd or netplan and take it from there.
In none of the recently updated images for these OS'es can I find any of these network managers present. I am not well versed enough to figure out what, if anything, is managing networking in these images - but it strikes me as extremely odd that these 3 OS's all start failing at the same time and all have their "usual" network configuration schemes disabled or removed or whatever it is that has happened.
Any input here would be greatly appreciated. I can provide access to a test system with these failing containers if need be.
jsnjack
(Yauhen Shulitski)
April 22, 2022, 3:14pm
3
Thanks for the explanation! Is there a way to use the image from the day before? It looks like all our base images were automatically updated!
tomp
(Thomas Parrott)
April 22, 2022, 3:14pm
4
@stgraber @monstermunchkin are older images stored on the image server do you know?
No, unfortunately we don’t keep older images.
jsnjack
(Yauhen Shulitski)
April 22, 2022, 4:19pm
6
I found a way to access older images:
Navigate to https://uk.lxd.images.canonical.com/streams/v1/images.json
find you image, expand versions
key (the last 3 days are available), select the date
expand items -> lxd.tar.xz
, copy combined_sha256
value
you can start a new container from this image like this lxc launch images:f32c4d30aad15b2ea5a97edb2691d31d7fcaa567d8fc700f0cfef3eb4f714e00 ww
I wonder if there is a way to override existing image in cache…
1 Like
stgraber
(Stéphane Graber)
April 22, 2022, 5:46pm
7
We keep 3 days of images, so it is possible to launch from an older image.
sdeziel
(Simon Deziel)
April 22, 2022, 9:04pm
8
Out of curiosity, is there an easy way to refer to an old image? Or is New containers not getting ipv4 address (broken base images) - #6 by jsnjack the only way?
stgraber
(Stéphane Graber)
April 22, 2022, 9:33pm
9
You need to refer to it by fingerprint
sdeziel
(Simon Deziel)
April 22, 2022, 9:39pm
10
I guess the question is where can we easily get the fingerprint?
Using the sha256 fingerprint of disk.qcow2 from https://us.lxd.images.canonical.com/images/ubuntu/bionic/amd64/default/20220420_07:42/SHA256SUMS doesn’t work:
lxc launch images:b662acb0a517a96c08a3c3c7aac7f4e365f31e8201d298191cb639531e816572 --vm
Creating the instance
Error: Failed instance creation: Failed getting image: The requested image couldn't be found
stgraber
(Stéphane Graber)
April 22, 2022, 9:40pm
11
lxc image list images: os=Ubuntu arch=amd64 release=focal variant=default
2 Likes
sdeziel
(Simon Deziel)
April 22, 2022, 9:50pm
12
lxc image list images: os=Ubuntu arch=amd64 release=bionic variant=default type=disk-kvm.img
worked thanks.
Not easily discovered though. I suspect many would point a browser to the images server and look for a fingerprint. Would it make sense to add a list of fingerprints to the artifacts uploaded along with other files?
stgraber
(Stéphane Graber)
April 22, 2022, 9:54pm
13
It’s technically there, just not super visible
stgraber@dakara:~$ curl https://us.lxd.images.canonical.com/images/ubuntu/bionic/amd64/default/20220420_07:42/.lxd_fingerprint
762b6c4e19fbba970c2c22710ba6d410de178b5ac8b26ebfd2bd853b57622261
sdeziel
(Simon Deziel)
April 23, 2022, 12:20am
14
Yeah, not super visible and I couldn’t guess the one for the VM. It’s also confusing that it doesn’t match what’s in the SHA256SUMS file. The lxc image list
with filters you provided is super handy though. If we have a list of tips and tricks, it should be in there.