File goes missing during image publish (/usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so)

Incus
1

This issue is really weird, but it recreates easily on my own system (ubuntu 22.04.4 on host, running incus 6.0 LTS)

$ incus create images:ubuntu/24.04/cloud foo1    # the exact image is 45b8d997e210
Creating foo1
$ incus publish foo1 --alias foo1 --reuse --public
Instance published with fingerprint: 025b078661e05e381ca802429eb65488ca578eedf5471a43e275e7befbd27d95
$ incus create foo1 foo2
Creating foo2
$ incus start foo1 foo2
$ incus exec foo1 -- dpkg -V dhcpcd-base
$ incus exec foo2 -- dpkg -V dhcpcd-base
missing     /usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so
$

The image “foo1” is created directly from instance “foo1” (which hasn’t even been started). Then instance “foo2” is created from this image. But somewhere along that process, the file /usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so goes missing.

Are there any other differences? No - except they create different journal files at startup.

# find /var/lib/incus/containers/nsrc-builder_foo1/ -type f | sed 's/foo1/fooX/g' | sort >a
# find /var/lib/incus/containers/nsrc-builder_foo2/ -type f | sed 's/foo2/fooX/g' | sort >b
# diff -u a b
--- a	2024-05-17 10:21:59.207955347 +0000
+++ b	2024-05-17 10:22:02.963982709 +0000
@@ -5884,7 +5884,6 @@
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/cryptsetup/libcryptsetup-token-systemd-fido2.so
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/cryptsetup/libcryptsetup-token-systemd-pkcs11.so
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/cryptsetup/libcryptsetup-token-systemd-tpm2.so
-/var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/e2fsprogs/e2scrub_all_cron
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/e2fsprogs/e2scrub_fail
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/usr/lib/x86_64-linux-gnu/engines-3/afalg.so
@@ -20497,7 +20496,7 @@
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/dmesg
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/dpkg.log
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/faillog
-/var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/journal/7665a148b1a24b708af1eb21d8dc20b1/system.journal
+/var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/journal/a728031f49f64d0d8c1aa845301ba362/system.journal
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/lastlog
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/syslog
 /var/lib/incus/containers/nsrc-builder_fooX/rootfs/var/log/unattended-upgrades/unattended-upgrades-shutdown.log

What about the image itself?

$ incus image list foo1
+-------+--------------+--------+---------------------------------------------+--------------+-----------+-----------+----------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 | ARCHITECTURE |   TYPE    |   SIZE    |     UPLOAD DATE      |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+-----------+----------------------+
| foo1  | 025b078661e0 | yes    | Ubuntu noble amd64 (cloud) (20240516_07:42) | x86_64       | CONTAINER | 194.09MiB | 2024/05/17 10:15 UTC |
+-------+--------------+--------+---------------------------------------------+--------------+-----------+-----------+----------------------+

# tar -tvzf /var/lib/incus/images/025b078661e05e381ca802429eb65488ca578eedf5471a43e275e7befbd27d95 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd
drwxr-xr-x root/root         0 2024-05-16 07:43 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd
drwxr-xr-x root/root         0 2024-05-16 07:43 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd/dev
-rw-r--r-- root/root     14488 2024-03-31 08:48 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so

So the file is in the image. Is something deleting when it starts up, or beforehand? Let’s create another instance and not start it.

$ incus create foo1 foo3
$ incus file pull foo3/usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so - | md5sum
Error: Not Found
d41d8cd98f00b204e9800998ecf8427e  -
$ md5sum </dev/null
d41d8cd98f00b204e9800998ecf8427e  -
$ incus file pull foo3/usr/lib/x86_64-linux-gnu/security/pam_unix.so - | md5sum
2ecd769e3a77af2afa1ce56428a5ab82  -

Therefore, it’s going missing before the container is even launched. How bizarre??

Aside: the symptom which brought this to my attention is LP#2039342.

2

Hah! I suddenly realised that there’s /dev/ in the middle of that path.

# tar -tvzf /var/lib/incus/images/025b078661e05e381ca802429eb65488ca578eedf5471a43e275e7befbd27d95 | grep /dev/
lrwxrwxrwx root/root         0 2024-05-16 07:44 rootfs/etc/systemd/system/ua-auto-attach.service -> /dev/null
lrwxrwxrwx root/root         0 2024-04-19 14:24 rootfs/etc/systemd/system-generators/systemd-gpt-auto-generator -> /dev/null
lrwxrwxrwx root/root                 0 2024-04-19 14:24 rootfs/usr/lib/systemd/system/cryptdisks-early.service -> /dev/null
lrwxrwxrwx root/root                 0 2024-04-19 14:24 rootfs/usr/lib/systemd/system/cryptdisks.service -> /dev/null
lrwxrwxrwx root/root                 0 2024-04-19 14:24 rootfs/usr/lib/systemd/system/hwclock.service -> /dev/null
lrwxrwxrwx root/root                 0 2024-04-08 14:50 rootfs/usr/lib/systemd/system/sudo.service -> /dev/null
lrwxrwxrwx root/root                 0 2024-04-19 14:24 rootfs/usr/lib/systemd/system/x11-common.service -> /dev/null
-rw-r--r-- root/root             14488 2024-03-31 08:48 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so

Pound to a penny says there’s some code attempting to protect /dev/.

I don’t know why this applies to my locally published image, but not to the one launched from the images: server. They are different formats though. Mine is a unified tarball:

# ls -l /var/lib/incus/images/025b078661e05e381ca802429eb65488ca578eedf5471a43e275e7befbd27d95*
-rw------- 1 root root 203519216 May 17 10:15 /var/lib/incus/images/025b078661e05e381ca802429eb65488ca578eedf5471a43e275e7befbd27d95

whereas the images: one is split:

# ls -l /var/lib/incus/images/45b8d997e210c6a3d20bd9fcd5158494f68c2f5217148a32363371497338fa22*
-rw-r--r-- 1 root root      1180 May 16 15:57 /var/lib/incus/images/45b8d997e210c6a3d20bd9fcd5158494f68c2f5217148a32363371497338fa22
-rw-r--r-- 1 root root 155787264 May 16 15:57 /var/lib/incus/images/45b8d997e210c6a3d20bd9fcd5158494f68c2f5217148a32363371497338fa22.rootfs
3

I thought I found it, in shared/archive/archive.go

        if strings.HasPrefix(extension, ".tar") {
                command = "tar"
                // We can't create char/block devices in unpriv containers so avoid extracting them.
                args = append(args, "--anchored")
                args = append(args, "--wildcards")
                args = append(args, "--exclude=dev/*")
                args = append(args, "--exclude=/dev/*")
                args = append(args, "--exclude=./dev/*")
                args = append(args, "--exclude=rootfs/dev/*")
                args = append(args, "--exclude=/rootfs/dev/*")
                args = append(args, "--exclude=./rootfs/dev/*")

Except with --anchored it seems to be fine:

# tar --anchored --wildcards "--exclude=dev/*" -tvzf /var/lib/incus/images/025b078661e05e381ca802429eb65488ca578eedf5471a43e275e7befbd27d95 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd
drwxr-xr-x root/root         0 2024-05-16 07:43 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd
drwxr-xr-x root/root         0 2024-05-16 07:43 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd/dev
-rw-r--r-- root/root     14488 2024-03-31 08:48 rootfs/usr/lib/x86_64-linux-gnu/dhcpcd/dev/udev.so

But when I check what incus is doing with ps, it’s missing that flag:

# ps auxwww | grep "tar "
root      463777 12.0  0.0   7216  3064 ?        S    11:01   0:00 tar --wildcards --exclude=dev/* --exclude=./dev/* --exclude=rootfs/dev/* --exclude=rootfs/./dev/* --restrict --force-local -C /var/lib/incus/storage-pools/default/containers/nsrc-builder_foo3 --numeric-owner --xattrs-include=* -zxf -
root      463778  4.0  0.0   7060   164 ?        R    11:01   0:00 tar --wildcards --exclude=dev/* --exclude=./dev/* --exclude=rootfs/dev/* --exclude=rootfs/./dev/* --restrict --force-local -C /var/lib/incus/storage-pools/default/containers/nsrc-builder_foo3 --numeric-owner --xattrs-include=* -zxf -
4

Ah, I see this was already raised in During tar image unpacking, all dev folders excluded not just /dev and /rootfs/dev · Issue #815 · lxc/incus · GitHub and was fixed about 2 weeks ago. So I just need to wait for a new incus 6.0 LTS patch :slight_smile: