Can't start VMs that worked days ago

mar · December 10, 2020, 5:27pm

Hi - I was able to start VMs several days ago but now VMs are having difficulty reading the ceph config file. What changed recently? The permissions were fine a couple days ago. Attempting to chmod to 755 or 777 as a test had no effect.

root@aa1-cptef101-n1:/# lxc --project <company>-staging2 start pod-890-1000-00
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited -- /snap/lxd/18546/bin/qemu-system-x86_64 -S -name pod-890-1000-00 -uuid <uuid> -daemonize -cpu host -nographic -serial chardev:console -nodefaults -no-reboot -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=deny,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/<company>-staging2_pod-890-1000-00/qemu.conf -pidfile /var/snap/lxd/common/lxd/logs/<company>-staging2_pod-890-1000-00/qemu.pid -D /var/snap/lxd/common/lxd/logs/<company>-staging2_pod-890-1000-00/qemu.log -chroot /var/snap/lxd/common/lxd/virtual-machines/<company>-staging2_pod-890-1000-00 -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: char device redirected to /dev/pts/1 (label console)
: Process exited with a non-zero value
Try `lxc info --show-log pod-890-1000-00` for more info
root@aa1-cptef101-n1:/# lxc --project<company>-staging2 info --show-log pod-890-1000-00
Name: pod-890-1000-00
Location: aa1-cptef101-n1
Remote: unix://
Architecture: x86_64
Created: 2020/06/18 23:54 UTC
Status: Stopped
Type: virtual-machine
Profiles: stagingtwo-virtual-vault-890
Snapshots:
  snap0 (taken at 2020/06/19 01:15 UTC) (stateless)
  pre-common-yml (taken at 2020/06/30 18:42 UTC) (stateless)
  phase3 (taken at 2020/06/30 23:34 UTC) (stateless)

Log:

2020-12-10 17:17:41.314100 7fdf8f96e3c0 -1 Errors while parsing config file!
2020-12-10 17:17:41.314105 7fdf8f96e3c0 -1 parse_file: cannot open /etc/ceph/ceph.conf: (13) Permission denied
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/<company>-staging2_pod-890-1000-00/qemu.conf:301: error reading conf file /etc/ceph/ceph.conf: Permission denied

root@aa1-cptef101-n1:/# ls -l /etc/ceph/ceph.conf
-rwxr-xr-x 1 root root 275 Oct  2 08:22 /etc/ceph/ceph.conf

stgraber · December 10, 2020, 5:33pm

Can you show the output of dmesg?

tomp · December 10, 2020, 5:42pm

Can you show output of lxc config show <instance> --expanded and also the any output about AppArmor from sudo dmesg please.

mar · December 10, 2020, 5:43pm

[4633369.211486] libceph: osd23 (1)[<IP>]:6805 socket closed (con state OPEN)
[4633531.681087] audit: type=1400 audit(1607620258.280:289338): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633531.681097] audit: type=1400 audit(1607620258.280:289339): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633531.681101] audit: type=1400 audit(1607620258.280:289340): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633531.681172] audit: type=1400 audit(1607620258.280:289341): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633531.681432] audit: type=1400 audit(1607620258.280:289342): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633545.664888] audit: type=1400 audit(1607620272.264:289343): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633545.664892] audit: type=1400 audit(1607620272.264:289344): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633545.664896] audit: type=1400 audit(1607620272.264:289345): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633545.664952] audit: type=1400 audit(1607620272.264:289346): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633545.665232] audit: type=1400 audit(1607620272.264:289347): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633565.268531] audit: type=1400 audit(1607620291.868:289348): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1569872/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270211] audit: type=1400 audit(1607620291.868:289349): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270268] audit: type=1400 audit(1607620291.868:289350): apparmor="ALLOWED" operation="file_mmap" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/usr/lib/x86_64-linux-gnu/nss/libsoftokn3.so" pid=1507213 comm="sssd_be" requested_mask="rm" denied_mask="rm" fsuid=0 ouid=0
[4633565.270730] audit: type=1400 audit(1607620291.868:289351): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/usr/lib/x86_64-linux-gnu/nss/libfreeblpriv3.so" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270753] audit: type=1400 audit(1607620291.868:289352): apparmor="ALLOWED" operation="file_mmap" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/usr/lib/x86_64-linux-gnu/nss/libfreeblpriv3.so" pid=1507213 comm="sssd_be" requested_mask="rm" denied_mask="rm" fsuid=0 ouid=0
[4633565.270925] audit: type=1400 audit(1607620291.868:289353): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270953] audit: type=1400 audit(1607620291.868:289354): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270966] audit: type=1400 audit(1607620291.868:289355): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270979] audit: type=1400 audit(1607620291.868:289356): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633565.270992] audit: type=1400 audit(1607620291.868:289357): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633572.541405] kauditd_printk_skb: 9 callbacks suppressed
[4633572.541408] audit: type=1400 audit(1607620299.140:289367): apparmor="DENIED" operation="mkdir" profile="/snap/snapd/10492/usr/lib/snapd/snap-confine" name="/home/madeline/" pid=1570013 comm="snap-confine" requested_mask="c" denied_mask="c" fsuid=1163 ouid=1163
[4633576.141007] audit: type=1400 audit(1607620302.740:289368): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1570033/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633579.636521] audit: type=1400 audit(1607620306.236:289369): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/proc/1570033/cmdline" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633579.640247] audit: type=1400 audit(1607620306.240:289370): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" pid=1507213 comm="sssd_be" capability=2  capname="dac_read_search"
[4633618.316914] libceph: osd3 (1)[<IP>]:6830 socket closed (con state OPEN)
[4633690.452650] br4: port 12(tapfac9c486) entered blocking state
[4633690.452652] br4: port 12(tapfac9c486) entered disabled state
[4633690.452818] device tapfac9c486 entered promiscuous mode
[4633690.533770] audit: type=1400 audit(1607620417.132:289371): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="lxd-backblaze-staging2_pod-890-1000-00_</var/snap/lxd/common/lxd>" pid=1570718 comm="apparmor_parser"
[4633690.660942] audit: type=1400 audit(1607620417.260:289372): apparmor="DENIED" operation="open" profile="lxd-backblaze-staging2_pod-890-1000-00_</var/snap/lxd/common/lxd>" name="/var/lib/snapd/hostfs/etc/ceph/ceph.conf" pid=1570736 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633690.790272] device tapfac9c486 left promiscuous mode
[4633690.790472] br4: port 12(tapfac9c486) entered disabled state
[4633691.439275] audit: type=1400 audit(1607620418.040:289373): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633691.439283] audit: type=1400 audit(1607620418.040:289374): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633691.439288] audit: type=1400 audit(1607620418.040:289375): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633691.439330] audit: type=1400 audit(1607620418.040:289376): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633691.439653] audit: type=1400 audit(1607620418.040:289377): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633713.679925] audit: type=1400 audit(1607620440.280:289378): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633713.679931] audit: type=1400 audit(1607620440.280:289381): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633713.679935] audit: type=1400 audit(1607620440.280:289379): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633713.679938] audit: type=1400 audit(1607620440.280:289380): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633713.680269] audit: type=1400 audit(1607620440.280:289382): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633723.822598] audit: type=1400 audit(1607620450.420:289383): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633723.822603] audit: type=1400 audit(1607620450.424:289384): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633723.822607] audit: type=1400 audit(1607620450.424:289385): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633723.822661] audit: type=1400 audit(1607620450.424:289386): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633723.822998] audit: type=1400 audit(1607620450.424:289387): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633742.373008] audit: type=1400 audit(1607620468.972:289388): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" pid=1371424 comm="sssd_pam" capability=2  capname="dac_read_search"
[4633742.373033] audit: type=1400 audit(1607620468.972:289389): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/proc/1569872/cmdline" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633801.448589] libceph: osd23 (1)[<IP>]:6805 socket closed (con state OPEN)
[4633807.343838] libceph: osd19 (1)[<IP>]:6812 socket closed (con state OPEN)
[4633885.351681] audit: type=1400 audit(1607620611.953:289390): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633885.351687] audit: type=1400 audit(1607620611.953:289391): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633885.351787] audit: type=1400 audit(1607620611.953:289393): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633885.351792] audit: type=1400 audit(1607620611.953:289392): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633885.352015] audit: type=1400 audit(1607620611.953:289394): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633889.050981] audit: type=1400 audit(1607620615.653:289395): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633889.050988] audit: type=1400 audit(1607620615.653:289396): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633889.050992] audit: type=1400 audit(1607620615.653:289398): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633889.050995] audit: type=1400 audit(1607620615.653:289397): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633889.051324] audit: type=1400 audit(1607620615.653:289399): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633895.255784] audit: type=1400 audit(1607620621.857:289400): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1570865/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633895.257719] audit: type=1400 audit(1607620621.857:289401): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1570866/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633904.827585] audit: type=1400 audit(1607620631.429:289402): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1570874/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633904.995510] audit: type=1400 audit(1607620631.597:289403): apparmor="ALLOWED" operation="capable" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" pid=1371424 comm="sssd_pam" capability=2  capname="dac_read_search"
[4633904.995551] audit: type=1400 audit(1607620631.597:289404): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/proc/1570874/cmdline" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633905.042659] audit: type=1400 audit(1607620631.645:289405): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/proc/1570900/cmdline" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633907.140069] audit: type=1400 audit(1607620633.741:289406): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1570984/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633911.802316] audit: type=1400 audit(1607620638.405:289407): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/proc/1570984/cmdline" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633934.629355] br4: port 12(tap96a47682) entered blocking state
[4633934.629357] br4: port 12(tap96a47682) entered disabled state
[4633934.629561] device tap96a47682 entered promiscuous mode
[4633934.638904] audit: type=1400 audit(1607620661.241:289408): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxd-backblaze-staging2_pod-890-1000-00_</var/snap/lxd/common/lxd>" pid=1571094 comm="apparmor_parser"
[4633934.709182] audit: type=1400 audit(1607620661.309:289409): apparmor="DENIED" operation="open" profile="lxd-backblaze-staging2_pod-890-1000-00_</var/snap/lxd/common/lxd>" name="/var/lib/snapd/hostfs/etc/ceph/ceph.conf" pid=1571107 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633934.838078] device tap96a47682 left promiscuous mode
[4633934.838283] br4: port 12(tap96a47682) entered disabled state
[4633935.615195] audit: type=1400 audit(1607620662.217:289410): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633935.615242] audit: type=1400 audit(1607620662.217:289411): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633935.615247] audit: type=1400 audit(1607620662.217:289413): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4633935.615250] audit: type=1400 audit(1607620662.217:289412): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4633935.615588] audit: type=1400 audit(1607620662.217:289414): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634073.698232] audit: type=1400 audit(1607620800.302:289415): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634073.698237] audit: type=1400 audit(1607620800.302:289416): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634073.698241] audit: type=1400 audit(1607620800.302:289417): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634073.698244] audit: type=1400 audit(1607620800.302:289418): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634073.698607] audit: type=1400 audit(1607620800.302:289419): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634075.129791] audit: type=1400 audit(1607620801.734:289420): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634075.129799] audit: type=1400 audit(1607620801.734:289421): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634075.129803] audit: type=1400 audit(1607620801.734:289422): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634075.129807] audit: type=1400 audit(1607620801.734:289423): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634075.130125] audit: type=1400 audit(1607620801.734:289424): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634094.292276] libceph: osd23 (1)[2605:72c0:a600:ce:aa1:d:0:f104]:6805 socket closed (con state OPEN)
[4634107.005541] audit: type=1400 audit(1607620833.606:289425): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/proc/1572749/cmdline" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634107.012266] audit: type=1400 audit(1607620833.610:289426): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/proc/1572749/cmdline" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634232.276216] audit: type=1400 audit(1607620958.875:289427): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634232.276224] audit: type=1400 audit(1607620958.875:289428): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634232.276229] audit: type=1400 audit(1607620958.875:289429): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634232.276271] audit: type=1400 audit(1607620958.875:289430): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634232.276633] audit: type=1400 audit(1607620958.879:289431): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634254.203313] audit: type=1400 audit(1607620980.803:289432): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634254.203359] audit: type=1400 audit(1607620980.803:289433): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634254.203405] audit: type=1400 audit(1607620980.803:289434): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634254.203743] audit: type=1400 audit(1607620980.803:289435): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634254.203770] audit: type=1400 audit(1607620980.803:289436): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634279.403821] libceph: osd22 (1)[<IP>]:6803 socket closed (con state OPEN)
[4634365.730589] br4: port 12(tap60e915fe) entered blocking state
[4634365.730592] br4: port 12(tap60e915fe) entered disabled state
[4634365.730821] device tap60e915fe entered promiscuous mode
[4634365.740107] audit: type=1400 audit(1607621092.343:289437): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="lxd-backblaze-staging2_pod-890-1000-00_</var/snap/lxd/common/lxd>" pid=1572896 comm="apparmor_parser"
[4634365.813715] audit: type=1400 audit(1607621092.415:289438): apparmor="DENIED" operation="open" profile="lxd-backblaze-staging2_pod-890-1000-00_</var/snap/lxd/common/lxd>" name="/var/lib/snapd/hostfs/etc/ceph/ceph.conf" pid=1572911 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634365.945493] device tap60e915fe left promiscuous mode
[4634365.945701] br4: port 12(tap60e915fe) entered disabled state
[4634366.717323] audit: type=1400 audit(1607621093.319:289439): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634366.717328] audit: type=1400 audit(1607621093.319:289440): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634366.717364] audit: type=1400 audit(1607621093.319:289441): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634366.717370] audit: type=1400 audit(1607621093.319:289442): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634366.717687] audit: type=1400 audit(1607621093.319:289443): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634417.646804] audit: type=1400 audit(1607621144.247:289445): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634417.646809] audit: type=1400 audit(1607621144.247:289444): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634417.646813] audit: type=1400 audit(1607621144.247:289446): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634417.646964] audit: type=1400 audit(1607621144.247:289447): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634417.647139] audit: type=1400 audit(1607621144.247:289448): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634454.036661] audit: type=1400 audit(1607621180.640:289449): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634454.036666] audit: type=1400 audit(1607621180.640:289450): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634454.036669] audit: type=1400 audit(1607621180.640:289451): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634454.036706] audit: type=1400 audit(1607621180.640:289452): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634454.037016] audit: type=1400 audit(1607621180.640:289453): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634499.825394] libceph: osd22 (1)[<IP>]:6803 socket closed (con state OPEN)
[4634500.331089] libceph: osd16 (1)[<IP>]:6801 socket closed (con state OPEN)
[4634500.354580] libceph: osd4 (1)[<IP>]:6850 socket closed (con state OPEN)
[4634521.426781] libceph: osd22 (1)[<IP>]]:6803 socket closed (con state OPEN)
[4634584.060481] audit: type=1400 audit(1607621310.664:289454): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634584.060488] audit: type=1400 audit(1607621310.664:289455): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634584.060492] audit: type=1400 audit(1607621310.664:289456): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634584.060497] audit: type=1400 audit(1607621310.664:289457): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634584.060819] audit: type=1400 audit(1607621310.664:289458): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634625.269337] audit: type=1400 audit(1607621351.872:289459): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634625.269353] audit: type=1400 audit(1607621351.872:289461): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634625.269354] audit: type=1400 audit(1607621351.872:289460): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634625.269444] audit: type=1400 audit(1607621351.872:289462): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634625.269662] audit: type=1400 audit(1607621351.872:289463): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634660.426130] libceph: osd23 (1)[<IP>]:6805 socket closed (con state OPEN)
[4634679.110733] libceph: osd23 (1)[<IP>]:6805 socket closed (con state OPEN)
[4634776.095837] audit: type=1400 audit(1607621502.701:289464): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634776.095845] audit: type=1400 audit(1607621502.701:289465): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634776.095850] audit: type=1400 audit(1607621502.701:289466): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634776.095854] audit: type=1400 audit(1607621502.701:289467): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634776.096209] audit: type=1400 audit(1607621502.701:289468): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634796.770882] libceph: osd23 (1)[<IP>]:6805 socket closed (con state OPEN)
[4634819.335789] libceph: osd18 (1)[<IP>]:6858 socket closed (con state OPEN)
[4634819.624094] audit: type=1400 audit(1607621546.229:289469): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634819.624102] audit: type=1400 audit(1607621546.229:289470): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/dev/urandom" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[4634819.624162] audit: type=1400 audit(1607621546.229:289471): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_nss" name="/run/systemd/resolve/stub-resolv.conf" pid=1371423 comm="sssd_nss" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634819.624168] audit: type=1400 audit(1607621546.229:289472): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_pam" name="/run/systemd/resolve/stub-resolv.conf" pid=1371424 comm="sssd_pam" requested_mask="r" denied_mask="r" fsuid=0 ouid=101
[4634819.624415] audit: type=1400 audit(1607621546.229:289473): apparmor="ALLOWED" operation="open" profile="/usr/sbin/sssd//null-/usr/libexec/sssd/sssd_be" name="/run/systemd/resolve/stub-resolv.conf" pid=1507213 comm="sssd_be" requested_mask="r" denied_mask="r" fsuid=0 ouid=101

Here’s an excerpt. Let me know if more is needed. There’s a lot there and most of it is osd up/down and the sssd apparmor messages.

mar · December 10, 2020, 5:45pm

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian stretch amd64 (20200618_16:58)
  image.os: Debian
  image.release: stretch
  image.serial: "20200618_16:58"
  image.type: disk-kvm.img
  limits.cpu: "8"
  limits.memory: 16GB
  security.secureboot: "false"
  volatile.base_image: 214c4284fea0055f1563e75a54dc349259428d43ea5e8502a78092b63d3852ec
  volatile.eth0.hwaddr: 00:16:3e:4d:81:a2
  volatile.last_state.power: RUNNING
  volatile.uuid: <UUID>
devices:
  audit:
    ceph.cluster_name: ceph
    ceph.user_name: lxd
    source: ceph:lxd-virtualvaultfs-01/disk_pod-890-1000-00_audit
    type: disk
  disk_a:
    ceph.cluster_name: ceph
    ceph.user_name: lxd
    source: ceph:lxd-virtualvaultfs-01/disk_pod-890-1000-00_a
    type: disk
  disk_b:
    ceph.cluster_name: ceph
    ceph.user_name: lxd
    source: ceph:lxd-virtualvaultfs-01/disk_pod-890-1000-00_b
    type: disk
  disk_c:
    ceph.cluster_name: ceph
    ceph.user_name: lxd
    source: ceph:lxd-virtualvaultfs-01/disk_pod-890-1000-00_c
    type: disk
  disk_d:
    ceph.cluster_name: ceph
    ceph.user_name: lxd
    source: ceph:lxd-virtualvaultfs-01/disk_pod-890-1000-00_d
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: br4
    type: nic
  root:
    path: /
    pool: ceph
    size: 1500GB
    type: disk
ephemeral: false
profiles:
- stagingtwo-virtual-vault-890
stateful: false
description: ""

tomp · December 10, 2020, 5:54pm

@stgraber I think AppArmor is blocking qemu from accessing the ceph config file specified here:

github.com

lxc/lxd/blob/master/lxd/device/disk.go#L483


			fields = strings.SplitN(fields[1], "/", 2)
			poolName := fields[0]
			volumeName := fields[1]
			clusterName, userName := d.cephCreds()
			// Configuration values containing :, @, or = can be escaped with a leading \ character.
			// According to https://docs.ceph.com/docs/hammer/rbd/qemu-rbd/#usage
			optEscaper := strings.NewReplacer(":", `\:`, "@", `\@`, "=", `\=`)
			opts := []string{
				fmt.Sprintf("id=%s", optEscaper.Replace(userName)),
				fmt.Sprintf("conf=/etc/ceph/%s.conf", optEscaper.Replace(clusterName)),
			}
			runConf.Mounts = []deviceConfig.MountEntryItem{
				{
					DevPath: fmt.Sprintf("rbd:%s/%s:%s", optEscaper.Replace(poolName), optEscaper.Replace(volumeName), strings.Join(opts, ":")),
					DevName: d.name,
				},
			}
		} else {
			srcPath := shared.HostPath(d.config["source"])

It can be any filename based on the cluster name from cephCreds function, so likely need to add an exception for /etc/ceph

stgraber · December 10, 2020, 5:56pm

That’s the issue, I’ll sort something out.

stgraber · December 10, 2020, 5:58pm

@mar in the meantime, you can use the raw.apparmor config key to get yourself out of this issue.

lxc config set NAME raw.apparmor="/var/lib/snapd/hostfs/etc/ceph/** r," should do the trick.

stgraber · December 10, 2020, 6:01pm

https://github.com/lxc/lxd/pull/8239 for the actual fix

mar · December 10, 2020, 6:02pm

Thanks - that got the VMs started.