Cannot create root password in privileged container


#1

host - ubuntu cosmic with 4.18.0-11 kernel / systemd 239-7 / apparmor 2.12

There is no issue with creating a root password via lxc-attach and passwd in an unprivileged container but very curiously it is not possible to create a password the same way for a privileged container (tried centos 7 and ubuntu cosmic),

The error reads

passwd: System error
passwd: Authentication token manipulation error

From journalctl -f it reports

passwd[7799]: PAM audit_log_acct_message() failed: Operation not permitted

Tried also chroot /srv/lxc/container/rootfs passwd but that is not working for either unprivileged or privileged container.

Now why would setting a password in an unprivileged container work but not in a privileged one and how to remedy, or is it a (nother systemd) bug perhaps?

This sounds similar of an issue also considering the patch - removing NoNewPrivileges=true and adding CAP_AUDIT_WRITE to CapabilityBoundingSet

Tried with lxc.cap.keep = CAP_AUDIT_WRITE but the container would not boot.


#2

@brauner @stgraber

Is here anything that can be done to debug this further?


#3

Switched kernel to 4.19.6-041906-generic but the issue persists.


with lxc.cap.keep = CAP_AUDIT_WRITE and lxc-start test -F -L /tmp/test.log just nothing happens, no output on the console, the log file is created but empty.

From the host log it appears that the network device gets engaged but that is all (container not booting). Instead there appears this line

new mount options do not match the existing superblock, will be ignored


#4

Tried lxc.apparmor.profile = unconfined but that did not remedy the issue. At least kernel and apparmor are eliminated as potential cause.

Suppose all that is left now is to file a bug report at the ubuntu package repo then.


#5

Had hoped that the update to 3.0.3 would relief the issue but it has not :slightly_frowning_face:

Bug report filed at downstream https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1807134


(Stéphane Graber) #6

@brauner any idea?

@n8v8r did you try manually downgrading to 3.0.1 to confirm that it’s not instead a kernel/libc change that’s causing the issue?


#7

there is only 3.0.2 and 3.0.3 in the Cosmic repo available and thus no


(Stéphane Graber) #8

Cosmic released with 3.0.2, so you never had this working then?


#9

Used only unprivileged containers thus far and there is no so issue with it, that was prior prior to 3.0.2 and on bionic. LXC been upgraded through the cycles to now 3.0.3.

Now I need a server monitoring app to run a in a privileged container and hence 3.0.2 been indeed the first time that privileged container was tried.

From a user perspective it baffles that it happens with privileged containers but not unprivileged. But suppose that is because of different uid/gid involved?


(Stéphane Graber) #10

Yeah, feels like some component in the PAM stack doing something odd in some cases.

You may want to go look for something related to audit in /etc/pam.d and try commenting it out, see if that gets rid of the problem.


#11

Nothing odd there and no issues with PAM otherwise or elsewhere. Also ran pam-auth-update

It seems more like a CAP issue, like the link in the initial post or this one

Each been resolved with CAP_AUDIT_WRITE and which I tried.


#12

Unless I am suffering a misconception the root user of the privileged container is the same as on the host (INIT ID - Defaults to: UID(0), GID(0)).

Thus should it not be possible with lxc-console to log into the privileged container with the host’s root credentials? Having tried this it produces however

Login incorrect


(Stéphane Graber) #13

Nope, privileged containers mean that uid 0 in the container is mapped to uid 0 outside of it.

This doesn’t however relate in any way with credentials as login credentials are purely a userspace concept implemented by the shadow tools and login command.


#14

Thanks for the clarification/education. I am loss now how to create a root passwd in the privileged container.

The container has the same CAP set as the host

grep ^Cap /proc/$$/status

CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000

capsh --decode=0000003fffffffff

0x0000003fffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read


(Christian Brauner) #15

Can you show the permissions on the containers rootfs files (A simple ls -al will do.).
This seems like a very odd error. The only reasonable explanation seems wrong permissions
or ownership of needed files.


#16

rdrwxrwx— 1 100000 100000 24 Dec 5 12:33 .
drwxr-xr-x 1 root root 78 Dec 15 14:50 …
-rw-r–r-- 1 root root 1749 Dec 15 13:48 config
drwxr-xr-x 1 100000 100000 132 Dec 15 15:39 rootfs


(Christian Brauner) #17

You seem to be using the rootfs of an unprivileged container but with a config without id mappings set up. So what’s happening is perfectly in order:

root@wittgenstein|~
> vim /home/brauner/.local/share/lxc/c1/

[1]+  Stopped                 nvim /home/brauner/.local/share/lxc/c1/
root@wittgenstein|~
> sudo lxc-start c1 -P /home/brauner/.local/share/lxc/
root@wittgenstein|~
> sudo lxc-attach c1 -P /home/brauner/.local/share/lxc/
root@c1:/# passwd
Enter new UNIX password: 
Retype new UNIX password: 
passwd: System error
passwd: password unchanged
root@c1:/# ls -al /etc/passwd 
-rw-r--r-- 1 100000 100000 1249 Aug 21 07:45 /etc/passwd

#18

In the privileged container settings these are commented out (disabled)

# lxc.idmap = u 0 100000 65536
# lxc.idmap = g 0 100000 65536

Thus I would expect the container being privileged and not unprivileged.

However in the default.conf they are not commented out. And what appears to happen is that lxc-create is applying the settings from the default.conf as unprivileged container and the container specific settings (privileged) cannot be applied anymore.

I am not sure that this logic well documented. I am closing the downstream bug.