Postfix doesn't run in an LXD Fedora 31 container: permission denied

LXD
1

I’m in the process to reinstall an LXD postfix service container based on Fedora 31 and CentOS 8 LXD runtime host. That container was running fine with previous versions of Fedora and a CentOS7 runtime.

When I start Postfix inside Fedora 31 container I get:

>     Mär 30 06:15:44 mail-mtaservice systemd[1]: Starting Postfix Mail Transport Agent...
>     Mär 30 06:15:45 mail-mtaservice postfix[86]: /usr/libexec/postfix/postfix-script: line 127: /dev/null: Permission denied
>     Mär 30 06:15:45 mail-mtaservice postfix/postfix-script[93]: fatal: the Postfix mail system is already running

Inside the mentioned script

start|start-fg)
            $daemon_directory/master -t 2>/dev/null || {.          <===  line 127
                    $FATAL the Postfix mail system is already running
                    exit 1
            }

In dev:

drwx--x--x.  2 nobody nobody       40 Mär 30 06:15 .lxd-mounts
drwxrwxrwt.  2 nobody nobody       40 Mär 30 06:14 mqueue
drwxr-xr-x.  2 root   root         60 Mär 30 06:15 net
crw-rw-rw-.  1 nobody nobody   1,   3 Mär 30 06:14 null         <====
crw-rw-rw-.  1 root   root     5,   2 Mär 30 06:15 ptmx
drwxr-xr-x.  2 root   root          0 Mär 30 06:15 pts

Of course, in native Fedora it works fine (/dev/null is owned by root).

There is no second postfix instance inside the container nor on the host.
It’s a plain standard installation of the container (i.e. just a lxc launch …) and the postfix software.

Any help appreciated.

2

What’s the owner & permissions on /dev/null on your host?
And also the content of /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/devices/NAME/

3

Regarding owner and permissions on /dev/null:

[root@mail-mtaservice ~]# ls -al /dev/null 
crw-rw-rw-. 1 nobody nobody 1, 3 Mär 30 06:14 /dev/null

which looks strange to me. But practically all files in /dev are owned by nobody. That is true vor all containers I created since installation on CentOS 8.

I don’t use snap but the Fedora copr repository maintained by ganto
(https://copr.fedorainfracloud.org/coprs/ganto/lxc3/)

All the LXD stuff is stored in /var/lib/lxd and the
/var/lib/lxd/devices/mail-mtaservice directory is empty.

4

I’ve run a few more tests:

  • I installed snapd on a CentOS 8 machine and created a CentOS 8 container: The latest LXD 4.0 has the same issue:
    /usr/libexec/postfix/postfix-script: line 127: /dev/null: Permission denied
    Postfix will not run in an LXD 4.0 container.
    Same is true for version 3.21 which I used when I ran into the issue. I checked with SELinux in permissive as well as in enforcing mode

  • I installed snapd on a Fedora 31 machine. LXD version 4.0 worked, the postfix problem did not occur. (but LXD version 3.21 didn’t work at all)

  • The problem didn’t occur in CentOS 7 either.

So probably the permission problem may be specific to CentOS 8.

Is there a chance that such a CentOS 8 specific problem can be fixed in the foreseeable future?

5

Did you run snap LXD 4 on the same CentOS 8 host where you tried my packages? In this case it might be the same error caused e.g. by my lxcfs package… Did you have SELinux enabled?

6

Originally there were 2 different machines but installed in the very same way, just a minimal CentOS 8 server installation. But to be on the safe side, I reverted the test machine with snap to a snapshot before I started the snap installation and installed copr lxd 3.21. I encountered the same permission issue with Postfix.

I’d be happy to run more tests if I can help with that. In any case, I have a certain amount of testing equipment at my disposal.

7

I just noticed that you released 3.22 on COPR. I updated our systems but found the permission denied issue again. Just for information, could you not address the issue or is the issue not resolvable?

Sorry, I don’t want to push at all, I just have to decide what containerisation system we will use on our systems in the future.

8

Hi all,
any progress here? I have installed CentOS 8 container in LXD (also driven by CentOS 8):

:; lxc version
Client version: 4.12
Server version: 4.12

Postfix (postfix-3.3.1-12.el8.x86_64.rpm ) does not starts:

Mar 17 10:38:17 CT postfix[92]: /usr/libexec/postfix/postfix-script: line 127: /dev/null: Permission denied
Mar 17 10:38:17 CT postfix/postfix-script[116]: fatal: the Postfix mail system is already running
Mar 17 10:38:18 CT systemd[1]: postfix.service: Control process exited, code=exited status=1
Mar 17 10:38:18 CT systemd[1]: postfix.service: Failed with result 'exit-code'.

Container runs in unprivileged mode, so:

:; ls -al /dev/null
crw-rw-rw-. 1 nobody nobody 1, 3 Feb 22 22:39 /dev/null

Device IS usable (no errors on this command):

:; ls -al /usr/ > /dev/null

I am able to start postfix with:

:; sudo /usr/sbin/postfix start

It fails, when using:

 :; sudo systemctl start postfix

Unit file (from postfix rpm):

[Unit]
Description=Postfix Mail Transport Agent
After=syslog.target network.target
Conflicts=sendmail.service exim.service

[Service]
Type=forking
PIDFile=/var/spool/postfix/pid/master.pid
EnvironmentFile=-/etc/sysconfig/network
PrivateTmp=true
CapabilityBoundingSet=~ CAP_NET_ADMIN CAP_SYS_ADMIN CAP_SYS_BOOT CAP_SYS_MODULE
ProtectSystem=true
PrivateDevices=true
ExecStartPre=-/usr/libexec/postfix/aliasesdb
ExecStartPre=-/usr/libexec/postfix/chroot-update
ExecStart=/usr/sbin/postfix start
ExecReload=/usr/sbin/postfix reload
ExecStop=/usr/sbin/postfix stop

[Install]
WantedBy=multi-user.target

To me, it looks like combination of unit file and way unprivileged container works in LXD.
Any advice?

Best regards

9

Can you try using systemctl edit postfix and then setting PrivateDevices to false, and also try ProtectSystem=false to see if it is one of those causing the issue.

10

Also what is on line 127 of /usr/libexec/postfix/postfix-script?

11

Hi @tomp:

here it is:

:; awk ‘{if (NR == 127) {printf(“%s\n”, $0)}}’ /usr/libexec/postfix/postfix-script
$daemon_directory/master -t 2>/dev/null || {

Regards.

12

Hi @tomp,

sudo systemctl edit postfix resulted in creation of /etc/systemd/system/postfix.service.d/override.conf file, with contents:

[Service]
PrivateDevices=false
ProtectSystem=false

After that, service was restarted w/o error:

:; sudo systemctl status postfix
● postfix.service - Postfix Mail Transport Agent
Loaded: loaded (/usr/lib/systemd/system/postfix.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/postfix.service.d
└─override.conf
Active: active (running) since Wed 2021-03-17 11:22:50 CET; 5s ago
…
Mar 17 11:22:50 CT postfix/master[1317]: daemon started – version 3.3.1, configuration /etc/postfix

Thanks, but I would expect that postfix service starts normally, with distribution unit file.

Regards.

13

Which one of the two fixed it though?

14

sudo systemctl edit postfix (see changes in my last comment, actually creation of override.conf).
Regards

15

I expected you to try each one of these two options and then report back which one worked, as trying them both together doesn’t tell us which one worked. Unless they are both required to make it work?

Either way please can you confirm.

16

Sorry, I did not catch that properly, so final override.conf made it working:

[Service]
PrivateDevices=false

Regards

1 Like
17

@Predrag_Zecevic when you try to start postfix with the default system unit, do you see anything from AppArmor on the host via sudo dmesg?

@brauner @stgraber is there any reason why systemd’s PrivateDevices option wouldn’t work inside a container (well restrict writing to /dev/null at least)?

18

II got an error:

:; sudo dmesg
dmesg: read kernel buffer failed: Operation not permitted

So, not sure why that too.
Regards.

19

That’s normal on most distro kernels for a long time and new for Ubuntu: unprivileged users aren’t able to read dmesg.

20

What systemd version are you on, afaict PrivateDevices should either ignore permissions errors and move on or simpley fallback to bind-mount. I’m sure I even fixed that in upstream systemd a while ago.