Dbus hangs in Jessie with Gentoo (Openrc host)

Hello,

I am using gentoo and trying to use jessie. Unfortunately, DBUS hangs in jessie

# lxc exec jessie bash      
root@jessie:~# systemctl status
Failed to get D-Bus connection: No such file or directory
root@jessie:~# apt-get install systemd
Reading package lists... Done
Building dependency tree       
Reading state information... Done
systemd is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@jessie:~# systemctl status
● jessie
    State: degraded
     Jobs: 0 queued
   Failed: 1 units
    Since: Mon 2018-01-01 17:08:56 UTC; 36s ago
   CGroup: /
           ├─ 1 /sbin/init
           ├─ 4 bash
           ├─97 systemctl status
           ├─98 pager
           └─system.slice
             ├─console-getty.service
             │ └─90 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 linux
             ├─networking.service
             │ └─68 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
             └─systemd-journald.service
               └─21 /lib/systemd/systemd-journald

root@jessie:~# apt-get install dbus
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libcap-ng0 libdbus-1-3 libexpat1
Suggested packages:
  dbus-x11
The following NEW packages will be installed:
  dbus libcap-ng0 libdbus-1-3 libexpat1
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 555 kB of archives.
After this operation, 1737 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
0% [Connecting to deb.debian.org (2001:41c8:1000:21::21:4)]^C
root@jessie:~# apt-get install dbus -y
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libcap-ng0 libdbus-1-3 libexpat1
Suggested packages:
  dbus-x11
The following NEW packages will be installed:
  dbus libcap-ng0 libdbus-1-3 libexpat1
0 upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 555 kB of archives.
After this operation, 1737 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian/ jessie/main libcap-ng0 amd64 0.7.4-2 [13.2 kB]
Get:2 http://deb.debian.org/debian/ jessie/main libdbus-1-3 amd64 1.8.22-0+deb8u1 [170 kB]
Get:3 http://deb.debian.org/debian/ jessie/main libexpat1 amd64 2.1.0-6+deb8u4 [80.2 kB]
Get:4 http://deb.debian.org/debian/ jessie/main dbus amd64 1.8.22-0+deb8u1 [292 kB]
Fetched 555 kB in 1s (318 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package libcap-ng0:amd64.
(Reading database ... 8627 files and directories currently installed.)
Preparing to unpack .../libcap-ng0_0.7.4-2_amd64.deb ...
Unpacking libcap-ng0:amd64 (0.7.4-2) ...
Selecting previously unselected package libdbus-1-3:amd64.
Preparing to unpack .../libdbus-1-3_1.8.22-0+deb8u1_amd64.deb ...
Unpacking libdbus-1-3:amd64 (1.8.22-0+deb8u1) ...
Selecting previously unselected package libexpat1:amd64.
Preparing to unpack .../libexpat1_2.1.0-6+deb8u4_amd64.deb ...
Unpacking libexpat1:amd64 (2.1.0-6+deb8u4) ...
Selecting previously unselected package dbus.
Preparing to unpack .../dbus_1.8.22-0+deb8u1_amd64.deb ...
Unpacking dbus (1.8.22-0+deb8u1) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libcap-ng0:amd64 (0.7.4-2) ...
Setting up libdbus-1-3:amd64 (1.8.22-0+deb8u1) ...
Setting up libexpat1:amd64 (2.1.0-6+deb8u4) ...
Setting up dbus (1.8.22-0+deb8u1) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Processing triggers for systemd (215-17+deb8u7) ...
root@jessie:~# systemctl status
● jessie
    State: degraded
     Jobs: 1 queued
   Failed: 2 units
    Since: Mon 2018-01-01 17:08:56 UTC; 4min 49s ago
   CGroup: /
           ├─  1 /sbin/init
           ├─  4 bash
           ├─274 systemctl status
           ├─275 pager
           └─system.slice
             ├─console-getty.service
             │ └─90 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 linux
             ├─networking.service
             │ └─68 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
             └─systemd-journald.service
               └─21 /lib/systemd/systemd-journald
root@jessie:~# systemctl --failed
  UNIT                LOAD   ACTIVE SUB    JOB   DESCRIPTION
● dev-hugepages.mount loaded failed failed       Huge Pages File System
● dbus.service        loaded failed failed start D-Bus System Message Bus

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
JOB    = Pending job for the unit.

2 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

Cgroups on host

$ mount|grep cgroup
cgroup_root on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,size=10240k,mode=755)
openrc on /sys/fs/cgroup/openrc type cgroup (rw,nosuid,nodev,noexec,relatime,release_agent=/lib64/rc/sh/cgroup-release-agent.sh,name=openrc)
none on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cpuset on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cpu on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cpuacct on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
blkio on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
memory on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
devices on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
perf_event on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
pids on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,relatime,name=systemd)

mount inside container

# mount
/var/lib/lxd.img on / type btrfs (rw,relatime,space_cache,subvolid=5,subvol=/storage-pools/default/containers/jessie/rootfs)
none on /dev type tmpfs (rw,relatime,size=492k,mode=755,uid=1000000,gid=1000000)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /sys type sysfs (rw,relatime)
devtmpfs on /dev/net/tun type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/lxd type tmpfs (rw,relatime,size=100k,mode=755)
tmpfs on /dev/.lxd-mounts type tmpfs (rw,relatime,size=100k,mode=711)
devtmpfs on /dev/null type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
devtmpfs on /dev/zero type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
devtmpfs on /dev/full type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
devtmpfs on /dev/urandom type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
devtmpfs on /dev/random type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
devtmpfs on /dev/tty type devtmpfs (rw,nosuid,relatime,size=10240k,nr_inodes=1012556,mode=755)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /dev/pts type devpts (rw,relatime,gid=1000005,mode=620,ptmxmode=666)
devpts on /dev/ptmx type devpts (rw,relatime,gid=1000005,mode=620,ptmxmode=666)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,uid=1000000,gid=1000000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755,uid=1000000,gid=1000000)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,uid=1000000,gid=1000000)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755,uid=1000000,gid=1000000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)

If i try to use dbus in Debian Stretch container it works,

root@stretch:~# systemctl status dbus
● dbus.service - D-Bus System Message Bus
   Loaded: loaded (/lib/systemd/system/dbus.service; static; vendor preset: enabled)
   Active: active (running) since Mon 2018-01-01 17:20:04 UTC; 25s ago
     Docs: man:dbus-daemon(1)
 Main PID: 242 (dbus-daemon)
   CGroup: /system.slice/dbus.service
           └─242 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation

Jan 01 17:20:04 stretch systemd[1]: dbus.service: Failed to reset devices.list: Operation not permitted
Jan 01 17:20:04 stretch systemd[1]: dbus.service: Failed to set invocation ID on control group /system.slice/dbus.service, ignoring: Operation not permitted
Jan 01 17:20:04 stretch systemd[1]: Started D-Bus System Message Bus.

Finally,

$ lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
api_status: stable
api_version: "1.0"
auth: trusted
public: false
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  driver: lxc
  driver_version: 2.0.9
  kernel: Linux
  kernel_architecture: x86_64
  kernel_version: 4.9.72-gentoo
  server: lxd
  server_pid: 5189
  server_version: "2.19"
  storage: dir
  storage_version: "1"

Do you get anything else if you do journalctl -u dbus?

Here it is,

# journalctl -u dbus
-- Logs begin at Sat 2018-01-06 00:51:42 UTC, end at Sat 2018-01-06 00:54:35 UTC. --
Jan 06 00:53:13 jessie systemd[1]: Starting D-Bus System Message Bus...
Jan 06 00:53:13 jessie systemd[1]: Started D-Bus System Message Bus.
Jan 06 00:54:03 jessie systemd[1]: dbus.service: main process exited, code=exited, status=206/OOM_ADJUST
Jan 06 00:54:03 jessie systemd[1]: Unit dbus.service entered failed state.

Commented out ‘OOMScoreAdjust=-900’ in /lib/systemd/system/dbus.service and now i get

# journalctl -u dbus
-- Logs begin at Sat 2018-01-06 01:13:44 UTC, end at Sat 2018-01-06 01:13:45 UTC. --
Jan 06 01:13:45 jessie systemd[1]: Starting D-Bus System Message Bus...
Jan 06 01:13:45 jessie systemd[1]: Started D-Bus System Message Bus.
Jan 06 01:13:45 jessie dbus[93]: [system] org.freedesktop.DBus.Error.AccessDenied: Failed to set fd limit to 65536: Operation not permitted
Jan 06 01:13:45 jessie dbus[93]: [system] Successfully activated service 'org.freedesktop.systemd1'

and

# systemctl --failed
  UNIT                LOAD   ACTIVE SUB    DESCRIPTION
● dev-hugepages.mount loaded failed failed Huge Pages File System

# systemctl status dev-hugepages.mount
● dev-hugepages.mount - Huge Pages File System
   Loaded: loaded (/lib/systemd/system/dev-hugepages.mount; static)
   Active: failed (Result: exit-code) since Sat 2018-01-06 01:13:44 UTC; 3min 57s ago
    Where: /dev/hugepages
     What: hugetlbfs
     Docs: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
  Process: 20 ExecMount=/bin/mount -n hugetlbfs /dev/hugepages -t hugetlbfs (code=exited, status=32)

Jan 06 01:13:44 jessie systemd[1]: dev-hugepages.mount mount process exited, code=exited status=32
Jan 06 01:13:44 jessie systemd[1]: Failed to mount Huge Pages File System.
Jan 06 01:13:44 jessie systemd[1]: Unit dev-hugepages.mount entered failed state.
Jan 06 01:13:44 jessie mount[20]: mount: permission denied
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

In Debian stretch container i get

# systemctl status dev-hugepages.mount
● dev-hugepages.mount - Huge Pages File System
   Loaded: loaded (/lib/systemd/system/dev-hugepages.mount; static; vendor preset: enabled)
   Active: inactive (dead)
Condition: start condition failed at Sat 2018-01-06 01:36:47 UTC; 39s ago
           └─ ConditionVirtualization=!private-users was not met
    Where: /dev/hugepages
     What: hugetlbfs
     Docs: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

After adding ConditionVirtualization=!container to /lib/systemd/system/dev-hugepages.mount, the error goes away

#cat /lib/systemd/system/dev-hugepages.mount
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Huge Pages File System
Documentation=https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
Documentation=http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
DefaultDependencies=no
Before=sysinit.target
ConditionPathExists=/sys/kernel/mm/hugepages
ConditionVirtualization=!container

[Mount]
What=hugetlbfs
Where=/dev/hugepages
Type=hugetlbfs

# systemctl status dev-hugepages.mount
● dev-hugepages.mount - Huge Pages File System
   Loaded: loaded (/lib/systemd/system/dev-hugepages.mount; static)
   Active: inactive (dead)
           start condition failed at Sat 2018-01-06 01:58:56 UTC; 9min ago
           ConditionVirtualization=!container was not met
    Where: /dev/hugepages
     What: hugetlbfs
     Docs: https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
           http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems

# systemctl status
● jessie
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2018-01-06 01:58:31 UTC; 10min ago
   CGroup: /
           ├─  1 /sbin/init
           ├─ 92 bash
           ├─101 systemctl status
           ├─102 pager
           └─system.slice
             ├─dbus.service
             │ └─87 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
             ├─systemd-logind.service
             │ └─85 /lib/systemd/systemd-logind
             ├─console-getty.service
             │ └─89 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 linux
             ├─networking.service
             │ └─65 dhclient -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
             └─systemd-journald.service
               └─18 /lib/systemd/systemd-journald

The OOM issue was fixed in systemd upstream I believe, so you may be able to convince the systemd maintainer in Debian to cherry-pick that particular fix.

I’ve never seen the hugepages one before but indeed it looks like the hugetlbfs filesystem isn’t supported inside an unprivileged container and so having that particular unit be marked as unsuitable for containers is probably the easiest, least disruptive fix for this (not that it’s likely to have any actual impact other than being listed in --failed).

The postgres package also requires OOMScoreAdjust=-900 to be commented out at /lib/systemd/system/postgresql@.service before it could work.

1 Like