Running podman on NixOS guest

Hello,

I try to get podman running on my NixOS guest:

asbachb@ubuntu-8gb-nbg1-1:/sys/fs/cgroup/pids$ lxc config show nixos-jenkins
architecture: x86_64
config:
  security.nesting: "true"
  volatile.base_image: d5e2a0b1ddb4c5bc36ced85dd3472dabf4e58e9b3a9aa03de22839e333d3cd34
  volatile.eth0.host_name: veth432d5bb8
  volatile.eth0.hwaddr: 00:16:3e:20:f9:2f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 374b7195-a3af-4b2f-bf88-e37a78fa8d02
devices: {}
ephemeral: false
profiles:
- nixos
stateful: false
description: ""
asbachb@ubuntu-8gb-nbg1-1:/sys/fs/cgroup/pids$ nix^C
asbachb@ubuntu-8gb-nbg1-1:/sys/fs/cgroup/pids$ lxc config show nixos-jenkins -e
architecture: x86_64
config:
  raw.lxc: |-
    lxc.init.cmd = /sbin/init
    lxc.mount.entry = proc mnt/proc proc create=dir 0 0
    lxc.apparmor.profile = unconfined
  security.nesting: "true"
  volatile.base_image: d5e2a0b1ddb4c5bc36ced85dd3472dabf4e58e9b3a9aa03de22839e333d3cd34
  volatile.eth0.host_name: veth432d5bb8
  volatile.eth0.hwaddr: 00:16:3e:20:f9:2f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 374b7195-a3af-4b2f-bf88-e37a78fa8d02
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: storage1
    type: disk
  shared-config:
    path: /etc/nixos/shared
    pool: storage1
    source: nixos-shared-config
    type: disk
ephemeral: false
profiles:
- nixos
stateful: false
description: ""

But launching a container fails:

[root@nixos:~]# podman run hello-world
Error: OCI runtime error: the requested cgroup controller `pids` is not available

Any ideas to resolve this issue without switchting to a privileged container?

@brauner should the pids namespace be available inside a container? Thanks

You’re mixing up the cgroup pids controller and pid namespaces. :slight_smile:

1 Like

So you’re running NixOS in a LXD container and you want to run Podman inside that LXD container?

What mounts are available inside the container, i.e. what does findmnt in the NixOS container show? Or if that tool doesn’t work on NixOS what does cat <procfs-mount>/1/mountinfo show?

Yeah. That’s correct.

[root@nixos:~]# findmnt 
TARGET                       SOURCE                 FSTYPE      OPTIONS
/                            /dev/sda1[/var/snap/lxd/common/lxd/storage-pools/storage1/containers/nixos-jenkins/rootfs]
β”‚                                                   ext4        rw,relatime,errors=remount-ro
β”œβ”€/nix/store                 /dev/sda1[/var/snap/lxd/common/lxd/storage-pools/storage1/containers/nixos-jenkins/rootfs/nix/store]
β”‚                                                   ext4        ro,relatime,errors=remount-ro
β”œβ”€/run                       tmpfs                  tmpfs       rw,nosuid,nodev,size=1989544k,mode=755,uid=1000000,gid=1000000
β”‚ β”œβ”€/run/keys                none                   ramfs       rw,nosuid,nodev,relatime,mode=750
β”‚ └─/run/wrappers            tmpfs                  tmpfs       rw,nodev,relatime,mode=755,uid=1000000,gid=1000000
β”œβ”€/dev                       none                   tmpfs       rw,nosuid,size=397912k,mode=755,uid=1000000,gid=1000000
β”‚ β”œβ”€/dev/shm                 tmpfs                  tmpfs       rw,nosuid,nodev,uid=1000000,gid=1000000
β”‚ β”œβ”€/dev/fuse                udev[/fuse]            devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/net/tun             udev[/net/tun]         devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/mqueue              mqueue                 mqueue      rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/dev/.lxc/proc           proc                   proc        rw,relatime
β”‚ β”œβ”€/dev/.lxc/sys            sys                    sysfs       rw,relatime
β”‚ β”œβ”€/dev/lxd                 tmpfs                  tmpfs       rw,relatime,size=100k,mode=755
β”‚ β”œβ”€/dev/.lxd-mounts         tmpfs[/nixos-jenkins]  tmpfs       rw,relatime,size=100k,mode=711
β”‚ β”œβ”€/dev/full                udev[/full]            devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/null                udev[/null]            devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/random              udev[/random]          devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/tty                 udev[/tty]             devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/urandom             udev[/urandom]         devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/zero                udev[/zero]            devtmpfs    rw,nosuid,noexec,relatime,size=3961460k,nr_inodes=990365,mode=755
β”‚ β”œβ”€/dev/console             devpts[/1]             devpts      rw,relatime,gid=5,mode=620,ptmxmode=666
β”‚ β”œβ”€/dev/pts                 devpts                 devpts      rw,nosuid,noexec,relatime,gid=1000003,mode=620,ptmxmode=666
β”‚ └─/dev/ptmx                devpts[/ptmx]          devpts      rw,nosuid,noexec,relatime,gid=1000003,mode=620,ptmxmode=666
β”œβ”€/proc                      proc                   proc        rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/proc/sys/fs/binfmt_misc binfmt_misc            binfmt_misc rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/proc/cpuinfo            lxcfs[/proc/cpuinfo]   fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ β”œβ”€/proc/diskstats          lxcfs[/proc/diskstats] fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ β”œβ”€/proc/loadavg            lxcfs[/proc/loadavg]   fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ β”œβ”€/proc/meminfo            lxcfs[/proc/meminfo]   fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ β”œβ”€/proc/stat               lxcfs[/proc/stat]      fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ β”œβ”€/proc/swaps              lxcfs[/proc/swaps]     fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ β”œβ”€/proc/uptime             lxcfs[/proc/uptime]    fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”‚ └─/proc/sys/kernel/random/boot_id
β”‚                            none[/.lxc-boot-id]    tmpfs       ro,nosuid,nodev,noexec,relatime,size=397912k,mode=755,uid=1000000,gid=10
β”œβ”€/sys                       sysfs                  sysfs       rw,relatime
β”‚ β”œβ”€/sys/fs/cgroup           cgroup2                cgroup2     rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/fs/fuse/connections fusectl                fusectl     rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/fs/pstore           pstore                 pstore      rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/kernel/config       configfs               configfs    rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/kernel/debug        debugfs                debugfs     rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/kernel/security     securityfs             securityfs  rw,nosuid,nodev,noexec,relatime
β”‚ β”œβ”€/sys/kernel/tracing      tracefs                tracefs     rw,nosuid,nodev,noexec,relatime
β”‚ └─/sys/devices/system/cpu/online
β”‚                            lxcfs[/sys/devices/system/cpu/online]
β”‚                                                   fuse.lxcfs  rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other
β”œβ”€/mnt/proc                  proc                   proc        rw,relatime
└─/etc/nixos/shared          /dev/sda1[/var/snap/lxd/common/lxd/storage-pools/storage1/custom/default_nixos-shared-config]
                                                    ext4        rw,relatime,errors=remount-ro

[root@nixos:~]# cat /proc/1/mountinfo 
1505 807 8:1 /var/snap/lxd/common/lxd/storage-pools/storage1/containers/nixos-jenkins/rootfs / rw,relatime master:353 - ext4 /dev/sda1 rw,errors=remount-ro
1506 1505 0:74 / /dev rw,nosuid - tmpfs none rw,size=397912k,mode=755,uid=1000000,gid=1000000
1507 1505 0:73 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
1508 1505 0:75 / /sys rw,relatime - sysfs sysfs rw
1509 1506 0:6 /fuse /dev/fuse rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1510 1506 0:6 /net/tun /dev/net/tun rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1511 1507 0:66 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime master:357 - binfmt_misc binfmt_misc rw
1512 1508 0:48 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime master:32 - fusectl fusectl rw
1513 1508 0:32 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
1514 1508 0:22 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime master:33 - configfs configfs rw
1515 1508 0:8 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime master:29 - debugfs debugfs rw
1516 1508 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
1517 1508 0:12 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime master:30 - tracefs tracefs rw
1518 1506 0:21 / /dev/mqueue rw,nosuid,nodev,noexec,relatime master:28 - mqueue mqueue rw
1519 1506 0:73 / /dev/.lxc/proc rw,relatime - proc proc rw
1520 1506 0:75 / /dev/.lxc/sys rw,relatime - sysfs sys rw
1521 1506 0:61 / /dev/lxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755
1522 1505 0:73 / /mnt/proc rw,relatime - proc proc rw
1523 1505 8:1 /var/snap/lxd/common/lxd/storage-pools/storage1/custom/default_nixos-shared-config /etc/nixos/shared rw,relatime master:353 - ext4 /dev/sda1 rw,errors=remount-ro
1524 1506 0:60 /nixos-jenkins /dev/.lxd-mounts rw,relatime master:356 - tmpfs tmpfs rw,size=100k,mode=711
1525 1507 0:59 /proc/cpuinfo /proc/cpuinfo rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1526 1507 0:59 /proc/diskstats /proc/diskstats rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1527 1507 0:59 /proc/loadavg /proc/loadavg rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1528 1507 0:59 /proc/meminfo /proc/meminfo rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1529 1507 0:59 /proc/stat /proc/stat rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1530 1507 0:59 /proc/swaps /proc/swaps rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1531 1507 0:59 /proc/uptime /proc/uptime rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1532 1508 0:59 /sys/devices/system/cpu/online /sys/devices/system/cpu/online rw,nosuid,nodev,relatime master:355 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
1533 1506 0:6 /full /dev/full rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1534 1506 0:6 /null /dev/null rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1535 1506 0:6 /random /dev/random rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1536 1506 0:6 /tty /dev/tty rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1537 1506 0:6 /urandom /dev/urandom rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1538 1506 0:6 /zero /dev/zero rw,nosuid,noexec,relatime master:2 - devtmpfs udev rw,size=3961460k,nr_inodes=990365,mode=755
1539 1506 0:51 /1 /dev/console rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
1540 1507 0:74 /.lxc-boot-id /proc/sys/kernel/random/boot_id ro,nosuid,nodev,noexec,relatime - tmpfs none rw,size=397912k,mode=755,uid=1000000,gid=1000000
1541 1506 0:76 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=1000003,mode=620,ptmxmode=666
1542 1506 0:76 /ptmx /dev/ptmx rw,nosuid,noexec,relatime - devpts devpts rw,gid=1000003,mode=620,ptmxmode=666
794 1505 8:1 /var/snap/lxd/common/lxd/storage-pools/storage1/containers/nixos-jenkins/rootfs/nix/store /nix/store ro,relatime master:353 - ext4 /dev/sda1 rw,errors=remount-ro
805 1506 0:79 / /dev/shm rw,nosuid,nodev - tmpfs tmpfs rw,uid=1000000,gid=1000000
809 1505 0:80 / /run rw,nosuid,nodev - tmpfs tmpfs rw,size=1989544k,mode=755,uid=1000000,gid=1000000
973 809 0:81 / /run/keys rw,nosuid,nodev,relatime - ramfs none rw,mode=750
974 809 0:84 / /run/wrappers rw,nodev,relatime - tmpfs tmpfs rw,mode=755,uid=1000000,gid=1000000
1080 1508 0:30 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup2 rw

This is a pure cgroup2 system. Idk if Podman can deal with pure cgroup2 layouts yet. You could try and set:

lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=false

to force systemd to fallback to the legacy cgroup hierarchy and then try again.

Please also show me ls -al /sys/fs/cgroup from within the container.

Podman fully supports cgroups v2 with the crun runtime instead of runc (dunno if supported with runc > v1.0.0-rc93 which should supports cgroups v2). It’s the default on Fedora with cgroups v2 (it’s not an hybrid hierarchy IIRC) but idk if crun is packaged on other distributions such as Nix.

I guess the installation already uses crun

  ociRuntime:
name: crun
package: Unknown
path: /nix/store/swab9gv1nb11ds0l2rxapzrs7rrp0bzj-crun-0.19.1/bin/crun
version: |-
  crun version 0.19.1
  commit: 0.19.1
  spec: 1.0.0
  +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL

@brauner I tried to set that configuration but I guess it since findmnt still shows cgroup2

asbachb@ubuntu-8gb-nbg1-1:~$ lxc config show nixos-jenkins2 -e
architecture: x86_64
config:
  raw.lxc: |-
    lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy=false
    lxc.mount.entry = proc mnt/proc proc create=dir 0 0
    lxc.apparmor.profile = unconfined
  security.nesting: "true"
  volatile.base_image: d5e2a0b1ddb4c5bc36ced85dd3472dabf4e58e9b3a9aa03de22839e333d3cd34
  volatile.eth0.host_name: veth598851a6
  volatile.eth0.hwaddr: 00:16:3e:f1:2a:70
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: 7d98e619-1624-4f94-b573-433df46bce70
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: storage1
    type: disk
  shared-config:
    path: /etc/nixos/shared
    pool: storage1
    source: nixos-shared-config
    type: disk
ephemeral: false
profiles:
- nixos
stateful: false
description: ""
[root@nixos:~]# findmnt | grep cgroup
β”‚ β”œβ”€/sys/fs/cgroup                    cgroup2                                                                                                                        cgroup2     rw,nosuid,nodev,noexec,relatime
[root@nixos:~]# ls -al /sys/fs/cgroup
total 0
drwxrwxr-x 7 nobody root    0 May 26 23:52 .
drwxr-xr-x 9 nobody nogroup 0 May 26 23:51 ..
-r--r--r-- 1 nobody nogroup 0 May 26 23:51 cgroup.controllers
-r--r--r-- 1 nobody nogroup 0 May 26 23:55 cgroup.events
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 cgroup.freeze
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 cgroup.max.depth
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 cgroup.max.descendants
-rw-rw-r-- 1 nobody root    0 May 26 23:51 cgroup.procs
-r--r--r-- 1 nobody nogroup 0 May 26 23:55 cgroup.stat
-rw-rw-r-- 1 nobody root    0 May 26 23:51 cgroup.subtree_control
-rw-rw-r-- 1 nobody root    0 May 26 23:51 cgroup.threads
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 cgroup.type
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 cpu.pressure
-r--r--r-- 1 nobody nogroup 0 May 26 23:55 cpu.stat
drwxr-xr-x 2 root   root    0 May 26 23:51 init.scope
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 io.pressure
drwxr-xr-x 2 root   root    0 May 26 23:51 .lxc
drwxr-xr-x 2 root   root    0 May 26 23:52 machine.slice
-rw-r--r-- 1 nobody nogroup 0 May 26 23:55 memory.pressure
drwxr-xr-x 9 root   root    0 May 26 23:51 system.slice
drwxr-xr-x 2 root   root    0 May 26 23:51 user.slice

Ah, the pids controller is indeed not enabled in the cgroup2 hierarchy here. Otherwise there would be pids.current entry in there. Can you please show me the output of findmnt on the host and the output of cat /proc/<container-init-pid>/cgroup?