Cannot start containers on btrfs

danielyrovas · November 24, 2021, 4:16am

I cannot start containers on btrfs with the same error mentioned here
I have tried on arch linux and alpine.
Thanks for your help

tomp · November 24, 2021, 8:59am

What error do you get when you start the instance?

danielyrovas · November 24, 2021, 9:50am

Hmm sorry I believe the issues aren’t the same.

this is the archlinux issue

Name: lxd-ide
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2021/11/24 15:05 AEDT
Last Used: 2021/11/24 20:40 AEDT

Log:

lxc lxd-ide 20211124094005.831 ERROR    conf - conf.c:lxc_map_ids:3654 - newuidmap failed to write mapping "": newuidmap 2830 0 1000000 1000000000
lxc lxd-ide 20211124094005.831 ERROR    start - start.c:lxc_spawn:1790 - Failed to set up id mapping.
lxc lxd-ide 20211124094005.831 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING"
lxc lxd-ide 20211124094005.831 ERROR    start - start.c:__lxc_start:2073 - Failed to spawn container "lxd-ide"
lxc lxd-ide 20211124094005.831 WARN     start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 17 for process 2830
lxc 20211124094010.857 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20211124094010.858 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors

this is the alpinelinux issue

Name: alps
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2021/11/16 20:01 AEDT
Last Used: 2021/11/16 20:16 AEDT

Log:

lxc alps 20211116091608.731 ERROR    conf - conf.c:lxc_map_ids:3654 - newuidmap failed to write mapping "newuidmap: uid range [0-1000000000) -> [1000000-1001000000) not allowed": newuidmap 3735 0 1000000 1000000000
lxc alps 20211116091608.731 ERROR    start - start.c:lxc_spawn:1785 - Failed to set up id mapping.
lxc alps 20211116091608.731 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state "ABORTING" instead of "RUNNING"
lxc alps 20211116091608.731 ERROR    start - start.c:__lxc_start:2068 - Failed to spawn container "alps"
lxc alps 20211116091608.731 WARN     start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 43 for process 3735
lxc 20211116091613.774 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20211116091613.774 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors
lxc 20211116091757.676 TRACE    commands - commands.c:lxc_cmd:509 - Connection refused - Command "get_state" failed to connect command socket
lxc 20211116091757.751 TRACE    commands - commands.c:lxc_cmd:509 - Connection refused - Command "get_state" failed to connect command socket
lxc 20211116091820.921 TRACE    commands - commands.c:lxc_cmd:509 - Connection refused - Command "get_state" failed to connect command socket

tomp · November 24, 2021, 9:56am

The alpine issue is that your /etc/subuid and /etc/subgid files don’t contain the values you’re trying to use for your container’s idmap. You can fix those files or remove the newuidmap tool so that LXC manages idmaps directly.

tomp · November 24, 2021, 9:57am

The arch linux issue looks like you’re trying to use an empty idmap.

danielyrovas · November 24, 2021, 9:59am

Ahh I see

I updated my /etc/sub[gu]id to match the alpine install

───────┬────────
       │ File: /etc/subgid
───────┼────────
   1   │ root:100000:65536
───────┴────────
───────┬────────
       │ File: /etc/subuid
───────┼────────
   1   │ root:100000:65536
───────┴────────

and now the error is the same

lxc info --show-log lxd-ide

``` Name: lxd-ide Status: STOPPED Type: container Architecture: x86_64 Created: 2021/11/24 15:05 AEDT Last Used: 2021/11/24 20:53 AEDT

Log:

lxc lxd-ide 20211124095332.250 ERROR conf - conf.c:lxc_map_ids:3654 - newuidmap failed to write mapping “newuidmap: uid range [0-1000000000) → [1000000-1001000000) not allowed”: newuidmap 8791 0 1000000 1000000000
lxc lxd-ide 20211124095332.250 ERROR start - start.c:lxc_spawn:1790 - Failed to set up id mapping.
lxc lxd-ide 20211124095332.250 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:867 - Received container state “ABORTING” instead of “RUNNING”
lxc lxd-ide 20211124095332.250 ERROR start - start.c:__lxc_start:2073 - Failed to spawn container “lxd-ide”
lxc lxd-ide 20211124095332.250 WARN start - start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 17 for process 8791
lxc 20211124095337.288 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20211124095337.288 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors

</details>

danielyrovas · November 24, 2021, 10:01am

If I use a default install of lxd without modifying the sub[gu]ids should it work? I hadn’t touched anything on my arch install before just now, so I don’t know why it would be empty.

tomp · November 24, 2021, 10:10am

Any ideas @brauner ?

tomp · November 24, 2021, 10:11am

Please can you show the lxc config show <instance> --expanded for each.

danielyrovas · November 24, 2021, 10:20am

for archlinux:

lxc config show lxd-ide --expanded

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian bullseye amd64 (20211123_05:24)
  image.os: Debian
  image.release: bullseye
  image.serial: "20211123_05:24"
  image.type: squashfs
  image.variant: default
  volatile.base_image: 8054e116e0c707476b67d2c6a10186d708b219da94371c4297edff26167b163e
  volatile.eth0.host_name: veth80143ce1
  volatile.eth0.hwaddr: 00:16:3e:cc:5f:8d
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 89724dee-5228-47ce-b508-a54361716cdb
devices:
  eth0:
    name: eth0
    network: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

danielyrovas · November 24, 2021, 10:25am

my ubuntu server install, which uses the snap version, is currently working (and updated to 4.20, inline with both alpine and arch), and the containers can launch on btrfs. ID mapping is working on a shared btrfs disk. I can’t check the alpine install because that would require me to restart the Ubuntu server.

tomp · November 24, 2021, 10:26am

You need to allow a larger range, as your container’s have a Maprange entry up to:

And you’re only allowing up to 65536

tomp · November 24, 2021, 10:27am

On the snap package, it doesn’t use the host’s newuidmap tool and so isn’t beholden to the restrictions set in /etc/subuid and /etc/subgid.

This is why I also suggested removing the newuidmap tool entirely.

danielyrovas · November 24, 2021, 10:35am

how can I remove it? by simply deleting the /etc/sub[gu]id ? I think that left me with an empty id mapping.

tomp · November 24, 2021, 10:46am

No, the newuidtool command, not the /etc/subuid and /etc/subgid files.

brauner · November 24, 2021, 10:50am

Other tools might depend on new{g,u}idmap. You should simply add a large enough map for root and lxd in /etc/sub{g,u}id. Usually:

lxd:100000:1000000000
root:100000:1000000000

in both /etc/subuid and /etc/subgid should suffice.

danielyrovas · November 24, 2021, 11:55am

I needed to add one more zero in the first group in your answer.
Thanks so much for all your help