Cannot start LXD container within LXD container

LXD
,
1

We have been running LXD and LXC on RHEL 7 for some time. Until recently, it’s been lxc-2.0.0 and lxd-2.19.

We’re looking to upgrade to LXD 4.6 & LXC 4.0.4.

I’ve made changes re: configuration keys found here: LXC 2.1 has been released

We’re utilizing packer to create our custom RHEL container images. No issues launching/starting a container from an image on a physical host running the upgraded LXD/LXC. However, we run into problems starting another container within one, which is fairly crucial to our workflow.

log on start:
[root@lxlc-bwil02 ~]# lxc info --show-log local:foo3
Name: foo3
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/11 03:20 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.857 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.858 ERROR cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo3 20210211032049.861 ERROR utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds
lxc foo3 20210211032049.861 ERROR start - start.c:lxc_spawn:1747 - Failed to setup cgroup limits for container “foo3”
lxc foo3 20210211032049.861 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:850 - Received container state “ABORTING” instead of “RUNNING”
lxc foo3 20210211032049.861 ERROR start - start.c:__lxc_start:1999 - Failed to spawn container “foo3”
lxc foo3 20210211032049.861 WARN start - start.c:lxc_abort:1024 - No such process - Failed to send SIGKILL to 2042

[root@lxlc-bwil02 ~]# lxc config show --expanded foo3
architecture: x86_64
config:
image.architecture: x86_64
image.description: rhel7 x86_64 base cloud-init
image.name: rhel7-lxd-base
image.os: rhel
image.platform: rhel7
image.release: “”
image.variant: cloud-init
linux.kernel_modules: loop,ip_tables,ip6_tables
security.nesting: “true”
security.privileged: “true”
volatile.base_image: 41acaa4a3a51d5dc8e87fb963eb22660df5e21e7d5b5f628bb3bcb8a86ce74be
volatile.eth0.hwaddr: 00:16:3e:c4:21:cc
volatile.idmap.base: “0”
volatile.idmap.current: ‘[]’
volatile.idmap.next: ‘[]’
volatile.last_state.idmap: ‘[]’
volatile.last_state.power: STOPPED
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdnet0
type: nic
libmodules:
optional: “true”
path: /usr/lib/modules/3.10.0-957.el7.x86_64
source: /usr/lib/modules/3.10.0-957.el7.x86_64
type: disk
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

[root@lxlc-bwil02 ~]# cat /proc/self/cgroup
11:blkio:/lxc.payload.lxlc-bwil02
10:perf_event:/lxc.payload.lxlc-bwil02
9:hugetlb:/lxc.payload.lxlc-bwil02
8:memory:/lxc.payload.lxlc-bwil02
7:cpuacct,cpu:/lxc.payload.lxlc-bwil02
6:cpuset:/lxc.payload.lxlc-bwil02
5:freezer:/lxc.payload.lxlc-bwil02
4:pids:/lxc.payload.lxlc-bwil02
3:devices:/lxc.payload.lxlc-bwil02
2:net_prio,net_cls:/lxc.payload.lxlc-bwil02
1:name=systemd:/lxc.payload.lxlc-bwil02

Thoughts?

Let me know if more info is needed. Thanks!

2

Can you show lxc info in both the host and the container where you have nested containers?

3

physical host:

config:                                                                                                                                                                                                                                                                                                          [173/298]
  core.https_address: '[::]:8443'
  core.trust_password: true
  images.auto_update_interval: "0"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- container_mount_propagation                                                                                                                                                                                                                                                                                     [87/298]
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- firewall_driver                                                                                                                                                                                                                                                                                                  [1/298]
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 10.224.64.89:8443
  - 192.168.122.1:8443
  architectures:
  - x86_64
  - i686
  certificate: redacted
  certificate_fingerprint: redacted
  driver: lxc
  driver_version: 4.0.4
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    seccomp_listener_continue: "false"
    shiftfs: "false"
    uevent_injection: "false"
    unpriv_fscaps: "true"
  kernel_version: 3.10.0-957.el7.x86_64
  lxc_features:
    cgroup2: "false"
    devpts_fd: "false"
    mount_injection_file: "false"
    network_gateway_device_route: "false"
    network_ipvlan: "false"
    network_l2proxy: "false"
    network_phys_macvlan_mtu: "false"
    network_veth_router: "false"
    pidfd: "false"
    seccomp_allow_deny_syntax: "false"
    seccomp_notify: "false"
    seccomp_proxy_send_notify_fd: "false"
  os_name: Red Hat Enterprise Linux Server
  os_version: "7.6"
  project: default
  server: lxd
  server_clustered: false
  server_name: redacted
  server_pid: 9549
  server_version: "4.6"
  storage: dir
  storage_version: "1"
4

first container (running on physical):

config:
  core.https_address: '[::]:8443'
  core.trust_password: true
  images.auto_update_interval: "0"
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- nvidia_runtime                                                                                                                                                                                                                                                                                                  [89/671]
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 172.16.81.1:8443
  - '[fd42:5823:ba6e:dbc8::1]:8443'
  - 10.224.64.81:8443
  architectures:
  - x86_64
  - i686
  certificate: redacted
  certificate_fingerprint: redacted
  driver: lxc
  driver_version: 4.0.4
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "false"
    seccomp_listener: "false"
    seccomp_listener_continue: "false"
    shiftfs: "false"
    uevent_injection: "false"
    unpriv_fscaps: "true"
  kernel_version: 3.10.0-957.el7.x86_64
  lxc_features:
    cgroup2: "false"
    devpts_fd: "false"
    mount_injection_file: "false"
    network_gateway_device_route: "false"
    network_ipvlan: "false"
    network_l2proxy: "false"
    network_phys_macvlan_mtu: "false"
    network_veth_router: "false"
    pidfd: "false"
    seccomp_allow_deny_syntax: "false"
    seccomp_notify: "false"
    seccomp_proxy_send_notify_fd: "false"
  os_name: Red Hat Enterprise Linux Server
  os_version: "7.6"
  project: default
  server: lxd
  server_clustered: false
  server_name: redacted
  server_pid: 456
  server_version: "4.6"
  storage: dir
  storage_version: "1"
5

Any chance you could try with liblxc 4.0.6 or at least 4.0.5 inside the container rather than your current 4.0.4?

6

Yeah - will take a bit to package it up, but will report back. Is there a bug or compatibility issue in 4.0.4 you’re considering?

7

We’ve had a lot of work and bugfixes in both cgroup handling and pidfds in the past two releases, seeing both of those mentioned in the error log makes me think we should check whether 4.0.6 doesn’t just fix those for you already.

8

I upgraded LXC to 4.0.6 on both the physical host and the container. It’s still failing to start the sub-container with the same error.

$ lxc info --show-log foo
Name: foo
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/12 17:03 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.569 ERROR    cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3234 - Failed to find current cgroup
lxc foo 20210212170335.573 ERROR    utils - utils.c:lxc_can_use_pidfd:1834 - Kernel does not support pidfds
lxc foo 20210212170335.573 ERROR    start - start.c:lxc_spawn:1747 - Failed to setup cgroup limits for container "foo"
lxc foo 20210212170335.573 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:850 - Received container state "ABORTING" instead of "RUNNING"
lxc foo 20210212170335.574 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "foo"
lxc foo 20210212170335.574 WARN     start - start.c:lxc_abort:1024 - No such process - Failed to send SIGKILL to 2028

Also may be of note:

$ lxc start foo
Error: Common start logic: Failed to start device "eth0": open /proc/sys/net/ipv6/conf/vethd2e7dd41/disable_ipv6: read-only file system
9

That error is a bit different actually, it’s a regression introduced by 4.0.6 which we’ve seen fixed.

Can you rebuild your LXC 4.0.6 using the same list of cherry-picks that we’re using?
That will fix that one for sure:

10

We will be releasing 4.0.7 earlier than expected to address that issue in places where users didn’t cherry-pick the follow-up fixes, but we have a lot more cleanup happening now and want to see that stabilize a bit before we cut a new release.

So for now, use 4.0.6 and cherry-pick those commits on top to fix any regression it introduced.

11

Could I use stable-4.0?

12

Yep, you can build directly from stable-4.0 for a similar result.

13

Ok - the Error that came to stdout about starting eth0 is gone, but sub-container still does not start. Same messages in log.

14

Did you update liblxc on the host and rebooted the parent container?
The nested container is failing to start because of an issue with the parent container, so you need to update liblxc on both host and container and reboot the parent container.

15

Yes, updated both on physical host and parent container. Restarted the parent container.

Physical host:

LXC version 4.0.6
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-3.10.0-957.el7.x86_64
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
newuidmap is not installed
newgidmap is not installed
Network namespace: enabled
Multiple /dev/pts instances: enabled

--- Control groups ---
Cgroups: enabled

Cgroup v1 mount points:
/sys/fs/cgroup/systemd
/sys/fs/cgroup/net_cls,net_prio
/sys/fs/cgroup/devices
/sys/fs/cgroup/pids
/sys/fs/cgroup/freezer
/sys/fs/cgroup/cpuset
/sys/fs/cgroup/cpu,cpuacct
/sys/fs/cgroup/memory
/sys/fs/cgroup/hugetlb
/sys/fs/cgroup/perf_event
/sys/fs/cgroup/blkio

Cgroup v2 mount points:


Cgroup v1 clone_children flag: enabled
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_NF_NAT_IPV4: enabled, loaded
CONFIG_NF_NAT_IPV6: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities:

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /opt/bats/bin/lxc-checkconfig

Parent container:

$ lxc-checkconfig -v
LXC version 4.0.6
Kernel configuration not found at /proc/config.gz; searching...
lxc-checkconfig: unable to retrieve kernel configuration

Try recompiling with IKCONFIG_PROC, installing the kernel headers,
or specifying the kernel configuration path with:
  CONFIG=<path> lxc-checkconfig
16

And the error you’re getting is still about things being read-only?

17

If so, what’s in cat /proc/mounts in the container?

18

/var/log/lxd/lxd.log from container:

t=2021-02-12T13:57:02-0600 lvl=info msg="LXD 4.6 is starting in normal mode" path=/var/lib/lxd
t=2021-02-12T13:57:02-0600 lvl=info msg="Kernel uid/gid map:"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - u 0 0 4294967295"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - g 0 0 4294967295"
t=2021-02-12T13:57:02-0600 lvl=info msg="Configured LXD uid/gid map:"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - u 0 100000 65536"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - g 0 100000 65536"
t=2021-02-12T13:57:02-0600 lvl=warn msg="AppArmor support has been disabled because of lack of kernel support"
t=2021-02-12T13:57:02-0600 lvl=info msg="Kernel features:"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - closing multiple file descriptors efficiently: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - netnsid-based network retrieval: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - pidfds: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - uevent injection: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - seccomp listener: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - seccomp listener continue syscalls: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - seccomp listener add file descriptors: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - attach to namespaces via pidfds: no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - safe native terminal allocation : no"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - unprivileged file capabilities: yes"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - cgroup layout: legacy"
t=2021-02-12T13:57:02-0600 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored"
t=2021-02-12T13:57:02-0600 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
t=2021-02-12T13:57:02-0600 lvl=info msg=" - shiftfs support: no"
t=2021-02-12T13:57:02-0600 lvl=info msg="Initializing local database"
t=2021-02-12T13:57:03-0600 lvl=info msg="Starting /dev/lxd handler:"
t=2021-02-12T13:57:03-0600 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2021-02-12T13:57:03-0600 lvl=info msg="REST API daemon:"
t=2021-02-12T13:57:03-0600 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/lib/lxd/unix.socket
t=2021-02-12T13:57:03-0600 lvl=info msg=" - binding TCP socket" socket=[::]:8443
t=2021-02-12T13:57:03-0600 lvl=info msg="Initializing global database"
t=2021-02-12T13:57:03-0600 lvl=info msg="Firewall loaded driver \"xtables\""
t=2021-02-12T13:57:03-0600 lvl=info msg="Initializing storage pools"
t=2021-02-12T13:57:03-0600 lvl=info msg="Initializing daemon storage mounts"
t=2021-02-12T13:57:03-0600 lvl=info msg="Initializing networks"
t=2021-02-12T13:57:03-0600 lvl=warn msg="Skipping AppArmor for dnsmasq due to raw.dnsmasq being set" driver=bridge name=lxdnet0 network=lxdnet0 project=default
t=2021-02-12T13:57:03-0600 lvl=info msg="Pruning leftover image files"
t=2021-02-12T13:57:03-0600 lvl=info msg="Done pruning leftover image files"
t=2021-02-12T13:57:03-0600 lvl=info msg="Loading daemon configuration"
t=2021-02-12T13:57:03-0600 lvl=info msg="Pruning expired images"
t=2021-02-12T13:57:03-0600 lvl=info msg="Done pruning expired images"
t=2021-02-12T13:57:03-0600 lvl=info msg="Pruning expired instance backups"
t=2021-02-12T13:57:03-0600 lvl=info msg="Done pruning expired instance backups"
t=2021-02-12T13:57:03-0600 lvl=info msg="Expiring log files"
t=2021-02-12T13:57:03-0600 lvl=info msg="Updating instance types"
t=2021-02-12T13:57:03-0600 lvl=info msg="Done expiring log files"
t=2021-02-12T13:57:03-0600 lvl=info msg="Done updating instance types"
t=2021-02-12T13:57:03-0600 lvl=eror msg="Error reading host's cpuset.cpus"
t=2021-02-12T13:57:11-0600 lvl=info msg="Starting container" action=start created=2021-02-12T11:03:03-0600 ephemeral=false name=foo project=default stateful=false used=2021-02-12T13:33:53-0600
t=2021-02-12T13:57:11-0600 lvl=eror msg="Error reading host's cpuset.cpus"
t=2021-02-12T13:57:11-0600 lvl=eror msg="Failed to stop device 'libmodules': remove /var/lib/lxd/devices/foo/disk.libmodules.usr-lib-modules-3.10.0--957.el7.x86_64: device or resource busy"
t=2021-02-12T13:57:11-0600 lvl=eror msg="Failed starting container" action=start created=2021-02-12T11:03:03-0600 ephemeral=false name=foo project=default stateful=false used=2021-02-12T13:33:53-0600
t=2021-02-12T13:57:11-0600 lvl=info msg="Container initiated stop" action=stop created=2021-02-12T11:03:03-0600 ephemeral=false name=foo project=default stateful=false used=2021-02-12T13:57:11-0600
t=2021-02-12T13:57:11-0600 lvl=info msg="Shut down container" action=stop created=2021-02-12T11:03:03-0600 ephemeral=false name=foo project=default stateful=false used=2021-02-12T13:57:11-0600
t=2021-02-12T13:57:11-0600 lvl=eror msg="Error reading host's cpuset.cpus"

log from failed sub-container:

$ lxc info --show-log foo
Name: foo
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/02/12 17:03 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.454 WARN     cgfsng - cgroups/cgfsng.c:cg_hybrid_init:3250 - Failed to find current cgroup
lxc foo 20210212195711.458 ERROR    utils - utils.c:lxc_can_use_pidfd:1905 - Kernel does not support pidfds
lxc foo 20210212195711.458 ERROR    start - start.c:lxc_spawn:1741 - Failed to setup cgroup limits for container "foo"
lxc foo 20210212195711.458 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:860 - Received container state "ABORTING" instead of "RUNNING"
lxc foo 20210212195711.459 ERROR    start - start.c:__lxc_start:1999 - Failed to spawn container "foo"
lxc foo 20210212195711.459 WARN     start - start.c:lxc_abort:1018 - No such process - Failed to send SIGKILL to 1405

/proc/mounts on container:

$ cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/sda2 / ext4 rw,relatime,stripe=64,data=ordered 0 0
none /dev tmpfs rw,relatime,size=492k,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys/net proc rw,nosuid,nodev,noexec,relatime 0 0
proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
sysfs /sys/devices/virtual/net sysfs rw,relatime 0 0
sysfs /sys/devices/virtual/net sysfs rw,nosuid,nodev,noexec,relatime 0 0
mqueue /dev/mqueue mqueue rw,relatime 0 0
devtmpfs /dev/fuse devtmpfs rw,nosuid,size=49382244k,nr_inodes=12345561,mode=755 0 0
devtmpfs /dev/net/tun devtmpfs rw,nosuid,size=49382244k,nr_inodes=12345561,mode=755 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
proc /dev/.lxc/proc proc rw,relatime 0 0
sys /dev/.lxc/sys sysfs rw,relatime 0 0
tmpfs /dev/lxd tmpfs rw,relatime,size=100k,mode=755 0 0
tmpfs /dev/.lxd-mounts tmpfs rw,relatime,size=100k,mode=711 0 0
/dev/sda2 /usr/lib/modules/3.10.0-957.el7.x86_64 ext4 rw,relatime,stripe=64,data=ordered 0 0
none /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,size=10240k,mode=755 0 0
none /sys/fs/cgroup/systemd tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/systemd/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd 0 0
none /sys/fs/cgroup/net_cls,net_prio tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/net_cls,net_prio/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,net_prio,net_cls 0 0
none /sys/fs/cgroup/devices tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/devices/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0
none /sys/fs/cgroup/pids tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/pids/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0
none /sys/fs/cgroup/freezer tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/freezer/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0
none /sys/fs/cgroup/cpuset tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/cpuset/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0
none /sys/fs/cgroup/cpu,cpuacct tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/cpu,cpuacct/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,cpuacct,cpu 0 0
none /sys/fs/cgroup/memory tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/memory/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0
none /sys/fs/cgroup/hugetlb tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/hugetlb/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0
none /sys/fs/cgroup/perf_event tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/perf_event/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0
none /sys/fs/cgroup/blkio tmpfs ro,nosuid,nodev,noexec,relatime,size=10240k,mode=755 0 0
cgroup /sys/fs/cgroup/blkio/lxc.payload.lxlc-bwil02 cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0
devpts /dev/console devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
none /proc/sys/kernel/random/boot_id tmpfs ro,nosuid,nodev,noexec,relatime,size=492k,mode=755 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024 0 0
devpts /dev/ptmx devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666,max=1024 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
tmpfs /run tmpfs rw,nosuid,nodev,mode=755 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0
tmpfs /var/lib/lxd/shmounts tmpfs rw,relatime,size=100k,mode=711 0 0
tmpfs /var/lib/lxd/devlxd tmpfs rw,relatime,size=100k,mode=755 0 0
tmpfs /run/user/1515800089 tmpfs rw,nosuid,nodev,relatime,size=9879252k,mode=700,uid=1515800089,gid=1515800089 0 0
19

Are you sure you’re running a liblxc built from stable-4.0?
I’m seeing the double mounted /proc/sys in there so that looks more like vanilla 4.0.6 to me.

@brauner am I reading this right?

20

Yes - this error is now gone with lxc built off stable-4.0