Starting container fails

Today I installed LXD on my Lenovo ThinkPad T530 running Voidlinux. However, trying to start containers fails:

> lxc launch images:voidlinux test
Error: Failed to run: /usr/bin/lxd forkstart test /var/lib/lxd/containers /var/log/lxd/test/lxc.conf:
Try lxc info --show-log local:test for more info

> lxc info --show-log local:test
Name: test
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/06/28 12:53 UTC
Status: Stopped
Type: container
Profiles: default

Log:
lxc test 20210628125337.487 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:979 - Set process title to [lxc monitor] /var/lib/lxd/containers test
lxc test 20210628125337.488 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 4
lxc test 20210628125337.488 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 5
lxc test 20210628125337.488 INFO start - start.c:lxc_check_inherited:286 - Closed inherited fd 6
lxc test 20210628125337.488 INFO lsm - lsm/lsm.c:lsm_init:40 - Initialized LSM security driver nop
lxc test 20210628125337.488 INFO conf - conf.c:run_script_argv:333 - Executing script “/proc/762/exe callhook /var/lib/lxd “default” “test” start” for container “test”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “[all]”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “reject_force_umount # comment this to allow umount -f; not recommended”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “[all]”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “kexec_load errno 38”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327718:errno] arch[0]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327718:errno] arch[1073741827]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327718:errno] arch[1073741886]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “open_by_handle_at errno 38”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327718:errno] arch[0]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327718:errno] arch[1073741827]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327718:errno] arch[1073741886]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “init_module errno 38”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327718:errno] arch[0]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327718:errno] arch[1073741827]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327718:errno] arch[1073741886]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “finit_module errno 38”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327718:errno] arch[0]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327718:errno] arch[1073741827]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327718:errno] arch[1073741886]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing “delete_module errno 38”
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327718:errno] arch[0]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327718:errno] arch[1073741827]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327718:errno] arch[1073741886]
lxc test 20210628125337.679 INFO seccomp - seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
lxc test 20210628125337.679 INFO start - start.c:lxc_init:837 - Container “test” is initialized
lxc test 20210628125337.685 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.monitor.test”
lxc test 20210628125337.686 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1368 - The monitor process uses “lxc.monitor.test” as cgroup
lxc test 20210628125337.687 WARN cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1152 - File exists - Failed to create directory “/sys/fs/cgroup/cpuset//lxc.payload.test”
lxc test 20210628125337.688 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1471 - The container process uses “lxc.payload.test” as cgroup
lxc test 20210628125337.689 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUSER
lxc test 20210628125337.689 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc test 20210628125337.689 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc test 20210628125337.689 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc test 20210628125337.689 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc test 20210628125337.697 INFO start - start.c:do_start:1085 - Unshared CLONE_NEWNET
lxc test 20210628125337.713 NOTICE utils - utils.c:lxc_setgroups:1420 - Dropped additional groups
lxc test 20210628125337.713 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1570 - No such file or directory - Failed to fchownat(29, memory.oom.group, 65536, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc test 20210628125337.726 NOTICE utils - utils.c:lxc_setgroups:1420 - Dropped additional groups
lxc test 20210628125337.726 NOTICE utils - utils.c:lxc_switch_uid_gid:1398 - Switched to gid 0
lxc test 20210628125337.726 NOTICE utils - utils.c:lxc_switch_uid_gid:1407 - Switched to uid 0
lxc test 20210628125337.726 INFO start - start.c:do_start:1198 - Unshared CLONE_NEWCGROUP
lxc test 20210628125337.727 ERROR conf - conf.c:lxc_mount_rootfs:1245 - No such file or directory - Failed to access to “/var/lxc/containers”. Check it is present
lxc test 20210628125337.727 ERROR conf - conf.c:lxc_setup_rootfs_prepare_root:3142 - Failed to setup rootfs for
lxc test 20210628125337.727 ERROR conf - conf.c:lxc_setup:3278 - Failed to setup rootfs
lxc test 20210628125337.727 ERROR start - start.c:do_start:1218 - Failed to setup container “test”
lxc test 20210628125337.728 ERROR sync - sync.c:__sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc test 20210628125337.731 WARN network - network.c:lxc_delete_network_priv:3183 - Failed to rename interface with index 0 from “eth0” to its initial name “veth0ffdf889”
lxc test 20210628125337.731 ERROR start - start.c:__lxc_start:1999 - Failed to spawn container “test”
lxc test 20210628125337.731 WARN start - start.c:lxc_abort:1012 - No such process - Failed to send SIGKILL via pidfd 31 for process 6090
lxc test 20210628125337.731 INFO conf - conf.c:run_script_argv:333 - Executing script “/usr/bin/lxd callhook /var/lib/lxd “default” “test” stopns” for container “test”
lxc test 20210628125337.732 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:859 - Received container state “ABORTING” instead of “RUNNING”
lxc test 20210628125337.894 NOTICE utils - utils.c:lxc_setgroups:1420 - Dropped additional groups
lxc 20210628125337.903 WARN commands - commands.c:lxc_cmd_rsp_recv:124 - Connection reset by peer - Failed to receive response for command “get_state”
lxc test 20210628125337.903 INFO conf - conf.c:run_script_argv:333 - Executing script “/usr/bin/lxd callhook /var/lib/lxd “default” “test” stop” for container “test”

Same thing when trying to start an Ubuntu container. Please let me know, if you need any further information.

 lxc test 20210628125337.727 ERROR conf - conf.c:lxc_mount_rootfs:1245 - No such file or directory - Failed to access to “/var/lxc/containers”. Check it is present

This is a very weird path as LXD should never use that…

Can you show /var/log/lxd/test/lxc.conf?

lxc.log.file = /var/log/lxd/test/lxc.log
lxc.log.level = info
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/log/lxd/test/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.include = /usr/share/lxc/config/common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/762/exe callhook /var/lib/lxd “default” “test” start
lxc.hook.stop = /usr/bin/lxd callhook /var/lib/lxd “default” “test” stopns
lxc.hook.post-stop = /usr/bin/lxd callhook /var/lib/lxd “default” “test” stop
lxc.tty.max = 0
lxc.uts.name = test
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/test
lxc.idmap = u 0 1000000 65536
lxc.idmap = g 0 1000000 65536
lxc.mount.auto = shmounts:/var/lib/lxd/shmounts/test:/dev/.lxd-mounts
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = veth0ffdf889
lxc.rootfs.path = dir:/var/lib/lxd/containers/test/rootfs

What’s in /usr/share/lxc/config/common.conf.d/ and in /etc/lxc/ ?

I’m trying to figure out where this /var/lxc/containers path comes from as that’s very non-standard.

/usr/share/lxc/config/common.conf.d/ just contains a README file.

/etc/lxc does not exist

Maybe I messed something up when I had legacy LXC installed a few years ago. But the package is long gone from my system already. Other than that there is only docker running for testing my builds.

What’s the LXC and LXD version in use here?

LXD is 4.15
LXC I don’t remember, but I think it was something like 4.0 or something

Can you show lxc info? That would show all the relevant versions.

Thank you for your patience. Really appreciate it :slight_smile:

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
   ...
    -----END CERTIFICATE-----
  certificate_fingerprint: f685f13d79455d4d0c990d91ee5643fce09b8747ccf9c90382e117493ec2e712
  driver: lxc | qemu
  driver_version: 4.0.6 | 6.0.0
  firewall: xtables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.8.17_1
  lxc_features:
    cgroup2: "true"
    devpts_fd: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: void
  os_version: ""
  project: default
  server: lxd
  server_clustered: false
  server_name: ares
  server_pid: 762
  server_version: "4.15"
  storage: btrfs
  storage_version: 5.11.1

Any ideas?

@brauner anything that comes to mind?

This isn’t the actual rootfs path but rather where the rootfs will be mounted to and where we pivot_root() into. So LXC seems to think that /var/lxc/containers is where to mount the rootfs to whereas on Ubuntu this would be /usr/lib/x86_64-linux-gnu/lxc . So this might just be Voidlinux specific? In this case just a mkdir /var/lxc/containers might be enough to fix this.

Indeed I manually created that directory yesterday and everything seems to work now. Weirdly enough my other Void system does not show that behavior so maybe it was just specific to this machine.

Thank you @stgraber for your efforts.

Oh, that’s very weird, I don’t know why void would use such a weird mountpoint, especially if it’s not guaranteed to exist…