LXD does't want to boot, permission problem

Hi

I attempt to start the container and it just fails to start:lxc info --show-log test001

Name: test001
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/03/24 11:21 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc test001 20200324115103.494 WARN cgfsng - cgroups/cgfsng.c:chowmod:1525 - No such file or directory - Failed to chown(/sys/fs/cgroup/unified//lxc.payload/test001/memory.oom.group, 65536, 0)
lxc test001 20200324115103.512 ERROR conf - conf.c:mount_autodev:1158 - Permission denied - Failed to create “/dev” directory
lxc test001 20200324115103.512 ERROR conf - conf.c:lxc_setup:3584 - Failed to mount “/dev”
lxc test001 20200324115103.512 ERROR start - start.c:do_start:1321 - Failed to setup container “test001”
lxc test001 20200324115103.515 ERROR sync - sync.c:__sync_wait:61 - An error occurred in another process (expected sequence number 5)
lxc test001 20200324115103.515 WARN network - network.c:lxc_delete_network_priv:3372 - Failed to rename interface with index 10 from “eth0” to its initial name “vethb7f6aa4e”
lxc test001 20200324115103.515 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:872 - Received container state “ABORTING” instead of “RUNNING”
lxc test001 20200324115103.515 ERROR start - start.c:__lxc_start:2036 - Failed to spawn container “test001”
lxc 20200324115103.778 WARN commands - commands.c:lxc_cmd_rsp_recv:134 - Connection reset by peer - Failed to receive response for command “get_state”

The debug log

lxc start --debug test001
lxc start --debug test001
DBUG[03-24|19:51:00] Connecting to a local LXD over a Unix socket
DBUG[03-24|19:51:00] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
DBUG[03-24|19:51:00] Got response struct from LXD
DBUG[03-24|19:51:00]
{
“config”: {},
“api_extensions”: [
“storage_zfs_remove_snapshots”,
“container_host_shutdown_timeout”,
“container_stop_priority”,
“container_syscall_filtering”,
“auth_pki”,
“container_last_used_at”,
“etag”,
“patch”,
“usb_devices”,
“https_allowed_credentials”,
“image_compression_algorithm”,
“directory_manipulation”,
“container_cpu_time”,
“storage_zfs_use_refquota”,
“storage_lvm_mount_options”,
“network”,
“profile_usedby”,
“container_push”,
“container_exec_recording”,
“certificate_update”,
“container_exec_signal_handling”,
“gpu_devices”,
“container_image_properties”,
“migration_progress”,
“id_map”,
“network_firewall_filtering”,
“network_routes”,
“storage”,
“file_delete”,
“file_append”,
“network_dhcp_expiry”,
“storage_lvm_vg_rename”,
“storage_lvm_thinpool_rename”,
“network_vlan”,
“image_create_aliases”,
“container_stateless_copy”,
“container_only_migration”,
“storage_zfs_clone_copy”,
“unix_device_rename”,
“storage_lvm_use_thinpool”,
“storage_rsync_bwlimit”,
“network_vxlan_interface”,
“storage_btrfs_mount_options”,
“entity_description”,
“image_force_refresh”,
“storage_lvm_lv_resizing”,
“id_map_base”,
“file_symlinks”,
“container_push_target”,
“network_vlan_physical”,
“storage_images_delete”,
“container_edit_metadata”,
“container_snapshot_stateful_migration”,
“storage_driver_ceph”,
“storage_ceph_user_name”,
“resource_limits”,
“storage_volatile_initial_source”,
“storage_ceph_force_osd_reuse”,
“storage_block_filesystem_btrfs”,
“resources”,
“kernel_limits”,
“storage_api_volume_rename”,
“macaroon_authentication”,
“network_sriov”,
“console”,
“restrict_devlxd”,
“migration_pre_copy”,
“infiniband”,
“maas_network”,
“devlxd_events”,
“proxy”,
“network_dhcp_gateway”,
“file_get_symlink”,
“network_leases”,
“unix_device_hotplug”,
“storage_api_local_volume_handling”,
“operation_description”,
“clustering”,
“event_lifecycle”,
“storage_api_remote_volume_handling”,
“nvidia_runtime”,
“container_mount_propagation”,
“container_backup”,
“devlxd_images”,
“container_local_cross_pool_handling”,
“proxy_unix”,
“proxy_udp”,
“clustering_join”,
“proxy_tcp_udp_multi_port_handling”,
“network_state”,
“proxy_unix_dac_properties”,
“container_protection_delete”,
“unix_priv_drop”,
“pprof_http”,
“proxy_haproxy_protocol”,
“network_hwaddr”,
“proxy_nat”,
“network_nat_order”,
“container_full”,
“candid_authentication”,
“backup_compression”,
“candid_config”,
“nvidia_runtime_config”,
“storage_api_volume_snapshots”,
“storage_unmapped”,
“projects”,
“candid_config_key”,
“network_vxlan_ttl”,
“container_incremental_copy”,
“usb_optional_vendorid”,
“snapshot_scheduling”,
“container_copy_project”,
“clustering_server_address”,
“clustering_image_replication”,
“container_protection_shift”,
“snapshot_expiry”,
“container_backup_override_pool”,
“snapshot_expiry_creation”,
“network_leases_location”,
“resources_cpu_socket”,
“resources_gpu”,
“resources_numa”,
“kernel_features”,
“id_map_current”,
“event_location”,
“storage_api_remote_volume_snapshots”,
“network_nat_address”,
“container_nic_routes”,
“rbac”,
“cluster_internal_copy”,
“seccomp_notify”,
“lxc_features”,
“container_nic_ipvlan”,
“network_vlan_sriov”,
“storage_cephfs”,
“container_nic_ipfilter”,
“resources_v2”,
“container_exec_user_group_cwd”,
“container_syscall_intercept”,
“container_disk_shift”,
“storage_shifted”,
“resources_infiniband”,
“daemon_storage”,
“instances”,
“image_types”,
“resources_disk_sata”,
“clustering_roles”,
“images_expiry”,
“resources_network_firmware”,
“backup_compression_algorithm”,
“ceph_data_pool_name”,
“container_syscall_intercept_mount”,
“compression_squashfs”,
“container_raw_mount”,
“container_nic_routed”,
“container_syscall_intercept_mount_fuse”,
“container_disk_ceph”,
“virtual-machines”,
“image_profiles”,
“clustering_architecture”,
“resources_disk_id”,
“storage_lvm_stripes”,
“vm_boot_priority”,
“unix_hotplug_devices”,
“api_filtering”,
“instance_nic_network”,
“clustering_sizing”
],
“api_status”: “stable”,
“api_version”: “1.0”,
“auth”: “trusted”,
“public”: false,
“auth_methods”: [
“tls”
],
“environment”: {
“addresses”: ,
“architectures”: [
“x86_64”,
“i686”
],
“certificate”: “-----BEGIN CERTIFICATE-----\nMIICJjCCAaugAwIBAgIRAMkqtxxC4aJiUPzC66I1ErgwCgYIKoZIzj0EAwMwQzEc\nMBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEjMCEGA1UEAwwacm9vdEBsb2Nh\nbGhvc3QubG9jYWxkb21haW4wHhcNMTkwNjEwMjMzNTI4WhcNMjkwNjA3MjMzNTI4\nWjBDMRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMSMwIQYDVQQDDBpyb290\nQGxvY2FsaG9zdC5sb2NhbGRvbWFpbjB2MBAGByqGSM49AgEGBSuBBAAiA2IABLg+\n2UlNfDyvOQRFW5hnBUPmqO1Nu043lkXqBNkrDASL64D362gwd6AOP3ahKbSsZEft\nJnwx85c8yFTzCmYZg/VwYc9MpsNSnsEknH/AZJg+XTBN68umawLpISinzwZ2o6Nj\nMGEwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB\n/wQCMAAwLAYDVR0RBCUwI4IVbG9jYWxob3N0LmxvY2FsZG9tYWluhwTAqAoRhwTA\nqHoBMAoGCCqGSM49BAMDA2kAMGYCMQDJJS3I8HogCC/4LsLqrNDeBofDFnfipEEK\nqp6MEt7e+2PFdampjAs4R1LzOV0KQ/gCMQCdiMBrQP1OVtnonjHsSchbBXtljLVe\niUuxE6+WuD40037S7LVuPhWZP6t0TuhKvCk=\n-----END CERTIFICATE-----\n”,
“certificate_fingerprint”: “1f5be0b01e220edee163326101f95b077c3f5ab481aeb762653a2af5cc8b04ab”,
“driver”: “lxc”,
“driver_version”: “3.2.1”,
“kernel”: “Linux”,
“kernel_architecture”: “x86_64”,
“kernel_features”: {
“netnsid_getifaddrs”: “true”,
“seccomp_listener”: “true”,
“seccomp_listener_continue”: “true”,
“shiftfs”: “false”,
“uevent_injection”: “true”,
“unpriv_fscaps”: “true”
},
“kernel_version”: “5.5.10-200.fc31.x86_64”,
“lxc_features”: {
“cgroup2”: “false”,
“mount_injection_file”: “true”,
“network_gateway_device_route”: “true”,
“network_ipvlan”: “true”,
“network_l2proxy”: “true”,
“network_phys_macvlan_mtu”: “true”,
“network_veth_router”: “true”,
“seccomp_notify”: “true”
},
“project”: “default”,
“server”: “lxd”,
“server_clustered”: false,
“server_name”: “localhost.localdomain”,
“server_pid”: 3718,
“server_version”: “3.21”,
“storage”: “dir”,
“storage_version”: “1”
}
}
DBUG[03-24|19:51:00] Sending request to LXD method=GET url=http://unix.socket/1.0/instances/test001 etag=
DBUG[03-24|19:51:00] Got response struct from LXD
DBUG[03-24|19:51:00]
{
“architecture”: “x86_64”,
“config”: {
“image.architecture”: “amd64”,
“volatile.eth0.hwaddr”: “00:16:3e:44:c1:c7”,
“volatile.idmap.base”: “0”,
“volatile.idmap.current”: “[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]”,
“volatile.idmap.next”: “[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]”,
“volatile.last_state.idmap”: “[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]”,
“volatile.last_state.power”: “STOPPED”
},
“devices”: {},
“ephemeral”: false,
“profiles”: [
“default”
],
“stateful”: false,
“description”: “”,
“created_at”: “2020-03-24T19:21:05.850829247+08:00”,
“expanded_config”: {
“image.architecture”: “amd64”,
“volatile.eth0.hwaddr”: “00:16:3e:44:c1:c7”,
“volatile.idmap.base”: “0”,
“volatile.idmap.current”: “[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]”,
“volatile.idmap.next”: “[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]”,
“volatile.last_state.idmap”: “[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]”,
“volatile.last_state.power”: “STOPPED”
},
“expanded_devices”: {
“eth0”: {
“name”: “eth0”,
“nictype”: “bridged”,
“parent”: “lxdbr0”,
“type”: “nic”
},
“root”: {
“path”: “/”,
“pool”: “default”,
“type”: “disk”
}
},
“name”: “test001”,
“status”: “Stopped”,
“status_code”: 102,
“last_used_at”: “2020-03-24T19:41:06.826931765+08:00”,
“location”: “none”,
“type”: “container”
}
DBUG[03-24|19:51:00] Connected to the websocket: ws://unix.socket/1.0/events
DBUG[03-24|19:51:00] Sending request to LXD method=PUT url=http://unix.socket/1.0/instances/test001/state etag=
DBUG[03-24|19:51:00]
{
“action”: “start”,
“timeout”: 0,
“force”: false,
“stateful”: false
}
DBUG[03-24|19:51:01] Got operation from LXD
DBUG[03-24|19:51:01]
{
“id”: “21c0cf6c-a64b-4917-9a33-8137061c5209”,
“class”: “task”,
“description”: “Starting container”,
“created_at”: “2020-03-24T19:51:00.890441903+08:00”,
“updated_at”: “2020-03-24T19:51:00.890441903+08:00”,
“status”: “Running”,
“status_code”: 103,
“resources”: {
“containers”: [
“/1.0/containers/test001”
]
},
“metadata”: null,
“may_cancel”: false,
“err”: “”,
“location”: “none”
}
DBUG[03-24|19:51:01] Sending request to LXD method=GET url=http://unix.socket/1.0/operations/21c0cf6c-a64b-4917-9a33-8137061c5209 etag=
DBUG[03-24|19:51:01] Got response struct from LXD
DBUG[03-24|19:51:01]
{
“id”: “21c0cf6c-a64b-4917-9a33-8137061c5209”,
“class”: “task”,
“description”: “Starting container”,
“created_at”: “2020-03-24T19:51:00.890441903+08:00”,
“updated_at”: “2020-03-24T19:51:00.890441903+08:00”,
“status”: “Running”,
“status_code”: 103,
“resources”: {
“containers”: [
“/1.0/containers/test001”
]
},
“metadata”: null,
“may_cancel”: false,
“err”: “”,
“location”: “none”
}
Error: Failed to run: /usr/bin/lxd forkstart test001 /var/lib/lxd/containers /var/log/lxd/test001/lxc.conf:
Try lxc info --show-log test001 for more info

I have subuid and suigid files are of these values:

root:100000:65536
lxd:100000:65536
root:1000:1

Using LXD version 3.21

Hmm, that’s a bit of a weird one.
It suggests that your container’s rootfs may not be reachable or be reachable but without a dev directory in it.

Can you show lxc config show --expanded NAME for that container as well as lxc storage list and the content of /var/snap/lxd/common/lxd/logs/lxd.log?

lxc config show --expanded test001

architecture: x86_64
config:
image.architecture: amd64
image.description: Ubuntu eoan amd64 (20200323_07:42)
image.os: Ubuntu
image.release: eoan
image.serial: “20200323_07:42”
image.type: squashfs
volatile.eth0.hwaddr: 00:16:3e:44:c1:c7
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:100000,“Nsid”:0,“Maprange”:65536},{“Isuid”:false,“Isgid”:true,“Hostid”:100000,“Nsid”:0,“Maprange”:65536}]’
volatile.last_state.power: STOPPED
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: default
type: disk
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

lxc storage list

+---------+-------------+--------+------------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |               SOURCE               | USED BY |
+---------+-------------+--------+------------------------------------+---------+
| default |             | dir    | /var/lib/lxd/storage-pools/default | 2       |
+---------+-------------+--------+------------------------------------+---------+

lxd.log

t=2020-03-24T19:50:33+0800 lvl=info msg=“LXD 3.21 is starting in normal mode” path=/var/lib/lxd
t=2020-03-24T19:50:33+0800 lvl=info msg=“Kernel uid/gid map:”
t=2020-03-24T19:50:33+0800 lvl=info msg=" - u 0 0 4294967295"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - g 0 0 4294967295"
t=2020-03-24T19:50:33+0800 lvl=info msg=“Configured LXD uid/gid map:”
t=2020-03-24T19:50:33+0800 lvl=info msg=" - u 0 100000 65536"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - g 0 100000 65536"
t=2020-03-24T19:50:33+0800 lvl=warn msg=“AppArmor support has been disabled because of lack of kernel support”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Kernel features:”
t=2020-03-24T19:50:33+0800 lvl=info msg=" - netnsid-based network retrieval: yes"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - uevent injection: yes"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - seccomp listener: yes"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - seccomp listener continue syscalls: yes"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - unprivileged file capabilities: yes"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - cgroup layout: hybrid"
t=2020-03-24T19:50:33+0800 lvl=warn msg=" - Couldn’t find the CGroup blkio.weight, I/O weight limits will be ignored"
t=2020-03-24T19:50:33+0800 lvl=info msg=" - shiftfs support: no"
t=2020-03-24T19:50:33+0800 lvl=info msg=“Initializing local database”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Starting /dev/lxd handler:”
t=2020-03-24T19:50:33+0800 lvl=info msg=" - binding devlxd socket" socket=/var/lib/lxd/devlxd/sock
t=2020-03-24T19:50:33+0800 lvl=info msg=“REST API daemon:”
t=2020-03-24T19:50:33+0800 lvl=info msg=" - binding Unix socket" inherited=true socket=/run/lxd.socket
t=2020-03-24T19:50:33+0800 lvl=info msg=“Initializing global database”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Initializing storage pools”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Initializing networks”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Pruning leftover image files”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Done pruning leftover image files”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Loading daemon configuration”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Started seccomp handler” path=/var/lib/lxd/seccomp.socket
t=2020-03-24T19:50:33+0800 lvl=info msg=“Pruning expired images”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Done pruning expired images”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Pruning expired container backups”
t=2020-03-24T19:50:33+0800 lvl=info msg=“Done pruning expired container backups”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Updating instance types”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Done updating instance types”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Expiring log files”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Done expiring log files”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Updating images”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Done updating images”
t=2020-03-24T19:50:34+0800 lvl=info msg=“Downloading image” alias=ubuntu/19.10/amd64 server=https://images.linuxcontainers.org
t=2020-03-24T19:51:02+0800 lvl=info msg=“Starting container” action=start created=2020-03-24T19:21:05+0800 ephemeral=false name=test001 project=default stateful=false used=2020-03-24T19:41:06+0800
t=2020-03-24T19:51:03+0800 lvl=eror msg=“Failed starting container” action=start created=2020-03-24T19:21:05+0800 ephemeral=false name=test001 project=default stateful=false used=2020-03-24T19:41:06+0800
t=2020-03-24T19:51:04+0800 lvl=info msg=“Container initiated stop” action=stop created=2020-03-24T19:21:05+0800 ephemeral=false name=test001 project=default stateful=false used=2020-03-24T19:51:03+0800
t=2020-03-24T19:51:07+0800 lvl=info msg=“Image downloaded” alias=ubuntu/19.10/amd64 server=https://images.linuxcontainers.org
t=2020-03-24T20:50:34+0800 lvl=info msg=“Pruning expired container backups”
t=2020-03-24T20:50:34+0800 lvl=info msg=“Done pruning expired container backups”
t=2020-03-24T21:36:18+0800 lvl=info msg=“Starting container” action=start created=2020-03-24T19:21:05+0800 ephemeral=false name=test001 project=default stateful=false used=2020-03-24T19:51:03+0800
t=2020-03-24T21:36:18+0800 lvl=eror msg=“Failed starting container” action=start created=2020-03-24T19:21:05+0800 ephemeral=false name=test001 project=default stateful=false used=2020-03-24T19:51:03+0800
t=2020-03-24T21:36:18+0800 lvl=info msg=“Container initiated stop” action=stop created=2020-03-24T19:21:05+0800 ephemeral=false name=test001 project=default stateful=false used=2020-03-24T21:36:18+0800
t=2020-03-24T21:50:34+0800 lvl=info msg=“Pruning expired container backups”
t=2020-03-24T21:50:34+0800 lvl=info msg=“Done pruning expired container backups”
t=2020-03-24T22:50:34+0800 lvl=info msg=“Pruning expired container backups”
t=2020-03-24T22:50:34+0800 lvl=info msg=“Done pruning expired container backups”

Can you show ls -lh /var/lib/lxd/containers/test001/rootfs?

sudo ls -lh /var/lib/lxd/containers/test001/rootfs

total 60K
lrwxrwxrwx. 1 100000 100000 7 Mar 24 18:52 bin → usr/bin
drwxr-xr-x. 2 100000 100000 4.0K Oct 14 22:35 boot
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:50 dev
drwxr-xr-x. 61 100000 100000 4.0K Mar 24 18:52 etc
drwxr-xr-x. 3 100000 100000 4.0K Mar 23 15:44 home
lrwxrwxrwx. 1 100000 100000 7 Mar 24 18:52 lib → usr/lib
lrwxrwxrwx. 1 100000 100000 9 Mar 24 18:52 lib32 → usr/lib32
lrwxrwxrwx. 1 100000 100000 9 Mar 24 18:52 lib64 → usr/lib64
lrwxrwxrwx. 1 100000 100000 10 Mar 24 18:52 libx32 → usr/libx32
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 media
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 mnt
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 opt
drwxr-xr-x. 2 100000 100000 4.0K Oct 14 22:35 proc
drwx------. 2 100000 100000 4.0K Mar 24 21:27 root
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:45 run
lrwxrwxrwx. 1 100000 100000 8 Mar 24 18:52 sbin → usr/sbin
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 srv
drwxr-xr-x. 2 100000 100000 4.0K Oct 14 22:35 sys
drwxrwxrwt. 7 100000 100000 4.0K Mar 24 23:00 tmp
drwxr-xr-x. 13 100000 100000 4.0K Mar 23 15:43 usr
drwxr-xr-x. 12 100000 100000 4.0K Mar 23 15:44 var

Can you show:

  • cat /var/log/lxd/test001.conf
  • stat /
  • stat /var
  • stat /var/lib
  • stat /var/lib/lxd
  • stat /var/lib/lxd/containers
  • stat /var/lib/lxd/containers/test001
  • stat /var/lib/lxd/containers/test001/rootfs

There may be bad permissions somewhere on the way causing this.

sudo cat /var/log/lxd/test001/lxc.conf

lxc.log.file = /var/log/lxd/test001/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/log/lxd/test001/console.log
lxc.mount.auto = proc:rw sys:rw cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.include = /usr/share/lxc/config/common.conf.d/
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/3718/exe callhook /var/lib/lxd 20 start
lxc.hook.stop = /usr/bin/lxd callhook /var/lib/lxd 20 stopns
lxc.hook.post-stop = /usr/bin/lxd callhook /var/lib/lxd 20 stop
lxc.tty.max = 0
lxc.uts.name = test001
lxc.mount.entry = /var/lib/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.seccomp.profile = /var/lib/lxd/security/seccomp/test001
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.mount.auto = shmounts:/var/lib/lxd/shmounts/test001:/dev/.lxd-mounts
lxc.rootfs.path = dir:/var/lib/lxd/containers/test001/rootfs
lxc.net.0.name = eth0
lxc.net.0.type = phys
lxc.net.0.flags = up
lxc.net.0.link = veth71488ac4

stat /

File: /
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 2 Links: 20
Access: (0555/dr-xr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:root_t:s0
Access: 2020-03-24 21:43:19.870811028 +0800
Modify: 2020-03-22 11:49:30.244035303 +0800
Change: 2020-03-22 11:49:30.244035303 +0800
Birth: 2019-06-11 00:53:15.000000000 +0800

stat /var

File: /var
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 57409537 Links: 21
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:var_t:s0
Access: 2020-03-24 17:20:31.517821271 +0800
Modify: 2020-03-22 16:23:38.102162221 +0800
Change: 2020-03-22 16:23:38.102162221 +0800
Birth: 2019-06-11 00:53:29.495537580 +0800

stat /var/lib

File: /var/lib
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 57409551 Links: 60
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:var_lib_t:s0
Access: 2020-03-24 17:23:54.666429054 +0800
Modify: 2020-03-24 17:23:42.871335704 +0800
Change: 2020-03-24 17:23:42.871335704 +0800
Birth: 2019-06-11 00:57:53.152962810 +0800

stat /var/lib/lxd

File: /var/lib/lxd
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 2097934 Links: 16
Access: (0711/drwx–x–x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:container_var_lib_t:s0
Access: 2020-03-25 21:36:28.919565000 +0800
Modify: 2020-03-24 19:50:33.944960158 +0800
Change: 2020-03-25 21:36:17.104545311 +0800
Birth: 2020-03-24 17:23:42.870335696 +0800

stat /var/lib/lxd/containers

File: /var/lib/lxd/containers
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fd00h/64768d Inode: 2097936 Links: 2
Access: (0711/drwx–x–x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:container_var_lib_t:s0
Access: 2020-03-25 21:36:28.927565013 +0800
Modify: 2020-03-24 19:21:09.624383676 +0800
Change: 2020-03-25 21:36:28.927565013 +0800
Birth: 2020-03-24 17:23:42.871335704 +0800

stat /var/lib/lxd/containers/test001>

File: /var/lib/lxd/containers/test001 → /var/lib/lxd/storage-pools/default/containers/test001
Size: 49 Blocks: 0 IO Block: 4096 symbolic link
Device: fd00h/64768d Inode: 2097993 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:container_var_lib_t:s0
Access: 2020-03-25 21:04:56.986414536 +0800
Modify: 2020-03-24 19:21:09.624383676 +0800
Change: 2020-03-24 19:21:09.624383676 +0800
Birth: 2020-03-24 19:21:09.624383676 +0800

sudo stat /var/lib/lxd/containers/test001/rootfs

File: /var/lib/lxd/containers/test001/rootfs
Size: 4096 Blocks: 16 IO Block: 4096 directory
Device: 40h/64d Inode: 2368336 Links: 1
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:container_var_lib_t:s0
Access: 2020-03-25 21:36:50.338600663 +0800
Modify: 2020-03-24 19:27:15.854646362 +0800
Change: 2020-03-25 21:36:45.080591909 +0800
Birth: 2020-03-24 19:22:07.713901195 +0800

All seems good to me.

The last one seems incorrect actually, can you show:

  • ls -lh /var/lib/lxd/storage-pools/default/containers
  • ls -lh /var/lib/lxd/storage-pools/default/containers/test001
  • ls -lh /var/lib/lxd/storage-pools/default/containers/test001/rootfs

sudo ls -lh /var/lib/lxd/storage-pools/default/containers

total 8K
d–x------. 4 root root 4.0K Mar 24 18:52 ubuntu
d–x------. 6 root root 4.0K Mar 24 19:41 test001

sudo ls -lh /var/lib/lxd/storage-pools/default/containers/test001

total 16K
-r--------. 1 root root 2.2K Mar 25 21:37 backup.yaml
-rw-r–r–. 1 root root 523 Mar 23 15:50 metadata.yaml
drwxr-xr-x. 1 root root 4.0K Mar 24 19:27 rootfs
drwxr-xr-x. 2 root root 4.0K Mar 23 15:50 templates

sudo ls -lh /var/lib/lxd/storage-pools/default/containers/test001/rootfs

total 64K
lrwxrwxrwx. 1 100000 100000 7 Mar 24 18:52 bin → usr/bin
drwxr-xr-x. 2 100000 100000 4.0K Oct 14 22:35 boot
drwxr-xr-x. 1 100000 100000 4.0K Mar 25 21:06 dev
drwxr-xr-x. 61 100000 100000 4.0K Mar 24 18:52 etc
drwxr-xr-x. 3 100000 100000 4.0K Mar 23 15:44 home
lrwxrwxrwx. 1 100000 100000 7 Mar 24 18:52 lib → usr/lib
lrwxrwxrwx. 1 100000 100000 9 Mar 24 18:52 lib32 → usr/lib32
lrwxrwxrwx. 1 100000 100000 9 Mar 24 18:52 lib64 → usr/lib64
lrwxrwxrwx. 1 100000 100000 10 Mar 24 18:52 libx32 → usr/libx32
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 media
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 mnt
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 opt
drwxr-xr-x. 2 100000 100000 4.0K Oct 14 22:35 proc
drwx------. 2 100000 100000 4.0K Mar 24 21:27 root
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:45 run
lrwxrwxrwx. 1 100000 100000 8 Mar 24 18:52 sbin → usr/sbin
drwxr-xr-x. 2 100000 100000 4.0K Mar 23 15:43 srv
drwxr-xr-x. 2 100000 100000 4.0K Oct 14 22:35 sys
drwxrwxrwt. 7 100000 100000 4.0K Mar 24 23:00 tmp
drwxr-xr-x. 13 100000 100000 4.0K Mar 23 15:43 usr
drwxr-xr-x. 12 100000 100000 4.0K Mar 23 15:44 var

The inode value seems odd.

chown 100000:100000 /var/lib/lxd/storage-pools/default/containers/test001
chown 100000:100000 /var/lib/lxd/storage-pools/default/containers/test001/rootfs

That should sort the current issue.

Did it, but still same the same problem.

I recreate the container once again, but now different problem/issue:

lxc test001 20200408041845.994 ERROR    dir - storage/dir.c:dir_mount:198 - Permission denied - Failed to mount "/var/lib/lxd/containers/test001/rootfs" on "/usr/lib64/lxc/rootfs"
lxc test001 20200408041845.997 ERROR    conf - conf.c:lxc_mount_rootfs:1351 - Failed to mount rootfs "/var/lib/lxd/containers/test001/rootfs" onto "/usr/lib64/lxc/rootfs" with options "(null)"

Still couldn’t understand the purpose of /usr/lib64/lxc/rootfs.