Cannot start privileged container

I am trying to create a privileged container to run an application (parrot-sphinx) inside a Ubuntu 18.04 container. The main reason behind this is because it will not install on a 20.04 system and requires 18.04.

I’m thinking I need a privileged container because of how some tools work in support of that simulator - in particular the firmwared daemon. It tries to mount some things to /var/cache and fails to do so with an “Operation Not Permitted” error from mount. If there is a better way to solve this I am all ears and would love some guidance!

So - back to the topic - when I try to create a privileged container I get the following:
$ lxc launch images:ubuntu/18.04 parrot-sphinx -p default -c security.privileged=true
Creating parrot-sphinx
Starting parrot-sphinx
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart parrot-sphinx
/var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.conf:
Try lxc info --show-log local:parrot-sphinx for more info

$ lxc info --show-log local:parrot-sphinx
Name: parrot-sphinx
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/06/30 03:49 UTC
Status: Stopped
Type: container
Profiles: default

Log:

lxc parrot-sphinx 20210630034952.353 ERROR    conf - conf.c:turn_into_dependent_mounts:3340 - No such file or directory - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc parrot-sphinx 20210630034952.429 ERROR    conf - conf.c:run_buffer:316 - Script exited with status 1
lxc parrot-sphinx 20210630034952.429 ERROR    conf - conf.c:lxc_setup:3686 - Failed to run mount hooks
lxc parrot-sphinx 20210630034952.429 ERROR    start - start.c:do_start:1265 - Failed to setup container "parrot-sphinx"
lxc parrot-sphinx 20210630034952.429 ERROR    sync - sync.c:sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc parrot-sphinx 20210630034952.435 WARN     network - network.c:lxc_delete_network_priv:3621 - Failed to rename interface with index 0 from "eth0" to its initial name "veth73187f1d"
lxc parrot-sphinx 20210630034952.435 ERROR    start - start.c:__lxc_start:2073 - Failed to spawn container "parrot-sphinx"
lxc parrot-sphinx 20210630034952.435 WARN     start - start.c:lxc_abort:1016 - No such process - Failed to send SIGKILL via pidfd 45 for process 64261
lxc parrot-sphinx 20210630034952.435 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:868 - Received container state "ABORTING" instead of "RUNNING"
lxc 20210630034957.604 ERROR    af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:207 - Connection reset by peer - Failed to receive response
lxc 20210630034957.604 ERROR    commands - commands.c:lxc_cmd_rsp_recv_fds:129 - Failed to receive file descriptors

Any thoughts on what is going on and why the container will not start when running as privileged?
I am running LXD 4.15 from snap.

Thanks!

Can you show lxc info and the content of /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.conf?

lxc info
config:
  core.https_address: '[172.31.43.217]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses:
  - 172.31.43.217:8443
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICBDCCAYmgAwIBAgIQTKnVKYCuzEcZXGlzmDuNXjAKBggqhkjOPQQDAzA0MRww
    GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QEMwMjM5
    MjAeFw0yMTAzMzAxNTQ1NTlaFw0zMTAzMjgxNTQ1NTlaMDQxHDAaBgNVBAoTE2xp
    bnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAQzAyMzkyMHYwEAYHKoZI
    zj0CAQYFK4EEACIDYgAEGrsMd+4y3co/b7E7sKrnZoddJx8yUD/kqLhsaBhBp045
    bjyTintX1aSQl7EmOhNxzNhpWWZmOO2p7tyHEYUcaA8Ato0JY6PbdmZeRDRhrDLF
    fU5SEypz2m0tA/UUi5H+o2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI
    KwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZDMDIzOTKHBH8AAAGH
    EAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxAN0VSP+YAXPA2Rvm
    CnmOHsYfGDmmcpPPoZRHSUiZ37/ENAYzRIil16rcjQZic56ZbQIxAI8f7X+1xt21
    teyG1yb2VxMCArhRYaQ/Tx+jjSAtzEb9fWt+tLrmqRHfYstn+7DE1g==
    -----END CERTIFICATE-----
  certificate_fingerprint: bff915ce77462dceca7acd4088444d43ecb04ffdf9c452e3caff544a5adcbaf7
  driver: lxc | qemu
  driver_version: 4.0.9 | 5.2.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.8.0-59-generic
  lxc_features:
    cgroup2: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "false"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "20.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: C02392
  server_pid: 3079
  server_version: "4.15"
  storage: zfs
  storage_version: 0.8.4-1ubuntu11.2

And contents of /var/snap/lxd/common/lxd/containers:

lrwxrwxrwx 1 root root 68 Jun  3 16:02 centos-bld -> /var/snap/lxd/common/lxd/storage-pools/default/containers/centos-bld
lrwxrwxrwx 1 root root 71 Jun 29 21:49 parrot-sphinx -> /var/snap/lxd/common/lxd/storage-pools/default/containers/parrot-sphinx
lrwxrwxrwx 1 root root 64 Jun 10 16:13 sigma1 -> /var/snap/lxd/common/lxd/storage-pools/default/containers/sigma1
lrwxrwxrwx 1 root root 64 Jun 10 16:17 sigma2 -> /var/snap/lxd/common/lxd/storage-pools/default/containers/sigma2
lrwxrwxrwx 1 root root 64 Jun 15 09:08 sigma3 -> /var/snap/lxd/common/lxd/storage-pools/default/containers/sigma3

And contents of /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.conf:

lxc.log.file = /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/parrot-sphinx/console.log
lxc.cap.drop = sys_time sys_module sys_rawio
lxc.mount.auto = proc:mixed sys:mixed cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = b *:* m
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/3366/exe callhook /var/snap/lxd/common/lxd "default" "parrot-sphinx" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "parrot-sphinx" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "parrot-sphinx" stop
lxc.tty.max = 0
lxc.uts.name = parrot-sphinx
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-parrot-sphinx_</var/snap/lxd/common/lxd>//&:lxd-parrot-sphinx_<var-snap-lxd-common-lxd>:
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/parrot-sphinx
lxc.environment = PULSE_SERVER=unix:/home/ubuntu/pulse-native
lxc.environment = DISPLAY=:0
lxc.environment = NVIDIA_VISIBLE_DEVICES=none
lxc.environment = NVIDIA_DRIVER_CAPABILITIES=all
lxc.environment = NVIDIA_REQUIRE_CUDA=
lxc.environment = NVIDIA_REQUIRE_DRIVER=
lxc.hook.mount = /snap/lxd/current/lxc/hooks/nvidia
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/parrot-sphinx:/dev/.lxd-mounts
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = veth73187f1d
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/parrot-sphinx/rootfs
lxc.cgroup.devices.allow = c 226:0 rwm
lxc.cgroup.devices.allow = c 226:128 rwm
lxc.cgroup.devices.allow = c 226:1 rwm
lxc.cgroup.devices.allow = c 226:129 rwm
lxc.cgroup.devices.allow = c 195:0 rwm
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-card0 dev/dri/card0 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-renderD128 dev/dri/renderD128 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-card1 dev/dri/card1 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-renderD129 dev/dri/renderD129 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-nvidia0 dev/nvidia0 none bind,create=file 0 0

Looks like you may have nvidia.runtime=true set. I believe the nvidia-container-runtime logic still doesn’t play nice with privileged containers, so that may be what’s causing the container to fail to start early.

That took care of it! Thanks for the help!

Unfortunately it did not get the parrot-sphinx simulation running, so I’m back to square one…