jhaws1982
(jhaws1982)
June 30, 2021, 4:00am
1
I am trying to create a privileged container to run an application (parrot-sphinx) inside a Ubuntu 18.04 container. The main reason behind this is because it will not install on a 20.04 system and requires 18.04.
I’m thinking I need a privileged container because of how some tools work in support of that simulator - in particular the firmwared daemon. It tries to mount some things to /var/cache and fails to do so with an “Operation Not Permitted” error from mount. If there is a better way to solve this I am all ears and would love some guidance!
So - back to the topic - when I try to create a privileged container I get the following:
$ lxc launch images:ubuntu/18.04 parrot-sphinx -p default -c security.privileged=true
Creating parrot-sphinx
Starting parrot-sphinx
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart parrot-sphinx
/var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.conf:
Try lxc info --show-log local:parrot-sphinx
for more info
$ lxc info --show-log local:parrot-sphinx
Name: parrot-sphinx
Location: none
Remote: unix://
Architecture: x86_64
Created: 2021/06/30 03:49 UTC
Status: Stopped
Type: container
Profiles: default
Log:
lxc parrot-sphinx 20210630034952.353 ERROR conf - conf.c:turn_into_dependent_mounts:3340 - No such file or directory - Failed to recursively turn old root mount tree into dependent mount. Continuing...
lxc parrot-sphinx 20210630034952.429 ERROR conf - conf.c:run_buffer:316 - Script exited with status 1
lxc parrot-sphinx 20210630034952.429 ERROR conf - conf.c:lxc_setup:3686 - Failed to run mount hooks
lxc parrot-sphinx 20210630034952.429 ERROR start - start.c:do_start:1265 - Failed to setup container "parrot-sphinx"
lxc parrot-sphinx 20210630034952.429 ERROR sync - sync.c:sync_wait:36 - An error occurred in another process (expected sequence number 5)
lxc parrot-sphinx 20210630034952.435 WARN network - network.c:lxc_delete_network_priv:3621 - Failed to rename interface with index 0 from "eth0" to its initial name "veth73187f1d"
lxc parrot-sphinx 20210630034952.435 ERROR start - start.c:__lxc_start:2073 - Failed to spawn container "parrot-sphinx"
lxc parrot-sphinx 20210630034952.435 WARN start - start.c:lxc_abort:1016 - No such process - Failed to send SIGKILL via pidfd 45 for process 64261
lxc parrot-sphinx 20210630034952.435 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:868 - Received container state "ABORTING" instead of "RUNNING"
lxc 20210630034957.604 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:207 - Connection reset by peer - Failed to receive response
lxc 20210630034957.604 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:129 - Failed to receive file descriptors
Any thoughts on what is going on and why the container will not start when running as privileged?
I am running LXD 4.15 from snap.
Thanks!
stgraber
(Stéphane Graber)
June 30, 2021, 4:08am
2
Can you show lxc info
and the content of /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.conf
?
jhaws1982
(jhaws1982)
June 30, 2021, 2:19pm
3
lxc info
config:
core.https_address: '[172.31.43.217]:8443'
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- 172.31.43.217:8443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIICBDCCAYmgAwIBAgIQTKnVKYCuzEcZXGlzmDuNXjAKBggqhkjOPQQDAzA0MRww
GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QEMwMjM5
MjAeFw0yMTAzMzAxNTQ1NTlaFw0zMTAzMjgxNTQ1NTlaMDQxHDAaBgNVBAoTE2xp
bnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAQzAyMzkyMHYwEAYHKoZI
zj0CAQYFK4EEACIDYgAEGrsMd+4y3co/b7E7sKrnZoddJx8yUD/kqLhsaBhBp045
bjyTintX1aSQl7EmOhNxzNhpWWZmOO2p7tyHEYUcaA8Ato0JY6PbdmZeRDRhrDLF
fU5SEypz2m0tA/UUi5H+o2AwXjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI
KwYBBQUHAwEwDAYDVR0TAQH/BAIwADApBgNVHREEIjAgggZDMDIzOTKHBH8AAAGH
EAAAAAAAAAAAAAAAAAAAAAEwCgYIKoZIzj0EAwMDaQAwZgIxAN0VSP+YAXPA2Rvm
CnmOHsYfGDmmcpPPoZRHSUiZ37/ENAYzRIil16rcjQZic56ZbQIxAI8f7X+1xt21
teyG1yb2VxMCArhRYaQ/Tx+jjSAtzEb9fWt+tLrmqRHfYstn+7DE1g==
-----END CERTIFICATE-----
certificate_fingerprint: bff915ce77462dceca7acd4088444d43ecb04ffdf9c452e3caff544a5adcbaf7
driver: lxc | qemu
driver_version: 4.0.9 | 5.2.0
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
shiftfs: "false"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.8.0-59-generic
lxc_features:
cgroup2: "true"
devpts_fd: "true"
idmapped_mounts_v2: "false"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Ubuntu
os_version: "20.04"
project: default
server: lxd
server_clustered: false
server_name: C02392
server_pid: 3079
server_version: "4.15"
storage: zfs
storage_version: 0.8.4-1ubuntu11.2
And contents of /var/snap/lxd/common/lxd/containers:
lrwxrwxrwx 1 root root 68 Jun 3 16:02 centos-bld -> /var/snap/lxd/common/lxd/storage-pools/default/containers/centos-bld
lrwxrwxrwx 1 root root 71 Jun 29 21:49 parrot-sphinx -> /var/snap/lxd/common/lxd/storage-pools/default/containers/parrot-sphinx
lrwxrwxrwx 1 root root 64 Jun 10 16:13 sigma1 -> /var/snap/lxd/common/lxd/storage-pools/default/containers/sigma1
lrwxrwxrwx 1 root root 64 Jun 10 16:17 sigma2 -> /var/snap/lxd/common/lxd/storage-pools/default/containers/sigma2
lrwxrwxrwx 1 root root 64 Jun 15 09:08 sigma3 -> /var/snap/lxd/common/lxd/storage-pools/default/containers/sigma3
And contents of /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.conf:
lxc.log.file = /var/snap/lxd/common/lxd/logs/parrot-sphinx/lxc.log
lxc.log.level = warn
lxc.console.buffer.size = auto
lxc.console.size = auto
lxc.console.logfile = /var/snap/lxd/common/lxd/logs/parrot-sphinx/console.log
lxc.cap.drop = sys_time sys_module sys_rawio
lxc.mount.auto = proc:mixed sys:mixed cgroup:mixed
lxc.autodev = 1
lxc.pty.max = 1024
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/config sys/kernel/config none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/tracing sys/kernel/tracing none rbind,create=dir,optional 0 0
lxc.include = /snap/lxd/current/lxc/config//common.conf.d/
lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = b *:* m
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
lxc.cgroup.devices.allow = c 1:7 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:2 rwm
lxc.cgroup.devices.allow = c 10:229 rwm
lxc.cgroup.devices.allow = c 10:200 rwm
lxc.arch = linux64
lxc.hook.version = 1
lxc.hook.pre-start = /proc/3366/exe callhook /var/snap/lxd/common/lxd "default" "parrot-sphinx" start
lxc.hook.stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "parrot-sphinx" stopns
lxc.hook.post-stop = /snap/lxd/current/bin/lxd callhook /var/snap/lxd/common/lxd "default" "parrot-sphinx" stop
lxc.tty.max = 0
lxc.uts.name = parrot-sphinx
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.apparmor.profile = lxd-parrot-sphinx_</var/snap/lxd/common/lxd>//&:lxd-parrot-sphinx_<var-snap-lxd-common-lxd>:
lxc.seccomp.profile = /var/snap/lxd/common/lxd/security/seccomp/parrot-sphinx
lxc.environment = PULSE_SERVER=unix:/home/ubuntu/pulse-native
lxc.environment = DISPLAY=:0
lxc.environment = NVIDIA_VISIBLE_DEVICES=none
lxc.environment = NVIDIA_DRIVER_CAPABILITIES=all
lxc.environment = NVIDIA_REQUIRE_CUDA=
lxc.environment = NVIDIA_REQUIRE_DRIVER=
lxc.hook.mount = /snap/lxd/current/lxc/hooks/nvidia
lxc.mount.auto = shmounts:/var/snap/lxd/common/lxd/shmounts/parrot-sphinx:/dev/.lxd-mounts
lxc.net.0.type = phys
lxc.net.0.name = eth0
lxc.net.0.flags = up
lxc.net.0.link = veth73187f1d
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/parrot-sphinx/rootfs
lxc.cgroup.devices.allow = c 226:0 rwm
lxc.cgroup.devices.allow = c 226:128 rwm
lxc.cgroup.devices.allow = c 226:1 rwm
lxc.cgroup.devices.allow = c 226:129 rwm
lxc.cgroup.devices.allow = c 195:0 rwm
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-card0 dev/dri/card0 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-renderD128 dev/dri/renderD128 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-card1 dev/dri/card1 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-dri-renderD129 dev/dri/renderD129 none bind,create=file 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/parrot-sphinx/unix.mygpu.dev-nvidia0 dev/nvidia0 none bind,create=file 0 0
stgraber
(Stéphane Graber)
July 1, 2021, 4:02am
4
Looks like you may have nvidia.runtime=true
set. I believe the nvidia-container-runtime logic still doesn’t play nice with privileged containers, so that may be what’s causing the container to fail to start early.
That took care of it! Thanks for the help!
Unfortunately it did not get the parrot-sphinx simulation running, so I’m back to square one…