stgraber:
lxc.log.level=trace
I added that to the config and started it with --debug in Proxmox.
pct start 108 --debug
This is the output, I basically see that same permission denied error that sticks out, but also that it’s attempting to run privileged? I do have it configured as unprivileged. I thought that the nvidia-container-toolkit would only work with unprivileged containers.
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1508 - Running privileged, not using a systemd unit
Config:
arch: amd64
features: nesting=1
hostname: gpu-test
memory: 512
nameserver: 192.168.0.8
net0: name=eth0,bridge=vmbr5,firewall=1,gw=192.168.5.1,hwaddr=BC:24:11:FC:F5:12,ip=192.168.5.7/24,type=veth
ostype: debian
rootfs: local:108/vm-108-disk-0.raw,size=8G
swap: 512
unprivileged: 1
lxc.log.level: trace
lxc.hook.pre-start: sh -c '[ ! -f /dev/nvidia0 ] && /usr/bin/nvidia-modprobe -c0 -u'
lxc.environment: NVIDIA_VISIBLE_DEVICES=all
lxc.environment: NVIDIA_DRIVER_CAPABILITIES=compute,graphics,utility,video
lxc.hook.mount: /usr/share/lxc/hooks/nvidia
Log:
pct start 108 --debug
run_buffer: 571 Script exited with status 1
lxc_setup: 3948 Failed to run mount hooks
do_start: 1268 Failed to setup container "108"
sync_wait: 34 An error occurred in another process (expected sequence number 4)
__lxc_start: 2114 Failed to spawn container "108"
art 108 20250823161118.578 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "108", config section "lxc"
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "sh -c '[ ! -f /dev/nvidia0 ] && /usr/bin/nvidia-modprobe -c0 -u'" for container "108", config section "lxc"
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1508 - Running privileged, not using a systemd unit
DEBUG seccomp - ../src/lxc/seccomp.c:parse_config_v2:664 - Host native arch is [3221225534]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "[all]"
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "kexec_load errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "open_by_handle_at errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "init_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "finit_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "delete_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "keyctl errno 38"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[250:keyctl] action[327718:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[250:keyctl] action[327718:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[250:keyctl] action[327718:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:1036 - Merging compat seccomp contexts into main context
INFO start - ../src/lxc/start.c:lxc_init:882 - Container "108" is initialized
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1679 - The monitor process uses "lxc.monitor/108" as cgroup
DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1787 - The container process uses "lxc/108/ns" as inner and "lxc/108" as limit cgroup
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUSER
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWNS
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWPID
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUTS
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWIPC
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWCGROUP
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved user namespace via fd 17 and stashed path as user:/proc/3968769/fd/17
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/3968769/fd/18
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/3968769/fd/19
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/3968769/fd/20
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/3968769/fd/21
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved cgroup namespace via fd 22 and stashed path as cgroup:/proc/3968769/fd/22
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:178 - Functional newuidmap and newgidmap binary found
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3538 - Limits for the unified cgroup hierarchy have been setup
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:176 - Caller maps host root. Writing mapping directly
NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
INFO start - ../src/lxc/start.c:do_start:1105 - Unshared CLONE_NEWNET
NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1453 - Switched to gid 0
NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1462 - Switched to uid 0
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved net namespace via fd 5 and stashed path as net:/proc/3968769/fd/5
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "108", config section "net"
DEBUG network - ../src/lxc/network.c:netdev_configure_server_veth:879 - Instantiated veth tunnel "veth108i0 <--> vethqSGoJX"
DEBUG conf - ../src/lxc/conf.c:lxc_mount_rootfs:1244 - Mounted rootfs "/var/lib/lxc/108/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
INFO conf - ../src/lxc/conf.c:setup_utsname:683 - Set hostname to "gpu-test"
DEBUG network - ../src/lxc/network.c:setup_hw_addr:3866 - Mac address "BC:24:11:FC:F5:12" on "eth0" has been setup
DEBUG network - ../src/lxc/network.c:lxc_network_setup_in_child_namespaces_common:4007 - Network device "eth0" has been setup
INFO network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4064 - Finished setting up network devices with caller assigned names
INFO conf - ../src/lxc/conf.c:mount_autodev:1027 - Preparing "/dev"
INFO conf - ../src/lxc/conf.c:mount_autodev:1088 - Prepared "/dev"
DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:543 - Invalid argument - Tried to ensure procfs is unmounted
DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:566 - Invalid argument - Tried to ensure sysfs is unmounted
DEBUG conf - ../src/lxc/conf.c:mount_entry:2223 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
DEBUG conf - ../src/lxc/conf.c:mount_entry:2242 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
DEBUG conf - ../src/lxc/conf.c:mount_entry:2286 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
DEBUG conf - ../src/lxc/conf.c:mount_entry:2286 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
DEBUG conf - ../src/lxc/conf.c:mount_entry:2286 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
DEBUG cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroupfs_mount:2197 - Mounted cgroup filesystem cgroup2 onto 19((null))
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "108", config section "lxc"
INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/nvidia" for container "108", config section "lxc"
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: mkdir: cannot create directory ‘/var/lib/lxc/108/hook’
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: : Permission denied
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: + exec nvidia-container-cli --user configure --no-cgroups --ldconfig=@/usr/sbin/ldconfig --device=all --compute --graphics --utility --video /usr/lib/x86_64-linux-gnu/lxc/rootfs
DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: nvidia-container-cli: mount error: open failed: /usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/1/ns/mnt: permission denied
ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
ERROR conf - ../src/lxc/conf.c:lxc_setup:3948 - Failed to run mount hooks
ERROR start - ../src/lxc/start.c:do_start:1268 - Failed to setup container "108"
ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
DEBUG network - ../src/lxc/network.c:lxc_delete_network:4220 - Deleted network devices
ERROR start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "108"
WARN start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 16 for process 3968889
startup for container '108' failed
I also did a lxc-start --foreground --logpriority=trace 108 --logfile=108.log, here is the output of that log:
lxc-start 108 20250823162413.820 INFO confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type u nsid 0 hostid 100000 range 65536
lxc-start 108 20250823162413.820 INFO confile - ../src/lxc/confile.c:set_config_idmaps:2273 - Read uid map: type g nsid 0 hostid 100000 range 65536
lxc-start 108 20250823162413.821 TRACE commands - ../src/lxc/commands.c:lxc_cmd_timeout:525 - Connection refused - Command "get_init_pid" failed to connect command socket
lxc-start 108 20250823162413.821 TRACE commands - ../src/lxc/commands.c:lxc_cmd_timeout:525 - Connection refused - Command "get_state" failed to connect command socket
lxc-start 108 20250823162413.821 TRACE commands - ../src/lxc/commands.c:lxc_server_init:2138 - Created abstract unix socket "/var/lib/lxc/108/command"
lxc-start 108 20250823162413.821 TRACE start - ../src/lxc/start.c:lxc_init_handler:755 - Unix domain socket 4 for command server is ready
lxc-start 108 20250823162413.824 TRACE start - ../src/lxc/start.c:lxc_start:2228 - Doing lxc_start
lxc-start 108 20250823162413.825 INFO lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc-start 108 20250823162413.825 TRACE start - ../src/lxc/start.c:lxc_init:779 - Initialized LSM
lxc-start 108 20250823162413.825 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to STARTING
lxc-start 108 20250823162413.825 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc-start 108 20250823162413.825 TRACE start - ../src/lxc/start.c:lxc_init:785 - Set container state to "STARTING"
lxc-start 108 20250823162413.825 TRACE start - ../src/lxc/start.c:lxc_init:841 - Set environment variables
lxc-start 108 20250823162413.825 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "108", config section "lxc"
lxc-start 108 20250823162413.576 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "sh -c '[ ! -f /dev/nvidia0 ] && /usr/bin/nvidia-modprobe -c0 -u'" for container "108", config section "lxc"
lxc-start 108 20250823162413.578 TRACE start - ../src/lxc/start.c:lxc_init:846 - Ran pre-start hooks
lxc-start 108 20250823162413.578 TRACE start - ../src/lxc/start.c:setup_signal_fd:371 - Created signal file descriptor 6
lxc-start 108 20250823162413.578 TRACE start - ../src/lxc/start.c:lxc_init:859 - Set up signal fd
lxc-start 108 20250823162413.578 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1508 - Running privileged, not using a systemd unit
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:462 - Adding cgroup hierarchy mounted at and base cgroup (null)
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the cpuset controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the cpu controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the io controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the memory controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the hugetlb controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the pids controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the rdma controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the misc controller
lxc-start 108 20250823162413.578 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the dmem controller
lxc-start 108 20250823162413.578 TRACE cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_program_load_kernel:335 - Loaded bpf program: func#0 @0
0: R1=ctx() R10=fp0
0: (61) r2 = *(u32 *)(r1 +0) ; R1=ctx() R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
1: (54) w2 &= 65535 ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
2: (61) r3 = *(u32 *)(r1 +0) ; R1=ctx() R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3: (74) w3 >>= 16 ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
4: (61) r4 = *(u32 *)(r1 +4) ; R1=ctx() R4_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
5: (61) r5 = *(u32 *)(r1 +8) ; R1=ctx() R5_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
6: (b7) r0 = 1 ; R0_w=1
7: (95) exit
mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r0 stack= before 6: (b7) r0 = 1
processed 8 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
lxc-start 108 20250823162413.578 TRACE cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_devices_cgroup_supported:553 - The bpf device cgroup is supported
lxc-start 108 20250823162413.578 TRACE cgroup - ../src/lxc/cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc-start 108 20250823162413.578 TRACE cgroup - ../src/lxc/cgroups/cgroup.c:cgroup_init:48 - Unified cgroup layout
lxc-start 108 20250823162413.578 TRACE start - ../src/lxc/start.c:lxc_init:866 - Initialized cgroup driver
lxc-start 108 20250823162413.578 DEBUG seccomp - ../src/lxc/seccomp.c:parse_config_v2:664 - Host native arch is [3221225534]
lxc-start 108 20250823162413.578 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:478 - Added arch 2 to main seccomp context
lxc-start 108 20250823162413.578 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:486 - Removed native arch from main seccomp context
lxc-start 108 20250823162413.578 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:478 - Added arch 3 to main seccomp context
lxc-start 108 20250823162413.578 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:486 - Removed native arch from main seccomp context
lxc-start 108 20250823162413.578 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:491 - Arch 4 already present in main seccomp context
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "[all]"
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "kexec_load errno 1"
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
lxc-start 108 20250823162413.578 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "open_by_handle_at errno 1"
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "init_module errno 1"
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "finit_module errno 1"
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "delete_module errno 1"
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "keyctl errno 38"
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[250:keyctl] action[327718:errno] arch[0]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[250:keyctl] action[327718:errno] arch[1073741827]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[250:keyctl] action[327718:errno] arch[1073741886]
lxc-start 108 20250823162413.579 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:1036 - Merging compat seccomp contexts into main context
lxc-start 108 20250823162413.579 TRACE seccomp - ../src/lxc/seccomp.c:parse_config_v2:1046 - Merged first compat seccomp context into main context
lxc-start 108 20250823162413.579 TRACE seccomp - ../src/lxc/seccomp.c:parse_config_v2:1062 - Merged second compat seccomp context into main context
lxc-start 108 20250823162413.579 TRACE start - ../src/lxc/start.c:lxc_init:873 - Read seccomp policy
lxc-start 108 20250823162413.655 TRACE start - ../src/lxc/start.c:lxc_init:880 - Initialized LSM
lxc-start 108 20250823162413.655 INFO start - ../src/lxc/start.c:lxc_init:882 - Container "108" is initialized
lxc-start 108 20250823162413.655 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Reusing 8(lxc.monitor) cgroup
lxc-start 108 20250823162413.655 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened cgroup lxc.monitor as 9
lxc-start 108 20250823162413.655 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Created 9(108) cgroup
lxc-start 108 20250823162413.655 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened newly created cgroup 108 as 10
lxc-start 108 20250823162413.655 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1679 - The monitor process uses "lxc.monitor/108" as cgroup
lxc-start 108 20250823162413.655 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3630 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc +dmem" controllers in the unified cgroup 8
lxc-start 108 20250823162413.655 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3630 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc +dmem" controllers in the unified cgroup 9
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_enter:1829 - Moved monitor (4019444) into cgroup 10
lxc-start 108 20250823162413.668 DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
lxc-start 108 20250823162413.668 TRACE conf - ../src/lxc/conf.c:lxc_rootfs_init:365 - Not pinning because container runs in user namespace
lxc-start 108 20250823162413.668 DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
lxc-start 108 20250823162413.668 TRACE sync - ../src/lxc/sync.c:lxc_sync_init:139 - Initialized synchronization infrastructure
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Reusing 8(lxc) cgroup
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened cgroup lxc as 14
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Created 14(108) cgroup
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened newly created cgroup 108 as 15
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:775 - Created limit cgroup 15->8(lxc/108)
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Created 15(ns) cgroup
lxc-start 108 20250823162413.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened newly created cgroup ns as 14
lxc-start 108 20250823162413.668 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1787 - The container process uses "lxc/108/ns" as inner and "lxc/108" as limit cgroup
lxc-start 108 20250823162413.668 TRACE start - ../src/lxc/start.c:lxc_spawn:1709 - Spawned container directly into target cgroup via cgroup2 fd 14
lxc-start 108 20250823162413.668 TRACE start - ../src/lxc/start.c:lxc_spawn:1749 - Cloned child process 4019469
lxc-start 108 20250823162413.669 TRACE start - ../src/lxc/start.c:core_scheduling:1567 - No new core scheduling domain requested
lxc-start 108 20250823162413.669 TRACE utils - ../src/lxc/utils.c:lxc_can_use_pidfd:1931 - Kernel supports pidfds
lxc-start 108 20250823162413.669 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUSER
lxc-start 108 20250823162413.668 TRACE start - ../src/lxc/start.c:lxc_spawn:1709 - Spawned container directly into target cgroup via cgroup2 fd 14
lxc-start 108 20250823162413.669 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWNS
lxc-start 108 20250823162413.669 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWPID
lxc-start 108 20250823162413.669 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUTS
lxc-start 108 20250823162413.669 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWIPC
lxc-start 108 20250823162413.669 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWCGROUP
lxc-start 108 20250823162413.669 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved user namespace via fd 17 and stashed path as user:/proc/4019444/fd/17
lxc-start 108 20250823162413.669 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/4019444/fd/18
lxc-start 108 20250823162413.669 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/4019444/fd/19
lxc-start 108 20250823162413.669 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/4019444/fd/20
lxc-start 108 20250823162413.669 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/4019444/fd/21
lxc-start 108 20250823162413.669 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved cgroup namespace via fd 22 and stashed path as cgroup:/proc/4019444/fd/22
lxc-start 108 20250823162413.669 DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start 108 20250823162413.669 DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start 108 20250823162413.669 DEBUG idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:178 - Functional newuidmap and newgidmap binary found
lxc-start 108 20250823162413.669 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_parent:110 - Child waiting for parent with sequence startup
lxc-start 108 20250823162413.671 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:246 - newuidmap wrote mapping "newuidmap 4019469 0 100000 65536"
lxc-start 108 20250823162413.673 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:246 - newgidmap wrote mapping "newgidmap 4019469 0 100000 65536"
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3630 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc +dmem" controllers in the unified cgroup 8
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3630 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc +dmem" controllers in the unified cgroup 5
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3630 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc +dmem" controllers in the unified cgroup 9
lxc-start 108 20250823162413.673 TRACE cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_list_add_device:467 - Device cgroup will allow ("denylist") all devices by default
lxc-start 108 20250823162413.673 TRACE cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_list_add_device:475 - Resetting cgroup device list
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3535 - Set "devices.allow" to "a"
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3535 - Set "memory.max" to "536870912"
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3535 - Set "memory.high" to "532676608"
lxc-start 108 20250823162413.673 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3535 - Set "memory.swap.max" to "536870912"
lxc-start 108 20250823162413.673 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3538 - Limits for the unified cgroup hierarchy have been setup
lxc-start 108 20250823162413.673 TRACE conf - ../src/lxc/conf.c:get_minimal_idmap:4484 - Allocated minimal idmapping for ns uid 0 and ns gid 0
lxc-start 108 20250823162413.673 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4548 - Establishing uid mapping for "4019474" in new user namespace: nsuid 65536 - hostid 0 - range 1
lxc-start 108 20250823162413.673 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4548 - Establishing uid mapping for "4019474" in new user namespace: nsuid 0 - hostid 100000 - range 65536
lxc-start 108 20250823162413.673 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4548 - Establishing gid mapping for "4019474" in new user namespace: nsuid 65536 - hostid 0 - range 1
lxc-start 108 20250823162413.673 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4548 - Establishing gid mapping for "4019474" in new user namespace: nsuid 0 - hostid 100000 - range 65536
lxc-start 108 20250823162413.673 DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
lxc-start 108 20250823162413.673 DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
lxc-start 108 20250823162413.673 INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:176 - Caller maps host root. Writing mapping directly
lxc-start 108 20250823162413.673 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:251 - Wrote mapping "65536 0 1
0 100000 65536
"
lxc-start 108 20250823162413.673 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:251 - Wrote mapping "65536 0 1
0 100000 65536
"
lxc-start 108 20250823162413.673 TRACE conf - ../src/lxc/conf.c:run_userns_fn:4420 - Calling function "chown_cgroup_wrapper"
lxc-start 108 20250823162413.673 NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
lxc-start 108 20250823162413.674 TRACE sync - ../src/lxc/sync.c:lxc_sync_barrier_child:97 - Parent waking child with sequence startup and waiting with sequence configure
lxc-start 108 20250823162413.674 INFO start - ../src/lxc/start.c:do_start:1105 - Unshared CLONE_NEWNET
lxc-start 108 20250823162413.674 NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
lxc-start 108 20250823162413.674 NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1453 - Switched to gid 0
lxc-start 108 20250823162413.674 NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1462 - Switched to uid 0
lxc-start 108 20250823162413.674 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence configure
lxc-start 108 20250823162413.674 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5239 - Set environment variable: TERM=linux
lxc-start 108 20250823162413.674 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5239 - Set environment variable: NVIDIA_VISIBLE_DEVICES=all
lxc-start 108 20250823162413.674 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved net namespace via fd 5 and stashed path as net:/proc/4019444/fd/5
lxc-start 108 20250823162413.674 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5239 - Set environment variable: NVIDIA_DRIVER_CAPABILITIES=compute,graphics,utility,video
lxc-start 108 20250823162413.674 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_parent:110 - Child waiting for parent with sequence post-configure
lxc-start 108 20250823162413.674 TRACE start - ../src/lxc/start.c:lxc_spawn:1841 - Allocated new network namespace id
lxc-start 108 20250823162413.679 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "108", config section "net"
lxc-start 108 20250823162414.237 DEBUG network - ../src/lxc/network.c:netdev_configure_server_veth:879 - Instantiated veth tunnel "veth108i0 <--> vethKOEfcL"
lxc-start 108 20250823162414.237 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_child:122 - Parent waking child with sequence post-configure
lxc-start 108 20250823162414.237 TRACE network - ../src/lxc/network.c:lxc_network_send_to_child:4105 - Sent network device name "vethKOEfcL" to child
lxc-start 108 20250823162414.237 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_child:116 - Parent waiting for child with sequence idmapped-mounts
lxc-start 108 20250823162414.238 TRACE conf - ../src/lxc/conf.c:turn_into_dependent_mounts:3463 - Turned all mount table entries into dependent mount
lxc-start 108 20250823162414.238 TRACE mount_utils - ../src/lxc/mount_utils.c:can_use_mount_api:582 - Kernel supports mount api
lxc-start 108 20250823162414.238 TRACE mount_utils - ../src/lxc/mount_utils.c:can_use_bind_mounts:607 - Kernel supports bind mounts in the new mount api
lxc-start 108 20250823162414.238 TRACE mount_utils - ../src/lxc/mount_utils.c:move_detached_mount:328 - Attach detached mount 19 to filesystem at 17
lxc-start 108 20250823162414.238 TRACE dir - ../src/lxc/storage/dir.c:dir_mount:197 - Mounted "/var/lib/lxc/108/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs"
lxc-start 108 20250823162414.238 DEBUG conf - ../src/lxc/conf.c:lxc_mount_rootfs:1244 - Mounted rootfs "/var/lib/lxc/108/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start 108 20250823162414.238 TRACE conf - ../src/lxc/conf.c:lxc_mount_rootfs:1252 - Container uses separate rootfs. Opened container's rootfs
lxc-start 108 20250823162414.238 INFO conf - ../src/lxc/conf.c:setup_utsname:683 - Set hostname to "gpu-test"
lxc-start 108 20250823162414.238 TRACE network - ../src/lxc/network.c:lxc_network_recv_from_parent:4130 - Received network device name "vethKOEfcL" from parent
lxc-start 108 20250823162414.240 TRACE network - ../src/lxc/network.c:__netdev_configure_container_common:1320 - Renamed network device from "vethKOEfcL" to "eth0"
lxc-start 108 20250823162414.240 DEBUG network - ../src/lxc/network.c:setup_hw_addr:3866 - Mac address "BC:24:11:FC:F5:12" on "eth0" has been setup
lxc-start 108 20250823162414.240 DEBUG network - ../src/lxc/network.c:lxc_network_setup_in_child_namespaces_common:4007 - Network device "eth0" has been setup
lxc-start 108 20250823162414.240 INFO network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4064 - Finished setting up network devices with caller assigned names
lxc-start 108 20250823162414.240 INFO conf - ../src/lxc/conf.c:mount_autodev:1027 - Preparing "/dev"
lxc-start 108 20250823162414.240 TRACE mount_utils - ../src/lxc/mount_utils.c:__fs_prepare:177 - Finished initializing new tmpfs filesystem context 18
lxc-start 108 20250823162414.240 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_set_property:215 - Set "mode" to "0755" on filesystem context 18
lxc-start 108 20250823162414.240 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_set_property:215 - Set "size" to "500000" on filesystem context 18
lxc-start 108 20250823162414.240 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_attach:266 - Mounted 20 onto 19
lxc-start 108 20250823162414.240 INFO conf - ../src/lxc/conf.c:mount_autodev:1088 - Prepared "/dev"
lxc-start 108 20250823162414.240 DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:543 - Invalid argument - Tried to ensure procfs is unmounted
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:550 - Created procfs mountpoint under 17
lxc-start 108 20250823162414.240 DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:566 - Invalid argument - Tried to ensure sysfs is unmounted
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:573 - Created sysfs mountpoint under 17
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc" read-write with flags 14
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/net" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/tty" read-write with flags 4096
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" read-write with flags 4096
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:625 - Remounted automount "(null)" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" read-only with flags 4129
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/tty" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/net" read-write with flags 8192
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sysrq-trigger" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sysrq-trigger" read-write with flags 4096
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:625 - Remounted automount "(null)" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sysrq-trigger" read-only with flags 4129
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "sysfs" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys" read-write with flags 14
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "sysfs" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys" read-only with flags 15
lxc-start 108 20250823162414.240 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:627 - Mounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys/devices/virtual/net" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/devices/virtual/net" read-write with flags 4096
lxc-start 108 20250823162414.241 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:602 - Unmounted automount "/usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/sys"
lxc-start 108 20250823162414.241 DEBUG conf - ../src/lxc/conf.c:mount_entry:2223 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start 108 20250823162414.241 DEBUG conf - ../src/lxc/conf.c:mount_entry:2242 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
lxc-start 108 20250823162414.241 DEBUG conf - ../src/lxc/conf.c:mount_entry:2286 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start 108 20250823162414.241 DEBUG conf - ../src/lxc/conf.c:mount_entry:2286 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
lxc-start 108 20250823162414.241 DEBUG conf - ../src/lxc/conf.c:mount_entry:2286 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
lxc-start 108 20250823162414.241 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence idmapped-mounts
lxc-start 108 20250823162414.241 TRACE conf - ../src/lxc/conf.c:lxc_idmapped_mounts_child:2907 - Finished setting up idmapped mounts
lxc-start 108 20250823162414.241 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_mount:2267 - Mixed cgroup mounts requested
lxc-start 108 20250823162414.241 TRACE conf - ../src/lxc/conf.c:lxc_idmapped_mounts_parent:3663 - Finished receiving idmapped mount file descriptors (-9 | -9) from child
lxc-start 108 20250823162414.241 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_child:116 - Parent waiting for child with sequence cgroup-limits
lxc-start 108 20250823162414.241 TRACE mount_utils - ../src/lxc/mount_utils.c:__fs_prepare:177 - Finished initializing new cgroup2 filesystem context 20
lxc-start 108 20250823162414.241 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_attach:266 - Mounted 21 onto 19
lxc-start 108 20250823162414.241 DEBUG cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroupfs_mount:2197 - Mounted cgroup filesystem cgroup2 onto 19((null))
lxc-start 108 20250823162414.241 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_mount:2365 - Force mounted cgroup filesystem in new cgroup namespace
lxc-start 108 20250823162414.241 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "108", config section "lxc"
lxc-start 108 20250823162414.264 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/nvidia" for container "108", config section "lxc"
lxc-start 108 20250823162414.274 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: mkdir: cannot create directory ‘/var/lib/lxc/108/hook’
lxc-start 108 20250823162414.274 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: : Permission denied
lxc-start 108 20250823162414.277 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: + exec nvidia-container-cli --user configure --no-cgroups --ldconfig=@/usr/sbin/ldconfig --device=all --compute --graphics --utility --video /usr/lib/x86_64-linux-gnu/lxc/rootfs
lxc-start 108 20250823162414.296 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/nvidia 108 lxc mount produced output: nvidia-container-cli: mount error: open failed: /usr/lib/x86_64-linux-gnu/lxc/rootfs/proc/1/ns/mnt: permission denied
lxc-start 108 20250823162414.299 ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
lxc-start 108 20250823162414.299 ERROR conf - ../src/lxc/conf.c:lxc_setup:3948 - Failed to run mount hooks
lxc-start 108 20250823162414.299 ERROR start - ../src/lxc/start.c:do_start:1268 - Failed to setup container "108"
lxc-start 108 20250823162414.299 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence error
lxc-start 108 20250823162414.299 ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_USER_NS=/proc/4019444/fd/17
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_MNT_NS=/proc/4019444/fd/18
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_PID_NS=/proc/4019444/fd/19
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_UTS_NS=/proc/4019444/fd/20
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_IPC_NS=/proc/4019444/fd/21
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_NET_NS=/proc/4019444/fd/5
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_CGROUP_NS=/proc/4019444/fd/22
lxc-start 108 20250823162414.299 DEBUG network - ../src/lxc/network.c:lxc_delete_network:4220 - Deleted network devices
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to ABORTING
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc-start 108 20250823162414.299 ERROR start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "108"
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to ABORTING
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc-start 108 20250823162414.299 WARN start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 16 for process 4019469
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to STOPPING
lxc-start 108 20250823162414.299 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc-start 108 20250823162414.378 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_remove:491 - Removed cgroup tree 8(lxc/108)
lxc-start 108 20250823162414.378 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Reusing 8(lxc.pivot) cgroup
lxc-start 108 20250823162414.378 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened cgroup lxc.pivot as 9
lxc-start 108 20250823162414.396 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:927 - Removed cgroup tree 8(lxc.monitor/108)
lxc-start 108 20250823162414.396 TRACE start - ../src/lxc/start.c:lxc_end:964 - Closed command socket
lxc-start 108 20250823162414.396 TRACE start - ../src/lxc/start.c:lxc_end:975 - Set container state to "STOPPED"
lxc-start 108 20250823162414.397 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "108", config section "lxc"
lxc-start 108 20250823162414.901 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-poststop-hook" for container "108", config section "lxc"
lxc-start 108 20250823162415.326 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 108 lxc post-stop produced output: umount: /var/lib/lxc/.pve-staged-mounts/mp0: not mounted.
lxc-start 108 20250823162415.326 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-poststop-hook 108 lxc post-stop produced output: command 'umount -- /var/lib/lxc/.pve-staged-mounts/mp0' failed: exit code 32
lxc-start 108 20250823162415.435 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:307 - The container failed to start
lxc-start 108 20250823162415.435 ERROR lxc_start - ../src/lxc/tools/lxc_start.c:lxc_start_main:312 - Additional information can be obtained by setting the --logfile and --logpriority options