The trace is below, with one line of relevance appearing to be:
lxc pytorch-test 20250110160537.569 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /opt/incus/share/lxc/hooks/nvidia produced output: ERROR: Missing tool nvidia-container-cli, see https://github.com/NVIDIA/libnvidia-container
I do have nvidia-container-tools (1.17.3+dfsg-0lambda0.22.04.1) installed on the host, but I take it there is more to it than that.
Full log:
Name: pytorch-test
Status: STOPPED
Type: container (application)
Architecture: x86_64
Location: lbd-vector03
Created: 2025/01/08 21:13 PST
Last Used: 2025/01/10 08:05 PST
Snapshots:
+-------+----------------------+----------------------+----------+
| NAME | TAKEN AT | EXPIRES AT | STATEFUL |
+-------+----------------------+----------------------+----------+
| snap0 | 2025/01/09 15:50 PST | 2025/01/16 15:50 PST | NO |
+-------+----------------------+----------------------+----------+
Log:
lxc pytorch-test 20250110160537.432 TRACE commands - ../src/lxc/commands.c:lxc_cmd_timeout:525 - Connection refused - Command "get_state" failed to connect command socket
lxc pytorch-test 20250110160537.432 TRACE start - ../src/lxc/start.c:lxc_init_handler:739 - Created anonymous pair {3,6} of unix sockets
lxc pytorch-test 20250110160537.432 TRACE commands - ../src/lxc/commands.c:lxc_server_init:2138 - Created abstract unix socket "/var/lib/incus/containers/pytorch-test/command"
lxc pytorch-test 20250110160537.432 TRACE start - ../src/lxc/start.c:lxc_init_handler:755 - Unix domain socket 8 for command server is ready
lxc pytorch-test 20250110160537.433 INFO lxccontainer - ../src/lxc/lxccontainer.c:do_lxcapi_start:959 - Set process title to [lxc monitor] /var/lib/incus/containers pytorch-test
lxc pytorch-test 20250110160537.434 INFO start - ../src/lxc/start.c:lxc_check_inherited:326 - Closed inherited fd 4
lxc pytorch-test 20250110160537.434 INFO start - ../src/lxc/start.c:lxc_check_inherited:326 - Closed inherited fd 5
lxc pytorch-test 20250110160537.434 INFO start - ../src/lxc/start.c:lxc_check_inherited:326 - Closed inherited fd 19
lxc pytorch-test 20250110160537.434 TRACE execute - ../src/lxc/execute.c:lxc_execute:49 - Doing lxc_execute
lxc pytorch-test 20250110160537.434 INFO lsm - ../src/lxc/lsm/lsm.c:lsm_init_static:38 - Initialized LSM security driver AppArmor
lxc pytorch-test 20250110160537.434 TRACE start - ../src/lxc/start.c:lxc_init:779 - Initialized LSM
lxc pytorch-test 20250110160537.434 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to STARTING
lxc pytorch-test 20250110160537.434 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc pytorch-test 20250110160537.434 TRACE start - ../src/lxc/start.c:lxc_init:785 - Set container state to "STARTING"
lxc pytorch-test 20250110160537.434 TRACE start - ../src/lxc/start.c:lxc_init:841 - Set environment variables
lxc pytorch-test 20250110160537.434 INFO utils - ../src/lxc/utils.c:run_script_argv:590 - Executing script "/proc/1562/exe callhook /var/lib/incus "default" "pytorch-test" start" for container "pytorch-test"
lxc pytorch-test 20250110160537.434 TRACE utils - ../src/lxc/utils.c:run_script_argv:633 - Set environment variable: LXC_HOOK_TYPE=pre-start
lxc pytorch-test 20250110160537.434 TRACE utils - ../src/lxc/utils.c:run_script_argv:638 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc pytorch-test 20250110160537.435 DEBUG lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:818 - First child 1097300 exited
lxc pytorch-test 20250110160537.469 TRACE start - ../src/lxc/start.c:lxc_init:846 - Ran pre-start hooks
lxc pytorch-test 20250110160537.470 TRACE start - ../src/lxc/start.c:setup_signal_fd:371 - Created signal file descriptor 5
lxc pytorch-test 20250110160537.470 TRACE start - ../src/lxc/start.c:lxc_init:859 - Set up signal fd
lxc pytorch-test 20250110160537.470 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1498 - Running privileged, not using a systemd unit
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:462 - Adding cgroup hierarchy mounted at and base cgroup (null)
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the cpuset controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the cpu controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the io controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the memory controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the hugetlb controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the pids controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the rdma controller
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_hierarchy_add:465 - The hierarchy contains the misc controller
lxc pytorch-test 20250110160537.470 TRACE cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_program_load_kernel:335 - Loaded bpf program: func#0 @0
0: R1=ctx() R10=fp0
0: (61) r2 = *(u32 *)(r1 +0) ; R1=ctx() R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
1: (54) w2 &= 65535 ; R2_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
2: (61) r3 = *(u32 *)(r1 +0) ; R1=ctx() R3_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
3: (74) w3 >>= 16 ; R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=0xffff,var_off=(0x0; 0xffff))
4: (61) r4 = *(u32 *)(r1 +4) ; R1=ctx() R4_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
5: (61) r5 = *(u32 *)(r1 +8) ; R1=ctx() R5_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff))
6: (b7) r0 = 1 ; R0_w=1
7: (95) exit
mark_precise: frame0: last_idx 7 first_idx 0 subseq_idx -1
mark_precise: frame0: regs=r0 stack= before 6: (b7) r0 = 1
processed 8 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
lxc pytorch-test 20250110160537.470 TRACE cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_devices_cgroup_supported:553 - The bpf device cgroup is supported
lxc pytorch-test 20250110160537.470 TRACE cgroup - ../src/lxc/cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc pytorch-test 20250110160537.470 TRACE cgroup - ../src/lxc/cgroups/cgroup.c:cgroup_init:48 - Unified cgroup layout
lxc pytorch-test 20250110160537.470 TRACE start - ../src/lxc/start.c:lxc_init:866 - Initialized cgroup driver
lxc pytorch-test 20250110160537.470 DEBUG seccomp - ../src/lxc/seccomp.c:parse_config_v2:664 - Host native arch is [3221225534]
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:478 - Added arch 2 to main seccomp context
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:486 - Removed native arch from main seccomp context
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:478 - Added arch 3 to main seccomp context
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:486 - Removed native arch from main seccomp context
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:get_new_ctx:491 - Arch 4 already present in main seccomp context
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "[all]"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "[all]"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "kexec_load errno 38"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[246:kexec_load] action[327718:errno] arch[0]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327718:errno] arch[1073741827]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327718:errno] arch[1073741886]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "open_by_handle_at errno 38"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[304:open_by_handle_at] action[327718:errno] arch[0]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327718:errno] arch[1073741827]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327718:errno] arch[1073741886]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "init_module errno 38"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[175:init_module] action[327718:errno] arch[0]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327718:errno] arch[1073741827]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327718:errno] arch[1073741886]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "finit_module errno 38"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[313:finit_module] action[327718:errno] arch[0]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327718:errno] arch[1073741827]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327718:errno] arch[1073741886]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "delete_module errno 38"
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[176:delete_module] action[327718:errno] arch[0]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327718:errno] arch[1073741827]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327718:errno] arch[1073741886]
lxc pytorch-test 20250110160537.470 INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:1036 - Merging compat seccomp contexts into main context
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:parse_config_v2:1046 - Merged first compat seccomp context into main context
lxc pytorch-test 20250110160537.470 TRACE seccomp - ../src/lxc/seccomp.c:parse_config_v2:1062 - Merged second compat seccomp context into main context
lxc pytorch-test 20250110160537.470 TRACE start - ../src/lxc/start.c:lxc_init:873 - Read seccomp policy
lxc pytorch-test 20250110160537.470 TRACE start - ../src/lxc/start.c:lxc_init:880 - Initialized LSM
lxc pytorch-test 20250110160537.470 INFO start - ../src/lxc/start.c:lxc_init:882 - Container "pytorch-test" is initialized
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Created 10(lxc.monitor.pytorch-test) cgroup
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened newly created cgroup lxc.monitor.pytorch-test as 11
lxc pytorch-test 20250110160537.470 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1669 - The monitor process uses "lxc.monitor.pytorch-test" as cgroup
lxc pytorch-test 20250110160537.470 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3620 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc" controllers in the unified cgroup 10
lxc pytorch-test 20250110160537.493 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_enter:1819 - Moved monitor (1097301) into cgroup 11
lxc pytorch-test 20250110160537.493 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_enter:1833 - Moved transient process into cgroup 11
lxc pytorch-test 20250110160537.493 DEBUG storage - ../src/lxc/storage/storage.c:get_storage_by_name:209 - Detected rootfs type "dir"
lxc pytorch-test 20250110160537.493 TRACE conf - ../src/lxc/conf.c:lxc_rootfs_init:361 - Not pinning because container runs in user namespace
lxc pytorch-test 20250110160537.493 DEBUG storage - ../src/lxc/storage/storage.c:get_storage_by_name:209 - Detected rootfs type "dir"
lxc pytorch-test 20250110160537.493 TRACE sync - ../src/lxc/sync.c:lxc_sync_init:139 - Initialized synchronization infrastructure
lxc pytorch-test 20250110160537.494 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Created 10(lxc.payload.pytorch-test) cgroup
lxc pytorch-test 20250110160537.494 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened newly created cgroup lxc.payload.pytorch-test as 16
lxc pytorch-test 20250110160537.494 INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1777 - The container process uses "lxc.payload.pytorch-test" as inner and "lxc.payload.pytorch-test" as limit cgroup
lxc pytorch-test 20250110160537.495 TRACE start - ../src/lxc/start.c:lxc_spawn:1709 - Spawned container directly into target cgroup via cgroup2 fd 16
lxc pytorch-test 20250110160537.495 TRACE start - ../src/lxc/start.c:lxc_spawn:1749 - Cloned child process 1097319
lxc pytorch-test 20250110160537.495 TRACE start - ../src/lxc/start.c:core_scheduling:1589 - Created new core scheduling domain with cookie 3565788268
lxc pytorch-test 20250110160537.495 TRACE utils - ../src/lxc/utils.c:lxc_can_use_pidfd:1931 - Kernel supports pidfds
lxc pytorch-test 20250110160537.495 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUSER
lxc pytorch-test 20250110160537.495 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWNS
lxc pytorch-test 20250110160537.495 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWPID
lxc pytorch-test 20250110160537.495 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUTS
lxc pytorch-test 20250110160537.495 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWIPC
lxc pytorch-test 20250110160537.495 INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWCGROUP
lxc pytorch-test 20250110160537.495 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved user namespace via fd 18 and stashed path as user:/proc/1097301/fd/18
lxc pytorch-test 20250110160537.495 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved mnt namespace via fd 19 and stashed path as mnt:/proc/1097301/fd/19
lxc pytorch-test 20250110160537.495 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved pid namespace via fd 20 and stashed path as pid:/proc/1097301/fd/20
lxc pytorch-test 20250110160537.495 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved uts namespace via fd 21 and stashed path as uts:/proc/1097301/fd/21
lxc pytorch-test 20250110160537.495 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved ipc namespace via fd 22 and stashed path as ipc:/proc/1097301/fd/22
lxc pytorch-test 20250110160537.495 TRACE start - ../src/lxc/start.c:lxc_spawn:1709 - Spawned container directly into target cgroup via cgroup2 fd 16
lxc pytorch-test 20250110160537.495 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved cgroup namespace via fd 23 and stashed path as cgroup:/proc/1097301/fd/23
lxc pytorch-test 20250110160537.495 INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc pytorch-test 20250110160537.495 INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc pytorch-test 20250110160537.495 DEBUG idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:186 - No newuidmap and newgidmap binary found. Trying to write directly with euid 0
lxc pytorch-test 20250110160537.495 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:251 - Wrote mapping "0 1000000 1000000000
"
lxc pytorch-test 20250110160537.495 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:251 - Wrote mapping "0 1000000 1000000000
"
lxc pytorch-test 20250110160537.495 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_parent:110 - Child waiting for parent with sequence startup
lxc pytorch-test 20250110160537.495 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgfsng_delegate_controllers:3620 - Enabled "+cpuset +cpu +io +memory +hugetlb +pids +rdma +misc" controllers in the unified cgroup 10
lxc pytorch-test 20250110160537.495 TRACE conf - ../src/lxc/conf.c:get_minimal_idmap:4476 - Allocated minimal idmapping for ns uid 0 and ns gid 0
lxc pytorch-test 20250110160537.496 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4540 - Establishing uid mapping for "1097320" in new user namespace: nsuid 1000000000 - hostid 0 - range 1
lxc pytorch-test 20250110160537.496 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4540 - Establishing uid mapping for "1097320" in new user namespace: nsuid 0 - hostid 1000000 - range 1000000000
lxc pytorch-test 20250110160537.496 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4540 - Establishing gid mapping for "1097320" in new user namespace: nsuid 1000000000 - hostid 0 - range 1
lxc pytorch-test 20250110160537.496 TRACE conf - ../src/lxc/conf.c:userns_exec_1:4540 - Establishing gid mapping for "1097320" in new user namespace: nsuid 0 - hostid 1000000 - range 1000000000
lxc pytorch-test 20250110160537.496 INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:165 - newuidmap binary is missing
lxc pytorch-test 20250110160537.496 INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:171 - newgidmap binary is missing
lxc pytorch-test 20250110160537.496 INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:176 - Caller maps host root. Writing mapping directly
lxc pytorch-test 20250110160537.496 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:251 - Wrote mapping "1000000000 0 1
0 1000000 1000000000
"
lxc pytorch-test 20250110160537.496 TRACE idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:251 - Wrote mapping "1000000000 0 1
0 1000000 1000000000
"
lxc pytorch-test 20250110160537.496 TRACE conf - ../src/lxc/conf.c:run_userns_fn:4412 - Calling function "chown_cgroup_wrapper"
lxc pytorch-test 20250110160537.496 NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
lxc pytorch-test 20250110160537.497 TRACE sync - ../src/lxc/sync.c:lxc_sync_barrier_child:97 - Parent waking child with sequence startup and waiting with sequence configure
lxc pytorch-test 20250110160537.497 INFO start - ../src/lxc/start.c:do_start:1105 - Unshared CLONE_NEWNET
lxc pytorch-test 20250110160537.497 NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
lxc pytorch-test 20250110160537.497 NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1453 - Switched to gid 0
lxc pytorch-test 20250110160537.497 NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1462 - Switched to uid 0
lxc pytorch-test 20250110160537.497 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence configure
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: LIBRARY_PATH=/usr/local/cuda/lib64/stubs:
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUBLAS_VERSION=11.4.1.1026
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUDA_VERSION=11.2.1.007
lxc pytorch-test 20250110160537.497 DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved net namespace via fd 4 and stashed path as net:/proc/1097301/fd/4
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NSIGHT_SYSTEMS_VERSION=2020.4.3.7
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: HOME=/root
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: OPENUCX_VERSION=1.9.0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: LD_LIBRARY_PATH=/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_REQUIRE_CUDA=cuda>=9.0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NCCL_VERSION=2.8.4
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_PYTORCH_VERSION=21.03
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: DALI_BUILD=2054952
lxc pytorch-test 20250110160537.497 TRACE start - ../src/lxc/start.c:lxc_spawn:1841 - Allocated new network namespace id
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: PYTORCH_VERSION=1.9.0a0+df837d0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUSOLVER_VERSION=11.1.0.135
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: PYTORCH_BUILD_NUMBER=0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: TRT_VERSION=7.2.2.3+cuda11.1.0.024
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: DLPROF_VERSION=21.03
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: BASH_ENV=/etc/bash.bashrc
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NPP_VERSION=11.3.2.139
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: OPENMPI_VERSION=4.0.5
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUDNN_VERSION=8.1.1.33
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVJPEG_VERSION=11.4.0.135
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: TRTOSS_VERSION=21.03
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUDA_DRIVER_VERSION=460.32.03
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: TORCH_CUDA_ARCH_LIST=5.2 6.0 6.1 7.0 7.5 8.0 8.6+PTX
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CURAND_VERSION=10.2.3.135
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: COCOAPI_VERSION=2.0+nv0.4.0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: MOFED_VERSION=5.1-2.3.7
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVM_DIR=/usr/local/nvm
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: PATH=/opt/conda/bin:/opt/cmake-3.14.6-Linux-x86_64/bin/:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/tensorrt/bin
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: DALI_VERSION=0.31.0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NSIGHT_COMPUTE_VERSION=2020.3.1.3
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: PYTORCH_BUILD_VERSION=1.9.0a0+df837d0
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: _CUDA_COMPAT_PATH=/usr/local/cuda/compat
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUSPARSE_VERSION=11.4.0.135
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_BUILD_ID=21060478
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUDA_CACHE_DISABLE=1
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: TERM=xterm
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: PYTHONIOENCODING=utf-8
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: JUPYTER_PORT=8888
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_VISIBLE_DEVICES=all
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: LC_ALL=C.UTF-8
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: CUFFT_VERSION=10.4.0.135
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: ENV=/etc/shinit_v2
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: TENSORBOARD_PORT=6006
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_VISIBLE_DEVICES=none
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_DRIVER_CAPABILITIES=compute,utility
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_REQUIRE_CUDA=
lxc pytorch-test 20250110160537.497 TRACE conf - ../src/lxc/conf.c:lxc_set_environment:5231 - Set environment variable: NVIDIA_REQUIRE_DRIVER=
lxc pytorch-test 20250110160537.497 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_parent:110 - Child waiting for parent with sequence post-configure
lxc pytorch-test 20250110160537.497 DEBUG network - ../src/lxc/network.c:netdev_configure_server_phys:1250 - Instantiated phys "vethfda708be" with ifindex "99"
lxc pytorch-test 20250110160537.500 TRACE network - ../src/lxc/network.c:create_transient_name:3542 - Created transient name physD5EH2j for network device
lxc pytorch-test 20250110160537.525 DEBUG network - ../src/lxc/network.c:lxc_network_move_created_netdev_priv:3593 - Moved network device "vethfda708be" with ifindex 99 to network namespace of 1097319 and renamed to physD5EH2j
lxc pytorch-test 20250110160537.525 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_child:122 - Parent waking child with sequence post-configure
lxc pytorch-test 20250110160537.525 DEBUG storage - ../src/lxc/storage/storage.c:get_storage_by_name:209 - Detected rootfs type "dir"
lxc pytorch-test 20250110160537.525 TRACE mount_utils - ../src/lxc/mount_utils.c:can_use_mount_api:582 - Kernel supports mount api
lxc pytorch-test 20250110160537.525 TRACE mount_utils - ../src/lxc/mount_utils.c:can_use_bind_mounts:607 - Kernel supports bind mounts in the new mount api
lxc pytorch-test 20250110160537.525 TRACE mount_utils - ../src/lxc/mount_utils.c:create_detached_idmapped_mount:286 - Idmapped mount "/var/lib/incus/storage-pools/local/containers/pytorch-test/rootfs" requested with user namespace fd 12
lxc pytorch-test 20250110160537.525 TRACE conf - ../src/lxc/conf.c:lxc_rootfs_prepare_parent:458 - Created detached idmapped mount 24
lxc pytorch-test 20250110160537.525 TRACE network - ../src/lxc/network.c:lxc_network_send_to_child:4105 - Sent network device name "physD5EH2j" to child
lxc pytorch-test 20250110160537.525 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_child:116 - Parent waiting for child with sequence idmapped-mounts
lxc pytorch-test 20250110160537.525 TRACE conf - ../src/lxc/conf.c:lxc_rootfs_prepare_child:3634 - Received detached idmapped mount 17
lxc pytorch-test 20250110160537.526 TRACE conf - ../src/lxc/conf.c:turn_into_dependent_mounts:3455 - Turned all mount table entries into dependent mount
lxc pytorch-test 20250110160537.526 TRACE mount_utils - ../src/lxc/mount_utils.c:can_use_mount_api:582 - Kernel supports mount api
lxc pytorch-test 20250110160537.526 TRACE mount_utils - ../src/lxc/mount_utils.c:can_use_bind_mounts:607 - Kernel supports bind mounts in the new mount api
lxc pytorch-test 20250110160537.526 TRACE mount_utils - ../src/lxc/mount_utils.c:move_detached_mount:328 - Attach detached mount 17 to filesystem at 19
lxc pytorch-test 20250110160537.526 TRACE dir - ../src/lxc/storage/dir.c:dir_mount:197 - Mounted "/var/lib/incus/storage-pools/local/containers/pytorch-test/rootfs" onto "/opt/incus/lib/lxc/rootfs"
lxc pytorch-test 20250110160537.526 DEBUG conf - ../src/lxc/conf.c:lxc_mount_rootfs:1240 - Mounted rootfs "/var/lib/incus/storage-pools/local/containers/pytorch-test/rootfs" onto "/opt/incus/lib/lxc/rootfs" with options "idmap=container"
lxc pytorch-test 20250110160537.526 TRACE conf - ../src/lxc/conf.c:lxc_mount_rootfs:1248 - Container uses separate rootfs. Opened container's rootfs
lxc pytorch-test 20250110160537.526 INFO conf - ../src/lxc/conf.c:setup_utsname:679 - Set hostname to "pytorch-test"
lxc pytorch-test 20250110160537.526 TRACE network - ../src/lxc/network.c:lxc_network_recv_from_parent:4130 - Received network device name "physD5EH2j" from parent
lxc pytorch-test 20250110160537.538 TRACE network - ../src/lxc/network.c:__netdev_configure_container_common:1320 - Renamed network device from "physD5EH2j" to "eth0"
lxc pytorch-test 20250110160537.538 DEBUG network - ../src/lxc/network.c:setup_hw_addr:3866 - Mac address "00:16:3e:e1:41:9b" on "eth0" has been setup
lxc pytorch-test 20250110160537.538 DEBUG network - ../src/lxc/network.c:lxc_network_setup_in_child_namespaces_common:4007 - Network device "eth0" has been setup
lxc pytorch-test 20250110160537.538 INFO network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4064 - Finished setting up network devices with caller assigned names
lxc pytorch-test 20250110160537.538 INFO conf - ../src/lxc/conf.c:mount_autodev:1023 - Preparing "/dev"
lxc pytorch-test 20250110160537.538 TRACE mount_utils - ../src/lxc/mount_utils.c:__fs_prepare:177 - Finished initializing new tmpfs filesystem context 20
lxc pytorch-test 20250110160537.538 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_set_property:215 - Set "mode" to "0755" on filesystem context 20
lxc pytorch-test 20250110160537.539 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_set_property:215 - Set "size" to "500000" on filesystem context 20
lxc pytorch-test 20250110160537.539 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_attach:266 - Mounted 22 onto 21
lxc pytorch-test 20250110160537.539 INFO conf - ../src/lxc/conf.c:mount_autodev:1084 - Prepared "/dev"
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:539 - Invalid argument - Tried to ensure procfs is unmounted
lxc pytorch-test 20250110160537.539 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:546 - Created procfs mountpoint under 19
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:562 - Invalid argument - Tried to ensure sysfs is unmounted
lxc pytorch-test 20250110160537.539 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:569 - Created sysfs mountpoint under 19
lxc pytorch-test 20250110160537.539 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:623 - Mounted automount "proc" on "/opt/incus/lib/lxc/rootfs/proc" read-write with flags 14
lxc pytorch-test 20250110160537.539 TRACE conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:623 - Mounted automount "sysfs" on "/opt/incus/lib/lxc/rootfs/sys" read-write with flags 0
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/dev/fuse" on "/opt/incus/lib/lxc/rootfs/dev/fuse" to respect bind or remount options
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/dev/fuse" were 4098, required extra flags are 2
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/dev/fuse" on "/opt/incus/lib/lxc/rootfs/dev/fuse" with filesystem type "none"
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/dev/net/tun" on "/opt/incus/lib/lxc/rootfs/dev/net/tun" to respect bind or remount options
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/dev/net/tun" were 4098, required extra flags are 2
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/dev/net/tun" on "/opt/incus/lib/lxc/rootfs/dev/net/tun" with filesystem type "none"
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/firmware/efi/efivars" on "/opt/incus/lib/lxc/rootfs/sys/firmware/efi/efivars" to respect bind or remount options
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/firmware/efi/efivars" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/firmware/efi/efivars" on "/opt/incus/lib/lxc/rootfs/sys/firmware/efi/efivars" with filesystem type "none"
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/fs/fuse/connections" on "/opt/incus/lib/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/fs/fuse/connections" on "/opt/incus/lib/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/fs/pstore" on "/opt/incus/lib/lxc/rootfs/sys/fs/pstore" to respect bind or remount options
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/fs/pstore" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.539 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/fs/pstore" on "/opt/incus/lib/lxc/rootfs/sys/fs/pstore" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/kernel/config" on "/opt/incus/lib/lxc/rootfs/sys/kernel/config" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/kernel/config" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/kernel/config" on "/opt/incus/lib/lxc/rootfs/sys/kernel/config" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/kernel/debug" on "/opt/incus/lib/lxc/rootfs/sys/kernel/debug" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/kernel/debug" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/kernel/debug" on "/opt/incus/lib/lxc/rootfs/sys/kernel/debug" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/kernel/security" on "/opt/incus/lib/lxc/rootfs/sys/kernel/security" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/kernel/security" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/kernel/security" on "/opt/incus/lib/lxc/rootfs/sys/kernel/security" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/sys/kernel/tracing" on "/opt/incus/lib/lxc/rootfs/sys/kernel/tracing" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/sys/kernel/tracing" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/sys/kernel/tracing" on "/opt/incus/lib/lxc/rootfs/sys/kernel/tracing" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/dev/mqueue" on "/opt/incus/lib/lxc/rootfs/dev/mqueue" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/dev/mqueue" were 4110, required extra flags are 14
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/dev/mqueue" on "/opt/incus/lib/lxc/rootfs/dev/mqueue" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/var/lib/incus/guestapi" on "/opt/incus/lib/lxc/rootfs/dev/incus" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/var/lib/incus/guestapi" were 4096, required extra flags are 0
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2247 - Mountflags already were 4096, skipping remount
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/var/lib/incus/guestapi" on "/opt/incus/lib/lxc/rootfs/dev/incus" with filesystem type "none"
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:parse_vfs_attr:2090 - Raising nosuid
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:parse_vfs_attr:2090 - Raising noexec
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:parse_vfs_attr:2090 - Raising nodev
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "shm" on "/opt/incus/lib/lxc/rootfs/dev/shm" with filesystem type "tmpfs"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "none" on "/opt/incus/lib/lxc/rootfs/run" with filesystem type "tmpfs"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/var/lib/incus/containers/pytorch-test/network/hosts" on "/opt/incus/lib/lxc/rootfs/etc/hosts" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/var/lib/incus/containers/pytorch-test/network/hosts" were 4096, required extra flags are 0
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2247 - Mountflags already were 4096, skipping remount
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/var/lib/incus/containers/pytorch-test/network/hosts" on "/opt/incus/lib/lxc/rootfs/etc/hosts" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/var/lib/incus/containers/pytorch-test/network/hostname" on "/opt/incus/lib/lxc/rootfs/etc/hostname" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/var/lib/incus/containers/pytorch-test/network/hostname" were 4096, required extra flags are 0
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2247 - Mountflags already were 4096, skipping remount
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/var/lib/incus/containers/pytorch-test/network/hostname" on "/opt/incus/lib/lxc/rootfs/etc/hostname" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/var/lib/incus/containers/pytorch-test/network/resolv.conf" on "/opt/incus/lib/lxc/rootfs/etc/resolv.conf" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/var/lib/incus/containers/pytorch-test/network/resolv.conf" were 4096, required extra flags are 0
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2247 - Mountflags already were 4096, skipping remount
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/var/lib/incus/containers/pytorch-test/network/resolv.conf" on "/opt/incus/lib/lxc/rootfs/etc/resolv.conf" with filesystem type "none"
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2219 - Remounting "/var/lib/incus/shmounts/pytorch-test" on "/opt/incus/lib/lxc/rootfs/dev/.incus-mounts" to respect bind or remount options
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2238 - Flags for "/var/lib/incus/shmounts/pytorch-test" were 4096, required extra flags are 0
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2247 - Mountflags already were 4096, skipping remount
lxc pytorch-test 20250110160537.540 DEBUG conf - ../src/lxc/conf.c:mount_entry:2282 - Mounted "/var/lib/incus/shmounts/pytorch-test" on "/opt/incus/lib/lxc/rootfs/dev/.incus-mounts" with filesystem type "none"
lxc pytorch-test 20250110160537.540 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence idmapped-mounts
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:parse_vfs_attr:2090 - Raising nosuid
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:parse_vfs_attr:2090 - Raising noexec
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:parse_vfs_attr:2090 - Raising nodev
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:lxc_idmapped_mounts_child:2903 - Finished setting up idmapped mounts
lxc pytorch-test 20250110160537.540 TRACE conf - ../src/lxc/conf.c:lxc_idmapped_mounts_parent:3655 - Finished receiving idmapped mount file descriptors (-9 | -9) from child
lxc pytorch-test 20250110160537.540 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_mount:2254 - Read-write cgroup mounts requested
lxc pytorch-test 20250110160537.540 TRACE sync - ../src/lxc/sync.c:lxc_sync_wait_child:116 - Parent waiting for child with sequence cgroup-limits
lxc pytorch-test 20250110160537.540 TRACE mount_utils - ../src/lxc/mount_utils.c:__fs_prepare:177 - Finished initializing new cgroup2 filesystem context 22
lxc pytorch-test 20250110160537.540 TRACE mount_utils - ../src/lxc/mount_utils.c:fs_attach:266 - Mounted 23 onto 21
lxc pytorch-test 20250110160537.540 DEBUG cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroupfs_mount:2187 - Mounted cgroup filesystem cgroup2 onto 21((null))
lxc pytorch-test 20250110160537.540 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_mount:2355 - Force mounted cgroup filesystem in new cgroup namespace
lxc pytorch-test 20250110160537.540 INFO utils - ../src/lxc/utils.c:run_script_argv:590 - Executing script "/opt/incus/share/lxcfs/lxc.mount.hook" for container "pytorch-test"
lxc pytorch-test 20250110160537.540 TRACE utils - ../src/lxc/utils.c:run_script_argv:633 - Set environment variable: LXC_HOOK_TYPE=mount
lxc pytorch-test 20250110160537.540 TRACE utils - ../src/lxc/utils.c:run_script_argv:638 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc pytorch-test 20250110160537.566 INFO utils - ../src/lxc/utils.c:run_script_argv:590 - Executing script "/opt/incus/share/lxc/hooks/nvidia" for container "pytorch-test"
lxc pytorch-test 20250110160537.566 TRACE utils - ../src/lxc/utils.c:run_script_argv:633 - Set environment variable: LXC_HOOK_TYPE=mount
lxc pytorch-test 20250110160537.566 TRACE utils - ../src/lxc/utils.c:run_script_argv:638 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc pytorch-test 20250110160537.569 DEBUG utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /opt/incus/share/lxc/hooks/nvidia produced output: ERROR: Missing tool nvidia-container-cli, see https://github.com/NVIDIA/libnvidia-container
lxc pytorch-test 20250110160537.569 ERROR utils - ../src/lxc/utils.c:run_buffer:571 - Script exited with status 1
lxc pytorch-test 20250110160537.569 ERROR conf - ../src/lxc/conf.c:lxc_setup:3940 - Failed to run mount hooks
lxc pytorch-test 20250110160537.569 ERROR start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "pytorch-test"
lxc pytorch-test 20250110160537.569 TRACE sync - ../src/lxc/sync.c:lxc_sync_wake_parent:104 - Child waking parent with sequence error
lxc pytorch-test 20250110160537.569 ERROR sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_USER_NS=/proc/1097301/fd/18
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_MNT_NS=/proc/1097301/fd/19
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_PID_NS=/proc/1097301/fd/20
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_UTS_NS=/proc/1097301/fd/21
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_IPC_NS=/proc/1097301/fd/22
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_NET_NS=/proc/1097301/fd/4
lxc pytorch-test 20250110160537.569 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_CGROUP_NS=/proc/1097301/fd/23
lxc pytorch-test 20250110160537.574 WARN network - ../src/lxc/network.c:lxc_delete_network_priv:3674 - Failed to rename interface with index 0 from "eth0" to its initial name "vethfda708be"
lxc pytorch-test 20250110160537.574 DEBUG network - ../src/lxc/network.c:lxc_delete_network:4220 - Deleted network devices
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_socket_pair:545 - Sent container state "ABORTING" to 6
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to ABORTING
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc pytorch-test 20250110160537.574 ERROR lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:837 - Received container state "ABORTING" instead of "RUNNING"
lxc pytorch-test 20250110160537.574 ERROR start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "pytorch-test"
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to ABORTING
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc pytorch-test 20250110160537.574 WARN start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 17 for process 1097319
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:484 - Set container state to STOPPING
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_serve_state_clients:487 - No state clients registered
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_USER_NS=/proc/1097301/fd/18
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_MNT_NS=/proc/1097301/fd/19
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_PID_NS=/proc/1097301/fd/20
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_UTS_NS=/proc/1097301/fd/21
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_IPC_NS=/proc/1097301/fd/22
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_NET_NS=/proc/1097301/fd/4
lxc pytorch-test 20250110160537.574 TRACE start - ../src/lxc/start.c:lxc_expose_namespace_environment:907 - Set environment variable LXC_CGROUP_NS=/proc/1097301/fd/23
lxc pytorch-test 20250110160537.574 INFO utils - ../src/lxc/utils.c:run_script_argv:590 - Executing script "/opt/incus/bin/incusd callhook /var/lib/incus "default" "pytorch-test" stopns" for container "pytorch-test"
lxc pytorch-test 20250110160537.574 TRACE utils - ../src/lxc/utils.c:run_script_argv:633 - Set environment variable: LXC_HOOK_TYPE=stop
lxc pytorch-test 20250110160537.574 TRACE utils - ../src/lxc/utils.c:run_script_argv:638 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc pytorch-test 20250110160537.656 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_remove:491 - Removed cgroup tree 10(lxc.payload.pytorch-test)
lxc pytorch-test 20250110160537.656 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:726 - Reusing 10(lxc.pivot) cgroup
lxc pytorch-test 20250110160537.656 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:741 - Opened cgroup lxc.pivot as 3
lxc pytorch-test 20250110160537.668 TRACE cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_destroy:927 - Removed cgroup tree 10(lxc.monitor.pytorch-test)
lxc pytorch-test 20250110160537.668 TRACE start - ../src/lxc/start.c:lxc_end:964 - Closed command socket
lxc pytorch-test 20250110160537.668 TRACE start - ../src/lxc/start.c:lxc_end:975 - Set container state to "STOPPED"
lxc 20250110160537.668 ERROR af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20250110160537.668 ERROR commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_init_pid"
lxc pytorch-test 20250110160537.668 INFO utils - ../src/lxc/utils.c:run_script_argv:590 - Executing script "/opt/incus/share/lxcfs/lxc.reboot.hook" for container "pytorch-test"
lxc pytorch-test 20250110160537.668 TRACE utils - ../src/lxc/utils.c:run_script_argv:633 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc pytorch-test 20250110160537.668 TRACE utils - ../src/lxc/utils.c:run_script_argv:638 - Set environment variable: LXC_HOOK_SECTION=lxc
lxc pytorch-test 20250110160538.172 INFO utils - ../src/lxc/utils.c:run_script_argv:590 - Executing script "/opt/incus/bin/incusd callhook /var/lib/incus "default" "pytorch-test" stop" for container "pytorch-test"
lxc pytorch-test 20250110160538.172 TRACE utils - ../src/lxc/utils.c:run_script_argv:633 - Set environment variable: LXC_HOOK_TYPE=post-stop
lxc pytorch-test 20250110160538.172 TRACE utils - ../src/lxc/utils.c:run_script_argv:638 - Set environment variable: LXC_HOOK_SECTION=lxc