Weird situation here. After upgrading a LXC host from Debian 10 to 11, I am unable to start ONE specific container (all the others have worked without a glitch).
Launching the container in foreground shows:
root@host ~ # lxc-start -n container -F
/sbin/init: /lib/x86_64-linux-gnu/libapparmor.so.1: version `APPARMOR_2.13' not found (required by /sbin/init)
Launched the start with TRACE logging and it shows an error on error.c:lxc_error_set_and_log:28 but was unable to identify what exactly this is.
lxc-start container 20230628041657.418 TRACE commands - commands.c:lxc_cmd:301 - Connection refused - Command "get_init_pid" failed to connect command socket
lxc-start container 20230628041657.418 TRACE commands - commands.c:lxc_cmd:301 - Connection refused - Command "get_state" failed to connect command socket
lxc-start container 20230628041657.418 TRACE start - start.c:lxc_init_handler:692 - Created anonymous pair {4,5} of unix sockets
lxc-start container 20230628041657.418 TRACE commands - commands.c:lxc_cmd_init:1770 - Created abstract unix socket "/var/lib/lxc/container/command"
lxc-start container 20230628041657.418 TRACE start - start.c:lxc_init_handler:708 - Unix domain socket 6 for command server is ready
lxc-start container 20230628041657.418 INFO lxccontainer - lxccontainer.c:do_lxcapi_start:979 - Set process title to [lxc monitor] /var/lib/lxc container
lxc-start container 20230628041657.418 DEBUG lxccontainer - lxccontainer.c:wait_on_daemonized_start:840 - First child 260320 exited
lxc-start container 20230628041657.418 TRACE start - start.c:lxc_start:2114 - Doing lxc_start
lxc-start container 20230628041657.419 INFO lsm - lsm/lsm.c:lsm_init:40 - Initialized LSM security driver AppArmor
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:732 - Initialized LSM
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to STARTING
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:738 - Set container state to "STARTING"
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:794 - Set environment variables
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:799 - Ran pre-start hooks
lxc-start container 20230628041657.419 TRACE start - start.c:setup_signal_fd:331 - Created signal file descriptor 8
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:808 - Set up signal fd
lxc-start container 20230628041657.419 DEBUG terminal - terminal.c:lxc_terminal_peer_default:665 - No such device - The process does not have a controlling terminal
lxc-start container 20230628041657.419 DEBUG terminal - terminal.c:lxc_terminal_create_log_file:837 - Using "console.log" as terminal log file
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:816 - Created console
lxc-start container 20230628041657.419 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:351 - Loaded bpf program: (null)
lxc-start container 20230628041657.419 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_devices_cgroup_supported:547 - The bpf device cgroup is supported
lxc-start container 20230628041657.419 TRACE cgroup - cgroups/cgroup.c:cgroup_init:41 - Initialized cgroup driver cgfsng
lxc-start container 20230628041657.419 TRACE cgroup - cgroups/cgroup.c:cgroup_init:48 - Running with unified cgroup layout
lxc-start container 20230628041657.419 TRACE start - start.c:lxc_init:823 - Initialized cgroup driver
lxc-start container 20230628041657.419 DEBUG seccomp - seccomp.c:parse_config_v2:656 - Host native arch is [3221225534]
lxc-start container 20230628041657.419 TRACE seccomp - seccomp.c:get_new_ctx:470 - Added arch 2 to main seccomp context
lxc-start container 20230628041657.419 TRACE seccomp - seccomp.c:get_new_ctx:478 - Removed native arch from main seccomp context
lxc-start container 20230628041657.419 TRACE seccomp - seccomp.c:get_new_ctx:470 - Added arch 3 to main seccomp context
lxc-start container 20230628041657.419 TRACE seccomp - seccomp.c:get_new_ctx:478 - Removed native arch from main seccomp context
lxc-start container 20230628041657.419 TRACE seccomp - seccomp.c:get_new_ctx:483 - Arch 4 already present in main seccomp context
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "[all]"
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
lxc-start container 20230628041657.419 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
lxc-start container 20230628041657.420 INFO seccomp - seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
lxc-start container 20230628041657.420 TRACE seccomp - seccomp.c:parse_config_v2:1027 - Merged first compat seccomp context into main context
lxc-start container 20230628041657.420 TRACE seccomp - seccomp.c:parse_config_v2:1043 - Merged second compat seccomp context into main context
lxc-start container 20230628041657.420 TRACE start - start.c:lxc_init:828 - Read seccomp policy
lxc-start container 20230628041657.883 TRACE start - start.c:lxc_init:835 - Initialized LSM
lxc-start container 20230628041657.883 INFO start - start.c:lxc_init:837 - Container "container" is initialized
lxc-start container 20230628041657.934 INFO cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1368 - The monitor process uses "lxc.monitor.container" as cgroup
lxc-start container 20230628041657.935 TRACE cgfsng - cgroups/cgfsng.c:__cgfsng_delegate_controllers:3087 - Enable "+cpuset +cpu +io +memory +hugetlb +pids +rdma" controllers in the unified cgroup "/sys/fs/cgroup//cgroup.subtree_control"
lxc-start container 20230628041657.935 TRACE sync - sync.c:lxc_sync_init:113 - Initialized synchronization infrastructure
lxc-start container 20230628041657.935 INFO cgfsng - cgroups/cgfsng.c:cgfsng_payload_create:1471 - The container process uses "lxc.payload.container" as cgroup
lxc-start container 20230628041657.936 TRACE start - start.c:lxc_spawn:1663 - Spawned container directly into target cgroup via cgroup2 fd 17
lxc-start container 20230628041657.936 TRACE start - start.c:lxc_spawn:1684 - Cloned child process 260397
lxc-start container 20230628041657.936 TRACE utils - utils.c:lxc_can_use_pidfd:1880 - Kernel supports pidfds
lxc-start container 20230628041657.936 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNS
lxc-start container 20230628041657.936 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWPID
lxc-start container 20230628041657.936 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWUTS
lxc-start container 20230628041657.936 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWIPC
lxc-start container 20230628041657.936 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWNET
lxc-start container 20230628041657.936 INFO start - start.c:lxc_spawn:1700 - Cloned CLONE_NEWCGROUP
lxc-start container 20230628041657.936 TRACE start - start.c:lxc_spawn:1663 - Spawned container directly into target cgroup via cgroup2 fd 17
lxc-start container 20230628041657.936 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved mnt namespace via fd 19
lxc-start container 20230628041657.936 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved pid namespace via fd 20
lxc-start container 20230628041657.936 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved uts namespace via fd 21
lxc-start container 20230628041657.936 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved ipc namespace via fd 22
lxc-start container 20230628041657.936 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved net namespace via fd 23
lxc-start container 20230628041657.936 DEBUG start - start.c:lxc_try_preserve_namespaces:166 - Preserved cgroup namespace via fd 24
lxc-start container 20230628041657.936 TRACE sync - sync.c:lxc_sync_wake_child:96 - Child waking parent with sequence startup
lxc-start container 20230628041657.936 TRACE sync - sync.c:lxc_sync_wait_child:90 - Child waiting for parent with sequence configure
lxc-start container 20230628041657.937 TRACE sync - sync.c:lxc_sync_wait_parent:84 - Parent waiting for child with sequence startup
lxc-start container 20230628041657.937 TRACE sync - sync.c:lxc_sync_barrier_parent:64 - Child waking parent with sequence configure and waiting for sequence post-configure
lxc-start container 20230628041657.937 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2852 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:__cgfsng_delegate_controllers:3087 - Enable "+cpuset +cpu +io +memory +hugetlb +pids +rdma" controllers in the unified cgroup "/sys/fs/cgroup//cgroup.subtree_control"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.deny" to "a"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c *:* m"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "b *:* m"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 1:3 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 1:5 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 1:7 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 5:0 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 5:1 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 5:2 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 1:8 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 1:9 rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 136:* rwm"
lxc-start container 20230628041657.937 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2953 - Set "devices.allow" to "c 10:229 rwm"
lxc-start container 20230628041657.937 INFO cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits:2956 - Limits for the unified cgroup hierarchy have been setup
lxc-start container 20230628041657.937 TRACE start - start.c:lxc_spawn:1780 - Allocated new network namespace id
lxc-start container 20230628041657.941 INFO network - network.c:instantiate_veth:285 - Retrieved mtu 1500 from virbr1
lxc-start container 20230628041657.942 INFO network - network.c:instantiate_veth:333 - Attached "veth-c04" to bridge "virbr1"
lxc-start container 20230628041657.942 DEBUG network - network.c:instantiate_veth:449 - Instantiated veth tunnel "veth-c04 <--> vethuB4QBS"
lxc-start container 20230628041657.942 TRACE network - network.c:lxc_network_send_to_child:3565 - Sent network device name "vethuB4QBS" to child
lxc-start container 20230628041657.942 TRACE sync - sync.c:lxc_sync_barrier_child:71 - Parent waking child with sequence post-configure and waiting with sequence cgroup
lxc-start container 20230628041657.942 TRACE network - network.c:lxc_network_recv_from_parent:3592 - Received network device name "vethuB4QBS" from parent
lxc-start container 20230628041657.942 TRACE sync - sync.c:lxc_sync_barrier_parent:64 - Child waking parent with sequence cgroup and waiting for sequence cgroup-unshare
lxc-start container 20230628041657.942 TRACE sync - sync.c:lxc_sync_barrier_child:71 - Parent waking child with sequence cgroup-unshare and waiting with sequence cgroup-limits
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.942 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3052 - Recursively turned old root mount tree into dependent mount
lxc-start container 20230628041657.943 TRACE conf - conf.c:turn_into_dependent_mounts:3054 - Turned all mount table entries into dependent mount
lxc-start container 20230628041657.945 DEBUG storage - storage/storage.c:storage_query:233 - Detected rootfs type "lvm"
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:327 - Trying to mount "/dev/vgdata/container"->"/usr/lib/x86_64-linux-gnu/lxc/rootfs" with FSType "btrfs"
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:336 - Invalid argument - Failed to mount
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:327 - Trying to mount "/dev/vgdata/container"->"/usr/lib/x86_64-linux-gnu/lxc/rootfs" with FSType "ext3"
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:336 - Device or resource busy - Failed to mount
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:327 - Trying to mount "/dev/vgdata/container"->"/usr/lib/x86_64-linux-gnu/lxc/rootfs" with FSType "ext2"
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:336 - Device or resource busy - Failed to mount
lxc-start container 20230628041657.945 DEBUG storage_utils - storage/storage_utils.c:find_fstype_cb:327 - Trying to mount "/dev/vgdata/container"->"/usr/lib/x86_64-linux-gnu/lxc/rootfs" with FSType "ext4"
lxc-start container 20230628041657.945 INFO storage_utils - storage/storage_utils.c:find_fstype_cb:343 - Mounted "/dev/vgdata/container" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs", with FSType "ext4"
lxc-start container 20230628041657.945 DEBUG conf - conf.c:lxc_mount_rootfs:1261 - Mounted rootfs "/dev/vgdata/container" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
lxc-start container 20230628041657.945 INFO conf - conf.c:setup_utsname:749 - Set hostname to "container"
lxc-start container 20230628041657.974 TRACE network - network.c:__instantiate_ns_common:884 - Renamed network device from "vethuB4QBS" to "eth%d"
lxc-start container 20230628041657.974 DEBUG network - network.c:setup_hw_addr:3367 - Mac address "54:52:00:12:01:04" on "eth0" has been setup
lxc-start container 20230628041657.975 DEBUG network - network.c:lxc_network_setup_in_child_namespaces_common:3517 - Network device "eth0" has been setup
lxc-start container 20230628041657.975 INFO network - network.c:lxc_setup_network_in_child_namespaces:3539 - Network has been setup
lxc-start container 20230628041657.975 TRACE network - network.c:lxc_network_send_name_and_ifindex_to_parent:3625 - Sent network device names and ifindices to parent
lxc-start container 20230628041657.975 TRACE conf - conf.c:lxc_setup:3310 - Created detached mount for container's console "/dev/pts/14"
lxc-start container 20230628041657.975 INFO conf - conf.c:mount_autodev:1056 - Preparing "/dev"
lxc-start container 20230628041657.975 DEBUG conf - conf.c:mount_autodev:1059 - Using mount options: size=500000,mode=755
lxc-start container 20230628041657.975 TRACE conf - conf.c:mount_autodev:1089 - Mounted tmpfs on "/usr/lib/x86_64-linux-gnu/lxc/rootfs"
lxc-start container 20230628041657.975 INFO conf - conf.c:mount_autodev:1106 - Prepared "/dev"
lxc-start container 20230628041657.976 DEBUG conf - conf.c:mount_entry:1945 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
lxc-start container 20230628041657.976 DEBUG conf - conf.c:mount_entry:1964 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
lxc-start container 20230628041657.976 DEBUG conf - conf.c:mount_entry:2008 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
lxc-start container 20230628041657.976 DEBUG conf - conf.c:mount_entry:2008 - Mounted "proc" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/proc" with filesystem type "proc"
lxc-start container 20230628041657.977 DEBUG conf - conf.c:mount_entry:2008 - Mounted "sys" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/.lxc/sys" with filesystem type "sysfs"
lxc-start container 20230628041657.977 DEBUG cgfsng - cgroups/cgfsng.c:__cg_mount_direct:1789 - Mounted "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/cgroup" with cgroup filesystem type cgroup2
lxc-start container 20230628041657.977 INFO conf - conf.c:run_script_argv:330 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "container", config section "lxc"
lxc-start container 20230628041658.450 INFO conf - conf.c:lxc_fill_autodev:1149 - Populating "/dev"
lxc-start container 20230628041658.454 DEBUG conf - conf.c:lxc_fill_autodev:1159 - Created device node "full"
lxc-start container 20230628041658.456 DEBUG conf - conf.c:lxc_fill_autodev:1159 - Created device node "null"
lxc-start container 20230628041658.457 DEBUG conf - conf.c:lxc_fill_autodev:1159 - Created device node "random"
lxc-start container 20230628041658.458 DEBUG conf - conf.c:lxc_fill_autodev:1159 - Created device node "tty"
lxc-start container 20230628041658.458 DEBUG conf - conf.c:lxc_fill_autodev:1159 - Created device node "urandom"
lxc-start container 20230628041658.459 DEBUG conf - conf.c:lxc_fill_autodev:1159 - Created device node "zero"
lxc-start container 20230628041658.460 INFO conf - conf.c:lxc_fill_autodev:1221 - Populated "/dev"
lxc-start container 20230628041658.461 INFO utils - utils.c:lxc_mount_proc_if_needed:1256 - I am 1, /proc/self points to "1"
lxc-start container 20230628041658.463 DEBUG conf - conf.c:lxc_setup_ttydir_console:1711 - Created directory for console and tty devices at "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc"
lxc-start container 20230628041658.466 DEBUG conf - conf.c:lxc_setup_ttydir_console:1745 - Moved mount "/dev/pts/14" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc/console"
lxc-start container 20230628041658.471 DEBUG conf - conf.c:lxc_setup_ttydir_console:1765 - Mounted "/dev/pts/14" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc/console"
lxc-start container 20230628041658.471 DEBUG conf - conf.c:lxc_setup_ttydir_console:1767 - Console has been setup under "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/lxc/console" and mounted to "/usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/console"
lxc-start container 20230628041658.571 TRACE conf - conf.c:lxc_pivot_root:1434 - pivot_root("/usr/lib/x86_64-linux-gnu/lxc/rootfs") successful
lxc-start container 20230628041658.605 DEBUG conf - conf.c:lxc_setup_devpts_child:1547 - Mount new devpts instance with options "gid=5,newinstance,ptmxmode=0666,mode=0620,max=1024"
lxc-start container 20230628041658.607 TRACE conf - conf.c:lxc_setup_devpts_child:1560 - Sent devpts file descriptor 21 to parent
lxc-start container 20230628041658.609 DEBUG conf - conf.c:lxc_setup_devpts_child:1575 - Created dummy "/dev/ptmx" file as bind mount target
lxc-start container 20230628041658.610 DEBUG conf - conf.c:lxc_setup_devpts_child:1580 - Bind mounted "/dev/pts/ptmx" to "/dev/ptmx"
lxc-start container 20230628041658.620 DEBUG conf - conf.c:lxc_allocate_ttys:936 - Created tty "/dev/pts/0" with ptx fd 21 and pty fd 22
lxc-start container 20230628041658.626 DEBUG conf - conf.c:lxc_allocate_ttys:936 - Created tty "/dev/pts/1" with ptx fd 23 and pty fd 24
lxc-start container 20230628041658.633 DEBUG conf - conf.c:lxc_allocate_ttys:936 - Created tty "/dev/pts/2" with ptx fd 25 and pty fd 26
lxc-start container 20230628041658.639 DEBUG conf - conf.c:lxc_allocate_ttys:936 - Created tty "/dev/pts/3" with ptx fd 27 and pty fd 28
lxc-start container 20230628041658.640 INFO conf - conf.c:lxc_allocate_ttys:953 - Finished creating 4 tty devices
lxc-start container 20230628041658.641 TRACE conf - conf.c:lxc_send_ttys_to_parent:994 - Sent tty "/dev/pts/0" with ptx fd 21 and pty fd 22 to parent
lxc-start container 20230628041658.641 TRACE conf - conf.c:lxc_send_ttys_to_parent:994 - Sent tty "/dev/pts/1" with ptx fd 23 and pty fd 24 to parent
lxc-start container 20230628041658.642 TRACE conf - conf.c:lxc_send_ttys_to_parent:994 - Sent tty "/dev/pts/2" with ptx fd 25 and pty fd 26 to parent
lxc-start container 20230628041658.643 TRACE conf - conf.c:lxc_send_ttys_to_parent:994 - Sent tty "/dev/pts/3" with ptx fd 27 and pty fd 28 to parent
lxc-start container 20230628041658.643 TRACE conf - conf.c:lxc_send_ttys_to_parent:1001 - Sent 4 ttys to parent
lxc-start container 20230628041658.646 DEBUG conf - conf.c:lxc_setup_ttys:867 - Bind mounted "/dev/pts/0" onto "/dev/lxc/tty1"
lxc-start container 20230628041658.649 DEBUG conf - conf.c:lxc_setup_ttys:867 - Bind mounted "/dev/pts/1" onto "/dev/lxc/tty2"
lxc-start container 20230628041658.651 DEBUG conf - conf.c:lxc_setup_ttys:867 - Bind mounted "/dev/pts/2" onto "/dev/lxc/tty3"
lxc-start container 20230628041658.654 DEBUG conf - conf.c:lxc_setup_ttys:867 - Bind mounted "/dev/pts/3" onto "/dev/lxc/tty4"
lxc-start container 20230628041658.655 INFO conf - conf.c:lxc_setup_ttys:898 - Finished setting up 4 /dev/tty<N> device(s)
lxc-start container 20230628041658.657 INFO conf - conf.c:setup_personality:1611 - Set personality to "0x0"
lxc-start container 20230628041658.658 DEBUG conf - conf.c:setup_caps:2419 - Dropped mac_admin (33) capability
lxc-start container 20230628041658.658 DEBUG conf - conf.c:setup_caps:2419 - Dropped mac_override (32) capability
lxc-start container 20230628041658.659 DEBUG conf - conf.c:setup_caps:2419 - Dropped sys_time (25) capability
lxc-start container 20230628041658.659 DEBUG conf - conf.c:setup_caps:2419 - Dropped sys_module (16) capability
lxc-start container 20230628041658.660 DEBUG conf - conf.c:setup_caps:2419 - Dropped sys_rawio (17) capability
lxc-start container 20230628041658.660 DEBUG conf - conf.c:setup_caps:2422 - Capabilities have been setup
lxc-start container 20230628041658.661 NOTICE conf - conf.c:lxc_setup:3446 - The container "container" is set up
lxc-start container 20230628041658.662 TRACE apparmor - lsm/apparmor.c:apparmor_process_label_fd_get:1136 - On-exec not supported with AppArmor
lxc-start container 20230628041658.664 TRACE apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1156 - Changing AppArmor profile on exec not supported
lxc-start container 20230628041658.666 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set_at:1169 - Set AppArmor label to "lxc-container_</var/lib/lxc>//&:lxc-container_<-var-lib-lxc>:"
lxc-start container 20230628041658.667 INFO apparmor - lsm/apparmor.c:apparmor_process_label_set:1225 - Changed AppArmor profile to lxc-container_</var/lib/lxc>//&:lxc-container_<-var-lib-lxc>:
#
# pseudo filter code start
#
# filter for arch x86_64 (3221225534)
if ($arch == 3221225534)
# filter for syscall "finit_module" (313) [priority: 65535]
if ($syscall == 313)
action ERRNO(1);
# filter for syscall "open_by_handle_at" (304) [priority: 65535]
if ($syscall == 304)
action ERRNO(1);
# filter for syscall "kexec_load" (246) [priority: 65535]
if ($syscall == 246)
action ERRNO(1);
# filter for syscall "delete_module" (176) [priority: 65535]
if ($syscall == 176)
action ERRNO(1);
# filter for syscall "init_module" (175) [priority: 65535]
if ($syscall == 175)
action ERRNO(1);
# filter for syscall "umount2" (166) [priority: 65533]
if ($syscall == 166)
if ($a1.hi32 & 0x00000000 == 0)
if ($a1.lo32 & 0x00000001 == 1)
action ERRNO(13);
# default action
action ALLOW;
# filter for arch x86 (1073741827)
if ($arch == 1073741827)
# filter for syscall "finit_module" (350) [priority: 65535]
if ($syscall == 350)
action ERRNO(1);
# filter for syscall "open_by_handle_at" (342) [priority: 65535]
if ($syscall == 342)
action ERRNO(1);
# filter for syscall "kexec_load" (283) [priority: 65535]
if ($syscall == 283)
action ERRNO(1);
# filter for syscall "delete_module" (129) [priority: 65535]
if ($syscall == 129)
action ERRNO(1);
# filter for syscall "init_module" (128) [priority: 65535]
if ($syscall == 128)
action ERRNO(1);
# filter for syscall "umount2" (52) [priority: 65534]
if ($syscall == 52)
if ($a1 & 0x00000001 == 1)
action ERRNO(13);
# default action
action ALLOW;
# filter for arch x32 (3221225534)
if ($arch == 3221225534)
# filter for syscall "kexec_load" (1073742352) [priority: 65535]
if ($syscall == 1073742352)
action ERRNO(1);
# filter for syscall "finit_module" (1073742137) [priority: 65535]
if ($syscall == 1073742137)
action ERRNO(1);
# filter for syscall "open_by_handle_at" (1073742128) [priority: 65535]
if ($syscall == 1073742128)
action ERRNO(1);
# filter for syscall "delete_module" (1073742000) [priority: 65535]
if ($syscall == 1073742000)
action ERRNO(1);
# filter for syscall "init_module" (1073741999) [priority: 65535]
if ($syscall == 1073741999)
action ERRNO(1);
# filter for syscall "umount2" (1073741990) [priority: 65534]
if ($syscall == 1073741990)
if ($a1 & 0x00000001 == 1)
action ERRNO(13);
# default action
action ALLOW;
# invalid architecture action
action KILL;
#
# pseudo filter code end
#
lxc-start container 20230628041658.714 TRACE sync - sync.c:lxc_sync_barrier_parent:64 - Child waking parent with sequence cgroup-limits and waiting for sequence ready-start
lxc-start container 20230628041658.718 WARN cgfsng - cgroups/cgfsng.c:cgfsng_setup_limits_legacy:2852 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
lxc-start container 20230628041658.720 TRACE start - start.c:lxc_spawn:1831 - Set up legacy device cgroup controller limits
lxc-start container 20230628041658.721 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type a, major -1, minor -1, access , allow -1, global_rule 0
lxc-start container 20230628041658.722 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major -1, minor -1, access m, allow 1, global_rule -1
lxc-start container 20230628041658.723 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type b, major -1, minor -1, access m, allow 1, global_rule -1
lxc-start container 20230628041658.724 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 1, minor 3, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.724 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 1, minor 5, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.725 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 1, minor 7, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.725 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 5, minor 0, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.726 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 5, minor 1, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.727 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 5, minor 2, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.728 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 1, minor 8, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.728 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 1, minor 9, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.729 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 136, minor -1, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.729 TRACE cgfsng - cgroups/cgfsng.c:cgfsng_devices_activate:3007 - Added rule to bpf device program: type c, major 10, minor 229, access rwm, allow 1, global_rule -1
lxc-start container 20230628041658.730 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_finalize:313 - Implementing allowlist bpf device cgroup program
lxc-start container 20230628041658.773 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_load_kernel:351 - Loaded bpf program: (null)
lxc-start container 20230628041658.776 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_cgroup_attach:407 - Loaded and attached bpf program to cgroup /sys/fs/cgroup//lxc.payload.container
lxc-start container 20230628041658.777 TRACE start - start.c:lxc_spawn:1837 - Set up cgroup2 device controller limits
lxc-start container 20230628041658.777 TRACE start - start.c:lxc_spawn:1854 - Finished setting up cgroups
lxc-start container 20230628041658.778 TRACE sync - sync.c:lxc_sync_barrier_child:71 - Parent waking child with sequence ready-start and waiting with sequence restart
lxc-start container 20230628041658.785 NOTICE utils - utils.c:lxc_setgroups:1420 - Dropped additional groups
lxc-start container 20230628041658.786 NOTICE start - start.c:start:2087 - Exec'ing "/sbin/init"
lxc-start container 20230628041658.820 TRACE conf - conf.c:lxc_setup_devpts_parent:1493 - Received devpts file descriptor 13 from child
lxc-start container 20230628041658.822 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:245 - index: 0
lxc-start container 20230628041658.823 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:246 - ifindex: 2
lxc-start container 20230628041658.824 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:250 - type: veth
lxc-start container 20230628041658.824 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:253 - veth pair: veth-c04
lxc-start container 20230628041658.825 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:261 - host side ifindex for veth device: 43
lxc-start container 20230628041658.825 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:310 - flags: up
lxc-start container 20230628041658.826 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:314 - link: virbr1
lxc-start container 20230628041658.826 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:318 - l2proxy: false
lxc-start container 20230628041658.826 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:321 - name: eth0
lxc-start container 20230628041658.827 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:324 - hwaddr: 54:52:00:12:01:04
lxc-start container 20230628041658.827 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:335 - ipv4 gateway auto: false
lxc-start container 20230628041658.828 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:338 - ipv4 gateway dev: false
lxc-start container 20230628041658.829 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:344 - ipv4 gateway: 192.168.12.1
lxc-start container 20230628041658.830 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:351 - ipv4 addr: 192.168.12.104
lxc-start container 20230628041658.830 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:354 - ipv6 gateway auto: false
lxc-start container 20230628041658.831 TRACE confile_utils - confile_utils.c:lxc_log_configured_netdevs:357 - ipv6 gateway dev: false
lxc-start container 20230628041658.832 TRACE start - start.c:lxc_recv_ttys_from_child:1434 - Received pty with ptx fd 25 and pty fd 26 from child
lxc-start container 20230628041658.832 TRACE start - start.c:lxc_recv_ttys_from_child:1434 - Received pty with ptx fd 27 and pty fd 28 from child
lxc-start container 20230628041658.833 TRACE start - start.c:lxc_recv_ttys_from_child:1434 - Received pty with ptx fd 29 and pty fd 30 from child
lxc-start container 20230628041658.834 TRACE start - start.c:lxc_recv_ttys_from_child:1434 - Received pty with ptx fd 31 and pty fd 32 from child
lxc-start container 20230628041658.834 TRACE start - start.c:lxc_recv_ttys_from_child:1440 - Received 4 ttys from child
lxc-start container 20230628041658.835 NOTICE start - start.c:post_start:2098 - Started "/sbin/init" with pid "260397"
lxc-start container 20230628041658.836 TRACE start - start.c:lxc_serve_state_socket_pair:502 - Sent container state "RUNNING" to 5
lxc-start container 20230628041658.838 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to RUNNING
lxc-start container 20230628041658.838 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc-start container 20230628041658.839 TRACE lxccontainer - lxccontainer.c:wait_on_daemonized_start:864 - Container is in "RUNNING" state
lxc-start container 20230628041658.843 TRACE start - start.c:lxc_poll:592 - Mainloop is ready
lxc-start container 20230628041658.844 NOTICE start - start.c:signal_handler:404 - Received 17 from pid 260322 instead of container init 260397
lxc-start container 20230628041658.956 DEBUG start - start.c:signal_handler:422 - Container init process 260397 exited
lxc-start container 20230628041658.959 TRACE start - start.c:lxc_poll:605 - Closed console mainloop
lxc-start container 20230628041658.961 TRACE start - start.c:lxc_poll:610 - Closed mainloop
lxc-start container 20230628041658.962 TRACE start - start.c:lxc_poll:613 - Closed signal file descriptor 8
lxc-start container 20230628041658.967 TRACE network - network.c:lxc_restore_phys_nics_to_netns:3306 - Moving physical network devices back to parent network namespace
lxc-start container 20230628041658.973 INFO error - error.c:lxc_error_set_and_log:28 - Child <260397> ended on error (1)
lxc-start container 20230628041658.985 INFO network - network.c:lxc_delete_network_priv:3232 - Removed interface "veth-c04" from "virbr1"
lxc-start container 20230628041658.985 DEBUG network - network.c:lxc_delete_network:3672 - Deleted network devices
lxc-start container 20230628041658.985 TRACE start - start.c:lxc_serve_state_clients:438 - Set container state to STOPPING
lxc-start container 20230628041658.985 TRACE start - start.c:lxc_serve_state_clients:441 - No state clients registered
lxc-start container 20230628041658.505 TRACE cgroup2_devices - cgroups/cgroup2_devices.c:bpf_program_cgroup_detach:442 - Detached bpf program from cgroup /sys/fs/cgroup//lxc.payload.container
lxc-start container 20230628041658.527 TRACE start - start.c:lxc_end:936 - Closed command socket
lxc-start container 20230628041658.527 TRACE start - start.c:lxc_end:947 - Set container state to "STOPPED"
lxc-start container 20230628041658.528 INFO conf - conf.c:run_script_argv:330 - Executing script "/usr/share/lxcfs/lxc.reboot.hook" for container "container", config section "lxc"
The container OS is Debian 11, too.
Any help is much appreciated, thanks.