Forkexec - Failed to load config file

brettmilford · January 23, 2023, 12:55am

Hi everyone,

I have a problem where lxc exec gives the error

Failed to retrieve PID of executing child process

After setting debug
snap set lxd daemon.debug=true
snap set lxd daemon.verbose=true
systemctl reload snap.lxd.daemon

And trying again I see in /var/snap/lxd/common/lxd/logs/forkexec.log

Failed to load config file /var/snap/lxd/common/lxd/logs/ams-ceqsttroh003taa8cu10/lxc.conf for /var/snap/lxd/common/lxd/containers/ams-ceqsttroh003taa8cu10

Indeed it appears that lxd.conf is missing at this location for all the containers on this node and the only file present for the other containers is lxc.log

From snap list:
lxd 4.0.9-a29c6f1 24065 5.0/stable canonical** disabled,in-cohort
lxd 5.0.1-9dcf35b 23545 5.0/stable canonical** in-cohort

Any thoughts on how this came to be?

Thanks.

tomp · January 23, 2023, 9:02am

Sounds like:

github.com/lxc/lxd

QEMU failed to run a feature check for virtual-machine

opened 06:30AM - 11 Aug 22 UTC

closed 04:36AM - 12 Aug 22 UTC

gozssky

Incomplete

# Required information * Distribution: CentOS 7 * Distribution version: 7 * The output of "lxc info" or if that fails: * Kernel version: 3.10.0-957.el7.x86_64 * LXC version: 5.3 * LXD version: 5.3 * Storage backend in use: lvm # Issue description LXD failed to run a feature check for `virtual-machine`. As a result, I am unable to create a virtual machine. This node supports creating virtual machines. There are 4 existing virtual machines running on this node. They were created by LXD. LXD log: ``` Aug 11 14:01:08 KS-22 lxd.daemon[3064906]: time="2022-08-11T14:01:08+08:00" level=error msg="Unable to run io_uring check during QEMU initialization: open /tmp/464998873: no such file or directory" Aug 11 14:01:08 KS-22 lxd.daemon[3064906]: time="2022-08-11T14:01:08+08:00" level=warning msg="Instance type not operational" driver=qemu err="QEMU failed to run a feature check" type=virtual-machine ``` LXC error when creating a virtual machine: ``` [root@KS-22 ubuntu]# lxc launch images:ubuntu/22.04/cloud dev04 --vm --storage nvme0n1 --target=ks-22 Creating dev04 Error: Failed instance creation: Failed creating instance record: Instance type "virtual-machine" is not supported on this server: QEMU failed to run a feature check ``` # Steps to reproduce 1. Step one 2. Step two 3. Step three # Information to attach - [ ] Any relevant kernel output (`dmesg`) - [ ] Container log (`lxc info NAME --show-log`) - [ ] Container configuration (`lxc config show NAME --expanded`) - [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log) - [ ] Output of the client with --debug - [ ] Output of the daemon with --debug (alternatively output of `lxc monitor` while reproducing the issue)

brettmilford · January 23, 2023, 9:42pm

Sort of. We aren’t in tmp though.

tomp · January 24, 2023, 8:07am

Does rebooting the host fix it for some time and then it happens again?

brettmilford · January 27, 2023, 12:52am

Restarting snap.lxd.daemon appears to restore exec (at least the 1 container we exec’d has its files back) but this also restarts containers which isn’t an option for the user.

Is it possible to prompt LXD to recreate those files? It is at least possible with a service and container restart.

tomp · January 27, 2023, 11:53am

See if this helps Lxc unable to connect to running container - #3 by CanuteTheGreat

brettmilford · February 9, 2023, 8:43am

Hey @tomp thanks for the assist so far.

So it looks like that link indicates a full restart of LXD and the containers is required.

Trying to avoid this I tried this based in info documented at [1]

Send SIGQUIT to the lxd daemon
sudo kill -QUIT $(pidof -s lxd)
Start the lxd daemon again.
sudo systemctl start snap.lxd.daemon

According to [1] this should run the startup sequence again where directory structures are checked.

Having tried this, it seems to have a similar effect except now the error message on exec is:
“Error: Instance not found”
Even though the container is listed on the host as running.

Thoughts on this?

[1] https://linuxcontainers.org/lxd/docs/master/daemon-behavior/

Thanks

tomp · February 9, 2023, 8:45am

You can cleanly reload LXD using:

sudo systemctl reload snap.lxd.daemon

Rather than uncleanly killing it (which may cause inconsistency or DB data loss).

tomp · February 9, 2023, 8:45am

Please show lxc list and lxc project list output.