Time to fix this once and for all : Failed preparing container for start: Failed to run: zfs set mountpoint=none canmount=noauto

restarting or stoping a cointainer and starting it or restoring happens ,
I will write a script to do thousands of times.

Because it had happened about 5 times already within 4 months , and we didn’t stop much after the first one.

1 Like

I am ready , wish me badluck

from pylxd import Client
import os,random
c = Client()

instances = c.instances.all()
instance_names= [x.name for x in instances]
inst_name = random.choice(instance_names)
for i in range(0,1000):
    print("stopping  ", inst_name)
    os.popen(f"lxc stop {inst_name} -f").read()
    os.popen(f"lxc start {inst_name}").read()
    os.popen(f"lxc restart {inst_name} -f").read()

You probably want to add a stop -f and restart -f so that in case the container isn’t ready to handle signals to shutdown, it wont hang.

thanks , gonna try now.

over 100 times , still doing fine.

1000 times of starting stoping and restarting done , no problem at all …
now i am going to test restoring from snapshots . Tired for today , gonna try next days.

Stars haven’t align enough to cause errors i think.

Same problem happend in lxd 4.22 with zfs, I can’t find the reason why it happened:

root@rainyun:~# lxc start bt611252
Error: Failed to run: zfs set mountpoint=none canmount=noauto data/containers/bt611252: umount: /var/snap/lxd/common/shmounts/storage-pools/data/containers/bt611252: no mount point specified.
cannot unmount ‘/var/snap/lxd/common/shmounts/storage-pools/data/containers/bt611252’: umount failed
Try lxc info --show-log bt611252 for more info

What host OS are you using?

Same here :confused:

Everything began with a snapshot on a running container (don’t know if it’s the root cause) :

lxc snapshot gautier
Error: Create instance snapshot (mount source): Failed to run: zfs set mountpoint=none canmount=noauto tank/containers/gautier: umount: /var/snap/lxd/common/shmounts/storage-pools/default/containers/gautier: no mount point specified.
cannot unmount '/var/snap/lxd/common/shmounts/storage-pools/default/containers/gautier': umount failed

Then I stop instance, no error but snapshot failed too.

Now I can’t start container :cry:, always the same error :

lxc start gautier
Error: Failed to run: zfs set mountpoint=none canmount=noauto tank/containers/gautier: umount: /var/snap/lxd/common/shmounts/storage-pools/default/containers/gautier: no mount point specified.
cannot unmount '/var/snap/lxd/common/shmounts/storage-pools/default/containers/gautier': umount failed
Try `lxc info --show-log gautier` for more info

zfs list show some container rootfs have mount point, some not :thinking: :

sudo zfs list
[sudo] password for ydavid:
NAME                                                                                   USED  AVAIL     REFER  MOUNTPOINT
tank                                                                                   403G  1,18T       96K  none
tank/containers                                                                        398G  1,18T       96K  none
tank/containers/akkalia-php73                                                         42,9G  1,18T     43,4G  none
tank/containers/php73                                                                 10,5G  1,18T     10,5G  /var/snap/lxd/common
tank/containers/gautier                                                               11,8G  1,18T     12,3G  none

gautier one was not mounted, try to set mount point sudo zfs set mountpoint=/var/snap/lxd/common/shmounts/storage-pools/default/containers/gautier canmount=noauto tank/containers/gautier, then try to start it : failure

What can I do ?
I Can’t reboot this server now

My OS : Ubuntu 20.04
LXC : 4.23 (snap)


We continue to try and workaround the snap/zfs issue, the latest attempt is here and will be in LXD 4.24 (if not already cherry-picked into LXD 4.23 by @stgraber ).

As for what you can do about this now, it seems to be triggered when the LXD snap base package (core20) refreshes (from chatting with @stgraber) seemingly causing an issue with the ZFS tooling not recognising the existing mounts inside the snap mount namespace.

Rebooting fixes the issue as it clears the mount table inside the snap mount namespace and synchronises it with the host. The latest PR tries to go further in avoiding the issue by trying to avoid getting the ZFS tooling from executing mounts/unmounts by setting the mount mode to legacy.

As to why the snap mount namespace gets out of sync, this may well be a snap bug, but so far a reproducer has not been found.

Any way to manually sync the mounts in the snap namespace @stgraber ?

Very welcome move, taking back the control, visibility and manageability and not being lost in snap user spaces / outer space.

Unfortunately yes, second time snap make me a bad joke with LXD :confused:

Hopefully it will improve things, but we are relying on ZFS not trying to perform any mount/unmount operations, which according to the manual legacy will stop it doing so, but then really none should have had the same effect and apparently it didn’t so we’ll have to see.

Good transition, which is the filesystem you recommand for a production server ?

And for my current problem, my container is unavailable, what do you think, is there any chance there is a command to remap mount point ?
Is 4.24 corrected version will remap the mount points, or is it just a patch to avoid this mess happen ?
When will it be released ?
Or must I start another container and reinstall my website in it (host reboot is not an option for now) ?


Is it now possible to legacy mount a container somewhere on host n order to access files without risk of interfering with LXD?
Or at least a read-only legacy mount for the purpose of outbound copy/rsync files?

Yes you can use the normal mount commands.

This problem strikes again
Error: Failed to run: zfs set mountpoint=legacy canmount=noauto default/containers/training-zulip
I cannot afford a reboot …

lxd --version

I tried this and it won’t help too.

The conversation for this issue has moved to:

There is a non ideal workaround you could try described there:

can’t find how :thinking:
Tried sudo mount tank/containers/odoo-new /mnt/lxd-temp/ but it doesn’t work, can you please help me ?