Time to fix this once and for all : Failed preparing container for start: Failed to run: zfs set mountpoint=none canmount=noauto

LXD
1

This ZFS bug exist in LXD for so long and it is becoming harder to use on production . We have to reboot whole server just to start a container. That is totally killing the purpose of using LXD .
This bug happens frequently and we need to come up as a community to resolve this.

Unless this is fixed , LXD shouldn’t be considered Production Ready.

There are many issues regarding this on the lxd github but main Devs couldn’t able to come up with a proper solution yet.

  • One theory is it is due to snap builds , the distro build is fine ( AFAIR , i cannot find the thread @stgraber mentioned about that ).
  • It was definately due to ZFS ? Then should we stop using it?
  • 4.20 have fixes by @tomp trying to resolve this by not using zfs mount , which seems to work for some but dosen’t work for me .

I love LXD , it make me able to avoid Kubernetes’ complexity , and i am using LXD + PyLXD in a product , as a core feature / lib to provide multiple classrooms to improvished area , to give online education, which is going to launch in a week.

But if this happen during production , i will have a lot of trouble.

So what can we do to avoid this?

  • Provide official Debian Packages?
  • Use Btrfs instead of ZFS ?
  • or Stop using ZFS as default?

Lets discusss.

Error: Failed preparing container for start: Failed to run: zfs set mountpoint=none canmount=noauto default/containers/magento-dev: umount: /var/snap/lxd/common/shmounts/storage-pools/default/containers/magento-dev: no mount point specified.
cannot unmount '/var/snap/lxd/common/shmounts/storage-pools/default/containers/magento-dev': umount failed
Try `lxc info --show-log magento-dev` for more info
2

Please show output of sudo zfs list on the host.

3

i already had rebooted host for now .

this is the container that happened before , after reboot it starts normally.

this is current results.

default/containers/CCOMMERCE1                                                                   696M   251G      910M  none
default/containers/Default-Moodle                                                               590M   251G      642M  none
default/containers/Seafile                                                                     2.07G   251G     2.27G  none
default/containers/blockchain-peer-1                                                           5.78G   251G     9.57G  none
default/containers/chain-peer-1                                                                2.66G   251G     12.9G  none
default/containers/chain-peer-3                                                                2.64G   251G     12.8G  none
default/containers/clean-groovy1                                                                286M   251G      492M  none
default/containers/database                                                                     865M   251G     1.05G  none
default/containers/development-repo                                                            4.63G   251G     7.40G  none
default/containers/kothte-ever-co                                                              25.8G   251G     26.0G  none
default/containers/moodle-nginx-test-debug-2                                                   56.8M   251G      558M  none
default/containers/zulip-clean                                                                  110M   251G     1.39G  none
default/custom                                                                                   24K   251G       24K  none
default/deleted                                                                                7.38G   251G       24K  none
default/deleted/containers                                                                     8.39M   251G       24K  none
default/deleted/containers/a364dfe9-1cd3-4d01-be39-27ea7486eda2                                3.66M   251G      207M  /var/snap/lxd/common/lxd/storage-pools/default/containers/thirsty-hermann
default/deleted/containers/c5065b4b-2e11-4e41-96f1-c7f190c30533                                4.71M   251G      242M  /var/snap/lxd/common/lxd/storage-pools/default/containers/blockchain-dev-21
default/deleted/custom                                                                           24K   251G       24K  none
default/deleted/images                                                                         7.37G   251G       24K  none
default/deleted/images/0305143b362cd76b78fdbde52cd62c1bad17dbe3a4c424aa028323a41600ae1a         249M   251G      249M  /var/snap/lxd/common/lxd/storage-pools/default/images/0305143b362cd76b78fdbde52cd62c1bad17dbe3a4c424aa028323a41600ae1a
default/deleted/images/0aebdf0c6b135fd6bd4ff5b5e8e9d10f6bcb9ac4f459d3734441b2aed6133749         207M   251G      207M  /var/snap/lxd/common/lxd/storage-pools/default/images/0aebdf0c6b135fd6bd4ff5b5e8e9d10f6bcb9ac4f459d3734441b2aed6133749
default/deleted/images/29b8978ab17db48f4a91ae2ca2874ee78f046fecc675cb677db39f5f9779fd1c         555M   251G      555M  none
default/deleted/images/529063e70b84ea2370805ead1a116352deba39bbfc4297cd0e2299897af98fac         249M   251G      249M  /var/snap/lxd/common/lxd/storage-pools/default/images/529063e70b84ea2370805ead1a116352deba39bbfc4297cd0e2299897af98fac
default/deleted/images/6899d205b9aec137bb8a26928c88b502d15fcbebf15aaec6b652ec6379d2312c         242M   251G      242M  /var/snap/lxd/common/lxd/storage-pools/default/images/6899d205b9aec137bb8a26928c88b502d15fcbebf15aaec6b652ec6379d2312c
default/deleted/images/7e12b43a20d8a2da69bb9d0ca99db1936499f9bd50bb286cabffbf6326e8f4d1         212M   251G      212M  /var/snap/lxd/common/lxd/storage-pools/default/images/7e12b43a20d8a2da69bb9d0ca99db1936499f9bd50bb286cabffbf6326e8f4d1
default/deleted/images/acdca2c8a2034d70d2a04b48f56816085b3f4321908fbbeb379e23cd3f954873        5.08G   251G     5.08G  /var/snap/lxd/common/lxd/storage-pools/default/images/acdca2c8a2034d70d2a04b48f56816085b3f4321908fbbeb379e23cd3f954873
default/deleted/images/aec64fea576d80f4281fc4dee526402e0b5762c11853a78c1ac0ac64753f56e7          27K  95.3M       26K  /var/snap/lxd/common/lxd/storage-pools/default/images/aec64fea576d80f4281fc4dee526402e0b5762c11853a78c1ac0ac64753f56e7
default/deleted/images/aec64fea576d80f4281fc4dee526402e0b5762c11853a78c1ac0ac64753f56e7.block   386M   251G      386M  -
default/deleted/images/ed06053d47a7019c128d175c345bf2d59378d07be81f3380d8293ca23cca400c         249M   251G      249M  /var/snap/lxd/common/lxd/storage-pools/default/images/ed06053d47a7019c128d175c345bf2d59378d07be81f3380d8293ca23cca400c
default/deleted/virtual-machines                                                                 24K   251G       24K  none
default/images                                                                                 18.9G   251G       24K  none
default/images/2012f3f777507159bb8c84595594841c3b59bcef5c3f94fc0ba976f9addd8708                 631M   251G      631M  none
default/images/2fba70a9af05091a585c294726a9065bc294f756e553da2a90c07ef7d0a62883                 631M   251G      631M  none
default/images/4e7aab0f185b77984fcb42d0eb975ce50c2c83e217f1a6a4962f33c206661ee0                1.39G   251G     1.39G  none
default/images/8d89e69a7da64084eabc8734ae3d8adcbc294f15f275673895faafae4aa26d55                 256M   251G      256M  /var/snap/lxd/common/lxd/storage-pools/default/images/8d89e69a7da64084eabc8734ae3d8adcbc294f15f275673895faafae4aa26d55
default/images/c1b1e1df58a0cba4a9456f03bce1dafa2ef4877e1d9259aeaecba18a4d7c7887                 367M   251G      367M  none
default/images/e5b0f357c7a1cb4c2309f47bcfc5d1319714ab75d0327af6c1f2d602ae3adc37                15.7G   251G     15.7G  none
default/virtual-machines                                                                       24.2G   251G       24K  none
default/virtual-machines/classvmin                                                             5.50M  89.9M     5.51M  none
default/virtual-machines/classvmin.block                                                       24.2G   251G     24.5G  -
4

Can you run:

sudo zfs set mountpoint=none,canmount=noauto <dataset>

for each one of the datasets that has a mount point?

I wonder if that is causing ZFS to get confused when running inside the snap mount namespace.

For comparison, this is my dev env:

NAME                                                                                       MOUNTPOINT  CANMOUNT  MOUNTED
zfs                                                                                        none              on       no
zfs/containers                                                                             none              on       no
zfs/containers/test_c1                                                                     none          noauto      yes
zfs/custom                                                                                 none              on       no
zfs/deleted                                                                                none              on       no
zfs/deleted/containers                                                                     none              on       no
zfs/deleted/custom                                                                         none              on       no
zfs/deleted/images                                                                         none              on       no
zfs/deleted/images/069dcc820d85617eef4ed163f4b6c0fc01bf07b10638ff6dc122af79f4e40bfd        none          noauto       no
zfs/deleted/images/069dcc820d85617eef4ed163f4b6c0fc01bf07b10638ff6dc122af79f4e40bfd.block  -                  -        -
zfs/deleted/images/d5b2972fae2bb20ec1db4976b154301366db24aedc7fde9dad892b37329b3f7e        none          noauto       no
zfs/deleted/virtual-machines                                                               none              on       no
zfs/images                                                                                 none              on       no
zfs/images/43d0790025a971f6a3b5caa9b959bc598115f8621c1b6bfe1fbf079c1ca8b23d                none          noauto       no
zfs/images/57198c6ec93dfffe64dc77daef1529f22c3654022c7968b1d24b11c64eb45b39                none          noauto       no
zfs/images/5af500c10166ccb930880a905497be71dc16aa9105c2703343299cd79e4d4c1c                none          noauto       no
zfs/images/5af500c10166ccb930880a905497be71dc16aa9105c2703343299cd79e4d4c1c.block          -                  -        -
zfs/images/771fd3b5b520c63707ec35e7e8dd822f0200ad73b9a2b9715740b5634564a313                none          noauto       no
zfs/images/771fd3b5b520c63707ec35e7e8dd822f0200ad73b9a2b9715740b5634564a313.block          -                  -        -
zfs/images/c42196d8cf695c6c6a1e5877f053d76768c5b8ad26349e29a6ccaf8ebc9a25ca                none          noauto       no
zfs/images/c42196d8cf695c6c6a1e5877f053d76768c5b8ad26349e29a6ccaf8ebc9a25ca.block          -                  -        -
zfs/images/d42f95c41ca31aef9bd8d0c3ceeb7432af07c6dae82450e62a9fdd01bc2b4a36                none          noauto       no
zfs/images/d42f95c41ca31aef9bd8d0c3ceeb7432af07c6dae82450e62a9fdd01bc2b4a36.block          -                  -        -
zfs/images/fcefefd52b451a6307d6b6525c25f890d1ef3afe6abf33f72380c42e178d5795                none          noauto       no
zfs/virtual-machines                                                                       none              on       no
zfs/virtual-machines/test_v1                                                               none          noauto      yes
zfs/virtual-machines/test_v1.block                                                         -                  -        -
5

Discussion moved to

https://github.com/lxc/lxd/issues/7119

6

dateset means zfs/containers ?
for me default/containers

7

Each of the lines from “zfs list” output is a separate dataset.

So for example:

zfs set mountpoint=none,canmount=noauth default/deleted/containers/a364dfe9-1cd3-4d01-be39-27ea7486eda2

Repeat this for every dataset with a mountpoint other than none

8

I think we should discuss here , in the forum.

ah , i will write a quick python script , then.

9

it hast to be /var/snap/lxd/common/lxd/storage-pools/default/containers-snapshot ?

cannot set property for 'default': 'mountpoint' must be an absolute path, 'none', or 'legacy'

cannot set property for 'default/containers': 'mountpoint' must be an absolute path, 'none', or 'legacy'

cannot set property for 'default/containers/CCOMMERCE1': 'mountpoint' must be an absolute path, 'none', or 'legacy'
10

Please show the exact command you typed, as well as the error that came back.

11

sudo zfs set mountpoint=none,canmount=noauto default/containers/moodle-nginx-test

result

cannot set property for 'default/containers/moodle-nginx-test': 'mountpoint' must be an absolute path, 'none', or 'legacy'
12

Try removing ,canmount=noauto

13

its works without canmount=noauto , why? it is not needed?

14

I don’t know why.

But the mountpoint=none is the most important part, lets see if that helps.

15

zfs set mountpoint=none canmount=noauto this works i think because of comma

16 
import os
zfs_datasets = os.popen("zfs list -o name").read().splitlines()[1:]

if __name__ == "__main__":
    for ds in zfs_datasets:
        print ( os.popen (f"zfs set mountpoint=none canmount=noauto {ds}").read())

It works , and here its set now ::

NAME                                                                                           MOUNTPOINT  CANMOUNT  MOUNTED
default                                                                                        none          noauto       no
default/containers                                                                             none          noauto       no
default/containers/CCOMMERCE1                                                                  none          noauto       no
default/containers/Default-Moodle                                                              none          noauto       no
default/containers/ELK                                                                         none          noauto       no
default/containers/Focalboard                                                                  none          noauto       no
default/containers/Grafama                                                                     none          noauto       no
default/containers/MagentoFinalTest                                                            none          noauto       no
default/containers/Moodle                                                                      none          noauto       no
default/containers/Seafile                                                                     none          noauto       no
default/containers/auth                                                                        none          noauto       no
default/containers/blockchain                                                                  none          noauto       no
default/containers/blockchain-dev-1                                                            none          noauto       no
default/containers/blockchain-peer-1                                                           none          noauto       no
default/containers/chain-hub                                                                   none          noauto       no
default/containers/chain-peer-0                                                                none          noauto       no
default/containers/chain-peer-1                                                                none          noauto       no
default/containers/chain-peer-2                                                                none          noauto       no
default/containers/chain-peer-3                                                                none          noauto       no
default/containers/chat-clean-hexcode-zulip                                                    none          noauto       no
default/containers/clean-groovy1                                                               none          noauto       no
default/containers/data-analytics-moodle                                                       none          noauto       no
default/containers/database                                                                    none          noauto       no
default/containers/development-repo                                                            none          noauto       no
default/containers/docs                                                                        none          noauto       no
default/containers/eloquent-kilby                                                              none          noauto       no
default/containers/erp                                                                         none          noauto       no
default/containers/hexchat-training                                                            none          noauto       no
default/containers/hexcode-dev-darna                                                           none          noauto       no
default/containers/hexcode-dev-saydanah                                                        none          noauto       no
default/containers/hexcode-development                                                         none          noauto       no
default/containers/hexcode-matrix-dark                                                         none          noauto       no
default/containers/hexgen                                                                      none          noauto       no
default/containers/hexgen-rewnew                                                               none          noauto       no
default/containers/kothet-ecommerce-ios-app                                                    none          noauto       no
default/containers/kothte-ever-co                                                              none          noauto       no
default/containers/m-commerce                                                                  none          noauto       no
default/containers/magento                                                                     none          noauto       no
default/containers/magento-dev                                                                 none          noauto       no
default/containers/magento-training                                                            none          noauto       no
default/containers/magento2                                                                    none          noauto       no
default/containers/magentoTest3                                                                none          noauto       no
default/containers/moodle-nginx-test                                                           none          noauto       no
default/containers/moodle-nginx-test-debug                                                     none          noauto       no
default/containers/moodle-nginx-test-debug-2                                                   none          noauto       no
default/containers/quizzical-johnson-1                                                         none          noauto       no
default/containers/sonic-reader-ubuntu1                                                        none          noauto       no
default/containers/superset                                                                    none          noauto       no
default/containers/susan-server                                                                none          noauto       no
default/containers/training-zulip                                                              none          noauto       no
default/containers/vmin                                                                        none          noauto       no
default/containers/vscode                                                                      none          noauto       no
default/containers/zulip-clean                                                                 none          noauto       no
default/custom                                                                                 none          noauto       no
default/deleted                                                                                none          noauto       no
default/deleted/containers                                                                     none          noauto       no
default/deleted/containers/a364dfe9-1cd3-4d01-be39-27ea7486eda2                                none          noauto       no
default/deleted/containers/c5065b4b-2e11-4e41-96f1-c7f190c30533                                none          noauto       no
default/deleted/custom                                                                         none          noauto       no
default/deleted/images                                                                         none          noauto       no
default/deleted/images/0305143b362cd76b78fdbde52cd62c1bad17dbe3a4c424aa028323a41600ae1a        none          noauto       no
default/deleted/images/0aebdf0c6b135fd6bd4ff5b5e8e9d10f6bcb9ac4f459d3734441b2aed6133749        none          noauto       no
default/deleted/images/29b8978ab17db48f4a91ae2ca2874ee78f046fecc675cb677db39f5f9779fd1c        none          noauto       no
default/deleted/images/529063e70b84ea2370805ead1a116352deba39bbfc4297cd0e2299897af98fac        none          noauto       no
default/deleted/images/6899d205b9aec137bb8a26928c88b502d15fcbebf15aaec6b652ec6379d2312c        none          noauto       no
default/deleted/images/7e12b43a20d8a2da69bb9d0ca99db1936499f9bd50bb286cabffbf6326e8f4d1        none          noauto       no
default/deleted/images/acdca2c8a2034d70d2a04b48f56816085b3f4321908fbbeb379e23cd3f954873        none          noauto       no
default/deleted/images/aec64fea576d80f4281fc4dee526402e0b5762c11853a78c1ac0ac64753f56e7        none          noauto       no
default/deleted/images/aec64fea576d80f4281fc4dee526402e0b5762c11853a78c1ac0ac64753f56e7.block  -                  -        -
default/deleted/images/ed06053d47a7019c128d175c345bf2d59378d07be81f3380d8293ca23cca400c        none          noauto       no
default/deleted/virtual-machines                                                               none          noauto       no
default/images                                                                                 none          noauto       no
default/images/2012f3f777507159bb8c84595594841c3b59bcef5c3f94fc0ba976f9addd8708                none          noauto       no
default/images/2fba70a9af05091a585c294726a9065bc294f756e553da2a90c07ef7d0a62883                none          noauto       no
default/images/4e7aab0f185b77984fcb42d0eb975ce50c2c83e217f1a6a4962f33c206661ee0                none          noauto       no
default/images/8d89e69a7da64084eabc8734ae3d8adcbc294f15f275673895faafae4aa26d55                none          noauto       no
default/images/c1b1e1df58a0cba4a9456f03bce1dafa2ef4877e1d9259aeaecba18a4d7c7887                none          noauto       no
default/images/e5b0f357c7a1cb4c2309f47bcfc5d1319714ab75d0327af6c1f2d602ae3adc37                none          noauto       no
default/virtual-machines                                                                       none          noauto       no
default/virtual-machines/classvmin                                                             none          noauto       no
default/virtual-machines/classvmin.block                                                       -                  -        -

What should i try next to make the error happen again?

17

Why has your zfs list output got shorter, it doesn’t have the zfs/deleted/* and zfs/images/* entries anymore.

18

my bad , paste got cut half way.

19

I don’t know, what ever you normally do to trigger it.

20

That was my mistake, sorry.

root@nuc2:~# zfs set mountpoint=none,canmount=noauto zfs/lxd/containers
cannot set property for 'zfs/lxd/containers': 'mountpoint' must be an absolute path, 'none', or 'legacy'
root@nuc2:~# zfs set mountpoint=none canmount=noauto zfs/lxd/containers
root@nuc2:~#