Snap set lxd shiftfs.enable=true breaks snapd

lhprojects · March 17, 2021, 3:52pm

This morning I ran snap set lxd shiftfs.enable=true which lead to snapd stumbling itself error: cannot communicate with server: Get "http://localhost/v2/changes/184": dial unix /run/snapd.socket: connect: connection refused

AppArmor status: apparmor is enabled but some kernel features are missing: dbus, network
2021/03/17 11:38:45.553940 patch.go:64: Patching system state level 6 to sublevel 1...
2021/03/17 11:38:45.565885 patch.go:64: Patching system state level 6 to sublevel 2...
2021/03/17 11:38:45.575843 patch.go:64: Patching system state level 6 to sublevel 3...
2021/03/17 11:38:45.597315 daemon.go:347: started snapd/2.49-1.12 (series 16; classic; devmode) opensuse-tumbleweed/20210219 (amd64) linux/5.11.2-1-default.
2021/03/17 11:38:45.703533 daemon.go:440: adjusting startup timeout by 1m55s (pessimistic estimate of 30s plus 5s per snap)
2021/03/17 11:38:46.983230 stateengine.go:150: state ensure error: cannot sections: got unexpected HTTP status code 403 via GET to "https://api.snapcraft.io/api/v1/snaps/sections"
panic: assignment to entry in nil map

goroutine 28 [running]:
github.com/snapcore/snapd/overlord/configstate/config.applyChanges(0x0, 0xc0008ab230)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/configstate/config/transaction.go:306 +0x146
github.com/snapcore/snapd/overlord/configstate/config.(*Transaction).Commit(0xc00055fa80)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/configstate/config/transaction.go:293 +0x1f4
github.com/snapcore/snapd/overlord/configstate.ContextTransaction.func1(0x4500000003, 0x7467fef0971e)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/configstate/hooks.go:53 +0x35
github.com/snapcore/snapd/overlord/hookstate.(*Context).Done(0xc0001dd960, 0x0, 0x0)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/hookstate/context.go:264 +0x84
github.com/snapcore/snapd/overlord/hookstate.(*HookManager).runHook(0xc0004961e0, 0xc0001dd960, 0xc0004e4700, 0xc000822400, 0xc000715d10, 0x0, 0x0)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/hookstate/hookmgr.go:416 +0x650
github.com/snapcore/snapd/overlord/hookstate.(*HookManager).runHookForTask(0xc0004961e0, 0xc0003d2c60, 0xc000715d10, 0xc0004e4700, 0xc000822400, 0x0, 0x0)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/hookstate/hookmgr.go:312 +0xba
github.com/snapcore/snapd/overlord/hookstate.(*HookManager).doRunHook(0xc0004961e0, 0xc0003d2c60, 0xc000715d10, 0x228f178, 0x7467f72ccb0b)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/hookstate/hookmgr.go:264 +0x168
github.com/snapcore/snapd/overlord/state.(*TaskRunner).run.func1(0x0, 0x0)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/overlord/state/taskrunner.go:203 +0xe3
github.com/snapcore/snapd/vendor/gopkg.in/tomb%2ev2.(*Tomb).run(0xc000715d10, 0xc00083e030)
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/vendor/gopkg.in/tomb.v2/tomb.go:163 +0x3a
created by github.com/snapcore/snapd/vendor/gopkg.in/tomb%2ev2.(*Tomb).Go
        /home/abuild/rpmbuild/BUILD/snapd-2.49/gopath/src/github.com/snapcore/snapd/vendor/gopkg.in/tomb.v2/tomb.go:159 +0xbc

Is there alternative method to enabling shiftfs? Ultimately, I like to be able to read and write to a mount point that is alias to directory on the host?

I’ve already open a thread at snapcraft here and waiting on what avenue to take moving forward resolving the broken snapd.

stgraber · March 17, 2021, 3:59pm

Definitely sounds like a snapd bug, hopefully they can easily give you a workaround to fix this. We have this very command automatically done on very many test systems so it’s not a generic snapd issue affecting everyone but instead something odd happening on your system.

lhprojects · March 17, 2021, 4:34pm

Mean time, while I wait for a response, is:

devices:
  dev:
    path: /usr/local/dev
    source: /home/user/Documents/Programming/
    type: disk

enough to achieve a read/writeable mount point?

Update:

There doesn’t seem to be a standardize method aside from shiftfs to deploy a rw mount within containers, see thread here https://github.com/lxc/lxd/issues/1879 and here (looking to mount your home directory rw) https://gist.github.com/bloodearnest/ebf044476e70c4baee59c5000a10f4c8. After evaluation, rw is unnecessary for me atm, so I’m dropping this.

lhprojects · March 17, 2021, 8:02pm

I haven’t gotten a response from the snapd folks, so I’ll proceed and purge snapd and snaps and start from scratch.

lhprojects · March 18, 2021, 9:44pm

So circling back to this, I build my own kernel to resolve a local issue and initially while researching this topic. I missed:

At this time, shiftfs isn’t mainline. … Note that in addition to shiftfs, the Ubuntu kernel also carries changes to overlayfs

So I would recommend avoiding snap set lxd shiftfs.enable=true on non-ubuntu kernel as that leaves you with incomplete task that breaks snapd (see bug), unless you were to build the kernel from Ubuntu sources if you on a different distribution.

toby63 · March 19, 2021, 8:38pm

Could you explain that a bit more?
Is the problem the missing shiftfs module or the implementation of shiftfs in other kernels?

In variant 1: I guess snapd should detect that shiftfs is missing .

I also maintain a repo for shiftfs installation via dkms (https://github.com/toby63/shiftfs-dkms) and it works on multiple distros, including Arch Linux with native LXD and also snap LXD (at least the last time I checked) (with arch kernel of course).
So it should work I guess.

stgraber · March 19, 2021, 8:52pm

I don’t see how this snapd failure would be specific to the shiftfs option being set though. The hook being called when you call snap set doesn’t do any special handling of any of the options in the LXD case.

All it does is generate a text file (/var/snap/lxd/common/config) based on the information given by snapd. So whether you’re setting shiftfs.enable=true or daemon.debug=true should not make any difference.

lhprojects · March 20, 2021, 2:23pm

First and foremost, and it appears I wasn’t clear. And:

If you look at the bug report, it seems that it’s related to snap snapshots, so it is bug in snapd. The aim was to say that:

shiftfs isn’t main line
enalibing that option can break snapd

I am on 5.12-rc3 and I would love to give this another try however:

toby63 · March 20, 2021, 4:46pm

You are not really explaining anything here.

As @stgraber pointed out, this is likely not related to shiftfs, so there is no reason for you to mark something as a solution, that obviously is not necessary.

I am on 5.12-rc3 and I would love to give this another try however:

Note that in addition to shiftfs, the Ubuntu kernel also carries changes to overlayfs

So you use overlayfs inside your container? As that is the only scenario where the overlayfs changes would be necessary.

lhprojects · March 20, 2021, 11:59pm

That’s not what I said, what I said is that enabling that option may break snapd.

I assumed that overlayfs and shiftfs were coupled. Knowing this, and when I touch this again, I’ll give shiftfs another try assuming snapd doesn’t break unless I compile snapd with the fix or the fix get’s released.