SIGSEGV with get-client-certificate

Incus
1

Hi there! First of all, thank you for Incus, this truly impressive!

For various reasons, I compiled my own version of Incus starting from the Ubuntu 26.04 packages for inspiration and upgrading to Incus 6.0.6. I use the vendor supplied directory and do not rely on the packaged Go modules. So, I end up with a nice set of packages compiled from the pristine 6.0.6 upstream version. Everything works smoothly so far except for one small thing I can’t find the root cause of.

incus remote get-client-certificate client.crt works perfectly and exports a valid PEM cert, but :

incus remote get-client-certificate client.pfx -f pfx fails with

Password for client.pfx:
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xd08a63]

goroutine 1 [running]:
github.com/lxc/incus/vendor/software.sslmate.com/src/go-pkcs12.(*Encoder).Encode(0x1a94be0, {0x10ed040, 0xa9578ca2780}, 0x0, {0x0, 0x0, 0x0?}, {0x1206500?, 0x0?})
        github.com/lxc/incus/vendor/software.sslmate.com/src/go-pkcs12/pkcs12.go:667 +0x123
main.(*cmdRemoteGetClientCertificate).Run(0xa9578b88d00, 0xa9578d07af8?, {0xa9578d20e70, 0x1, 0x0?})
        github.com/lxc/incus/cmd/incus/remote.go:802 +0x719
github.com/lxc/incus/vendor/github.com/spf13/cobra.(*Command).execute(0xa9578c7b508, {0xa9578d20e40, 0x3, 0x3})
        github.com/lxc/incus/vendor/github.com/spf13/cobra/command.go:1015 +0xb14
github.com/lxc/incus/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xa9578b10008)
        github.com/lxc/incus/vendor/github.com/spf13/cobra/command.go:1148 +0x465
github.com/lxc/incus/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        github.com/lxc/incus/vendor/github.com/spf13/cobra/command.go:1071
main.main()
        github.com/lxc/incus/cmd/incus/main.go:357 +0x1f5

Adding --verbose and/or --debug did not change the output

Again, for various reasons, I must stick to 6.0.6 and can not upgrade to 7.x so I would appreciate any pointer/patch to fix this issue. Thank you.

2

I just built and ran 6.0.6 here:

stgraber@orilla-d13:~/Code/lxc/incus (incus:s-shf-cluster/github) (git:(HEAD detached at v6.0.6))$ incus remote get-client-certificate -f pfx out.p12
Password for out.p12: 
stgraber@orilla-d13:~/Code/lxc/incus (incus:s-shf-cluster/github) (git:(HEAD detached at v6.0.6))$ incus --version
6.0.6
stgraber@orilla-d13:~/Code/lxc/incus (incus:s-shf-cluster/github) (git:(HEAD detached at v6.0.6))$
3

Also rebuilt from our release .tar.xz and getting the same result:

stgraber@orilla-d13:~/Downloads/incus-6.0.6$ incus --version
6.0.6
stgraber@orilla-d13:~/Downloads/incus-6.0.6$ incus remote get-client-certificate -f pfx out.p12
Password for out.p12: 
stgraber@orilla-d13:~/Downloads/incus-6.0.6$

Maybe it’s related to the Go version you’re building with somehow?

4

Or it could be related to the format of your client.key?

5

I don’t have any previous key to put in the config. After installing Incus, I simply type the commands exactly as you did but I get the SIGSEGV.

I’ll describe exactly how I build Incus, maybe your keen eye will see something I’m missing :smile: The build happens inside a Ubuntu 26.04 rootfs prepared with debootstrap (I’ll soon switch to distrobuilder :smile: ) which I enter using systemd-nspawn.

  1. I download incus-6.0.6.tar.zx, rename it to incus_6.0.6.orig.tar.xz and unpack it
  2. Inside incus-6.0.6, I unpack http://archive.ubuntu.com/ubuntu/pool/universe/i/incus/incus_6.0.5-8.debian.tar.xz
  3. I remove all the quilt patches and adjust the changelog to reflect the new version
  4. In debian/control I remove all golang-* packages in Build-Depends except golang-any of course. I want to use the vendor supplied dir and not rely on the distro packages. I also add libudev-dev, libcowsql-dev, pkg-config and lxc-dev to Build-Depends. In the same way, I remove all golang-* from the Depends field of package golang-github-lxc-incus-dev even though I won’t use this package so it does not really matter.
  5. In debian/rules I have to add the following files to DH_GOLANG_INSTALL_EXTRA because DebHelper only copies *.go files from the vendor directory so the build process won’t find other files and produce an error
  • vendor/golang.org/x/net/publicsuffix/data/
  • vendor/github.com/zitadel/oidc/v3/pkg/op/form_post.html.tmpl
  • vendor/google.golang.org/protobuf/internal/editiondefaults/editions_defaults.binpb
  • vendor/github.com/opencontainers/umoci/VERSION
    and finally, I blank the contents of override_dh_auto_test to speed up the build process

Now all is well, I install the build dependencies with apt, nothing fancy here, and I fire up dpkg-buildpackage and wait for it to complete. Then I have my packages which I manually copy to a qemu VM installed with a Ubuntu 26.04 ISO where I can play around and tests everything.

6

You’re taking a really hacky approach here. Any reason why you’re not just using the Zabbly lts-6.0 packages? That will keep you on the 6.0.x series of releases, and is available for Ubuntu 26.04.

7

Actually, we don’t yet have lts-6.0 builds for 26.04.

1 Like
8

Anyway, from an upstream point of view, we support and can help with building from source using either the release tarball and git tag, using a currently supported version of the Go compiler and building in a clean environment (none of the special Go environment variables that can alter the result of the build).

Your approach of applying an old Debian packaging on top of a modern source tree, when that packaging was designed to specifically avoid all vendored/pinned dependencies, really isn’t something that we can support I’m afraid.

I’d recommend trying to build at least the client directly from source without going through the packaging logic, if that build works correctly, it will confirm that there’s something going on with the Debian packaging which is causing your issue.

1 Like
9

Don’t worry, I wasn’t expecting some special support :slight_smile: I posted here in case someone already had experienced that error so I could have a clue where to look.

You are right about something weird in the Debian build process. I tried countless permutations where I put breakpoints at certain stages to see where the problem could be (CGO_CFLAGS, etc.) but nothing helped in any way. I also of course tried a simple go build -v -tags libsqlite3 ./cmd/incus and… sure enough, it does work perfectly on the destination host exporting a PFX file.

So yes, something in the Debian build process does not work as it should and yes, trying to bend the rules to re-enable the vendoring system did not end well in my case. No problem though as throughout all my tests, everything I need works perfectly so I’ll stick with it.

As to answer gibmat : Long story short : I’m taking over the administration of several servers for a friend who owns a small business and lost his previous admin. Almost everything is running inside LXC containers “by hand” with the bare commands (lxc-* tools) in privileged mode. The servers are running very old Ubuntu OSes (20.04 and 22.04) and I want to move everything to Incus as a first step (at least to run unprivileged containers). I need to familiarize myself with the Incus build process and everything around it (LXC, AppArmor, etc.). I started with 26.04 since 6.0.0 is close enough the to the upstream 6.0.5 so I can fiddle with the backport process. Fortunately Incus uses Go which facilitates a lot the process of backporting it as far back as 20.04! That’s why I can’t use Zabbly packages which don’t provide the 6.0 LTS branch anyway. So far, I backported everything I need to 20.04 and 22.04 (cowsql, raft, LXC 5.0.5, LXCFS, meson, etc) and everything works fine so far, that’s the part where Incus really impresses me, it runs flawlessly :smiley: Thankfully 20.04 have access to kernel 5.15 which allows CGroupsV2 so I’m quite happy how things are going.

BTW, “converting” plain LXC containers to Incus is incredibly easy and effortless! lxc-stop -n XXX ; incus create --empty XXX ; mv /var/lib/lxc/XXX/rootfs/* /var/lib/incus/storage-pools/default/containers/XXX/rootfs/ ; incus start XXX and voilà! A nice unprivileged container, splendid!

The only small problem was this SIGSEGV issue but I don’t mind because everything else works perfectly. Getting the PFX with a navigator in the UI works as it should so definitely not an issue. I can now start the process of careful migration of everything step by step and modernize everything while the staff continues to use their services as usual :slight_smile:

Sorry for the long post :grin: and thank you for the help and tests.

10

Note that we also have lxc-to-incus which does roughly the same thing but also handles a bunch of the config keys. May be useful if you have more complex cases.

11

Yes, I tried it for once, it does the job “too” well :slight_smile: As the LXC containers are privileged, lxc-to-incus keeps them privileged too. My manual method ensures that I end up with exactly the same container but now unprivileged.

12

Ah yeah, it would indeed do that :slight_smile:

1 Like
13

I just bumped into this myself using the Debian packaging of Incus 7.0.0; guess I’ll have to dig into what’s different in the package build that’s causing the panic when trying to generate a pfx client certificate.

14

Maybe the GODEBUG thingy which puts Go in legacy mode, you may enjoy a little chat with ajqlee who’s dealing with that in Debian

15

Yes, there’s been a little bit of conversation on the d-go mailing list about how to modernize Debian’s packaging of Go programs. :grin:

16

Is there a strong reason not to start with incus-7.0 LTS?

There are 7.0-LTS packages in the Zabbly repo for Ubuntu 26.04. While 6.0 will remain supported for 3 years, starting on 7.0 buys you an extra couple of years. And there are some useful features in 7.0, such as the ability to run OCI containers.

17

I’d be very interested in the solution if one is found as I really struggled a lot trying to find the root cause. I compared everything I could (environment variables, dh_* calls, etc) and I except the GO111MODULE=off which lead to issues I couldn’t resolve (I’m not fluent in Go environments), everything else had no influence from what I could observe.

18

Well, I described the reasons in the very message you quoted :slight_smile:
The servers I manage are running Ubuntu 20.04 and 22.04. 20.04 isn’t supported by the Zabbly packages for a good reason (EOL) and they both use quite old software Incus depend on (LXC/LXCFS 5, etc). As I stated, I had to backport all the packages I needed to make Incus a reality on those distros. Thankfully, there’s a PPA maintaining recent GoLang packages for those distros and I heavily used them to properly compile Incus.

Incus 7 LTS requires LXC 6, Linux 6.12, etc which are too far ahead of what could run on those old distros.

I’m already very happy that 20.04 could have access to kernel 5.15 so I could get rid of CGroupV1. I believe those a very strong reasons for the first step of migration. Once everything is under control (unprivileged LXC containers migrated), I take a new step towards the whole modernization process and then I’ll use Incus latest and shiniest LTS release :slight_smile:

PS: OCI images are of no use to me for now so not a feature I need nor require and since I use Incus 6.0.6 LTS, they’re already supported anyway (all Incus 6.x non LTS features where backported to LTS).

19

Unfortunately that GO111MODULE variable has very dramatic effects, so it’s very possible that it’s the culprit. There’s an ongoing effort to deal with it in the way Debian packages for the Go ecosystem are built, but I don’t have my feet in the Go team :slight_smile:

1 Like
20

Ah right, I thought you were just going to wrap the existing servers in containers, in which case you could rebuilt the hosts as 26.04 but keep 20.04/22.04 inside the containers. That would be doable using incus-migrate. However, it sounds like that’s more of your long-term plan.

I don’t think that’s true. If it were, then there would be no difference at all between incus 6.23 and 6.0.6.

However, all the systems I had running 6.0.6 have now been upgraded to 7.0, so I can’t test this easily.