I recently took a stab at statically building LXD in an Alpine Linux container, with the intent to use the compiled binaries on a Container Linux (CoreOS) machine.
I am currently having some issues with the build process, but in particular two libraries that won’t go build properly:
Make errors and those are the only things I can see that affect it.
Here’s a chunk of the hacked Makefile:
.PHONY: debug
debug:
go get -t -v -d ./...
CGO_ENABLED=0 GOOS=linux go install -a -ldflags '-extldflags "-static"' -v $(TAGS) -tags logdebug $(DEBUG) ./...
#go install -v $(TAGS) -tags logdebug $(DEBUG) ./...
@echo "LXD built successfully"
Is what I’m trying to do just silly to start with? Anyone have any ideas on what could have caused these issues? I think go-sqlite3 is a cgo package which would explain it but I didn’t expect lxc/lxd/shared to have this issue…
the go-sqlite3 bindings require CGO, so the CGO_ENABLED=0 won’t work.
Note also that according to our experience the SQLite C code bundled in the go-sqlite3 package and statically compiled by the cgo tool (via gcc) can be up to 10x slower then when building a dynamically linked Go executable using the system’s libsqlite shared library (with go build -tags libsqlite3).
If want to build the shared libsqlite library you’ll need to do it from this SQLite fork:
On top of what @freeekanayaka said, LXD itself contains CGO code which specifically relies on dynamic linking to the C library, so between our own code and the code in some of the packages we’re using, you’d need quite a lot of effort to have valid static .a files for all those external libraries which you could then in theory use to produce a single static binary.
All that only relates to the daemon though, the client can be statically built which I believe is what’s done for Windows and macOS.
Thanks for the answer @freekanayaka – I realized that shortly after looking at the go-sqlite3 bindings… was more so surprised that there was some work inside LXD itself, and @stgraber helped covered that…
@stgraber Just to make sure I’m not misunderstanding, the CGO code inside LXD is the bit interfacing with the LXC shared libraries, correct? ( the lxc-go package?)
In the case of go-sqlite3, theoretically a pure-go replacement (or use of a different db/persistence mechanism) would fix that requirement, but I’m way less clear on the code inside LXD that is using CGO code. With the current codebase, how feasible would you say it is to shim in a different implementation?
Looking inside my Container Linux distribution I can’t find the lxc shared libraries so it looks like to get LXD working on I’d have to find a way to either install both of these shared libraries or shim them somehow.
In LXD itself, we have go-lxc which is pure CGO interfacing with liblxc but we also have quite a bit of C for things like attaching to containers, messing with mounts inside containers, various network operations, efficient uid/gid shifting, …
We have no plan to switch to pure-Go as that’d require us to rewrite our entire runtime code (which is a C library) and would result in a worse experience as Go isn’t really meant for very low level kernel interactions in the first place.
@stgraber thanks for your detailed answer, again. Yeah it definitely doesn’t make sense to ask you guys to switch to rewrite the entire runtime code in pure go, I definitely wouldn’t be so selfish as to ask something like that – I was more wondering if it would be pointless to go off and try something myself. Looking at the binding (https://github.com/lxc/go-lxc/blob/v2/lxc-binding.go), the amount of work doesn’t seem insurmountable but obviously would take a very long time to actually get right.
I’m a bit past my knowledge/comfort zone, but do you think it would be possible to statically compile the LXC shared libraries? LXC is supported on alpine which is what gives me hope, as that means it must have been compiled with musl libc, which is a lot more amenable to static compilation.
From the discussion so far it seems like what I should be trying to do is statically compile go-lxc on something like Alpine and then seeing if it could be used at all from container linux, then go about finding a way to use it as a static library in lxd’s compilation.
[EDIT] - thinking about it, I will also try to run lxd from a privileged container and see if that works. I think I’ve tried it before and it didn’t but maybe I didn’t try hard enough
liblxc will indeed build fine with musl and provided that all its own dependencies are available as static objects (.a) you should be able to get a static liblxc and get that included into a static Go binary.
I’ve never done that myself (only part of liblxc a while back) but there’s no technical reason why it would be impossible. Do note that the resulting binary is going to be rather big as it’d included the LXD code, a copy of the C library, a copy of the libcap library, a copy of the libseccomp binary and quite possibly a few others.
Thanks for confirming – it seemed possible in my own head so I’m going to give it a try and will report back once I find a way to run LXD on container linux. I think I can actually leave libseccomp dynamically linked, CoreOS should have it.
Thanks for taking so much time to answer my questions!