I am working on a Gentoo ebuild for distrobuilder and need to specify the dependencies with version and hash.
distrobuilder clearly has external dependencies, but I have not been able to determine what they are. Usually they are in a go.sum file or in a vendors directory, but neither is true for distrobuilder.
Not everyone in the Go community uses vendoring or Go modules.
In both LXD and distrobuilder, we don’t use those mechanisms as, so far at least, they cause more problems for us than they’re worth.
The vast majority of Go projects do not issue security advisories when a critical security issue is resolved. As a result, pinning on old versions of those external dependencies means that you may well be vulnerable to critical security issues.
Short of having our external dependencies maintain proper stable branches, stable APIs and issue security notices, the best alternative is to just always build with what’s latest and deal with the occasional breakages (that are very few as we have low tolerance for external dependencies breaking API and will very actively move away from them).
For distributions, our official release tarballs do include a copy of all external dependencies at the time of release so this is what you’d use to be able to rebuild a given release of our projects in a consistent way.
For distributions, our official release tarballs do include a copy of all external dependencies at the time of release so this is what you’d use to be able to rebuild a given release of our projects in a consistent way.
Okay, I tried to verify the release tarball found here with gpg using the signature provided and I get this output:
$ LANG=C gpg --verify distrobuilder-1.0.tar.gz.asc distrobuilder-1.0.tar.gz
gpg: Signature made Mon Oct 21 21:12:09 2019 CEST
gpg: using RSA key 602F567663E593BCBD14F338C638974D64792D67
gpg: Can't check signature: No public key
GPG says that in such a case I either I have the wrong keys or the file is suspicious:
If the output of the above command is similar to the following, then either you don’t have our distribution keys (our signing keys are here) or the signature was generated by someone else and the file should be treated suspiciously.
Where can I find the official public key?
Tarballs are signed by the maintainers, that’d usually be either myself or @brauner.
Our keys are:
And are both widely trusted in the GPG web of trust through our relations with Ubuntu/Debian and the Linux development community.
Thanks for the response. Surely those aren’t the entire keys. Where can I find them?
Yeah, you can find them from any GPG server as they all sync to each other (or are supposed to).
gpg --recv-keys KEYID will normally fetch it from whatever server your distribution has set as the default.
I’ve consulted with some Go developers on #go-nuts on freenode, and the consensus there is that the way distrobuilder is packaged is not really accepted practice.
It should be possible to keep dependencies up to date while also using modules. I’ll quote a contributor: “bumping all your dependencies is significantly easier with modules than with that horrible kludge of theirs…”
Sadly, this is entirely consistent with my experience. What should have been a trivial matter has now consumed more than a week of effort. Not using modules makes it really difficult to package distrobuilder for other distributions. Is that your intent?
No, it is not our intent to make it hard on packagers, our intent is to distribute code that we feel is safe to consume.
We’ve used this method for both distrobuilder and lxd ever since we started those projects and we have lxd packaged in quite a variety of distros and even on Windows and macOS without this ever being a real problem.
Perhaps it is just a particular challenge for distros which package on the fly, as Gentoo does.
I tried getting all the dependencies using a script to generate a list that I can pass to the ebuild. Unfortunately, some of them have been moved:
>>> Downloading 'https://proxy.golang.org//errors/@v/v0.8.0.zip'
--2020-08-18 17:38:23-- https://proxy.golang.org//errors/@v/v0.8.0.zip
Resolving proxy.golang.org... 172.217.16.81, 2a00:1450:4005:800::2011
Connecting to proxy.golang.org|172.217.16.81|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: /errors/@v/v0.8.0.zip [following]
--2020-08-18 17:38:23-- https://proxy.golang.org/errors/@v/v0.8.0.zip
Reusing existing connection to proxy.golang.org:443.
HTTP request sent, awaiting response... 410 Gone
2020-08-18 17:38:23 ERROR 410: Gone.
>>> Downloading 'https://goproxy.io//errors/@v/v0.8.0.zip'
--2020-08-18 17:38:23-- https://goproxy.io//errors/@v/v0.8.0.zip
Resolving goproxy.io... 119.28.201.50
Connecting to goproxy.io|119.28.201.50|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://goproxy.onetool.net//errors/@v/v0.8.0.zip [following]
--2020-08-18 17:38:24-- https://goproxy.onetool.net//errors/@v/v0.8.0.zip
Resolving goproxy.onetool.net... 203.205.224.59, 203.205.137.254
Connecting to goproxy.onetool.net|203.205.224.59|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-18 17:38:27 ERROR 404: Not Found.
>>> Downloading 'https://gocenter.io//errors/@v/v0.8.0.zip'
--2020-08-18 17:38:27-- https://gocenter.io//errors/@v/v0.8.0.zip
Resolving gocenter.io... 35.230.74.213
Connecting to gocenter.io|35.230.74.213|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2020-08-18 17:38:28 ERROR 404: Not Found.
This causes the ebuild to fail when creating the manifest. Perhaps go handles this in a more robust manner, I don’t get errors like this when getting the deps manually.
I cannot pull the deps automatically at build time because Gentoo builds in a restricted sandbox for security reasons. The sandbox does not have network access; Gentoo expects the ebuild to provide URIs for all required sources and handles retrieval and checksumming itself, there’s no provision for letting upstream scripts do things in their own way. This is for good reasons, but obviously a problem in a case like this. Note that this wouldn’t affect binary packaging.
A Gentoo contributor suggested that I get all the dependencies manually, as the Makefile does, then put everything including deps in my own github repo and fetch from there. What do you think of that solution? What might go wrong with that approach?
It will increase the maintenance burden for the package maintainer, so it is not my favourite solution, but I am running out of ideas.
Why don’t you just use the release tarballs from https://linuxcontainers.org/distrobuilder/downloads/ which do include the _dist directory with all the deps?
We just released distrobuilder 1.1 today (release announcement still being written) so that should include everything you need.
I have some good news. I have succeeded in getting working Gentoo ebuilds of both the distribution tarball (also 1.1) and of a commit from the repo (323598eee2c3ac10e165bec95315574efe0be53f). Now that this works in principle with this particular commit, it should be possible to make ebuilds that work off of any commit that works in testing. I did have to determine the dependencies in advance and the result is an enormous manifest, but I’ve been told that many go ebuilds look like this and there’s not anything we can do about it. The main point is that it works now.
The next task is to put together an ebuild for the image templates from lxc-ci, as distrobuilder isn’t much use (at least to naive users) without them.
I will see about getting these into the Gentoo main repository and keep you informed.
There is now a Gentoo ebuild for distrobuilder that works off of the 1.1 release tarball: app-emulation/distrobuilder
Regarding the snapshot-based builds: portage (Gentoo’s package manager) retrieves all the sources to generate a manifest, and it does so for all versions of the package. We discovered that the snapshot-based ebuild was picking up the distribution tarball and silently using it instead of the repo, which is the only reason it worked.
It will require some hacking to get snapshot ebuilds to work without modules.