Upgrade from Ubuntu 18.04 to 18.10 broke my system because of LXD

I just finished the rescue process after unsuccessful upgrade from Ubuntu 18.04 to 18.10. The process hanged, leaving me with half upgraded unbootable system. I believe the culprit to be the LXD, specifically the failed transition from deb to snap. Luckily for me, Linux already has taught me to do backups and thanks to LVM I only wasted 15 minutes reinstalling grub from LiveCD.

Steps to reproduce:

  1. backup
  2. apt update
  3. apt dist-upgrade
  4. reboot
  5. do-release-upgrade

Here is the last visible terminal output before the process hanged:

dpkg: error processing archive /tmp/apt-dpkg-install-JR1TQB/107-lxd_1%3a0.4_all.deb (--unpack):
 new lxd package pre-installation script subprocess returned error exit status 1
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/apt/cache.py", line 265, in __getitem__
 rawpkg = self._cache[key]
 KeyError: '107-lxd'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 126, in _apt_pkg
return self._cache()[package]
File "/usr/lib/python3/dist-packages/apt/cache.py", line 267, in __getitem__
raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named '107-lxd'"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/share/apport/package_hook", line 48, in <module>
pr['SourcePackage'] = apport.packaging.get_source(options.package)
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 159, in get_source
  if self._apt_pkg(package).installed:
File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 128, in _apt_pkg
 raise ValueError('package %s does not exist' % package)
ValueError: package 107-lxd does not exist

I will gladly report it in a proper place if @stgraber or @simos would be kind enough to point me where.

This constitutes a bug report, likely on the lxd transition package in Ubuntu 18.10.

The referenced deb package /tmp/apt-dpkg-install-JR1TQB/107-lxd_1%3a0.4_all.deb has a hex character for :, so it’s really /tmp/apt-dpkg-install-JR1TQB/107-lxd_1:0.4_all.deb. Looks weird, but still the error is not about bad filename but just key errors in Python.

As a workaround, I would suggest to migrate from the deb package to the snap package, then upgrade from 18.04.1 to 18.10.

Yep. After restore, I simply purged lxd package (wasn’t using it on that particular machine anyway - maybe that was also a problem, that there was no configuration preseed by lxd init?) and then followed same steps. The rest of the upgrade went almost* flawless.

@simos, do you want me to report it as a bug? If yes, then where? Launchpad, github?

  • I was kicked out from X session and had to restart but that was probably due to me having replaced gnome with unity desktop.

So looks like there are two bugs, one is that apport is blowing up on packages that use an epoch in their version?

The other bug is that LXD somehow failed to upgrade. Do you have an update log for that part?

mmmm… sorry, after failed upgrade I did a rollback via LVM. The only log I have is the part you see above that I just copy-pasted from terminal to a file on another disk. Recent update log won’t be useful since it went without lxd package at all. If you find it useful I can (later this or next week) spinup a VM with vanilla bionic and check how it goes again.

Pretty sure vanilla bionic will work fine with and without containers, as that’s what we’ve been using for our upgrade tests and what’s part of the automated CI.

So you most likely hit some odd edge case, either in the upgrade tool itself, or hit a snapd issue in the middle of the upgrade (though that’d have made it hang and print a retry message normally) or hit something weirder (I/O issue, apt ordering problem, …). Without either a reliable way to reproduce this or an upgrade log, it’s going to be pretty hard to figure out what happened.

I still have the backup zip before the upgrade so I can try to spinup vm with this image and redo the process. Which log files do you need?

I’d need the terminal output of the upgrader, it should show the lxd package upgrade followed by the automated migration code that will pull in the LXD snap.

If the upgrade fails, you should still be able to then attempt a manual upgrade with apt dist-upgrade and see only the failing packages be retried.

Normally I just manually select the output with a mouse and copy-paste it to a file. I’m afraid output od dist-upgrade might exceed the terminal limit of lines displayed. Is there a way to log the output to external file? Would simple do-release-upgrade > /home/upgrade.log redirection work?

I don’t think that it would as it’s running screen/tmux. Maybe look under /var/log, I’d expect it to log a copy there.

Ok, I will try to do it the following weekend, will paste logs from /var/log.