[SOLVED] Best way to repackage a running lxc container?

Hi everyone,

I am currently using Packer with the LXC builder to bring up and provision containers. This works well but is slow to iterate on when developing as it requires rerunning the full build each time, which downloads the entire LXC image, applies OS updates, etc something which is a waste of time when those do not change (the container is destroyed when a build/provisioning step fails).

I had been previously using Docker instead of LXC with the same workflow and the scheme I was using was the following:

  1. Run the build once with Packer, pulling the standard OS Docker image, applying OS updates, etc without running any user modifications/scripts.
  2. Packer produces a Docker image. Push that resulting image to a Docker registry.
  3. Change the Packer config file to use that as a new base image instead of the OS Docker one.
  4. I can then quickly iterate and debug my provisioning scripts until all is working. At that point, I can revert to the original OS Docker image, rerun a build from scratch for sanity and I am good to go.

I am trying to replicate that workflow using LXC. We currently use the LXC download template in Packer with sth like:

"builders": [{
      "type": "lxc",
      "config_file": "/etc/lxc/default.conf",
      "template_name": "download",
      "template_parameters": [
          "-d", "ubuntu",
          "-r","18.04",
          "-a", "amd64"
      ],

I figured that if I can repackage a running LXC container (the one with the base OS updates applied), I could then use the local LXC template to iterate from that base.

I have thus been looking at distrobuilder, especially the pack-lxc command. The documentation is pretty scarce but I have managed to make it generate a rootfs/meta with the following vincent.yaml:

image: 
  description: Vincent's Linux container
  distribution: ubuntu 
  release: 0.1

source:
  downloader: ubuntu-http

packages:
  manager: apt

My command is:

sudo distrobuilder pack-lxc ~/tmp/vincent.yaml /var/lib/lxc/lxc_app-backend_fedora/ /tmp/vincent/

This correctly generates the LXC files:

$ ls -al                                                   
total 432M                                                                                                             
drwxrwxr-x.  2 vincent vincent   80 Jan  2 14:49 ./                                                                    
drwxrwxrwt. 35 root    root    1.6K Jan  2 14:51 ../                                                                   
-rw-r--r--.  1 root    root     228 Jan  2 14:40 meta.tar.xz                                                           
-rw-r--r--.  1 root    root    432M Jan  2 14:40 rootfs.tar.xz

The problem is that I cannot start the container as it is missing some configuration bits:

$ lxc-create -n vincent-new -t local -- -m ./meta.tar.xz -f rootfs.tar.xz 
Unpacking the rootfs
ERROR: meta tarball is missing the configuration file
lxc-create: vincent-new: lxccontainer.c: create_run_template: 1648 Failed to create container from template
lxc-create: vincent-new: tools/lxc_create.c: main: 331 Failed to create container vincent-new

Can somebody tell me whether this is a sensible approach? In particular, it seemed a little strange to me that to pack an existing container/rootfs, one needed the YAML file. Maybe this is the root of my issue, with the missing configuration which should have been in there… If so, is there a way to get the required information from the running container?

Thanks much for your help!

Vincent

PS: this is Fedora 31, with LXC coming from Ganto’s Copr:

$ cat /etc/redhat-release
Fedora release 31 (Thirty One)
$ lxc-create --version
3.2.1

You need some lxc specific bits in your distrobuilder yaml to generate that config that lxc is looking for.

You can find our production yaml files at https://github.com/lxc/lxc-ci (in the images directory) which include the config bits suitable for a variety of LXC versions.

Hi Stephane,

Thanks for your quick reply. Are you specifically referring to the config collection of the YAML file, under the lxc block of the target entry at https://github.com/lxc/lxc-ci/blob/master/images/ubuntu.yaml#L247?

Also, is there a way to know the minimal set of entries there which will generate the config file?

Thanks for the guidance!

Yep, the config chunk is what you need.
You’re unlikely to need the before 5 ones, so could just keep the remaining three and drop the after 4 to have a tidier set for new images.

Hi Stephane,

First, Happy New Year 2020 to you all the whole Linux Containers team! It looks like your will be very busy with lots of projects on this front :slight_smile:

Thanks for your answer concerning the configuration part, I have tried updating my yaml file and indeed the error about the configuration part has disappeared.

Here is the updated YAML file (quickpath.yml below), updated as per your suggestions:

image: 
  description: Shortcut for Packer development
  distribution: ubuntu 
  release: 0.1

source:
  downloader: ubuntu-http

packages:
  manager: apt

targets:
  lxc:
    create-message: |-
        This "shortcut" container was created out of a Packer build by distrobuilder.

    config:
      - type: all
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/common.conf

          # For Ubuntu 14.04
          lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
          lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
          lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
          lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0

      - type: user
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/userns.conf

          # For Ubuntu 14.04
          lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none bind,optional 0 0
          lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none bind,optional 0 0

      - type: all
        content: |-
          lxc.arch = {{ image.architecture_personality }}

The container creation does not output any error anymore:

$ sudo distrobuilder pack-lxc quickpath.yml /var/lib/lxc/lxc_app-backend_ubuntu/ /tmp/vincent/
$ sudo lxc-create -n backend-respawn -t local -- -m /tmp/vincent/meta.tar.xz -f /tmp/vincent/rootfs.tar.xz

The created container does not start though. With logging enabled, I can see this is because /sbin/init cannot be found:

$ sudo lxc-start -n backend-respawn --logfile=/tmp/lxc-start.txt
lxc-start: backend-respawn: lxccontainer.c: wait_on_daemonized_start: 872 Received container state "ABORTING" instead of "RUNNING"
lxc-start: backend-respawn: tools/lxc_start.c: main: 329 The container failed to start
lxc-start: backend-respawn: tools/lxc_start.c: main: 332 To get more details, run the container in foreground mode                                                                                                                            
lxc-start: backend-respawn: tools/lxc_start.c: main: 334 Additional information can be obtained by setting the --logfile and --logpriority options
$ cat /tmp/lxc-start.txt
lxc-start backend-respawn 20200107124444.562 ERROR    start - start.c:start:2121 - No such file or directory - Failed to exec "/sbin/init"
lxc-start backend-respawn 20200107124444.562 ERROR    sync - sync.c:__sync_wait:61 - An error occurred in another process (expected sequence number 7)
lxc-start backend-respawn 20200107124444.582 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:872 - Received container state "ABORTING" instead of "RUNNING"
lxc-start backend-respawn 20200107124444.583 ERROR    start - start.c:__lxc_start:2036 - Failed to spawn container "backend-respawn"
lxc-start backend-respawn 20200107124444.583 ERROR    lxc_start - tools/lxc_start.c:main:329 - The container failed to start
lxc-start backend-respawn 20200107124444.583 ERROR    lxc_start - tools/lxc_start.c:main:332 - To get more details, run the container in foreground mode
lxc-start backend-respawn 20200107124444.583 ERROR    lxc_start - tools/lxc_start.c:main:334 - Additional information can be obtained by setting the --logfile and --logpriority options

Looking at the new container root FS, I can see some of its rootfs got expanded outside the rootfs directory, which appear to contain the original rootfs:

$ sudo ls /var/lib/lxc/backend-respawn/rootfs -ls
total 32
4 -rw-r--r--.  1 root root  363 Jan  7 12:03 config
4 drwxr-xr-x.  3 root root 4096 Jan  7 13:43 dev
4 drwxr-xr-x.  2 root root 4096 Jan  7 12:09 etc
4 drwxr-xr-x.  2 root root 4096 Jan  7 12:09 proc
4 drwxr-xr-x. 22 root root 4096 Jan  7 12:03 rootfs
4 drwxr-xr-x.  2 root root 4096 Jan  7 12:09 run
4 drwxr-xr-x.  2 root root 4096 Jan  7 12:09 sys
4 drwxr-xr-x.  2 root root 4096 Jan  7 12:09 tmp

$ sudo ls /var/lib/lxc/backend-respawn/rootfs/rootfs -al
total 88
drwxr-xr-x. 22 root    root   4096 Jan  7 12:03 .
drwxr-xr-x.  9 root    root   4096 Jan  7 13:44 ..
drwxr-xr-x.  2 root    root   4096 Jan  7 10:35 bin
drwxr-xr-x.  2 root    root   4096 Apr 24  2018 boot
drwxr-xr-x.  3 root    root   4096 Jan  7 10:35 dev
drwxr-xr-x. 74 root    root   4096 Jan  7 10:37 etc
drwxr-xr-x.  4 root    root   4096 Jan  7 10:36 home
drwxr-xr-x. 12 root    root   4096 Jan  7 10:37 lib
drwxr-xr-x.  2 root    root   4096 Dec 30 08:43 lib64
drwxr-xr-x.  2 root    root   4096 Dec 30 08:42 media
drwxr-xr-x.  2 root    root   4096 Dec 30 08:42 mnt
drwxr-xr-x.  2 root    root   4096 Dec 30 08:42 opt
drwxr-xr-x.  2 root    root   4096 Apr 24  2018 proc
drwx------.  7 root    root   4096 Jan  7 12:03 root
drwxr-xr-x.  2 root    root   4096 Dec 30 08:45 run
drwxr-xr-x.  2 root    root   4096 Jan  7 10:35 sbin
drwxr-xr-x.  2 root    root   4096 Dec 30 08:42 srv
drwxr-xr-x.  2 root    root   4096 Apr 24  2018 sys
drwxrwxrwt.  2 root    root   4096 Jan  7 10:44 tmp
drwxr-xr-x. 10 root    root   4096 Dec 30 08:42 usr
drwxr-xr-x. 12 root    root   4096 Dec 30 08:44 var
drwxr-xr-x. 10 vincent docker 4096 Jan  7 10:36 xdt

I also note that the original container got polluted in the same vein with those additional directories (that did not exist before distrobuilder was called):

$ ls -l /var/lib/lxc/lxc_redpesk-backend_ubuntu/
total 32K
-rw-r--r--.  1 root root  363 Jan  7 11:37 config
drwxr-xr-x.  2 root root 4.0K Jan  7 11:40 dev/
drwxr-xr-x.  2 root root 4.0K Jan  7 11:40 etc/
drwxr-xr-x.  2 root root 4.0K Jan  7 11:40 proc/
drwxr-xr-x. 22 root root 4.0K Jan  7 11:37 rootfs/
drwxr-xr-x.  2 root root 4.0K Jan  7 11:40 run/
drwxr-xr-x.  2 root root 4.0K Jan  7 11:40 sys/
drwxr-xr-x.  2 root root 4.0K Jan  7 11:40 tmp/

I have checked the new container config file and the rootfs path appears correct:

$ sudo cat /var/lib/lxc/backend-respawn/config
# Template used to create this container: /usr/share/lxc/templates/lxc-local
# Parameters passed to the template: -m /tmp/vincent/meta.tar.xz -f /tmp/vincent/rootfs.tar.xz
# Template script checksum (SHA-1): 381a40a454bdaf18de159d6d50bba0cf7425af4b
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf

# For Ubuntu 14.04
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none bind,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none bind,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none bind,optional 0 0
lxc.mount.entry = mqueue dev/mqueue mqueue rw,relatime,create=dir,optional 0 0
lxc.arch = linux64

# Container specific configuration
lxc.rootfs.path = dir:/var/lib/lxc/backend-respawn/rootfs
lxc.uts.name = backend-respawn

# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:a3:74:f9

Do you know what could be causing this directory messup? Is there a particular layout the container directory should abide by when distrobuilder is called to pack it into config/rootfs?

Thanks much for your help!

Edit: I tried to use distrobuilder build-dir as per the doc at https://distrobuilder.readthedocs.io/en/latest/building/ to inspect the input state of pack-lxc but this fails:

$ sudo distrobuilder build-dir ~/Documents/Dev/repos/community/lxc-ci/images/ubuntu.yaml /tmp/vincent-lxc-ci
I: usage: [OPTION]... <suite> <target> [<mirror> [<script>]]
I: Try `debootstrap --help' for more information.
E: You must specify a suite and a target.
Error: Error while downloading source: exit status 1

According to the Jenkins logs at https://jenkins.linuxcontainers.org/job/image-ubuntu/architecture=amd64,release=bionic,restrict=lxc-priv,variant=default/2/console, it looks like the doc might be missing some parameters. I can indeed see the Jenkins invocation as being:

distrobuilder --timeout 3600 build-dir image.yaml rootfs -o image.serial=20200107_07:42 -o image.architecture=amd64 -o image.release=bionic -o image.variant=default -o source.url=http://us.archive.ubuntu.com/ubuntu

Hi there,

Just a quick note to mention that I solved my issue. The problem was that I was giving distrobuilder pack-lxc the path to the output directory of Packer (/var/lib/lxc/lxc_app-backend_ubuntu/ in my messages above) instead of the actual container rootfs.

In my example above, this should have been /var/lib/lxc/lxc_app-backend_ubuntu/rootfs instead. When this is done, distrobuilder works as expected and generates the correct meta.tar.xz and rootfs.xz, which can then be used w/ lxc-create -t local.

Thanks again for all the support and the help, distrobuilder is a great tool. I’ll see if I can submit a PR for the doc bits that could be improved about this behavior.