Incus OCI Container startup failure after adding volume as persistent data

Incus
,
1

Hey all, trying to get started using app containers with Incus. I’ve managed to get them launched without anything extra, e.g. storage but now I’m looking to keep persistent data and allow me to update the containers in the future so I can run applications like VaultWarden without worrying about losing my passwords on upgrade. I’ve been testing this with the inductive automation ignition container running latest 8.1.47. Here are my configs:

ignition0 container:

gage@dl380g10:~$ incus config show ignition0 --expanded 
architecture: x86_64
config:
  environment.HOME: /home/ignition
  environment.IGNITION_INSTALL_LOCATION: /usr/local/bin/ignition
  environment.LANG: en_US.UTF-8
  environment.LANGUAGE: en_US
  environment.LC_ALL: en_US.UTF-8
  environment.PASS_THROUGH: both
  environment.PATH: /usr/local/bin/ignition/lib/runtime/jre/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  environment.TERM: xterm
  image.architecture: x86_64
  image.description: docker.io/inductiveautomation/ignition (OCI)
  image.id: inductiveautomation/ignition
  image.type: oci
  limits.cpu: "4"
  limits.memory: 4GiB
  security.privileged: "false"
  volatile.base_image: fa81e8cac62d743bfa42f61973abd91cb1b255b2aeda6b128f0801d37fd932f3
  volatile.cloud-init.instance-id: 9cd000cb-5c20-4cbb-809d-6966e2950274
  volatile.container.oci: "true"
  volatile.eth30.hwaddr: 00:16:3e:ac:48:84
  volatile.eth30.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: e5c6fd9e-b380-403f-9aaa-bed220def196
  volatile.uuid.generation: e5c6fd9e-b380-403f-9aaa-bed220def196
devices:
  data:
    path: /usr/local/bin/ignition/data
    pool: default
    source: ignition0-data
    type: disk
  eth30:
    mtu: "1500"
    nictype: bridged
    parent: br10
    type: nic
    vlan: "30"
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- defaultContainer
- vlan30
stateful: false
description: ""

ignition0-data

config:
  security.shifted: "true"
  volatile.idmap.last: '[]'
  volatile.idmap.next: '[]'
description: ""
name: ignition0-data
type: custom
used_by:
- /1.0/instances/ignition0
location: none
content_type: filesystem
project: default
created_at: 2025-03-27T14:42:13.381621056Z

I’ve passed security.shifted true in the volume config, but it appears to not have any effect on the errors that I receive.

incus info ignition0 --show-log

gage@dl380g10:~$ incus info ignition0 --show-log
Name: ignition0
Description: 
Status: STOPPED
Type: container (application)
Architecture: x86_64
Created: 2025/03/27 14:41 UTC
Last Used: 2025/03/27 15:54 UTC

Log:

lxc ignition0 20250327155416.741 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:747 - File exists - Creating the final cgroup 10(lxc.monitor.ignition0) failed
lxc ignition0 20250327155416.741 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:807 - File exists - Failed to create monitor cgroup 10(lxc.monitor.ignition0)
lxc ignition0 20250327155416.741 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroup_tree_create:747 - File exists - Creating the final cgroup 10(lxc.monitor.ignition0-1) failed
lxc ignition0 20250327155416.741 WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgroup_tree_create:807 - File exists - Failed to create monitor cgroup 10(lxc.monitor.ignition0-1)
lxc ignition0 20250327155417.167 ERROR    af_unix - ../src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc ignition0 20250327155417.167 ERROR    commands - ../src/lxc/commands.c:lxc_cmd_rsp_recv_fds:128 - Failed to receive file descriptors for command "get_state"

I think it’s more the errors with cgroups. I’ve tried setting it as a privileged container but that didn’t help either.

I’ll happily provide information that is requested.

Thank you,
Gage

2

incus console --show-log ignition0 would be useful here as it’s unlikely to be an LXC startup failure more than something failing in the entrypoint of the container.

3

You’re onto something:

init     | 2025/03/27 15:54:16 | Creating init.properties file
init     | 2025/03/27 15:54:16 | Creating gateway.xml
cp: cannot stat '/usr/local/bin/ignition/data/gateway.xml_clean': No such file or directory

Am I mounting my volume into the container correctly? If we can get this going I’d love to write some documentation.

P.S. thank you for that I was wondering how to get that information.

4

You are, but I think that ignition is expecting some pre-existing data in that volume… So the blank volume is causing it to fail.

5

In that case then, how can I have the volume mounted when it does the data expansion during build? My guess is that they include default files in the data directory that they expect to be expanded into the persistent volume when the container is launched like when docker would do it.

docker run -d -p 9088:8088 --name ignition-test \
    -v gw-data:/usr/local/bin/ignition/data \
    --pull always inductiveautomation/ignition:latest \
    -n docker-test -a localhost -h 9088 -s 9043

My assumption would be that when docker run does it’s thing and starts, it mounts the folder before expanding the containers filesystem. Since I can’t initialize one of these with the volume mounted it breaks that process.

I’m going to try to make a profile for this and I’ll report back with my findings, perhaps having the volume existing during creation is the key to getting this to work.

6

Easiest trick is to mount the volume at another location, inside the container, copy the data, then put the volume at the correct location

7

Alright, that lead to the same result. I’m going to file a bug report with inductive on it to let them know that they can’t count on that directory to contain data in edge cases. Is there a way to have it have the volume in place when it expands the OCI container onto the disk? that’d solve the problem too. Is there a way to get a shell in the container? Doesn’t look like they include one I can interact with in the console so I may just be out of luck on copying the data over.

8

Run into similar issues with other OCI images…

It is an permission issue after mounting the volume:

drwxr-xr-x 10 ignition ignition     23 Mar 28 04:30 .
drwxr-xr-x  3 root     root          7 Feb 26 21:20 ..
drwx--x--x  5 root     root         14 Mar 28 04:28 data
...

On default it is mounted by root:root where it needs to be ignition:ignition to work properly.

Unfortunately there is no option, as far as I’m aware, to define the correct user:group for mounting a volume. May be raw.mount.options supports to add correct mount permissions, haven’t tried.

As a workaround you could attach the volume under a different folder like /mnt/data. Change permissions to chown -R ignition:ignition /mnt/data, stop container, change mount point to correct place /usr/local/bin/ignition/data and start container again. Now it should startup as designed.

All in all not pretty but works on my lab system.