Lxd and cloudinit restart error

LXD
#1

Hello @all,

I’ve been using lxd with cloudinit for many months.
Today I noticed that there is currently an unexpected side effect.

I reduced my profile to a minimum:

config:
user.user-data: |
#cloud-config
description: Connect container to public bridge br0.
devices:
eth0:
nictype: bridged
parent: br0
type: nic
roots:
paths: /
pool: btrfs
type: disc
name: devpublicbridge
used_by: []

When I create an instance:

$ lxc launch ubuntu:21.10 lab-kubuntu -p devpublicbridge

Then the container is created and started:
| lab-kubuntu | RUNNING | | | CONTAINERS |

If I do a restart now:

$ lxc restart lab-kubuntu
Error: Failed unmounting instance: In use
Try lxc info --show-log lab-kubuntu for more info

But the most important thing is that the container is no longer started:

| lab-kubuntu | STOPPED | | | CONTAINERS |

However, when I start the container by hand, it works as usual.

I get the following output in the log:

$ lxc info --show-log lab-kubuntu
Name: lab-kubuntu
Status: STOPPED
Type: containers
Architecture: x86_64
Location: gramophone
Created: 2022/04/10 21:14 UTC
Last Used: 2022/04/10 21:14 UTC
Log:
lxc lab-kubuntu 20220410211451.145 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc lab-kubuntu 20220410211451.145 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc lab-kubuntu 20220410211451.147 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc lab-kubuntu 20220410211451.147 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc lab-kubuntu 20220410211515.770 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc lab-kubuntu 20220410211515.770 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing

I tryed ubuntu 20.04, too. I have the same issue.
So I think it is a lxd specific problem.

Thanks con

#2

Parameters:

  • lxd 5.0 snap version
  • Host: Ubuntu Server 21.10

Reboot on all containers gererated by lxd 4.xx works.
Reboot on containers by 5.0 didnt work and go in to STATE - STOPPED.

Thanks Frank

#3

What happens if you use -f with lxc restart?

#4

The same behavior:

$ lxc restart lab-kubuntu -f
Error: Failed unmounting instance: In use
Try lxc info --show-log lab-kubuntu for more info

and

$ lxc ls
| lab-kubuntu | STOPPED | | | CONTAINER |

#5

Please show lxc config show <instance> --expanded and lxc storage show <pool>

Also, have you tried it without a cloud init config and just a minimal fresh instance?

#6

Config:

$ lxc config show lab-kubuntu
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 21.10 amd64 (release) (20220309)
image.label: release
image.os: ubuntu
image.release: impish
image.serial: “20220309”
image.type: squashfs
image.version: “21.10”
user.network-config: |-
#cloud-config
version: 2
renderer: networkd
ethernets:
eth0:
dhcp4: False
addresses:
- 192.168.178.33/24
gateway4: 192.168.178.1
nameservers:
addresses:
- 192.168.178.13
user.user-data: |-
#cloud-config
runcmd:
- [‘touch’, ‘/tmp/user_data_was_run’]
package_update:
- True
packages: openssh-server
users:
- name: memyselfandi
groups:
- adm
- sudo
- dip
- plugdev
- cdrom
passwd:
lock_passwd: False
shell: /bin/bash
ssh_import_id:
- gh:memyselfandi
ssh_authorized_keys:
- ssh-rsa
sudo:
- ALL=(ALL) ALL
prefer_fqdn_over_hostname: True
fqdn: lab-kubuntu.bodomos.lan
user.vendor-data: ‘#cloud-config’
volatile.base_image: b75ddd1dcba22127cc0013c01a4a89f387995d74f96b328b83e9ef69e9362fc0
volatile.cloud-init.instance-id: 6ac8024a-eb6d-4884-a17d-5d4c8049923b
volatile.eth0.hwaddr: 00:16:3e:48:e8:8c
volatile.eth0.name: eth0
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: STOPPED
volatile.uuid: 49ab5d9e-030e-44e4-9566-02539ff2fec2
devices: {}
ephemeral: false
profiles:
- devpublicbridge
stateful: false
description: “”

Pool:

$ lxc storage show btrfs
config: {}
description: “”
name: btrfs
driver: btrfs
used_by:

  • /1.0/images/1b4e4d326603c8150bcd00857539bc4a5b0f90703f1ec316c6be76749c9651e6?target=gramophone
  • /1.0/images/7e8bbf2f24234c9414314019cfa275870f945a6a2ed52c944c3c11a16a98e4ef?target=gramophone
  • /1.0/images/b75ddd1dcba22127cc0013c01a4a89f387995d74f96b328b83e9ef69e9362fc0?target=gramophone
    […]
  • /1.0/instances/lab-kubuntu?target=gramophone
    […]
  • /1.0/profiles/DMZ
  • /1.0/profiles/default
  • /1.0/profiles/devpublicbridge
  • /1.0/profiles/publicbridge
    status: Created
    locations:
  • gramophone
#7

Do you see the same issue if you launch the instance on an lvm storage pool?

#8

You’re right, I thought I tested it before posting yesterday. Even without cloudinit parameters I now have the same error. I create a new pool and then test.

#9

$ lxc storage create lvm lvm
$ lxc launch ubuntu:21.10 lab-kubuntu --storage lvm
Creating lab-kubuntu
Starting lab-kubuntu
$ lxc restart lab-kubuntu
$ lxc ls
| lab-kubuntu | RUNNING | 10.137.13.217 (eth0) | fd42:2a4e:bf70:bc7:216:3eff:fe9f:2f64 (eth0) | CONTAINER |

It seams to be anything with the btrfs backend?
But my old Conaine didnt have this error.

#10

Please can you open an issue at Issues · lxc/lxd · GitHub with as minimal reproducer steps as possible (i.e don’t include cloud-init if thats not part of the problem). Thanks :slight_smile:

#11

Hi tomp,

after your support i started creating new containers with different names, which worked without any problems. Then I rebooted the server and now everything is working as it should. I don’t know what was stuck. But I can’t reproduce the error. Should I still open the ticket?

Thanks Frank

#12

Only if you can get a reproducer, sounds like you may have had something stuck in the kernel.