Trying LXD virtual machines

I wanted to be able to run lxc exec on VMs without rebooting for automation purposes, I achieved this with the following profile in case anyone need this:

config:
user.user-data: |
#cloud-config
ssh_pwauth: yes
apt_mirror: http://us.archive.ubuntu.com/ubuntu/
users:
- name: ubuntu
passwd: “$6$SaltSalt$/oLlBqBTXG/VR/9XsOXUUcMqwTBd5gzRCcs7Xtbr3cplQCh3Kkq33nyYml9MAF9C3zjfpAvySydIMD9q4KciX/”
lock_passwd: false
groups: lxd
shell: /bin/bash
sudo: ALL=(ALL) NOPASSWD:ALL
runcmd:
- mount -t 9p config /mnt/
- cd /mnt/ && ./install.sh
- cd / && umount /mnt
- systemctl start lxd-agent-9p lxd-agent
description: VM specific configuration
devices:
config:
source: cloud-init:config
type: disk

1 Like

Note that we now have the new images on the images: remote which have the agent pre-baked.

See Initial set of distrobuilder VM images for more info.

Thanks it works! Definitely easier that tweaking with cloud init.

There is still something I don’t get though. I created a VM with a dir storage backend:

lxc storage info virtpool
info:
description: “”
driver: dir
name: virtpool
space used: 20.73GB
total space: 103.08GB

lxc launch images:ubuntu/bionic b1 --vm --storage virtpool

But when I df inside the VM:

df -h
Filesystem Size Used Avail Use% Mounted on
udev 482M 0 482M 0% /dev
tmpfs 99M 532K 98M 1% /run
/dev/sda2 3.8G 704M 3.1G 19% /
tmpfs 492M 0 492M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 492M 0 492M 0% /sys/fs/cgroup
/dev/sda1 99M 3.6M 95M 4% /boot/efi
config 96G 20G 72G 22% /run/lxd_config/9p

I don’t get why the / is only 3.8G while /run/lxd_config/9p is 96G ?

Also is “only the dir storage backend works with VMs” still true ? I checked the release posts for 3.20 and 3.21 but did not find anything about this.

Thanks for your help.

Unless you’re using a cloud image, the image is going to have a 4GB / as you have above.
The 96GB is for the agent config drive and shows disk information from the host rather than your VM.

We allocate a 10GB large volume by default, but unless you’re using the cloud images, you need to manually handle any resizing yourself as LXD doesn’t want to go resize partitions on you.

I’ve followed your instructions to the letter but keep getting login incorrect. I generated my password using the command mkpasswd -m sha-512 and plonked into my profile for virtual-machines which looks like this:

$ lxc profile show vms
config:
  user.user-data: |
    #cloud-config
    ssh_pwauth: yes

    users:
      - name: ubuntu
        password: "$6$d32nUte9$vaImUHkBw1DrB1ntxMkqYu1OPaDZ0zU/IyVzpPBdj.Tdw9eM3PJpNfIf0Kg539r39mfME8lMtjcFWcRkfX23i1"
        lock_passwd: false
        groups: lxd
        shell: /bin/bash
        sudo: ALL=(ALL) NOPASSWD:ALL
description: LXD Profile for VM's
devices:
  config:
    source: cloud-init:config
    type: disk
  root:
    path: /
    pool: default
    type: disk
name: vms
used_by: []

During boot, I see a warning that may or may not be indicating something wrong with my profile:

14.560321] cloud-init[805]: 2020-03-20 23:48:32,911 - util.py[WARNING]: Failed at merging in cloud config part from part-001

Any ideas?

Also, I had to add the root: part in devices. If not present, I get this error:

$ lxc launch ubuntu:b --vm --profile vms bionic
Creating bionic
Error: Failed instance creation: Create instance: Create instance: Invalid devices: Failed detecting root disk device: No root device could be found

Any pointers appreciated.

Are you referring to this, https://blog.simos.info/how-to-use-virtual-machines-in-lxd/ ?

When you create a container or a VM, you need to specify the storage device (i.e. the root:).
You could very well copy the root: entry from the default profile to all other profiles, like the VM profile. Or, when you create a container or VM, you can always specify the default profile, and on top of that, add a vm profile that has additional configuration. This is about stacking two separate profiles together. In my blog post I suggest the latter.

I also get the

[    9.294197] cloud-init[719]: 2020-03-21 19:32:12,041 - util.py[WARNING]: Failed at merging in cloud config part from part-001

but can get a shell into the container. When I say I can get a shell, I mean either with lxc console or ssh. lxc exec requires to setup the LXD Agent first.

Here are the details of the warning/error.

2020-03-21 19:32:12,040 - __init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (text/cloud-config, part-001, 3) with frequency once-per-instance
2020-03-21 19:32:12,041 - util.py[DEBUG]: Attempting to load yaml from string of length 13 with allowed root types (<class 'dict'>,)
2020-03-21 19:32:12,041 - util.py[DEBUG]: loaded blob returned None, returning default.
2020-03-21 19:32:12,041 - util.py[WARNING]: Failed at merging in cloud config part from part-001
2020-03-21 19:32:12,047 - util.py[DEBUG]: Failed at merging in cloud config part from part-001
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 140, in handle_part
    self._merge_part(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 116, in _merge_part
    (payload_yaml, my_mergers) = self._extract_mergers(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 95, in _extract_mergers
    mergers_yaml = mergers.dict_extract_mergers(payload_yaml)
  File "/usr/lib/python3/dist-packages/cloudinit/mergers/__init__.py", line 83, in dict_extract_mergers
    raw_mergers = config.pop('merge_how', None)
AttributeError: 'NoneType' object has no attribute 'pop'
2020-03-21 19:32:12,050 - __init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (__end__, None, 3) with frequency once-per-instance

@simos,

Yes, that’s the one. I must be missing something if you are using those same instructions and ending up successfully logged in while I fail.

I’ll try the process again; maybe I missed a letter - or two.

Thanks.

OK.

I’ve been throwing everything (including the proverbial kitchen sink) at this problem and I haven’t made any progress. So, the obvious conclusion is that I have very little understanding of what I’m doing when I’m modifying this profile.

So, to that effect, can someone point me to the documentation where it documents what the lines below are really doing (as in line by line) and what files (if any) are being created by these actions in the container. Once all this is understood, then we all (really I) stand a better chance of getting stuff working instead of fighting the system.

What exactly am I changing in the container when I add this profile to it:

config:
  user.user-data: |
    #cloud-config
    ssh_pwauth: yes

    users:
    - name: ubuntu
      password: "$6$d32nUte9$vaImUHkBw1DrB1ntxMkqYu1OPaDZ0zU/IyVzpPBdj.Tdw9eM3PJpNfIf0Kg539r39mfME8lMtjcFWcRkfX23i1"
      lock_passwd: false
      groups: lxd
      shell: /bin/bash
      sudo: ALL=(ALL) NOPASSWD:ALL
description: LXD Profile for VM's
devices:
  config:
    source: cloud-init:config
    type: disk
  root:
    path: /
    pool: default
    type: disk
name: vms
used_by:

https://cloudinit.readthedocs.io/en/latest/ for the full cloud-init documentation.

ssh_pwauth authorizes ssh password authentication.
The rest creates an ubuntu user, who’s part of the lxd group, has password authentication enabled, full sudo rights and the password you specified (hashed).

Note that cloud-init runs exactly once, on first boot of an instance.

So every time you want to test a change to your cloud-init config, you must create a new instance with that profile applied at launch time.

Holy smoke - I’ve been doing nothing all this time.

That’s a big revelation and thanks for the link too.

ak.

@stgraber is there a website or place in official documentation which holds howtos like this one?
If there isn’t, I would suggest creating the howto section of the official documentation and moving a lot of content from your personal website (thank you for examples there) and examples such as this one from your post, there. So it’s all in one place, what do you think?

We have a Tutorials section for this on the forum, but this post is a bit too outdated to make it there, we’ll want to write a new one.

thanks! please do.
I currently use multipass for vms, but having lxd on my machine for everything else, I’d also like to use it for vms.
It would be great if vm support in lxd is improved so launching vms is as simple as with multipass.

btw, maybe this is a nice place to put all tutorials: https://ubuntu.com/tutorials ?

1 Like

wow that was quick! Thank you!

is this why you cannot copy a VM that has been booted up once and expect the copy to work?

The copy would work but it may not run cloud-init again which could lead to duplicate SSH host keys, incorrect hostname, …

Normally the cloud-init logic is to re-run on copies where the instance name is changed though and LXD implements that. But it won’t run if you use a move operation or if copying with the same name but a different remote or project.

yep, I get what no cloud init means for a duplicated machine, but i used latest Focal from both ubuntu: and images: and the copy would not start. I moved on because I was pressed for time then but I will file a bug report.