[Solved] Nova LXD + Openstack

LXD
1

I been tasked with job of add a dedicated fleet of nodes with Nova LXD on a already running Openstack KVM Cluster.

That scenario is nothing new and I already have Nova LXC up and running, but sadly the integration between libvirtd and LXC offer limited functionallity.
Now I’m trying Nova LXD but I’m having some issues (probably related to my limited experiencie with LXD).

The most rare, is that when I launch a instance nova-compute consume a lot of CPU, takes a lot of time and fails.

Strace shows me that nova-compute is processing an anormal amount of data (probably the image). What is the correct format in this scenario? rootfs.tar.gz? or a typical cloud image in RAW format?

Checking this site I found people with Clouds implemented using Openstack and Nova LXD. Can you share tips and configurations?

Thanks in advance.

Edit:

In the end I successfully extended my Openstack cluster to support LXD along side KVM on production. Some problems where related to my poor knowledge of LXD and other where bugs still presents in the the old version of the software (in my case, the control plane was Ocata) that needed fixes and special configurations on nova-compute and pylxd.

Thanks to all.

2

@ajkavanagh can you take this one?

1 Like
3

@ajkavanagh do think can help me with tips or config references of workings clouds with NovaLXD?

Thanks

4

You might simultaneously ask about this topic to

communitymngr@openstack.org

Just a thought.

5

Hi @Sergio_Morales_Acuna

That sounds a bit odd. To help you a bit more, please could you indicate what versions you are running of:

  • Linux (and vendor/version)
  • LXD
  • OpenStack release (or master)
  • How you install OpenStack (Ansible, Juju, Devstack, something else?)

Also, if you could attach your nova.conf for the host with the nova-lxd config, that would be great.

Thanks.
Alex.

1 Like
6

@Sergio_Morales_Acuna I deployed OpenStack Rocky on Bionic using Juju bundle OpenStack on LXD and having Nova LXD as the hypervisor.

There are some caveats and pitfalls. In this type of deployment (nested containers) you have to use btrfs as storage backend. You will have to enable bluestore with ceph-osd as well. Storage pool on the Nova Compute node would not be initialised properly and you will have to manually create btrfs storage pool on the Nova Compute node.

Have in mind that you cannot run both KVM and Nova LXD on the same Nova Compute node.

Let me know if I could provide further details.

7

Thanks!

I’m Running Ubuntu 16.04 with Ocata from Cloud Archive. For this node I’m using a manual installation of the components. The rest of the cluster has Kolla-Ansible.

Do I need a special configuration on LXD? In the previous iterations I encounter some issues like HostNotFound: Unable to connect to LXD daemon on nova compute.

Thanks!

Here’s the configuration:

[DEFAULT]
debug = True
osapi_compute_listen = 10.90.1.174
osapi_compute_listen_port = 8774
osapi_compute_workers = 5
metadata_workers = 5
metadata_listen = 10.90.1.174
metadata_listen_port = 8775
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
default_availability_zone = cl-stgo-1
dhcp_domain = internal.NOOOP
default_floating_pool = Santiago
block_device_allocate_retries = 300
linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver
allow_resize_to_same_host = true
my_ip = 10.90.1.174
transport_url = NOOOP
compute_driver = lxd.LXDDriver

[api]
use_forwarded_for = true

[conductor]
workers = 5

[vnc]
novncproxy_host = 10.90.1.174
novncproxy_port = 6080
vncserver_listen = 10.90.1.174
vncserver_proxyclient_address = 10.90.1.174
novncproxy_base_url = NOOOP

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[glance]
api_servers = http://172.20.1.6:9292
num_retries = 1
debug = False

[cinder]
catalog_info = volumev2:cinderv2:internalURL
os_region_name = RegionOne

[neutron]
url = NOOOP:9696
metadata_proxy_shared_secret = NOOOP
service_metadata_proxy = true
auth_url = NOOOP:35357/v3
auth_type = password
project_domain_name = default
user_domain_id = default
project_name = service
username = neutron
password = NOOOP

[database]
connection = NOOOP/nova
max_pool_size = 50
max_overflow = 1000
max_retries = -1

[api_database]
connection = NOOOP/nova_api
max_retries = -1

[cache]
backend = oslo_cache.memcache_pool
enabled = True
memcache_servers = 172.20.1.6:11211

[keystone_authtoken]
auth_uri = NOOOP:5000
auth_url = NOOOP:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = nova
password = NOOOP
memcache_security_strategy = ENCRYPT
memcache_secret_key = NOOOP
memcached_servers = 172.20.1.6:11211

[upgrade_levels]
compute = auto

[oslo_messaging_notifications]
driver = noop

[privsep_entrypoint]
helper_command = sudo nova-rootwrap /etc/nova/rootwrap.conf privsep-helper --config-file /etc/nova/nova.conf

[guestfs]
debug = False

[wsgi]
api_paste_config = /etc/nova/api-paste.ini

[scheduler]
max_attempts = 10
discover_hosts_in_cells_interval = 60

[placement]
auth_type = password
auth_url = NOOOP:35357
username = placement
password = NOOOP
user_domain_name = default
project_name = service
project_domain_name = default
os_region_name = RegionOne
os_interface = internal

[keymgr]
fixed_key = NOOOP
8

Can you share your LXD configuration? and, What is the correct image format? rootfs tar? Thanks!

9

So the error HostNotFound: Unable to connect to LXD daemon essentially means that the nova-lxd <-> pylxd <-> lxd daemon is not being found, and I suspect, it’s pylxd not finding the socket for the local (to the host) LXD daemon.

Please could you provide the version for

  • LXD
  • OpenStack versions (e.g. for Ocata it will be something like 15.0.x.x for nova and nova-lxd)
  • pylxd on the host (either use pip freeze or the apt python package is python-pylxd)

If you log into the host and run lxc version it’ll tell you the version of LXD installed on that host.

Also, if you could describe how LXD is configured on the host, that would be very useful.

10

Thanks for the answer.

I found a problem with the LXD socket and its permissions (root:root). After a quick fix Nova (pylxd) is capable of connecting to socket and is sending the glance Image correctly to the LXD deamon (lxc image list show the correct UUID).

Now I’m facing a problem where neutron-linuxbridge-agent is not creating the tap interface (the bridge is been created just fine). With LXC+Libvirt NeutronLinuxBrigeAgent is working fine so I’m doing more troubleshooting on this environment.

About the versions:
NovaLXD: 15.0.2
NovaCompute: 15.1.5
pylxd: 2.2.4
LXD: 2.0.11 (default installation. profile: nictype: bridged, parent: lxdbro, type: nic)

Now it’s look like the creation of the neutron port is my main problem.

With NovaLXD, neutron agent is still responsible of creating the port and attaching it to the bridge?

Thanks for your help.

11

Quick update.

I’m in the same boat of: https://lists.linuxcontainers.org/pipermail/lxc-users/2018-August/014467.html

I’m looking into it but I think the problem is related to the versions involved.

Cheers

12

Hi @Sergio_Morales_Acuna

Christmas intervened in our conversation! Did you manage to sort it out? I’m still wondering what the problem might have been?

Yes, the nova lxd driver uses the standard ComputeDriver.vif_driver.plug(...) function call to plug the network in.

13

But can you run KVM and Nova LXD in two separate compute nodes on the same host? Using the openstack-on-lxd yaml files, you can create as separate compute nodes with each have a different virt-type

something like this:
nova-compute-kvm:
annotations:
gui-x: ‘250’
gui-y: ‘250’
charm: cs:nova-compute
num_units: 1
options:
enable-live-migration: False
enable-resize: False
migration-auth-type: ssh
force-raw-images: False
virt-type: kvm
cpu-mode: ‘host-passthrough’
ceph-osd-replication-count: 1
nova-compute-lxd:
annotations:
gui-x: ‘255’
gui-y: ‘255’
charm: cs:nova-compute
num_units: 1
options:
enable-live-migration: False
enable-resize: False
migration-auth-type: ssh
force-raw-images: False
virt-type: lxd
ceph-osd-replication-count: 1

14

You could have two compute nodes deployed in LXD containers on the same host. However, you will have no control over selecting a node to which an application will be deployed.

A better way to deploy two or more compute nodes would be to create additional cells. I used Juju charms to deploy additional cells.

15

Do you have an example I can review? As far I as knew, host aggregates was the only way to go for segregating virt-types. Using metadata tags and host aggregates is suppose to prompt the scheduler to spin up the instance where the correct virt-type is deployed.