dbergloev
(Daniel B)
October 4, 2022, 11:32pm
1
Hi.
I am trying to move my network settings out of the containers (I have custom vlan setup) to organize it in the profiles that I have instead. However cloud-init does not seam to work for some reason.
I have flashed a fresh install of Ubuntu Server 22.04 to a test server where I have tried both LXD 5.0.1 and the latest 5.6. I have tried creating containers from ubuntu:22.04
and images:ubuntu/22.04/cloud
, but nothing works. I have also tried using cloud-init.network-config
as stated here: cloud-init - LXD documentation as well as user.network-config
as stated here: Custom network configuration with cloud-init - LXD documentation . But nothing is changed at all.
I tried using some of the other options, just to see if I could get something to work:
config:
cloud-init.user-data: |
runcmd:
- [touch, /cloud.init.ran]
But this fails as well. No file is created.
The only way to affect the network from a profile, was to create a mount from the host to /etc/netplan on the container, with a yaml file. This works, but I would much rather have cloud-init work properly.
tomp
(Thomas Parrott)
October 5, 2022, 7:51am
2
I just tested using LXD 5.6 and a fresh ubuntu/jammy/cloud
image and it works for me using this config:
lxc init images:ubuntu/jammy/cloud c1
lxc config edit c1 #Adds `cloud-init.user-data` key
lxc config show c1
architecture: x86_64
config:
cloud-init.user-data: |
#cloud-config
runcmd:
- touch /root/cloud.init.ran
image.architecture: amd64
image.description: Ubuntu jammy amd64 (20221004_08:30)
image.os: Ubuntu
image.release: jammy
image.serial: "20221004_08:30"
image.type: squashfs
image.variant: cloud
volatile.base_image: 3f53f1fd15686a9a7388cff159e79f87ec3c3435930af9fd529b76923b71ce39
volatile.cloud-init.instance-id: bf93214b-0af1-4df8-baaa-64d4774bd4da
volatile.eth0.host_name: veth7b7f1a0f
volatile.eth0.hwaddr: 00:16:3e:55:47:72
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
volatile.uuid: 999b441c-29b7-45e7-b87a-1e75bd68638e
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
lxc start c1
lxc exec c1 -- ls /root
cloud.init.ran
So make sure that the cloud-init config is set before the first start of the instance (as it only runs on first start) and make sure you’ve got the #cloud-config
line in your cloud-init config.
1 Like
dbergloev
(Daniel B)
October 5, 2022, 8:42am
3
I don’t get it, #cloud-config
is just a comment, why would this be required?
Also, if cloud-init only runs once, then it’s not really useful for my usecase. If I change or add profiles, the configurations for that profile should change along with it. This is no different from my current static netplan setup.
tomp
(Thomas Parrott)
October 5, 2022, 9:02am
4
My understanding that in both cases this is how cloud-init is designed, not a LXD specific thing.
You can see this in the cloud-init examples:
https://cloudinit.readthedocs.io/en/latest/topics/examples.html
As for when cloud-init runs, this is up to cloud-init, although I do believe that cloud-init will re-run when it thinks the ID of the instance has changed.
This was added fairly recently (and is in LXD 5.6):
lxc:master
← stgraber:cloud-init
opened 02:36AM - 17 Mar 22 UTC
And you can see what triggers an ID change here:
lxc:master
← stgraber:cloud-init
opened 02:36AM - 17 Mar 22 UTC
There’s a description of what changes trigger an ID change here:
opened 06:39PM - 19 Jan 22 UTC
closed 06:39PM - 18 Mar 22 UTC
# Issue description
Feature request:
**Requirements**:
Currently LXD p… ublishes a [static metadata.instance-id which matches the hostname](https://github.com/lxc/lxd/blob/master/lxd/devlxd.go#L104-L108) via the dev LXD API.
Cloud-init would like the `instance-id` to change at `/dev/lxd/sock /1.0/meta-data` anytime the underlying container or vm
configuration changes which tells cloud-init to reconfigure the system on next boot:
- 1.0/devices # new device added/removed to the machine (side-effect of a `lxc config device add|remove` call)
- 1.0/config/cloud-init.* # any cloud-init config keys
- ?? 1.0/config/user.meta-data # if still supported by LXD
- ??Any additional container migration/profile change that would need a require an update of network config within the container/VM
**Not a requirement**: Do not need any updates to NoCloud datasource seed files on a running container/VM.
# Steps to reproduce (once feature is implemented)
1. lxc launch ubuntu-daily:jammy test-instance-id-updates
2. lxc exec test-instance-id-updates -- cloud-init status --wait --long
3. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # check current instance-id
4. lxc config set test-instance-id-updates cloud-init.user-data
5. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # assert instance-id changed
6. lxc config set test-instance-id-updates cloud-init.network-config
7. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # assert instance-id changed
8. lxc config set test-instance-id-updates cloud-init.vendor-data
9. lxc exec test-instance-id-updates -- curl --unix-socket /dev/lxd/sock http://x/1.0/meta-data # assert instance-id changed
# background
On many clouds, cloud-init treats changes to `meta-data.instance-id` as an dirty-cache check from the platform that cloud-init needs to re-run initial system configuration because network config, user-data or meta-data on the system has changed which requires cloud-init to re-run as if it were a new deployment. Many clouds instrument `meta-data.instance-id` as a UUID that changes when underlying network, meta-data, vendor-data or user-data change for a given VM.
The new [LXDDatasource in cloud-init](https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceLXD.py#L257) consumes updated system instance-data from the LXD dev API @ /dev/lxd/sock and it will check for`instance-id` changes across reboot in order to determine of a config update is needed.
Required information
* Distribution: Looking to generically support *nix distributions this Dev LXD API behavior in cloud-init
* The output of "lxc info" or if that fails:
```
csmith@uptown:~$ lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIIB9zCCAX2gAwIBAgIQU3bmUn7ogbX2pOjlALWf7TAKBggqhkjOPQQDAzA0MRww
GgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRQwEgYDVQQDDAtyb290QHVwdG93
bjAeFw0xOTEwMjEyMDMzMzRaFw0yOTEwMTgyMDMzMzRaMDQxHDAaBgNVBAoTE2xp
bnV4Y29udGFpbmVycy5vcmcxFDASBgNVBAMMC3Jvb3RAdXB0b3duMHYwEAYHKoZI
zj0CAQYFK4EEACIDYgAEzZhp2YHDbTeZIhdxUaKj6JZ5Cpqi2K4qzAdZ243TIo6r
Ojzh9Q+7Ux9ZUBAXDuzDQEjpN2rSMzFLXxjy6PUUqn+ukSzNGERQYKKv3Rx2qKFs
3zgt3WGTP/CPvf0fHTY6o1QwUjAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYI
KwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHREEFjAUggZ1cHRvd26HBMCoAguH
BAqpKAEwCgYIKoZIzj0EAwMDaAAwZQIwDBSN4Xu1B8zP0oU3qDnvdltOmf95EBLf
s9CJwiovmwxAuyaFwIYJyGC/pWYH35uHAjEAwlIPBWQzs2qjP0L3Z+JI54al2N1N
iACdzozxVE9eHgGFNOTMppNbBpeKo4tudOab
-----END CERTIFICATE-----
certificate_fingerprint: 6aaceb8d2e522b490ee1d95bf2a31be73f1ed8bcf503709e55f630f1b7be0c58
driver: lxc | qemu
driver_version: 4.0.0 (devel) | 6.2.0
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
shiftfs: "false"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.4.0-71-generic
lxc_features:
cgroup2: "true"
core_scheduling: "true"
devpts_fd: "true"
idmapped_mounts_v2: "true"
mount_injection_file: "true"
network_gateway_device_route: "true"
network_ipvlan: "true"
network_l2proxy: "true"
network_phys_macvlan_mtu: "true"
network_veth_router: "true"
pidfd: "true"
seccomp_allow_deny_syntax: "true"
seccomp_notify: "true"
seccomp_proxy_send_notify_fd: "true"
os_name: Ubuntu
os_version: "20.04"
project: default
server: lxd
server_clustered: false
server_name: uptown
server_pid: 14127
server_version: "4.22"
storage: zfs | lvm
storage_version: 0.8.3-1ubuntu12.6 | 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30)
/ 4.41.0
storage_supported_drivers:
- name: ceph
version: 15.2.14
remote: true
- name: btrfs
version: 5.4.1
remote: false
- name: cephfs
version: 15.2.14
remote: true
- name: dir
version: "1"
remote: false
- name: lvm
version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.41.0
remote: false
- name: zfs
version: 0.8.3-1ubuntu12.6
remote: false
```
# Information to attach
- [ ] Any relevant kernel output (`dmesg`)
- [ ] Container log (`lxc info NAME --show-log`)
- [ ] Container configuration (`lxc config show NAME --expanded`)
- [ ] Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
- [ ] Output of the client with --debug
- [ ] Output of the daemon with --debug (alternatively output of `lxc monitor` while reproducing the issue)
Which may be sufficient for your use case.
dbergloev
(Daniel B)
October 5, 2022, 8:52pm
5
Thanks, this is great info. Also I was not aware that cloud-init was a one-time initialization feature. I thought it ran on every init instance.
1 Like