eraserix
April 13, 2022, 3:36pm
1
Hi all,
For reasons we use a rather dated 4.9 kernel. Currently we use Ubuntu 18.04 and LXD 3.0.3 for software development.
Is there any advice on what version of LXD I could use on 4.9?
I already built LXD 5.0, but it does not seem to really work (Kernel 5.4 being required, that was kind of anticipated). I don’t require security features, I use LXD as a lightweight (realtime capable) development VM isolated inside the host (apparmor disabled, containers privileged).
Thanks for any hints!
sdeziel
April 13, 2022, 6:51pm
2
Prior to LXD 5.0, the minimum kernel requirement was 3.13 so you can probably just install LXD 4.0/stable snap.
eraserix
April 14, 2022, 7:50am
3
Thanks. (Probably) due to my old kernel, snaps don’t work anymore on 22.04.
I built lxd-4.0.9, but I’m stuck pretty much at the same place as before (with LXD 5)
~$ lxc init ubuntu:18.04 u3 -c security.privileged=true
Creating u3
~$ lxc ls
+------+-------+------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+-----------+-----------+
| u3 | ERROR | | | CONTAINER | 0 |
+------+-------+------+------+-----------+-----------+
This happens on stock 5.15 as well as on my old 4.9. From here on I can do nothing at all with the container, not even delete it. Any ideas?
tomp
April 14, 2022, 8:09am
4
If you’re building LXD yourself, have you also built liblxc?
What does lxc info and lxc info u3 --show-log show?
eraserix
April 14, 2022, 8:34am
5
Nope, I have not, I figured I could use the one provided by the system? (liblxc1, 1:5.0.0~git2209-g5a7b9ce67-0ubuntu1)
Logs below. This is on 5.15.0-25-generic with apparmor=0 and systemd.unified_cgroup_hierarchy=0
When I try to set a property on the container I get the following error from the server.
$ lxc config set u3 security.privileged=true
Error: Initialize LXC: Failed to set LXC config: lxc.logfile=/var/log/lxd/u3/lxc.log
$ lxc info u3 --show-log
Name: u3
Location: none
Remote: unix://
Architecture: x86_64
Created: 2022/04/14 09:45 CEST
Status: Error
Type: container
Profiles: default
Error: open /var/log/lxd/u3/lxc.log: no such file or directory
$ lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- resources_system
- usedby_consistency
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- storage_rsync_compression
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_state_vlan
- gpu_sriov
- migration_stateful
- disk_state_quota
- storage_ceph_features
- gpu_mig
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- network_counters_errors_dropped
- image_source_project
- database_leader
- instance_all_projects
- ceph_rbd_du
- qemu_metrics
- gpu_mig_uuid
- event_project
- instance_allow_inconsistent_copy
- image_restrictions
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses: []
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
MIICDTCCAZOgAwIBAgIRALGghmh6d5nJkFNMnwGdcbowCgYIKoZIzj0EAwMwNzEc
MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEXMBUGA1UEAwwOcm9vdEBjaDEw
ZGQzMDcwHhcNMjIwNDE0MDU0NzU5WhcNMzIwNDExMDU0NzU5WjA3MRwwGgYDVQQK
ExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QGNoMTBkZDMwNzB2
MBAGByqGSM49AgEGBSuBBAAiA2IABDxXKLmf+5eKhJwu2aTJg01NYU6EkVJyNKdw
8q4HB7X3CXGIVnMKa1tb3USG41l8zmngCBL3fI8ihK2K1j/7T8ng9A7wroUI2oEQ
gvuQ43tqaLuEZryWrqrB364J0vd7n6NjMGEwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
JQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwLAYDVR0RBCUwI4IJY2gxMGRk
MzA3hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMQDE
bPlRGgc0d2hfrEW6tk9yHq+L/9yQU5mTz6bfcSsCfvE3kJVSC252TBHLrTTTscQC
MHKlIlGemNARrNJpPhRit6FjXW3cOBmmaRdV/muURXvk5pNPDbUMWPa5uRBL+ykD
zA==
-----END CERTIFICATE-----
certificate_fingerprint: 8c63dc0a809807c78d86a64f9c8d9c5ddfc904bb6757acaad5f235a92e66cd3c
driver: lxc
driver_version: 5.0.0~git2209-g5a7b9ce67
firewall: nftables
kernel: Linux
kernel_architecture: x86_64
kernel_features:
netnsid_getifaddrs: "true"
seccomp_listener: "true"
seccomp_listener_continue: "true"
shiftfs: "true"
uevent_injection: "true"
unpriv_fscaps: "true"
kernel_version: 5.15.0-25-generic
lxc_features:
cgroup2: "false"
core_scheduling: "false"
devpts_fd: "false"
idmapped_mounts_v2: "false"
mount_injection_file: "false"
network_gateway_device_route: "false"
network_ipvlan: "false"
network_l2proxy: "false"
network_phys_macvlan_mtu: "false"
network_veth_router: "false"
pidfd: "false"
seccomp_allow_deny_syntax: "false"
seccomp_notify: "false"
seccomp_proxy_send_notify_fd: "false"
os_name: Ubuntu
os_version: "22.04"
project: default
server: lxd
server_clustered: false
server_name: ch10dd307
server_pid: 2965
server_version: 4.0.9
storage: btrfs
storage_version: 5.16.2
storage_supported_drivers:
- name: btrfs
version: 5.16.2
remote: false
- name: dir
version: "1"
remote: false
tomp
April 14, 2022, 8:36am
6
Does /var/log/lxd/ exist?
eraserix
April 14, 2022, 8:48am
8
Yes, it contains dnsmasq.lxdbr1.log and nothing else.
I just tried to run the tests in go-lxc (v2 branch), and they seem fine (even though I get awarded a non-zero exit status).
$ make test-privileged
==> Running tests for privileged user
[sudo] password for ch1010832:
=== RUN TestVersion
lxc_test.go:115: LXC version: 5.0.0~git2209-g5a7b9ce67
--- PASS: TestVersion (0.00s)
=== RUN TestDefaultConfigPath
--- PASS: TestDefaultConfigPath (0.00s)
=== RUN TestSetConfigPath
--- PASS: TestSetConfigPath (0.00s)
=== RUN TestAcquire
--- PASS: TestAcquire (0.00s)
=== RUN TestConcurrentDefined_Negative
lxc_test.go:154: Skipping concurrent tests for now
--- SKIP: TestConcurrentDefined_Negative (0.00s)
=== RUN TestDefined_Negative
--- PASS: TestDefined_Negative (0.00s)
=== RUN TestExecute
lxc_test.go:206: executing the command in a temporary container failed
--- FAIL: TestExecute (0.04s)
=== RUN TestSetVerbosity
--- PASS: TestSetVerbosity (0.00s)
=== RUN TestCreate
Using image from local cache
Unpacking the rootfs
---
You just created an Alpinelinux edge x86_64 (20220413_13:00) container.
--- PASS: TestCreate (0.20s)
=== RUN TestClone
--- PASS: TestClone (0.09s)
=== RUN TestCloneUsingOverlayfs
lxc_test.go:248: skipping test as overlayfs support is missing.
--- SKIP: TestCloneUsingOverlayfs (0.00s)
=== RUN TestCreateSnapshot
--- PASS: TestCreateSnapshot (0.09s)
=== RUN TestCreateSnapshots
--- PASS: TestCreateSnapshots (0.28s)
=== RUN TestRestoreSnapshot
--- PASS: TestRestoreSnapshot (0.09s)
=== RUN TestConcurrentCreate
lxc_test.go:312: Skipping concurrent tests for now
--- SKIP: TestConcurrentCreate (0.00s)
=== RUN TestSnapshots
--- PASS: TestSnapshots (0.00s)
=== RUN TestConcurrentStart
lxc_test.go:353: Skipping concurrent tests for now
--- SKIP: TestConcurrentStart (0.00s)
=== RUN TestConfigFileName
--- PASS: TestConfigFileName (0.00s)
=== RUN TestDefined_Positive
--- PASS: TestDefined_Positive (0.00s)
=== RUN TestConcurrentDefined_Positive
lxc_test.go:408: Skipping concurrent tests for now
--- SKIP: TestConcurrentDefined_Positive (0.00s)
=== RUN TestInitPid_Negative
--- PASS: TestInitPid_Negative (0.00s)
=== RUN TestStart
--- PASS: TestStart (0.12s)
=== RUN TestWaitIPAddresses
--- PASS: TestWaitIPAddresses (4.01s)
=== RUN TestControllable
--- PASS: TestControllable (0.00s)
=== RUN TestContainerNames
--- PASS: TestContainerNames (0.00s)
=== RUN TestDefinedContainerNames
--- PASS: TestDefinedContainerNames (0.00s)
=== RUN TestActiveContainerNames
--- PASS: TestActiveContainerNames (0.00s)
=== RUN TestContainers
--- PASS: TestContainers (0.00s)
=== RUN TestDefinedContainers
--- PASS: TestDefinedContainers (0.00s)
=== RUN TestActiveContainers
--- PASS: TestActiveContainers (0.00s)
=== RUN TestRunning
--- PASS: TestRunning (0.00s)
=== RUN TestWantDaemonize
--- PASS: TestWantDaemonize (0.00s)
=== RUN TestWantCloseAllFds
--- PASS: TestWantCloseAllFds (0.00s)
=== RUN TestSetLogLevel
--- PASS: TestSetLogLevel (0.00s)
=== RUN TestSetLogFile
--- PASS: TestSetLogFile (0.00s)
=== RUN TestInitPid_Positive
--- PASS: TestInitPid_Positive (0.00s)
=== RUN TestName
--- PASS: TestName (0.00s)
=== RUN TestFreeze
--- PASS: TestFreeze (0.00s)
=== RUN TestUnfreeze
--- PASS: TestUnfreeze (0.00s)
=== RUN TestLoadConfigFile
--- PASS: TestLoadConfigFile (0.00s)
=== RUN TestSaveConfigFile
--- PASS: TestSaveConfigFile (0.00s)
=== RUN TestConfigItem
--- PASS: TestConfigItem (0.00s)
=== RUN TestSetConfigItem
--- PASS: TestSetConfigItem (0.00s)
=== RUN TestRunningConfigItem
--- PASS: TestRunningConfigItem (0.00s)
=== RUN TestSetCgroupItem
--- PASS: TestSetCgroupItem (0.00s)
=== RUN TestClearConfigItem
--- PASS: TestClearConfigItem (0.00s)
=== RUN TestConfigKeys
--- PASS: TestConfigKeys (0.00s)
=== RUN TestInterfaces
--- PASS: TestInterfaces (0.00s)
=== RUN TestInterfaceStats
--- PASS: TestInterfaceStats (0.00s)
=== RUN TestMemoryUsage
--- PASS: TestMemoryUsage (0.00s)
=== RUN TestKernelMemoryUsage
--- PASS: TestKernelMemoryUsage (0.00s)
=== RUN TestMemorySwapUsage
--- PASS: TestMemorySwapUsage (0.00s)
=== RUN TestBlkioUsage
--- PASS: TestBlkioUsage (0.00s)
=== RUN TestMemoryLimit
--- PASS: TestMemoryLimit (0.00s)
=== RUN TestSoftMemoryLimit
--- PASS: TestSoftMemoryLimit (0.00s)
=== RUN TestKernelMemoryLimit
--- PASS: TestKernelMemoryLimit (0.00s)
=== RUN TestMemorySwapLimit
--- PASS: TestMemorySwapLimit (0.00s)
=== RUN TestSetMemoryLimit
--- PASS: TestSetMemoryLimit (0.00s)
=== RUN TestSetSoftMemoryLimit
--- PASS: TestSetSoftMemoryLimit (0.00s)
=== RUN TestSetKernelMemoryLimit
lxc_test.go:951: skipping the test as it requires memory.kmem.limit_in_bytes to be set
--- SKIP: TestSetKernelMemoryLimit (0.00s)
=== RUN TestSetMemorySwapLimit
--- PASS: TestSetMemorySwapLimit (0.00s)
=== RUN TestCPUTime
--- PASS: TestCPUTime (0.00s)
=== RUN TestCPUTimePerCPU
--- PASS: TestCPUTimePerCPU (0.00s)
=== RUN TestCPUStats
--- PASS: TestCPUStats (0.00s)
=== RUN TestRunCommandNoWait
--- PASS: TestRunCommandNoWait (1.02s)
=== RUN TestRunCommand
--- PASS: TestRunCommand (0.03s)
=== RUN TestCommandWithEnv
--- PASS: TestCommandWithEnv (0.00s)
=== RUN TestCommandWithEnvToKeep
--- PASS: TestCommandWithEnvToKeep (0.00s)
=== RUN TestCommandWithCwd
--- PASS: TestCommandWithCwd (0.00s)
=== RUN TestCommandWithUIDGID
--- PASS: TestCommandWithUIDGID (0.00s)
=== RUN TestCommandWithArch
--- PASS: TestCommandWithArch (0.00s)
=== RUN TestConsoleFd
--- PASS: TestConsoleFd (0.00s)
=== RUN TestIPAddress
--- PASS: TestIPAddress (0.00s)
=== RUN TestIPv4Address
--- PASS: TestIPv4Address (0.00s)
=== RUN TestIPv46ddress
lxc_test.go:1298: skipping test since lxc bridge does not have ipv6 address
--- SKIP: TestIPv46ddress (0.00s)
=== RUN TestAddDeviceNode
lxc_test.go:1318: skipping the test as it requires/dev/network_latency
--- SKIP: TestAddDeviceNode (0.00s)
=== RUN TestRemoveDeviceNode
lxc_test.go:1338: skipping the test as it requires/dev/network_latency
--- SKIP: TestRemoveDeviceNode (0.00s)
=== RUN TestIPv4Addresses
--- PASS: TestIPv4Addresses (0.00s)
=== RUN TestIPv6Addresses
lxc_test.go:1366: skipping test since lxc bridge does not have ipv6 address
--- SKIP: TestIPv6Addresses (0.00s)
=== RUN TestReboot
--- PASS: TestReboot (0.00s)
=== RUN TestConcurrentShutdown
lxc_test.go:1394: Skipping concurrent tests for now
--- SKIP: TestConcurrentShutdown (0.00s)
=== RUN TestShutdown
--- PASS: TestShutdown (4.06s)
=== RUN TestStop
--- PASS: TestStop (0.69s)
=== RUN TestDestroySnapshot
--- PASS: TestDestroySnapshot (0.02s)
=== RUN TestDestroyAllSnapshots
--- PASS: TestDestroyAllSnapshots (0.05s)
=== RUN TestDestroy
--- PASS: TestDestroy (0.04s)
=== RUN TestConcurrentDestroy
lxc_test.go:1537: Skipping concurrent tests for now
--- SKIP: TestConcurrentDestroy (0.00s)
=== RUN TestBackendStore
--- PASS: TestBackendStore (0.00s)
=== RUN TestState
--- PASS: TestState (0.00s)
=== RUN TestSupportedConfigItems
--- PASS: TestSupportedConfigItems (0.00s)
FAIL
coverage: 49.9% of statements
exit status 1
FAIL github.com/lxc/go-lxc 10.868s
make: *** [Makefile:13: test-privileged] Error 1
eraserix
April 14, 2022, 8:57am
9
I think the error comes from this place in go-lxc:
if !bool(C.go_lxc_set_config_item(c.container, ckey, cvalue)) {
return ErrSettingConfigItemFailed
}
lxd calls this function from driver_lxc.go:692 (initLXC()).
Can I crank up debug output in liblxc or go-lxc using some environment variable to get more insight on what exactly fails in this simple call?
tomp
April 14, 2022, 9:12am
10
Please show ls -la /var/log/lxd and ls -la /var/lib/lxd?
eraserix
April 14, 2022, 9:21am
11
lxdbr0 is from an earlier attempt with lxd 5. I forgot to delete the bridge and used lxdbr1 for lxd 4.
$ sudo ls -la /var/log/lxd
total 8
drwx------ 2 root root 4096 Apr 14 10:02 .
drwxrwxr-x 15 root syslog 4096 Apr 14 09:56 ..
-rw-r--r-- 1 root root 0 Apr 13 17:09 dnsmasq.lxdbr0.log
-rw-r--r-- 1 root root 0 Apr 14 11:07 dnsmasq.lxdbr1.log
$ sudo ls -la /var/lib/lxd
total 64
drwx--x--x 16 root root 4096 Apr 14 11:07 .
drwxr-xr-x 73 root root 4096 Apr 14 10:23 ..
drwx------ 2 root root 4096 Apr 14 07:47 backups
drwx--x--x 2 root root 4096 Apr 14 09:45 containers
drwx------ 3 root root 4096 Apr 14 11:07 database
drwx--x--x 2 root root 4096 Apr 14 11:07 devices
drwxr-xr-x 2 root root 60 Apr 14 11:07 devlxd
drwx------ 2 root root 4096 Apr 14 07:49 disks
drwx------ 2 root root 4096 Apr 14 07:50 images
drwx--x--x 3 root root 4096 Apr 14 07:49 networks
srwx------ 1 root root 0 Apr 14 11:07 seccomp.socket
drwx------ 4 root root 4096 Apr 14 07:47 security
-rw-r--r-- 1 root root 774 Apr 14 07:47 server.crt
-rw------- 1 root root 288 Apr 14 07:47 server.key
drwx--x--x 2 root root 40 Apr 14 11:07 shmounts
drwx------ 2 root root 4096 Apr 14 07:47 snapshots
drwx--x--x 3 root root 4096 Apr 14 07:49 storage-pools
srw-rw---- 1 root sudo 0 Apr 14 11:07 unix.socket
drwx--x--x 2 root root 4096 Apr 14 07:47 virtual-machines
drwx------ 2 root root 4096 Apr 14 07:47 virtual-machines-snapshots
eraserix
April 14, 2022, 9:51am
12
util.RuntimeLiblxcVersionAtLeast(2, 1, 0) does not go well with 5.0.0~git2209-g5a7b9ce67 and will return false, because it fails to convert 0~git2209-g5a7b9ce67 to a number. It will then try to set unsupported keys in lxc (lxc.logfile, which is now lxc.log.file).
I now get a different error, so progress
eraserix
April 14, 2022, 11:18am
14
Thank you very much for your help!
$ lsb_release -a; uname -r; lxc ls
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Jammy Jellyfish (development branch)
Release: 22.04
Codename: jammy
4.9.178-realtime-6-rt131
+------+---------+-------------------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+-------------------+------+-----------+-----------+
| u1 | RUNNING | 10.0.3.199 (eth0) | | CONTAINER | 0 |
+------+---------+-------------------+------+-----------+-----------+
1 Like
