Which LXD on a really old Kernel?

Hi all,

For reasons we use a rather dated 4.9 kernel. Currently we use Ubuntu 18.04 and LXD 3.0.3 for software development.

Is there any advice on what version of LXD I could use on 4.9?

I already built LXD 5.0, but it does not seem to really work (Kernel 5.4 being required, that was kind of anticipated). I don’t require security features, I use LXD as a lightweight (realtime capable) development VM isolated inside the host (apparmor disabled, containers privileged).

Thanks for any hints!
Christoph

Prior to LXD 5.0, the minimum kernel requirement was 3.13 so you can probably just install LXD 4.0/stable snap.

Thanks. (Probably) due to my old kernel, snaps don’t work anymore on 22.04.

I built lxd-4.0.9, but I’m stuck pretty much at the same place as before (with LXD 5)
I can run lxd (init was successul), but when I try to init a new container, it seems to work, but the new container is created in ERROR state.

~$ lxc init ubuntu:18.04 u3 -c security.privileged=true
Creating u3
~$ lxc ls
+------+-------+------+------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+------+-------+------+------+-----------+-----------+
| u3   | ERROR |      |      | CONTAINER | 0         |
+------+-------+------+------+-----------+-----------+

This happens on stock 5.15 as well as on my old 4.9. From here on I can do nothing at all with the container, not even delete it. Any ideas?

If you’re building LXD yourself, have you also built liblxc?

What does lxc info and lxc info u3 --show-log show?

Nope, I have not, I figured I could use the one provided by the system? (liblxc1, 1:5.0.0~git2209-g5a7b9ce67-0ubuntu1)

Logs below. This is on 5.15.0-25-generic with apparmor=0 and systemd.unified_cgroup_hierarchy=0

When I try to set a property on the container I get the following error from the server.

$ lxc config set u3 security.privileged=true
Error: Initialize LXC: Failed to set LXC config: lxc.logfile=/var/log/lxd/u3/lxc.log
$ lxc info u3 --show-log
Name: u3
Location: none
Remote: unix://
Architecture: x86_64
Created: 2022/04/14 09:45 CEST
Status: Error
Type: container
Profiles: default
Error: open /var/log/lxd/u3/lxc.log: no such file or directory
$ lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- resources_system
- usedby_consistency
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- storage_rsync_compression
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_state_vlan
- gpu_sriov
- migration_stateful
- disk_state_quota
- storage_ceph_features
- gpu_mig
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- network_counters_errors_dropped
- image_source_project
- database_leader
- instance_all_projects
- ceph_rbd_du
- qemu_metrics
- gpu_mig_uuid
- event_project
- instance_allow_inconsistent_copy
- image_restrictions
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICDTCCAZOgAwIBAgIRALGghmh6d5nJkFNMnwGdcbowCgYIKoZIzj0EAwMwNzEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzEXMBUGA1UEAwwOcm9vdEBjaDEw
    ZGQzMDcwHhcNMjIwNDE0MDU0NzU5WhcNMzIwNDExMDU0NzU5WjA3MRwwGgYDVQQK
    ExNsaW51eGNvbnRhaW5lcnMub3JnMRcwFQYDVQQDDA5yb290QGNoMTBkZDMwNzB2
    MBAGByqGSM49AgEGBSuBBAAiA2IABDxXKLmf+5eKhJwu2aTJg01NYU6EkVJyNKdw
    8q4HB7X3CXGIVnMKa1tb3USG41l8zmngCBL3fI8ihK2K1j/7T8ng9A7wroUI2oEQ
    gvuQ43tqaLuEZryWrqrB364J0vd7n6NjMGEwDgYDVR0PAQH/BAQDAgWgMBMGA1Ud
    JQQMMAoGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwLAYDVR0RBCUwI4IJY2gxMGRk
    MzA3hwR/AAABhxAAAAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2gAMGUCMQDE
    bPlRGgc0d2hfrEW6tk9yHq+L/9yQU5mTz6bfcSsCfvE3kJVSC252TBHLrTTTscQC
    MHKlIlGemNARrNJpPhRit6FjXW3cOBmmaRdV/muURXvk5pNPDbUMWPa5uRBL+ykD
    zA==
    -----END CERTIFICATE-----
  certificate_fingerprint: 8c63dc0a809807c78d86a64f9c8d9c5ddfc904bb6757acaad5f235a92e66cd3c
  driver: lxc
  driver_version: 5.0.0~git2209-g5a7b9ce67
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "true"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-25-generic
  lxc_features:
    cgroup2: "false"
    core_scheduling: "false"
    devpts_fd: "false"
    idmapped_mounts_v2: "false"
    mount_injection_file: "false"
    network_gateway_device_route: "false"
    network_ipvlan: "false"
    network_l2proxy: "false"
    network_phys_macvlan_mtu: "false"
    network_veth_router: "false"
    pidfd: "false"
    seccomp_allow_deny_syntax: "false"
    seccomp_notify: "false"
    seccomp_proxy_send_notify_fd: "false"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: false
  server_name: ch10dd307
  server_pid: 2965
  server_version: 4.0.9
  storage: btrfs
  storage_version: 5.16.2
  storage_supported_drivers:
  - name: btrfs
    version: 5.16.2
    remote: false
  - name: dir
    version: "1"
    remote: false

Does /var/log/lxd/ exist?

That should be fine.

Yes, it contains dnsmasq.lxdbr1.log and nothing else.

I just tried to run the tests in go-lxc (v2 branch), and they seem fine (even though I get awarded a non-zero exit status).

$ make test-privileged
==> Running tests for privileged user 
[sudo] password for ch1010832: 
=== RUN   TestVersion
    lxc_test.go:115: LXC version: 5.0.0~git2209-g5a7b9ce67
--- PASS: TestVersion (0.00s)
=== RUN   TestDefaultConfigPath
--- PASS: TestDefaultConfigPath (0.00s)
=== RUN   TestSetConfigPath
--- PASS: TestSetConfigPath (0.00s)
=== RUN   TestAcquire
--- PASS: TestAcquire (0.00s)
=== RUN   TestConcurrentDefined_Negative
    lxc_test.go:154: Skipping concurrent tests for now
--- SKIP: TestConcurrentDefined_Negative (0.00s)
=== RUN   TestDefined_Negative
--- PASS: TestDefined_Negative (0.00s)
=== RUN   TestExecute
    lxc_test.go:206: executing the command in a temporary container failed
--- FAIL: TestExecute (0.04s)
=== RUN   TestSetVerbosity
--- PASS: TestSetVerbosity (0.00s)
=== RUN   TestCreate
Using image from local cache
Unpacking the rootfs

---
You just created an Alpinelinux edge x86_64 (20220413_13:00) container.
--- PASS: TestCreate (0.20s)
=== RUN   TestClone
--- PASS: TestClone (0.09s)
=== RUN   TestCloneUsingOverlayfs
    lxc_test.go:248: skipping test as overlayfs support is missing.
--- SKIP: TestCloneUsingOverlayfs (0.00s)
=== RUN   TestCreateSnapshot
--- PASS: TestCreateSnapshot (0.09s)
=== RUN   TestCreateSnapshots
--- PASS: TestCreateSnapshots (0.28s)
=== RUN   TestRestoreSnapshot
--- PASS: TestRestoreSnapshot (0.09s)
=== RUN   TestConcurrentCreate
    lxc_test.go:312: Skipping concurrent tests for now
--- SKIP: TestConcurrentCreate (0.00s)
=== RUN   TestSnapshots
--- PASS: TestSnapshots (0.00s)
=== RUN   TestConcurrentStart
    lxc_test.go:353: Skipping concurrent tests for now
--- SKIP: TestConcurrentStart (0.00s)
=== RUN   TestConfigFileName
--- PASS: TestConfigFileName (0.00s)
=== RUN   TestDefined_Positive
--- PASS: TestDefined_Positive (0.00s)
=== RUN   TestConcurrentDefined_Positive
    lxc_test.go:408: Skipping concurrent tests for now
--- SKIP: TestConcurrentDefined_Positive (0.00s)
=== RUN   TestInitPid_Negative
--- PASS: TestInitPid_Negative (0.00s)
=== RUN   TestStart
--- PASS: TestStart (0.12s)
=== RUN   TestWaitIPAddresses
--- PASS: TestWaitIPAddresses (4.01s)
=== RUN   TestControllable
--- PASS: TestControllable (0.00s)
=== RUN   TestContainerNames
--- PASS: TestContainerNames (0.00s)
=== RUN   TestDefinedContainerNames
--- PASS: TestDefinedContainerNames (0.00s)
=== RUN   TestActiveContainerNames
--- PASS: TestActiveContainerNames (0.00s)
=== RUN   TestContainers
--- PASS: TestContainers (0.00s)
=== RUN   TestDefinedContainers
--- PASS: TestDefinedContainers (0.00s)
=== RUN   TestActiveContainers
--- PASS: TestActiveContainers (0.00s)
=== RUN   TestRunning
--- PASS: TestRunning (0.00s)
=== RUN   TestWantDaemonize
--- PASS: TestWantDaemonize (0.00s)
=== RUN   TestWantCloseAllFds
--- PASS: TestWantCloseAllFds (0.00s)
=== RUN   TestSetLogLevel
--- PASS: TestSetLogLevel (0.00s)
=== RUN   TestSetLogFile
--- PASS: TestSetLogFile (0.00s)
=== RUN   TestInitPid_Positive
--- PASS: TestInitPid_Positive (0.00s)
=== RUN   TestName
--- PASS: TestName (0.00s)
=== RUN   TestFreeze
--- PASS: TestFreeze (0.00s)
=== RUN   TestUnfreeze
--- PASS: TestUnfreeze (0.00s)
=== RUN   TestLoadConfigFile
--- PASS: TestLoadConfigFile (0.00s)
=== RUN   TestSaveConfigFile
--- PASS: TestSaveConfigFile (0.00s)
=== RUN   TestConfigItem
--- PASS: TestConfigItem (0.00s)
=== RUN   TestSetConfigItem
--- PASS: TestSetConfigItem (0.00s)
=== RUN   TestRunningConfigItem
--- PASS: TestRunningConfigItem (0.00s)
=== RUN   TestSetCgroupItem
--- PASS: TestSetCgroupItem (0.00s)
=== RUN   TestClearConfigItem
--- PASS: TestClearConfigItem (0.00s)
=== RUN   TestConfigKeys
--- PASS: TestConfigKeys (0.00s)
=== RUN   TestInterfaces
--- PASS: TestInterfaces (0.00s)
=== RUN   TestInterfaceStats
--- PASS: TestInterfaceStats (0.00s)
=== RUN   TestMemoryUsage
--- PASS: TestMemoryUsage (0.00s)
=== RUN   TestKernelMemoryUsage
--- PASS: TestKernelMemoryUsage (0.00s)
=== RUN   TestMemorySwapUsage
--- PASS: TestMemorySwapUsage (0.00s)
=== RUN   TestBlkioUsage
--- PASS: TestBlkioUsage (0.00s)
=== RUN   TestMemoryLimit
--- PASS: TestMemoryLimit (0.00s)
=== RUN   TestSoftMemoryLimit
--- PASS: TestSoftMemoryLimit (0.00s)
=== RUN   TestKernelMemoryLimit
--- PASS: TestKernelMemoryLimit (0.00s)
=== RUN   TestMemorySwapLimit
--- PASS: TestMemorySwapLimit (0.00s)
=== RUN   TestSetMemoryLimit
--- PASS: TestSetMemoryLimit (0.00s)
=== RUN   TestSetSoftMemoryLimit
--- PASS: TestSetSoftMemoryLimit (0.00s)
=== RUN   TestSetKernelMemoryLimit
    lxc_test.go:951: skipping the test as it requires memory.kmem.limit_in_bytes to be set
--- SKIP: TestSetKernelMemoryLimit (0.00s)
=== RUN   TestSetMemorySwapLimit
--- PASS: TestSetMemorySwapLimit (0.00s)
=== RUN   TestCPUTime
--- PASS: TestCPUTime (0.00s)
=== RUN   TestCPUTimePerCPU
--- PASS: TestCPUTimePerCPU (0.00s)
=== RUN   TestCPUStats
--- PASS: TestCPUStats (0.00s)
=== RUN   TestRunCommandNoWait
--- PASS: TestRunCommandNoWait (1.02s)
=== RUN   TestRunCommand
--- PASS: TestRunCommand (0.03s)
=== RUN   TestCommandWithEnv
--- PASS: TestCommandWithEnv (0.00s)
=== RUN   TestCommandWithEnvToKeep
--- PASS: TestCommandWithEnvToKeep (0.00s)
=== RUN   TestCommandWithCwd
--- PASS: TestCommandWithCwd (0.00s)
=== RUN   TestCommandWithUIDGID
--- PASS: TestCommandWithUIDGID (0.00s)
=== RUN   TestCommandWithArch
--- PASS: TestCommandWithArch (0.00s)
=== RUN   TestConsoleFd
--- PASS: TestConsoleFd (0.00s)
=== RUN   TestIPAddress
--- PASS: TestIPAddress (0.00s)
=== RUN   TestIPv4Address
--- PASS: TestIPv4Address (0.00s)
=== RUN   TestIPv46ddress
    lxc_test.go:1298: skipping test since lxc bridge does not have ipv6 address
--- SKIP: TestIPv46ddress (0.00s)
=== RUN   TestAddDeviceNode
    lxc_test.go:1318: skipping the test as it requires/dev/network_latency
--- SKIP: TestAddDeviceNode (0.00s)
=== RUN   TestRemoveDeviceNode
    lxc_test.go:1338: skipping the test as it requires/dev/network_latency
--- SKIP: TestRemoveDeviceNode (0.00s)
=== RUN   TestIPv4Addresses
--- PASS: TestIPv4Addresses (0.00s)
=== RUN   TestIPv6Addresses
    lxc_test.go:1366: skipping test since lxc bridge does not have ipv6 address
--- SKIP: TestIPv6Addresses (0.00s)
=== RUN   TestReboot
--- PASS: TestReboot (0.00s)
=== RUN   TestConcurrentShutdown
    lxc_test.go:1394: Skipping concurrent tests for now
--- SKIP: TestConcurrentShutdown (0.00s)
=== RUN   TestShutdown
--- PASS: TestShutdown (4.06s)
=== RUN   TestStop
--- PASS: TestStop (0.69s)
=== RUN   TestDestroySnapshot
--- PASS: TestDestroySnapshot (0.02s)
=== RUN   TestDestroyAllSnapshots
--- PASS: TestDestroyAllSnapshots (0.05s)
=== RUN   TestDestroy
--- PASS: TestDestroy (0.04s)
=== RUN   TestConcurrentDestroy
    lxc_test.go:1537: Skipping concurrent tests for now
--- SKIP: TestConcurrentDestroy (0.00s)
=== RUN   TestBackendStore
--- PASS: TestBackendStore (0.00s)
=== RUN   TestState
--- PASS: TestState (0.00s)
=== RUN   TestSupportedConfigItems
--- PASS: TestSupportedConfigItems (0.00s)
FAIL
coverage: 49.9% of statements
exit status 1
FAIL	github.com/lxc/go-lxc	10.868s
make: *** [Makefile:13: test-privileged] Error 1

I think the error comes from this place in go-lxc:
container.go:937

	if !bool(C.go_lxc_set_config_item(c.container, ckey, cvalue)) {
		return ErrSettingConfigItemFailed
	}

lxd calls this function from driver_lxc.go:692 (initLXC()).

Can I crank up debug output in liblxc or go-lxc using some environment variable to get more insight on what exactly fails in this simple call?

Please show ls -la /var/log/lxd and ls -la /var/lib/lxd?

lxdbr0 is from an earlier attempt with lxd 5. I forgot to delete the bridge and used lxdbr1 for lxd 4.

$ sudo ls -la /var/log/lxd
total 8
drwx------  2 root root   4096 Apr 14 10:02 .
drwxrwxr-x 15 root syslog 4096 Apr 14 09:56 ..
-rw-r--r--  1 root root      0 Apr 13 17:09 dnsmasq.lxdbr0.log
-rw-r--r--  1 root root      0 Apr 14 11:07 dnsmasq.lxdbr1.log
$ sudo ls -la /var/lib/lxd
total 64
drwx--x--x 16 root root 4096 Apr 14 11:07 .
drwxr-xr-x 73 root root 4096 Apr 14 10:23 ..
drwx------  2 root root 4096 Apr 14 07:47 backups
drwx--x--x  2 root root 4096 Apr 14 09:45 containers
drwx------  3 root root 4096 Apr 14 11:07 database
drwx--x--x  2 root root 4096 Apr 14 11:07 devices
drwxr-xr-x  2 root root   60 Apr 14 11:07 devlxd
drwx------  2 root root 4096 Apr 14 07:49 disks
drwx------  2 root root 4096 Apr 14 07:50 images
drwx--x--x  3 root root 4096 Apr 14 07:49 networks
srwx------  1 root root    0 Apr 14 11:07 seccomp.socket
drwx------  4 root root 4096 Apr 14 07:47 security
-rw-r--r--  1 root root  774 Apr 14 07:47 server.crt
-rw-------  1 root root  288 Apr 14 07:47 server.key
drwx--x--x  2 root root   40 Apr 14 11:07 shmounts
drwx------  2 root root 4096 Apr 14 07:47 snapshots
drwx--x--x  3 root root 4096 Apr 14 07:49 storage-pools
srw-rw----  1 root sudo    0 Apr 14 11:07 unix.socket
drwx--x--x  2 root root 4096 Apr 14 07:47 virtual-machines
drwx------  2 root root 4096 Apr 14 07:47 virtual-machines-snapshots

util.RuntimeLiblxcVersionAtLeast(2, 1, 0) does not go well with 5.0.0~git2209-g5a7b9ce67 and will return false, because it fails to convert 0~git2209-g5a7b9ce67 to a number. It will then try to set unsupported keys in lxc (lxc.logfile, which is now lxc.log.file).

I now get a different error, so progress :slight_smile:

You want this:

:partying_face:
Perfect! The error I mentioned previously disappeared by itself.
LXD 4 seem to work now on Ubuntu 22.04 with Kernel 5.15 and 4.9!

Thank you very much for your help!

$ lsb_release -a; uname -r; lxc ls
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu Jammy Jellyfish (development branch)
Release:        22.04
Codename:       jammy
4.9.178-realtime-6-rt131
+------+---------+-------------------+------+-----------+-----------+
| NAME |  STATE  |       IPV4        | IPV6 |   TYPE    | SNAPSHOTS |
+------+---------+-------------------+------+-----------+-----------+
| u1   | RUNNING | 10.0.3.199 (eth0) |      | CONTAINER | 0         |
+------+---------+-------------------+------+-----------+-----------+
1 Like