Introduction
The LXD team is very excited to announce the release of LXD 3.16!
This release includes a number of new features, configuration options and improvements to the command line tool.
Behind the scenes, a lot of work has gone into reworking the infrastructure used for container devices with the nic, infiniband and proxy devices having switched over to the new logic. This should result in much cleaner code that is easier to debug, better tests and more thorough error handling and configuration validation.
Enjoy!
New features
shift property for disk devices
It is now possible to request LXD setup a shiftfs overlay on any disk device.
This effectively allows exposing any disk or directory from your host system into an unprivileged container without having to resort to ACL/chown trickery to get the ownership right.
stgraber@castiana:~$ lxc config device add c1 home disk path=/home source=/home
Device home added to c1
stgraber@castiana:~$ lxc exec c1 -- ls -lh /home
total 537K
drwxr-xr-x 14 nobody nogroup 18 Mar 13 20:32 blah
drwx--x--x 33 nobody nogroup 86 Aug 9 22:25 stgraber
stgraber@castiana:~$ lxc config device remove c1 home
Device home removed from c1
stgraber@castiana:~$ lxc config device add c1 home disk path=/home source=/home shift=true
Device home added to c1
stgraber@castiana:~$ lxc exec c1 -- ls -lh /home
total 537K
drwxr-xr-x 14 ubuntu ubuntu 18 Mar 13 20:32 blah
drwx--x--x 33 201105 200512 86 Aug 9 22:25 stgraber
stgraber@castiana:~$
NOTE: This relies on shiftfs which requires an Ubuntu kernel of version 5.0 or higher and for snap users, must be opted into through a snap configuration. More details here.
security.shifted property for custom storage volumes
Building onto the new shift logic for disk devices, itâs also now possible to configure custom storage volumes to be attached to containers using shiftfs.
This now makes it possible to attach a shared custom volume to a mix of privileged, unprivileged and isolated containers.
stgraber@castiana:~$ lxc launch ubuntu:18.04 c1
Creating c1
Starting c1
stgraber@castiana:~$ lxc launch ubuntu:18.04 c2 -c security.privileged=true
Creating c2
Starting c2
stgraber@castiana:~$ lxc launch ubuntu:18.04 c3 -c security.idmap.isolated=true
Creating c3
Starting c3
stgraber@castiana:~$ lxc storage volume create default demo security.shifted=true
Storage volume demo created
stgraber@castiana:~$ lxc storage volume attach default demo c1 demo /demo
stgraber@castiana:~$ lxc storage volume attach default demo c2 demo /demo
stgraber@castiana:~$ lxc storage volume attach default demo c3 demo /demo
stgraber@castiana:~$ lxc exec c1 -- touch /demo/blah
stgraber@castiana:~$ lxc exec c1 -- chown 123:456 /demo/blah
stgraber@castiana:~$ lxc exec c2 -- ls -lh /demo
total 512
-rw-r--r-- 1 123 456 0 Aug 9 23:17 blah
stgraber@castiana:~$ lxc exec c3 -- ls -lh /demo
total 512
-rw-r--r-- 1 123 456 0 Aug 9 23:17 blah
stgraber@castiana:~$
NOTE: This relies on shiftfs which requires an Ubuntu kernel of version 5.0 or higher and for snap users, must be opted into through a snap configuration. More details here.
Empty container creation
Up until now, the only way to create a new container with the default command line tool was to use an existing image, be it local or remote.
LXD has long supported creating an empty container, but that mechanism was only available directly through the API and used by a number of data migration tools.
Following some user request for this, we now have a --empty option to lxc init which lets you create an empty container. Such a container cannot start and its filesystem must be manually populated, either through lxc file or by directly modifying it on the host system.
stgraber@castiana:~$ lxc init --empty c1
Creating c1
stgraber@castiana:~$ lxc list c1
+------+---------+------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+---------+------+------+------------+-----------+
| c1 | STOPPED | | | PERSISTENT | 0 |
+------+---------+------+------+------------+-----------+
Syscall interception configuration
The system call interception logic was extended to support setxattr on top of the existing mknod. At the same time, the feature was moved under a configuration key with each system call being individually toggable.
The two new options are:
- security.syscall.intercept.mknod
- security.syscall.intercept.setxattr
Both of them default to false and turning on this feature requires a 5.0 kernel, LXC 3.2 and an upstream snapshot of libseccomp.
Added infiniband data to resources API
Building onto the rework of the resources API from LXD 3.15, infiniband devices now report a bit more information, specifically the character devices used to drive them.
The relevant section of lxc info --resources now looks like:
Card 0:
NUMA node: 1
Vendor: Mellanox Technologies (15b3)
Product: MT27500 Family [ConnectX-3] (1003)
PCI address: 0000:82:00.0
Driver: mlx4_core (4.0-0)
Ports:
- Port 1 (ethernet)
ID: enp130s0d1
Address: 00:02:c9:a0:00:91
Supported modes: 1000baseKX/Full, 10000baseKX4/Full, 10000baseKR/Full
Supported ports: fibre
Port type: fibre
Transceiver type: internal
Auto negotiation: false
Link detected: false
- Port 0 (infiniband)
ID: ib0
Address: 80:00:0a:80:fe:80:00:00:00:00:00:00:00:02:c9:03:00:a0:00:91
Auto negotiation: false
Link detected: false
Infiniband:
IsSM: issm0 (231:64)
MAD: umad0 (231:0)
Verb: uverbs0 (231:192)
SR-IOV information:
Current number of VFs: 4
Maximum number of VFs: 31
VFs: 31
- NUMA node: 1
Vendor: Mellanox Technologies (15b3)
Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (1004)
PCI address: 0000:82:00.1
Driver: mlx4_core (4.0-0)
Ports:
- Port 1 (ethernet)
ID: enp130s0f1d1
Address: 5e:93:07:c6:ae:4c
Auto negotiation: false
Link detected: false
- Port 0 (infiniband)
ID: ib1
Address: 80:00:0a:81:fe:80:00:00:00:00:00:00:6a:fc:bc:b5:23:4f:ba:c9
Auto negotiation: false
Link detected: false
Infiniband:
IsSM: issm2 (231:66)
MAD: umad2 (231:2)
Verb: uverbs1 (231:193)
- NUMA node: 1
Vendor: Mellanox Technologies (15b3)
Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (1004)
PCI address: 0000:82:00.2
Driver: mlx4_core (4.0-0)
Ports:
- Port 1 (ethernet)
ID: enp130s0f2d1
Address: c2:21:28:88:3a:00
Auto negotiation: false
Link detected: false
- Port 0 (infiniband)
ID: ib2
Address: 80:00:0a:82:fe:80:00:00:00:00:00:00:ae:12:68:fa:cd:db:53:f1
Auto negotiation: false
Link detected: false
Infiniband:
IsSM: issm4 (231:68)
MAD: umad4 (231:4)
Verb: uverbs2 (231:194)
- NUMA node: 1
Vendor: Mellanox Technologies (15b3)
Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (1004)
PCI address: 0000:82:00.3
Driver: mlx4_core (4.0-0)
Ports:
- Port 1 (ethernet)
ID: enp130s0f3d1
Address: c6:f0:fb:b2:0b:81
Auto negotiation: false
Link detected: false
- Port 0 (infiniband)
ID: ib3
Address: 80:00:0a:83:fe:80:00:00:00:00:00:00:0a:94:39:75:2d:fe:6e:19
Auto negotiation: false
Link detected: false
Infiniband:
IsSM: issm6 (231:70)
MAD: umad6 (231:6)
Verb: uverbs3 (231:195)
- NUMA node: 1
Vendor: Mellanox Technologies (15b3)
Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] (1004)
PCI address: 0000:82:00.4
Driver: mlx4_core (4.0-0)
Ports:
- Port 1 (ethernet)
ID: enp130s0f4d1
Address: fa:4a:c2:9d:f0:2d
Auto negotiation: false
Link detected: false
- Port 0 (infiniband)
ID: ib4
Address: 80:00:0a:84:fe:80:00:00:00:00:00:00:0a:73:ab:6d:2c:c6:62:df
Auto negotiation: false
Link detected: false
Infiniband:
IsSM: issm8 (231:72)
MAD: umad8 (231:8)
Verb: uverbs4 (231:196)
This matches similar reporting used for DRM and NVIDIA device node information on GPUs.
Reworked set commands in client
In order to make things more consistent in the command line tool, all set commands now accept multiple key=value options. This makes it easier to set configuration that requires multiple keys be changed at the same time, without having to rely to the edit commands and using an interactive text editor.
Both new and old syntax work in parallel, with the key=value now being the preferred one.
stgraber@castiana:~$ lxc config set c1 user.foo 1
stgraber@castiana:~$ lxc config get c1 user.foo
1
stgraber@castiana:~$ lxc config set c1 user.foo=2 user.bar=3
stgraber@castiana:~$ lxc config get c1 user.foo
2
stgraber@castiana:~$ lxc config get c1 user.bar
3
--format option for all lists in client
Another command line tool improvement is that all of the list commands now behave the same and support the same --format option, allowing output in:
- table (default)
- csv
- json
- yaml
The full list of such commands is:
- lxc alias list
- lxc cluster list
- lxc config template list
- lxc config trust list
- lxc image list
- lxc image alias list
- lxc list
- lxc network list
- lxc network list-leases
- lxc operation list
- lxc profile list
- lxc project list
- lxc remote list
- lxc storage list
- lxc storage volume list
Support for combined images in simplestreams
It is now possible to publish a LXD combined image (single tarball) on a simplestreams image server by using the filetype lxd_combined.tar.gz.
This is an alternative to the more common (and flexible) split image which is made of distinct metadata (lxd.tar.xz) and rootfs (root.tar.xz or squashfs) files.
Bugs fixed
- bash: Update contain er options
- bash: Update device options
- bash: Update network options
- bash: Update server options
- bash: Update storage options
- doc: Update Github URLs
- doc/containers: Fix markdown escaping
- doc/containers: Remove incorrect host_name setting from ipvlan and macvlan
- doc/containers: Update container volatile keys for host_name
- doc/containers/: Update proxy bind settings to host or guest
- doc/containers: Update proxy default mode value to 0644
- doc/storage: Make descriptions consistent
- doc/userns: Update to match current behavior
- i18n: Update translations from weblate
- i18n: Update translation templates
- lxc/list: If no snapshots, show 0
- lxd: Update Github URLs
- lxd/cluster: Use hook for initial heartbeat
- lxd/cluster: Workaround watchFunc kicking heartbeat a bit too early
- lxd/cluster/hearbeat: Add new RaftID field to heartbeat Members struct
- lxd/cluster/membership: Fix new DB server id
- lxd/containers: Add access to containerLoadByProjectAndName from device
- lxd/containers: Add DevicesPath() function to interface
- lxd/containers: Add InstanceLoadNodeAll link to device package
- lxd/containers: Add state to containerValidDevices and updates references
- lxd/containers: Add Type function
- lxd/containers: Add Type() function to interface
- lxd/containers: Delete on creation failure
- lxd/containers: Donât validate liblxc version during config parsing
- lxd/containers: Fix infiniband support
- lxd/containers: Link device interface into LXD, removes unused functions
- lxd/containers: Move fillNetworkDevice into startInfiniband
- lxd/containers: NIC device validation to device interface
- lxd/containers: Remove migrated infiniband validation
- lxd/containers: Remove old infiniband logic
- lxd/containers: Remove proxy validation
- lxd/containers: Restore update of disk limit options
- lxd/containers: Restructure deviceStop to support post stop hooks
- lxd/containers: Rework MAAS calls
- lxd/containers: Update infiniband to use device package
- lxd/containers: Update references for proxy to use device package
- lxd/containers: Update references to deviceGetAttributes to device package
- lxd/containers: Update references to proxy functions and vars after move
- lxd/containers: Update use of device.New with device name
- lxd/containers: Update use of device.New() with device name
- lxd/daemon: Check for non-empty members list from heartbeat
- lxd/device: Add device interface and common device implementation
- lxd/device: Add device name to device structure
- lxd/device: Hook up proxy device
- lxd/device: Link up infiniband device
- lxd/device: Move device related functions into device package
- lxd/device: Remove single line if assignments
- lxd/device: Remove old static update fields list
- lxd/device: Update interface for Stop() to return RunConfig
- lxd/device/config: Move types.* to device.config.*
- lxd/device/errors: Add errors file for device error definitions
- lxd/device/infiniband: Add infiniband device loader
- lxd/device/infiniband/physical: Add new infiniband physical implementation
- lxd/device/infiniband/sriov: Add infiniband sriov device implementation
- lxd/device/instance/id: Add DevicesPath() function
- lxd/device/instance/id: Add functions to interface to expose config
- lxd/device/instance/id: Add instanceIdentifier interface
- lxd/device/instance/id: Add LogPath() to instance identifier interface
- lxd/device/nic: Add NIC device loader, nic type map and validation
- lxd/device/nic: Rename runConfig to runConf for consistency
- lxd/device/nic: Update comments to remove âcontainerâ references
- lxd/device/nic: Update nic devices to use new RunConfig format
- lxd/device/nic/bridged: Add bridged NIC device implementation
- lxd/device/nic/bridged: Add checks for DHCP being enabled if no static IP
- lxd/device/nic/bridged: Fix issue with non-dhcp, non-addressed parent device
- lxd/device/nic/bridged: Update for post stop hooks
- lxd/device/nic/ipvlan: Add IPVLAN NIC device implementation
- lxd/device/nic/macvlan: Add MACVLAN NIC device implementation
- lxd/device/nic/macvlan: Update for post stop hooks
- lxd/device/nic/p2p: Add P2P NIC device implementation
- lxd/device/nic/p2p: Update for post stop hooks
- lxd/device/nic/physical: Add physical NIC device implementation
- lxd/device/nic/physical: Update for post stop hooks
- lxd/device/nic/sriov: Add SR-IOV NIC device implementation
- lxd/device/nic/sriov: Fix mac_filtering when no hwaddr specified
- lxd/device/nic/sriov: Switch to use shared instanceGetReservedDevices
- lxd/device/nic/sriov: Update for post stop hooks
- lxd/device/nic/vlan: Update for post stop hooks
- lxd/device/proxy: Add proxy device implementation
- lxd/device/proxy: Implement default listen file mode of 0644
- lxd/device/proxy: Remove unnecessary CanHotPlug function
- lxd/device/proxy: Update for post stop hooks
- lxd/device/runconfig: Add CGroups slice to RunConfig
- lxd/device/runconfig: Add MountEntryItem struct definition for mounts
- lxd/device/runconfig: Add PostStartHooks and simplifies NetworkInterface
- lxd/device/runconfig: Add the struct types returned when a device is started
- lxd/device/runconfig: Rename PostStartHooks to PostHooks
- lxd/device/utils: Add cidr list validation functions
- lxd/device/utils: Add InstanceLoadNodeAll var
- lxd/device/utils: Add network{Snapshot,Restore}PhysicalNic and networkRestorePhysicalNic functions
- lxd/device/utils: Add NetworkSRIOVGetFreeVFInterface function
- lxd/device/utils: Add veth management functions
- lxd/device/utils: Bring VLAN parent interface up
- lxd/device/utils: Move IP validation functions from network_utils
- lxd/device/utils: Move NetworkAttachInterface to device_utils
- lxd/device/utils: Move networkGetDevMAC and networkSetDevMAC to device_utils
- lxd/device/utils: Move networkGetDevMTU and networkSetDevMTU to device_utils
- lxd/device/utils: Move networkGetHostDevice to device_utils
- lxd/device/utils: Move networkSysctlGet to device_utils
- lxd/device/utils: Move networkSysctlSet to device_utils
- lxd/device/utils: Rename createVlanDeviceIfNeeded to NetworkCreateVlanDeviceIfNeeded
- lxd/device/utils: Rename deviceNextVeth to NetworkRandomDevName
- lxd/device/utils: Rename deviceRemoveInterface to NetworkRemoveInterface
- lxd/device/utils/disk: Add disk management utils file
- lxd/device/utils/infiniband: Add infiniband utils file
- lxd/device/utils/instance: Add access to InstanceLoadByProjectAndName function
- lxd/device/utils/instance: Add instanceGetReservedDevices function
- lxd/device/utils/network: Move proxy related network functions into device package
- lxd/device/utils/proxy: Add proxy specific shared functions
- lxd/device/utils/unix: Add unix device utils file
- lxd/device/validate: Add device config validation framework
- lxd/device/validate: Update validation to understand infiniband has nictype
- lxd/dnsmasq: Adds dnsmasq package and updates usage
- lxd/dnsmasq: Adds RemoveStaticEntry function
- lxd/dnsmasq: Donât fail file deletion if missing
- lxd/dnsmasq: Removes RebuildConfig function link to networkUpdateStatic
- lxd/images: Fix crash on refresh error
- lxd/instance: Add new type instance
- lxd/internal: Remove OnNetworkUp hook command
- lxd/iptables: Moves iptables helper functions into own package
- lxd/maas: Do more configuration validation
- lxd/main_checkfeature: Add ifdef SECCOMP_GET_ACTION_AVAIL
- lxd/main_forkmount: Error on invalid calls
- lxd/main_forkmount: Fix cobra parsing
- lxd/main_forkmount: Properly exit on success
- lxd/main_forkproxy: Fix crash when listener cannot be setup
- lxd/main_forkproxy: Rework log messages to better define the different types
- lxd/main_forkproxy: Update references to shared types in device package
- lxd/main_forksyscall: Add and use setnsat()
- lxd/main_forksyscall: Add chdirchroot()
- lxd/main_forksyscall: Avoid calling close on garbage fd
- lxd/main_forksyscall: Donât break chdirchroot() with setns(CLONE_NEWNS)
- lxd/main_forksyscall: Fix variable declarations
- lxd/main_forksyscall: Harden open()-calls via O_PATH and O_DIRECTORY
- lxd/main_forksyscall: Introduce acquire_basic_creds()
- lxd/main_forksyscall: Protect CLONE_NEWCGROUP with ifdef
- lxd/main_forksyscall: Re-introduce setns(CLONE_NEWNS) properly
- lxd/main_forksyscall: Remove same_fsinfo() logic completely
- lxd/main_forksyscall: Remove st_ino check from same_fsinfo()
- lxd/main_forksyscall: Replace target_fd with cwd_fd
- lxd/main_forksyscall: Switch chdirchroot() and setns() order
- lxd/main_forksyscall: Use correct error handling for chdirchroot()
- lxd/networks/config: Update references to NetworkValidAddress
- lxd/networks/config: Update to use IP validation in device_utils
- lxd/networks/utils: Remove functions that are moved to device implementations
- lxd/networks/utils: Remove networkUpdateStaticContainer
- lxd/networks/utils: Remove networkValidAddress
- lxd/networks/utils: Remove unused IP validation functions
- lxd/project: Adds project package and updates references to it
- lxd/proxy: Remove unused code
- lxd/response: Fix SmartError
- lxd/seccomp: Abstract syscall handling
- lxd/seccomp: Always use setfattr
- lxd/seccomp: Bugfix, cleanup, and simplify
- lxd/seccomp: Donât hardcode ns type
- lxd/seccomp: Donât mask errors
- lxd/seccomp: Fix broken setxattr
- lxd/seccomp: Fix setattr of directories
- lxd/seccomp: Fix whiteout detection
- lxd/seccomp: Handle setxattr syscall
- lxd/seccomp: Only define Go arch (and include elf)
- lxd/seccomp: Remove shiftfs special-casing
- lxd/seccomp: Rename getSeccompProfileContent to seccompGetPolicyContent
- lxd/seccomp: Retrieve fs{g,u}id for mknod{at}() syscalls
- lxd/seccomp: Retrieve fs{g,u}id for setxattr() syscalls
- lxd/seccomp: Use int64 for uid/gid
- lxd/seccomp: Use LXD uidmap functions
- lxd/storage: Fix hangs on volume migration failures
- lxd/storage/ceph: Handle EBUSY on unmap
- lxd/storage/ceph: Slightly speed up creation
- lxd/storage/zfs: Fix transfer on encrypted pool
- lxd/storage/zfs: Properly wrap mount error
- lxd/storage/zfs: Properly wrap mount error
- lxd/task: Attempt to run tasks on schedule
- lxd/task/group: Move wait group Done() after g.running update to avoid race on task end
- Makefile: Include libraft and libco
- Makefile: Make it easier to build from tarball
- Makefile: Rename dist to _dist to avoid Go recursion
- Makefile: Update Github URLs
- shared: Donât open files to get their mode
- shared/container: Add IsNotEmpty to help with validation required fields
- shared/container: Add IsUnixUserID and IsOctalFileMode functions
- shared/osarch: Add more aliases
- shared/util: Remove BlockFsDetect as moved into device package
- tests: Add basic infiniband tests
- tests: Add nic bridged filtering tests for when DHCP is disabled
- tests: Add sleep for DHCP release tests for slower machines
- tests: Always pass -f to stop
- tests: Always use pg_num=1 during tests
- tests: Avoid ceph pool conflict
- tests: Avoid event forwarding race condition
- tests: Ensure SR-IOV tests remove all containers
- tests: Fix bridge tests detection of busybox udhcpc6 presence
- tests: Fix CEPH pool names
- tests: Fix proxy device unix tests on Ubuntu Eoan
- tests: Make shellcheck happy
- tests: Rename ct_name to ctName for consistent naming in NIC tests
- tests: Rename the proxy device tests to fit with other device tests
- tests: Update forkproxy tests
- tests: Update nic bridged filtering tests for non-IP addressed parent
- tests: Update NIC SR-IOV test to check for device reservation
- tests: Update NIC tests to check for volatile key cleanup
- tests: Update static_analysis.sh
- tests: Workaround race condition in image import event listener
Try it for yourself
This new LXD release is already available for you to try on our demo service.
Downloads
The release tarballs can be found on our download page.
Binary builds are also available for:
- Linux: snap install lxd
- MacOS: brew install lxc
- Windows: choco install lxc
