LXD 3.16 has been released

Introduction

The LXD team is very excited to announce the release of LXD 3.16!

This release includes a number of new features, configuration options and improvements to the command line tool.

Behind the scenes, a lot of work has gone into reworking the infrastructure used for container devices with the nic, infiniband and proxy devices having switched over to the new logic. This should result in much cleaner code that is easier to debug, better tests and more thorough error handling and configuration validation.

Enjoy!

New features

shift property for disk devices

It is now possible to request LXD setup a shiftfs overlay on any disk device.

This effectively allows exposing any disk or directory from your host system into an unprivileged container without having to resort to ACL/chown trickery to get the ownership right.

stgraber@castiana:~$ lxc config device add c1 home disk path=/home source=/home
Device home added to c1
stgraber@castiana:~$ lxc exec c1 -- ls -lh /home
total 537K
drwxr-xr-x 14 nobody nogroup 18 Mar 13 20:32 blah
drwx--x--x 33 nobody nogroup 86 Aug  9 22:25 stgraber
stgraber@castiana:~$ lxc config device remove c1 home
Device home removed from c1

stgraber@castiana:~$ lxc config device add c1 home disk path=/home source=/home     shift=true
Device home added to c1
stgraber@castiana:~$ lxc exec c1 -- ls -lh /home
total 537K
drwxr-xr-x 14 ubuntu ubuntu 18 Mar 13 20:32 blah
drwx--x--x 33 201105 200512 86 Aug  9 22:25 stgraber
stgraber@castiana:~$ 

NOTE: This relies on shiftfs which requires an Ubuntu kernel of version 5.0 or higher and for snap users, must be opted into through a snap configuration. More details here.

security.shifted property for custom storage volumes

Building onto the new shift logic for disk devices, it’s also now possible to configure custom storage volumes to be attached to containers using shiftfs.

This now makes it possible to attach a shared custom volume to a mix of privileged, unprivileged and isolated containers.

stgraber@castiana:~$ lxc launch ubuntu:18.04 c1
Creating c1
Starting c1
stgraber@castiana:~$ lxc launch ubuntu:18.04 c2 -c security.privileged=true
Creating c2
Starting c2
stgraber@castiana:~$ lxc launch ubuntu:18.04 c3 -c security.idmap.isolated=true
Creating c3
Starting c3

stgraber@castiana:~$ lxc storage volume create default demo security.shifted=true
Storage volume demo created
stgraber@castiana:~$ lxc storage volume attach default demo c1 demo /demo
stgraber@castiana:~$ lxc storage volume attach default demo c2 demo /demo
stgraber@castiana:~$ lxc storage volume attach default demo c3 demo /demo

stgraber@castiana:~$ lxc exec c1 -- touch /demo/blah
stgraber@castiana:~$ lxc exec c1 -- chown 123:456 /demo/blah
stgraber@castiana:~$ lxc exec c2 -- ls -lh /demo
total 512
-rw-r--r-- 1 123 456 0 Aug  9 23:17 blah
stgraber@castiana:~$ lxc exec c3 -- ls -lh /demo
total 512
-rw-r--r-- 1 123 456 0 Aug  9 23:17 blah
stgraber@castiana:~$ 

NOTE: This relies on shiftfs which requires an Ubuntu kernel of version 5.0 or higher and for snap users, must be opted into through a snap configuration. More details here.

Empty container creation

Up until now, the only way to create a new container with the default command line tool was to use an existing image, be it local or remote.

LXD has long supported creating an empty container, but that mechanism was only available directly through the API and used by a number of data migration tools.

Following some user request for this, we now have a --empty option to lxc init which lets you create an empty container. Such a container cannot start and its filesystem must be manually populated, either through lxc file or by directly modifying it on the host system.

stgraber@castiana:~$ lxc init --empty c1
Creating c1
stgraber@castiana:~$ lxc list c1
+------+---------+------+------+------------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+------+---------+------+------+------------+-----------+
| c1   | STOPPED |      |      | PERSISTENT | 0         |
+------+---------+------+------+------------+-----------+

Syscall interception configuration

The system call interception logic was extended to support setxattr on top of the existing mknod. At the same time, the feature was moved under a configuration key with each system call being individually toggable.

The two new options are:

  • security.syscall.intercept.mknod
  • security.syscall.intercept.setxattr

Both of them default to false and turning on this feature requires a 5.0 kernel, LXC 3.2 and an upstream snapshot of libseccomp.

Added infiniband data to resources API

Building onto the rework of the resources API from LXD 3.15, infiniband devices now report a bit more information, specifically the character devices used to drive them.

The relevant section of lxc info --resources now looks like:

Card 0:
    NUMA node: 1
    Vendor: Mellanox Technologies (15b3)
    Product: MT27500 Family [ConnectX-3] (1003)
    PCI address: 0000:82:00.0
    Driver: mlx4_core (4.0-0)
    Ports:
      - Port 1 (ethernet)
        ID: enp130s0d1
        Address: 00:02:c9:a0:00:91
        Supported modes: 1000baseKX/Full, 10000baseKX4/Full, 10000baseKR/Full
        Supported ports: fibre
        Port type: fibre
        Transceiver type: internal
        Auto negotiation: false
        Link detected: false
      - Port 0 (infiniband)
        ID: ib0
        Address: 80:00:0a:80:fe:80:00:00:00:00:00:00:00:02:c9:03:00:a0:00:91
        Auto negotiation: false
        Link detected: false
        Infiniband:
          IsSM: issm0 (231:64)
          MAD: umad0 (231:0)
          Verb: uverbs0 (231:192)
    SR-IOV information:
      Current number of VFs: 4
      Maximum number of VFs: 31
      VFs: 31
      - NUMA node: 1
        Vendor: Mellanox Technologies (15b3)
    Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]     (1004)
        PCI address: 0000:82:00.1
        Driver: mlx4_core (4.0-0)
        Ports:
          - Port 1 (ethernet)
            ID: enp130s0f1d1
            Address: 5e:93:07:c6:ae:4c
            Auto negotiation: false
            Link detected: false
          - Port 0 (infiniband)
            ID: ib1
            Address: 80:00:0a:81:fe:80:00:00:00:00:00:00:6a:fc:bc:b5:23:4f:ba:c9
            Auto negotiation: false
            Link detected: false
            Infiniband:
              IsSM: issm2 (231:66)
              MAD: umad2 (231:2)
              Verb: uverbs1 (231:193)
      - NUMA node: 1
        Vendor: Mellanox Technologies (15b3)
    Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]     (1004)
        PCI address: 0000:82:00.2
        Driver: mlx4_core (4.0-0)
        Ports:
          - Port 1 (ethernet)
            ID: enp130s0f2d1
            Address: c2:21:28:88:3a:00
            Auto negotiation: false
            Link detected: false
          - Port 0 (infiniband)
            ID: ib2
            Address: 80:00:0a:82:fe:80:00:00:00:00:00:00:ae:12:68:fa:cd:db:53:f1
            Auto negotiation: false
            Link detected: false
            Infiniband:
              IsSM: issm4 (231:68)
              MAD: umad4 (231:4)
              Verb: uverbs2 (231:194)
      - NUMA node: 1
        Vendor: Mellanox Technologies (15b3)
    Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]     (1004)
        PCI address: 0000:82:00.3
        Driver: mlx4_core (4.0-0)
        Ports:
          - Port 1 (ethernet)
            ID: enp130s0f3d1
            Address: c6:f0:fb:b2:0b:81
            Auto negotiation: false
            Link detected: false
          - Port 0 (infiniband)
            ID: ib3
            Address: 80:00:0a:83:fe:80:00:00:00:00:00:00:0a:94:39:75:2d:fe:6e:19
            Auto negotiation: false
            Link detected: false
            Infiniband:
              IsSM: issm6 (231:70)
              MAD: umad6 (231:6)
              Verb: uverbs3 (231:195)
      - NUMA node: 1
        Vendor: Mellanox Technologies (15b3)
    Product: MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]     (1004)
        PCI address: 0000:82:00.4
        Driver: mlx4_core (4.0-0)
        Ports:
          - Port 1 (ethernet)
            ID: enp130s0f4d1
            Address: fa:4a:c2:9d:f0:2d
            Auto negotiation: false
            Link detected: false
          - Port 0 (infiniband)
            ID: ib4
            Address: 80:00:0a:84:fe:80:00:00:00:00:00:00:0a:73:ab:6d:2c:c6:62:df
            Auto negotiation: false
            Link detected: false
            Infiniband:
              IsSM: issm8 (231:72)
              MAD: umad8 (231:8)
              Verb: uverbs4 (231:196)

This matches similar reporting used for DRM and NVIDIA device node information on GPUs.

Reworked set commands in client

In order to make things more consistent in the command line tool, all set commands now accept multiple key=value options. This makes it easier to set configuration that requires multiple keys be changed at the same time, without having to rely to the edit commands and using an interactive text editor.

Both new and old syntax work in parallel, with the key=value now being the preferred one.

stgraber@castiana:~$ lxc config set c1 user.foo 1
stgraber@castiana:~$ lxc config get c1 user.foo
1
stgraber@castiana:~$ lxc config set c1 user.foo=2 user.bar=3
stgraber@castiana:~$ lxc config get c1 user.foo
2
stgraber@castiana:~$ lxc config get c1 user.bar
3

--format option for all lists in client

Another command line tool improvement is that all of the list commands now behave the same and support the same --format option, allowing output in:

  • table (default)
  • csv
  • json
  • yaml

The full list of such commands is:

  • lxc alias list
  • lxc cluster list
  • lxc config template list
  • lxc config trust list
  • lxc image list
  • lxc image alias list
  • lxc list
  • lxc network list
  • lxc network list-leases
  • lxc operation list
  • lxc profile list
  • lxc project list
  • lxc remote list
  • lxc storage list
  • lxc storage volume list

Support for combined images in simplestreams

It is now possible to publish a LXD combined image (single tarball) on a simplestreams image server by using the filetype lxd_combined.tar.gz.

This is an alternative to the more common (and flexible) split image which is made of distinct metadata (lxd.tar.xz) and rootfs (root.tar.xz or squashfs) files.

Bugs fixed

  • bash: Update contain er options
  • bash: Update device options
  • bash: Update network options
  • bash: Update server options
  • bash: Update storage options
  • doc: Update Github URLs
  • doc/containers: Fix markdown escaping
  • doc/containers: Remove incorrect host_name setting from ipvlan and macvlan
  • doc/containers: Update container volatile keys for host_name
  • doc/containers/: Update proxy bind settings to host or guest
  • doc/containers: Update proxy default mode value to 0644
  • doc/storage: Make descriptions consistent
  • doc/userns: Update to match current behavior
  • i18n: Update translations from weblate
  • i18n: Update translation templates
  • lxc/list: If no snapshots, show 0
  • lxd: Update Github URLs
  • lxd/cluster: Use hook for initial heartbeat
  • lxd/cluster: Workaround watchFunc kicking heartbeat a bit too early
  • lxd/cluster/hearbeat: Add new RaftID field to heartbeat Members struct
  • lxd/cluster/membership: Fix new DB server id
  • lxd/containers: Add access to containerLoadByProjectAndName from device
  • lxd/containers: Add DevicesPath() function to interface
  • lxd/containers: Add InstanceLoadNodeAll link to device package
  • lxd/containers: Add state to containerValidDevices and updates references
  • lxd/containers: Add Type function
  • lxd/containers: Add Type() function to interface
  • lxd/containers: Delete on creation failure
  • lxd/containers: Don’t validate liblxc version during config parsing
  • lxd/containers: Fix infiniband support
  • lxd/containers: Link device interface into LXD, removes unused functions
  • lxd/containers: Move fillNetworkDevice into startInfiniband
  • lxd/containers: NIC device validation to device interface
  • lxd/containers: Remove migrated infiniband validation
  • lxd/containers: Remove old infiniband logic
  • lxd/containers: Remove proxy validation
  • lxd/containers: Restore update of disk limit options
  • lxd/containers: Restructure deviceStop to support post stop hooks
  • lxd/containers: Rework MAAS calls
  • lxd/containers: Update infiniband to use device package
  • lxd/containers: Update references for proxy to use device package
  • lxd/containers: Update references to deviceGetAttributes to device package
  • lxd/containers: Update references to proxy functions and vars after move
  • lxd/containers: Update use of device.New with device name
  • lxd/containers: Update use of device.New() with device name
  • lxd/daemon: Check for non-empty members list from heartbeat
  • lxd/device: Add device interface and common device implementation
  • lxd/device: Add device name to device structure
  • lxd/device: Hook up proxy device
  • lxd/device: Link up infiniband device
  • lxd/device: Move device related functions into device package
  • lxd/device: Remove single line if assignments
  • lxd/device: Remove old static update fields list
  • lxd/device: Update interface for Stop() to return RunConfig
  • lxd/device/config: Move types.* to device.config.*
  • lxd/device/errors: Add errors file for device error definitions
  • lxd/device/infiniband: Add infiniband device loader
  • lxd/device/infiniband/physical: Add new infiniband physical implementation
  • lxd/device/infiniband/sriov: Add infiniband sriov device implementation
  • lxd/device/instance/id: Add DevicesPath() function
  • lxd/device/instance/id: Add functions to interface to expose config
  • lxd/device/instance/id: Add instanceIdentifier interface
  • lxd/device/instance/id: Add LogPath() to instance identifier interface
  • lxd/device/nic: Add NIC device loader, nic type map and validation
  • lxd/device/nic: Rename runConfig to runConf for consistency
  • lxd/device/nic: Update comments to remove “container” references
  • lxd/device/nic: Update nic devices to use new RunConfig format
  • lxd/device/nic/bridged: Add bridged NIC device implementation
  • lxd/device/nic/bridged: Add checks for DHCP being enabled if no static IP
  • lxd/device/nic/bridged: Fix issue with non-dhcp, non-addressed parent device
  • lxd/device/nic/bridged: Update for post stop hooks
  • lxd/device/nic/ipvlan: Add IPVLAN NIC device implementation
  • lxd/device/nic/macvlan: Add MACVLAN NIC device implementation
  • lxd/device/nic/macvlan: Update for post stop hooks
  • lxd/device/nic/p2p: Add P2P NIC device implementation
  • lxd/device/nic/p2p: Update for post stop hooks
  • lxd/device/nic/physical: Add physical NIC device implementation
  • lxd/device/nic/physical: Update for post stop hooks
  • lxd/device/nic/sriov: Add SR-IOV NIC device implementation
  • lxd/device/nic/sriov: Fix mac_filtering when no hwaddr specified
  • lxd/device/nic/sriov: Switch to use shared instanceGetReservedDevices
  • lxd/device/nic/sriov: Update for post stop hooks
  • lxd/device/nic/vlan: Update for post stop hooks
  • lxd/device/proxy: Add proxy device implementation
  • lxd/device/proxy: Implement default listen file mode of 0644
  • lxd/device/proxy: Remove unnecessary CanHotPlug function
  • lxd/device/proxy: Update for post stop hooks
  • lxd/device/runconfig: Add CGroups slice to RunConfig
  • lxd/device/runconfig: Add MountEntryItem struct definition for mounts
  • lxd/device/runconfig: Add PostStartHooks and simplifies NetworkInterface
  • lxd/device/runconfig: Add the struct types returned when a device is started
  • lxd/device/runconfig: Rename PostStartHooks to PostHooks
  • lxd/device/utils: Add cidr list validation functions
  • lxd/device/utils: Add InstanceLoadNodeAll var
  • lxd/device/utils: Add network{Snapshot,Restore}PhysicalNic and networkRestorePhysicalNic functions
  • lxd/device/utils: Add NetworkSRIOVGetFreeVFInterface function
  • lxd/device/utils: Add veth management functions
  • lxd/device/utils: Bring VLAN parent interface up
  • lxd/device/utils: Move IP validation functions from network_utils
  • lxd/device/utils: Move NetworkAttachInterface to device_utils
  • lxd/device/utils: Move networkGetDevMAC and networkSetDevMAC to device_utils
  • lxd/device/utils: Move networkGetDevMTU and networkSetDevMTU to device_utils
  • lxd/device/utils: Move networkGetHostDevice to device_utils
  • lxd/device/utils: Move networkSysctlGet to device_utils
  • lxd/device/utils: Move networkSysctlSet to device_utils
  • lxd/device/utils: Rename createVlanDeviceIfNeeded to NetworkCreateVlanDeviceIfNeeded
  • lxd/device/utils: Rename deviceNextVeth to NetworkRandomDevName
  • lxd/device/utils: Rename deviceRemoveInterface to NetworkRemoveInterface
  • lxd/device/utils/disk: Add disk management utils file
  • lxd/device/utils/infiniband: Add infiniband utils file
  • lxd/device/utils/instance: Add access to InstanceLoadByProjectAndName function
  • lxd/device/utils/instance: Add instanceGetReservedDevices function
  • lxd/device/utils/network: Move proxy related network functions into device package
  • lxd/device/utils/proxy: Add proxy specific shared functions
  • lxd/device/utils/unix: Add unix device utils file
  • lxd/device/validate: Add device config validation framework
  • lxd/device/validate: Update validation to understand infiniband has nictype
  • lxd/dnsmasq: Adds dnsmasq package and updates usage
  • lxd/dnsmasq: Adds RemoveStaticEntry function
  • lxd/dnsmasq: Don’t fail file deletion if missing
  • lxd/dnsmasq: Removes RebuildConfig function link to networkUpdateStatic
  • lxd/images: Fix crash on refresh error
  • lxd/instance: Add new type instance
  • lxd/internal: Remove OnNetworkUp hook command
  • lxd/iptables: Moves iptables helper functions into own package
  • lxd/maas: Do more configuration validation
  • lxd/main_checkfeature: Add ifdef SECCOMP_GET_ACTION_AVAIL
  • lxd/main_forkmount: Error on invalid calls
  • lxd/main_forkmount: Fix cobra parsing
  • lxd/main_forkmount: Properly exit on success
  • lxd/main_forkproxy: Fix crash when listener cannot be setup
  • lxd/main_forkproxy: Rework log messages to better define the different types
  • lxd/main_forkproxy: Update references to shared types in device package
  • lxd/main_forksyscall: Add and use setnsat()
  • lxd/main_forksyscall: Add chdirchroot()
  • lxd/main_forksyscall: Avoid calling close on garbage fd
  • lxd/main_forksyscall: Don’t break chdirchroot() with setns(CLONE_NEWNS)
  • lxd/main_forksyscall: Fix variable declarations
  • lxd/main_forksyscall: Harden open()-calls via O_PATH and O_DIRECTORY
  • lxd/main_forksyscall: Introduce acquire_basic_creds()
  • lxd/main_forksyscall: Protect CLONE_NEWCGROUP with ifdef
  • lxd/main_forksyscall: Re-introduce setns(CLONE_NEWNS) properly
  • lxd/main_forksyscall: Remove same_fsinfo() logic completely
  • lxd/main_forksyscall: Remove st_ino check from same_fsinfo()
  • lxd/main_forksyscall: Replace target_fd with cwd_fd
  • lxd/main_forksyscall: Switch chdirchroot() and setns() order
  • lxd/main_forksyscall: Use correct error handling for chdirchroot()
  • lxd/networks/config: Update references to NetworkValidAddress
  • lxd/networks/config: Update to use IP validation in device_utils
  • lxd/networks/utils: Remove functions that are moved to device implementations
  • lxd/networks/utils: Remove networkUpdateStaticContainer
  • lxd/networks/utils: Remove networkValidAddress
  • lxd/networks/utils: Remove unused IP validation functions
  • lxd/project: Adds project package and updates references to it
  • lxd/proxy: Remove unused code
  • lxd/response: Fix SmartError
  • lxd/seccomp: Abstract syscall handling
  • lxd/seccomp: Always use setfattr
  • lxd/seccomp: Bugfix, cleanup, and simplify
  • lxd/seccomp: Don’t hardcode ns type
  • lxd/seccomp: Don’t mask errors
  • lxd/seccomp: Fix broken setxattr
  • lxd/seccomp: Fix setattr of directories
  • lxd/seccomp: Fix whiteout detection
  • lxd/seccomp: Handle setxattr syscall
  • lxd/seccomp: Only define Go arch (and include elf)
  • lxd/seccomp: Remove shiftfs special-casing
  • lxd/seccomp: Rename getSeccompProfileContent to seccompGetPolicyContent
  • lxd/seccomp: Retrieve fs{g,u}id for mknod{at}() syscalls
  • lxd/seccomp: Retrieve fs{g,u}id for setxattr() syscalls
  • lxd/seccomp: Use int64 for uid/gid
  • lxd/seccomp: Use LXD uidmap functions
  • lxd/storage: Fix hangs on volume migration failures
  • lxd/storage/ceph: Handle EBUSY on unmap
  • lxd/storage/ceph: Slightly speed up creation
  • lxd/storage/zfs: Fix transfer on encrypted pool
  • lxd/storage/zfs: Properly wrap mount error
  • lxd/storage/zfs: Properly wrap mount error
  • lxd/task: Attempt to run tasks on schedule
  • lxd/task/group: Move wait group Done() after g.running update to avoid race on task end
  • Makefile: Include libraft and libco
  • Makefile: Make it easier to build from tarball
  • Makefile: Rename dist to _dist to avoid Go recursion
  • Makefile: Update Github URLs
  • shared: Don’t open files to get their mode
  • shared/container: Add IsNotEmpty to help with validation required fields
  • shared/container: Add IsUnixUserID and IsOctalFileMode functions
  • shared/osarch: Add more aliases
  • shared/util: Remove BlockFsDetect as moved into device package
  • tests: Add basic infiniband tests
  • tests: Add nic bridged filtering tests for when DHCP is disabled
  • tests: Add sleep for DHCP release tests for slower machines
  • tests: Always pass -f to stop
  • tests: Always use pg_num=1 during tests
  • tests: Avoid ceph pool conflict
  • tests: Avoid event forwarding race condition
  • tests: Ensure SR-IOV tests remove all containers
  • tests: Fix bridge tests detection of busybox udhcpc6 presence
  • tests: Fix CEPH pool names
  • tests: Fix proxy device unix tests on Ubuntu Eoan
  • tests: Make shellcheck happy
  • tests: Rename ct_name to ctName for consistent naming in NIC tests
  • tests: Rename the proxy device tests to fit with other device tests
  • tests: Update forkproxy tests
  • tests: Update nic bridged filtering tests for non-IP addressed parent
  • tests: Update NIC SR-IOV test to check for device reservation
  • tests: Update NIC tests to check for volatile key cleanup
  • tests: Update static_analysis.sh
  • tests: Workaround race condition in image import event listener

Try it for yourself

This new LXD release is already available for you to try on our demo service.

Downloads

The release tarballs can be found on our download page.

Binary builds are also available for:

  • Linux: snap install lxd
  • MacOS: brew install lxc
  • Windows: choco install lxc
2 Likes

Thanks again!

Is LXD 3 fully compatible / tested with Ubuntu 18.04.3 LTS - 5.0 kernel? Tested - Works fine.

I will try shiftfs later with the NVMe SSDs.

Yeah, the vast majority of the production deployments I handle are on 18.04 with the hwe 5.0 kernel. Our own CI also uses that combination and we have automated testing for most other distributions/kernel combinations including daily testing of development upstream kernels.

1 Like

I’ve been having issue with this latest update I think. my alpine Linux container can’t seem to establish a TLS handshake at all. All traffic from SSL is failing. Actually inside any container all SSL network is failing. I just tried ubuntu container, alpine container. It just can’t seem to establish any network on port 443

Confirmed this is a bug in LXD 3.16 (although it appears to have existed in some form before that but only when devices were hotplugged into a container, rather than started on boot).

Fix is here https://github.com/lxc/lxd/pull/6078

Awesome Sauce. Extremely Happy with LXD so far. I’ve been spreading the LXD love to all my clients. They seem to be very happy with it.

1 Like

There is a breaking change in lxc list --format=csv. This now displays a header line. so when the command

lxc list comtainer_name -cs --format=csv

used to return just

RUNNING

it now returns

STATE
RUNNING

and this is breaking a number of my scripts

I do not see an option to suppress the header.

Oh, oops, this certainly wasn’t an intentional change. I’ll be sending a fix to get rid of the headers in all CSV output. If we find that some people do want the headers, we can add a separate output format for it.

Fix has now rolled out, run snap refresh lxd and you should be fixed.

wow, thank you!