Article about running LXD on the new ARM64 Scaleway cloud servers

LXD
1

tl;dr:

  1. These are inexpensive cloud servers. Not baremetal, but KVM. On ARM64 (Cavium ThunderX) hardware.
  2. The Linux kernel lacks ZFS support, thus requires to compile it by hand. Takes time and effort (-1). Have script (+1).
  3. LXD complains that AppArmor is not complete, thus requires config option to continue. Not sure what is missing.
  4. Overall it works, however Scaleway need to step up their game and offer a proper Ubuntu Linux kernel.

Link: https://blog.simos.info/a-closer-look-at-the-new-arm64-scaleway-servers-and-lxd/

2

Did anyone try to just switch to the Ubuntu kernel on those?

I believe the default Ubuntu kernel works fine on Cavium hardware (or virtual hardware) and if you switch to it, then you’ll get ZFS support built-in and full AppArmor support.

We do regular LXD tests on arm64 machines using Ubuntu 16.04, so we know it’s all working fine, so long as you run the right kernel :slight_smile: (we test on bare-metal and VMs on APM CPUs)

3

You can’t switch to the default kernel because they have a custom solution in place to mount storage at boot time.

Scaleway cut many corners to provide such cheap hardware.

4

That’s unfortunate considering that most arm64 VMs I’ve played with so far had a very familiar UEFI based firmware with grub and all the other bits you’d normally expect on a Linux server.

5

There are a few ways to tackle this.

According to https://community.online.net/t/official-linux-kernel-new-modules-optimizations-hacks/226 they are OK to receive changes to the kernel configuration. Specifically, it says on that page If you are missing some kernel modules or optimizations, give us info about your needs (the best would be .config lines) At the end of that page, a user reported that xz compression was not present in the kernel (an issue with snaps) and it was resolved in a week.

The kernel .config can be found at http://mirror.scaleway.com/kernel/aarch64/4.9.23-std-1/ What needs to change in the .config in order to get proper AppArmor support? What is needed to get ZFS to be compiled in?

6

There is a guide to use kexec to run a custom kernel on Scaleway,

Ideally though, Scaleway should support Ubuntu officially.

7

Unfortunately neither ZFS nor AppArmor are something you can do with just config changes. Both are rather complex and big patchsets to apply to the kernel sources before you can turn on the needed config options.

For AppArmor that delta is slowly being upstreamed so eventually that won’t be a problem.
For ZFS that’s unlikely to ever change due to the licence difference.

8

Scaleway uses the term “bootscript” to describe a set of kernel and initrd (https://www.scaleway.com/docs/bootscript-and-how-to-use-it/). A user cannot set up a custom “bootscript” but can only use one of the provided bootscripts (also, kexec is not enabled currently on the ARM64 kernel).

The way forward would be to have a git repo (or source tarball) with the Ubuntu Linux kernel source(with ZFS and Apparmor included), and ask kindly Scaleway to produce a bootscript for that kernel. Which git repo/source tarball would you suggest to use?

9

For Ubuntu 16.04, that’d be: https://code.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial/+ref/master

10

Reported at


and

KEXEC support has been requested at https://github.com/scaleway/kernel-tools/issues/348

11

What do you think about using BTRFS instead of ZFS since BTRFS support is included in Scaleway Ubuntu 16.04 kernel?

12

Every storage backend comes with its issues, https://github.com/lxc/lxd/blob/master/doc/storage.md has an overview of what to expect.

btrfs is one of the best storage backends and should be fine so long as you don’t expect quotas to work in any kind of useful way and aren’t using a complex RAID setup which btrfs still has some issue running without the occasional crash and data loss.

2 Likes
13

The bigger issue with the Scaleway Linux kernel for ARM64 is that it does not support all the AppArmor features that LXD requires.

If you see at https://blog.simos.info/a-closer-look-at-the-new-arm64-scaleway-servers-and-lxd/
it is possible to relax the requirements for LXD so that it runs with the limited AppArmor support at Scaleway.

Indeed, for simplicity, it is OK to try out btrfs instead of ZFS. In my post I show that it is possible to get ZFS to compile on the ARM64, with quite a bit of involvement.

1 Like