LXD + BlockStorage + ZFS

LXD
1

Hi Guys,

I’m stucked since few days with my LXD setup.
Maybe blockstorage is not suitable for this usage…!:thinking:
Any idea ?

Configuration
VPS (Hetzner) + Attached BlockStorage@ZFS
Ubuntu 18.04
Snap Install

user01@dev:~$ sudo lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: pool-lxd  
Name of the storage backend to use (btrfs, ceph, cephfs, dir, lvm, zfs) [default=zfs]: 
Create a new ZFS pool? (yes/no) [default=yes]: 
Would you like to use an existing block device? (yes/no) [default=no]: yes
Path to the existing block device: /dev/sdb1 <--- Block Storage
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like LXD to NAT IPv4 traffic on your bridge? [default=yes]: yes
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: none
Would you like LXD to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: no

user01@dev:~~$ groups
user01 sudo lxd

user01@dev:~~$ ls -la /var/lib/lxd/unix.socket
srw-rw---- 1 root lxd 0 Jul 20 20:08 /var/lib/lxd/unix.socket

user01@dev:~~$  sudo lxd --version
3.15

user01@dev:~$ zpool list
NAME             SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
pool-lxd	 199G   350K   199G         -     0%     0%  1.00x  ONLINE  -

user01@dev:~$ lxc list
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first container, try: lxc launch ubuntu:18.04

Error: Get http://unix.socket/1.0: dial unix /var/lib/lxd/unix.socket: connect: connection refused

LXD log :

    user01@dev:~$ sudo cat /var/snap/lxd/common/lxd/logs/lxd.log
    t=2019-07-21T18:09:25+0200 lvl=info msg="LXD 3.15 is starting in normal mode" path=/var/snap/lxd/common/lxd
    t=2019-07-21T18:09:25+0200 lvl=info msg="Kernel uid/gid map:" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - u 0 0 4294967295" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - g 0 0 4294967295" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Configured LXD uid/gid map:" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - u 0 1000000 1000000000" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - g 0 1000000 1000000000" 
    t=2019-07-21T18:09:25+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Kernel features:" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - netnsid-based network retrieval: no" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - uevent injection: no" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - seccomp listener: no" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - unprivileged file capabilities: yes" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - shiftfs support: no" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Initializing local database" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Starting /dev/lxd handler:" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
    t=2019-07-21T18:09:25+0200 lvl=info msg="REST API daemon:" 
    t=2019-07-21T18:09:25+0200 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
    t=2019-07-21T18:09:25+0200 lvl=info msg="Initializing global database" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Initializing storage pools" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Initializing networks" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Pruning leftover image files" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Done pruning leftover image files" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Loading daemon configuration" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Pruning expired images" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Done pruning expired images" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Pruning expired container backups" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Done pruning expired container backups" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Updating images" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Expiring log files" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Updating instance types" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Done updating images" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Done expiring log files" 
    t=2019-07-21T18:09:25+0200 lvl=info msg="Done updating instance types" 
    t=2019-07-21T18:12:05+0200 lvl=info msg="Creating ZFS storage pool \"pool-lxd\"" 
    t=2019-07-21T18:12:06+0200 lvl=info msg="Created ZFS storage pool \"pool-lxd\""

Thx !

2

This is the only error that I can see. Which is, connection refused.

And this error means that your non-root account, user01, is not a member of the lxd Unix group.
Run groups to view if this account is a member of lxd group.
If it is, then you may log out and log in again so that the new membership is enabled.
If it is not, then add it with sudo usermod -a -G lxd user01. And log out, log in again.

3

Hi @simos,

Thank for your help !
I did it, my user is already a member of the lxd unix group.
I’ve rebooted the VPS but it didn’t solve the issue.

4

Hi @simos,

I did some tests with another Cloud Provider and it’s works.
It seems to be an issue with hetzner’s block storage

V

5

Do not use NVMe, try Ceph with Hetzner Cloud to prevent corruption issues with ZFS.
At the moment I only use dedicated servers from hetzner for LXD hosting.

6

Hi,

Thank for this tip !
From my understanding Block storage Volume is using the same technology than Ceph to avoid data corruption.

“How does Hetzner Online store the data in volumes?
Every block of data is stored on three different physical servers (triple replication)”

But I have a strange behavior (see above), same setup is working fine with another Cloud Provider.

You’re right, I heard that a dedicated server with an SSD/NVMe disk is the best option for ZFS.
Hetzner have strong offers (EX Series) but oversized for my purpose.:sweat_smile:

7

I’ve been using this configuration for a while now, with no problem.
Hetzner VPS + attached block storage + zfs + snap LXD.
I think the only thing that I did differently was to create the zfs dataset myself:

sudo apt install zfsutils-linux
sudo zpool create z /dev/sdb
sudo zfs create z/lxd

sudo apt-get remove lxd lxd-client
sudo snap install lxd
sudo adduser ubuntu lxd # assumming you are using user "ubuntu"
# logout and login again
lxd init

Do you want to configure a new storage pool? (yes/no) [default=yes]: yes
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, ceph, cephfs, dir, lvm, zfs) [default=zfs]: zfs
Create a new ZFS pool? (yes/no) [default=yes]: no
Name of the existing ZFS pool or dataset: z/lxd
8

Hi @votsalo

Thank I’ll try this way !

9

You can also repartition the VPS, as described here: https://blog.simos.info/how-to-repartition-a-hetzner-vps-disk-for-zfs-on-its-own-partition-for-lxd/
I followed this procedure and it’s easier than I thought.
I’ll be doing some benchmarks in the next few days to find out how much faster the local ZFS partition is than the attached volume, for running a mysql database.

10

Thank you !
However blockstorage offer more flexibility (growth zfs pool) but it will be interesting to see a benchmark.
I know that someone here (@angristan) is running PostegreSQL with ZFS+Ceph (Local)