Issue with deleted containers and ZFS pool size

LXD
1

Hi all,

today I run into 2 issues with my ZFS pool.

First I found that my ZFS pool was running out of space:

$ lxc storage list
+------+-------------+--------+--------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
+------+-------------+--------+--------+---------+
| lxd  |             | zfs    | lxd    | 13      |
+------+-------------+--------+--------+---------+

$ zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd   19.9G  18.6G  1.30G         -    77%    93%  1.00x  ONLINE  -

Digging a little bit more, I found that a container moved to another host is still eating space under the “deleted” path:

# zfs list
NAME                                                                          USED  AVAIL  REFER  MOUNTPOINT
lxd                                                                          18.6G   698M    24K  none
lxd/containers                                                               14.2G   698M    24K  none
[...]
lxd/deleted                                                                  3.08G   698M    24K  none
lxd/deleted/containers                                                       3.08G   698M    24K  none
lxd/deleted/containers/c8ad47df-a92d-4b4c-b0fb-09fbb718da27                  3.08G   664M  2.32G  none
[...]

I’m not sure, but I suppose that something happened due to lack of space, and LXD was unable to delete the moved container.
Is it possibile?
Is there a chance to reclaim the space?

Then, I decided to grow the ZFS pool, so I first resized the underlying disk (on AWS) from 20G to 40G.
Note that I had already run these same commands previously on the same host without any issue, but this time it din’t work as expected.
Here are the commands I executed, the drive is: /dev/nvme0n1p1

$ zpool status -v
  pool: lxd
 state: ONLINE
  scan: scrub repaired 0B in 0h3m with 0 errors on Sun May 12 00:27:19 2019
config:

	NAME        STATE     READ WRITE CKSUM
	lxd         ONLINE       0     0     0
	  nvme0n1   ONLINE       0     0     0

errors: No known data errors

$ parted -l
Warning: Not all of the space available to /dev/nvme0n1 appears to be used, you
can fix the GPT to use all of the space (an extra 41943040 blocks) or continue
with the current setting? 
Fix/Ignore? Fix                           

$ growpart  /dev/nvme0n1 1
CHANGED: partition=1 start=2048 old: size=41940959 end=41943007 new: size=83883999,end=83886047

$ fdisk -l /dev/nvme0n1
Disk /dev/nvme0n1: 40 GiB, 42949672960 bytes, 83886080 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 847423D4-75AD-0444-9E63-8A4319EF6750

Device         Start      End  Sectors Size Type
/dev/nvme0n1p1  2048 83886046 83883999  40G Solaris /usr & Apple ZFS

$ zpool list
NAME   SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
lxd   19.9G  19.2G   753M         -    81%    96%  1.00x  ONLINE  -

$ zpool online -e lxd nvme0n1
cannot expand nvme0n1: cannot relabel '/dev/nvme0n1': unable to read disk capacity

I tried do execute partprobe again, but it didn’t work.
Any idea?

Thank you.
Alessandro

2

can’t say for sure but I remember dimly that something like that happened to me with btrfs, there was a remaining pseudo-container that did not exist for LXD, I got rid of it with some btrfs command and all was well in the world again. There is no reason you could not do the same with ZFS.

3

Kinda hard to tell what’s going on, but my guess is that some partition is in used and therefore prevents the Linux kernel from re-reading the partition table.

You may be able to confirm that with hdparm -z /dev/nvme0n1, if that fails, then you would need to unmount whatever is using the drive, or alternatively reboot to force the system to re-read the partition table.