Blkio limits on Ceph RBD volume

Hi guys!

I’m not sure if I didn’t search enough but I can’t seem to make blkio limits work on Ceph RBD volumes.

  • I first tried to set limits in the container config using config edit however that did not appear to work.
  • I then created a new volume manually and attached it and edited the config again. This did not give any errors but it also is not limiting my blkio as far as I can tell (by using dd and inspecting the created blkio rules)

Can anybody point me in the right direction?

Config

    architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20190918)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20190918"
  image.type: squashfs
  image.version: "18.04"
  limits.cpu: 5,6
  limits.cpu.allowance: 50%
  limits.memory: "512000000"
  limits.memory.enforce: hard
  limits.memory.swap: "false"
  volatile.base_image: 9ff5784302bfd6d556ac4c4c1176a37e86d89ac4d1aced14d9388919fa58bee8
  volatile.eth0.host_name: vethba820ffd
  volatile.eth0.hwaddr: 00:16:3e:4d:bc:a0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices:
  maran-test:
    limits.read: 1MB
    limits.write: 1MB
    path: /mnt/external
    pool: default
    source: maran-test
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

Syntax looks correct, so I suspect the issue may be with tracking down the block device backing the volume. What LXD version is that?

# lxd --version
3.18

Perhaps I’m looking at the wrong cgroup to verify it? What cgroup folder name should I expect in this case?

/sys/fs/cgroup/blkio/lxc.payload/NAME, I’d expect to see read_bps_device and write_bps_device populated in there.

It would also be interesting to restart the container and then look at /var/snap/lxd/common/lxd/logs/NAME/lxc.conf as I would also expect to see some lxc.cgroup lines in there to match.

I think your initial suspicion might be correct. I build lxd from source to get some debug lines in and found a few issues.

The initial problem is here it tries to check if the device path maran-test exists. Which it obviously does not since it’s not a real path but a storage volume. If I remove the check all-together it still won’t work properly, it does create a blkio cgroup record but it’s for the wrong major:minor identifier.

I’m not comfortable enough with the project yet to say how to solve this, I guess some translation from the Ceph rbd name is required, if you know a solution I would be happy to try and provide a PR for it.

Can you file an issue at https://github.com/lxc/lxd/issues?

It should be reasonably straightforward for me to reproduce it here and take a quick look at that resolution logic.

Issue created.

Is there some way to apply the limits.write/read to the ‘root’ volume? I wonder if that would work but I couldn’t work out where to set the limits on the rootfs. (not sure the right terminology here but basically the volume that’s mounted on / inside the container)

To answer my own question in case anybody else comes upon it.

You can create a root key inside the devices config. I.e.

devices:
  root:
    limits.read: 10MB
    limits.write: 10MB
    path: /
    pool: default
    type: disk

Yup, that’s how you’d do it. Though in my testing for this issue, I’ve found it to also be broken for the root volume.

I’ll soon do a round of cherry-picks for the 3.18 snap, I expect this fix will be in it.