Question
I just started using LXC and I’m trying to figure out how I can get near to the baremetal performance in my containers. Is there anything that needs to be configured or installed?
Currently Podman is winning over LXC and I’d like to know how I can fix that
Benchmark setup
I’m running everything as root to avoid any kinds of bottlenecks in this test since Podman is known to cause performance penalties in userspace like using the slirp4netns network driver, 1024 FD limit and VFS file system (more details here)
$ sudo -i
Podman installation: (with crun for best performance)
$ pacman -S netavark crun podman
$ echo 'unqualified-search-registries = ["docker.io"]' | tee -a /etc/containers/registries.conf
$ podman run \
--rm \
--name=benchmark \
--privileged \
-it \
archlinux:latest
LXC installation:
$ pacman --noconfirm -S lxc
# Network Bridge
echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/99-ipv.conf && sysctl --system
nmcli connection add type bridge ifname br0 stp no
nmcli connection add type bridge-slave ifname enp5s0 master br0
nmcli connection down "Wired connection 1"
nmcli connection up bridge-br0
nmcli connection up bridge-slave-enp5s0
# /etc/lxc/default.conf
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
# /etc/subuid
root:100000:65536
# /etc/subgid
root:100000:65536
$ lxc-create -n playtime -t download -- --dist archlinux --release current --arch amd64
$ lxc-start -n playtime
$ lxc-attach -n playtime
Benchmark commands:
$ pacman --noconfirm -Syu time
$ echo '#!/bin/bash
for i in $(seq 1 1000);
do bash -c ":" ;
done' > bash_benchmark_script.sh
$ /usr/bin/time bash bash_benchmark_script.sh
Benchmark result
I’ve re-run them 3 times and the results are very consistent (lower result is better):
# Arch Linux Baremetal (bash)
0.48user 0.15system 0:00.62elapsed 102%CPU (0avgtext+0avgdata 3584maxresident)k
0.47user 0.14system 0:00.60elapsed 102%CPU (0avgtext+0avgdata 3584maxresident)k
0.47user 0.12system 0:00.59elapsed 102%CPU (0avgtext+0avgdata 3840maxresident)k
0.46user 0.13system 0:00.58elapsed 102%CPU (0avgtext+0avgdata 3584maxresident)k
# Arch Linux Rootful Podman (bash)
0.49user 0.16system 0:00.63elapsed 102%CPU (0avgtext+0avgdata 3584maxresident)k
0.48user 0.13system 0:00.60elapsed 101%CPU (0avgtext+0avgdata 3584maxresident)k
0.48user 0.14system 0:00.61elapsed 101%CPU (0avgtext+0avgdata 3584maxresident)k
0.48user 0.16system 0:00.62elapsed 102%CPU (0avgtext+0avgdata 3584maxresident)k
# Arch Linux LXC (bash)
1.15user 0.25system 0:01.40elapsed 101%CPU (0avgtext+0avgdata 3840maxresident)k
1.13user 0.28system 0:01.40elapsed 101%CPU (0avgtext+0avgdata 3840maxresident)k
1.12user 0.29system 0:01.39elapsed 101%CPU (0avgtext+0avgdata 4096maxresident)k
1.13user 0.27system 0:01.39elapsed 101%CPU (0avgtext+0avgdata 4096maxresident)k
Troubleshooting
Host kernel (Arch Linux installation with nothing in it apart from defaults and LXC):
# uname -r
6.4.7-arch1-1
LXC debug
$ lxc-checkconfig
LXC version 5.0.3
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Warning: newuidmap is not setuid-root
Warning: newgidmap is not setuid-root
Network namespace: enabled
--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled
Cgroup v1 mount points:
Cgroup v2 mount points:
- /sys/fs/cgroup
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled
--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, not loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, loaded
--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: enabled
/var/lib/lxc/playtime/config is default:
Click to view
# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: --dist archlinux --release current --arch amd64
# Template script checksum (SHA-1): 78b012f582aaa2d12f0c70cc47e910e9ad9be619
# For additional config options, please look at lxc.container.conf(5)
# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)
# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.include = /usr/share/lxc/config/userns.conf
lxc.arch = x86_64
# Container specific configuration
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536
lxc.rootfs.path = dir:/var/lib/lxc/playtime/rootfs
lxc.uts.name = playtime
# Network configuration
lxc.net.0.type = veth
lxc.net.0.link = br0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:7e:6e:4b
/usr/share/lxc/config/common.conf is default:
Click to view
# Default configuration shared by all containers
# Setup the LXC devices in /dev/lxc/
lxc.tty.dir = lxc
# Allow for 1024 pseudo terminals
lxc.pty.max = 1024
# Setup 4 tty devices
lxc.tty.max = 4
# Drop some harmful capabilities
lxc.cap.drop = mac_admin mac_override sys_time sys_module sys_rawio
# Ensure hostname is changed on clone
lxc.hook.clone = /usr/share/lxc/hooks/clonehostname
# Default legacy cgroup configuration
#
# CGroup allowlist
lxc.cgroup.devices.deny = a
## Allow any mknod (but not reading/writing the node)
lxc.cgroup.devices.allow = c *:* m
lxc.cgroup.devices.allow = b *:* m
## Allow specific devices
### /dev/null
lxc.cgroup.devices.allow = c 1:3 rwm
### /dev/zero
lxc.cgroup.devices.allow = c 1:5 rwm
### /dev/full
lxc.cgroup.devices.allow = c 1:7 rwm
### /dev/tty
lxc.cgroup.devices.allow = c 5:0 rwm
### /dev/console
lxc.cgroup.devices.allow = c 5:1 rwm
### /dev/ptmx
lxc.cgroup.devices.allow = c 5:2 rwm
### /dev/random
lxc.cgroup.devices.allow = c 1:8 rwm
### /dev/urandom
lxc.cgroup.devices.allow = c 1:9 rwm
### /dev/pts/*
lxc.cgroup.devices.allow = c 136:* rwm
### fuse
lxc.cgroup.devices.allow = c 10:229 rwm
# Default unified cgroup configuration
#
# CGroup allowlist
lxc.cgroup2.devices.deny = a
## Allow any mknod (but not reading/writing the node)
lxc.cgroup2.devices.allow = c *:* m
lxc.cgroup2.devices.allow = b *:* m
## Allow specific devices
### /dev/null
lxc.cgroup2.devices.allow = c 1:3 rwm
### /dev/zero
lxc.cgroup2.devices.allow = c 1:5 rwm
### /dev/full
lxc.cgroup2.devices.allow = c 1:7 rwm
### /dev/tty
lxc.cgroup2.devices.allow = c 5:0 rwm
### /dev/console
lxc.cgroup2.devices.allow = c 5:1 rwm
### /dev/ptmx
lxc.cgroup2.devices.allow = c 5:2 rwm
### /dev/random
lxc.cgroup2.devices.allow = c 1:8 rwm
### /dev/urandom
lxc.cgroup2.devices.allow = c 1:9 rwm
### /dev/pts/*
lxc.cgroup2.devices.allow = c 136:* rwm
### fuse
lxc.cgroup2.devices.allow = c 10:229 rwm
# Setup the default mounts
lxc.mount.auto = cgroup:mixed proc:mixed sys:mixed
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none bind,optional 0 0
# Block some syscalls which are not safe in privileged
# containers
lxc.seccomp.profile = /usr/share/lxc/config/common.seccomp
# Lastly, include all the configs from /usr/share/lxc/config/common.conf.d/
lxc.include = /usr/share/lxc/config/common.conf.d/
/usr/share/lxc/config/userns.conf is default:
Click to view
# CAP_SYS_ADMIN in init-user-ns is required for cgroup.devices
#
# Default legacy cgroup configuration
#
lxc.cgroup.devices.deny =
lxc.cgroup.devices.allow =
# Default unified cgroup configuration
#
lxc.cgroup2.devices.deny =
lxc.cgroup2.devices.allow =
# Start with a full set of capabilities in user namespaces.
lxc.cap.drop =
lxc.cap.keep =
# We can't move bind-mounts, so don't use /dev/lxc/
lxc.tty.dir =
# Setup the default mounts
lxc.mount.auto = sys:rw
# Lastly, include all the configs from /usr/share/lxc/config/userns.conf.d/
lxc.include = /usr/share/lxc/config/userns.conf.d/
Have I missed something? Is there anything that needs to be removed or added to get the right performance?
Please let me know if I should check anything else!