Container slower first boot time compared to VMs

I have noticed that when launching or booting a container for the first time, the startup is significantly longer:

$ time lxc launch ubuntu:20.04 container-test
Creating container-test
Starting container-test
real	0m4,076s
user	0m0,086s
sys	0m0,080s

$ lxc stop container-test
$ time lxc start container-test

real	0m0,417s
user	0m0,035s
sys	0m0,039s

VMs while also taking a bit longer don’t show such a large difference and actually startup faster than containers on the first boot:

$ time lxc launch ubuntu:20.04 vm-test -p vm --vm
Creating vm-test
Starting vm-test

real	0m2,264s
user	0m0,086s
sys	0m0,049s

$ lxc stop vm-test
$ time lxc start vm-test

real	0m1,143s
user	0m0,035s
sys	0m0,119s

I am guessing this has to do with the extra Remapping container filesystem step that is not present on VM instances?

This should only occur on first boot, it is required for unprivileged containers to remap the filesystem UID/GIDs to the unprivileged user.