Profile slow LXD container creation

Hi!

I suggest to try with 900 LXD containers. There are some Linux kernel limits that may make it more complicated to diagnose what’s going on. You can have a look at production suggestions at https://github.com/lxc/lxd/blob/master/doc/production-setup.md because when you create so many containers, you may end up hitting soft and hard limits for resources on your server.

Note that when you launch a LXD container, LXD will set it up and then start it. When LXD starts a container, the container runs on its own. This means that lxc launch ubuntu:b mycontainer1 may finish in a few seconds but in reality the moment you get back your shell prompt, the container runtime just started booting up. This means that as soon as LXD launched 100 containers and you got back your shell prompt, your host has just started booting up each one of them.

What you are experiencing as a bottleneck, is the backlog of the many containers booting up.

There is something called shiftfs. See Trying out `shiftfs` It helps a lot if you enable it, because without it, it takes quite more time for each container to setup (therefore the backlog becomes bigger).

Also, depending on the container image that you are using, you may hit the memory limit of the host. If you get too many containers running, and there is not enough memory, you may crash the host.