Benchmarking LXD: I can only get 84 IPs

Hi guys,

I was following this great article by @simos

I just rented a large dedicated server and wanted to see how many containers I could get running in there without issues.

I was able to run 256 containers and just use 1/3rd of the available RAM (256GB)
Only 4 containers returned errors and all the containers were marked as “Running” except those 4.

However, I was only able to get an IP for 84 of these containers. I tried a different network, I tried different parameters, For some reason it seems to stop at 84 IPs.

Any reason that would be happening?
Thank you.

First thing that comes to mind is that a container network with LXD default bridge use at least a pair of sockets. So if you take the idea a bit further and speculate that each networked container uses 3 resources, you get (84+1)*3 = 255. This looks suspiciously like a system limitation. So did you follow the advice in the LXD doc for big installations ? I’d post a link but since I don’t have big install myself I’m too lazy to look it up for you :slight_smile:

That’s interesting indeed:
*each networked container uses 3 resources, you get (84+1)3 = 255.

re: So did you follow the advice in the LXD doc for big installations ?
@gpatel-fr, I’ve googled, among several other things:
LXD documentation settings configuration for big large installations deployments
And I didn’t get anything like what you describe in the first 3 pages…
Can you give me any other clue?
Thank you!

1 Like

Thanl you @gpatel-fr

Yeah, you definitely need production-setup for scale benchmarking of LXD.

systemd is pretty good at using some global kernel resources, so if inotify isn’t bumped, you do hit that wall at a bit under 100 containers.

Note that you’ll then hit the limit of the number of IPs in the default LXD bridge.
You can increase that by making it a /16 or /8 subnet.

After that, the usual next limit is 1024 containers per Linux bridge, at which point you’ll need to create additional bridges.

In our own scale tests, we’ve found that using LXD projects with up to 200 containers or so per project makes the API more responsive as it doesn’t need to list all containers every time.

Also, if you’re just looking at running a lot of containers, using something lighter like Alpine can be quite beneficial.

1 Like

Yes, the network/subnet was the first thing I tried, I was so surprised to see it didn’t solve the issue, lol.
Truth is I doubt I will never go over 100-150 or so containers, so definitely under 200, but 84 seemed a bit too low for that much RAM.

Will try the production settings and report back.

Thank you @stgraber and @gpatel-fr