LXD in production environment


(Vitalino Borges) #1

Hi,

Just out of curiosity, anyone who uses LXD in production environment here?

[]'s


(Fridolin Somers) #2

Yes I’am.
Currently Xenial + LXD 2.21, upgrade to 3.0 soon.
One server has 60 containers.
I prepare containers on a stage server and then move to production server.
There is an automatic image creation for bakup on a dedicated server.


#3

Been running LXC1.0 in production since 2014 and currently in the process of migrating my ~250 containers on Xenial with LXD 3.0.

Running like a charm still, but managing the old LXC way is becoming a pain.


(Vitalino Borges) #4

I’m deciding migrate to LXD.

Your containers work with different network services or all 60 containers work with same application (like apache)?

The server running LXD in your environment are configured with bridge or macvlan interfaces?


(Vitalino Borges) #5

You host your ~250 containers in more than one host server? Live migration is a common task in your day-to-day?

The storage pool used are in a block device?


#6

I am hosting in one LXD host. My existing setup was LVM storage but the new build I am working on is going to be ZFS block device.


(Druggo Yang) #7

LXC in production since 2012, and migrate to LXD (dir) in 2017.


(Vitalino Borges) #8

What kind of services run on their containers?

I intend run Postgres, MySQL, NFS server (and NFS mount point), Glusterfs mount point, Apache, and Wildfly in some LXC containers (but i’m have a bit of fear).

I’m studing all documentation available in:

I’m trying to enumerate a set of best practices for use LXD in production environment!

Thanks!


(Druggo Yang) #9

running nginx, php, tomcat, mysql, hadoop, docker and almost everything :smiley:


(Fridolin Somers) #10

Hi.
We use bridge, with default configuration.

Because of that big number of containers, we had to increase some sysctl values. I will tell you which if you need.

Regards


(Fridolin Somers) #11

Here is what we set in /etc/sysctl.conf :

fs.inotify.max_queued_events = 1048576
fs.inotify.max_user_instances = 1048576
fs.inotify.max_user_watches = 1048576
vm.max_map_count = 262144
kernel.dmesg_restrict = 1

And in /etc/security/limits.conf :

*               soft    nofile          1048576
*               hard    nofile          1048576
root            soft    nofile          1048576
root            hard    nofile          1048576
*               soft    memlock         unlimited
*               hard    memlock         unlimited

(Vitalino Borges) #12

Thanks for the clarifications! I’ll prepare my setup with these adjustments.

You use one profile per container? Or all containers share the same profile?

Your disk limits are applied via profile or manual via command line?


#13

These might also be a good idea to add in /etc/sysctl.conf:

net.ipv4.neigh.default.gc_thresh3=8192
net.ipv6.neigh.default.gc_thresh3=8192

They increase the maximum entries limit in ARP tables, IPv4 and IPv6 respectively and will allow you to have a lot more containers.

On that note, I believe the standard linux bridge has a hard limit of 1024 hosts, so in case you might go over, either balance your hosts in different bridges, or, my preference, got with openvswitch.


(Ron Kelley) #14

We have been running LXD in production mode for about 2 years now. Each container host a WordPress site for our customers (nginx and PHP) - we have about 650 sites spread across 10 LXD container servers. We also run MariaDB servers for the WordPress sites in other containers.

So far, so good. We have run into minor issues along the way (more of a learning-curve for containers), but overall, the containers run very well.

Things we really like about LXD:

  • For our workloads (web site hosting) the containers offer a great way to isolate each site from another
  • Using BTRFS filesystem allows us to take snapshots of each site in less than a couple of seconds
  • Starting/stopping containers is very fast; very little downtime for our customers
  • Very easy to spin up a new container

Things we struggle with:

  • No centralized management tool to see all container servers with their running containers (LXDUI is a per-server only at this time)
  • Getting per-container stats is difficult - especially to see which sites are misbehaving
  • Until LXD 3.7 (just released), no easy way to incrementally move a site from one container server to another