LXD in production environment

LXD
13

These might also be a good idea to add in /etc/sysctl.conf:

net.ipv4.neigh.default.gc_thresh3=8192
net.ipv6.neigh.default.gc_thresh3=8192

They increase the maximum entries limit in ARP tables, IPv4 and IPv6 respectively and will allow you to have a lot more containers.

On that note, I believe the standard linux bridge has a hard limit of 1024 hosts, so in case you might go over, either balance your hosts in different bridges, or, my preference, got with openvswitch.

1 Like
14

We have been running LXD in production mode for about 2 years now. Each container host a WordPress site for our customers (nginx and PHP) - we have about 650 sites spread across 10 LXD container servers. We also run MariaDB servers for the WordPress sites in other containers.

So far, so good. We have run into minor issues along the way (more of a learning-curve for containers), but overall, the containers run very well.

Things we really like about LXD:

  • For our workloads (web site hosting) the containers offer a great way to isolate each site from another
  • Using BTRFS filesystem allows us to take snapshots of each site in less than a couple of seconds
  • Starting/stopping containers is very fast; very little downtime for our customers
  • Very easy to spin up a new container

Things we struggle with:

  • No centralized management tool to see all container servers with their running containers (LXDUI is a per-server only at this time)
  • Getting per-container stats is difficult - especially to see which sites are misbehaving
  • Until LXD 3.7 (just released), no easy way to incrementally move a site from one container server to another
3 Likes
15

Do you connect WordPress to the MariaDB servers using tcp/ip, or do you somehow share/bind the MariaDB unix socket between containers?

Why did you choose BTRFS instead of ZFS?

16

All containers share same profile because its the same softwares inside.
We defined CPU and RAM limits in container configuration but you may create dedicated profiles.

1 Like
17

You mean disk size ? We are not using ZFS so we dont have limits.

1 Like
18

We run the MariaDB on a different server and connect over TCP.

We have been using BTRFS for so long, it is not even a question anymore. I tried ZFS in the past, and it required lots more RAM than BTRFS to get similar performance. Maybe things have changed over the past couple of years. For us, BTRFS requires minimal overhead to perform well, and the snapshots are almost instantaneous. We don’t use BTRFS to create any RAID devices. The underlying storage for the VM already does this.

1 Like
19

@rkelleyrtp @fridobox @druggo @datablitz7 Have you thought about use Kubernetes/Docker?

20

My use case requires persistent system containers rather than application hosting or stateless containers. Pets vs cattle!

21

Same for me

22

Thanks everyone!

Your clarifications was much important for me.

I hope that post helped much other people (~400 views).

Solved.

23

Hi,
Thanks for writing this down. Can I know how you make the containers talk out to network. Like is there any configuration you have done in containers to add domain, I am using a proxy container which then further routes the requests to each domain and this configuration seems not good to me.

24

What is the origin and meaning of this phrase?

25

I’m not sure who coined it first to be honest. It is a way to convey the shift in how we treat our servers: as pets, meaning we nurture them and help them live for a long time, or as cattle, where we have them fulfill their purpose and then kill them. Here is an interesting post on this distinction.

https://blog.octo.com/en/pet-vs-cattle-from-server-craftsman-to-software-craftsman/

26

Yes , I Do.
Look at this post : https://unit42.paloaltonetworks.com/attackers-tactics-and-techniques-in-unsecured-docker-daemons-revealed/
You can’t do firewall particularly within a docker container.
But with lxd you can.
ufw enable possible in lxd container but not docker.

1 Like
27

You can run stateful workloads in docker containers, you mount external persistent storage (persistent volumes). These can be local folders or shared storage, many plugins available.

If your using Kubernetes you can run persistent storage with something like Rook or OpenEBS which can deploy ceph on your kube worker nodes and will create the ceph osds, the rbd images and mount persistent volumes to your pods. Quite nifty really.

Firewalling with docker containers would be more akin to policy based rules and an SDN, everything with iptables is automated in Kubernetes, similar to how it is with Openstack Neutron or contrail, its designed to scale so you don’t touch the things at the individual conatiner level.

The downside is running docker on mass with kubernetes requires a lot more planning to have it running correctly. LXD is easier to setup, lower barrier for entry, so for smaller scale seems to fit the bill perfectly and its easier to get your head around it if your coming from the world of VM’s.

I don’t know why more people don’t use LXD, it still seems most people only know VM’s or Docker/kubernetes these days.

Cheers!
Jon.

1 Like
28

Hello,
I plan to use LXD for creating and selling VPS. is that the right choice?

1 Like
29

It can definitely be used for that and a few others have used it for this before.
You’ll need to be careful about security and networking, but that’s the case regardless of the platform you use.

1 Like
30

Hello Stgraber
How are you?
do you think it is the right choice?
also, did people use that succeeded and still use it?

1 Like
31

See also Long-term (3+year) LXD commitment?

32

very good