Hello! I’m a newbie to LXC and after a few weeks of google’ing and playing with containers have a few questions to fill in some knowledge gaps. I apologize in advance as they are probably very basic, I’ve been unable to find a solid answer in my searching.
Questions:
I want to assign a public and private IP to each container and give them inbound/outbound net access. Should I use a bridge or macvlan? I’ve seen that bridges look to be more common (bonus: I’d like to provide basic firewall support from the host to block inbound traffic on specific ports to containers)
My understanding it’s more secure to run unprivileged containers with security.idmap.isolated=true?
Running security.idmap.isolated=true would give each container 65536 UIDs? Would I still be able to run a decent amount of containers per host? (My understanding via https://ubuntu.com/blog/custom-user-mappings-in-lxd-containers says yes, but wanted to confirm)
Security wise: is it acceptable to run containers (with the above security settings) with untrusted code? If someone was able to break out of the container, they’d effectively be running as a nobody user with a unique UID so they wouldn’t be able to access any resources or cause damage?
Would an unprivileged container be able to run a full “LAMP” stack (i.e. DB, web server, php/node/ruby, redis, etc)?
Resource quotas/limits work for CPU/Memory/Disk usage, but disk IOs/MBs is not yet supported?
Software wise, is the kernel the only thing shared between the host and a container? If I update libs/dependancies on the host will it affect the container’s libs/dependancies for packages? (I’ll be running Ubuntu as host and different flavors of supported linux distros as guests)
This article (https://www.zdnet.com/article/canonical-claims-lxd-crushes-kvm/) states that:
“If, however, your VMs are running many different software programs, KVM would be a better choice.”
Is this still true (article is from 2015). Is it uncommon to run a “LAMP” stack (or variation of that) in a single container? I’m curious what “many different software programs” is referring to.
First of all, I think your question is a LXD question and not a LXC question, therefore I changed the post’s category to LXD.
LXC is when you run commands like lxc-create ... while LXD is when you run lxc launch .... See more at Comparing LXD vs. LXC In a nutshell, LXD is like LXC but with a useful manager/hypervisor service that helps you manage the containers.
Also, the containers in LXD are called system containers in contrast to the application containers in Docker and elsewhere. A system container is like a VM; you start it and it stays running until you stop it.
In terms of security, some users prefer to isolate the host from the containers. In the host there will be no services running, and all services will be in the unprivileged containers. In that case, macvlan suits quite well. In addition, I think it is more straightforward to setup firewall rules in individual containers rather than setting them all onto the host. If you feel more comfortable with the bridge though, go for it.
security.idmap.isolated=true makes the containers to have different ranges of ID between each other. I have not seen some practical security issue on this yet.
On Linux, the UIDs are unsigned int (/usr/include/x86_64-linux-gnu/bits/typesizes.h), which means that they go up to about 4 billion. You can fit more than 60000 containers in there.
On untrusted code, you would consider between a VM (hardware virtualization) and a system container (software virtualization). On both you have cases of escapes, VM escapes and container escapes. For container escapes, see https://brauner.github.io/2019/02/12/privileged-containers.html which describes the most recent incident, and gives you a good background on privileged/unprivileged. In addition, see https://shenaniganslabs.io/2019/05/21/LXD-LPE.html which describes that a member of the lxd group can become root on the host (members of the lxd group are considered administrators in LXD, therefore it is wrong to give lxd membership to non-admins).
An unprivileged container can run the full LAMP stack. Ideally, you can create multiple containers so that one is the db, another the www, a third the proxy (reverse proxy) and so on. Each gets automatically an internal hostname of the form db.lxd, www.lxd, etc. so that you can configure your, let’s say, WordPress to use the database server db.lxd. You would only expose the reverse proxy (port 443) to the Internet.
Regarding the resource limits, see https://lxd.readthedocs.io/en/latest/search.html?q=limits There is a background there on what resources can currently be limited. In practice, (at least for some common workloads) you would not put limits in the system containers so that they can use as much resources as needed.
Indeed, the kernel is only shared between the containers and the host. LXD is the manager or hypervisor that manages these system containers. If LXD crashed for some reason, the running containers would still keep running. If you run the snap package of LXD, the full set of software is included in the package and has no (or minimal?) dependencies from the host. Each container is created from a distro runtime, which are fully separate from the host.
I do not know what is meant to say with that sentence. It does not look like a direct quote, and I do not think that it refers to whether you can run the full LAMP stack on LXD system containers. It might refer to running software like OpenStack.