Hi guys, wanted to ask about your experience with the minimal images.
It is incredible how little information there is about the minimal server images.
So, what is your experience with minimal and lxd?
Which packages do you usually have to add for daily operations? (if any).
On Hetzner, I use dedis, which come with the minimal Ubuntu version by default. I’ve been using them for a long time and I currently have both In production 18.04 and 20.04 on the host with just LXD installed on them.
Because LXD Snap comes with everything it’s needed to run LXC containers and I do not run anything else on the host except LXD, I’ve never missed any packages or noticed anything unusual…
With that excellent experience, I want to start using minimal on the containers too, but I am concerned about what could be missing for my use case (Serving a PHP-based web app with Apache and MariaDB).
Please do share your experience or if you deliberately do not use minimal, please share why.
Personally we started with ubuntu minimal and then slowly migrated to normal images as thats what became necessity over a period of time. The reasons were following:
We were using BTRFS and it needed additional utilities. Later on when migrated to clustered file systems (e.g. moosefs, seaweedfs, gluterfs and NFS), needed to install utilities and that needed additional packages.
Initially used haproxy containers for load balancing, but then instead of adding overhead of using LXD proxy device and getting the source IP. We decided to move haproxy to the host, that required additional packages, but gave a better performance.
Wanted to try our own networking setup with SDN and in order to use new OVN, we again needed additional packages.
Need to install monitoring on each host machine to make sure disk space, CPU and RAM are not the bottleneck for application performance. After initially using glances ((python monitoring utility) with influx and grafana moved to netdata.io (https://github.com/netdata/netdata) with influx and grafana dashboards.
So depending on your use-case minimal images might work, but slowly you might need to keep on adding additional packages to get it working and come to the realization that it’s easier to just use the normal images.
I understand your point. Truth is as soon as you start installing what you need in the containers, they’re gonna have dependencies and you’re gonna end installing some of the removed packages in the end.
Because no one from Ubuntu ever gave an explanation of what the (implied in the marketing) technical differences are, I am assuming there are none except for the removed weight.
Python seems to be a good chunk of the extra weight, if you need to add it later anyway you end with little gains."
Initially used haproxy containers for load balancing, but then instead of adding overhead of using LXD proxy device and getting the source IP. We decided to move haproxy to the host, that required additional packages, but gave a better performance.
Very interesting, I am using one HA Proxy per host as a reverse proxy, it lives in a container, so I am very interested in this. What is the level of overhead proxy devices add and how big are the gains from running HA Proxy on the host, in your experience?
…moved to netdata.io with influx and grafana dashboards.
I’m using Netdata too and I was planning to connect it to Prometheus + Grafana (cause everyone is doing that)… but something feels wrong about using Prometheus just as a database when you have specific tools like Influx for that.
I totally understand the benefits of using Prometheus + Grafana, probably the best possible mix for FOSS monitoring (or otherwise).
However, if you have to use Netdata, if you already have Netdata on the systems, in your opinion, is there any possible benefit from using Netdata + Prometheus compared to Netdata + Influx.
Now LXD proxy devices have a nat=true option so that it uses iptables/nftables transparently for you and does not spawn a fork_proxy process. The original fork_proxy process method is still supported. The reason for the two different options is that nat=true may not be supported in all setups.
Normally you would use Ubuntu Minimal LXD container images for the containers only. And Ubuntu Server for the host.
Understood, I didn’t know about the nat=true parameter, will implement it immediately.
Since I already install UFW everywhere, I’m already using IP Tables, which makes this parameter even sweeter.
Yes, I know minimal is meant for containers, vms and similar purposes. It just happens that on the Hetzner dedis, you can only install minimal versions, that’s all they offer, and that’s how I got interested in the topic, because I was forced to using it on the host (and it works just fine).
I am very happy they did it this way, cause in my opinion, it makes a lot of sense to use minimals in the host if you take into account the reduced attack surface, and that you usually run absolutely nothing on the host except LXD and some inevitable tools, right?
Is there any specific reason not to use minimal in the host?
There is a Minimal Ubuntu ISO, and there are Minimal Ubuntu cloud images. These two are different, and the first difference is that the cloud images do not obviously have a Linux kernel in them.