Introduction
Some of you may have noticed some bumpiness with our UK and US mirrors over the past few weeks/months. This was caused by a mix of hardware failures and network issues on those mirror servers operated by Canonical Ltd.
We have now designed a new, simpler, less resource intensive, much more reliable way to handle our regional mirrors.
Unfortunately due to other priority work taking precedence, the Canonical IS team has indicated that they are unlikely to have time to change their side of the infrastructure over the coming months.
As a result, we will be phasing out the mirror servers operated by Canonical Ltd and will initially be serving the traffic directly from our main servers located in Canada and are hoping that some of our community members will lend us a hand and offer to operate some regional servers to improve latency for our users.
Infrastructure
The updated image distribution infrastructure looks like this:
- Image builders generating our images daily (by Canonical Ltd. and @stgraber)
- Test servers validating a subset of our images for quality (by @stgraber)
- Image signing and publishing server (by @stgraber)
- Cluster of 3 fully redundant web servers serving the original copy of our images (by @stgraber)
Those main web servers run in a datacenter in Canada with a variety of Tier 1 transit providers (Zayo, HE, Cogent, Arelion) and 20Gbps of peak transit capacity.
However as good and diversified a set of transit providers you may have, you canât beat physics and to provide a great experience to our worldwide user base, we need servers closer to our users, at minimum some Europe and Asia capacity.
The main servers in Canada do GeoIP lookups on all client requests and can then dispatch users to a server closer to them when available.
While our servers have very fast connectivity, the total daily throughput of our mirrors is pretty low at somewhere between 300Mbps and 500Mbps for the total worldwide traffic. The faster connectivity is mostly useful to handle spikes in demand and to always be able to provide a download as fast as a clientâs own internet connection.
Security
Before going into much more details about running regional mirrors, we need to touch on how any of this can be safe. After all, we never want one of our users to download an altered, compromised image.
To avoid such issues, the main servers that we operate will ALWAYS be the ones serving the index files which LXD then fetches over HTTPS. This means LXD can trust the index files it downloaded, after that, downloading the actual image artifacts is where regional mirrors come into play. LXD will download the artifacts from the regional mirror over HTTP (with fallback on HTTPS), validating the hash (SHA256) of all the downloaded files against the hash contained in the index file.
Any attempt at altering an image file will therefore cause a download validation failure when LXD checks the downloaded hash against the expected hash from the trusted index file.
This effectively means that while we definitely want our mirrors to be functional and well maintained, we donât have to strictly trust their operators as the worst that can happen is a denial of service by returning broken files to LXD. Should this happen, weâd simply change the GeoIP rules in the main servers and stop sending users to the affected server.
Running a mirror
Running a mirror for internal company or personal use is effectively the same as running a public regional mirror, so you can follow the same instructions and just skip the last step 
To do so, weâve come up with a pretty simply nginx configuration file you can use.
This has been tested on Ubuntu 22.04 LTS though it should work just as well on other platforms.
If you intend to become a public regional mirror, youâll need the following:
- At least 1Gbps of symmetric internet access
- A DNS record pointing to your server
- Letâs Encrypt or another valid TLS certificate for your DNS record
- 100GB of spare disk space
- System TCP congestion control set to
bbr(net.ipv4.tcp_congestion_control=bbr)
The nginx config is as follow:
# Setup a local cache of 60GB with expiry after 3 days.
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=STATIC:10m inactive=3d max_size=100g;
server {
# Listeners
listen 80;
listen 443 ssl;
listen [::]:80 ipv6only=on;
listen [::]:443 ssl ipv6only=on;
server_name lxd-images.example.net;
# Enable HTTPS with Let's Encrypt
ssl_certificate /etc/letsencrypt/live/lxd-images.example.net/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/lxd-images.example.net/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
# Disable all client logging
access_log off;
# Default to bouncing everything back to the original server.
location / {
return 302 "$scheme://images.linuxcontainers.org$request_uri";
}
# For the images themselves, serve directly from the local cache.
location /images {
proxy_pass https://images.linuxcontainers.org/images;
proxy_cache STATIC;
proxy_cache_key $proxy_host$request_uri;
proxy_cache_lock on;
proxy_cache_valid 200 3d;
proxy_cache_valid 301 302 1m;
proxy_cache_valid 404 5m;
proxy_http_version 1.1;
proxy_max_temp_file_size 2048m;
proxy_set_header "Connection" "";
proxy_set_header "X-LXC-No-GeoIP" "on";
proxy_ssl_protocols TLSv1.3;
proxy_ssl_server_name on;
proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
proxy_ssl_verify on;
}
}
This configuration is for a virtual-host of lxd-images.example.net, itâs configured to operate as a caching reverse proxy for the LXD image artifacts, sending everything else back to the main servers. The cache is configured to not exceed 100GB and to cache files for up to 3 days (which is the maximum life time of a file in our case).
The benefit of this caching reverse proxy approach is that only the files that are being accessed will be downloaded to your server. Images that nobody in your region uses, will simply never be downloaded, saving a whole bunch of space.
To use your image server directly, you can use:
lxc remote add my-images https://lxd-images.example.net --protocol=simplestreams --public
And then launch an instance using it with:
lxc launch my-images:ubuntu/22.04 foo
Then if you want us to send you some traffic, send a private message to @stgraber with:
- Who you are (who to credit)
- Where your server is
- What kind of connectivity you have
- Whatâs the DNS record for your server
Weâll then do some tests to make sure connectivity is working as expected and will then add it to the rotation on our side. You can ask us to remove you from GeoIP at any point and should also notify us of any planned maintenance lasting more than a couple of minutes so we can send the traffic elsewhere during that time.
Regional mirrors
Here is the list of currently operating mirrors:
| Server | Location | Speed | Operator | Protocol | Notes |
|---|---|---|---|---|---|
images.linuxcontainers.org |
Canada | 20Gbps | @stgraber | IPv4/IPv6 | Primary |
lxd-images.server1.bg |
Bulgaria | 1Gbps | https://server1.bg | IPv4 | |
fra1lxdmirror01.do.letsbuildthe.cloud |
Germany | 2Gbps | @mpontillo | IPv4/IPv6 | |
sgp1lxdmirror01.do.letsbuildthe.cloud |
Singapore | 2Gbps | @mpontillo | IPv4/IPv6 | |
blr1lxdmirror01.do.letsbuildthe.cloud |
India | 2Gbps | @mpontillo | IPv4/IPv6 | |
syd1lxdmirror01.do.letsbuildthe.cloud |
Australia | 2Gbps | @mpontillo | IPv4/IPv6 | |
sfo3lxdmirror01.do.letsbuildthe.cloud |
United States | 2Gbps | @mpontillo | IPv4/IPv6 | |
za.images.linuxcontainers.org |
South Africa | 1Gbps | @stgraber | IPv4/IPv6 |
Please note that we NEVER recommend directly connecting to a particular mirror.
Our users should always interact with https://images.linuxcontainers.org which will then redirect to a regional mirror if applicable. Mirrors can get decommissioned or change addresses at any time.