Image server infrastructure

Introduction

Some of you may have noticed some bumpiness with our UK and US mirrors over the past few weeks/months. This was caused by a mix of hardware failures and network issues on those mirror servers operated by Canonical Ltd.

We have now designed a new, simpler, less resource intensive, much more reliable way to handle our regional mirrors.

Unfortunately due to other priority work taking precedence, the Canonical IS team has indicated that they are unlikely to have time to change their side of the infrastructure over the coming months.

As a result, we will be phasing out the mirror servers operated by Canonical Ltd and will initially be serving the traffic directly from our main servers located in Canada and are hoping that some of our community members will lend us a hand and offer to operate some regional servers to improve latency for our users.

Infrastructure

The updated image distribution infrastructure looks like this:

  • Image builders generating our images daily (by Canonical Ltd. and @stgraber)
  • Test servers validating a subset of our images for quality (by @stgraber)
  • Image signing and publishing server (by @stgraber)
  • Cluster of 3 fully redundant web servers serving the original copy of our images (by @stgraber)

Those main web servers run in a datacenter in Canada with a variety of Tier 1 transit providers (Zayo, HE, Cogent, Arelion) and 20Gbps of peak transit capacity.

However as good and diversified a set of transit providers you may have, you can’t beat physics and to provide a great experience to our worldwide user base, we need servers closer to our users, at minimum some Europe and Asia capacity.

The main servers in Canada do GeoIP lookups on all client requests and can then dispatch users to a server closer to them when available.

While our servers have very fast connectivity, the total daily throughput of our mirrors is pretty low at somewhere between 300Mbps and 500Mbps for the total worldwide traffic. The faster connectivity is mostly useful to handle spikes in demand and to always be able to provide a download as fast as a client’s own internet connection.

Security

Before going into much more details about running regional mirrors, we need to touch on how any of this can be safe. After all, we never want one of our users to download an altered, compromised image.

To avoid such issues, the main servers that we operate will ALWAYS be the ones serving the index files which LXD then fetches over HTTPS. This means LXD can trust the index files it downloaded, after that, downloading the actual image artifacts is where regional mirrors come into play. LXD will download the artifacts from the regional mirror over HTTP (with fallback on HTTPS), validating the hash (SHA256) of all the downloaded files against the hash contained in the index file.

Any attempt at altering an image file will therefore cause a download validation failure when LXD checks the downloaded hash against the expected hash from the trusted index file.

This effectively means that while we definitely want our mirrors to be functional and well maintained, we don’t have to strictly trust their operators as the worst that can happen is a denial of service by returning broken files to LXD. Should this happen, we’d simply change the GeoIP rules in the main servers and stop sending users to the affected server.

Running a mirror

Running a mirror for internal company or personal use is effectively the same as running a public regional mirror, so you can follow the same instructions and just skip the last step :slight_smile:

To do so, we’ve come up with a pretty simply nginx configuration file you can use.
This has been tested on Ubuntu 22.04 LTS though it should work just as well on other platforms.

If you intend to become a public regional mirror, you’ll need the following:

  • At least 1Gbps of symmetric internet access
  • A DNS record pointing to your server
  • Let’s Encrypt or another valid TLS certificate for your DNS record
  • 100GB of spare disk space
  • System TCP congestion control set to bbr (net.ipv4.tcp_congestion_control=bbr)

The nginx config is as follow:

# Setup a local cache of 60GB with expiry after 3 days.
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=STATIC:10m inactive=3d max_size=100g;

server {
    # Listeners
    listen 80;
    listen 443 ssl;
    listen [::]:80 ipv6only=on;
    listen [::]:443 ssl ipv6only=on;
    server_name lxd-images.example.net;

    # Enable HTTPS with Let's Encrypt
    ssl_certificate /etc/letsencrypt/live/lxd-images.example.net/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/lxd-images.example.net/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    # Disable all client logging
    access_log off;

    # Default to bouncing everything back to the original server.
    location / {
        return 302 "$scheme://images.linuxcontainers.org$request_uri";
    }

    # For the images themselves, serve directly from the local cache.
    location /images {
        proxy_pass                      https://images.linuxcontainers.org/images;
        proxy_cache                     STATIC;
        proxy_cache_key                 $proxy_host$request_uri;
        proxy_cache_lock                on;
        proxy_cache_valid               200 3d;
        proxy_cache_valid               301 302 1m;
        proxy_cache_valid               404 5m;
        proxy_http_version              1.1;
        proxy_max_temp_file_size        2048m;
        proxy_set_header                "Connection" "";
        proxy_set_header                "X-LXC-No-GeoIP" "on";
        proxy_ssl_protocols             TLSv1.3;
        proxy_ssl_server_name           on;
        proxy_ssl_trusted_certificate   /etc/ssl/certs/ca-certificates.crt;
        proxy_ssl_verify                on;
    }
}

This configuration is for a virtual-host of lxd-images.example.net, it’s configured to operate as a caching reverse proxy for the LXD image artifacts, sending everything else back to the main servers. The cache is configured to not exceed 100GB and to cache files for up to 3 days (which is the maximum life time of a file in our case).

The benefit of this caching reverse proxy approach is that only the files that are being accessed will be downloaded to your server. Images that nobody in your region uses, will simply never be downloaded, saving a whole bunch of space.

To use your image server directly, you can use:

lxc remote add my-images https://lxd-images.example.net --protocol=simplestreams --public

And then launch an instance using it with:

lxc launch my-images:ubuntu/22.04 foo

Then if you want us to send you some traffic, send a private message to @stgraber with:

  • Who you are (who to credit)
  • Where your server is
  • What kind of connectivity you have
  • What’s the DNS record for your server

We’ll then do some tests to make sure connectivity is working as expected and will then add it to the rotation on our side. You can ask us to remove you from GeoIP at any point and should also notify us of any planned maintenance lasting more than a couple of minutes so we can send the traffic elsewhere during that time.

Regional mirrors

Here is the list of currently operating mirrors:

Server Location Speed Operator Protocol Notes
images.linuxcontainers.org Canada 20Gbps @stgraber IPv4/IPv6 Primary
lxd-images.server1.bg Bulgaria 1Gbps https://server1.bg IPv4
fra1lxdmirror01.do.letsbuildthe.cloud Germany 2Gbps @mpontillo IPv4/IPv6
sgp1lxdmirror01.do.letsbuildthe.cloud Singapore 2Gbps @mpontillo IPv4/IPv6
blr1lxdmirror01.do.letsbuildthe.cloud India 2Gbps @mpontillo IPv4/IPv6
syd1lxdmirror01.do.letsbuildthe.cloud Australia 2Gbps @mpontillo IPv4/IPv6
sfo3lxdmirror01.do.letsbuildthe.cloud United States 2Gbps @mpontillo IPv4/IPv6
za.images.linuxcontainers.org South Africa 1Gbps @stgraber IPv4/IPv6

Please note that we NEVER recommend directly connecting to a particular mirror.
Our users should always interact with https://images.linuxcontainers.org which will then redirect to a regional mirror if applicable. Mirrors can get decommissioned or change addresses at any time.

4 Likes

Just a quick update that we have now phased out the servers provided by Canonical and are currently serving everything from the primary servers in Canada.

To offer ideal coverage, adding a mirror in Europe and one in Asia would be ideal, though Asia could already be improved with something on the North American west coast.

Another quick update. We’ve now added a local mirror in Bulgaria serving those users in the region with IPv4 connectivity, the server doesn’t have IPv6, so IPv6 traffic is still served from Canada.

As part of this, we’ve removed the requirement to have IPv6 connectivity from the list above. Instead, the GeoIP logic can now be different for IPv4 and IPv6 users, making it possible to handle IPv4 or IPv6 only servers.

1 Like

i’m curious but who is we? :slight_smile:

The LXD team

Right, the bulk of what’s done as far as operating the infrastructure is done directly by me, image building is looked after by @monstermunchkin and both @tomp and @sdeziel help a bunch with testing, validation and working on some of the nginx, haproxy, … configs that we use.

1 Like

And thanks to a fleet of cloud instances setup by @mpontillo we now have pretty good coverage of the whole planet.

We’re obviously very much open to adding more mirrors, whether additional country-specific ones or just adding more capacity and redundancy to the network, so get in touch if you want to provide one!

One area we could still do with a mirror is Africa. It’s currently served by the server up in Germany, but having something on the continent would be pretty good.

3 Likes

We now have a status page that keeps track of the uptime of all the mirrors.
https://status.images.linuxcontainers.org/

4 Likes

An additional server covering Southern Africa has been added.

It’s located in Johannesburg, South Africa and serves:

  • South Africa
  • Zimbabwe
  • Zambia
  • Mozambique
  • Mauritius
  • Malawi
  • Madagascar
  • Comoros
  • Angola
  • Namibia
  • Lesotho
  • Eswatini
  • Botswana

The rest of Africa is served from Frankfurt, Germany as it’s expected to generally provide better latency and throughput. If that’s inaccurate for your country, let us know and we can adjust the GeoIP rules.

4 Likes

Hi,

just wondering, is there a way to download images directly from the primary mirror images.linuxcontainers.org without being redirected?
We’re currently running a large LXD deployment for a customer, but due to security reasons, their web access is limited through a proxy. However, we don’t have access to this proxy and can’t add other mirrors ourself. Additionally, our customer prefers not to add a long list of new domains.

Thanks in advance :slight_smile:

I’ve now setup https://ca.images.linuxcontainers.org which is the same as https://images.linuxcontainers.org but will not do GeoIP redirects. So that one is stable and can therefore be used in places where stable IPs are required.

4 Likes

With the removal of some “exotic” arches, I think this can be reduced by a fair bit, what do you think would be an appropriate size?

Well, those were never downloaded so never used any actual space, so requirements haven’t really changed, though I do run the Africa mirror with just 50GiB I believe.

1 Like

I guess the right question would have been: how big is the full image set (either a single or 3 days) now that less arches are supported?

root@shell01:~# du -sch /data/images.linuxcontainers.org/
406G	/data/images.linuxcontainers.org/
406G	total
1 Like

This infrastructure is also used by ‘raw’ LXC. At least, lxc-download template does. It seems that the security you described for LXD does not apply for LXC. If that’s true, it might be worth mentioning it.

The same is true for LXC downloads as the LXC downloader only does HTTPS.

What is the recommended way to handle mirrors with a firewall (nft output policy drop and dnsmasq nftset to enable dynamically ip associated with precise fqdn)? The http(s), notably 302 redirection, is not seen by dnsmasq and therefore connection to the redirection location is stopped.