I don't think the image mirrors are doing very well

Distrobuilder
1

So I have tried downloading devuan/excalibur/amd64 and devuan/daedalus/amd64 right now and they both return 502 Bad Gateway (nginx) error.

Before that, about 8 months ago, I tried downloading gentoo amd64 and the image was totally borked, didn’t work at all, did not start/boot.

I think the distrobuilder might have some serious issues, my suspicion would be that the issues could be related to building distros without systemd, because I try to use no-systemd distros and they always fail somehow when I try to use them with incus.

2

Let me add that building the distro myself using incus-migrate worked for the gentoo.

Right now I am downloading Devuan and installing it on a VM so that I can also use incus-migrate to create an instance.

I hope that works

3

Are you still getting this behavior? We operate a global network of mirrors/proxies for the image server, so issues tend to be pretty location dependent.

That said our global monitoring hasn’t seen any issue in the past couple of days:

4

Every single image we produce gets tested to ensure it boots, gets IPv4 and IPv6 connectivity and functional DNS. An image cannot appear on the image server unless those tests have passed.

That doesn’t always translate on a guarantee that it will work on your system however as local firewalls, kernel and OS configuration can affect the behavior of containers, but in general, anything you find on our image servers, whether container or VM images will have been tested with the latest stable Incus release on a Debian 13 system.

5

https://fra1lxdmirror01.do.letsbuildthe.cloud is responding with 502 Bad Gateway since yesterday for AlmaLinux, Alpine, Fedora, Amazon Linux and FreeBSD images.

2 Likes
6

Ah, that’s fun…

@mpontillo can you take a look?

1 Like
7

Hi friends; not sure what happened, but I saw a lot of TLS errors in the logs. I ran certbot renew --force-renewal on the Frankfurt mirror and that seems to have resolved it! Apologies for the glitch.

1 Like
8

Positive, tested 5 hours ago.

Tested from 2 different IPs, both via Incus and also tried downloading manually from a browser. Returns 502 Bad Gateway

Let me re-check right now…

9

Following URL gives 502 bad gateway right now:

https://fra1lxdmirror01.do.letsbuildthe.cloud/images/devuan/excalibur/amd64/default/20260602_11:50/rootfs.squashfs

I don’t think the “mirror testing scripts” you mentioned before are doing what they are supposed to be doing.

10

Thanks for the link; I think there is still a problem I’ll need to resolve with the upstream mirror configuration; but I worked around it and can fetch this URL now. How does it look on your side? I haven’t seen errors in the log in awhile.

1 Like
11 
stgraber@orilla-d13:~$ wget https://fra1lxdmirror01.do.letsbuildthe.cloud/images/devuan/excalibur/amd64/default/20260602_11:50/rootfs.squashfs
--2026-06-03 18:34:06--  https://fra1lxdmirror01.do.letsbuildthe.cloud/images/devuan/excalibur/amd64/default/20260602_11:50/rootfs.squashfs
Resolving fra1lxdmirror01.do.letsbuildthe.cloud (fra1lxdmirror01.do.letsbuildthe.cloud)... 2a03:b0c0:3:d0::3b:1, 164.90.187.92
Connecting to fra1lxdmirror01.do.letsbuildthe.cloud (fra1lxdmirror01.do.letsbuildthe.cloud)|2a03:b0c0:3:d0::3b:1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93851648 (90M) [application/octet-stream]
Saving to: ‘rootfs.squashfs’

rootfs.squashfs                           100%[=====================================================================================>]  89.50M  26.3MB/s    in 3.4s    

2026-06-03 18:34:10 (26.3 MB/s) - ‘rootfs.squashfs’ saved [93851648/93851648]

stgraber@orilla-d13:~$ wget https://fra1lxdmirror01.do.letsbuildthe.cloud/images/devuan/excalibur/amd64/default/20260602_11:50/rootfs.squashfs -4
--2026-06-03 18:34:12--  https://fra1lxdmirror01.do.letsbuildthe.cloud/images/devuan/excalibur/amd64/default/20260602_11:50/rootfs.squashfs
Resolving fra1lxdmirror01.do.letsbuildthe.cloud (fra1lxdmirror01.do.letsbuildthe.cloud)... 164.90.187.92
Connecting to fra1lxdmirror01.do.letsbuildthe.cloud (fra1lxdmirror01.do.letsbuildthe.cloud)|164.90.187.92|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 93851648 (90M) [application/octet-stream]
Saving to: ‘rootfs.squashfs.1’

rootfs.squashfs.1                         100%[=====================================================================================>]  89.50M  20.8MB/s    in 4.8s    

2026-06-03 18:34:18 (18.5 MB/s) - ‘rootfs.squashfs.1’ saved [93851648/93851648]

stgraber@orilla-d13:~$
1 Like
12

I’m testing some new droplets as replacements; it would be great if the community could help try these out and let me know if they look good:

13

Works right now, I will try installing Devuan from the image, thanks

14

BTW Is there way to choose another mirror manually in case the one automatically assigned is faulty?

15

You can add or modify a remote to test; I think something like this would work (for example, if you wanted to hit the fra1 mirror specifically):

incus remote add fra1 https://fra1mirror01.do.images.linuxcontainers.org \
    --protocol=simplestreams --public
incus launch fra1:ubuntu/26.04 testing
1 Like
16

I confirm the default image mirror now works.

Thanks for help guys.