Image download speed is slow as heck

1 
$ lxc launch ubuntu:18.04 test-lp-1802345
Creating test-lp-1802345
Retrieving image: rootfs: 95% (16.86kB/s)

Where are those images hosted? Is there any way to speed up the download process?

ISP: Hinet(TW)
Internet Plan: 100Mbps/40Mbps

2

Hi!

You are launching a container from ubuntu:18.04 which means that you are using the ubuntu remote.
Here it is,

$ lxc remote list
+-----------------+---------------------------------------------------+---------------+-----------+--------+--------+
|      NAME       |                        URL                        |   PROTOCOL    | AUTH TYPE | PUBLIC | STATIC |
+-----------------+---------------------------------------------------+---------------+-----------+--------+--------+
| images          | https://images.linuxcontainers.org                | simplestreams |           | YES    | NO     |
+-----------------+---------------------------------------------------+---------------+-----------+--------+--------+
| local (default) | unix://                                           | lxd           | tls       | NO     | YES    |
+-----------------+---------------------------------------------------+---------------+-----------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases          | simplestreams |           | YES    | YES    |
+-----------------+---------------------------------------------------+---------------+-----------+--------+--------+
...

Therefore, the server is https://cloud-images.ubuntu.com/releases, and is accessible through the simplestreams protocol.
This means that if you could find a mirror of cloud-images, you could change the remote: and use the new one instead.

A proper mirror of cloud-images needs:

  1. the container images shown at https://cloud-images.ubuntu.com/releases
  2. the necessary files (signatures) for the container images, shown at https://cloud-images.ubuntu.com/releases/streams/v1/

I do not know whether there is an official list of mirrors for cloud-images.
Google has this unofficial list, https://www.howtodojo.com/2017/06/ubuntu-cloud-images-mirror-list/
From there, you need to find the appropriate mirror that has this *streams/ subdirectory.
For example, http://mirrors.nju.edu.cn/ubuntu-cloud-images/releases/ is OK.

So, how do you add this new faster remote to LXD?

$ lxc remote add fasterubuntu https://mirrors.nju.edu.cn/ubuntu-cloud-images/releases/ --protocol simplestreams

See the list of container images with

$ lxc image list fasterubuntu:

And launch a container from the new remote,

$ lxc launch fasterubuntu:18.04 test-lp-1802345

If you are happy with the new remote, you can replace the existing ubuntu: remote with your local one.

2 Likes
3

Thanks for the detailed explanation! I’m also wonder if the cloud-images archive is hosted on any CDN?

4

cloud-images currently isn’t CDN backed, this may change in the future.

I know there is a ticket open for it to at least be GEOIP-ed the same way images.linuxcontainers.org is (with a US and UK server).

2 Likes
5

Here is my list of verified (they are up-to-date) mirrors of https://cloud-images.ubuntu.com/releases

  1. US (Illinois), http://mirrors.servercentral.com/ubuntu-cloud-images/releases/
  2. US (Oregon), http://ubuntu.mirrors.tds.net/ubuntu-cloud-images/releases/
  3. China (Jiangsu), http://mirrors.nju.edu.cn/ubuntu-cloud-images/releases/
  4. Taiwan, https://ftp.yzu.edu.tw/Linux/ubuntu-cloud-images/releases/
  5. Japan (Honshu), http://ubuntutym2.u-toyama.ac.jp/cloud-images/releases/
3 Likes
6

I have contacted The Free Software Lab of the National Center for High-performance Computing in Taiwan and they are willing to mirror the Ubuntu Cloud Images archive which is now at http://free.nchc.org.tw/ubuntu-cloud-images/releases.

2 Likes
7

@simos I tried using the aforementioned free.nchc.org.tw mirror today but the lxc launch command end up with the folllowing error:

Creating ubuntu-18-04
Error: Failed container creation: Unable to fetch https://free.nchc.org.tw/ubuntu-cloud-images/releases/server/releases/bionic/release-20190131/ubuntu-18.04-server-cloudimg-amd64-lxd.tar.xz: 404 Not Found

would you mind checking out what went wrong in the mirror?

8

Hi!

The URL should be a bit different:

$ lxc remote add faster https://free.nchc.org.tw/ubuntu-cloud-images/releases/server --protocol simplestreams
$ lxc init faster:c mycontainer
Retrieving image: metadata: 100% (1.26GB/s)
Retrieving image: rootfs: 2% (810.06kB/s)   
Error: Remote operation canceled by user    
$

That is, for free.nchc.org.tw, the URL for lxc remote should instead be https://free.nchc.org.tw/ubuntu-cloud-images/releases/server

1 Like
9

Hello!
So it should be autobalanced? Any list of active mirrors somewhere? My googling-fu is not that strong, I couldn’t find actual one, just random mirror like https://mirrors.tuna.tsinghua.edu.cn/status/ (lxc-images part)
I’m particularly interested in mirror in SEA region, Singapore/Hong Kong may be a good places to start with

10

I recently had the same issue Super slow distro downloads
-Reminder -Minimal images are much much fast than standard distro for lxd/lxc/kvm’s.

lxc remote add --protocol simplestreams ubuntu-minimal https://cloud-images.ubuntu.com/minimal/releases/

lxc launch ubuntu-minimal:bionic