Unable to retrieve container images

mxxsuse · June 28, 2024, 8:52am

Hi,

I use openSUSE LEAP and tried to setup LXD for some software development project, but then ran into networking problems and switched to Incus. But now I seem to have similar problems:
I cannot retrieve images from https://images.linuxcontainers.org:
incus image list images:
results in:

Error: Failed parsing stream: Get "https://images.linuxcontainers.org/streams/v1/index.json": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host

My setup:
Since there was no other possibility, I compiled Incus from sources. After some trial and error, I disabled IPv6 (which isn’t working anyway for me) and now have the following network configuration:

incus network ls
+----------+----------+---------+----------------+------+-------------+---------+---------+
|   NAME   |   TYPE   | MANAGED |      IPV4      | IPV6 | DESCRIPTION | USED BY |  STATE  |
+----------+----------+---------+----------------+------+-------------+---------+---------+
| eth0     | physical | false   |                |      |             | 0       |         |
+----------+----------+---------+----------------+------+-------------+---------+---------+
| incusbr1 | bridge   | true    | 10.76.124.1/24 | none |             | 1       | CREATED |
+----------+----------+---------+----------------+------+-------------+---------+---------+
| lo       | loopback | false   |                |      |             | 0       |         |
+----------+----------+---------+----------------+------+-------------+---------+---------+

Also after start of the Incus daemon incusd, I get this:

WARNING[2024-06-28T08:42:19+02:00]  - Couldn't find the CGroup memory swap accounting, swap limits will be ignored
ERROR  [2024-06-28T08:42:19+02:00] Unable to run feature checks during QEMU initialization: Unable to locate a UEFI firmware
WARNING[2024-06-28T08:42:19+02:00] Instance type not operational                 driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
WARNING[2024-06-28T08:42:20+02:00] Failed to update instance types: Get "https://images.linuxcontainers.org/meta/instance-types/aws.yaml": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host
ERROR  [2024-06-28T08:42:20+02:00] Failed updating instance types                err="Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host"

Stopping the firewall of the host, didn’t help either. There was some similar post a while ago:

, but I don’t know if this is really the same problem here ?

simos · June 28, 2024, 9:08am

Here it says that your DNS server (the one that your Incus host is configured to get, probably through a DHCP request from your router), cannot resolve the hostname images.linuxcontainers.org.

You would need to figure out what device is on 192.168.1.1, and why it does not resolve the specific hostname.

Do you run Incus in a VM?

Install bind9-host and run the following on your Incus host, and possibly from another system on your network.

host images.linuxcontainers.org
host -d images.linuxcontainers.org

mxxsuse · June 28, 2024, 11:23am

192.168.1.1 is my router, and I can get to the URL with my web-browser.

Do you run Incus in a VM?

No

Install bind9-host and run the following on your Incus host, and possibly from another system on your network.

I installed monitoring-plugins-bind9 this opensuse-package (I hope this is the correct one ?).

host images.linuxcontainers.org
images.linuxcontainers.org has address 45.45.148.8
images.linuxcontainers.org has IPv6 address 2602:fc62:a:1::8
;; communications error to 192.168.1.1#53: timed out

and

host -d images.linuxcontainers.org
Trying "images.linuxcontainers.org"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53856
;; flags: qr; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;images.linuxcontainers.org.    IN      A

;; ANSWER SECTION:
images.linuxcontainers.org. 786 IN      A       45.45.148.8

Received 60 bytes from fe80::1%2#53 in 0 ms
Trying "images.linuxcontainers.org"
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24815
;; flags: qr; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;images.linuxcontainers.org.    IN      AAAA

;; ANSWER SECTION:
images.linuxcontainers.org. 787 IN      AAAA    2602:fc62:a:1::8

Received 72 bytes from fe80::1%2#53 in 0 ms
Trying "images.linuxcontainers.org"
;; communications error to 192.168.1.1#53: timed out
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18687
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0

;; QUESTION SECTION:
;images.linuxcontainers.org.    IN      MX

;; AUTHORITY SECTION:
linuxcontainers.org.    2495    IN      SOA     samba03.stgraber.net. hostmaster.stgraber.net. 1719570260 900 600 86400 3600

Received 111 bytes from 192.168.1.1#53 in 19 ms

I did the same from another system and there was no timeout error.

mxxsuse · July 1, 2024, 8:39am

I tried host on another system on the same PC without LXD and Incus and there it worked without error.
Also I noticed that now Incus is trying to use Ipv6 even though I disabled it on my host and on the network bridge incusbr1:

incus image list images:
Error: Failed parsing stream: Get "https://images.linuxcontainers.org/streams/v1/index.json": Unable to connect to: images.linuxcontainers.org:443 ([dial tcp [2602:fc62:a:1::8]:443: connect: network is unreachable])

stgraber · July 1, 2024, 5:32pm

You’ll most likely get that error after a failed connection over IPv4. Go will always try both and fail if they both fail.

mxxsuse · July 8, 2024, 8:20am

Okay, now the following happened:
The command incus image list images: works sporadically (maybe in 10% of all tries).
When I try to launch an image with:
incus launch -s incus_mypool_diff images:debian/12/cloud test
, I get the following error:

Launching test2
Error: Failed instance creation: Failed to run: /usr/local/share/go/bin/incusd forkstart test2 /var/lib/incus/containers /run/incus/test2/lxc.conf: exit status 1

Then I can get a more specific error message by trying to start the image:

incus start test test2
test2: error: Failed to run: /usr/local/share/go/bin/incusd forkstart test2 /var/lib/incus/containers /run/incus/test2/lxc.conf: exit status 1
test2: Try `incus info --show-log test2` for more info
test: error: Failed to run: /usr/local/share/go/bin/incusd forkstart test /var/lib/incus/containers /run/incus/test/lxc.conf: exit status 1
test: Try `incus info --show-log test` for more info

Error: Some instances failed to start

Also, I was wondering if Network Manager has something to with this problem ?

stgraber · July 8, 2024, 3:03pm

What does the suggested incus info --show-log test2 show you?

mxxsuse · July 8, 2024, 3:22pm

Sorry, I have forgotten to include this:

incus info --show-log test2
Name: test2
Status: STOPPED
Type: container
Architecture: x86_64
Created: 2024/07/08 10:04 CEST
Last Used: 2024/07/08 17:19 CEST

Log:

lxc test2 20240708151902.778 ERROR    idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:245 - newuidmap failed to write mapping "newuidmap: write to uid_map failed: Invalid argument": newuidmap 13226 0 400000000 500000001 0 1000000 1000000000
lxc test2 20240708151902.779 ERROR    start - ../src/lxc/start.c:lxc_spawn:1795 - Failed to set up id mapping.
lxc test2 20240708151902.779 ERROR    lxccontainer - ../src/lxc/lxccontainer.c:wait_on_daemonized_start:838 - Received container state "ABORTING" instead of "RUNNING"
lxc test2 20240708151902.781 ERROR    start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "test2"
lxc test2 20240708151902.781 WARN     start - ../src/lxc/start.c:lxc_abort:1038 - No such process - Failed to send SIGKILL via pidfd 17 for process 13226

stgraber · July 8, 2024, 3:34pm

Can you show your /etc/subuid and /etc/subgid?
Seems like there’s something wrong in there.

mxxsuse · July 9, 2024, 9:08am

This looks weird to me also:

/etc/subgid:
otheruser:100000:65536
normaluser:165536:65536
root:400000000:500000001
root:1000000:1000000000

/etc/subuid:
otheruser:100000:65536
normaluser:165536:65536
root:400000000:500000001
root:1000000:1000000000

normaluser is my ordinary, unprivileged account, otheruser is just a different account that has nothing to do with Incus.

I want to use the Incus for compiling and testing sofware as a normal (not root) user (iow with an “unprivileged container”?).
Do I need entries for normaluser in subgid/subuid ?

stgraber · July 9, 2024, 7:05pm

Incus only cares about the root user.

The two maps look a bit confusing, unless you have a good reason for having both of those for root, I’d simplify things and just put root:1000000:1000000000 in both subuid and subgid. Then run systemctl restart incus to have Incus reload the configuration.

mxxsuse · July 10, 2024, 8:11am

I’ve changed subgid/subuid as you suggested, but the problem persists.
Moreover, I haven’t configured Incus as a systemd service, so I get:

Failed to start incus.service: Unit incus.service not found.

Instead, I do:

sudo -E PATH=${PATH} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} $(go env GOPATH)/bin/incusd --group incussudo &

Is it neccessary to use a systemd service ?

stgraber · July 10, 2024, 2:25pm

Ah yeah, that’s fine. WIthout systemd you’ll need to manually restart it.
I’d recommend starting it with --verbose so you can see what it’s reading from subuid/subgid.

mxxsuse · July 10, 2024, 4:30pm

Sure. I did:

sudo -E PATH=${PATH} LD_LIBRARY_PATH=${LD_LIBRARY_PATH} $(go env GOPATH)/bin/incusd -v --group incussudo &

and this is the result:

INFO   [2024-07-10T18:14:38+02:00] Starting up                                   mode=normal path=/var/lib/incus version=6.2
INFO   [2024-07-10T18:14:38+02:00] System idmap (root user):
INFO   [2024-07-10T18:14:38+02:00]  - u 0 1000000 1000000000
INFO   [2024-07-10T18:14:38+02:00]  - g 0 1000000 1000000000
INFO   [2024-07-10T18:14:38+02:00] Selected idmap:
INFO   [2024-07-10T18:14:38+02:00]  - u 0 1000000 1000000000
INFO   [2024-07-10T18:14:38+02:00]  - g 0 1000000 1000000000
INFO   [2024-07-10T18:14:38+02:00] Kernel features:
INFO   [2024-07-10T18:14:38+02:00]  - closing multiple file descriptors efficiently: yes
INFO   [2024-07-10T18:14:38+02:00]  - netnsid-based network retrieval: yes
INFO   [2024-07-10T18:14:38+02:00]  - pidfds: yes
INFO   [2024-07-10T18:14:38+02:00]  - pidfds for threads: no
INFO   [2024-07-10T18:14:38+02:00]  - core scheduling: no
INFO   [2024-07-10T18:14:38+02:00]  - uevent injection: yes
INFO   [2024-07-10T18:14:38+02:00]  - seccomp listener: yes
INFO   [2024-07-10T18:14:38+02:00]  - seccomp listener continue syscalls: yes
INFO   [2024-07-10T18:14:38+02:00]  - seccomp listener add file descriptors: yes
INFO   [2024-07-10T18:14:38+02:00]  - attach to namespaces via pidfds: yes
INFO   [2024-07-10T18:14:38+02:00]  - safe native terminal allocation: yes
INFO   [2024-07-10T18:14:38+02:00]  - unprivileged binfmt_misc: no
INFO   [2024-07-10T18:14:38+02:00]  - unprivileged file capabilities: yes
INFO   [2024-07-10T18:14:38+02:00]  - cgroup layout: cgroup2
WARNING[2024-07-10T18:14:38+02:00]  - Couldn't find the CGroup memory swap accounting, swap limits will be ignored
INFO   [2024-07-10T18:14:38+02:00]  - idmapped mounts kernel support: yes
INFO   [2024-07-10T18:14:38+02:00] Instance type operational                     driver=lxc features="map[]" type=container
ERROR  [2024-07-10T18:14:38+02:00] Unable to run feature checks during QEMU initialization: Unable to locate a UEFI firmware
WARNING[2024-07-10T18:14:38+02:00] Instance type not operational                 driver=qemu err="QEMU failed to run feature checks" type=virtual-machine
INFO   [2024-07-10T18:14:38+02:00] Initializing local database
INFO   [2024-07-10T18:14:38+02:00] Set client certificate to server certificate  fingerprint=9809a00ab2b20addae5bfe6240c09c7b542351028bf1e018f5ba4d8e30facc74
INFO   [2024-07-10T18:14:38+02:00] Starting database node                        id=1 local=1 role=voter
INFO   [2024-07-10T18:14:39+02:00] Loading daemon configuration
INFO   [2024-07-10T18:14:39+02:00] Binding socket                                socket=/var/lib/incus/unix.socket type="REST API Unix socket"
INFO   [2024-07-10T18:14:39+02:00] Binding socket                                socket=/var/lib/incus/guestapi/sock type="devIncus socket"
INFO   [2024-07-10T18:14:39+02:00] Initializing global database
INFO   [2024-07-10T18:14:39+02:00] Connecting to global database
INFO   [2024-07-10T18:14:39+02:00] Connected to global database
INFO   [2024-07-10T18:14:39+02:00] Initialized global database
INFO   [2024-07-10T18:14:39+02:00] Firewall loaded driver                        driver=nftables
INFO   [2024-07-10T18:14:39+02:00] Initializing storage pools
INFO   [2024-07-10T18:14:40+02:00] Initialized storage pool                      pool=incus_mypool_diff
INFO   [2024-07-10T18:14:40+02:00] All storage pools initialized
INFO   [2024-07-10T18:14:40+02:00] Initializing daemon storage mounts
INFO   [2024-07-10T18:14:40+02:00] Initializing networks
INFO   [2024-07-10T18:14:40+02:00] Initialized network                           name=incusbr1 project=default
INFO   [2024-07-10T18:14:40+02:00] All networks initialized
INFO   [2024-07-10T18:14:40+02:00] Cleaning up leftover image files
INFO   [2024-07-10T18:14:40+02:00] Done cleaning up leftover image files
INFO   [2024-07-10T18:14:40+02:00] Starting device monitor
INFO   [2024-07-10T18:14:40+02:00] Initialized filesystem monitor                driver=fanotify path=/dev
INFO   [2024-07-10T18:14:40+02:00] Started seccomp handler                       path=/run/incus/seccomp.socket
INFO   [2024-07-10T18:14:40+02:00] Pruning expired images
INFO   [2024-07-10T18:14:40+02:00] Done pruning expired images
INFO   [2024-07-10T18:14:40+02:00] Pruning expired backups
INFO   [2024-07-10T18:14:40+02:00] Done pruning expired backups
INFO   [2024-07-10T18:14:40+02:00] Pruning resolved warnings
INFO   [2024-07-10T18:14:40+02:00] Daemon started
INFO   [2024-07-10T18:14:40+02:00] Updating images
INFO   [2024-07-10T18:14:40+02:00] Updating instance types
INFO   [2024-07-10T18:14:40+02:00] Done pruning resolved warnings
INFO   [2024-07-10T18:14:40+02:00] Expiring log files
INFO   [2024-07-10T18:14:40+02:00] Done expiring log files
WARNING[2024-07-10T18:14:41+02:00] Failed to update instance types: Get "https://images.linuxcontainers.org/meta/instance-types/aws.yaml": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host
ERROR  [2024-07-10T18:14:41+02:00] Failed updating instance types                err="Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host"
INFO   [2024-07-10T18:14:46+02:00] Done updating images

stgraber · July 10, 2024, 4:56pm

Okay, so that’s looking good here. What do you get now when you do say incus launch images:alpine/edge test1?

simos · July 10, 2024, 4:58pm

mxxsuse:

WARNING[2024-07-10T18:14:41+02:00] Failed to update instance types: Get "https://images.linuxcontainers.org/meta/instance-types/aws.yaml": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host
ERROR  [2024-07-10T18:14:41+02:00] Failed updating instance types                err="Get \"https://images.linuxcontainers.org/meta/instance-types/aws.yaml\": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host"

There’s a DNS issue, images.linuxcontainers.org does not resolve.

mxxsuse · July 11, 2024, 11:11am

incus launch images:alpine/edge test1

gives the following error:

Launching test1
Error: Failed instance creation: Get "https://images.linuxcontainers.org/images/alpine/edge/amd64/default/20240707_13:00/incus.tar.xz": lookup images.linuxcontainers.org on 192.168.1.1:53: no such host

mxxsuse · July 11, 2024, 11:24am

I’m wondering if NetworkManager is at the root of this. If I try to manually activate the bridge:

ip link set incusbr1 up

, it doesn’t work:

3: incusbr1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:16:3e:7d:0a:8a brd ff:ff:ff:ff:ff:ff

The state is still down.
Maybe, one has to do something different for NetworkManager, but I just tried this command:

nmcli con up incusbr1

, and this didn’t help either.

mxxsuse · August 5, 2024, 8:54am

Now, I tried to install the new Release 6.3 according to the instructions to see if I have more luck with that, but now the compiler aborts with an error:
make results in:

go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.2.2
# cd /home/berzem; git status --porcelain
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
error obtaining VCS status: exit status 128
        Use -buildvcs=false to disable VCS stamping.
error obtaining VCS status: exit status 128
        Use -buildvcs=false to disable VCS stamping.
....
make: *** [Makefile:37: build] Error 1

and there is no go/bin folder created.

mxxsuse · August 7, 2024, 8:45am

OK. I figured out how to insert -buildvcs=false into the makefile, so Incus is now built successfully…