Linuxcontainers.org image server appears to be broken?

LXD
1

I’ve tested this on two machines: one is 4.0.5 LTS and one is 4.13:

$ lxc launch images:ubuntu/focal/cloud testcontainer
Creating testcontainer
Error: Failed instance creation: invalid character '\x00' after top-level value

It occurs with any image on that server, but I can create a container from the ubuntu-minimal image server without issues.

I tried issuing lxc image refresh images:ubuntu/focal/cloud but it seems that is only for private remotes.

I haven’t dug into the code yet to see what might actually be failing but I suspect there’s a metadata issue somewhere on the image server?

1 Like
2

Update: this appears to be resolved? I never worked out what the cause was.

3

I am still affected by this:

$ lxc image list images:
Error: invalid character '\x00' after top-level value

I am running lxd 4.13 on archlinux fully up to date. The ‘images’ remote is the only one that is broken for me i.e.:

$ lxc image list ubuntu:

works as expected. I can ‘launch’ or ‘init’ images from the ‘ubuntu:’ remote, but not from the ‘images:’ remote.

Please let me know if there is anything I can do to help resolve the issue.

Thanks,
L.

1 Like
4

I have just tried again.

Now, this works (I get a list of images):

$ lxc image list images:

However, this still fails:

$ lxc launch images:archlinux blah
Creating blah
Error: Failed instance creation: invalid character '\x00' after top-level value

I also tried with another image than arch, just in case. Same thing:

$ lxc launch images:alpine/3.13 blah
Creating blah
Error: Failed instance creation: invalid character '\x00' after top-level value

HTH
L.

1 Like
5

Are you accessing over IPv6?

CC @stgraber

1 Like
6

Also, where abouts are you located, are you nearer US or UK?

7

I’m also getting this same error. Same version. Multiple host machines. Started some time within the last 15 hours. Located in Australia.

8

Tested working OK on a fresh LXD 4.13 install here in the UK using:

lxc launch images:ubuntu/focal c1
lxc launch images:archlinux blah

Perhaps the US is mirror playing up.

9

One of my machines is now working. Came good about 08:38 UTC. The other one located in a different location (but still in Aus) still has the error. Perhaps it will come good soon also.

Both machines seem to use images.linuxcontainers.org.

10

Yeah, so images.linuxcontainers.org redirects clients to either us.images.linuxcontainers.org or uk.images.linuxcontainers.org, and us.images.linuxcontainers.org seems to be down ATM.

It connects, but just hangs:

telnet us.images.linuxcontainers.org 80
Trying 2001:67c:1562::41...
Connected to gible.canonical.com.
Escape character is '^]'.
GET / HTTP/1.1
Host: us.images.linuxcontainers.org

...

But its ok over IPv4:

telnet 91.189.91.21 80
Trying 91.189.91.21...
Connected to 91.189.91.21.
Escape character is '^]'.
GET / HTTP/1.1
Host: us.images.linuxcontainers.org

HTTP/1.1 200 OK
Date: Wed, 14 Apr 2021 08:50:15 GMT
Server: Apache/2.4.7 (Ubuntu)
Last-Modified: Wed, 14 Apr 2021 07:27:09 GMT
ETag: "e36c-5bfe9acbb05ba"
Accept-Ranges: bytes
Content-Length: 58220
Vary: Accept-Encoding
Content-Type: text/html

<!DOCTYPE html>
<html lang="en">
...
1 Like
11

If you add an /etc/hosts entry for us.images.linuxcontainers.org pointing to 91.189.91.21 it should fix it temporarily.

12

Just checked the internal Canonical IS team and there are known IPv6 issues at the US data center currently.

13

I added the /etc/hosts entry and it has not resolved the issue.

14

Check that has taken effect using tcpdump, also can you replicate the telnet tests I did above and confirm the issue is only affecting IPv6. Thanks

sudo tcpdump -nn -i any host 91.189.91.21
16

It was using IPv4 confirmed using tcpdump. However on my second machine the issue has been resolved at about 9:09 UTC.

1 Like
17

No, I am not accessing over IPv6

18

I am in London, UK

19

Can you provide output of:

curl -v -I images.linuxcontainers.org
20 
$ curl -v -I images.linuxcontainers.org
*   Trying 170.39.196.167:80...
* Connected to images.linuxcontainers.org (170.39.196.167) port 80 (#0)
> HEAD / HTTP/1.1
> Host: images.linuxcontainers.org
> User-Agent: curl/7.76.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Moved Temporarily
HTTP/1.1 302 Moved Temporarily
< server: nginx/1.18.0 (Ubuntu)
server: nginx/1.18.0 (Ubuntu)
< date: Wed, 14 Apr 2021 09:35:34 GMT
date: Wed, 14 Apr 2021 09:35:34 GMT
< content-type: text/html
content-type: text/html
< content-length: 154
content-length: 154
< location: http://uk.images.linuxcontainers.org/
location: http://uk.images.linuxcontainers.org/
< access-control-allow-origin: *
access-control-allow-origin: *

< 
* Connection #0 to host images.linuxcontainers.org left intact
21

It is now working for me. Location: London (UK), using IPv4:

I didn’t have to apply the /etc/hosts workaround.

Many thanks for solving this.

L.