Failed to connect to LXD server

,

Hi,
I am unable to launch new on remote host , i have 2 servers one in google cloud and one in linode.
when i try to execute some command like “lxc exec remote:testcontainer bash” , it works flawless but if i try to create a new container in gives and error.

root@staging2-api-clone:~# lxc launch proxy_image-v8 lin:test1234
Creating test1234
Error: Failed instance creation:
 - https://10.175.234.1:8443: Failed to connect to LXD server "https://10.175.234.1:8443": Get "https://10.175.234.1:8443/1.0": Unable to connect to: 10.175.234.1:8443
 - https://[fd42:88ad:c57:161d::1]:8443: Failed to connect to LXD server "https://[fd42:88ad:c57:161d::1]:8443": Get "https://[fd42:88ad:c57:161d::1]:8443/1.0": Unable to connect to: [fd42:88ad:c57:161d::1]:8443
 - https://192.168.1.4:8443: Failed to connect to LXD server "https://192.168.1.4:8443": Get "https://192.168.1.4:8443/1.0": Unable to connect to: 192.168.1.4:8443

remote host is listening on 8443 flawlessly.

root@staging2-api-clone:~# curl 45.x.x.x:8443
Client sent an HTTP request to an HTTPS server.

P.S : i have already added remote on public IPs below are the details:

server1
server_version: "4.13"
server2
server_version: "4.13"

Can you SSH into the LXD server and run:

lxc monitor --type=logging --pretty

Then try again to launch the container from the remote host and see what gets outputted, this will confirm the remote client is connecting to the LXD server.

i ran it got below logs:

DBUG[04-30|08:13:34] New event listener: 9cca4e26-87e4-4580-8c21-202ae18aaa15
DBUG[04-30|08:13:42] Handling                                 method=GET protocol=tls url=/1.0 username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.x:42188
DBUG[04-30|08:13:42] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:13:42] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:13:42] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:13:42] New event listener: 381b31ce-5d2d-4377-ad79-83a28187b69f
DBUG[04-30|08:13:42] Handling                                 username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.x:42190 method=GET protocol=tls url=/1.0/events
DBUG[04-30|08:13:42] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:13:42] Handling                                 url=/1.0/instances username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.x:42192 method=POST protocol=tls
DBUG[04-30|08:13:42] Responding to instance create
DBUG[04-30|08:13:42] New task Operation: 71fa7a4e-f8df-4215-bef7-467776abf67f
DBUG[04-30|08:13:42] Started task operation: 71fa7a4e-f8df-4215-bef7-467776abf67f
DBUG[04-30|08:13:42] Connecting to a remote public LXD over HTTPs
DBUG[04-30|08:13:42] Sending request to LXD                   url=https://192.168.1.4:8443/1.0 etag= method=GET
DBUG[04-30|08:13:42] Handling                                 ip=35.184.49.x:42194 method=GET protocol=tls url=/1.0/operations/71fa7a4e-f8df-4215-bef7-467776abf67f username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:13:42] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:14:02] Failure for task operation: 71fa7a4e-f8df-4215-bef7-467776abf67f: Failed to connect to LXD server "https://192.168.1.4:8443": Get "https://192.168.1.4:8443/1.0": Unable to connect to: 192.168.1.4:8443
DBUG[04-30|08:14:02] Handling                                 url=/1.0/instances username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.x:42222 method=POST protocol=tls
DBUG[04-30|08:14:02] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:14:02] Responding to instance create
DBUG[04-30|08:14:02] New task Operation: 599f6dc8-8974-4019-ab1a-262f35f35bcd
DBUG[04-30|08:14:02] Started task operation: 599f6dc8-8974-4019-ab1a-262f35f35bcd
DBUG[04-30|08:14:02] Connecting to a remote public LXD over HTTPs
DBUG[04-30|08:14:02] Sending request to LXD                   etag= method=GET url=https://10.175.234.1:8443/1.0
DBUG[04-30|08:14:03] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:14:03] Handling                                 url=/1.0/operations/599f6dc8-8974-4019-ab1a-262f35f35bcd username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.26:42224 method=GET protocol=tls
DBUG[04-30|08:14:23] Failure for task operation: 599f6dc8-8974-4019-ab1a-262f35f35bcd: Failed to connect to LXD server "https://10.175.234.1:8443": Get "https://10.175.234.1:8443/1.0": Unable to connect to: 10.175.234.1:8443
DBUG[04-30|08:14:23] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:14:23] Handling                                 username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.26:42254 method=POST protocol=tls url=/1.0/instances
DBUG[04-30|08:14:23] Responding to instance create
DBUG[04-30|08:14:23] New task Operation: 6788763b-42f0-4881-80ed-d8c0cbe0d7d9
DBUG[04-30|08:14:23] Started task operation: 6788763b-42f0-4881-80ed-d8c0cbe0d7d9
DBUG[04-30|08:14:23] Connecting to a remote public LXD over HTTPs
DBUG[04-30|08:14:23] Sending request to LXD                   etag= method=GET url=https://[fd42:88ad:c57:161d::1]:8443/1.0
DBUG[04-30|08:14:23] Found cert                               name=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2
DBUG[04-30|08:14:23] Handling                                 url=/1.0/operations/6788763b-42f0-4881-80ed-d8c0cbe0d7d9 username=0ed0d9b739fd983fc7ff2d1f4a825226518da43857ac55580593b9d0c907e8c2 ip=35.184.49.26:42256 method=GET protocol=tls
DBUG[04-30|08:14:43] Failure for task operation: 6788763b-42f0-4881-80ed-d8c0cbe0d7d9: Failed to connect to LXD server "https://[fd42:88ad:c57:161d::1]:8443": Get "https://[fd42:88ad:c57:161d::1]:8443/1.0": Unable to connect to: [fd42:88ad:c57:161d::1]:8443
DBUG[04-30|08:14:43] Disconnected event listener: 381b31ce-5d2d-4377-ad79-83a28187b69f
DBUG[04-30|08:14:43] Event listener finished: 381b31ce-5d2d-4377-ad79-83a28187b69f
DBUG[04-30|08:14:45] Handling                                 ip=@ method=GET protocol=unix url=/1.0 username=root
DBUG[04-30|08:14:45] Handling                                 method=GET protocol=unix url="/1.0/instances?recursion=2" username=root ip=@
DBUG[04-30|08:14:45] GetInstanceUsage started                 driver=zfs instance=test123 pool=default project=default
DBUG[04-30|08:14:45] GetInstanceUsage finished                pool=default project=default driver=zfs instance=test123

@tomp also got the same error on server A also

Looks like your LXD host is trying to connect to another remote host.

What host has the IPs

fd42:88ad:c57:161d::1
10.175.234.1

@tomp its the server A ip which is configured on lxdbr0
lxc info command results on server A listed below.

environment:
  addresses:
  - 192.168.1.4:8443
  - 10.175.234.1:8443
  - '[fd42:88ad:c57:161d::1]:8443'

So we need to start using consistent naming so I can understand your setup :slight_smile:

You’ve mentioned server1 and server2, but your remote name in lxc launch proxy_image-v8 lin:test1234 is lin (I assume Linode). But does that represent server1, server2 or serverA?

Also, what are the public IPs (not lxdbr0) of the two servers?

Finally, can you show the output of lxc remote ls on your local client machine?

Thanks

@tomp ok for your ease i am listing all details of both servers
serverA

name=staging2-api-clone
lxc info=
environment:
  addresses:
  - 192.168.1.4:8443
  - 10.175.234.1:8443
  - '[fd42:88ad:c57:161d::1]:8443'
public IP= 35.184.x.x

Server B =which is remote server in my case

name=lin (name in my lxc remote list on server A)
lxc info=
environment:
  addresses:
  - 45.79.x.x:8443
  - 192.168.161.244:8443
  - '[2600:3c00::f03c:92ff:febf:5a9e]:8443'
  - 10.185.26.1:8443
  - '[fd42:91:f32c:454f::1]:8443'

public IP =45.79.x.x

Image name i am using = proxy_image-v8
command i am running = lxc launch proxy_image-v8 lin:test1234

i hope it makes things clear for you. if still thing there is any ambiguity or less information feel free to ask.
lxc remote list is given below

root@staging2-api-clone:~# lxc remote list
+-----------------------+------------------------------------------+---------------+-------------+--------+--------+--------+
|         NAME          |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC | GLOBAL |
+-----------------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| images                | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     | NO     |
+-----------------------+------------------------------------------+---------------+-------------+--------+--------+--------+
| lin                   | https://45.79.x.x:8443                | lxd           | tls         | NO     | NO     | NO     |

I see so you are running lxc launch proxy_image-v8 lin:test1234 on staging2-api-clone which is itself an LXD server.

Can you show me the output of lxc image ls on staging2-api-clone please?

And to confirm was the lxc monitor command you run earlier run on lin?

yes you got it right staging2-api is its self an lxd server

lxc image ls

root@staging2-api-clone:~# lxc image ls
+----------------+--------------+--------+---------------------------------------+--------------+-----------+----------+-------------------------------+
|     ALIAS      | FINGERPRINT  | PUBLIC |              DESCRIPTION              | ARCHITECTURE |   TYPE    |   SIZE   |          UPLOAD DATE          |
+----------------+--------------+--------+---------------------------------------+--------------+-----------+----------+-------------------------------+
| mariadb          | 21861b5b5cdc | no     | Debian stretch amd64 (20200811_05:24) | x86_64       | CONTAINER | 744.66MB | Mar 17, 2021 at 9:27am (UTC)  |
+----------------+--------------+--------+---------------------------------------+--------------+-----------+----------+-------------------------------+
| proxy_image-v8 | 2e3574cc6022 | no     | Debian buster amd64 (20210119_05:24)  | x86_64       | CONTAINER | 237.21MB | Mar 17, 2021 at 9:45am (UTC)  |

yes i ran lxc monitor command on lin server

OK thanks, and does proxy_image-v8 only exist on staging2-api-clone and not on lin?

yes it only exists on staging2-api-clone server

Right so that’s the issue I think, you’re asking lin to launch a container using an image on staging2-api-clone but lin cannot connect to staging2-api-clone to get the image (admittedly the error could be clearer in my view).

i guess but can we confirm if launch an instance through public images like

lxc launch images:ubuntu lin:test1232

???

1 Like

@tomp but lin server can also connect to staging server see below results

root@localhost:~# telnet 35.184.49.x 8443
Trying 35.184.49.x...
Connected to 35.184.49.x.
Escape character is '^]'.
root@localhost:~# curl 35.184.49.x:8443
Client sent an HTTP request to an HTTPS server.

above command ran on lin server

Yes, try that and see if it works.

I suspect the issue is that the LXD running on staging2-api-clone doesn’t know that its public IP is 35.184.49.x, and only it sees the local IPs:

  - 192.168.1.4:8443
  - 10.175.234.1:8443
  - '[fd42:88ad:c57:161d::1]:8443'

Which it then informs the remote server lin where it is reachable.

I suspect this is because staging2-api-clone is on GCP and GCP don’t give you a proper public IP on the VM, but instead use 1:1 DNAT, whereas Linode give you a proper public IP on the VM.

If you run ip a on staging2-api-clone this will show you the IPs LXD will know about.

yes you are right that is the case with GCP it does not list the public IP on interface same with AWS how to resolve it ?? so if there is anything we can configure in LXD to add public IP then I think it will work.

@stgraber have you come across this issue before where we need the lxc client to inject an manually specified public IP as one of the remotely reachable addresses it provides in order for the image transfer to proceed?

i have been using LXD for 7-8 months and i used on private networks in Gcp. but now we have requirement to use multi cloud. I guess it will also fail if i put servers across different VPCs