Dynamic switch between the different remotes

LXD
, , ,
#1

Hello!

I need failsafe LXD Remotes

For example, if one of my mirrors falls, i need to automatically change the default remote

At the moment, when my server “worker1” is disconnected, the following error occurs (Error: Get “https://worker1:8443/1.0”: lookup worker1 on 10.197.158.1:53: server misbehaving)

this means it will not automatically switch to worker2

How can I do that?
I will be very grateful for any help!
Thank you!

image
image1600×899 162 KB

#2

Does LXD support this feature?

If not, then how can i solve it?

Regards.

#3

Best option is deploy haproxy in tcp proxy mode and have it point to all your servers and do health tracking.

1 Like
#4

@stgraber Will haproxy automatically switch me between servers? (lxc remote switch worker2)

Is it possible to somehow use lxd’s internal tools for this, or will there be such a possibility in the future?

Regards.

#5

Ah no, I was assuming those systems were clustered.

If you’re dealing with standalone systems, then you’d need to build some other tool that does what you want using our API, or write some wrapper scripts around the CLI.

1 Like
#6

@stgraber Thanks for the answer!

I will try to write a script that will automatically switch me to another server if there is no connection

If it works I’ll post it here

Regards.

#7

I created a script (MicroHA) :slight_smile:
The script can be put in cron to run every 30 seconds

The structure of the script is

  1. Checking local LXD installation
  2. Checking the connection to the server using Ping
  3. Check port activity with Netcat
  4. Switching to the active server
  5. Checking if LXD is active on a remote server
  6. Select Master LXD Remote

The output will have the following format

Checking local server...
LXD installed and running on this machine ✅
Checking ping on the worker1 server...
Ping to worker1 is not active ❌
Checking ping on the worker2 server...
Ping to worker2 is active ✅
Cheking port on the worker2 server...
Port 8443 on worker2 is active ✅
Switching to another server...
Switching to worker2 server ✅
Checking LXD service on worker2 server...
LXD running on worker2 server ✅
worker2 server selected as master server ✅

Contents of the script


#!/bin/bash

# Local server check
echo "Checking local server..."
if lxc info --force-local > /dev/null 2>&1; then
    echo "LXD installed and running on this machine ✅"
else
    echo "LXD is not installed or not running on this machine ❌"
fi

# Define list of servers
SERVERS=("worker1" "worker2" "worker3" "worker4")

# Loop through servers
for SERVER in "${SERVERS[@]}"; do
    echo "Checking ping on the $SERVER server..."
    if ping -c 1 "$SERVER" > /dev/null 2>&1; then
        echo "Ping to $SERVER is active ✅"
    else
        echo "Ping to $SERVER is not active ❌"
        continue
    fi

    echo "Cheking port on the $SERVER server..."
    if nc -z -w 1 "$SERVER" 8443 > /dev/null 2>&1; then
        echo "Port 8443 on $SERVER is active ✅"
    else
        echo "Port 8443 on $SERVER is not active ❌"
        continue
    fi

    echo "Switching to $SERVER server..."
    if lxc remote switch "$SERVER" > /dev/null 2>&1; then
        echo "Switched to $SERVER server ✅"
    else
        echo "Not switched to $SERVER server ❌"
        continue
    fi

    echo "Checking LXD service on $SERVER server..."
    if lxc info > /dev/null 2>&1; then
        echo "LXD running on $SERVER server ✅"
        echo "$SERVER server selected as master server ✅"
        break
    else
        echo "LXD is not running on $SERVER server ❌"
        continue
    fi
done