[LXD] Cluster member evacuation

LXD
22

If evacuate is used, then return, admit,restore seem to be easy to work with and describe what is being done.

Another idea, is drop the evacuation term, and use maintenance so
lxc cluster maintenance start
lxc cluster maintenance end

Just throwing some ideas.

24

We also need a new column evacuated (bool) in the nodes table.

25

I think we need to add a state column and get rid of the current pending column.

26

OK. Possible states would then be pending, evacuated and operational?

27

We’d probably line things up with network and storage, so:

  • pending
  • created
  • evacuated

But as integers in the DB with consts in lxd/db/

1 Like
28

But as integers in the DB with consts in lxd/db/

Yeah, we can simply rename the pending column as that’s an integer.

29

The API will be a operation (async) and should include state updates (UpdateMetadata) to indicate what instance is being evacuated (Stopping instance XYZ in project XYZ, Migrating instance XYZ in project XYZ to XYZ, Starting instance XYZ in project XYZ).

This will then show up in the CLI when you run lxc cluster evacuate

30

Marked this as implemented. The code was merged today and I just sent an additional PR to improve documentation coverage for it. It’ll be available to users in LXD 4.17.

31

I’ve been waiting for something like a “maintenance mode” for a long time, and I’m excited to see it happening!

Can you give some technical details surrounding the requirements for this?

I’m wondering if this is some sort of backend lxc move where the container has to stop, move, and then start. Or if it’s a live migration where I need to figure out CRIU and Ceph ahead of time for each node in my cluster.

32

Currently there are 3 options (for cluster.evacuate):

  • auto (will migrate unless the instance relies on devices which can’t be moved)
  • migrate (forces a migration but may fail if devices are incompatible with target)
  • stop (just stops the instance, won’t attempt to move)

Currently all our migrate are cold migration, so the evacuation will perform a clean shutdown of the instance (up to timeout from boot.host_shutdown_timeout), then move it to a new cluster member and then start it back up. Stopped instances are also moved away.

If you’re on ceph, the downtime is much shorter as no data needs to be transferred over the network. For those instances that aren’t on ceph, then they’ll get migrated the same way lxc move would do it.

1 Like
33

FYI, Ganeti (another cluster manager) uses failover (offline move) and migrate (online/live-move).

34

In our case, the intent is to add live-migrate once we have it working with VMs and have auto default tolive-migrate for any instance matching the required config, then migrate for those that don’t and finally stop for those that can’t be migrated.

That way you should always get the best possible behavior.

2 Likes