| Project | LXD |
| Status | Implemented |
| Author(s) | @tomp |
| Approver(s) | @stgraber |
| Release | LXD 4.17 |
| Internal ID | LX001 |
Abstract
Simplify the recovery of a LXD installation when the database has been removed but the storage pool(s) still exist with a new interactive lxd recover tool that will assist with accessing the storage pools and attempting to recreate the database records for those instances and custom volumes present.
The goal of lxd recover is disaster recovery. It is not intended to be a substitute for taking backups of the LXD database and its files. It will not be able recover all contents of a LXD database (such as profiles, networks and images). Things will be missing and the user will need to manually reconfigure them.
What it will need to handle are the bits which cannot be re-created/re-configured, which are: instances, instance snapshots, custom volumes and custom volumes snapshots.
Rationale
Currently the lxd import command provides the ability to recover instances that are stored on storage pools after the LXD database has been removed. However it requires the storage pools to be mounted (inside the LXD snapâs mount namespace) by the user before running the tool. This is non-trivial and requires the user to understand the intricacies of both snap packaging mount namespaces and LXDâs mount path layouts. This is a bad user experience at the best of times, and during a disaster recovery situation it is even more important for the user to be able to recover their LXD installation quickly and easily.
Specification
Design
Instances store a copy of their DB configuration (including a copy of the storage pool and volume config) in a file called backup.yaml in their storage volume. This can be used as the basis of recreating the missing DB records during a disaster recovery scenario.
However some storage pool types do not allows access to these backup.yaml files without first activating the pool itself and mounting the per-instance storage volume.
This creates something of a chicken/egg problem, as in order to access the backup.yaml file to restore the DB records we need the info that was in the DB previously.
In order to overcome this the recovery tool will need to ask the user questions about the storage pool(s) they want to recover in order to ascertain the minimum amount of info to allow the storage pool to be activated and the storage volume(s) mounted.
This design would introduce a new lxd recover command that would provide an interactive CLI (similar to lxd init) that would use both the new /internal/recover API endpoints (for pool and volume discovery and recovery) and the existing public API endpoints (to validate whether there are any existing or conflicting entities).
First it will list the existing pools in the DB (if any). Then it will ask the user to add any additional pools to recover.
For each it will ask the user the following questions:
- Name of storage pool to recover.
- Driver type of storage pool to recover (
dir,zfsetc). - Source of the storage pool.
At this point the tool will check whether the storage pool already exists in the DB and that it matches the specified driver type.
- If it does, then it will use the existing DB record config to access the storage pool and will not require asking the user additional questions.
- If the storage pool exists in the DB already but the driver types do not match then an error will be displayed and recovery cannot continue.
- If the storage pool doesnât exist in the DB then the user will be asked additional questions:
- Source of storage pool to recover (this should match what was in the
sourceproperty of the original storage pool DB record). - Depending on the driver type there will also need to be per-driver questions (to be defined). These will be asked in a loop with each question/answer step providing one option value until the user is done. E.g. âAdditional storage pool configuration property (KEY=VALUE, empty to skip)â.
- Source of storage pool to recover (this should match what was in the
Once these questions have been answered and all storage pools have been defined, the tool will make a call to the /recover/validate endpoint with a POST body containing the requested pool info. LXD will proceed to load the relevant storage pool drivers and attempt to mount the requested storage pools (without creating any new DB records).
If this succeeds then the storage pools will be inspected for their volumes and these will be compared to the current DB records and the following validation checks will be run for each instance found on the storage pool that is not already in the DB (in this way lxd recover can be run on an existing partially imported storage pool):
- Check for entities referenced in its config file that donât exist (e.g. networks, profiles, projects) - if any are found then an error will be shown and the user will be expected to fix this manually using the existing
lxccommands. The user will be asked if they want to retry the validation once they have fixed the problem (to avoid them having to enter in all the information again). - Check that a conflicting record doesnât exist on a different storage pool - if it does then an error will be shown, and it may be that the user decides to manually delete the conflicting instance or remove the volume from the storage pool that is being recovered.
Once these checks succeed, then a list of instances and volumes to be recovered from all pools will be displayed. At this point no modifications have been made to the database.
The user will be asked if they wish to proceed with the import.
If the user wishes to proceed then a request is made to the /recover/import endpoint with info about each pool, instance and custom volume DB record that will be recreated.
If the recovery fails the DB records created for that pool will be rolled back.
API changes
As this is an internal command and associated API routes there will be no public API changes.
There will be some additional internal API endpoints:
Validate
POST /internal/recover/validate
LXD iterate through all the existing and new pools, look for unknown volumes and report any missing dependencies that would prevent importing those volumes.
// RecoverValidatePost is used to initiate a recovery validation scan.
type RecoverValidatePost struct {
Pools []StoragePoolsPost // Pools to scan (minimum population is Name for existing pool and Name, Driver and Config["source"] for unknown pools).
}
// RecoverValidateVolume provides info about a missing volume that the recovery validation scan found.
type RecoverValidateVolume struct {
Name string // Name of volume.
Type string // Same as Type from StorageVolumesPost (container, custom or virtual-machine).
SnapshotCount int // Count of snapshots found for volume.
Project string // Project the volume belongs to.
}
// RecoverValidateResult returns the result of the validation scan.
type RecoverValidateResult struct {
UnknownVolumes []RecoverValidateVolume // Volumes that could be imported.
DependencyErrors []string // Errors that are preventing import from proceeding.
}
The request would only fail if a pool canât be found at all, in all other cases it would return a RecoverValidateVolume which contains all that was found and all thatâs missing (dependencies).
Import
POST /internal/recover/import
LXD will proceed with the recovery and have all unknown volumes imported into the database.
// RecoverImportPost initiates the import of missing storage pools, instances and volumes.
type RecoverImportPost struct {
Pools []StoragePoolsPost // Pools to scan (minimum population is Name for existing pool and Name, Driver and Config["source"] for unknown pools).
}
The request would fail if any dependency is still missing (shouldnât be possible) and then will process the import, failing if something fails to import for some reason. The Pools property will be used to:
- Check if the storage pool already exists in the database, and doesnât need to be created.
- If the storage pool doesnât exist, then it will be mounted from the info provided.
- If there are no instances to be imported then the config supplied will be used to create a new storage pool DB record.
- Otherwise LXD will prefer to recreate the storage pool DB record from the backup.yaml file stored with one of instances being recovered.
CLI changes
A new interactive lxd recover command will be added with an example user experience below:
This LXD server currently has the following storage pools:
- "local" (backend="zfs", source="castiana/lxd")
- "remote" (backend="ceph", source="ceph")
Would you like to recover another storage pool? (yes/no) [default=no]: **yes**
Name of the storage pool: **foo**
Name of the storage backend (btrfs, dir, lvm, zfs, ceph, cephfs): **btrfs**
Source of the storage pool (block device, volume group, dataset, path, ... as applicable): **/dev/sdb**
Additional storage pool configuration property (KEY=VALUE, empty when done): **btrfs.mount_options=noatime**
Additional storage pool configuration property (KEY=VALUE, empty to skip): ****
Would you like to recover another storage pool? (yes/no) [default=no]: **no**
The recovery process will be scanning the following storage pools:
- EXISTING: "local" (backend="zfs", source="castiana/lxd")
- EXISTING: "remote" (backend="ceph", source="ceph")
- NEW: "foo" (backend="btrfs", source="/dev/sdb")
Would you like to continue with scanning for lost volumes? (yes/no) [default=yes]: **yes**
The following unknown volumes have been found:
- Instance "bar" on pool "local" in project "blah"
- Volume "blah" on pool "remote" in project "demo"
- Instance "a1" on pool "foo" in project "default"
- Instance "a2" on pool "foo" in project "default" (includes 3 snapshots)
- Instance "a3" on pool "foo" in project "default"
- Volume "vol1" on pool "foo" in project "blah" (includes 2 snapshots)
- Volume "vol2" on pool "foo" in project "blah"
- Volume "vol3" on pool "foo" in project "blah"
You are currently missing the following:
- Network "lxdbr1" in project "default"
- Project "demo"
- Profile "bar" in project "blah"
Please create those missing entries and then hit ENTER:
You are currently missing the following:
- Profile "bar" in project "blah"
Please create those missing entries and then hit ENTER:
The following unknown volumes have been found:
- Instance "bar" on pool "local" in project "blah"
- Volume "blah" on pool "remote" in project "demo"
- Instance "a1" on pool "foo" in project "default"
- Instance "a2" on pool "foo" in project "default" (includes 3 snapshots)
- Instance "a3" on pool "foo" in project "default"
- Volume "vol1" on pool "foo" in project "blah" (includes 2 snapshots)
- Volume "vol2" on pool "foo" in project "blah"
- Volume "vol3" on pool "foo" in project "blah"
Would you like those to be recovered? (yes/no) [default=no]: **yes**
All unknown volumes have now been recovered!
Database changes
No database changes are required.
Upgrade handling
The plan is to remove the lxd import command and it will return an error instructing the user to use lxd recover. The documentation will also be updated to reference lxd recover.
Further information and considerations
Unlike instance volumes, custom volumes do not have their DB configuration written to a backup.yaml file. This means we have to be able to derive all information required to recreate their DB records using just the supplied pool configuration and the name of the custom volume on the storage pool.
An issue exists due to the way we encode the project and LXD custom volume name into the underlying storage pool volume name using the underscore as a delimiter. The issue is that, unlike instance names (which must be valid hostnames), both projects and custom volume names are currently allowed to contain underscores.
This means it is impossible to ascertain where the project name ends and the custom volume name starts.
An example of the problem reversing the custom storage volume names back into database records can be exemplified with the ZFS driver. Currently creating a project called test_test and then creating a custom storage volume on a ZFS pool inside that project results in a ZFS volume called:
zfs/custom/test_test_test_test
However without having the database record available, it is impossible to ascertain where the project name ends and the custom volume name starts. The project could equally be called test with a volume called test_test_test.
To workaround this whilst trying to support as many existing custom volumes are possible we will take the following steps:
- If the custom storage volume name only has 1 underscore in it we can know that the part before the underscore is the project name and the part after is the LXD volume name (because custom volumes always have their project prefixed).
- If there are >1 underscores in the custom storage volume name we cannot know if we are splitting it correctly, so a warning will be displayed and the recovery of the volume will be skipped.
In order to prevent the 2nd scenario in the future we will prevent the use of underscores in new project names.
