Migrating a volume from an lxd 3.0 container to an lxd 5.0 container with a couple hundred million files in it

1

I have volumes currently being used in an lxd 3.0 container. They’re zfs datasets, and the issue is about uid/gid mapping. The files in the dataset all have uid/gids like 1011001, 101002, and so on. The newer version of lxd uses 1000000 for a uid/gid offset instead of 100000 like in lxd 3. Using the migration tools are not really an option because of the number of files involved; one of the datasets has somewhere between 150 and 200 million files, and the migration process pretty much always breaks while it’s (I think?) rewriting the UID/GIDs to match the new offset. I have considered a few things, including trying to tell the new containers to use a different offset, but they seem to end up trying to traverse the file system and changing them all, which experience and estimates tell us will take between a week or two, which is not acceptable for the live service that these datasets are used for.

Is there a way for me to use say shiftfs to fix this? I’ve tried enabling shiftfs, but it doesn’t seem to result in leaving the actual filesystem alone; eventually I see “remapping filesystem” and then it just grinds to a halt on the rock of trying to change so many files.

Thoughts?

Cheers,

Jack

2

Yeah, as your source container was already shifted, using shiftfs or VFS idmap shifting won’t help as it would need to first be unshifted anyway…

When you say it’s a volume, do you mean a custom storage volume managed by LXD, so external to the instance? I think your best bet probably is to have the instance itself start with its new map. Then tweak the volume config to make LXD think it’s using the correct map. Confirm that the instance starts up properly and sees the volume, albeit with broken uid/gid.

If that all looks good, then I’d do the long process of shifting manually on the host by using fuidshift. It’s the exact same logic LXD has, but running as a separate command which makes it a bit easier to do dry-run and to see what’s going on.