Sudo snap remove lxd seems to take a very long time

You should make sure to delete all storage pools prior to running snap remove.
Otherwise snapd will make a snapshot of all your data, including the potentially massive sparse file backing your pool.