The package manager can’t always untangle all of the system integration, filesystems, … which you might have defined.
One way is to remove the package and purge it, then reboot you system to make sure all kernel state is gone. Another way is to use the LXD API to stop and delete all containers, images, networks and storage pools. At which point the kernel state should effectively be gone and uninstalling/reinstalling should be fine.
I personally tend to go with the former, mostly out of laziness