Yes you’re going to have some trouble removing the entry now because the underlying storage volumes have been manually removed. So LXD’s database has got out of sync with reality.
Try lxd sql global "delete from instances where name = 'container-temp'"