That sounds like a kernel bug.
Normally when the last process in a container dies, the kernel destroys all the namespaces, including the network namespace which contains the container side interface of the veth pair used to give your container connectivity.
In your case, something is keeping that network namespace active, which then keeps that veth device active in the container including its IP address.
Unfortunately you deleting the host side device just papers over the issue, you’re still leaking kernel resources in the background with little you can really do to track and fix this.
We have planned kernel work which should make it easier to identify such issues in the future.