That’s most likely a shiftfs caching bug. This can happen in some cases when the underlay is changed but the overlay has cached the file.
I’ve personally had better luck with this with more recent kernels (5.8), so you may want to install the linux-generic-hwe-20.04 kernel on your system and see if that helps.
In general, those kind of issues is what’s caused us to go a completely different route when upstreaming this feature in the Linux kernel and instead doing things at the VFS layer rather than as a standalone filesystem.
The Linux 5.12 kernel should release with this new approach and LXC and LXD will automatically use the new logic when available, hopefully making this a problem of the past over the next year or so (we need some time for things to mature and distributions to adopt suitable kernels).