This rule is there because you pretty much never want to machines that are on the same L2 network to have their source addresses rewritten through NAT.
So in your case you have all addresses on your host redirecting to 10.0.4.104:5000 when hit on port 10450.
When connecting from inside a container, the host processes the prerouting rule, replacing the destination with the right container ip and port and sends the traffic through. The container then attempts to directly reply to the source without going through the host, which causes the packets to be dropped.
You are correct that the only real way to avoid the issue is to also SNAT that traffic, though rather than doing it for all container traffic, which as I mentioned isn't a good idea to do in the middle of a subnet, you could just do a specific rule to MASQUERADE just that one port.
iptables -t nat -A POSTROUTING -p tcp -d 10.0.4.104 --dport 5000 -j SNAT --to 10.0.4.1