So unlike using routed or bridged NIC types, the ipvlan NIC type will get filtered in the INPUT and OUTPUT chains of your firewall I believe (rather than FORWARD like the other NIC types mentioned).
So you need to add the relevant rules to those chains as if they were local IPs.
The alternative is to use routed NIC type which behaves similarly to ipvlan except it allows communication with the host and will use the FORWARD chain of your firewall.
See How to get LXD containers get IP from the LAN with routed network