Docker - Unpriv LXC - weird php-fpm and nginx issue

I have an upriv lxc that is ment to host some docker images. I have created a docker image that utilizes Nvidia docker toolkit to passthrough a GPU. I also have a usb TPU passed through to the docker container and both the devices work. There is also a flask web server running that works no issues.

My issue comes when I try and run a container that uses nginx and php-fpm. All my HTTP requests end up timing out. The nginx logs only say that upstream times out while trying to communicate using phpfpms Unix socket. I switched php-fpm and fcgiwraps to TCP sockets and ended up with the same result, so I am assuming it is not an issue with using Unix sockets internal to the image for it to communicate with its internal services.

When I run the image on are metal host it works as expected so it is an lxc issue I am overlooking somewhere. I am using proxmox 7.1 as host.

Any insight would be great, this is the last issue I have to make my application dockerized with lxc host.