Ansible Connection

Been trying to work on a workflow in Ansible that would connect to an LXD host and then use the lxd_connection plugin in Ansible to run remote tasks on that host. Trouble is when I create an inventory of containers using the lxd connection plugin in Ansible, it will want to run on my localhost and not on the remote host. My local host does not have the remotes setup for lxd. I want to loop through my container inventory in Ansible, but I want it to execute on the remote host to perform the lxd exec. Is this possible?