LXC Container unreachable using LXD Remote & Ansible

LXD
1

I am attempting to utilize the LXD Remote Connection in Ansible to setup and configure LXC containers on a server but I am running into an issue.

After setting up the inventory file below, creating the LXC container (which does work fine), and ensuring my computer I am running ansible on has the server added as a LXD remote I get an error when running a task against the LXC container.

Ansible Inventory:

all:
  children:
    servers:
      hosts:
        10.10.11.1:
    containers:
      vars:
        ansible_connection: lxd
        ansible_lxd_remote: kubuntu
        ansible_user: root
        ansible_become: no
      hosts:
        znas:

Ansible Output Error:

redirecting (type: connection) ansible.builtin.lxd to community.general.lxd
<znas> ESTABLISH LXD CONNECTION FOR USER: root
<znas> EXEC /bin/sh -c 'echo ~root && sleep 0'
<znas> EXEC /bin/sh -c 'echo "`pwd`" && sleep 0'
<znas> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp `"&& mkdir "` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `" && echo ansible-tmp-1658411856.6940753-5825-74320748781232="` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `" ) && sleep 0'
fatal: [znas]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp `\"&& mkdir \"` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `\" && echo ansible-tmp-1658411856.6940753-5825-74320748781232=\"` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `\" ), exited with result 1",
    "unreachable": true
}

Any help troubleshooting this would be great.

2

Hi… so znas is the name of the container in LXD ?

If Ansible is running as a different user on the controller node it will need to have it’s own LXD Remote added… in the user’s context, otherwise it runs the LXD connection as a user without the Remote you set up for it.

Check whether the Remote name matches, in the user context that it runs in.

Check whether the container name in LXD matches “znas” in lxc list

Sorry starting from the bottom based on my knowledge acquired by the Homer DOH method :wink:

3

yes, znas is the container that has successfully been created using ansible. So that part is done and easy to replicate at least.

As for the user and Remote, I’m not sure how to check that. For example the user Ansible runs as to deploy/run items on the server is “zuser” which does have sudo privilege’s set up correctly. At the same time LXD in the Ansible script is installed using the zuser account (not root).

It does seem like the error is saying the root user is failing to reach the remote container but when changing ansible_user to zuser that also fails and tells me lxd does not support remote users.

“lxc list” does work correctly when running it on the control pc, at least when running “sudo lxc list “remotename”:”

Any advice on how to check some of the items you listed would be great (types of commands to test for them). Thanks for the ideas so far.

4

On the control node so if you run:

sudo -u zuser lxc list

do you see the kubuntu remote you are referencing in ansible_lxd_remote ?

It sounds like the user that Ansible is running as (zuser) does not have the LXD Remote setup… i.e. you’ve set it up for your account or the root account but not for the zuser account.

5

Thank you for the clarification.

Running that on the control node provides this:

sudo: unknown user zuser
sudo: error initializing audit plugin sudoers_audit

Looks to me like your hunch was right and to add to that, on the control node I do not have zuser as a user (this was intentional). Is there a way to add the remote lxd server (kubuntu) with zuser if that account does not exist on the control node?

I haven’t seen any documentation on doing a remote add with a user on the remote machine. This is a first I’ve seen “lxc list” with a “-u” flag used as well.

6

So what I understand is that whatever the user that Ansible runs as on the control node, that user account has to have the LXC client configured for it on the control node… I was assuming this was your zuser on the control node.

When LXC Connection connects to your container it’ll do it via the LXD client (lxc) and get into the container as the root user by default… like when you are on the LXD host and run lxc exec znas bash.

Imagine you are Ansible on your control node and you want to open a terminal session in your LXD container, you need to:

  1. have the lxc client and LXD Connection
  2. in lxc set up a remote to your LXD host (looks like you’re wanting to call it kubuntu)
  3. open a session in your container by running lxc exec kubuntu:znas bash

LXD Connection effectively does #3 for you in the above list, putting you into the container as the root user by default, in conjunction with your vars below the containers group, which all looks good to me.

If you are running the ansible-playbook command as yourself or as root (sudo) on the control node, then it could be that this account you’re using doesn’t have lxc remote set up… e.g.:

  • if you’re running sudo ansible-playbook ... then on the control node try running sudo lxc exec kubuntu:znas bash to see if you get into the container fine. If you do then something else is amiss here.
  • if you are logged into the control node as zerogravitas and run the anisble-playbook... command as is (no sudo), then on the control node try running lxc exec kubuntu:znas bash to see if you get into the container fine. If you do then something else is amiss here.
7

Now we’re getting closer I think. Thank you for the detailed explanation. It makes perfect sense after thinking about Ansible, LXD, and the users running the different pieces of software.

Knowing that, I realized that I was logged in as say zerogravitas in the control node and was running ansible-playbook as that user (ssh setup for kubuntu in the Ansible Config is set to log in as zuser which I think is fine).

At the same time the LXC setup/commands required “sud0 lxc list” meaning it was running as privileged. I fixed that and now can run both ansible-playbook and lxc list kubuntu:znas bash without issues under zerogravitas. It is able to remote into the znas container successfully.

Having said that, attempting to run the playbook, it still provides the same exact error: unreachable with the same output as original. Any further ideas/thoughts?

8

Hi,
Can you post your /etc/ansible/ansible.cfg? And what you want to achieve? An example lxd ansible example.
Regards.

9

yeah, below is the ansible.cfg, the main playbook, and the znas task. Hopefully this can shed some light as to why it isn’t working.

ansible.cfg:

[defaults]
inventory = inventory.yml
private_key_file = ~/.ssh/ansible
remote_user = zuser

playbook (site-test.yml):

---

- hosts: all
  become: yes
  pre_tasks:

  - name: update & upgrade repository index (Ubuntu)
    tags: always
    apt:
      upgrade: 'yes'
      update_cache: yes
    when: ansible_distribution == "Ubuntu"

- hosts: servers
  become: true
  roles:
    - setup-test

- hosts: containers
  become: true
  roles:
    - znas

znas task(main.yml):

---

- name: ZNAS Server Setup Tasks
  tags: znas
  block:
  # Sudo Group Setup
  - name: Make sure we have a 'wheel' group
    group:
      name: wheel
      state: present

  - name: Allow 'wheel' group to have passwordless sudo
    lineinfile:
      path: /etc/sudoers
      state: present
      regexp: '^%wheel'
      line: '%wheel ALL=(ALL) NOPASSWD: ALL'
      validate: '/usr/sbin/visudo -cf %s'

  # zuser + Key Setup
  - name: Create a new regular user (zuser) with sudo privileges
    user:
      name: "zuser"
      state: present
      password: "{{ zuser_password | password_hash('sha512')}}"
      update_password: on_create
      uid: 1111
      groups: wheel
      append: true
      create_home: true
      shell: /bin/bash

error message that occurs (happens during Gathering Facts when running ansible-playbook site-test.yml --ask-become-pass -l containers --tags znas):

redirecting (type: connection) ansible.builtin.lxd to community.general.lxd
<znas> ESTABLISH LXD CONNECTION FOR USER: root
<znas> EXEC /bin/sh -c 'echo ~root && sleep 0'
<znas> EXEC /bin/sh -c 'echo "`pwd`" && sleep 0'
<znas> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo ~/.ansible/tmp `"&& mkdir "` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `" && echo ansible-tmp-1658411856.6940753-5825-74320748781232="` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `" ) && sleep 0'
fatal: [znas]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo ~/.ansible/tmp `\"&& mkdir \"` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `\" && echo ansible-tmp-1658411856.6940753-5825-74320748781232=\"` echo ~/.ansible/tmp/ansible-tmp-1658411856.6940753-5825-74320748781232 `\" ), exited with result 1",
    "unreachable": true
}
10

After looking at Stephen Graber’s/LXD Youtube video again I tried something extremely basic which I thought was already done (as usual, always start with the most basic steps when investigating these things).

I re-installed the Ansible Community.General Collection (ansible-galaxy collection install community.general -f). After making sure that was up to date, then re-running the ansible playbook with the znas tag everything worked perfect. So for some reason that collection was not up to date on my controller node.

Thank you @mratt & @cemzafer for helping dig through these setup items and making me think about how I have both systems setup.

By the way, this project and community is awesome. After using docker for quite a few years I was looking for a more permanent solution for long-run services/applications and LXD/LXC fills that exact need. I think with more work on the documentation side, this could very well rival docker containers for most home users and self-hosters.

1 Like
11

I glad you have resolved the issue, you dont need to define a group name wheel, just create a specific user for ansible and add to sudo group or wheel group depends on your distribution.
Regards.

12

Just made this account to say BIG THANK YOU to ZeroGravitas. YOU ARE SO AWESOME i was having trouble with:
"Ansible_loop_var attribute error: nonetype object has no attribute items" leading to lxd_container " error when installing xroad using ansible playbook

Here’s the log

line 99, in _ansiballz_main invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
File "/home/vboxuser/.ansible/tmp/ansible-tmp-1728036128.1841958-3528-12931979 1180256/Ansiballz_lxd_container.py", line 47, in invoke_module runpy.run_module(mod_name='ansible_collections.community.general.plugins.mod
ules.lxd_container', init_globals-dict(_module_fqn='ansible_collections.communit y.general.plugins.modules.lxd_container', _modlib_path=modlib_path), File "/usr/lib/python3.10/runpy.py", line 224, in run_module return _run_module_code(code, init_globals, run_name, mod_spec) File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code _run_code(code, mod_globals, init_globals,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals)
File "/tmp/ansible_lxd_container_payload_g1eotpsb/ansible_lxd_container_payloa d.zip/ansible_collections/community/general/plugins/modules/lxd_container.py", l ine 901, in <module>

All that was needed was:

ansible-galaxy collection install community.general -f

THANK YOU SO VERY MUCH. YOU ARE AN AMAZING INDIVIDUAL