Inconsistent behaviour lxc exec in shell script

Hi,

In my shell script I have this: lxc exec $container_name -- passwd -d ubuntu
Which has no effect on the container.
My guts tell me this has something to do with the user who executing the command so I changed it to su - $USER -c "lxc exec $container_name -- passwd -d ubuntu" and apparently it works.

Is it intended ? I’m not sure why this doesn’t work since it’s all fine for other commands.

I have tried that on an ubuntu: container and I managed to run that initial command from the shell and reset the password. I also got a warning about an unrelated issue,

$ lxc exec mycontainer -- passwd -d ubuntu
passwd: password expiry information changed.

Have a look at this also, though: Only one lxc exec in a shell while loop

Hi,
Thanks for the quick response. You run it directly in cli or through a shell script ? Because through the cli it works fine. I also got that warning whether it is run through script or cli. It juts doesn’t change the passwd when being run through script. So I’m not really sure what happened.
Is there a log somewhere ?

Here is my test case. It works for me from shell and also in a script.

$ lxc exec mycontainer -- grep ubuntu /etc/shadow
ubuntu:$6$1kSOEWuTDz3avfOv$ixt/9DWaeuaa78riKKFvJm93UOemzSdjWgJ/4o.pnNnT9eEG64lk7Dy1LoFRwo6AbUS5QRO5.xcZVA0wCVi6y0:18690:0:99999:7:::
$ cat mytest.sh 
#!/bin/sh
lxc exec mycontainer -- passwd -d ubuntu
$ bash mytest.sh 
passwd: password expiry information changed.
$ lxc exec mycontainer -- grep ubuntu /etc/shadow
ubuntu::18690:0:99999:7:::
$ 

What’s good to do, is provide that test case of yours that it did not work, so that we can try to replicate as well.

When you run a process in a container, there is no default logging as to what really happened inside the container. Some commands may fail silently if they need input. See the link I posted earlier about this issue.

Hello, here the test case

init/configs/lxd 
❯ cat test.sh 
#!/bin/sh
lxc copy base test
lxc start test 2> /dev/null
lxc exec test -- passwd -d ubuntu

init/configs/lxd 
❯ ./test.sh 
passwd: password expiry information changed.

And then

init/configs/lxd took 7s 
❯ lxc console test
To detach from the console, press: <ctrl>+a q

Ubuntu 18.04.5 LTS test console

test login: ubuntu
Password: 

Login incorrect

Which means the password is still there. If I replace lxc exec test -- passwd -d ubuntu with su - $USER -c "lxc exec test -- passwd -d ubuntu" then it works as intended.

In such cases, it is good to shorten the replication steps as much as possible so that it becomes apparent in the end which line causes the issue.
If you were to run just the command lxc exec mycontainer -- passwd -d ubuntu in a script, does it work for you?

Here is my minimal script,

#!/bin/sh
lxc launch ubuntu: mycontainer
sleep 5
lxc exec mycontainer -- passwd -d ubuntu
lxc exec mycontainer -- grep ubuntu /etc/shadow

The output in my case is

Creating mycontainer
Starting mycontainer
passwd: password expiry information changed.
ubuntu::18690:0:99999:7:::

The last line says ubuntu::18690..... Those :: after ubuntu means that the ubuntu account does not have a password. And you should be able to login with lxc console without a password.

indeed if I only put the line with passwd -d in the script it works fine. Any lxc command precede it and it fails.

Give me an example of a reproducible script that fails, preferably with only one extra preceding command. Or, whichever is the minimal script that reproduces the problem.

The line lxc start test 2> /dev/null redirects the standard error. Normally, you wouldn’t get any error if the container starts OK. Considering that that is the preceding line of the failed passed -d, have you tried with removing the redirection altogether?

Hi, sorry for the delay. Here’s the script

#!/bin/sh
read -p "New container name: " container_name
lxc copy $container_name test
echo "created container test"
lxc start test
lxc exec test -- passwd -d ubuntu

Do you have any update ?

Hi!

Sorry for the delay.

The gist of the issue is that when you lxc start a container, the command returns quickly, but the container may not have actually finished booting. The reason is that LXD gives the instruction to the container to boot and the container is now on its own. It might get ready straight away, or it may take a few moments.

How can you be sure when to start running lxc exec on a container that you have just lxc start-ed?

You could put some delay for a few seconds, with lxc exec test -- sleep 5.

But the elegant way is to make use of cloud-init. The ubuntu: repository containers, and also the /cloud containers from the images: repository support cloud-init. cloud-init is a service that runs at the very end of the startup process. If you run the following, it will wait (will print dots while waiting) until cloud-init finishes.

lxc exec test -- cloud-init status --wait

Then, if you lxc exec test -- passwd -d ubuntu will work as expected.

But do you need to wait for the container to boot up before you can run commands?
The answer is no. If the command is lxc exec test -- touch /i_was_here, then it runs successfully even as soon as you run lxc start test.

Let’s avoid using passwd -d to reset the password of an account, but rather run the following. This is a shell command, does not invoke the passwd command, and directly edits the /etc/shadow file to remove the password lock.

lxc exec mytest -- sed -i '/^ubuntu\:/{s/\!//g}' /etc/shadow

We run it and let’s see how does the /etc/shadow line for ubuntu looks like. That’s weird. Now there are two characters. We are supposed to have removed ! and now we see that character and then an asterisk!

ubuntu:!*:18713:0:99999:7:::

When you copy a LXD container, the copied container apparently runs cloud-init as soon as it is started for the first time. And per /var/log/cloud-init.log, the ubuntu: container images create a non-root account and then lock it. Here is the relevant fragment of /var/log/cloud-init.log,

2021-03-27 20:14:11,966 - __init__.py[DEBUG]: Adding user ubuntu
2021-03-27 20:14:11,966 - subp.py[DEBUG]: Running hidden command to protect sensitive input/output logstring: ['useradd', 'ubuntu', '--comment', 'Ubuntu', '--groups', 'adm,audio,cdrom,dialout,dip,floppy,lxd,netdev,plugdev,sudo,video', '--shell', '/bin/bash', '-m']
2021-03-27 20:14:12,001 - subp.py[DEBUG]: Running command ['passwd', '-l', 'ubuntu'] with allowed return codes [0] (shell=False, capture=True)

I’ll stop here. The importance of this whole exercise is that the user account files should be considered not ready when a container is starting up. If you want to edit those file, you would need to wait until cloud-init does its own work, and then you can proceed and make changes.

Thank you for the throughout response ! it totally fixed the problem.