Run command across all containers?

How do you run commands across all of your running containers?

For example, doing an apt update/upgrade across all of them. Do you consider that a problem outside of LXD or would it make sense to allow an --all flag for exec that would just execute across all of them?

for i in `lxc list -f compact | tail -n +2 | awk '{print $1}'` ; do lxc exec $i -- sh -c "apt update && apt -y upgrade" ; done

seems tedious.

Not aware of anything that does what your asking native to LXD.

Ansisble / teraform / cloud-init is probably the more typical approach.

I toyed with something 2 years ago but didn’t communicate it clearly and never bothered to see it through.

Right, something like an Ansible script is probably better suited for this in practice.

In concept, having a lxc exec --all type command doesn’t seem very difficult, but it gets very messy very quickly. That’s because a command like apt update and apt upgrade may need some user input, think things like configuration file prompts and the like. Both commands also print out a bunch of stuff.

Dealing with all of the input and output can get pretty messy and the right behavior may very well depend on the commands being run. For some, you just don’t want an active stdin, for some others, you do but then need to be pretty clever to allow the user to respond to a particular prompt while the other commands are still running in the background.

Same goes for how to mix stdout and stderr of all commands. Just dumping everything on screen isn’t ideal as that’d cause a big confusing mess. Adding a prefix with what instance ran what is nice, but blows up pretty badly because of shell escape sequences, 


Anyway, most of that refers to running things in parallel which is what most folks have been requesting. The same doesn’t really apply when done sequentially as you do with your command.

Personally, I’d done something similar to what you did, just a bit more compact with:

for i in $(lxc ls -cn -fcsv); do lxc exec $i -- apt update; done

The csv output combined with column selector saves you from having to deal with the table and columns.

2 Likes

Thanks for your answers - appreciate it. :slight_smile:

Agree for larger installations, config management is the right approach.

Thanks for the tip with the column selector, I did not know about this one - handy!

enter GNU parallel

example:

parallel -j+0 lxc exec {} -- pacman -Syu --noconfirm ::: $(lxc ls -f json | jq '.[] | select(.type = "container") | select(.config."image.os" == "Archlinux") | select(.status = "Running") | .name' | cut -d \" -f 2)

linux vservers had some tooling around it which was very helpful

It started with a “vsomething” command which could basically be used to build all the other commands. eg start/stop/info/exec etc

Vservers can have tags applied to them, and you can filter on the tags with the vsomething command.

This was quite useful for doing updates or restarts or whatever. So you could tag containers as say “www”, “nginx”, “webmail”. Then you can do some edit to all containers which might run “nginx”, or some different edit only on “webmail” containers, etc. Other uses I’ve found were to tag machines that need special update commands (a “be-careful” tag), or those that should start at boot, or not start at boot, those with unusual cpu architectures, etc

This quite closely, (probably close enough for practical purposes) maps to profiles on LXD. What seems to be missing is a way to filter commands to run only against certain profiles. eg stop all “www” or run exec against all “build-servers” or whatever

Would this be a relatively trivial feature request if I were to file something? So introducing a param across most lxc commands that applies only to certain profiles? (are there any perceived issues with creating a bunch of empty profiles, purely for their benefit of applying something like a “tag” to an instance?)

This code example does not work. Here is my Grok3 session that was extremely beneficial to assist in getting it to work.

The parallel -q for quote handling fixed it. I changed from Arch to ubuntu because the containers return lowercase ‘ubuntu’ and do not locate ‘Ubuntu’.

I am sharing this for future google searches. This should also work with the LXD fork of INCUS.

The --verbose is useful to see the commands being sent to containers you can remove it.

The -q is critical to prevent parallel from stripping the quotes

The backslashes can be removed they are useful for line breaks on a long single line command.

# Parallel be verbose, handle quotes, jobs use all cores
# If you have many containers consider limiting cores -j4
parallel --verbose -q -j+0  \ 
# The ::: is a Parallels parameter 
# The {} is replaced by the container name from the commands after :::
lxc exec {} -- bash -c "apt-get update && apt-get upgrade -y" ::: \ 
# List json data pipe into jq command, parse containers that are Ubuntu
# alter these values for alpine, Arch, etc. 
# currently running then trim to clean it up.
$(lxc ls -f json | jq '.[] | \
select(.type == "container") | \
select(.config."image.os" == "ubuntu") | \ 
select(.status == "Running") | .name' | tr -d '"')
# Results in:
# <container1>
# <container2>
# ...

Each container replaces {} and executes the commands.

Here it is on a single line:

parallel -q -j+0 lxc exec {} -- bash -c "apt-get update && apt-get upgrade -y" ::: $(lxc ls -f json | jq '.[] | select(.type == "container") | select(.config."image.os" == "ubuntu") | select(.status == "Running") | .name' | tr -d '"')

You can turn this into a shell script or use the pet command to load it into a collection of long commands for easy access. The pet command can also do snippets so if you wanted this to run for a specific host you can do so. Or if you want to change the inputs to jq query. i.e. prompt for these values before running the shell one-liner.

1 Like

An even more minimalistic approach in terms of dependencies would be: xargs -P0 -I {} <command> {} instead of parallel

1 Like

You can also skip jq and use this command to get a list of containers:

incus list type=container status=RUNNING image.os="(U|u)buntu" -c n -f csv
1 Like

GNU Parallel is different and more powerful than just xargs. It can handle far more complex scenarios. Learning a complex took has its advantages but only if you use it all the time as you will forget.

Here’s a comparison of what’s different about Parallel: