How do you run commands across all of your running containers?
For example, doing an apt update/upgrade across all of them. Do you consider that a problem outside of LXD or would it make sense to allow an --all flag for exec that would just execute across all of them?
for i in `lxc list -f compact | tail -n +2 | awk '{print $1}'` ; do lxc exec $i -- sh -c "apt update && apt -y upgrade" ; done
seems tedious.
Not aware of anything that does what your asking native to LXD.
Ansisble / teraform / cloud-init is probably the more typical approach.
I toyed with something 2 years ago but didnât communicate it clearly and never bothered to see it through.
Right, something like an Ansible script is probably better suited for this in practice.
In concept, having a lxc exec --all type command doesnât seem very difficult, but it gets very messy very quickly. Thatâs because a command like apt update and apt upgrade may need some user input, think things like configuration file prompts and the like. Both commands also print out a bunch of stuff.
Dealing with all of the input and output can get pretty messy and the right behavior may very well depend on the commands being run. For some, you just donât want an active stdin, for some others, you do but then need to be pretty clever to allow the user to respond to a particular prompt while the other commands are still running in the background.
Same goes for how to mix stdout and stderr of all commands. Just dumping everything on screen isnât ideal as thatâd cause a big confusing mess. Adding a prefix with what instance ran what is nice, but blows up pretty badly because of shell escape sequences, âŠ
Anyway, most of that refers to running things in parallel which is what most folks have been requesting. The same doesnât really apply when done sequentially as you do with your command.
Personally, Iâd done something similar to what you did, just a bit more compact with:
for i in $(lxc ls -cn -fcsv); do lxc exec $i -- apt update; done
The csv output combined with column selector saves you from having to deal with the table and columns.
Thanks for your answers - appreciate it. 
Agree for larger installations, config management is the right approach.
Thanks for the tip with the column selector, I did not know about this one - handy!
enter GNU parallel
example:
parallel -j+0 lxc exec {} -- pacman -Syu --noconfirm ::: $(lxc ls -f json | jq '.[] | select(.type = "container") | select(.config."image.os" == "Archlinux") | select(.status = "Running") | .name' | cut -d \" -f 2)
linux vservers had some tooling around it which was very helpful
It started with a âvsomethingâ command which could basically be used to build all the other commands. eg start/stop/info/exec etc
Vservers can have tags applied to them, and you can filter on the tags with the vsomething command.
This was quite useful for doing updates or restarts or whatever. So you could tag containers as say âwwwâ, ânginxâ, âwebmailâ. Then you can do some edit to all containers which might run ânginxâ, or some different edit only on âwebmailâ containers, etc. Other uses Iâve found were to tag machines that need special update commands (a âbe-carefulâ tag), or those that should start at boot, or not start at boot, those with unusual cpu architectures, etc
This quite closely, (probably close enough for practical purposes) maps to profiles on LXD. What seems to be missing is a way to filter commands to run only against certain profiles. eg stop all âwwwâ or run exec against all âbuild-serversâ or whatever
Would this be a relatively trivial feature request if I were to file something? So introducing a param across most lxc commands that applies only to certain profiles? (are there any perceived issues with creating a bunch of empty profiles, purely for their benefit of applying something like a âtagâ to an instance?)
This code example does not work. Here is my Grok3 session that was extremely beneficial to assist in getting it to work.
The parallel -q for quote handling fixed it. I changed from Arch to ubuntu because the containers return lowercase âubuntuâ and do not locate âUbuntuâ.
I am sharing this for future google searches. This should also work with the LXD fork of INCUS.
The --verbose is useful to see the commands being sent to containers you can remove it.
The -q is critical to prevent parallel from stripping the quotes
The backslashes can be removed they are useful for line breaks on a long single line command.
# Parallel be verbose, handle quotes, jobs use all cores
# If you have many containers consider limiting cores -j4
parallel --verbose -q -j+0 \
# The ::: is a Parallels parameter
# The {} is replaced by the container name from the commands after :::
lxc exec {} -- bash -c "apt-get update && apt-get upgrade -y" ::: \
# List json data pipe into jq command, parse containers that are Ubuntu
# alter these values for alpine, Arch, etc.
# currently running then trim to clean it up.
$(lxc ls -f json | jq '.[] | \
select(.type == "container") | \
select(.config."image.os" == "ubuntu") | \
select(.status == "Running") | .name' | tr -d '"')
# Results in:
# <container1>
# <container2>
# ...
Each container replaces {} and executes the commands.
Here it is on a single line:
parallel -q -j+0 lxc exec {} -- bash -c "apt-get update && apt-get upgrade -y" ::: $(lxc ls -f json | jq '.[] | select(.type == "container") | select(.config."image.os" == "ubuntu") | select(.status == "Running") | .name' | tr -d '"')
You can turn this into a shell script or use the pet command to load it into a collection of long commands for easy access. The pet command can also do snippets so if you wanted this to run for a specific host you can do so. Or if you want to change the inputs to jq query. i.e. prompt for these values before running the shell one-liner.
An even more minimalistic approach in terms of dependencies would be: xargs -P0 -I {} <command> {} instead of parallel
You can also skip jq and use this command to get a list of containers:
incus list type=container status=RUNNING image.os="(U|u)buntu" -c n -f csv
GNU Parallel is different and more powerful than just xargs. It can handle far more complex scenarios. Learning a complex took has its advantages but only if you use it all the time as you will forget.
Hereâs a comparison of whatâs different about Parallel: