Run command across all containers?

How do you run commands across all of your running containers?

For example, doing an apt update/upgrade across all of them. Do you consider that a problem outside of LXD or would it make sense to allow an --all flag for exec that would just execute across all of them?

for i in `lxc list -f compact | tail -n +2 | awk '{print $1}'` ; do lxc exec $i -- sh -c "apt update && apt -y upgrade" ; done

seems tedious.

Not aware of anything that does what your asking native to LXD.

Ansisble / teraform / cloud-init is probably the more typical approach.

I toyed with something 2 years ago but didn’t communicate it clearly and never bothered to see it through.

Right, something like an Ansible script is probably better suited for this in practice.

In concept, having a lxc exec --all type command doesn’t seem very difficult, but it gets very messy very quickly. That’s because a command like apt update and apt upgrade may need some user input, think things like configuration file prompts and the like. Both commands also print out a bunch of stuff.

Dealing with all of the input and output can get pretty messy and the right behavior may very well depend on the commands being run. For some, you just don’t want an active stdin, for some others, you do but then need to be pretty clever to allow the user to respond to a particular prompt while the other commands are still running in the background.

Same goes for how to mix stdout and stderr of all commands. Just dumping everything on screen isn’t ideal as that’d cause a big confusing mess. Adding a prefix with what instance ran what is nice, but blows up pretty badly because of shell escape sequences, …

Anyway, most of that refers to running things in parallel which is what most folks have been requesting. The same doesn’t really apply when done sequentially as you do with your command.

Personally, I’d done something similar to what you did, just a bit more compact with:

for i in $(lxc ls -cn -fcsv); do lxc exec $i -- apt update; done

The csv output combined with column selector saves you from having to deal with the table and columns.

2 Likes

Thanks for your answers - appreciate it. :slight_smile:

Agree for larger installations, config management is the right approach.

Thanks for the tip with the column selector, I did not know about this one - handy!

enter GNU parallel

example:

parallel -j+0 lxc exec {} -- pacman -Syu --noconfirm ::: $(lxc ls -f json | jq '.[] | select(.type = "container") | select(.config."image.os" == "Archlinux") | select(.status = "Running") | .name' | cut -d \" -f 2)

linux vservers had some tooling around it which was very helpful

It started with a “vsomething” command which could basically be used to build all the other commands. eg start/stop/info/exec etc

Vservers can have tags applied to them, and you can filter on the tags with the vsomething command.

This was quite useful for doing updates or restarts or whatever. So you could tag containers as say “www”, “nginx”, “webmail”. Then you can do some edit to all containers which might run “nginx”, or some different edit only on “webmail” containers, etc. Other uses I’ve found were to tag machines that need special update commands (a “be-careful” tag), or those that should start at boot, or not start at boot, those with unusual cpu architectures, etc

This quite closely, (probably close enough for practical purposes) maps to profiles on LXD. What seems to be missing is a way to filter commands to run only against certain profiles. eg stop all “www” or run exec against all “build-servers” or whatever

Would this be a relatively trivial feature request if I were to file something? So introducing a param across most lxc commands that applies only to certain profiles? (are there any perceived issues with creating a bunch of empty profiles, purely for their benefit of applying something like a “tag” to an instance?)