Network Issue with AlmaLinux 8/9 on GitHub Actions using Incus

otegami · May 30, 2024, 7:35am

The following error occurred when accessing the external network on AlmaLinux 8/9 with Incus on GitHub Actions.
Could you help me to resolve the following error?

AlmaLinux 8 - BaseOS                            0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name for https://mirrors.almalinux.org/mirrorlist/8/baseos [Could not resolve host: mirrors.almalinux.org]
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.almalinux.org/mirrorlist/8/baseos [Could not resolve host: mirrors.almalinux.org]
Error: Process completed with exit code 1.

ref: Add new line at the end of file · otegami/sandbox-incus@9c38885 · GitHub

How to reproduce it

We can reproduce by re-running Add new line at the end of file · otegami/sandbox-incus@9c38885 · GitHub

Environment

GitHub Actions runner: ubuntu-latest
VM OS: AlmaLinux 8/9
Virtualization tool: Incus

What I tried to do

On Github Actions, Docker runs.
It doesn’t work for me although I tried to follow ‘Prevent connectivity issues with Incus and Docker’

Uninstall Docker
- Uninstall docker by otegami · Pull Request #3 · otegami/sandbox-incus · GitHub
Enable IPv4 forwarding
- Enable IPv4 forwarding by otegami · Pull Request #4 · otegami/sandbox-incus · GitHub
Allow egress network traffic flows
- Allow egress network traffic flows by otegami · Pull Request #5 · otegami/sandbox-incus · GitHub

Expected behavior

We can access the external network on AlmaLinux 8/9 with Incus on GitHub Actions.

stgraber · May 30, 2024, 7:36am

See Can I run incus from Github Actions to spin up an archlinux vm? - #6 by stgraber

otegami · May 30, 2024, 7:39am

Thank you so much for addressing it.

I tried to reset the firewall setting at the following PR.
But we cannot access the external network like using dnf upate.

Reset firewall settings by otegami · Pull Request #6 · otegami/sandbox-incus · GitHub

If I misunderstood something, could you tell me?

stgraber · May 30, 2024, 8:26am

Works fine for me following the instructions from the post I linked.

otegami · May 30, 2024, 8:51am

Thank you so much!

When I added the sleep 10 before executing dnf update with resetting wirefall, it works on GitHub Actions.
ref: Retry to reset firewall with sleep 10 by otegami · Pull Request #7 · otegami/sandbox-incus · GitHub

Could you explain me why you add sleep 10?
If there is a better way to wait for something, could you give me suggestions?

otegami · May 30, 2024, 9:55am

When launching a new instance and immediately attempting to run dnf update, I encountered errors related to downloading metadata from the repository.
The issue was due to DNS resolution failure because the network services were not fully operational yet. Here’s the error message I received:

$ sudo incus launch images:almalinux/8 target && sudo incus exec target -- sudo dnf update -y
Launching target
AlmaLinux 8 - BaseOS                                                                                                                         0.0  B/s |   0  B     00:00
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name for https://mirrors.almalinux.org/mirrorlist/8/baseos [Could not resolve host: mirrors.almalinux.org]
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.almalinux.org/mirrorlist/8/baseos [Could not resolve host: mirrors.almalinux.org]

Waiting a few seconds before running the command resolves this error, as shown below.
I received this advice from stgraber, and it worked wonders—thank you so much!

sudo incus launch images:almalinux/8 target && sleep 5 && sudo incus exec target -- sudo dnf update -y
Launching target
AlmaLinux 8 - BaseOS                                                                                                                         352 kB/s | 2.9 MB     00:08
AlmaLinux 8 - AppStream                                                                                                                      2.8 MB/s |  10 MB     00:03
AlmaLinux 8 - Extras                                                                                                                         1.4 kB/s |  21 kB     00:14
Dependencies resolved.
Nothing to do.
Complete!

To understand why this brief delay is necessary, I ran the following command sequence to monitor the system processes immediately after launch.
kou helped me to figure it out. Thanks!

% sudo incus stop target; sudo incus delete target; sudo incus launch images:almalinux/8 target && sudo incus exec target -- bash -c 'for x in {1..10}; do echo $x; ps axuwww; sleep 1; done' | tee /tmp/log

log

Launching target
1
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  89292  7808 ?        Ss   09:07   0:00 /sbin/init
root          21  0.0  0.0  89292  2492 ?        S    09:07   0:00 (sd-execut
root          22  0.0  0.0  12132  2432 ?        S    09:07   0:00 /bin/sh /etc/systemd/system-generators/lxc /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root          44  0.0  0.0  12132  1892 ?        S    09:07   0:00 /bin/sh /etc/systemd/system-generators/lxc /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late
root          45  0.0  0.0 264668  3456 ?        R    09:07   0:00 ps axuwww
2
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90096  9472 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89648  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
root         498  0.0  0.0 225520  3840 ?        Ss   09:07   0:00 /bin/bash /etc/rc.d/init.d/network start
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         541  0.0  0.0 225520  3968 ?        S    09:07   0:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-lo
root         576  0.0  0.0 224860  3072 ?        S    09:07   0:00 /bin/bash /etc/sysconfig/network-scripts/ifup-ipv6 ifcfg-lo
root         577  0.0  0.0 264668  3584 ?        R    09:07   0:00 ps axuwww
3
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90096  9472 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89648  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
root         498  0.0  0.0 225520  3840 ?        Ss   09:07   0:00 /bin/bash /etc/rc.d/init.d/network start
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         646  0.0  0.0 225520  3840 ?        S    09:07   0:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-eth0 boot
root         665  0.0  0.0  82004  8960 ?        S    09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         671  0.0  0.0  12396  3072 ?        S    09:07   0:00 /bin/bash /usr/sbin/dhclient-script
root         677  0.0  0.0  16560  1792 ?        S    09:07   0:00 arping -D -q -c2 -I eth0 10.222.242.89
root         678  0.0  0.0 264668  3712 ?        R    09:07   0:00 ps axuwww
4
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90096  9472 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89648  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
root         498  0.0  0.0 225520  3840 ?        Ss   09:07   0:00 /bin/bash /etc/rc.d/init.d/network start
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         646  0.0  0.0 225520  3840 ?        S    09:07   0:00 /bin/bash /etc/sysconfig/network-scripts/ifup-eth ifcfg-eth0 boot
root         665  0.0  0.0  82004  8960 ?        S    09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         671  0.0  0.0  12396  3072 ?        S    09:07   0:00 /bin/bash /usr/sbin/dhclient-script
root         677  0.0  0.0  16560  1792 ?        S    09:07   0:00 arping -D -q -c2 -I eth0 10.222.242.89
root         680  0.0  0.0 264668  3456 ?        R    09:07   0:00 ps axuwww
5
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90204  9600 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89640  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         700  0.0  0.0  79176  6784 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-hostnamed
root         705  0.0  0.0  82004  3616 ?        Ss   09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         778  0.0  0.0 211724  4800 ?        Ssl  09:07   0:00 /usr/sbin/rsyslogd -n
root         779  0.0  0.0 235704  3200 ?        Ss   09:07   0:00 /usr/sbin/crond -n
root         780  0.0  0.0 219328  1920 pts/0    Ss+  09:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         785  0.0  0.0 264668  3584 ?        R    09:07   0:00 ps axuwww
6
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90204  9600 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89640  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         700  0.0  0.0  79176  6784 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-hostnamed
root         705  0.0  0.0  82004  3616 ?        Ss   09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         778  0.0  0.0 211724  4800 ?        Ssl  09:07   0:00 /usr/sbin/rsyslogd -n
root         779  0.0  0.0 235704  3200 ?        Ss   09:07   0:00 /usr/sbin/crond -n
root         780  0.0  0.0 219328  1920 pts/0    Ss+  09:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         787  0.0  0.0 264668  3584 ?        R    09:07   0:00 ps axuwww
7
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90204  9600 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        R    09:07   0:00 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89640  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         700  0.0  0.0  79176  6784 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-hostnamed
root         705  0.0  0.0  82004  3616 ?        Ss   09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         778  0.0  0.0 211724  4800 ?        Ssl  09:07   0:00 /usr/sbin/rsyslogd -n
root         779  0.0  0.0 235704  3200 ?        Ss   09:07   0:00 /usr/sbin/crond -n
root         780  0.0  0.0 219328  1920 pts/0    Ss+  09:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         789  0.0  0.0 264668  3456 ?        R    09:07   0:00 ps axuwww
8
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90204  9600 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:01 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89640  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         700  0.0  0.0  79176  6784 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-hostnamed
root         705  0.0  0.0  82004  3616 ?        Ss   09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         778  0.0  0.0 211724  4800 ?        Ssl  09:07   0:00 /usr/sbin/rsyslogd -n
root         779  0.0  0.0 235704  3200 ?        Ss   09:07   0:00 /usr/sbin/crond -n
root         780  0.0  0.0 219328  1920 pts/0    Ss+  09:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         791  0.0  0.0 264668  3584 ?        R    09:07   0:00 ps axuwww
9
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90204  9600 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        R    09:07   0:01 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89640  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         700  0.0  0.0  79176  6784 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-hostnamed
root         705  0.0  0.0  82004  3616 ?        Ss   09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         778  0.0  0.0 211724  4800 ?        Ssl  09:07   0:00 /usr/sbin/rsyslogd -n
root         779  0.0  0.0 235704  3200 ?        Ss   09:07   0:00 /usr/sbin/crond -n
root         780  0.0  0.0 219328  1920 pts/0    Ss+  09:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         793  0.0  0.0 264668  3712 ?        R    09:07   0:00 ps axuwww
10
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.0  0.0  90204  9600 ?        Ss   09:07   0:00 /sbin/init
root          36  0.0  0.0 231888  2688 ?        S    09:07   0:01 bash -c for x in {1..10}; do echo $x; ps axuwww; sleep 1; done
root         483  0.0  0.0  89640  9856 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-journald
root         489  0.0  0.0  93184  7296 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-udevd
dbus         500  0.0  0.0  54220  4736 ?        Ss   09:07   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root         501  0.0  0.0  79408  7040 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-logind
root         700  0.0  0.0  79176  6784 ?        Ss   09:07   0:00 /usr/lib/systemd/systemd-hostnamed
root         705  0.0  0.0  82004  3616 ?        Ss   09:07   0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth0.lease -pf /run/dhclient-eth0.pid -H target eth0
root         778  0.0  0.0 211724  4800 ?        Ssl  09:07   0:00 /usr/sbin/rsyslogd -n
root         779  0.0  0.0 235704  3200 ?        Ss   09:07   0:00 /usr/sbin/crond -n
root         780  0.0  0.0 219328  1920 pts/0    Ss+  09:07   0:00 /sbin/agetty -o -p -- \u --noclear --keep-baud console 115200,38400,9600 linux
root         795  0.0  0.0 264668  3328 ?        R    09:07   0:00 ps axuwww

Based on the log data, it’s evident that network configuration commands take a few seconds to complete. Key processes involved include:

/etc/rc.d/init.d/network start - Initializes network services.
ifup-eth ifcfg-eth0 boot - Activates the Ethernet interface.
dhclient - Acquires an IP address from the DHCP server.
arping - Ensures no IP address conflicts on the network.

These processes are essential for establishing a network connection that allows external commands like dnf update to execute successfully.
Thus, inserting a brief pause after the instance launch and before executing network-dependent commands ensures the system is fully prepared and connected.

stgraber · May 30, 2024, 2:43pm

Right, that. incus start or incus launch will return as soon as the init system has started in the container. We don’t have visibility on when it has finished booting, so immediately running commands in there will likely not result in connectivity for the first few seconds.

qkiel · May 30, 2024, 5:00pm

You could try systemctl is-system-running --wait instead of sleep. Documentation says if --wait is in use, states initializing or starting will not be reported, instead the command will block until a later state (such as running or degraded) is reached.

otegami · May 31, 2024, 3:30am

Thank you so much for your help! However, it appears that the command doesn’t consistently wait for the system to reach the “running” or “degraded” status, despite using is-system-running --wait.

Here’s a link to a job where this issue occurred:

Use shell format · otegami/sandbox-incus@992f150 · GitHub

Here is the script I used:

sudo incus admin init --auto
sudo incus launch images:almalinux/8 target
sleep 1 # Wait for systemd to be ready
sudo incus exec target -- bash -c "sudo systemctl is-system-running --wait"
sudo incus exec target -- sudo dnf update -y
sudo incus stop target
sudo incus delete target

When I run ./test.sh, the results vary:
Sometimes it shows the initializing state and fails:

Launching target
initializing
Errors during downloading metadata for repository 'baseos':
  - Curl error (6): Couldn't resolve host name for https://mirrors.almalinux.org/mirrorlist/8/baseos [Could not resolve host: mirrors.almalinux.org]
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.almalinux.org/mirrorlist/8/baseos [Could not resolve host: mirrors.almalinux.org]

Other times, it successfully reaches the running state:

Launching target
running
AlmaLinux 8 - BaseOS                                                                                                             2.6 MB/s | 3.0 MB     00:01
AlmaLinux 8 - AppStream                                                                                                          4.3 MB/s |  10 MB     00:02
AlmaLinux 8 - Extras                                                                                                              28 kB/s |  21 kB     00:00
Dependencies resolved.
Nothing to do.
Complete!

I’m not sure whether the is-system-running --wait command is effectively waiting for the system to stabilize.

candlerb · May 31, 2024, 7:14am

I tried it for scripted container builds, and I found that the container was still initializing when it returns.

I have a couple of options that I use instead.

One is just to wait for an IPv4 address to be present. (In this build environment the IP addresses start 192, so I just check for that, but you can adjust the logic)

incus-wait() {
  container="$1"

  for i in {1..120}; do
    incus exec "$container" -- ip addr list | grep 'inet 192' && return
    sleep 1
  done
  echo "Timed out waiting for container networking"
  exit 1
}

This could probably also be done from outside the container, using some variant of incus list or incus info instead of incus exec.

Another is to wait until cloud-init has finished:

incus-wait-cloud-init() {
  vm="$1"

  for i in {1..120}; do
    incus exec "$vm" -- systemctl is-active cloud-init && incus exec "$vm" -- cat /run/cloud-init/result.json && return
    sleep 1
  done
  echo "Timed out waiting for cloud-init to complete"
  exit 1
}

And for a VM with incus running inside, I wait until the incus daemon is running:

incus-wait-incus-daemon() {
  vm="$1"

  for i in {1..120}; do
    incus exec "$vm" -- systemctl is-active incus.socket && incus exec "$vm" -- incus version | grep "^Server version: [^u]" && return  # disallow "Server version: unreachable"
    sleep 1
  done
  echo "Timed out waiting for incus to be ready"
  exit 1
}

qkiel · May 31, 2024, 7:16am

How about (link):

incus exec target -- bash -c "while ! systemctl is-system-running --wait; do sleep 1; done"

There’s a catch: if the system never enters the running state but immediately transitions to say degraded, it will never terminate.

But now it becomes more complicated than a simple sleep 10.

Edit:
This worked for me, and it checks for degraded status:

incus launch images:almalinux/8 target
incus exec target -- bash -c 'for i in {1..10}; do system_status=$( systemctl is-system-running --wait 2> /dev/null ); if [[ ${system_status} == "running" || ${system_status} == "degraded" ]]; then dnf update -y; break; else sleep 1; fi; done'
incus stop target
incus delete target

otegami · June 1, 2024, 9:50am

I really appreciate your suggestions. I tried polling using systemctl is-system-running --wait and it worked for me. I apologize for not trying the other ways.

ref: Allow egress network traffic flows by otegami · Pull Request #5 · otegami/sandbox-incus · GitHub

I’ve also left a note below explaining why is-system-running --wait didn’t work on AlmaLinux8.

sudo incus admin init --auto
sudo incus launch images:almalinux/8 target
sleep 1 # Wait for systemd to be ready
sudo incus exec target -- sudo systemctl is-system-running --wait
sudo incus exec target -- sudo dnf update -y
sudo incus stop target
sudo incus delete target

In the AlmaLinux 8 image, it uses systemd version 239. According to this commit, the is-system-running --wait feature was introduced in version 240. This explains why we couldn’t use it.

% sudo incus stop target; sudo incus delete target; sudo incus launch images:almalinux/8 target && sudo incus exec target -- sudo systemctl --version Launching target systemd 239 (239-82.el8) +PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy

So if you tried it using AlmaLinux9, it works using like the following.

sudo incus admin init --auto
sudo incus launch images:almalinux/9 target
sleep 1 # Wait for systemd to be ready
sudo incus exec target -- sudo systemctl is-system-running --wait
sudo incus exec target -- sudo dnf update -y
sudo incus exec target -- sudo systemctl --version
sudo incus stop target
sudo incus delete target

./test.sh
Launching target
running
AlmaLinux 9 - AppStream                                                                                      2.1 MB/s | 8.8 MB     00:04
AlmaLinux 9 - BaseOS                                                                                         3.0 MB/s | 4.6 MB     00:01
AlmaLinux 9 - Extras                                                                                         9.3 kB/s |  18 kB     00:01
Dependencies resolved.
Nothing to do.
Complete!
systemd 252 (252-32.el9_4.alma.1)
+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN -IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified