toni
December 30, 2025, 2:33am
1
I’ve built a container image with Trixie which seems to work well on my Trixie dev machine. I have a habit of running SSH on random ports to reduce the noise. Using this container on a Bookworm prod machine, I am running into the following problem:
sshd starts on port 22, even though I have configured a different port. To make it use the configured port, I have to
$ sudo systemctl stop ssh.socket
$ sudo systemctl stop ssh
$ sudo systemctl start ssh
which will make sshd pick up the configured port. If I stop the container, then start it again, sshd listens on Port 22 again. I couldn’t find where it might do that, though.
I’ll try to work around it with another systemd unit, but this should not be necessary. However, I’m a bit at a loss as to where to look.
On the Bookworm machine, I have
ii incus 1:6.19.1-debian12-202512140349 amd64 Incus - Container and virtualization daemon
ii incus-base 1:6.19.1-debian12-202512140349 amd64 Incus - Container and virtualization daemon (container-only)
ii incus-client 1:6.19.1-debian12-202512140349 amd64 Incus - Command line client
Currently, the Zabbly servers are not responding, so I can’t update. On the dev machine, I have
$ dpkg -l |grep incus
ii incus 1:6.20-debian12-202512212020 amd64 Incus - Container and virtualization daemon
ii incus-agent 6.0.4-2+deb13u2 amd64 Incus guest agent
ii incus-base 1:6.20-debian12-202512212020 amd64 Incus - Container and virtualization daemon (container-only)
ii incus-client 1:6.20-debian12-202512212020 amd64 Incus - Command line client
ii incus-extra 1:6.20-debian12-202512212020 amd64 Incus - Extra tools
stgraber
December 30, 2025, 2:45am
2
Not sure what you mean by not responding. We have monitoring on all those servers and there’s no reported outage anywhere right now.
stgraber
December 30, 2025, 2:46am
3
toni:
which will make sshd pick up the configured port. If I stop the container, then start it again, sshd listens on Port 22 again. I couldn’t find where it might do that, though.
That sounds like a generic packaging/systemd change in Debian. Moving from sshd always running to having it be socket activated by systemd. The consequence of that is likely that to change the port, rather than just change sshd_config, you need to also edit the ssh.socket unit to change the port it listens to.
1 Like
alangeb
December 30, 2025, 8:18am
4
I just tried this in one of my debian 13 containers - for me it works as expected. Has there been a very recent change?
root@dbgvm:~# apt update
Hit:1 http://deb.debian.org/debian trixie InRelease
Get:2 http://deb.debian.org/debian trixie-updates InRelease [47.3 kB]
Get:3 http://deb.debian.org/debian-security trixie-security InRelease [43.4 kB]
Get:4 http://deb.debian.org/debian-security trixie-security/main Sources [116 kB]
Get:5 http://deb.debian.org/debian-security trixie-security/main amd64 Packages [93.7 kB]
Get:6 http://deb.debian.org/debian-security trixie-security/main Translation-en [58.6 kB]
Get:7 https://pkgs.zabbly.com/incus/stable trixie InRelease [8,953 B]
Get:8 https://pkgs.zabbly.com/incus/stable trixie/main amd64 Packages [3,589 B]
Fetched 372 kB in 2s (199 kB/s)
2 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@dbgvm:~# apt upgrade
Upgrading:
incus-client libcares2
Summary:
Upgrading: 2, Installing: 0, Removing: 0, Not Upgrading: 0
Download size: 6,654 kB
Space needed: 54.3 kB / 50.6 GB available
Continue? [Y/n]
Get:1 http://deb.debian.org/debian-security trixie-security/main amd64 libcares2 amd64 1.34.5-1+deb13u1 [98.3 kB]
Get:2 https://pkgs.zabbly.com/incus/stable trixie/main amd64 incus-client amd64 1:6.20-debian13-202512212057 [6,556 kB]
Fetched 6,654 kB in 1s (4,587 kB/s)
(Reading database ... 109787 files and directories currently installed.)
Preparing to unpack .../incus-client_1%3a6.20-debian13-202512212057_amd64.deb ...
Unpacking incus-client (1:6.20-debian13-202512212057) over (1:6.19.1-debian13-202512140425) ...
Preparing to unpack .../libcares2_1.34.5-1+deb13u1_amd64.deb ...
Unpacking libcares2:amd64 (1.34.5-1+deb13u1) over (1.34.5-1) ...
Setting up libcares2:amd64 (1.34.5-1+deb13u1) ...
Setting up incus-client (1:6.20-debian13-202512212057) ...
Processing triggers for libc-bin (2.41-12) ...
root@dbgvm:~# apt install openssh-server
Installing:
openssh-server
Installing dependencies:
libwrap0 libwtmpdb0 ncurses-term openssh-sftp-server
Suggested packages:
molly-guard monkeysphere ssh-askpass
Summary:
Upgrading: 0, Installing: 5, Removing: 0, Not Upgrading: 0
Download size: 1,252 kB
Space needed: 8,329 kB / 50.6 GB available
Continue? [Y/n]
Get:1 http://deb.debian.org/debian trixie/main amd64 openssh-sftp-server amd64 1:10.0p1-7 [65.3 kB]
Get:2 http://deb.debian.org/debian trixie/main amd64 libwrap0 amd64 7.6.q-36 [55.3 kB]
Get:3 http://deb.debian.org/debian trixie/main amd64 libwtmpdb0 amd64 0.73.0-3+deb13u1 [13.1 kB]
Get:4 http://deb.debian.org/debian trixie/main amd64 openssh-server amd64 1:10.0p1-7 [601 kB]
Get:5 http://deb.debian.org/debian trixie/main amd64 ncurses-term all 6.5+20250216-2 [518 kB]
Fetched 1,252 kB in 0s (33.5 MB/s)
Preconfiguring packages ...
Selecting previously unselected package openssh-sftp-server.
(Reading database ... 109789 files and directories currently installed.)
Preparing to unpack .../openssh-sftp-server_1%3a10.0p1-7_amd64.deb ...
Unpacking openssh-sftp-server (1:10.0p1-7) ...
Selecting previously unselected package libwrap0:amd64.
Preparing to unpack .../libwrap0_7.6.q-36_amd64.deb ...
Unpacking libwrap0:amd64 (7.6.q-36) ...
Selecting previously unselected package libwtmpdb0:amd64.
Preparing to unpack .../libwtmpdb0_0.73.0-3+deb13u1_amd64.deb ...
Unpacking libwtmpdb0:amd64 (0.73.0-3+deb13u1) ...
Selecting previously unselected package openssh-server.
Preparing to unpack .../openssh-server_1%3a10.0p1-7_amd64.deb ...
Unpacking openssh-server (1:10.0p1-7) ...
Selecting previously unselected package ncurses-term.
Preparing to unpack .../ncurses-term_6.5+20250216-2_all.deb ...
Unpacking ncurses-term (6.5+20250216-2) ...
Setting up openssh-sftp-server (1:10.0p1-7) ...
Setting up libwrap0:amd64 (7.6.q-36) ...
Setting up libwtmpdb0:amd64 (0.73.0-3+deb13u1) ...
Setting up ncurses-term (6.5+20250216-2) ...
Setting up openssh-server (1:10.0p1-7) ...
Creating config file /etc/ssh/sshd_config with new version
Creating SSH2 RSA key; this may take some time ...
3072 SHA256:7db6bsB5xyINDl6+4f3zMuqAHhSAp8IYNEGrqY1aRWg root@dbgvm (RSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:haXGdiGQM2C1iHebDMvTy0NxqZxYmQJajsuaWwkWVaE root@dbgvm (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:l90363Ea6JHqtZON6CbaEQh2uVXUndc/D1feBAcV4sE root@dbgvm (ED25519)
Creating user 'sshd' (sshd user) with UID 987 and GID 65534.
Created symlink '/etc/systemd/system/sshd.service' → '/usr/lib/systemd/system/ssh.service'.
Created symlink '/etc/systemd/system/multi-user.target.wants/ssh.service' → '/usr/lib/systemd/system/ssh.service'.
ssh.socket is a disabled or a static unit, not starting it.
Created symlink '/etc/systemd/system/ssh.service.wants/sshd-keygen.service' → '/usr/lib/systemd/system/sshd-keygen.service'.
Created symlink '/etc/systemd/system/sshd.service.wants/sshd-keygen.service' → '/usr/lib/systemd/system/sshd-keygen.service'.
Created symlink '/etc/systemd/system/sshd@.service.wants/sshd-keygen.service' → '/usr/lib/systemd/system/sshd-keygen.service'.
Created symlink '/etc/systemd/system/ssh.socket.wants/sshd-keygen.service' → '/usr/lib/systemd/system/sshd-keygen.service'.
Processing triggers for ufw (0.36.2-9) ...
Processing triggers for man-db (2.13.1-1) ...
Processing triggers for libc-bin (2.41-12) ...
root@dbgvm:~# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 996 15421 495/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 15758 1422/sshd: /usr/sbi
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 996 15433 495/systemd-resolve
tcp 0 0 127.0.0.54:53 0.0.0.0:* LISTEN 996 15435 495/systemd-resolve
tcp6 0 0 :::22 :::* LISTEN 0 15760 1422/sshd: /usr/sbi
udp 0 0 0.0.0.0:5355 0.0.0.0:* 996 15420 495/systemd-resolve
udp 0 0 127.0.0.54:53 0.0.0.0:* 996 15434 495/systemd-resolve
udp 0 0 127.0.0.53:53 0.0.0.0:* 996 15432 495/systemd-resolve
udp 0 0 10.1.84.171:68 0.0.0.0:* 998 881 999/systemd-network
udp6 0 0 :::5355 :::* 996 16618 495/systemd-resolve
root@dbgvm:~# vi /etc/ssh/sshd_config
root@dbgvm:~# systemctl restart ssh
root@dbgvm:~# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 996 15421 495/systemd-resolve
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 996 15433 495/systemd-resolve
tcp 0 0 127.0.0.54:53 0.0.0.0:* LISTEN 996 15435 495/systemd-resolve
tcp 0 0 0.0.0.0:10022 0.0.0.0:* LISTEN 0 13004 1544/sshd: /usr/sbi
tcp6 0 0 :::10022 :::* LISTEN 0 13006 1544/sshd: /usr/sbi
udp 0 0 0.0.0.0:5355 0.0.0.0:* 996 15420 495/systemd-resolve
udp 0 0 127.0.0.54:53 0.0.0.0:* 996 15434 495/systemd-resolve
udp 0 0 127.0.0.53:53 0.0.0.0:* 996 15432 495/systemd-resolve
udp 0 0 10.1.84.171:68 0.0.0.0:* 998 881 999/systemd-network
udp6 0 0 :::5355 :::* 996 16618 495/systemd-resolve
# cat /etc/ssh/sshd_config
# This is the sshd server system-wide configuration file. See
# sshd_config(5) for more information.
# This sshd was compiled with PATH=/usr/local/bin:/usr/bin:/bin:/usr/games
# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented. Uncommented options override the
# default value.
Include /etc/ssh/sshd_config.d/*.conf
Port 10022
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key
# Ciphers and keying
#RekeyLimit default none
# Logging
#SyslogFacility AUTH
#LogLevel INFO
# Authentication:
#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10
#PubkeyAuthentication yes
# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
#AuthorizedPrincipalsFile none
#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody
# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes
# To disable tunneled clear text passwords, change to "no" here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
# Change to "yes" to enable keyboard-interactive authentication. Depending on
# the system's configuration, this may involve passwords, challenge-response,
# one-time passwords or some combination of these and other methods.
# Beware issues with some PAM modules and threads.
KbdInteractiveAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the KbdInteractiveAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via KbdInteractiveAuthentication may bypass
# the setting of "PermitRootLogin prohibit-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and KbdInteractiveAuthentication to 'no'.
UsePAM yes
#AllowAgentForwarding yes
#AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none
# no default banner path
#Banner none
# Allow client to pass locale and color environment variables
AcceptEnv LANG LC_* COLORTERM NO_COLOR
# override default of no subsystems
Subsystem sftp /usr/lib/openssh/sftp-server
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# PermitTTY no
# ForceCommand cvs server
candlerb
December 30, 2025, 9:37am
5
I’m guessing the configuration change you made was setting “Port NNNNN” in /etc/ssh/sshd_config or /etc/ssh/sshd_config.d/something.conf?
Ubuntu 24.04 has the following comment in its sshd_config:
# When systemd socket activation is used (the default), the socket
# configuration must be re-generated after changing Port, AddressFamily, or
# ListenAddress.
#
# For changes to take effect, run:
#
# systemctl daemon-reload
# systemctl restart ssh.socket
#
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
1 Like
alangeb
December 30, 2025, 11:22am
6
@candlerb Thanks! Interesting - and good to know!
@toni I’ve just tried this on debian 12 (using a incus init images:debian/12).
IF I install “task-ssh-server” I get the behavior you describe, and @stgraber identified as socket activated (you’ll find you have ssh.sock in your system).
IF I install “openssh-server” I get the direct sshd behavior you expected (config file has immediate effect on port changes).
candlerb
December 30, 2025, 1:47pm
7
Trying this:
incus launch images:debian/13/cloud test1
incus exec test1 -- apt install -y openssh-server
echo "Port 22222" | incus file push - test1/etc/ssh/sshd_config.d/port.conf
incus stop test1
incus start test1
root@nuc3:~# incus list test1
+-------+---------+--------------------+---------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------+---------+--------------------+---------------------------------------------+-----------+-----------+
| test1 | RUNNING | 10.12.13.74 (eth0) | 2a07:244:36:ca13:1266:6aff:fec2:4a92 (eth0) | CONTAINER | 0 |
+-------+---------+--------------------+---------------------------------------------+-----------+-----------+
root@nuc3:~# nc 10.12.13.74 22222
SSH-2.0-OpenSSH_10.0p2 Debian-7
That works as expected. Trying @alangeb ’s alternative:
incus launch images:debian/13/cloud test2
incus exec test2 -- apt install -y task-ssh-server
echo "Port 22222" | incus file push - test2/etc/ssh/sshd_config.d/port.conf
incus stop test2
incus start test2
root@nuc3:~# incus list test2
+-------+---------+---------------------+---------------------------------------------+-----------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------+---------+---------------------+---------------------------------------------+-----------+-----------+
| test2 | RUNNING | 10.12.13.250 (eth0) | 2a07:244:36:ca13:1266:6aff:fe41:765a (eth0) | CONTAINER | 0 |
+-------+---------+---------------------+---------------------------------------------+-----------+-----------+
root@nuc3:~# nc 10.12.13.250 22222
SSH-2.0-OpenSSH_10.0p2 Debian-7
That also works exactly as expected. So I’m at a loss to reproduce the problem. The OP @toni still hasn’t said exactly what configuration change they tried, to set the listening port.
The underlying host here is Ubuntu 22.04 with incus 6.0.5 (Zabbly LTS repo)
toni
December 30, 2025, 1:56pm
8
Yes, that’s what I did, I made a file that’s being included and also shows the correct port number when I run sshd -G.
toni
December 30, 2025, 1:58pm
9
It’s the same container on both host systems, but they behave differently. That’s what’s irking me.
toni
December 30, 2025, 1:59pm
10
I mean that if I run apt update, it hangs while trying to access your server(s). Oh, and it hangs for minutes (I aborted it at some point). Might also be a routing problem somewhere, it doesn’t have to be your servers. It also works now.
candlerb
December 30, 2025, 2:07pm
11
What happens if you try the exact steps that I posted? (Containers “test1” and “test2”)
If they don’t work for you, then there’s something different about your system (host and/or incus), and we’ll need to narrow that down.
But if they do work, then what is the exact set of steps that you need to do to reproduce the problem on your system?
toni
December 30, 2025, 2:42pm
12
In my image, I have openss-server, not task-ssh-server.
toni
December 30, 2025, 3:33pm
13
As @stgraber indicated, modifying the socket activation file solved the problem.
candlerb
December 30, 2025, 3:56pm
14
I showed both: openssh-server in the test1 container, task-ssh-server in test2 container. Both worked as expected.
But why didn’t I have to do so? What’s different about the way you started the container and configured the ssh port, versus the way I did?
In any case, it seems clear this is not an issue with incus, but just configuration of openssh in Debian 13.