SSH resets to default port on reboot

Question

I changed my default SSH port on my home server (in the /etc/ssh/sshd_config file) to port 54747, then restarted the ssh and sshd services (never sure which one so I did both just to be safe). To test my configuration, I logged out and then back in without any problem.

A couple days later, I installed apt updates, and then rebooted my server. When I tried to SSH back in (on port 54747), I got a connection refused error.

For some reason, I tried to SSH on default port, and it worked ! I went back to check on the sshd_config, but it still had the custom port. So I restarted the sshand sshdservices, and it got back to "regular" behaviour (ssh on port 54747). I tried rebooting again, and connection refused again...

Anyone knows what I did wrong ?

Extra details :

• Ubuntu 16.04.2 LTS
• Server is also used a HTPC, with an open session (same user as SSH) on my TV
• I SSH using my laptop's RSA key, and have disabled password auth
• I used to reboot with sudo reboot -h now, but after searching, I discovered it was discouraged by some people, so I tried sudo reboot, but no differences

EDIT Sequence of events :

1. Change SSH port from 22 to 54747 in /etc/ssh/sshd_config
2. Restart ssh and sshd services
3. End current SSH session
4. SSH back in successfully on port 54747
5. Reboot
6. SSH connection error on port 54747, but successful on port 22
7. Restart ssh and sshd services
8. SSH back in successfully on port 54747, connection error on port 22
9. Reboot and go back to 6

EDIT 1 : netstat output

rgo@ATLAS:~$ sudo netstat -lntp | grep :54747
rgo@ATLAS:~$ sudo netstat -lntp | grep :22
tcp6       0      0 :::22                   :::*                    LISTEN      1/init  

EDIT 2 : service sshd status

● ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

EDIT 3 : lsof -i | grep ssh

systemd      1     root   46u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
systemd      1     root   49u  IPv6  14641      0t0  TCP *:ssh (LISTEN)
sshd      4088     root    3u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
sshd      4088     root    4u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
sshd      4202      rgo    3u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)
sshd      4202      rgo    4u  IPv6  42724      0t0  TCP ATLAS:ssh->192.168.1.27:49837 (ESTABLISHED)

For reference, ATLAS is the remote server hostname, 192.168.1.27 is my laptop's LAN IP, and command was executed between steps 6 and 7

ufw status

Status: inactive

EDIT 4 : ps -ef |grep sshd

root      4088     1  0 22:40 ?        00:00:00 sshd: rgo [priv]
rgo       4202  4088  0 22:40 ?        00:00:00 sshd: rgo@pts/1 sshd

Answer 1

ssh may be "socket activated" by systemd depending on configuration, which means that initially it is systemd that sets up the listening port, and sshd is only started when a client first connects. This is to speed up startup time: service daemons are only started on demand.

However this means that you must also configure systemd to the matching port. You'll find the system configuration in /lib/systemd/system/ssh.socket which lists ListenStream=22. To override this, create a file /etc/systemd/system/ssh.socket.d/port.conf (creating the directory ssh.socket.d if needed) that contains:

[Socket]
ListenStream=
ListenStream=54747

Change the number to the port desired. The first blank entry erases the previous default, and the subsequent entry adds the new one. This overrides the default shipped in /lib/systemd/system/ssh.socket and must be done in addition to changing /etc/ssh/sshd_config.

Then run sudo systemctl daemon-reload to tell systemd about your changes, and sudo systemctl reload ssh if your ssh daemon was previously running.

Answer 2

Possible causes that I can think of

1. A different sshd binary is started on boot or sshd is started with a different config. Maybe systemd is the culprit here - it has a different way to change port, via file /usr/lib/systemd/system/sshd.socket apparently: https://www.vultr.com/docs/how-to-change-ssh-port-on-coreos
2. The correct /etc/ or /etc/ssh isn't mounted yet when sshd starts, is it a separate volume on your machine that gets mounted later in the boot process?
3. sshd is lacking read permissions to the config file at boot time, although I don't know if sshd would even start then at all.

Answer 3

ssh is the client process that arbitrates and maintains a user session connection to the ssh server. sshd is the daemon that runs on the ssh server to listen for and authenticate ssh connection requests.

The configuration file on the sshd server that is read when starting the sshd service (which requires sudo privileges to edit) is

/etc/ssh/sshd_config

The service should start out of

/etc/systemd/system/sshd.service

To restart sshd which would involve re-reading the sshd_config file

sudo service sshd restart

To see what port the sshd daemon is listening to, as well as other helpful information, on the ssh server type

sudo service sshd status

Do these steps in the specified order:

Reboot the ssh server

Open a terminal session on the ssh server (not a ssh connection into it)

Type hostname

If hostname does not return the name of the ssh server (Atlas in this case) redo the previous step correctly.

grep Port /etc/ssh/sshd_config - note the port number. Should be the one you specified

sudo service sshd status

If status reports that it is active, running and listening on the custom port you specified, then you are good on that end. If not, the service startup may not be calling the sshd_config file you modified but another config file that contains default info. If the service didn't start (says dead and not active and running, then this is a different problem than what you asked about.

These steps will likely identify the root cause of the problem you are asking about.

For testing purposes and for simplicity: On the client side, from a terminal session you would ssh into the ssh server as follows

ssh -l username -p 54747 hostname

Based on OP feedback, I suspect that sshd isn't starting up on bootup but does start correctly when manually invoked. Successful ssh connections via port 22 may well NOT be connecting to the ssh server but to something else (e.g. localhost). To prove or debunk this, after connecting via ssh type

hostname

Based on what OP is saying, I'm guessing hostname won't be the ssh server atlas.

To further isolate this, after rebooting the ssh server but before doing anything further, from a terminal session on the ssh server (Atlas) type

ssh localhost

If this fails, as it should, then

ssh -p 54747 localhost

If this doesn't work either that will confirm the results obtained when running

sudo service sshd status

Answer 4

Verify your port settings in the /etc/ssh/sshd_config file. Make sure you are editing as sudo or a user in the sudo group. All you have to do to set the port is, on one line type Port 54747. Now, restart the ssh service by running service sshd restart. Then verify that ssh is listening on that port by running sudo netstat -lntp | grep ssh. Reboot and test.

Also check your network settings. If you are on a corporate network, make sure you are in the correct vlan.

Answer 5

Sometimes things just go wrong. If I were on your place, I would try with:

cp /etc/ssh/sshd_config $HOME
sudo apt-get --reinstall install openssh-server

Answer 6

Probably you just answered Y when apt detected differences between your sshd_config and package's one. It asks if you want to install package mantainer's version or keep yours.