Should I change the default SSH port on linux servers?

The Internet is a wild and scary place, full of malcontents whose motives range from curiosity all the way to criminal enterprise. These unsavories are constantly scanning for computers running services they hope to exploit; usually the more common services such as SSH, HTTP, FTP, etc. The scans typically fall into one of two categories:

1. Recon scans to see what IP address have those services open.
2. Exploit scans against IP addresses who have been found to be running a specific service.

Considering how large the Internet is it is typically infeasible to look on every port of every IP address to find what's listening everywhere. This is the crux of the advice to change your default port. If these disaffected individuals want to find SSH servers they will start probing each IP address on port 22 (they may also add some common alternates such as 222 or 2222). Then, once they have their list of IP addresses with port 22 open, they will start their password brute force to guess usernames/passwords or launch their exploit kit of choice and start testing known (at least to them) vulnerabilities on the target system.

This means that if you change your SSH port to 34887 then that sweep will pass you on by, likely resulting in you not being targeted by the followup break-in.

Seems rosy right? There are some disadvantages though.

1. Client Support: Everybody who connects to your server will need to know and use the changed port. If you are in a heavily managed environment, this configuration can be pushed down to the clients, or if you have few enough users it should be easy to communicate.
2. Documentation Exceptions: Most network devices, such as firewalls and IDSes, are pre-setup for common services to be run on common ports. Any firewall rules related to this service on this device will need to be inspected and possibly modified. Similarly, IDS signatures will be tweaked so as to only perform SSH inspection on port 22. You will need to modify every signature, every time they are updated, with your new port. (As a data point there are currently 136 VRT and ET snort signatures involving SSH).
3. System Protections: Modern Linuxes often ship with an kernel layer MAC and/or RBAC systems (e.g. SELinux on RedHat based or AppAmor on Debian based) and that are designed to only allow applications to do exactly what they're intended to do. That could range from accessing the /etc/hosts file, to writing to a specific file, or sending a packet out on the network. Depending on how this system is configured it may, by default, forbid sshd from binding to a non-standard port. You would need to maintain a local policy that would allow it.
4. Other Party Monitoring: If you have an external Information Security division, or outsource monitoring, then they will need to be made aware of the change. When performing a security assessment, or analyzing logs looking for security threats, if I see an SSH server running on a non-standard port (or an SSH server on a non-UNIX/Linux for that matter) I treat it as a potential backdoor and invoke the compromised system part of incident handling procedure. Sometimes it is resolved in 5 minutes after making a call to the administrator and being told it's legitimate, at which point I update documentation, other times it really is badness that gets taken care of. In any event, this can result in down-time for you or, at the least, a nerve racking call when you answer your phone and hear, "Hi, this is Bob from the Information Security Office. I have a few questions for you."

Before changing your port you need to take all of this into account so you know you're making the best decision. Some of those disadvantages may not apply, but some certainly will. Also consider what you're trying to protect yourself against. Often times it is simply easier to just configure your firewall to only allow access to 22 from specific hosts, as opposed to the whole Internet.

Yes it can be, not by increasing security but you can reduce the amount of failed login attempts on your server. I always change my default ssh to reduce the warnings I get from ossec. Also if you use a really random port and someone still tries to access your machine, chances are higher that it is a targeted attack rather than a random scanner.

As others said, putting SSH on a port other than 22 will make more unlikely to be hit with a random scan. You will be targetted if the attacker is trying to get your server, not any server.

I have a server with ssh bound to a random high port. And I have a ssh honeypot on port 22, that will reply to any and every login attempt with a 'access denied' message.

I don't think it's a defense by obscurity, but defense in depth: to attack my server, the attacker must first find the port. If I have a few fake honeypots (lots of ports redirecting to the same honeypot), the attacker will hit a lot of fakes and have no way to know if it hit the real ssh or not.

It's only the defense, though. I have portsentry active too, so if someone try a portscan, they will be blocked for an hour. If they bruteforce the password on the right ssh, they will receive 'access denied' for an hour. If they successfully guess the password, they will be presented with a PAM prompt asking for the Google Auth token.

The cost of changing the port is very low, and I think the downsides are justified by the upsides.