Is changing the SQL Server port really that much safer?

It helps against common port scans that can be initiated through port scanning websites. But it won't help against a committed attacker. It's just another layer, but doesn't add much over the firewall, as you mentioned.

If your SQL Servers are directly connected to the Internet (which they shouldn't be) then it can offer you some protection as most of the generic attack scripts only use the default port numbers.

If your SQL Servers are not accessible directly from the Internet this is pretty pointless. Any firewall will have to allow connectivity to the remote port. As soon as I run the client software on the machine I can see what port the SQL Server is using via the NET STAT command. At this point you've slowed me down exactly 2-3 seconds.

It will break applications that expect port 1433.
Some apps can be configured to deal with this but this has to be deployed.

I'd just leave it. If they "hack" your default instance on port 1433 then you're already bollixed.

You can specify the port for named instances but then port 1434 needs opened to resolve instance to port...