ssh delay when connecting

It's most often dns problem. Try setting 'UseDNS no' in sshd_config.

It could possibly be a reverse-dns lookup delay. If your connecting host doesn't have an DNS entry, try adding an entry for your source system in /etc/hosts on the server you're connecting to.