pam service(sshd) ignoring max retries
Solution 1:
While the other answers are correct in elimiating the error message you got, consider that this error message may just be a symptom of another underlying problem.
You get these messages because there are many failing login attempts via ssh on your system. There may be someone trying to brute-force into your box (was the case when I got the same messages on my system). Read your var/log/auth.log for research...
If this is the case, you shoud consider installing a tool like 'fail2ban' (sudo apt-get install fail2ban on Ubuntu). It automatically reads the log files of your system, searches for multiple failed login attempts and blocks the malicious clients for a configurable time via iptables...
Solution 2:
PAM is telling you that it is configured with "retry=3" and it will ignore any further auth requests from sshd within the same session. SSH however will continue trying until it exhausts MaxAuthTries setting (which defaults to 6).
You should probably set both of these (SSH and PAM) to same value for maximum auth retries.
Updated
To change this behaviour:
For sshd you edit /etc/ssh/sshd_config and set MaxAuthTries 3. Also restart SSH server for the setting to take effect.
For PAM, you have to look for configuration in /etc/pam.d directory (I think it's common-password file in Ubuntu), you have to change retry= value.
Note: I would strongy suggest to also check Peter Hommel's answer regarding the reason of these requests as it's possible your SSH is being brute-forced.
Solution 3:
It seems the above analysis is not completely correct. There doesn't seem to be a retry= option for pam authentication (I did find one for pam_cracklib, but that only concerns changing password in the "password" section, not authentication in the "auth" section of pam). Instead, pam_unix contains a builtin maximum number of retries of 3. After 3 retries, pam returns the PAM_MAXRETRIES error code to inform sshd of this.
sshd should really stop trying in this case, regardless of its own MaxAuthTries. It doesn't, which I think is a bug (which I just reported with openssh).
Until that bug is fixed, it seems that setting MaxAuthTries to <= 3 is the only way to prevent this message from showing up.
Solution 4:
The ssh client may attempt to authenticate with one or more keys. Any keys which are not listed in authorized_keys will fail, consuming one of sshd's retries. The client will try every ssh key it has until one succeeds or all fail, so it's good that sshd lets you try several.
If no keys match, sshd may allow you to try a password. Each of these attempts also consumes one of sshd's allowed retries. But, it also consumes one of PAM's allowed retries.
So, the combination of 6 ssh auth tries and 3 pam auth tries is a good thing: it means that ssh will allow 6 auth tries total (key or password) but only 3 password tries.
As others have said, if you often see these in your logs, someone is trying to brute force their way in to your system. Consider using fail2ban to completely block packets from IP addresses from which these attempts originate.