How to disable "Shake password dialog when authentication fails" of Gnome Screensaver?
For a moment, I thought that this might be inherited from the GDM configuration (since the GDM login screen does the same thing), but apparently it's not.
After checking a few other places without any luck, I decided to find out for myself and took a look at the source code(v2.30). The code responsible for the shaking only checks to make sure the dialog isn't already being shaken. It makes no checks against any configuration, so there doesn't appear to be a way to disable it without changing the code itself.
You might try switching to xscreensaver and see if that helps.