Can a malicious actor lock the real user out by deliberately trying incorrect passwords every X minutes?

Login brute-force protection can be enforced in three ways:

• Temporary Lockout
• Permanent Lockout
• CAPTCHA

In my perspective, CAPTCHA is the most reasonable solution to avoid the risk of bruteforce as well as denial of service due to account lockout. You might have seen a CAPTCHA appearing on the login pages of Facebook and Gmail in case you enter wrong password more than three or four times. That's a decent way of restricting bots from bruteforcing and at the same time avoiding locking users out.

Permanent lockout is not a novel solution, and it adds a lot of operational overhead to customer support team if they have to manually unlock the account for the user. Temporary lockout on the other hand deter bruteforcing, but like the scenario you mentioned, it can lock out a genuine user.

Yes, some websites do that in order to prevent bruteforce or password guessing attacks. Instead of banning the user, the website should ban the IP address from accessing the website.

If the attacker is hopping from one IP address to another and doing the brute force attack, then in that case the website can ban the user id and notify the banned user through email or some other way or may be banning the user for a short duration of time will also do the needful.

Other answerers have already offered some valuable means by which a website can prevent abuse of anti-password-guessing measures to lock people out. Here is another one: IP whitelisting.

@Skynet already suggested blacklisting , but with attackers nowadays being able to build botnets well over a million devices in size, that might not be very effective against a resourceful attacker. (Note that an attacker wouldn't necessary have to build such a botnet themselves: many cybercrooks rent out access to their bots.)

So whilst blacklisting could offer a defense against a less powerful attacker, if an attack scales up to a large number of IP addresses, an alternative would be to try to restrict access to IP addresses which have previously been used to successfully log in to the account.

Of course, care still needs to be taken: if an attacker has access to a device that uses a whitelisted IP address, a website should look to ensure this will not allow the attacker to escape brute-force detection. Also, since a user may have a dynamic IP address or for some other reason try to login from a new IP, there needs to be some alternate mechanism by which the user can bypass the login blocking, such as:

• By entering a CAPTCHA (though be warned: a determined attacker could hire people to solve these at a surprisingly low price);
• Clicking a link in an e-mail (you'll also want to build in some checks to prevent this from being abused to flood people's inboxes); or
• Using some kind of two-factor authentication to prove that they are the legitimate owner of the account.