Securing linux servers: iptables vs fail2ban
should I use fail2ban or iptables?
You use fail2ban in addition to a firewall solution, to extend on-demand those existing firewall rules with rules to block the specific ip-addresses of systems that perform undesirable actions on otherwise public services. They work in concert with each other.
Simplified: a firewall only sees network connections and packets and can make some sense of the patterns therein but it doesn't have the application level insight to distinguish desired and valid requests from malicious, malformed and undesirable requests. For instance your firewall can't tell the difference between a bunch of HTTP API requests and a number incorrect login attempts caused by brute force password guessing on your Wordpress admin page, to the firewall they both are only TCP connections to port 80 or 443.
Fail2ban is a generic and extensible approach to provide that application level insight to your firewall, albeit somewhat indirectly.
Frequently applications will register and log malicious, malformed and undesirable requests as such, but only rarely will they have the native ability to prevent further abuse. Although it is slightly decoupled Fail2ban can then act on those logged malicious events and limit the damage and prevent further abuse, typically by dynamically reconfiguring your firewall to deny further access. In other words Fail2ban gives your existing applications, without modifying them, the means to fend off abuse.
A different method to provide firewalls with application level insights would be by means of a intrusion detection/prevention system.
For instance a webserver is a common public service and in your firewall TCP ports 80 and 443 are open for the internet at large.
Typically you don't have any rate-limiting on the HTTP/HTTPS ports because multiple valid users can have a single origin when they are for instance behind a NAT gateway or a web proxy.
When you detect undesirable and/or malicious actions towards your webserver you use fail2ban to automate blocking such an offender (either block them completely or by only locking their access to ports 80 & 443).
On the other hand SSH access is not a public service, but if you're not in a position to restrict SSH access in your firewall to only white-listed ip-address ranges, rate-limiting incoming connections is one way to slow down brute-force attacks. But your firewall still can't distinguish between user bob successfully logging in 5 times because he's running ansible playbooks and 5 failed attempts to log in as root by a bot.
should I use fail2ban or iptables?
This is somewhat akin to asking "should I use a seatbelt or a car?".
First off, remember that fail2ban really is only a tool to automatically detect recurring entries in text files and execute some command when those meet a specified threshold.
We often use it for blocking hosts that violate some policy as evidenced by recurring log entries that indicate a policy violation, but that's not the only thing you can use it for.
You can use fail2ban to add (and remove) iptables rules on demand. You can also add and remove iptables rules by hand, or you can use fail2ban to do something entirely different in response. That's all about how you configure it.
You should have general firewalling in place regardless of whether you are running fail2ban or not. Such firewalling would be, for example, to block off (incoming or outgoing) traffic that you know is never going to be legitimate. For example, does that database server really need to deal with incoming connections on port 25 from the whole Internet?
On top of that, having fail2ban respond to policy violations by cutting off the offending IP(s) for a while won't do much to secure your server per se (a good exploit only needs to get through your firewall once) but it will cut down on the noise level on your system, including but not limited to in the system logs. The simple way to do that is to have fail2ban run iptables to configure the kernel to drop the packets for a while. If you can reconfigure your perimeter firewall instead of just the host firewall, then only so much the better.
In other words, to the extent that they can be readily separated in the first place, you want both.
could it be an overkill if all I am trying to achieve is to 'block' an IP from the server if they make 2 failed access attempts on any service/protocol over a x amount of time?
That is exactly the use case fail2ban is designed to solve. Using a tool for its intended purpose is almost never overkill.
The goal here is to open daily logwatch reports and not have to scroll through pages of attempted failed connections to the server.
An aside, not directly related to your question: Whenever you are filtering logs for review, consider what you will do about some particular entry. If all you are going to do about an entry is to say "meh" and move on, then you probably want to filter it out. Make sure to save the full logs for review should that be needed, but only push into your regular monitoring workflow the stuff that you are actually going to do something with when it shows up. If you set up fail2ban to block a host after a few failed connection attempts, chances are pretty good that you won't need to review those manually, and can drop them from your monitoring notifications. If a legitimate user complains about loss of access, just pull out the complete logs and take a look.
I solved the same question some years ago. I decided to use iptables with recent modul because of performance and easy setup. I had to protect a lot of virtual containers on hosts. Only keep in mind not to open any DOS vector with your rules. Also use ipset to match network lists or ip lists in rules. I use it for white lists. All networks of one country in one list is great for fine tuning. And it is very easy to protect an other service with same ruleset only by adding one more port to match. So I do not like to change with fail2ban but maybe someone with other needs will be happy with fail2ban.
Here is some sample:
#
# SSH tracking sample
#
#################################################################################
iptables -X IN_SSH
iptables -N IN_SSH
iptables -A IN_SSH -m set --match-set net_blacklist src -p tcp -j REJECT
iptables -A IN_SSH -m set --match-set net_whitelist src -p tcp --match limit --limit 5/second -j LOG --log-prefix whitelist_de_prefix
iptables -A IN_SSH -m set --match-set net_whitelist src -p tcp -j ACCEPT
# filter update
iptables -A IN_SSH -m recent --name sshbf --set --rsource
# connlimit
iptables -A IN_SSH -m connlimit --connlimit-above 4 --match limit --limit 5/second -j LOG --log-prefix ssh_connlimit_per_ip_above_4
iptables -A IN_SSH -m connlimit --connlimit-above 4 -j REJECT
# filter
iptables -A IN_SSH -m recent --name sshbf --rttl --rcheck --hitcount 13 --seconds 60 --match limit --limit 5/second -j LOG --log-prefix ssh_filtered_13in60sec
iptables -A IN_SSH -m recent --name sshbf --rttl --rcheck --hitcount 13 --seconds 60 -j REJECT
iptables -A IN_SSH -j ACCEPT
iptables -A FORWARD -p tcp --dport ssh --syn --jump IN_SSH
# iptables -A INPUT -p tcp --dport ssh --syn --jump IN_SSH
The output of your logging messages could be combined with fail2ban. You can use it for INPUT rules, too.