Debugging iptables and common firewall pitfalls?

Solution 1:

In general:

Viewing and modifying the firewall configuration requires administrator privileges (root) as does opening services in the restricted port number range. That means that you should either be logged in as root or alternatively use sudo to run the command as root. I'll try to mark such commands with the optional [sudo].

Contents:

1. Order matters or the difference between -I and -A
2. Display the current firewall configuration
3. Interpreting the ouput of iptables -L -v -n
4. Know your environment
5. The INPUT and FORWARD chains
6. Kernel modules

1. Order matters or the difference between -I and -A

The thing to remember is that firewall rules are checked in the order they are listed. The kernel will stop processing the chain when a rule is triggered that will either allow or dis-allow a packet or connection.

I think the most common mistake for novice firewall administrators is that they follow the correct instructions to open a new port, such as the one below:

[sudo] iptables -A INPUT -i eth0 -p tcp --dport 8080 -j ACCEPT

and then discover that it won't take effect.

The reason for that is that the -A option adds that new rule, after all existing rules and since very often the final rule in the existing firewall was one that blocks all traffic that isn't explicitely allowed, resulting in

...
7    2515K  327M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited
8        0  0    ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:8080

Or equivalent in iptables-save:

...
iptables -A INPUT  -j REJECT
iptables -A INPUT  -p tcp --dport 8080 -j ACCEPT

and the new rule opening TCP port 8080 will never be reached. (as evidenced by the counters stubbornly remaining at 0 packets and zero bytes).

By inserting the rule with -I the new rule would have been the first in the chain and will work.

2. Display the current firewall configuration

My recommendation for the firewall administrator is to look at the actual configuration the Linux kernel is running, rather than trying to diagnose firewall issues from userfriendly tools. Often once you understand the underlying issues you can easily resolve them in a matter supported by those tools.

The command [sudo] iptables -L -v -n is your friend (although some people like iptables-save better). Often when discussing configurations it is useful to use the --line-numbers option as well to number lines. Refering to rule #X makes discussing them somewhat easier.
Note: NAT rules are included in the iptables-save output but have to be listed separately by adding the -t nat option i.e, [sudo] iptables -L -v -n -t nat --line-numbers.

Running the command multiple times and checking for incrementing counters can be a useful tool to see if a new rule actually gets triggered.

[root@host ~]# iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1     784K   65M fail2ban-SSH  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22
2    2789K  866M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED
3       15  1384 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
4    44295 2346K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22
5    40120 2370K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:80
6    16409  688K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:443
7    2515K  327M REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT 25 packets, 1634 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain fail2ban-SSH (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 REJECT     all  --  *      *       117.239.37.150       0.0.0.0/0           reject-with icmp-port-unreachable
2        4   412 REJECT     all  --  *      *       117.253.208.237      0.0.0.0/0           reject-with icmp-port-unreachable

Alternatively the output of iptables-save gives a script that can regenerate the above firewall configuration:

[root@host ~]# iptables-save
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [441:59938]
:fail2ban-SSH - [0:0]
-A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A fail2ban-SSH -s 117.239.37.150/32 -j REJECT --reject-with icmp-port-unreachable
-A fail2ban-SSH -s 117.253.208.237/32 -j REJECT --reject-with icmp-port-unreachable
COMMIT

It is a matter of preference what you'll find easier to understand.

3. Interpreting the ouput of iptables -L -v -n

The Policy sets the default action the chain uses when no explicit rule matches. In the INPUT chain that is set to ACCEPT all traffic.

The first rule in the INPUT chain is immediately an interesting one, it sends all traffic (source 0.0.0.0/0 and destination 0.0.0.0/0) destined for TCP port 22 (tcp dpt:22) the default port for SSH to a custom target (fail2ban-SSH). As the name indicates this rule is maintained by fail2ban (a security product that among other things scans system log files for possible abuse and blocks the IP-address of the abuser).

That rule would have been created by an iptables commandline similar to iptables -I INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH or is found in the output of iptables-save as -A INPUT -p tcp -m tcp --dport 22 -j fail2ban-SSH. Often you'll find either of those notations in documentation.

The counters indicate that this rule has matched 784'000 packets and 65 Megabytes of data.

Traffic that matches to this first rule is then processed by the fail2ban-SSH chain that, as a non-standard chain, gets listed below the OUTPUT chain.

That chain consists of two rules, one for each abuser (source ip-address 117.253.221.166 or 58.218.211.166) that is blocked (with a reject-with icm-port-unreachable).

 -A fail2ban-SSH -s 117.253.221.166/32 -j REJECT --reject-with icmp-port-unreachable
 -A fail2ban-SSH -s 58.218.211.166/32 -j REJECT --reject-with icmp-port-unreachable

SSH packets that aren't from those blocked hosts are neither allowed nor dis-allowed yet and will now that the custom chain is completed will be checked against the second rule in the INPUT chain.

All packets that weren't destined for port 22 passed the first rule in the INPUT chain and will also be evaluated in INPUT rule #2.

The INPUT rule number 2 means this is intended to be a statefull firewall, which tracks connections. That has some advantages, only the packets for new connections need to be checked against the full rule-set, but once allowed, additional packets belonging to an established or related connection are accepted without further checking.

Input rule #2 matches all open and related connections and packets matching that rule will not need to be evaluated further.

Note: rule changes in the configuration of a stateful firewall will only impact new connections, not established connections.

In contrast, a simple packet filter tests every packet against the full rule-set, without tracking connection state. In such a firewall no state keywords would be used.

INPUT rule #3 is quite boring, all traffic connecting to the loopback (lo or 127.0.0.1) interface is allowed.

INPUT rules 4, 5 and 6 are used to open TCP ports 22, 80 and 443 (the default ports for resp. SSH, HTTP and HTTPS) by granting access to NEW connections (existing connections are already allowed by INPUT rule 2).

In a stateless firewall those rules would appear without the state attributes:

4    44295 2346K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
5    40120 2370K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0
6    16409  688K ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0

or

-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT

The final INPUT rule, #7 is a rule that blocks all traffic that was NOT granted access in INPUT rules 1-7. A fairly common convention: everything not allowed is denied. In theory this rule could have been omitted by setting the default POLICY to REJECT.

Always investigate the whole chain.

4. Know your environment

4.1 . The settings in a software firewall won't effect security settings maintained elsewhere in the network, i.e. despite opening up a network service with iptables the unmodified access control lists on routers or other firewalls in your network may still block traffic...

4.2 . When no service is listening you won't be able to connect and get a connection refused error, regardless of firewall settings. Therefore:

• Confirm that a service is listening (on the correct network interface/ip-address) and using the port numbers you expect with [sudo] netstat -plnut or alternatively use ss -tnlp.
• If your services are not yet supposed to be running, emulate a simple listener with for instance netcat: [sudo] nc -l -p 123 or openssl s_server -accept 1234 [options] if you need a TLS/SSL listener (check man s_server for options).
• Verify that you can connect from the server itself i.e. telnet <IP of Server> 123 or echo "Hello" | nc <IP of Server> 123 or when testing TLS/SSL secured service openssl s_client -connect <IP of Server>:1234, before trying the same from a remote host.

4.3 . Understand the protocols used by your services. You can't properly enable/disable services you don't sufficiently understand. For instance:

• is TCP or UDP used or both (as with DNS)?
• is the service using a fixed default port (for instance something like TCP port 80 for a webserver)?
• alternatively is a dynamic port number chosen that can vary (i.e. RPC services like classic NFS that register with Portmap)?
• infamous FTP even uses two ports, both a fixed and a dynamic port number when configured to use passive mode...
• the service, port and protocol descriptions in /etc/services do not necessarily match with the actual service using a port.

4.4 . The kernel packet filter is not the only thing that may restrict network connectivity:

• SELinux might also be restricting network services. getenforce will confirm if SELinux is running.
• Although becoming slightly obscure TCP Wrappers are still a powerful tool to enforce network security. Check with ldd /path/to/service |grep libwrap and the /hosts.[allow|deny] control files.

5. INPUT or FORWARD Chains

The concept of chains is more thoroughly explained here but the short of it is:

The INPUT chain is where you open and/or close network ports for services running locally, on the host where you issue the iptables commands.

The FORWARD chain is where you apply rules to filter traffic that gets forwarded by the kernel to other systems, actual systems but also Docker containers and Virtual guest Servers servers when your Linux machine is acting as a bridge, router, hypervisor and/or does network address translation and port forwarding.

A common misconception is that since a docker container or KVM guest runs locally, the filter rules that apply should be in the INPUT chain, but that is usually not the case.

6. Kernel modules

Since the packet filter runs within the Linux kernel it can also be compiled as dynamic module, multiple modules actually. Most distributions include netfilter as modules and the required netfilter modules will get loaded into the kernel as needed, but for some modules a firewall administrator will need to manually ensure they get loaded. This primarily concerns the connection tracking modules, such as nf_conntrack_ftp which can be loaded with insmod.

The modules currently loaded into the running kernel can be displayed with lsmod.

The method to ensure modules are loaded persistently across reboots depends on the Linux distribution.

Solution 2:

Iptables/Firewall "introduction"

A Firewall is basically a policy-based network filter. Linux firewalls are built around Netfilter; the kernel's network packet processing framework which is made of several kernel modules performing specific tasks:

1. The FILTER module (always loaded by default) mainly allows us to ACCEPT or DROP IP packets based on a certain matching criteria.
2. The NAT module set allows us to perform Network Address Translations (SNAT, DNAT, MASQUERADE).
3. The MANGLE module allows us to alter certain IP packet fields (TOS, TTL).

Users configure the Netfilter framework to suit their firewall needs using iptables from the command line. With iptables we define rules that instruct the kernel what to do with IP packets when they arrive into, pass through, or leave our Linux box. Each Netfilter main process is represented by a TABLE (FILTER, NAT, MANGLE) on iptables lingo. They have several specific hook points on the network packet flow map where they are invoked by the kernel to perform their duties. Certain specifically located sequences of TABLE calls are generically called built-in CHAINS receiving the names of PREROUTING, INPUT, FORWARD, OUTPUT, and POSTROUTING. It is easy to remember if we associate a TABLE with a "type of process" and a CHAIN with the "location" on the network packet flow map where instances of those processes are invoked.

Since an IP packet is received on a network interface, or created by a local process, until it is finally delivered or discarded the Netfilter engine will sequentially test and apply the rules contained along the network packet flow map. At each block identified by a TABLE@CHAIN pair the user can add one or more of these consecutive rules containing an IP packet matching criteria and a corresponding course of action. There are actions (i.e. ACCEPT, DROP, etc.) that can be performed by more than one TABLE and other actions (i.e. SNAT, DNAT, etc.) that are TABLE specific.

i.e. when an IP packet arrives from a network interface it is first processed by the PREROUTING chain invoking the MANGLE table user defined rules if any. If there are not rules that match the current packet the corresponding MANGLE@PREROUTING default course of action or "policy" applies. At this point if the packet was not dropped the process will continue now invoking the rules of the NAT table at the PREROUTING chain (see the map) and so on. In order to facilitate the rules layout, users can also create their own custom chains and "jump" into them from different points of the map as they wish.

While built-in chains can have user defined policies of either ACCEPT or DROP packets, user defined chains have allways an unchangeable default policy of RETURN to the caller to continue the process.

Iptables commands

The iptables main commands populate the network packet flow map with the required processing rules.

The generic iptables rule can be written as:

# iptables <table> <Add/Insert/Delete> <CHAIN> <PKT_MATCHING_CRITERIA> <ACTION>

It could be read like:

Netfilter (kernel module) please <Add/Insert/Delete> this rule for <table> at <CHAIN> where packets matching <PKT_MATCHING_CRITERIA> have to be <ACTION>ed

<table>
  -t filter       (the filter table is assumed when omitted)
  -t nat
  -t mangle 

<Add/Insert/Delete>
  -A              (append rule at the end of the chain list)
  -I              (insert rule at the begining of the chain list)
  -D              (Delete rule)

<CHAIN>
  PREROUTING
  INPUT
  FORWARD
  OUTPUT
  POSTROUTING
  USER_DEFINED_CHAIN

<PKT_MATCHING_CRITERIA>
ISO Level-2 matching:
  -i [!] <if_name>    or --in-interface [!] <if_name>
          (OUTPUT and POSTROUTING chains cannot match on input  interfaces)
  -o [!] <if_name>    or --out-interface [!] <if_name>
          (INPUT  and PREROUTING  chains cannot match on output interfaces) 
    -mac-source [!] <xx-xx-xx-xx-xx-xx>
            (OUTPUT and POSTROUTING chains cannot match on input  interfaces)

ISO Level-3 matching:
  -s [!] <src_ip>     or --src [!] <src_ip>   or --source [!] <src_ip>
  -d [!] <dst_ip>     or --src [!] <dst_ip>   or --destination [!] <dst_ip>

ISO Level-4 matching:
  -p [!] <prot_name>    or --protocol [!] <prot_name>  (udp|tcp|icmp)

  Also available when ICMP protocol is defined
  --icmp-type [!] <icmp_type>

  Also available when UDP protocol is defined
  --source-port [!] <udp_src_port>      or --sport [!] <udp_src_port>
  --destination-port [!] <udp_dst_port> or --dport [!] <udp_dst_port>

  Also available when TCP protocol is defined
  --source-port [!] <tcp_src_port>      or --sport [!] <tcp_src_port>
  --destination-port [!] <tcp_dst_port> or --dport [!] <tcp_dst_port>
  --tcp-flags [!] <tcp_flags>   (SYN|ACK|FIN|RST|URG|PSH|ALL|NONE)
    --syn
  --tcp-option [!] <tcp_option#>

  --state [!] <state>
  -m <match> [options]

    note: [!] = negation operator

<ACTION>                (also called TARGET)
  -j ACCEPT             (process continues with rules of the next table in map)
  -j DROP               (discard current packet)
  -j REJECT             (discard current packet with ICMP notification)
      option:
      --reject-with <reject_type>
  -j USER_DEFINED_CHAIN   (start traversing USER_DEFINED_CHAIN rules)
  -j RETURN               (return from USER_DEFINED_CHAIN)
  -j LOG                  (log to syslog, then process next rule in table)
      options:
      --log-level <level>
      --log-prefix <prefix>
      --log-tcp-sequence
      --log-tcp-options
      --log-ip-options
      --log-uid

nat table specific
  -j SNAT             (rewrite the source IP address of the packet)
      option:
      --to <ip_address>
  -j SAME             (idem SNAT; used when more than one source address)
      options:
      --nodst 
      --to <a1-a2>
  -j MASQUERADE       (idem SNAT; used when the replace IP is dynamic)
  -j DNAT             (rewrite the destination IP address of the packet)
      option:
      --to <ip_address>
  -j REDIRECT         (rewrite dst IP to 127.0.0.1, PREROUTING and OUTPUT only)
      option:
      –-to-port <port#>

mangle table specific
  -j ROUTE            (explicitly route packets, valid at PREROUTING)
      options:
      --iface <iface_name>
      --ifindex <iface_idx>
  -j MARK             (set Netfilter mark values)
      options:
      --set-mark <value>
      --and-mark <value>
      --or-mark <value> 
  -j TOS              (set the IP header Type of Service field) 
      option:
      --set-tos <value>
  -j DSCP             (set the IP header Differentiated Services Field)
      options:
      --set-dscp <value>
      --set-dscp-class <class>
  -j TTL              (set the IP header Time To Live field)
      options:
      --ttl-set <value>
      --ttl-dec <value>
      --ttl-inc <value>

The iptables auxiliary commands complete the scenario setting default conditoins, listing rules, flushing rules, etc.

#iptables -t <table> -L             
       (Lists the <table> rules in all chains)
#iptables -t <table> -L <CHAIN>     
       (Lists the <table> rules in <CHAIN>)

#iptables -t <table> -N <CHAIN>     
       (Creates a user-defined <CHAIN> for holding <table> rules)
#iptables -t <table> -E <CHAIN> <NEWCHAIN>  
       (Renames <CHAIN> that holds <table> rules to <NEWCHAIN>)

#iptables -t <table> -X   
       (Deletes all user-defined chains created for holding <table> rules)
#iptables -t <table> -X <CHAIN>
       (Deletes user-defined <CHAIN> created for holding <table> rules)

#iptables -t <table> -P <CHAIN> <ACTION>     where <ACTION> = ACCEPT|DROP
       (Sets the default policy of <table> rules at <CHAIN> to <ACTION>)

#iptables -t <table> -F             
       (Flushes (deletes) all <table> rules in all chains)
#iptables -t <table> -F <CHAIN>
       (Flushes (deletes) all <table> rules in <CHAIN>)

#iptables -t <table> -R <CHAIN> <INDEX> <NEWRULE>
       (Replaces <table> rule at position <INDEX> in <CHAIN> with <NEWRULE>

Iptables loads our commands into Netfilter engine at runtime, Netfilter inmediatelly enforces the loaded rules and settings but they are not persistant. Afeter reboot all previously loaded Netfilter rules and settings will be lost. For this reason there are iptables utilities that allow to save the currently active ruleset to a file and to reload it later.

#iptables-save > fileName
      (Save the currently active Netfilter ruleset to fileName)

#iptables-restore < fileName
      (Restore Netfilter ruleset to the one saved in fileName)

Iptables Summary

Netfilter is an extremely flexible and powerful framework but there is a price to pay for it; Iptables is complex. From an user point of view certain terms like TABLE, CHAIN, TARGET do not really match very well the concept they represent and do not make much sense at first. The topic is long, commands seem to have an endless list of parameters. To make things worse there is not a single book that really masters Iptables. They mostly fall into two categories: "recipe book" or "manpage book". I think this introduction gives you a snapshot of the Netfilter/Iptables landscape plus the necessary dose of pre-digested manpage stuff. If you are new at iptables, after reading these paragraphs a couple of times you will be ready to read iptables examples. With some practice you will soon find yourself writting your own rules.

Firewalls

A firewall is mainly designed to dynamically permit or deny network traffic based upon a set of rules. At this point it is easy to understand why the Linux Netfilter/Iptables framework is perfect for firewall construction. Looking at the network packet flow map we find two particularly interesting spots on the FILTER table at the INPUT and FORWARD chains; We can decide there upon IP source address, IP protocol (UDP/TCP), destination port (80, 21, 443, etc), etc, if we ACCEPT, REJECT, or just DROP a particular IP packet. This is what a firewall does 80% of the time when i.e. protects a web server from unauthorized network requests. The other 20% of the time is manipulating (NAT, MANGLE) network packets.

Firewalls Scenarios

There are hundreds of different firewall layouts addresing different needs but 3 of them could be considered the most typical firewall scenarios.

1. Simple web server with one or more interfaces connected to Internet. Policy includes basic rules to permit restricted inbound access, unrestricted outbound access, and anti-spoofing rules. IP forwarding is off.
2. This firewall connects to Internet and to a protected internal area. Policy includes basic rules to permit restricted inbound access, unrestricted outbound access, and anti-spoofing rules. Since the protected area uses private IP addresses source NAT is needed. IP forwarding is on.
3. This firewall connects to Internet, internal protected and demilitarized area. Policy includes basic rules to permit restricted inbound access, unrestricted outbound access and anti-spoofing rules. Since protected and DMZ areas use private IP addresses they need source and destination NAT. IP forwarding is on.

I have written this for: http://www.vercot.com/~jeoss/howto/JeossEasyFirewall.html

---

Solution 3:

Common issues with different protocols

DNS: DNS uses port 53 UDP by default, but messages that won't fit in a single UDP datagram will be transmitted using TCP instead (typically zone transfers and such) requiring port 53 TCP to be opened as well when you run a name server.

Email: Many consumer ISP's block SMTP traffic (or at least the default port TCP 25), making it impossible to directly receive or send email and their customers are forced to use the ISP's SMTP relay for all outgoing email and sometimes for incoming email as well. Relates to §1.1.

FTP: FTP is an odd protocol in the regard that two connections are used. The first is the control connection, by default an FTP server will listen on TCP port 21 for that. The control connection is used for authentication and issueing commands. The actual file transfers and things such as the output of a directory listing go over a second TCP connection, the DATA connection. In active FTP that DATA connection would be initiated from the FTP server from TCP port 20 and connect to the FTP client. Active FTP doesn't work too well with users behind firewalls and NAT gateways so it has mostly fallen into disuse. Most FTP servers suppport Passive FTP instead. With Passive FTP the FTP server opens a listener for the DATA connection on a second port, to which FTP client can then connect. The problem for a firewall is that the DATA port can be any available unprivilged port between 1024-65536.

In a stateless firewall that is typically resolved by restricting the number of passive ports that the FTP server may assign and then explicitly opening those ports. i.e.

iptables -A INPUT -p tcp --match multiport --dports 21000:21050 -j ACCEPT

In a stateful firewall you do not need to explicitly open the DATA port, the netfilter helper module will recognize the dynamic port that gets assigned and dynamically open that port for the correct client by marking the DATA connection as RELATED after which it will match the generic rule:

  iptables -I INPUT -p tcp -m state ESTABLISHED,RELATED -j ACCEPT

This requires that the correct kernel module is loaded, in the FTP case manually by running for instance insmod nf_conntrack_ftp, making that persistent depends accross reboot depends on the distribution.

Note: The FTP connection tracking module will fail when FTP is used with SSL, as the control connection will be encrypted and the nf_conntrack_ftp won't be able to read the PASV repsonse anymore.

NFS and similar RPC services: The problem with RPC services is that by design they don't use a specific fixed port. They can pick any available port at random, which will then be registered with the RPC Portmap daemon. A client trying to connect will query the Portmap daemon and then connect directly to the correct port. That solved the problem of running out of reserved ports...

From a firewall perspective TCP/UDP port 111 needs to be opened and and the actual port that the RPC service is currently using. The problem of opening such a random port in a firewall is typically solved by restricting the RPC service, such as the NFS server, to use a predfined fixed port.