FIN Attack- What is this type of attack really?

It's an older attack originally intended to be a "sneaky, firewall bypass" that was dependent on a few factors that are now uncommon today: old Unix OSes, lack of stateful firewalls, lack of NIDS/NIPS, etc. It can still be useful when testing (i.e., as a fingerprinting technique not an attack per se) completely new or novel TCP/IP stacks (or just new to you or your environment), which is rare but can happen.

Here is a modern replacement, the TCP protocol scan:

nmap --reason -n -Pn --packet-trace -g 80 -sO -p 6 <target ip>

Which is almost exactly the same as the TCP ACK scan (which can be used to map hosts, open ports, firewall rulesets, etc with the caveat that some NIPS, IDS, and modern firewalls will detect -- with another situation-specific event where perhaps it will not notify incident responders or Security Operations Centers because they have more important things to look at these days):

nmap --reason -n -Pn --packet-trace -g 80 -sA -p 80 <target ip>

But the outputs are slightly different and you can see the other packet-level differences as well.

What you are looking for in order to develop a more advanced technique is to identify the subtleties in the RST packets and their window sizes. If you get non-zero window sizes, then you may want to switch to using the TCP Window scan instead of the TCP ACK scan. For more information, see http://nmap.org/book/man-port-scanning-techniques.html

Some other techniques are found in the NSE guide, such as the firewalk and firewall-bypass scripts. However, there are many other techniques including BNAT, fragroute, osstmm-afd, 0trace, lft, and potentially others that detect other inline, non-firewall devices such as WAFs, IDS, IPS, reverse proxies, gateways, and deception systems such as honeypots or active defenses. You will want to be aware of all of this and more if you are performing a network penetration test, but they come in handy for troubleshooting all sorts of network and security issues.

FIN Attack(I assume you mean FIN Scan) is a type of TCP Port Scanning.

According to RFC 793: "Traffic to a closed port should always return RST". RFC 793 also states if a port is open and segment does not have flag SYN, RST or ACK set. The packet should be dropped. It could be an old datagram from an already closed session.

So what the FIN Attack does is to abuse this. If we send a FIN packet to a closed port we get a RST back. If we get no response we know that is either dropped by the firewall or the port is open. Also, the FIN Attack is more invisible than the SYN Scan(sending SYN to see the respons).

However, many system always return RST. And then it is not possible to know if the port is open or closed, for example Windows does this but not UNIX.

FIN scans, as NULL, XMAS, or custom-flags scans --were and-- are used for bypassing firewall and sometimes evading IDS, I quote:

FIN Scan: The key advantage to these scan types is that they can sneak through certain non-stateful firewalls and packet filtering routers. Such firewalls try to prevent incoming TCP connections (while allowing outbound ones) Demonstrating the full , firewall-bypassing power of these scans requires a rather lame target firewall configuration. With a modern stateful firewall, a FIN scan should not produce any extra information.

SYN/FIN One interesting custom scan type is SYN/FIN. Sometimes a firewall administrator or device manufacturer will attempt to block incoming connections with a rule such as "drop any incoming packets with only the SYN Hag set". They limit it to only the SYN flag because they don't want to block the SYN/ACK packets which are returned as the second step of an outgoing connection. The problem with this approach is that most end systems will accept initial SYN packets which contain other (non-ACK) flags as well. For example, the Nmap OS fingerprinting system sends a SYN/FIN/URG/PSH packet to an open port. More than half of the fingerprints in the database respond with a SYN/ACK. Thus they allow port scanning with this packet and generally allow making a full TCP connection too. Some systems have even been known to respond with SYN/ACK to a SYN/RST packet! The TCP RFC is ambiguous as to which flags are acceptable in an initial SYN packet, though SYN/RST certainly seems bogus. Example 5.13 shows Ereet conducting a successful SYNIFIN scan of Google. He is apparently getting bored with scanme.nmap.org.

NMAP Network Discovery by Gordon "Fyodor" Lyon