Why is {a^nb^n | n >= 0} not regular?

What you're looking for is Pumping lemma for regular languages.

Here is an example with your exact problem:

Examples:
Let L = {a^mb^m | m ≥ 1}.
Then L is not regular.
Proof: Let n be as in Pumping Lemma.
Let w = aⁿbⁿ.
Let w = xyz be as in Pumping Lemma.
Thus, xy²z ∈ L, however, xy²z contains more a’s than b’s.

Because you can't write a finite state machine that will 'count' identical sequences of 'a' and 'b' symbols. In a nutshell, FSMs cannot 'count'. Try imagining such a FSM: how many states would you give to symbol 'a'? How many to 'b'? What if your input sequence has more?

Note that if you had n <= X with X an integer value you could prepare such FSM (by having one with a lots of states, but still a finite number); such language would be regular.

The reason is, you have to reach the final state only when no. of 'a' and no. of 'b' are equal in the input string. And to do that you have to count both, the no. of 'a' as well as no. of 'b' but because value of 'n' can reach infinity, it's not possible to count up to infinity using a Finite automata.

So that's why {a^n b^n | n >= 0} is not regular.