Number of ways to arrange 5 monkeys in a row?

Here's how I'd approach this particular problem. I'll solve for $k$ monkeys afterwards.

You have $5!$ ways for the monkeys to be arranged in a line without restriction.

There are $8$ ways that $A$ and $B$ can be positioned next to each other; there are $4$ pairs of adjacent spaces, and either $A$ or $B$ can be on the left.

For each of these cases, there are $3! = 6$ ways to arrange the other three monkeys.

So the answer is $5! - 8 \cdot 3! = 72$ ways.

Now, just apply to $k$ monkeys using the same argument:

$$P(k) = k! - 2(k-1)(k-2)! = k! - 2(k-1)!.$$

(Hat tip to user471297 for the last simplification.)

Recursive solution (the complementary counting solution is outlined in John's answer):

Let $f(k)$ be the number of ways to arrange $k$ monkeys, including $a$ and $b$ such that these two aren't next to each other.

Case 1: $a$ or $b$ is at the beginning of the line.

Counting this case directly, we first choose the leading monkey in one of $2$ ways. Then we find that the other of these two monkeys is in one of $k-2$ positions (any spot except for the one occupied by the first monkey, and the one immediately behind it). The other $k-2$ monkeys can be in any order, so we get $2 (k-2) (k-2)!$ ways.

Case 2: Neither are at the beginning of the line.

There are a total of $k-2$ choices for the monkey to lead the line; after that, we have the $k-1$ subproblem, so we find $(k-2)f(k-1)$ ways here.

Combining these, we have a total of $f(k) = (k-2)(2(k-2)! + f(k-1))$ good arrangements. Starting with $f(2) = 0$, an inductive argument should show that this matches the closed form answer.

@John and @platty have both supplied good answers. Here is another approach.

$a$ is at an end of the row: Since $a$ can be at the left or right end of the row, there are two ways to place $a$. For each such choice, there are three ways to place $b$ so that $b$ is not adjacent to $a$. The remaining three monkeys can be arranged in the three remaining positions in $3!$ ways. Hence, there are $$2 \cdot 3 \cdot 3!$$ arrangements in which $a$ is at an end of the row.

$a$ is not at the end of the row: Since there are five positions including the two ends of the row, there are three choices for the position of $a$. Since $b$ cannot be adjacent to $a$, there are two ways to place $b$. The remaining three monkeys can be arranged in the three remaining positions in $3!$ ways. Hence, there are $$3 \cdot 2 \cdot 3!$$ arrangements in which $a$ is not at an end of the row.

Total: Since the two cases are mutually exclusive and exhaustive, the five monkeys can be arranged in $$2 \cdot 3 \cdot 3! + 3 \cdot 2 \cdot 3! = 72$$ ways if $a$ and $b$ are not in adjacent positions.