British Maths Olympiad (BMO) 2005 Round 1 Question 1 how to make progress?

First note that we have some lower bounds: Not only are $x,y,n$ positive, but the amounts $y-3$ and $x-n$ must also be positive. In fact, if $y=4$, we get $x=n-3$, contradicting $x>n$; we conclude that $y\ge 5$

Plugging the value of $x$ from the first into the second equation, we get $$\tag1y+n=3(n(y-3)-3-n) $$ which we can rearrange into the following equivalent equations, where we first expand and bring all to one side and then play around to find a product of linear factors that resembles the total as closely as poyyible: $$\tag23yn-13n-y-9=0.$$ $$\tag3(y-4)(3n-1)=n+13$$ $$\tag4(y-5)(3n-1)=14-2n.$$ Especially $(4)$ raises our interest: We already know that $y\ge 5$. Hence the left hand side is non-negative and we must have $14-2n\ge 0$, i.., $n\in\{1,2,3,4,5,6,7\}$. For each of these cases, we can compute $y=\frac{14-2n}{3n-1}+5$ and then $x=n(y-3)-3$: $$\begin{matrix} n=1,&y=11,&x=5\\ n=2,&y=7,&x=5\\ n=3,&y=6,&x=6\\ n=4,&y\notin \Bbb N\\ n=5,&y\notin \Bbb N\\ n=6,&y\notin \Bbb N\\ n=7,&y=5,&x=11\\ \end{matrix}$$ among which we detect precisely four solutions.

I think you are trying to look only at how $n$ is expressed in terms of the rest of the variables. Indeed, such an approach is good, but you can do with other variables as well! So let's try to write $n$ in terms of $y$, by eliminating $x$ from both equations.

From the first, $x = ny-3n-3$. From the second, $x = \frac{y+n}{3} + n$.

Combining these, we get $\frac{y+n}{3} + n = ny-3n-3$,so multiplying by $3$, $y+4n = 3ny-9n-9$. Isolating $n$ gives $y + 9 = n(3y-13)$, and hence $$ n = \frac{y+9}{3y-13} $$

Note that $n$ is a positive integer. If $y > 12$, then $3y-13 > 22$ while $y + 9 < 22$, so that $n$ cannot possibly be an integer! Furthermore, $3y-13 < 0$ if $y < 5$, so we have $5 \leq y \leq 11$.

Now, we can test : $y = 5$ gives $n = 7$ and $x = 11$, which works out.

$y = 6$ gives $n = 3$ and $x = 6$, which works out.

$y = 7$ gives $n = 2$ and $x = 5$, which works out.

$y = 8,9,10$ don't work out. Finally, $y = 11$ gives $n = 1$ and $x = 5$ which works out.

Finally, $n = 1,2,3,7$.

Concluding, the difference of approach was in isolating a different variable from that of $n$. Remember that all the variables are related in a manner such that knowing one means you know the rest, so it was sufficient to deal with any of the variables. Finally, expressions involving $\frac{ay+b}{cy+d}$ are restrictive if integers, when $a,c$ are small.