System of two Equations: $\sqrt x+y=11$ and $\sqrt y+x=7$

Assume $x$ and $y$ are integers. Notice that, in this case, if $\sqrt x +y=11$, an integer, then $\sqrt x $ must be an integer. A similar argument can be made for $y$. So if they're integers then they're both perfect squares. Rephrasing in terms of the square roots (still integers) $X=\sqrt x,Y=\sqrt y$ $$X+Y^2=11$$ $$Y+X^2=7$$ subtracting the second equation from the first: $$X-Y+Y^2-X^2=4$$ $$(X-Y)+(Y-X)(Y+X)=4$$ $$(Y-X)(X+Y-1)=4$$ Both of the brackets are integers, so the only values they can take are the factors of $4$. So either $$Y-X=2,X+Y-1=2$$ or $$Y-X=4,X+Y-1=1$$ or$$Y-X=1,X+Y-1=4$$ Solving each of these is simple. The only one that gives positive integer values (the conditions of our little set up here) is the $3^{rd}$ one, which gives the answer you found. Keep in mind that there's nothing wrong with guessing and playing around with the problem first, then coming to a more structured argument later. If you want a full analytic solution you could use the quartic equation on the one you have and rule out the other solutions as involving the wrong branches of $\sqrt x$, but it's pointlessly messy.

Once you guessed the solutions, you can easily prove that there are no others. Rewrite the equations as $y=11-\sqrt x=F(x)$ and $x=7-\sqrt y=G(y)$. Note that both $x,y\le 11$, so their square roots are at most $4$, which means that $x,y\ge 3$. Now just observe that $z\mapsto \sqrt z$ is a contraction on $[3,\infty)$ (the difference of values is less than the difference of arguments). Thus, $F$ and $G$ are also contractions whence if we had two different solutions $(x_1,y_1)$ and $(x_2,y_2)$, we would get $$ |x_1-x_2|=|G(y_1)-G(y_2)|<|y_1-y_2|=|F(x_1)-F(x_2)|<|x_1-x_2| $$ which is absurd.