(Dis)prove that this system has only integral solutions: $\sqrt x+y=7$and $\sqrt y+x=11$

You want \begin{cases} x=(7-y)^2\\ y=(11-x)^2\\ 0\le x\le 11\\ 0\le y\le 7 \end{cases}

The equation becomes $$ x=49-14(121-22x+x^2)+(121-22x+x^2)^2 $$ which reduces to $$ (x-9)(x^3-35x^2+397x-1444)=0 $$ (courtesy of WolframAlpha). The polynomial $f(x)=x^3-35x^2+397x-1444$ has at least one real root. It has indeed three. One of them satisfies the condition $0\le x\le 11$ and it's approximately $7.87$. With this value of $x$ we get $y\approx 9.79$ that doesn't satisfy $0\le y\le 7$.

One can be more precise: call $\alpha$ the least root of $f$. Then $7<\alpha<8$, so that $3<11-\alpha<4$ and so $9<(11-\alpha)^2<16$, which shows that the limitation on $y$ is not fulfilled.

How can you know this? Compute

• $f(7)=-37$
• $f(8)=4$
• $f(12)=8$
• $f(13)=-1$
• $f(15)=11$

Thus you know that the three roots of $f$ are $7<\alpha<8$, $12<\beta<13$ and $13<\gamma<15$.

If you move things around and square the equations, you have the following two starting equations. $$x=(7-y)^2$$ $$y=(11-x)^2$$

If you plug one of these into the other and factor out a term, you get $$ (x-9)(x^3-35x^2+397x-1444)=0. $$

This is a quartic equation, with four solutions in general. There is a linear term, $x-9$, and the remaining cubic term. If $x=9$, the linear term goes to zero and the equation is solved. If you plug $x=9$ back into the original equations, you get $y=4$, so this represents the original solution you found. To find the other solutions, we only have to focus on the roots of the remaining cubic term. So we're left with finding the roots of this equation

$$ x^3-35x^2+397x-1444=0. $$

All cubics have three roots, if you count complex roots and double roots. The possibilities go like this for any cubic:

There are three real, distinct roots

There are three real roots, but two of them are merged into a double root

There is one real root and a pair of complex conjugate roots

You can follow along with this procedure from Wikipedia to calculate the exact roots of the equation. Or you can use a computer solver for the roots that does a similar procedure internally:

This shows you that this cubic is in the first family with three real, distinct roots, with $x \approx 7.8687, 12.848, 14.283$ and corresponding $y\approx 9.80504, 3.4151, 10.7781$.

Edit

As a user pointed out, the values we just found should be plugged into the original equations to see if they actually solve the equation, or if they are extraneous. It turns out that these three solutions are indeed extraneous, meaning that they are spurious results of squaring the original equations. This leaves the only solution we've found as $(9,4)$. It is necessary that the solutions are a subset of the four presented, because the system is fundamentally fourth order (two quadratic equations), so it can have at most four solutions. The only one of the necessary solutions that is sufficient is $(9,4)$, so this is the unique solution. It consists of two integers, so we have proof that this equation only has integer solutions.

Solve the first equation for $x,$ the second for $y,$ to get

$$ x = (y-7)^2.$$

$$ y = (x-11)^2.$$

Substitute the value for $x$ into the second equation, to get:

$$y = ((y-7)^2 -11)^2 = (y^2-14 y + 38)^2.$$ This is a quartic equation, but dividing through by $y-4,$ you get an irreducible cubic $y^3-24y^2+176 y - 361.$ Now, being cubic, it has at least one real solution, which would indicate that there is at least one more real pair $(x, y).$

EDIT Mathematica seems to think that the only solution is $9, 4.$