Is my method of solving equation correct?

Spoiler alert: this is the solution.

Let's take $ a = \sqrt[5]{16+\sqrt x} $ and $ b = \sqrt[5]{16-\sqrt x} $ . Note that because $ \sqrt x \ge 0 $, $ a $ is always positive (technically greater than or equal to $ \sqrt[5]{16} $), but $ b $ can be positive, zero, or negative. We have: $$ a + b = 2 $$ $$ (a + b)^5 = 32 $$ We also have: $$ a^5 = 16 + \sqrt x $$ $$ b^5 = 16 - \sqrt x $$ $$ a^5 + b^5 = 32 $$ Aha! Let's see where this takes us: $$ (a + b)^5 = a^5 + b^5 $$ $$ a^5 + 5a^4b + 10a^3b^2 + 10a^2b^3 + 5ab^4 + b^5 = a^5 + b^5 $$ $$ 5a^4b + 10a^3b^2 + 10a^2b^3 + 5ab^4 = 0 $$ For this sum to be zero, either all of its terms should be zero, or some of them should be positive and some others negative. The former case happens if $ b $ is zero (we noted earlier that $ a $ is always positive, so it can't be zero). The latter case can only happen if $ b $ is negative, because again as we noted earlier, $ a $ is always positive and cannot make negative terms. Let's keep that in mind and continue. $$ 5ab(a^3 + 2a^2b + 2ab^2 + b^3) = 0 $$ The expression in the parentheses looks a bit like $ (a+b)^3 $, but not exactly! Let's add $ a^2b + ab^2 - a^2b - ab^2 $ to it: $$ 5ab[(a+b)^3 - a^2b - ab^2] = 0 $$ $$ 5ab[(a+b)^3 - ab(a+b)] = 0 $$ $$ 5ab(a+b)[(a+b)^2-ab] = 0 $$ And recalling that $ a+b = 2 $: $$ 10ab[4-ab]=0 $$ Now, for this equality to hold, we must have either $ ab = 4 $ or $ ab = 0 $.

But $ ab = 4 $ is not possible, because if it were so, considering that $ a $ is positive, $ b $ should necessarily be positive as well (otherwise $ ab $ would not be positive). And if both $ a $ and $ b $ were positive, then the sum $ 5a^4b + 10a^3b^2 + 10a^2b^3 + 5ab^4 $ would necessarily be positive, whereas it should be zero.

So we are left with the conclusion that $ ab = 0 $. As we mentioned before, $ a $ is positive and not zero. So: $$ b = 0 $$ $$ \sqrt[5]{16-\sqrt x} = 0 $$ $$ 16 - \sqrt x = 0 $$ $$ x = 16^2 = 256 $$

When you went from : $$a^5+b^5+5ab(a^3+b^3)+10a^2b^2(a+b)=32$$ to $$a^5+b^5+5ab\biggl(\frac{32}{(a+b)^2}\biggr)+10a^2b^2(a+b)=32$$ You incorrectly assumed that $$a^3 + b^3 = (a+b)^3 = \biggl(\frac{32}{(a+b)^2}\biggr)$$ Therefore you cannot say that $$a^3 + b^3 = \frac{32}{2^2}$$

You can correct the error that Ryan points out. Instead, note that $$a^3 + b^3 = (a + b)^3 - 3ab(a + b) = 8 - 6ab.$$ So, $$a^5+b^5+5ab(a^3+b^3)+10a^2b^2(a+b)=32$$ becomes $$a^5 + b^5 + 5ab(8 - 6ab) + 20a^2b^2 = 32.$$ Using $a^5 + b^5 = 32$, then expanding and collecting like terms, $$40a^2b^2 - 10ab = 0 \implies ab = 0 \text{ or } ab = 4.$$ So, now we know that $$(r - a)(r - b) = r^2 - (a + b)r + ab$$ equals either $r^2 - 2r + 4$, or $r^2 - 2r$, depending on the value of $ab$. Note that the former has no real roots (though it does have complex solutions, but that's whole other kettle of fish), but the latter does. The two numbers would have to be the roots of the former polynomial: $0$ and $2$. We must therefore have, putting the numbers in ascending order, $$\sqrt[5]{16 - \sqrt{x}} = 0 \quad \sqrt[5]{16 + \sqrt{x}} = 2.$$ It's not hard to see that $\sqrt{x} = 16 \implies x = 256$.

Note: this is the same answer as your method. The reason is, because as it turned out $b = 0$, we actually had $(a + b)^3 = a^3 + b^3$ for that particular $a$ and $b$.