An olympic mathematics problem regarding Cauchy-Schwarz

Let $x^2+y^2=m$.

Thus, $$(m+6)m\geq A(m-2)$$ or $$m^2+(6-A)m+2A\geq0,$$ for which we need $$(6-A)^2-8A\leq0,$$ which gives $2\leq A\leq18$.

For $A=18$ we get $m=6$ or $x^2+y^2=6$, which says that the equality occurs.

Thus, $18$ is our answer.

Done!

Note that your first expression:

$$(x+y)^2-8\ge A\cdot (x-y)^2$$

must be: $$(x+y)^4+2(x+y)^2-8\ge A\cdot (x-y)^2.$$ Alternative solution: You can also denote: $$(x+y)^2=z; \ \ \ (x-y)^2=(x+y)^2-4xy=z-4.$$ Then: $$(z+4)(z-2)\ge A\cdot (z-4) \Rightarrow$$ $$z^2+(2-A)z+4A-8\ge 0$$ This inequality is true for all $z$ (all $x,y$) if: $$(2-A)^2-4(4A-8)\le 0 \Rightarrow (A-2)(A-18)\le 0 \Rightarrow 2\le A\le 18.$$ Hence the maximum value of $A$ is $18$. P.S. Was there any other condition, like $x\ne y$? Otherwise if $x=y=1$, the left hand side is $16$, while the right hand side is $0$, implying $A$ can be any number, hence no max.