Multivariate polynomial functional equation

As you ask for a hint, here are two hints to help you along. Below is a sketch of a full proof. Let me know when I can 'unhide' all the hidden text to make the answer more legible for future readers.

Hint 1:

For all $a,b\in\Bbb{R}$ find $c\in\Bbb{R}$ such that the second identity becomes of the form $$P(u,-u)+P(v,-v)+P(w,-w)=0.$$

Hint 2:

Deduce that if $\deg{P}>1$ then $P$ is divisible by $X+Y$.

---

Full solution: The polynomials that satisfy the conditions are precisely the polynomials $$(X-2Y)(X+Y)^n,$$ with $n\in\Bbb{N}$. It is not hard to verify that these polynomials satisfy the conditions. Showing that there are no other solutions is more work. Below is a proof is by induction on the degree.

Observation 1: The unique solution $P\in\Bbb{R}[X,Y]$ with $\deg P\leq1$ is $P=X-2Y$.

Proof. There are no constant solutions, and for $n=1$ setting $P=uX+vY$ shows that $$(2u+v)(a+b+c)=0,$$ holds for all $a,b,c\in\Bbb{R}$, and together with $P(1,0)=1$ this implies $P=X-2Y$.$\hspace{10pt}\square$

Observation 2: If $P\in\Bbb{R}[X,Y]$ satisfies the conditions and $\deg P>1$ then $X+Y$ divides $P$.

Proof. Suppose $P\in\Bbb{R}[X,Y]$ satisfies the conditions and $\deg P>1$. Plugging in $c=-a-b$ shows that for all $a,b\in\Bbb{R}$ we have $$0=P(a+b,-a-b)+P(-a,a)+P(-b,b)=((a+b)^n+(-a)^n+(-b)^n)P(1,-1),$$ which implies that $P(1,-1)=0$ because $n>1$, and hence that $P(X,-X)=0$. This means $P$ is divisible $X+Y$.$\hspace{10pt}\square$

Proof of full solution. Now we can prove by induction that for all $n\in\Bbb{N}$ we have

If $P\in\Bbb{R}[X,Y]$ satisfies the conditions and $\deg P=n+1$ then $P=(X-2Y)(X+Y)^n$.

The base case $n=0$ is covered by observation 1. So let $n\in\Bbb{N}$ and suppose that the statement above holds for $n$.

Suppose $P\in\Bbb{R}[X,Y]$ satisfies the conditions and $\deg P=n+2$. Then $P$ is divisible by $X+Y$ by observation 2, which means there exists $Q\in\Bbb{R}[X,Y]$ such that $P=(X+Y)Q$. Then clearly $\deg Q=n+1$, and we verify that $Q$ also satisfies the conditions:

• Because $P$ and $X+Y$ are homogeneous, also $Q$ is homogeneous.
• For all $a,b,c\in\Bbb{R}$ we have \begin{eqnarray*} 0&=&P(a+b,c)+P(b+c,a)+P(c+a,b)\\ &=&(a+b+c)(Q(a+b,c)+Q(b+c,a)+Q(c+a,b)), \end{eqnarray*} which shows that for all $a,b,c\in\Bbb{R}$ with $a+b+c\neq0$ we have $$Q(a+b,c)+Q(b+c,a)+Q(c+a,b)=0.$$ Because $Q$ is a polynomial, it follows that this holds for all $a,b,c\in\Bbb{R}$.
• Clearly $P(1,0)=1$ implies $Q(1,0)=1$.

This shows that $Q$ satisfies the conditions and $\deg Q=n+1$, so by induction hypothesis $$Q=(X-2Y)(X+Y)^n \qquad\text{ and hence }\qquad P=(X-2Y)(X+Y)^{n+1},$$ which completes the proof by induction.

From $P_n(t x, t y) = t^nP_n(x,y)$ we conclude that $P_n(x,y) = \sum_{k=0}^n a_k x^k y^{n-k}$

now considering

$$ P_n(2x,y) + 2P_n(x+y,x) = 0\Rightarrow \sum_{k=0}^n 2^k a_k x^k y^{n-k}+2\sum_{k=0}^na_k\left(x+y\right)^k x^{n-k} $$

or

$$ \sum_{k=0}^n 2^k a_k x^k y^{n-k}+2\sum_{k=0}^na_k\left(\sum_{j=0}^k C_j^k x^j y^{k-j}\right) x^{n-k} = \sum_{k=0}^n 2^k a_k x^k y^{n-k}+2\sum_{k=0}^na_k\left(\sum_{j=0}^k C_j^k x^{n-(k-j)} y^{k-j}\right)=0 $$

or making $\nu=k-j$

$$ \sum_{k=0}^n 2^k a_k x^k y^{n-k}+2\sum_{k=0}^na_k\left(\sum_{\nu=k}^0 C_{k-\nu}^k x^{n-\nu} y^{\nu}\right)=0 $$

so to guarantee the polynomial identity we conclude

$$ 2^{n-k}a_{n-k}+2 \sum_{j=0}^{n-k}a_{j+k}C_j^{j+k}= 0 $$

with $a_n = 1$ due to $P_n(1,0) = 1$

This is a almost triangular linear system. For $n = 3$ we have

$$ \begin{cases} 2^3a_3+2\left(a_0C_0^0+a_1C_1^1+a_2C_2^2+a_3C_3^3\right)=0\\ 2^2a_2+2\left(a_1C_0^1+a_2C_1^2+a_3C_2^3\right)=0\\ 2^1a_1+2\left(a_2C_0^2+a_3C_1^3\right)=0\\ 2^0a_0+2a_3C_0^3 = 0 \end{cases} $$

or

$$ \left\{ \begin{array}{rcl} 2 a_0+2 a_1+2 a_2+10 a_3& = &0 \\ 2 a_1+8 a_2+6 a_3&=&0 \\ 2 a_1+2 a_2+6 a_3&=&0 \\ a_0+2 a_3&=&0 \\ \end{array} \right. $$

and solving we have

$$ a_0 = -2, a_1 = -3, a_2 = 0, a_3 = 1 $$

or

$$ P_3(x,y) = (x-2y)(x+y)^2 $$

and for $n$

$$ P_n(x,y) = (x-2y)(x+y)^{n-1} $$

as can be easily verified.