Show that $\int_{0}^{1}\{P(x)\}^{2}\,dx = (n + 1)^{2}\left(\int_{0}^{1}P(x)\,dx\right)^{2}$

Since, $\displaystyle \int_0^1 x^kP(x)\,dx=\frac{a_0}{n+k+1}+\frac{a_1}{n+k}+\ldots+\frac{a_n}{k+1}=0$, for each $k=1,2,\ldots,n$

Then $f(x)=\dfrac{a_0}{n+x+1}+\dfrac{a_1}{n+x}+\ldots+\dfrac{a_n}{x+1}=\dfrac{Q(x)}{(n+x+1)\ldots(x+1)}$ (where, Q is a polynomial of degree atmost n), has $n$ zeros $x=1,2,\ldots,n$.

Thus, $Q(x)=c(x-1)(x-2)\ldots(x-n)$, for some constant $c$.

Also, $(x+1)f(x)=\dfrac{a_0(x+1)}{n+x+1}+\dfrac{a_1(x+1)}{n+x}+\ldots+a_n=\dfrac{Q(x)}{(n+x+1)\ldots(x+2)}$

Setting, $x=-1$ in the above expression $a_n = \dfrac{Q(-1)}{n!}=\dfrac{c(-1)^n(n+1)!}{n!}=c(-1)^n(n+1)$

and, setting $x=0$ we have $\displaystyle \int_0^1 P(x)\,dx = \dfrac{a_0}{n+1}+\ldots+a_n=\dfrac{Q(0)}{(n+1)!}=\dfrac{c(-1)^n}{n+1}$.

Thus $a_n=\displaystyle (n+1)^2\int_0^1 P(x)\,dx$, implying $\displaystyle \int_{0}^{1}\{P(x)\}^{2}\,dx = (n + 1)^{2}\left(\int_{0}^{1}P(x)\,dx\right)^{2}$.

Here is yet another solution; it is a little long, but is in fact very simple and non-technical.

Outline of the proof

The crucial observation is that the polynomial $Q(x):=((1-x)P(1-x))'$ also has the property in question: integrating by parts, changing the variable, and expanding the binomial, for any $1\le k\le n$ we get $$ \int_0^1 x^kQ(x)\,dx = -k\int_0^1 (1-x)P(1-x)x^{k-1}\,dx = -k\int_0^1 P(y) (1-y)^{k-1} y\,dy = 0. $$ We will see that this forces $Q(x)=cP(x)$ with an appropriate coefficient $c$. This is a very strong relation; the result will follow by integrating it against $x^{n+1}$.

Detailed argument

With $Q(x)$ defined above, observing that $$ \int_0^1 Q(x)\,dx = (1-x)P(1-x)\Big\vert_0^1 = -P(1), $$ and letting $$ I:=\int_0^1 P(x)\,dx, $$ we conclude that $$ \int_0^1 (IQ(x)+P(1)P(x))x^k\,dx = 0,\qquad 0\le k\le n. $$ As a result, for any polynomial $T$ of degree at most $n$, we have $$ \int_0^1 (IQ(x)+P(1)P(x))T(x)\,dx = 0. $$ Applying this with $T(x)=IQ(x)+P(1)P(x)$, we conclude that $$ \int_0^1 (IQ(x)+P(1)P(x))^2\,dx = 0. $$ Therefore, $IQ(x)+P(1)P(x)$ is the zero polynomial: $$ IQ(x) = -P(1)P(x). $$ Recalling the definition of $Q(x)$ and switching from $x$ to $1-x$, we get $$ I(xP(x))' = P(1)P(1-x). \tag{1} $$ Expanding, $I(xP'(x)+P(x))=P(1)P(1-x)$, and substituting $x=0$ gives $$ IP(0)=(P(1))^2. \tag{2} $$ Write $$ J := \int_0^1 x^{n+1} P(x)\, dx. $$ Notice that $J\ne 0$, as otherwise we would have $$ \int_0^1 x(P(x))^2\,dx = 0. $$

To complete the proof, we integrate (1) against $x^{n+1}$. In the LHS we get $$ I\int_0^1 (xP(x))'x^{n+1}\,dx = I x^{n+2} P(x)\Big\vert_0^1 - (n+1)I \int_0^1 x^{n+1}P(x)\,dx = IP(1)-(n+1)IJ, $$ in the RHS $$ P(1) \int_0^1 x^{n+1}P(1-x)\,dx = P(1) \int_0^1 (1-x)^{n+1}P(x)\,dx = P(1) (I+(-1)^{n+1}J) $$ (for the last step, we expand the binomial and integrate termwise, observing that all intermediate terms vanish). Hence, $$ -(n+1)IJ = (-1)^{n+1} P(1)J; $$ therefore, $$ P(1) = (-1)^n(n+1)I. $$ Squaring and using (2), $$ IP(0) = (n+1)^2 I^2, $$ and the result follows readily.

Note that $P(x)$ is a linear combination of $1$ and $x, x^2$, $\ldots$, $x^n$ and is perpendicular to the subspace generated by $x, x^2$, $\ldots$, $x^n$. Therefore $P$ gives the direction of the vertical component of $1$.

Now, we must show that $$\frac{\langle 1, P\rangle ^2 }{\|P\|^2} = \frac{1}{(n+1)^2}$$ that is, $d(1, \langle x,x^2, \ldots, x^n\rangle)^2=\frac{1}{(n+1)^2}$, in words, the distance from $1$ to the subspace generated by $x,x^2, \ldots,x^n$ equals $\frac{1}{n+1}$. Now we have $$d^2 = \frac{G(1,x,\ldots, x^n)}{G(x,x^2, \ldots, x^n)}$$ where the $G$'s are Gram determinants that can be calculated with the formula for Cauchy determinants.

$\bf{Added:}$ One sees easily that the Gram matrix for the system of functions $(x^{\alpha_i})$ is the symmetric Cauchy matrix $(\frac{1}{\alpha_i + \alpha_j+1})$, with determinant $V(\alpha_i)\cdot\prod_{i,j} \frac{1}{\alpha_i + \alpha_j+1}$, where $V$ is the Vandermonde determinant. We get for the distance squared $d^2$ from $x^{\alpha}$ to the span of $(x^{\alpha_i})_i$

$$d^2=\frac{G(x^{\alpha},x^{\alpha_i})}{G(x^{\alpha_i})}=\frac{1}{2\alpha+1}\prod_i \frac{(\alpha_i-\alpha)^2}{(\alpha_i+\alpha+1)^2}$$ For $\alpha=0$ and $\alpha_i=i$, with $1\le i \le n$, we get the desired value $\frac{(n!)^2}{(n+1)!^2} = \frac{1}{(n+1)^2}$.