How prove this inequality $\left(\int_{0}^{1}f(x)dx\right)^2\le\frac{1}{12}\int_{0}^{1}|f'(x)|^2dx$

Using integration by parts and $f(0)=f(1)=0$, we have: $$ \int_0^y f(x)dx = \int_0^y(y-x)f'(x) dx$$ and $$ \int_y^1 f(x)dx = \int_y^1(y-x)f'(x) dx$$ for all $y\in [0,1]$.

Taking $y=1/2$ and using Cauchy-Schwarz inequality we get: $$ \left(\int_0^{1/2} f(x)dx\right)^2 = \left(\int_0^{1/2}({1/2}-x)f'(x) dx\right)^2 $$ $$\leq \int_0^{1/2}({1/2}-x)^2 dx \int_0^{1/2}(f'(x))^2 dx = \frac{1}{24} \int_0^{1/2}(f'(x))^2 dx$$ and $$ \left(\int_{1/2}^1 f(x)dx\right)^2 = \left(\int_{1/2}^1({1/2}-x)f'(x) dx\right)^2 $$ $$\leq \int_{1/2}^1({1/2}-x)^2 dx \int_{1/2}^1(f'(x))^2 dx = \frac{1}{24} \int_{1/2}^1(f'(x))^2 dx.$$ So:

$$\frac{1}{2} \left(\int_0^{1} f(x)dx\right)^2\leq \left(\int_0^{1/2} f(x)dx\right)^2 + \left(\int_{1/2}^1 f(x)dx\right)^2 $$ $$\leq \frac{1}{24} \int_0^{1/2}(f'(x))^2 dx+\frac{1}{24} \int_{1/2}^1(f'(x))^2 dx = \frac{1}{24} \int_0^{1}(f'(x))^2 dx.$$

In fact $$ \left(\int_0^1(2x-1)f'(x)dx\right)^2\le\int_0^1(2x-1)^2dx\int_0^1(f'(x))^2dx. \tag1 $$ But $$ \int_0^1(2x-1)f'(x)dx =-2\int_0^1f(x)dx$$ by using the Integration-by-Parts formula and $$ \int_0^1(2x-1)^2dx=\frac{1}{3}. $$ Putting these two into (1), you can get the desired inequality.

Using integration by parts we find that $$\int_0^1 f(x)dx = - \int_0^1 xf'(x)dx$$

Then Cauchy-Schwarz will give $$\left( \int_0^1 x f'(x) dx \right)^2 \le \left( \int_0^1 |x f'(x)| dx \right)^2 \le \int_0^1 x^2 dx \cdot \int_0^1 |f'(x)|^2 dx$$