Prove that $\int_{0}^{1}f(x)\arctan x dx=\frac{π}{8}\int_{0}^{1}f(x)dx$

Assuming that you meant:$$2f\left(\frac{1-x}{1+x}\right)=\color{orange}{f(x)}(1+x)^2$$ We can start with the given integral and let $x=\frac{1-t}{1+t}\Rightarrow dx=-\frac{2}{(1+t)^2}dt$ $$I=\int_{0}^{1}f(x)\arctan xdx=\int_0^1 \color{blue}{f\left(\frac{1-t}{1+t}\right)}\color{red}{\arctan \left(\frac{1-t}{1+t}\right)}\frac{\color{blue}{2}}{(1+t)^2}dt$$ $$=\int_0^1 \color{blue}{f(t)}\left(\color{red}{\frac{\pi}{4}-\arctan t}\right)dt\overset{t=x}=\frac{\pi}{4}\int_0^1 f(x)dx-I$$ $$\Rightarrow 2I=\frac{\pi}{4}\int_0^1 f(x)dx \Rightarrow I=\frac{\pi}{8}\int_0^1 f(x)dx$$

Your approach is correct but, as Zacky pointed out, a factor $f(x)$ is missing in the stated property of $f$.

Let $x=\frac{1-t}{1+t}$, then $dx=\frac{-2dt}{(1+t^2)}$, $$\arctan(x)=\arctan(1)-\arctan(t),$$ and, by using the given property $2f(x)=f(t)(1+t)^2$ (with $f$), we get $$\int_{0}^{1}f(x)\arctan(x)\,dx=\int_{0}^{1}f(t)(\arctan(1)-\arctan(t))dt\\ = \frac{\pi}{4}\int_{0}^{1}f(t)\,dt-\int_{0}^{1}f(t)\arctan(t))\,dt$$ which implies that $$\int_{0}^{1}f(x)\arctan(x)\,dx=\frac{\pi}{8}\int_{0}^{1}f(t)\,dt.$$