A problem related to $2x^3-3x^2-x+\frac{3}{2}=0$

You have the right idea about exploiting some sort of symmetry in $f(f(x))$, but the integral doesn't evaluate zero.

Anyway, for all $y \in \mathbb{R}$, we have $$f(1-y) = 2(1-y)^3-3(1-y)^2-(1-y)+\dfrac{3}{2} = -2y^3+3y^2+y-\dfrac{1}{2} = 1-f(y).$$

Hence, for all $x \in \mathbb{R}$, we have$$f(f(1-x)) = f(1-f(x)) = 1-f(f(x)),$$ where we have used the above property for $y = x$, and then for $y = f(x)$.

Now, let $I = \displaystyle\int_{1/8}^{7/8}f(f(x))\,dx$. Replace $x$ with $1-x$ to get $I= \displaystyle\int_{1/8}^{7/8}f(f(1-x))\,dx$.

By adding these expressions for $I$ together, we get $$2I = \int_{1/8}^{7/8}f(f(x))+f(f(1-x))\,dx = \int_{1/8}^{7/8}1\,dx = \dfrac{3}{4},$$ and thus, $I = \dfrac{3}{8} \neq \dfrac{3}{4}$.