"Proof" all integrals are $0$
In integration theory when you want to use a u-subsitution you have to use a function which is a $C^1$-diffeomorphism. Otherwise you could end up with funny things.
Edit Actually a diffeomorphism is not necessary as pointed out by @zhw. To apply the substitution $x=\psi(u)$ from $u \in [c,d]$ to $x \in [a,b]$ you "only" need to have $\psi$ to be $C^1$ and $\psi([c,d])$ included in the domain of $f$, with $\psi(c)=a$ and $\psi(d)=b$.
For example if you want to compute $\int_{-1}^1 x^2\mathrm{d}x$, using a u-substitution $u=x^2$ will change the bounds to $1$ and $1$ and give 0, which is not true. This is because the function $x \in [-1,1] \mapsto x^2$ is not a bijection.
In your case your function $x \in [a, b] \mapsto x(x-a-b)$ has the following graph for $a=-1$ and $b=1$. You clearly see that this function is not 1-1. 
The fact that your substitution is not 1-1 is clearly due to the square. Let $\phi: x \in [a,b] \mapsto x(x-a-b)$. The function $\phi$ reaches its minimum in $c=(a+b)/2$ and is bijective on $[a, c]$ and on $[c, b]$. If you split your integral into 2 integrals on these intervals you can use your substitution.
You would get $\int_a^b f(x) \mathrm{d}x=\int_a^c f(x)\mathrm{d}x + \int_b^c f(x)\mathrm{d}x$
Now use $u=\phi(x)=x(x-a-b)$. You have therefore $x=\dfrac{a+b-\sqrt{(a+b)^2+4u}}{2}=\psi_1(u)$ on $[a,c]$ and $x=\dfrac{a+b+\sqrt{(a+b)^2+4u}}{2}=\psi_2(u)$ on $[c,b]$
This would give $$\int_a^b f(x) \mathrm{d}x=\int_{-ab}^{-(a+b)^2/4}f(\psi_1(u))\psi_1'(u)\mathrm{d}u + \int_{(a+b)^2/4}^{-ab}f(\psi_2(u))\psi_2'(u)\mathrm{d}u$$
And you see that you cannot "merge" these two integrals because their content is now different.
Of course this u-substitution does not seem to help a lot to compute the integral, particularly because $f$ is general. But if $f$ has a particular form, maybe this could help.
The only way you could do such a substitution to compute $\int_a^b f(x)\,dx$ is if there is some function $g$ such that: $$f(x)\,dx=g(u)\,du=g(x(x-a-b))(2x-a-b)\,dx$$
So if $f(x)=g(x(x-a-b))(2x-a-b)$, then let $h(x)=f\left(\frac{a+b}{2}+x\right)$, then $h$ is defined on $\left[-\frac{b-a}{2},\frac{b-a}{2}\right]$, and $h(-x)=-h(x)$. So you get that if $f(x)$ is of this form, then:
$$\int_{a}^{b} f(x)\,dx \int_{-\frac{b-a}{2}}^{\frac{b-a}{2}}h(t)\,dt= 0$$ since $h$ is an odd function.
So when the substitution can be done, you do get $0$. The problem is that, for the general function $f$, you can't find a function $g$ - you end up, instead, with a multi-valued function $g$.
Similarly, if you use $u=\sin x$, to calculate $\int_{0}^{\pi} f(x)\,dx$ then there should be a $g$ such that $f(x)=g(\sin x)\cos x$. And you can show that $$\int_{0}^{\pi}g(\sin x)\cos x\,dx = 0.$$ This is because $f(\pi-x)=-f(x)$.
More generally, if $g(x)$ is a continuous function and $u(x)$ a differentiable function and $u(a)=u(b)$ then any: $$\int_{a}^{b} g(u(x))u'(x)\,dx = 0$$
Proof: Let $G(x)=\int_{a}^{x} g(t)\,dt$. Then $g(u(x))u'(x)=\frac{d}{dx}(G(u(x))$ by the chain rule and the property that $G'=g$. So, by the Fundamental Theorem of Calculus, $$\int_{a}^{b} g(u(x))u'(x)\,dx = G(u(x))\big\vert_{a}^{b}=0$$
This is essentially the long form of substitution - substitution is doing the reverse of the chain rule.