Integration of $ \int x^{2} \sqrt{2x-6} dx $

Your result is correct as soon as you change a sign (typo?): the result can be written as $$A(2x-6)^{7/2} + B(2x-6)^{5/2} + C(2x-6)^{3/2} + D$$

Note that $$\frac{2x}{15}(2x-6)^{5/2}=\frac{2x-6+6}{15}(2x-6)^{5/2}=\frac{1}{15}(2x-6)^{7/2}+\frac{2}{5}(2x-6)^{5/2}.$$ Similarly, you can write $\frac{x^2}{3}(2x−6)^{3/2}$ as a linear combination of $(2x-6)^{7/2}$, $(2x-6)^{5/2}$, and $(2x-6)^{3/2}$: $$\frac{x^2}{3}(2x−6)^{3/2}=\frac{(2x-6)^2/4+3(2x-6)+9}{3}(2x−6)^{3/2} \\=\frac{1}{12}(2x−6)^{7/2}+(2x−6)^{5/2}+3(2x−6)^{3/2}.$$ Hence, your result can be written as $$\underbrace{\left(\frac{1}{12}-\frac{1}{15}+\frac{2}{105}\right)}_{1/28}(2x-6)^{7/2} + \underbrace{\left(1-\frac{2}{5}\right)}_{3/5}(2x-6)^{5/2} + 3(2x-6)^{3/2} + C.$$ However, the final result in that form can be obtained more easily by applying the substitution $t=\sqrt{2x-6}$. Then $x=\frac{t^2}{2}+3$, $dx=tdt$ and $$\int x^{2} \sqrt{2x-6} dx =\frac{1}{4}\int (t^2+6)^2 t (t dt)=\int \left(\frac{t^6}{4}+3t^4+9t^2\right) dt\\ =\frac{t^7}{28}+\frac{3t^5}{5}+3t^3+C =\frac{1}{28}(2x-6)^{7/2} + \frac{3}{5}(2x-6)^{5/2} + 3(2x-6)^{3/2} + C.$$

Hint:Substituting $$t=\sqrt{2x-6}$$ then we get $$x=\frac{t^2+6}{2}$$ then we get $$dx=tdt$$