Taking the derivative of $y = (\frac{x}{1-\sqrt{x}})^3$ using the chain rule

A quotient can be expressed as a product of the function to the power of $-1$, e.g., $\frac{f(x)}{g(x)} = f(x)g^{-1}(x)$. You can then apply your "differentiating short-cut" using $n = -1$. In particular, this gives $h(x) = g^{-1}(x) \implies h'(x) = -g^{-2}(x)$, which is the same as the quotient rule.

As for the specific question,

$$\begin{equation}\begin{aligned} f(x) & = x^3\left(1-\sqrt{x}\right)^{-3} \\ f'(x) & = 3x^2\left(1-\sqrt{x}\right)^{-3} + x^3(-3)\left(1-\sqrt{x}\right)^{-4}\left(-\frac{1}{2}x^{-1/2}\right) \\ & = 3x^2\left(1-\sqrt{x}\right)\left(1-\sqrt{x}\right)^{-4} + \frac{3}{2}x^{5/2}\left(1-\sqrt{x}\right)^{-4} \\ & = \frac{3x^2\left(1 - \sqrt{x} + \frac{1}{2}\sqrt{x}\right)}{\left(1-\sqrt{x}\right)^{4}} \\ & = \frac{3x^2\left(1 - \frac{1}{2}\sqrt{x}\right)}{\left(1-\sqrt{x}\right)^{4}} \\ & = \frac{3x^2\left(2 - \sqrt{x}\right)}{2\left(1-\sqrt{x}\right)^{4}} \\ \end{aligned}\end{equation}\tag{1}\label{eq1}$$

This basically agrees with what you got. As for the maths book solution, I don't see how they got that.

Solving using only the chain rule

From the chain rule, $$y = \left(\frac{x}{1-\sqrt{x}}\right)^3\implies\frac{dy}{dx}=3\left(\frac x{1-\sqrt x}\right)^2\cdot\color{red}{\frac d{dx}\left(\frac{x}{1-\sqrt{x}}\right)}\tag1.$$ Now \begin{align}\color{red}{\frac d{dx}\left(\frac{x}{1-\sqrt{x}}\right)}&=\frac d{dx}\left(\left(\frac{1-\sqrt x}{x}\right)^{-1}\right)=-\left(\frac{1-\sqrt x}{x}\right)^{-2}\cdot\color{blue}{\frac d{dx}\left(\frac{1-\sqrt x}{x}\right)}\tag2\end{align} and $$\color{blue}{\frac d{dx}\left(\frac{1-\sqrt x}{x}\right)}=\frac d{dx}\left(x^{-1}-x^{-1/2}\right)=-\frac1{x^2}+\frac1{2x\sqrt x}\tag3$$ so \begin{align}\frac{dy}{dx}&=3\left(\frac x{1-\sqrt x}\right)^2\cdot\left(-\left(\frac{1-\sqrt x}{x}\right)^{-2}\right)\cdot\left(-\frac1{x^2}+\frac1{2x\sqrt x}\right)\\&=-3\left(\frac x{1-\sqrt x}\right)^2\cdot\left(\frac x{1-\sqrt x}\right)^2\cdot\left(-\frac1{x^2}+\frac1{2x\sqrt x}\right)\\&=\frac{3x^4}{(1-\sqrt x)^4}\left(\frac1{x^2}-\frac1{2x\sqrt x}\right)\\\vphantom{2cm}\\\implies\frac{dy}{dx}&=\frac{3x^2-\frac32x^{5/2}}{(1-\sqrt x)^4}\tag4\end{align} which is what you have. As for the book's answer, it's wrong, as can be seen here.

We have $$f(x)=\frac{x^3}{(1-\sqrt{x})^3}=x^3\cdot (1-\sqrt{x})^{-3}$$ So we get $$f'(x)=3x^2\cdot (1-\sqrt{x})^{-3}+x^3\cdot (-3)(1-\sqrt{x})^{-4}\cdot (-1)\frac{1}{2}x^{-1/2}$$ It can be simplified to $$f'(x)=-\frac{3 \left(\sqrt{x}-2\right) x^2}{2 \left(\sqrt{x}-1\right)^4}$$