How to find the derivative of this function?

Note that: $$\begin{align} f(x) &= \int_{a(x)}^{b(x)} g(t)\,dt = G\bigl(b(x)\bigr)-G\bigl(a(x)\bigr) \\ f'(x) &= g\bigl(b(x)\bigr)\cdot b'(x)-g\bigl(a(x)\bigr)\cdot a'(x) \end{align}$$ Hence: $$\begin{align} f(x) &= \int_0^{\cos x}t^2 \, dt=G(\cos x)-G(0) \\ f'(x)&=(\cos x)^2\cdot (\cos x)'-(0)^2\cdot 0' \\ &=\cos^2x \cdot (-\sin x) \\ &=-\cos^2x\cdot \sin x \end{align}$$

If $$I=\int_a^{g(x)} f(t) \, dt$$ where $a$ is a constant, the fundamental theorem of calculus gives $$\frac{dI}{dx}=f(g(x))\, g'(x)$$ In you case where $g(x)=\cos(x)$ and $f(t)=t^2$, this leads to $$\frac{dI}{dx}=-\sin(x)\, \cos(x)^2$$

\begin{align} y & = \int_0^{\cos x} t^2\, dt \\[10pt] y & = \int_0^u t^2 \, dt & u & = \cos x \\[10pt] \frac{dy}{du} & = u^2 & \frac{du}{dx} & = -\sin x \\[10pt] \frac{dy}{dx} & = \frac{dy}{du} \cdot \frac{du}{dx} \\[10pt] & = u^2 \cdot(-\sin x) \\[10pt] & = (\cos x)^2(-\sin x). \end{align} So your result is correct.

However, in order to know whether it was correctly graded, we would have to see exactly how the problem was stated. For example, if there was some standing instruction to the class to mention the chain rule explicitly each time it is used, then omitting to do that could count against you, and there are lots of other things like that that we might not know about without further information.