Converting cartesian to polar double integral
The integration area is shaded and their polar boundaries are marked in the graph. Moreover, in polar coordinates $$x^2y dy dx= r^4\cos^2\theta\sin\theta drd\theta=f(r,\theta)drd\theta$$ and the integral is given as
$$\int_{\frac\pi4}^0 \int_3^{\frac3{\cos\theta}} f(r,\theta) drd\theta +\int_{\frac\pi4}^{\pi} \int_0^{3} f(r,\theta)drd\theta +\int_{\pi}^{\frac{5\pi}4} \int_0^{-\frac3{\cos\theta}} f(r,\theta)drd\theta = - \frac{81}5 $$
Remark that $x\ge \sqrt{9-x^2}$ when $\frac 3{\sqrt 2}\le x\le 3$.
By a well-known theorem (see, for instance, [Fich, 596]), the given double integral is equal to $\int_{S} x^2y dx dy-\int_{S’} x^2y dx dy $, where $$S=\left\{(x,y)\in\Bbb R^2: -3\le x\le 3,\, x\le y\le \sqrt{9-x^2}\right\}$$ and $$S’=\left\{(x,y)\in\Bbb R^2: \frac 3{\sqrt 2}\le x\le 3,\, \sqrt{9-x^2}\le y\le x\right\}.$$
The correspondence between Cartesian coordinates $(x,y)\in\Bbb R^2$ and polar coordinates $(r,\varphi)\in\Bbb R^+\times [0,2\pi)$ is $x=r\cos\varphi $, $y=r\sin\varphi$, $$S=\{(r,\varphi): 0\le r\le 3,\, \pi/4\le \varphi\le \pi\}\cup$$ $$\{(r,\varphi): 0\le r\le - 3/\cos\varphi,\, \pi\le \varphi\le 5\pi/4\},$$ $$S’=\{(r,\varphi): 3\le r\le 3/\cos\varphi,\, 0\le \varphi\le \pi/4\},$$
and $dS=r dr d\varphi$. Thus
$$\int_S x^2y dx dy=\int_{\pi/4}^{\pi}\int_0^3 r^4\cos^2\varphi\sin\varphi dr d\varphi+$$ $$\int_{\pi}^{5\pi/4}\int_0^{-3/\cos\varphi} r^4\cos^2\varphi\sin\varphi dr d\varphi- \int_{0}^{\pi/4}\int_0^{3/\cos\varphi} r^4\cos^2\varphi\sin\varphi dr d\varphi.$$
References
[Fich] Grigoriy Fichtenholz, Differential and Integral Calculus, v. III, 4-th edition, Moscow: Nauka, 1966, (in Russian).