Probability that rectangle is inside a circle

For a given position of $X$, say at coordinates $(\cos\theta,\sin\theta)$ in the first quadrant, the area of the rectangle is indeed

$$A=4\sin\theta\cos\theta.$$

As $Y$ is drawn uniformly in the disk, the probability to fall in $A$ is $A/\pi$.

Then as $X$ is drawn uniformly on the circumference and independently of $Y$, you take the expectation on the first quadrant,

$$P=\frac4\pi\frac 2\pi\int_{\theta=0}^{\pi/2}\sin\theta\cos\theta\,d\theta=\frac4{\pi^2}.$$

Let the coordinates of the first point be $(x_0,y_0)$ and those of the second point be $(x_1, y_1)$. Then the coordinates of the other two points on the rectangle are $(x_0, y_1)$ and $(x_1, y_0)$.

Let us suppose the first point must fall on the quarter-circle in the first quadrant. This does not affect the probability at all because of the symmetry of the circle. Then we have $y_0=\sqrt{1-x_0^2}$. In order to have the rectangle fully inside of the circle, all of the following must be true: $$x_0^2+y_1^2 \le 1$$ $$x_1^2+y_0^2 \le 1$$ We can substitute into the second requirement $y_0=\sqrt{1-x^2}$ to get $$x_1^2+1-x_0^2 \le 1$$ $$x_1^2-x_0^2 \le 0$$ $$x_0^2 \ge x_1^2$$ $$|x_0| \ge |x_1|$$ Similarly, the first requirement can be replaced by $$|y_0| \ge |y_1|$$ and our requirements are $$|x_0| \ge |x_1|$$ $$|y_0| \ge |y_1|$$ Try and visualize this:

Let $F$ be the event in which the rectangle fits, and $P(F|x_0)$ be the probability that the rectangle falls entirely inside of the circle given the value of $x_0$. All points $(x_1, y_1)$ for which the rectangle would fit in the circle are inside of the green rectangle shown with width $2x_0$ and height $2y_0$. This is now a spatial probability problem. Given $x_0$, we can now say that the probability of the rectangle fitting is the probability that $(x_1, y_1)$ lands in that green rectangle. This probability is given by the area of the rectangle over the area of the circle: $$P(F|x_0)=\frac{4x_0y_0}{\pi}$$ $$P(F|x_0)=\frac{4x_0\sqrt{1-x_0^2}}{\pi}$$ We will have to integrate over this to find the total probability. Our first instinct would be to try $$\int_{-1}^{1} P(F|x_0)dx_0$$ but this counts the points on the circumference of the circle unevenly, and not with equal probability. Instead, we should try $$\int_{0}^{\frac{\pi}{2}} P(F|\cos x_0)dx_0$$ Because this sweeps the circumference of the quarter-circle counting all points equally. Thus the probability should be $$\frac{2}{\pi}\int_{0}^{\frac{\pi}{2}} \frac{4\cos x_0\sqrt{1-\cos^2 x_0}}{\pi}dx_0$$ We can simplify this to $$\frac{2}{\pi}\int_{0}^{\frac{\pi}{2}} \frac{4\cos x_0|\sin x_0|}{\pi}dx_0$$ This value turns out to be about $0.405$.

Due to symmetry, without loss of generality the point on circumference is uniformly selected to by in the first quadrant, that is it makes an angle $\theta$, uniform in $(0,\pi/2)$.

The $x$-component of this point is then $X_0= \cos \theta$ and the $y$-component is $Y_0=\sin \theta$.

As for the second point, write $X$ and $Y$ for its $x$- and $y$- components, respectively. In order for the rectangle to be contained in the circle, we must have $ (X,Y) \in [-X_0,X_0]\times [-Y_0,Y_0]$.

Because $(X,Y)$ is uniform in the disk, and is independent of $\theta$, conditioning on $\theta$, the probability of $(X,Y)$ landing in this rectangle is the area of this rectangle divided by $\pi$. That is,

$$ \frac{ 2 \cos \theta \times 2 \sin \theta}{\pi} = \frac{ 2 \sin (2\theta) }{\pi}.$$

Integrate with respect to $\theta$ to get the answer:

$$p= \frac{2}{\pi} \int_0^{\pi/2} \frac{ 2\sin (2\theta)}{\pi}d\theta= \frac 2{\pi^2} \int_0^{\pi}\sin(u)du =\frac{4}{\pi^2}. $$