A square appears randomly within a square of ten time its area. What is probability that the smaller square contains the larger square's center?

Without Rotation

Suppose a square of side $s\le\frac12$ is placed randomly inside a unit square, without rotation. The probability that the $x$-coordinate of the center of the unit square is within the $x$-span of the smaller square is $\frac{s}{1-s}$. That is, when covering the center of the unit square, the left side of the smaller square can be $s$ to $0$ units to the left of the center of the unit square, while, without restriction, the left side of the smaller square can go from $1$ to $s$ units to the left of the right side of the unit square. The same is true for the $y$-coordinate. So the probability that the center of the larger square is within the smaller square is $$ \left(\frac{s}{1-s}\right)^2\tag1 $$ In your question, $s=\frac1{\sqrt{10}}$, which gives a probability of $$ \bbox[5px,border:2px solid #C0A000]{\frac1{11-2\sqrt{10}}\approx0.213883399}\tag2 $$

---

With Rotation

Geometry and Probability

The smaller square rotated by $0\le\theta\le\frac\pi4$ will span a width and height of $$ d=s(\cos(\theta)+\sin(\theta))\tag3 $$ Note that $(3)$ says that $d:\left[0,\frac\pi4\right]\mapsto\left[s,s\sqrt2\right]$ monotonically.

If the smaller square is to stay inside the unit square, its center is constrained to a non-rotated square of side $1-d$ concentric with the unit square (shown below as a red square). If the center of the unit square is to be inside the smaller square, the center of the smaller square needs to be inside a square with the same size and rotation, but concentric with the unit square (shown below as a green square).

Thus, the locus of the center of the smaller square so that the smaller square is contained in the unit square, and so that the smaller square contains the center of the unit square is the intersection of the red and green squares (shown below in purple).

For a given size and rotation of the smaller square, given that the smaller square is contained in the unit square, the probability that the center of the unit square is inside the smaller square is the ratio of the area of the purple region divided by the area of the red square.

However, to get the proper weighting of the probabilities for the different angles, we need to weight the probabilities by the area of the red square since that is the probability that the rotated square is contained in the unit square. This means we compute the integral of the area of the purple region and divide that by the integral of the area of the red region.

Computation of the Areas

The area of the red square is $$ \begin{align} \text{area}_\text{red} &=(1-d)^2\tag{4a}\\[6pt] &=(1-s(\sin(\theta)+\cos(\theta)))^2\tag{4b}\\[6pt] &=1-2s(\sin(\theta)+\cos(\theta))+s^2(1+2\sin(\theta)\cos(\theta))\tag{4c}\\ \int_0^{\pi/4}\text{area}_\text{red}\,\mathrm{d}\theta &=\frac\pi4-2s+\left(\frac\pi4+\frac12\right)s^2\tag{4d} \end{align} $$ If $d\le\frac12$, then, because $d\lt1-d$, the green square is completely contained within the red square and so the area of the purple region is $$ \begin{align} \text{area}_\text{purple} &=s^2\tag{5a}\\ \int\text{area}_\text{purple}\,\mathrm{d}\theta &=\theta s^2\tag{5b}\\ &=a_1(s,\theta)\tag{5c} \end{align} $$ We leave $\text{(5b)}$ indefinite since we may need to restrict $\theta$ due to the condition $d\le\frac12$.

For $d\gt\frac12$, we will refer to the following diagram:

Since the side length of the red square $1-d$ and the height of the rotated green square is $d$, the four corners of the green square that extend beyond the red square have altitude $\frac{d-(1-d)}2=d-\frac12$ and base $\frac{d-1/2}{\sin(\theta)\cos(\theta)}$. If the bases of these triangles are smaller than the sides of the red square, then $\frac{d-1/2}{\sin(\theta)\cos(\theta)}\lt1-d$, which is equivalent to $d\lt\frac{1+\sin(2\theta)}{2+\sin(2\theta)}$.

Thus, if $\frac12\lt d\lt\frac{1+\sin(2\theta)}{2+\sin(2\theta)}$, the area of the purple region is $$ \begin{align} \text{area}_\text{purple} &=s^2-\frac{2(d-1/2)^2}{\sin(\theta)\cos(\theta)}\tag{6a}\\ &=s^2-\frac{2\left(s(\sin(\theta)+\cos(\theta))-\frac12\right)^2}{\sin(\theta)\cos(\theta)}\tag{6b}\\[3pt] &=\scriptsize-\frac12\csc(\theta)\sec(\theta)+2(\csc(\theta)+\sec(\theta))s-(3+2\csc(\theta)\sec(\theta))s^2\tag{6c}\\[6pt] \int\text{area}_\text{purple}\,\mathrm{d}\theta &=\scriptsize-\frac12\log(\tan(\theta))+2\log\left(\tan\left(\tfrac\theta2\right)\tan\left(\tfrac\pi4{+}\tfrac\theta2\right)\right)s-\left(3\theta+2\log(\tan(\theta))\right)s^2\tag{6d}\\[9pt] &=a_2(s,\theta)\tag{6e} \end{align} $$ If $d\ge\frac{1+\sin(2\theta)}{2+\sin(2\theta)}$, the red square is completely contained within the green square and so the area of the purple region is the same as the area of the red square: $$ \begin{align} \text{area}_\text{purple} &=(1-d)^2\tag{7a}\\[3pt] &=(1-s(\sin(\theta)+\cos(\theta)))^2\tag{7b}\\[3pt] &=1-2(\sin(\theta)+\cos(\theta))s+(1+2\sin(\theta)\cos(\theta))s^2\tag{7c}\\ \int\text{area}_\text{purple}\,\mathrm{d}\theta &=\theta-2(\sin(\theta)-\cos(\theta))s+\left(\theta+\sin^2(\theta)\right)s^2\tag{7d}\\ &=a_3(s,\theta)\tag{7e} \end{align} $$ Conditional Ranges

We need to apply each of $a_1$, $a_2$, or $a_3$ over particular parts of $\left[0,\frac\pi4\right]$. That is, we use $a_1$ over the green region of the chart below, $a_2$ over the yellow region, and $a_3$ over the red region.

To simplify the usage of the information in the chart above, define $$ \theta_1(s)=\frac12\sin^{-1}\left(\frac{1-4s^2}{4s^2}\right)\tag8 $$ and $$ \theta_2(s)=\frac12\sin^{-1}\left(\frac{1-4s^2+\sqrt{1-4s^2}}{2s^2}\right)\tag9 $$

If $s\le\frac1{2\sqrt2}$, we use $a_1$ for all $\theta\in\left[0,\frac\pi4\right]$. That is, the probability of the randomly rotated and positioned smaller square containing the center of the unit square is $$ \frac{a_1\!\left(s,\frac\pi4\right)-a_1\!\left(s,0\right)}{a_3\!\left(s,\frac\pi4\right)-a_3\!\left(s,0\right)}=\frac{\frac\pi4s^2}{\frac\pi4-2s+\left(\frac\pi4+\frac12\right)s^2}\tag{10} $$ If $\frac1{2\sqrt2}\lt s\le\frac{\sqrt2}3$, we use $a_1$ over $\theta\in[0,\theta_1(s)]$ and $a_2$ over $\theta\in\left[\theta_1(s),\frac\pi4\right]$. That is, the probability of the randomly rotated and positioned smaller square containing the center of the unit square is $$ \frac{a_2\!\left(s,\frac\pi4\right)-a_2(s,\theta_1(s))+a_1(s,\theta_1(s))-a_1(s,0)} {a_3\!\left(s,\frac\pi4\right)-a_3\!\left(s,0\right)}\tag{11} $$ If $\frac{\sqrt2}3\lt s\le\frac12$, we use $a_1$ over $\theta\in[0,\theta_1(s)]$, $a_2$ over $\theta\in[\theta_1(s),\theta_2(s)]$, and $a_3$ over $\theta\in\left[\theta_2(s),\frac\pi4\right]$. That is, the probability of the randomly rotated and positioned smaller square containing the center of the unit square is $$ \frac{a_3\!\left(s,\frac\pi4\right)-a_3(s,\theta_2(s))+a_2(s,\theta_2(s))-a_2(s,\theta_1(s))+a_1(s,\theta_1(s))-a_1(s,0)} {a_3\!\left(s,\frac\pi4\right)-a_3\!\left(s,0\right)}\tag{12} $$ In the question, $s=\frac1{\sqrt{10}}$, so we apply $(10)$ and get a probability of $$ \bbox[5px,border:2px solid #C0A000]{\frac\pi{11\pi+2-8\sqrt{10}}\approx0.279022074}\tag{13} $$ Here is a plot of the probability that a randomly rotated and positioned square of side $s$ inside the unit square contains the origin of the unit square:

---

Simulation

I ran a simulation placing the center of the smaller square, with side $s$, uniformly in a square, concentric with the unit square, and with side $1-s$, and rotating uniformly with $\theta\in\left[0,\frac\pi4\right]$.

All trials where the translated and rotated square did not fit completely in the unit square were rejected. The remaining trials were tallied counting those where the center of the unit square fell within the smaller square.

The results of $10,\!000,\!000$ trials with $s=\frac1{\sqrt{10}}$ were

non-rejected trials: $7,\!664,\!102$
successful trials: $2,\!136,\!729$
ratio: $0.278797046$

This is close to $0.279022074$.

Here is the Mathematica code for the simulation:

test[s_, x_, y_, \[Theta]_] := 
 Module[{c = {x - 1/2, y - 1/2} (1 - s), 
   m = s {{Cos[\[Theta]], Sin[\[Theta]]}, {-Sin[\[Theta]], Cos[\[Theta]]}},
   u = 1/2 {{-1, 1}, {1, 1}, {1, -1}, {-1, -1}}, 
   cross = (#1.{{0, 1}, {-1, 0}}.#2) &, ul, ur, lr, ll},
   {ul, ur, lr, ll} = u.m + {c, c, c, c}; 
  If[Max[Abs[ul], Abs[ur], Abs[lr], Abs[ll]] > 1/2, {0, 0}, 
   If[Min[cross[ur, ul], cross[ul, ll], cross[ll, lr], cross[lr, ur]] < 0,
    {0, 1}, {1, 1}]]]

Module[{s = 1/Sqrt[10], n = 10000000, data = {0, 0}}, 
 Do[data += test[s, RandomReal[1], RandomReal[1], RandomReal[Pi/4]], n];
 data]

Let me see if I understand you problem.

You are looking at a square $S = [-R/2, R/2]^2$ of side length $R$ (for simplicity centred at the origin. I will write $C(x;\theta)$ for the smaller square of side length $r$ centred in $x\in\mathbb{R}^2$ at rotation $\theta$. You want to know what the probability is of the event that $0\in C(x;\theta)$ under the constraint $C(x;\theta)\subseteq S$.

For simplicity, we take $r \ll R$ in a suitable sense ($=$ so that everything works just fine). The first thing I would like to do is to wrap it up mathematically. For this, we define our probability space to be $$ \Omega := \{(x,\theta)\in \mathbb{R}^2\times [0,\pi/4]\;\vert\; C(x;\theta)\subseteq S\}. $$ Note that I only consider angles up to 45 degrees, because the situation is symmetric for angles between 45 and 90 degrees. To model "random appearance", we take the uniform distribution on $\Omega$.

First things first: Can we describe $\Omega$? For this, fix the angle $\theta$. Write $h(\theta)$ for the height ($=$ the width) of the skewed square $C(x;\theta)$. Then $$ h(\theta) = \cos\left(\dfrac{\pi}{4} - \theta\right)\cdot\sqrt{2}\cdot r. $$ We conclude that $$ \Omega = \{(x,\theta)\in \mathbb{R}^2\times [0,\pi/4]\;\vert\; \Vert x\Vert_\infty \leq (R - h(\theta))/2\}. $$ In particular, $$ m(\Omega) = \int_0^{\pi/4} \big{(} R - r\cos(\theta)\sqrt{2}\big{)}^2\mathrm{d}{\theta} = \dfrac{\pi}{4}R^2 - 2Rr + \dfrac{2+\pi}{4}r^2, $$ where I wrote $m$ for the Lebesgue measure on $\mathbb{R}^2$.

Now to the event we are interested in: $$ A = \{(x,\theta)\in \Omega\;\vert\; 0\in C(x;\theta)\}. $$ Since we took $r\ll R$ we do not need to worry about whether the squares touching $0$ are indeed included in $S$. We further simplify the problem by noticing that the size of the set of suitable angles $\theta$ for a given centre $x$ only depends on $\rho := \Vert x\Vert_2$. So let's fix $0\leq \rho \leq \dfrac{\sqrt{2}}{2}r$. (The upper bound is given by the half diagonal.) To get some visual intuition, take $x = (\rho, 0)$. Let us start in the most "favorable" position, i.e. at $\theta=\pi/4$. The question is by how much we are allowed to skew the square. Note that the point closest to the origin is at distance $$ d(\theta) = \dfrac{r}{2\cos\theta}. $$ Hence, we need $$ \cos\theta \leq \dfrac{r}{2\rho}. $$ This means that we have the constraint $\cos^{-1}\left(\dfrac{r}{2\rho}\right)\leq \theta \leq \dfrac{\pi}{4}$ for $\rho \geq r/2$ and bo constraint for $\rho < r/2$. This means that $$ m(A) = \dfrac{\pi^2r^2}{8} - 2\pi\int_{r/2}^{\sqrt{2}r/2} \rho \cos^{-1}\left(\dfrac{r}{2\rho}\right)\mathrm{d}\rho = \dfrac{\pi^2r^2}{8} - \dfrac{\pi(\pi - 2)r^2}{8}. $$

If I have not made any mistake in any of those integrals, we should get the probability $$ \dfrac{m(\Omega)}{m(A)} = \text{something very ugly}. $$

Edit: The following comments try to give a bit more intuitive insight into the notation.

First, note that the uniform distribution on a interval means that the probability of the event is the "length" or "size" of the event, divided by the total length of the interval. The same is going on here: Once we have understood that we want to take $(x,\theta)$ uniformly in $\Omega$, the probability of an event $A$ is the "volume" (because we operate in 3 dimensions now) of $A$, divided by the volume of $\Omega$.

Now, to obtain the volume of a set, we usually integrate, especially if the bodies are complex as in this case. But in the end, it is only "measuring the volume". The $m$ I use is just "the volume of". This "usual" volume is called Lebsegue measure, because there other ways to make sense of the word "volume", but in this case this doesn't matter.

I don't think that there is a straightforward way to obtain the explicit expression of the result without using integrals, i.e. by simply using geometric surface/volume formula, but you can always try!

For the other notation: $\Vert (x,y)\Vert_2 = \sqrt{x^2 + y^2}$ is the usual Euclidean distance, $\Vert (x,y)\Vert_\infty = \max\{\vert x\vert, \vert y\vert\}$ is the so-called infinity norm. (There are $\Vert\cdot\Vert_p$-norms for any $p\geq 1$.) When I write $r\ll R$, I mean that $r$ is a lot smaller than $R$. Here, we need this assumption when figuring out how we may describe $A$.

Also, I mainly used WolframAlpha for those horrendous integrals.

Edit 2: There is a lot easier way to calculate $m(A)$. Indeed, mjw (https://math.stackexchange.com/q/3966261) correctly points out that, for any given angle $\theta$, the centre must be in a square of side length $r$. That means that $$ m(A) = \dfrac{\pi}{4}r^2. $$ I corrected the above mistake.