Radius of a circle touching a rectangle both of which are inside a square

It is just using the pythagorean theorem:
$a=10$ $cm$
$b=20$ $cm$
$(r-a)^2+(r-b)^2=r^2$
$(r-10)^2+(r-20)^2=r^2$
$r^2+100-20r+r^2+400-40r=r^2$
$r^2-60r+500=0$
$r=50$ $cm$
$r=10$ $cm$
The $r=50$ $cm$ is the acceptable answer.

Place the center of the cricle at $O.$

Let the radius be $R$

The corner of the square is $(R,R)$ I have made a small alteration to the picture to create fewer negative numbers.

Offsetting by the rectange, the corer of the rectangle is $(R-20, R-10)$

And the distance from this point equals the $R.$

That should put you on your way to the solution.

$(x, y) = (R-20, R-10)$ as a point on the circle $y = \sqrt{R^2 - x^2}$

$R - 10 = \sqrt{R^2 - (R-20)^2}$

$(R- 10)^2 = R^2 - (R-20)^2$

$R^2 - 20R + 100 = R^2 - (R^2 - 40R + 400)$

$R^2 - 60R + 500 = 0$

$(R - 50)(R-10) = 0$

$R = 50$ is the only sensible option.