Fit biggest square inside a polygon

It is an almost trivial computation, which I added to the picture. The sizes of the bigger square, H, and the smaller square, h, are stored in functions, which you can adjust. Everything is computed from those.

\documentclass[tikz,border=3mm]{standalone}
\begin{document}
\begin{tikzpicture}[>=latex,
    declare function={H=10;h=2;hprime=(H-sqrt(H*H-4*h*H))/2;}]
 \draw (-H/2,H/2-h) -| (-H/2+h,H/2) -|  (H/2-h,H/2-h) -| (H/2,-H/2+h) -| (H/2-h,-H/2)
 -| (-H/2+h,-H/2+h) -| cycle;
 \draw (-H/2+hprime,H/2)  -- (H/2,H/2-hprime) -- (H/2-hprime,-H/2) -- (-H/2,-H/2+hprime)
 --cycle;
 \draw[|<->|] (-H/2-2,H/2) -- node[fill=white] {$H-h'$} ++ (0,-H+hprime);
 \draw[|<->|] (-H/2-1,H/2-h) -- node[fill=white] {$H-h-h'$} ++ (0,-H+hprime+h);
 \draw[|<->|] (H/2-h+0.3,H/2) -- node[fill=white] {$h$} ++ (0,-h);
 \draw[|<->|] (-H/2,H/2-h+0.3) -- node[fill=white] {$h$}  ++(h,0);
 \draw[|<->|] (-H/2,H/2+0.3) -- node[fill=white] {$h'$}  ++(hprime,0);
 \path (current bounding box.north) node[above=2em]
 {$\displaystyle\frac{h}{H-h-h'}=\frac{h'}{H-h'}\quad
 \Rightarrow\quad 
 h'=\frac{1}{2}(H-\sqrt{H^2-4\,h\,H})$ };
\end{tikzpicture}
\end{document}

EDIT: Fixed typo in formula that is displayed, the drawing code is unchanged.

BTW. obviously one needs to demand that h<H/4.

\documentclass[tikz,border=3mm]{standalone}
\begin{document}
\foreach \X in  {0.5,0.6,...,2.4,2.41,2.42,...,2.5,2.49,2.48,...,2.4,2.3,2.2,...,0.6}
{\begin{tikzpicture}[declare function={H=10;h=\X;
hprime=(H-sqrt(abs(H*H-4*h*H)))/2;}]
 \draw (-H/2,H/2-h) -| (-H/2+h,H/2) -|  (H/2-h,H/2-h) -| (H/2,-H/2+h) -| (H/2-h,-H/2)
 -| (-H/2+h,-H/2+h) -| cycle;
 \draw (-H/2+hprime,H/2)  -- (H/2,H/2-hprime) -- (H/2-hprime,-H/2) -- (-H/2,-H/2+hprime)
 --cycle;
\end{tikzpicture}}
\end{document}

The animation can be made smoother by solving for h as a function of hprime.

\documentclass[tikz,border=3mm]{standalone}
\begin{document}
\foreach \X in  {5,15,...,355}
{\begin{tikzpicture}[declare function={H=10;hprime=(1+sin(\X))*H/2;
h=((H-hprime)*hprime)/H;}]
 \draw (-H/2,H/2-h) -| (-H/2+h,H/2) -|  (H/2-h,H/2-h) -| (H/2,-H/2+h) -| (H/2-h,-H/2)
 -| (-H/2+h,-H/2+h) -| cycle;
 \draw (-H/2+hprime,H/2)  -- (H/2,H/2-hprime) -- (H/2-hprime,-H/2) -- (-H/2,-H/2+hprime)
 --cycle;
\end{tikzpicture}}
\end{document}

Here's a solution in Metapost. This is wrapped up in luamplib, so you need to compile it with lualatex. I've left the dotlabels as an exercise-for-the-reader.

\documentclass[border=5mm]{standalone}
\usepackage{luamplib}
\begin{document}
\mplibtextextlabel{enable}
\begin{mplibcode}
beginfig(1);
    path poly, sq, c;
    poly = (for i=0 upto 3: subpath (3, 1) of unitsquare scaled 2 
            shifted (-5, -5) rotated 90i -- endfor cycle) scaled 1cm;

    c = fullcircle rotated 184 scaled abs(point 1 of poly - point 4 of poly) 
                   shifted 1/2[point 1 of poly, point 4 of poly];
    (t, u) = poly intersectiontimes c;

    sq = for i=0 upto 3: point t + 3i of poly -- endfor cycle;

    fill poly withcolor 7/8 [blue, white];
    draw poly;

    fill sq withcolor background;
    draw sq withcolor 2/3 blue;

endfig;
\end{mplibcode}
\end{document}

Notes

• the poly path is constructed by taking the top and right sides of the standard unitsquare path scaled and shifted to the bottom left, then rotated four times. The -- in the loop is enough to draw the long sides.

• to solve the puzzle you make a circle between the first two inner corners and find where it cuts the polygon. This is time t returned from the intersectiontimes command. Rotating the circle by 184 turns out to be important; otherwise you find the intersection at the corner rather than the intersection along the long side.

• armed with the time along the path, you just have to connect the points t, t+3, t+6 and t+9 of the polygon.

• to get the "other" solution, rotate the circle by 274 instead of 184

• and if you want to see the circle, try adding draw c dashed withdots scaled 1/2 withcolor 2/3 red;

• finally if you want MP to find the actual area for you, then try adding this at the end: label(decimal round((abs(point 1 of sq - point 0 of sq)/cm)**2), origin);. On my system this shows "60".

With tkz-euclide

\documentclass{standalone} 
\usepackage{tkz-euclide}

\begin{document} 

\begin{tikzpicture}
\pgfmathsetmacro{\h}{sqrt(5)}
\tkzDefPoints{0/0/A,10/0/B,2/0/A_1,8/0/B_3,10/8/C_3,2/10/D_3,0/5-\h/Z}
\tkzDefSquare(A,B)\tkzGetPoints{C}{D}
\tkzDefSquare(A,A_1)\tkzGetPoints{A_2}{A_3}
\tkzDefSquare(B_3,B)\tkzGetPoints{B_1}{B_2}
\tkzDefSquare(C_3,C)\tkzGetPoints{C_1}{C_2}
\tkzDefSquare(D_3,D)\tkzGetPoints{D_1}{D_2}
\tkzInterLL(Z,A_2)(A,B)\tkzGetPoint{W}
\tkzDrawPolygon(A_3,A_2,A_1,B_3,B_2,B_1,C_3,C_2,C_1,D_3,D_2,D_1)
\tkzDefSquare(Z,W) \tkzGetPoints{X}{Y}
\tkzDrawPolygon(X,Y,Z,W)
\end{tikzpicture}

\end{document}

But why XYZW is the biggest square ?