How to draw a square on cylinder?

Consider three points on the square, say B, C and D. Without loss of generality we can assume that they have the coordinates

 C = (r \cos \phi,r \sin\phi,h)
 D = (r \cos (180-\phi),r \sin(180-\phi),h)
 B = (r \cos \phi,-r \sin\phi,0)

The conditions that they form a square mean that

 CD = BC

where

 CD = 2r \cos \phi
 BC = \sqrt{(2r \sin\phi)^2+h^2}

This yields

 \phi = \acos(h^2/(4r^2))/2

This confirms hpechristiansens numerical result:

 CD = 2 r \cos \left(\frac{1}{2} \acos\left(\frac{h^2}{4 r^2}\right)\right)

MWE

\documentclass[border=2mm,12pt,tikz]{standalone}
\usepackage{tikz-3dplot} 
\usetikzlibrary{3d,calc,backgrounds,patterns}
\begin{document}
\pgfmathsetmacro{\myr}{3}
\pgfmathsetmacro{\h}{2}
\pgfmathsetmacro{\angA}{acos(\h*\h/(4*\myr*\myr))/2}
\tdplotsetmaincoords{65}{100}
\begin{tikzpicture}[tdplot_main_coords,scale=1,line cap=butt,line join=round]
\begin{scope}[canvas is xy plane at z=0]
\draw[dashed] (\tdplotmainphi:\myr) arc(\tdplotmainphi:\tdplotmainphi+180:\myr);
\coordinate (O) at (0,0);
\coordinate (A) at (\angA:\myr);
\coordinate (B) at (180-\angA:\myr);
\draw[thick] (\tdplotmainphi:\myr) coordinate(BR) arc(\tdplotmainphi:\tdplotmainphi-180:\myr)
coordinate(BL);
\end{scope}
\begin{scope}[canvas is xy plane at z=\h]
\coordinate (O') at (0,0);
\coordinate (C) at (-180+\angA:\myr);
\coordinate (D) at (-\angA:\myr);
\draw[thick]  (O') circle[radius=\myr];
\draw [thick](BR) -- (\tdplotmainphi:\myr) (BL) -- (\tdplotmainphi-180:\myr); 
\end{scope}
\fill (A) circle[radius=1pt] node[above] {$A$};
\fill (B) circle[radius=1pt] node[above] {$B$};
\fill (C) circle[radius=1pt] node[above] {$C$};
\fill (D) circle[radius=1pt] node[above] {$D$};
\draw[dashed]  (C) -- (B) --   (A);
\draw[thick] (A) -- (D) -- (C);
\end{tikzpicture}
\end{document}

In this code, the point B can run everywhere on the circle. You can change it at \def\angB{-20}, then you have an option of the square ABCD.

\documentclass[border=2mm,12pt,tikz]{standalone}
\usepackage{tikz-3dplot} 
\usetikzlibrary{3d,calc,backgrounds,patterns}
\usepackage{fouriernc}
\begin{document}
    \pgfmathsetmacro{\myr}{3}
    \pgfmathsetmacro{\h}{2}
    \pgfmathsetmacro{\d}{sqrt(\h*\h+4*\myr*\myr)/sqrt(2)}
    \def\angB{-20}
    \def\angA{{\angB + acos((2*\myr*\myr-\d*\d)/(2*\myr*\myr)}}
    \tdplotsetmaincoords{65}{100}
    \begin{tikzpicture}[tdplot_main_coords,scale=1,line cap=butt,line join=round]
    \begin{scope}[canvas is xy plane at z=0]
    \draw[dashed] (\tdplotmainphi:\myr) arc(\tdplotmainphi:\tdplotmainphi+180:\myr);
    \coordinate (O) at (0,0);
    \coordinate (A) at (\angA:\myr);
    \coordinate (B) at (\angB:\myr);
    \coordinate (A') at ($ 2*(O) - (A) $);
    \coordinate (B') at ($ 2*(O) - (B) $);

    \draw[thick] (\tdplotmainphi:\myr) coordinate(BR) arc(\tdplotmainphi:\tdplotmainphi-180:\myr)
    coordinate(BL);
    \end{scope}
    \begin{scope}[canvas is xy plane at z=\h]
    \coordinate (O') at (0,0);
    \coordinate (C) at  ($ (O') - (O) +(A')$);
    \coordinate (D) at  ($ (O') - (O) +(B')$);
    \draw[thick]  (O') circle[radius=\myr];
    \draw [thick](BR) -- (\tdplotmainphi:\myr) (BL) -- (\tdplotmainphi-180:\myr); 
    \end{scope}
    \foreach \v/\position in { B/below,O/below,A/below,B'/above,A'/above,C/above,D/above} {\draw[draw =black, fill=black] (\v) circle (1pt) node [\position=0.2mm] {$\v$};
    }
    \draw[thick]    (C) -- (D) ;
    \draw[dashed] (A) -- (A') (B) -- (B') (A) -- (B) -- (A') -- (B') -- cycle (B') -- (D) (A') -- (C) (A) --  (D) (B) -- (C); 
    \end{tikzpicture}
\end{document} 

With some calculations by using Maple, coordinates of point A, B, C, D can be choose

then, the code

\documentclass[border=2mm,12pt,tikz]{standalone}
\usepackage{tikz-3dplot} 
\begin{document}
    \pgfmathsetmacro{\myr}{3}
    \pgfmathsetmacro{\h}{2}
    \tdplotsetmaincoords{65}{100}
    \begin{tikzpicture}[tdplot_main_coords,scale=1,line cap=butt,line join=round]
    \begin{scope}[canvas is xy plane at z=0]
    \draw[dashed] (\tdplotmainphi:\myr) arc(\tdplotmainphi:\tdplotmainphi+180:\myr);
    \coordinate (O) at (0,0);
    \coordinate (A) at ({1/4*sqrt(8*\myr*\myr-2*\h*\h)}, {sqrt(1/2*\myr*\myr+(1/8)*\h*\h)});
    \coordinate (B) at ({1/4*sqrt(8*\myr*\myr-2*\h*\h)}, -{sqrt(1/2*\myr*\myr+(1/8)*\h*\h)});
    \draw[thick] (\tdplotmainphi:\myr) coordinate(BR) arc(\tdplotmainphi:\tdplotmainphi-180:\myr)
    coordinate(BL);
    \end{scope}
    \begin{scope}[canvas is xy plane at z=\h]
    \coordinate (O') at (0,0);
    \coordinate (C) at ({-1/4*sqrt(8*\myr*\myr-2*\h*\h)}, -{sqrt(1/2*\myr*\myr+(1/8)*\h*\h)});
    \coordinate (D) at ({-1/4*sqrt(8*\myr*\myr-2*\h*\h)}, {sqrt(1/2*\myr*\myr+(1/8)*\h*\h)});
    \draw[thick]  (O') circle[radius=\myr];
    \draw [thick](BR) -- (\tdplotmainphi:\myr) (BL) -- (\tdplotmainphi-180:\myr); 
    \end{scope}
    \foreach \v/\position in { B/below,O/below,A/below,C/above,D/above} {\draw[draw =black, fill=black] (\v) circle (1pt) node [\position=0.2mm] {$\v$};
    }
    \draw[thick]    (C) -- (D) ;
    \draw[dashed]    (C) -- (B) -- (A) --  (D)  ; 
    \end{tikzpicture}
\end{document} 

This is just to help/check a real answer. I drew the figure in 3D CAD(Fusion 360) and put the constraints from you description. This is what I got (scale(10:1)) :

The side length of the square is 4.4721