Convert a rotated ellipse to sheared ellipse

Recall that the standard form of an ellipse centered at the origin is $$\frac{x^2}{a^2} + \frac{y^2}{b^2} = 1,$$ where $a$ is half the length of the horizontal axis and $b$ half the vertical (so we only need $a, b > 0$ and there is no need to specify $a > b$). Upon counterclockwise rotation by an angle $\theta$ about the origin, that is to say, $$(x,y) \to (x \cos \theta - y \sin \theta, x \sin \theta + y \cos \theta),$$ this equation takes the form $$Ax^2 + Bxy + Cy^2 = 1,$$ where $$\begin{align*} A &= \frac{\cos^2 \theta}{a^2} + \frac{\sin^2 \theta}{b^2}, \\ B &= \left(\frac{1}{a^2} - \frac{1}{b^2}\right) \sin 2\theta, \\ C &= \frac{\sin^2 \theta}{a^2} + \frac{\cos^2 \theta}{b^2}. \end{align*}$$

Now it should be obvious that the intersection of the red diameter with the ellipse in your second figure is simply the solution to $Ax^2 = 1,$ i.e. $$x = \pm \frac{1}{\sqrt{A}} = \pm \frac{ab}{\sqrt{a^2 \sin^2 \theta + b^2 \cos^2 \theta}}$$ hence the positive value is also half the length of the red diameter.

The blue diameter in the second figure corresponds to the vertical extrema; e.g., what is the largest possible choice of $y$ satisfying $Ax^2 + Bxy + Cy^2 = 1$ for any valid choice of $x$? Some thought should lead you to conclude that this occurs precisely when the quadratic in $x$ $$Ax^2 + (By)x + (Cy^2 - 1) = 0$$ has a repeated root; i.e., the discriminant $$\Delta = (By)^2 - 4A(Cy^2 - 1) = 0.$$ Thus $$y = \pm \frac{2\sqrt{A}}{\sqrt{4AC - B^2}} = \pm \sqrt{a^2 \sin^2 \theta + b^2 \cos^2 \theta}.$$ and the vertical height of the green parallelogram is twice the positive value. The slope of the blue line (after some calculations) should be $$\frac{a^2 \tan \theta + b^2 \cot \theta}{a^2 - b^2},$$ the proof of which is left as an exercise.

I strongly advise you to verify these calculations as I have not spent the time and effort to do so myself.

Let the input parameters be $(a, b, \theta)$, which are respectively the length of major and minor axis and the angle of rotation. Of course the first thing is to rotate the ellipse to the "standard" one $$\frac{x^2}{a^2} + \frac{y^2}{b^2}=4.$$

Let the new ellipse touches the green parallelogram at $(\pm p/2, 0)$ and $\pm (q/2m, q/2)$. Here $p$ is the width (length of the red line), $q$ is the height of the bounding parallelogram and $m$ is the slope of the non-horizontal green line.

Indeed we need only to find a $2\times 2$ matrix $A$ so that $$A\begin{bmatrix} a/2 \\ 0\end{bmatrix} =\begin{bmatrix} p/2 \\ 0\end{bmatrix}, \ \ \ A\begin{bmatrix} 0\\ b/2\end{bmatrix} = \begin{bmatrix} q/2m \\ q/2\end{bmatrix}.$$ Then the rectangle bounding the old ellipse will be sent under $A$ to the green parallelogram, and the standard ellipse will be sent to the new one, as suggested in the comment.

It is easy to see that

$$A = \begin{bmatrix} p/a & q/bm \\ 0 & q/b\end{bmatrix}.$$

Thus the $ 2\times 2$ matrix

$$\begin{bmatrix} p/a & q/bm \\ 0 & q/b\end{bmatrix} \begin{bmatrix} \cos\theta & \sin\theta \\-\sin\theta & \cos\theta\end{bmatrix}$$

will send the ellipse given by $(a, b, \theta)$ to the ellipse given by $(p,q,m)$.