When does a sequence of rotated-and-circumscribed rectangles converge to a square?
This is a particularly good first question!
Using a little linear algebra, we can actually write sufficiently explicit formulas for $A_n$ and $B_n$ to confirm your conjecture, namely that the divergence of $\sum_{i = 1}^{\infty} \alpha_i$ is sufficient and necessary.
First, by writing the recurrence relations for $A_n$ and $B_n$ in matrix notation and applying a straightforward induction, we get that, for all nonnegative integers $n$, $$\require{cancel} \pmatrix{A_n \\ B_n} = \left(\prod_{i = 1}^n \underbrace{\pmatrix{\cos \alpha_i & \sin \alpha_i \\ \sin \alpha_i & \cos \alpha_i}}_{\Gamma_i}\right) \pmatrix{a \\ b} . $$ We can make this more explicit by employing the standard trick of diagonalization: The eigenvalues of $\Gamma_i$ are $\lambda_i^{\pm} := \cos \alpha_i \pm \sin \alpha_i$, so for $\alpha_i$ in the given range, we have that $$0 < \lambda_i^- < \lambda_i^+ ,$$ and in particular the eigenvalues are distinct, guaranteeing that we can write $$\Gamma_i = P_i \pmatrix{\lambda_i^- & 0 \\ 0 & \lambda_i^+} P_i^{-1}$$ for some $P_i$. In fact, since the rotations $\Gamma_i$ all commute with one another (and are diagonalizable) they are simultaneously diagonalizable, that is, we can choose all of the $P_i$ to be the same matrix $P$. Computing eigenvectors corresponding to the eigenvalues shows that we may take $$P = \pmatrix{\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}} \\ -\frac{1}{\sqrt{2}} & \frac{1}{\sqrt{2}}},$$ the rotation matrix for a clockwise rotation of $\frac{\pi}{4}$ (see the remark below).
Our observation about being able to use the same $P$ for all rotations pays off immediately, as the expression for the product $\prod_{i = 1}^n \Gamma_i$ of rotations simplifies dramatically: $$ \prod_{i = 1}^n \Gamma_i = \prod_{i = 1}^n \left[P \pmatrix{\lambda_i^- & 0 \\ 0 & \lambda_i^+} P^{-1} \right] = P \pmatrix{\prod_{i = 1}^n \lambda_i^- & 0 \\ 0 & \prod_{i = 1}^n \lambda_i^+} P^{-1}. $$ This yields explicit formulas for $A_n, B_n$ in terms of $a, b, (\alpha_i)$: $$\pmatrix{A_n \\ B_n} = \left(\prod_{i = 1}^n \Gamma_i\right) \pmatrix{a \\ b} = \pmatrix{\frac{1}{2}(a + b) \prod_{i = 1}^n \lambda_i^+ + \frac{1}{2} (a - b) \prod_{i = 1}^n \lambda_i^- \\ \frac{1}{2}(a + b) \prod_{i = 1}^n \lambda_i^+ + \frac{1}{2}(b - a) \prod_{i = 1}^n \lambda_i^-} .$$
Dividing and rewriting gives $$\frac{A_n}{B_n} = \frac{\frac{1}{2}(a + b) \prod_{i = 1}^n \lambda_i^+ + \frac{1}{2} (a - b) \prod_{i = 1}^n \lambda_i^-}{\frac{1}{2}(a + b) \prod_{i = 1}^n \lambda_i^+ + \frac{1}{2}(b - a) \prod_{i = 1}^n \lambda_i^-} = \frac{1 + \mu_n}{1 - \mu_n},$$ where $$ \mu_n := \frac{a - b}{a + b} \prod_{i = 1}^n \frac{\lambda_i^-}{\lambda_i^+} = \frac{a - b}{a + b} \prod_{i = 1}^n \frac{\cos \alpha_i - \sin \alpha_i}{\cos \alpha_i + \sin \alpha_i} = \frac{a - b}{a + b} \prod_{i = 1}^n \tan \left(\frac{\pi}{4} - \alpha_i\right) , $$ and $0 < \mu_n < 1$.
So, if $\limsup \alpha_i > 0$, then $\mu_n \to 0$ and hence $\frac{A_n}{B_n} \to 1$. If, on the other hand, $\limsup \alpha_i = 0$, we need not have $\mu_n \to 0$. For small $\alpha_i$, $\tan \left(\frac{\pi}{4} - \alpha_i\right) \sim 1 - 2 \alpha_i$, so for $\frac{A_n}{B_n} \to 1$ I believe it's sufficient and necessary for $\sum_{i = 1}^{\infty} \alpha_i$ to diverge.
Similarly, we get \begin{align*}\lim_{n \to \infty} (A_n - B_n) &= (a - b) \prod_{i = 1}^n (\cos \alpha_i - \sin \alpha_i) . \end{align*} Again, $\limsup \alpha_i > 0$ is sufficient to guarantee vanishing. For the case $\limsup \alpha_i = 0$ we can use $\cos \alpha_i - \sin \alpha_i \sim 1 - \alpha_i$ to conclude that $A_n - B_n \to 0$ iff $\sum_{i = 1}^{\infty} \alpha_i$ diverges.
Remark The Jordan decomposition $\Gamma_i = P \pmatrix{\lambda_i^- & 0 \\ 0 & \lambda_i^+} P^{-1}$ is in this problem more than a convenient tool---it also gives some geometrical insight into why rotating and circumscribing makes rectangles more squarelike. The matrix $P^{-1}$ rotates a vector in the $ab$-plane (which we can view as a vector from the rectangle to one of its corners) by $\frac{\pi}{4}$. Then $\Gamma_i$ shears the rotated vector, lengthening its component in the direction that corresponds to the quantity $a + b$ and shortening its component in the direction that corresponds to $a - b$, which we may view as the absolute defect from squareness. Finally, the matrix $P$ rotates back the one-eighth of a turn, which we may view as returning back to the original $ab$-coordinates.
This will not explain all your observations, but might shed some first light on what is going on.
As you noted we can find this nice recursive definition for the sequence of the side lengths:
\begin{align} A_{n+1} &= \cos\alpha \cdot A_n+\sin\alpha \cdot B_n,\\ B_{n+1} &= \sin\alpha \cdot A_n+\cos\alpha \cdot B_n. \end{align}
This looks pretty linear and can be expressed via some vectors and matrices:
$$ \begin{pmatrix}A_{n+1}\\B_{n+1}\end{pmatrix} = \underbrace{\begin{pmatrix} \cos \alpha &\sin\alpha \\ \sin\alpha &\cos \alpha \end{pmatrix}}_{R(\alpha)} \begin{pmatrix}A_{n}\\B_{n}\end{pmatrix}. $$
So the problem reduces to studying the behavior of the matrices $R(\alpha)$ and their compositions $\prod_i R(\alpha_i)$. This is because
$$ \begin{pmatrix}A_{n+1}\\B_{n+1}\end{pmatrix} = R(\alpha_n)R(\alpha_{n-1})\cdots R(\alpha_0) \begin{pmatrix}A_{0}\\B_{0}\end{pmatrix}. $$
Note that
$$\det R(\alpha)=\cos^2\alpha-\sin^2\alpha\in(0,1),\qquad\text{for }\alpha\in(0,\pi/4).$$
This means that (at least for $\alpha_i$ bounded from below by some $\epsilon$) the matrix defined by the product
$$\prod_{i=0}^n R(\alpha_n)$$
becomes more and more singular for $n\to\infty$ as $\prod_i \det R(\alpha_i)\to 0$. Further, note that $R(\alpha)$ is in the form
$$\begin{pmatrix} a&b\\b&a \end{pmatrix}$$
and that the product of two matrices in such a form is also of such a form. Hence the product convergec to a singular matrix of such a form (this is not rigorous but certainly can be made to). All such matrices are of the form
$$\begin{pmatrix} c&c\\c&c \end{pmatrix}=c\cdot\mathbf 1$$
Applying this to the vector $(A_0,B_0)^\top$ describing your initial rectangle will give you
$$c\begin{pmatrix} A_0+B_0\\A_0+B_0 \end{pmatrix}.$$
And this is indeed a square as both side lengths are equal (in the limit of course).
As I said, this is neither complete nor fully rigorous. For example, this does not discuss the case $\alpha_i=1/i$ as I assumed $\alpha_i>\epsilon>0$. In such a case we probably have $A_n,B_n\to\infty$ and cannot expect $A_n-B_n\to 0$ (e.g. choose $\alpha_i$ always close to $\pi/4$), but my reasoning shows that we still expect $A_n/B_n\to 1$.