Matrix diagonalization in SU(2) and SO(3)
There is nothing being diagonalized here. The passage you quote does use the notation $$ AXA^\dagger = AXA^{-1} $$ (since $A$ is unitary), which does appear often in diagonalization problems, but the transformation is much broader than that. This type of transformation, $$ A\mapsto BAB^{-1}, $$ is known as a matrix similarity transformation, and two matrices $A$ and $C$ are said to be similar if and only if there exists an invertible matrix $B$ such that $C=BAB^{-1}$.
Diagonalization is the process of taking a matrix $A$ and finding a matrix $C$ which is similar to $A$ and also diagonal.
However, the uses of similarity as a relationship and a transformation go way beyond just diagonalization - in essence, two matrices are similar if and only if they represent the same linear map on two different bases. As such, a wide array of matrix properties are preserved by similarity (well summarized in the Wikipedia link above), which is part of what makes the relationship so useful.
The Lie algebra structure of $su(2)$ is entirely irrelevant here; all that matters is that $su(2)$ is a three-dimensional real vector space, so start by forgetting about Lie algebras.
An element $A\in SU(2)$ acts on that 3-d vector space by mapping $X$ to $AXA^{-1}$.
Therefore an element $A\in\pmatrix{P&Q\cr -\overline{Q}&\overline{P}\cr}$ in $SL(2)$ can be represented as a 3 by 3 real matrix $\rho(A)$. It would be a very very good exercise for you to write down an explicit formula for $\rho(A)$ in terms of $P$ and $Q$ --- not that the final result is important, but this will fix in your head exactly what's going on here. Start, of course, by computing how $A$ acts on each of the three known basis vectors for $su(2)$; those are the columns of $\rho(A)$.
Finally, check that $\rho(A)$ is in $SO(3)$, so you've mapped $SU(2)$ to $SO(3)$. It will be obvious that $A$ and $-A$ go to the same place, so the mapping is (at least) two-to-one. You can check further that it's exactly two-to-one.
Optional next step: To fix in your head the idea that nothing about $su(2)$ matters except for its 3-dimensionality, identify $A$ with the quaternion $P+Qj$ and let it act on the 3-d real vector space of pure-imaginary quaternions via conjugation. This is a different route to the same outcome, and clearly has nothing to do with Lie algebras.
Finally, I don't understand any of your questions about diagonalization or why you are so eager to diagonalize something. Since you're the one bringing diagonalization to the table, only you can know what you want to diagonalize or why.
In a nutshell, the first main point is that (up to some conventional constants) there is an isometric Lie algebra isomorphism $X\mapsto [X]$ between
the 3-dimensional Lie algebra $(su(2),[\cdot,\cdot],\det(\cdot))$ of traceless anti-Hermitian $2\times 2$ matrices equipped with the determinant as a norm square, and
the 3D space $(\mathbb{R}^3, \times, |\cdot|^2)$ equipped with the standard vector cross product and the standard norm square.
(The Lie algebra structure plays no role in what follows, so it is enough to think of the map $X\mapsto [X]$ as an isometric vector space isomorphism.)
The second crucial point is now that for each group element $A\in SU(2)$, the map $\rho(A):\mathbb{R}^3\to \mathbb{R}^3$ (which Nadri Jeevanjee defines above) is a linear isometry. Therefore it is an orthogonal transformation in 3D space $\mathbb{R}^3$, which can be represented by a $3\times3$ orthogonal matrix (where we use the standard orthonormal basis in $\mathbb{R}^3$). In other words, the map $\rho$ is a map from $SU(2)$ to $O(3)$.
The above can now be tighten up to show that $SU(2)$ is a double cover of $SO(3)$. No diagonalization is used anywhere, cf. OP's questions.