Showing the Lie Algebras $\mathfrak{su}(2)$ and $\mathfrak{sl}(2,\mathbb{R})$ are not isomorphic.
Your approach works without problems, if you write the condition $[Ax,Ay]=A[x,y]$ for all $x,y$ in terms of the $9$ coefficients of the matrix $A$. The polynomial equations in these $9$ unknowns over $\mathbb{R}$ quickly yield $\det(A)=0$, a contradiction.
Another elementary argument is the following. $\mathfrak{sl}(2,\mathbb{R})$ has a $2$-dimensional subalgebra, e.g., $\mathfrak{a}=\langle f_1,f_2\rangle$, but $\mathfrak{su}(2)$ has no $2$-dimensional subalgebra. Hence they cannot be isomorphic.
This is a Q&A style answer not meant to be the final answer to the question. It fleshes out one of the techniques suggested by Dietrich Burde for future readers.
Another elementary argument is the following. $\mathfrak{sl}(2,\mathbb{R})$ has a $2$-dimensional subalgebra, e.g., $\mathfrak{a}=\langle f_1,f_2\rangle$, but $\mathfrak{su}(2)$ has no $2$-dimensional subalgebra. Hence they cannot be isomorphic.
$\mathfrak{sl}(2, \mathbb{R})$ has a two dimensional subspace.
Consider matrices of the form $\alpha_1 f_1 + \alpha_2 f_2$. Clearly this is a subspace of $\mathfrak{sl}(2)$. We need to show the commutation operation is closed in this subspace: $$[\alpha_1 f_1 + \alpha_2 f_2, \beta_1 f_1 + \beta_2 f_2] = 2(\alpha_1\beta_2 - \alpha_2\beta_1)f_2$$
$\mathfrak{su}(2)$ does not have a two dimensional subspace.
Consider a two dimensional subspace with basis $g_1, g_2$. Then $$[\alpha_1 g_1 + \alpha_2 g_2, \beta_1 g_1 + \beta_2 g_2] = (\alpha_1\beta_2 - \alpha_2\beta_1)[g_1, g_2]$$ We must show that $g_1, g_2$ cannot be chosen such $[g_1, g_2]$ is in the span of $g_1, g_2$. To this end let $g_1 = \sum_i a_i e_i, g_2 = \sum b_i e_i$. It can be shown through direct calculation that $$[g_1, g_2] = \begin{vmatrix} 2 e_1 & a_1 & b_1 \\ 2 e_2 & a_2 & b_2 \\ 2 e_3 & a_3 & b_3 \notag \end{vmatrix}$$ In other words, the commutator of $g_1$ and $g_2$ is twice their cross product. Since the cross product is perpendicular to $g_1, g_2$ we are done.
(Warning: See comments below this post - this answer is incorrect currently. A completed answer has been posted at math.stackexchange.com/a/4032652/96384)
This is a Q&A style answer not meant to be the final answer to the question. It completes the original technique for future readers. Thanks to Dietrich Burde for the motivation to continue with it.
As above, suppose $A$ is an isomorphism from $\mathfrak{su}(2) \to \mathfrak{sl}(2, \mathbb{R})$. Then $$[AX, AY] = A[X, Y]$$
Let $A_i$ denote the column vectors of $A$. Then $$Ae_i = \sum_j A_{ij} f_j$$ We use $[Ae_1, Ae_2] = A[e_1, e_2]$ to obtain $$2\begin{vmatrix} A_{11} & A_{21} \\ A_{12} & A_{22} \end{vmatrix}f_2 + 2\begin{vmatrix} A_{11} & A_{21} \\ A_{13} & A_{23} \end{vmatrix}(-f_3) + \begin{vmatrix} A_{12} & A_{22} \\ A_{13} & A_{23} \end{vmatrix}f_1 = 2 (A_{31}f_1 + A_{32}f_2 + A_{33}f_3)$$ Combining these three implied equations with the cofactor expansion of the determinant: $$\begin{vmatrix} A_{11} & A_{21} & A_{31} \\ A_{12} & A_{22} & A_{32} \\ A_{13} & A_{23} & A_{33} \end{vmatrix} = A_{31}\begin{vmatrix} A_{12} & A_{22} \\ A_{13} & A_{23} \end{vmatrix} - A_{32}\begin{vmatrix} A_{11} & A_{21} \\ A_{13} & A_{23} \end{vmatrix} + A_{33}\begin{vmatrix} A_{11} & A_{21} \\ A_{12} & A_{22} \end{vmatrix} $$ we obtain: $$\det(A) = 2 A_{31}^2 + A_{32}A_{33} + A_{33}A_{32}$$ Using the other two commutivity relations we get: $$\det(A) = 2 A_{11}^2 + 2 A_{12} A_{13}$$ $$\det(A) = 2 A_{21}^2 + 2 A_{22} A_{23}$$