Which methods of optimization are possible with this piece of LaTeX?

Keeping the documentation of amsmath package next to your monitor will go a long way for you, if you are using lot of equations.

BTW you can use align* and split the equation in to different lines.

\documentclass{article}
\usepackage{amsmath}   

\begin{document}
 \begin{align*}
 \left(\sqrt{2}+\sqrt{3}\right)^2 
            &=\left(\sqrt{2}\right)^2+2\times\sqrt{2}\times\sqrt{
                3}+\left(\sqrt{3}\right)^2\\
            &=2+2\sqrt{6}+3\\
            &=5+2\sqrt{6}
\end{align*}
\end{document}

Edit-1 As noted by Gonzalo in his comments, use of \left and \right gives slightly bigger delimiters. Instead, one can use \bigl and \bigr. But it is to be noted that these are not dynamic in nature (i.e., the size of parenthesis stays same).

\documentclass{article} 
\usepackage{amsmath} 
\begin{document} 
\begin{align*} 
\bigl(\sqrt{2}+\sqrt{3}\bigr)^2
            &=\bigl(\sqrt{2}\bigr)^2+2\times\sqrt{2}\times\sqrt{
                3}+\bigl(\sqrt{3}\bigr)^2\\
            &=2+2\sqrt{6}+3\\
            &=5+2\sqrt{6}
\end{align*} 
\end{document}

Edit-2 As per the comments of Peter Grill and Mico, You may opt to remove \times and add a thinspace \, after the sqrt, just before the closing parenthesis as in:

\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{align*}
\bigl(\sqrt{2}+\sqrt{3}\,\bigr)^2
            &=\bigl(\sqrt{2}\,\bigr)^2+2\sqrt{2}\sqrt{
                3}+\bigl(\sqrt{3}\,\bigr)^2\\
            &=2+2\sqrt{6}+3\\
            &=5+2\sqrt{6}
\end{align*}
\end{document}

There is no need for larger parentheses in such expressions. Only a few strategically placed \, will give room and make the expression perfectly readable.

The places to look out for are the ends of radicals. If they are next to another "ordinary" object (not a relation or operation symbol), it's usually correct to insert the thin space \,. For instance, notice the difference in

\sqrt{2}\,x \quad x\sqrt{2}

What about parentheses? Larger ones are not needed, in general; compare

(\sqrt{2}\,)^{2} \quad \bigl(\sqrt{2}\,\bigr)^{2}

The left one is perfectly readable, while the one on the right has the exponent too high and the vinculum clashes with the parenthesis even with the thin space. Here is the version with \left and \right, which is clearly wrong:

\left(\sqrt{2}\,\right)^{2}

gives

and this should be enough to discard it. Use \left and \right only when really needed.

Your mathematical reasoning can be typeset with the help of the align* environment:

\begin{align*}
(\sqrt{2}+\sqrt{3}\,)^{2}
&=(\sqrt{2}\,)^{2}+2\sqrt{2}\,\sqrt{3}+(\sqrt{3}\,)^{2}\\
&=2+2\sqrt{6}+3\\
&=5+2\sqrt{6}
\end{align*}

Notice that there's no \, after \sqrt{6} in the second line, because it precedes a binary operation.