How to break long equations while keeping alignment?

I would suggest you make several changes. The first one will fix the error messages you're getting; the remaining ones should help make it easier for your readers to parse the material in the equations.

• You can't have a \left( ... \right) pair crossing a line break. In the first and third lines of the equation construct, replace the first \left( with \Big[, and in the second and fourth lines replace \right) with \Big].

• The remaining \left( ... \right) pairs -- I believe there are 18 of them... -- currently generate parentheses of varying sizes. This happens because not all of the expressions they enclose are large enough to trigger an increase in the size of the fences beyond the basic size. I suggest you change all 18 remaining \left( instructions to \bigl(, and all 18 remaining \right) instructions to \bigr). This will ensure consistent sizing.

• The Sp terms in the third and fourth lines are probably math operators, right? If so, they should be typeset in an upright rather than italic font.

• In the first set of two lines, the first line has 6 additive terms and the second has 3; in the second set of two lines, the lines have 5 and 4 additive terms, respectively. I suggest you use the same number of terms across the two sets.

• Use \mid instead of | in the left-hand term. This will provide a bit of spacing around the vertical bar.

• To indicate multiplication, don't use a "dot". Use either \cdot or (better yet) omit the dots entirely.

• To provide some visual grouping, it's a good idea to indent the 2nd and 4th lines a bit relative to the 1st and 3rd lines. In the example below I use \qquad to indent the two lines; if that's not enough for you, you could try \qquad\quad.

• Last but not least, since you're already using the amsmath package, use the align* environment instead of nested displaymath and aligned environments.

\documentclass[]{article}
\usepackage{amsmath}
\DeclareMathOperator{\Sp}{Sp}
\begin{document}
\begin{align*}
(\chi_1\mid \chi_1)
&=\frac{1}{24}\Big[\bigl(\chi_1(1)\bigr)^2+\bigl(\chi_1(b^2)\bigr)^2+2\bigl(\chi_1(a)\bigr)^2+2\bigl(\chi_1(a^3)\bigr)^2+2\bigl(\chi_1(a^5)\bigr)^2\\
&\qquad +2\bigl(\chi_1(a^2)\bigr)^2+2\bigl(\chi_1(a^2b^2)\bigr)^2+6\bigl(\chi_1(b)\bigr)^2+6\bigl(\chi_1(ab)\bigr)^2\Big]\\
&=\frac{1}{24}\Big[\bigl(\Sp(I_2)\bigr)^2+\bigl(\Sp(B^2)\bigr)^2+2\bigl(\Sp(A_1)\bigr)^2+2\bigl(\Sp({A_1}^3)\bigr)^2+2\bigl(\Sp({A_1}^5)\bigr)^2\\
&\qquad  +2\bigl(\Sp({A_1}^2)\bigr)^2+2\bigl(\Sp({A_1}^2B^2)\bigr)^2+6\bigl(\Sp(B)\bigr)^2+6\bigl(\Sp(A_1B)\bigr)^2\Big]\\
\end{align*}
\end{document}

You can use \phantoms to move yourself into the correct spot, and avoid bracket-complications by using the "big" alternatives:

\documentclass{article}
\usepackage{amsmath}% http://ctan.org/pkg/amsmath
\newcommand{\FaSymb}{\chi_1}\newcommand{\FbSymb}{\text{Sp}}
\newcommand{\Fa}[1]{\FaSymb(#1)}
\newcommand{\Fb}[1]{\FbSymb(#1)}
\begin{document}

\begin{align*}
  (\FaSymb \mid \FaSymb)
    & = \tfrac{1}{24}
        \Bigl(\bigl(\Fa{1}\bigr)^2
            + \bigl(\Fa{b^2}\bigr)^2
           + 2\bigl(\Fa{a}\bigr)^2
           + 2\bigl(\Fa{a^3}\bigr)^2
           + 2\bigl(\Fa{a^5}\bigr)^2
           + 2\bigl(\Fa{a^2}\bigr)^2 \\
    & \phantom{=\tfrac{1}{24}\Bigl(} + 2\bigl(\Fa{a^2b^2}\bigr)^2
      + 6\bigl(\Fa{b}\bigr)^2
      + 6\bigl(\Fa{ab}\bigr)^2\Bigr) \\
    & = \tfrac{1}{24}
        \Bigl(\bigl(\Fb{I_2}\bigr)^2
            + \bigl(\Fb{B^2}\bigr)^2
           + 2\bigl(\Fb{A_1}\bigr)^2
           + 2\bigl(\Fb{A_1^3}\bigr)^2
           + 2\bigl(\Fb{A_1^5}\bigr)^2 \\
    & \phantom{=\tfrac{1}{24}\Bigl(} + 2\bigl(\Fb{A_1^2}\bigr)^2
      + 2\bigl(\Fb{A_1^2B^2}\bigr)^2
      + 6\bigl(\Fb{B}\bigr)^2
      + 6\bigl(\Fb{A_1B}\bigr)^2\Bigr)
\end{align*}

\end{document}

I defined \Fa and \Fb to promote consistency.

Your second attempt didn't work because every \left has to be paired up with \right on the same line. To fix this, you can just use \Bigl( and \Bigr) instead. To get the alignment right, you can use the IEEEtrantools package (see Section 3.5.2):

\documentclass{article}
\usepackage{amsmath}
\usepackage[retainorgcmds]{IEEEtrantools}

\begin{document}
\begin{IEEEeqnarray*}{rClCl}
  (\chi_1|\chi_1)
  &=& \frac{1}{24}
      \Bigl(
           \left(\chi_1(1)\right)^2
        &+&\left(\chi_1(b^2)\right)^2
        +2.\left(\chi_1(a)\right)^2
        +2.\left(\chi_1(a^3)\right)^2
        +2.\left(\chi_1(a^5)\right)^2 \\
  & & &+&2.\left(\chi_1(a^2)\right)^2
        +2.\left(\chi_1(a^2b^2)\right)^2
        +6.\left(\chi_1(b)\right)^2
        +6.\left(\chi_1(ab)\right)^2
      \Bigr) \\
  &=& \frac{1}{24}
      \Bigl(
           \left(Sp(I_2)\right)^2
        &+&\left(Sp(B^2)\right)^2
        +2.\left(Sp(A_1)\right)^2
        +2.\left(Sp({A_1}^3)\right)^2
        +2.\left(Sp({A_1}^5)\right)^2 \\
  & & &+&2.\left(Sp({A_1}^2)\right)^2
        +2.\left(Sp({A_1}^2B^2)\right)^2
        +6.\left(Sp(B)\right)^2
        +6.\left(Sp(A_1B)\right)^2
      \Bigr)
\end{IEEEeqnarray*}
\end{document}