Typesetting the square of a mathematical operator

The reason for the difference is that TeX typesets superscripts differently whether it follows a character or a box, as described in rule 18a of Appendix G of The TeXbook. As the macro \operatorname boxes its contents (because it calls \mathop which does), that's why \operatorname{A}^2 and \operatorname{A^2} differ (the first superscript concerns a box, whereas the second only the preceding A). You can easily see that an \operatorname and an \hbox behave similarly:

\documentclass{article}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{xcolor}

\begin{document}

\begin{tabular}{ccc}
\scalebox{5}{$\operatorname{A}^2$} & \scalebox{5}{$\hbox{A}^2$} & \scalebox{5}{$\operatorname{A^2}$} \\
\verb"$\operatorname{A}^2$" & \verb"$\hbox{A}^2$" & \verb"$\operatorname{A^2}$" \\
\end{tabular}

\raisebox{1.22cm}[0pt]{\color{red}\rule{\textwidth}{0.4pt}}

\end{document}

---

Here are the technical details of the actual computations made by TeX in the present case:

\documentclass[a4paper]{article}

\usepackage{graphicx}
\usepackage{xcolor}
\usepackage{geometry}

\begin{document}

\setbox0=\hbox{$a$}% to initialize the maths fonts

\begingroup
\newdimen\h
\newdimen\q
\newdimen\boxedu
\newdimen\unboxedu
\newdimen\sigmafourteen
\newdimen\sigmafive
\q=\the\fontdimen18\scriptfont2
\sigmafourteen=\the\fontdimen14\textfont2
\sigmafive=\the\fontdimen5\textfont2
\def\tabularheading{\itshape\color{red!70!black}}

\noindent List of relevant font parameters and their values:
\begin{quote}
\begin{tabular}{lll}
\tabularheading Name & \tabularheading Symbol & \tabularheading Value \\
\texttt{x\_height} & $\sigma_5$ & \the\sigmafive \\
\texttt{sup2}      & $\sigma_{14}$ & \the\sigmafourteen \\
\texttt{sup\_drop} & $q$ (it's $\sigma_{18}$ of superscript font) & \the\q \\
\end{tabular}
\end{quote}
Comparison of the amount the superscript is shifted up for a boxed and unboxed $A$:
\begin{quote}
\setbox0=\hbox{$A$}
\h=\the\ht0
\def\maxof#1#2{%
  \ifdim#1>#2%
      #1%
    \else
      #2%
    \fi}
\begin{tabular}{lll}
& \tabularheading Boxed $A$ & \tabularheading Unboxed $A$ \\
\tabularheading height $h$ & \the\h & \the\h \\
\tabularheading base superscript shift $u_0$ & $h-q = \mathrm{\the\dimexpr\h-\q\relax}$ & 0pt \\
\tabularheading real shift $u = \max(u_0,\sigma_{14},\frac{1}{4}\sigma_5)$ &
\boxedu=\dimexpr\h-\q\relax
\boxedu=\maxof{\boxedu}{\sigmafourteen}%
\global\boxedu=\maxof{\boxedu}{.25\sigmafive}%
\the\boxedu
&
\unboxedu=0pt
\unboxedu=\maxof{\unboxedu}{\sigmafourteen}%
\global\unboxedu=\maxof{\unboxedu}{.25\sigmafive}%
\the\unboxedu
\end{tabular}
\end{quote}
Comparision of the calculations with the real typesetting:
\begin{quote}
\begin{tabular}{cc}
\scalebox{5}{$\hbox{$A$}^2$\hbox{$A$\raise\boxedu\hbox{$\scriptstyle2$}}} & \scalebox{5}{$A^2$\hbox{$A$\raise\unboxedu\hbox{$\scriptstyle2$}}} \\
\tabularheading boxed $A$ & \tabularheading unboxed $A$ \\
\end{tabular}

\raisebox{1.35cm}[0pt]{\color{blue}\rule{9.5cm}{0.4pt}}
\end{quote}
\endgroup

\end{document}

Here's a fairly detailed explanation of what goes on in the execution of an \operatorname instruction. Note that this explanation is simplified to the case of the use of this command without the * ("star") qualifier. (See amsopn.sty for the full details.)

The \operatorname instruction (without the "star" qualifier) is set up as

\DeclareRobustCommand{\operatorname}{{\qopname\newmcodes@ o}}

where \qopname, in turn, is defined as

\DeclareRobustCommand{\qopname}[3]{%
    \mathop{#1\kern\z@\operator@font#3}%
    \csname n#2limits@\endcsname},

\operator@font is given by

\def\operator@font{\mathgroup\symoperators},

and \newmcodes@ is given -- inside a TeX group for which " has catcode 12 -- by

\gdef\newmcodes@{\mathcode`\'39\mathcode`\*42\mathcode`\."613A%
  \ifnum\mathcode`\-=45 \else
    \mathchardef\std@minus\mathcode`\-\relax
  \fi
  \mathcode`\-45\mathcode`\/47\mathcode`\:"603A\relax} 

(Basically, the \newmcodes@ command modifies the meanings to the characters ' * . - / and : from their "regular" math-mode settings.) Finally, the command \z@ is equivalent to 0pt (zero length).

Hence, executing the command \operatorname{xyz} is equivalent to executing

{\qopname\newmcodes@ o xyz}

which boils down to executing, after (i) recognizing that none of the special characters affected by the \newmodes@ command are involved in the current example, (ii) resolving the construct in the \csname ... \endcsname complex to \nolimits, and (iii) noting that \nolimits has no effects if we don't specify limits:

{\mathop{\kern0pt \operator@font xyz}

Therefore, $\operatorname{A}^2$ resolves to

${\mathop{\kern0pt \operator@font A}^2$

whereas $\operatorname{A^2}$ resolves to

${\mathop{\kern0pt \operator@font A^2}$

If the "squaring instruction" is inside the \mathop instruction, it appears that the height of the letter(s) that precede the superscript-2 do not affect the vertical positioning of the 2. E.g., check out the positions of the 2 glyph in

$\mathop{\kern0pt \operator@font ln^2}$
$\mathop{\kern0pt \operator@font sin^2}$
$\mathop{\kern0pt \operator@font cos}^2$`

They're all the same.

Conversely, if the "squaring instruction" is not inside the \mathop instruction, what comes into play is the height of the entire box that contains the "name" part of the \operatorname instruction; if the "name" part contains letters with ascenders, the box's height increases, and this will affect the positioning of the superscript-2. E.g., for $\ln^2$, $\det^2$, and $\cos^2$, the superscript is at different heights because of the differences in the heights of the boxes that contain ln, sin, and cos, respectively.

You got some great answers explaining the TeXnicalities (and thus answering your question). I'd like to point out that you should never use \(\operatorname{A^{2}}\), and that you probably just want \(A^2\):

If you have some mathematical operator, then you can use the variable A to denote that operator. In that case you should just use A^2. Only for special (non-variable) operators one should use \operatorname, e.g. \operatorname{E} for the expected value. (In this example it happens that \operatorname{E}^{2} doesn't really make sense, but you'd always put the square outside the \operatorname.)