\text vs \scalebox

When \medcup is used in a display equation, it normally produces a scaled version of a text style \bigcup. What is happening in your first equation, however, is that a scaled version of the display style \bigcup is used instead because \text causes a \displaystyle command to be inserted in $\bigcup$.

The \text command is designed so that any equations in its argument are typeset in the current math style, which is \displaystyle in this case. Its definition contains a \mathcoice, whose \displaystyle branch is effectively given by

\hbox{{\everymath{\displaystyle}\let\f@size\f@size\selectfont #1}}

Because this sets \everymath={\displaystyle}, \displaystyle will be inserted at the start of every equation inside the argument of \text. This includes the first $\bigcup$ in the definition of \medcup. Your first equation therefore effectively becomes

\[
  \hbox{Inside \cs{text}: 
    $\displaystyle
      \operatornamewithlimits{
        \vcenter{\hbox{{\scalebox{1.2}{$\displaystyle\bigcup$}}}}
      }_{n = 1}^{\infty}
    $
  }
\]

and thus produces a scaled version of the display style version of \bigcup, rather than the text style version.


To ensure that the text style version of \bigcup is always used, you can explicitly insert a \textstyle in the definition of \medcup:

\newcommand*\medcup{%
  \operatorname*{%
    \mathchoice{\vcenter{\hbox{\scalebox{1.2}{$\textstyle\bigcup$}}}}%
    {\bigcup}%
    {\bigcup}%
    {\bigcup}%
  }%
}

(I also replaced the deprecated \operatnamewithlimits by \operatorname* and simplified the \textstyle branch, as suggested by egreg in this comment.)


It has to do with the fact that \[\text{$\mathchoice{D}{T}{S}{s}$}\] produces Displaystyle and not Textstyle.

Thus, adding \textstyle in the definition is needed to overcome this behavior.

\documentclass[a4paper]{memoir}
\usepackage{amsmath}%,scalerel}
\usepackage{graphicx}

\def\medcup{%
  \operatornamewithlimits{%
    \mathchoice{\vcenter{\hbox{\scalebox{1.2}{$\bigcup$}}}}
    {\vcenter{\hbox{$\bigcup$}}}%
    {\bigcup}%
    {\bigcup}%
  }%
}

\begin{document}

\[
  \text{Inside \cs{text}: $\textstyle \medcup_{n = 1}^{\infty} $}
\]

\[
  \text{Not inside \cs{text}: }  \medcup_{n = 1}^{\infty}
\]

\[\text{$\mathchoice{D}{T}{S}{s}$}\]
\[\bigcup\scalebox{1.2}{$\bigcup$}\text{$\scalebox{1.2}{$\bigcup$}$}\]
\[\bigcup\scalebox{1.2}{$\bigcup$}\text{$\scalebox{1.2}{$\textstyle\bigcup$}$}\]

\def\medcup{%
  \operatornamewithlimits{%
    \mathchoice{\vcenter{\hbox{\scalebox{1.2}{$\textstyle\bigcup$}}}}
    {\vcenter{\hbox{$\textstyle\bigcup$}}}%
    {\bigcup}%
    {\bigcup}%
  }%
}

\[
  \text{Inside \cs{text}: $\textstyle \medcup_{n = 1}^{\infty} $}
\]

\[
  \text{Not inside \cs{text}: }  \medcup_{n = 1}^{\infty}
\]
\end{document}

enter image description here

The first two lines of output are from the OP's original definition. Then I demonstrate the display/text-style issue with the "D" line.

Then I show on the next line how the 3rd bigcup should be the same as the 2nd but isn't because of this behavior.

The next line fixes that by explicitly invoking \textstyle.

Finally, I change the OP's definition, explicitly adding \textstyle, to fix the overall issue.