Problems getting \expandafter\<controlsequence>\endgroup to work as expected

You should say:

\begingroup
  \def\helloworld{\newcommand\aemoon{this is the moon}}
  \expandafter
\endgroup\helloworld

The idea here is that the \expandafter causes \helloworld to be expanded before TeX sees the \endgroup, so first \helloworld gets expanded and TeX sees \endgroup\newcommand\aemoon{this is the moon}. Then it evaluates the \endgroup and \helloworld goes out of scope, then it evaluates the \newcommand and \aemoon is defined.

You probably confused this situation with the case where there's a command in a conditional that takes an argument, and you don't want it to use \fi or \else and the false clause as its argument so you say: \ifsomething \expandafter\somecommand\fi. Since \else and \fi are fully expandable and when expanded delete themselves (and in the case of \else the false clause), this fixes that problem.

The problem with saying \expandafter\helloworld\endgroup is that \endgroup is not expandable, so the result is the \expandafter expands the \endgroup, which changes it not at all -- \endgroup acts like a } here. Then what TeX sees is \mycommand\endgroup, which doesn't do what you want.

Roughly speaking, TeX has two stages of evaluation -- expansion, where it takes macros and repeatedly expands them, and evaluation, where it takes unexpandable primitives (either characters or commands like \bgroup, \relax, \def, \show etc) and evaluates them. \expandafter only interacts with the expansion stage. Many nonintuitive behaviors of TeX result from this expansion evaluation distinction.

However, even if \expandafter\helloworld\egroup did behave as you expected, it would not be right, because that would then be the same as saying \egroup\helloworld, which would cause an error because the definition of \helloworld went out of scope.

\endgroup is not expandable, which makes

\begingroup
  \def\helloworld{\newcommand\aemoon{this is the moon}}
  \expandafter\helloworld
\endgroup

equivalent to

\begingroup
  \def\helloworld{\newcommand\aemoon{this is the moon}}
  \helloworld
\endgroup

As such, \helloworld is defined, executed and then everything within the group that isn't \global does not survive. This includes \aemoon. You may consider an alternative

\begingroup
  \def\helloworld{\endgroup\newcommand\aemoon{this is the moon}}
  \helloworld

where \helloworld ends the group implicitly, leaving \aemoon still defined.

   \expandafter\helloworld\endgroup

applies \expandafter to \endgroup that is not an error but it does nothing as \endgroup is not expandable so it is equivalent to

  \helloworld\endgroup

So \helloworld acting inside the group is the expected behaviour.