Different macro behavior when used in another macro

You can check (ignoring spaces) whether the first token in the argument of \outerbox is \innerbox and, in this case, insert a negative space for compensation.

At the end you can recursively remove all glue nodes.

\documentclass{article}

\newcommand{\mymarginwidth}{3em}
\newcommand{\mymargin}{\hspace*{\mymarginwidth}}
\newcommand{\negmymargin}{\hspace*{-\mymarginwidth}}

\newcommand{\innerbox}[1]{\mymargin\fbox{#1}\mymargin}
\makeatletter
\newcommand{\outerbox}[1]{%
  \fbox{%
    \@ifnextchar\innerbox{\negmymargin}{}#1%
    \forever@unskip
  }%
}
\newcommand{\forever@unskip}{%
  \ifnum\lastnodetype=11
    \expandafter\unskip\expandafter\forever@unskip
  \fi
}
\makeatother

\begin{document}

test \innerbox{Hello} test

test \outerbox{\innerbox{Hello} test \innerbox{Hello} test \innerbox{Hello}}

test \outerbox{\fbox{Hello}\mymargin{} test \innerbox{Hello} test \mymargin\fbox{Hello}}

test \outerbox{test \innerbox{Hello} test \innerbox{Hello}}

test \outerbox{\innerbox{Hello} test \innerbox{Hello} test}

test \outerbox{ test \innerbox{Hello} test \innerbox{Hello} }

test \outerbox{ \innerbox{Hello} test \innerbox{Hello} test }

\end{document}

This version omits the space if \innerbox is in any hbox, not just \outerbox

\documentclass{article}

\newcommand{\mymargin}{\hspace*{3em}}
\newcommand{\innerbox}[1]{\ifhmode\ifinner\else\mymargin\fi\fi\fbox{#1}\ifhmode\ifinner\else\mymargin\fi\fi}
\newcommand{\outerbox}[1]{\fbox{#1}}

\begin{document}

test \innerbox{Hello} test% OK

test \outerbox{\innerbox{Hello} test \innerbox{Hello} test \innerbox{Hello}}%   I don't want space at beginning and end of box

test \outerbox{\fbox{Hello}\mymargin{} test \innerbox{Hello} test \mymargin\fbox{Hello}}%   what i'd like

\end{document}

egreg notes that you may not be what you wanted, so..

\documentclass{article}

\newcommand{\mymargin}{\hspace*{3em}}
\newcommand{\innerbox}[1]{\ifhmode\ifnum\lastpenalty=-1\else\mymargin\fi\fi\fbox{#1}\mymargin}
\newcommand{\outerbox}[1]{\fbox{\penalty-1#1\unskip\unskip}}

\begin{document}

test \innerbox{Hello} test% OK

test \outerbox{\innerbox{Hello} test \innerbox{Hello} test \innerbox{Hello}}%   I don't want space at beginning and end of box

test \outerbox{\fbox{Hello}\mymargin{} test \innerbox{Hello} test \mymargin\fbox{Hello}}%   what i'd like

\end{document}