Condition "if within another command" in LaTeX

You can use the \newif command to define a new (very low-level) conditional and commands that alter the conditional test:

\documentclass{article}

\newif\ifinsideaaa

\newcommand{\aaa}[1]{%
    \insideaaatrue
    #1%
    \insideaaafalse
}
\newcommand{\bbb}[1]{
    \ifinsideaaa
    {\itshape #1}%
    \else
    {\bfseries #1}%
    \fi
}

\begin{document}

\bbb{Some text outside of \texttt{\string\aaa}}

\aaa{\bbb{Some text inside of \texttt{\string\aaa}}}

\end{document}

The new commands defined by \newif\if<foo> are:

• \if<foo>: the conditional check to be used with \if<foo> ... \else ... \fi,
• \<foo>true: the command that sets the condition to true, and
• \<foo>false: the command that sets the condition to false.

As a side note, you should use \itshape and \bfseries instead of \it and \bf in LaTeX. The latter are outdated.

EDITED to reflect helpful insight from Jonas that the original method (setting 0/1) could be spoofed with repeated invocations of \bbb inside of \aaa. Thus I converted over to an analogous counter approach, such that the counter value reflects the depth of nesting.

\documentclass{article}

\newcounter{inaaa}
\newcommand{\aaa}[1]{\stepcounter{inaaa}#1\addtocounter{inaaa}{-1}}
\newcommand{\bbb}[1]{%
  \ifnum\value{inaaa}=0\textbf{#1}\else\relax\textit{#1}\fi}
\begin{document}
\bbb{sometext}

\aaa{\bbb{sometext} text}

\aaa{\aaa{\bbb{foo}}\bbb{bar}}
\end{document}

I tend to avoid the TeX primitives \if.. whenever avoiding them is possible.

\documentclass{article}

\makeatletter
\newcommand\My@CheckWhetherinAAA{}
\newcommand\My@SetInAAATrue{\let\My@CheckWhetherinAAA=\@firstoftwo}
\newcommand\My@SetInAAAFalse{\let\My@CheckWhetherinAAA=\@secondoftwo}
\global\My@SetInAAAFalse

\newcommand{\aaa}[1]{%
  \My@CheckWhetherinAAA{#1}%
                       {\My@SetInAAATrue#1\My@SetInAAAFalse}%
}%

\newcommand{\bbb}[1]{%
  \My@CheckWhetherinAAA{\textit}%
                       {\textbf}%
  {#1}%
}
%
% The result of the following is the same but LaTeX has to shuffle
% around more tokens:
%
% \newcommand{\bbb}[1]{%
%   \My@CheckWhetherinAAA{\textit{#1}}%
%                        {\textbf{#1}}%
%   }%
%
\makeatother


\begin{document}
\bbb{some text}

\aaa{\bbb{some text}}

\aaa{\aaa{\bbb{some text}} \aaa{\bbb{some more text}}}

\begingroup \aaa{\aaa{\bbb{some text}} \endgroup\aaa{\bbb{some more text}}}

\bbb{some text}

\end{document}

With this approach, the tokens forming the argument of \aaa get doubled and gobbled.

This can be avoided by defining another helper-macro:

\documentclass{article}

\makeatletter
\newcommand\My@CheckWhetherinAAA{}
\newcommand\My@SetInAAATrue{\let\My@CheckWhetherinAAA=\@firstoftwo}
\newcommand\My@SetInAAAFalse{\let\My@CheckWhetherinAAA=\@secondoftwo}
\newcommand\My@SetAAAconditions[1]{\My@SetInAAATrue#1\My@SetInAAAFalse}%
\global\My@SetInAAAFalse

\newcommand{\aaa}[1]{%
  \My@CheckWhetherinAAA{\@firstofone}%
                       {\My@SetAAAconditions}%
                       {#1}%
}%

\newcommand{\bbb}[1]{%
  \My@CheckWhetherinAAA{\textit}%
                       {\textbf}%
  {#1}%
}
%
% The result of the following is the same but LaTeX has to shuffle
% around more tokens:
%
% \newcommand{\bbb}[1]{%
%   \My@CheckWhetherinAAA{\textit{#1}}%
%                        {\textbf{#1}}%
%   }%
%
\makeatother


\begin{document}
\bbb{some text}

\aaa{\bbb{some text}}

\aaa{\aaa{\bbb{some text}} \aaa{\bbb{some more text}}}

\begingroup \aaa{\aaa{\bbb{some text}} \endgroup\aaa{\bbb{some more text}}}

\bbb{some text}

\end{document}