Check for group as argument (expandable)
This doesn't ignore space, I think it's easier to use strcmp, I added one extra test for {a} at the end:
\documentclass{article}
\makeatletter
\def\zz#1\zz{#1}
\def\MyIfGroup#1{%
\ifnum\expandafter\pdfstrcmp\expandafter{\zz#1\zz}{#1}=0
\expandafter\@secondoftwo\else\expandafter\@firstoftwo
\fi}
\makeatother
\begin{document}
\MyIfGroup{{}}{true}{false}
\MyIfGroup{ {}}{true}{false}
\MyIfGroup{{} }{true}{false}
\MyIfGroup{ {} }{true}{false}
\MyIfGroup{ }{true}{false}
\MyIfGroup{a}{true}{false}
\MyIfGroup{{}a}{true}{false}
\MyIfGroup{{a}}{true}{false}
\end{document}
Based on the expandable loop code in expl3, but as requested using no packages:
\documentclass{article}
\makeatletter
\newcommand\MyIfGroupTF[1]{%
\MyIfGroup@loop@i#1\q@mark\q@stop
}
\long\def\MyIfGroup@loop@i#1\q@stop{%
\MyIfGroup@if@brace{#1}%
{\MyIfGroup@brace@i}
{%
\MyIfGroup@if@normal{#1}
{\MyIfGroup@end}
{\MyIfGroup@space@i}%
}%
#1\q@stop
}
\expandafter\long\expandafter\def\expandafter\MyIfGroup@space@i\space#1\q@stop{%
\MyIfGroup@loop@i#1\q@stop
}
\newcommand\MyIfGroup@brace@i[1]{\MyIfGroup@loop@ii}
\long\def\MyIfGroup@loop@ii#1\q@stop{%
\MyIfGroup@if@brace{#1}%
{\MyIfGroup@end}
{%
\MyIfGroup@if@normal{#1}
{\MyIfGroup@normal@ii}
{\MyIfGroup@space@ii}%
}%
#1\q@stop
}
\newcommand\MyIfGroup@normal@ii[1]{%
\ifx#1\q@mark
\expandafter\MyIfGroup@normal@cleanup
\else
\expandafter\MyIfGroup@end
\fi
}
\long\def\MyIfGroup@normal@cleanup#1\q@stop{\@firstoftwo}
\expandafter\long\expandafter\def\expandafter\MyIfGroup@space@ii\space#1\q@stop{%
\MyIfGroup@loop@ii#1\q@stop
}
\long\def\MyIfGroup@end#1\q@stop{\@secondoftwo}
\newcommand\MyIfGroup@if@brace[1]{%
\ifcat\expandafter\@gobble\expandafter{\expandafter{\string#1?}}**%
\expandafter\@secondoftwo
\else
\expandafter\@firstoftwo
\fi
}
\newcommand\MyIfGroup@if@normal[1]{%
\ifcat\iffalse{\fi\MyIfGroup@if@normal@aux?#1 }%
\expandafter\@gobble\expandafter{\expandafter{\string#1?}}**%
\expandafter\@firstoftwo
\else
\expandafter\@secondoftwo
\fi
}
\long\def\MyIfGroup@if@normal@aux#1 {%
\if\relax\detokenize\expandafter{\@gobble#1}\relax
^%
\fi
\expandafter\@gobble\expandafter{\iffalse}\fi
}
\newcommand*\q@mark{\q@mark}
\newcommand*\q@stop{\q@stop}
\makeatother
\begin{document}
Works: \MyIfGroupTF{{i}}{true}{false}.
Works: \MyIfGroupTF{ {}}{true}{false}.
Works: \MyIfGroupTF{ }{true}{false}.
Works: \MyIfGroupTF{ a }{true}{false}.
Works: \MyIfGroupTF{a}{true}{false}.
Works: \MyIfGroupTF{{}{}}{true}{false}.
Works: \MyIfGroupTF{{} }{true}{false}.
\end{document}
The basic idea is to use a delimited argument to allow us to test the head of the input as one of space/group/'normal', then to branch as required. (See l3tl.dtx and in particular 'Token by token changes' in the code for more.) There are two loops, the first to check we have a brace group, the second to make sure it's the only one and that there are no other tokens outside of it. One could extend the code to expand any macros, though this makes life more 'interesting'.
One could also use a version of David's answer combined with a space-trimming approach (see https://tex.stackexchange.com/a/69771):
\documentclass{article}
\makeatletter
\long\def\trim@spaces#1{%
\@@trim@spaces{\q@mark#1}%
}
\def\@tempa#1{%
\long\def\@@trim@spaces##1{%
\@@trim@spaces@i##1\q@nil\q@mark#1{}\q@mark
\@@trim@spaces@ii
\@@trim@spaces@iii
#1\q@nil
\@@trim@spaces@iv
\q@stop
}%
\long\def\@@trim@spaces@i##1\q@mark#1##2\q@mark##3{%
##3%
\@@trim@spaces@i
\q@mark
##2%
\q@mark#1{##1}%
}%
\long\def\@@trim@spaces@ii\@@trim@spaces@i\q@mark\q@mark##1{%
\@@trim@spaces@iii
##1%
}%
\long\def\@@trim@spaces@iii##1#1\q@nil##2{%
##2%
##1\q@nil
\@@trim@spaces@iii
}%
\long\def\@@trim@spaces@iv##1\q@nil##2\q@stop{%
\unexpanded\expandafter{\@gobble##1}%
}%
}
\@tempa{ }
\def\zz#1\relax{#1}
\newcommand\MyIfGroup[1]{%
\ifnum\expandafter\pdfstrcmp\expandafter
{\expandafter\zz\romannumeral-`0\trim@spaces{#1}\relax}%
{\trim@spaces{#1}}%
=0 %
\expandafter\@secondoftwo\else\expandafter\@firstoftwo
\fi}
\begin{document}
\MyIfGroup{{}}{true}{false}
\MyIfGroup{ {}}{true}{false}
\MyIfGroup{{} }{true}{false}
\MyIfGroup{ {} }{true}{false}
\MyIfGroup{ }{true}{false}
\MyIfGroup{a}{true}{false}
\MyIfGroup{{}a}{true}{false}
\MyIfGroup{{a}}{true}{false}
\end{document}
I don't understand what you exactly want, but maybe the following code can help you. (Edit: I added new version of the code which is full expandable)
\def\MyIfGroup#1{\expandafter\MyIfGroupA\romannumeral-`\.#1.{}\end}
\def\MyIfGroupA#1#{\ifx\end#1\end\expandafter\MyIfGroupB
\else\expandafter\MyIfGroupE\fi}
\def\MyIfGroupB#1#2#3\end{\ifx\end#3\end\expandafter\MyIfGroupT
\else\expandafter\MyIfGroupF\fi}
\def\MyIfGroupE#1\end{\MyIfGroupF}
\def\MyIfGroupF#1#2{#2}
\def\MyIfGroupT#1#2{#1}
\tt
\MyIfGroup{{}}{true}{false} -> true
\MyIfGroup{ {}}{true}{false} -> true
\MyIfGroup{{} }{true}{false} -> true
\MyIfGroup{ {} }{true}{false} -> true
\MyIfGroup{ }{true}{false} -> false
\MyIfGroup{a}{true}{false} -> false
\MyIfGroup{{}a}{true}{false} -> false
\MyIfGroup{ {a} }{true}{false} -> true
\MyIfGroup{ {a}{} }{true}{false} -> false
\MyIfGroup{{}{}}{true}{false} -> false
\bye
Edit: How it works: The potential first space is removed by \romannumeral-``\. The rest is read by \MyIfGroupA macro which reads a parameter #1 to the first occurrence of { (without it). If there is no such brace then the original parameter followed by dot is read because we added .{}\end to the input stream. This means that the #1 of \MyIfGroupA is not empty, so \MyIfGroupE is processed (this macro removes the rest from input stream and goes to the false branch. If thee exist { but something is before it then the result is the same. Else #1 of \MyIfGroupA is empty. Then the \MyIfGroupB is processed. It reads the (first) braced parameter to #1 then reads one token to #2 and the rest of input stream to \end in #3. The reading of #2 ignores potentially spaces and if there is nothing else the . is read to #2 and nothing is in #3 because outer braces are removed when #3 is read. And this is exactly the situation where we need to go to the true branch. Else there is something else after first braced parameter and we go to the false branch.