How to write a macro that is braces sensitive?

Fundamentally you just need to use \futurelet as you do for any other look ahead

\def\foo{\futurelet\footoken\fooaux}
\def\fooaux{%
  \ifx\footoken\bgroup
     % Brace group
   \else
     % Something else
   \fi
}

The only reason this 'looks different' to other peek ahead situations is that you can't use an explicit {, but rather the implicit token \bgroup.

You can use \futurelet

\let\leftbracechar={
\def\foo{%
  \begingroup
  \futurelet\footemp\innerfoo
}%
\def\innerfoo{%
  \expandafter\endgroup
  \ifx\footemp\leftbracechar
  \expandafter\fooatleftbrace
  \else
  \expandafter\fooatnoleftbrace
  \fi
}%
\def\fooatleftbrace#1{Argument in braces is: {\bf #1}}
\def\fooatnoleftbrace#1{Argument without braces is: {\bf #1}}


\foo a

\foo{a}

\bye

, but be aware that this can be confused by implicit characters, i.e., by things like \foo\bgroup huh?...

Besides this, the check is only about tokens (be they explicit or implicit character tokens) where the category code is 1 (begin group) and the character code equals the character code of the curly-opening-brace-character. The check does not work out with character tokens where the category code is 1 (begin group) but the character code is different.

But you can implement a full expandable check which tells you whether the first token inside a macro argument is an explicit character token of category code 1 (begin group) no matter what its character code might be:

%%-----------------------------------------------------------------------------
%% Check whether argument's first token is an explicit catcode-1-character
%%.............................................................................
%% \UDCheckWhetherBrace{<Argument which is to be checked>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked has leading
%%                       catcode-1-token>}%
%%                     {<Tokens to be delivered in case that argument
%%                       which is to be checked has no leading
%%                       catcode-1-token>}%
\long\def\firstoftwo#1#2{#1}%
\long\def\secondoftwo#1#2{#2}%
\long\def\UDCheckWhetherBrace#1{%
  \romannumeral0\expandafter\secondoftwo\expandafter{\expandafter{%
  \string#1.}\expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
  \firstoftwo}{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
}%

\UDCheckWhetherBrace{Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{{}Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{{Test}}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\leavevmode\hrulefill\null

% Now let's have some fun: Give [ the same functionality as {:
\catcode`\[=\the\catcode`\{

\UDCheckWhetherBrace{Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{[}Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{[Test}}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\leavevmode\hrulefill\null

% Now let's see that the test on explicit characters is not fooled by implicit characters:
\let\bgroup={

\UDCheckWhetherBrace{Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{\bgroup\egroup Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{\bgroup Test\egroup}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\leavevmode\hrulefill\null    

% The test is also not fooled by implicit active characters:
\catcode`\X=13
\let X={

\UDCheckWhetherBrace{Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{X\egroup Test}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%

\UDCheckWhetherBrace{X Test\egroup}%
                    {The first token of the arg is an explicit catcode 1 char.}%
                    {The first token of the arg is not an explicit catcode 1 char.}%


\bye

---

In order to see how \UDCheckWhetherBrace works, let's write it with different line-breaking and different indentation:

The gist is: Have the argument's first token hit by \string and use TeX's catching of brace-balanced arguments for finding out whether an opening brace or something else was neutralized/was turned into a catcode-12-sequence:

\long\def\UDCheckWhetherBrace#1{%
  \romannumeral0%
  \expandafter
  \secondoftwo%← This is the interesting \secondoftwo.
  \expandafter{%← This is the interesting \secondoftwo's first argument's opening brace.
  \expandafter{%
  \string#1.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
             %  , which itself must be brace-balanced, had an opening-brace as first token.
  \expandafter\firstoftwo\expandafter{\expandafter
  \secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
  \firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
              %  did not have an opening-brace as first token
  {\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
}%

For example

\UDCheckWhetherBrace{test}{brace}{no brace}%

yields:

\romannumeral0%
\expandafter
\secondoftwo%← This is the interesting \secondoftwo.
\expandafter{%← This is the interesting \secondoftwo's first argument's opening brace.
\expandafter{%
\string test.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
              %  , which itself must be brace-balanced, had an opening-brace as first token.
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter
\secondoftwo%← This is the interesting \secondoftwo.
\expandafter{%← This is the interesting \secondoftwo's first argument's opening brace.
\expandafter{%
\string test.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
              %  , which itself must be brace-balanced, had an opening-brace as first token.
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields carrying out the \expandafter-chain and \string and thus stringifying "t" from the phrase "test":

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\secondoftwo%← This is the interesting \secondoftwo.
{%← This is the interesting \secondoftwo's first argument's opening brace.
{%
⟨character t, due to \string now of category code 12 (other)⟩est.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument
                                                                  %  , which itself must be brace-balanced, had an opening-brace as first token.
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields carrying out the interesting \secondoftwo:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter\expandafter\firstoftwo{ }{}\secondoftwo%
{brace}{no brace}%

yields carrying out \expandafter and \firstoftwo and thus delivering the space that was inside braces:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter⟨space token⟩\secondoftwo%
{brace}{no brace}%

yields carrying out \expandafter and \secondoftwo:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
⟨space token⟩no brace

Now \romannumeral finds the space token and discards it and aborts gathering digits. As by now it has only found the digit "0" which does not form a positive number, it does silently not deliver any token:

%\romannumeral expansion done:
no brace

---

For example

\UDCheckWhetherBrace{{test}}{brace}{no brace}%

yields:

\romannumeral0%
\expandafter
\secondoftwo%← This is the interesting \secondoftwo.
\expandafter{%← This is the interesting \secondoftwo's first argument's opening brace.
\expandafter{%
\string{test}.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
               %  , which itself must be brace-balanced, had an opening-brace as first token.
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter
\secondoftwo%← This is the interesting \secondoftwo.
\expandafter{%← This is the interesting \secondoftwo's first argument's opening brace.
\expandafter{%
\string{test}.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
               %  , which itself must be brace-balanced, had an opening-brace as first token.
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields carrying out the \expandafter-chain and \string and thus stringifying the left curly brace/the openening curly brace from the phrase "{test}":

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\secondoftwo%← This is the interesting \secondoftwo.
{%← This is the interesting \secondoftwo's first argument's opening brace.
{%
⟨character {, due to \string now of category code 12 (other)⟩test}.}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument
                                                                    %  , which itself must be brace-balanced, had an opening-brace as first token.
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields carrying out the interesting \secondoftwo:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter\firstoftwo\expandafter{\expandafter
\secondoftwo\string}\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields carrying out the \expandafter-chain and \string:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\firstoftwo{%
\secondoftwo⟨character }, due to \string now of category code 12 (other)⟩\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo}%← This is the interesting \secondoftwo's first argument's closing brace in case the argument 
            %  did not have an opening-brace as first token
{\expandafter\expandafter\firstoftwo{ }{}\secondoftwo}%
{brace}{no brace}%

yields carrying out \firstoftwo:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\secondoftwo⟨character }, due to \string now of category code 12 (other)⟩\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo
{brace}{no brace}%

yields carrying out \secondoftwo:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter\expandafter\firstoftwo{ }{}%
\firstoftwo
{brace}{no brace}%

yields carrying out \expandafter and \firstoftwo and thus delivering the space that was inside braces:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
\expandafter⟨space token⟩\firstoftwo%
{brace}{no brace}%

yields carrying out \expandafter and \firstoftwo:

%\romannumeral expansion in progress as by now \romannumeral only found the digit 0 and searches for more digits:
⟨space token⟩brace

Now \romannumeral finds the space token and discards it and aborts gathering digits. As by now it has only found the digit "0" which does not form a positive number, it does silently not deliver any token:

%\romannumeral expansion done:
brace