How is beamer's {frame} title optional, yet in braces (not brackets)?

The bottom line is that this is a design decision. To understand what is going on, we need to look at the technical detail.

When Leslie Lamport wrote LaTeX, he decided that as a user having a difference between optional and mandatory arguments was a 'good thing'. He did that by using the { ... } pair for mandatory arguments and the [ ... ] pair for optional ones. However, other people writing LaTeX code can use other conventions.

In LaTeX, the characters [ and ] have TeX category codes 'other'. In contrast to { and }, which are 'begin group' and 'end group', respectively. TeX does brace matching when grabbing an argument inside { ... } (a LaTeX mandatory argument) as the rules for 'begin'/'end' group tokens are clear: they must balance. On the other hand, TeX does not do that for [ ... ] as they are just 'other' tokens. LaTeX2e implements optional arguments using some look-ahead code (to find a [) and what is called a delimited argument

\def\foo[#1]{% Code here

When TeX looks for an argument surrounded by tokens in this way, it does it in a 'lazy' sense. Thus

\foo[[bar]]

will grab [bar as #1 and leave the second ] 'dangling'.

It is possible to add additional code to do matching of [ ... ] tokens: xparse does this. The 'cost' (lines of code) is too high to have been realistic when LaTeX2e was released but is now perfectly acceptable. Thus commands produced using xparse do not have the issue with nested square brackets.

In the beamer case, Till Tantau uses some look-ahead code for { rather than [. It's slightly more tricky to do this, but it's is quite workable if commands do not need to be 'expandable'. There are a few minor issues with verbatim slides (the look-ahead is a bit of a pain!), but other than that the reason to not favour Till's approach is that it's not at all obvious reading the code which arguments are required.

It is only a hint, but the mechanism of using the same kind of braces in different meanings is well-known in basic LaTeX. In

\begin{picture}(a,b)(c,d)

(a,b) are mandatory, (c,d) optional. The trick is the following:

\long\gdef\picture#1{\pictur@#1}
\gdef\pictur@(#1){%
  \@ifnextchar({\@picture(#1)}{\@picture(#1)(0,0)}}

The construction in beamerbaseframe.sty is similar:

  \def\beamer@checkframetitle{\@ifnextchar\bgroup\beamer@inlineframetitle{}}
  \def\beamer@inlineframetitle#1{%
    \@ifnextchar\bgroup{\frametitle{#1}\framesubtitle}{\frametitle{#1}\relax}%

but ( is replaced by \bgroup.