How to make \tfrac work in >{}?

You can not use \bgroup to enclose macro arguments.

As noted in comments \sqrt doesn't as it happen take its argument as a macro argument.

The last column is different because & is a TeX primitive which causes the text from <{...} to be inserted as soon as it is seen, however \\ is a macro and macros are not expanded while scanning for arguments. So you would need to use a dummy last column that is always empty so your scan always ends with & or (with some loss of functionality) use \cr primitive rather than \\. Or (best) insert \\ rather than \endgroup into the preamble with <{\\} and in the >{...} don't use \sqrt directly but use a macro using delimited arguments \foo#1\\ this delimited argument will then always pick up the content (and you have to put a real \ back if you are in the last row)

---

\documentclass[border=3pt]{standalone}
\usepackage{booktabs}
\usepackage{array}
\usepackage{amsmath}

\setlength\extrarowheight{2pt}

\def\getNum#1\relax{\gdef\num{#1}$\tfrac{1}{2}\sqrt{#1}$}

\def\getDenum#1\\{\gdef\denum{#1}$\tfrac{1}{2}\sqrt{#1}$&$\sqrt{\tfrac{\num}{\denum}}$\\}

\begin{document}

\begin{tabular}
{
    >{$}c<{^\circ$}
    >{\getNum}c
    >{\getDenum}c
    c
}
\toprule
\multicolumn{1}{c}{$\theta$}&
    \multicolumn{1}{c}{$\sin\theta$}&
        \multicolumn{1}{c}{$\cos\theta$}&
            \multicolumn{1}{c}{$\tan\theta$}\\
\midrule
0&
    0&
        4\\
30&
    1&
        3\\
45&
    2&
        2\\
60&
    3&
        1\\
90&
    4&
        0\\
\bottomrule
\end{tabular}

\end{document}

The collcell package by Martin Scharrer solves the problem:

\documentclass[border=3pt]{standalone}
\usepackage{booktabs}
\usepackage{array}
\usepackage{amsmath}
\usepackage{collcell}

\newcommand\getnumerator[1]{$\tfrac{1}{2}\sqrt{#1}$\gdef\numerator{#1}}
\newcommand\getdenominator[1]{$\tfrac{1}{2}\sqrt{#1}$\gdef\denominator{#1}}

\begin{document}

\begin{tabular}
  {
   >{$}c<{^\circ$}
   >{\collectcell\getnumerator}c<{\endcollectcell}
   >{\collectcell\getdenominator}c<{\endcollectcell}
   >{$\sqrt{\tfrac{\numerator}{\denominator}}$}c
  }
\toprule
\multicolumn{1}{c}{$\theta$}&
\multicolumn{1}{c}{$\sin\theta$}&
\multicolumn{1}{c}{$\cos\theta$}&
\multicolumn{1}{c}{$\tan\theta$}\\
\midrule
0  & 0 & 4 & \\\addlinespace
30 & 1 & 3 & \\\addlinespace
45 & 2 & 2 & \\\addlinespace
60 & 3 & 1 & \\\addlinespace
90 & 4 & 0 & \\
\bottomrule
\end{tabular}

\end{document}