How does TeX figure out whether a `-` should be typeset as unary, or binary?

There are six basic atom types: Ord, Op, Bin, Rel, Open, Close and Punct. There are other atom types, but they're treated like Ord ones as far as spacing is concerned (a somewhat special case is Inner).

An atom of type Bin is turned into Ord if it isn't surrounded on both sides by atoms compatible with its nature. The following table shows what happens when the atom to the right of the Bin (here +) is an Ord; there are 36 possible combinations, you can try the other 30. For instance, “Ord Bin Open” is the counterpart of “Close Bin Ord”, where the Bin remains such, like also in “Close Bin Open”.

\baselineskip=1.2\baselineskip

\halign{#\unskip\hfil &\qquad #\unskip\hfil\cr
  $x+a$    & Ord Bin Ord   \cr
  $\sum+a$ & Op Bin Ord    \cr
  $++a$    & Bin Bin Ord   \cr
  $=+a$    & Rel Bin Ord   \cr
  $(+a$    & Open Bin Ord  \cr
  $)+a$    & Close Bin Ord \cr
  $,+a$    & Punct Bin Ord \cr
}

\bye

Compile with pdftex.

As you see from the picture, only in lines 1, 3 and 6 the middle + behaves like a binary operator, in the other lines it becomes an Ord.

Note also that a trailing Bin will always be treated like an Ord (say when you type \lim_{x\to0-}, because it has nothing after it, similarly to a leading one, like in $-1$.

This is not available at the macro level and is decided upon at a deep stage when TeX is interpreting a math list for later transforming it into a horizontal list.

I know this does not answer the question, but my comment turned out too long.

For a + or - to decide if it is binary or unary it must get information from what was before it. Compare 1-3 with 1^{-3} or even 1\times-3. How can you achieve that at macro level? It is impossible if only + and - are macros. Basically you would need a parser of the whole math expression. To start with, the math mode may end with $ or \) or \] or $$ or \end{equation} or \end{align} etc... thus you will have to go for something like <start of math>\PARSE{....}<end of math> input, or hack very deep into $. Of course math-engines like l3fp know how to distinguish a unary minus from a binary one, but you have to write a full parser for that, and the math expressions they handle are not quite the same as the math TeX can typeset. Thus at macro level it is very difficult, and not feasible from having only + and - converted to macros.

The math list atoms manipulated by TeX are not accessible at macro level (I don't think e-TeX \lastnodetype would help much).

However as per @HenriMenke's comment you may use LuaTeX's callbacks.