How to define a macro inside a macro via \if?

In your second example, when the \ifnum takes the first branch, TeX executes \def\foo\else..., which is quite problematic as we'll explain below. Things are similar when the other branch is taken: \def\bar\fi....

What happens precisely

Let's take for instance the case where the \ifnum test is true. In this case, \foo gets defined, but not the way you expected. The actual definition is:

\def\foo\else\def\bar\fi{hello}

This is a valid macro definition where the parameter text is \else\def\bar\fi, and the replacement text is hello. So, whenever \foo is expanded after this definition, TeX wants to see the tokens \else\def\bar\fi right afterwards. But in your example, the token after \foo's call is a control space \, therefore you get the error:

./faulty.tex:11: Use of \foo doesn't match its definition.
l.11 \foo\ 
           \bar
Runaway argument?

The other case defines \bar as:

\def\bar\fi{world}

which, when expanded towards the end of the TeX run, fails for the same reason (\bye isn't the same token as the expected \fi).

Remedy

A simple way to fix that is to expand the \else or \fi at the appropriate time, in order to remove the unwanted tokens from the input stream before TeX gets to execute the \def (i.e., before the \def from the chosen branch reaches TeX's stomach):

\count255=0
\def\newdef{\advance\count255 by1
  \ifnum\count255=1
    \expandafter\def\expandafter\foo
  \else
    \expandafter\def\expandafter\bar
  \fi
}
\newdef{hello}
\newdef{world}
\foo\ \bar
\bye

This is essentially the same trick as the classical one using LaTeX's \@firstoftwo and \@secondoftwo.

Expanding the \else when the condition is true removes all tokens between the \else and the matching \fi (both inclusive; note that if the condition were still undecided, a frozen \relax would be inserted). Expanding the \fi simply removes it. So, in the first case, what is left in the input stream is:

\def\foo{hello}
\newdef{world}
\foo\ \bar
\bye

and in the second case:

\def\bar{world}
\foo\ \bar
\bye

Going a little further

I said above that if you try to expand the \else token when the condition is still undecided, a frozen \relax is inserted. In this case, making this become reality only requires adding a percent sign after \ifnum\count255=1, because then TeX would continue expanding tokens after the 1 (the 〈number〉 following the = sign wouldn't be finished yet) and would therefore expand the two \expandafters before this 〈number〉 has been fully read. So, removing the end part of the code that doesn't matter here, the test code could be:

\count255=0
\def\newdef{\advance\count255 by1
  \ifnum\count255=1% bug here: the second <number> isn't finished!
    \expandafter\def\expandafter\foo
  \else
    \expandafter\def\expandafter\bar
  \fi
}
\newdef{hello}\show\foo
\bye

which shows the inserted frozen \relax inside \foo's definition:

> \foo=macro:
\relax \else \expandafter \def \expandafter \bar \fi ->hello.
l.9 \newdef{hello}\show\foo

The problem is, as described in Frougon's very fine answer, that conditionals in TeX work in quite a peculiar way: when the test is evaluated true, \else...\fi will only disappear as a result of expanding \else, which removes everything up to the matching \fi (without expansion). When the test is evaluated false, everything up to \else (or \fi) will be removed (without expansion). A dangling \fi will disappear anyway, because its expansion is void.

The method with \csname works, because expansion is performed all the way until only character tokens remain (other unexpandable tokens will throw a Missing \endcsname error), so \else and \fi will disappear as described above.

Workaround? Since you're doing definitions, expandability is not a concern and you can use Knuth's preferred construction with tail recursion and \next.

\count255=0
\def\newdef{\advance\count255 by1
  \ifnum\count255=1
    \def\next{\def\foo}%
  \else
    \def\next{\def\bar}%
  \fi
  \next
}
\newdef{hello}
\newdef{world}
\foo\ \bar

\bye