What is the difference between \over and \frac?

The command \over is a so-called "primitive" command that's available in Plain TeX and in LaTeX. \frac is a LaTeX-only command that builds on the \over macro to provide something that's much less likely to throw unexpected errors if the user doesn't handle the syntax of the \over command just right.

Assuming you have the amsmath package loaded, the \frac command is defined as follows:

\DeclareRobustCommand{\frac}[2]{{\begingroup#1\endgroup\@@over#2}}

where \@@over is an amsmath-internal version of \over. Note that the first argument of \frac (the numerator) is carefully encased in a \begingroup ... \endgroup construct and that the entire command's "content" is surrounded by a double set of curly braces. The braces serve to remove potential ambiguities that might arise from a less than fully careful use of the \over command by itself.

If you're a user of LaTeX, there's no reason whatsoever for not using the \frac command -- and several good reasons for not using the \over command directly. In short, if you use LaTeX, use \frac.

The latter is preferable. It corrects a behavior that in some situations can reveal itself in a bad way. The LaTeX definition is

\def\frac#1#2{{\begingroup #1\endgroup\over #2}}

In some cases an assignment made in the numerator can affect also the denominator. It's difficult to produce it with standard LaTeX commands, but you can compare the result of

\[
\fam0 a\over b
\]

with

\[
\frac{\fam0 a}{b}
\]

In the first case both "a" and "b" will be upright, which is probably not expected.

Another relevant argument against the \over syntax is that it's foreign to LaTeX, which always uses first the command and then its argument and never an "infix" syntax. Moreover the grouping around numerator\over denominator is automatically provided, avoiding mistakes such as

1\over 2 \over 3

Update

Starting from the 2019-10-01 release of LaTeX, the definition of \frac uses \DeclareRobustCommand, as part of a program to remove as many fragile commands as possible:

% latex.ltx, line 4705:
\DeclareRobustCommand\frac[2]{{\begingroup#1\endgroup\over#2}}

In olden times it was not possible to make all commands robust, because of memory restrictions. These considerations are no longer a problem.

The amsmath package has used \DeclareRobustCommand for its modified definition of \frac for many years.

In User’s Guide for the amsmath Package p.14 it is written the following:

The primitive generalized fraction commands \over, \overwithdelims, \atop, \atopwithdelims, \above, \abovewithdelims produce warning messages if used with the amsmath package, for reasons discussed in technote.tex.

And in Technical notes on the amsmath package p.2:

Not only is the unusual syntax of the TeX primitives rather out of place in LaTeX, but furthermore that syntax seems to be responsible for one of the most significant flaws in TeX's mathematical typesetting capabilities: the fact that the current mathstyle at any given point in a math formula cannot be determined until the end of the formula, because of the possibility that a following generalized fraction command will change the mathstyle of the preceding material.

(...)

There are additional bad consequences following from the syntax of those generalized fraction commands that only become evident when you do some writing of nontrivial macros for math use. For example, as things currently stand you cannot measure the size of any object in math without going through \mathchoice and leaving and reentering math mode via \hbox{$ (which then introduces complications regarding \everymath and \mathsurround). And it seems that uncertainty about the current mathstyle is the only barrier to allowing the use of mu units with \vrule, to make vertical struts in constructing compound symbols or notation. And so on and so forth.