When to use \par and when \\, \newline, or blank lines
\par is a TeX primitive and is the same as a blank line (except in special environments such as verbatim where the usual rules don't apply). It ends horizontal mode, causes TeX to break the horizontal text into lines placed on the current vertical list, and exercises the page breaker which may possibly cause the next page to be shipped out.
\\ is different in almost every respect. It is a macro not a primitive, and its definition changes wildly in almost every LaTeX definition. The definition in normal text, a center environment, a flushleft environment and a table are all different.
In normal running text when it forces a linebreak it is essentially a shorthand for \newline. This does not end horizontal mode or end the paragraph. It just inserts some glue and penalties at that point into the horizontal material, so that when the paragraph does end, a line-break will occur at that point, with the short line padded with white space.
\\at the end of a paragraph causes bad output with an empty, maximally under-full, box, and so you get a warning about badness 10000, the visual effect looks a bit like extra vertical space but it is not: it is an extra spurious line at the end of the paragraph, and for example it is not dropped at a page break and will break widow/club line calculations.
You should rarely need to use \\ in documents apart from its use in alignments (where it is a macro based on the \cr primitive), and you should rarely need \par in documents as a blank line should suffice.
In text mode, \par starts a new paragraph, while \\ ends the current line.
So when to use each one?
\\ should be used with parsimony, when you want to break of the rhythm of your current argumentation, but without jumping to a new idea / argument (which would require starting a new paragraph).
\par should actually be used even less frequently. Not that you do not want to start new paragraphs (you do want to keep paragraphs to reasonable lengths!), but you should start new paragraphs by leaving an empty line in your TeX code. This has the exact same effect as using \par, but will make your code much more readable. \par is therefore mostly used when defining macros.
Note that outside of text mode, \par will usually lead to an error, while \\ usually works as expected to start a new line (like while in an tabular or array).