Using \textbf vs. \mathbf in Math mode

Spaces are ignored in the arguments of \mathbf and \mathit, but not in the arguments of \textbf and \textit.

Also, you can't combine \mathbf and \mathit. E.g., \mathbf{\mathit{xyz}} produces xyz rather than xyz.

And, of course, the outputs of \mathbf and \textbf will differ if you use different fonts for math-mode and text-mode material.

\documentclass{article}
\begin{document}
$\textbf{abc def}$ 

$\mathbf{abc def}$

$\textit{abc def}$

$\mathit{abc def}$
\end{document}

\textbf uses the text font (Libertine in the example) and \mathbf uses the math font (computer modern) -- same happens for \textrm and \mathrm. Libertine has no math, the reason why TeX takes the default math font. And, of course, text and math font maybe looking very different.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage{libertine}
\begin{document}

$\textbf{some text}~\mathbf{some text}~some text$

\end{document}

To extend Herberts answer, the \textbf furthermore keeps the “surrounding” font style, for example when using a theorem, which typesets its content in italics, so will the \textbf will set the text bold and italic, while \mathbf will not only use the math font but furthermore only the bold font and not an italic one.

For example with

\documentclass{scrartcl}
\usepackage{amsthm}
\newtheorem{thm}{Theorem}
\begin{document}
    In normal text
    \[
        \mathbf{A}\textbf{B}
    \]
    \begin{thm}
    In theorems, or other italic texts
    \[
        \mathbf{A}\textbf{B}
    \]
    \end{thm}
\end{document}

The result is which might or might not be what you want (for example when defining own commands).