Using Linux Libertine fonts for math without newtxmath

The right order to load those packages is amsthm, newtxmath, mathspec

\usepackage{amsthm}
\usepackage[libertine]{newtxmath}
\usepackage{mathspec}

You don't need to load amsmath since it is already loaded by newtxmath. That's also the reason why mathspec complains when you load newtxmath after it.

Moreover, note that both amsthm and newtxmath define the \openbox command. But, while the former defines it through a \newcommand, the latter does it through a \DeclareRobustCommand, so it does not complain...