DeclareMathOperator does not respect math font

Using T1-encoding with xelatex is not a good idea. See e.g. https://tex.stackexchange.com/a/470987/2388.

Changing the order works for me:

\documentclass{article}

\usepackage{amsmath}
\usepackage[libertine]{newtxmath}
\usepackage{libertine}

\DeclareMathOperator*{\argmin}{arg\,min}

\begin{document}

This is some text. arg min

\begin{equation*}
  x = y + \sum_{z \in \mathcal{Z}} \beta_z
\end{equation*}

\begin{equation*}
  x = \argmin_{y \in Y} \bigg \{ f(y) + \frac{1}{g(y) - h(y)} \bigg\}
\end{equation*}

\end{document}

If you get an error, show your log-file.