Avoid specifying font family in PGF export of matplotlib figure

I cannot offer a solution but a workaround building on @samcarter's comment: You can redefine \sffamily locally, e.g.:

\documentclass{article}

\usepackage{pgf}
\usepackage{fontspec}
\setmainfont{DejaVu Serif}
\setsansfont{DejaVu Sans}
\setmonofont{DejaVu Sans Mono}

\begin{document}
Lorem ipsum {\sffamily Lorem ipsum}
\begin{center}
    \renewcommand\sffamily{}
    \input{fig.pgf}
\end{center}
Lorem ipsum {\sffamily Lorem ipsum}
\end{document}

Instead of center you can use any environment or \begingroup and \endgroup.

Building on https://matplotlib.org/users/pgf.html#font-specification you could use:

import matplotlib as mpl
import matplotlib.pyplot as plt

pgf_with_rc_fonts = {
    "font.family": "serif",
}
mpl.rcParams.update(pgf_with_rc_fonts)


fig = plt.figure()
plt.xlabel('a label')
fig.savefig('fig.pgf')

This way \rmfamily is used instead of \sffamily.

There is another workaround by replacing the font specification with sed before importing the pgf file in your tex-document.

\documentclass{article}
\usepackage{pgf}
\usepackage{pgfplots}
\pgfplotsset{compat=1.8}
\usepackage{filecontents}

\begin{document}
\begin{figure}
    \begin{filecontents*}{tmpfile.sed}
# sed command file
s/\\color{textcolor}\\sffamily\\fontsize{.*}{.*}\\selectfont //\end{filecontents*}
    \immediate\write18{sed -i -f tmpfile.sed yourplot.pgf}
    \import{yourplot.pgf}
\end{figure}
\end{document}