Problem of amsart and microtype under xelatex

There is imho a bug in microtype. When fontspec is not loaded it retrieve the wrong glyph slots in some cases and this leads to errors. This is not related to amsmart. You would get the same error with article, if you use \scshape. I already made a bug report. As a work-around I would advise to load fontspec:

\documentclass{amsart}
\usepackage{fontspec}
\usepackage{microtype}

\begin{document}

 \begin{abstract}
    This is a test.
 \end{abstract}
 \end{document}

If you take a copy of

/usr/local/texlive/2017/texmf-dist/tex/latex/microtype/mt-LatinModernRoman.cfg

and save it in the directory with your document then comment out lines 192-196 so it looks like.

%    /one.oldstyle   = {100,100},
%    /two.oldstyle   = { 50, 50},
%    /three.oldstyle = { 30, 80},
%    /four.oldstyle  = { 50, 50},
%    /seven.oldstyle = { 50, 80},

Then your document runs without error.