Ignore Greek letters using `\mathcal`
Since you said XeLaTeX, just load unicode-math
:
\documentclass{article}
\usepackage{amsmath}
\usepackage{fontspec}
\usepackage{unicode-math}
\setmainfont{Latin Modern Roman}
\setmathfont{Latin Modern Math}
\newcommand{\set}[1]{\mathcal{#1}}
\begin{document}
\begin{itemize}
\item $\set{A} = ...$
\item $\set{\Omega} = ...$
\end{itemize}
\end{document}
It is possible to test the slot number of each character in your argument and work with that, but this is a really simple fix.
To bypass math alphabet commands like \mathcal
you can simply make all uppercase greek letters mathord
instead of mathalpha
. If conflict arises revert the letters with \mathchardef
.
\documentclass{article}
\DeclareMathSymbol{\Gamma}{\mathord}{operators}{"00}
\DeclareMathSymbol{\Delta}{\mathord}{operators}{"01}
\DeclareMathSymbol{\Theta}{\mathord}{operators}{"02}
\DeclareMathSymbol{\Lambda}{\mathord}{operators}{"03}
\DeclareMathSymbol{\Xi}{\mathord}{operators}{"04}
\DeclareMathSymbol{\Pi}{\mathord}{operators}{"05}
\DeclareMathSymbol{\Sigma}{\mathord}{operators}{"06}
\DeclareMathSymbol{\Upsilon}{\mathord}{operators}{"07}
\DeclareMathSymbol{\Phi}{\mathord}{operators}{"08}
\DeclareMathSymbol{\Psi}{\mathord}{operators}{"09}
\DeclareMathSymbol{\Omega}{\mathord}{operators}{"0A}
\begin{document}
\[\mathcal{\Gamma\Delta\Theta\Lambda\Xi\Pi\Sigma\Upsilon\Phi\Psi\Omega}\]
\mathchardef\Gamma="7000
\mathchardef\Delta="7001
\mathchardef\Theta="7002
\mathchardef\Lambda="7003
\mathchardef\Xi="7004
\mathchardef\Pi="7005
\mathchardef\Sigma="7006
\mathchardef\Upsilon="7007
\mathchardef\Phi="7008
\mathchardef\Psi="7009
\mathchardef\Omega="700A
\[\mathcal{\Gamma\Delta\Theta\Lambda\Xi\Pi\Sigma\Upsilon\Phi\Psi\Omega}\]
\mathchardef\Gamma="00
\mathchardef\Delta="01
\mathchardef\Theta="02
\mathchardef\Lambda="03
\mathchardef\Xi="04
\mathchardef\Pi="05
\mathchardef\Sigma="06
\mathchardef\Upsilon="07
\mathchardef\Phi="08
\mathchardef\Psi="09
\mathchardef\Omega="0A
\[\mathcal{\Gamma\Delta\Theta\Lambda\Xi\Pi\Sigma\Upsilon\Phi\Psi\Omega}\]
\end{document}