Font selection in XeTeX for specific characters

Yes, you can. But I only know a tricky method --- to use \XeTeXinterchartoks and related commands. There have been some packages, for example xeCJK and ucharclasses, that use this mechanism for multilingual support. (I wrote a few code for xeCJK.) For more information, you can read XeTeX's reference.

Set \XeTeXinterchartokenstate=1 to enable the mechanism.

There are already some predefined char classes. 0 for normal westen alphabets and symbols, 1 for CJK ideographs, 4095 for boundary (255 in older versions of XeTeX), etc. And you can use \newXeTeXintercharclass to allocate a new class.

Use \XeTeXcharclass to set the numbers to the new character class, and use \XeTeXinterchartoks to do the trick.

Full example:

\documentclass{article}
\usepackage{fontspec}
\XeTeXinterchartokenstate=1
\chardef\CharNormal=0
\makeatletter
% Test for old and new versions of the latex kernel
\ifx\e@alloc@intercharclass@top\@undefined
    \chardef\CharBound=255
\else
    \chardef\CharBound=\e@alloc@intercharclass@top
\fi
\makeatother
\newXeTeXintercharclass\CharNumbers
\XeTeXcharclass`0=\CharNumbers
\XeTeXcharclass`1=\CharNumbers
\XeTeXcharclass`2=\CharNumbers
\XeTeXcharclass`3=\CharNumbers
\XeTeXcharclass`4=\CharNumbers
\XeTeXcharclass`5=\CharNumbers
\XeTeXcharclass`6=\CharNumbers
\XeTeXcharclass`7=\CharNumbers
\XeTeXcharclass`8=\CharNumbers
\XeTeXcharclass`9=\CharNumbers
\newtoks\TokSetfont
\TokSetfont={\begingroup\fontspec{Latin Modern Mono}}
\XeTeXinterchartoks\CharNormal\CharNumbers=\TokSetfont
\XeTeXinterchartoks\CharBound\CharNumbers=\TokSetfont
\XeTeXinterchartoks\CharNumbers\CharNormal={\endgroup}
\XeTeXinterchartoks\CharNumbers\CharBound={\endgroup}
\begin{document}
0123

abc123

123abc

abc123def

123abc456
\end{document}

BTW, this does not affect math fonts.

Note Code has been updated to reflect changes in XeTeX (but not actually reflected in the documentation.) If this code fails, remove the test for XeTeX version and just use \chardef\CharBound=4095. (There was one version of the kernel that did not have \e@alloc@intercharclass@top defined, but the class was still 4095.)

I don't know if this works in LaTeX, but at least with Plain:

\font\mydigitsfont="Helvetica"
\font\mydigitssfont="Helvetica/S=7" at 7pt
\font\mydigitsssfont="Helvetica/S=5" at 5pt
\newfam\mydigitsfam
\textfont\mydigitsfam=\mydigitsfont
\scriptfont\mydigitsfam=\mydigitssfont
\scriptscriptfont\mydigitsfam=\mydigitsssfont
\XeTeXmathcode`0="0\mydigitsfam`0
\XeTeXmathcode`1="0\mydigitsfam`1
\XeTeXmathcode`2="0\mydigitsfam`2
\XeTeXmathcode`3="0\mydigitsfam`3
\XeTeXmathcode`4="0\mydigitsfam`4
\XeTeXmathcode`5="0\mydigitsfam`5
\XeTeXmathcode`6="0\mydigitsfam`6
\XeTeXmathcode`7="0\mydigitsfam`7
\XeTeXmathcode`8="0\mydigitsfam`8
\XeTeXmathcode`9="0\mydigitsfam`9
\def\mydigits{\fam\mydigitsfam\mydigitsfont}
$ 1^22^{23^3}3^1 $ 123 {\mydigits 123}
\bye

So, automatic only for mathmode.