Coloring maths inside a tcolorbox

As it is rather improbable that there are page breaks in the math you could simply use \color:

\documentclass{article}
\usepackage{mathspec}
\setmainfont{TeX Gyre Pagella}
\usepackage{blindtext}
\usepackage[breakable]{tcolorbox}

\begin{document}

    \begin{tcolorbox}[breakable]
      \blindtext[3]
      \addfontfeatures{Color=blue}
      \blindtext[3]
      \addfontfeatures{Color=red}
      \blindtext
      \addfontfeatures{Color=brown}
      \blindtext\color{brown}
      \[F=ma\]
      \blindtext[3]
      \addfontfeatures{Color=blue}
      \blindtext[3]
      \addfontfeatures{Color=red}
      \blindtext
      \addfontfeatures{Color=brown}
      \blindtext
    \end{tcolorbox}

\end{document}

With lualatex instead of xelatex you could use the luacolor package. This will solve both problems:

\documentclass{article}
\usepackage{unicode-math}
\setmainfont{TeX Gyre Pagella}
\setmathfont{TeX Gyre Pagella Math}
\usepackage{blindtext}
\usepackage[breakable]{tcolorbox}
\usepackage{luacolor}

\begin{document}

    \begin{tcolorbox}[breakable]
      \blindtext[3]
      \color{blue}      
      \blindtext[3]
      \color{red}
      \blindtext
      \color{brown}
      \blindtext
      \[F=ma\]
      \blindtext[3]
      \color{blue}
      \blindtext[3]
      \color{red}
      \blindtext
      \color{brown}
      \blindtext
    \end{tcolorbox}

\end{document}

With XeLaTeX, you could patch the math commands (both display and inline) to include the color command. This ensures that the color is set on the page where the math starts.

This does not work with the TeX commands $ and $$ but only with the LaTeX equivalents \(\) and \[\]. It also does not work if your math crosses a page break (but of course it will work if a new math expression starts on the next page). If you want to use it with other math-like environments (equation, align, etc) then you should patch those separately.

Note that you should set \mycolor (or use \boxcolor) before you use math, otherwise you'll get an error. Also note that this takes effect outside of the tcolorbox as well, so you should probably put \def\mycolor{black} after each tcolorbox (or do that automatically with \AtEndEnvironment).

With a small macro you can set \color and \addfontfeatures with a single command.

MWE:

\documentclass{article}
\usepackage{mathspec}
\setmainfont{Linux Libertine O}
\usepackage{etoolbox}
\pretocmd{\[}{\color{\mycolor}}{}{}
\pretocmd{\(}{\color{\mycolor}}{}{}
\newcommand{\boxcolor}[1]{\def\mycolor{#1}\addfontfeatures{Color=\mycolor}}
\usepackage{blindtext}
\usepackage[breakable]{tcolorbox}

\begin{document}

    \begin{tcolorbox}[breakable]
      \blindtext[3]
      \boxcolor{red}
      \blindtext[3]
      \boxcolor{orange}
      \blindtext[3]
      \[F=m\times a\]
       \blindtext[3]
      \boxcolor{blue}
      \blindtext[3]
      \(1+1=2\)
      \boxcolor{red}
      \blindtext
      \boxcolor{brown}
      \blindtext
    \end{tcolorbox}

\end{document}

Result: