Angle brackets in non-math-mode text

You could employ the commands \textlangle and \textrangle provided by the textcomp package. If that's too many letters to type each time, you could define shortcuts for them, say \la and \ra. The following MWE illustrates this idea, also

% !TEX TS-program = xelatex
\documentclass{article}
\usepackage{fontspec}
% \setmainfont{Linux Libertine O} % select your main text font here
\usepackage{textcomp,    % for \textlangle and \textrangle macros
            xspace}
\newcommand\la{\textlangle\xspace}  % set up short-form macros
\newcommand\ra{\textrangle\xspace}
\begin{document}
\noindent
\textlangle\ and \textrangle; \newline 
\la and \ra.   % abbreviated macros; note: no need to insert explicit whitespace after "\la"
\end{document}

fontspec doesn't have an interface for substituting some characters with characters from a different font, but thankfully egreg provided a package to do just that:

\documentclass{article}
\usepackage{fontspec}
\usepackage{newunicodechar}
\setmainfont{Linux Libertine O}

\newfontfamily\anglefont{Free Serif}    % Change to whatever font you prefer that has ⟨⟩.
\newunicodechar{⟨}{{\anglefont ⟨}}
\newunicodechar{⟩}{{\anglefont ⟩}}

\begin{document}
This is Linux Libertine with ⟨ and ⟩ from {\anglefont Free Serif}.
\end{document}