Is there a way to make a symbol behave as either mathord or mathrel depending on context?

No, a relation symbol will behave as such in all contexts and there is no “look back to see what came before me”. You could do a look forward for / to see whether what follows is a relation symbol and, in this case, treat it as an ordinary one. However this method has several limitations, because it would always require / (in math mode) to be followed by something still belonging to math mode.

The specific problem, quite common when dealing with equivalence relations, can be solved in a different way:

\newcommand{\quotient}[2]{#1/{#2}}

so that the call

$\quotient{X}{\sim}$

will do what you want and will also provide a good markup for possible improvement of the typesetting (a different slash, some spacing, whatever). The trick is bracing #2, so it will be a subformula and become an ordinary atom.

Here is a possible implementation of the “look forward”, which assumes that / will always be followed by something that should be considered as an ordinary atom:

\documentclass{article}
\usepackage{amsmath}

\newcommand{\qslash}[1]{\normalslash{#1}}
\mathchardef\normalslash=\mathcode`/
\begingroup\lccode`~=`/ \lowercase{\endgroup\let~}\qslash
\AtBeginDocument{\mathcode`/=\string"8000 }

\begin{document}

$X/\sim$

\end{document}

I don't recommend it.

Here's a LuaLaTeX-based solution, which sets up a Lua function which, by being assigned the to process_input_buffer callback, operates as a pre-processor, i.e., before TeX itself does any processing. The Lua function scans all input lines and encases all instances of \sim that are preceded by / (plus, possibly, some whitespace) in curly braces, "on the fly". Encasing a math atom in curly braces changes its status to "math-ordinary", which affects its spacing relative to adjacent math-atoms.

\documentclass{article}
\usepackage{luacode}
\begin{luacode}
function slash_sim ( line )
  return ( string.gsub ( line , "/%s-\\sim", "/{\\sim}" ) )
end
luatexbase.add_to_callback ( "process_input_buffer", slash_sim, "slash_sim" )
\end{luacode}

\begin{document}
$x\sim y$ 

$U /\sim$   % no whitespace between "/" and "\sim"

$U / \sim$  % some whitespace between "/" and "\sim"

$U /{\sim}$ % status of "\sim" is set to math-ord manually 
\end{document}