Define tcolorbox in math mode

You can do this using \tcboxmath from the theorems library of tcolorbox:

\documentclass{article}
\usepackage{tcolorbox}
\tcbuselibrary{theorems}

\newcommand*{\mywbox}{%
  \tcboxmath[colback=white, colframe=black, size=fbox, arc=3pt, boxrule=0.8pt]%
}

\begin{document}

This \mywbox{3x} is in line math.
This
\[ y = \mywbox{-5x} - 5 + 6 \]
is not in line math.

\end{document}

You may want to abstract things a little bit using a style too, so that you can apply it to other boxes should the need arise:

\documentclass{article}
\usepackage{tcolorbox}
\tcbuselibrary{theorems}

\tcbset{my math box/.style={
          colback=white, colframe=black, size=fbox, arc=3pt, boxrule=0.8pt}
}

\newcommand*{\mywbox}{\tcboxmath[my math box]}

\begin{document}

This \mywbox{3x} is in line math.
This
\[ y = \mywbox{-5x} - 5 + 6 \]
is not in line math.

\end{document}

I would say

\newtcbox{\mywboxtext}{on line,colback=white,colframe=black,size=fbox,arc=3pt,boxrule=0.8pt}
\newcommand{\mywboxmath}[1]{\mywboxtext{$#1$}}

Your example can become

\documentclass{article}
\usepackage{tcolorbox}

\newtcbox{\mywboxtext}{on line,colback=white,colframe=black,size=fbox,arc=3pt,boxrule=0.8pt}
\newcommand{\mywboxmath}[1]{\mywboxtext{$#1$}}

\begin{document}

This \mywboxmath{3x} is in line math and this
\[
y=\mywboxmath{-5x}-5+6
\]
is display math.

\end{document}

Change the names to your liking.

Avoid $$ in LaTeX (and of course also \obeylines).

Should you need the boxes to behave in subscripts or superscripts, change the code into

\documentclass{article}
\usepackage{tcolorbox}

\newtcbox{\mywboxtext}{on line,colback=white,colframe=black,size=fbox,arc=3pt,boxrule=0.8pt}

\makeatletter
\newcommand{\mywboxmath}[1]{\mathpalette\mywboxmath@do{#1}}
\newcommand{\mywboxmath@do}[2]{\mywboxtext{$\m@th#1#2$}}
\makeatother

\begin{document}

This $\mywboxmath{3x}$ is in line math and this
\[
y_{\mywboxmath{0}}=\mywboxmath{-5x}-5+6
\]
is display math.

\end{document}

Note that in this case you need that \mywboxmath is in math mode.