Plotting function ℝ² →ℝ with pole at (0,0) smoothly

The answer to the technical part of the question whether or not one can specify a special value at special points is yes. One possible way to do that is to plot something like

ifthenelse(x^2+y^2>0.05,(x*y)/(x^2+y^2),0.5*sin(2*atan2(y,x)))

which switches the expression to 0.5*sin(2*atan2(y,x)) if you are close to the origin.

The perhaps more interesting question is what to put there, or why I put 0.5*sin(2*atan2(y,x)). I added the explanation in LaTeX.

\documentclass[tikz,border=3mm]{standalone}
\usepackage{pgfplots}
\pgfplotsset{compat=1.17}
\begin{document}
\begin{tikzpicture}[]
\begin{axis}[axis lines=center,
axis on top,
xtick=\empty,
ytick=\empty,
ztick=\empty,
xrange=-2:2,
yrange=-2:2
]
% function
\addplot3[domain=-2:2,y domain=-2:2,colormap/viridis,surf,opacity=0.5,samples = 55]
{ifthenelse(x^2+y^2>0.05,(x*y)/(x^2+y^2),0.5*sin(2*atan2(y,x)))};
\end{axis}
\node[align=left,above,text width=10cm] at (current axis.north) 
{In polar coordinates,
\[x=r\,\cos\varphi\quad\mbox{and}\quad y=r\,\sin\varphi\;,\]
such that
\[\frac{x\,y}{x^2+y^2}=\frac{r^2\,\cos\varphi\,\sin\varphi}{r^2}=\cos\varphi\,\sin\varphi\]
with $\varphi=\arctan(y/x)$. So we can replace
\[\frac{x\,y}{x^2+y^2}\to \sin(2\arctan(y/x))/2\;.\]
};
\end{tikzpicture}

\begin{tikzpicture}[]
\begin{axis}[axis lines=center,
axis on top,
xtick=\empty,
ytick=\empty,
ztick=\empty,
xrange=-2:2,
yrange=-2:2
]
% function
\addplot3[domain=-2:2,y domain=-2:2,colormap/viridis,surf,opacity=0.5,samples = 55]
{0.5*sin(2*atan2(y,x))};
\end{axis}

\end{tikzpicture}
\end{document}

As you can see from the lower plot, you do not need x*y/(x^2+y^2) at all, you can plot 0.5*sin(2*atan2(y,x)) over the full domain and get a non-jagged result.

There is some jaggedness to the graph when I run it through the computer algebra system (CAS) called SAGE, so you should expect some in pgfplots

To increase the precision of the result you can have SAGE generate the x and y coordinates or you can do them yourself. I used the command xcoords = [i for i in srange(-2,2,.06)] but you could just have easily specified xcoords = [-2,-.75,-.1,.1,.36,1,2]. By forcing the pgfplots through SAGE you have more control over the plot. (EDIT: Although the sagetex package is part of the LaTeX distribution, SAGE is not). The easiest way to work with it is through Cocalc. If you are working with mathematical topics this is a good tool to get familiar with.

\documentclass[border=5pt]{standalone}
\usepackage{pgfplots}
\usepackage{sagetex}
\pgfplotsset{compat=1.15} 
\begin{document}
\begin{sagesilent}
xcoords = [i for i in srange(-2,2,.06)]
ycoords = [i for i in srange(-2,2,.05)]

output = ""
output += r"\begin{tikzpicture}[scale=1.0]"
output += r"\begin{axis}[view={30}{45},xmin=%d, xmax=%d, ymin=%d, ymax=%d]"%(-2,2,-2,2)
output += r"\addplot3[colormap/viridis,surf,opacity=0.5,mesh/rows=%d] coordinates {"%(len(ycoords))
# the length of ycoords is the number of y values
for y in ycoords:
    for x in xcoords:
        output += r"(%f, %f, %f) "%(x,y,x*y/(x^2+y^2))

output += r"};"
output += r"\end{axis}"
output += r"\end{tikzpicture}"
\end{sagesilent}
\sagestr{output}
\end{document}

which looks like this:

I thought the plot looked better with just surf

The documentation for SAGE 3d plots is here.

I know it is not perfect but...

With the help of @Schrödinger's cat i came up with the following:

The line on the Inputspace that causes the problem is defined by x=y. So using the "ifthenelse" function i made:

if |x-y|< small value, then set output to 0.5

\begin{tikzpicture}[]
\begin{axis}[axis lines=center,
axis on top,
xtick=\empty,
ytick=\empty,
ztick=\empty,
xrange=-2:2,
yrange=-2:2
]
% function
\addplot3[domain=-2:2,y domain=-2:2,colormap/viridis,surf,opacity=0.5,
samples = 55]
{ifthenelse(abs(x-y)<0.1,0.5,0.5*sin(2*atan2(y,x)))};
\end{axis}
\end{tikzpicture}

So that if a input point is close to the line it just sets the output to 0.5. It still doesn't look perfect put if i now increase the resolution the Graph should look at least kind of smooth: