Math formula to check two integers

Yes, you have two equations $$\begin{align}x+y&=n\\xy&=m\end{align}$$ where $n$ and $m$ are known. From the first, $y=n-x$ Substitute this into the second, and use the quadratic formula to check if the solutions are integers.

EDIT The substitution gives $$x^2-nx+m=0$$ By the quadratic formula, $$x={n\pm\sqrt{n^2-4m}\over2}$$ So now you can check if this is an integer. You need $n^2-4m$ to be a perfect square, and the fraction to work out to a whole number. If $x$ is an integer, so is $y,$ since $n$ and $m$ have to be integers. (If $m$ and $m$ aren't integers, there's no possibility of integral solution.)

I overlooked the line beginning "So is there any exact way ..." before, and I'm afraid I don't understand what you mean.

There are a lot of functions that meet your criteria.

First, there is this function. It is boring, as it defeats the point of the question, but it is worth noting that it is a mathematically valid function:

1) $f(x, y) = \cases{1 & if $x$ and $y$ are integers \\ 0 & otherwise}$

Similarly, there's

2) $f(x, y) = \{x\} + \{y\}$ where $\{x\}$ denotes the fractional part of $x$. It is 0 iff both are integers.

So it would be a good idea to set some additional restrictions on our functions to have a meaningful question.

A typical restriction could be to ask for functions that are continuous and differentiable (smooth), as most functions and operation we use (addition, subtraction, multiplication, trig, etc.) have this property. Here is a continuous and differentiable (smooth) function that works:

3) $f(x, y) = \sin^2(\pi x) + \sin^2(\pi y)$

This is 0 iff both $x$ and $y$ are integers.

Let's consider continuous and differentiable functions with a bit more generality. Suppose $f(m, n) = k$ for some integers $m$ and $n$. Consider a contour plot line at height $k$. We can then then move slightly along the line, and make $m$ and/or $n$ not integer while preserving the value $k$... Unless $(m, n)$ is a maximum or a minimum, and so the contour line is actually just a point. This means, we need functions that have maxima or minima at every integer point.

This means, there aren't any such functions that can be expressed in terms of only elementary operations (addition, subtraction, multiplication, division, roots).