What is the identity arrow in this category?

You are looking for the Kronecker delta $\delta(x,y)$ which is $1$ if $x =y $ and $0$ otherwise.

Writing down the relation that the identity $I_A$ would have to satisfy on one side : $$f(a,a') = (f \circ I_A)(a,a') = \sum_{a'' \in A} f(a,a'')I_A(a'',a'),$$ so a natural candidate would be $I_A(a'',a') = 1$ if $a'' = a'$ and $0$ otherwise. (This is known as the Kronecker delta function.)

$\newcommand{\id}{\mathsf{id}}$ You'd like a function $f: A\times A \rightarrow \mathbb{R}$ such that for any other function $g: A\times B \rightarrow \mathbb{R}$, you have

$$\begin{eqnarray*} f \circ g &= g & \qquad A\times B\rightarrow \mathbb{R}\\ (f \circ g)(a,b) &= g(a,b) &\qquad \forall \,(a,b) \in A\times B\\ \sum_{y \in A} f(a, y)\cdot g(y, b) &= g(a,b) & \\ \end{eqnarray*}$$

We can define $f$ in such a way that most terms in the sum vanish. In particular, if we set $f(a,y) \equiv 0$ unless $y=a$, then the sum collapses to a single term:

$$f(a,a)\cdot g(a,b) = g(a,b)$$

Then if we define $f(a,a) = 1$, we get the desired result: $g(a,b) = g(a,b)$, so $f\circ g = g$ in general.

The definition we've arrived at is:

$$f(a,y) \equiv \begin{cases}0 & \text{if }a \neq y\\1 & \text{if }a=y\end{cases}$$

which you can prove is also a right handed identity: $h\circ f = h$.