Are Euclidean domains exactly the ones which we can define "mod" on?

As you say, you’re using the binary function “mod” of the computer scientists. But in mathematics, one finds that this function is of very limited utility, and we use the equivalence relation $a\equiv b\pmod I$ instead. It’s read, “$a$ is congruent to $b$ modulo $I$”, and it’s a sentence that will be either true or false.

To your specific question, consider the Euclidean domain $\Bbb Z[i]$, the Gaussian integers. If you take a complex modulus like $\zeta=6+3i$, it makes perfect sense to say whether two Gaussian numbers $\alpha$ and $\beta$ are congruent modulo $\zeta$, namely you check to see whether $\beta-\alpha$ is a multiplie of $\zeta$ by another Gaussian number. But there is no good and consistent way of defining a “mod” function here. You can make up an artificial “mod” function in this situation, but it will be neither good nor consistent.

To your title: No. Any ring works for a mod operation, you just make things the standard coset way. Let $I\subseteq R$ be an ideal. Then we define

$$a\equiv b\mod I\iff a-b\in I.$$

And this is a well-defined equivalence relation.

For the integers this reduces to

$$a\equiv b\mod n\iff a-b\in (n)\iff a-b=nk$$

for some $k\in\Bbb Z$.

So to the question in the body (which is not the same thing) as to if we can define mod for EDs: yes you can, but they're not the most general setting.

In a Euclidean domain, there is no requirement that $r$ be uniquely determined by $a$ and $b$, so you cannot define mod in the way you proposed.