Number of ways to put $n$ unlabeled balls in $k$ bins with a max of $m$ balls in each bin

As a check I did it with an inclusion-exclusion argument, getting $$\sum_i(-1)^i\binom{k}i\binom{n+k-1-i(m+1)}{k-1}\,.$$

Setting $r=i$ and $r_2=n-i(m+1)$ to match your notation, I make this $$\sum_r (-1)^r\binom{k}r\binom{k+r_2-1}{r_2}\,.$$ It appears that you’ve the wrong exponent on $-1$.

Let's denote the problem as $D(n,k,m)$, then when $mk-n \leq m$, the answer can be given as: $D(n,k,m)=\binom{km-n+k-1}{k-1}$, where $m \leq n$.

It can be easily verified using $D(5,2,3)=2$, or $D(6,3,3)=10$, etc..

Let me try to explain it in more details with my poor English (sorry for that).

Suppose we start from the state that all bins are filled with $m$ balls in each. Then the task for us is to eliminate $mk-n$ balls from these $k$ bins. To ditribute the $mk-n$ "elimination" into $k$ bins, we have $\binom{km-n+k-1}{k-1}$ different ways if $mk-n \leq m$, i.e. the number of elimination in each bin will not exceed $m$.