What's the "limit" in the definition of Riemann integrals?

It is the limit of a net. Nets are a generalization of sequences which make all the familiar statements about sequences true for spaces that are not first-countable (for example a point lies in the closure of a subspace if and only if there is a net converging to it, and so forth), so any time you want to prove something about general spaces and you would like to use sequences but can't, you can use nets instead (although there are some subtleties here; one cannot just replace "sequence" with "net" in a proof).

One way of thinking about it is that you have a function defined on the set of partitions of $[a,b]$ into the real numbers called the Riemann sum. You put an order on partitions by defining the notion of mesh ($|\sigma|$ in your notation) and defining an order on the set of partitions by $\sigma\succeq\tau$, if and only if $|\sigma| \leq |\tau|$ and say that $\sigma$ is finer than $\tau$. So now you can make a definition similar to the limit of sequences: $\lim_{|\sigma|\rightarrow 0} R(\sigma)=J$ if and only if for all $\epsilon>0$ there exists a partition $\Lambda$ such that for all partitions $\sigma$ such that $\sigma\succeq\Lambda$ one has $|R(\sigma)-J|<\epsilon$.

The more general context for this is that we are making the set of partitions into a directed set, and so Riemann sum becomes a net from the set of partitions into $\mathbb{R}$.

It can be stated in terms of the ordinary definition of limit. Let $A(\sigma)$ and $B(\sigma)$ respecively be the supremum and infimum of $\sum_i f(\xi_i) (x_i - x_{i-1})$ over all subdivisions of "norm" $\sigma$ and all choices of the $\xi_i$. Then if $\lim_{\sigma \to 0} A(\sigma) = \lim_{\sigma \to 0} B(\sigma)$, i.e. both limits exist and are equal, the common value is the Riemann integral.