Functions that are Riemann integrable but not Lebesgue integrable
The following is a bit of a ramble, but I hope you find it a useful collection of information.
The Riemann integral is only defined for bounded functions on bounded intervals, which are all Lebesgue-integrable. It's the extension to the improper Riemann integral that can integrate functions that are not Lebesgue-integrable.
We recall that a function $f$ is improperly Riemann-integrable on $(a,b)$ if $\int_c^d f$ exists for all $c,d$ with $a<c<d<b$ and $\lim_{\substack{c \to a \\ d \to b }}\int_c^d f$
An easy way to understand the difference is that if $f$ is Lebesgue-integrable, $\lvert f \rvert$ must also be Lebesgue-integrable (basically because this is how we define the Lebesgue integral for nonpositive functions). $\sin{x}/x$ does not satisfy this, so cannot be Lebesgue-integrable.
Another way to understand the Lebesgue integral is via the Daniell construction: we take a vector space of basic functions $\mathcal{F}$ (so that in addition $f \in \mathcal{F}$ if and only if $|f| \in \mathcal{F}$) and a map $I: \mathcal{F} \to \mathbb{R}$ that satisfies linearity, positivity ($f \geq 0 \implies I(f) \geq 0$) and continuity (if $f_n$ is a nonincreasing sequence converging pointwise to $0$, then $I(f_n) \to 0$). One can then define an integral on functions representable as a monotone limit of the functions in $\mathcal{F}$, with some subtleties to deal with negative functions. The advantage here is that one may choose the basic integral $I$ as the Riemann integral and the elementary functions as the continuous functions on a finite interval, or compactly supported continuous functions, and then this construction can be proven to give Lebesgue's integral.
The usefulness of the Lebesgue integral does not really lie in extending the Riemann integral unilaterally. For this the Gauge/Henstock–Kurzweil integral is a much better idea, and indeed, functions like the characteristic function of the rationals are not common in the applications of integration for which the Lebesgue theory is favoured (that's most of them). Instead, we have found that it has other advantages:
- Spaces of Lebesgue-integrable functions tend to be better than spaces of Riemann-integrable functions, the main advantage being that limits of Lebesgue integrable functions are normally also Lebesgue-integrable (pointwise limits, for example, are, by one or other of the Convergence Theorems), whereas this is not the case for Riemann-integrable functions: you need uniform convergence for the limit to be Riemann-integrable. This means that, for example, $L^1$ is a Banach space with norm given by the integral of the absolute value, whereas one would have to use the uniform norm to turn a space of Riemann-integrable functions into a Banach space, with which comes a lack of flexibility: uniform convergence is hard to prove, and generally too much to ask!
- The other main advantage is that the Lebesgue integral can be defined over much more general spaces than the Riemann integral (the Riemann integral requires an order structure of some sort on the underlying set, so is essentially limited to $\mathbb{R}^n$), which makes it useful in a much more general context.
The above are the modern reasons for using the Lebesgue integral: this can hardly have been Lebesgue's motivation, which was likely rather more rooted in the extension you suggest. It's also very difficult to exhibit a non–locally-Lebesgue-integrable function on a finite interval: one needs a lot of the axiom of choice.
We've essentially found there are two common sorts of integral one can define on the real numbers: "absolute" integrals, that tend to be the Lebesgue integral if they are sufficiently general (In this category we have Daniell, Mikusiński and McShane, for example). The other sort are the more general non-absolute ones, such as the gauge integral.
The way I see it, the Lebesgue integral is indeed an "improvement" of the Riemann integral, and the function you give is in fact an example of that. In other words, the fact that this function has a Riemann integral on $\mathbb{R}$ just shows that Riemann integration has some flaws. This is directly related to the following fact:
Let $\sum a_n$ be a conditionally convergent series. Then for any real number $\alpha$, there is a rearrangement of $\sum a_n$ which converges to $\alpha$.
In your example, you can take the function on every interval $[n\pi,(n+1)\pi]$ and reorder the intervals. The above fact implies that any real number could be obtained as a "reordered" Riemann integral of the function in this way. The Lebesgue integral on the other hand, ignores anything such as order. Indeed, it is additive for any sequence of measurable sets. Hence, if your function were Lebesgue integrable, its Lebesgue integral would have to be equal to any real number.
For $r>0$, your function has both a Riemann and a Lebesgue integral over $[0,r]$, and they have the same value in terms of $r$. Now let $r\to\infty$. This common $r$-dependent value tends to a limit, which is traditionally called the improper Riemann integral over $[0,\infty)$. It might just as well be called the improper Lebesgue integral over $[0,\infty)$, but for (good) historical reasons it is not. As others have commented, such terminology is misleading: the value is a limit of a (parameter-dependent) integral and not itself an integral. That said, there is seldom any problem in treating such limits as integrals; so the terminology usually doesn't give rise to confusion in practice.