In which senses can an integral exist?
In addition to other useful answers... Before giving the standard example of an integral whose convergence is problemmatical, but which does have some sense... we should ask ourselves what it is we are expecting "integrals" to do, what properties the process should have. For example, "integration" should not just produce random numerical outcomes. And which functions, on which intervals, should be expected to be acceptable inputs? At the very least, if $f,g$ are acceptable, then linear combinations $af+bg$ should be, and, letting $I$ denote the integration procedure, $I(af+bg)=aI(f)+bI(g)$. ("Linearity".)
Riemann integration works best on finite intervals with nearly-continuous functions, while Lebesgue integration accommodates very-discontinuous functions, etc. In both cases, the integral of $f$ on a set or interval is a limit of finite sums, and the set-up for the game consists of proving these limits will exist (and be finite numbers!) under various assumptions on $f$.
A simpler standard example similar to one in your other post is $\int_0^\infty \sin(x^2)\;dx$. This has the disturbing feature that the function doesn't go to $0$ at infinity, so if we think of Cauchy's criterion for convergence of a series (rather than integral), we might conclude that this would diverge, meaning that $\lim_N \int_0^N \sin(x^2)\;dx$ might be $\pm\infty$? Or not exist? However, the oscillation produces enough self-cancellation so that this doesn't happen. In fact, changing variables, replacing $x$ by $\sqrt{x}$, gives integral $\int_0^\infty {\sin(x)\over \sqrt{x}}\;dx$. Now, at least it looks like it decays at infinity, and there is still cancellation due to the oscillation. In fact, the limit can be evaluated by various tricks: I think it is $\pi/2$ or something similar.
In fancier circumstances, it often happens that "an integral" is not meant to be taken literally, but only to indicate the structure of some operation on functions. The basic case is with Fourier or Laplace transforms on the real line. Fourier transforms expressed as integrals $\hat{f}(\xi)=\int_{\mathbb R} e^{-i\xi x}\,f(x)\;dx$ make best sense for $\int_{\mathbb R} |f(x)|\;dx<\infty$, but, in fact, via the Plancherel theorem for Fourier transforms, we know that $\int_{\mathbb R} \hat{f}(\xi)\;\hat{g}(\xi)\;dx=\int_{\mathbb R}f(x)\,g(x)\;dx$ (maybe up to a constant multiple), so there is a unique extension of Fourier transform to square-integrable functions $f$, that is, such that $\int_{\mathbb R}|f(x)|^2\;dx<\infty$. This extension has the same properties as the Fourier transform that is literally an integral, but is not quite given by that integral.
Similarly, Fourier transforms can be extended to ("tempered") "generalized functions" (="distributions", not in the probabilistic sense), in a way that is completely sensible structurally, but in which the integrals are wildly not-convergent. For example, $\int_{\mathbb R} x^n\cdot e^{-i\xi x}\;dx$ doesn't converge at all, but by other means we can conclude that it is (a constant multiple of) the $n$th derivative of the Dirac delta distribution.
And, in case there were any doubt, computer algebra systems have their limitations, especially in dealing with "divergent" (not numerically docile!) integrals.
Basically, the definite (Riemann) integral $\int_a^b f(x)\,\mathrm dx$ exists if, well, $f$ is Riemann integrable. The most important class of Riemann integrable functions are the continuous functions. Your example is an improper integral, i.e. one or (in your case both) ends are infinite, and that can only be evalutaed in the sense of $$\tag1\int_{-\infty}^\infty f(x)\,\mathrm dx:=\lim_{a\to-\infty\atop b\to+\infty}\int_a^bf(x)\,\mathrm dx$$ (There are other cases of improperness, e.g. when $f$ is not continuous at $a$ and/or $b$ and again you have to take a limit of definite integrals). Although $f$ is continuous and hence all $\int_a^b$ exist, the (double) limit is a completely different question.
You may also want to have a look at Lebesgue integral, a different integration theory altoether that handles some problems of Riemann integral with a different systematic. Lebesgue treats the integral "all at once" but requires some other conditions to treat oscillating functions (as yours), namely that $\int_{-\infty}^\infty|f(x)|\,\mathrm dx$ be finite - which is not the case with your example.
The function you are considering, $$ f(x):=\exp \left ( i\left ( ax^3+bx^2 \right ) \right ), $$ is not Lebesgue integrable on $(-\infty,\infty)$, as $|f(x)|=1$ for all $x\in(-\infty,\infty)$. However, the limit $$ \lim_{M,N\to\infty}\int_{-M}^Nf(x)\,dx $$ does exist. This is called an improper integral.