Why are some non elementary integrals defined and others are not?

I can define one right now. I hereby declare that $$ \mathrm{Xi}(x) := \int_0^xt^tdt $$ Now, the only thing that remains is to see whether other mathematicians pick it up and start using it.

As with any other mathematical notation, there isn't really a committee somewhere who sits and weighs criteria to decide what mathematical notation is correct and should be accepted. What decides whether a piece of notation gets accepted and conventional is simply whether enough other mathematicians find it useful and start using it.

And what is the point if we cannot solve them?

That's actually a big part of the motivation for giving some integral a name. An integral that can be solved exactly doesn't really need a name – you can just go ahead, solve it right now and thus avoid having to write out the integral anymore. Or else, keep writing it out in verbatim, and anybody can calculate the results themselves later.

OTOH, an integral whose result we can't write in elementary form can be a pain in the back, if you have to keep carrying around. You may still be able to obtain numerical results, derive differential relationships etc., but all that is tedious work that'll need to be done over and over again by everybody who encounters the integral.

Much better is if you realise “ah, that's the Sine Integral”, then you can invoke some standard formulas / implementations / tables to look up values.

"Cannot solve them" might not be the right point of view. Example: consider $$\tag1 \text{erf}(x)=\frac2{\sqrt\pi}\,\int_{0}^xe^{-t^2}\,dt. $$ This is the error function, the prototypical function that cannot be expressed in terms of elementary functions. Now, if you had a formula, it would probably involve the exponential; and any time you want to obtain numbers from the exponential, you need to use its Taylor series. So, approximations of the series $$\tag2 \text{erf}\,(x)=\frac2{\sqrt\pi}\sum_{n=0}^\infty\frac{(-1)^nx^{2n+1}}{n!(2n+1)} $$ are actually the mosta direct way of calculating it (and fairly efficient for $x$ not too big).

More importantly, just the fact that a function has a name, doesn't really make it "easier": compare $(2)$ with $$\tag3 \sin x=\sum_{n=0}^\infty\frac{(-1)^nx^{2n+1}}{(2n+1)!}. $$ Do you think that having $\sin x$ in a formula is much better than having $\text{erf}\,(x)$?