Which are the mathematical problems in non-standard analysis? (If any)

I'm not entirely sure what you're asking, but let me take a stab at it:

First of all, there's nothing standard analysis can do that nonstandard analysis can't. A nonstandard analyst could always decide to just study the standard hyperreals, and this would correspond to standard analysis. (The converse is also true, but nontrivially so.) So you won't find a mathematical problem in a deep sense; anytime the nonstandard approach is less useful than the standard one, a nonstandard analyst could always just use the standard approach inside nonstandard analysis.

That said, there are mathematical features of the hyperreals which are (in my opinion) less than ideal. Topologically, they are ugly: there are multiple natural topologies to put on them, and they all have odd features (see here). And I would consider the presence of lots of automorphisms to be a negative feature as well - it means that if someone asks for an example of an infinitesimal, we can't really give a satisfying answer; however, this arguably reflects my own standard bias.

I suspect there are also algebraic properties the reals have which the hyperreals lack, although at the moment I can't think of any (my previous example was incorrect and silly).

Basically, I think the bottom line is this:

  • Anything a standard analyst can do, a nonstandard analyst can also do - sometimes more easily. (Although my understanding is that that gain in ease rapidly drops off once one is comfortable with standard analysis. Some exceptions exist, however: the invariant subspaces problem was originally solved via nonstandard analysis, and I think there are some nonstandard proofs of esoteric results for which no proof via standard analysis is currently known, although we know that such a proof must exist.)

  • That said, the hyperreal field is a much less nice object than $\mathbb{R}$: the price of having a nice infinitesimal structure is that we lose good properties elsewhere. And it lacks - in my opinion - the compellingness of the structure $\mathbb{R}$. Note that this objection is completely unrelated to the question of whether nonstandard analysis and the hyperreals are useful: something need not be philosophically compelling to be a good tool. I actually think there is a really interesting philosophical phenomenon here: I find the hyperreals completely uncompelling, but the language and techniques of nonstandard analysis to be very compelling! The subject is somehow more compelling to me than its subject matter. No idea what that says about me.

  • I think there are extremely good reasons to learn standard analysis, but no good ones besides personal preference and limitations of time to not learn nonstandard analysis. I would argue that for most mathematicians, learning nonstandard analysis would not necessarily be a good use of time (a combination of unpopularity and - I suspect - a low benefit to their already-existing research interests), but the reasons for this are at least largely sociological, and not inherent to the subject.


About the likeability issue raised in the question: indeed, A. Robinson's original version of non-standard analysis (which is not quite necessary to follow J. Keisler's treatment of calculus!) does require understanding a bit of model theory. In contrast, E. Nelson's "IST" version is a much more user-friendly version (see A. Robert's very nice little introductory book, and/or Nelson's essay), that has managed to package most of Robinson's stuff in a fashion that does not require frequent interaction with (or understanding of) the model theory. (Nevertheless, I gather that there are some possibilities in Robinson's version that are not fully represented in Nelson's).

I have not quite used non-standard analysis "in research", instead finding that L. Schwartz' notion of "distribution/generalized function", as expanded-on by A. Grothendieck's early work, and Gelfand-Graev-et al, is adequate (so far) for my purposes (and in my context, obviously). Nevertheless, non-standard analysis is interesting to me for at least two reasons. First, it approximately shows (in a revisionist and anachronistic way, of course) that L. Euler's and A. Cauchy's use of "infinitesimals" can be turned into a completely legitimate argument (in contrast to various naive dismissals that often claim that epsilon-delta arguments are the only legitimate way to do analysis). Second, non-standard analysis does seem to better capture certain intuitions about "orders of magnitude" that are a bit clumsy to formulate epsilon-delta-wise... Not that it is impossible to capture them, but that it requires a priori an understanding that might only be achieved by thinking in non-standard terms. Not that I've made anything of this myself, only that I have a vague feeling in this direction. Again, A. Robert's book gives the delightful example of canards.