Which branch of mathematics rigorously defines infinitesimals?

Your perception is wrong. Non-standard analysis is grounded on Logic and it's as solid as any other field of Mathematics. I suggest that you read Abraham Robinson's Non-standard Analysis.

Most objections to non-standard analysis seem to be about the use of the axiom of choice in the construction of the field of hyperreals. Non-standard analysis is completely rigorous, but if you're a hardcore constructivist then you may be a bit squeamish about it. Then again, there's always some things you need to take on faith in any branch of maths:

• If you're a hardcore finitist then you have to be really careful about analysis in general, since the conventional $\mathbb{R}$ as an object doesn't exist at all.
• If you don't accept the axiom of dependent choices then you're pretty limited in what you can do in real analysis, because many arguments rely on taking a sequence chosen arbitrarily.
• If you don't believe there is a nonprincipal ultrafilter on $\mathbb{N}$ then you can't construct the ultrapower required to create the hyperreals.

If you choose to allow more axioms ("there is an infinite set", "dependent choices", "there is a nonprincipal ultrafilter on $\mathbb{N}$") then you get access to correspondingly more interesting things you can do, but it's all still rigorous.

Note, however, that if you accept Choice then in a certain sense "anything you can do in non-standard analysis, you can also do without the hyperreals" (see https://math.stackexchange.com/a/51480/259262). It's an extra proof technique to make things easier by hiding many of the $\forall \exists$ quantifiers, rather than allowing you to prove genuinely new things that you couldn't prove before.

Amusingly, one of the answers to the question you ask is that elementary calculus rigorously defines infinitesimals.

How does it do so? Via the notion of differential. The problem you're struggling with is almost backwards; the standard, traditional track is:

• Define the notion of derivative
• Use multivariable derivatives to define the notions of (tangent) vector and differential
• Conceptualize a notion of an "infinitesimal" neighborhood of a point

To elaborate on that last point, you're supposed to envision the points of the infinitesimal neighborhood to be enumerated by tangent vectors — the intuitive idea is that you take an "infinitesimal" step proportional to the tangent vector. Differentials are the functions on the infinitesimal neighborhood.

But this conceptualization is not trying to define anything new — it is merely a way of thinking about calculus. (albeit a very useful one!)

But the second point is bog standard. For example, in multivariable calculus, one incarnation of these notions is

• Tangent vectors to points in $\mathbb{R}^n$ are $n \times 1$ column vectors — the sort of thing you get when differentiating a vector function of one variable
• Differentials at points in $\mathbb{R}^n$ are $1 \times n$ row vectors — the sort of thing you get when differentiating a scalar function of $n$ variables

Both the subjects of differential geometry and algebraic geometry treat this sort of thing much more explicitly and more in-depth.