Is physics rigorous in the mathematical sense?
No, physics is not rigorous in the sense of mathematics. There are standards of rigor for experiments, but that is a different kind of thing entirely. That is not to say that physicists just wave their hands in their arguments [only sometimes ;) ], but rather that it does not come even close to a formal axiomatized foundation like in mathematics.
Here's an excerpt from R.Feynman's lecture The Relation of Mathematics and Physics, available on youtube, which is also present in his book, Character of Physical Law (Ch. 2):
There are two kinds of ways of looking at mathematics, which for the purposes of this lecture, I will call the the Babylonian tradition and the Greek tradition. In Babylonian schools in mathematics, the student would learn something by doing a large number of examples until he caught on to the general rule. Also, a large amount of geometry was known... and some degree of argument was available to go from one thing to another. ... But Euclid discovered that there was a way in which all the theorems of geometry could be ordered from a set of axioms that were particularly simple... The Babylonian attitude... is that you have to know all the various theorems and many of the connections in between, but you never really realized that it could all come up from a bunch of axioms... [E]ven in mathematics, you can start in different places. ... The mathematical tradition of today is to start with some particular ones which are chosen by some kind of convention to be axioms and then to build up the structure from there. ... The method of starting from axioms is not efficient in obtaining the theorems. ... In physics we need the Babylonian methods, and not the Euclidean or Greek method.
The rest of the lecture is also interesting and I recommend it. He goes on (with an example of deriving conservation of angular momentum from Newton's law of gravitation and having it generalized):
We can deduce (often) from one part of physics, like the law of gravitation, a principle which turns out to be much more valid than the derivation. This doesn't happen in mathematics, that the theorems come out in places where they're not supposed to be.
Physics is usually not rigorous. But there is a branch of physics, called mathematical physics, in which physics is treated with full mathematical rigor. There everything begins with formally stated assumptions (axioms) from which everything else is rigorously deduced.
In particular, there are fully rigorous treatments of phenomenological thermodynamics (see, e.g., my paper http://www.mat.univie.ac.at/~neum/ms/phenTherm.pdf), of classical mechanics, of fluid mechanics, and of quantum mechanics.
A possible set of axioms for quantum mechanics is given in my ''Postulates for the formal core of quantum mechanics'' from Chapter A4: The interpretation of quantum mechanics of my theoretical physics FAQ at
http://www.mat.univie.ac.at/~neum/physfaq/physics-faq.html
This chapter also contains a discussion of ''What is the meaning of axioms for physics?''. See also ''Why bother about rigor in physics?'' in Chapter C2: Some philosophy of physics.
Some parts of physics are rigorous in the sense of mathematics - i.e. they are treated as mathematics, with physically-motivated entities. And it is not that uncommon that for some papers to make it rigorously, explicitly stating assumptions and proving things.
However, most of the time physics in not mathematically rigorous. It stems from a few things:
Physics, typically, work the other way around than mathematics. That is, knowing some effects you try to figure out assumptions so the effects can be explained.
Physics is related to the real world. And many times it is tricky to relate a pure mathematical concept to (reasonably) objectively measurable quantities.
In physics, there is not much difference if you fail at predicting because of error in mathematics, or using unphysical assumptions.
Most of things in physics started as a hand waving arguments, which were mathematically dubious, but "they worked in most cases". In that way it was possible to explain or predict many phenomena. Their mathematical grounding often went later (a few years, decades or... is still an open problem); and more than often, in a utterly unpractical form for any physical calculations.
Think of things like Lebesgue integral (still to get the actual numerical value you need to do summation or Riemann integration), delta Dirac as a distribution (for physical calculations it is treated as "narrow enough" function), formalization of path integrals (well, not suitable for calculations), ...
How does one come up with seemingly simple equations that describe physical processes in nature?
It is a big question.
There are some answer in the spirit of emergency like:
"Physics is this part of the reality that is easily described with mathematics."
Or even more cynically (it's more-or-less a quotation, but I've forgotten the author):
"Physics is this part of the reality that can be approximated as coupled harmonic oscillators."
Also, it is highly recommendable to read a classical text on that issue: Eugene Wigner, The Unreasonable Effectiveness of Mathematics in the Natural Sciences.