Is calculus not rigorous?

You've identified one of the biggest issues already:

we considered $\Bbb R$ (and $\Bbb R^n$) to be given and just kind of "the number line of every number you possibly can think of"

As a result, there are some very important theorems that I'll bet you didn't prove formally (although you probably did draw pictures corresponding to them), such as the Intermediate Value Theorem. Ultimately, the IVT is a topological statement about how the reals behave that, in a way, formalizes what we mean by "every number you possibly can think of," at least in terms of filling in holes and making $\mathbb{R}$ complete. Not discussing what a complete metric space actually is means that there's going to be quite a bit missing.

Likewise, the Extreme Value Theorem, which is the key step in proving the Mean Value Theorem, is something that one usually doesn't approach without some basic topological knowledge (or a significantly more in-depth knowledge of sequences than is typical for a general calculus course). Both of these two theorems I've mentioned really rely on that concept of "completeness," or "not having any holes."

But otherwise, the course is probably pretty rigorous - the $\epsilon-\delta$ approach isn't lacking of anything from a technical view, and there's quite a bit you can do only using it.