What really is mathematical rigor? How can I be more rigorous?

You should be able to delineate the precise mathematical theorems that allow you to make each step in a proof. For example, if you have $(x,y) \in \mathbb{R}^2$ and you write: let $r,\theta$ satisfy $x = r\cos \theta,y=r\sin \theta$ with $r\geq 0$ and $2\pi > \theta \geq 0$, you are using a theorem that says that:

Proposition. For all $x,y \in \mathbb{R}$, there exists $r \in \mathbb{R}_{\geq 0}$ and $\theta \in \mathbb{R}$ such that $x=r\cos \theta$ and $y = r \sin \theta$ and $2\pi > \theta \geq 0$.

Make sure you can explicitly write out the theorems that allow you to make the steps you are making.

The other thing is that you need to develop an intuition for what the instructor (reader, etc.) expects you to take for granted. You may need to write some follow-up lemmas if there are steps in your proof which invoke theorems that you really shouldn't be taking for granted.

I can relate. Throughout the years, many of my exams and homework assignments had deductions for not being rigorous enough.

The trick to mathematical rigour is managing the leaps that you make from statement to statement. The size of the leaps is inversely proportional to the rigour.

Let me mention some heuristics that tend to work well for me:

• If you spend a long time thinking on a step, also spend a long time on writing it down — apparently it isn't clear at first sight.
• When using theorems relevant to the subject matter and the level of the proof, mention them. If necessary, explicitly verify that the hypotheses are met.
• Try to verify how your proof works in a small or otherwise manageable example.
• Whenever you give in to the temptation of using "obviously", "trivially", "clearly" and their brethren, or just leaving an intermediary step out, pay extra attention to make sure that the leap you're making is indeed small enough to be disposed of using these terms.

That should hopefully be of some assistance in assessing your own rigour.

Two obvious pointers remain:

• Have others read your work. It's always a good sign if others can follow your proof.
• Practice. You will get there.

As a side remark, I've always found writing answers on Maths.SE a good means to combine these.

A proof of a proposition is rigorous if it convinces the reader that the proposition is true beyond a reasonable doubt. In math, as in everything else, what constitutes "reasonable doubt" is flexible. It depends on the cultural context:

• What was rigorous to Euclid was not always rigorous to Hilbert.
• What was rigorous to Gauss was not always rigorous to Whitehead & Russel.
• What's rigorous in physics is not always rigorous in math.

It depends on the situation at hand:

• What's rigorous in a published paper, meant to convince experts of a new result, may not be rigorous in a homework assignment, meant to verify that a student really knows what they're talking about.
• What's rigorous when teaching middle school students, who are often very willing to rely on intuition, may not be rigorous when teaching graduate students, who know from bitter experience how intuition can lead them astray.

Knowing what level of rigor to use in a given context is a social skill, and it can only be learned through social interaction.

To get acculturated to the standard of rigor in your courses, look at proofs written by other people—your classmates, your teachers, your textbooks' authors. Look at places that teachers have flagged as gaps in students' work, and see how other people bridged those gaps. As you suggested, ask your teachers about how you can strengthen your arguments in places where they're too weak for the course.

I think it's normal, in an undergraduate course, for a proof to get a good grade but also a note that it ought to be more rigorous. As an analogy, if I were marking an essay for a planetary geology class written in English by a student who was still learning English, I would flag awkward phrasing and grammatical errors to help the student improve their writing, but I wouldn't punish the student grade-wise for them. In an essay for a English writing class, of course, it might be a different story. I think most math teachers feel that students should be nudged toward an appropriate standard of rigor to help them improve their writing, but not punished harshly for straying from it, except of course in a class devoted to teaching a certain standard of rigor.

TL;DR. The bad news is that rigor, like all human social standards, is somewhat vague and arbitrary, and thus frustrating to learn. The good news is that you're a human, so you'll learn it. Good luck!

p.s. If you enjoyed this answer, you might also enjoy Eugenia Cheng's "Mathematics, morally," which touches on many related topics.