What are good "math habits" that have improved your mathematical practice?$ $
I think this is a great question and you've already made an important step in addressing the problem - realizing that you are not satisfied with your math working process and searching for ways to improve it. Here are some ideas and suggestions which I found helpful:
- Understand well the basic objects of the game. This means that you should be able to give many interesting examples and non-examples of the objects you work on. Make a (mental or physical) list of such examples. What are the most important examples of vector spaces? Of subspaces? Can you give an example of something which is not a subspace? What kind of constructions generate subspaces? What kind of integrable functions are there? What do you know about them? And so on.
- Make sure you understand everything about the statement of the problem first before trying to approach it. If you don't, go back and review what you have learned. There is no point in trying to solve an exercise about nilpotent linear operators if you can't give an example of a nilpotent operator and an example of a non-nilpotent operator. This will only cause you to halt and feel depressed.
Play with simplified models. This is something I really learned in graduate school and I wish I would have been told explicitly much earlier. If you are facing a problem that you have no idea how to approach and you feel paralyzed, try to work on a simplified (even trivial) model. For example, let's say you need to prove some statement about a linear map $T$ on some vector space $V$ and you have no idea what to do. Can you solve the problem if you assume in addition that $V$ is one-dimensional? Even better, if $V$ is zero-dimensional? Can you do it if $T$ is diagonalizable? If you are asked to prove something about a continuous function, can you do it if the function is a particularly simple one? Say a constant one? Or a linear one? Or a polynomial? Or maybe you can do it if you assume in addition it is differentiable?
Applying this idea has two advantages. First, more often than not you'll actually manage to solve the simplified problem (and if not, try to simplify even more!). This will increase your self-confidence and help you feel better so that you won't give up early on the harder problem. In addition, the solution of the simplified problem will often give you some hints on how to tackle the general one. You might be able to perform an induction argument, or identify which properties you needed to use and then realize those properties actually apply in a more general context, etc.
- When working on a problem, try to drop an assumption and see what goes wrong. Often this will help you to identify the crucial property which you need to actually solve the exercise and then you can review the theorems and results you learned to see if it actually holds.
- Try to have some mental image associated to any important object and concept you meet. This way, when you'll work on a problem which involves various objects and concepts, you'll already feel familiar with them and won't halt and feel paralyzed. Review the images as you make progress and make adjustments as necessary. For example, for the notion of a direct sum decomposition you can hold in your head the image of $\mathbb{R}^3$ decomposed as the "sum" of the $xy$-plane and the $z$-axis. This is, of course, a particular example of a direct sum decomposition but it helps you to feel much more at ease with the concept.
- Build a mental (or physical) map of relations between various results and concepts. For example, let's say you want to determine whether a series converges or not. A useful thing to realize is that it is easier to determine whether a series with positive terms converges than an arbitrary series because there are more tests available for this case. Another useful thing to know is that if the series converges absolutely, it also converges; so in some cases even if the series doesn't have positive terms you can reduce it to the easier case. Knowing all those relations and results before you start the problem will help you to decide on a good strategy to attack the problem. Not knowing them in advance will often cause to to go astray.
- Don't be afraid of writing something wrong. Be hesitant of writing something that you don't really understand. It's not that bad if you write something like "All operators are diagonalizable, hence $X$" because once you understand that not all operators are diagonalizable, you'll immediately see the error. But if you write a convoluted argument two pages long which uses somewhere the fact that your operator is diagonalizable, it will be much more difficult to discover and learn from the error.
- Develop decent computational skills. Math is hard enough without being bogged down in computation errors and wrong applications of techniques. For example, when learning how to solve a general linear system of equations, sit down and solve $7$ different systems. If you got a wrong result in $5$ of the $7$ cases, something is fishy. Identify clearly the origin of the mistake in each case (is it an arithmetic error? did you apply the algorithm incorrectly?). Then repeat with $7$ other systems until you get at least $6$ correct.
- Try to work on math problems with other people. By that I don't mean asking other people for solutions to exercises you couldn't solve. Try to find someone which is more or less your level and has good communication and interpersonal skills and work together with them all the way through a few problems. Be active, propose some ideas, listen to the other person's ideas and work together. This way, you'll get exposed to techniques that work for other people, their mental maps and ideas about the concepts involved and you'll be able to adapt and implement what you learn as part of your own skill set if you find it helpful.
EDIT: I misunderstood the OP at first, and the first half of my answer gives advice on how to approach proving an unknown problem. I then tie this into the organizational question the OP is really asking below the line.
Examples, examples, examples! For me, pretty much all of mathematics is driven visually and by example.
Every time you see a theorem, first seriously commit yourself to finding a counter example. Find almost-counterexamples that show why every assumption in the problem is necessary. Then for each of those almost-counterexamples find an example that is extremely similar, except satisfies the assumption the counterexample was missing. Now you're ready to prove the theorem or read its proof, and in all likelihood you're already close to the proof.
There's a great anecdote about this by Keith Kendig about Hassler Whitney
One day in his office, I happened to mention Bezout's theorem which basically says that two curves of degree $m$ and $n$ respectively intersect in $nm$ points. He says he never heard of it and seems galvanized by it. He jumps up and heads to the blackboard, saying "Let's see if I can disprove that" Disprove it?! "Wait a minute!" I say, "that theorem is nearly two centuries old! You can't disprove anything... really..." As he begins to working on some counterexamples at the blackboard I see my well-meant words are simply static.
His first tries were easy to demolish, but he was a fast learner, and ideas soon surfaced about the complex line at infinity and how to count multiple points of intersection. After a while it got harder for me to justify the theorem, and when he asked "What about two concentric circles?" I had no answer. He argued his way through and eventually found all four points. Finally he was satisfied, and the piece of chalk was given a rest. He backed away from the blackboard and said "Well, well - that is quite a theorem, isn't it?"
I think I mostly kept my cool during all of this, but after I left his office I realized I was pretty shaken. I remember thinking to myself. "Golly, Kendig, you just saw how one of the giants does it!" He'd taken the theorem to the mat, wrestled with it, and the theorem won. I'd known about that result for at least two years, but in 15 or 20 minutes he'd gained a deeper appreciation of it than I'd ever had. In retrospect, it represented a turning point for me: I began to think examples, examples, examples. Whitney worked by finding an example that contained the essential crux of a problem, and then worked relentlessly on it until he crack it.
Doing mathematics by example has taught me how to feel the shape of a theorem, to naturally divide mathematical objects into collections based on how the theorem divides them into "examples" and "non-examples" and those lines that the theorem draw shows you how to prove the theorem. This will also massively help you recreate the proof in the future.
Thinking about the theorem from the "wrong direction" will teach you to think in unusual ways help you falsify conjectures and assumptions easier. People have a strong bias towards looking for confirmation of facts, but struggle to remember to look for disconfirmation. This can make it hard to understand theorems, because $\mathbb{Z}[\sqrt{-5}]$ tells you a hell of a lot more about the nature of prime factorization than $\mathbb{Z}$ does. There's a famous quote about the importance of this kind of thinking about the philosopher and logician Wittgenstein (source):
Tell me," Wittgenstein asked a friend, "why do people always say, it was natural for man to assume that the sun went round the earth rather than that the earth was rotating?"
His friend replied, "Well, obviously because it just looks as though the Sun is going round the Earth."
Wittgenstein replied, "Well, what would it have looked like if it had looked as though the Earth was rotating?
Here's a MathOverflow link about counterexamples to get to know and love.
Now, to tie this into the actual questions in the OP:
For example, do you have two separate pieces of paper for intermediate results and for details?
That depends on the flow of the proof. I'm a big chalk-boarder, and often sketch out my proofs and my examples on chalkboards before committing them to paper (or TeX, more commonly). If proving an intermediate result seriously disrupts the flow of the proof (which tends to mean "requires more than a paragraph") then in my proof sketch I'll just write "by Padding Lemma" or whatever and then prove the "Padding Lemma" on a different pane of the chalkboard. This is largley because I want to be able to look at my examples and proof sketch without getting bogged down in the combinatorics of this lemma I happen to be using. In the actual write up of the proof, learning how to properly organize your "digressions" is a very important part of learning to do academic writing, but is heavily contextual. Like organizing stories or essays, it's more of an art than a skill.
Are there any specific ways of organizing your derivations on paper, or in notebooks, that help clear your mind?
For me, the process of designing a proof (and in fact, thinking in general) is like a conversation. I create an interlocutor in my mind and argue with them. I walk them through example and non-example, explaining why in each case the theorem holds or fails. I find that doing so helps me focus on the similarity between the examples and find the underlying logical thread that contains the "real reason" that the theorem is true. I might draw diagrams or do a few calculations on paper if I can fit them in my head, but I generally don't start really writing the proof until I know what's going to happen. At that point, I chart out a few examples that have been particularly enlightening and that can guide my thinking on a chalkboard. This is usually a few pictures and equations, the equations written with the pictures, and each example separated in space. Then I'll sit down somewhere that I can see the whole chalkboard and begin to write.
Do you write everything linearly, from top to bottom of your notebook, or do you go back and forth on your scrap paper, only writing it linearly when you've found the result?
As I mentioned, I sort my thoughts by example, with each example being an explication of why the theorem is true for that example. I tend to separate examples horizontally, and organize their explication either vertically (especially for formula-heavy problems) or circularly (especially for graphic-heavy problems). I don't worry too much about the arrangement of my chalkboard though, and just place things "where they obviously fit."
Do you scratch formulae completely if you've made a mistake, and start over, or do you just correct the formulae?
I prefer working on a chalkboard and erasing mistakes and fixing them. In particular, I avoid slashing through or using other markings to indicate that a term is incorrect, because my equations are often adjacent to arrows and other markings that indicate how they fit together, and I use crossing out to indicate cancellation.
Do you write derivations quickly on a scratchbook, until you've found the final answer, or do you write them neatly from start to finish?
My handwriting is not that neat, and I ususally TeX the final. If it's legible to me (and any collaborators), that's all that's necessary for scratch work.
Everyone is confused when learning a new subject, so don't get discouraged if you don't pick things up immediately. My advice would be: instead of letting the confusion scare you away, harness your confusion for all it is worth.
There are many different kinds of confusion. Sometimes it may be just a sort of general haziness around a subject - this might just indicate you need to re-read the textbook because you don't remember the details.
But there is another kind of confusion, one that is potentially useful. Often you will have some sort of cognitive dissonance. That is, you're learning something new - but it does not jive with your current understanding. Something does not feel right. One difference between an excellent student and a mediocre one is that the excellent student will simply refuse to let this feeling go until it is resolved. It's very easy to accept what you've just learned, even if it doesn't make sense to you. But if you're feeling this kind of cognitive dissonance, it means something is wrong in your understanding, either of the previous material or of what you're currently learning.
The trick is to be able to precisely pinpoint what the issue is. At first what you might feel is only an emotion, a vague uncertainty. But this uncertainty may be meaningful. If you double down on this, and keep thinking about it, eventually some more concrete questions will bubble to the surface. It's okay to not have the answer immediately - the important thing is to sharpen the confusion into a very precise question. This will often involve taking away all of the extraneous details surrounding your question and focusing only on precisely where the issue is. If you're having an issue with a very general theorem or idea, try coming up with a specific, concrete example that demonstrates the problem.
Once you've funneled the haziness into a very specific question, you'll often find that the answer is not as difficult as you thought. If you can't come up with it after some thought, you should ask your a classmate, instructor or a TA - or ask here on Stack Exchange. Generally speaking, I think teachers appreciate getting very well-formed questions that show the student has put a lot of thought into it.
It is possible to take this too far, getting bogged down in minor issues instead of moving on with the material. Sometimes it's necessary to table a question so you can get on with your studying. In that case, my advice would be to write down your confusion and remember to come back to it later. Sometimes you just need to take a walk, or sleep on it. Channeling confusion into a meaningful question is a difficult skill that takes time and practice.
Edit:
Here's an illustration of the process. Many students don't actually get from Step 3 to Step 4. (Cartoon due to Fan Wei, found on the page for Richard Stanley's Enumerative Combinatorics.)