How should I deal with becoming discouraged as a graduate student?

The fact is research is hard. It appears to consist primarily of staring at a problem for days and days and days without getting anywhere. Sometimes, rarely, I do figure something out and that feels wonderful, but the overwhelming majority of my time appears to be spent banging my head against a mostly figurative wall.

Yes. This. And it wouldn't be so damn tempting if those bricks didn't wiggle just a little bit every time I slammed my forehead into them. Sometimes I think my eyes must be playing tricks on me, what with the repeated cranial trauma and all. But then I remember how good it felt the last time my head actually went through the wall, and so I keep plugging away.

I've found it extremely useful to have two or three walls to bang my head against at any given time. Surprisingly, sometimes banging my head against one wall actually makes one of the other walls weaker. But most walls prove considerably stronger than my head; so it's helpful to have options, so I don't feel so bad about walking away with some scalp intact.

If you're very lucky, one good smack on the bricks will actually bring the ceiling crashing around your ears. That takes a long time to clean up, but sometimes the debris will knock down other walls for you. And then you have a whole new set of even bigger walls to bang your head against!

I find myself thinking along the lines of "Oh, if only someone actually smart were thinking about this problem they would have solved it in moments" and so on.

Do not listen to the Impostor Syndrome. Everyone "actually smart" is hearing exactly the same voice in their head saying "Oh, if only someone who actually knew how to hit walls with their forehead hit this wall, it would come down like a stack of cards." when in fact the wall really is made of brick.

Does it get better with time and experience?

Yes. Eventually, you'll move from hoping that you'll be able to knock down a wall with your head someday, to being surprised at how often the walls you hit with your head actually fall, to finally believing that you really can knock down walls with your head sometimes. (For me, the second transition happened some time after tenure.)

But your head will still hurt.

It definitely does not mean research isn't for you. Research is hard, and it takes some getting used to. Your experiences sound normal, and it will indeed get better.

Part of the problem is that it's tempting to focus too much on the destination: proving theorems, writing papers. These things happen only occasionally, and thinking about them (or their absence) too much is an easy way to become depressed. Instead, you want to reach the point of enjoying the journey itself. This takes some perspective and confidence, but it will come with time.

For example, imagine that one day you mention a difficulty you're having to your office mate, who tells you about a wonderful theorem that's relevant. If you're feeling insecure, this is awful: you think about the time you wasted not knowing about this theorem, and you worry that your office mate knew it and therefore you should have. On the other hand, if you're confident in yourself, then it feels great: you learned something beautiful that will help your research, and how can it be a bad day when you learned something like that? This confidence can take time to develop, but as you feel more relaxed and bolder, everything will become more enjoyable.

Another thing to keep in mind is that progress is difficult to measure when you don't know where you're going (which is what research is!). Grad students sometimes feel bad because they don't think they're en route to solving their thesis problems. Often they're right, but that's not a problem. If you knew in advance that you were going to solve it, then it wouldn't be research. The goal isn't to solve the problem you started with, and indeed you often won't. Instead, the goal is to find something exciting along the way. Once you're used to this, you can say to yourself, "OK, probably I'm not going to solve this problem, but it's worth a try, and in any case I'm sure that if I think hard enough about it, something interesting and worthwhile will come out of that work."

Basically, I think of this as a phase transition that happens in a certain point in one's development as a researcher. Before the transition, you think "Oh no, there's so much to learn. How can I ever learn enough to be a good researcher?" Afterwards, you think "Well, I don't know that much in the overall scheme of things, but I seem to be doing research anyway. And I'm so glad there's a lifetime supply of great mathematics to learn, so I'll never be bored." The key is to relax and trust that everything will work out, even when it feels overwhelming.

I know this is easier said than done, and I struggled with it myself. For years, I felt like I wasn't a real mathematician, and I would tell myself I would be one if only I could accomplish some goal: publishing a paper, learning some difficult topic, publishing a paper I was proud of, getting a job, etc. However, it was never enough. I thought the end game was deciding I was a real mathematician, but it turns out it's developing the confidence not to worry about this, and I've been much happier since that point.

Does it get better with time and experience?

sort of, in that you become smart enough to realize that there ISN'T anyone smarter who would have figured it out in a couple of minutes :)

Is this a sign that research is not for me and that I should seriously consider a life outside academia?

Certainly not !

Research is hard work. You're on the cutting edge, charting territory no one has explored before. It takes courage, persistence, energy and a VERY THICK SKIN for rejection. After all, (and this pertains to CS), probably 95% of your job applications will be turned down, 75% of your papers will be rejected the first time, a grant proposal has a 1 in 10 change of succeeding.

But it's the small sublime moments of joy when you realize that you've discovered something that no one else knows that make it fun. And the feeling, as time goes on, that you're immersed in a wonderful lake of , with beautiful new ideas around you as far as you can see.

p.s the advice you were given is very sound. Take breaks, find fulfilling things to do outside of work, and realize that everyone (even seasoned researchers) feel the same frustrations and highs that you do.