Will I look bad if I'm never in my office as a graduate student?

Look bad, not necessarily. Will it be good for you? Maybe not.

Assuming that you don't have a really significant reason not to be on the lab, you should be. Stuff only happens with people that show up. If you are not in the lab, you won't talk to people as much. I literally can't count how many good ideas (and papers) came out of random coffee talk, that wouldn't happen on an otherwise "official" meeting.

full disclosure: That's something I didn't do during my phd and I regret it. I had my reasons, probably not really good enough, but I was almost never at my office and that hindered my development. I could have done more if I was there.

Never is an absolute. Having said that, you have outlined reasons why you should use your office (office hours, spending time between department events and classes etc.).

Unless your adviser makes it mandatory (highly unusual), I don't see why you shouldn't work wherever you feel most comfortable in. I was very fortunate, in graduate school, to have a spacious, very well lit and clean office. In spite of these structural advantages, I chose to work from coffee shops most of the time. I had no issues with the administration, faculty or my peers.

To summarize, there are some (usually official) reasons why you would need to go to your office but there is usually no compulsion to have to continue using the office at all other times.

i had a terrible office for the first 3 years of my phd. I spent some time there initially. I eventually decided it was much smarter to use it to dump my stuff and then worked in the departmental reading room which was next to the common room. I always went to afternoon tea and ate my lunch in the common room.

I therefore had plenty of social contact and mathematical interaction with my peers when I wanted it.

Work time in maths is about solo concentration and is best done in a quiet, airy space with natural light and minimal distractions. This tends to mean a library unless you are a professor.

I now have phd students. Some of the best ones only come in when they need to. That is fine by me.