Is it frowned upon to go to office hours as a graduate student?

Grad students who come to office hours with good questions that show serious engagement with the material (e.g. attempts to solve the problem themself) are generally seen as mature, hard-working students.

It is also a good way to get to know faculty in your department, which can be useful e.g. if you are looking for a research assistantship, or recommendation letters, etc.

Personally, when I teach, I like when students come to office hours with good questions - it helps me see how students are understanding the lesson, where common points of friction are, and thereby improves the quality of my instruction. It also helps them do better in the course, which is of course a goal we have in common. But, I enjoy teaching and engaging with students. Some faculty in my department don't like teaching, and don't like office hours - but even those faculty members don't look down on students who make (good) use of them.

One common mistake grad students make is not asking for help when they need it, either from professors or advisors, out of a misguided belief that they're supposed to demonstrate independence or ability by not taking advantage of the resources available.

No, it is not frowned on to go to office hours. If a professor goes out of their way to make a resource - like office hours - available to you, you should assume that it's intended to actually be available.

My experience and answer only covers UK academia; the situation in the US might be different (or not).

In the UK, graduate students are paying fees to University and also often contribute to research. I would be unpleasantly surprised to see a colleague who would frown upon such student seeking guidance and support. But to be honest I never yet seen anyone who would frown upon a student coming to office hours.

What is expected of graduate students, though, is an ability to become independent learners and researchers through the course of their studies. So I would expect such students, in particular, to demonstrate a significant attempt to solve the problem, including (but not limited to) a list of literature they attempted to search/study to solve the problem. I would not frown on a student who came to my office hours completely unprepared, but I would perhaps send them back to the library to attempt to find an answer to their question first.