Is it wise to select a PhD topic before choosing an advisor?

In brief: no, in mathematics in the U.S., you do not "have to choose" a research topic prior to beginning your PhD. At the same time it is _of_course_ good to have something to be enthusiastic about, curious about, fixated upon. But that can change, and probably should change as you learn more. It is apparently the case that in the U.S., and to some degree elsewhere, in mathematics, undergrads and even M.S. students really don't get enough background to understand contemporary mathematics at all, except very superficially. Fine, that's not a moral failing... maybe just evidence that there's a lot happening in mathematics... (!?!) But, then, in that light, the enthusiasms that novices have are necessarily (by-far-typically) ill-informed... and will change if/when they learn more.

This is not to say that one should not have an opinion, or should not admit interest in things because one knows one's interest will change... but only that one should anticipate that change, rather than believe that one "has arrived" at age 22 or so, and there's nothing left to learn, and now it's just "trying to solve problems/do research". I understand the appeal of that, but, if we think about it, it's better in the long run if mathematics is not that shallow or superficial.

More specifically, spectral graph theory is a quite viable research area. "Expander graphs" are popular and google-able. Lots of people work on this, both from the "spectral side", and from the "graph theory" side.

And, perhaps counter-intuitively, the possible fact that there are not so many "books" treating a topic can in fact be a very positive sign... it's unclear. That is, if there are zillions of books on something, then chances are good that it's an older topic and has been worked over pretty hard... leaving less room for beginners to contribute.

So, sure, it's entirely reasonable to have specific and fine-grained interest... as opposed to "general" interest. Like having a specific lunch rather than a lunch-in-general. :) But be utterly open to changes in your viewpoint... as you learn more, which will be for many years. :)

True research is driven by passion. It would be better to select a topic that interests you and then search for an advisor who could work on that topic. All that matters is how fine-grained your initial topic ought to be. If it is too narrow, it might be difficult to find a suitable advisor.

Coming back to your problem, Spectral Graph Theory is actually a subtopic of Graph Theory. There are many professors researching on that topic. Just because it is not stated in their research profile doesn't necessarily mean they would not work on it.

The key for a good PhD candidate is to be flexible and open to new ideas. Your initial idea may be readily accepted for further research by your prospective supervisor, or you may be assigned with a new one. But the field of the topic is your choice to make.

Best wishes on your PhD journey!