Reporting a student who may be psychologically disturbed

The "administration" is probably the wrong place to look. Instead, see if you your institution has some mental health staff, whether just associated contacts, or counselors, or even a dedicated behavior concerns advice function, to get proper advice. If the university is well-organized enough to have one of the latter, chances are they've also established policies requiring you to pursue this. And such an advice line would be well suited to tell you how to go about it. If your university doesn't, figure out where to report it.

Note that, unless your institution is highly dysfunctional, reporting concerning behavior and having a professional talk to the student in question shouldn't be considered a punishment, but a safety net.

As Buffy points out in a comment, the law may further require you to do some things, and avoid doing other things. While one would hope that the mental health support staff are well-informed, contacting the legal staff (University Counsel) may be useful. That said, as a non-lawyer person, I imagine most legal pitfalls concern naming the student when you shouldn't. Your initial inquiries need not include the name of the student, until you're convinced that you should.

Not to be dismissive of this, but to be honest he sounds like the average "edgy" teenager/young adult. Maybe he's a fan of horror/slasher movies/history or just interested in serial killers specifically. I'm very interested in this type of things myself, and I'm not violent in the slightest. I also did some questionable school projects when I was younger, not only because of my interest in these subjects but also partly for shock factor and to stand out from the other works (which seems to have happened here).

That being said, there's always a chance he could become a serial killer or show signs of violence, etc., but anyone in your class could without explicitly demonstrating it. If anything, show it to your superiors or talk to him about it just to clear your conscience, and if you notice any other worrying signs maybe suggest an appointment with a mental health specialist.

Grade wise, I do think he deserves a high grade if he did a good/accurate and detailed work, independent of opinion or personal taste on the matter.

[A student wrote] a diary for a famous American serial killer... graphically detail[ing] the killings this man performed, including hand sketches of female genetilia... [The student will likely receive] a fairly high grade on the project.

It seems like the student should be commended for their account of historic events.

It is a horrifically disturbing piece of work.

That seems like an excellent portrayal of a serial killer.

(as far as we can tell) splatters of real blood

You've acknowledged that it might not be real....

I'm left thinking: The student should be commended. That said, I don't have the full picture and you certainly "do not want to become part of history ourselves by being 'that one professor.'"

The OP has been commended (in comments, e.g., by Nic) for acknowledging that they feel somewhat out of their depth. So, this seemingly isn't an issue that the OP should deal with alone: I recommend delegating responsibility to your department head. This isn't shirking; it is the right thing to do. (At least in my opinion.)

There are also ramifications to consider: As commented below, "[I]f it all blows up, you can point fingers. If you don't, and it all blows up, fingers will be pointed at you."