Test answer rejected for saying more than asked for
I know from personal experience that it's very frustrating not to get proper credit for knowledge that you think you demonstrated correctly, so let me start by saying I sympathize with your situation. Now let me address your questions:
Is it justifiable to reject an entire answer because more was said, without even partial credit?
To give a literal, completely general answer: obviously yes. If the correct answer is, say, one sentence long, and your answer quotes the complete works of Shakespeare that happen to contain that sentence, I personally would not give you any points (but I would still be very impressed :-)). Of course, that is an absurd example, but it illustrates the point that what likely matters to your professor (or if not, certainly what should matter to a reasonable professor) is that your answer demonstrates an understanding of the topic the question asks about. It is certainly possible, and happens pretty frequently, that an answer Y contains the correct answer X but by adding more irrelevant things demonstrates a less good level of understanding than just the shorter answer X. The way this typically happens is that a student doesn't understand the material well and upon being asked about a certain topic, regurgitates everything they memorized about the topic, not knowing which part is relevant to the question. Clearly that does not leave a very good impression.
Now of course, the above does not address what happened in your specific situation. It is completely possible that your longer answer Y still demonstrated as good of an understanding of the material as the official answer X, and deserves to get partial or full credit; or not - you haven't given us enough information to say.
Is it justifiable to reject an answer that gives something better than what is required, perfectly equivalent, even if relying on the student's intervention? (in this case, it would be a compiler optimization, thus not an entirely direct interpretation, but an elegant and industrially valuable workaround)
I think this question is too specific for anyone here to be able to answer without knowing more details. As I said above, the points you get for the answer should be correlated with the perceived mastery of the topic being asked about that your answer demonstrates. If including the optimization demonstrates that mastery, you should get the points. But I can imagine a situation where using the optimized method instead of the official method actually demonstrates less mastery, for example if it appears that you used the optimized method to avoid revealing that you don't know the simpler, less sophisticated method the question asked about. In that case, it may make sense for your answer to be rejected.
Is it acceptable to direct students to example answers and not allow them to explain why they think their answer is correct?
I think it may be acceptable in certain cases. For example, if you have already asked many questions of the professor, argued with him in a way that he thought was unreasonable, overly argumentative, or delusional, and if the professor is very busy, for example is also teaching a class of 500 students in addition to the smaller class you're taking with him, at some point he may decide that spending more time discussing your exam with you can limit his ability to perform his other duties effectively, and be a disservice to the other students.
Summary. As you can see from my answers above, whether the professor's behavior is unreasonable depends on many details that you haven't provided. It's certainly possible that it is unreasonable, but it's also possible that it is less unreasonable than you may think. Statistically speaking, I've found that students often overestimate the extent to which they understand the material and the degree of correctness of their exam answers. That doesn't mean it's what happened in your case, but it's a possibility you should keep in mind as you consider your next steps. In any case, good luck! As for what steps you should take to address the situation, this answer is already quite long so I think it makes sense to save that for a follow-up question if you care to post one.
Dan Romik's answer is excellent, but I'll add a bit more perspective. Your instructor sounds a bit severe but not entirely unjustified.
Is it justifiable to reject an entire answer because more was said, without even partial credit?
Yes. I do this (perhaps in smaller doses). Here's an example: If I ask the question, "find the margin of error for this estimation", and a student works out an entire confidence-interval estimation -- which includes the margin of error as an included term -- and then boxes the C.I. as the answer, then I think: "This student doesn't know what the margin of error is", and needs to be corrected. In that case I'd take off partial points, and probably receive the exact complaint from the student that you're presenting here; after a short discussion, the student usually sees that their understanding really was lacking and needs some improvement.
Sometimes for shorter explanatory questions, expecting maybe two lines, a student will write a whole half-page regurgitation of all the subject matter of a particular chapter. If it's a 1-point question, and it seems like the student really can't incisively identify the particular item asked, then off comes the (entire) 1 point.
When I see responses like this, I'm being taken back to bad interactions I had in the past with my own teachers or coworkers in school or industry. E.g.: a teacher that when asked a tough question, wouldn't respond on-topic but would give to a random dump of other information so they'd look smart but wouldn't answer your question. Or a coworker who would behave similarly. Either case is a big aggravation, so if I can I'd like to help escort people out of thinking that that's useful/rewarded behavior. It's so much more helpful to have a colleague say, "I don't know", so you can go search elsewhere, rather than do a brain-dump of random information on you, which is just wasting your time.
Is it acceptable to direct students to example answers and not allow them to explain why they think their answer is correct?
Yes. Especially in large courses, such as your "amphitheater" course. Presumably the instructor spent time writing up the example answers and distributing them, specifically to save time from interacting with students one-at-a-time on the issue. (See also: "it's on the syllabus" comic, "it's on the syllabus" T-shirt, etc.)
Remember that at most research universities (likely ones with large amphitheater courses), teaching is not the professor's top professional priority (in terms of evaluation, promotion, etc.). Published research is required to come first, and then teaching is effectively a secondary part of the employment -- so the time spent on the teaching side, especially on large introductory courses, has to be kept very constrained as a necessity. (Personally, I'm a dedicated lecturer, not a research professor, so I'm very happy that I get to spend more time responding to individual questions from students, providing personal feedback on tests with partial credit, etc.)
It's hard to say without seeing the specific question and answer.
I can easily imagine scenarios where too much information would be wrong, because it would indicate that you don't know which portion of the answer you gave is relevant.
To take a silly example, if a history test asked, "When was George Washington elected president?", and a student answered, "One of the years 1750, 1751, 1752, 1753, ..." etc, listing every year from 1750 to 1850, to say, "But the correct year is in there somewhere! I am being penalized for giving too much information!" would be true, but clearly the student does not know the correct answer. It is properly marked wrong with no partial credit.
On the other hand, if a student answered, "In 1789, and his vice president was John Adams", I'd say the bit about John Adams is not relevant to the question and I don't know why he mentioned it, but I probably wouldn't take points off.
But in general, any information that is not relevant to the question likely indicates a lack of understanding of the subject. If a test asked, "What loops are available in Java?", and a student wrote, "FOR, WHILE, and IF", that's simply wrong, because "IF" is not a loop. Yes, you could say it's extra information in addition to the right answer, but it's flatly wrong. It indicates that the student either doesn't know what a loop is or doesn't know what an IF does.
The best case I can think of is if a question is potentially ambiguous. Life if a history test asked, "When did Barrack Obama become president?", a careful student might answer, "He won the election in 2008 and took office in 2009", because the question is not entirely clear which it is asking for.
And ultimately, if you give more information than asked for, you are wasting the professor's time asking him to read it. If a test asked, "How much is 2+2?", I'd expect the answer "4". Anything more than that is wasting my time, and at some point I'd start penalizing for it.
Update several years later
I just got an upvote (thanks!) that drew my attention back to this old post, and looking at it again, I came up with a better example than my George Washington one.
Suppose a test asked, "What causes a compass needle to point north?", and a student answered, "The gravitational pull of Jupiter, the magnetic field of the earth, and the Coriolis effect." Yes, the right answer is in there, but the fact that the student gave additional information that is irrelevant shows that he does not know what IS relevant. He doesn't understand the subject that the test is asking about. A teacher might give partial credit, thinking the student has some understanding, but he'd be justified in saying no, you don't get it.