What are appropriate questions for a closed-door (non-public) part of a PhD (or master's) defense?
Generally speaking anything and everything is on the table unless you are told otherwise. There are no limits. It is up to individuals what is asked and up to those and other individuals to evaluate the answers. The candidate may not be able to answer everything.
But it probably isn't required to answer everything. What is required is to say sensible things, including "I never studied that".
But for the foundational knowledge of your field, you'd better be able to give pretty good if not absolutely ideal answers.
The (second) worst story I ever heard, though it may be apocryphal, is one in which the candidate in biology or chemistry gave his dissertation presentation in which he repeatedly mentioned pH. There was an outside member of the English department on the committee who said she didn't have any real questions, not understanding the subject, but would like some layperson's explanation of pH from the candidate. The person, who had long ago studied it formally and was able to use, froze up and gave no answer. He failed.
But maybe that was just a horror story told around grad students to scare them into working harder, something like a campfire ghost story.
In a private oral exam of my own I was asked a question and started to develop an answer (math - algebraic topology) and it went nowhere. I stopped and admitted that I wouldn't be able to finish it, but was able to say exactly what the flaw in my argument was. This actually impressed the examiners more than if I'd given the correct answer immediately, as they told me later.
So, you don't have to be perfect, but you do have to make sense. Do that and you will probably be fine, subject to the vagaries of personality.
As the norms and rules vary enormously from institution to institution and department to department even within the US, ask your supervisor.
In my experience the supervisor and committee do give at least broad outlines of what's expected/what topics will be covered. (My personal opinion is that we [academic supervisors] give way too little guidance about what to expect, making the process far too stressful.) You can and should at least ask for this guidance.
Also in my experience, the broad "anything goes" questioning you describe in your question applies more to comprehensive or qualifying exams, where the goal is to find out if the student has an adequate level of foundational knowledge etc. to begin a research program, than to the defense. The PhD (or master's) defense is usually much more focused on the material in the thesis/project itself, although it can of course stray into related topics (see the answers to the question you linked about PhD defenses for more discussion of the typical scope of these exams).
As for
Is it appropriate to fail someone for not answering seemingly random questions - as if they are to have encyclopedic knowledge?
When you put it that way, of course not. But ... a student's and a professor's view of "seemingly random" is often different. A professor might legitimately feel that a researcher in field X should know about topic Y, even if that seems unrelated to the student. Again, I think it comes down to reasonable expectations being set and communicated before the student starts to study for the exam. (As I say in my third paragraph above, that applies to comprehensive/qualifying exams. Students usually shouldn't even have to study for their thesis defenses — the defense is about exploring details of a topic that they've just spent several years immersed in ...)
I agree with @Buffy's points that making any reasonable effort to answer the question will count in your favour (e.g. "well I don't know that but it seems related to ..." or "let me start working that on the blackboard and see how far I can get ...") Again in my experience your examiners are actually on your side and want you to succeed, they will usually provide hints if you make a good-faith effort to tackle the question.
One of the best pieces of advice I have received is to defend only what you have done. That means that you have to be an expert in your thesis and to a reasonable level in the field as a whole. This perspective enables answering tough questions:
Why haven't you done X? We thought about doing it, however after analyzing our case specifically we opted for Y because of A, B, and C.
Why X is missing? Agreed! It would be good to have it. I did Y instead because it seemed the right way at the time because of A, B, and C.
Your result X goes against state-of-the-art! True, however we take into account A, B, and C that others have overlooked.
Any tough question can be turned into a silent agreement and "selling" what you did instead of defending what you could have done.