How to screen out candidates for faculty jobs who don't know the subject

Presumably your interview is doing a good job screening out the individuals who you feel "don't know the subject" and you are trying to screen them out prior to the interview. I think a reasonable screening tool could be a a phone interview. You should probably conduct between 10-15 phone interviews to find the 6 candidates you want to interview.

While I say "phone interview", it is most often now a "Skype" interview. These interviews could be as short as 30 minutes and should be no longer than 1 hour scheduled back-to-back such that they all get completed in two long days. The phone interviews I know about have had 3-6 people (presumably the majority of the committee) present, but I think depending on department politics that you could reduce it to a smaller group (maybe even just the committee chair).

While you could simply focus the phone interview on the questions that cause candidates the most trouble during on campus interviews, given 1/2 your candidates have difficulties recalling key concepts of "random" classes, you may want to help them out a little. If you require that applicants submit a sample syllabus for freshman physics and another for an upper level elective, then the phone interview should focus on the teaching statement and sample syllabus.

During the phone interview you should also mention some of the other key classes of you department that they might teach on. If a candidate is truly clueless, this will not matter, but good candidates will realize that they need to brush up on those classes for the on campus interview.

In summary, an inability to talk about a syllabus the candidate has written would be your screening tool and the hints during the phone interview will help you minimize throwing out potentially good candidates.

You think that asking one or two questions on the spot in an interview is a good way to assess an applicant's knowledge of basic concepts in the discipline. However, this sounds like a relatively poor approach to me.

There are basic principles of psychometrics that are relevant here. Basic principles of item response theory are relevant. First, people differ on a distribution in latent ability (i.e., knowledge of the domain). Second, items differ in difficulty. Third, even with these two bits of information, there is a random component. So if you want to measure latent ability in applicants, then you can improve your assessment by having more items. For example, give them a written test with a bunch of items.

More importantly, I think you need to be be careful with assuming that most people can recall basic facts from the undergraduate curriculum. If you teach that curriculum, then you are likely to recall many such details. If you don't teach such material, even if you've learnt the material and would know how to look it up and solve problems when required, it may not be available to immediate working memory. It might be easy to relearn and if you were given the task of teaching a class related to that content, you may still be able to do a good job presenting it.

In summary, you may want to think about the degree which being able to recall undergraduate knowledge at the time of hire is predictive of job performance. I imagine that it might be relevant, but I also think that a whole range of other indicators would be relevant too. So in addition to trying to measure the construct better, I'd recommend that you treat it as one of several indicators of potential job performance.

There's a famous test in the CS world called "FizzBuzz" that companies these days are using to efficiently screen out candidates that managed to get degrees without actually learning anything. FizzBuzz (and its variants) focus on what are considered to be core concepts that any programmer regardless of language should be able to do in a matter of minutes. As reported, the results are astounding. While material on the effectiveness of FizzBuzz in recruiting has never been extensively peer-reviewed (to my knowledge), coming from that world and even being a self-taught programmer I can tell you that anyone worth their salt should be able to complete the test in at least one language, even if the solution is inelegant.

It sounds like you're running into a similar problem in your neck of the woods, and are already onto a similar solution. I'd recommend trying to find that core set of problems that any undergrad should be able to solve one way or another and use that as your screening. Let them use a textbook, maybe hint at a list of possible problems ahead of time as others have mentioned so they aren't caught off-guard, and be willing to help the way you've been doing. But make the first interviews (possibly over Skype as was also suggested) quick and to the point so you don't waste time. Establish either the base knowledge, or at least the ability to pick up on it quickly while under pressure, before proceeding to full, in-person interviews.

Those tricks have worked in my industry, hopefully they can help you in yours. We've seen that it doesn't matter who graduated with what degrees and has what recommendations or grades, the ones who can actually live up to their title and do their job are sometimes the least-decorated.

Edit: These "quick" interviews can be done on Skype in at most 30 minutes, possibly more like 15 depending on the questions. You can knock out a lot of candidates in a short period of time before dedicating serious resources to second-round interviews (which may still be online and only last an hour), assuming you think the process through in the beginning.