Research on strategies for student recruitment

There is a nice survey report online: "What Matters to Student Success", a report from the Government of Kentucky.

The work analyses the factors most crucial to student success in a college. It also discusses recruitment of students. For example,

Consumerism colors virtually all aspects of the college experience, with many colleges and universities “marketizing” their admissions approach to recruit the right “customers” — those who are best prepared for college and can pay their way (Fallows et al. 2003)

It does not exactly answer your question on strategies for selection, but gives some insight to what makes a successful student pool, for which admission process is one determinant.

I am sure the reference papers cited in the journal (and their citations) will assist you in an extensive literature survey.

For a relevant journal, I found this: Journal of College Admission. I do not know about the quality, but I was able to find this discussion (or thereabout) in a few papers herein.