How to motivate students you supervise to produce high-quality research when other graduate students in the program don't?

This is, as you say, a common problem. It certainly is realistic to try to get more out of your students than the other faculty. However, depending on how substantial the gap is between the norm and your expectations, getting into a better school may turn out to be the only realistic and sustainable way of aligning goals and reality. If the statement that the PhD theses in the new school are like bachelor theses in the old school is not just hyperbole, this may certainly turn out to be the case for your friend.

how can he motivate his students to work harder and produce better work when other students in the program don't?

Step 1 is certainly to figure out why they should work harder than their peers (who still graduate!) in the first place. It's likely that your friend never really gave this much thought, because in his old school, working your ass off was simply what PhD students do. Here it isn't, so he needs to find a motivator that works for them. The two solutions offered by Brian (tell them that they won't find an academic job if they don't publish, and show them what cool things they could be doing) are good, but they will work only if they actually have aspirations to stay in academia, or respectively have substantial internal drive to do good research. Both may have been a given in the old school, but not necessarily so in the new one. Other potential motivators may include:

• Getting to see nice conference venues (don't laugh - this works much better than you think as a motivator)
• Serving as a stepping stone for a cool industry internship, e.g., at Microsoft.
• Finding a problem that they personally really want to solve, i.e., real curiosity-driven research.
• Bonuses! Most likely difficult to implement in many universities, but depending on his financial capabilities, there should be some goodies that he can offer in exchange for going beyond the work of peers. It does not need to be a super-formal, but knowing that the students that work hard and get published are also the ones that get new laptops first is certainly a motivator for some students.

The most important step is really to start giving it a thought why they should be working hard in the first place, rather than lamenting why they don't.

Step 2, and strongly connected to Step 1, is then to find the right students. This was again something he has maybe not given too much thought, as in his old school most students were good to excellent. In his new school, there are almost certainly also people for which some of the above motivators will work and compel them to produce good research, but he will need to identify and actively recruit them to his cause. The most important part of this is to make clear that things in his lab are different than in the rest of the department. Students are not graduating without papers. Students are expected to work on their research X hours a week, for whatever a realistic X is, and so on. If this is the message that your friend is communicating plausibly, most of the students will stay far away from working with him, but that is ok - he does not want those students anyway. However, even at a weaker school, there are likely going to be students who are dissatisfied with the status quo and who are actually going to be appealed by your friend's less lenient approach to PhD school, especially once he builds up a track record of students actually going beyond the norm in the school.

Step 3 is going to be the most difficult, but maybe the most important, step. He then needs to meet his students half-way. Despite all of the above, he is still not going to be working at a top-tier research university. That means he will need to re-evaluate his expectations on his students and the papers they produce. This may include both, quality and quantity of results. He needs to realise that it actually is much harder for his students to produce good work in this environment than what it was for him. He likely had a stimulating and high-competition environment to work in. They just have him, and maybe one or two other more ambitious students. All the other students (who plan to graduate without papers) are frankly more of a distraction than support for them. All the other faculty are simply of not much help. At the end, he may be ok with his students not, or not immediately, submitting to the very best venues. Instead, he needs to ease them into it (without teaching them bad practices). In computer science, a good way may be to initially submit to workshops at the top conferences. This is allows students to go to the top conferences and see what the work is like that is published there, but getting a paper accepted at those workshops is achievable for most students given a little supervision. Really, the core in this step is to get away from the mindset that is common at top schools that only the very best publication venues are good enough.

What is also important in this step is the right kind of expectation management. Telling stories about your friend's old lab may be a good way to reframe the expectations his students have on how a PhD works, but he needs to pick his time and place. For instance, when his first student gets a paper in the major conference of the field, this is a big deal and should be celebrated as such. This is certainly not the right time to let everyone know that in his old lab they had papers in this conference every year for the last decade.

Finally, Step 4 is to make alliances within the faculty. It is likely that there are other (younger?) faculty in the department that are also not happy with how the department currently operates. They won't be able to turn the department as a whole around, but it will certainly be easier to improve their own labs if they share their experiences. From a motivational point of view, it is also a good idea to try to socially align the labs of the higher-aspiration professors more tightly - if your students are friends primarily with the students of the other faculty who expect their students to actually work, it will be much easier for your students to keep up motivation. Ways to achieve this include joint social events or retreats, joint talk series, and of course, if possible, joint research and publications.

In my experience, one of the best ways consists in leading by example. He can try and publish excellent work on a regular basis, and he can require high standards for any publication from his team. He can say that it is a matter of reputation and even point out the importance of international competition in the field. After a while, I bet that there will be at least one student who would like to be part of this "excellence" group and show that his or her work is worthy of the requested high quality. This could become a positive upward spiral, since other students in the team won't like to be left behind.

One strategy would be to explain to the students that their success on the job market after graduation will depend greatly on the quality of the research that they've done as graduate students.

Another strategy would be to expose the students to high quality research being conducted by students at other universities (e.g. by taking them to conferences) so that they can see beyond the provincial confines of their department.