Should professors pay undergraduate students doing research in their labs?
Generally, yes undergraduates should be paid for their research. This is an equity issue. At many institutions, a large portion of the student body has to work while they study in order to survive. If student research is unpaid, those students will not be able to participate. Students who have to work are often also members of other underrepresented groups such as racial minorities.
It is also a justice issue. If students are doing valuable work, it is only fair to pay them. It is also unjust to expect students to work more than the assigned hours.
I should add that it is not appropriate to pay students for work that earns course credit.
If your research students are not achieving what you want them to do, you need to be sure you are setting clear expectations. You also need to be willing to part with under-achieving students.
I was an undergraduate who was paid for his work (rather than receiving credit; I had the option of one or the other, but research credits in the department I worked in were useless to me). I worked at that lab throughout undergrad and am now a staff scientist in the same group.
You really should give your students one or the other, or both, if your university allows it. It seems to me fairly unethical to have someone do work for you without compensating them. Of course, there are realities of budgets, but you really should strive to pay people for work they do.
(Honestly, it's ridiculous that you have to pay to take research credits, but that's a different post.)
They also work exactly the assigned hours (says 10 hours/week). You know, like the regular employees in a company, not like a researcher in academia.
Well, yeah. Before I was hired, we set an expectation that I would work 10 hours a week in my mentor's lab. (In fact there may be a limit on students' working hours, my university's was 19.) I don't understand why you think anyone would work hours for free when they are paid hourly. If you need more than 10 hours/week out of them, then hire them for more hours.
Even if you had volunteers; you should set some sort of expectation with them for how many hours (on average) they should put in. Yes, some weeks will be more, some weeks less, but a volunteer should know how much work you want out of them.
Only the students who are really interested in doing and learning research will volunteer to work in the lab for free
This is not true; let me be a counterpoint. Only the students who can afford to will volunteer to work in your lab for free. That is, students who have support from their parents, or who don't already work a job for money will work for you for free. I am sure that the best undergraduate research assistants do not come solely from the pool of people with wealthier parents.
Am I doing it wrong? I'm not cheap, but if paying the students to do research results in wrong motivation and expectation for them, and if not paying them is a good filtering mechanism to select good students, maybe I should do that.
No, not paying them is not "a good filtering mechanism to select good students." You sound like a new professor, and I think your own "filtering mechanisms" will improve as you interview and hire more undergrads over the years. At my lab, we went through about three for a position over three quarters before settling on a good, motivated undergraduate.
Edit:
After thinking about this for a few days, I would also point out that some types of relationships don't need to be paid. For example, students at my university can do honors projects in their department. Supervising these students is more of a service from professors than an employer/employee relationship, and undergrads seek out professors to supervise them, rather than professors recruiting students.
I assumed from your description of your students having been hired for 10 hours/week that this isn't the relationship you have with them. To put it another way, if you are getting value out of them that either you or a graduate student would have had to perform (e.g. feeding rats, interviewing human subjects) you should pay them. If you put out an ad looking for undergraduates, you should pay them.
In terms of equity, you should always be trying to pay your undergraduates, but I don't want to deny the existence of some relationships where not paying them may not be unethical.
It's unfair to take valuable labor from anyone, including students, without compensation. On the other hand, new undergraduates just starting in a research lab are often 90% training and 10% productivity (*). For new graduate students, it might be 75%/25%. So the training alone might be pay enough when a student is first starting out.
(*) 1 undergraduate summer = 1 graduate student month = 1.5 postdoc weeks = 1 professor week. That may sound harsh, but it's my own experience after 30 years spread across all four of those categories. It also varies by student.
A typical progression might look like this:
Join the group for a month without pay or credit, and demonstrate that you can contribute and collaborate. Solve a simple technical problem or two.
Get paid relatively low wages for a quarter/semester/summer to be a simple assistant to others in the group: an apprentice. Just do what needs to be done that day. Become an expert? Higher pay.
Get research credit for a project that's actually yours. You're expected to produce publishable results, if not a full paper. Increased future career earnings are worth the cost of tuition.
Get a summer or other research fellowship that combines the best of 2 (pay) and 3 (interesting project).
Those who want more pay or more freedom will work hard to earn it. Those who don't aren't rehired next term. I had one mentor tell me to "fire" one student each year. Looking back, I wish I had accepted that advice much earlier in my career. Be explicit from Day 1 what you expect so they're not surprised and you're not disappointed.