Is it unethical for a professor to demand students buy their book then force autographing it before they can submit any work for the semester?

Yes, some aspects of this are unethical. I'm surprised that a university would permit it to happen. They would, I hope, object if they learn of it. Especially if they learn of it from complaints to higher authorities.

In my personal view as a professor and author, I can, certainly, assign one of my own books in a course I teach. But I believe that ethically, I should return to the student any royalty payment that I receive from the publisher. To avoid student cheating, I can purchase their receipt for the book from them so that they can't simply return it to be sold again as new.

I think that the professor in question is trying to do something similar in part of this (autographing the book), but in an unethical way.

I would think that autographed copies can be sold back to the bookstore, but not simply returned as new. Some autographed works are worth more, actually. The professor doesn't profit from the sale of used copies, of course. Only the "first sale" results in any royalties.

I don't know how the professor "verifies" the purchase. Does he keep a list, or just ask to see an autographed copy?

But, the issue of profiting from students when you require a purchase in such a way as to guarantee a profit is clearly a breach. As you describe it, it sounds like a clear form of coercion that should not occur and actually can pollute the relationship between student and professor.

And obviously, most professor-authors think their own books are the best.

Yes I think it is unethical. If your university has an ombudsman or similar office, go to them and ask their opinion. You may be able to report this anonymously.