Professor refuses to write a recommendation letter to students who haven't written a research paper with him

I wanted to ask whether his refusal for recommendation letters is reasonable

My personal opinion is that if you worked in his lab for a year (satisfactorily), it is unreasonable to refuse a letter. All the more so if you had other good interactions (TAing, etc.) with him.

and also ask for some advice on how to handle this situation!

Unfortunately, my opinion counts for absolutely nothing. You cannot force this professor to write you a letter, and would be unwise to try. All you can do is find someone else. Preferably someone who can comment on your research, or at least someone who can confirm that your advisor almost never writes letters for undergrads (i.e., vouching that the lack of a letter from your advisor should not reflect poorly on you).

We cannot say if they are reasonable. We only know the details you provide and, moreover, we are not judges.

But this is the wrong question. Why should it matter for you if we say the prof is reasonable or not? There are many unreasonable people in the world. Among those there are professors, students, astronauts, beggars, social workers, farmers,.. with one word: everyone. So don't ask whether they are reasonable. The answer does not help you.

How to handle this: Ask another prof if available.

Yes, the refusal seems reasonable in general. The professor is the one who should decide who he writes reference letters to, and if he has a policy of providing letters only to collaborators that is his right. Though unfriendly to undergraduate, it does have some merit: his letters can be trusted to a very high degree.

It is also possible that the professor does not think he can provide you with a strong letter. He already saw you in his lab, and may have not formed a positive view of your achievements at this stage. He may try politely to avoid the need to say this to you hiding behind other excuses. He may have suggested you become his master's student since he believes you are about to decline, or since he genuinely wishes to give you a second chance to prove yourself.

More importantly: any kind of hesitation from a potential reference letter writer should be an immediate warning sign that the person may not give you a good letter.

Conclusion: find another professor.

Important note: The OP has clarified that "Direct PhDs are not possible in my country." This means the OP is not from the US. Therefore, the Professor may not be accustomed to providing undergrads letters, only master's students, making the case even stronger for the professor to refuse a letter.