What to do about referral for student I would not recommend?

Why do you dread his reaction? You have no responsibility of giving anyone a referral. I doubt "this professor did not write me a referral" would stand as an excuse to allege persecution in any tribunal. Don't get bullied by a student, just plainly refuse to do a referal if you are not up for it.

There are ways of writing a recommendation :

1) this student attended my course on snail migration in semester X.

2) this student attended my course on snail migration in semester X and was always on time , with the homework completed.

3) this student attended my course on snail migration in semester X, was always on time, always came prepared and actively and positively contributed to each session.

One other method is the “omission” method ie if timekeeping is not mentioned at all in the reference - then that says they are often late. Some do not like to put negative comments and just saying “attended” compared to “regularly attended” or “always attended” can speak volumes...

Have you considered that this student may be oblivious to their problems, not realizing they exist and hence not able to take any action to solve this? I have seen this a number of times, where nobody would like to work together with a certain student, but nobody tells the student in question this, and hence the student is not even aware that people feel this way.