Writing a letter of recommendation for a faculty colleague I cannot highly recommend

There is quite a bit going on here.

First things first: you seem to suggest that it is a code of honor among academics to never write a weak or negative recommendation letter or to omit all weak points in a letter. I think that's too strong. Indeed, when people suggest declining to write letters, I think they are mostly thinking of letters for students of whom they do not have a strong opinion, rather than a clear, qualified negative opinion. I would say rather that you can write a letter for whomever you want, and omitting weak points that you feel are crucially relevant to the position being applied to is an ethical lapse.

In your case, you say that you have strong feelings and strong evidence that your colleague Steve should not get another tenure track job. Given that, I think the correct thing to do is to write a letter for him. In that letter, you should take extra care that your criticism be 100% factual. Indeed, I would mostly or entirely restrict yourself to reporting the facts. You should be able to trust a hiring committee to put the pieces together correctly.

As an aside: getting a tenure track job is a significant achievement in the current job market. Getting a tenure track job after being denied tenure is much harder still. The idea that Steve can get another tenure track job using letters from the department that denied him tenure boggles my mind. I think that's essentially impossible: any positive things said in such letters will have to be outweighed by the fact that the department as a whole denied him tenure! (Or possibly the department supported him and his tenure was denied at a higher level. Even so, it would be hard for department members to make a convincing case.) A nice letter from the chair about grant support is not going to cut it.

There are some other aspects to your question that I find more alarming.

Steve has already informed this student that if she does not comply with helping him obtain a new position, he will not allow her to defend her dissertation in a few months.

That's really horrible. It's so horrible that in my view if you know about that and are not doing anything about it, you become somewhat complicit. Can you not take this student as your own? I think you should (or find some other way to ensure the student lands on her feet).

By declining to write Steve a LOR, I fear that he will refuse to cooperate on the pending publications we have. (And he has not exactly been cooperative to begin with. Again, there is a reason he is being dismissed). This would obviously affect my publications that I have worked on for the past two years. (I regret even getting tangled up with Steve. But that's a discussion for a different day).

Indeed maybe you should not have collaborated with someone who is as uncooperative and immoral as you say Steve is. Some academic investments don't pay out. You don't need, and apparently don't want, to be held hostage to this person for the foreseeable future. It sounds like you have no guarantee that the work will come through successfully no matter what you do and that it very much depends on Steve's future career. I would seriously consider decoupling from him, even if that costs you a certain amount of academic work.

In summary: I suggest honesty all around while making extricating Steve's poor student from the situation as your top priority. Steve is trying to strongarm his student from a position of no power -- the university administration is not going to support him; on the contrary, they have already dimissed him! You should not pretend that you approve of Steve's actions. Neither should you try to "blacklist" him: just let the facts speak for themselves.

First off, I would get some kind of documentation of him threatening to prevent the Student from defending her dissertation. Either capture an e-mail he sent to the student, or get some kind of recording of the conversation. That is a unconscionable course of action. Your department chair should be able to tell you what they can do for the student to help secure their future.

I've had similar problems in the past where coworkers asked me for letters that I could not in good conscience provide. My method was to provide a full copy of the letter to the coworker first, before sending it to anyone. Then, I told them to their face "This is the letter that I wrote. You can decide for yourself if this is what you would like submitted on your behalf, or if you'd like to ask someone else." This method would avoid Steve having any defamation case, since he has consented to the "publishing" (giving it to anyone) of the piece. However, make sure to be fair. List his positive qualities (like being very good at getting grants) also.

In fact, I follow this practice for all of my letters of recommendation. I don't like talking about people behind their backs, good or bad. Also, since this you mentioned the "revenge" aspect in your edit, I will also say that delivering not-so-glowing recommendations to someone's face is pretty satisfying.

As for your collaborative work, I would avoid the sunk-cost fallacy. I would be wary of the fact that Steve is clearly already not well thought-of at your institution, and I wouldn't want to put my name on a collaborative work with someone like that regardless. If he does end up job hopping to other institutions, it becomes pretty likely that he won't be well thought of in those places as well, and having your name on a work with someone like that could harm your reputation more than help it. The same is true of the student.

This turned out to be too big of a comment, so I post it as an answer.

It seems like you should cut your losses. As Pete L. Clark's (great) answer says, not all investments work out in the end. If you're already tenured this should be fine.

The really worrying aspect of the situation is the grad student's situation. If it comes to it, wouldn't it be possible to compensate the hypothetical lack of publications with a strong letter of recommendation for her, explaining why these publication weren't submitted? You're probably the only person with power who knows about the situation. Help her.

As a final note I'd like to mention that another important thing to take into account is that Steve is not the only one with bargaining chips here. Withholding these publications will certainly affect his possibilities of getting a new job in academia.