How to deal with sources whose authors I don't have good relationship with?

If you feel that the work is relevant, you must cite it - not citing would be unethical.

Also, most people are happy if they are cited - regardless of their relationship to the person who cites them. Citing someone does not mean liking someone - even when criticizing someone's work, you have to cite.

And if you can think of no particular reason why they should not want to be cited by you, there probably is no such reason.

Yes, you should cite their work. The only reason I can think of for not wanting to cite it is if the other person believes that you have misinterpreted their work and have told you not to cite it.

Even then, if your work is based on theirs you need to cite them. I would be wary, of course of making disparaging comments about their work in a future publication. If you think you have done an improvement over what they have done, let your own work speak for itself. Of course, you shouldn't misrepresent what they have said, but that is true whether you have any personal relationship or not.

Who you cite is your business and nobody else's. You don't need to ask permission to cite(unless you're reproducing images or something*), and nobody can deny you the right to cite them, for whatever reason.

More importantly, if your work draws from someone's work, you must cite them, it's imperative. No personal equations can change that.

*In this case too, as pointed out by corey979, the permission is for copyright rather than citation, and the parties involved could include the publisher.