Should I withdraw a paper if the Editor-in-Chief doesn't like it, but the reviewers think it's great?

The basic issue is this:

The Editor-in-Chief's excessive involvement in your paper represents a serious conflict of interest on the EIC's part.

The reason it's a conflict of interest is that the editor-in-chief is a proponent of theory X, and wants to force you to rewrite your paper in a way consistent with his "pet" theory. This is entirely inappropriate behavior, because if the paper was written on a different topic, the editor-in-chief likely would not have intervened in such a manner.

Given that essential fact, it is entirely appropriate for you to withdraw your paper and submit elsewhere. At the same time, I would also bring this matter to the attention of the journal's publisher, to alert them of an editor-in-chief who is clearly overstepping bounds by inserting themselves into a publication in a highly inappropriate way. (I interpret the editor D's comment to be a warning to the editor-in-chief that he's not happy with the way things are proceeding. Therefore, I would explain to editor D in an email why you feel forced to withdraw the paper again.)

Your question splits into several, which I will try to address separately.

• Is it ethical to withdraw a paper after peer review? It depends. Sometimes authors use peer review to improve a paper, withdraw and resubmit to a better journal. Effectively, they exploit reviewers of one journal to improve their paper for another - clearly an unethical approach. Sometimes authors merely disagree with editors' decisions and do not want to make changes as suggested. In this case withdrawal is not unethical, but as Alexander Woo mentioned it would be nicer if you sent some final version to the editor asking them to accept or reject, instead of withdrawing.
• Is Editor D right to be concerned? Yes. Editor's role is to safeguard journal's resources (peers included) and making sure they are used for the best of this journal (and only then for the best of the whole community). As academics we may care about wider benefits first, but editor's role implies some administrative duties, and Editor D is right to be concerned about it.
• Is Editor-in-Chief right to promote his theory? No, I don't think so. First of all, I never heard of a Editor-in-Chief be so involved with a particular paper. In my experience, this role assumes leadership, not hands-on-work, so receiving comments from Chief is highly unusual. It is not clear it the comments should be treated simply as comments of another peer reviewer, or if they are more authoritative. This makes the whole situation a bit weird and possibly implies some potential conflict of interests here.

tl;dr: What to do? I would suggest to publish anything related to this line of research in other journals to avoid deepening the conflict of interest which is already sparkling between you and Editor-in-Chief mentioned.