Academia - What should be done about highly controversial answers which attract many comments?

First of all

It […] deprives the community at large of important follow-up content.

Almost nobody wants to read a discussion spanning twenty or more comments. This apparently even applies to comment authors – going by comments that add nothing to existing comments. In fact I would wager that the only people who read all the comments on the answer in question are its author and some moderators. The point of moving comments to chat is to make those comments visible that are the few ones that are of high interest to future readers on their own – be these existing or potential future comments.

Also, please remember that comments are mostly intended to be temporary, i.e., to be made obsolete with an edit or similar. The main exceptions to this are relevant links, but even those can be edited into the post. If you want to say something of lasting value, do not say it in a comment.

What should have happened

Ideally, almost none of the fifty-something comments following the initial moving to chat should have been posted as a comment but in chat. This is not because those comments were entirely pointless, but these comments were either very likely to incite further replies or were only of value in the context of the entire debate surrounding this answer. (Please see this as to what kind of comments I consider worthy of being comments after comments have been moved to chat.)

What should happen in a similar situation in the future

In the future, moderators should include a link to this FAQ to every moved-to-chat notice. If you post a comment after such a notice you have to live with the possibility that your comment is removed without warning. If you do not like this, do not post a comment but in chat.

Comments that do not follow the FAQ above should be removed as soon as possible to avoid the strong broken-window effect that this situation has on some comment authors (“If that opinion deserves the honour of being a comment, so does my contrasting opinion.”).

What to do with the post in question

After the initial moving to chat, not a single message was posted in chat. Any of the comment authors could have said or thought:

We are having a longer discussion here, let’s move to chat.

My comment is essentially a reply to a chat message (being a comment moved to chat); I’ll post it as such.

This comment will likely incite a longer discussion, I’ll post it in chat.

They didn’t – despite a moderator’s comment saying that comments are not for extended discussion.

On the other hand, we moderators are not completely without fault either. We could have intervened earlier or more clearly.

I exploited some special properties of the situation, abused quite a few features, and spent a considerable amount of time to move the entirety of comments into one chatroom. This is a one-time thing. Do not expect this to ever happen again.

I know your answer has been controversial and you received a lot of both positive and negative comments. You have handled that, often hostile, feedback well. Your responses to the feedback have been on point. The main issue that I see as a moderator is your point

The deletion of the comments thus seems quite detrimental to a high-quality discussion and contrary to the goals of the site.

I strongly disagree with this statement. The goal of this site is not for high quality discussion, it is for high quality answers. The point of comments on answers is to help improve the answer. That improvement process might require a back and forth discussion, but the end result should be an improvement to the answer.

When reading the comments and responses, it was clear to me that you thought about the comments and were not going to integrate the concepts in your answer. Given the nature of many of the comments, that is a reasonable decision and yours to make. At that point all the comments became obsolete as they were no longer going to make your answer better. If someone felt strongly about the views expressed in those comments, they could have written an answer.

The answer is simple, of course. The comments should be moved into the existing chat. If the moderator can't because comments can only be moved once, then this strikes me as a serious flaw in the system.

I had a related issue recently. I posted a comment on an answer explaining my vote and ways to improve the answer, and the comment received several dozen upvotes. A long series of irrelevant comments followed. The entire block of comments was moved to chat, including mine. I queried that and was told that there's no way of selectively moving comments to chat. To me that seems to defeat the point of the comment system, which is supposed to be used for suggesting improvements to the answer.

I would suggest that moderators need finer tuned controls for deleting and/or moving comments, and site policies should also be finer tuned. Comments should be deleted or moved based on their individual merits, and not as an entire block (which is akin to throwing out the baby with the bathwater).

As a final point, deletion really ought only to be used on comments that have serious problems (e.g. are abusive). With the availability of move-to-chat, there is little reason to delete a comment.