How detailed should one be in writing the referee's report?

When I write a report I try to be as detailed as possible in my response to the authors. This is important as they will need my comments to improve the manuscript, and if the comments are vague they will fail to do so. However, many of the journals I have experience with in my field (life sciences and bioinformatics) ask referees to avoid including comments regarding the suitability for publication in the letter to the authors. For that purpose they provide the score sheet that needs to be filled when submitting the report (at least for journals using electronic systems).

However, those score sheets cannot convey all the information about the impressions of the referee after reading the paper. Therefore, there is always a section with comments to the editors. In this section you should add any relevant information that is not clearly represented in the score sheet. Personally, I always try to add specific comments to the editors. For example, I may have complained in my report about the poor description of the statistical methods. Then I will let know the editor that that is, in my opinion, a mandatory change for the suitability of the manuscript for publication. Finally, when writing my comments to the editor, I try to be as brief as possible (but never just 'yes' or 'no'), so that I do not overwhelm the editor's likely busy inbox with unnecessary comments.

In summary, the comments to the editor should include any information that is not properly represented in the score sheet, cannot go into the response to the authors, and may be relevant for the editor in order to make an appropriate decision on the manuscript.

Reviewing is a "Golden Rule" situation: "Treat others as you would like to be treated." The authors have (probably) invested a lot of time and effort in the paper, and being accepted or dismissed with a terse couple of sentences is neither helpful nor particularly respectful. Even if the paper is terrible, the authors deserve a thorough explanation of why you think that it is terrible, so that they can understand the fairness of the decision---remember, many papers that you encounter will come from young authors or authors from institutions without a very strong scientific research culture, and your feedback is critical to helping them improve.

Furthermore, as a referee, your job is not to make the final judgement (even if you have a form that says "reject" or "accept"), but to provide the editor or program chair evidence in support of their decision. I thus find the following template useful for organizing my reviews, to make sure that I convey the right information:

1. Start with a few sentences summarizing what you think the main idea and impact of the paper is.
2. Give a general feeling about whether you think the paper is ready for publication or not.
3. If not, explain what are the biggest general problems that have to be dealt with.
4. Major issues list: all significant scientific issues that have to be addressed before publication
5. Minor issues list: all the scientific side-issues, nitpicks, typos, nice-to-haves, etc.

The distinction between 3, 4, and 5 is one of triage and communication, and should be made very clear in a review. #3 is the main reason for your judgement. #4 is for all of the rest of the problems that impact the scientific content of the paper. #5 is for all the little stuff: I always appreciate it when a reviewer reads carefully enough to provide nitpicks and notice typos, but it's also important that it be clear (to the editor, as well as the authors) that such minor problems are, in fact, not the basis of judgement.

A place that this distinction is particularly critical is regarding requests for additional experiments. Many reviewers seem to feel that it is their job to assign "homework" for the authors, and this is seriously problematic. You should only request new experiments if the work as presented will not stand scientifically without it. If you have suggestions for experiments that you think would be cool or would strengthen the point, put them in "minor issues" and clearly label them as suggestions not required for publication. It doesn't matter if you would have done it differently or if you think the new experiment wouldn't be too much work: your job is to evaluate whether the paper is scientifically sound, not to pretend that you are the authors' Ph.D. advisor.

Another important point, with regards to structure and grammar: remember that many papers you read will not be written in the authors' native language. Difficult as it may be, it is critical to try to separate language and presentation problems from scientific problems. I always remind myself that I would be writing much worse prose in the authors' native language. Thus, my section #5 may often contain a comment like, "The paper has many problems with English grammar and needs attention from a proof-reader," but unless the problems are so bad that they impair my ability to understand the scientific meat, it stays as a "minor issue."

In addition to the textual feedback, there are often numerical or qualitative ratings to select. I view the textual feedback as the meat, and always prepare it first, especially since writing up the textual feedback often changes my opinions of what the numbers should be. Moreover, numbers are not very informative, because different people interpret them differently: the text is the only opportunity you have to clearly communicate your reasons for liking or disliking a paper.

Finally, there is one case where all of this goes out the window: if I find that I am dealing with a plagiarized or multiply submitted paper, then the authors are wasting everybody's time and I have no problem giving a short and harsh rejection, e.g., "I didn't like this paper the last time we rejected it, and the authors haven't changed anything since then."

A review is supposed to provide the view of a peer on the manuscript submitted for publication. An editor needs to take one or several, usually two, reviews to form the basis on deciding on the fate of the submitted manuscript and to form the basis for revisions by the authors. There are very few instances, I have yet to see one, where a paper is publishable as is. There is, however, no clear correlation between usefulness and length of a review.

The key is to provide feedback for improvements to ensure the manuscript becomes as good a contribution as possible. Some reviewers spend much time correcting language or other formal aspects of manuscripts. These sorts of comments are always welcome but the core issues are focussing on the science itself and the clarity with which it is disseminated.

So a shorter form of report contains the core scientific and clarity issues encountered and perhaps general comments on other shortcomings, a longer report details also the detailed issues. The size of the reports obviously depend on the quality of the manuscript but also on your expertise. Some times editors appoint reviewers to cover a narrower but vital aspect of the manuscript. The good review should be critical, fair and objective and provide both the editor and authors with suggestions for improvement. By necessity this will be more than just a few paragraphs of comments