How much should reviewers care about other things than an article's content?

I think the gut feeling is helpful during the evaluation process - one can probably not fight it anyway, so I think one should just accept that it is there. However, the gut feeling must not hinder a review based on the content of the paper.

So, what should not happen is, e.g.:

• Stop reading after a few pages because you don't like the look, typesetting, language, variable names… (you may, however, stop reading if the style hinders your understanding, which is something completely different). Even if you hate Times New Roman or Arial, be professional and keep reading.

• Keep worrying about issues as above when it comes to write the comments (with the exception of the language, but I am always very careful about comments on language). Comments should mostly address the content of the paper, especially your recommendation should not be influenced by gut feeling. So once you read everything carefully, form an opinion not based on the look of the paper, but on the content (which you have digested at that point and hence, you should be able to evaluate it without taking the look into account).

One exception is figures: If figures are poorly formatted and don't allow to extract the information, this is a serious issue (too small font, pixel graphics with poor resolution, colors that are indistinguishable for common color blindness…). But also here it is more about "can I extract the information" and not about "do I like the look".

Be extra careful with gut feeling if you review something which is borderline for you field. As can be seen in the above comments, the styles of submission vary greatly between fields (extreme example: Some field have all floats, i.e. tables and figures and such, on extra pages at the end of the documents, other field have submission that basically look like a final article, some fields use double spacing, some don't…).

We're supposed to review papers, not authors. But, even in fields with blind review, things like formatting can reveal (or just suggest) a lot of information about the author, which we should try to ignore.

If it's a feature of the formatting which is actually going to disappear in publication (like font choice or number of columns) it's genuinely irrelevant: it tells you nothing about what the published article would be like, so any reaction is personal judgement about the author rather than the work.

Formatting decisions that will remain, like some features of figures and tables, are harder. Those, I think, should be treated like any other expository feature: if you don't like it, you make yourself focus on the question of whether it actually makes the paper less clear or harder to read, or is merely unpleasant. In the former case, you hold it against the paper, in the latter you don't.

One thing to keep in mind is that we don't want to reject people from nearby fields with different writing conventions who have something good to say. Many of the signals your seeing are mostly indications that the person is an outsider to the area, which really shouldn't be held against them. (Note that an outsider is already at a disadvantage, because conventions that matter to the refereeing, like how to organize a paper; how to handle that is, I think, a more difficult question.)

Reviewers should care about matter of text, not aesthetics. Most things you describe belong to the editorial board and editorial staff competences.

That said, I believe that, most of the times, there are some guidelines that a submission must fulfill in order to be passed for review. I find it a basic decency towards reviewers to hand out a properly formatted document. And usually it's not the author who does so.

I would say it's not a matter of pedantry but rather hygiene. It's simply easier and more pleasant to read a document with basic consistent type setting. You can focus better.

"Authors who used LaTeX to write a 5-pages article with 6-7 equations and 10 citations felt like pedantic undergrads with too much time on their hands. Or at least it didn't look like they had a lot of experience publishing in this field."

I don't know, but my personal experience is that it's easier to write in latex anything more than 3 pages, or with at least 1 equation, especially if you have some experience. You do not care about anything else than the matter. You get free bibliography and cross-references. Very easy. And it looks half decent, you get basic type setting for free too. I guess it's just a matter of workflow.

Your sentiment can be turned around very easily. Would you be happy if students handed in assignments written on scrap paper? Probably not. Would you say, with out reading, the content is flawed. Also probably not.