How much trust are you supposed to give a peer-reviewed article?

  • Should one trust that citations are claiming what is in the articles they cite?

    No. Too many people don't read their sources carefully enough (or even at all).

  • If there isn't anyone who has disagreed yet, should I assume it's true for the purpose of discussions, without further examination?

    There is a wide range of possible levels of trust between something I read in a blog post and something on the level that "the square root of 2 is irrational". I'd put a random peer-reviewed article somewhere between the two extremes, depending on the discipline, the journal, the state of the art and so forth.

    "Without further examination" should not be part of a scientist's vocabulary. Except maybe for the square root of 2.

  • Should public policy be based on it, without further examination?

    No. In particularly not if the article reports on experimental findings in psychology, economics, sociology, medicine and so forth. Most published research findings are false. These disciplines always need replications and meta analyses, because they cannot perform experiments as tightly controlled as, say, in physics.

  • Should I trust data to have been collected correctly?

    I do my best, but I wouldn't trust myself 100% to have collected my data correctly for published work. Stuff always happens.

    In addition, peer review doesn't really enter into this question. Peer reviewers cannot easily assess your data collection - only your description of it. You could have made horrendous errors in good faith, and the reviewer wouldn't know.

  • Should I trust data to have been processed correctly?

    See above. Would you trust software to be bug-free? You shouldn't. And again, reviewers don't review your data analysis as such - usually, your analysis scripts are not part of the bundle you submit.


Bottom line: peer review will increase my level of trust in an article, but not infinitely.

In addition, like a good Bayesian, I trust more surprising (which have a better chance of appearing in the more prestigious journals) findings less. Therefore, I usually expect articles in Nature and Science to be less easily replicable than less "sexy" findings published in other venues.


Should one trust that citations are claiming what is in the articles they cite?

Simkin and Roychowdhury (2003) estimated that authors read only 20% of the works they cite. You cannot even assume that the cited articles exist:

The Most Influential Paper Gerard Salton Never Wrote https://www.ideals.illinois.edu/handle/2142/1697

Should public policy be based on it, without further examination?

No.

Why Most Published Research Findings Are False http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Should I trust data to have been collected/processed correctly?

Hauser was a leading scientist in his field and he was making up the data:

https://en.wikipedia.org/wiki/Marc_Hauser#Scientific_misconduct

God help you if Excel was used:

http://lemire.me/blog/2013/04/24/you-probably-shouldnt-use-a-spreadsheet-for-important-work/


I see peer reviewed articles as a discussion. A discussion with some rules to discourage people from advancing untested or ridiculous arguments, rules that require one to acknowledge previous papers and present evidence about the new idea, but not much more than that. Published letters are rarely a summary of the community's shared conclusion about something. They are simply a fresh argument or an interesting interpretation of some new idea/data/evidence.

(The longer review articles are often an attempt to form such a shared conclusion, but even they may have some political slant to them.)

Should one trust that citations are claiming what is in the articles they cite?

Usually but not absolutely. I have seen a very believable citation in a front page Nature publication that turned out not to mean what the authors asserted it did. Naturally I only noticed this when I required the information for my own work.

If there isn't anyone who has disagreed yet, should I assume it's true for the purpose of discussions, without further examination?

Should public policy be based on it, without further examination?

In both cases, never.

Should I trust data to have been collected correctly?

Should I trust data to have been processed correctly?

It depends for what you require those data. Data processing and collecting are often done by the least trained and most stressed people in the laboratory, who are sometimes under pressure to cut corners to get something their boss or advisor likes. There is often a great deal of cherry picking in the hard science disciplines to show the data in an exciting way, so to justify a top tier publication.

If the data are going to be supporting your safety case for the airworthiness of a new passenger aircraft, you probably should get the data again and talk to the original authors (in person) about the experiment. If the data are merely supporting a limb of your hypothesis in a new paper, and are widely in agreement with other data published independently, then it is probably sufficient to accept them as they are.

Tags:

Peer Review