feedback-3677258_1920 2.jpg

Just because a paper passes peer review, it doesn't mean it's right.

One of the most important aspects of science is to actually communicate your results to your peers. This can be done through a variety of formal and informal ways. Emails are always popular, as per usual. And conferences are great places too to give quick status updates - although the most interesting conversations don't happen during the presentations, but in the hallways and over dinners. But science has enjoyed centuries of tried-and-true tradition for widely disseminating research results: the journal article.

Publishing a paper is considered, for good reason, a major accomplishment. It means that you've advanced the field, you've made your mark, you've upped your game. You've made enough progress and learned enough about nature that you think that your peers need to hear about it, and the publishers of the journal agree.

But it doesn't mean you're right.

The major hurdle for publishing a paper is peer review, when a (usually) anonymous member of your community (somebody who knows what they're talking about, hopefully) reads the draft of your paper and offers critiques.  Now these critiques are usually guided by the editor and the publishing standards of the journal. Most importantly, the reviewer has to determine if a significant scientific advancement was made, and whether that advancement is worthy enough for publication.

What's more, the reviewer has to determine if there are any major flaws in the paper. This is where things get interesting. It's impossible for the reviewer to completely duplicate the experimental setup or observing program. And it's usually very difficult to replicate the entire methods and analyses that were used to arrive at the conclusions. So the reviewer has to do their best job of following the logic and arguments laid out in the paper and seeing if they agree with the conclusions.

But the reviewer isn't getting paid. There's no compensation involved except the kudos of the editor (who in gratitude will simply send you more papers to review in the future) and a sense of obligation that you've done your duty and service to the larger scientific enterprise.

So in a system where the reviewer is volunteering their time, and can only do their best, what are we to make of peer review in general?

Does it mean a paper is correct? No.

Does it mean a paper is the final word? No.

Does it mean that the scientific community has accepted the results? No.

It means that one or a couple random reviewers thought the paper was interesting, new, and not obviously wrong.

That's it.

So just because a paper has "passed" peer review, it doesn't transmute into the scientific equivalent of Gospel Truth. Instead, peer review is the first step, not the last. Now that the paper is out in the wild, it's time for it to be analyzed and picked apart - the real guts of scientific progress.