Post Publication Peer Review: Check the Critique

Post publication peer review is the evaluation of research after it has been published. It can happen through journal letters, public scholarly comments or invited reports in a publish-then-review model. A critique may expose a problem, clarify a method or support a finding. Its value depends on the evidence, not on the fact that someone posted it.

For your own reading, keep the objection, paper passage and reply together. You need to know what each source supports before changing how you use the paper.

Identify the paper and review channel

Confirm the version and dates

Start with the paper's title, DOI and version. A preprint, accepted manuscript and version of record may have different text or figures. Check which version the critique addresses before comparing its claim with a newer copy.

Record the date of the critique and the date you checked it. A reply or correction may have appeared later. Save the supplied link and a usable copy when permitted so your note points to the material you actually read.

Do not treat a comment count as a measure of reliability. Several comments may repeat one concern. A paper with no public comments has not thereby passed an extra scientific check.

Keep the title and DOI in each note even when a thread links to the paper. Titles can be similar, and replies may discuss a different version. If the supplied comment quotes a line that you cannot find, record that mismatch and ask for the version it used.

Separate discussion from editorial action

PubPeer asks for facts that readers can check. It also says it does not scientifically review comments. A comment that passed moderation can still be wrong. Check its sources before you use it to judge a claim.

A report invited by a journal follows a different process from a public comment. The ACS explanation describes models in which a paper is reviewed after it first appears. Check its status label to learn which review steps it has passed.

The 2021 review by O'Sullivan, Ma and Doran groups journal-based review, letters and academic networks. The original excerpt below also states that it excludes informal mainstream social-media reviews. That scope limits what the review can tell you about all public discussion.

Read the scope statement with the listed channels. The review does not cover every place where readers might discuss a paper.

O'Sullivan, Ma and Doran, An Overview of Post-Publication Peer Review, page 4, CC BY 4.0. Original excerpt cropped and converted to WebP.

Richard Smith's BMJ commentary uses the term more broadly, including how studies enter later scholarship. Authors use the term in these different ways. Name the channel you are checking so the reader knows what kind of review your note covers.

Work through a dated critique

Trace one objection at a time

The following example is fictional. It shows how to read a paper and replies without making allegations about a real author. Suppose a public comment raises concerns about a survey that compares two groups.

The comment may contain several objections that need different checks. Split them before judging the paper as a whole. Keep each concern tied to the result or interpretation it might affect.

Use the note below to keep the comment date, paper locator and reply together. A missing source remains a gap, not a confirmed finding.

Table 1: The status follows the material available for each objection. It does not describe the character or intent of the authors.

Distinguish a reply from a resolution

In the first row, a paper passage answers the factual question. In the second, the authors supply a reason for their analysis choice. That reason still needs method expertise and, where necessary, the underlying data.

A reply can explain what the authors did without showing that the choice was sound. Likewise, a reader can identify an exclusion rule without knowing how much it changed the result.

For the image concern, record the missing raw files. Do not infer copying from an AI description of visual similarity. If a proper assessment requires image files or specialist work you have not done, retain that limit in the note.

Decide what the evidence changes

Keep the consequence claim-specific

Ask what you planned to use from the paper. A concern about one group may change the claim you need from that group. It may not address a result from a different part of the study. The note should explain that connection.

Use limitations of the study to tell a stated scope limit apart from an error. A study can have a narrow sample or method and still answer a bounded question. Check whether your intended claim extends beyond what it studied.

State the decision in terms of the claim you need: use it as reported, use it with a limit, or leave it out pending a check. Include the reason and passage. These are reading choices, not grades for the paper. A later reply may change the choice if it gives the missing support.

If a concern changes your reading, write the new wording and cite the passage that led to it. You might change “the method works for all groups” to “the paper reports this result in the studied sample.” Do not attach a wider conclusion to a narrow correction.

If a comment claims that a study did not replicate, distinguish replicability from reproducibility. A new-data study and a rerun of the original data test different questions.

Check formal notices separately

Visit the journal's article page to see whether a notice exists. PLOS policy distinguishes editorial notes, corrections, expressions of concern and retractions. Read the actual notice to see which statements or results it addresses.

Keep the notice apart from the first comment and your own judgment. Note its date and scope. A public thread may stay online afterward. Its presence does not show that the journal ignored the concern.

Use your institution's process for a concern that needs a formal inquiry. KU Leuven's guidance keeps that process distinct from public discussion. A reading note cannot settle an inquiry.

Check new material when it arrives and retain the earlier state of the note. Record which question a reply answered, which claim you changed and what remains open. This helps you trace later changes without treating your first judgment as final.

Save the checked note in Atlas

Compare the supplied sources

Create a project and add the paper, dated public critiques and author replies that you may use. Wait for source processing, open a chat and use @ mentions to name those sources. Keep unrelated paper threads outside this comparison.

Ask: “Compare these critiques with this paper version and these replies. For each concern, give the date, paper passage, reply passage and what remains unresolved. Cite the sources. Do not infer misconduct or treat a missing reply as proof.”

Verify and preserve the result

Open each numbered citation and read the passage in context. An AI source checker can help organize this review, but you must decide whether the cited text supports the wording. This is a proposed workflow, not a measured test of Atlas on a real dispute.

If Atlas labels the fourth fictional concern “confirmed duplication,” remove that judgment. The supplied material does not settle it. Change the status to “raw files unavailable” and retain the question for specialist review.

Select New, then Note, and save the checked comparison with dates and source locators. Confirm Saved. For a later review across papers, use the research synthesis workflow and carry forward only the claims your note supports.