Reporting bias and the outcomes a paper leaves out

Reporting bias occurs when the results people can read depend on what those results show. Findings may be left out, delayed or shown by choice because of their direction, size or test results.

To check a supplied study, compare its planned outcomes with its full record. Record exact matches, changes and open questions before deciding what they mean. A mismatch deserves a question, not a charge.

Atlas can help compare a protocol, report and extra files with citations. Keep a gap note that separates what each file says from your reading of why the difference occurred.

What reporting bias changes

Reporting bias changes the sources a reader can see. If results reach readers by choice, a synthesis can reflect the reporting process as well as the subject being studied.

Cochrane distinguishes two related problems needing varied checks. An entire study or outcome result may be missing. Alternatively, authors may report a chosen measure or analysis from several possible options. Neither is resolved by reading an abstract alone.

Cochrane's current missing-evidence guidance separates those problems. Its scope matters because assessing missing evidence across a review needs more than checking one paper.

Publication bias concerns which studies reach readers. Selective outcome reporting concerns choices within studies. Both differ from selection bias, which concerns entry into a study or analysis. Check the sampling strategy separately when a claim depends on who the study reached.

Match outcomes before naming a discrepancy

Use a source-evidence matching method that keeps the support for each source check. Record the protocol identifier, registration record, change dates, report DOI and supplement filenames. Keep earlier and later versions to check when a change appeared.

Match more than an outcome label. “Quality of life” could refer to varied instruments, change scores or follow-up times. Compare the domain, measure, metric and time point before deciding that two entries state the same outcome.

Also check whether the result concerns the same group of people and analysis. A report may state both a planned analysis and an exploratory one. Keep their status separate instead of treating every extra analysis as a protocol violation.

Search the full supplied report set. An outcome missing from the main paper may appear in an appendix or separate linked report. Record where you looked and what remains missing, so the next reviewer can repeat the check.

The outcome definition and multi-file approach follow Cochrane chapter 13. Use them to make the source check precise, while leaving the assessment for the right method.

A fictional protocol and report

In a made-up training-program study, protocol version 1 names confidence at six weeks as its primary outcome and course completion at twelve weeks as a secondary outcome. The main report highlights course completion.

A first note says, “The primary outcome was left out and replaced by a favorable secondary outcome.” That sentence makes two unchecked claims: the result is absent from the full record, and its omission depended on the findings.

On checking the made-up extra file, the reviewer finds the six-week confidence result in appendix B. The primary outcome was absent from the main results table, but it was found in the supplied extra file.

The revised note says, “The six-week confidence result appears in appendix B. Course completion receives greater focus in the main report. No reason for that focus was found in the supplied files.”

This revision does not decide whether focus created a bias concern. It shows a sounder finding. A formal assessment would consider the planned analysis, real results, order of events and reason under its own rules.

Keep the source note short enough to read:

• Planned: confidence score, stated instrument and metric, six weeks, primary status, protocol version and location.
• Reported: matching confidence result in appendix B, with the report and page location.
• Difference: course completion given more space in the main report despite its secondary status in the supplied protocol.
• Unknown: whether a further dated change or reason exists outside the supplied record.

All files and outcomes in this example are invented. The purpose is to show how a missing-source check changes the wording you can defend, without accusing any real authors or guessing unseen results.

Build the discrepancy note in Atlas

Create a project containing the allowed protocol, report, extra files and changes on record. Include dates and version numbers in source names. Add your own note stating which outcome or claim you are checking.

Mention the sources with @ and choose Project only for new retrieval. Earlier outside evidence remains in the chat, so check that citations point to the files you named. Ask Atlas to compare planned and reported outcomes, citing both sides of each match and marking any fields that the supplied files do not show.

A helpful prompt is: “For each planned outcome, compare its measure, metric, time point and primary or secondary status with these reports. Cite each finding. Separate results absent from the main paper from results absent from all supplied files.”

Open each citation and read the source passage. Check that the cited report belongs to the same study, that its time point matches, and that a summary has not mistaken an appendix reference for a missing result.

Read each line of the note from both ends. Can you find the planned outcome in the first file and the reported result in the second? If one line says six weeks and the other says twelve, do not mark them as a match just because both concern confidence. Keep the two times in view and state what has yet to be found.

For the made-up example, the note must now point to appendix B. A reader should be able to open that file and see the six-week result. The old claim that it was missing must go. The remaining question is why the main report gave more space to course completion. Keep that question open unless the files contain a dated reason for the choice.

The PDF and answer can be read side by side. This existing capture illustrates the source view, rather than a protocol/report test. The visible AI research paper does not support the made-up outcomes in this guide. Visible paper: AI Scientist-v2, Yutaro Yamada et al., CC BY 4.0. Capture copied unchanged.

Correct the note whenever a citation fails to support the source check. If the location is missing, find the file by name and record that limit. Do not mark a result missing solely because a citation fails to open.

Select New, then Note, and save the checked gap with source locations and open questions. Wait for Saved before closing. Researchers decide how the finding enters a formal assessment.

Separate observation from risk judgment

For Cochrane-style assessment, an observed gap is a fact about the files you read. A reason suggests why it occurred. A risk judgment applies a defined assessment method to the result or synthesis being checked. Keep all three visible.

A documented change made before results were known can have a different meaning from an unexplained change after analysis. If order of events is unclear, keep that doubt rather than supplying a story about what drove the authors.

Cochrane chapter 7 stresses clear risk judgments about given results. It also separates reporting completeness from the broader assessment of bias.

A CONSORT checklist helps locate what a randomized trial report states. Satisfying a reporting item does not prove that all results were chosen without bias, just as a missing item alone does not show intent.

Carry uncertainty into your interpretation

Write the limit at the level the files support. “No twelve-week result was located in the supplied paper and extra file” is clear enough to repeat. “The authors concealed an unfavorable result” needs support the file check has not shown.

When needed, name an exact follow-up source: the earlier registration history, an analysis plan, a linked report or a clearer reason from an author. State which doubt it could resolve, rather than asking vaguely for more details.

If you are preparing a commentary article, connect the critique to the exact target claim and keep any documented reason. The gap note supports a fair response even when the remaining doubt cannot be resolved.

Keep the compared versions alongside your note. A newly located supplement or amendment can change the finding, so revise the assessment input and the wording of your reading together.