Umbrella Review: Compare Reviews Without Double Counting

An umbrella review asks one question by comparing past reviews. These often include systematic reviews and meta-analyses. The reviews are the main sources of evidence. A standard systematic review often studies the original papers instead.

That choice creates a trap: two reviews may agree because they use the same studies. Before you call this independent support, check each review's scope, methods, and study list.

Suppose Review A cites studies P1 through P8, while Review B cites P4 through P10. Both report a similar result. Five studies appear in both lists. The team must decide what that shared set means for its question. The two review conclusions are not fully separate lines of support.

What an umbrella review covers

The JBI Manual for Evidence Synthesis treats an umbrella review as a planned summary of past reviews. It separates the plan from the final account of findings. You select reviews, then read their study lists to check for shared sources.

Begin with a question that can be answered across several reviews. State who or what is studied, the setting, any intervention, and the results or themes of interest. Then say which types of review qualify. A narrative opinion piece and a systematic review may both discuss a topic, but they do not meet the same rule.

A review of reviews does not make the original studies disappear. Their dates, designs, and duplication affect how much weight you can put on the review-level account. If you are grouping included primary studies instead, see our narrative synthesis guide. Keep a link from each included review to its source document and extract the study identifiers it reports.

For the boundary between this method and reviewing primary research, see our systematic-review tools guide. The tools page helps with software choices; the method decision comes first.

When the method fits

An umbrella review can help when several systematic reviews ask related questions. Check where the groups, dates, or measures differ. Those gaps may help explain why the reviews reach different conclusions.

It is a poor fit when no reviews meet your rules. It may also be a poor fit when the reviews are old and new studies change the picture. A new, narrow systematic review may serve you better.

Use clear criteria for the reviews you include. Keep those rules in a written plan. This helps the team explain each choice.

Check whether the candidate reviews answer the same question. “Workplace wellbeing” may cover different groups, measures, and study designs. A broad shared label does not make those findings directly comparable. Note the mismatch before creating a summary table.

Set review eligibility and scope

Write a protocol, or study plan, before you select reviews. Record your question and the kinds of review you will include. Name the search sources, dates, and rules for choosing reviews. Also plan which data to take, how to judge review methods, and how to handle studies that appear in more than one review. A scoping review protocol needs a different plan: it maps a field instead of bringing together past reviews.

The JBI protocol section can help the team set those rules. A search of the agreed sources gives you a record of what you found. A review found by chance may be relevant, but it does not replace that search. For each full text, note why it was included or left out. Another team member should be able to follow your reasoning, even for close calls.

Before extraction, decide which review-level facts matter. Typical fields include the review question, search end date, study designs, population, study IDs, main findings, and limits.

JBI's data collection guidance shows why these fields are planned before the comparison.

Do not confuse the publication date with the search end date. A review published last year may have searched for studies several years earlier. Mark that gap in the matrix. It tells readers what evidence the review could not have covered.

Appraise included reviews

A review can be relevant yet weak in its methods. The team should choose an appraisal tool and rules suited to the eligible review types, then apply them to the review methods.

JBI's appraisal guidance calls for a check of each review's methods. Record which criteria are met, unmet, or unclear.

Record the judgment for each criterion and the evidence in the review text. If two reviewers disagree, preserve the reason and the resolution rule. A single “high quality” label without a trace to methods will be hard to audit.

Do not turn a missing methods detail into an invented score. “Not reported” may be the right entry until the team checks an appendix or contacts an author. This check does not prove a review's claim is true. It shows how its methods help or limit trust in that claim.

The JBI critical-appraisal checklist is one concrete tool. The team still owns the tool choice and threshold in its protocol.

Atlas can help locate text in supplied reviews, but it does not assign a valid appraisal score.

Check primary-study overlap

Create a list of study IDs for each included review. Use a stable citation, registry ID, or other clear identifier. Check the source before you merge names that look alike. Two similar author-year entries may be distinct studies. One trial may also have several reports.

For example, suppose Review A includes P1–P8 and Review B includes P4–P10. They share P4–P8. If both report the same direction of effect, part of that agreement rests on the same underlying evidence.

A step-by-step method guide covers shared-study checks and how to read the results.

Keep uncertain matches visible. Write “possible duplicate: check trial ID” instead of quietly merging them. For a review of numeric results, the team may use an overlap measure or a rule set in advance. This example does not give a new estimate or tell you which review to prefer.

The overlap check also helps explain why reviews disagree. One may add newer trials, use other measures, or study a narrower group. Compare those choices before you call the results a conflict.

Build a review-level matrix

The table is hypothetical. The review labels, study IDs, and findings are invented to show what to record. They are not published study results or an Atlas performance test.

Table 1: In real work, tie each entry to a page, table, or appendix in its review. “Not reported” is a valid cell when the team cannot locate a fact. A citation to a review paragraph may support its stated finding but not the accuracy of its own primary studies.

In Atlas, add the selected review texts to a project, mention those sources, and ask for a review-by-review comparison of scope, search date, included-study IDs, findings, and missing fields. Then open each citation. Confirm the statement in the review and inspect tables or appendices for study IDs.

Save a checked note with the confirmed rows, possible duplicates, and unresolved questions. Atlas can compare supplied documents and point back to cited text.

It does not search every database or decide which reviews to use, how sound they are, or where studies repeat. Our research-paper synthesis guide covers source comparison at a broader level.

The screenshot shows Atlas beside an open research paper. It illustrates where to check a cited source. It does not depict an umbrella review or a verified overlap result.

Next steps for interpreting an umbrella review

Before you trust matching results, check each review’s methods, question, shared studies, and search date. Say which reviews back the claim and which do not. If the evidence is too mixed to combine, say why instead of forcing one headline.

The JBI results section shows how to report findings from the reviews you chose. In your report, keep a clear trail from each claim to its review, and from overlap decisions to the study lists that support them. Give readers the appraisal results and the uncertainty you could not resolve.

A cited source check can help sort this work. The review team still chooses the method and decides what the findings mean.

For software that helps manage citations and review documents, see our literature-review software guide and broader AI research assistants guide.