Quasi Experimental Design: Check the Comparator

Quasi experimental design studies a treatment without the usual random assignment to groups. To read one well, check who got the treatment, how they got it, and which group or time period they were compared with. The design name or the word “matched” cannot replace those source checks.

This guide builds a note from real papers: how treatment was assigned, what it was compared with, and what remains unclear. You will see why planned matching is not a result and why “random” needs a stated action and unit.

The note helps you read the paper; it is not a new analysis of its data.

What quasi experimental design means

A quasi-experiment seeks to learn about a treatment when it was not assigned through the usual random process. CASP's design guide explains this point and ways to compare groups. Start with what happened, then ask why the groups should give you a fair basis for the claim.

The comparator is the group, time period, or pattern used to judge what would happen without treatment. It might be a nearby school, people just outside a cutoff, or the trend before a new rule began. Ask why it should stand in for the treated group. Being untreated is not enough to make it a good match.

The Handley review abstract covers design types such as nonequivalent groups and time series. These names help you choose what to check. But two studies that look alike may assign treatment in distinct ways, so read the rule used in each one before choosing a label.

Quasi-experiments can support claims about cause when the design and assumptions are sound. A lack of random assignment does not make that task impossible. Nor does the label make the claim true. State the treatment, the comparison, what must be true, and what else might explain the result so a reader can check your reasoning.

Trace who received the treatment

Begin with the Methods section and find the actual action that created treatment groups. Who made the choice, and what rule governed it?

A course timetable, a policy cutoff, or a person's choice to enroll can create different comparisons even when each paper uses the same broad design label.

1. Name the treatment. State what changed or what people received. Separate an invitation, enrollment, attendance, and completion if the paper does so.
2. Name the assignment rule. Find how units entered each arm, including any choice, cutoff, rollout, or random process.
3. Name the unit. Was the action applied to a person, class, school, clinic, or region? Keep sampling and assignment at each level apart.
4. Name the comparator. Record who or what stands in for the alternative, and how that comparator was selected.
5. Name the timing. Find when exposure and outcomes occurred, along with any baseline or trend data.

GOV.UK's comparator guidance highlights selection differences and the challenge of choosing a comparison group. If a program begins in one region first, you need to ask what else differs between regions. Shared geography, age, or a broad group label alone cannot settle the question.

The word “random” needs an object. Randomly drawing people from a list concerns who enters the sample; randomly assigning treatment concerns what they receive. A study can use one without the other. Record the action and level beside the word so the next reader can see which process it describes.

Keep these fields before attempting a broad synthesis of research papers. Merging findings too early can hide differences in exposure, comparator, and time. A source-by-source note makes those differences easier to review.

Build a published-study evidence note

The table gives my reading notes on two real sources. One is a protocol: a plan for a study. The other is a report of a study. Each row states what the methods say and what still needs checking. It does not copy results, estimate effects, or prove a health benefit.

Table: Source-located assignment and comparator notes. The open questions are reading checks, not study ratings, reanalysis results, or proof of an effect.

The protocol rows need words that show a plan. They do not show matching that has been done or the final result. Keep that status in its own field: a clean table should not turn “will do” into “did.” You can check a later report when it becomes available.

The digital media rows need a check of the design name against the actions in Methods. Choosing centers for arms and drawing women from lists are distinct steps. Keep both passages and ask how centers entered each arm. The sampling sentence alone cannot tell you that women were assigned treatment at random.

In your next paper, the assignment rule may be clear but the comparison weak. Or the paper may name the groups without saying how they were formed. These are distinct gaps. State the gap you found and link to the source. Do not fill a blank field just because you know the design name.

Correct the claim about matching

Suppose a draft note says: “Matching made the groups equivalent, so the program's effect is established.” The protocol cannot support that claim. It plans matching but does not show the balance or effect found after doing it. First fix what the source says was done; then ask what a result could mean.

Use three criteria: was matching planned or done, which measures did it use, and how well do the groups match on those measures? The method name cannot answer these questions. Find the report that shows what was done and what was found. Read its limits before deciding that the groups support the claim.

Replace the note with: “The protocol plans to match people who chose treatment with a control group. We need to check the balance and outcomes in a later report before judging the effect.” This leaves the study's result open.

It states the check this source cannot yet supply, rather than claiming the study failed.

The WWC matching guidance makes a similar point: a method name does not replace showing baseline equivalence on the required measures. Those rules serve a defined review task. Check the current handbook and protocol before giving a study a rating or applying a cutoff from an older guide.

Matching can account for measured traits under assumptions. It does not prove that the team measured each factor that matters. Ask what might affect both the choice to take treatment and the outcome but be missing from the data. State that threat rather than simply saying the groups were alike.

Check what the comparator can support

A comparator should answer the alternative relevant to the claim. If a program is compared with services as usual, ask what those services contain. If it is compared with nonparticipants, ask why they did not participate. A claim about enrollment also differs from a claim about attending every session.

Read the baseline as well as the final outcome. Which differences existed before treatment, and how were they handled? The WWC FAQ also notes that changing group composition can affect interpretation.

Do not treat last year's class as the same people merely because the school name is unchanged.

Time creates further questions. Did another policy, staffing change, or seasonal shift occur alongside the intervention? GOV.UK's guidance on before-and-after comparisons describes background changes and repeated-measure concerns. A change after treatment is not, by timing alone, proof that treatment caused it.

For a time-series design, check the pattern before the change and how the study models the interruption. For a cutoff design, check the rule and the units near it.

These designs need different assumptions. Do not apply a simple two-group checklist as though it resolves every form of quasi-experiment.

Missing outcomes and crossover can alter the group you end up comparing. Record the exclusions and the treatment contrast the analysis actually uses. When that is unclear, write “not found in this text” and identify the report or supplement needed next.

The empirical research guide helps with the wider task of tracing data to claims. Keep that evidence trace alongside this assignment note. Knowing that a study used observations does not tell you whether its comparator supports the causal claim under review.

Inspect assignment passages in Atlas

Add the selected papers and any permitted supplements to a project. Wait for processing to finish, then start a chat and type @ to choose each source. Use distinct titles and versions so a protocol cannot silently stand in for a completed report.

Choose Project only beside + to stop new outside retrieval if that matches your review scope. Earlier conversation context remains available, including outside sources already in the chat. Start a fresh chat when that earlier context could blur the evidence you want to inspect.

Ask for a comparison with separate mechanism and status fields:

For each selected source, identify the treatment, assignment rule, assignment unit, comparator, timing, and remaining limitation. Cite each supporting passage. Distinguish sampling from assignment and planned methods from completed results. Mark missing details as not found; do not certify causality.

Open each citation and inspect the surrounding text, checking that it describes this study's action rather than a method cited as background.

The citation-tracking guide explains why keeping that source link matters when reviewing a claim.

Existing Atlas screenshot: the ColPali PDF and citation preview show source inspection. They do not depict these studies or a test of causal-assessment accuracy.

Correct any row that turns planned matching into completed equivalence. Ask a focused follow-up about the source status or assignment unit, then check the new passage. Keep your reason for the correction so another reader can assess it rather than seeing only a cleaner label.

For dense layouts or scans, the AI PDF reader guide can help you assess source-reading needs. Confirm the relevant page yourself.

Atlas assists comparison and notes; it does not run the statistics, decide the appraisal rating, or establish causality for you.

Save the claim with its limits

Before saving, follow one claim back through its assignment and comparator fields. Check the level at which treatment was determined and whether the cited evidence is a plan or a result. If either field is unresolved, narrow the claim and keep the question open.

Create a note through New then Note, and wait for Saved before closing. Retain source versions, methods locations, the corrected claim, and remaining threats. A short explanation of why matching is still a plan is more useful than a bare design label.

Share the note with its limits intact. When a later report arrives, check the completed methods and results before updating the row. A sound reading decision preserves what the source supports today while making the next check clear.