Natural Experiment With a Worked Study Appraisal

A natural experiment uses a change outside a researcher's control that puts people or places under different levels of exposure.

A new law, service rule, or change in water supply can create this chance to learn. The word natural refers to where the change comes from; it need not mean a storm or an event in nature.

The harder question is whether the groups can be fairly compared. A change made outside the study can still leave groups with different risks, records, or past events. Calling the study an experiment does not make those gaps go away.

Start with the event, who received the change, and the result the study tracks. The worked table below keeps those parts distinct in Snow's water-supply study.

Atlas can help compare chosen sources; you check the cited text and keep the open questions in view.

What makes an event a natural experiment

This guide follows the broad definition in the 2025 MRC-NIHR framework: an event or process outside researcher control creates differing exposure. A study uses that change to assess what happens. This meaning puts the event first, rather than tying the label to one method of data analysis.

An outside change can leave unequal groups

For example, imagine a town changes a bus fare while a nearby town keeps its fare. This is a made-up teaching case, not a reported study. The change may let you compare travel patterns, but the towns could differ in jobs, routes, or past trends. The fare change alone would not show its effect.

Plan the study around the event

Keep the event distinct from the study. A town council might choose where a service changes; the research team might choose which records to collect and how to analyze them. The team can plan its study in advance even though it does not control the policy. Outside control does not mean the research must begin after the event.

This chance to learn can be useful when a research team cannot safely or fairly assign people to a change. It still needs a clear question, sound data, and a reason to compare the groups. A strong story about the event cannot make up for weak records of who received what.

Distinguish the event from the design label

Authors do not all use natural experiment in the same way. Some use it only when the way people receive exposure could work like random assignment. Others include a wider range of outside events.

Titiunik's scholarly chapter reviews this debate and puts forward her own meaning; it is not a rule shared by every field.

Read the rule behind each label

The original de Vocht and colleagues figure shows four views. Panel 1a separates natural events from planned changes; 1b asks whether exposure acts like random assignment.

Panel 1c contrasts outside exposure with self-selection and distinguishes weaker and stronger quasi experiments. Panel 1d groups natural and quasi experiments together. These are competing definitions, not measured effects or a ranking of your study.

Original Figure 1, de Vocht and colleagues (2021), CC BY 4.0. Full figure unchanged; the surrounding text explains all four panels.

The figure helps when two sources seem to disagree about a label. Read what each author means by an outside change, a planned action, or random assignment before treating the terms as the same. The newer broad framework used here does not require every event to fit the narrow view in panel 1b.

For the broader family of nonrandom intervention studies, see quasi-experimental design. Use that guide to describe the study's setup. Neither label tells you whether the groups can be fairly compared, so keep the rule for who received what even when an author gives a familiar design name.

Inspect the assignment evidence and comparison

Assignment means the process that decides who receives the exposure. Look for the rule, who made it, the date it took effect, and the choices people had. A city border may split groups, but people on either side may also differ in ways that matter for the result.

Keep an offer distinct from receipt. In the made-up bus case, living in the town with lower fares would not mean every resident rides the bus. A study of access to the lower fare answers a different question from a study of actual travel. Do not switch from one to the other halfway through your note.

Identify the missing outcome

Next find the counterfactual: what would have happened to the exposed group without the change. You cannot observe both paths for the same group at the same time. The study uses another group, a past trend, or another well-founded approach to estimate the missing path. Your task is to find why that substitute might be fair.

The current framework's method guidance, in Concepts and definitions and Quantitative methods, explains how the question, exposure rule, and data shape the choice of method. Read the study's reasons before writing that a control group solves bias. A group with a different history can bring its own problems into the study.

Use a short reading record: who received what, why the groups were compared, the result, dates, and remaining threat. Give a source location for each part.

If a passage only calls the groups similar, keep that as the author's claim and look for the facts behind it. Shared measured traits cannot show that every unmeasured cause was also shared.

A worked water-supply exposure comparison

Snow's 1855 account describes the Lambeth company's move to an upstream water intake in 1852. The Southwark and Vauxhall supply continued to draw from a different stretch of the Thames. Snow later studied cholera deaths among households served by the two companies.

This is the company-supply inquiry in Part 3 of his book. Keep it distinct from the Broad Street pump study. Removing a pump handle and comparing households served by two companies are different changes. Each raises its own questions about who received what and how to assess the result.

The table puts source facts in plain language and adds new reading questions. Its right-hand column helps you review the source. It is not a list of other risks Snow measured or findings from a new analysis. No death rate or effect estimate has been worked out here.

Table 1: Five parts of Snow's water-supply inquiry to retain in a study appraisal.

The event and mixed supply appear in the original Part 3 transcription. The UCLA historical account reproduces Snow's passage about supplier checks. Those checks help show which water a house received. They do not, by themselves, show that all risks were balanced across households.

Snow describes earlier supplier choice by owners or occupiers. Mixed pipes alone do not document random assignment. Retain his comparison argument beside that exposure history, rather than adding a randomization procedure absent from the account.

A bounded note would say that an intake move changed water exposure and that Snow used the existing supply pattern to compare deaths. It would then state what the account shows about the groups and water records. Keep the author's argument distinct from your own review of it.

Keep causal assumptions beside the claim

A causal claim says what would change because of an exposure, compared with a stated alternative. It needs more than a gap in results.

Write the contrast the author intends before reviewing the grounds for it. This keeps a narrow claim from turning into a claim about every group or every form of the change.

Confounding occurs when other causes of the outcome are also related to exposure. In the made-up bus case, changes in local jobs could affect both fare policy and travel.

This is a possible cause to check, not a finding that such a change happened. Keep possible threats distinct from known facts throughout the record.

The framework's quantitative methods section stresses that different methods rely on different assumptions. When a study compares changes between groups, it needs grounds for the relevant trends to have behaved in the same way without the event.

Near an eligibility cutoff, it needs grounds for nearby units to offer a fair contrast. Naming the method does not show that these conditions hold.

Read the author's checks beside the claim. Past trends, balance on measured traits, or results under other choices can help support the account. As the original conceptual paper's limitations explain, they cannot show every hidden difference or prove what would have happened without the change.

Record what each check tests and what it leaves open, so that several checks do not become a blanket claim of certainty.

Describe the intended trial

The 2021 conceptual paper discusses a target-trial approach. Set out who would enter, which strategies would be compared, how assignment would work, follow-up, outcomes, the causal contrast, and analysis.

This describes the trial you would want to run if it were feasible. It helps clarify the actual study, without turning the event you observe into that trial.

You can also compare evidence with different sources of bias. Data triangulation explains that separate job. Agreement across records or methods can help, but it does not repair a shared missing assumption.

Ask whether each source offers a distinct check or simply repeats the same claim. Two accounts built from the same flawed record may agree for the same reason.

Align the event, outcome and study population

The policy date, exposure date, and outcome date may differ. A law can be announced before it takes effect; a service can take time to reach users.

In Snow's case, the intake move came before the later study of deaths. Keep that sequence visible rather than putting all the events under one study year.

Ask whether people changed what they did before the formal start. If they acted on news of the change, the start date may not mark a clean break.

Also check follow-up: a result measured too soon may miss the change, while a later result may include more events. The right time window depends on the question and the setting, rather than a fixed waiting period.

Check who appears in the records before and after. If new people enter the sample, the result may partly reflect who was studied. If the same people are followed, missing records or moves between groups may still matter.

Keep the author's account of these issues in view. If the paper does not say how a gap was handled, mark that detail as missing.

Preserve the unit and the whole change

Keep the unit of analysis beside the outcome. A household, person, district, or school can be the unit in the study. Company counts of supplied houses are not counts of residents.

Do not rewrite a result for a district as a person's risk or forecast. Ask which unit was assigned, which unit was measured, and which unit the claim describes.

Finally, review other changes around the event. A made-up example would be a lower fare introduced with a new route network.

If the study compares the whole package, describe it that way. Do not give all credit to the fare unless the design offers a sound way to split its effect from the route change.

Report what the comparison leaves unresolved

The first limit is source access. An abstract may name a natural experiment without explaining who received what or how the groups were compared.

Record the missing detail and seek the paper, appendix, policy record, or another source. A gap in the source set should remain a gap in the review; do not fill it with a confident guess.

Measurement is a separate limit. A policy's existence, who qualifies, who receives it, and how much they receive may be different measures.

A result can also change because records improve or the meaning of a recorded event changes. Keep these details attached to the study, especially when two papers use the same label for unlike measures.

Match the claim to its scope

Scope matters even when the groups can be fairly compared. A result may concern one place, period, group, or level of exposure.

Report that scope before drawing lessons for a new setting. The range around an estimate does not, by itself, account for every design problem or show that the same effect will hold elsewhere.

Use language that matches your review. You can describe a measured gap without endorsing a causal estimate. If you report the author's causal claim, attribute it and name the key assumption still under review.

A cautious note should name the gap. Vague phrases about uncertainty are less useful than a clear account of what the study does and does not show.

Review an exposure change in Atlas

Add the paper and useful supporting sources to Atlas, then wait for processing. Mention the chosen sources in a focused question.

For example: “Compare the external event, exposure rule, groups, outcome dates, and stated assumptions in these sources. Cite each row and mark details you cannot locate.”

Keep the request tied to the papers you need. A Project can limit new retrieval to its sources, but earlier chat context can still matter. When reviewing a new study, name the source set you mean.

Choosing a set does not ensure that every method detail has been found. Check missing fields rather than treating a neat table as a complete record.

Check the cited assignment account

Open each citation and read the passage with nearby text. For Snow's case, check the move date, supply pattern, and period of deaths in turn.

A citation to a whole study may fit the topic without supporting the exact sentence. Record a section, table, or paragraph location where the source provides one, and compare the words with the claim you plan to keep.

If an answer calls supply random, ask a named-source follow-up: “What passage documents the assignment process? Separate Snow's comparison argument from evidence of a known randomization procedure.” If the passage remains unclear, read the source directly. A smooth summary should not stand in for the missing proof of how people received their water.

Revise the table so that source facts, author claims, your review, and open questions remain distinct. Atlas can help organize and compare the supplied text; it does not test causal assumptions or estimate effects.

Create a Note from the checked work and confirm it shows Saved before relying on it later. Keep the source links with the claims they support.

Save a bounded study appraisal

Finish with a paragraph that names the event, exposure rule, groups compared, result window, and main open assumption. Attach source locations to the parts that support your conclusion.

For the water inquiry, keep the intake move and company supply account without adding a known random procedure. The note should let another reader see where your claim comes from.

Choose the next check from the actual gap. That might mean reading an appendix on the exposure rule, finding a start date, or checking what a death rate counts.

Name that part of the study so that the next reader knows what to seek. A broad demand for more evidence gives them less to act on.

When comparing several papers, synthesize research papers around those same details. Keep each study's exposure and assumptions visible before combining conclusions. Save the checked review with its limits intact, so that a useful summary does not lose the reason you trust it or the question you still need to answer.