Undercoverage Bias: Find Who the Frame Leaves Out

Undercoverage occurs when a sampling frame misses people or units the study aims to describe. It can produce undercoverage bias when their missing answers affect the estimate for that target group.

Start with who belongs to the target group and who can enter through the frame. A large sample or a random draw cannot reach people absent from that source. Record who is missing before guessing what that gap does to the result.

What undercoverage bias means

A survey may claim to describe all adults living in a town while drawing names only from a list of home owners. Adult renters belong to the target group but have no route into that draw. The frame’s scope is narrower than the claim.

Statistics Canada’s archived coverage guidance distinguishes missing units from those listed twice or wrongly added. These are different frame errors; state which problem the source shows.

A missing group is a known gap in reach. Its effect on an estimate also depends on the group’s missing answers and how the estimate is made. Do not infer the size or direction of the effect just from the group’s name.

A small group count does not prove undercoverage. Chance, group size or a planned sampling design may explain it. Check the frame and rules for the draw before treating unequal counts or unequal chances as a sign of missing people.

Locate where people become missing

Missing from the frame

Ask whether each target group has a route into the draw. A landline-only frame cannot reach homes with only mobile phones through that source. An old roster may miss people who joined after the list was made.

The sampling-frame guide explains the source used to draw units. Census quality guidance includes a mismatch between a frame and its target group among coverage concerns.

Selected but not responding

If people are on the frame, selected and invited but do not supply data, examine nonresponse. Sending another reminder may help reach those people. It does not put people missing from the frame onto the original list.

Use the nonresponse guide to trace missing selected replies. The Census SIPP methods page also keeps coverage and loss of participants over time distinct.

Choosing to enter an open poll

An open poll may reach only one social group, then rely on people choosing to reply. Limited reach and self-selection can occur together. Record both stages rather than using one label to explain the final sample. That makes clear which question a new source would need to answer.

Use the voluntary response guide to check opt-in entry. A selection-process review can link the frame, replies and excluded data without treating them all as one cause.

Worked check of a neighborhood survey

Suppose a fictional study asks all adults in a neighborhood about local services. Names were drawn from a homeowners association list. The report gives no other list of residents, update date or route for renters to join.

No counts or findings are supplied. Build an inclusion-gap note from these facts. Mark who can enter, who is left out and whose status is unclear. Each judgment needs a rule from the source to support it.

Table 1: The scenario is fictional. In a real note, attach the relevant source passage to each row.

Avoid expanding what the list shows

“All home owners are covered” may still be too broad. The list only shows a route for listed owners who live in the study area. It does not show that every owner belongs to the association or that each entry is current.

Ask for the list description before extending that claim. Official frame documentation guidance calls for stating the groups covered and known gaps.

Correct the all-residents claim

Replace “the survey captures all residents’ views” with “the draw used the homeowners association list.” Record that renters have no stated route into the draw and that the report does not show whether recent movers are on the list.

Do not add “renters disagree more” unless the sources provide evidence. A paper-analysis workflow helps keep the reported result, its named group and the missing source facts in distinct fields.

Assess what a remedy can change

Random selection uses the available frame

A random draw from the same list still cannot choose people who are not on it. Increasing the number selected from that list does not repair the missing route for renters. First ask which source change would make those target units reachable.

AAPOR’s survey practices discuss frame choice and multiple sources. Their probability-sampling distinction concerns known selection chances, not a claim that every valid design must use equal chances.

Add sources with overlap checks

A list of addresses might reach homes outside the owners list. It would still need checks for new addresses, shared homes and who is chosen within each home. Two sources may also list the same people.

For a plan to join frames, look for overlap and rules on who counts. The UN household sampling handbook explains drawing samples in stages. Choosing areas does not itself show that every home was listed.

Weighting is not direct observation

Weighting can bring known group totals into line with controls. It does not directly reveal missing groups’ answers. Ask which features were used, where the controls came from and how the new estimates were checked.

The Census SIPP coverage discussion describes weighting but also notes limits in what is known about its effect on bias. Keep those limits clear when a report claims to fix a gap.

Use multi-source synthesis to compare the study with its frame appendix and benchmarks. Group names, dates and scope need to match before totals can help assess reach.

Save an inclusion-gap note in Atlas

1. Add the target-population statement, frame description and reported limitations to one project. Use approved descriptions or extracts; personal names and contact details are not needed for this reading task.
2. In Ask a question, type @ to select the sources. Ask: “Compare the target groups with the sampling frame. Cite the inclusion rule for each group. Distinguish omitted groups from groups whose status is not reported.”
3. Open each citation and read the surrounding text. Check whether a passage names the target population, the frame or the respondents. Ask for the exact rule if an answer claims all residents were covered.
4. Correct the homeowners example to listed eligible owners. Keep renters’ missing route distinct from recent movers’ unknown status. Do not convert either gap into an invented result about their views.
5. Select New → Note, save the inclusion statuses, checked claim and unanswered questions, and wait for Saved. Revisit the note when an updated frame description or author clarification arrives.

The screenshot shows a source beside a cited answer. Read the text to check a claim about who is covered. Its AI paper is not the neighborhood study; the image shows how to inspect text, not a completed frame test.

A citation can support a stated inclusion rule; it cannot establish coverage the source does not report. Visible paper: The AI Scientist-v2, Yamada and colleagues, CC BY 4.0. Screenshot copied unchanged.

Researchers still judge the design and its effects. The note keeps the stated scope and gaps clear. A later source can change the claim while you retain the facts on which the earlier note was based.

Keep the population claim bounded

Report who can enter through the stated frame and which target groups are missing or unclear. Ask how the list was updated and who was added when those facts could change the claim. Keep missing views unknown.

For several studies, a literature-review process helps compare each frame with its own target. Samples of the same size need not have the same reach. A narrower claim does not prove the estimate free of bias.