Predictive Validity: Trace a Measure to Its Later Outcome

Predictive validity asks whether a measure relates to a relevant criterion assessed later. To read that claim, trace the first score, the later outcome, the time between them and the people in the result. The word predicts is not enough.

A study can support a score-to-outcome link without telling you what will happen to a given person. This guide shows how to build a cited timing and limit table. The training-readiness case is fictional.

What predictive validity means

A test before training could be compared with a score at the end of the course. The later score is the criterion: the outcome used to assess the link. Boateng and colleagues place this within criterion validity.

The open research-methods textbook explains the role of the criterion. Name the exact outcome. A final exam score, task rating and course completion are distinct results even if they all occur later.

Timing separates this from concurrent validity. Two tests in the same session ask a different question from a first test linked to a later course outcome.

Ask what later outcome this paper used and what its result supports. “Is the test predictive?” needs a group, an outcome and a time frame before it can be answered with care.

Reconstruct the measure-to-outcome timeline

Find when the first score was obtained. Record the measure version and study stage. Baseline can be a useful label, but it may not tell you when all the inputs were known.

Next, name the later outcome and how it was scored. Success after training is too broad if the paper measures only attendance or a written exam. Keep the source's exact outcome in the note.

Record the time gap in the authors' terms. They may say six months after entry or at the final exam. If they say only later, do not add a date they did not give.

The date a file was opened is not always the date of the outcome. A team could obtain older school records after giving a test. Check when the event took place, rather than when its record was retrieved.

Use five fields to trace the timeline:

1. First measure: what was known at the prediction point?
2. Later criterion: what outcome was assessed, and how?
3. Time gap: how far apart were the two events?
4. Sample: whose first and later scores entered the result?
5. Claim: what link or model result did the authors state?

This note records the study; it does not fit a model. The textbook's criterion-validity account helps place those fields in the wider measurement question.

If the report concerns a multivariable prediction model, read the TRIPOD scope. It distinguishes model development from evaluation in other data. Its reporting scope is broader than a simple pair of test scores, and it is not a validity certificate.

Check who reaches the later criterion

The people tested first may differ from those with a later score. Some may not enter the course, may leave it or may have missing records. Find the participant flow and the final analysis sample.

Selection can narrow the range of scores in that sample. McManus and colleagues' study discusses this issue in medical-student selection and supplies the diagram below.

The green entrant group occupies a narrower score range than the applicant group.

The horizontal axis is the selection score, and the vertical axis is the outcome score. The green ellipse shows entrants. The red boundary shows applicants, and the dashed outer line shows a wider group.

The diagram shows why data for entrants and a claim about all applicants are different things. It is a source-owned conceptual figure, not this article's dataset or forecast. Figure 1: McManus and colleagues, BMC Medicine, 2013, reused unaltered under CC BY 2.0.

Keep the observed sample in your note. A result among people who finish training does not by itself cover all applicants or all people who start. The original selection paper gives the full method argument behind the figure.

Also note missing later scores and how the authors handled them. If that detail is absent, keep the gap. Do not assume that missing scores were harmless or that a correction was made.

Whether these issues change the result needs a methods judgment. Your note should expose the sample limits so someone can make that judgment, rather than hide them behind a strong coefficient.

Write the timing and limitation table

Suppose a fictional survey is given before a training course. The paper links it to a practical test at the end, among people who finish the course. It does not assess whether each person will finish.

A colleague writes: “The survey predicts who will complete the course.” That changes the outcome. The source concerns a practical score among completers, rather than completion status for all starters.

This original teaching table has invented details and no empirical coefficient. Replace its entries with checked source text. Keep absent details unknown.

Table 1: Repair the claim before using it in a review. A better note says: “The paper links the first survey score to the later practical-test score among completers with records. It does not establish who will finish the course.”

This does not reject the survey. It names the outcome that was assessed and the one that was not. The scale-development primer explains why the intended use matters to a validity argument.

Keep a separate row when another paper assesses a new outcome. Do not turn exam marks, attendance and completion into one field called success. That would erase the distinction the note is meant to keep.

Check the later-outcome passages in Atlas

Gather the measure description, follow-up methods, outcome definition, results and limits. Add the supplement if it gives participant flow or missing-score details. The abstract alone may omit the facts you need.

Create an Atlas project and add permitted sources. Select the relevant material and ask:

Trace the first measure to its later criterion. Find the version, outcome definition, time gap, analyzed sample and author-reported result. Cite each field. Keep missing details unknown and do not forecast individual outcomes.

Check the chronology and criterion

Use the returned answer as a draft and make these source checks:

1. Open each citation and read the text behind the field.
2. Check whether the cited date is a measurement point or just the year of publication.
3. Keep a practical score distinct from completion, attendance or satisfaction.
4. If the answer says all participants but the source says completers with records, correct the sample.
5. Save the corrected table with links and a separate note for your judgment.

Keep sample limits even if the authors describe their result positively. A reviewer should see the source behind each factual field and know which statements are your own reading.

Atlas supports reading and comparing supplied sources. You are responsible for fitting any predictor, evaluating its accuracy, and deciding whether the evidence supports a selection policy or person-level forecast.

Keep the prediction claim within its evidence

Name the predictor, later outcome, time gap and observed group. Then state what the result leaves open. This makes a narrow finding usable without turning it into a guarantee.

Prediction differs from explanation. A feature can help forecast an outcome without showing that changing it causes that outcome to change. Shmueli's paper develops that distinction and helps frame the claim you are reading.

For the fictional training case, keep practical scores distinct from completion. Keep the completer-only sample too. Use discriminant-validity appraisal when the question concerns distinct constructs. If it changes to a new group or setting, use the external-validity guide.

Before finishing the note, check that your claim keeps the source's outcome, sample and time gap. Find the missing evidence or narrow the sentence if it does not. A precise limit table remains useful even when the stronger claim is untested.