Process Evaluation With a Delivery Evidence Example

Process evaluation checks how a program ran: what staff did, who took part, and what helped or got in the way. It helps a team understand what happened before judging the program's results.

Suppose a plan promises six workshops, but the folder contains five session logs. That leaves a question about the sixth session. It does not, by itself, prove the session was canceled. A useful review keeps the plan, available records, and unanswered question together.

Atlas can compare chosen files and help you inspect the passages behind a draft claim. You check the records and decide what they support.

What process evaluation examines

Check what happened

A program can change between its plan and its start. Staff may change the time, leave out a task, or use new materials for a group. Process evaluation asks how the work took place and why it changed. AIFS's practice guide includes checks on where the work differed from the plan.

The MRC framework groups questions under implementation, mechanisms, and context. Implementation asks what staff did and how they did it. Mechanisms concern how people respond and how the program might bring about change. Context covers the setting and events that can affect the work or its results.

The diagram links the program's design to what staff do, how people respond, and what changes. Its arrows show how the setting affects those parts and can be affected in turn. Figure 1 by Moore and colleagues is reproduced under CC BY 4.0, cropped from the published PDF with the diagram intact.

Distinguish fidelity, dose, and reach

Three terms help you choose evidence for what happened. A check against the plan, a count of visits, and a list of people reached answer distinct questions:

• Fidelity: Did staff follow the key parts of the plan? A workshop may run but leave out its practice task.
• Dose: How much did staff provide or people receive? The time spent teaching and the number of visits measure distinct things. State which you mean.
• Reach: Did the people the program meant to serve take part? A list of attendees helps, but you also need to know who could have joined.

The government method guide places these questions beside people's views and the setting. A full review may need new data as well as existing records. File comparison is one part of that work.

For example, a staff note that people lacked transport suggests one reason they missed sessions. To learn how many faced that problem, ask the people affected, including those who never came. AIFS's reach guidance helps frame whose views and access the review should examine.

Process evaluation versus outcome evaluation

Process evaluation asks how the program ran. Outcome evaluation asks what changed for the people or issue it aimed to help. The outcome evaluation guide shows how to review the measures, timing, and claim limits for that second question. James Bell Associates separates feedback on how the work was done from checks on its results.

A workshop team might ask whether people got the planned practice tasks. That is a process question. Asking whether they knew more after the workshops is an outcome question. Each needs its own evidence. A log can show that staff ran a task, while a test can measure what people learned.

Both kinds of inquiry can run together. Knowing how the work went may help explain a poor result or why sites differ. It can also suggest a change to test next. TSNE's practitioner account connects findings to changes a team could make in its work.

Enjoying a workshop does not show that people learned more. Higher test scores alone do not prove the workshop caused the rise; a study must address other possible causes. The MRC guide explains how process findings fit alongside work on outcomes.

Plan questions around a delivery decision

Choose the decision and scope

Start with a choice someone must make, such as whether to change workshop times next term. That gives the review a purpose. A request to collect all program data gives staff no clear point at which to stop.

Name who will use the findings and when they need them. Set the site, group, dates, and plan version to review. Compare the work with the plan in force at that time. A later plan may list new dates, which could make the earlier work look wrong if you use it as the baseline.

Use the program theory to find the parts expected to matter. For each one, ask how you would know staff did it and people received it. Note which parts staff could change and which had to stay. The MRC framework links these checks to the way the program is expected to work.

Assign questions and owners

Choose a small set of questions, such as:

1. Which planned sessions and activities were delivered?
2. Who attended, and which intended groups may be missing?
3. What changes did staff make, and why?
4. What barriers did participants and staff describe?
5. Which delivery changes should the team consider next term?

Fit the questions to the choice at hand and the data you can gather. The evaluation questions guide helps link them to the people who will use the findings.

Give each question an owner and a date for gathering evidence. RAND's process evaluation planner links questions with methods, timing, and staff roles. Do this before the program starts so the review has the records it needs, instead of just the files left over at the end.

Choose evidence for each question

Match each record to the claim it can support. A timetable shows the plan, while a dated staff log says what happened. Watching a session can show whether staff ran a task, and people's accounts can explain what a checklist misses.

State what you count. A count of people tells you how many took part. A count of visits tells you how often they came. Someone who comes four times is one person and four visits. Keep those totals distinct when you describe reach.

To work out reach as a share, you need to know who could have joined. That group supplies the denominator, the total against which you compare the count.

A list of attendees alone cannot show the share of local carers reached. The government guide treats reach as a question about the intended group.

For feedback, record who you asked and who replied. Did you include people who left early or never came? Praise from attendees can help you see what they liked, while leaving the reasons other people could not join unknown.

RAND's guide includes counts, feedback, staff views, and checks against the plan. Use each for the question it can answer. A single success score would hide which part of the work went well and where gaps remain.

Before adding files to a tool, check that you may use them for the review. Remove names or other details you do not need and follow the program's access rules. Keep dates, authors, versions, and the time span covered so someone else can trace each finding.

Work through a process evaluation example

Read the plan beside the records

This program and its records are fictional teaching examples. Every table entry was invented to show how to check records against a plan.

A community center plans six weekly workshops. Each should include an explanation, a practice exercise, and participant discussion. Its coordinator must decide whether to change the schedule and log format next term.

The selected record set contains a plan, session logs, a narrative summary, a room notice, and optional feedback. Compare their claims before deciding which delivery changes are supported.

Table 1: The table distinguishes a documented omission from an unrecorded event and limits each finding to the available records.

Turn gaps into checks

The practice row supports a claim that staff logged a missed task in session three. It does not show that practice never took place. The reach row describes who came, but tells us little about people who tried and failed to join.

The table applies the question-and-evidence approach in RAND's planner. The gap column gives the team a check to make for each open question. It keeps missing facts visible.

The first useful change might be to add fields for canceled, moved, and makeup sessions to the log. Before changing times, the team needs to hear from people whose needs were missed. These changes concern how the workshops run. Neither shows that they improved what people know or how they feel.

Resolve conflicts without filling gaps

The fictional summary S1 says all workshops ran. Log L1 lists five dates, and notice N1 says the room was closed. The files do not agree, and none yet settles whether the sixth session ran.

Check what each file covers before choosing an account. S1 may include a makeup session after the dates covered by L1. The log may have gaps. The room notice may explain a move to a new site. Ask staff to confirm the dates or supply a makeup record.

Until that check is done, revise the claim that all six workshops ran. Say that five dates are logged and that the sixth session remains unconfirmed in these files. That states both the evidence and its limit.

Use three labels in the working note:

• Supported finding: a claim the record supports.
• Possible explanation: a reason that still needs a check.
• Missing evidence: the record or account needed to settle the question.

Keep these labels when you copy a finding into the report. Without them, a possible reason can start to read like a fact. AIFS's guide calls for examining both what happened and what may have affected it.

The MRC guide also distinguishes ideas formed during the process review from reasons offered after outcomes are known. Note when you checked the process data and when you first saw the outcome results. Readers can then tell whether a reason was expected or came to mind after the result.

Compare implementation records in Atlas

Use Atlas to compare chosen files and keep the passage behind each draft claim within reach. Start with records you may hold in the project. For this fictional example, choose P1, L1, L2, S1, R1, N1, and F1. In your own review, use clear file names and dates.

Once the relevant sources have finished processing, open the project chat. In Ask a question, type @ and select each source for a bounded question:

Compare this plan, the session logs, the summary, and the feedback. Use one row for each part of the plan. State what was meant to happen, what the files say happened, and the source date. Cite the passage for each claim. Show where accounts differ and what evidence is missing. Do not assume a session was canceled just because there is no log. Keep possible reasons distinct from findings.

Treat the answer as a draft. Open the numbered citation for each claim and read the text around it.

Has a planned task become a finished task in the answer? Do the dates match? Does a finding from feedback speak only for those who replied? Correct the claim if its source does not support it.

If Atlas says six sessions ran, check the cited source. S1 says that in its summary, while L1 logs five dates. Ask Atlas to show each account on its own, then correct the finding by hand. If a citation is missing or unhelpful, find the record by name and read it yourself.

Save the checked table in a note with the makeup-session question and who will check it. When a new record arrives, revise the claims it affects and keep the source behind the change.

The checked table helps you see what to ask for next and which claims you can draft. The team still chooses methods, gathers missing evidence, and decides what to report.

Report findings and next checks

A process evaluation report should let readers see what happened, why it may have happened, and what the team wants to change. For each finding, state the question, evidence, dates, and limit. Where a choice is still open, name who will check it and what evidence they need.

In this workshop review, the report could say that one practice task was logged as missed and the sixth date is unconfirmed. It could suggest a better log and ask people who could not come about workshop times. Those findings do not show that the program helped people learn more.

State which files you could not get and whose views are missing. If only people who kept coming gave feedback, say so beside the finding. Readers deciding whether to expand the program need to know who was left out as well as what people liked.

Use process findings to change how the work is done or plan another inquiry. A contribution analysis asks about the program's role in a change that took place. It may use process data, but it needs more reasoning and evidence than this table provides.

Before closing the review, check the basis for each proposed change. Better makeup-session records follow from gaps in the log. A claim that the program works well enough to expand needs outcome evidence and a sound study design. James Bell Associates' process and outcome guide helps keep those claims distinct.