Experimental research design sets out what researchers change, who gets each task, and what they test. When reading a paper, check the method behind each design label.
A study may call group placement random without explaining the steps. A control group may do an active task. Keep these details in a source-linked methods table so you can describe what was tested.
The example below reads two published learning studies. Use Atlas to compare their cited methods and save a checked note. Researchers still judge the design, the analysis, and what caused the findings.
Check reported methods in Atlas
Compare study conditions and save a checked methods note.
What experimental research design reports
In an experiment, researchers make a planned change and then observe what happens. The manipulation is the change they make. Assignment is how a person, class, or other unit enters a study condition. The comparison condition is the task or state used as a reference. If the paper only measures an existing exposure, use the correlational versus experimental research guide to check that distinction first.
Read these as distinct questions. One label may tell you about the tasks while leaving group placement unclear. In a between-subjects design, each person receives one of the conditions being compared. In a within-subjects design, the same people receive those conditions. The psychology methods textbook explains these terms and how groups are formed.
Other labels tell you how changes are combined. A factor is something varied in the study, with each setting called a level. NIST's full factorial guide describes a design that includes every mix of the factor settings. Four conditions could be two factors with two levels each, or one factor with four levels. The factorial research design guide helps map those levels and combinations before interpreting effects.
This guide reviews methods in published papers. To choose a study approach, first clarify your research objectives. You will need a plan for your own question, setting, and resources.
Set criteria for the methods review
Start with a focused question: What changed between groups, how were they formed, and what was being compared? Use your literature review process to choose papers that address it.
Use the abstract as an overview, then find the detailed method. If a needed detail is missing, read linked protocols and extra files. A protocol sets out the plan. Keep any changes to that plan in view when reading what the team did.
Use these criteria for each entry:
- Change: Name the task, dose, or instruction that differed.
- Unit: Record who or what received each condition.
- Entry: Describe how groups were formed and keep the authors' label.
- Comparison: State what the other group did.
- Outcome: Keep what was tested, when, and who took the test.
- Support: Add the source section and any detail still unclear.
The operationalization guide links a broad phrase such as "learning method" to the task people received. Keep the concrete task in your note. A broad idea may lose the detail you need to compare the studies.
A worked manipulation-assignment-control evidence table
The rows below read the methods in two chosen papers. Each paper has more than one row to keep distinct parts of its design in view. The selection shows how to build a methods note and leaves a full study appraisal for further work.
Read each row with its cited method section. The last column keeps a correction or open check beside the method it concerns.
| Paper and passage | Deliberate change | Reported assignment | Comparison and review note |
|---|---|---|---|
| Ritchie et al., Experiment 1, Learning phase | Take notes with the factsheet turned over | Pupils placed in alternating groups along the class register, called random by the authors | Other pupils kept the sheet visible. Keep both the label and method |
| Same experiment, mapping passage | Mind-map notes | One class from each year randomly selected for mapping | Other classes used notes except mind maps. Mapping was assigned at class level |
| Okano et al., Procedure | Unguided discussion, guided discussion, or quizzes during a video | Volunteers reported randomly assigned to four conditions | Baseline watched the video as usual. This passage does not specify how the sequence was made |
| Same paper, next-day assessment | Later knowledge test is the outcome | Return for testing was optional | Those who did not return were excluded. Keep follow-up loss in the note |
Table 1: These rows keep the change, group placement, and later test distinct. Each field answers a different question about the study.
The school study crosses retrieval and mapping factors. The workplace study compares four learning conditions. Both have four conditions, but the tasks are arranged in different ways.
Keep the chosen school experiment distinct from later ones in the same paper. A broad summary could mix tasks or time points from more than one study. Check the section name when you add a row.
Check the procedure behind assignment labels
Preserve the label and method
For the school row, "all groups were randomly assigned" would lose two details. Retrieval groups followed alternate names on the register, while mapping was set at class level. Keep both in the note and inspect the original learning-phase account.
The label and method leave a question to review. Keep the described steps rather than replace them with a random-number process. To judge group placement fully, seek how the sequence and register order were formed.
NIST's randomized-design guidance describes chance-based sequences made by software or physical means. Taking turns alone does not describe such a process. Keep what the source says in view before deciding how to classify the method.
Keep missing detail open
For the workplace row, keep the random label and mark the sequence method as not specified in the reviewed passage. That states what you found in the passage. It leaves open whether more detail exists elsewhere.
Random assignment also differs from random sampling. Assignment places units in study conditions. Sampling concerns who enters the study. The textbook's assignment discussion keeps these steps distinct and notes that chance can still leave groups uneven. The label alone tells you neither that the sample reflects a whole population nor that groups are identical.
When an event outside the research team creates differing exposure, the natural experiment guide shows how to check that event, the exposure rule, and the comparison before choosing a design label.
Read what comparison groups actually did
A control or baseline label needs a task description. In the table, the groups with visible sheets or the usual video still took part in learning tasks. Calling either group "no learning" would change what the studies compared.
Read what both groups received and what differed. Check time on the task, instructions, feedback, staff contact, and access to the content. Ask which details were meant to change and which might make the contrast harder to explain.
Blinding concerns who knew the condition. It is a distinct part of the method. OpenStax's methods chapter explains that people taking part and those scoring tests may know different things. People might recognize their task while a scorer remains unaware of their group. Check the paper before making that claim about a study.
Taking tests more than once does not show that each person received all treatment conditions. Read which tasks each person did before using a within-subjects label. When the same people receive treatments in successive periods, check the crossover study design for sequence, period and carryover details.
Separate learning tasks from outcome tests
The workplace paper's Figure 1 shows a question during learning in panel a and a later test question in panel b. Both concern DevOps content, but their roles differ. The quiz during learning is part of a treatment. The later test measures the outcome.
Original question examples from Okano, Kaczmarzyk, and Gabrieli (2018), Figure 1, reproduced under CC BY 4.0. Full figure converted to WebP.
Keep the test format with the outcome. A score on taught content does not itself show how people later perform at work. Keep who returned for testing in the note, too. The participants and procedure sections describe who was left out after optional return.
A workplace label does not show that a task took place during ordinary work. The field experiment guide helps check the actual setting, tasks and staff routines before drawing conclusions about behavior in that setting.
The same care applies when studies use different delays or tests. Keep those details with the outcome before comparing effects. This table leaves result pooling and rankings for further work.
Compare cited methods in Atlas
Use the papers and permitted protocols as the source set. Name each experiment clearly so you can check a response against the right method.
- Open the project containing the studies. Start a chat and type @ in Ask a question to select each source.
- Choose Project only beside + to keep new retrieval in the project. Earlier chat context remains available, so state the source boundary in your question.
- Ask: "Compare what changed, who got each task, how groups were formed, what the control groups did, and when outcomes were tested. Keep authors' labels distinct from the described steps. Cite each row and mark gaps." Select Send.
- Open the citations for claims you need. Read the full method passage and surrounding text. PDF citations can open a precise passage when its location is known. Find the section by name if needed.
- Correct any row that merges pupil and class assignment, invents a sequence, or calls an active baseline no exposure. Keep the source passage with the revised row.
- Select New, then Note. Save the reviewed table and unresolved checks, and wait for Saved before closing it.
A first response might call all school assignment random. Revise it to keep the label, described steps, and class-level mapping detail shown above. Make the methods judgment after reading the original account.
Save limits with the checked table
Use research notes you can revisit to keep source versions, study names, method sections, revised rows, and open checks together. State whether a detail came from the paper, a protocol, an extra file, or your own reading.
A table can cover your chosen fields while leaving cause-and-effect questions open. Review how groups were formed, what tasks they received, missing test results, and the way scores were analyzed before judging the claim. Add formal appraisal findings once that work is done.
When a protocol, correction, or reviewer question arrives, reopen the row and its source. Revise the note with the evidence that changed it. The next reading can then start from a checked account.
Check reported methods in Atlas
Compare study conditions and save a checked methods note.