Mixed Methods Appraisal Tool: Check Evidence by Criterion

The Mixed Methods Appraisal Tool (MMAT) helps you check the quality of a study's methods. It covers studies that use words, numbers, or both. Use the checks that fit the study to record what its findings support, what remains unclear, and why you chose each response.

For a mixed methods paper, keep three parts in view: the work with words, the work with numbers, and how the authors join them. A quote can support an interview theme without showing whether a survey reached the right people.

It also cannot show how the two parts were joined. Keep those issues clear in your notes instead of hiding them in one score.

When MMAT fits your review

McGill University's MMAT resource places the tool in systematic mixed studies reviews. These reviews can include qualitative studies that work with words, quantitative studies that work with numbers, and mixed methods studies that use both. Each paper need not use both.

The 2018 user guide contains both the checklist and a guide to each item. Read both when you check a paper. MMAT is for studies that collect or use data. It is not designed for theory articles or review papers, and it does not cover every study design. Check whether the tool fits before you start to rate the paper.

Start with the manual's two screening questions: is the research question clear, and can the data answer it? If you cannot tell, check whether it makes sense to go on with the appraisal. These questions help you decide whether MMAT fits. Your review's rules for which papers to include still apply.

McGill links to the authors' project. If its download fails, use a copy from a known institution and check the cover, authors, and version. The copy linked here says version 2018, with an August 1, 2018 update. A recent upload date on another site does not mean the tool itself is new.

Choose criteria from the study methods

The MMAT manual defines five categories:

• Qualitative research.
• Quantitative randomized controlled trials.
• Quantitative non-randomized studies.
• Quantitative descriptive studies.
• Mixed methods studies.

The authors' explanation of the 2018 tool describes five criteria in each group. A qualitative paper uses that group's checks. For a quantitative paper, choose the group that fits its design. For a mixed methods paper, use three groups: qualitative, the right quantitative group, and mixed methods.

Read how the authors chose people, gathered data, studied the results, and joined the parts. The manual's design categories guide that choice. “Survey” alone does not tell you which group to use. The survey might describe a group or test a link between an exposure and an outcome. Nor do numbers and a few quotes, by themselves, make a mixed methods study.

Save the passage that supports your choice of group. If reviewers disagree about the design, settle that issue before they rate the checks. Two careful readers can reach different ratings because they used different groups, even when they agree on what the paper says.

For an appraisal focused on qualitative research, the CASP qualitative checklist guide explains a different instrument and its evidence record. Choose the instrument through your review's purpose; do not substitute CASP items for MMAT criteria.

Look for the link between the parts. Survey results might guide who gets interviewed. Or the authors might compare interview themes with survey results to explain what they found. Read how that link works, rather than relying on the title. To check a design that brings the two sets of results together, see convergent mixed methods design.

Keep evidence separate from the judgment

An evidence map is a set of working notes to use with the official tool.

It helps you find the source text again and see why you chose a response. Keep the official checks; the map adds a record of your reasoning.

For each criterion that fits the study, keep these details:

• The criterion number and the part of the study you checked.
• The report and place you read, including any extra files.
• The passage or a close account of what it says.
• The reviewer's response and reason.
• Any missing details, other ways to read the text, or checks still needed.

Choose a response from the source text

The 2018 author paper sets out three responses: Yes, No, and Can't tell. Yes means the criterion is met; No means it is not. Can't tell means there is too little detail to judge. A missing account does not prove the authors skipped a step. It also does not prove they did it well.

For example, “the report describes interviews” tells you how the authors got some of their data. It does not tell you whether interviews fit the question or whether the data support their claims.

Write your reason beside the source text, so a second reader can check how you moved from the text to the rating.

Work through a mixed methods example

Consider a fictional paper that uses a patient survey and interviews to study access to an outpatient service. The authors describe how they found survey users, show interview themes with quotes, and discuss both sets of results. All study details and section labels below are invented. This example shows how to read a paper; it is not an appraisal of a real study.

One reader might call the paper “well integrated” because its discussion mentions both parts. Before you accept that view, map what the text supports and what you still need to know.

These four rows show different questions to ask of the source text. A full appraisal must still cover all the official criteria that apply.

Table 1: The map keeps the source text apart from the question it leaves open. A quote that supports a theme cannot tell you whether missing survey replies skewed the results. It also cannot tell you how well the two parts were joined.

Check the missing information

Suppose the main report says how many people completed the survey but says nothing about those who could have replied and did not. Record what you know and what remains unclear. Many completed surveys would not, alone, show whether people who did not reply had different access needs.

Before you choose a response, check other reports and extra files allowed by your review's rules. The MMAT manual guides how to handle details that are hard to judge. Save each source location so a second reader can check the same text. If details are still missing, explain what you cannot judge. Do not guess what the authors meant to do.

Examine disagreement between components

Imagine that most survey users report easy access, while people in interviews describe large barriers. The gap could come from who took part, what they were asked, when they were asked, or what they had been through. Check the report for signs of these causes before you choose one for your appraisal note.

The manual's mixed methods checks cover why the authors use both methods, how they join them, what the joint results mean, how they handle results that clash, and the quality of each part.

Look at what they do with the gap. Saying that the results differ still leaves a question: how does their account of that gap follow from the data?

A useful note could say that the discussion spots a gap but gives too little detail to judge its cause. Another note might point to work that links the gap to who took part. Choose the account you can support from the source.

Report the criteria without a total

The 2018 authors discourage an overall score. A percentage can hide which parts of a study cause concern. Two papers with the same number of Yes responses may have quite different problems for the question your review aims to answer.

For the fictional paper, missing replies may affect a claim about how common poor access is. A weak account of how the parts were joined may affect a claim about why access differs. One total score cannot tell the reader which claim has which problem.

The author paper's reporting advice keeps each criterion visible. Use that detail to explain a concern about a particular finding rather than replace it with a number.

Some discovery resources retain conflicting advice. The NCCMT registry entry describes a step to give an overall score. The reviewed manual and author paper advise reporting each criterion in detail.

Follow the authors' advice for the 2018 tool and state which version your review used.

If a course, journal, or review team asks for a total, resolve that request before adding one. Explain the tool's advice and offer ratings for each check with comments.

Do not invent an official pass mark or average unlike parts. If you call a study “high quality,” explain why.

Find and check passages in Atlas

Atlas can help you find and compare passages in the reports you add. Keep your rating apart from the text it finds. A correct source passage still needs a reviewer to judge whether it meets the MMAT check.

Build a checked source record

1. Add the paper, extra files, and MMAT guide that you have permission to use to the project. Wait until each source is ready.

2. Open a chat and use @ to choose the exact sources to compare.

3. Ask a grounded question for source text: “Find the passages about who joined the survey, missing replies, how the interviews were studied, and how the authors joined the survey and interview findings. Give the source and location for each passage. Keep each part separate. Do not assign MMAT responses.”

4. Open the citation beside each claim. Read the text around it to check that it supports the claim. Check that the answer has not joined text from different studies or parts of a study.

5. Fix your source notes before you rate the paper. If a theme quote is given as proof that the parts were joined, find where the authors explain how they brought the findings together.

6. Select New, then Note, and save the checked passages with their source details. Read the manual before you add your own response and reason. Wait for Saved before you close the note.

The screenshot shows a research paper beside an Atlas answer. This is the view you use to check a claim against its source. It shows a reading step, rather than a completed MMAT appraisal.

Read the cited passage and nearby text before you use it to support a rating.

The Atlas screenshot includes ColPali by Faysse et al., made available under CC0 1.0, with the paper content unchanged.

The pictured paper is unrelated to the fictional survey and interview example. These steps follow Atlas's public guides to source comparison, citations, and notes. Reviewers must read each passage and make their own judgments before they use the text in their appraisal record.

If a citation is missing or opens the wrong place, find the source by name and read it directly.

Use the AI citation checker workflow to check whether a cited passage supports the claim. A sentence can be on topic without answering the whole MMAT check. Keep “source not found” and “criterion not met” as separate issues in your notes.

Carry the appraisal into your synthesis

Follow your review's rules for separate appraisals and how to resolve them. The manual recommends at least two independent reviewers with the right skills or training. Software can help them find source text. Sharing one AI response does not give you two independent judgments. The authors' explanation is the instrument reference for criterion responses. Keep that reference beside the study passages so a second reviewer can assess the same question.

Keep the first ratings, the reason for any gap between them, and the agreed response. If reviewers used different passages, go back to the sources. If they read the same passage in different ways, note why they chose the final rating. Agreement does not, on its own, remove a concern about the study.

Link concerns to affected claims

In the review report, name the tool version and groups used. State how reviewers worked and how their concerns shaped the final claims. Link each concern to the claim it affects. A sample that misses some groups may weaken a claim about how common a problem is. That issue may matter in a different way for a claim about one person's experience.

The Royal College of Nursing's resource guide lists appraisal tools alongside reporting guides. They answer different questions. Reporting guides help authors describe their work. Appraisal checks whether the methods support the claims. A complete reporting checklist does not settle the MMAT ratings.

Use the checked profile to plan the next stage of your literature review process. Keep each study's limits in view as you compare findings. Explain how gaps or weak methods affect your trust in each joint claim. If you drop papers based on an invented score cutoff, you lose the reasons the appraisal was meant to keep.