Market Research Challenges: Diagnose the Evidence Gap

The hardest market research challenges often come from an unclear decision, data that miss the target audience, or claims that have lost their source context. The remedy depends on the cause. A weak sample needs better evidence; two reports with different scopes may need a careful comparison first.

For example, a team may see one report favoring self-service onboarding and another favoring guided setup. Before choosing a direction, it should check whom each report studied, when, and what “favoring” meant.

Atlas can compare the selected reports with citations you open and check. An unsuitable sample still needs better data.

What makes market research fail

Start by locating the break in the chain from decision to evidence to claim. Four common failure modes need different work. The question may be vague. The people or the survey may be a poor fit. Key sources may be missing. Or the final claim may go beyond what the sources show.

The Greenbook practitioner guide describes vague goals, weak samples, poor questions, late work, and findings that teams cannot use. Those are symptoms. Find which one affects the decision at hand.

Table 1: Four market research failure symptoms with the diagnostic question and next action for each.

The rubric is a triage tool. It does not rank the size of each problem. An unclear question can make a perfect sample irrelevant, and a well-defined question cannot repair a poor sample.

Diagnose the decision and sample

Write the decision as a sentence. “Should we offer guided setup to US small businesses next term?” is more useful than “What does the market want?” It identifies the choice, audience, and time frame. Then ask what evidence could change that choice.

Check who the data represent

Feveile et al. (2007) report how 4,000 sampled adults moved through contact, response, completion, and analysis in one randomized comparison of telephone and postal survey modes. The counts are specific to that Danish study. The flow chart illustrates what to disclose when checking sample coverage; it says nothing about market representativeness. Source article, CC BY 2.0.

Find the study's target buyers, sample source, recruitment method, field dates, and exclusions. A report on large firms may say little about small firms you have not reached. A few interviews can reveal reasons to investigate. They do not show how common those reasons are in the market.

AAPOR's disclosure standards call for the population, recruitment method, sample size, dates, question wording, data quality steps, and limitations. These details let a reader judge whether the evidence fits the decision.

Inspect the question itself

A survey can recruit the right people yet ask a question that steers them. Compare the exact options and wording. Does “prefer self-service” mean choosing from a forced list, ranking several options, or describing an actual setup experience?

The Greenbook guide flags confusing questions and poor method fit. The ICC/ESOMAR code calls for research that fits its purpose and names its sources and limits. A summary cannot repair a bad survey question.

Resolve conflicting market evidence

Put each claim beside its source before calling the reports contradictory. Check geography, buyer segment, field date, population, definition, and method. Two headlines may use the same word for different measures.

Align the scope first

Suppose one fictional report covers US firms with fewer than 50 staff this year. Another describes interviews with UK enterprise buyers two years earlier. Their views may differ. That is a scope difference, so neither report directly overturns the other.

Even when the scope looks similar, read the method. Two surveys may reach different types of firms through different panels. AAPOR's disclosure requirements make those design details a required part of the report.

Investigate the remaining conflict

If both sources ask the same people the same question at the same time, an opposite result needs a closer look. Check how each found its people, how it weighted them, what answers it offered, who it left out, and the passage behind each headline. If a method detail is missing, keep the conflict open.

The AAPOR Transparency Initiative asks researchers to show their methods. AAPOR says that showing a method does not by itself make a study good. Use the details to make your own call.

Table 3 from Tonkin-Crine et al. counts 234 paired comparisons in a health trial review. The study found three cases where findings clashed, beside cases of agreement, partial agreement, silence, and no comparison. It shows how to compare two kinds of evidence, and it is from health research rather than a market. Source article (CC BY 4.0).

For a broader method of comparing cases on the same questions, see comparative analysis. The scope check here answers a narrower question. Can the gap be closed with the market sources you already have?

Check data quality and provenance

Before trusting a headline, ask how the data were gathered. For surveys, check who joined, how they were screened, whether the questions were tested, and how poor responses were handled. For interviews, check whom the team spoke to, what it asked, and whether quotes retain their context.

A Bureau of Labor Statistics comparison shows how source definitions and coverage can change what a consumer-spending figure means.

Appendix A, printed page 52, from the BLS report prepared for the March 2003 Federal Economic Statistics Advisory Committee meeting. The page describes source definitions, populations, and collection methods; it is a historical methods example, not current market evidence. Source report (PDF).

Read the method and limits

AAPOR's survey disclosure standards ask researchers to report checks for attention, logic, and duplicate responses. They also ask for exclusions and study limits.

The ICC/ESOMAR code keeps oversight with the people running the research, whatever tools they use.

These standards help you ask for the method. They do not certify a report just because it has a methods page. Check whether the disclosed design serves your decision and whether a stated limitation changes your conclusion.

Trace a claim to its passage

For each claim in the brief, save the report title, page or section, date, and a short account of what the passage says. Separate a source's observation from your inference. If the source is silent about a buyer segment, mark the gap. Silence cannot support a claim that the segment has no demand.

This government release links a reported finding to its chart and source information. Its $316.1 billion estimate covers U.S. retail e-commerce sales for 2025 Q4, seasonally adjusted but not adjusted for price changes; it is not an estimate of a fictional buyer segment or total market demand. Source: U.S. Census Bureau, Q4 2025 Quarterly Retail E-Commerce Sales.

Scattered documents make this work slow. An industry guide from AlphaSense describes the effort of finding useful information. Its product remedy is a vendor claim. Your source trail still needs direct checks. A guide to cited AI answers explains why a citation starts the review.

If the source set is mainly published studies, synthesizing research papers can help turn checked findings into a coherent account. The sample and scope checks above still apply.

Choose the next research action

After the checks, make one of four decisions. State who will own the next step and what evidence would close the gap.

1. Clarify the question when the requested research does not map to a real decision, audience, or date.
2. Collect better data when the current sample, question, or coverage cannot answer it.
3. Reconcile existing sources when claims differ because of definitions, dates, or segments you can identify.
4. Retain a stated uncertainty when a material conflict remains and new data cannot be gathered before the decision.

When the checked finding points to a design problem, a How Might We statement can turn it into an open question for ideas. Keep the source and any counterexample beside the prompt so the team does not skip from one quote to a favored feature.

The fourth choice is still a decision. Say what the team knows, what it does not know, and how the unknown could change the action. Avoid giving a single confident answer just because a report has a sharp headline.

The ICC/ESOMAR code requires clear methods, sources, and limits. People remain responsible for the work.

If your next task is software choice after you name the evidence problem, see market research AI tools.

For a wider comparison of research workspaces, use the AI research assistants guide. Match the tool to the gap you found: source review, new data collection, or analysis.

A worked discrepancy log

The following reports are wholly hypothetical. The claims, dates, and document IDs were invented to show a review process. They are not observed market findings or a real Atlas output. A team is deciding how to support US small businesses during onboarding next term.

Table 2: Hypothetical discrepancy log comparing three reports by scope, term meaning, unresolved issue and next action.

The log changes the next action. A and B warrant a closer sample and question check. C supplies a possible question for new interviews, but it cannot vote in a US small-firm preference estimate. Nothing here justifies a fabricated percentage or a market-size forecast.

For a real but historical example, the U.S. Bureau of Labor Statistics compared Consumer Expenditure Survey (CE) and Personal Consumption Expenditures (PCE) aggregates. Its 1997 comparison shows that the series differ across categories and flags which items were considered comparable. The report explains that the sources have different coverage and methods; neither column is a universal market estimate or automatic ground truth.

Table C1 from the BLS report prepared for the March 21, 2003 Federal Economic Statistics Advisory Committee meeting. It compares 1997 CE and PCE aggregates using a 1992 PCE benchmark; the asterisk marks items considered comparable. Historical U.S. consumption-statistics example, not a current market estimate. Source report (PDF).

Write the decision brief

A checked note could say: “The supplied surveys make opposing preference claims for US small firms. Their eligible groups differ, and A's recruitment is not shown. We will request A's method and compare the overlapping segment before using either headline. C gives interview prompts and cannot supply a US estimate.”

That note retains the claim, its scope, the missing method, and the owner of the next step. It is more useful than merging the three headlines into one average.

Next step: check sources in Atlas

Add the reports you may use to one Atlas project. Mention the selected survey reports and interview summary with @. Ask for a source-by-source comparison using the exact fields in the log.

Compare these selected market reports on geography, buyer segment, field date, sample recruitment, question wording, and the meaning of “preferred.” Cite the passage behind each entry. Mark missing method details as unknown. Separate a scope mismatch from a remaining conflict, and list the next evidence request. Do not estimate market size.

Open each citation and read the method and surrounding text. If Atlas joins two sources or fills an empty field by guesswork, correct the log from the documents. Save the checked discrepancy and next-action note. The researcher still decides whether to seek new data or make a limited decision now.

Atlas can help compare the supplied text and retain checked notes. It cannot determine whether a sample represents the intended market, acquire missing respondents, validate a survey's statistics, or turn document agreement into market truth.