Before a consumer goods team acts on a research report, check nine things: the screener, the achieved base for the cut you care about, the field dates, the weighting scheme, the question wording behind the headline, whether differences are actually significant, how open ends were coded, what the response rate and quality controls were, and which recommendations are supported by the data rather than by the author. Most bad launch decisions trace back to one of these, not to bad fieldwork.
Fieldwork is the part everyone scrutinises and the part most vendors do competently. The report is where a defensible dataset quietly becomes an indefensible recommendation. A 42 per cent purchase intent on a base of 61 becomes "four in ten shoppers would buy" in a slide headline, and by the third meeting it is a fact.
These nine checks take about twenty minutes on a typical deck. Run them before the debrief, not after, so the questions land while the analyst is still in the room.
Who was allowed into this study? Category buyers, or anyone who says they might buy? Past-three-months purchasers, or ever-tried? The screener defines what the numbers mean, and it is almost always in an appendix nobody opens. If the screener lets in non-buyers, every purchase intent figure is inflated.
Total sample base sizes are reassuring and often irrelevant. If the decision is about young urban mothers, the number that matters is the achieved base for that group, not n=1,000 overall. Anything under about 80 in a subgroup should be treated as directional and labelled as such in the slide, not just the footnote.
Fieldwork run over a major festive period, a competitor launch or a price rise reads differently from fieldwork run in a quiet month. In Southeast Asia this matters more than most global templates allow for, given Ramadan, Chinese New Year and Deepavali all move.
Weighting is normal and usually sensible. What you need to know is which variables were weighted, to what source, and how extreme the weights got. A small cell weighted up heavily can swing a total. Ask for the effective sample size after weighting, not just the raw count.
Chart titles are written by humans with a point of view. "Consumers prefer our pack" often sits on top of a question that asked which of two packs stands out more on shelf, which is a different claim. Ask for the questionnaire and read the three or four questions behind the headline slides.
A 4-point gap on a base of 200 is usually noise. Reports often show differences without testing them, or test them at a confidence level buried in a methodology slide. If a recommendation rests on one gap, ask explicitly whether that gap is significant and at what level.
Verbatim responses get grouped into themes, and the grouping decisions shape the story. Ask to see the code frame and a sample of raw verbatims. If coding was automated, ask whether a human reviewed the frame. This is where the most interesting and most distorted findings both live.
What was removed and why: speeders, straight-liners, duplicate devices, nonsense open ends. A vendor that cannot tell you how many responses were discarded has probably not looked. On Vase.ai responses are AI-validated before they reach the dashboard, and we can show the exclusion counts; expect any serious provider to do the same.
Recommendations are valuable, but they are judgement layered on evidence. Ask which recommendations would survive if one finding were reversed. The ones that would not are the ones to interrogate.
| Check | What to ask for | Red flag |
|---|---|---|
| Screener | The full screening criteria | Non-buyers admitted to a purchase intent study |
| Subgroup base | Achieved n for your key cut | Under 80 presented as a firm number |
| Field dates | Start and end dates | Fieldwork over a festive or launch period, unflagged |
| Weighting | Variables, source and effective sample size | No answer, or very large weights on small cells |
| Wording | The questionnaire | Chart title makes a stronger claim than the question |
| Significance | Test level on key differences | Differences shown but never tested |
| Open ends | Code frame and raw verbatims | Themes with no visible coding logic |
| Quality controls | Number and reason for exclusions | Vendor does not know |
Push back hardest on base sizes and question wording, because those are factual and fixable. Push back on significance claims, because they are cheap to verify. Be more generous about weighting and coding judgement calls, which are legitimate craft decisions where reasonable researchers differ.
And separate the report from the platform. If you are also reviewing who should run the next study, our piece on what mid-market CPG teams should look for in research platforms covers the vendor question rather than the deliverable.
What base size is too small to report in a consumer goods study?
As a working rule, under 30 should not be shown as a percentage at all, and 30 to 80 should be labelled directional with the raw count visible. Above 100 you can present a percentage with a margin of error, and above 300 you can start comparing subgroups with reasonable confidence.
Should I trust AI-generated commentary in a research report?
Treat it as a first draft, not a finding. AI summaries are good at describing what the chart shows and poor at knowing which differences are significant or which context matters. Ask whether a researcher reviewed the commentary and who is accountable for it being correct.
How do I know if a research report was weighted correctly?
Ask for three things: which variables were weighted, what population source was used as the target, and the effective sample size after weighting. If the effective sample size is much lower than the raw count, the weighting is doing heavy lifting and the confidence intervals are wider than they look.