appraising-qualitative-studies

Appraising Qualitative Studies: A Practical Guide for Reviewers

Qualitative research is often treated as the awkward guest in evidence synthesis. Reviewers trained on trials and cohort studies reach for checklists designed for numbers, find that they do not fit, and either exclude qualitative work or include it without any appraisal at all. Neither response is good scholarship. Qualitative studies answer questions about experience, meaning, process and context that quantitative work cannot, and they can be appraised rigorously on their own terms. This guide sets out a practical approach to reading and appraising qualitative studies, with notes on where a source-linked research assistant like Yila can help a review team work faster without flattening nuance.

Start with the question and the fit

The first appraisal question is whether a qualitative approach suits the research question. Questions about how people experience a service, why an intervention works in some settings and not others, or what a concept means to a particular group are natural fits. Questions about how much or how often usually are not.

Then look for coherence between the question, the methodology and the methods. A study claiming to use grounded theory should show iterative sampling and constant comparison. A phenomenological study should focus on lived experience and describe how the researchers handled their own assumptions. Mismatches between the stated approach and the actual procedures are a common weakness.

Sampling and recruitment

Qualitative samples are not meant to be statistically representative, but they should be purposeful. Ask:

• Who was recruited, and why those people?
• Were there groups whose perspectives might have been missed?
• How was the sample size decided? Appeals to "saturation" are common; good studies explain what saturation meant in practice.
• How were participants approached, and could the recruitment route have shaped who agreed?

A study of patient experiences that recruited only through a support group may capture a particularly engaged subset. That is not a fatal flaw, but it affects how far the findings travel.

Data collection

Look for enough detail to understand what participants were asked and in what circumstances. Interview guides, observation protocols and the setting of data collection all matter. Note who collected the data and what relationship they had with participants; a clinician interviewing their own patients raises different issues from an independent researcher.

Duration and depth are relevant too. Ten-minute interviews rarely produce rich data. Multiple methods, such as interviews combined with observation or documents, can strengthen a study if they are integrated thoughtfully.

Analysis and interpretation

This is the heart of appraisal and the part most often under-reported. A credible analysis section explains:

1. The analytic approach, such as thematic analysis, framework analysis or interpretative phenomenological analysis.
2. How codes and themes were developed, by whom, and how disagreements were handled.
3. How the researchers moved from data to interpretation.
4. How they looked for disconfirming cases or alternative explanations.

Then read the findings with the analysis in mind. Are themes supported by quotations from a range of participants, or do a few voices dominate? Do the quotations actually illustrate the theme, or do they seem chosen for colour? Are the interpretations plausible given the data presented?

Reflexivity

Qualitative researchers are part of the instrument. Good studies describe the researchers' backgrounds, assumptions and relationship to the topic, and how these might have influenced the work. A short reflexivity statement that says nothing beyond "the researchers were aware of their biases" adds little. A thoughtful one explains specific risks and how they were managed.

Ethics and context

Check that ethical approval and consent are reported, and pay attention to how sensitive topics were handled. Context also matters for transferability: the country, the health or education system, the time period and the cultural setting all shape what participants say. Thick description of context lets readers judge whether findings might apply elsewhere.

Using an appraisal tool

Structured tools such as the CASP qualitative checklist or the JBI critical appraisal checklist provide a consistent set of questions. Use them as prompts, not as scoring systems. Adding up "yes" answers to produce a quality score obscures the fact that some weaknesses matter far more than others. A study with a minor reporting gap and a study with no description of analysis should not receive similar scores.

Record, for each study, the main concerns and how they might affect confidence in its findings. If you plan a confidence assessment of synthesised findings, for example with the GRADE-CERQual approach, these notes feed directly into the methodological limitations component.

Working as a team

Two reviewers appraising each study independently, then discussing differences, is good practice. Disagreements in qualitative appraisal often reflect genuinely different readings, and discussing them improves the review. Keep a short record of how disagreements were resolved.

Where AI assistance helps and where it does not

Reading qualitative studies is time-consuming because the relevant information is spread through long methods and findings sections. An AI research workspace can speed up the mechanical parts: locating the description of sampling, extracting the interview topics, listing the themes and the quotations used to support each, and identifying whether a reflexivity statement is present.

Yila is designed to do this with traceable answers. When it reports that a study used purposive sampling of nurses in three hospitals, it shows the passage it drew from, so the reviewer can confirm and read the surrounding context. That keeps the reviewer in the interpretive role while removing some of the searching.

What AI should not do is make the appraisal judgement. Whether an analysis is convincing, whether quotations genuinely support a theme, and whether a study's context limits transferability are interpretive decisions. They require reading the study as a whole, and they belong to the review team.

Reporting your appraisal

In the review, describe the tool used, how appraisal was conducted and how it influenced the synthesis. Present the results in a table with a brief narrative of the main strengths and concerns across studies. Readers should be able to see which studies contributed most to each synthesised finding and how confident you are in them.

Closing thought

Appraising qualitative research well requires a shift from asking "is this unbiased?" to asking "is this credible, coherent and transparent?" With clear questions, careful reading and honest records, qualitative studies can take their proper place in evidence synthesis, contributing understanding that numbers alone cannot provide.