The Industry Researcher’s Guide to Effective Literature Searching

We critically appraise information constantly, formally or informally, to determine if something is going to be valuable for our purpose and whether we trust the content it provides.

In the context of a literature search, critical appraisal is the process of systematically evaluating and assessing the research you have found in order to determine its quality and validity. It is essential to evidence-based practice.

More formally, critical appraisal is a systematic evaluation of research papers in order to answer the following questions:

• Does this study address a clearly focused question?
• Did the study use valid methods to address this question?
• Are there factors, based on the study type, that might have confounded its results?
• Are the valid results of this study important?
• What are the confines of what can be concluded from the study?
• Are these valid, important, though possibly limited, results applicable to my own research?

Once you have narrowed down your pool of results to those relevant enough to get the full text, it's time to begin critically appraising your articles.  Using a checklist helps you scrutinise articles in a consistent, structured way.  

Questions to consider include: 

• Are the aims of the study clearly stated?
• Is the study design suitable for the aims?
• Are the measurements and methods used clearly described?
• Are the correct measurement tools used?
• Are the statistical methods described?
• Was the sample size adequate? 
• Are the methods overall described in enough detail that you could replicate the study?
• Does the discussion overall reflect the results?
• Who funded this study?
• What are the specific limitations of what can be concluded from the study?

Working through the questions will help you identify the strengths and weakness of each article, and inform how much weight you put on its research in your projects. 

Levels of hierarchies can be useful for assessing the quality of evidence. In health sciences, these are portrayed as a pyramid with levels for the different types of study design.

Understanding study designs will help you judge the limitations of what can be concluded from a particular study. 

While in health related research this hierarchy of study types can be used as a guideline, you cannot rely on the hierarchy to substitute for critical appraisal.  A strong cohort study would be more useful than a flawed systematic review.  There are times when evidence is better sorted by its usefulness for your own research question than by type of study design.

In food research, there is no consensus around the hierarchy of evidence.* 

When you plan your search, think about the type of research you are looking for and what study designs would be appropriate for it. 

For example, if you are interested in qualitative studies of people’s behaviour towards nutrition, or in animal studies, you are unlikely to find large RCTs as it isn’t ethical or possible to run randomised studies in certain areas, so they may not exist. You are more likely to find qualitative studies that may include survey and interview data, write-ups from focus groups, or other types of studies that exist involving crops or animals (often observational or non-randomised studies).  Similarly, you might decide that challenge studies are the most appropriate way to research packaging and shelf life.

A systematic review is a study of studies, where researchers follow a predetermined and published protocol to find all the primary research studies done on a question, weigh the reliability of each one, and, if possible, extract the data from the studies in order to draw a conclusion from the combined evidence.  

Randomised control trials (RCTs)  randomly allocate participants to either an intervention or a control group, so that conclusions can be drawn about the efficacy of an intervention.  

Cohort studies are observational, longitudinal studies that look at a group of people with a shared experience or characteristic to see how they fare over time in regards to a particular factor. 

Case-controlled studies are observational studies in which a group of cases (i.e. people with a condition or disease) is compared to an analogous  group (similar to the case group except that they don't have the condition) to see if a causal attribute can be found for the case group.   

Cross-sectional studies are observational studies that describe a population at a certain point in time. These studies can locate correlations but not causal relationships.  

*Reported in Research Synthesis Methods and EFSA Journal.

As you closely examine full articles, you will be making judgements about why to include or exclude each study from your work.  Documenting your reasoning will help you reassure yourself and demonstrate to others that you have been systematic and unbiased in your appraisal decisions.  You can use a worksheet like the one below, or capture notes within reference management software if you are using it. The more you practice critical appraisal, the quicker and more confident you will become at it.

Keeping track of what you have excluded, and why, can help you develop robust knowledge on a topic. 

You can also capture the key points for why an article is useful for your needs. 

BEST PRACTICE RECOMMENDATION: If you include any direct quotes in your chart (or in any notes) be sure to use quotation marks so that you don’t later mistake the words for your own.