6.3 Evaluating sources

In the previous section, we discussed how to formulate search queries to get the most relevant results. At this point, if you still find yourself staring at a Google Scholar window with 1,000,000 search results, you may want to refine your queries using the suggestions in section 6.2. Hopefully, you can find at least a few different queries that provide relevant resources to help you answer your working question or introduce new ideas that might revise or update your working question. Remember that revising and updating your working research question as you learn more about your topic is part of the process.

Skim abstracts

All databases will give you access to an article’s abstract. The abstract is a summary of the main points of an article. It will provide the purpose of the article and summarize the author’s conclusions. Once you have a few good search queries, start skimming through abstracts and find the articles that are most relevant to your working question. Soon enough, you will find articles that are so relevant that you may decide to read the full text.

It is a good idea to cast a wide net at this point, since your project is just beginning. If you like the article, make sure to download the full text PDF to your computer so you can read it later. Save it in a new folder using a descriptive title. It is useful to include the first author’s last name, year, and the first few words of the title since the file names assigned by journals are often unclear.

What do you do with all of those PDFs? You can keep them in folders on a hard drive or cloud storage drive, arranged by topic. For those who are more ambitious, you may want to use a reference manager like Mendeley or Zotero, which can help keep your sources and notes organized. Both of them can also help you build a proper bibliography and organize your notes on each article. At the very least, write some notes on paper or in a word processor reflecting on the articles you skim and how they might be of use in your study.

Which articles to download?

Keeping your working question in mind, you should look at your potential sources and evaluate whether they are relevant to your inquiry. To assess the relevance of a source, ask yourself if the source will help you answer or think more deeply about your working question. Does the information help you answer this question, challenge your assumptions, or connect your question with another topic? Does the information present an opposing point of view, so you can show that you have addressed all sides of the argument in your paper? Does the article just provide a broad overview of your topic or does it have a specific focus on what you want to study? If the article isn’t helpful to you, it’s okay not to read it. No matter how good your searching skills, some articles won’t be relevant. You don’t need to read and include everything you find!

In addition to relevance, you should ask yourself about the quality of the source you’ve found. Is the information outdated? Is the source more than 5-10 years old? If so, it will not provide what we currently know about the topic–just what we used to know. Older sources are helpful for historical information, such as how our understanding of a topic has changed over time or how the prevalence of an issue has increase or decreased. However, unless historical analysis is the focus of your literature review, try to limit your sources to those that are current. Doing so will also narrow down your list of results considerably in a database. Also, you may want to consult with your professor to see if they have additional guidelines on which articles are “too old” to include in a literature review.

Older sources may be important, however, if they are foundational articles. Foundational articles are often cited in the literature because were the first article (or one of the early articles) to introduce a theory, method, concept, or topic. They are clearly important to a lot of scholars in the field. While not all articles are foundational, you can get a quick sense of how important an article is to the broader literature on a topic by looking at how many other sources cited it. If you search for the article on Google Scholar, you can see how many other sources cited this information. Generally, the higher the number of citations, the more influential the article. Of course, articles that were recently published will likely have fewer citations than older articles, and the citation count is only one indicator of an article’s influence.

What evidence is in each type of article?

Literature searching is about finding evidence to inform your ideas, supporting or refuting what you think about a topic. Each type of empirical article (we reviewed some of them in section 6.1) provides different kinds of evidence. As we talked about in section 6.2, a good place to start your literature search is with review articles—meta-analyses, meta-syntheses, and scoping reviews, and systematic reviews. These types of articles give you a birds-eye view of the literature in a topic area, and with meta-analyses and meta-syntheses, conduct empirical analyses on enormous datasets comprised of the raw data from multiple studies. As a result, their conclusions represent what is broadly true about the topic area. They also have comprehensive reference lists that you can browse for sources relevant to your topic.

In my experience, students are often tempted to read short articles because they can complete assignments more quickly that way. This is a trap for students starting a literature review. Short articles take less time to read, sure. But this isn’t about reading a certain number of articles, but finding the information you need to write your literature review as efficiently as possible. You’ll save time by reading more relevant articles with lengthier and more comprehensive literature reviews—in particular, review articles.

Review articles are the best place to start for any literature search, but you should also look for specific types of articles based on your working question. If your working question asks about an intervention, like a therapeutic technique or program, randomized controlled trials (RCTs) provide the strongest evidence if a meta-analysis or systematic review of relevant RCTs is unavailable. Quasi-experimental designs are considered inferior because researchers have less control over the research process. You will want to avoid relying heavily on articles that use non-experimental designs and include words like “pilot study,” “convenience sample,” or “exploratory study” in their methods section. These are preliminary studies that are done prior to a more rigorous experiment like an RCT, and their conclusions are tentative and collected for the purpose of informing future inquiry, not establishing what is true for broader populations. We will discuss experimental design in Chapter 13, but for now, it’s important to know that the purpose is often to establish the efficacy of an intervention and the truth value of the evidence contained in them varies based on the design of the experiment, with RCTs being the gold standard.

Experiments are one of two quantitative designs explored in this book. The other design is survey research. Looking at survey research is a good idea in any project, as it provides evidence about broader populations by generalizing from a smaller sample of people. For example, surveys can tell you about the risk and protective factors for a social problem by querying the people who are likely to experience that problem over time. Longitudinal surveys are often the most helpful in understanding causality because you have a record of how things have changed over time. Cross-sectional surveys are more limited in establishing causal relationships, as they only query people at one point in time. We discuss the differences between these types of surveys in Chapter 12. Both cross-sectional and longitudinal surveys are very commonly cited types of sources in social work literature reviews because their results are often applicable across broad populations. However, they are limited in the degree to which they can establish causality, as they lack the controlled environment of an experiment. As with experiments, students should be very cautious about using survey results that are “exploratory” or a “pilot study,” as the purpose of those studies is to inform future research rather than understand what is true about broader populations.

The previous few paragraphs can be summarized in this hierarchy of evidence, as described by McNeese & Thyer (2004).^[1] The higher a type of article on this list, the better they can reliably and directly inform evidence-based practice in social work.

1. Systematic reviews and meta-analyses
2. Randomized controlled trials
3. Quasi-experimental studies
4. Case-control and cohort studies
5. Pre-experimental (or non-experimental) group studies
6. Surveys
7. Qualitative studies

The hierarchy of evidence is a useful heuristic, and it is based on sound reasoning. A systematic review or meta-analysis will provide you with a better picture of what is generally true for most people about a given topic than a qualitative study. However, generalizable objective truth is not the only thing researchers want to know. For example, if you wanted to research the impact of gentrification on a community, systematic reviews would not provide you the depth you need to understand the stories of people impacted, displaced, and discriminated against in housing policy. That’s not what systematic reviews are designed to do.

To find evidence like that, the most relevant evidence is in qualitative studies, at the bottom of the hierarchy. For this reason, it is better to think of the hierarchy of evidence within the broader context of your working question and the knowledge you need to investigate it. Different types of sources are useful depending on what your question is. Table 6.1 contains a suggested starting point for evaluating what types of literature will provide the most relevant evidence for your project.

Qualitative studies are not designed to provide information about a broader population. As a result, you should treat their results as related to the specific time and place in which they occurred. If a study’s context is similar to the one you plan to research, then you might expect similar results to emerge in your research project. Qualitative studies provide the lived experience and personal reflections of people knowledgeable about your topic. These subjective truths provide evidence that is just as important as other studies in the hierarchy of evidence.

Scholarly Literature

Insert a paragraph to introduce Table 6.2 & Scholarly Literature

Empirical Articles

[[add info here]]

Review Articles

Along with empirical articles, review articles are an important type of article you will read as part of conducting your review of the literature. Review articles are journal articles that summarize the findings other researchers and establish the state of the literature in a given topic area. They are so helpful because they synthesize a wide body of research information in a (relatively) short journal article. Review articles include literature reviews, like this one from Reynolds & Bacon (2018)^[2] which summarizes the literature on integrating refugee children in schools. You may find literature reviews with a special focus like a critical review of the literature, which may apply a perspective like critical theory or feminist theory while surveying the scholarly literature on a given topic.

These days, basic literatures do not get published as often as they used to. Many journals require review papers to be empirical articles, meaning the authors applied systematic methods to conducting their literature search to reduce bias and support replication of their efforts. These empirical review articles include systematic reviews and scoping reviews which are like literature reviews but more clearly specify the criteria by which the authors search for and include scholarly literature in the review (Uman, 2011).^[3] For example, this recent systematic review by Leonard, Hafford-Letchfield, and Couchman (2018)^[4] discusses how to use strategies from arts education to inform social work education. As you can see in Figure 1 of this article, systematic reviews are very specific about which articles they include in their analysis. Because systematic reviews try to address all scholarly literature published on a given topic, researchers specify how the literature search was conducted, how many articles were included or excluded, and the reasoning for their decisions. This way, researchers make sure there is no relevant source excluded from their analysis.

The two final kinds of review articles, meta-analysis and meta-synthesis go even further than systematic reviews in that they analyze the raw data from all of the articles published in a given topic area, not just the published results. A meta-analysis is a study that combines raw data from multiple sources and analyzes the pooled data using statistics. Meta-analyses are particularly helpful for intervention studies, as it will pool together the raw data from multiple samples and studies to create a super-study of thousands of people, which has greater explanatory power. For example, this recent meta-analysis by Kennedy and colleagues (2016)^[5] analyzes pooled data from six separate studies on parent-child interaction therapy to see if it is effective.

A meta-synthesis is similar to a meta-analysis but it pools qualitative results from multiple studies for analysis. For example, this meta-synthesis by Hodge and Horvath (2011)^[6] pooled data from 11 qualitative studies that addressed spiritual needs in healthcare settings. While meta-analyses and meta-syntheses are the most methodologically robust type of review article, any recently published review article that is highly relevant to your topic area is a good place to start reading literature. Because review articles synthesize the results of multiple studies, they can give you a broad sense of the overall literature on your topic. We’ll review a few kinds of non-empirical articles next.

Theoretical articles

Theoretical articles discuss a theory, conceptual model, or framework for understanding a problem. They may delve into philosophical or ethical analysis as well. For example, this theoretical article by Carrillo & Grady (2018)[1] discusses structural social work theory and anti-oppressive practice approaches. Theoretical articles can help you understand how to think about a topic and may help you make sense of the results of empirical studies.

Practical articles describe “how things are done” from the perspective of on-the-ground practitioners. They are usually shorter than other types of articles and are intended to inform practitioners of a discipline on current issues. They may also reflect on a “hot topic” in the practice domain, a complex client situation, or an issue that may affect the profession as a whole.  Methods articles explain research methods and can address a range of topics including difficulties with data collection, lessons learned from implementing an intervention, special considerations when conducting research with a certain population. Both practical and methods articles can be informative , but tend not to help you develop specific research questions.

What articles should you read?

No one type of article is better than the other, as each serves a different purpose. Foundational articles relevant to your topic area are important to read because the ideas in them are deeply influential in the literature. Theoretical articles will help you understand the social theory behind your topic. Empirical articles should test those theories quantitatively or create those theories qualitatively, a process we will discuss in greater detail in Chapter 8. Review articles help you get a broad survey of the literature by summarizing the results of multiple studies, which is particularly important at the beginning of a literature search. And finally, practical articles will help you understand a practitioner’s on-the-ground experience. Pick the kind of article that gives you the kind of information you need.

In one sense, to be a critical consumer of research is to be discerning about which sources of information you read and why. If you have followed along with the exercises in this chapter, you hopefully have about 30, 50, or even 100 PDFs of articles that are relevant to your project in a folder on your computer. You do not have to read every article or source that discusses your topic.

At some point, reading another article won’t add anything new to your literature review. We’ve discussed in this chapter how to read sources strategically, getting the most out of each one without wasting your time. Ultimately, the number of sources you need should be guided, at a minimum, by  what you need to provide a comprehensive review of the literature on your topic. The purpose of the literature review is to give an overview of the research relevant to your project—everything a reader would need to understand the importance, purpose, and thought behind your project—not for you to restate everything that has ever been studied about your topic.