How can education professionals identify and select evidence-based practices or programs?
Page 7: No Suitable EBP: Now What?
If you have searched the list(s) of recommended sources for an EBP and have not found one that meets your criteria (i.e., has a sufficient evidence level, matches your student population and setting, and can be implemented with your available resources) what do you do next? There are several options:
- Select the practice or program that best meets your criteria. In this case, be aware that you might not get the same results as the studies in which the EBP was validated.
- Review professional organizations and federally funded centers and agencies for discipline- or disability specific resources about practices or programs.
- Review the research literature yourself.
Many of the recommended sites in this module use a systematic process to evaluate a body of research and to rate a practice or program. However, there may be effective practices or programs that are not included on these sites. Of course, not all available practices and programs have been evaluated by the sites overviewed in this module. Some practices and programs have demonstrated positive outcomes, though the body of supporting research is not yet sufficient for them to be considered EBPs. This could be due to issues such as the type of research design, lack of replication, or small number of participants. In other cases, the practice or program has not been formally reviewed at all by a reputable organization. Therefore, a review of the research literature might help you to locate potentially effective practices or programs.
Review the Research Literature
If you cannot find a practice or program that has been deemed an EBP, one option is to review the research and identify a practice or program that has the potential to improve the academic or behavioral outcomes of your students. This initially can be intimidating: Teachers might not know how or where to start, might worry about the time it takes, or might have difficulty understanding the research. However, the suggestions below can simplify this task and help to guide you through this process.
Perform a Literature Search
Once you have identified the skill or behavior you want to address (e.g., reading fluency, social skills), you need to locate research articles. Some sources to search are:
- Google scholar (a search engine that searches for scholarly articles)
- A library database (e.g., ProQuest)
- Professional journals
- Federally funded centers and agencies (e.g., U.S. Department of Education, Center for Disease Control)
For Your Information
- Research articles in peer-reviewed professional journals have been through a review process that helps to ensure that the study’s methods followed good scientific procedures.
- Some helpful terms to use when beginning your search include meta-analysis, research synthesis, and review. These terms will often yield articles that review the existing research on a given topic.
peer-reviewed professional journal
A journal that publishes articles that have been through a process of review by experts in the field, who also make recommendations regarding possible revisions and whether or not the article is worthy of publication.
meta-analysis
A method of reviewing research on a given practice or program in which a systematic and reproducible literature search is conducted, specific criteria are used for including research studies in the analysis, and the combined statistical results of these studies yield an effect size for the practice or program across the studies reviewed.
research synthesis
A method of reviewing research on a given practice or program in which a systematic and reproducible literature search is conducted and specific criteria are used for including research studies in the analysis. This method of reviewing research studies does not calculate an effect size.
review
A summary of research literature for a given practice or program. This typically involves reviewing several studies to summarize the findings for that practice or program.
Identify Potential Practices or Programs
Reviewing research articles can be overwhelming. The information in the table below describes the content found in different sections of a research article and can help you identify information related to choosing a practice or program.
| Research Article Section | Information Found in the Section | What To Look For | ||||||||||||||||||||
| Abstract | A brief summary of the research design, participants, and outcomes |
|
||||||||||||||||||||
| Introduction | A literature review that can help you identify other studies that have been conducted for a given practice or program |
|
||||||||||||||||||||
| Methods | An in-depth description of the study’s design, setting, participants, and implementation procedures |
|
||||||||||||||||||||
| Results |
A description of the study findings Note: Though this section might be difficult for novice readers, the information is often summarized in a non-statistical fashion in the Discussion section. |
|
||||||||||||||||||||
|
x
effect size A statistical calculation that, when used in intervention studies, measures the effectiveness of a practice or program. Below are general guidelines for interpreting effect sizes in group-design intervention studies.
Adapted from “A Teacher’s Guide to Meta-Analysis” by D. R. Banda and W. J. Therrien, 2008, TEACHING Exceptional Children, 41(2), 66–71, and J. W. Lloyd, S. R. Forness, and K. A. Kavale, 1998, “Some Methods Are More Effective than Others,” Intervention in School and Clinic 33(4), 195–200. x
statistically significant effect An indication that a practice or program is effective to some degree and that the positive student outcomes are not solely due to chance. However, to determine how effective a practice or program is, one also needs to know the effect size. x
Percentage of non-overlapping data (PND) One common method of calculating the effectiveness of a practice or program using the results of single-case design studies. Below are general guidelines for interpreting a PND.
Adapted from “A Teacher’s Guide to Meta-Analysis” by D. R. Banda and W. J. Therrien, 2008, TEACHING Exceptional Children, 41(2), 66–71. x
single-case design studies Research studies in which an individual or small group serves as its own comparison across conditions (e.g., baseline, intervention). |
||||||||||||||||||||||
| Discussion | A summary of the procedures and the outcomes |
|
||||||||||||||||||||
For Your Information
When researching a practice or program on the Internet or on a publisher’s website, be cautious of their claims. Sometimes practices or programs are described as evidence–based even when there is no strong research evidence to support the claim.
It is often beneficial to group articles for a given practice or program. By reviewing multiple studies, you can get a better sense of how effective it is. Click here for table to help you determine the level of evidence for a given practice or program.
You might find that you are interested in more than one potential practice or program. When this occurs, click here for a worksheet to help you compare them and make an informed decision about which best meets your criteria.
If you decide to use a given practice or program, you might need to further explore implementation by obtaining and reading a manual, visiting the publisher’s website, or attending a training.
Level of Evidence
Use the information below to help determine the level of evidence that supports a given practice or program. This information can usually be found in the “Methods” and “Discussion” sections of a research article.
| Level of Evidence | Criteria |
| Strong Evidence |
Randomized controlled trial (RCT) studies
|
|
Single-case design studies
|
|
| Possible Evidence |
Quasi-experimental studies
|
|
Single-case design studies
|
|
| No Evidence |
Expert opinion articles
|
|
Practitioner article
|
For a more thorough description of the items in this table, view Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide.
(Close this panel)
Bryan Cook and Lisa Sanetti discuss considering single-case design studies when identifying and selecting a practice or program, especially for students with disabilities.
Bryan Cook, PhD
Professor, Special Education
University of Hawai’i at Mānoa
(time: 1:50)
Lisa Sanetti, PhD
Co-PI, Project PRIME
Associate Professor, Neag School of Education
University of Connecticut
(time: 3:01)
Transcript: Bryan Cook, PhD
Single-case designs can be a great source for determining the effectiveness of a practice or identifying evidence-based practices, especially for learners with disabilities. Because for many low-incidence populations where there’s just not a lot of individuals with the particular disability or condition in a school setting or in a district or in an area, it’s very difficult if not impossible to do a group research study.
A well-designed, single-case design study establishes reasonably whether the practice causes changes in student outcomes, so they’re very useful, and it’s very appropriate to use single-case designs to determine the evidence-base of a practice and identify evidence-based practices Typically, it’s a small number of individual learners that are involved in single-case studies. And so, as opposed to a group design study where you might see, say, 50 participants in a study, you may see only three or four individuals in a single-case design study. It’s especially important when you’re looking at single-case designs to look at findings across multiple studies and not just say, “I found one study where it works,” because oftentimes that’s only for a few individuals that it worked for. And to really have confidence that a practice works, we want to see it work across multiple studies for a relatively large number of individuals to really have confidence in it. Typically, we look for at least five single-case design studies that support the practice and that the practice is effective for at least 20 individuals across those studies.
Transcript: Lisa Sanetti, PhD
Well, I think that there’s really quite a few benefits to using evidence-based practices, and the first of which is just that if we’re going to expend a lot of time and energy to try to improve outcomes for the students in our schools, we want to be able to use things that we know are more likely to be effective. We know that evidence-based practices have been rigorously evaluated and are known to be effective with certain populations of students. And so we’re more likely to get improved student outcomes if we implement those evidence-based practices with fidelity. So if you’re trying to find an evidence-based practice to help a particular student or group of students, and you come across single-case design research, that can be a great way to look to see whether a practice or a program may help the students that you’re interested in helping. So you certainly want to be sure that you’re looking at the methods of the study to see if the target problem is very similar to that that you’re trying to address, and also to look and see if the setting and the students are very similar to those that you’re trying to work with and if you would have the resources to implement the intervention that’s described. After looking at that, if you think that it looks like a fairly good alignment with your setting and students and target problem then certainly you want to go ahead and read through the rest of the study, look at those results, see how much progress the students made over what course of time. It gives you an idea of what rate of progress you might hope to expect, and many times you might be able to implement the intervention based on what the information that’s available in the method’s section.
Sometimes there’s just not enough information, and I really think that educators should feel free to email the lead author. Their contact information is always on the front page of the journal article. So email them and ask for more information. Researchers are generally always more than willing to share the methods of effective interventions with practitioners. Generally, we certainly want to see something that’s worked across multiple groups of students, ideally across multiple research groups. But that may not always be possible to find, or you might not have access to those articles. So if you can find that multiple articles have been published and show favorable results, that’s certainly better than an intervention that only has one evaluation with favorable results. So you can feel more confident that it’s likely to work with your students. But if there’s only one intervention that’s sort of aligned with the target issue that seems like something you’d be able to do then again contacting that researcher and asking if they do have other data to support its effectiveness can be really beneficial. There might be other studies out there that you’re just not finding or you might not have access to, and so that can provide some helpful information to you as well.
