How can education professionals identify and select evidence-based practices or programs?
Page 3: Considerations When Identifying or Selecting an EBP
Now that you know what an EBP is and how it could benefit you and your students, you are ready to identify and select one to use in your classroom. To begin the identification and selection process, you need to know what type of practice or program you are looking for (e.g., one that addresses social-emotional development, behavior, reading skills). More specifically, you need to identify the skill or behavior you want to address (e.g., overall reading achievement, reading fluency). Once you have identified the skill or behavior you want to address, you must choose a practice or program that is right for you. You need to consider:
- Students and setting
- Resources
- Evidence level
Students and Setting
You need to take into account the unique characteristics of the children and families or students you are working with and the setting in which the practice or program will be implemented. For example, a third-grade teacher might be looking for a math or reading program for her grade level. On the other hand, a high school social studies teacher might be looking for a practice that will improve his students’ reading comprehension. The closer you can match a practice or program to your students’ needs, the greater the possibility it will lead to the desired outcome. Below are some questions to consider when trying to identify an EBP.
- Does the practice or program address skills I’m interested in? Are the outcomes the same ones I am interested in?
- Are the research study participants comparable to my students (e.g., age, ethnicity, socio-economic status, demographic location)? Were subgroups, such as children with disabilities and English language learners (ELLs) included in the sample and part of the analysis?
- Is the research setting(s) similar to my instructional setting?
- Is the practice or program aligned with my state and district standards?
- Were conditions in the research similar to the ones in my setting? For instance, a study evaluating the effectiveness of a reading intervention conducted in classrooms using a collaborative teaching model may or may not reflect the conditions and resources that exist in your school.
It may not be possible to find a practice or program that exactly matches your student and setting characteristics. If this is the case, you should identify a practice or program that matches as many characteristics as possible. For example, if the goal is to increase reading achievement scores among elementary ELLs, you should identify a practice or program shown to be effective with these students. However, if you cannot identify a relevant EBP, you might select a reading practice shown to be effective with typically developing elementary students as well as those with disabilities, but for which research has not been conducted yet with ELLs.
Resources
For Your Information
When identifying and selecting an EBP, educators should consider collaborating or seeking the support of their peers, such as another teacher or a grade-level planning team. Support could include helping you research an EBP or sharing the costs associated with an EBP.
Implementing any new practice or program requires resources: time, costs, and training resources. As you might expect, some require more, some less. Below are some guiding questions, developed by the U.S. Department of Education, relevant to figuring out what resources are required to implement an EBP.
- How much time will it take to implement? Will I have to adjust my schedule?
- Are there costs associated with the practice or program?
- What is included with the program?
- Teacher and student materials?
- Training materials?
- Other resources (e.g., assessments, fidelity checklist)?
x
fidelity checklist
A checklist that can be used to accurately and systematically collect data on how well an individual is implementing a practice or program; can be used either as a self-report measure or by an observer.
- What kind of training is required to implement the practice or program?
Evidence Level
For Your Information
Just because a practice or program is deemed evidence-based does not guarantee that it will be effective for every student.
For many skills and behaviors, no EBP has yet been identified. In these cases, you need to choose a practice or program with the strongest available evidence. Because reviewing the research literature is a difficult and time-intensive task, a good way to investigate the evidence level of practices or programs is to consult a number of “trusted sources.” These organizations and agencies conduct a literature review and either summarize the research or indicate the level of evidence that exists for a given practice or program. Each organization has its own rating system, so you need to become familiar with the way each of them evaluates the available research (e.g., What Works Clearinghouse rates programs or practices with strong evidence as “++”, whereas Best Evidence Encyclopedia rates them with a completely filled in hexagon 
).
Listen as Larry Wexler and Tom Kratochwill raise additional considerations related to matching student characteristics, setting, resources, and level of evidence of a practice or program to your needs.
Larry Wexler, PhD
Director, Research to Practice Division
Office of Special Education Programs
US Department of Education
(time: 2:55)
Tom Kratochwill, PhD
Professor, Educational Psychology
Co-PI, Project PRIME
University of Wisconsin-Madison
(time: 1:50)
Transcript: Larry Wexler, EdD
If you’re choosing a curriculum or an intervention, you want to look at a number of things. One is the context or the set of circumstances, including place in which the curriculum or intervention will be implemented. And you might ask questions, such as how widely used is this curriculum? Was it developed to address issues similar to mine? For which grades or ages or sub-groups, including children with disability, is the curriculum or intervention appropriate?
If the curriculum is for learning-disabled children, and you decide that you want to implement it for children with visual impairments then you’re implementing something that it wasn’t designed to do. You might ask, is the curriculum or intervention aligned with state and district standards? Are you reaching too far? Is it too intense? Does it go beyond the resources that you have? The second question you’d asked is, does the evidence exist to support the efficacy of the curriculum or intervention? Were conditions in the study similar to the ones in my setting? So if the study that supported the development of this curriculum was done with groups of thirty and you want to do it with a group of a 100, are you likely to get the same results? If the study was done only in one type of environment and your environment’s completely different, you would have to question whether it will be effective and is it the right one to purchase. Do studies show that this curriculum or intervention has a positive impact on student outcomes? And are those outcomes the same outcomes that you’d want impacted by an intervention under consideration?
There’s a lot of great programs out there that don’t necessarily do what you need to have done. And another question you might ask is, does the curriculum or intervention show consistent success over a period of time? You want more than just a slice, more than a snapshot that it was implemented with a small group for a short period of time, and that that’s the basis of the evidence. Other questions you would probably ask are, how much does it cost? Do you have the support to do that? If it’s a program that requires a one-to-three ratio and you have a one-to-ten ratio, is it the right one for you to get? What kind of equipment or software are included, as opposed to you having to purchase? How in-depth is the training required to implement and how expensive is that? And will the developer of the curriculum offer some kind of extended support?
Transcript: Tom Kratochwill, PhD
It’s an interesting and, actually, a much more complex process than I think we have all realized in the past. What is the particular student need or issue that needs to be addressed? In other words, what are the characteristics that are of concern? Are you trying to improve student behavior, student skills? Are you trying to reduce particular difficulties or issues that the student is experiencing? So really understanding what the target of concern is is a first, primary issue in thinking about identification of an evidence-based practice. The intervention really should match the student issue or characteristics that you’re trying to address. In addition, I think it’s always helpful for teachers to think about comparisons among different interventions. In other words, there may be several different interventions that are available for a particular problem. And so thinking about what comparison options are available is critical. And the critical elements of that would include things like cost, efficiency, logistics, support to implement the intervention, etc., etc. Another really important issue is what outcome is desired when you are identifying and selecting an evidence-based practice. In other words, there has to be a good match between the student characteristics and the criteria for selecting an intervention and what your outcomes going to focus on.
Activity
Ms. McAdory, a fourth-grade general education teacher in a suburban school setting, is concerned about the reading comprehension skills of a number of her students, including some with disabilities and English language learners. She wants to find an EBP to address this issue. Ms. McAdory visits the websites of several trusted organizations and records her findings on a comparison worksheet.
- Click here to review her completed worksheet. To assist you with this activity, a column containing information about Ms. McAdory’s students and resources has been included on this worksheet.
- Based on the information recorded on the worksheet, which program do you think Ms. McAdory should select? Justify your answer.
Bryan Cook shares his thoughts on what practice Ms. McAdory should consider. (time: 1:56).
Bryan Cook, PhD
Professor, Special Education
University of Hawai’i at Mānoa
Transcript: Bryan Cook, PhD
Sometimes, when you’re identifying or prioritizing an evidence-based practice, there’s going to be only one evidence-based practice out there that is a good match with your learners and your resources. Sometimes there really will be just one clear answer. In a lot of cases such as this one, I don’t think there is one right answer. There aren’t clear guidelines here about what aspects of the decision to prioritize over others. So is “matched with student characteristics” more important than “matched with available resources”? In some level, if you don’t have the resources to participate in the training or to buy the curriculum or the products, you absolutely can’t do it, so that kind of rules it out. Similarly, the level of evidence is really important even if something is a very good fit. If there’s not evidence that it works well, I wouldn’t select it.
And so, ultimately, I think it really depends, because we don’t know the specific of Miss McAdory’s situation. Does she have unlimited resources, which then opens up some different practices that may otherwise not be available to many teachers? What exactly are the characteristics and needs of her student, and how well does that match up? So some of the practices may be ruled out, depending on the availability of resources to Miss McAdory. I’d really look at either PALS or CIRC as prioritized EBPs, because of their higher rating from organizations like the What Works Clearinghouse in terms of their effectiveness. So I would probably look at those two practices, and once I knew more about Miss McAdory’s classroom, her students, her resources, try to find which is the best fit for her.
The next several pages present lists of “trusted sources” for various age groups: birth to three, three to five, and five to twenty-one. Proceed to the page(s) that pertain to the age group(s) with which you will be working.
