How can school personnel determine that they have effectively implemented evidence-based practices or programs?
Page 11: Promote Sustainability
Having implemented a new practice or program, school staff may find it difficult to maintain its use over time. In such instances, school staff may continue using the practice or program but do so with low fidelity, or they may discontinue the practice or program altogether. In general, the inability to maintain a practice or program often results from changes in staff, time constraints, lack of funding, insufficient leadership, an emphasis on high-stakes testing, and resistance to change. Therefore, school personnel need to plan for sustainability—the use of a classroom or school-wide practice or program with fidelity over time.
sustainability requires constant vigilance and continued attention to system supports, starting with the exploration stage and continuing through the life of the innovation.
Listen as each panelist below discusses a factor that may influence whether a practice or program is sustained over time.
Expert Panel
Daryl Mellard, PhD
Co-Principal Investigator,
National Center on
Response to Intervention
(time: 1:20)
Transcript: Joseph Torgesen, PhD
Implementing an evidence-based reading program requires a degree of self-discipline and attention to details that many teachers experience as really hard work that even might go against their natural grain of wanting to have fun with the kids or go off and do some other kinds of activities. They have this program that’s requiring them to do things that they experience as kind of hard to do. Unless they have strong leadership in which this is encouraged, then oftentimes it doesn’t get sustained. When you’re demanding new behaviors or difficult behaviors from people, you’ve got to provide leadership. You’ve got to reward people to do it well. You’ve got to make adjustments for people who don’t do it too well. And so I think, often, high-quality instruction is not sustained in schools because of a lack of effective leadership on the part of the principal or the reading coach or the assistant principal, who’s ever sort of the instructional leader in the school. There’s also the idea that no one program’s going to work for every child, and oftentimes schools are looking for the magic bullet, and so they’ll switch away from an effective program even if it’s working well in hopes that they’ll find a magic one that will solve the problems for everybody. I think that problem is less common than the one that I mention first, which high-fidelity implementation of a complex reading program is not easy, particularly when you are working with students who struggle in schools that have a lot of high-risk kids who need lots of review, and careful monitoring of their reading growth, that’s hard work. And unless we have strong leadership, unless we get rewards for this hard work, oftentimes it kind of goes away.
Transcript: Lynn Fuchs, PhD
An important question is why implementation of evidence-based practices sometimes is not sustained. And in my experience it has a lot to do with the ongoing, changing culture in school buildings. So I’ve seen schools that are using evidence-based practices very well, and often the motivation or the impetus for the school’s use of that practice is either a principal who’s encouraging his or her teachers to use the practice, or sometimes even a single teacher in that building who has become convinced over time about the value of that practice and has a forceful positive personality and brings the school on board so you get a schoolwide implementation of a practice that results in dramatically improved student outcomes. And you would think that the clear association between the use of that practice and the better student outcomes would be a very strong motivation for schools to continue to use the practice. But sometimes the principal leaves; a new principal comes in, has a different agenda. Sometimes that leadership teacher leaves or has personal issues that force her to not have as big a role in a school, and as the leadership for the use of the practice wanes there’s always coming and going of a school body staff. So it may be that there are no routine opportunities for new teachers to become trained in the practice that at one time was universally used in that school. And it just may be that the incentives and motivators for what practices a school is being encouraged to use is changing. I think that a lot of the times when you’ve got a strong implementation of a good practice that’s clearly resulting in improved student outcomes, when that begins to fail, the lack of sustainability is often due to leadership issues.
Transcript: Daryl Mellard, PhD
We need to think of sustainability as a multi-pronged effort. There’s no one magic tool. When we want to sustain a practice, we know we have to make it a priority, be sure that everyone realizes this is a long-run solution, not just a practice that we’re going to implement today. We’ll make sure that we look at our agenda for professional development. I think too often we rely on the awareness level training and assume, well, you’re an experienced teacher, you won’t have any trouble picking this up. I think it’s pretty clear that even something as basic as administering a standardized assessment can prove to be problematic for folks experienced in assessments. So continued efforts along the line of professional development can help us sustain the practice and help folks develop a deeper understanding of the practice. Let’s see this as a gradual improvement, and look at our observation systems, look at our evaluation systems, look at our coaching, and see how those also can support sustaining particular practices.
Research has shown that even when teachers see a practice or program producing positive outcomes for their students, this is, surprisingly, not enough to sustain its use. More effort is needed to maintain the use of a practice or program. When planning to implement a new practice or program, school staff need to consider all of the factors associated with its continued use. Click on each tab below to learn more about some of the factors that help to sustain a practice or program.
Classroom feasibility
School personnel are more likely to sustain the use of a practice or program if it is feasible—in other words, it:
- Is defined by specific guidelines
- Can be integrated with other teaching practices
- Can be adapted to fit specific classroom needs
- Aligns with state standards
Ongoing support
School personnel are more likely to sustain the use of a practice or program if:
- It promotes mastery of the practice or program over time
- Discussions with colleagues provide opportunities to problem solve implementation issues as well as opportunities for support and feedback
- Administrators or implementation coordinators support and promote continued implementation
Materials availability
School personnel are more likely to sustain the use of a practice or program if:
- All materials needed to effectively implement it are readily available
Required change
School personnel are more likely to sustain the use of a practice or program if it:
- Augments their skills—does not require them to learn a completely new set of skills
- Engages their interest—does not overwhelm them with minor details
- Addresses the needs of a wide range of students—does not target a small number of students
Visible benefits
School personnel are more likely to sustain the use of a practice or program if they:
- Link improvements in student performance to the practice or program
- Observe improvements for students with and without disabilities
Student enjoyment
School personnel are more likely to sustain the use of a practice or program if:
- The students find the practice or program engaging
Conceptual knowledge
School personnel are more likely to sustain the use of a practice or program if they can:
- Learn the underlying theory or principles behind the practice or program
- Understand how the practice or program is affecting student performance
Although researchers and program developers agree that it is important for school personnel to implement a program with a high degree of fidelity, research indicates that the ability to adapt a program increases its sustainability. Some researchers believe that adapting programs can be beneficial, perhaps even essential—schools need to have the ability to adapt the program to better meet the needs of their students and to align with available resources. However, even those who advocate for adaptations believe that teachers should adhere to the critical program components, only making changes to its nonessential parts. If staff alter or stray too much from the implementation procedures used in the validation research, their changes will likely affect diminish the approach’s effectiveness and may consequently cause the approach to fail. In some cases, programs that have been adapted have produced positive student outcomes, while in other cases students have not shown progress.
Listen to each panelist below discuss making adaptations to a practice or program. More specifically, Joseph Torgesen and George Sugai describe situations for which it is acceptable to make adaptations, while Daryl Mellard gives an example of how an adaptation can have adverse effects on student outcomes.
Expert Panel
George Sugai, PhD
Center on Positive Behavioral
Interventions & Supports
(time: 0:47)
Daryl Mellard, PhD
Co-Principal Investigator,
National Center on
Response to Intervention
(time: 0:35)
Transcript: Joseph Torgesen, PhD
If your goal is to implement research-based instruction, I think it’s important to understand that there needs to be a sensible balance between needs for fidelity and the need to respond to the needs of the growth rate of individual students, specifically with regard to pacing. Some programs have a kind of pacing guide that say you ought to spend a week on this and a week on this and a week on this. And they do that with the average student in mind, but in some case a teacher might have a group of students that she’s working with that learn extraordinarily fast or that learn extraordinarily slowly, and that just would not make any sense to sort of slavishly follow that pacing guide in a way that would go too fast for students or go way too slow for them. So it’s really important to have a sensible approach to this. Do not insist that every class learn at the same rate or every student learn at the same rate. It’s really important for leaders and for teachers to understand the basic principles that are implemented in a program and then be faithful to those in a way that also allows teachers to make adjustments for individual students. But, again, you might go too far. Or the teacher might exercise poor judgment. That’s why we need to have checks and balances. We need to have a leader or a coach that observes, because some teachers are really skillful at making adjustments and other teachers might think they need to make adjustments when really they don’t. And so this is an element of teacher skill that we can’t always be sure is present in the classroom. It’s just one of those things that a good leader needs to be aware of. But a good leader needs to be aware that his teachers should be making some adjustments.
Transcript: George Sugai, PhD
Implementation of a practice and the adaptations of those can be acceptable under certain kind of conditions. It’s important to stay with the expectations or requirements of the implementation, because that’s how we’ve demonstrated, through the research anyway, that we can produce the outcomes that we want. However, we’ve also learned that adaptations can be considered based on a number of reasons; most of them are related to the students who are experiencing the practice or the intervention. We can imagine a kid who’s got this evidence-based reading program that they’re experiencing but not having the social skills to be able to engage in that, and maybe there are some modifications that are required to ensure that the kid can benefit from that instruction. Adaptations are only as good or bad as our ability to produce those outcomes.
Transcript: Daryl Mellard, PhD
Let’s just pretend we’ve got four students in the group, and someone says, well, let’s add a fifth student. Well, okay, not so bad, unless the addition of that fifth student is so disruptive that the adherence drops. If managing behavior within that small group becomes a major issue, you don’t have the same treatment, and it isn’t just the difference of adding one more student. The intensity of the teaching moment is reduced, as well.
For Your Information
The process of implementing a new program or practice is recursive. Depending on the complexity of the program or practice, the implementation process may take 2–4 years to complete in most schools. Achieving the sustained use of a practice or program may take several more years.
Click the school logos below to learn how each school develops a system to monitor fidelity of implementation.
|
|
|
Promote Sustainability
Paige Elementary School
Because district personnel know that RTI is a large undertaking, they have given much thought to issues related to sustainability. As each school in the district begins implementing components of RTI, such as progress monitoring (PM), district personnel plan to provide the necessary supports to make sure the teachers are successful. District personnel discuss the following factors with the implementation team, with an emphasis on how to address each to maintain the use of PM at Paige.
Factor |
Strategies to address sustainability at Paige |
||||||
| Classroom feasibility |
|
||||||
| Ongoing support |
|
||||||
| Materials availability |
|
||||||
| Extent of change |
|
||||||
| Visible benefits for students |
|
||||||
| Student enjoyment |
|
||||||
| Conceptual understanding |
|
(Close this panel)
Promote Sustainability
Grafton Middle School
The implementation team at Grafton Middle is aware of the need to plan for the sustainability of the reading program at their school. They think they are off to a good start by having strong PD training, ongoing support, and available materials. They also plan to train new reading teachers to make sure everyone uses the evidence-based program. The table below lists factors that support the sustained use of a new program and the ways in which the Grafton Middle team is addressing them.
Factor |
Addressed by Reading Program |
Addressed by Grafton Middle |
| Feasibility in the classroom | Aligns with state standards; Easily implemented in the classroom setting | N/A |
| Ongoing support | Publisher provides ongoing support via Webinars and unlimited phone support | Grade-level lead teachers serve as coaches |
| Availability of materials | Includes all needed materials | Purchase materials as needed (e.g., student workbooks each year) |
| Extent of change | Very little; meshes with current instructional practices | N/A |
| Visible benefits for students | Manual provides research evidence for positive student effects | Track students’ reading performance and note improvements |
| Student enjoyment | N/A | Assess students’ perceptions of the reading program |
| Conceptual understanding | The PD and ongoing support helps teachers gain the conceptual understanding of how the program works | Provide coaching to help teachers gain a conceptual understanding of how the program works |
(Close this panel)
Promote Sustainability
DuBois High School
As the implementation team at DuBois learns more about PBIS, they discover that many of the factors they typically consider when thinking about how to sustain a practice or program are already addressed by the approach.
Factor |
Addressed by PBIS Approach |
||||||
| Feasibility in the classroom |
|
||||||
| Ongoing support |
|
||||||
| Availability of materials |
|
||||||
| Extent of change |
|
||||||
| Visible benefits for students |
|
||||||
| Student enjoyment |
|
As the team learns more about PBIS, they discover too that the PBIS Center recommends additional factors for sustaining schoolwide implementation of the approach. As the team reviews these factors, they create a list of ideas about how DuBois can address each of them.
Factors |
Ideas |
||||||
| PBIS is a priority for all school staff |
|
||||||
| The schoolwide behavior program is effective for all students |
|
||||||
| The program is efficient for school staff |
|
||||||
| The program is adaptive to change |
|
||||||
| Effective leadership |
|
||||||
| Use of data |
|
(Close this panel)
