How do you know whether an EBP is effective with your children or students?
Page 8: Evaluating the Relation Between Outcomes and Fidelity
Now that you have both progress monitoring data and fidelity data, you can judge the effectiveness of an EBP for your children or students. You can do so by examining the relation between the two sets of data.
- If fidelity is high, increases in performance can be attributed to the evidence-based practice or program.
- If fidelity is high and there is no change in performance, it can be inferred that the practice or program was not effective with your children or students.
- If fidelity is low, increases in performance cannot be not necessarily be attributed to the practice or program.
- If fidelity is low and there is no change in performance, it is unclear whether the practice or program would lead to improved outcomes if implemented correctly.
The table below describes the instructional decision for each case.
| Improved Child/Student Outcomes (i.e., above or on the goal line) |
Inadequate Child/Student Outcomes (i.e., below the goal line) |
|
| High Fidelity | Continue using the EBP and continue monitoring performance | Change instruction because the EBP is not effective for your children or students |
| Low Fidelity | Decision is unclear: 1) Continue using the EBP and monitoring performance OR 2) Improve implementation fidelity and collect progress monitoring data to see whether outcomes further improve. | Improve implementation fidelity and collect more progress monitoring data to see whether outcomes improve. |
As the table above highlights, when fidelity is low, the effect of an EBP on child or student outcomes cannot be determined with confidence because:
- It is difficult to identify which aspects of the practice or program are leading to positive outcomes or whether something else is responsible for the positive outcomes.
- It is uncertain whether positive outcomes might further improve if the practice or program were implemented as intended.
- It is uncertain whether poor outcomes are due to low fidelity or because the EBP is not effective for given children or students.
By examining both progress monitoring and fidelity data, you can judge whether an EBP is effective for your children or students. Monitoring only one component will provide only partial data. Without monitoring both components, you will not be able to determine whether the EBP is effective for your children or students and subsequently to make informed intervention or instructional decisions.
Listen as Tom Kratochwill elaborates on how the combination of progress monitoring data and fidelity data can inform instructional decisions. Next, Lisa Sanetti explains why educators should be concerned about low fidelity even when learner outcomes are improving.
Tom Kratochwill, PhD
Professor, Educational Psychology
Co-PI, Project PRIME
University of Wisconsin-Madison
(time: 2:40)
Lisa Sanetti, PhD
Co-PI, Project PRIME
Associate Professor, Neag School of Education
University of Connecticut
(time: 1:08)
Transcript: Tom Kratochwill, PhD
If you have really high treatment fidelity or integrity and the student is doing really terrific then there is some cause for continuing the program and the belief that it is contributing to the kind of positive outcomes that you’re observing in the student. It’s also possible that you could have a program with very high fidelity, but the student is not making progress on the measures that you have selected, and that could mean one of several things. One is that perhaps the program, even though it’s being implemented with high fidelity, is not particularly well matched to the nature of the problem. That is, there are perhaps elements of the child’s problem that have not been considered. Perhaps there are certain measures that are being used to assess the outcome that are not capturing the kinds of outcomes you’d like to observe. So that is a possible perplexing problem that the educator has to consider, but it could very well result in changing the intervention program in some way or the instructional procedures to make it more potent or stronger.
If low fidelity is there, you’re either seeing or not seeing change in a child. It really means that you need to focus on addressing the low fidelity, because low fidelity of the program means you have to engage in a problem-solving process to determine whether you can improve the fidelity and then make a decision about whether the child improves or is not improving. If the child improves then, obviously, you’re making the kinds of decisions that will result in positive student outcomes.
But if the child after you improved the fidelity of the program is not improving then you may have to consider the nature of the program, the context of implementation, the types of measures that are being used, possibly adding another program, etc. So the issue of low fidelity causes more complexity actually in decision-making and the problem-solving process. You have a number of different forks in the road to consider under conditions of low fidelity, which by the way attest to the need for implementing programs with high fidelity.
Transcript: Lisa Sanetti, PhD
Sometimes we’ll find that teachers are implementing interventions and students are improving, but really their fidelity data don’t show that they’re implementing at a very high level. We can see positive student outcomes with lower levels of implementation with really strong and robust interventions. And so it’s not unexpected to have a teacher say, “Why should I bother implementing at a higher level if I’m getting good outcomes?” The reason is that we have emerging data to show that higher levels of implementation result in more efficient outcomes and in better outcomes and higher rates of outcomes for students. And so any increases we can get in fidelity are going to result in potentially having to implement the intervention for less time, getting students to close the gap in a shorter period of time. So any increases in implementation will only result in better student outcomes in the end.
