How do you know whether you are correctly implementing an EBP?
Page 6: Monitoring Fidelity of Implementation
When educators initially implement an EBP, they often do so with fidelity. However, over time they might inadvertently modify or omit procedures or activities that make up a practice or program. This tendency, referred to as drift, typically occurs when educators believe that they have mastered the procedures and consequently stop referring to the implementation guidelines.
Research Shows
A majority of teachers had difficulty maintaining high levels of fidelity for more than two or three weeks after initiating an intervention or practice.(Sanetti, Fallon, & Collier-Meek, 2013)
For the reasons mentioned above, it is important for you to monitor your fidelity of implementation. Doing so early and frequently will create the opportunity to improve your implementation. Research indicates:
- Fidelity data should be collected frequently at first (e.g., twice per week).
- As you become more familiar with a practice or program and achieve and maintain high levels of fidelity, monitoring can occur less frequently (e.g., once per month).
As mentioned previously, fidelity is best measured through the use of observational data. One way to conduct an observation is to ask a colleague to watch you implementing the EBP. Alternately, you can video yourself implementing the EBP and review it later. In either case, the observer needs to complete the fidelity checklist, recording each procedural step to which you adhere, any steps that you miss, and the addition of a step or component. In general, objective observers are more accurate sources of fidelity data than are educators who monitor their own implementation.
Listen as Tom Kratochwill and Lisa Sanetti discuss ways in which educators can assess their fidelity of implementation.
Tom Kratochwill, PhD
Professor, Educational Psychology
Co-PI, Project PRIME
University of Wisconsin-Madison
(time: 2:17)
Lisa Sanetti, PhD
Co-PI, Project PRIME
Associate Professor, Neag School of Education
University of Connecticut
(time: 1:01)
Transcript: Tom Kratochwill, PhD
There are many different strategies that educators could use to assess the integrity of a program they are implementing. One immediate thing to consider is that, in an instructional or intervention program that is being used, has the developer or the researcher provided a protocol that can be readily used and implemented or adapted for purposes of integrity assessment. And many programs actually have integrity measures that accompany the program. Beyond that, there are many different formats of assessment that people can use. One is observation. One could have a colleague observe the educator implementing the program. That observational data could be used to actually provide feedback about implementation fidelity. Another way is that the educator might self-monitor the process of implementation. The assessment would be direct, inasmuch as they would check off the intervention component at the time the program is being implemented. So that is a very useful strategy to use, and often the reactivity of self-recording can actually promote and cause positive fidelity.
If you want to do something that’s a little bit more indirect, creating and monitoring a checklist subsequent to or at a time removed from the actual implementation could be completed. So people might complete a little checklist of the elements of the intervention program and reflect back on what they did during that process. So those are some of the more common examples of how treatment fidelity or integrity can actually be monitored, and certainly one should have as an overarching concern what is practical, what can you do within the confines of time and resources and the logistics of monitoring integrity.
Transcript: Lisa Sanetti, PhD
There are a couple of ways that educators can assess their own treatment integrity. One is to look at the intervention and come up with a task analysis, which is just a big word for saying “write down all of the list of behaviors that you have to engage in to deliver that intervention and create a checklist for yourself and at the end of every day or every few days take a look and complete that checklist and see which steps are you able to do pretty consistently and which steps are more challenging.” Once you get a sense of the pieces you are struggling to implement then you can really use those to set goals for yourself around implementing those at a higher rate the following week. Another thing is you can work with your colleagues to try to observe each other or check in with each other about how things are going. We know that being accountable to somebody else can be helpful in changing behavior. So that’s another strategy that I’ve seen some educators use pretty efficiently.
Now that you have learned how to develop a fidelity checklist, watch the movie below to learn how to collect observational data. In this simulation, you will see how fidelity data are collected on Mr. Braxton (portrayed by PALS expert Devin Kearns), a middle-school teacher implementing PALS 2–6. For illustrative purposes, the teacher in this video is not implementing with complete fidelity (time: 13:22).
Transcript: Video
Narrator: In this simulation, you will see how fidelity data are collected by a middle school teacher implementing PALS 2–6. Note that the observer has completed all the information at the top of the form, with the exception of Stop Time. Although the observer will record items as they occur, for demonstration purposes we will stop at the end of each activity and discuss the teacher’s implementation.
Mr. Braxton: Okay, everybody, it’s time for PALS. So go ahead, second readers, got to switch. Go ahead and switch now. Do that now. All right, we’ve got to make sure you get started. So, you know, you need to move to your PALS seats pretty quickly and everything. Question?
Student: Can readers stay in their seats?
Mr. Braxton: Yeah! First readers, stay in your seats. You know we always do it that way, don’t we?
Student: Yeah.
Mr. Braxton: Yeah! So it’s just like we usually do, okay?
[A student shows her bracelet to Mr. Braxton.]
Mr. Braxton: Very nice bracelet. Okay, quickly to your seats. All right. Okay, it’s time for PALS, everybody. So all right. So, first readers, I need to give you your folders.
[Mr. Braxton begins handing out folders]
Mr. Braxton: Pair A, there you are. Thank you. Pair B…wait, where’s Pair B’s folder? Um, okay, Pair B, sorry…wait, who’s Pair B? Oh, you’re pair B! Pair B, um, okay. You can get out your stuff. Don’t forget to do that, too. Pair E…where’s Pair E? Where’s your folder? I’ll give you guys yours now. Oh, thank you. Wait…you’re Pair E? E? There you go. Sorry. Uh, Pair G, and you guys are F, right? Okay, cool.
Narrator: The teacher has just finished the Introduction. This section consists of three items and should be completed in two minutes or less. The teacher introduced the PALS session, so the observer puts a checkmark by the first item. However, the teacher did not have a system in place for the students to transition to pairs quickly and quietly, nor did they have an organized system for the students to access materials. Because of this, the observer will not put checks next to the other two items under Introduction.
Mr. Braxton: All right, go ahead and get out your stuff. We’re going to start with Partner Reading like always. All right, and I’ll wait for you guys to get ready. Don’t forget to have out…oh, you guys have the book, have your books out. Don’t forget that, too. Those were in your…you know, we usually keep those in your desks. Okay, all right, you’ve got five minutes for Partner Reading. Okay, and…go.
[The students read aloud until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs.
[A student raises her hand.]
Mr. Braxton: Yeah? What’s up?
Student: Can I go to the bathroom?
Mr. Braxton: Yeah, go to the bathroom. Hurry up, hurry up, hurry up. I’ll be his partner for a little bit while you guys, while you’re in the restroom. All right, and…go. All right.
[Mr. Braxton takes a seat. The student reads.]
Mr. Braxton: You’re a second reader?
Student: Yes.
Mr. Braxton: Okay, go ahead.
[The students read until the timer sounds.]
Mr. Braxton: Okay, stop.
Narrator: For the Partner Reading activity, the teacher should adhere to four items. He is diligent about setting his timer and implements the activity for 10 minutes, making sure to instruct his students to switch jobs after 5 minutes. For this reason, he will get checks for items 4 and 6. However, because he fails to follow the script and monitor his students, he does not get checks for items 5 and 7.
Mr. Braxton: You guys, switch jobs. Okay, great work. So now we’re going to Paragraph Shrinking.
Narrator: The teacher instructs the students to switch to the Paragraph Shrinking activity. In fact, however, he should have asked them to engage in the Retell activity. Because he skips this activity altogether, items 8 through 10 do not receive checkmarks.
Mr. Braxton: Go ahead and switch materials again, and we’re going to start Paragraph Shrinking in just a second. Don’t forget to, um, start with the first reader. Make sure you have all the…first readers, make sure you have your books. Second readers, make sure to ask the questions. Ready…go.
[The students read aloud.]
Mr. Braxton: Yeah, don’t forget, you got to make sure you read a section first and then do them. Yeah. Okay, good.
[The students read.]
First student [reading]: “‘My name is Jonathan,’ said the boy. ‘We have been listening to your talk about going out.’ “
Second student: Jonathan asked Macadamia if he could go outside, too.
Mr. Braxton: Is that ten words? Is that right, do you think?
Second student [counting]: Jonathan asked Macadamia if he could go outside, too.
Mr. Braxton: So that was less than ten.
First student: Okay.
Mr. Braxton: All right, don’t forget to give her three points. Keep on going.
[The students continue reading.]
Mr. Braxton: Keep on going. You’re doing great.
First student: Okay, what’s next?
Student [reading]: “Bandit felt as if she had been summoned by the pirate emperor. This time…”
Mr. Braxton: And where’s the book supposed to be
Student: [Inaudible.]
Mr. Braxton: And on that. Okay.
[The students continue reading.]
Second student: Name the most important who or what.
First student: Melody.
Second student: Tell the most important thing about the who or what.
First student: She picked up…
Second student: Say the main idea in ten words or less.
Mr. Braxton: Melody…
Student: Melody picked up…
Mr. Braxton: Yeah, is that right? Is that the most important thing in that paragraph?
First student: Um-hum.
Mr. Braxton: Okay, she gets three points. Don’t forget to mark them.
[The students continue reading until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs. Get ready to start again. All right, and go ahead.
[The students continue reading.]
Mr. Braxton: It appears you guys…what is all this stuff? This is just junk. Let’s put this away. How come you have my sticky notes, anyway? All right, now, well, you guys lost your place in the book, too? You got it back. Okay, who’s reading? You’re reading? Oh, you’re thinking.
Student: I’m asking the questions.
Mr. Braxton: Remember you can have the book open, and you can be looking back as you’re doing it, okay? Keep on working hard.
[The students continue reading until the timer sounds.]
Mr. Braxton: Okay, stop. We didn’t switch jobs yet, did we? No, we did. Did we?
Student: Yeah.
Mr. Braxton: We already switched?
Student: I don’t know.
Student: Did we?
Mr. Braxton: I can’t remember.
Students: Yeah, we switched.
Mr. Braxton: We switched. Okay, so is it time for Prediction Relay?
Narrator: For the Paragraph Shrinking activity, the teacher should adhere to four items. Again, he is diligent about setting his timer and implements the activity for 10 minutes, making sure to instruct his students to switch jobs after 5 minutes. For this reason, he will get checks for items 11 and 13. He also walks around the classroom to monitor his students, so he receives a check for item 14. However, because he fails to follow the script, he does not receive a check for item 12.
Mr. Braxton: Okay, time for Prediction Relay, so go ahead and switch jobs, switch jobs and get ready for Prediction Relay. Okay, you have five minutes.
Student: I have to go to the bathroom
Mr. Braxton: Okay, that’s fine. I can be Tyler’s partner. All right, ready? You got it. Okay. Ready and…go. Five minutes.
[Mr. Braxton sits down with the student.]
Mr. Braxton: Okay is she the first reader?
Student: Uh, yes.
Mr. Braxton: Okay, where are you guys?
[The students continue working until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs. This is our last one, everybody, so make it good here. All right and…begin.
[The students continue working.]
Mr. Braxton: Okay, so did you make a prediction? Oh, no, you’re the second reader. Did you make a prediction?
Student: Um, not yet.
Mr. Braxton: Not yet. Okay, did you ask her what her prediction was? I forgot to tell you guys. I’m sorry. Make sure you make a prediction first. So what do you predict will happen next?
Student: Um, I think that maybe the mom will write a letter to the girl that, um, she’ll like, that she’s going to get to come home.
Mr. Braxton: Okay, does that sound like a prediction to you?
Student: Yes.
Mr. Braxton: Okay, good. So what do you say next?
[The students disagree about what comes next.]
Mr. Braxton: Girls, you can cooperate. Let’s keep working together. Did you make a prediction?
Student: Yes. She said, she said she was about to read half the page.
Mr. Braxton: About half the page. Okay.
[The timer sounds.]
Mr. Braxton: Oh, that’s time. Okay.
Narrator: For the Prediction Relay activity, the teacher should adhere to four items. Again, he implements the activity for 10 minutes, making sure to instruct his students to switch jobs after 5 minutes. For this reason, he will get checks for items 15 and 17. He also receives a check for item 18 because he monitors his students. However, he once more fails to follow the script and therefore does not receive a check for item 16.
Mr. Braxton: Okay. All right, so go ahead, put your folders and everything back in your folders and put your books back in your desk, and I’ll take your folders. And thank you. Thank you. I’m going to come around this way. May I have your folder, please? All right, so today everyone did a pretty good job. I think some pairs weren’t cooperating the best, but that’s okay. We’re going kind of fast today since we have a…we have to get to lunch. That’s great. All right, so, everyone, good job today. Thanks so much, and we will do more good work with PALS tomorrow.
Narrator: As he completes the Wrap Up, the teacher instructs his students to place their materials inside their folders, which he then collects quickly and efficiently. He completes this item in under 2 minutes and receives a check.
In terms of the quality items, the first one—how well did the teacher implement the PALS instruction activities?—the teacher should be rated one, “not well.” The reason for this is because, first of all, the teacher did not implement all of the instructional activities. Omitting an entire activity, as the teacher did with Retell, was a serious error. In a program like PALS, that is research-based, it’s important to complete all the activities and not eliminate any of them. In addition, the teacher did not follow the script. Not following the script caused the teacher to eliminate important instructions for the students. It also led to a chaotic transition that resulted in students not being on-task and as focused as they could have been.
In terms of the second item—how well did the teacher monitor students?—the teacher should probably be rated “average.” Although the teacher did not monitor during Partner Reading or during Retell, he did monitor the students for the last two activities, Paragraph Shrinking and Prediction Relay. Particularly in the case of Prediction Relay, the teacher made an attempt to help the students improve their implementation. Some of the student difficulties were caused by the teacher not introducing the activity, but the teacher made a good-faith effort to try and help the students. And for that reason the teacher should be rated “average” for monitoring.
Activity
Now that you have watched a demonstration of how to collect fidelity data, it is your turn to practice collecting this type of data. Print the PALS 2–6 Observation Form. Then watch the video below of Mr. Braxton (portrayed by Devin Kearns) implementing PALS 2–6 and record your observations on the fidelity checklist (time: 8:23).
Transcript: Video
Mr. Braxton: Okay, everybody, it’s time for PALS. Second readers, go ahead and move to your PALS positions. Do that now. All right. Okay, taking a little while. It’s all right. Okay, we’ll get started in just a minute. Pair A, first reader, thank you. And you can go ahead and get out your stuff for, um, Partner Reading. So go ahead and take out your points sheet, and your, um, and your correct question card. Pair B? Pair C, first reader? Thank you. Pair D? Thank you. Pair E? All right. Pair F? All right. And pair G? Okay, there you go. All right, so first activity today is Partner Reading; we’re going to do that first. Okay, so, first readers, get ready to read quickly and correctly. Make sure you’ve got out your materials. Don’t forget that. You’re going to read for five minutes. Second readers, you’re going to listen and correct mistakes if there are any, okay? Okay, so books right in the middle. Second readers, make sure you’ve got your point sheets and your pencils. Most pairs have their pencils out. All right, we’re going to do this for five minutes. Ready…begin.
[The students read aloud from their books.]
Mr. Braxton: Okay, I like how you’re following along when she’s reading. Excellent. Keep it up.
[The students continue to read aloud.]
Mr. Braxton: Excellent job following along. I like how hard you guys are working together. I’m going to give you three bonus points. Much better job than yesterday. Great work.
[The timer beeps.]
Mr. Braxton: Okay, stop. All right, time to switch jobs. Go ahead and switch materials. And you’re going to go back to where you started, so make sure you go back to the place where you started today. Second readers, get ready to read. Ready…begin.
[The students read aloud from their books until the timer beeps.]
Mr. Braxton: Okay, that is…whoa…that is time. Great job working hard on Paragraph, er, Partner Reading today. Okay, it’s time for Retell. Second readers, you’re going to retell everything. First readers, remember to ask them what happened first, what happened next. You have two minutes. Ready…go ahead.
[The students practice Retell until the timer beeps.]
Mr. Braxton: Okay, stop, everyone. All right, great job. Okay, time for Paragraph Shrinking. Switch materials again. Remember, first readers, you’re going to read and shrink. Second readers, remember to ask the Paragraph Shrinking questions, okay? All right, you’re going to have five minutes. Ready…begin.
[The students practice Paragraph Shrinking, with Mr. Braxton offering occasional suggestions.]
Mr. Braxton: That was less than ten words. All right, yeah, that’s good. Give her ten, give her three points and then go on to the next one. Great job asking questions. Make sure you have your card out so you don’t forget, okay? Uh-oh, my, I forgot to set my timer, guys. I think we went over time, but that’s okay, so let’s just go on. It’s time to switch jobs so, second readers, go ahead and take the books. First readers, go ahead and take the point-sheet and the question card. I noticed a couple pairs didn’t have out their question cards. I forgot to tell you guys to have them flipped over, so make sure that you have out the question card with the Paragraph Shrinking information on it, so if you forget you’ll remember what to say, okay? We’re going to do it for five minutes. I’m going to make sure I set my timer this time. Ready…begin.
[The students practice until the timer beeps.]
Mr. Braxton: Oh, that’s time. All right, it’s time for Prediction Relay. Switch jobs. I like everyone switching really quickly. Okay, it’s time for Prediction Relay. Ready…begin.
[The students practice Prediction Relay.]
Mr. Braxton [to a student]: Did you make a good prediction?
Student: Yes.
Mr. Braxton: Okay, so follow along carefully. Where’s the book supposed to be?
Student: The book’s supposed to be…
Mr. Braxton: In the middle. Perfect.
[The timer beeps.]
Mr. Braxton: Okay, that was the time. All right, time to switch jobs. Second readers, get ready. Do this quickly. All right, let me set the timer. Try not to forget again today. Ready…begin.
Mr. Braxton: For having his dad help him with his paper?
Students: Yes.
Mr. Braxton: Okay, let’s see if that happens on the next page. Go ahead. Good job working hard.
[The students switch readers and continue their practice until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs…oh, no, we…that’s it, right? We already switched jobs? Okay! Excellent job today, everyone. We are finished with PALS for today. You can put all of your materials back in your PALS folders. Put the books back in your desks. We’ll collect the PALS folders in a second. As soon as everyone’s quiet, I want to tell you guys a couple things about today. All right. Okay, again today, everyone, really good job.
After you have watched the video and recorded Mr. Braxton’s fidelity data, check your observation data using the provided answer key. For comments about Mr. Braxton’s performance listen to Devin Kearns’ audio below (time: 2:48).
Devin Kearns, PhD
Assistant Professor
University of Connecticut
Transcript: Devin Kearns, PhD
In this observation, the introduction to PALS was okay. You’ll notice that the teacher took a relativity long time to distribute the folders to the students. The folder distribution process was not as efficient as it could have been, although it was completed in the allotted amount of time. In this observation, we saw the teacher do a decent job implementing PALS. In terms of the fidelity checklist, the teacher followed the script for parts of the lessons. For Partner Reading, the teacher held the script for the beginning of the activity, but not during the transition. The teacher did not use the script for all of the remaining PALS activities. You’ll notice in this observation that the teacher repeatedly failed to use the script. That resulted in a significant lowering of the teacher’s fidelity. It’s a simple thing to fix, and if the teacher were to do that differently fidelity would definitely improve.
In addition, the teacher did not monitor students consistently, leaving out monitoring completely during the Retell activity focusing on some other tasks. You also notice that the teacher didn’t implement PALS for the correct amount of time during Paragraph Shrinking. If the teacher had also used the timer correctly during Paragraph Shrinking and had monitored the students during Retell, the fidelity would actually have been perfect. And so this demonstrates that it does not actually take that much additional effort to get your fidelity from relatively weak to relatively strong. On our fidelity checklist, you see that implementing PALS for ten minutes is one item, and instructing students to switch partners after five minutes is another item. Because of the way we structure the fidelity checklist, the teacher’s penalized twice for going over time on Paragraph Shrinking. When schools design fidelity checklists, it’s important to consider the implications of individual items for other items; in this case you can see how it created a double penalty. In terms of the quality items, for the first one—how well did the teacher implement the PALS instructional activities?—the teacher should be rated average. The teacher did implement the activities for the most part correctly, but the teacher did not consistently use the script to introduce the activities, and the teacher didn’t use a timer in the correct way, so that Paragraph Shrinking did not take the correct amount of time. So we should consider this an average implementation. In terms of student monitoring, the second item, the teacher should also be considered average. There was feedback given to some of the students, but monitoring was not done consistently, and the quality of the feedback could have been better. The teacher didn’t always provide clear instructional feedback to help the students improve their future implementation of the program, and [was] sometimes quick and off-the-cuff. The teacher also did not always use the orange Watching PALS Pairs checklist, which is a helpful tool for monitoring the students. It allows the teacher to decide what sort of things should be focused on.