How can school personnel determine that they have effectively implemented evidence-based practices or programs?
Page 10: Collect Data and Evaluate Fidelity of Implementation
Once the data collectors have proven themselves reliable, they can begin to collect fidelity data. They must do so in accordance with the established schedule and procedures. As was mentioned previously, school personnel should monitor fidelity and provide immediate feedback to teachers in a supportive, non-blaming manner that emphasizes problem solving. An observer should offer a teacher praise and positive reinforcement for implementing a practice or program with high fidelity. When a teacher implements a practice or program with low fidelity, the observer should not offer critical feedback (e.g., “You did it wrong.”). Rather, he or she should be supportive and provide feedback that informs the teacher about the observed lesson or instructional technique. The observer should briefly commend the teacher for what he or she did correctly and point out areas that need improvement. Instances of low fidelity and the response should be documented. This information could prove useful when examining student outcomes.
Data must be saved, preferably in a spreadsheet program, so that:
- Fidelity for each teacher/ implementer and for the program as a whole can be determined
- The relation between practice or program implementation and student outcomes can be evaluated
Once fidelity data have been collected, school personnel can use them in a number of ways. They can establish whether the teachers implemented the practice or program as it was intended, and, if not, can take steps to correct their implementation. Click here to learn how to calculate an individual’s fidelity of implementation. Because the school environment is constantly changing (e.g., through teacher turnover), fidelity checks need to become routine after a new practice or program is implemented.
Calculate an Individual’s Fidelity of Implementation
- Determine the total number of items on the fidelity measure. In the example measure below, the total number of items is four.
| Behavior Plan | Completed? | |||
| Yes | No | |||
|
✔ | |||
|
✔ | |||
|
✔ | |||
|
✔ | |||
- Count the number of items adhered to. In the example, the teacher completes three steps correctly.
- Divide the number of steps completed correctly by the total number of items. In the example, 3 is divided by 4.
- Change the decimal result in step three to a percentage.
(Close this panel)
Click on the movie below to join a classroom observer as she completes a fidelity checklist on a teacher implementing the POW writing strategy (time: 10:50).
Transcript: Fidelity Checklist
Teacher: Who can tell me what makes a good story? Will?
Will: Well, if it has a great action scene in the very middle of it.
Teacher: A great action scene. Good.
Student: A hook that could, like, get you into the book before you quit it.
Teacher: So you have kind of the reader that wants to be sucked into the story and be aware. Very good. Any other ideas on what makes a good story? Stuart?
Stuart: Make the story kind of make sense.
Teacher: The story needs to make sense. Have you ever read a story that you just read it and you said, “What was that about? I have no idea. It made no sense at all!” You want a story to make sense. Very good. Do stories have to be true?
Class: No.
Teacher: No. What kind of stories can you have? Sarah?
Sarah: Fantasy.
Teacher: Huh?
Sarah: Fantasy stories.
Teacher: Fantasy stories. Victor?
Victor: Fiction.
Teacher: Fiction. Ellie?
Ellie: Biography.
Teacher: Biography. Elias?
Elias: Um, Greek mythology.
Teacher: Greek mythology. Cameron?
Cameron: Maybe a novel?
Teacher: A novel. So, yeah, you have all different kinds of stories, and we want stories to have all their parts and make sense. We want them to be fun to read. We want to be able to hook our readers, and they need to have all their parts.
Narrator: The teacher has just completed the first item on the fidelity observation checklist. She has developed the students’ background knowledge about writing. Item 1, therefore, gets a check in the “Yes” column. The observer also needs to rate how well the teacher performed this task. The observer believes that the teacher did “very well” on this task, so she circles 3 on item 1a.
Teacher: Today, we’re gonna learn about how to use it with story writing, ok? So the “P” in POW is “pick my idea.” We need to pick an idea. What’s my story gonna be about. What can a story be about?
Narrator: The teacher has just discussed the “P,” pick a topic, step of POW. The observer places a check in the “Yes” column for item 2a.
Teacher: The second part of the trick is organize my notes. Does anybody write out notes when they make a story? You do? Do you write down all the ideas of how you’re gonna write your story? Yeah, it helps me a lot when I have to write my story, ’cause I can’t keep all my ideas in my head and sometimes I forget them, and if I don’t write them down, I won’t have them later when I go to write. So I am gonna teach you a trick on how to organize your notes for stories. Ok? And then, the third part, is write and say more. So I’m gonna take my notes, and I’m gonna write my story and I’m gonna add more to it.
Narrator: The teacher has just discussed steps “O,” organize my thoughts, and “W,” write and say more. The observer places a checkmark in the “Yes” column for items 2b. and 2c. Because the observer thinks the teacher did an average job of pointing out the benefits of the POW strategy, she circles 2 for item 2d.
Teacher: Okay, so today I’m going to share with you how I write, ’cause sometimes I have trouble writing. And I’m going to show you how I use my tricks. So I have this picture here, and now I have to think, “Hhhhmmm. What am I going to write a story about? What do I do first? This is hard. I know! I’ll use POW.” So first I have to…What’s “P”? I know: “Pick my idea.” I’m going to pick my idea. Okay, so he’s looking at a magnifying glass. It’s kind of a silly picture. I really don’t like it. Okay, keep my mind free, and I can think of a good idea. I think my idea’s going to be…he’s going on a big adventure. He’s searching for a treasure. That’s what my idea is going to be. So I’m doing POW, okay? So I picked my idea. He’s going on a treasure hunt. All right, that’s done.
Narrator: The teacher is now modeling the POW strategy. She has just completed modeling “P,” pick a topic, so the observer places a checkmark in the “Yes” column for item 3a.
Teacher: So next I have to “Organize my notes.” Writing is so tough. All right, what do I do to organize my notes? That’s right, I learned a trick. W-W-W, What = 2, How = 2. So I have this. This is my graphic organizer. This will help me remember my parts. So WWW. “Who?” Oh, I don’t know that one. I can skip that one. “When?” He looks kind of modern, but he has a canteen and a magnifying glass. I’m going to say, like, 1903. Okay. “Where?” Where would somebody go on a treasure hunt? Um, I have no idea. Who does treasure hunts anymore? Okay, keep my mind free. I can do this. Where? Ooh, the desert in Africa. That’ll be good. I like that idea! “Desert.” “Africa.” Okay. That’s it. I have all my parts. I have organized my notes. Yes…all right!
Narrator: The teacher has modeled a strategy for students to use for the “O” step–organize their thoughts. The observer places a checkmark in the “Yes” column for item 3b.
Teacher: All right! So I picked my idea, I organized my notes, and now I’m going to write and say more. So now I’m gonna use my notes, so I’m gonna say 1903, and I’m gonna check that off, ’cause I have that part done.
Narrator: The teacher has just finished modeling “W,” write and say more, so the observer places a checkmark in the “Yes” column for item 3c. The next item on the checklist asks the observer to rate how well the teacher explained the reasons for each step of POW as she was modeling them. Although the observer believes that the teacher did an excellent job of modeling the steps, she does not think that the teacher explained why the students should use each step. She circles “poorly” for item 3d.
Teacher: Okay, so you guys are sitting in pairs, and you’re going to work with your partner on memorizing the trick. Sometimes you might want to write a story at home or if you keep a journal at home, or if you go on an airplane and go far far away and decide to write a story about your trip, you’ll want to use the trick. You won’t always have the poster to help you out with the trick. So if you memorize it then you can take it with you and use it anywhere, and use it all the time. Okay, so you can turn to your partner and you have these little cards, and you can quiz each other to help you memorize the trick.
Narrator: Because the teacher has given the students an opportunity to memorize the steps of POW, the observer places a checkmark in the “Yes” column for item 4.
Teacher: We’re gonna write a story together, and I’m going to need your help. I have this picture here. So what am I going to do first? Victor?
Victor: Who’s the main character?
Teacher: Who’s the main character? That’s good, but what am I going to do before that?
Student: Pick your idea.
Teacher: Pick my idea. What’s the trick I use for all kinds of writing? POW. So I am going to pick my idea. What’s my idea going to be? Thank you. Anybody have an idea? Thomas.
Thomas: Family road trip.
Teacher: Family road trip. Ok, so I’ve picked my idea. So who’s my main character going to be on my family trip? (Children laughing.) I’m going to put “guy driving.” So do you notice I don’t write complete sentences? We wrote and said more. This is the story we wrote based off of our notes from the graphic organizer.
Narrator: The teacher has given the students an opportunity to use the steps of POW. She leads them in guided practice of using POW to write a story. The observer places a checkmark in the “Yes” column for item 5. Item 5a. asks the observer to rate how well the teacher provides corrective feedback to the students while they used POW. Because the observer does not think the teacher had the opportunity to provided corrective feedback she writes “NA” – not applicable on the observation form for item 5a.
Teacher: Okay, so you guys are going to practice working on these tricks by yourself, okay? You each have a picture on your table, and you have a graphic organizer just like the one that I’ve been using on the board. But you have a smaller one. So I want you to work on and think about using POW and W-W-W, What = 2, How = 2 to write a story about those pictures, okay? If you have any questions, raise your hands. I’ll be walking around to help you.
Narrator: The teacher has given the students an opportunity to use POW independently to write a story. The observer places a checkmark in the “Yes” column for item 6.
(Close this panel)
Click here for a copy of the POW Observation Checklist used to collect the observational data.
Activity
The third-grade teachers at Truman Elementary are now implementing PALS 2–6. The implementation team has begun to monitor the teachers’ fidelity of implementation. They observe Mr. Braxton several times, as is captured in the three videos below. For this activity, imagine that you are one of the observers monitoring Mr. Braxton’s implementation fidelity. Watch each video and complete the PALS 2–6 Observation Form to determine whether Mr. Braxton is implementing PALS 2–6 with fidelity.
To begin, click on the Observation 1 video to watch Mr. Braxton implement PALS with 100 percent fidelity. As you do so, be sure to record your observations on the PALS 2–6 Observation Form. When you are finished, listen to Devin Kearns’ commentary on Mr. Braxton’s implementation.
Transcript: Observation 1
Mr. Braxton: All right, everyone, time for PALS. Second readers, go ahead and get your folders and move. Go ahead. I like how quickly and quietly everyone is moving, getting their folders. Great job. And when you get to your seats, go ahead and get out your PALS materials. First readers, you can go ahead and get out the books, as well. Okay, everyone’s getting prepared nice and quickly. Thank you so much for your hard work. Point sheets are out. Question cards are out. Excellent job. Books right in the middle. People are turning to the correct page. You guys are already ready. Let me give you guys a couple of bonus points. That was perfect.
Mr. Braxton [cont.]: Okay, first readers, you’re going to read quickly and correctly. You’re going to read for five minutes. Second readers, remember to listen, give points, and fix mistakes, okay? Go ahead and open your folders. Get yourselves ready. First readers, open the books. Okay, you’re going to have five minutes. Ready…begin.
[The students begin reading. Mr. Braxton moves about the classroom, listening in and offering encouragement.]
Mr. Braxton: Great job. You guys are doing a great job following along while I’m marking points. Excellent work. Okay, you guys are following along so nice, I’m going to give you three bonus points. Nice work. Keep on working hard.
[The students continue reading.]
Mr. Braxton: I like how carefully you’re following along. Very nice work. Go ahead.
[The timer sounds.]
Mr. Braxton: Okay, everyone stop reading. Everyone switch jobs. Turn back to the first page you read today in Partner Reading. Second readers, get ready to read for five minutes. I’m going to watch and make sure all second readers have the books right in the middle. First readers, have your point sheets out. Everyone has the question card? Excellent work. Okay, you’re going to read for five minutes. Ready…begin.
[The students begin reading. Mr. Braxton moves about the classroom, listening in and offering encouragement.]
Mr. Braxton: Okay, so go back to what you were doing just a minute ago. Okay, go ahead.
[The timer sounds.]
Mr. Braxton: Okay, stop. Everyone, stop reading. Great job working hard on that. Now it’s time for Retell. I want you to retell what you read. First readers, get ready to ask what happened. Second readers, get ready to state what happened. You have two minutes for Retell. Go ahead.
[The students continue their work.]
Mr. Braxton: Remember, you want to make sure to ask her what happened first. What happened first?
Student: Um, Bandit was about to change her name.
Mr. Braxton: And then you ask her what…what happened next, right?
Student: Okay, what happened next?
Mr. Braxton: There you go. Keep on going. Great job. I can tell you’re following along really carefully. I’m going to give you guys a couple bonus points. Nice job.
Mr. Braxton [cont.]: Okay, stop retelling. Before we go on to Paragraph Shrinking, I just want to compliment everyone for doing such a great job with Retell today. I heard everyone doing a good job of asking the next thing that happened. I even heard a couple people correcting mistakes people made. Excellent work, everyone. All right, everyone, now it’s time for Paragraph Shrinking. You’re going to start reading where the second reader stopped during Partner Reading. First readers, I want you to get ready to read and shrink information into good main idea statements. Second readers, ask the Paragraph Shrinking questions. They’re on your question card. And make sure your partner’s answers are right. You have five minutes. Everyone switch materials. First readers, you may begin reading.
[The students read. Mr. Braxton moves about the classroom, offering guidance.]
Mr. Braxton: Okay, so I want you to have…make sure you guys have the book open, and who’s the first reader? You’re the first reader, right? Okay, so go ahead and read a paragraph and then you are going to answer the questions. I’ll listen while you guys do it, okay? Go ahead.
First student: “Mom and dad are going to be home any minute, so I thought we were doomed. But then Roderick had a genius idea. He said we could switch the door out and replace it with a closet door from the basement.”
Mr. Braxton: Okay.
Second student: Name the…
Mr. Braxton: Say “stop.”
Second student: Stop.
Mr. Braxton: Okay.
Second student: Name the what…the who or what the most important thing.
Mr. Braxton: Name the most important who or what. Go ahead.
First student: Um, Roderick and the doors.
Mr. Braxton: Do you agree?
Second student: Yes.
Mr. Braxton: Okay, ask the next question.
Second student: Tell the most important thing about the who or what.
First student: Roderick had the great idea and the doors were there because otherwise he wouldn’t have the idea.
Second student: And then say the main idea in ten words or less.
Mr. Braxton: Let’s count them. Go ahead.
First student: I think it was that…
Mr. Braxton: Roderick, because that is the most important who or what, right? Start with Roderick. Okay, go ahead.
First student: Roderick was going to…
[The timer sounds.]
Mr. Braxton: Okay, stop. It’s time to switch jobs. Second readers, it’s your turn to read and shrink information to good main idea statements. Go ahead. Switch materials. I like how quickly everyone is switching. Great work. Okay, ready…begin.
[The students continue working until the timer sounds.]
Mr. Braxton: Okay, stop. Okay, it’s time for Prediction Relay. First readers, you’re going to have five minutes to make predictions, read, and then check the predictions. Second readers, make sure you ask the Prediction Relay questions, listen carefully, and give points when they’re earned. Get ready. Don’t forget the first thing you need to do is make a prediction. Ready…begin. Switch materials and begin.
[The students continue working until the timer sounds.]
Mr. Braxton: Okay, stop. It’s time to switch jobs. Go ahead and switch your materials. Second readers, get ready to make predictions, read, and check your predictions. Don’t forget, second readers, the first thing you need to do is make a prediction. Ready…begin.
[The students continue working until the timer sounds.]
Mr. Braxton: Okay, stop. Great job, everybody. We have reached the end of PALS for today. I’d like you all to put your PALS materials back in your PALS folders. Close your books. Put them back in your desks, first readers. As soon as everyone has their PALS materials closed away then I want to talk to you a little bit about some of the things I thought went really well today when we did PALS. I like how Pair E has all their materials away. The folder should be folded up on your desk and the book should be inside. Great job. Okay, so a couple things I really noticed everyone did wonderfully today is, coaches, excellent work when you were coaching of listening carefully to your partners. I noticed every pair was listening carefully to each other, and that was excellent.
(Close this panel)
Commentary – Note: Devin Kearns plays the role of Mr. Braxton in the videos (time: 2:46).
Transcript: Devin Kearns, MA
As a reminder, this activity is a simulation. It’s important to know that before you become a real PALS observer, it’s important to be trained not only on the program itself but on how to fill out the fidelity checklists for the program. You’ve just watched Mr. Braxton doing a very good job implementing PALS. We’d like to point out that Mr. Braxton used this script really consistently. You noticed that green sheet that he had with all the PALS activities describes how to introduce each of them in really clear language. He used that not because he necessarily needs it but because having that consistent guide reminds him to do everything he’s supposed to. You also notice that he used his timer throughout the lesson, which is also really important because that way his times are consistent. And one other important aspect was he monitored the students throughout the lesson and he monitored them really well. Not only did he watch the students but he provided each pair with feedback, and he also made an effort to get to all the pairs in the class throughout the lesson. The combination of those factors makes this a really strong PALS lesson.
One thing I’d like to draw your attention to is the quality items. The first quality item—how well did the teacher implement the PALS instructional activities?—Mr. Braxton should be rated very well on this item because he implemented the activities in exactly the way they’re designed. You’ll notice that he used the script to guide him throughout—that was the green sheet that he was referring to—and he did all the activities the way that they’re designed, and had the students following the correct instructional procedures. The second quality item—how well did the teacher monitor the students?—Mr. Braxton should be considered to have done very well with that component for a number of reasons. First of all, he attempted to get to all of his pairs throughout the lessons so that he could watch to see whether his students were doing what they should be doing. And we can assume that he was strategic about the students that he worked with; in other words, he would have chosen to work with students who needed the most support and the activities they needed help with. What was also good about his monitoring was that he gave a really specific positive feedback to his students about what they were doing, and he provided clear instructions to them about how to improve their implementation. Both of those things make his monitoring very strong. You also notice that he was holding the orange Watching PALS Pairs checklist card, which is a tool for him to use to remember the key elements of the activity and what students should be doing throughout that helps him decide what sorts of things he needs to focus on with the students. That’s a great part of his monitoring strategy.
(Close this panel)
Now you are going to observe Mr. Braxton two more times and record his fidelity of implementation for each observation. Before watching the two video clips of Mr. Braxton implementing PALS, click here and print two copies of the PALS 2–6 Observation Form.
Transcript: Observation 2
Mr. Braxton: Okay, everybody, it’s time for PALS. So go ahead, second readers, got to switch. Go ahead and switch now. Do that now. All right, we’ve got to make sure you get started. So, you know, you need to move to your PALS seats pretty quickly and everything. Question?
Student: Can readers stay in their seats?
Mr. Braxton: Yeah! First readers, stay in your seats. You know we always do it that way, don’t we?
Student: Yeah.
Mr. Braxton: Yeah! So it’s just like we usually do, okay?
[A student shows her bracelet to Mr. Braxton.]
Mr. Braxton: Very nice bracelet. Okay, quickly to your seats. All right. Okay, it’s time for PALS, everybody. So all right. So, first readers, I need to give you your folders.
[Mr. Braxton begins handing out folders]
Mr. Braxton: Pair A, there you are. Thank you. Pair B…wait, where’s Pair B’s folder? Um, okay, Pair B, sorry…wait, who’s Pair B? Oh, you’re pair B! Pair B, um, okay. You can get out your stuff. Don’t forget to do that, too. Pair E…where’s Pair E? Where’s your folder? I’ll give you guys yours now. Oh, thank you. Wait…you’re Pair E? E? There you go. Sorry. Uh, Pair G, and you guys are F, right?
Mr. Braxton [cont.]: Okay, cool. All right, go ahead and get out your stuff. We’re going to start with Partner Reading like always. All right, and I’ll wait for you guys to get ready. Don’t forget to have out…oh, you guys have the book, have your books out. Don’t forget that, too. Those were in your…you know, we usually keep those in your desks. Okay, all right, you’ve got five minutes for Partner Reading. Okay, and…go.
[The students read aloud until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs.
[A student raises her hand.]
Mr. Braxton: Yeah? What’s up?
Student: Can I go to the bathroom?
Mr. Braxton: Yeah, go to the bathroom. Hurry up, hurry up, hurry up. I’ll be his partner for a little bit while you guys, while you’re in the restroom. All right, and…go. All right.
[Mr. Braxton takes a seat. The student reads.]
Mr. Braxton: You’re a second reader?
Student: Yes.
Mr. Braxton: Okay, go ahead.
[The students read until the timer sounds.]
Mr. Braxton: Okay, stop. You guys, switch jobs. Okay, great work. So now we’re going to Paragraph Shrinking. Go ahead and switch materials again, and we’re going to start Paragraph Shrinking in just a second. Don’t forget to, um, start with the first reader. Make sure you have all the…first readers, make sure you have your books. Second readers, make sure to ask the questions. Ready…go.
[The students read aloud.]
Mr. Braxton: Yeah, don’t forget, you got to make sure you read a section first and then do them. Yeah. Okay, good.
[The students read.]
First student [reading]: “‘My name is Jonathan,’ said the boy. ‘We have been listening to your talk about going out.’ “
Second student: Jonathan asked Macadamia if he could go outside, too.
Mr. Braxton: Is that ten words? Is that right, do you think?
Second student [counting]: Jonathan asked Macadamia if he could go outside, too.
Mr. Braxton: So that was less than ten.
First student: Okay.
Mr. Braxton: All right, don’t forget to give her three points. Keep on going.
[The students continue reading.]
Mr. Braxton: Keep on going. You’re doing great.
First student: Okay, what’s next?
Student [reading]: “Bandit felt as if she had been summoned by the pirate emperor. This time…”
Mr. Braxton: And where’s the book supposed to be
Student: [Inaudible.]
Mr. Braxton: And on that. Okay.
[The students continue reading.]
Second student: Name the most important who or what.
First student: Melody.
Second student: Tell the most important thing about the who or what.
First student: She picked up…
Second student: Say the main idea in ten words or less.
Mr. Braxton: Melody…
Student: Melody picked up…
Mr. Braxton: Is that right? Is that the most important thing in that paragraph?
First student: Um-hum.
Mr. Braxton: Okay, she gets three points. Don’t forget to mark them.
[The students continue reading until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs. Get ready to start again. All right, and go ahead.
[The students continue reading.]
Mr. Braxton: It appears you guys…what is all this stuff? This is just junk. Let’s put this away. How come you have my sticky notes, anyway? All right, now, well, you guys lost your place in the book, too? You got it back. Okay, who’s reading? You’re reading? Oh, you’re thinking.
Student: I’m asking the questions.
Mr. Braxton: Remember you can have the book open, and you can be looking back as you’re doing it, okay? Keep on working hard.
[The students continue reading until the timer sounds.]
Mr. Braxton: Okay, stop. We didn’t switch jobs yet, did we? No, we did. Did we?
Student: Yeah.
Mr. Braxton: We already switched?
Student: I don’t know.
Student: Did we?
Mr. Braxton: I can’t remember.
Students: Yeah, we switched.
Mr. Braxton: We switched. Okay, so is it time for Prediction Relay? Okay, time for Prediction Relay, so go ahead and switch jobs, switch jobs and get ready for Prediction Relay. Okay, you have five minutes.
Student: I have to go to the bathroom
Mr. Braxton: Okay, that’s fine. I can be Tyler’s partner. All right, ready? You got it. Ready and…go. Five minutes.
[Mr. Braxton sits down with the student.]
Mr. Braxton: Okay is she the first reader?
Student: Uh, yes.
Mr. Braxton: Okay, where are you guys?
[The students continue working until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs. This is our last one, everybody, so let’s make it good here. All right and…begin.
[The students continue working.]
Mr. Braxton: Okay, so did you make a prediction? Oh, no, you’re the second reader. Did you make a prediction?
Student: Um, not yet.
Mr. Braxton: Not yet. Okay, did you ask her what her prediction was? I forgot to tell you guys. I’m sorry. Make sure you make a prediction first. So what do you think will happen next?
Student: Um, I think that maybe the mom will write a letter to the girl that, um, she’ll tell, like, she’s going to get to come home.
Mr. Braxton: Okay, does that sound like a prediction to you?
Student: Yes.
Mr. Braxton: Okay, good. So what do you say next?
[The students disagree about what comes next.]
Mr. Braxton: Girls, you can cooperate. Let’s keep working together. Did you make a prediction?
Student: Yes. She said, she said she was about to read half the page.
Mr. Braxton: About half the page. Okay.
[The timer sounds.]
Mr. Braxton: Oh, that’s time. Okay. All right, so go ahead, put your folders and everything back in your folders and put your books back in your desk, and I’ll take your folders. And thank you. Thank you. I’m going to come around this way. May I have your folder, please? All right, so today everyone did a pretty good job. I think some pairs weren’t cooperating the best, but that’s okay. We’re going kind of fast today since we have a…we have to get to lunch. That’s great. All right, so, everyone, good job today. Thanks so much, and we will do more good work with PALS tomorrow.
(Close this panel)
- Using the fidelity data you collected for Observation 2, calculate Mr. Braxton’s fidelity (i.e., the percentage of steps he implemented correctly). Click here for feedback
Feedback – Observation 2, Question 1
Below is a copy of the completed PALS 2–6 Observation Form for Observation 2 and the calculation for Mr. Braxton’s fidelity. For more detailed comments about Mr. Braxton’s performance, listen to Devin Kearns’ audio at the bottom of the page.
PALS 2–6 Observation Form
Date: XX-XX-XX
Teacher: Mr. Braxton
Observer: DK
Introduction (2 minutes or less)
1. Introduce PALS session
2. Instruct students to transition to pairs quickly and quietly
3. Have system in place for students to quickly and easily
access materialsPartner Reading
4. Implement for 10 minutes
5. Follow script
6. Instruct students to switch partners after 5 minutes
7. Monitor studentsRetell
8. Implement for 2 minutes
9. Follow script
10. Monitor studentsQuality Items20. How well did the teacher
implement the PALS
instructional activities?1 2 3 Not well Average Very well 21. How well did the teacher
monitor the students?1 2 3 Not well Average Very well Paragraph Shrinking
11. Implement for 10 minutes
12. Follow script
13. Instruct students to switch partners after 5 minutes
14. Monitor studentsPrediction Relay
15. Implement for 10 minutes
16. Follow script
17. Instruct students to switch partners after 5 minutes
18. Monitor studentsWrap Up (2 minutes or less)
19. Have students put away materials
To calculate Mr. Braxton’s fidelity of implementation, the observer divides the number of items checked (i.e., 10) by the total number of items on the checklist (i.e., 19), not including the quality items. Dividing 10 by 19 yields .526, or 53 percent adherence to procedures.Listen as Devin Kearns offers detailed comments about Mr. Braxton’s performance (time: 3:19).
Devin Kearns, MA
PALS trainer
Vanderbilt UniversityTranscript: Devin Kearns, MA
For this observation, the teacher had some difficulty implementing PALS. From the beginning, it was clear the teacher was not well organized. The transition to pairs took a very long time. It did not happen in an organized fashion. The method for distributing the folders to the students was highly inefficient and took far longer than was necessary. In terms of the implementation of the individual activities, for Partner Reading the teacher never followed the script starting the activity in sort of an off-the-cuff way and taking so little time to begin the activity that the students were probably confused. In addition, the teacher did not monitor the students during Partner Reading. The teacher did not complete the Retell activity. That’s problematic because PALS is designed to be a set of components that all fit together, and when you have a research-based program like PALS, it’s important to implement all the components. Missing an entire component is a really serious error. For Paragraph Shrinking, the teacher did monitor the students. The quality of the monitoring may not have been great, but the monitoring did occur. The teacher did not follow the script at all and so the introduction to the activity, and the transition from one student to the other reading, was somewhat chaotic. This was also true for Prediction Relay. It resulted in the students not beginning the activity correctly. In PALS, students are supposed to begin Prediction Relay by making a prediction. On the green script card, it tells the teacher to begin by making a prediction. Because the teacher didn’t use a script card, the students did not know to begin by making a prediction, and many of the pairs neglected to do that. You may have noticed the teacher reminding students to start with a prediction.
In terms of the quality items, the first one—how well did the teacher implement the PALS instruction activities?—the teacher should be rated one, not well. The reason for this is because, first of all, the teacher did not implement all of the instructional activities. Omitting an entire activity, as the teacher did with Retell, was a serious error. In a program like PALS that’s research-based, it’s important to complete all the activities and not eliminate any of them. In addition, the teacher did not follow the script. Not following the script caused the teacher to eliminate important instructions for the students. It also resulted in a chaotic transition that resulted in students not being on-task and as focused as they could have been.
In terms of the second item—how well did the teacher monitor students?—the teacher should probably be rated average. Although the teacher did not monitor during Partner Reading or during Retell because the activity was not completed. The teacher did monitor the students for the last two activities, Paragraph Shrinking and Prediction Relay. Particularly in the case of Prediction Relay, the teacher made an attempt to help the students improve their implementation. Some of the student difficulties were caused by the teacher not introducing the activity, but the teacher made a good faith effort to try to help the students. And for that reason the teacher should be rated average for monitoring. It’s important to note, however, that it can be difficult to decide in a case like this whether or not the ratings should be one or two. If you went through training, it would be much easier to determine whether the teacher should be rated one or two, because the training would identify the key characteristics of a one-level or a two-level implementation.
(Close this panel [Kearns audio])
(Close this panel [Observation 2, Question 1])
- If you were providing Mr. Braxton with corrective feedback, what suggestions would you make based on what you observed? Click here for feedback.
Feedback – Observation 2, Question 2
Listen as Devin Kearns discusses the type of feedback he would give to Mr. Braxton regarding his implementation of PALS (time: 0:43).
Devin Kearns, MA
PALS trainer
Vanderbilt UniversityTranscript: Devin Kearns, MA
To give feedback to the teacher, I would first highlight the positive, which in this case was that the teacher provided feedback to the students for Paragraph Shrinking and Prediction Relay. There are a number of pretty major concerns in this lesson; all of them appear to be driven by the fact that the teacher didn’t use the script. So in giving feedback to the teacher, I would emphasize the importance of using the script to guide them through the PALS activities so that they don’t forget to leave anything out that would have prevented them from missing Retell entirely. Telling the teacher to use the script would not improve the fidelity completely. The teacher might still have difficulty consistently monitoring the students. But considering there are so many concerns in this case, I focus on the biggest one, which is using the script.
(Close this panel [Kearns audio])
(Close this panel [Observation 2, Question 2])
Transcript: Observation 3
Mr. Braxton: Okay, everybody, it’s time for PALS. Second readers, go ahead and move to your PALS positions. Do that now. All right. Okay, taking a little while. It’s all right. Okay, we’ll get started in just a minute. Pair A, first reader, thank you. And you can go ahead and get out your stuff for, um, Partner Reading. So go ahead and take out your points sheet, and your, um, and your correct question card. Pair B? Pair C, first reader? Thank you. Pair D? Thank you. Pair E? All right. Pair F? All right. And pair G? Okay, there you go. All right, so first activity today is Partner Reading; we’re going to do that first. Okay, so, first readers, get ready to read quickly and correctly. Make sure you’ve got out your materials. Don’t forget that. You’re going to read for five minutes. Second readers, you’re going to listen and correct mistakes if there are any, okay? Okay, so books right in the middle. Second readers, make sure you’ve got your point sheets and your pencils. Most pairs have their pencils out. All right, we’re going to do this for five minutes. Ready…begin.
[The students read aloud from their books.]
Mr. Braxton: Okay, I like how you’re following along when she’s reading. Excellent. Keep it up.
[The students continue to read aloud.]
Mr. Braxton: Excellent job following along. I like how hard you guys are working together. I’m going to give you three bonus points. Much better job than yesterday. Great work.
[The timer beeps.]
Mr. Braxton: Okay, stop. All right, time to switch jobs. Go ahead and switch materials. And you’re going to go back to where you started, so make sure you go back to the place where you started today. Second readers, get ready to read. Ready…begin.
[The students read aloud from their books until the timer beeps.]
Mr. Braxton: Okay, that is…whoa…that is time. Great job working hard on Paragraph, er, Partner Reading today. Okay, it’s time for Retell. Second readers, you’re going to retell everything. First readers, remember to ask them what happened first, what happened next. You have two minutes. Ready…go ahead.
[The students practice Retell until the timer beeps.]
Mr. Braxton: Okay, stop, everyone. All right, great job. Okay, time for Paragraph Shrinking. Switch materials again. Remember, first readers, you’re going to read and shrink. Second readers, remember to ask the Paragraph Shrinking questions, okay? All right, you’re going to have five minutes. Ready…begin.
[The students practice Paragraph Shrinking, with Mr. Braxton offering occasional suggestions.]
Mr. Braxton: That was less than ten words. All right, yeah, that’s good. Give her ten, give her three points and then go on to the next one. Great job asking questions. Make sure you have your card out so you don’t forget, okay? Uh-oh, my, I forgot to set my timer, guys. I think we went over time, but that’s okay, so let’s just go on. It’s time to switch jobs so, second readers, go ahead and take the books. First readers, go ahead and take the point-sheet and the question card. I noticed a couple pairs didn’t have out their question cards. I forgot to tell you guys to have them flipped over, so make sure that you have out the question card with the Paragraph Shrinking information on it, so if you forget you’ll remember what to say, okay? We’re going to do it for five minutes. I’m going to make sure I set my timer this time. Ready…begin.
[The students practice until the timer beeps.]
Mr. Braxton: Oh, that’s time. All right, it’s time for Prediction Relay. Switch jobs. I like everyone switching really quickly. Okay, it’s time for Prediction Relay. Ready…begin.
[The students practice Prediction Relay.]
Mr. Braxton [to a student]: Did you make a good prediction?
Student: Yes.
Mr. Braxton: Okay, so follow along carefully. Where’s the book supposed to be?
Student: The book’s supposed to be…
Mr. Braxton: In the middle. Perfect.
[The timer beeps.]
Mr. Braxton: Okay, that was the time. All right, time to switch jobs. Second readers, get ready. Do this quickly. All right, let me set the timer. Try not to forget again today. Ready…begin.
Mr. Braxton: For having his dad help him with his paper?
Students: Yes.
Mr. Braxton: Okay, let’s see if that happens on the next page. Go ahead. Good job working hard.
[The students switch readers and continue their practice until the timer sounds.]
Mr. Braxton: Okay, stop. Switch jobs…oh, no, we…that’s it, right? We already switched jobs? Okay! Excellent job today, everyone. We are finished with PALS for today. You can put all of your materials back in your PALS folders. Put the books back in your desks. We’ll collect the PALS folders in a second. As soon as everyone’s quiet, I want to tell you guys a couple things about today. All right. Okay, again today, everyone, really good job.
(Close this panel)
- Using the fidelity data you collected for Observation 3, calculate Mr. Braxton’s fidelity (i.e., the percentage of steps he implemented correctly). Click here for feedback
Feedback – Observation 3, Question 1
Below is a copy of the completed PALS 2–6 Observation Form for Observation 3 and the calculation for Mr. Braxton’s fidelity. For more detailed comments about Mr. Braxton’s performance, listen to Devin Kearns’ audio at the bottom of the page.
PALS 2–6 Observation Form
Date: XX-XX-XX
Teacher: Mr. Braxton
Observer: DK
Introduction (2 minutes or less)
1. Introduce PALS session
2. Instruct students to transition to pairs quickly and quietly
3. Have system in place for students to quickly and easily
access materialsPartner Reading
4. Implement for 10 minutes
5. Follow script
6. Instruct students to switch partners after 5 minutes
7. Monitor students
Retell
8. Implement for 2 minutes
9. Follow script
10. Monitor studentsQuality Items20. How well did the teacher
implement the PALS
instructional activities?1 2 3 Not well Average Very well 21. How well did the teacher
monitor the students?1 2 3 Not well Average Very well Paragraph Shrinking
11. Implement for 10 minutes
12. Follow script
13. Instruct students to switch partners after 5 minutes
14. Monitor studentsPrediction Relay
15. Implement for 10 minutes
16. Follow script
17. Instruct students to switch partners after 5 minutes
18. Monitor studentsWrap Up (2 minutes or less)
19. Have students put away materials
To calculate Mr. Braxton’s fidelity of implementation, the observer divides the number of items checked (i.e., 13) by the total number of items on the checklist (i.e., 19), not including the quality items. Dividing 13 by 19 yields .684, or 68 percent adherence to procedures.Listen as Devin Kearns offers detailed comments about Mr. Braxton’s performance (time: 2:48).
Devin Kearns, MA
PALS trainer
Vanderbilt UniversityTranscript: Devin Kearns, MA
In this observation, the introduction to PALS was okay. You’ll notice that the teacher took a relativity long time to distribute the folders to the students. The folder distribution process was not as efficient as it could have been, although it was completed in the allotted amount of time. In this observation, we saw the teacher do a decent job implementing PALS. In terms of the fidelity checklist, the teacher followed the script for parts of the lessons. For Partner Reading, the teacher held the script for the beginning of the activity, but not during the transition. The teacher did not use the script for all of the remaining PALS activities. You’ll notice in this observation that the teacher repeatedly failed to use the script. That resulted in a significant lowering of the teacher’s fidelity. It’s a simple thing to fix, and if the teacher were to do that differently fidelity would definitely improve.
In addition, the teacher did not monitor students consistently, leaving out monitoring completely during the Retell activity focusing on some other tasks. You also notice that the teacher didn’t implement PALS for the correct amount of time during Paragraph Shrinking. If the teacher had also used the timer correctly during Paragraph Shrinking and had monitored the students during Retell, the fidelity would actually have been perfect. And so this demonstrates that it does not actually take that much additional effort to get your fidelity from relatively weak to relatively strong. On our fidelity checklist, you see that implementing PALS for ten minutes is one item, and instructing students to switch partners after five minutes is another item. Because of the way we structure the fidelity checklist, the teacher’s penalized twice for going over time on Paragraph Shrinking. When schools design fidelity checklists, it’s important to consider the implications of individual items for other items; in this case you can see how it created a double penalty. In terms of the quality items, for the first one—how well did the teacher implement the PALS instructional activities?—the teacher should be rated average. The teacher did implement the activities for the most part correctly, but the teacher did not consistently use the script to introduce the activities, and the teacher didn’t use a timer in the correct way, so that Paragraph Shrinking did not take the correct amount of time. So we should consider this an average implementation. In terms of student monitoring, the second item, the teacher should also be considered average. There was feedback given to some of the students, but monitoring was not done consistently, and the quality of the feedback could have been better. The teacher didn’t always provide clear instructional feedback to help the students improve their future implementation of the program, and [was] sometimes quick and off-the-cuff. The teacher also did not always use the orange Watching PALS Pairs checklist, which is a helpful tool for monitoring the students. It allows the teacher to decide what sort of things should be focused on.
(Close this panel [Kearns audio])
(Close this panel [Observation 3, Question 1])
- If you were providing Mr. Braxton with corrective feedback, what suggestions would you make based on what you observed? Click here for feedback.
Feedback – Observation 3, Question 2
Listen as Devin Kearns discusses the type of feedback he would give to Mr. Braxton regarding his implementation of PALS (time: 0:47).
Devin Kearns, MA
PALS trainer
Vanderbilt UniversityTranscript: Devin Kearns, MA
For this observation, the positive feedback I’ll give to the teacher is that they completed all the activities in a relatively efficient way. The procedure the teacher used for getting out materials wasn’t perfect, but it did finish in less than two minutes, and the implementation of all the activities went relatively smoothly. In terms of giving this teacher constructive feedback, I would highlight a couple of things. I would first mention that the teacher should consistently use a script. This is not likely to be difficult for the teacher since they used the script part of the time, but reminding the teacher to use the script consistently will help them improve their fidelity. The second piece of feedback I would give would be to monitor the students for every activity. The teacher did not monitor the students for the Retell activity, and monitoring in that case would have been helpful as well. This teacher is already doing an okay job implementing PALS, and these small suggestions will help improve their implementation even further.
(Close this panel [Observation 3, Question 2])
Ensure the Practice or Program Was Implemented as Intended
Whereas collecting data is part of the monitoring process, evaluating fidelity occurs when the data are analyzed. If the fidelity data indicate that school staff are implementing the practice or program as it was intended, they are off to a good start. Even if fidelity is high, staff should receive feedback on their implementation: Feedback is the most common method of maintaining fidelity and it helps motivate staff to improve or refine their implementation. Feedback also enables those implementing the practice or program to better judge to what extent they are adhering to program requirements.
Now listen to George Sugai as he discusses how schools can overcome one of the challenges related to collecting fidelity data (time: 0:43).
George Sugai, PhD
Center on Positive Behavioral
Interventions & Supports
Transcript: George Sugai, PhD
One of the challenges is making sure that the teams and the teachers and so forth understand that the purpose of fidelity of implementation checks are about improving the quality of the kid outcomes, and the kinds of interventions we put in place. That it’s not really for evaluation or accountability or to judge good or bad teaching; It’s really about maximizing kid outcomes. What we’ve tried to do is to make fidelity checks easy. We’ve used checklists, we’ve used self-assessments, we’ve asked people to videotape themselves, in ways that it gives them quick access to information about how they’re implementing, and allow them to match up to what the necessary requirements are of the intervention or practice.
Detect and Correct Implementation Errors Early
Because fidelity issues are fairly common when school staff begin implementing a practice or program, it is important in the early weeks of implementation to collect fidelity data frequently so as to correct implementation errors in a timely manner. If individuals are allowed to implement a practice or program for too long without constructive feedback, incorrect procedures may become ingrained and therefore be more difficult to correct.
In the event that the fidelity data indicate that one or more school personnel are not implementing the practice or program as it was intended, the corrective procedures should be followed without delay. The staff members in this instance may need to be retrained, coached, or mentored to help them to implement the practice or program with fidelity. Soon after the corrective action has been taken, the individuals need to be observed and fidelity data again collected. If the data indicate that the individual is now implementing with fidelity, data collection can proceed according to the established schedule. It may be necessary to collect fidelity data more often for those who show a tendency to drift from the procedures.
Listen as Daryl Mellard explains the benefits and drawbacks of several types of commonly used methods and measures for assessing fidelity of implementation (time: 1:49).
Daryl Mellard, PhD
Co-Principal Investigator,
National Center on Response to Intervention
Transcript: Daryl Mellard, PhD
We would hope that the fidelity measures occur frequently enough and that we’re able to use these observations to improve teacher performance or anyone who’s in this instructional role. It could be a peer from the same grade or class who knows this content. It could be that instructional coach. It could be just someone who serves as the mentor from within the school. I can imagine that beginning teachers might be paired with someone with more experience who seems to work well with students and knows the content. That person might be in a good position to offer the kinds of observations to improve that instructor’s fidelity of implementation. When we observe drift, I think it’s very appropriate to have clear observation data to be able to describe here was the shift in protocol that was observed and to address that through coaching, maybe through more professional development in the particular content or the particular skills. So those two—continued professional development, coaching, incorporating some element of modeling—seems very important in our efforts at improving fidelity of implementation. We want to model the ideal practice so that everyone can recognize this is what it should look like when we see high-fidelity delivery.
Evaluate the Relation Between Fidelity and Student Outcomes
To judge a program’s effectiveness within a given school, staff can examine the relation between fidelity of implementation and student outcomes. School staff cannot determine whether any change in students’ performance can be attributed to the practice or program unless it was implemented as intended. If fidelity is high, increases in students’ performance can be attributed to the evidence-based practice or program. If fidelity is high and there is no change in students’ performance, it can be inferred that the practice or program was not effective.
On the other hand, if fidelity is low, the relation between the practice or program and student outcome data cannot be interpreted with confidence because:
- It is uncertain whether poor student outcomes are due to low fidelity or inadequacies in the practice or program.
- It is uncertain whether positive outcomes might be improved still further if the practice or program was accurately implemented.
- It may be difficult to identify which aspects of the practice or program are leading to positive student outcomes when adaptations are made.
For Your Information
It is not uncommon for schools to experience a decrease in student performance—that is, an “implementation dip”—when implementing a new practice or program. As teachers refine and master their new skills, student performance usually increases.
Click the school logos below to learn about the process each school uses to collect data and evaluate fidelity of implementation.
|
|
|
Collect Data and Evaluate Fidelity of Implementation
Paige Elementary School
The team begins collecting data on the teachers. After observing one of them––Mrs. McGill––the observer briefly provides her with feedback about her implementation (see the Classroom Observation Form below), pointing out that she needs to hand back scored probes so that her students can graph their performance.
|
1. Probes are filed in folders labeled with date of administration.
To calculate Mrs. McGill’s fidelity of implementation, the observer divides the number of items checked (i.e., 12) by the total number of items on the checklist (i.e., 13). Dividing 12 by 13 yields .923, or 92 percent adherence to procedures.
In addition to the observation data, the implementation team reviews the computerized student data files for each teacher. As they review Mrs. McGill’s data, they notice that her students are performing unusually well (i.e., 3 to 4 times better) compared to the students in the other classes whose teachers are also implementing with high fidelity. After further investigation, they discover that Mrs. McGill has been entering the percent of words read correctly instead of the number of words read correctly. They determine that Mrs. McGill needs a brief booster session on how to correctly use the computer software program and on the appropriate data to enter.
(Close this panel)
Collect Data and Evaluate Fidelity of Implementation
Grafton Middle School
The team begins collecting data on the teachers, starting with Ms. Thomas. Of the twelve items on the observation checklist, Ms. Thomas adhered to nine (see the Fidelity of Implementation Checklist below). Although Ms. Thomas failed to adhere to items 4a, 5a, and 10, the observer felt that, overall, she was off to a promising start. At the end of the reading class, the observer briefly discussed the observation with the teacher and provided her with specific corrective feedback about the three items that she did not adhere to. Over the next week, the observer monitors her fidelity twice more.
When the implementation team meets to review the data, they begin with Ms. Thomas’s data (see the Fidelity of Implementation Checklist below). To calculate Ms. Thomas’s fidelity of implementation for her first observation, the team divides the number of items checked “Yes” (i.e., 9) by the total number of items on the checklist (i.e., 12). Dividing 9 by 12 yields .75, or 75 percent adherence to procedures. The team is concerned with Ms. Thomas’s observed fidelity score of 75 percent, but they continue to examine the rest of her data. For the next two observations, she received scores of 92. The average of her three observed fidelity scores (75 + 92 + 92) is 86 percent, which meets the established fidelity criteria.
|
Fidelity of Implementation Checklist
Date: __10/22/XX____ Teacher: __Ms. Thomas___ Observer: __JB__ |
|||||||||||||||||
| Instructions: Put a checkmark in the column labeled Yes or No depending on whether the teacher completed the task. | |||||||||||||||||
| Yes | No | ||||||||||||||||
| 1. Teacher conducted lesson at the scheduled time. | ✔ | ||||||||||||||||
| 2. Small-group size is 10 students or fewer. | ✔ | ||||||||||||||||
| 3. Teacher pre-teaches vocabulary. | ✔ | ||||||||||||||||
| 3a. Teacher followed the script. | ✔ | ||||||||||||||||
| 4. Teacher leads oral reading. | ✔ | ||||||||||||||||
| 4a. Teacher followed the script. | ✔ | ||||||||||||||||
| 4b. Teacher monitors oral reading and provides corrective feedback. |
✔ | ||||||||||||||||
| 5. Teacher conducts a comprehension check. | ✔ | ||||||||||||||||
| 5a. Teacher followed the script. | ✔ | ||||||||||||||||
|
|||||||||||||||||
|
|||||||||||||||||
|
|||||||||||||||||
| 9. Teacher assigned corresponding workbook lesson to students. | ✔ | ||||||||||||||||
| 10. Students exchanged workbooks and corrected each other’s work. Not completed; time ran out |
✔ | ||||||||||||||||
| 11. Lesson lasted between 40 and 50 minutes. | ✔ | ||||||||||||||||
The team also examines Ms. Thomas’s self-report data (see the Teacher Self-Report Checklist below). Ms. Thomas reported that she adhered to eleven of the twelve items (i.e., 92 percent) on the first self-report. Ms. Thomas’s scores on the next two self-report checklists were 92 percent and 100 percent. The average of her three self-reports of fidelity is 95 percent. The team realizes that self-reports of fidelity often yield higher scores than do observations of fidelity.
Because Ms. Thomas’s observed fidelity increased over the three observations and her average score met their requirements, as did her self-reports of fidelity, the team agreed that she was ready to continue implementing the reading program with monthly fidelity checks.
|
Teacher Self-Report Checklist
Teacher: __Ms. Thomas___ Date: __10/22/XX____ |
|||||||||||||||||
| Instructions: Put a checkmark in the column labeled Yes or No depending on whether you completed the task. | |||||||||||||||||
| Yes | No | ||||||||||||||||
| 1. I conducted lesson at the scheduled time. | ✔ | ||||||||||||||||
| 2. Small-group size is 10 students or fewer. | ✔ | ||||||||||||||||
| 3. I pre-taught vocabulary. | ✔ | ||||||||||||||||
| 3a. I followed the script. | ✔ | ||||||||||||||||
| 4. I led oral reading. | ✔ | ||||||||||||||||
| 4a. I followed the script. | ✔ | ||||||||||||||||
| 4b. I monitored oral reading and provided corrective feedback. |
✔ | ||||||||||||||||
| 5. I conducted a comprehension check. | ✔ | ||||||||||||||||
| 5a. I followed the script. | ✔ | ||||||||||||||||
|
|||||||||||||||||
|
|||||||||||||||||
|
|||||||||||||||||
| 9. I assigned corresponding workbook lesson to students. | ✔ | ||||||||||||||||
| 10. Students exchanged workbooks and corrected each other’s work. | ✔ | ||||||||||||||||
| 11. Lesson lasted between 40 and 50 minutes. | ✔ | ||||||||||||||||
(Close this panel)
Collect Data and Evaluate Fidelity of Implementation
DuBois High School
Fidelity of implementation for PBIS is based on two criteria:
- A score of 80 percent or higher on the School-wide Evaluation Tool (SET) general index (i.e., an average of scores from all the sections)
- A score of 80 percent or higher on the index for Behavioral Expectations Taught (section B)
For demonstration purposes, a portion of the SET is included below to illustrate how it could be filled out and how fidelity could be calculated.
| Feature | Evaluation Question | Data Source (circle sources used) P = product; I = interview; O = observation |
Score: 0–2 |
||
|---|---|---|---|---|---|
| A. Expectations Defined | 1. Is there documentation that staff has agreed to 5 or fewer positively stated school rules/ behavioral expectations? (0 = no; 1 = too many/ negatively focused; 2 = yes) |
|
2 | ||
| 2. Are the agreed upon rules & expectations publicly posted in 8 of 10 locations? (See interview & observation form for selection of locations). (0 = 0–4; 1 = 5–7; 2= 8–10) |
|
1 | |||
| B. Behavioral Expectations Taught | 1. Is there a documented system for teaching behavioral expectations to students on an annual basis? (0 = no; 1 = states that teaching will occur; 2 = yes) |
|
2 | ||
| 2. Do 90% of the staff asked state that teaching of behavioral expectations to students has occurred this year? (0 = 0–50%; 1 = 51–89%; 2 = 90%–100%) |
|
2 | |||
| 3. Do 90% of team members asked state that the schoolwide program has been taught/reviewed with staff on an annual basis? (0 = 0–50%; 1 = 51–89%; 2 = 90%–100%) |
|
1 | |||
| 4. Can at least 70% of 15 or more students state 67% of the school rules? (0 = 0–50%; 1 = 51–69%; 2 = 70–100%) |
|
2 | |||
| 5. Can 90% or more of the staff asked list 67% of the school rules? (0 = 0–50%; 1 = 51–89%; 2 = 90%–100%) |
|
2 | |||
| C. On-going System for Rewarding Behavioral Expectations |
1. Is there a documented system for rewarding student behavior? (0 = no; 1 = states to acknowledge, but not how; 2 = yes) |
|
2 | ||
| 2. Do 50% or more students asked indicate they have received a reward (other than verbal praise) for expected behaviors over the past two months? (0 = 0–25%; 1 = 26–49%; 2 = 50–100%) |
|
1 | |||
| 3. Do 90% of staff asked indicate they have delivered a reward (other than verbal praise) to students for expected behavior over the past two months? (0 = 0–50%; 1 = 51–89%; 2 = 90–100%) |
|
2 |
(From the Positive Behavioral Interventions & Support website, http://www.pbis.org/pbis_resource_detail_page.aspx?
Type=4&PBIS_ResourceID=222)
To obtain the score for the SET general index, calculate the average of the scores from all the sections (for illustration purposes, the example below uses only three sections). To do this, for each section first divide the score (e.g., 3 for Section A) by the highest possible score (e.g., 4 for Section A) to calculate the percentage for that section (e.g., 3 divided by 4 equals .75, or 75 percent). Next, add the percentage completed for each section (e.g., .75, .90, and .83) and divide by the number of sections (e.g., 3 sections) to calculate the level of adherence.
Recall that PBIS requires two scores above 80 percent in order to meet criteria for fidelity:
- The first is a general index, which is the average of all section scores. For this example, the average of Sections A–C is 83 percent.
- The second is the Behavioral Expectations Taught score. In this example, that score (Section B) is 90 percent.
These two scores are greater than the required 80 percent criteria, an indication that the school is implementing the program with fidelity.
(Close this panel)
