How do you measure infant, child, or student performance?
Page 4: Evaluating Progress
To get a reliable estimate of how a child or student is responding to an EBP, you need to implement it for at least three weeks and to score and graph a minimum of six probes. Only then should you evaluate each individual’s performance using the Four Point Method—examining the relationship between the four most-recent data points and the goal line. The figures below illustrate how you can interpret the information from the individual’s graph using the Four Point Method.
Listen as the experts further discuss analyzing progress monitoring data.
Scott McConnell, PhD
Professor, Educational Psychology
University of Minnesota
(time: 1:30)
Bryan Cook, PhD
Professor, Special Education
University of Hawai’i at Mānoa
(time: 1:12)
Transcript: Pamela Stecker, PhD
In order to make a judgment about how well an evidence-based practice is working for a particular student, teachers need to implement that instruction long enough for it to have an overall effect on student performance. The teacher is not just checking to see that the daily lesson objective has been mastered but is looking to see whether the student is growing in desired ways over time and that the student’s learning is generalized, which is reflected by improvement in scores on the progress monitoring tool. This rule may be applied especially when teachers are trying to accelerate student growth.
Transcript: Scott McConnell, PhD
There are a variety of decision rules for analyzing progress monitoring data as we collect it in determining whether we need to change intervention to better support the individual student. Those decisions, rules at this point, are in some cases empirically supported, and we know that they’re the right things to use. In other instances, I think they’re more based on practice and experience. And so the general principle here is that you want to collect information often enough that you can see the child’s development, and you want to monitor that at a level that, if you need to change what you’re doing, you do so quickly. So three data points that are not headed in the right direction, that’s probably enough to act on. In some cases, one or two data points may be sufficient. It depends on the thing being taught, its particular importance or urgency, and how effective you expect the intervention to be.
You know, the example I can think of is that if I had a terrible infection and my fever was very high, and I went to the hospital for treatment they probably wouldn’t wait for three days to change the drug they were giving me if my fever continued to be high. They would probably change it faster, because they know that this is a drug that should work, and so if it’s not working quickly we need to change our path. So I think it’s us being sophisticated about how we look at this information and use it. When in doubt, following simple rules like three data points going up is a good thing, and three data points not going up means we need to change. That’s fine, but I think we also need to be open to other possibilities.
Transcript: Bryan Cook, PhD
I think with a lot of issues in education, including progress monitoring, common sense is often your best guide. And the four-point rule is a great rule of thumb, where if you’ve got four points below the aim line it probably means that the intervention isn’t working. I do encourage people to look at the bigger picture. Even though the four most-recent data points may be below the aim line, if they’re are on a trajectory or they have a slope where it looks like they’re really catching up to the aim line, I’d encourage people to stay with the intervention and not discard it, especially for at-risk learners, kids with disabilities, students who are struggling in school for whatever reason. If we have a practice that is catching them up pretty quickly to where we want them to be, that’s pretty promising. Because we have four or more data points that are below the aim line, we also have to be very careful that we’re not sticking with something that isn’t getting them to where we want them to be, and so we’ve got to make sure we’re doing very frequent progress monitoring and that we don’t let them fall through the cracks and stay below that aim line for too long.
If an individual’s data are on or above the line, it might be tempting to conclude that the EBP is effective. However, without information about whether the EBP was implemented with fidelity, it is unclear whether:
- Something else could be causing the improved performance
- An individual could improve even more if the EBP were implemented as intended
Similarly, if an individual’s data are below the line, without fidelity data, you cannot conclude with certainty that the EBP was ineffective. For this reason, it is crucial to collect fidelity data along with the progress monitoring data to determine whether an EBP is effective with your children or students. The next section will cover how to monitor fidelity of implementation.