How can school personnel effectively implement evidence-based practices or programs?

Transcript: Mel Riddile, EdD

First of all, I want to make sure that our staff learn what it is they need to learn. We want to observe other people doing it. There should be opportunities for guided practice and for teachers to get feedback. In some cases, we visited other schools to watch them do it, and that was very important. We also built in a peer-observation component and a coaching component, so that the teachers would observe each other using the strategy and then give feedback to each other, and then the coach would observe them, go into their classroom, and work with them to make sure that they were actually implementing that strategy the way it was designed. But, more importantly, we wanted to know how much follow-up support we were going to get. And that’s a critical part, because, as I learned, nothing is perfect out-of-the-box, so to speak. And so we have to have some flexibility, and it has to be adapted to certain context. And to do that we want to know how much follow-up support we would get from the professional development organization, the program, whatever it was. The support from the vendor-provider organization that we are working with was a critical part of our choice of a path or a program. Do they have technical staff that can support us? The other thing is, can they come back and look at our needs and then provide follow-up support? Does the organization actually have a part of that organization that’s devoted to fidelity of implementation? Do they actually care if schools are implementing the programs as designed and what are they prepared to do to assist schools to do that.

Transcript: Cynthia Alexander

When considering finding the appropriate professional development for the staff, we go back to the data and the needs that are specifically for the school. The first thing you have to consider when selecting PD for a group of educators is what professional development will be the most beneficial for that group of teachers. So that’s what we do periodically, not just once a year. We have a professional-development plan, but it’s a living document, much like the school improvement plan, whereas we see needs evolve we also include those needs for PD in there. You have to be open to make changes as you go with professional development, because what you need at the beginning of the year you might find is not necessarily what you need. So it has to be an evolving process, and you have to make sure that they are relevant topics included in professional development for that time period in your school. There’s some things that will stay the same, but for the most part it’s an evolutionary process where you plan for that PD according to what the needs are. Now, as far as evaluation of the PD, at the end we do a needs assessment for the teachers to figure out was it useful PD, how will you use it, and then also offer them opportunities in team-leader meetings and faculty meetings to, kind of, present some of those high-points or lessons learned from the PD in the form of a lesson, a presentation, or just a quick blurb of what they got out of the PD and what they were able to translate back into their classroom. It’s always good to talk to your teachers to find out exactly what they need.

Pre-training Assessment

Peer Assisted Learning Strategies (PALS) Workshop Pre-Training Assessment
	Very Little		Some		A Lot
1. How much do you know about peer tutoring?	1	2	3	4	5
2. How much do you know about PALS?	1	2	3	4	5
3. Do you use PALS in your classroom?	1	2	3	4	5
	Not at All		Unsure		Definitely
7. Do you believe students can learn effectively from each other?	1	2	3	4	5
8. How comfortable are you letting students work together in pairs?	1	2	3	4	5
9. How comfortable are you with following scripted lessons?	1	2	3	4	5
	Very Little		Some		A Lot
14. How much would you like to learn about implementing PALS?	1	2	3	4	5
15. How much would you like to learn about the benefits of implementing PALS?	1	2	3	4	5

(Close this panel)

Post-training Assessment

Peer Assisted Learning Strategies (PALS) Workshop Post-Training Assessment
	Very Little		Some		A Lot
1. How much do you know about peer tutoring?	1	2	3	4	5
2. How much do you know about PALS?	1	2	3	4	5
	Not at All		Unsure		Definitely
7. Do you believe students can learn effectively from each other?	1	2	3	4	5
8. How comfortable are you letting students work together in pairs?	1	2	3	4	5
9. How comfortable are you with following scripted lessons?	1	2	3	4	5
	Very Little		Some		A Lot
14. How much did you learn about implementing PALS?	1	2	3	4	5
15. How much did you learn about the benefits of implementing PALS?	1	2	3	4	5

(Close this panel)

Overall Training Evaluation Example

Workshop Title: ________________________________________________________________________

(From the Out-of-School Time Resource Center [OSTRC],
http://www.sp2.upenn.edu/ostrc/resources/prodeveval/index.html)

(Close this panel)

Follow-up Assessment Example

Fall Philly PNM Follow Up Survey
Philadelphia Fall Peer Networking Meetings –Follow Up Survey
Thank you for your continued support of the Philadelphia Peer Networking Meetings. Please take a few moments to complete this follow-up evaluation. Your feedback will help enhance our meetings and enrich the out-of-school time field.
	1. How much did you learn from attending the Fall Peer Networking Meeting(s):
	I did not attend this meeting. A great deal A lot Something Very Little Nothing September 17th: Connecting with Schools October 16th: Life Skills for Youth November 14th: Family /Home Dynamics
Application
	2. I applied the knowledge or skill(s) that I learned at the PNM in my work with youth.
	Yes
	No, I did not have to time to apply the knowledge or skill yet.
	No, the knowledge or skill was not directly related to my work.
	No, the PNM did not provide me with any new knowledge or skill.
	Other (please specify)
	3. I shared knowledge or skills from the PNM with colleagues.
	Yes
	No, but I plan to.
	No, I did not learn any new knowledge or skill at the PNM to share with my colleagues.
	Other (please specify)
	4. I adapted the knowledge or skill from the PNM to other programs.
	Yes
	No, I did not have to time to adapt the knowledge or skill to other programs.
	No, but I plan to adapt the new knowledge or skill to other programs.
	No, I work with only one program.
	No, the PNM did not provide me with any new knowledge or skills.
	Other (please specify)
	5. I had the support of my organization to implement the knowledge or skill learned at the PNM.
	Yes
	No
	Other (please specify)
	6. I was held accountable to apply the knowledge or skill from the PNM in my work with youth.
	Yes
	No
	Other (please specify)
Knowledge, Skills and Belief
Now that some time has passed since attending a Fall Peer Networking meeting, please rate the following questions on a scale of 1 to 5:
	7. Now that some time has passed, my level of knowledge and/or skills from attending the PNM is:
	Excellent Above average Average Below average Poor Did not attend September 17th: Connecting with Schools October 16th: Life Skills for Youth November 14th: Family /Home Dynamics
	8. Now that some time has passed, my level of belief in the importance of the topic(s), in terms of benefiting the youth in my work is:
	Excellent Above average Average Below average Poor Did not attend September 17th: Connecting with Schools October 16th: Life Skills for Youth November 14th: Family /Home Dynamics
	9. Thank you for taking the time to complete this survey. If you have any questions, comments or suggestions, please contact Lisa Colby, Senior Research Coordinator at the Out-of-School Time Resource Center at [email protected] or 215.898.2505.Please share any additional comments, feedback and/or suggestions about the Peer Networking Meetings. _______________________________________________________________________________ _______________________________________________________________________________ _______________________________________________________________________________

(From the Out-of-School Time Resource Center [OSTRC],
http://www.sp2.upenn.edu/ostrc/resources/prodeveval/index.html)

(Close this panel)

Tools for Identifying and Evaluating Professional Development

Checklist for Providing PD	Use this handy checklist to make certain that your PD services are ensuring effective intervention. Source: Adapted from the Tennessee Reading First Intervention Guide.
Evaluating Professional Development Resources: Selection and Development Criteria	An extensive and thorough tool for the evaluation of PD resources, this publication seeks to streamline what is too often a difficult (if not haphazard) selection process. Included are detailed notes on the criteria development and organizational process, as well as suggestions for the criteria’s use. Source: Building Professional Development Partnerships for Adult Education Project, American Institutes for Research.
The Five Critical Levels of Professional Development Evaluation	Use this handy tool to track and assess the progress of professional development through five evaluation levels. The pertinent questions include “What Questions Are Addressed?” and “What Is Measured or Assessed?” Source: Evaluating Professional Development by Thomas R. Guskey.
Guidelines for Reviewing a Professional Development Program in Reading	This useful workbook allows users to evaluate indicators across a number of categories—”Content,” “Process,” “Context,” and “Materials”—for the purpose of assessing the completeness of a professional development program in reading. Source: Florida Center for Reading Research.
The Out-of-School Time Resource Center Survey Toolkit	This “compilation of resources and research-based surveys” is a handy tool for developing surveys to evaluate out-of-school time conferences and workshops. Included are tips for creating original surveys as well as customizable versions created and assessed by the OSTRC. Source: Out-of-School Time Resource Center, The University of Pennsylvania.
Steps for Providing PD	This brief outline of steps that might be taken in order to provide professional development can also be used to help school staff evaluate the usefulness of ongoing PD services. Source: Adapted from the Tennessee Reading First Intervention Guide.
Questions to Consider When Choosing Professional Development Training	Use this helpful list of questions to help choose the best and most appropriate professional development training for your school or district. Questions to consider are broken into broad categories related to need, quality, implementation requirements, and cost.

(Close this panel)

Provide Training

Paige Elementary School

Because the district is strongly encouraging all schools to adopt RTI for reading, district personnel are providing training on the core components of the approach (e.g., high-quality reading instruction, progress monitoring). The school implementation team at Paige Elementary contacts the district about training opportunities and learns that the district will conduct a number of summer institutes, one of which focuses on progress monitoring. The implementation team is excited to learn this; they know that most teachers at Paige are not familiar with progress monitoring. They also feel that a number of teachers might be intimidated by the process of examining the data and of subsequently making instructional decisions based on the data. In addition to the summer training, the district will, upon request, provide booster sessions and support regarding progress monitoring.

Although the implementation team believes that the district training will be invaluable, they believe that they should also find out more about any training or technical assistance provided by the company from which they are purchasing their progress monitoring materials. Viewing the company’s website, they see that the following supports are available:

• Video demonstrations
• Online training and conferencing
• Email support
• Phone support

Evaluating PD

At the end of the summer institute, district training personnel will conduct an overall training evaluation so that they can continuously improve their training. In addition, the district can use the evaluation data to provide follow-up support to target those skills that participants report they do not feel confident implementing.

(Created by the Out-of-School Time Resource Center at the University of Pennsylvania. May not be reproduced without permission. The Trustees of the University of Pennsylvania. All rights reserved. Copyright 2007.)

(Close this panel)

Provide Training

Grafton Middle School

Investigating PD

The school implementation team decides that the teachers will benefit from training on how to use the reading program as it was intended to be used. Because the reading program is a commercially available product, the implementation team visits the publisher’s website to learn more about training opportunities. They locate a list of approved trainers and a description of the type of training available. The team also finds the answers to a number of questions:

Does the training include a manual?

Yes, the training will include a manual. This is in addition to the guidelines included in the materials the school purchased.

What type of training is provided?

The initial training on how to use the program will be provided by a company representative; he or she will provide a customized workshop based on Grafton’s needs. The teachers who serve as coaches will participate in an additional session with the trainer in order to learn the necessary coaching skills.

For answers to the remaining questions, the team speaks to the representative on the phone. She supplies the following information.

Does the trainer or vendor offer follow-up training support?

The trainer will offer ongoing support in the form of Webinar booster sessions and phone support.

Does the trainer use multiple teaching formats or models during training?

The trainer will address different levels of skill development during training.

Does the trainer address fidelity of implementation?

The trainer will address implementation fidelity but will not provide any further supports (e.g., observation forms, teacher surveys).

Evaluating PD

The implementation team decides to evaluate the PD they receive from the trainer. Because the teachers have different levels of knowledge about reading instruction, they choose to do a pre- and post-training assessment. Some of the teachers were part of the initial attempt to implement the reading program several years ago, whereas others have no experience with the reading program at all.

Grafton Middle School Reading Program Pre-Training Survey
	Very Little		Some		A Lot
1. How much do you know about this reading program?	1	2	3	4	5
2. Have you ever used this reading program in your classroom?	1	2	3	4	5
3. Do you currently use this reading program in your classroom?	1	2	3	4	5
4. If you currently implement this reading program, how often do you do so?	1	2	3	4	5
	Not at All		Unsure		Definitely
7. Do you believe this reading program can help improve the reading performance of your students?	1	2	3	4	5
8. How comfortable are you implementing this reading program?	1	2	3	4	5
9. How comfortable are you with following scripted lessons?	1	2	3	4	5
	Very Little		Some		A lot
14. How much would you like to learn about implementing this reading program?	1	2	3	4	5
15. How much would you like to learn about the benefits of implementing this reading program.	1	2	3	4	5

Grafton Middle School Reading Program Post-Training Survey
	Very Little		Some		A Lot
1. How much do you know about this reading program?	1	2	3	4	5
	Not at All		Unsure		Definitely
7. Do you believe this reading program can help improve the reading performance of your students?	1	2	3	4	5
8. How comfortable are you implementing this reading program?	1	2	3	4	5
9. How comfortable are you with following scripted lessons?	1	2	3	4	5
	Very Little		Some		A Lot
14. How much did you learn about implementing this reading program?	1	2	3	4	5
15. How much did you learn about the benefits of implementing this reading program.	1	2	3	4	5

(Close this panel)

Provide Training

DuBois High School

As the school improvement team soon learns, the PBIS Technical Assistance Center requires a 3–5 year training commitment from participating schools. A team (which includes administrators and general and special education teachers) of ten school representatives will receive training and in turn will train the remainder of the staff. The training takes two or three days and is provided by skilled trainers. As they learn more about what is required to implement PBIS, the members of the school improvement team create the table below to help organize the new information.

(Close this panel)