How can school personnel identify and select evidence-based practices or programs?

Before you begin, consider first, who should be involved in this discussion? Are you choosing this curriculum or intervention on your own or will others be involved? Every district, school or program (e.g., a Part C Preschool program) is different as to who and how many people should be involved in choosing a new curriculum or intervention. No matter who initiates the search or how many people are on the committee, it is very important that all opinions and ideas be gathered and considered.

Before choosing a new curriculum or intervention for your district, school, or program you should consider questions about:

I. The context or set of circumstances, including place, in which the curriculum or intervention will be implemented;

II. The evidence that demonstrates the effectiveness of the curriculum or intervention; and

III. The resources, or sources of support, including funding and personnel, that will be required to successfully implement and sustain the use of the curriculum or program.

Questions about context:

Each school district has different needs, priorities, student populations, and resources. Choosing a curriculum or intervention that is right for you means taking into account the unique characteristics of your district, school, or program and making sure that the curriculum or intervention will match your needs, specific circumstances, and resources.

Questions to ask include:

• What are the historical origins of the curriculum/intervention? How long has it been around? Was it developed to address issues similar to mine?
• How common is this curriculum or intervention? How widely is it used, and in what kinds of settings does it appear most suited? Are the settings in which the curriculum or intervention is widely used similar to mine? (Factors to consider include demographics of children to which the curriculum or intervention is targeted, size of school, district, or program, resources needed to successfully implement the curriculum or intervention).
• Is the curriculum or intervention aligned with state and district standards?
• In what language can the curriculum or intervention be administered?
• For which grades, ages, or subgroups is the curriculum or intervention appropriate?
• Is the curriculum or intervention sensitive to and does it have flexibility built in to meet individual student needs, including those of children with disabilities?
• Are the teachers and administrators I am targeting prepared to implement this curriculum or intervention?
• Will the design of this curriculum or intervention alter the structure of the schools, district, or program I am targeting in any negative ways?
• Are the changes facilitated by this intervention manageable and/or gradual?

Questions about evidence:

Evidence offers data about how well a curriculum or intervention has been demonstrated to improve student outcomes. It is important to ask questions about what the evidence shows about student outcomes resulting from the use of the curriculum or intervention, as well as questions about the conditions under which studies were conducted. There are differing levels of evidence to consider when examining evidence-based practices. It is up to the decision makers to determine the level of evidence, both quality and quantity, that is most relevant to their needs. Based on the definitions used by the Institute of Education Sciences practice guide standards, these levels are:

1. Strong evidence, which includes research that supports the effectiveness of a curriculum or intervention through (a) several randomized controlled trials or quasi-experiments, (b) a systematic review of research, or (c) one large well-designed randomized controlled multi-site trial. This evidence must be proven to be reliable (gets the same result time after time) and valid (there is less than a 5% chance of the result being inaccurate). Example: A meta-analysis of 10 randomized control trial studies shows conclusive evidence that kindergarten and first grade students with learning disabilities who are exposed to a particular math intervention have significantly better outcomes than students with learning disabilities who were not exposed to the intervention.

2. Moderate evidence often comes from studies where there are strong causal conclusions (saying that one event caused another), but the results may only relate to the specific group that participated in the research. For example, results tested in an elementary setting may not apply to students in secondary settings. Moderate evidence includes that from (a) experiments or quasi-experiments that have small samples sizes or otherwise limit how much the results can be generalized to other populations, (b) comparison studies that do not demonstrate that two groups are equal at pre-test, but consistently show strong outcomes for groups being given a particular program or intervention and have no major flaws in their design. Example: Students taught using a reading program that is based on the National Reading Panel’s components of reading have better outcomes than students using another reading program in an elementary school classroom, but there is not demonstrated evidence on outcomes for students in a high school setting.

3. Low evidence comes from research or studies that do not meet the qualifications for moderate or strong evidence. These include practices for which original data have not been collected, but for which professional anecdotes and wisdom exist. These may also include practices for which some evidence exists, but not from a randomized or quasi-experimental study, and which cannot be generalized to other populations. Example: Teachers in a district implement a positive behavior system across all schools at the elementary school level. Data are collected before, during, and after implementation and indicate that office referrals are significantly reduced and hallway behavior improves at these schools nine months after full implementation of the program.

Questions to ask include:

• Does evidence exist to support the efficacy of the curriculum or intervention?
• Who conducted this research and which peer-reviewed journals have published findings from this evidence?
• Do the studies use adequate sample sizes? Are the samples comparable (age, ethnicity, socio-economic status, disability status, demographic location) to the population I plan to serve with this curriculum or program? Were subgroups, such as children with disabilities and English Language Learners included in the sample and part of the analysis?
• Were conditions in the study similar to the ones in my setting? For instance, if a study evaluated the effectiveness of a specific reading intervention and the study was conducted in classrooms that used a collaborative teaching model, do I have these same conditions and resources available to me?
• Do studies show that this curriculum or intervention has a positive impact on student outcomes? Are these outcomes the same outcomes that I want influenced by the intervention under consideration?
• Are these results comparable to the results of other, similar curricula or interventions? Are the gains in student outcomes larger than the gains we are currently achieving in our school, district, or program?
• Does the curriculum or intervention show consistent success over a period of time?

Questions about resources:

Implementing any new curriculum or intervention requires resources. It is important to consider the total cost of implementing a new program as compared to similar programs, including purchasing costs, staff requirements, training needs, and the cost of sustaining use of the program over time. Questions to ask include:

• How much does the curriculum or intervention cost? Are there costs associated with both implementation and maintenance, including staff training, data systems, etc.?
• Is equipment or software purchase required?
• What types of components are included with the curriculum or intervention?
• Are the materials teacher and student friendly?
• Do the teacher materials provide clear instructions and insight on the topic that is covered?
• What other resources are included?
• Are additional materials that are not included needed to implement the curriculum or intervention?
• What are the personnel and staffing resource needs of the curriculum or intervention? How much time (contracted or volunteer) will staff be required to give?
• What kind of training and inservice is required to implement and maintain the curriculum or intervention? Does the developer provide external support? What are the costs associated with this support?
• Does the developer offer an extended support plan that will provide technical assistance to our school, district, or program after the products have been purchased? How long does this extended support program last?
• If we implement this curriculum or intervention, will resources have to be taken from another program or area, or will another area or school need be compromised because of the resources devoted to this curriculum or intervention?
• Will we be able to use this program once ARRA funds have been expended? Have possible funding options been considered for continuing the work once ARRA funds have been expended?

When choosing a new curriculum or intervention related to behavior, it is important to first determine whether the district has a need related to social and emotional learning, positive behavior supports or both. Social and emotional learning is the process through which children become better at understanding and managing their emotions, learning to make choices, and developing friendships. Positive behavior supports impact the learning environments in a school to increase positive behaviors among students and decrease negative behaviors. It is important that districts examine the behavior challenges they are facing and their goals in terms of student behavior as they make choices about curricula and interventions.

Questions about district need:

• What are the biggest behavior challenges among children in the district that need to be addressed?
• Is there a common vision and consistent language throughout the district related to behavior?
• What behavior outcomes is the district most interested in?
• In what setting will the curriculum or intervention be implemented?
• What are the training needs of teachers and staff with respect to providing effective behavior instruction?
• What are the training needs for the community and families?
• Does the district have a leadership team that includes behavior experts? Does the team share the same set of values, vision, and purpose?
• Is there interest in or a need for a district-wide or schoolwide behavior program? Is there a system in place to support scaling-up efforts and training for a district-wide behavior program?
• Is there an interest in or need for curriculum related to social and emotional learning?
• Does the district have model demonstration schools that are effectively implementing positive behavioral interventions and supports?

Questions about program or intervention characteristics:

• Does the program require having a committed school team in place? If so, do schools considering the program have such a team?
• Does the curriculum have elements that address the needs of the community and families?
• Is the program or intervention based on positive behavior supports?
• Does the program or intervention teach skills for improving interpersonal relationships?
• Are the interventions and strategies evidence-based?
• What teacher training and support is needed to implement the program? Does the program include supports for teachers?
• Are procedures in place to monitor and evaluate the program’s effectiveness?
• Does the program include functional behavioral analysis (FBA) and behavior intervention plans (BIP)?
• Does the program or intervention foster individual accountability?
• Is the program set up for long-term effectiveness?
• Does the district or school have sufficient funding and resources to support the program?
• How does the program provide support to students with emotional or behavioral disabilities? Are there components that specifically address the needs of students with disabilities?
• Does the program include a full array of services and supports and coordinate the resources of the school with those of the community?
• Once purchased will there be additional costs to sustain the program? What is the total cost of this curriculum or intervention compared to similar programs?

Early childhood provides services for children from birth to Kindergarten age who qualify under part B or C of the Individuals with Disabilities Education Act (IDEA). When choosing an early childhood curriculum or intervention, it is important to understand the early learning and development needs of your district’s young children and their families and examine the characteristics of curricula and interventions in relation to those needs. Early childhood curricula and interventions may be comprehensive, support development in multiple domains, or be content specific

Questions about district need:

• What early literacy (e.g., vocabulary, phonological awareness, alphabet knowledge, print awareness) and early numeracy (e.g., classification, measurement concepts, counting, number identification, basic shapes) needs do children in the district have that may affect their ability to benefit from elementary instruction?
• What social-emotional needs do children in the district have that might affect transition into preschool or kindergarten (e.g., executive functioning, coping with negative feelings, peer interaction, staying on-task)?
• What physical development needs do children in the district have that can affect their development and learning in other domains (e.g., overweight/obesity, toxin exposure)?
• Is there a need to target specific age groups or other subgroups of children (e.g., children in poverty, English language learners, children with disabilities)?
• What needs do early childhood program staff have related to supporting children’s active learning through focused, intentional instruction in formal and informal (i.e., play) settings?

Questions about program characteristics:

• Are the goals of this curriculum or intervention understood and valued by all stakeholders, and are they consistent with state learning standards?
• Does the curriculum or intervention have clearly stated key components that will help teachers implement it with fidelity?
• Does the curriculum or intervention provide content and implementation supports that help teachers build on children’s prior learning and experiences, considering variations in developmental age, cultural and familial context, disability status, and language background?
• Does the curriculum or intervention provide guidance to help teachers involve families in children’s learning?
• Does the curriculum or intervention include multiple (i.e., data are gathered from more than one source or method), ongoing, developmentally appropriate assessment procedures to accurately measure children’s development and inform individualized instructional planning?
• Was the curriculum or intervention designed to be implemented in the type of setting in which you want to implement it, considering factors like duration, intensity, and staff qualifications?
• Once purchased will there be additional costs to sustain the program? What is the total cost of this curriculum or intervention compared to similar programs?

References:

Frede, E. & Ackerman, D. J. (2007, March). Curriculum decision-making: Dimensions to consider. NIEER Preschool Policy Brief, Issue 12. Retrieved April 17, 2009, from http://nieer.org/resources/policybriefs/12.pdf.

Institute for Education Sciences. (n.d.). WWC evidence review protocol for early childhood education interventions. Retrieved April 17, 2009, from http://ies.ed.gov/ncee/wwc/PDF/ECE_protocol.pdf.

National Association for the Education of Young Children. (November, 2003). Early childhood curriculum, assessment, and program evaluation. Retrieved April 20, 2009, from http://www.naeyc.org/about/positions/pdf/CAPEexpand.pdf.

Schweinhart, L. J. (2004). A school administrator’s guide to early childhood programs (2nd ed.). Ypsilanti, MI; High/Scope Press.

Questions about district need:

• What mathematical knowledge should children come away with at each level (pre-school, elementary, middle, high)?
• What mathematical knowledge do teachers need to acquire or strengthen to effectively implement the curriculum?
• What are the largest mathematical content (number and operations, algebra, geometry, measurement, data analysis and probability) and process (problem solving, reasoning and proof, communication, connections, representations) gaps, based on the NCTM standards, which need to be addressed?
• In what direction does the district want to move in the coming years with respect to mathematics and the improvement of student achievement? What curriculum shifts and interventions are needed to reach these goals?

Questions about program or intervention characteristics:

• How does the curriculum or program approach mathematics instruction (i.e., investigations of mathematical concepts and thinking, practical applications, technological perspective, or embedded in games, stories or dilemmas)?
• How well aligned is the curriculum or intervention with the district’s goals and expectations?
• How well are concepts, skills, and problem solving balanced within the development of the mathematical ideas?
• What mathematics content is particularly well developed in this program? Are there important math ideas that are not developed?
• How is the program effectively articulated with aspects of the existing program or is it likely that its adoption will create programmatic disequilibrium?
• How does the program help teachers understand the important mathematical ideas of the curriculum?
• Does the program provide specific guidance for its use with children with disabilities and children who are English Language Learners?
• Are the tasks and problems that children work on accessible to all children? Are there problems and exercises for children who need reinforcement?
• Are connections made between mathematical topics and real world applications reflective of children’s cultural background and experiences?
• What role do graphing calculators, computers, manipulatives, and other tools play in the curriculum or program? Are they used in meaningful ways? Are children encouraged to use them while carrying out assessment tasks?
• Once purchased will there be additional costs to sustain the program? What is the total cost of this curriculum or intervention compared to similar programs?

When choosing a new curriculum or intervention, it is important to first gain an understanding of the needs of your district, school, or program with respect to reading, and then to examine the characteristics of the curriculum or intervention you are considering and determine the extent to which those programs meet your needs. The recommendations of the National Reading Panel will provide valuable guidance when evaluating reading curricula and interventions.

Questions about district need:

• What are the biggest reading challenges among children in my district that need to be addressed?
• Is there a need to target reading challenges among specific age groups or other subgroups of children (e.g., student with disabilities, English language learners, children who are older but still cannot decode)?
• What reading outcomes are you most interested in (e.g., phonemic awareness, fluency, vocabulary, comprehension, spelling)?
• Does the district have evidence-based intensive and individualized interventions available for struggling readers and specialists who are trained to deliver them?
• What are the needs of my teachers and staff with respect to providing effective reading instruction?

Questions about program or intervention characteristics:

• Does this program address reading needs for all children (used as a core program or Tier 1 instruction), or does it target specific groups of children (used as a supplemental program or Tier 2 or 3 instruction)?
• Does this curriculum or intervention focus on specific aspects of reading skill (e.g., alphabetics, fluency, comprehension), as recommended by the National Reading Panel?
• For programs that focus on alphabetics, are both phonemic awareness and phonics instruction included as instructional components?
• For programs that focus on fluency, is guided oral reading an instructional component?
• For vocabulary instruction, are words taught both directly and indirectly? Are multiple exposures of the same words presented? Are words taught in context? For older students, is academic vocabulary a focus?
• For comprehension, is there integration of multiple instructional techniques (e.g., comprehension monitoring, cooperative learning, graphic and semantic organizers, question answering, question generation, story structure and summarization)?
• For comprehension programs that target older students, are students directly taught comprehension strategies and comprehension monitoring techniques? Are students guided to collaborate with one another around a variety of texts and given sufficient opportunity to engage in extended discussion of text meaning and interpretation?
• Does the curriculum or program provide ongoing, informal, formative assessments that are then used to make instructional decisions?
• How are writing tasks integrated with reading to increase reading comprehension and build writing skills?
• Does the curriculum or intervention allow for flexibility in meeting the varying reading strengths and weaknesses of children?
• Is the program engaging and motivating for children? Is it engaging and motivating for teachers?
• Does the program integrate reading instruction with instruction in other subject areas (e.g., social studies)? Especially for older students, does the program take a whole-school approach with language arts teachers using content-area texts and content-area teachers providing instruction and practice in reading?
• Particularly for older students, does the program include a technology component where technology is used as a tool and media literacy is taught?
• Once purchased will there be additional costs to sustain the program? What is the total cost of the curriculum or intervention compared to similar programs?

When choosing a new curriculum or intervention, it is important to first gain an understanding of your district, school, or program needs with respect to science, and then to examine the characteristics of the curricula you are considering and determine the extent to which those programs meet your needs. Before making a decision, it is important to determine whether you are looking for inquiry-based approaches that focus on student constructed learning as opposed to more traditional teacher-transmitted types of lessons.

Questions about district need:

• What are the science requirements at the elementary and secondary levels of my district? Which science courses are required for all students and which are taken by fewer students?
• Do teachers have a common understanding of K-12 science standards?
• Is there awareness among teachers of how elementary science curriculum builds into middle school and how middle school aligns with the high-school curriculum?
• Are high-school course offerings available to serve students of all interests and ability levels, including students with disabilities?
• What are the training needs of teachers and staff to provide effective science instruction?
• What are the laboratory resources (space and materials) available in my district? Do all schools have access to needed resources?
• Are safety issues (square feet per student) factored into class size limits in secondary science classes?

Questions about program or intervention characteristics:

• How systematic is the curriculum or program? Are key components built upon at each grade level? Is the curriculum or program easy to understand and implement or will extensive training be required?
• Is there a focus on both process skills and content? Is one emphasized more than the other?
• Are there hands-on laboratory activities? Do the labs use accessible and affordable materials or are more expensive materials required?
• Does the curriculum differentiate activities for students with disabilities, students who may have physical difficulties in the laboratory, students with reading difficulties, or English Language Learners?
• Does the curriculum reflect state and district standards for life science, earth science, and physical science?
• Do assessments align with state and district standards and provide a balanced measure of process skills in combination with concepts?
• How much adult supervision is required for laboratory and other curriculum activities? Is it appropriate by grade level?
• What level of teacher expertise is needed to implement the curriculum? Is there a training package to accompany the curriculum?
• Is the curriculum inquiry based? Is it characterized by interactive student-centered activities focused on questioning, exploring, and posing explanations?
• Once purchased will there be additional costs to sustain the program? What is the total cost of this curriculum or intervention compared to similar programs?

When choosing a new technology product (assistive or instructional), it is important to understand your district’s needs and capacities related to technology. Consider conducting a technology audit of your district that surveys the capacity (age, reliability, extensibility) of hardware, software, Internet capacity and connectivity, tech support, teacher expertise and training needs, and community support. When considering a purchase, be sure to question the “total cost of ownership” – or the life-long costs of obtaining, using and maintaining the product. Knowing your capacity will help you and your colleagues examine the characteristics of the purchase you are considering and determine the extent to which those products or programs meet your needs.

Questions about district need:

• How up-to-date are the technology equipment and capabilities in the district (broadband connections, wireless, technology replacement schedule, connectivity in all rooms, tech support)?
• Is the technology plan up-to-date in the district and aligned with teaching and learning goals?
• What are the potential barriers to technology integration in the district?
• What are the needs of teachers and staff with respect to integrating technology into teaching?
• Do priorities support the use of technology to differentiate instruction in the classroom?
• Do priorities support universally designed and accessible technology use throughout the district?
• What are the needs of administrators in terms of making purchasing decisions about technology?
• What role does technology play when you assess your students? Will this purchase support your assessment program?
• How much do you use technology to manage administrative classroom tasks (including individual education programs (IEPs))? Will this purchase support these efforts?
• Does your technology build self-sufficiency in students? Is it used to actively engage them in their own learning?
• How do teachers communicate with parents and the community? Will this purchase support these efforts?

Questions about program or intervention characteristics:

• Does the vendor provide ongoing support in aligning the curriculum or intervention to district curriculum standards? If so, what does this support consist of?
• Does the curriculum or intervention have a “try before you buy” option so that you can test drive products before purchasing?
• Is the curriculum or intervention compatible with adaptive device equipment, such as screen readers and alternative input devices?
• Is the curriculum or intervention compatible with existing platforms and resources? Can it be supported by existing tech support expertise?
• Does the curriculum or intervention and training model the use of technology as a seamless and expected component of instruction using principles of universal design and as a means to differentiate instruction?
• Once purchased will there be additional costs to sustain the technology? What is the total cost of this technology compared to similar programs?

When choosing a new curriculum or intervention, it is important to first gain an understanding of your district’s need with respect to transition, and then to examine the characteristics of the available curricula you are considering and determine the extent to which these programs meet your needs. When making decisions about transition, it is important to consider whether you are interested in programs and curricula designed to prepare students for post-school education settings, post-school employment, or both. In addition, you will want to determine whether school-based programs that prepare students for transition through simulated activities or programs that encourage community-based participation and involvement are more appropriate for your needs.

Questions about district need:

• What are the biggest challenges that need to be addressed among children transitioning in my district?
• Is there a need to address challenges related to the transition of specific subpopulations (e.g., students with disabilities, minority students, 9th grade students)?
• What are the areas of transition components or domains the district is most interested in (e.g., assessment, preparation for post-secondary education, career awareness, work skill development)?
• What are the skill knowledge and professional development needs of my teachers and staff with respect to providing transition activities?

Questions about program or intervention characteristics:

• Does the curriculum or intervention provide flexible program and curricular options to meet varying student needs?
• Is the goal of the curriculum or intervention to improve post-school outcomes for students?
• Does the curriculum and intervention encourage parent and family involvement?
• Does the curriculum or intervention enhance interagency collaboration with a variety of stakeholders (e.g., vocational rehabilitation, vocational educators, community members)
• Is there evidence that the curriculum or intervention is effective in meeting the needs of the target population?
• Does the curriculum or intervention prepare teachers to assist students in preparing and planning for post-school life?
• Do transition assessments match the students’ post-school ambitions and existing community opportunities?
• Does the curriculum or intervention integrate a broad range of transition skills or target specific skills (e.g., social skills, self-determination, study skills)
• Does the curriculum or intervention address multiple transition area domains (e.g., leisure and recreation, independent living, vocational, post-secondary training)?
• Is the curriculum or intervention appropriate for the learning characteristics of the target individuals, including cultural and linguistic differences?
• Does the program enhance connections between existing subject areas and positive post-school outcomes?
• Does the program or curriculum provide students opportunities to learn about and explore various post-school opportunities? Once purchased will there be additional costs to sustain the program? What is the total cost of this curriculum, intervention, or training compared to similar programs?