Performance Improvement Mapping A Handbook for School Planning Teams February 2006, Version 2 Massachusetts Department of Education School and District Improvement Planning and Support address 350 Main Street, Malden, MA 02148 telephone 781-338-3000 internet www.doe.mass.edu This document was prepared by the Massachusetts Department of Education Dr. David P. Driscoll, Commissioner of Education The Massachusetts Department of Education, an Affirmative Action employer, is committed to ensuring that all of its programs and facilities are accessible to all members of the public. We do not discriminate on the basis of age, color, disability, national origin, race, religion, sex, or sexual orientation. Copyright © 2006 Massachusetts Department of Education Permission is hereby granted to copy any or all parts of this document for non-commercial educational purposes. Please credit the "Massachusetts Department of Education." This document printed on recycled paper 350 Main Street, Malden, Massachusetts 02148-5023 #781-338-3000 Massachusetts Department of Education Internet address: www.doe.mass.edu Contents Performance Improvement Mapping 1 PIM as a continuous improvement process 2 Essential conditions for planning 2 PIM and the school improvement plan 3 Using the PIM process according to a school’s needs 5 Conducting PIM with a trained facilitator 6 Conducting PIM without a trained facilitator 7 The role of the principal in the PIM process 7 The role of the local school district in the PIM process 9 The written plan 11 How to use the PIM Handbook 11 Step Zero: Data organization and preliminary self-assessment 15 Expectations for this step 15 0.1 Establish a school planning team 18 0.2 Schedule the work of the planning team 19 0.3 Collect and prepare student performance data 20 0.4 Prepare disaggregated MCAS data 22 0.5 Graph MCAS item analysis data for the past two years 23 0.6 Conduct a preliminary school self-assessment 26 0.8 Establish group norms and set expectations 27 Checklist 29 Step One: Prioritize the needs of student groups 45 Expectations for this step 46 1.1 Identify which student groups have the lowest performance 48 1.2 Identify any overlap in the composition of these groups 51 1.3 Prioritize which groups will be the focus of the planning process 52 Checklist 56 Step Two: Set student performance goals aligned to adequate yearly progress (AYP) expectations 59 Expectations for this step 60 2.1 Determine the amount of improvement that needs to be made in the performance of identified student groups 61 2.3 Set goals across multiple grades (optional) 64 Checklist 66 Step Three: Define student learning objectives 74 Expectations for this step 75 3.1 Identify MCAS items on which targeted student groups performed poorly 77 3.2 Identify the skills and knowledge that targeted groups have not mastered 81 3.3 Consult other sources of data to verify theories about the skills and knowledge students lack 84 3.4 Set student learning objectives 87 Checklist 89 Step Four: Identify the most significant causes of the weaknesses in students’ knowledge and skills 94 Expectations for this step 95 4.1 Identify possible causes of students' lack of knowledge and skills 96 4.2 Identify aspects of teaching and learning that may affect students’ ability to master the skills and knowledge identified 98 4.3 Determine whether the causes identified pertain specifically to the skills and knowledge articulated in a student learning objective or whether they impact other areas of teaching and learning 101 4.4 Identify causes that affect each content area or student learning overall 103 4.5 Prioritize causes 104 4.6 Try to verify or disprove the possible causes generated 106 Checklist 107 Step Five: Establish improvement objectives 114 Expectations for this step 115 5.1 Select which causes to address in the plan 116 5.3 Set improvement objectives to address the causes of poor student achievement 117 5.4 Incorporate information and data from Steps 1-5 into the written plan 123 Checklist 124 Step Six: Assess the capacity of current strategies to address the improvement objectives 128 Expectations for this step 129 6.1 Assess the implementation and effectiveness of current strategies 130 Checklist 137 Step Seven: Investigate and evaluate possible new strategies to support improvement objectives 139 Expectations for this step 140 7.1 Consider whether the existing strategies the school identified will accomplish the improvement objectives 141 7.2 Investigate possible new strategies and determine their effectiveness 142 7.3 Select strategies to implement 144 7.4 Consider whether the strategies identified will accomplish the goals and objectives 144 Checklist 145 Step Eight: Develop action plans for meeting improvement objectives 148 Expectations for this step 149 8.1 Determine the activities needed to implement and ensure the success of the school’s strategies 149 8.2 Review action plans to ensure they are grounded in the previous work 155 Checklist 156 Step Nine: Establish benchmarks 160 Expectations for this step 161 9.1 Establish implementation benchmarks to measure whether the school is offering its students the learning experience it intended 162 9.2 Establish outcome benchmarks for each improvement objective 166 9.3 Review plans for collecting benchmark data 170 Checklist 172 Step Ten: Set a process and timeline for review of the school’s performance improvement plan 189 Expectations for this step 190 10.1 Determine what role the planning team will play and how it will function during the implementation and revision stage 191 10.2 Understand how to use benchmark data in the context of a continuous improvement cycle 192 10.3 Develop a schedule to review data and revise and expand the plan 195 Checklist 196 Appendix A: Documenting the plan 200 The executive summary 201 The narrative 201 Action plans and Benchmarks 204 Appendix B: Using MCAS item analysis data 206 Import student performance data into Testwiz 206 Generate item analysis reports and item analysis graphs 208 Appendix C: Assessing reading and literacy skills 212 The five components of reading instruction 213 Understanding and using reading assessment data 215 Introduction Performance Improvement Mapping (PIM) is an intensive, data-driven, results-oriented school planning model. It was developed by the Massachusetts Department of Education (MA DOE) to assist low-performing schools in raising student performance in English language arts (ELA) and mathematics. Its integrated steps guide school and district leaders to: • Set student performance goals aligned with state and federal expectations for adequate yearly progress. • Analyze student assessment data to determine student strengths and weaknesses. • Identify the specific skills and knowledge that students lack. • Analyze the root causes of low student performance. • Appraise current practices to determine what is and is not working. • Select new research-based strategies. • Write a prioritized action plan based on the school’s analysis of its needs and programs. • Evaluate the implementation and outcomes of the plan. PIM as a continuous improvement process The PIM process is more than a set of instructions for writing a plan; it is a framework for establishing a continuous improvement process that is focused on increasing student performance. Although the written document guides the school’s improvement efforts, the larger goal is to foster a school culture of ongoing self-analysis and improvement. A school that undertakes the PIM process can hope to achieve the following: • Principals will learn how to involve school staff in making informed decisions on program development. • Teachers will find new ways to interpret data and use it to inform their own teaching. • Communication about teaching and learning will increase at the grade, subject and school level, and evidence will be used as the basis for decision making. • School staff will look for root causes before attempting to solve problems. • Data on program implementation and outcomes will be collected regularly, shared collectively, and used to modify instruction and programming. Essential conditions for planning The way a school approaches the PIM process determines how much it will impact the culture of the school. The following conditions are most conducive to creating whole school change: • School staff must be involved in data analysis and decision making. Schools often choose to have representatives of the staff serve on a school planning team, sharing information and gathering input from the larger staff at critical junctures in the process. • The main focus of the planning process must be to continuously improve instruction by increasing the effectiveness of curricula, student assessment, professional development, teacher supervision and the supports necessary for learning. • Schools should direct their energy primarily toward changing what is within their control. • Any other planning conducted by the school should be integrated with the PIM process. • The planning process should both inform and respond to the district's improvement plan. PIM and the school improvement plan The plan resulting from the PIM process addresses core teaching and learning issues in English language arts (ELA) and mathematics. It requires additional components in order to meet all federal, state, and local expectations of a school improvement plan. For example, some schools include sections on building renovations, family involvement, or school climate in their school improvement plans. Schools receiving Title I Part A funds as Schoolwide programs are required by Federal No Child Left Behind (NCLB) legislation (§1114) must include specific components that may or may not be addressed as a school identifies causes of low student performance and appropriate strategies. IMPORTANT: Schools receiving Title I, Part A funds as Schoolwide programs must include the following components in their school improvement plans: (A) A comprehensive needs assessment of the entire school; (B) Schoolwide reform strategies; (C) Instruction by highly qualified teachers; (D) High-quality and ongoing professional development for teachers, principals, and paraprofessionals; (E) Strategies to attract highly qualified teachers to high-need schools; (F) Strategies to increase parental involvement in accordance with §1118, such as family literacy services; (G) Plans for assisting preschool children in the transition from early childhood programs; (H) Measures to include teachers in the decisions regarding the use of academic assessments...in order to provide information on, and to improve, the achievement of individual students and the overall instructional program; (I) Activities to ensure that students who experience difficulty mastering the proficient or advanced levels of academic achievement standards...be provided with effective, timely, and additional assistance; (J) Coordination and integration of Federal, State, and local services and programs. For more information on Title I requirements, visit the MA DOE Web site at www.doe.mass.edu/nclb/title_i.html. Appendix A articulates a suggested structure and format schools may use; however, MA DOE does not require schools to use a specific template for the written plan that results from the PIM process as long as it captures the essential information described below. • The Executive Summary is a brief (1-2 paragraphs) holistic description of the essential parts of the plan. It describes the school’s major priorities—the big changes and improvements that will be happening in the school—and the rationale for undertaking them. • The Narrative provides the reader with relevant background information about the school, describes the planning process the school used, and supports the rationale underlying the critical decisions the school made. • Action plans are step by step “how to guides” for implementing the strategies the school selected and the activities needed to implement them. Benchmarks describe how the school will monitor the quality of the implementation of its most critical strategies, indicate the tools used for measuring the plan's impact on students’ skills, knowledge and behaviors, and articulate the amount of incremental progress that should be made on student outcomes if the school is on track toward reaching its goal. Using the PIM process according to a school’s needs Schools with very low performance will benefit from conducting the PIM process in its entirety. Schools with more limited needs to raise student performance may use the PIM model more flexibly to improve their existing school improvement process. For these schools, the inquiry methods incorporated in the PIM process can serve as a model for how a thorough planning process may be conducted to produce a comprehensive plan to raise student achievement in specific areas. School and district leaders have inquired about using one or more PIM steps in isolation to strengthen weak links in their process. Because the findings in each PIM step form the basis for the next, the quality of work performed in early steps determines the success of subsequent steps. Therefore, schools should be sure that they have conducted the quality and type of analysis described in all earlier steps before undertaking an individual PIM step. Conducting PIM with a trained facilitator Facilitator: The facilitator will find guidance in this sidebar. The PIM process is designed for use by school planning teams working with an outside facilitator who is rigorously trained in the model. The facilitator’s role is critical in encouraging school staff to challenge their assumptions about student learning and think critically about their own practice. Furthermore, many of the activities are difficult to describe adequately in a text and are best understood through experience, and a trained facilitator understands the level of quality expected for the products of each step. Schools that try to conduct the process using only the handbook may misunderstand or have trouble completing the activities. MA DOE periodically offers facilitator training to district staff whose role is to assist schools in improvement planning. Priority is given to staff from districts identified for improvement or with schools identified for improvement. Some consultant facilitators are available. Contact MA DOE at ata@doe.mass.edu for more information. Conducting PIM without a trained facilitator Because the number of trained facilitators is limited, the PIM Handbook is available online for schools and districts that would like to try the PIM planning methods on their own. However, it is strongly recommended that the principal of the school not serve as the facilitator of the PIM process. The principal has a unique and important role as the leader of the planning team and must be free to participate in discussions in that capacity without the constraints of a facilitator (who must remain neutral). Furthermore, while analyzing necessary changes to the status quo of a school, an external facilitator may be able to ask more pointed questions about how things are currently done and why. Schools that use the handbook without a trained facilitator should consider identifying a neutral person with facilitation skills to keep the planning team on task and set a process for making decisions and resolving conflicts. Before beginning, the principal and facilitator should read the entire handbook so they will understand the process as a whole and how each step fits in. The logic model underlying the PIM process relies on the close connections between the steps. It is important to understand the complete process even if the school chooses to use only some of the PIM activities, because the facilitator needs to understand what the outputs from previous steps would be in order to assemble the right information as inputs into the current activity. The role of the principal in the PIM process Because he or she is ultimately accountable for the performance of the adults and students in the school, the principal is the leader of the planning team during the development of the written plan, and more importantly, the person with overall responsibility for implementing the plan. It is critical, then, that the principal is oriented to the process and the anticipated outcomes by someone in an authority role in the district. The conversation should include any potential roadblocks that might emerge, the selection of the team, and when and how the district will support the team through lending content and pedagogical expertise, securing time for the team to meet, etc. The principal and the facilitator have specific—and distinct—roles throughout the development of the plan. These individuals should meet before the first meeting of the planning team to clarify the roles and responsibilities of each. Roles and responsibilities of the principal and facilitator The principal... Identifies and recruits members of the planning team and makes sure that members attend and actively participate in all meetings. Directs the process by setting a vision for the plan. Constantly reinforces the importance of PIM through his/her words and actions. Is comfortable ceding the floor to other team members whose views and perspectives might differ from his/her own. Creates time for the planning activities. Does not claim to know all of the answers; instead, requests ideas from all group members, particularly when groupmembers have special skills and knowledge that they can contribute. Manages the communication and dissemination of the work of the planning team to the rest of the school and brings the findings of the team to the districtlevel as needed. Organizes and monitors the implementation of the plan and allocates the necessary time and human, material, and fiscal resources to embed the plan into the daily work of the school. Holds him/herself and all members of the school staff accountable for the success of the plan. May consult with the school on the collection of benchmark data and the ongoing evaluation and revision of the plan. The facilitator... Identifies rotating roles for members (e.g. time keeper, recorder, etc.). Provides an overview of the process, theexpected products and the time and effort that will be required. Ensures that uncomfortable issues raised during planning meetings are keptconfidential. Helps the group establish written norms and maintains a comfortable environment where risk-taking andreflection are encouraged and supported. Ensures that follow-up work is completed between team meetings and that documents and materials are organizedand readily accessible. Guides the activities at each step and makes sure the resulting products represent that the step was done thoroughly. Keeps discussion focused on the task at hand. Makes sure the team members base their views on data and evidence rather than perceptions. Helps the team document their discussions and decisions at each step. Helps the team plan and createdocuments and presentations summarizing the outcomes of the process. The role of the local school district in the PIM process Improvement planning should occur both at the district level and in schools; these processes should inform each other both in the analyses of needs and in the identification of improvement initiatives. Most schools are part of a larger school system and should enter the PIM process with support and direction from the district in both the planning and implementation phases. Districts should support schools in the following ways: • Handling, or ensuring, facilitation of the process • Providing data and time for school planning • Taking part in causal analysis discussions about the results of district reviews of schools and classrooms, curriculum alignment and implementation, professional development needs and the allocation and availability of resources • Providing content/pedagogical expertise in steps 3, 5, 6, 7 • Providing support for the implementation of identified improvement initiatives • Providing oversight of and problem-solving around the implementation of improvement initiatives Districts may choose to involve multiple schools in the PIM process as part of a larger strategy for system-wide improvement focused on its schools with lower performance. In engaging more than one school in the process, the district should: • Identify trends in student performance (such as the needs of student subgroups) that may not be apparent in individual schools • Better understand, evaluate, and address gaps in curriculum alignment and the implementation of key district-wide initiatives • Develop district-wide plans to address needs identified across multiple schools • Make decisions about the allocation of fiscal, material, human resources, and professional development, to schools based on identified needs The written plan School planning teams should note that the worksheets throughout the handbook are tools used to capture information during the work sessions. They are not effective vehicles for communicating the final, overall plan to those who have not participated in the process, such as other teachers, parents, district staff, school councils, school committee members, etc. At this time MA DOE does not require schools to use a specific template for the written plan that results from the PIM process. However, many schools have asked for guidance regarding the plan's structure and contents. Appendix A contains one possible format schools may use. Schools should first follow any instructions and/or templates for preparing the plan required by their district. How to use the PIM Handbook Facilitator: Work.sheets may be projected with an LCD projector or duplicated on poster-size paper. While using an LCD projector saves time, posters let planning teams view multiple worksheets at once, including those from previous steps. The PIM Handbook contains materials and guidance both for participants and the facilitator. Each activity is described so that participants will understand what they are supposed to do and what the end product should be. Guiding Questions at the beginning of each section frame the overall task the school planning team will undertake in that section. Discussion Questions in the text of each section, lead the team through the conversations they need to have to accomplish the work. Worksheets at the end of each step guide participants through the process and help organize the products of the team's work. Many of the worksheets contain a Rationale Box where teams record key points from their conversations. Notes from the Rational Boxes will be useful when the team writes the plan. (For guidance on documenting the plan, see Appendix A.) The Handbook is available for download from MA DOE Web site in Portable Document Format (PDF). Should a school decide to undertake the PIM process, it is recommended that the planning team print the entire handbook. To view or print only certain sections of the handbook (e.g., worksheets or specific steps) use the Adobe Bookmark feature, which displays a list of sections on the left-hand side of the screen. To view Bookmarks, select View from an Adobe Reader or Acrobat menu and then select Bookmarks. Navigate the handbook from the Index by clicking on an Index title. Electronic copies of the worksheets are available on the Web site in Microsoft Word and Excel. The PIM Handbook is periodically revised and updated. The date and version number of the handbook are listed at the bottom of each page. The beginning of each step contains a list of the worksheets and other materials and equipment needed to complete the step, as well as useful resources. Worksheets The worksheets used in each step are listed hereand are physically located at the end of each stepin the printed document. When viewed online in PDF format, click on the name of a worksheet to open it in its native format (such as Microsoft Word or Excel). Other Materials and Equipment This section lists some of the materials and equipment that the school planning team will need in advance to be prepared to conduct the activities in the step (e.g., overhead projector, LCD projector, etc.). Resources This section contains references to resources that are external to the handbook such as web sites or other documentation the team may find helpful in completing a particular step. Step 0: Data organization and preliminary self-assessment Facilitator: Meet with the principal and check that the school planning team has assembled its data in the appropriate format before work begins on Step 1. It will be helpful to become familiar with the data trends before the first work session. Expectations for this step • The principal will establish the school planning team and set a schedule for their meetings. • A team member (or a person identified by the team) will take an inventory of the student data for use in Steps 1-4 and organize it in a way that will be accessible to the planning team. • Team members (and optionally the entire school) will complete a school self-assessment to identify major issues in need of attention based on perception data. • The planning team will establish norms for working together, problem-solving and managing conflicts. The order of the activities in Step 0 may vary depending on the needs of the school, so long as each is completed before the team begins Step 1. For example, if the team elects to complete the school self-assessment as a group rather than individually, it may be necessary to establish group norms before that activity, whereas if they complete the school self-assessment individually, they can develop group norms at the beginning of the first Step 1 meeting. Worksheets 0-A Inventory and preparation of student performance data (ELA, mathematics) (Word) 0-B MCAS data disaggregated by student group (Excel) 0-C Cross-tabulation of student groups (Excel) 0-D School self-assessment (Excel) Other Materials and Equipment Item analysis templates (available online for 20012002 on the MA DOE Web site; accessible with TestWiz software for 2003 and later): www.doe.mass.edu/sdi/pim/steps.html?step=0 Data sources listed on Worksheet 0-A, including student-level accountability and MCAS data (available from MA DOE Security Portal) Computer with the latest version of TestWiz and spreadsheet software (e.g., Microsoft Excel, Lotus 1-2-3); optional database/analytical software (e.g., FoxPro, Access, SPSS, SAS or STATA) 3-ring binders for organizing data, overhead transparencies, and large-size poster paper ornewsprint Resources PIM online: www.doe.mass.edu/sdi/pim/ TestWiz: www.testwiz.com/ Massachusetts School and District Accountability System: www.doe.mass.edu/sda/ Grade 3 MCAS Reading cut scores and equivalent performance levels: www.doe.mass.edu/sdi/pim/g3mcasread_cutscores.pdf 0.1 Establish a school planning team Facilitator: Meet with the principal to discuss the composition of the planning team and to clarify roles in the planning process. (See "The role of the principal in the PIM process" on page 7.) The size of a school planning team will vary, but should be small enough to allow the group to work productively. The team should include the principal, teachers, lead teachers or other experts in the subject area(s) to be addressed, and staff who work with special education and limited English proficient students if applicable to the school. It is also very helpful to include a district level staff person such as a curriculum coordinator. Team members should have instructional expertise and knowledge of the school’s curriculum and instructional programs across the grade levels. It is also helpful to include members who are familiar with technology, data analysis, curriculum development, summative and formative assessments, student support services, and specific programs or initiatives underway in the school or district (e.g., Special Education, Sheltered English Immersion, Reading First, Title I, etc.). The team will do the labor-intensive planning work, but they must seek input and feedback from other teachers throughout the PIM process. Therefore, it is important that the team include teachers who can educate other staff about the PIM process, communicate the decisions the team has made, and run feedback sessions. Because it is difficult to introduce a new member to the team midway through the process, team members should commit to participating on the team for the full planning process (Steps 1-9). Some schools wish to have parents involved in the process. During the PIM process, teachers are asked to be self-reflective and consider ways that teaching practices in the school can be improved. In order to create an environment where teachers are comfortable suggesting areas for improvement, it is not recommended that parents serve on the team. However, parents can be involved in providing input and feedback to the team through the school council or special parent meetings. 0.2 Schedule the work of the planning team Once the school planning team is formed, the principal will need to establish a work schedule that includes team meetings as well as meetings with the staff at large, and meetings with parents and other constituencies if desired. Schools may decide to establish separate planning teams by subject (i.e., for English language arts and mathematics). Schools should not, however, have separate teams for student subgroups (unless the subgroup is in a substantially separate program) as this could hinder the integration of instructional strategies for those students within the established program. Schools that decide to form subject-specific teams should establish a clear method of communication between the teams, as individual teams will uncover issues that affect all content areas and must be addressed through common strategies. It is difficult to estimate the amount of time needed to undergo the PIM process because the number of priority areas addressed (e.g., student groups, grade levels, etc.) will vary by school. School planning teams should also consider the following: • If possible, arrange for the team to meet away from the school, or when school is not in session. It is important for the team to work in an environment that is free of distractions and interruptions, a problem that particularly plagues school principals. • Plan to spend a minimum of one half day (or four hours) of uninterrupted work when the team learns to conduct a step for the first time. This is especially important in Step 4, a step where conflict is common and the team will need time to reach a point of resolution. • Consider whether existing time (e.g., faculty meetings, curriculum meetings, professional development days, common planning time, etc.) can be restructured to conduct some of the work. • The amount of paperwork generated in the PIM process can be cumbersome. A team member should be assigned to keep track of materials and information between team meetings. This person will be responsible for saving and organizing computer files and organizing and transcribing worksheet posters. • Build time into the schedule to periodically disseminate the work of the team to the rest of the school for their feedback. 0.3 Collect and prepare student performance data Facilitator: Copies Although MCAS analysis is used as the entry point for the PIM process, of the data should the conclusions the team draws should be verified and refined by other be organized in a data sources. Worksheet 0-A (Inventory and preparation of student three-ring binder for each member of the performance data) organizes the data sources available in the school. Because the team cannot work effectively with raw data, use the worksheet to plan any necessary organization and summarization of the data. (Note that an additional worksheet is provided for high schools and vocational/technical schools.) 1. Before completing Worksheet 0-A, consider the following questions: • In what forms does the school and district gather data on student performance? • In what ways do classroom teachers record student progress? • What formative assessments does the school use (e.g., DIBELS, GRADE, Yearly Progress Pro, Stanford Learning First, etc.)? • What evidence do teachers and school leadership find helpful in making decisions about instruction, curriculum, and assessment? 2. Record the following on Worksheet 0-A for each content area: • Data type: List the specific data sources that will be assembled for the PIM process. • Grade(s): Indicate the grade level(s) associated with each data source. • Data preparation required: Describe any disaggregation, summarization, or charting needed to present the data in a usable way. If possible, this data should be disaggregated by student groups, including: limited English proficient, Special Education status, low income, first language not English, and racial groups. • Person responsible and date due: Indicate the person(s) responsible for preparing the data and the deadline. 3. Once the data on the list is prepared, organize it in a three-ring binder for the planning team. 0.4 Prepare disaggregated MCAS data NOTE: Student-Student-level data files must be downloaded from the MA DOE Security Portal by authorized personnel in each district. For more information, contact the district’s directory administrator. A complete list of administrators is available at www.doe.mass.edu/infoservices/data/diradmin/list.asp. 1. Obtain the school’s MCAS data. Accountability data files contain MCAS and MCAS-Alt data from the MA DOE Security Portal by along with identifying information for each student tested in the authorized school. Further student level data, such as a student’s Special personnel in each Education placement or first language, are contained in thedistrict. For more school’s Student Information Management System (SIMS) data information, contact files. Both files are available through the MA DOE Security the district’s directory Portal. 2. Complete one copy of Worksheet 0-B (MCAS data disaggregated administrators is by student group) for each grade tested in each content area. • Worksheet 0-B is a Microsoft Excel file that contains tables for Cycle II (2001 and 2002), Cycle III (2003 and 2004) and Cycle IV (2005 and 2006). Complete the tables for the previous two cycles (3 years if it is currently a mid-cycle; 4 years if a cycle has just ended). Complete tables for additional years if desired. • When the number of students in each performance category is entered, the table automatically calculates the percentage of students in that category as well as the composite performance index (CPI) for the students in that student group and grade. • Combine data for students who took the MCAS and MCAS-Alt or consider the MCAS-Alt data separately. • Be cautious of reporting data for student groups smaller than ten students, as the reports may expose students’ personal information. 3. Using Worksheet 0-C (Cross-tabulation of student groups), create a cross-tabulation of each group by listing the number of students represented in both the horizontal and vertical categories. 4. Place copies of the completed Worksheet 0-B and Worksheet 0-C in three-ring binders for each member of the planning team and copy a set onto transparencies for the facilitator. 0.5 Graph MCAS item analysis data for the past two years TestWiz is a Windows-based software application available to all Massachusetts educators to import, disaggregate, and present student-level data from a number of assessments and data sources, including MCAS, SIMS, DIBELS, GRADE, LAS R/W, and MELA-O. For instructions on using TestWiz to create item analysis graphs, see Appendix B. 1. Create MCAS item analysis graphs depicting student performance on each item sorted by the percent of students across the state who answered correctly. • The x-axis lists the question number and indicates the question type and reporting category for each question. • The y-axis represents the percent correct for each item. • The data should be organized in descending order of the State’s percent correct.(In other words, from left to right the graph will show the questions the most students across the state answered correctly to those the fewest students across the state answered correctly.) • Indicate the student group, grade and number of students tested on each graph. IMPORTANT: The state percent correct serves as a proxy for the relative difficulty of the test items and does not represent the expectation that students reach a state norm. MCAS is a criterion-referenced test that measures the progress of all students towards proficiency on the Curriculum Frameworks. Schools may also rank order the data using “State Proficient” as the comparison group, which would provide a profile of the responses of students deemed to have met the state standard. 2. Prepare a set of graphs for each grade tested for the previous two years. The set should include separate graphs for the following student groups in the school: • All students who scored Warning/Failing or Needs Improvement • Students with limited English proficiency • Students who are not limited English proficient • Students who are formerly limited English proficient • Students with special needs • Students without special needs • Students who receive free lunch (“low income”) • Students who do not receive free lunch (“not low income”) 3. Place copies of graphs into the data binder(s) to be used by the school planning team and make a complete set of graphs copied on to transparencies. 0.6 Conduct a preliminary school self-assessment Facilitator: If the school has had a review or audit by an outside entity, have school staff consider that feedback when recording their responses. NOTE: The team may also wish to use any existing self-assessments or surveys. High schools may wish to complete the self-assessment by academic department, program area, or shop area. The PIM process uses a data-driven investigation method to determine the causes of students’ low performance. However, recording team members’ perceptions ahead of time will highlight possible areas of weakness that should be investigated. It is also important to note the assumptions team members bring to the process that may be proved or disproved during the investigation process. Using Worksheet 0-D (School self-assessment) record preliminary assumptions about the school’s strengths and weaknesses and indicate any areas that may need special attention during the PIM process. Members of the planning team may either complete the self-assessment as a group or fill it out individually and meet to discuss their answers. Additionally, schools may decide to ask all staff to complete the self-assessment. 1. Complete Worksheet 0-D for both English language arts and mathematics. (Answers on some items will overlap content areas.) 2. Read each statement and consider how well it represents practices in the school and how consistently the statement applies to all staff and students. Record the strengths and weaknesses of the school as compared to the standard described in the statement. 3. In the last column, check the box if possible weaknesses exist that should be explored during the PIM process. 0.7 Establish group norms and set expectations 1. When the planning team assembles for the first time, explain the purpose of the planning process and what it will entail, and talk about the kinds of discussions the team will have. • The goal of the PIM process is to raise student performance in ELA and math by embedding a culture of ongoing investigation and improvement into the everyday work of the school. • PIM provides a context for school planning teams to have sustained, reflective discussions about their practice that are grounded in evidence and data. • These discussions can be difficult when participants are asked to identify weaknesses in school practices and make high stakes decisions about the how the school should change. Facilitator: Some tension is expected and helpful. Do not be tempted to use techniques that prematurely smooth-over conflict, such as trading, voting, coin-flips, or bargaining. Instead, require that each individual take responsibility for hearing others and being heard and ground their assertions in evidence and data. 2. Discuss the various methods for group decision-making. Emphasize that, to the extent practiable, the planning team should strive for consensus throughout the process. • Consensus promotes the growth of a professional community. Through consensus, teams work through differences to reach a mutually satisfactory position that is in the best interest of students and supported by the data. • Consensus does not mean that everyone thinks that the decision made is necessarily the best one possible or even that they are sure it will work. It does mean that all points of view were considered. • While the team should be committed to consensus, it may sometimes need to utilize other forms of decision making (e.g., compromise, majority rules, or the principal makes the final decision). 3. Develop a written set of group norms that will be posted at each meeting. Many facilitators have a preferred norm-setting activity. Or, the facilitator may simply ask, “What do you need from your fellow team members in order to work together effectively?” Norms often fall into two categories: • Procedural norms: These norms help participants stay focused on the work, ensuring each step of the process is done with fidelity and the available time is used effectively. Examples of procedural norms may include: “One person speaks at a time”, “Show up on time to all meetings”, “No mobile phones”, etc. • Interpersonal norms: These norms help ensure that participants feel comfortable discussing difficult issues. Examples of interpersonal norms may include: “Use open, honest, respectful communication”, “Keep conversations confidential”, “Keep in mind that our goal is to help students”, “Don’t change your mind to avoid conflict”, etc. Checklist Before moving on to Step 1, make the team has: • Formed a school planning team and set a planning schedule. • Completed Worksheet 0-A (Inventory and preparation of student performance data) for English language arts and mathematics. • Completed Worksheet 0-B (MCAS data disaggregated by student group). • Completed Worksheet 0-C (Cross-tabulation of student groups). • Created MCAS item analysis graphs and reports for the past two years. • Completed Worksheet 0-D (School self-assessment) and considered any existing school/district audits or surveys. • Established group norms for conducting the work of the team. • Assembled copies of the completed worksheets and other data for each member of the team and the necessary transparencies for the facilitator into data binders. Step 1: Prioritize the needs of student groups Facilitator: This step requires the team to reconcile philosophical and ethical dilemmas regarding which student groups and areas of weakness to target. The facilitator’s role is to resolve conflicting opinions in a way that allows all individuals to feel heard. In Step 1 the planning team will review data on the makeup of the student body and determine which students are most in need of different and/or additional instruction and support to meet state learning standards. The team will then prioritize which groups of students it will focus on in the current planning process. It may decide that the needs of the whole student body are great enough that the plan should target all students as a group. If so, next determine whether any subgroups will need additional assistance beyond what will be planned for all students. In Step 2 the team will set goals for the groups it chose to target. Before analyzing MCAS data, it is important to understand how it is used to measure school performance. The team should review the most current School Leaders’ Guide to Accountability and AYP Reports (www.doe.mass.edu/sda/ayp/) before beginning Step 1. Expectations for this step • The team will develop a common understanding of (1) the makeup of the student body, (2) the general performance of all students and (3) the performance of individual subgroups as defined by the No Child Left Behind Act (NCLB). • Team members will prioritize the student subgroups they will focus on in the current planning process. • Team members will begin to develop a better understanding of data analysis methods. • Principals will have the opportunity to establish their role in the process and to practice a collaborative leadership style with the team. Facilitator: Introduce the idea of the “waiting room”: A flip chart is used to record interesting insights and possible solutions as well as concerns that cannot be addressed immediately. The waiting room captures this information while keeping everyone focused on issues related to the current step. Worksheets 1-A Identifying the lowest performing groups(Word) 1-B Targeted student groups (Word) Other Materials and Equipment Copies of completed Step 0 worksheets for all team members Copies of Worksheets 0-B and 0-C on transparency film Copies of the school’s AYP Report Overhead projector and transparency markers Resources Current School Leaders’ Guide to Accountabilityand AYP Reports from MA DOE Web site: www.doe.mass.edu/sda/ayp/ 1.1 Identify which student groups have the lowest performance Facilitator: Looking at the data on an overhead projector helps focus the team. It is also helpful if team members have their own copies of the data. Guiding Question: Which student groups have the lowest performance? 1. Working with one content area at a time, look at the school's Adequate Yearly Progress (AYP) Report. Ask: • Are there any subgroups that did not make AYP? • If so, for what reason? • Did they fail to make AYP because they did not reach the state performance target or their improvement target, and/or • Did they fail to make AYP for other reasons such as their Participation rate or Attendance/Competency Determination (CD) rate? 2. If a subgroup did not make AYP based on Performance or Improvement, record the subgroup on Worksheet 1-A (Identifying the lowest achieving groups). 3. Look at Worksheet 0-B (MCAS data disaggregated by student group) for the current cycle. Discuss the relative performance of the subgroups of students at the school. Ask: • Which groups have the highest percentage of students in warning? • Which groups have the lowest percentage of students in proficient and advanced? • Which groups have the lowest Proficiency Index or Composite Performance Index? 4. Based on the discussion, record the lowest performing groups on Worksheet 1-A (if not already listed). 5. For each group listed on Worksheet 1-A, discuss the following: • How many students are represented in this group? • What is the difference between the Proficiency Index or Composite Performance Index of this group and that of the students who are not in this group? • How does the Proficiency Index or Composite Performance Index of this group compare to the current state performance target (Cycle IV: mathematics = 68.7 / ELA = 80.5)? 6. Record answers on Worksheet 1-A. NOTE: When comparing data for LEP students across years, be aware that the definition of students included in this group has changed several times. Please consult the “School Leaders’ Guide to the Cycle III Accountability and AYP Reports” at www.doe.mass.edu/sda/ayp. 7. Look at copies of Worksheet 0-B for the previous cycle to identify multiyear trends for subgroups. Ask: • Do similar patterns appear over previous years for the identified subgroups? • Which subgroups appear to have the lowest performance across the last few years? • Are there groups of students whose performance has remained flat or declined? 8. Add any newly identified subgroups to your list and make any relevant notes about subgroup performance in the previous cycle on Worksheet 1-A. 1.2 Identify any overlap in the composition of these groups Guiding Question: Is there significant overlap in the composition of the selected groups? 1. Looking at Worksheet 0-C (Cross-tabulation of student groups), consider the overlap between each targeted group and other subgroups. Ask: • What percent of students in the target group overlap with the other subgroup? • What percent of students who are NOT in the target group overlap with the other subgroup? • Are these percents comparable? Example: Is the percentage of LEP students who are Low Income similar to the percentage of Not LEP who are low income? 2. Record answers on Worksheet 1-A (Identifying the lowest achieving groups). 1.3 Prioritize which groups will be the focus of the planning process Guiding Question: Should the whole student population be a focus of the planning process, and/or should specific subgroups be targeted? In some cases the need to target all students may already be evident. If so proceed to Step 1.3.3. Other schools should consider the performance of their higher achieving groups to see if they require attention. 1. Looking at the data on the higher performing groups, ask: • Which student groups have the highest Proficiency Index or Composite Performance Index? • How many students are represented in the higher performing groups? • What percentage of these higher performing groups scored Proficient or Advanced? • How does the Proficiency Index or Composite Performance Index of this group compare to the current state performance target (Cycle IV: mathematics = 68.7 / ELA = 80.5)? 2. Record the answers on the bottom of Worksheet 1-A (Identifying the lowest performing groups). 3. Looking at Worksheets 1-A consider whether the plan should address the performance of the whole student population. • Given the performance of students in the higher performing groups, is the performance of the overall student population low enough to be a priority in the planning process at this time? If so, should students be targeted in the aggregate or in separate groups? • Do most of the students in the school fall into one of the low performing subgroups? If so, consider targeting the whole student population rather than making a distinction between students who are and are not in that subgroup. • Do the students in the higher and lower achieving groups participate in the same programs and receive the same curriculum and instruction? If not, consider targeting the groups separately. 4. If targeting all students in the aggregate, record “all students” as the first group on Worksheet 1-B (Targeted student groups). 5. Again, look at Worksheets 1-A and consider which subgroups should be targeted in the plan. • Which student groups have the lowest and/or declining performance? • Which student groups have the highest number of students? • Given the overlap between certain groups, could the needs of most of the students in two or more groups be addressed by targeting one group? • If targeting all students, which groups will likely need support in addition to what will be provided for the whole group? 6. Based on the answers to these questions, select up to three groups per content area to target in the current plan (including “all students” if applicable). • When deciding how many groups to target in the plan, remember that PIM is an ongoing process, and groups that are not targeted in the current plan can be targeted in future years. If the performance of all students is very low, it may make sense to make large scale changes that affect all students before targeting individual student groups. • Although students in some grades do not take the MCAS, they may be included by targeting a grade span (e.g., K-3), as the school’s improvement strategies will likely be designed for more grades than just the tested grade. 7. Record the selected groups on Worksheet 1-B, and record any key points from the discussion in the rationale box. Checklist Before moving on to Step 2, make sure the team has done the following for English language arts and mathematics: • Identified student groups with the lowest or declining performance using the school’s AYP report and Worksheet 0-B (MCAS data disaggregated by student group) • Used Worksheet 0-C (Cross-tabulation of student groups) to identify any overlap in the identified low performing groups • Considered the performance of the higher performing groups to determine whether all students at the school should be targeted in the plan • Decided to target the aggregate student population and identified up to two subgroups expected to need additional support, or chosen not to target the aggregate student population and selected up to three subgroups to target in the plan Step 2: Set student performance goals aligned to adequate yearly progress (AYP) expectations Facilitator: As in Step 1, philosophical and ethical dilemmas may arise regarding whether to target students at specific performance levels versus all students. In Step 2, the planning team will calculate an improvement target for each of the student groups it selected using the Proficiency Index (PI), Composite Performance Index (CPI) or the MCAS-Alt Index. This target will generate student performance goals. While the accountability system assigns targets for student groups schoolwide, the team will have the option to calculate a target for student groups at specific grade levels. Expectations for this step Facilitator: Teams may feel overwhelmed and discouraged by the amount of improvement they are expected to make and may be tempted to set goals lower than their improvement targets. Remind teams that whether or not they make their improvement targets in the next cycle, they need to set their sights high in order to drive teaching and learning to the highest levels possible. • The planning team will set student performance goals that meet state and federal expectations for improvement in student performance. • Team members will develop a better understanding of the gains needed to reach improvement targets set under the state accountability system. • Team members will identify which levels of student performance to target. Worksheets 2-A Student performance goal spreadsheet (Excel) 2-B Integrated goal spreadsheet (Excel) (Optional) Other Materials and Equipment Completed Steps 0-1 worksheets, LCD Projector Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Resources School Leaders’ Guide to the Cycle III Accountability and AYP Reports: www.doe.mass.edu/sda/ayp Massachusetts School and District Accountability System: www.doe.mass.edu/sda/ 2.1 Determine the amount of improvement that needs to be made in the performance of identified student groups Facilitator: To save time, consider pre-populating the worksheet by following steps 2.1.1-2.1.3 ahead of time. Guiding Question: How much would student performance have to time, consider pre-increase in the coming years in order for all students to achieve populating the proficieccy by 2014? For each group the team chose to target, the team will use Worksheet 2-A (Student performance goal spreadsheet) to determine the improvement targets for the next two years. The team will then identify a hypothetical distribution of scores that would allow the school to reach these targets. IMPORTANT: Worksheet 2-A can be used to calculate the PI, CPI, or MCAS-Alt Index depending on the information entered. Data may be combined for one or multiple grade levels. 1. Open Worksheet 2-A in Microsoft Excel and save it to the computer. In Part 1, type the name of the school, the first targeted student group, and the relevant grade level(s) in the spaces provided at the top. 2. In Part 2, estimate the likely number of students in the student group who will take the MCAS and/or MCAS-Alt in the next cycle (or the remainder of the current cycle if applicable). NOTE: Find the number of students in each performance level on copies of Worksheet 0-B. Enter the number of students into the spreadsheet, not the percentage of students. 3. In Part 3, enter the distribution of students tested in the previous cycle by performance level. Enter the number of students in each performance level for the years in the previous cycle (and the mid-cycle if applicable). The spreadsheet will automatically calculate the next improvement target for the group. 4. Generate a distribution of student performance that would allow the group to achieve its improvement target. • Using the distribution of scores over the past year as a starting point, think about how scores would have to improve in order to reach the next improvement target. Enter some hypothetical numbers in the performance levels in the column for the upcoming year. As the team explores various scenarios, ask: • What is a reasonable expectation for an increase or decrease in the number of students scoring at each level? • Will those changes allow the school to reach its improvement target? • Should the team focus improvement activities equally across performance levels, or should efforts be focused on one particular level? 5. When an appropriate distribution of scores for the coming year has been established, repeat the process by projecting distributions of scores for the remaining years. 6. Repeat steps 1-5 above for any remaining targeted subgroups, grade levels, or subjects. • Use the “Save As” function in Microsoft Excel to save additional versions of Worksheet 2-A. • If developing goals for the aggregate student population and for targeted subgroups, make sure that the distribution of students for individual subgroups are consistent with the aggregate. For example, if the school intends to move all students out of Warning by 2007, then the projected distribution for student subgroups should not include any student scoring Warning. NOTE: Worksheet 2-A is pre-formatted to print the Student Performance Goals chart. 7. Copy and paste the chart entitled “Student Performance Goals” for each student group identified in Step 1 into the School Improvement Plan Template, PIM Access Database, or an alternate format. 2.3 Set goals across multiple grades (optional) Facilitator: Do not introduce this worksheet until the school planning team thoroughly understands the CPI, PI, and MCAS-Alt Index, and is adept at using Worksheet 2-A. Once the team has learned to use Worksheet 2-A (Student performance goal spreadsheet), it may use Worksheet 2-B (Integrated goal spreadsheet) to examine student performance at multiple grade levels. For example, a K-8 school may wish to: • develop individual goals for ELA in Grades 3, 4, and 7 and also a cumulative goal for combined grades, • set CPI targets that approximate those reported on the school’s AYP Report, and • target specific grade(s) in the school that may require particular attention in the planning process. 1. Open Worksheet 2-B in Microsoft Excel and save it to the computer. Select either the “ELA” or the “Math” tab. 2. Parts 1 and 2: Enter the number of students scoring at each performance level in Cycle III (2003 and 2004) for every grade tested in the school. 3. Part 3: Enter the likely number of students in the student group at each grade level who will take the MCAS and MCAS-Alt in future years. 4. Part 4: Taking one grade level at a time, enter a distribution of students by performance level that meets or exceeds the target for each grade level. 5. Part 5: Verify the goals that were set for individual grade levels and for the content area as a whole. • For each grade level tested in the school, make sure that the goal (“Your Goal”) meets or exceeds the target calculated by the spreadsheet for each year (“CPI Target”). • The spreadsheet automatically calculates whether the group is on target for each grade level (horizontally) and for each year (vertically) by generating a “Yes” (black text) or a “No” (red text) in the line next to the text “On Target?”. • A “No” associated with the content area as a whole or a given year or grade level means that the school must adjust its projected goals in order for the group to be on target for making AYP. • A CPI Goal with a strikethrough means that the school has not distributed all of the students estimated to take the MCAS or MCAS-Alt in future years. Checklist Before moving on to Step 3, make sure to have: • Learned to use Worksheet 2-A (Student performance goal spreadsheet) • Used Worksheet 2-A to determine how much student performance must increase in order for the school to reach its improvement targets • Developed student performance goals for each student group identified in Step 1 and documented those goals in the School Improvement Plan Template, PIM Access Database, or an alternate format Step 3: Define student learning objectives Facilitator: Revisit the steps of the PIM process and show the team where they will have the opportunity to discuss causes and Worksheet 0-D (Step 4) and solutions (Step 5). In Step 3 the planning team will identify specific skills and knowledge that students have not mastered and define objectives for student learning based on those findings. The team will first examine the MCAS item analysis graphs prepared in Step 0, and look for patterns indicating the skills students may lack. As the team develops theories, it will also consider whether a lack of basic reading or literacy skills may have prevented students from engaging with the MCAS questions. Finally, the team will consult data from other assessments to confirm the skills and knowledge students have not mastered. Note that the test items on item analysis graphs are arranged according to the scores of students statewide in order to help the team judge the relative difficulty of each question. The line representing the state percent correct should not be viewed as the standard the school is trying to meet, but rather as an indication of each question's difficulty. Expectations for this step • Team members will increase their familiarity with MCAS questions and the skills and knowledge required to answer them correctly. • Team members will increase their understanding of students’ performance on individual MCAS items relative to the difficulty of the question. • Team members will become more familiar with the variety of data available to the school, including data from diagnostic reading assessments and the MEPA, and how to use it to expand on the information MCAS provides about students' skills and knowledge. • The planning team will be able to reflect on whether the assessments the school conducts provide sufficient information on students' skills and knowledge. • The planning team will create a set of student learning objectives outlining the skills and knowledge that most need to be addressed in order to raise students’ performance. Worksheets 3-A Identification of MCAS items (Word) 3-B Analysis of MCAS questions (Word) 3-C Skills and knowledge students lack (Word) 3-D Student learning objectives (Word) Other Materials and Equipment Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Appendix C: Assessing reading and literacy skills Copies of completed item analysis graphs(prepared in Step 0) on transparency film Copies of item analysis reports that include the percent of students giving each response Released MCAS items from MA DOE Web site: www.doe.mass.edu/mcas/testitems.html Copies of the state Curriculum Frameworks for English language arts and mathematics from MA DOE Web site: www.doe.mass.edu/frameworks/current.html Data listed on Worksheet 0-A, overhead projector,and transparency markers Resources Anderson, L., and Krathwohl, D., Eds. (2001). A taxonomy for learning, teaching, and assessing: arevision of Bloom’s taxonomy of educational objectives. New York: Addison Wesley Longman, Inc. 3.1 Identify MCAS items on which targeted student groups performed poorly Guiding Question: On which items did students perform most poorly given the difficulty of the question? Facilitator: Why show data for subgroups such as special education students against data for all students statewide instead of students statewide in that subgroup? The statewide data is not meant to be viewed as a comparison group. It is only an indicator of the relative difficulty of the questions based on statewide results. The team will use the item analysis graphs generated in Step 0 to identify the test items on which targeted student groups performed poorly. 1. Select the item analysis graphs for targeted student groups • Select the graphs for targeted student subgroups and the graphs for the students not in those subgroups. • If targeting all students, the team may use the graph for the aggregate population, or, if the performance of the aggregate is close to the state average, it may be more informative to use the graph for all students who scored warning/failing and needs improvement, as this is the group whose performance needs to improve. • Be aware of the number of students each group represents. Be cautious of drawing conclusions for groups of fewer than 20 students. Also, if the students in the warning/failing and needs improvement group is close to the number in the targeted subgroups, these graphs may represent the same students. Facilitator: The team will repeat this step using an additional year of MCAS data. Consider having two teams perform the work in parallel, and then meet to discuss their findings. 2. For each targeted student group, look at the appropriate item analysis graph. Identify eight to ten questions on which those students’ scores are lowest relative to the difficulty of the question and circle them on the graph. Select questions based on the following: • Questions where the school’s percent correct is lowest relative to the state percent correct • Questions on which the state percent correct is relatively high and the school’s students performance is significantly lower • Questions with the greatest distances between the performance of the target group and another student group at the school IMPORTANT: The state percent correct serves as a proxy for the relative difficulty of the test items and does not represent the expectation that students reach a state norm. MCAS is a criterion-referenced test that measures the progress of all students towards proficiency on the Curriculum Frameworks. 3. On Worksheet 3-A (Identification of MCAS items), record a check mark in the appropriate column for each question identified. 4. For each subgroup, using item analysis graphs for the students who are not in that group, use the same process to select questions that were problematic for those students. 5. If all students are targeted, look across the student groups on Worksheet 3-A and identify which items seem to have been problematic for the majority of student versus those that were problematic only for a particular subgroup. • If a check appears in multiple columns, including the “# not in group” columns, the question was probably difficult for all students. • If a check mark appears in a subgroup column, but not in the column for “All students,” the question was difficult mainly for that subgroup. • If a check mark appears in a subgroup column and in the column for “All students,” but does not appear in the “# not in group” column, the performance of the subgroup may account for the lower performance of the “All students” group. 6. If all students are targeted, the team should now have identified: • a set of questions to analyze for all students, and • a set of questions to analyze for each subgroup. 7. Using a copy of Worksheet 3-B (Analysis of MCAS questions) for each student group, list the questions identified for that group. IMPORTANT: If the planning team did not identify all students as a targeted group, only identify the 8-10 questions that appear most difficult for each targeted subgroup. 8. Repeat Step 3.1 using the previous year’s MCAS data. 3.2 Identify the skills and knowledge that targeted groups have not mastered Guiding Question: What skills and knowledge do students need in order to answer these questions? Facilitator: Make sure the team members have sufficient content expertise to conduct this analysis. In order to define more clearly what specific skills and knowledge students are unable to demonstrate, it is helpful to look at their performance on individual items. However, knowing how a particular student group performed on one item doesn't necessarily indicate what it was about that question that made it easier or more difficult for students. By looking for similarities in questions on which students performed in similar ways, the team will gain insight into the skills and knowledge different groups of students need to acquire to raise their performance. 1. For each targeted group, record the following on Worksheet 3-B (Analysis of MCAS questions): • the item numbers selected for that group, • the item types (short answer open response, or multiple choice), and • the relevant standards (listed in the released questions). 2. Using the released MCAS items, read through each question on the list and brainstorm the skills and knowledge needed to answer each question. Ask: • What do students need to know or be able to do to answer this question? • What about this question might have made it difficult for students to answer correctly? • Using the item analysis report, note whether a high percent of students chose a particular wrong answer. Why would students have chosen that response? • Does the terminology regularly used during instruction match that which is used on the test items and in the standards? • Consult the Curriculum Frameworks to better understand the skills and knowledge required to perform well on the question at that grade level. NOTE: For comparison, consider performing a similar analysis of questions on which students' scores are highest relative to the statewide score. This analysis will help identify students' strengths. Consider what may have these questions less difficult for students to answer. 3. Record all the skills and knowledge needed to answer the questions in the spaces provided on Worksheet 3-B. 4. After listing the skills and knowledge for each question, look for patterns. Ask: • Are there similarities between these questions? • What characteristics appear in a number of questions? • Examine test items on similar content that required different levels of thinking. Does student performance change as the thinking level requirements change? • Are there similar items on the test on which students’ performance was higher? If so, what differences between the items might account for the difference in performance? 5. Based on the discussion, record the skills and knowledge that seem to pose the most difficulty for students on Worksheet 3-C (Skills and knowledge students lack). Facilitator: Be sure to distinguish between early literacy skills (the foundation skills students need to be able to read and write) and early reading skills (techniques or strategies students use help them read and write actively). 6. Consider whether a lack of literacy skills contributed to the low student performance on the questions identified. Ask: • Is student performance for the targeted group generally low for all of the items on the test? • Does student performance seem to be lower on items that require more reading or higher-level reading? 7. If the answer to either of these questions is yes, record “literacy skills” as a placeholder on Worksheet 3-C. The team will have to perform further analysis with help from a literacy specialist to determine the specific literacy skills students lack. Consult Appendix C (Assessing reading and literacy skills) for guidance on incorporating this analysis into the PIM process. 3.3 Consult other sources of data to verify theories about the skills and knowledge students lack Facilitator: Involve people from outside the team with expertise in ELA and mathematics, reading, English language learners, special education, and other specialists as needed. Guiding Question: Do other sources of data verify or refute the team’s conclusions that students have difficulty with the skills and knowledge identified? Once the team has investigated students’ reading and literacy skills (if applicable), it will consult data to verify the other skills and knowledge students lack. 1. For each skill or area of knowledge listed on Worksheet 3-C (Skills and knowledge students lack), look at any relevant data from district or school assessments and examples of student work that may provide additional information on students’ level of mastery of these skills. These assessments might include: • Other standardized tests (e.g., Stanford, TerraNova, ITBS, MEPA, MELA-0, LAS-R/W, etc.) • District-based tests and cross-grade or classroom assessments scored using rubrics (e.g., quarterly assessments, etc.) • Classroom or school-based information on student performance, including both summative and formative assessments (e.g., portfolio assessments, samples of student work, etc.) IMPORTANT: Remember to look at disaggregated data for the student group currently targeted. 2. Try to isolate information on skills and areas of knowledge similar to those identified as weak in the initial MCAS item analysis. Ask: • Does the data from other assessments demonstrate weaknesses in the same skills and areas of knowledge identified using MCAS data? OR • Do other assessments seem to indicate that students’ performance on these skills and areas of knowledge is stronger than suggested by MCAS data? If so, to what might the discrepancy be attributed? 3. Consider whether to include MCAS Retest data as an additional source of information for high schools. High school planning teams should keep the following considerations in mind when examining data from MCAS Retests: • Students are not required to take the retest; therefore, item analysis results may be negatively influenced by the inclusion of scores from blank test booklets. • Students take retests for a variety of reasons. Some students take the retest because they transferred from out of state, while other students do so because they performed poorly on the standard spring test. • Because retests are focused on the skills and knowledge needed to earn a Competency Determination, they do not contain items that allow students to score proficient or advanced. The highest score one can earn on a retest is a scaled score of 238. If MCAS retest data is used to verify the skills and knowledge students lack, it is important to know which students took the test and why. 4. Record findings from more extensive data analysis on Worksheet 3-C in the space provided, along with any questions and any other data that should be considered before making a final statement about the gaps in students' skills and knowledge. 3.4 Set student learning objectives Guiding Question: Based on analysis, what skills and knowledge do students need to acquire? 1. If analysis of reading assessment data indicated that students’ lack of basic reading or literacy skills may interfere with their ability to demonstrate other skills, write one or more student learning objectives describing the specific reading or literacy skills students need on Worksheet 3-D (Student learning objectives). 2. Based on the other data sources consulted, determine which other skills and knowledge most need to be addressed in order to raise students’ performance. 3. Write student learning objectives describing the expectations for the skill or area of knowledge which the student will demonstrate. NOTE: The student learning objective may be tied to a specific learning standard, or it may represent a more specific skill. Where possible, the student learning objective should refer to specific language in the state Curriculum Frameworks. IMPORTANT: Write a maximum of three student learning objectives for each student group. Prioritize those skills that serve as building blocks to others, such as early literacy skills. 4. Record student learning objectives on Worksheet 3-D. Checklist Before moving on to Step 4, make sure the team has: • Used the MCAS item analysis to identify the characteristics of questions that students found difficult • Identified the skills and knowledge that students may lack based on the patterns observed in the item analysis • Investigated student’s early reading and literacy skills as necessary • Used additional sources of data to verify that students struggle with the skills and knowledge identified • Consulted the Curriculum Frameworks and created student learning objectives describing the skills and knowledge students need to master Step 4: Identify the most significant causes of the weaknesses in students’ knowledge and skills Facilitator: Step 4 requires self-reflection about how teaching and learning can be improved. Initially participants may feel threatened by this process or feel guilty about “blaming teachers.” This is a good time to remind them of the goals of the PIM process. Also remind the group not to focus yet on solutions. Now that the team has identified the specific skills and knowledge students lack, it will try to determine why students have been unable to master these skills. It will think critically about the learning experiences that students receive in the school and identify potential barriers to their learning. Step 4 has three distinct parts: (1) Working with each student learning objective individually, the team will brainstorm a number of possible causes that affect students’ ability to master that skill. The team will organize the causes into categories according to whether they pertain specifically to the student learning objective, affect the content area as a whole, or affect student learning across multiple content areas. (2) Once the team has brainstormed causes for every student learning objective, it will consolidate similar causes across the various student learning objectives and across student groups. (3) Finally, the team will prioritize which causes it will address and look for evidence to verify its hypotheses that these causes exist. Expectations for this step • Team members will become familiar with the principles implicit in the Dimensions of School Improvement and use them to frame future school improvement discussions. • The team will identify broad causes of low student achievement in one or multiple content areas. • The team will identify specific causes that affect the individual skills described in the student learning objectives. • The team will prioritize causes based on their level of impact on student achievement and the level of control the school or district has over each cause. • The team will collect evidence to verify or disprove its theories on the causes affecting student learning. Worksheets 4-A Possible causes of students’ lack of skills and knowledge (Word) 4-B Dimensions of school improvement (Word) 4-C Causes related to a student learning objective (Word) 4-D Causes related to one content area (Word) 4-E Causes related to all content areas (Word) Other Materials and Equipment Half-sheets of paper, masking tape, and completed copy of Worksheet 0-D (School self-assessment) 4.1 Identify possible causes of students' lack of knowledge and skills Facilitator: Revisit the group norms set at the beginning of the process, and make sure they are sufficient to manage the sometimes diffi.cult conversations that take place in Step 4. Allow 2-3 hours for the first Step 4 meeting. Guiding Question: Why haven't students in the targeted group learned the skills and knowledge described in the student learning objectives? If a goal was set for all students, begin with the student learning objectives for that group. Complete Step 4 through Section 4.3 before conducting the same process for subgroup goals. 1. Beginning with one student learning objective, brainstorm the possible underlying causes for the identified gaps in students' knowledge and skills. 2. As the team brainstorms, write a 3-5 word summary of each cause on a piece of paper and tape it on the wall where everyone can see it. Facilitator: Use the “5 Why’s” technique to focus the group on root .ple, if a participant says, “Kids can’t read,” ask “Why?” If team members are raising perceptions that may not be true, ask “How do you know?” IMPORTANT: Make sure that the written causes are specific enough to be interpreted after the discussion is over. For example, a cause written as “Curriculum” does not describe what was lacking in the curriculum. 3. When the group is finished with the brainstorm, use page 1 of Worksheet 4-A (Possible causes of students’ lack of skills and knowledge) to consider whether any causes listed there may also affect the knowledge and skills described in the student learning objective. 4. If the student learning objective is related to early reading skills, discuss the questions on page 2 of Worksheet 4-A and identify any relevant causes of students’ lack of reading skills. NOTE: There is no need to discuss the School Self-assessment again when working on subsequent student learning objectives in the same content area. 5. Finally, look at the copies of Worksheet 0-D (School self-assessment) the team completed for each content area in Step 0 and consider whether the weaknesses identified there may be significant causes of low student performance in general. 6. Throughout these discussions, add any new causes that arise to the list. 4.2 Identify aspects of teaching and learning that may affect students’ ability to master the skills and knowledge identified Facilitator: Though it is important to seek feedback from the rest of the staff; before sharing causes with teachers who have not participated in these intense discussions, determine the best way to introduce them to the concepts in the Dimensions of School Improvement. Guiding Question: What gaps or inconsistencies in teaching and learning may make it more difficult for students to master these skills and areas of knowledge? 1. When the brainstorming session is completed, refer to Worksheet 4-B (Dimensions of school improvement). • Core realm: contains factors that most directly affect student outcomes • Enabling realm: contains conditions that must be in place in order to make the core elements successful • Supporting realm: contains conditions that are helpful toward making the core elements successful One way to think about this diagram is that the core realm represents what happens at the classroom level, the enabling realm represents the school, and the supporting realm represents the community. Another way to think about this diagram is that the amount of control that teachers and the school have is greatest at the center. Facilitator: If the team feels frustrated by the limits imposed by the “Dimensions of school improvement” graphic, remind them that the purpose of the PIM process is to improve curriculum, instruction, and assessment at the school. 2. Sort the brainstormed causes into categories for the core, enabling, or supporting realms by moving the papers on the wall. 3. Consider the causes placed in the supporting category. These are likely factors over which the school has very little control and that are not directly related to teaching practice. Taking each cause individually, discuss the following: • Is this cause important enough for the school to focus time and energy on in the planning process? • Can this cause be stated in a way that relates it more directly to issues in the core or enabling realm? Facilitator: These waiting room issues may need to be discussed with district representatives and revisited later in the process. 4. After discussing each cause in the supporting realm, there are two options: • Put the cause aside to be dealt with at another time (Remove it from the wall and record it on a “waiting room” list.) • Rewrite the cause in a way that allows it to be placed in the core or enabling category 5. Next, turn to the causes placed in the core and enabling categories. If there are very few causes in these categories, ask: • Are there any other factors related to teaching and learning that might affect students’ ability to master the skills and knowledge described in the student learning objective? If necessary, brainstorm and add any new ideas to the list. 4.3 Determine whether the causes identified pertain specifically to the skills and knowledge articulated in a student learning objective or whether they impact other areas of teaching and learning. Facilitator: If the team has not identified any causes that specifically address the student learning objective, work further to identify some. In the case of some elemental skills, it may be that all the causes have a broad effect, but be sure to investigate specific causes by asking why students had particular trouble with the skill described in the student learning objective. Guiding Question: Which causes apply specifically to the student learning objective? Which affect students’ mastery of a content area as a whole? Which impact student learning across all content areas? 1. Discuss each cause in the core and enabling categories and sort them into three new categories: • Causes that specifically impact the student learning objective • Causes that impact the overall content area (English language arts or mathematics) for the student group targeted in the school’s goal • Causes that impact student learning across all content areas for the student group targeted in the school’s goal 2. Once each cause is placed in the appropriate category, consolidate any duplicate or very similar issues within each category; however, avoid consolidating causes in ways that make them too broad and vague. 3. Record the causes in each category on the appropriate Step 4 worksheet. Facilitator: The brainstorming done with each new student learning objective will build on previous discussions. There is not need to repeat the issues that fall in the “one content area” or “all content areas” categories. Stop here! Repeat Steps 4.1-4.3 for each student learning objective the team set for the student group it is working on. For each student learning objective, create a new copy of Worksheet 4-C. Add to the original copy of Worksheet 4-D when working in the same content area and begin a new one when working on the other content area. Use only one copy of Worksheet 4-E for both content areas. Repeat the process for the other student groups. For each new student group, start new copies of all of the worksheets. Do not conduct the remainder of Step 4 until Step 4.3 is completed for all student groups and content areas. 4.4 Identify causes that affect each content area or student learning overall Guiding Question: What common causes affect each content area or student learning overall? 1. Working with one student group at a time (beginning with all students if applicable), look at the copies of Worksheet 4-D (Causes related to one content area). Looking across the causes identified in both mathematics and English language arts, ask: • Are there any issues that arose in mathematics and English language arts that are similar and affect all content areas? • Do these causes primarily affect a subgroup of students, or do they affect all students? 2. Remove identified issues that affect multiple content areas from both Worksheets 4-D and record them on the copy of Worksheet 4-E (Causes related to all content areas) for the appropriate student subgroup. 4.5 Prioritize causes Guiding Question: Which causes have the greatest impact on student achievement? Which causes do the school and/or district have the most immediate control over? It would be impractical to try to address all of the causes identified. Therefore, narrow the list of causes to identify those which can be addressed most productively. 1. For each copy of Worksheets 4-C (Causes related to a student learning objective), 4-D (Causes related to one content area), and 4-E (Causes related to all content areas), rate each cause based on the amount of control the school and district have over it and the impact it is likely to have on student achievement. Use the following scales: • How much impact will the school have on improving student achievement if it addresses the cause? 1 = Minimal impact 2 = Some impact 3 = Substantial impact • How much control does the school have to address the cause? 1 = Very little control 2 = Some control 3 = A lot of control • How much control does the district have to address the cause? 1 = Very little control 2 = Some control 3 = A lot of control 2. Identify causes that rated high on the amount of impact on student achievement and the amount of control the school or district has over the issue. 3. Causes rated as having a substantial impact on student achievement over which the district (and not the school) has a lot of control should be raised with the district administration. 4.6 Try to verify or disprove the possible causes generated Guiding Question: What evidence verifies the planning team’s theories about why students lack this knowledge and/or these skills? 1. For each cause rated high on impact and control, gather evidence to verify or disprove that the cause exists and affects students’ skills and knowledge. To collect this evidence consider examining a number of different sources of information, such as: • Copies of the school's or district’s curricula • Lesson plans • Student schedules • Information on teacher qualifications • Information on student and staff attendance • Records of classroom observations • Information from interviews or surveys of teachers, parents or students • Discipline records • Information on participation and impact of student assistance programs • Information on teacher training and professional development 2. When the available information and data has been consulted, discuss whether significant evidence exists to confirm the team’s hypothesis about the causes, and record conclusions in the last column on the worksheets. Checklist Before moving on to Step 5, be sure the team has: • Identified causes in the core and enabling realm for each student learning objective • Sorted causes according to whether they pertain to all content areas, one whole content area, or a specific student learning objective • Considered whether the causes affect one student group or all students • Prioritized each set of causes based on level of impact and control • Collected evidence to verify the causes identified Step 5 Facilitator: The team will be relieved to be talking about solutions and may have strategies in mind. Keep team members from rushing ahead. Teams often think of improvements the school has already planned or those most in vogue. Push the group to think beyond “easy answers.” Make sure they brainstorm thoroughly before writing the improvement objective. In Step 4, the planning team identified possible causes of students' lack of knowledge and skills. Now it will select which causes to address in the plan and begin to develop solutions for improving student achievement. From the causes identified, the team will develop improvement objectives that explicitly describe the change it wishes to make in students' learning experiences. When identifying causes, the team started at the micro level, looking at individual skills and areas of knowledge, and from there moved to more macro levels, thinking about issues that affect whole content areas and multiple content areas. In Step 5 the team begins at the macro level. It will first identify the changes it would like to make in the learning experiences for students across all content areas. Then it will determine what additional changes need to be made in the individual content areas. Finally, it will determine what further changes need to be made to address the specific skills and knowledge in the student learning objectives. Because the plan should be integrated, there is not need to repeat any strategies at a micro level if they are addressed at a more macro level. Similarly, if targeting all students, the team will first develop improvement objectives for the whole population. Then when working at the subgroup level the team needs only to include improvement objectives describing the supports the school will provide in addition to what is planned for all students. Expectations for this step • Teams will identify the most significant causes to address. • Teams will consult key resources regarding ways to address the identified causes. • Teams will develop improvement objectives describing the learning experiences students need to improve student performance. Worksheets 5-A Improvement objectives (Word) 5-B Resources needed for developingimprovement objectives (Word) Other Materials and Equipment Newsprint Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Completed copies of Worksheet 4-C (Causes related to a student learning objective), Worksheet 4-D (Causes related to one content area), and Worksheet 4-E (Causes related to all content areas) 5.1 Select which causes to address in the plan Facilitator: Once the team decides which causes to address, have one person transcribe them on to Worksheet 5-A while the team continues working. Guiding Question: Which causes are most urgent to address and the team decides should take priority in the current planning process? 1. Beginning with all students, if applicable, look at the causes on .Worksheet 4-E (Causes related to all content areas). Ask: • Which causes were verified with evidence? • Which causes rate highest on level of impact and control? • Are there causes that must be addressed first in order to address other causes? 2. Based on the answers to these questions, select the most significant causes to address. Then ask: • Are there two or more causes that are closely related that should be combined or addressed together? (If combining two or more closely related causes, be careful not to combine them in ways that over-generalize them. Causes may also written separately, but grouped together.) 3. Record the most significant causes in the first column on Worksheet 5-A (Improvement objectives). 4. Ask the above questions again while looking at the causes on Worksheet 4-D (Causes related to one content area), and record the significant causes on a new copy of Worksheet 5-A. Again, repeat the process with the causes on Worksheet 4-C (Causes related to a student learning objective) and another copy of Worksheet 5-A. 5. Repeat the process for the other student groups. The team should now have, on multiple copies of Worksheet 5-A, a complete set of the issues it will consider for the rest of the planning process. 5.3 Set improvement objectives to address the causes of poor student achievement Guiding Question: Based on the causes of poor student achievement, what experiences do students need in order to improve their performance? What resources should be consulted to better understand the causes identified? Now that the team has carefully defined the problems it faces and identified the most significant causes of those problems, it will begin thinking about possible solutions. An improvement objective is a descriptive statement of the most important learning experiences the school wants its students to receive that directly address the cause of low student performance identified in Step 4. Good improvement objectives: • Clearly describe the learning experience the school wants its students to receive. • View learning from the students’ point of view, because they focus on what students need rather than what adults need. • Directly address the cause of low student performance. • Clearly describe to the reader (especially fellow teachers) what will be different from past practice. • Illustrate observable changes in the condition of teaching and learning whenever possible. Facilitator: Team members will likely raise ideas for specific strategies. Record them under “key brainstorm elements”, but urge the group to consider what learning experiences for students would result from the implementation of these strategies. In Step 7 the team can revisit this list of potential strategies. 1. Start with the causes listed on Worksheet 5-A (Improvement objectives) for all content areas. 2. Taking each cause individually, think about what learning experiences the school would like to offer students in order to address this cause. Brainstorm what the ideal learning situation would look like from the students’ point of view. Be creative and do not worry yet about what is realistic. Record these ideas on a piece of newsprint. 3. Consider the list of experiences that were generated. Ask: Which of these experiences are the most critical to improving student learning? 4. Record these experiences on Worksheet 5-A under “Key brainstorm elements.” Next, ask: • What is the general theme in the experiences the school hopes to offer students? • How might this theme be summarized in a way that addresses the cause(s) identified? 5. Record the general experience the school would like to offer students on Worksheet 5-A under “Improvement objective.” The statement, which may be written in one sentence or a few sentences, does not need to be exhaustive; however, it should describe the new learning experience in a way that is clear to the rest of the staff. 6. If appropriate, use the space provided on the bottom of Worksheet 5-A to write a rationale for the improvement objective chosen. The team may want to explain how it prioritized from among the particular brainstorm elements used to craft its improvement objective or why it feels the improvement objective will be different from past practice. 7. The team may find that it does not possess the expertise needed to choose the right learning experience and may need to talk with others in the district or consult additional resources outside the district. Use Worksheet 5-B (Resources needed for developing improvement objectives) to record information the team needs to gather, where it will look for it, who will be responsible for gathering it, and when that information will be due. Once it has collected this information, the team will complete or refine its improvement objectives. 8. Repeat the process for causes in each individual content area and record the improvement objective on additional copies of Worksheet 5-A. IMPORTANT: Remember not to repeat any improvement objectives already set on Worksheet 5-A for all content areas, as this would create unnecessary duplication in the plan. Facilitator: If causes addressed in 5.3 above require fundamental changes to curriculum and instruction in the school, the team may need to address them before developing strategies to target individual student learning objectives. In this case the school may delay creating plans for student learning objectives. The .ment its rationale and when it will revisit that part of the plan. 9. Now that the broader causes of low student achievement have been addressed, consider the causes of why students have not mastered the specific skills in the student learning objectives. 10. Repeat the process described in Step 5.3 for each cause listed on the copies of Worksheet 5-A for each student learning objective. 5.4 Incorporate information and data from Steps 1-5 into the written plan Guiding Question: What goals and objectives were chosen as the focus of the plan? NOTE: For instructions on using the PIM Access Database, click “Getting Started” on the main menu or click the “Help” button within any form. 1. If the team has decided to use the School Improvement Plan Template and/or the PIM Access Database to document its plan, enter the following information: • From Step 2: Student performance goals (if using the PIM Access Database) • From Step 3: Student learning objectives • From Step 5: Prioritized causes that will be addressed • From Step 5: Improvement objectives 2. Reflect on the decisions the planning team made, and ask: • Does the plan address the most important problems the school faces and the most important changes the school needs to make? • Is the scope of the list of objectives rigorous yet realistic for the school to undertake? • Will the objectives listed be sufficient to allow the school to reach it goals? 3. Based on this discussion, make any necessary changes or refinements to the work. Checklist Before moving on to Step 6, be sure the team has: • Determined which causes are most urgent and will be most productive to address and recorded them on copies of Worksheet 5-A (Improvement objectives) • Gathered information from other sources as listed on Worksheet 5-B (Resources needed for developing improvement objectives) • Recorded the changes the school intends to make to student learning experiences on Worksheet 5-A • Recorded goals, student learning objectives, causes, and improvement objectives in the School Improvement Plan Template, the PIM Access Database, or an alternate format Step 6 Facilitator: The process may become stalled if individuals are attached to certain programs that have outlived their usefulness. Try to help team members describe the value they think the program adds and whether those benefits may be sustained through other means. Save those ideas for Step 7. In Step 6 the planning team will identify what strategies are already in place that may help the school achieve its improvement objectives. For each strategy, the team will gauge how thoroughly it has been implemented and how effective it has been in improving student achievement. The team will then determine what modifications may be needed in order to make the strategies more effective and whether there are ineffective strategies that should be discontinued. Later, in Step 7 the team will determine if additional strategies are needed in order to achieve its improvement objectives, and in Step 8 it will develop action plans to guide school staff in making those changes. Thinking ahead to those steps, the first task in Step 6 is to group any improvement objectives the team would like to address together. Expectations for this step • Teams will identify current strategies that support the school’s improvement objectives and any modifications that should be made to improve their implementation and effectiveness. • Teams will identify any strategies that have not demonstrated effectiveness and plan to discontinue them. • Teams will have the opportunity to consider whether adequate data on program effectiveness is collected at the school. Worksheets 6-A Current strategies (Word) Other Materials and Equipment Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Completed copies of Worksheet 5-A (Improvement objectives) Criteria for Evaluating Instructional Materials and Programs in Reading (Pages 1-4), Mathematics (Pages 5-6) and Science and Technology / Engineering (Pages 7-8) from MA DOE Web site: www.doe.mass.edu/read/mrfp/criteria.pdf Additional resources and evaluation criteria for reading curricula from MA DOE Web site: www.doe.mass.edu/read/mrfp/links.html?section= centers 6.1 Assess the implementation and effectiveness of current strategies Facilitator: Here the paperwork and information involved become difficult to manage. To minimize confusion, the team should work on only one student group at a time, beginning with the largest student group. Remind the team of where it is in the process and help it develop systems for categorizing and tracking information. Guiding Question: What strategies are already in place to support the improvement objectives identified? Are changes needed to make these strategies more effective? 1. Looking at each copy of Worksheet 5-A (Improvement objectives) for one or all content areas, consider whether there are improvement objectives that are closely related and should be addressed as a group. Complete a copy of Worksheet 6-A (Current strategies) for each improvement objective or group of improvement objectives. 2. For each copy of Worksheet 5-A pertaining to a student learning objective, group all of the improvement objectives associated with the student learning objective on one copy of Worksheet 6-A. 3. In the “Current Strategies” column on Worksheet 6-A, list any strategies already planned or in place that might contribute to meeting the improvement objective(s). Depending on the scope of the improvement objective that the team identified, a strategy may be defined as a specific means, method, or approach to: • achieve the learning experiences described in a given improvement objective; and/or • help students master the specific skills and knowledge described in the student learning objectives. Strategies take many forms in the school. For one or multiple content areas, examples of strategies may include: • Policies, such as student discipline and suspension policies • Structures, such as time in the schedule for common planning • Systems, such as screening the reading skills of entering students • Curriculum materials, such as pacing guides, textbooks, and supplementary materials For student learning objectives, strategies will take the form of instructional strategies such as: • The use of graphic organizers to organize details in a story • Grouping students to increase student-teacher interaction when discussing a particular concept or skill • The assignment of specific academic tasks • The use of formative assessments to gauge student performance on a specific skill 4. As a group, rate each strategy on the following two scales: • Has the strategy been fully implemented as intended in the school? 0 = We don't know how fully the strategy has been implemented. 1 = The strategy is not taking place. 2 = The strategy has been partially implemented: some teachers are implementing the strategy and some are not and/or all teachers are implementing the strategy only partially. 3 = The strategy has been fully implemented and there is evidence to support this conclusion. • As implemented, has the strategy had the intended effect on student achievement? 0 = We don’t know how much effect the strategy has had. 1 = The strategy has had little effect; evidence indicates that few students benefit from the strategy. 2 = The strategy has had some effect: evidence indicates that many students benefit from the strategy but some do not or that the outcomes for those who benefit are less than expected. 3 = The strategy has had substantial effect: evidence strongly indicates that nearly all students benefit from the strategy and that the outcomes are significant. 5. Discuss the strengths and weaknesses of each strategy as it was implemented. Ask: • Were systems clearly defined? • Was there adequate training and professional support for teachers? • Was the timeline for implementation realistic or was it disrupted in any way? • Were resources sufficient? • Were staff overwhelmed by unclear or competing priorities? 6. Consider how the strategies could be improved. Ask: • What could be done to improve the implementation or effectiveness of this strategy? • How could this strategy be modified so it would more effectively support the student learning experiences articulated in the improvement objective? • How could the ability to gauge the level of implementation or degree of effectiveness of this strategy be improved? 7. Record any important notes or ideas for modifying the strategy in the appropriate column on Worksheet 6-A. Facilitator: Step 6 is a critical point for the team to involve the rest of the staff. Consider displaying or distributing the strategies and rating criteria by grade level and program area so that all staff have the opportunity provide feedback. 8. After rating strategies, determine which should be continued. In the last column indicate whether the school intends to: C = Continue the strategy M = Modify the strategy D = Discontinue the strategy 9. If appropriate, record the rationale for continuing, modifying, or discontinuing strategies in the space provided at the bottom of the worksheet. IMPORTANT: If the team has decided to discontinue certain strategies, it must establish a time and/or process for communicating this information to the rest of the staff at the school. 10. Enter the strategies that will be included in the plan in the School Improvement Plan Template, PIM Access Database or in an alternate format. • Enter the strategies that require modification, describing how each strategy will change. • Enter any strategies the school will continue that the team wishes to emphasize in the plan. Checklist Before moving on to Step 7, be sure the team has: • Used Worksheet 6-A (Current strategies) to assess the efficacy of current strategies in light of the improvement objectives identified • Decided which current strategies will be continued, which require modifications, and which, if any, should be discontinued STEP Seven: Investigate and evaluate possible new strategies to support improvement objectives Facilitator: Focus the team on how new strategies will specifically support the improvement objectives. Press them to investigate and weigh options before making decisions. Help the team keep track of the “threads” of logic linking the information and decisions generated in each step. Expectations for this step • Teams will investigate and select strategies that will support the improvement objectives. Worksheets 7-A Investigating possible new strategies (Word) Other Materials and Equipment Worksheet 5-A (Improvement objectives) Worksheet 6-A (Current strategies) Resources Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Criteria for evaluating instructional materials and programs in reading (Pages 1-4), mathematics(Pages 5-6) and science and technology / engineering (Pages 7-8) www.doe.mass.edu/read/mrfp/criteria.pdf Identifying and implementing educationalpractices supported by rigorous evidence: A user friendly guide: www.ed.gov/rschstat/research/pubs/rigorousevid/rigorousevid.pdf Resources and evaluation criteria for readingcurricula from MA DOE Web site: www.doe.mass.edu/read/mrfp/links.html?section= centers 7.1 Consider whether the existing strategies the school identified will accomplish the improvement objectives Guiding Question: In addition to the current strategies, are additional strategies needed by the school to ensure the learning experience described in its improvement objective? 1. Beginning with the improvement objectives and supporting strategies identified for all content areas, ask: • Will the current strategies—given the necessary modifications identified—offer students the learning experience described in the improvement objective(s)? 2. If the answer is no, brainstorm additional strategies on newsprint. (Remember to consult the waiting room and brainstorm elements listed on Worksheet 5-A (Improvement objectives) for ideas.) Ask: • In addition to these strategies, what else should teachers do to offer students the learning experience described in the improvement objective(s)? • What else should the school do to offer students the learning experience described in the improvement objective(s)? • What supports do teachers need in order to provide students with the learning experience described in the improvement objective(s)? 3. From the newsprint, identify potential new strategies to investigate and record them on Worksheet 7-A (Investigating possible new strategies). 4. Repeat this step for the improvement objectives that address one content area and then for the improvement objectives that address specific student learning objectives. Record these findings on additional copies of Worksheet 7-A as needed. 7.2 Investigate possible new strategies and determine their effectiveness Guiding Question: What does the team need to know before selecting new strategies for reaching the improvement objective? Look at the potential strategies listed on Worksheet 7-A (Investigating possible new strategies). Members of the planning team (or others in the school or district) will likely need to do some investigating before the team proceeds. For example, if the team has decided that teachers should use a standard writing program, it will want to take the time to investigate available programs and assess their strengths and weaknesses relative to the school’s needs. 1. Depending on the type of strategy being considered, plan to do the following: • Consult reliable sources of information1 from web sites, educational literature, and other sources. Consider contacting experts from district, state, and regional sources (such as local colleges, universities, or other educational organizations). • Identify which specific skills and knowledge the strategy has been shown to impact. • Weigh the strengths and weaknesses of each strategy. • Identify the conditions necessary for its successful implementation, including the human, material, and fiscal resources needed to implement the strategy and produce results. 2. In the spaces provided at the bottom of Worksheet 7-A, indicate the following: • A description of the proposed new strategy • Questions to ask about the strategy • The name(s) of the person(s) conducting the investigation • The date for making the final decision about whether to adopt the strategy 7.3 Select strategies to implement Guiding Question: Based on the investigation, which strategies will be most effective in supporting the improvement objective(s)? 1. Discuss the work performed in Step 7.2 above to assess the effectiveness of the proposed strategies. 2. Select the strategy or strategies that will be the most effective in supporting the improvement objective(s) identified, and indicate “adopt” in the right-hand column on Worksheet 7-A (Investigating possible new strategies). 3. Record the rationale for making these selections in the space provided on Worksheet 7-A. 7.4 Consider whether the strategies identified will accomplish the goals and objectives Guiding Question: Do the selected strategies form an effective plan? 1. Record any new strategies along with current strategies in the School Improvement Plan Template, PIM Access Database, or an alternate form. 2. Reflect on the list of strategies and ask: • Will the strategies be sufficient to allow the school to reach its goals? • Will the strategies result in the student learning experience described in the improvement objective? • Do the strategies directly address the causes of poor student performance identified? • Do the strategies support one another to form a cohesive, integrated approach to school improvement? 3. Make any necessary changes or additions to the strategies. Checklist • Before moving on to Step 8, make sure the team has: • Developed a list of potential new strategies • Thoroughly investigated and evaluated each potential strategy • Recorded the strategies the school decided to adopt—along with the rationale for doing so—on Worksheet 7-A (Investigating possible new strategies) • Recorded any new strategies the school selected along with its current strategies in the School Improvement Plan Template, PIM Access Database, or an alternate format Worksheet 7-A: Investigating possible new strategies Guiding Question: In addition to your current strategies, what new strategies are needed in order to create the learning experience you described in your improvement objective? Student Group: Content Area: Student Learning Objective (If applicable) Causes (From Worksheet 4-C, D, or E) Improvement Objective(s) (From Worksheet 5-A, B, or C) Notes on rationale: Step 8: Develop action plans for meeting improvement objectives Facilitator: During the development of the school’s action plans, the team is likely to confront constraints in time, funding, and district priorities. It will be important for the PIM team to present their requests to the district and support them with student data, evidence of the causes they have identified and justification for the strategies they have chosen. Now that the planning team has identified the strategies the school needs to reach its improvement objectives, the team will create action plans that serve as “how-to” guides for implementing those strategies. Any school staff person should be able to look at the action plans and understand what he or she is expected to contribute and how his/her work supports the improvement objectives. Follow these basic guidelines in writing action plans: • Include strategies at the classroom level as well as strategies at the school and district levels targeted to achieve the learning experience described in the improvement objective(s) (and student learning objectives where applicable). • Include step-by-step activities necessary to fully implement the strategies. • Indicate who will be responsible for ensuring that the activities are completed in a timely way in order to fully implement the strategies. Expectations for this step • The team will create action plans to guide the implementation of strategies that support the improvement objectives. • The team will be able to define and differentiate the terms “strategies” and “activities.” • The team will identify the activities needed to fully implement each strategy. Materials and Equipment Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Newsprint, markers, and tape Completed copies of Worksheet 4-A (Possiblecauses of students’ lack of skills and knowledge), Worksheet 6-A (Current strategies), and Worksheet 7-A (Investigating possible new strategies) 8.1 Determine the activities needed to implement and ensure the success of the school’s strategies Guiding Question: What steps are needed in order for teachers and school staff to carry out the strategies and reach the improvement objective? What support does school leadership need from the district in order to ensure that the school reaches the improvement objective? An activity is a specific, short-term task or series of tasks needed to implement a larger strategy for improvement. 1. Beginning with one improvement objective or group of improvement objectives, consider the strategies the team decided to continue from Step 6 and any new strategies identified from Step 7. For each strategy, brainstorm the activities needed to successfully implement those strategies. • For existing strategies, ask: What steps are needed to make the modifications identified in Step 6? • For new strategies, ask: What steps are needed to get this new strategy up and running? 2. To help identify all the activities necessary to make the strategy successful, ask: • Is there any further investigation or research that needs to be done before implementing these strategies? • Is approval required for any of these strategies? • Are costs associated with the strategies? • What materials, training, and/or systems are needed? • What support do teachers need in order to implement these strategies? • Are resources or coordinating activities needed from the district? How will this support be negotiated? NOTE: For instructions on completing the action plan form in the PIM Access Database, click the help button on that form. 3. Next consider which order these activities must be completed in. List them in order in the School Improvement Plan Template, PIM Access Database, or an alternate format. 4. Determine the time frame for each activity. • If the activity is a one-time event, record the deadline for its completion. • If the activity will be ongoing, record how frequently it will take place and the date it will commence. 8.2 Review action plans to ensure they are grounded in the previous work Guiding Question: Do the actions plan represent the sum of the work done in the previous seven steps? 1. When a draft of the action plan is completed, verify the decisions the team has made by asking the following questions: • Does the plan fully address the improvement objective(s)? • Will the plan result in the student learning experiences the school intends to provide? • Will the plan be sufficient to allow the school to reach its goal? • Does the plan address the significant causes of poor student performance the team identified? • If the action plan addresses a student learning objective, does the plan directly target that specific skill or area of knowledge? • Are the timelines for completing activities consistent with achieving the school’s student performance goals? Checklist Before moving on to Step 9, make sure the team has: • Developed a meaningful, measurable action plan that is grounded in the team’s previous work for each improvement objective or group of improvement objectives • Specified an activity or series of activities needed to accomplish each strategy • Recorded action plan information in the School Improvement Plan Template, PIM Access Database, or an alternate format Step Nine: Establish benchmarks Facilitator: When discussing the need for documentation, team members may feel overwhelmed by the amount of additional work this will require of school staff. Teams should identify what information is most important to collect and the most efficient means for collecting it. The word “benchmark” has several definitions, and other planning models may use the word slightly differently than it is used in the PIM process. Two common definitions are (1) a point of reference from which measurements may be made and (2) a standard by which something can be measured or judged. The PIM process employs two kinds of benchmarks: implementation benchmarks and outcome benchmarks. • Implementation benchmarks measure whether the school has changed the instruction and services that students receive as described in the plan. • Outcome benchmarks measure whether the changes the school has made have had an effect on students’ skills, knowledge and behavior. In the most general sense, it may help to think about them this way: • Implementation benchmarks measure what adults are doing and teaching. • Outcome benchmarks measure what students are learning. The planning team will use both kinds of benchmark data to evaluate the success of the plan and determine what revisions need to be made. Expectations for this step • Teams will identify the most important strategies to benchmark and determine how to monitor the quality of their implementation. • Teams will identify tools for measuring the plan's impact on students’ skills, knowledge and behaviors. • Teams will determine the amount of incremental progress that should be made on student outcomes if the school is on track toward reaching its goal. • Teams will begin to outline the processes that will be needed to monitor the implementation and effects of the plan on an ongoing basis. (These processes will be further defined in Step 10.) Worksheets Appendix A: Documenting the plan (School Improvement Plan Template, PIM Access Database, or an alternate format) Other Materials and Equipment School-based assessments, including bothsummative and formative assessments (e.g., DIBELS, GRADE), portfolio assessments, end-ofcourse exams, samples of student work, etc. 9.1 Establish implementation benchmarks to measure whether the school is offering its students the learning experience it intended Guiding Question: How will people in the school or other stakeholders know if the school has provided students with the learning experience it intended? Before judging whether the strategies were effective, the school will first need to determine whether the strategies have been successfully implemented. Implementation benchmarks help planning teams measure whether the strategies in the plan are being implemented consistently throughout the school as intended. This data will alert the team to any strategies that require more resources and support. Good implementation benchmarks: • Describe what quality implementation of the strategy looks like • Communicate to adults in the school their role in implementing the plan • Measure whether the strategies are being implemented to create the student learning experience described in the improvement objective It is not necessary to benchmark every strategy in the plan. For each improvement objective, benchmark the strategies that most directly affect the student learning experience. 1. For each improvement objective, select the strategies that are most important to benchmark by asking, “Which strategies most directly address the instruction or services students receive?” Select at least one strategy under each improvement objective. Facilitator: Remind the team that the method of collecting implementation benchmark data must produce documentation that can be reviewed by the planning team. 2. For each strategy the team has chosen to benchmark, determine one method for monitoring implementation. Useful methods include classroom observations, review of lesson plans, review of sample assignments, documentation of discussions during teacher meetings, and evaluation of professional development. Record the method under “Method of collecting evidence” in the School Improvement Plan Template, PIM Access Database, or an alternate format. NOTE: For instructions on completing the benchmark forms in the PIM Access Database, click the “help” button in these forms. 3. Based on the method selected, describe the criteria that will be used to determine whether the strategy is being implemented to an adequate level of quality. Record the description under “Criteria for Quality Implementation.” IMPORTANT: Be sure that the chosen method for collecting evidence will allow people to observe the criteria cited. For example, if the chosen method of collecting evidence is “review of lesson plans” and the criteria is “students will actively use content vocabulary in discussions”, people will not be able to observe student discussions when looking at written lesson plans. NOTE: If using the PIM Access Database, the team may record specific monitoring activities that will appear in the calendar. 4. Record how frequently the strategy will be monitored, who will be responsible for doing the monitoring, and the date the monitoring will begin. 9.2 Establish outcome benchmarks for each improvement objective Guiding Question: How will the school measure whether the plan is producing the student outcomes it desired? Outcome benchmarks measure whether the plan is producing the student outcomes the school desired. For each improvement objective, first define the intended outcome the school intends to reach and then choose a method for measuring progress toward that outcome. 1. For each improvement objective associated with a student learning objective, the outcome is already stated in the skills and knowledge described in the student learning objective. Record these skills and/or knowledge under “outcome” in the School Improvement Plan Template, PIM Access Database, or an alternate format. 2. For improvement objectives related to one or all content areas, specify the desired outcome. These outcomes may be framed as skills to be gained, such as improved writing skills, or behaviors to be changed, such as a decrease in student absenteeism. Reflect on what the team hopes students will gain as a result of the improvement objective and record the desired outcome in the School Improvement Plan Template, PIM Access Database, or an alternate format. IMPORTANT: If the Improvement objective is very broad, it may be difficult to define a specific outcome. For example, if an improvement objective deals with differentiating instruction, the desired o