Report of the Follow-up School Panel Review of
the Williams Middle School, Chelsea, MA

Introduction

The Williams North Middle School was one of twelve middle schools referred for panel review in the spring of 2001 as a result of critically low levels of student performance on State MCAS assessments in 1998 and declining MCAS results in 1999-2000. A Panel Review was conducted in March of 2001. At that time, the panel found there to be significant inadequacies in the Williams North Middle School's plan for improving student achievement. Moreover, panel members found that:

At the present time the conditions necessary for successful implementation of a sound plan to improve student performance at the Williams North School-stable and effective leadership at the school and district levels-do not appear to be in place.¹

Since that time, the Chelsea School District reorganized its middle school program, combining the Williams North Middle School and the Williams South Middle School to make the Williams Middle School. Williams Middle School has a new principal, new administrative structure, a substantially larger staff and student body, and a new schedule.

Upon consideration of the panel findings and changes made by the district, the Commissioner deferred action on the determination of under-performance for a period of six months, and provided a $25,000 grant to support planning and school improvement efforts during that time. The principal and a planning team from the Williams Middle School participated in facilitated work sessions during July, August, and September, at which Department technical assistance staff and data analysts guided the school's planning team through an inquiry-based process designed to help them develop a sound plan for improving student performance at their school. At the conclusion of the six-month deferral period, a follow-up review was conducted at the school by a four-member review panel. The Williams Middle School was in its seventh week of operation at the time of the panel review.

The Scope of the Follow-up Review Process

The follow-up review panel's charge was to review the original panel's findings, analyze current data and written information on the school's improvement efforts, visit the school, and meet with school and district officials in order to advise the Commissioner of its findings relative to the same two key questions that guided the original panel review:

1. Does the school have a sound plan for improving student performance?
2. Are the conditions in place for the successful implementation of the school's improvement plan?

The follow-up panel's responses to the two key questions that defined the scope of their review are included in this report. These findings and conclusions are the product of the panel's analysis, discussion, and observation, based on the evidence available to them. A list of panel members who participated in the Williams North Middle School follow-up review is provided in Appendix A. A detailed schedule of the panel's activities is provided in Appendix B.

The panel's findings and conclusions on the two key questions will be forwarded to the Commissioner of Education for consideration in determining whether the Williams North Middle School is deemed to be under-performing.

Williams Middle School Profile²

The newly formed Williams Middle School enrolled 1,035 students in grades six through eight as of fall 2001. Eighty-four percent of the students were eligible for free or reduced-price lunch, and 74 percent spoke a first language other than English. The school identified 16 percent of its students as Limited English Proficient and reported that 68 percent were Hispanic, 16 percent were white, eight percent were African-American, and five percent were Asian.

Williams North Middle School MCAS Test Results

Cycle 1 Results

In comparing the 1998 MCAS results to the 1999 and 2000 average, scores were level in English Language Arts, while declining four points in Mathematics and three points in Science & Technology. In 2000, 26 percent scored in the Failing performance level in English Language Arts, 74 percent scored in Failing in Mathematics, and 80 percent scored in Failing in Science & Technology. Of the 22 special education students tested in 2000, all scored in the Failing performance level in Mathematics and Science & Technology for a combined average scaled score of 201 in each subject. Another five special education students were exempted from the test.

2001 Results

A= Advanced
P= Proficient
NI= Needs Improvement
W= Warning (formerly Failing)

Panel Reponses to the Key Questions

Key Question 1: does the School Have a Sound Plan for Improving Student Performance?

Within the context of a substantial reorganization of the school, including its administrative structure, the Williams Middle School has a reasonably sound plan for improving student performance. While the plan lacks specificity in key areas (the lack of meaningful or timely progress indicators and ambiguity regarding the strategies to be used to address student weaknesses in Reading and Writing), it does provide a generally coherent framework for improving student achievement. Further, the process used to develop the plan appears to have engendered a good deal of enthusiasm among teachers for its implementation.

A. Has the school analyzed appropriate data and program information to accurately identity the gaps in student performance and determine why those gaps exist?

The Williams staff has analyzed key student performance data but has not conducted a systematic evaluation of its various programs. They have recently initiated several efforts to analyze standardized achievement data, decreasing their reliance on district-wide, performance-based exit exams as indicators of student progress. They have not, however, conducted a meaningful evaluation of their various programs nor do they appear well-positioned to do so. While examining multiple measures of student performance represents an important step toward improvement, they have not prioritized the relative importance of different data sources, clearly explained how the various data sets fit together or complement one another, or set up a system for assessing the effectiveness of specific initiatives.

The analysis of multiple sources of data has served as the basis for the improvement planning process. With support from the district as well as an outside consultant, teacher "Action Teams" examined several sources of data including 8th grade MCAS scores in English Language Arts and Math, district exit exams in ELA, Mathematics, and History, school report card grades in all core subject areas, and student placement forms for entering 7th grade students. An item analysis of the 7th and 8th grade MCAS results for English Language Arts and Mathematics, disaggregated by students who had passed and those in the Warning performance level, revealed serious weaknesses in student achievement in both subject areas. Further, a key component of the Williams' improvement strategy is the administration of the Stanford Diagnostic Reading Test to all 7th grade students. Although this test has yet to be administered, the analysis of this data will inform other decisions outlined by the plan such as the adoption of a specific literacy model.

The Williams staff and district personnel recognize the importance of looking at standardized student performance data. Focusing on standardized assessments such as the MCAS and the (planned) Stanford Diagnostic Reading Test marks a key departure from an exclusive reliance on performance based assessments-subject matter exit exams administered district wide and scored by teachers with established rubrics. Several people interviewed, including the Principal, noted that while the results of the performance assessments contain valuable information, the significant amount of time required to analyze trends in these assessments limits their practical usefulness. While a substantial amount of work remains to be done in terms of clarifying which data to focus on and how it should be analyzed, broadening the kind of data used represents an important step towards improvement planning based on comprehensive data analysis.

Data analysis was conducted by teacher Action Teams that each chose an area for investigation. Although the process of choosing areas to investigate did not appear well coordinated or strategic to the panel, the results of some Action Teams' work has provided useful analysis for planning purposes. For example, using a random sample of 8th grade students, a teacher Action Team examined the relationship between student report card grades and academic performance as measured by the MCAS. They discovered that a substantial disparity exists between student MCAS performance and teacher assigned grades Teacher assigned grades consistently suggested better performance than that indicated by MCAS scores. This Action Team then generated a list of reasons that this disparity exists and began brainstorming ways to address the disparities. While a direct link between this work and the school improvement plan submitted by the school is difficult to establish, this group went through a sound process of analyzing relevant data and determining why the patterns they observed exist. Other Action Teams followed a similar process in analyzing the district exit exams and student placement forms.

There is currently no formal evaluation of the various promising programs at the Williams School. For example, lead teachers are charged with developing a process writing program, providing demonstration lessons and informal evaluative feedback for teachers, and ongoing curriculum development. However, there does not appear to be a sound strategy for assessing the effectiveness of these efforts. The school lacks a detailed overview of its programs and a program evaluation plan based upon that overview. For example, while the Connected Math Program is currently being used by the math faculty, there is no system for measuring its effectiveness on an interim basis. Similarly, because the present literacy program lacks structure and specificity, it is currently not possible to evaluate its effectiveness across the school. A clearer specification of the programmatic initiatives designed to increase student skills in reading, writing, and math is needed in order for meaningful evaluation to occur.

In sum, while the school is to be commended for increasing the range and depth of its data analysis, further work is needed to develop a more systematic approach to using data in order to maintain a focus on the specific improvement objectives articulated in the plan. A more systematic approach would clarify the relationship between the improvement initiatives, the specific outcomes they are associated with, and the data source(s) that will be used to determine whether the desired outcomes are being reached.

B. Does the plan set out specific improvement objectives that are grounded in the school's analysis of the reasons for poor student performance?

Each of the improvement objectives specified in the Williams' improvement plan are linked to portions of their data analysis, and are clear, reasonable, achievable, and measurable. However, the link between the analysis of the reasons for poor student performance and the specific improvement objectives is clearer for some subject areas than in others. Specifically, the connection between data analysis and the improvement objectives is clearer for math and writing than it is for reading.

Although the relationship between data analysis efforts and the improvement objectives specified in the plan are not always clear, each of the improvement objectives can be linked to some data analysis and some of the reasons the school identified for the patterns of poor student performance observed in the data. After each Action Team conducted an analysis of performance data, they brainstormed reasons for poor student performance. This process yielded three clearly articulated problem statements: one in reading, one in writing, and one in math. A specific and measurable improvement objective was then developed for each problem statement. Based on the improvement objective, strategies and practices in the core academic areas and specific action steps were identified, and a reasonably clear course of action developed. The relationships between the problem statements and the improvement objectives are summarized in the following chart.

The panel review team found that the plan's goals were reasonable, achievable, specific and measurable. Because the areas of weaknesses identified in the analysis of data are broad, the improvement objectives specified in the School Improvement Plan are articulated in broad terms as well. Some of the time lines for goals are also quite general, allowing for the goal to be reached over a two-year period. While reasonable, achievable, specific and measurable, the goals lack clear benchmarks that articulate milestones of progress in the shorter term. Further development is needed to identify specific progress indicators to track movement toward goal attainment.

District personnel, building administrators and teachers agree that literacy is and should be the primary focus of school improvement. However, the reason that some of the barriers identified for poor student performance are targeted in the school improvement plan and not others is unclear. Although the Principal and some of the teachers interviewed reported that the decisions regarding which barriers to focus on in their improvement efforts were driven by the identified causes over which the school has direct control and those likely to result in more immediate results, the review team found it difficult to establish a consistent link between the identified barriers and the corresponding strategies. For example, the Action Team that looked at the relationship between MCAS results and report grades identified several reasons for the observed discrepancies, including a lack of study skills, few connections made between courses within content areas, lack of system-wide writing program, a lack of common standards for grading, and the need for professional development. However, only the need for a writing program is clearly addressed in the plan, and, for example, nowhere in the plan is there any mention of the need to improve student study skills, clearly something under the direct control of the Williams staff. While the panel review team recognizes the importance of prioritizing needs based on available resources, the rationale behind these decisions was difficult to establish.

The clearest links between the strategies for improvement and the reasons for poor student achievement are found in mathematics and writing. In mathematics, "ineffective training and in-service support of the math program" and "varying classroom models depending largely on the teacher's selection of style and material" are both clearly addressed in the school improvement plan with the focus on the training, support, and on-going evaluation of the Connected Math Program. The improvement objective in writing and the corresponding strategy of developing a process model of writing is clearly connected to the Action Team data analysis of the open-response questions on the MCAS and the district exit exams.

In sum, while each of Williams' improvement objectives are linked to data analysis and are clear, reasonable, achievable, and measurable, the panel review team found this connection stronger in math and writing than in reading.

C. In order to accomplish each improvement objective, does the plan specify strategies that appear likely to lead to improved student performance?

The strategies for improving student achievement in the area of mathematics are clearer than they are for either reading or writing. The school has adopted the Connected Math Program (CMP), a comprehensive and research-based program, and has articulated a promising implementation strategy. All of the math teachers interviewed by the team report having received program materials, and all but one reported receiving some initial training in CMP. Successful implementation of this strategy appears likely to lead to improved student performance in mathematics.

Identifying the need for a literacy model and a model for process writing are promising first steps. However, the lack of specificity regarding the structure of each model, how it will be implemented, and how it will be evaluated makes it difficult at this time to judge the likelihood of these approaches leading to improved student achievement results. For the improvement objective tied to reading, the school improvement plan identifies several actions steps designed to improve student achievement. While some of the action steps are quite specific-administering the Stanford Reading Diagnostic Test to assess the current reading levels of all 7th grade students and using the SRA Direct Instruction Reading Intervention for students who score significantly below grade level-others provide more of a framework for moving forward rather than specific strategies likely to lead to improved student achievement.

The panel received conflicting messages regarding the strategy of creating a literacy model. School administration described the current literacy model as an "eclectic" mix of best practices Although the panel review observed a clear effort to increase reading and writing in all content areas, there does not appear to be a shared understanding of what creating either a literacy model or a process writing model entails. For example, a lead literacy teacher described the effort of creating a model as pulling together many of the existing materials and then distributing them to teachers. But she then indicated that teachers could modify the model as they saw fit. The review team could not determine whether the idea of creating a literacy model is understood as a set of practices that every teacher will be expected to implement or as simply providing a set of practices that teachers can use as they see fit. Whether selecting a literacy model will lead to improved student performance depends in part on whether or not Williams teachers implement a set of consistent practices. Responses from teachers and staff interviewed by the review team indicated different understandings what level of consistency will be required and what exactly the practices will be in the area of literacy.

D. Are the school's written improvements planning document(s) clear and specific enough to guide their implementation of planned improvement initiatives?

The strengths of the Williams school improvement plan outweigh its weaknesses. It provides a solid framework for improvement and represents a sound first step toward developing a plan that will guide school improvement. The plan includes clear statements of problems, measurable improvement objectives, and a reasonable plan of action including action steps, the individual(s) responsible for carrying them out, required resources, timelines for each action step and general descriptions of how implementation will be monitored. Weaknesses of the plan are in the area of evaluation. Although benchmarks have been identified to measure progress, the review team found that they were either not directly aligned with the improvement objective or were not scheduled often enough to inform refinement of the plan or implementation steps. The lack of a clear strategy for assessing the degree and quality of implementation, as well as whether the programs implemented are leading to the desired student performance outcomes, is a significant weakness of the school improvement plan.

As mentioned in 1B and 1C above, the initiatives outlined in the school improvement plan vary widely in terms of clarity and specificity. Part of this ambiguity can be traced to the timing of the panel visit. The Williams School is still very much in the process of developing its plan, and the vagueness regarding how program implementation will be monitored is at least partially attributable to the fact that two of the three key initiatives-developing a literacy model and a model for process writing-have yet to be clearly defined. Developing greater specificity in terms of these two models, along with identifying more timely and specific formative assessments are critical steps necessary for successful implementation of the plan.

In the panel's opinion, the plan defines student weaknesses only in broad terms (e.g., students cannot read at grade level, students have problems in all areas of math, etc.). In order to address this lack of specificity, the school will develop a baseline of student performance in reading using the Stanford Reading Diagnostic Test. Efforts to develop a literacy model that addresses particular areas of weakness are on hold until the results of these assessments are available.

Because of the broad terminology used in the description of key action steps, significant work remains to be done on the plan in order for it to serve as an effective guide for school improvement. Further specification of both the literacy and writing models is needed in order to make a clear and meaningful evaluation of the plan possible.

E. Was the SIP developed through a process that will support its successful implementation?

A substantial amount of evidence indicates that key stakeholders, particularly the school and central office staff, were involved in the development of the school improvement plan. Over the summer and continuing this fall, approximately 50 staff members, building administrators, and central office personnel were included in different aspects of the planning process. It does not appear that parents and students were included in the planning process nor is it clear what role these groups will play in its implementation.

The majority of the Williams staff participated in some aspect of developing the school improvement plan. Several "Action Teams" were formed over the summer-all staff members were asked to sign-up for one of the ten teams (established at the end of the last school year). These action teams first analyzed the needs assessment that was conducted in the spring. The focus areas of the needs assessment had been identified by each of the parent, student and teacher surveys. This stage of the process only focused on the needs assessment.

The staff continues to be involved in various data analysis and program development efforts. In the fall, Action Teams were again established, this time organized by the time of day teachers were available to meet. One of the first tasks each team had was to decide which data they wanted to investigate further. While this decentralized approach is far from systematic, the rationale provided by the Principal is sound: by having staff "get their hands dirty" (in the data analysis process), teachers became more invested in the planning process and its implications for the direction of school improvement. The Principal reported that teams began a friendly competition over who could do the more sophisticated and comprehensive data analyses.

During this phase, teams were again asked to select and analyze a source of data, generate hypotheses regarding why the observed patterns exist, and identify strategies to address the gaps in student achievement that surfaced. The connection between the work of the Action Teams and the contents of the current plan are unclear-some of the work of the teams is more clearly reflected in the school improvement plan than others. Some teachers reported that their teams continue to meet to pilot programs that address the problems identified in the analysis of their data. For example, one team identified students' lack of vocabulary knowledge as a key barrier to improved achievement in both Reading and Writing. They are now in the process of developing a program to teach key vocabulary across the content areas. While this appears to be a promising strategy clearly related to the goal of improved student literacy, no mention of it appears in the plan. While it is perfectly reasonable to exclude some analyses from the plan as part of prioritization and equally acceptable, even commendable, to continue data analysis, the panel was not able to determine how these efforts are coordinated.

The process of deciding which teams' work became a part of the plan and who has final authority regarding its contents is unclear. The group that attended the Massachusetts Department of Elementary and Secondary Education summer training, which was comprised of the principal, the vice principal in charge of curriculum, a literacy lead teacher, a central office administrator, and two classroom teachers, authored the initial plan. This group used some of the work of the Action Teams to author the initial school improvement plan, presented it to the staff for feedback (at a faculty meeting), revised the plan, and again presented it to staff. This iterative process seems to have engendered general support for the direction spelled out in the school improvement plan. It is not clear, however, who has the final say regarding the school improvement plan. The revisions that occurred between the time the school sent their school improvement plan to the Massachusetts Department of Elementary and Secondary Education and the time the visit occurred (when the review team was presented with a new school improvement plan) were authored by the Assistant Superintendent. Clarifying who has the responsibility to modify the plan in the future and how the work of the Action Teams fits into the school improvement plan planning process will be critical to ensuring its continued support and successful implementation.

Although the administrative leadership team reported that the plan would be modified on an "as needed" basis, the school has not identified any process for evaluating the plan. For example, the means of evaluating the implementation of several of the strategies for improvement rest on classroom observations. However, there is no indication of how classroom practices will be evaluated. Although the school has created a template for teacher lesson plans, the focus on reading and writing across the curriculum is not yet reflected in it. Because the plan lacks periodic progress indicators and tools to measure those indicators, it is unclear what future modifications will be based on. More clearly specifying both the strategies and progress indicators for the desired outcomes are critical steps in developing a meaningful strategy for monitoring plan implementation.

¹ Quoted from a letter from Commissioner Driscoll to Principal DiGregorio dated April 26, 2001
² School indicator data not yet available.

Panel Reponses to the Key Questions

Key Question 2: Are the Conditions in Place for the Successful Implementation of the Improvement Plan(s)?

The conditions for successfully implementing improvement initiatives appear to be in place, although there are concerns about the guidance provided by the district. The leadership at the school has the support of the vast majority of stakeholders, and faculty express confidence in the both the Principal and the direction outlined in the plan. Teachers and school administrators are generally "on the same page." While the district offers substantial fiscal support, and plans to provide additional programmatic support, the link between the Williams improvement plan and the district plan is not well-established.

A. Does the school have effective leadership and sound management?

The new principal of Williams Middle School is energetic, confident, and purposeful. She has made some important changes while maintaining the confidence and support of the majority of staff and parents interviewed by the review team. One of her most significant early decisions was to reorganize the school schedule, grouping teachers into teaching teams and teaching teams within school clusters. This provided common planning time for teachers in the same cluster, in order to increase collegiality and decrease teacher isolation. Although teachers interviewed by the panel review team reported quite different understandings of what the common planning time should be used for, and could probably benefit from more professional development and/or guidance in this area, they were generally enthusiastic about having this time to plan with their colleagues.

The school decision-making process is unclear to many school stakeholders. While there are many different groups who have been involved in the planning process, a number of teachers interviewed by the review team indicated that they did not know where and how various decisions are made.

B. Is there evidence that the school's faculty supports the planned improvement efforts?

The evidence gathered by the team clearly indicates that the majority of the school staff support the direction reflected in the school improvement plan and have confidence in the Principal. Although the plan lacks specificity, the broad involvement of the staff in its development appears to have engendered a good deal of enthusiasm and support for the focus on literacy. When asked directly whether the planned efforts will lead to improved student achievement, the majority of staff and parents interviewed by the team expressed confidence that it will.

There is not a shared understanding of the specific goals of some of the improvement initiatives. Because the improvement initiatives are under-specified it is unclear whether teachers have a good sense of the implications of the plan. For example, when asked to explain what a literate student can do, several answers emerged. One teacher said simply, "read, write, and comprehend," while another defined literate as the ability to "learn on your own and expand on your existing knowledge." Others explained, "Literacy grows through meaningful personal experiences," that a literate person "can read a newspaper," and "can read at grade level." The ambiguity surrounding the specific goals of the focus on literacy may undermine improvement efforts as new expectations of staff emerge (e.g., that teachers in all content areas need to be reading and writing teachers as well, and that they will be held accountable for doing so). Clarifying the improvement initiatives, the specific student learning goals, and how progress will be monitored on a regular basis represent important next steps for the Williams School.

C. Is the school receiving adequate guidance and support from the district leadership?

The support and guidance provided by district leadership is mixed. The Superintendent has committed significant resources to improving Williams Middle School. As indicated by the panel report from last spring, the Superintendent has combined Chelsea's two middle schools and created a new administrative structure with two vice principals (one focused on academics and one on the day-to-day operations), two deans of students, and one special education coordinator. She also hired an energetic principal, provided various forms of support such as a mentor (an experienced retired middle school principal) and given the new principal "carte blanche" in terms of the fiscal resources needed for staff professional development. Further, the district is in the process of hiring for a new administrative position that will focus on literacy. The Superintendent has also created a central office position to coordinate after school programs, which are a part of the Williams' school improvement effort.

Although key district personnel have been involved in the planning process and the district has been generous in its fiscal support of several of the initiatives described in the plan, the review team was greatly concerned about the poor alignment between the district's improvement efforts and those of the William's school and how the lack of clear alignment might affect the district's ability to provide clear guidance and oversight in the implementation of the Williams's plan. When asked about the direction the district is headed, the Superintendent identified "academic rigor" as central to her vision. She further elaborated several initiatives underway, including one directly tied to supporting efforts to increase student literacy at schools, the new administrative position devoted to secondary literacy. This is clearly in alignment with the school's efforts. However, it was unclear to the review team, how district administration envisions this person supporting the implementation of the Williams plan.

Alignment between district and school efforts was less clear in review of the planning documents. The district improvement plan was reported to be a synthesis of the all of the school improvement plans in the district. It includes some 17 goals, only one of which pertains to student achievement ("Improve test scores for students in the school system.") It is not clear how these distinct priorities at the district level will be helpful in guiding the development, implementation, and monitoring of the Williams plan. The district plan does not provide clear guidance regarding instructional priorities, sound strategies for assessing programmatic effectiveness, or other forms of support that would directly help Williams in the implementation of its improvement plan. In describing the process of creating the school improvement plan, no one referred to the district plan.

Conclusion

The Williams Middle School has a reasonably sound plan for improving student performance. While the plan lacks specificity in key areas (the lack of meaningful or timely progress indicators, ambiguity regarding the strategies to be used to address student weaknesses in Reading and Writing), it does provide a generally coherent framework for improving student achievement. Further, the process used to develop the plan appears to have engendered a good deal of enthusiasm regarding its implementation.

The conditions for successfully implementing improvement initiatives appear to be in place, although there are concerns about the guidance provided by the district. The leadership at the school has the support of the vast majority of stakeholders, and faculty express confidence in the both the Principal and the initiatives outlined in the plan. Teachers and school administrators are generally "on the same page." While the district has provided substantial fiscal support, and plans to provide additional programmatic support, district planning efforts do not appear to be fully aligned with the development and implementation of the Williams improvement plan.

Appendix A
Team Members

Tom Buffett, Panel Chairperson, SchoolWorks Consultant, Beverly, MA

Dave Deschamps, Classroom teacher, Hardwick Elementary School, Gilbertville, MA

Joyce Harrington, Assistant Superintendent, Bourne, MA

Anthony Pope, SchoolWorks Consultant, Beverly, MA

Appendix B
Evaluating School Performance
Detailed Schedule for Review Panel School Site Visit