Report of the Follow-Up School Panel Review of the Edmond Talbot Middle School, Fall River, MA

Introduction

The Edmond Talbot Middle School was one of twelve middle schools referred for panel review in the Spring of 2001 as a result of critically low levels of student performance on State MCAS assessments in 1998 and declining MCAS results in 1999-2000. A Panel Review was conducted in March of 2001. At that time, the panel found there to be significant inadequacies in the Talbot School's plan for improving student achievement. In particular, panel members found:

The first part of the school's plan is a district-wide plan that specifies goals and strategies for all schools in the district, and the initiatives specified by Talbot under each goal are not clearly linked to identified school needs. The actions prescribed in the second part of the Talbot School plan are based on cursory analysis of data, are not detailed enough to clarify the action steps to be taken, do not always articulate measurement means tied to the desired outcomes, and do not outline meaningful timelines. Moreover, the panel is concerned that the school lacks the instructional leadership and supervision to ensure that improvement efforts impact teaching and learning.¹

Upon consideration of the panel findings, the Commissioner deferred action on the determination of under-performance for a period of six months, and provided a $25,000 grant to support planning and school improvement efforts during that time. The Department provided the school with the opportunity to participate in a series of guided school improvement planning sessions in June, August, and September. The principal and a planning team from the Talbot School participated in these facilitated work sessions, at which Department technical assistance staff and data analysts guided the school's planning team through an inquiry-based process designed to help them develop a sound plan for improving student performance at their school. At the conclusion of the six month deferral period, a follow-up review was conducted at the school by a four member review panel.

The Scope of the Follow-up Review Process

The follow-up review panel's charge was to review the original panel's findings, analyze data and written information on the school's performance and improvement efforts since the time of the first panel review, visit the school, and meet with school and district officials in order to advise the Commissioner of its findings relative to the same two key questions that guided the original panel review:

1. Does the school have a sound plan for improving student performance?
2. Are the conditions in place for the successful implementation of the school's improvement plan?

The follow-up panel's responses to the two key questions that defined the scope of their review are included in this report. These findings and conclusions are the product of the panel's analysis, discussion, and observation, based on the evidence available to them. A list of panel members who participated in the Talbot Middle School follow-up review is provided in Appendix A. A detailed schedule of the panel's activities is provided in Appendix B.

The panel's findings and conclusions on the two key questions will be forwarded to the Commissioner of Education for consideration in determining whether the Edmond Talbot Middle School is deemed to be under-performing.

Edmond P. Talbot Middle School Profile

Talbot Middle School enrolled 780 students in grades six through eight as of fall 2001. Fifty-two percent of the students were eligible for free or reduced-price lunch, and 39 percent spoke a first language other than English. The school identified 14 percent of its students as Limited English Proficient. The student body is identified as predominantly white (77%) with sizable Asian (9%), Hispanic (7%), and African-American (6%) populations.

Students missed on average 15 days of school in 2000-2001, for a daily attendance rate of 91.7 percent. In that same year, 186 students (24%) received out-of-school suspensions, while 436 (55%) were suspended in school. No students were excluded for more than 10 days. Additionally in 2001, no students dropped out, and 94 (12%) were retained in grade.

MCAS Test Results

Cycle I Results

MCAS results were flat over the period 1998- 2000, beginning and ending the cycle with an overall average score of 218.3. In each content area scaled scores were within a point of the 1999 & 2000 average. Failure rates were similar as well, falling slightly in Mathematics, rising slightly in Science & Technology and remaining the same in English Language Arts. In 1999 and 2000, two-thirds of Talbot's students have scored in the Failing performance level in Mathematics and Science & Technology, while 29 percent scored in Failing in English Language Arts. The one positive indication from the 2000 results was the near doubling (from 8% in 1998 & 1999 to 15%) of the percentage of students scoring in Proficient or Advanced in Science & Technology.

2001 Results

A= Advanced
P= Proficient
NI= Needs Improvement
W= Warning (formerly Failing)

Panel Reponses to the Key Questions

Key Question 1: does the School have a Sound Plan for Improving Student Performance?

As a result of intensive summer work, the current school improvement plan is sound. It is based on data analysis, has clear measurable goals, timelines, identified resources, responsible parties and staff ownership. The strengths of the plan significantly outweigh its weaknesses.

A. Has the school analyzed appropriate data and program information to accurately identify the gaps in student performance and determined why those gaps exist?

Yes. During the summer of 2001, over 65% of the Talbot staff and the school leadership team met to analyze MCAS and Terra Nova data for the purpose of refining their school improvement plan. The process of disaggregating data led to discussions of the causes of student achievement gaps. Given that 45% of the student population speaks a first language other than English and there is a significant special education population, discussions of data focused on language arts and math; specifically, vocabulary, critical reading skills across the content areas, basic math and number sense.

Since the summer, the Talbot school has added the STAR testing program for math and language arts. This program provides teachers with baseline information on each student in the form of individual student reports. From the reports, individual remedial packets are developed. Thus far, all students have been tested for math, and teachers now have individual student reports. According to the Principal, assessments from the STAR program reinforce the leadership team's findings regarding student learning weaknesses identified through MCAS and Terra Nova analysis conducted over the summer.

Reviewing the quality and adequacy of existing programs began during the summer. Content teachers scrutinized programs and the existing school improvement plan and then designed and/or revised strategies in each subject area. For example, their review of the math curriculum revealed that the Connected Math Program— a program that had been in place for 4 years— was not addressing all of the math content standards. Therefore, math teachers are now "using the data and designing strategies to plug in the holes." Program review also revealed inadequate curriculum to address students' need for vocabulary development and better reading comprehension skills. As a result, all teachers, with the support from the Title I teacher, will be implementing critical reading and vocabulary strategies across the curriculum. These strategies include, for example, word walls and reflective journals.

The school has a general understanding of the reasons for poor student performance, but further work could be done to ensure that all teachers have a good grasp of the analysis of data and that all aspects of the school's programming are reviewed. For example, during the panel review, teachers could only talk in vague terms about the reasons for poor student performance. In a focus group, teachers mentioned the students themselves, the task of curriculum alignment and the lack of teaching strategies for inclusion classrooms as some of the reasons for poor student performance. Teachers did not mention that the Talbot was the only middle school in the district with a bilingual program and that the school did not have in-house expertise on second language development. The panel concludes that a more critical examination of the issues surrounding the education of bilingual students would strengthen existing programmatic review.

B. Does the plan set out specific improvement objectives that are grounded in the school's analysis of the reasons for poor student performance?

Yes. An analysis of student gaps in achievement resulted in prioritizing improvement objectives. They include improving vocabulary and critical reading across the curriculum and improving computational skills. For example, one of the improvement objectives in math is students will "improve the raw score on number sense, relationships and computation questions by 3% by June 2002." All of the objectives relate directly to improving student performance on the MCAS. Moreover the objectives are specific and measurable.

C. In order to accomplish each improvement objective, does the plan specify strategies which appear likely to lead to improved student results?

Yes. The school has specified six major strategies that are likely to lead to improved student results. Two strategies focus on building level structure: the restructuring of common planning time and the implementation of a school-wide inclusion model. Four strategies focus on direct improvement of student results. They include: Critical reading strategies across the curriculum, Vocabulary development, Connected Math, and standards based instruction. With the possible exception of standards-based instruction, these strategies are being implemented with sufficient support and resources.

According to teachers, the leadership team and the administration, common planning time (CPT) has been a significant positive strategy for ultimately improving student performance. This is the first year that CPT has been instituted school-wide, and it provides needed time to problem solve and collaborate around new initiatives. Currently a variety of issues are discussed during CPT from the needs of individual students, to instructional strategies, to dealing with world events. In the future, CPT will include looking at student work. All teachers meet in CPT teams minimally once a week and many teams meet more frequently. Those teachers that teach more than one grade level have difficulty getting to all grade level meetings but attend at least one per week. The Principal, assistant Principals and district coordinator attend and are actively involved in CPT meetings as they set the agenda with teacher input.

The Talbot is implementing a school-wide inclusion model for special education students. Generally, staff is supportive of this programmatic shift but there is acknowledgment that more training and school-based support is needed. A few teachers interviewed commented about their insecurity implementing inclusion. Between 1998 and 2001, the district provided 6000 hours of professional development in inclusion practices. Approximately 1000 teachers have participated in one or more hours of training. A number of Talbot staff has been to at least one workshop on this topic; however, there is some concern that in order to have success with this model more school-based support may be necessary. The school is providing two workshops this school year with an outside consultant on inclusion. The faculty's willingness to move to a full inclusion model is evidence of its collaborative nature and willingness to adjust instructional practices to students' needs.

Critical reading is an instructional strategy that is targeted to improve reading across the curriculum. A special education inclusion teacher and the Title I specialist provide in-house expertise on critical reading, and they are presently providing 10 hours of training for teachers in each content area. Critical strategies include the explicit teaching of vocabulary with specific connection to content materials. There is high teacher turn out for this training. Math teachers have completed their training with all but two teachers participating. Science teachers are currently being trained. Social Studies and Language Arts will follow. The ESL, special education and alternative teachers are participating as well.

The development of content-specific and standardized test vocabulary is another school improvement strategy. The decision to focus on vocabulary development was driven by an analysis of MCAS and Terra Nova tests and the fact that 45% of the school population comes from families whose primary language is not English. Included in the vocabulary development strategies are plans to identify MCAS specific vocabulary, vocabulary notebooks to be used by all students in each content area, sufficient dictionaries for students in class and at home, and the purchase of libraries to support group and independent reading.

Connected Math (CMP) has been used for the past four years, and most teachers have been trained in its use. The faculty generally supports the program. Results from Terra Nova and MCAS have been used to evaluate CMP, and an analysis revealed the need to supplement CMP with basic math materials. The STAR program, a computer-based learning tool, is fulfilling this need. STAR assesses each student's needs and creates an individualized remedial packet to address those needs.

Instruction and assessment that is more closely aligned with the state frameworks is the final strategy that is being implemented to improve student performance. The panel has concerns that district capacity may not be sufficient to implement this strategy in a timely manner. The district has been working on finalizing the curriculum documents, and standards have been created for the four major subject areas. However, teaching strategies and assessments have yet to be added to the standards to create useful curriculum guides. Curriculum guides need to be completed to move forward effectively with standards-based teaching.

A District Curriculum Coordinator is assigned to the Talbot as one of her 12 schools in the district. At the school, the District Coordinator is responsible for running monthly curriculum meetings, attending CPT, and visiting classrooms. Teachers and administrators commented on the strong positive relationship between the District Coordinator and the school faculty and the importance of her efforts to assist them in the school improvement process. However, the panelists remain concerned that the Coordinator, given her demanding work load, does not have sufficient time to provide the necessary support at the school.

The above strategies represent the school's efforts towards improving student performance. In the aggregate, the school's strategies appear likely to lead to improved student performance. For some of the strategies --- inclusion and standards based teaching --- there is some concern among the panelists that there may not be enough school-based or district-based support to ensure success. While there is some professional development offered by the district it is not comprehensive for the long term. The Talbot is maximizing their in-house expertise but change in classroom practices frequently requires on-site coaching for some teachers.

D. Are the school's written improvement planning document (s) clear and specific enough to guide their implementation of planned improvement initiatives?

Yes. The plan's strengths significantly exceed its weaknesses. It has identified areas of need, a clear focus, stated objectives, action steps, responsibilities timelines, implementation steps and benchmarks. The plan is well written, and there is evidence that it is already guiding improvement efforts.

The plan could better assist future school improvement efforts if benchmarks and checkpoints for monitoring progress were clearer. Expansion of the implementation steps could include more detailed action steps. Finally, the timelines could be more specific; there are several places where the timeline merely states the span of the school year.

E. Was the School Improvement Plan developed through a process that will support its successful implementation?

Yes. There is strong evidence of an inclusive and effective school planning process that began in June 2001 and continued throughout the summer. The summer planning began with the creation of a leadership team that worked closely with the administration and teachers to ensure input from all who wanted to contribute. Over 65 % of the teachers participated in brainstorming sessions, data analysis, and plan development. Content area teachers met to scrutinize subject area concerns. At the start of the school year, there was sufficient ownership and understanding of the newly revised plan. The whole staff was encouraged to review the plan and give feedback. One teacher interviewed said, "I feel a strong ownership of the plan." During a teacher focus group one teacher commented, "It's our plan." Another teacher said, "I couldn't participate this summer, but I own [the plan]."

The summer work resulted in a critical mass of teachers starting school in September 2001 with knowledge of how data informs planning and an understanding of the school improvement plan. There is evidence of an attitude shift towards the benefits of using data to inform planning and instruction. Comments about the benefits of using data emerged frequently during interviews with the leadership team, Principal and teachers. Teachers have begun to see that "data analysis makes their job easier." The Talbot School has been using the services of the high school data center that provides data analysis services to the district. They realize that data analysis is fundamental to planning and now see that it is also a 'new way to look at teaching practices.' With the assistance of the high school data center, MCAS data analysis will be the focus of the November 2001 district-sponsored release day.

The school improvement plan calls for on-going evaluation through the use of performance benchmark outcomes. To date, there is no benchmark data. Plans to look at student work during CPT have yet to be realized. The panel review team has some concern that on-going evaluation is too informal at this stage.

Key Question 2: Are the Conditions in Place for the Successful Implementation of the Improvement Plan?

The conditions are generally in place for the successful implementation of the plan. School leaders guide the process with confidence, energy and purpose, and there is broad faculty support for the plan. The district is supportive of the changes being made at the school, but it is not clear that the district has the necessary resources to provide sufficient curricular and instructional guidance to the school.

A. Does the school have effective leadership and sound management?

Yes. School leaders communicate the school's vision in a positive way. The school's Principal has a broad base of support from the district, teachers, students and those parents that are involved in the school. Several teachers, parents and students commented on the Principal's effective leadership skills. The Principal has approached the school improvement process as an opportunity for school growth. He works closely with the leadership team and supports their efforts to address the school's needs. The administration has high visibility. The Principal and Assistant Principals attend all common planning meetings and the Assistant Principals each are assigned to oversee a curricular area.

While the panel finds organizational leadership to be strong, it is concerned about the instructional leadership at the school. Despite the fact that the Principal, Assistant Principals and District Curriculum coordinator are active participants in all aspects of the school, there is some question of the expertise needed to be effective instructional leaders in a school with high numbers of diverse students. For example, the Principal, Assistant Principals and District Curriculum Coordinator do not have professional backgrounds in second language development. Talbot teachers are shifting to standards based instruction, inclusion, and critical reading strategies; many teachers would benefit from more on-going support.

Communication of professional development opportunities could be improved. There are a number of school based professional development opportunities that are articulated in the school improvement plan. The panel believes that these offerings could be communicated in a more systematic way. It is unclear how teachers learn about these offerings as there are no advanced advertisements detailing dates and times.

Those teachers and students that were interviewed commented on the positive school climate. Students indicated that they feel safe in the school. The panel review team observed a very positive school climate. The teachers were hardworking and positive, and the students were positive and respectful. The hallways were safe and welcoming; no small feat for a middle school with 750 students. Those parents that were interviewed spoke highly of the school and felt that their children were getting a good education. There could be more parent involvement. The Principal recognizes this need and has been seeking ways to involve more parents.

B. Is there evidence that the school's faculty supports the planned improvement efforts?

Yes. As stated previously more than 65% of the staff participated in the summer planning process. In a teacher focus group one teacher said, "Now we row the boat together," when discussing the effectiveness of the school improvement planning process. Another teacher mentioned that the school improvement process has given teachers a voice. In addition, the planning process appears to have changed the attitude of the majority of staff interviewed; there is a clearer focus of what each person needs to do to increase student success. Some teachers expressed reservations about funding to support ongoing improvement efforts but were confident that the "things that did not cost money in the plan would be done."

Classroom observations revealed additional evidence that staff is supporting the school improvement plan. One 8th grade class was using reflective journals in science; another class had MCAS test vocabulary displayed prominently. Panel members observed teachers implementing critical reading strategies and vocabulary instruction. There was also evidence of the use of word walls and vocabulary notebooks. In addition, all ten students interviewed were well aware of the MCAS tests and could talk about the specific skills they needed to do well on them.

C. Is the school receiving adequate guidance and support from the district leadership?

Interviews with the Superintendent and Assistant Superintendent for Curriculum and Instruction revealed a commitment to the Talbot's school improvement plan. The district has supported the Talbot's restructuring to ensure common planning time, supported the move to inclusion, made the high school data center available to the Talbot school, provided a curriculum coordinator, and talked about shifting finances in order that the instructional programs are funded properly.

While the district is making efforts to support the school improvement process, the panel has questions about the district's capacity to do so in a number of areas. First, the district curriculum is not finished. Instructional and assessment strategies are yet to be outlined in the curriculum guides. These documents are essential if teachers are to have the necessary tools to adopt a standards-based approach in the classroom.

Second, the district provides a number of professional development offerings; however, they do not appear to be strategic. For example, the district's professional development plan is menu driven; that is teachers select from a variety of offerings that suit their individual needs. Moreover, participation is not mandatory. Finally, there is some question as to whether the five release days are sufficient for school improvement efforts.

Third, the district has named program evaluation as one of its four priority areas. Aside from the high school data center and the District Coordinator's involvement in the school, there does not appear to be a formal system in place for the District to assist schools in monitoring their progress on improvement efforts or the efficacy of their programs.

The panel did not find sufficient personnel support from the district office to meet the needs of Limited English Proficient (LEP) and First Language Not English (FLNE) students at the Talbot School. The District Administrator responsible for LEP students is also responsible for coordinating new school construction efforts. The District Coordinator assigned to the Talbot does not have second language expertise. Although five teachers hold bilingual certification at the Talbot School, there is no one at the building level with the job responsibility to guide teachers in the instruction of these students whether they are in a bilingual class or integrated into the mainstream.

In summary, the panel found school and district leadership to be supportive and committed to improving student achievement at the Talbot. Actions at both the school and district levels are solid evidence of their sincere engagement in the improvement process. However, the panel has concerns that personnel capacity, particularly in the area of working with limited English proficiency students, may need to be augmented to meet the challenge of implementing the plan.

Conclusion

The Talbot Middle School has done a significant amount of work towards the development of an effective school improvement plan based on sound data analysis. There is substantial staff ownership and support for the plan. The staff is building on a solid climate of collaboration. Under the Principal's and leadership team's efforts the staff has used the planning process as an opportunity for growth.

With the exception of capacity to support instruction for English language learners, the conditions are in place to ensure successful implementation of the plan. The panel review team has confidence that the Talbot School will perpetuate a cycle of systematic feedback: from planning to implementation to monitoring improvements.

Appendix A
Team Members

Dr. Nancy Clair, Panel Chairperson, consultant for the Center for Applied Linguistics, Washington, D.C. and SchoolWorks, Beverly, MA.

Neal Brown, consultant for SchoolWorks, Beverly, MA. and coordinator for the Coalition for Essential Schools.

Donna Dixon, Asst. Principal, Ford Middle School, Acushnet, MA.

Hazel Grenham, Director Pupil Personnel, Chelsea Public Schools, Chelsea, Ma.

Appendix B
Evaluating School Performance
Detailed Schedule for Review Panel School Site Visit

¹ Quoted from the Commissioner's letter to the Edmond Talbot School dated April 26, 2001.