Report of the Follow-up School Panel Review of the Dr. A. B. Consentino Middle School,
Haverhill, MA

Introduction

The Consentino Middle School was one of twelve middle schools referred for panel review in the spring of 2001 as a result of critically low levels of student performance on State MCAS assessments in 1998 and declining MCAS test results in 1999-2000. A Panel Review was conducted in March of 2001. At that time, the panel found:

The Consentino Middle School does not have a formal plan for improvement which sets forth clear goals, measurable benchmarks, timelines, identified resources and responsible parties. The present collection of improvement activities lacks sufficient focus and coordination to serve as a blueprint for better student results.¹

Upon consideration of the panel findings, the Commissioner deferred action on the determination of under-performance for a period of six months, and provided a $25,000 grant to support planning and school improvement efforts during that time. The Principal and a planning team from the Consentino Middle School participated in these facilitated work sessions during June, August, and September, at which Department technical assistance staff and data analysts guided the school's planning team through an inquiry-based process designed to help them develop a sound plan for improving student performance at their school. At the conclusion of the six-month deferral period, a follow-up review was conducted at the school by a four-member review panel.

The Scope of the Follow-up Review Process

The follow-up review panel's charge was to review the original panel's findings, analyze current data and written information on the school's improvement efforts, visit the school, and meet with school and district officials in order to advise the Commissioner of its findings relative to the same two key questions that guided the original panel review:

1. Does the school have a sound plan for improving student performance?
2. Are the conditions in place for the successful implementation of the school's improvement plan?

The follow-up panel's responses to the two key questions that defined the scope of their review are included in this report. These findings and conclusions are the product of the panel's analysis, discussion, and observation, based on the evidence available to them. A list of panel members who participated in the Consentino Middle School follow-up review is provided in Appendix A. A detailed schedule of the panel's activities is provided in Appendix B.

The panel's findings and conclusions on the two key questions will be forwarded to the Commissioner of Education for consideration in determining whether the Consentino Middle School is deemed to be under-performing.

Consentino Middle School Profile²

The Consentino Middle School enrolled 734 students in grades six through eight as of fall 2001. Thirty-nine percent of the students were eligible for free or reduced-price lunch, and 15 percent spoke a first language other than English. The school identified three percent of its students as Limited English Proficient. The student body is identified as predominantly white (69%) with sizable Hispanic (23%), African-American (5%), and Asian (3%) populations.

MCAS Test Results

Cycle 1 Results

Overall MCAS results showed a decline in 1999 and in 2000 were nearly identical to baseline scores. In comparing the 1998 baseline score to the 1999 and 2000 average, scores were flat in English Language Arts, decreased by two points in Mathematics, and increased by one point in Science & Technology. In 2000, 26 percent scored in the Failing performance level in English Language Arts, 70 percent scored in Failing in Mathematics, and 58 percent scored in Failing in Science & Technology.

2001 Results

A= Advanced
P= Proficient
NI= Needs Improvement
W= Warning (formerly Failing)

Panel Responses to the Key Questions

Key Question 1: does the School have a Sound Plan for Improving Student Performance?

Significant strides have been made at the Consentino School to put a school improvement plan in place for this academic year. With technical assistance from the Massachusetts Department of Elementary and Secondary Education during the summer, approximately twenty staff members worked to analyze data, identify gaps in learning, and create a plan intended to address those gaps. From this work, school staff identified five priority areas for school improvement and developed "Turnaround Plans" for three of these priority areas. While the plans represent a serious and substantive step forward, they do not reveal a solid understanding of the gaps in curriculum and instruction related to student learning nor do they set forth a coherent plan for changes in leadership, curriculum, and instruction that will address these gaps.

A. Has the school analyzed appropriate data and program information to accurately identify the gaps in student performance and determined why those gaps exist?

A volunteer team of Consentino staff worked over the summer in a school improvement planning process guided by the Massachusetts Department of Elementary and Secondary Education. Through this process, school staff used grade 8 MCAS scores in ELA and Math (2000), grade 7 ITBS scores in Reading, Language Arts, and Math (1999-2001), grade 6, 7, and 8 district writing assessment results, and grade 6 end-of-year math assessment results to identify gaps in student performance. This substantial effort on the part of the Consentino staff was a critical step in learning how to use data to identify student needs as part of school improvement planning. Through this process, the school identified five improvement areas, three of which were identified as priorities for this academic year: English language arts, math, and attendance. The school developed Turnaround Plans for each of these areas, using the process provided by the MA ESE. School climate and accountability were the two other areas to be addressed during the 2001-2002 school year.

The analysis and planning done to date represents a good faith beginning for Consentino in its efforts to improve student achievement. The school identified several areas of concern based on their analysis of student test data. This data-driven identification of the key areas of low student performance represents a solid first step in the school's efforts to improve. A critical missing link in the analysis is a serious, comprehensive effort to assess instruction, curriculum, and other school practices to determine how these factors may or may not be contributing to student performance. The link is partially made in some cases; the Turnaround Plan for mathematics does attempt to link curricular and instructional gaps to gaps in student performance, but the Plan for English Language Arts is less clear and does not strive to link school practices to student outcomes. Improvement in language arts relies heavily on the John Collins writing program.

Although reading was identified as "static" in terms of student growth, the school's analysis in this area is weak and does not address how curricular and instructional practices are tied to student performance. For example, the school identified vocabulary as a core weakness with poor reading skills as one of the underlying reasons for this weakness. The plan does not communicate that the school has a comprehensive understanding of students' gaps in reading skills or how the school's curriculum and instructional program may be related to those gaps. Also as part of its analysis of MCAS scores, the school reports that students faired poorly on multiple choice responses, yet it is not clear if these gaps were or were not associated with reading comprehension or other literacy skills.

Planning documents revealed inconsistent reading assessment practices at the school at this time. All teams are not presently using the Gates-MacGinitie. In the planning documents the school states a lack of information about the reading levels of students as an issue for analyzing student performance but does make use of available Gates-MacGinitie and ITBS data. The action plan for reading includes the full implementation of Gates-MacGinitie by the fall of 2002. In writing, the school identified a gap in students' ability to respond to open response questions (this is also identified in math). Weaknesses in grammar, language, general knowledge, and poverty were cited as the contributing factors. Again, gaps in curricular and instructional practices were not effectively considered in this analysis.

In general, the Mathematics Turnaround Plan provides a clear link between curriculum, instruction and student outcomes. It identifies staffing, curriculum, instructional, and professional development needs as contributing factors to the problems identified. The lack of math certified teachers is cited in the school's plan as a contributing factor to low performance in math. It was noted by the review team that the grouping of students in math (some are grouped by ability and some are not, depending on the will of grade-level teams) was not considered in the analysis of student performance in math. The Principal and some staff are clear proponents of heterogeneous grouping as a means of improving math performance, yet the school continues to practice homogeneous grouping in math for some students. This is the first year MCAS test data for heterogeneously grouped classes will be accessible, so no comparative analysis is presently available. The school presented no other analysis of the impact of various grouping practices on student performance in math.

Another weakness identified by the school in their analysis of student performance data was in the performance of English Language Learners (ELL) and Hispanic students. It is apparent from the written plans and from interviews with school personnel that there is no clear understanding of the relationship between the performance of ELL and Hispanic students and the school's curricular and instructional practices.

The school's Turnaround Plan for Attendance, the third priority area, does a credible job of identifying groups of students who have high rates of absenteeism. The school's analysis of absentee data identified students that: 1) live with one parent; 2) who receive free/reduced lunch; and, 3) were in Grade 7 had higher than average absentee rates. In identifying underlying causes, the school listed family lack of awareness and supports as possible reasons.

B. Does the plan set out specific improvement objectives that are grounded in the school's analysis of the reasons for poor student performance?

The efforts made by the school to analyze data and create action plans based on this analysis represent a good faith effort to move the school forward. The limits to the analysis of data as described in Section A, also limits the ability of the school to create objectives that result in the kinds of gains they are seeking to make in student performance. With that in mind, the school did set forth a series of improvement objectives related to their analysis of student performance. In reading, the school set forth several activities intended to bolster student reading. The activities represent good attempts to provide more support for reading, but they are discrete efforts that are not tied to a broader understanding of needs in reading and a comprehensive curricular and instructional approach to addressing those needs. At present the school has some assessment practices in the area of reading beyond MCAS, namely ITBS and some use of Gates-MacGinitie. The critical objective of assessing all students' reading skills through Gates-MacGinitie is not slated for full implementation until October 2002.

To address the low scores in open response questions and to address students' writing skills overall, the school set forth three objectives: writing across the curriculum, tools for the writing process, and the development and use of a writing prompt with rubrics. There is some confusion in the plan about the objective of addressing students' ability to answer open response items and the strategies and benchmarks that appear to primarily address skills needed in long composition.

In math, the action plan sets forth objectives in professional development, realignment of curriculum maps, and integrating open response writing in math. These efforts are clearly linked to the school's analysis gaps in student math performance. The one exception is that the lack of certified math teachers, a well noted area of concern, is not addressed in the school's plan.

The school attendance plan is comprised of the purchase and implementation of a school automatic caller system to address the issue of parental lack of awareness of absenteeism. This system was already in place at the time of the visit. To support this, objectives to communicate with parents about attendance and to monitor cases of chronic absenteeism are also set forth in the plan. The plan does not address fully the level of need for parent outreach and support that was identified in the school's analysis of the contributing factors.

C. In order to accomplish each improvement objective, does the plan specify strategies that appear likely to lead to improved student results?

The strategies set forth in the school's improvement plans are likely to lead to some improvements in student performance. At present, the school lacks a coherent vision for teaching and learning that can serve as the foundation for discrete efforts to improve student performance. The school lacks a coherent and cohesive curricular and instructional approach to reading based on the needs of students. While the infusion of John Collins writing across the curriculum is likely to provide more opportunities for students to write and to respond to open ended questions, it was not clear during this visit that the school has a solid understanding of the expectations for writing and how to instruct students in the process of developing those skills. The objective to develop and use a writing rubric is a valuable step in that taking place.

A critical missing link in the school's plan and in the school's discussion of planning is the integration of standards driven practices. The curriculum frameworks do not appear to serve as a core curricular and instructional foundation across the school. There is not a clear sense that there are shared school-wide expectations for student performance from grade to grade. The plans do not clearly link instruction, curriculum, and assessment practices with the learning standards inherent in the Massachusetts Frameworks. There appears to be a reliance on programs over standards-based practice.

There was a notable lack of awareness of research and current practice related to the needs the school is currently facing. Given the prevalence of research currently focused on meeting the literacy needs of middle school youth, on reaching out to families of middle school youth, and on standards driven practice, it was of concern to the panel review team that staff did not articulate an awareness of nor a link to bodies of research or best practice in this critical time of need at the school. One example is the low achievement of English language learners and Hispanic students.

D. Are the school's written improvement planning document (s) clear and specific enough to guide their implementation of planned improvement initiatives?

The school's plan sets forth several specific activities for implementation in each of the three targeted areas of reading, writing, and math. The activities represent a good first effort to increase student involvement in activities identified as weak in the school's analysis. The plan identifies responsible parties, resources needed, timelines, and implementation indicators for each improvement objective. The plan is clear about what teachers and staff will be doing in a broad sense and includes several credible activities to address student needs. However, the plan does not organize these discrete activities within an overall instructional design for reading writing and mathematics. In the panel's judgment, distinct improvement activities, however credible as independent strategies, are not likely to lead to improved student performance unless they are part of a cohesive approach. For example, the plan does not make clear to teachers how proposed writing, vocabulary, and reading strategies are part of a comprehensive approach to literacy. In summary, the plan has successfully begun to identify improvement strategies that relate to student needs but has yet to provide teachers with a framework that explains how improvement strategies are linked to an overall instructional approach for improving student achievement.

The plan does specify measures to gauge student progress over time, but the measures need to be more fully developed. In the case of mathematics, the plan depends almost solely on ITBS and MCAS for evaluation of student progress. In Language Arts, the practices are almost all school-based assessments, such as tests and quizzes. In both cases, these measures are good first steps but need to be linked to an overall approach to assessment.

E. Was the School Improvement Plan developed through a process that will support its successful implementation?

The Consentino administration took on the process of school improvement planning in a serious and committed fashion. The involvement of a school team in the ESE summer training, along with the establishment of school-based teams to further develop and support the plans represented a serious commitment on the part of the school and the district to the process of school improvement. The school team involved in the ESE training included school administration, guidance, a district curriculum administrator, and a social studies teacher. However, no English language arts or mathematics teachers attended the initial training.

The school, under the direction of an Assistant Principal, created five school-based teams to address each of the five problem areas identified in the training: English language arts, mathematics, attendance, school climate, and accountability (later named assessment). School staff indicated that three of the teams were successful in developing plans because they had data to work with.

Since that time, the Principal has introduced the plans to the entire staff through staff meetings and team meetings. Staff members were asked to review them and provide feedback. Some changes were made as a result of this review, primarily in pushing back some of the timelines. According to staff interviews, the teams are not presently operational, and it is unclear to staff how they will continue to play a role in ensuring that the improvement plans are being faithfully implemented. There is no clear and coherent system in place for the continuation and deepening of the planning work that has begun at Consentino.

There is evidence that activities in the school plans are being implemented. The school has purchased and implemented the Easy Caller system for reporting school absenteeism. Teachers have been trained in the John Collins writing program and are using the strategies across the curriculum with full implementation expected by December. Instances were noted of teachers addressing learning needs identified in the plans (multiple step problem solving, math concepts, etc).

The plans for professional development needed to ensure successful implementation are not well-developed. The two prominent mechanisms noted for professional development include workshop offerings and team based collegial work. It is not clear that all key staff will be trained, in what they will be trained, nor how they will be supported on an ongoing basis to ensure effective implementation of improvement efforts. In math, the school also uses a ESE consultant to provide assistance to teachers. There is no expert knowledge source built in to support reading efforts at the school. The approach does not incorporate best practices in professional development - focus, new knowledge, opportunities for practice, assessment, and refinement.

The faculty at Consentino has a good deal of choice in terms of their involvement in school improvement activities. Teachers still determine whether or not they will group students by ability. Teachers are asked to volunteer for school improvement work. If no English or Mathematics teachers are involved in key leadership for those plans, the likelihood of success is minimized. In a time of critical need, this policy for involvement in and adoption of improvement efforts may not be the most effective in leading the school toward successful implementation.

¹ Quoted from a letter from Commissioner Driscoll to Principal Boucher dated April 26, 2001.
² Most recent school indicator data not yet available.

Panel Responses to the Key Questions

Key Question 2: Are the Conditions in Place for the Successful Implementation of the Improvement Plan(s)?

Sound day-to-day management practices are in place at both the school and district levels; however, there is a lack of a common vision for teaching and learning that can effectively guide decision-making and planning throughout the system.

A. Does the school have effective leadership and sound management?

There is clear and unwavering confidence and support for the Principal and administrative team among the school staff and other stakeholders interviewed. Most people credit the school's positive morale, even in times of critical budget shortfall and uncertainty, to the Principal and the administrative team. The Principal is respected for his steadfast support given to the staff and his commitment to doing whatever it takes. There is a notable loyalty among staff to the school and to the Principal, in particular. The Superintendent communicated a high degree of confidence in the Principal as well. He appears to have engendered and maintained the commitment of his key stakeholders even in the midst of serious budget cuts and uncertainty.

The Principal has managed the process of school improvement well, given very formidable conditions. He advocated successfully with the Superintendent and School Board for school positions and acquisitions he deemed necessary to move the school forward. He ensured that there was back-to-back planning time built into the teachers' daily schedule to allow for team planning and collaboration. He took advantage of the opportunity presented by the ESE to train his school in school improvement planning. He is confident that the school will move forward. The administrative team has been successful in creating some of the conditions necessary for doing so. The school is a calm and conducive learning environment, and staff support and have confidence in one another.

While the Principal and his team are highly respected, there is a clear and critical gap in instructional leadership at the school. There is a diffusion of energy created by a void in common instructional vision and purpose at the school. There does not appear to be a clear and compelling vision for teaching and learning toward which all planning and decision-making are to be channeled. Interviews with stakeholders across the school revealed various notions of a school vision, with the most prominent being "to raise MCAS scores." This vision does not provide the kind of instructional direction that is needed at this time to move the school forward.

There is not a clear school-wide approach to instruction based on a solid rationale informed by research and data analysis. Consentino, at present, lacks a solid research and knowledge base by which it can effectively respond to the needs identified through data analysis and lacks the commitment to ensure that all stakeholders adhere to and support the school's vision. There was no articulation of how Consentino accesses research and expertise and then goes about developing and sustaining leadership to effectively support changes. This type of leadership was not clearly evident in the school's administrative team, nor in the School Site Council. There was some evidence of this leadership capacity within teaching staff.

B. Is there evidence that the school's faculty supports the planned improvement efforts?

There is clear evidence that the faculty supports the Principal. All stakeholders interviewed expressed support for and commitment to the planned improvement efforts. Stakeholders spoke highly of the efforts to implement John Collins and to address issues of rigor in the mathematics program. At the same time, there are indications that the support is not as widespread and deep as is verbally articulated. For example, some faculty indicated a desire to move out of mathematics teaching because of the low scores and pressure on that area. Teachers also spoke of wanting to be "off of the team" (school improvement team). There was also mention that "teachers will probably select the math program that is not standards based because they are afraid of it. They will probably choose a traditional program." While there is support for the school's leadership, these are all indications of wavering support and lack of commitment to the improvement plan.

C. Is the school receiving adequate guidance and support from the district leadership?

In times of severe budget shortfalls, the district has responded to the priority needs of Consentino as presented by the Principal to the Superintendent and to the School Board. The Superintendent expressed the utmost confidence in, and support of, the administration at Consentino.

The district of Haverhill is the midst of a severe financial crisis. This situation is a significant factor in the type and level of support that the district is providing to Consentino. The Superintendent is consumed in these budget negotiations on a daily basis, such that he is now reviewing and approving every expenditure and cannot forecast the district's budget from one week to the next. In addition, the first priority for the School Committee is the accreditation of the high school. With these two looming issues at hand at the district level, the support for Consentino, at present, is limited.

It appears that at the district level, much like at the school level, the bulk of the administrative focus is on management related issues. The district is necessarily focused on its budget crisis. Its stance on priorities for teaching and student learning are not sufficiently clear. The Superintendent explained that there are School Committee goals, but there is no operational district improvement plan to support or guide school improvement planning at the school level. The Superintendent indicated that he plans to use the Consentino plans as a model for other schools.

The Superintendent has restructured some of the roles and responsibilities at the district. There is a new Executive Director for Curriculum and Instruction. He indicated to the panel that this department's immediate priorities are the implementation of two new curricular series and the John Collins writing program. The professional development offered by the district consists of a menu of workshops for teachers to choose from. The quality of these offerings is high, yet he indicated a significant lack of enrollment in these offerings. Professional development for teachers at Consentino is not adequately supported at present by the district. In addition, the district indicated that while principals are expected to be instructional leaders of their school, the district has not done much to train them in that role.

Conclusion

The panel review team recognizes the enormous work and solid accomplishments made by the Consentino School in establishing school improvement plans. Staff learned, many for the first time, how to use data to identify gaps in performance and how to consider those gaps in light of planning for the future. The plans developed by the school to start the school year identify areas in need of improvement and articulate the ways in which the school is attempting to address them. However, these plans are likely to have limited success without a clear and coherent framework for teaching and learning at the Consentino. After having reviewed the plans and visited the school, the panel is unclear about what students are expected to know and be able to do and how the school's academic program is designed to meet their needs. The school's plans and actions to date represent a solid start to what should become a deeper and more comprehensive approach to improvement.

The issues of capacity at the school and district level are similar. Sound management practices are in place, however, there is a lack of a common vision for teaching and learning that can effectively guide decision-making and planning throughout the system. Decisions appear to be ad hoc, driven by the most pressing need at the moment rather than by a vision and long-term commitment to achieve it. It is not clear that staff fully understands standards driven instruction and the expectations for learning that are inherent in the Massachusetts Curriculum Frameworks. There are pockets of this knowledge at the Consentino Middle School, but no concrete mechanisms for engendering this leadership to affect school-wide change are in place. Given the current improvement plans and the capacity at the school and district to successfully implement them, the Consentino School students are likely to make some discrete, but not substantial, gains in achievement.

Appendix A
Team Members

Carol Keirstead, Panel Chairperson, Senior Research Associate, RMC Research Corporation, Portsmouth, NH

Jolane Roy, Teacher, Lowell Public Schools, Lowell, MA

Margaret Kelliher, Professional Development, Springfield Public Schools, Springfield, MA

Carolyn Richards, Testing and Assessment Coordinator, Lowell High School, Lowell, MA

Appendix B
Evaluating School Performance
Detailed Schedule for Review Panel School Site Visit