Report of the School Panel Review of the Rebecca Johnson K-8 School, Springfield, MA

Introduction

The purpose of the School Panel Review Process is to assist the Commissioner of Education in determining whether State intervention is needed to guide improvement efforts in schools where students' MCAS performance is critically low and no trend toward improved student performance is evident from MCAS data. The Rebecca Johnson K-8 School was one of twelve Massachusetts middle schools meeting this criterion that were selected for panel reviews in spring, 2001. The Panel Review of the Rebecca Johnson School was conducted on March 21, 2001.

The review panel's charge was to analyze data and written information on the school's performance and improvement efforts, visit the school, and meet with school and district officials in order to advise the Commissioner on the answers to the following two key questions:

1. Does the school have a sound plan for improving student performance?
2. Are the conditions in place for the successful implementation of the school's improvement plan?

The panel's responses to the two key questions that defined the scope of their review are included in this report. These findings and conclusions are the product of the panel's analysis, discussion, and observation, based on the evidence available to them. A list of panel members who participated in the Johnson school review is provided in Appendix A. A detailed schedule of the panel's activities is provided in Appendix B.

The panel's findings and conclusions on the two key questions will be forwarded to the Commissioner of Education for consideration, together with school performance data, in determining whether the Rebecca Johnson School is deemed under-performing. The panel was not asked to formulate a sound plan for school improvement where such a plan does not presently exist, or to recommend a course of action to create the conditions for successful implementation of sound improvement strategies where such conditions at present do not appear to exist. Diagnostic and/or prescriptive intervention, where needed to assist an under-performing school, occurs at the next stage of the school review process.

Rebecca Johnson School Profile

The Rebecca Johnson School is one of two Springfield schools serving students from kindergarten through grade eight. In addition, the school contains one of the district's 12 preschool programs. In 2000, the school enrolled 912 students, 40 percent of whom were enrolled in grades 6 to 8. Overall, 79 percent of the students were eligible for free or reduced-price lunch in 2000. The school reported that 24 percent spoke a first language other than English, and it identified 24 percent as Limited English Proficient. The school also reported that 51 percent of its students were Hispanic, 34 percent were African-American, and 15 percent were white.

In 1999 (the most recent year's data available to the panel), students missed on average 10 days of school, for a daily attendance rate of 94.5 percent. The school also reported no dropouts among its sixth to eighth graders that year. In 2000, 48 students (13.6%) in grades 6-8 received one or more out-of-school suspensions, and 128 (36.3%) were suspended in-school one or more times. The school reported seven incidents of student exclusion in 2000. In addition, eight students (2.3%) in grades 6-8 were repeating a grade in 2000.

Staffing

Rebecca Johnson's 393 middle school students are taught by 26 full-time teachers and 22 K-8 specialty teachers for an approximate average pupil-to-teacher ratio of 12 to 1. The central staff consists of Principal Veta A. Daley, three assistant principals, two guidance counselors, and a librarian/media specialist. Of the 45 full-time teachers, eight (18%) have attained a master's degree. Among the eight teaching math or general science, two hold degrees in their subject and one is certified to teach middle school in the subject. Three middle school math teachers possess elementary teaching certificates.

MCAS Results

Overall, grade 8 MCAS results have declined in all content areas since 1998. Comparing the 1998 baseline score to the 1999 and 2000 average, scores declined one point in English Language Arts, three points in Mathematics, and one point in Science & Technology. In 2000, 41 percent scored in the Failing performance level in English Language Arts, 86 percent scored in Failing in Mathematics, and 83 percent scored in Failing in Science & Technology. MCAS participation rates decreased in 2000, especially in ELA where approximately 11 percent of the regular and special education students were exempted from participating.

Panel Reponses to the Key Questions

Key Question 1: does the School Have a Sound Plan for Improving Student Performance?

A. Has the school analyzed appropriate data and program information to accurately identify the gaps in student performance and determined why those gaps exist?

The school has completed some analysis of its test data in an attempt to identify gaps in student performance. However, these analyses are limited in breadth and scope, particularly the MCAS item analysis. The school has proposed reasons for poor student performance, but many of those reasons are not the product of a deep analysis of student needs. In many aspects, there are weak connections between the products of the school's data analysis, proposed recommendations for further action, and the reasons mentioned by various school personnel as the likely cause of poor student performance.

In the report of its MCAS item analysis, staff identified specific topic areas where students performed poorly. In many cases, their attribution of a reason for particular weaknesses does not seem connected with the academic concern identified by the test scores. For example, Johnson eighth graders scored poorly in mathematics in the areas of Number Sense, Computation and Estimation, and Patterns and Functions. The report lists "regressive and progressive reasoning that have been applied appropriately" as the reason for the low scores. Recommendations responding to this identified need include examination of student's cognitive styles. In another section of the analysis, the reasons for student weaknesses in Mathematics -- Written Responses are listed as "not enough training in the early grades." In the Panelist's judgment, the school's analysis of available data is inadequate, and the reasons the school identifies for poor student performance have little relationship to student's academic weaknesses.

In the Principal's Leadership Report, she lists high mobility rates, lack of parental involvement, high teacher absenteeism and the high number of low functioning students as key barriers to improved student performance on assessment tests. Few of the responses she lists to address these barriers consider the impact of school-related factors, such as course curricula and instructional practices, on the quality of student performance. Her assessment of the needs of students differs in many ways from the recommendations of the MCAS analysis and her responses address student characteristics the school cannot change. Neither the Leadership Report nor the MCAS analysis look closely at actions the school can take to help students succeed.

Teacher survey responses list a third set of reasons for poor student performance, even more distinct from the Principal's response and from the MCAS analysis. Teachers identify inconsistent approaches to student discipline and ineffective leadership as the primary barriers to student success. When asked to elaborate, several teachers explained that the challenges they face managing large numbers of students with special needs overwhelms their ability to provide meaningful instruction. From their perspective, little academic improvement is possible until students and teachers receive the support they need to provide a safe and calm learning environment.

When considering the causes for the gaps in student performance, the school has not undertaken a systematic analysis of school programs to determine how weaknesses in teaching, curriculum, or class assignment might contribute to disappointing academic achievement. District staff rely heavily on data from periodic "learning walk-throughs" to assess the school's progress toward instructional improvement goals. However, the Panel suspects that the data from these visits is an unreliable measure of classroom practice. Students reported that teachers prepare for the "walk through" visits and present lessons unlike their regular instruction. When the Principal was asked to share the data from these visits, the only report made available for review was an anecdotal, generalized summary of the comments from district observers. Attributions of "70% improvement" heard from the Principal cannot be verified from the information available to the Review Panel. It is unclear what program data about instructional practices served as the source for these claims.

District level curriculum coordinators did report completing a review of staff qualifications for the content area they are assigned. Their review indicated weaknesses in teachers' content area knowledge, and, in response, district coordinators are planning additional content-focused professional development in coming years. Content area weaknesses among the faculty are not addressed as a possible reason for poor student performance in the current or previous Rebecca Johnson Improvement Plan, but it seems likely to be a contributing factor.

Many of the analyses provided to the Panel described the school's examination of the ITBS (Iowa Test of Basic Skills). The school's use of the ITBS as a measure of student performance is inappropriate in this application. ITBS compares students to a national norm, providing a generalized comparison of student's proficiency in basic skills to the proficiency of similar students in other settings. The ITBS cannot identify specific student knowledge gains or proficiency against particular standards such as those required in the MA Curriculum Frameworks. Student scores on ITBS may or may not compare with their MCAS performance, because the two tests measure different student characteristics. The ITBS gives the school a general idea that Johnson students are below the national average, but it does not provide critical program information about specific student strengths and weaknesses.

B. Does the plan set out a course of action that is: (1) responsive to the school leader's analysis of the nature and reasons for poor student performance, and (2) is likely to lead to improved student results?

The Rebecca Johnson School Improvement Plan reflects the goals identified by the school leader as meriting attention. However, there are discrepancies between the school leaders' identified needs and the recommendations listed in analyses of student performance. Additionally, the leaders' goals differ in some important aspects from those mentioned by teachers as well as those noted by the Superintendent.

The vague connection between the improvement strategies listed in the Rebecca Johnson School Improvement Plan and the needs of students reduces the likelihood that the plan will lead to improved student results. Some of the strategies, such as the school's association with the Coalition of Essential Schools, are described as 'promising' by the teaching staff. The Coalition strategies engage faculty in collaborative examination of student work and similar activities promoting professional growth. Some of the Coalition work focuses on instructional methods, other sessions focus on classroom management and classroom climate. Teachers judge these projects to have had a positive impact on their work.

Other strategies mentioned in the Johnson Improvement Plan receive mixed reviews. One effort begun last school year, the "learning walk-throughs," allows district staff and outside consultants to derive a general assessment of the learning activities in the building by 'walking through' the school and observing a brief segment of every class. District personnel rate this tool highly, citing its efficiency and accuracy in generating useful data for the school to use in refining its improvement efforts. Challenging the claim of accuracy, though, students interviewed for the Panel Review explained that teachers prepare for the "walk throughs" and for visits like ours by creating special lessons unlike their normal routine. One student complained that when we leave, they will be back to "reading out of books and doing worksheets" instead of the interesting labs and collaborative lessons planned for days when visitors are expected.

Some members of the faculty expressed appreciation for the individual feedback provided by the 'walk throughs.' They felt the reviewer's comments were constructive and helpful. Other staff complained that the "walk-throughs" are used to evaluate teachers based on a brief glimpse into a larger lesson session. Since the principal is held accountable for moving instructional methods toward more student-focused ones, some teachers expressed concern that the principal would be displeased if the "walk through" team was critical of an individual teacher's performance. Summaries of the Principal's report of the "walk through" to her staff reflect the mixed perceptions of the faculty, including some positive reinforcement for effective lessons and some bluntly critical statements for unsatisfactory performance.

In the leadership report submitted for the Panel Review, the Principal identifies the large influx of special needs students at the sixth grade as a significant barrier to success. However, a review of information provided by the school to the Department of Elementary and Secondary Education shows no dramatic change in student population by ethnicity, poverty or special needs from 1998 through the present. MCAS scores for all student groups show little change, with special needs students slightly increasing their average scaled score in English Language Arts and Science and Technology. The school's data does not support the Principal's interpretation of population changes as one of the causes of poor student performance.

One of the most revealing pieces of data that could indicate the likelihood that the Johnson Improvement plan will improve student performance is the effectiveness of the school's previous improvement efforts. Many of the strategies listed in the 2000-2003 plan are the same as those listed in the 1998-2000 Johnson Improvement Plan. Unfortunately, the school has not completed a systematic study of the impact of its past efforts and cannot reliably distinguish those that were helpful to students from those that were not. It is generally understood that similar efforts will yield similar results, and in the Panel's judgment, past improvement plans have not yielded higher student performance. Continuing those efforts in the new Improvement Plan without significant change is unlikely to support improved student achievement. The newest Johnson Improvement Plan describes no meaningful change in the school's previous strategies.

C. Was the School Improvement Plan developed through a process that will support its successful implementation?

The Johnson School Improvement Plan was developed by the SCDM (School Centered Decision-Making Team) consisting of the Principal, representative teachers, parent representatives and a community liaison from Bayer, Inc. The team constructed the draft under the Principal's guidance then brought the draft to the full faculty during Extended Day (75 minutes added to the school day). Faculty comments were solicited, and teachers confirmed that their ideas were incorporated within the suggested strategies under each improvement goal.

The school has been less effective in communicating the steps required for implementation of the strategies listed in the plan. Teachers report some awareness of what is expected of them, but they are unable to cite specific actions or measures that they are expected to take to help move the plan into action. Even though their ideas were included, teacher's exposure to the draft report has not elicited commitment from them to its beliefs and its achievement. Comments in interviews and in survey responses indicate that most faculty are passive reactors to the Improvement Plan, and they have little sense of responsibility for the plan's success.

Despite the mention of student surveys in the previous and current improvement plans, students interviewed during the visit reported limited knowledge of the school's improvement efforts and recalled no opportunities to have input into its content or direction. Students did speak emphatically about a decline in school climate, mentioning the elimination of school dances, the infrequent meetings of the student council, and expressing special disappointment in the loss of "weird hat days" as factors leading to poor student attitude about the school. One student complained, "We have nothing to look forward to." The students praised the teachers as caring and supportive - complaining that they will "not let you go on until you GET IT" - but expressed a desire for more school spirit and more opportunities to be heard. From the small sample interviewed during the visit, it does not appear that the school has fostered a sense of commitment and dedication to academic success among the middle school students.

Commitment to the Johnson School Improvement Plan is even more diffuse among parents. Those parents who participated in the development of the plan speak enthusiastically of its quality. However, among parents not involved in the plan's development, there is vague awareness of the plan's commitments or their role in achieving the plan's goals. Parents speak highly of their regard for the school and for their child's positive experience at Rebecca Johnson, but particular knowledge of the improvement plan is absent.

Based on the weak commitment of faculty, students and parents to the Johnson Improvement Plan, there are serious questions whether the plan has been developed within a process that will support its successful implementation. The community does not see the plan as a guiding document in school improvement efforts.

D. Is the School Improvement Plan document clear and specific?

The Rebecca Johnson School Improvement Plan lacks the essential elements necessary to make it a usable guide for school change. In particular, the plan does not demonstrate a clear linkage between strategies for change and reasons for poor student performance. It also lacks clear measurement of progress toward implementation and systems for evaluating the impact of the plan's strategies on student academic performance.

As presented to the Panel for review prior to the visit, the Johnson Action Plan does not contain a reliable analysis of the causal factors related to poor student performance. As mentioned in response to question 1A, there has been a general analysis of ITBS scores, a limited item analysis of MCAS scores, and some inquiry into teacher qualifications and instructional practices. The results of these analyses do not directly relate to the particular strategies listed in the Improvement Plan. For example, only one of the six objectives relate directly to academic programs. Objective 1 sets a target of 80% of students at Rebecca M. Johnson achieving at or above grade level across the curriculum. None of the strategies identified to achieve that goal define specific student needs, particularly the special needs of students with physical and cognitive challenges, as they relate to student learning targets. The MCAS analysis identified specific strengths and weaknesses in student performance, but that data is not referenced in the list of action steps within the academic performance objective.

The Johnson Improvement Plan also lacks the specificity and clarity to make it a usable tool for teachers and parents. In Objective 1, "positive reinforcement" is identified as the means to the goal of 80% performance at or above grade level, but no program or model is identified to link positive reinforcement to student academic improvement. The school plans to increase opportunities to share "best practices," but no realistic plan is indicated that would provide the time for teachers to meet or develop the expertise among the faculty to make that a useful activity.

The Plan presented to the panel before the visit did not contain critical components - a timeline for each major action, definition of persons responsible for management and oversight of each strategy and an indication of resources needed for implementation of the plan. On the day of the site visit, the Review Panel received an addendum to the Improvement Plan with a brief listing of these elements. The addendum identifies the Principal as the primary contact person for most strategies, even though many of the action steps relate to changes that are to occur in the classroom. The timeline on the addendum is broad and long-term, with no indication of checkpoints or progress/ status dates. Also missing from the Improvement Plan is an indication of meaningful ways to evaluate the impact of each action step on student performance. It was clear during the visit that the school has not conducted thorough analyses of its previous improvement plan, so the omission of an evaluation process in the current plan is consistent.

The major part of the Johnson 2000-2003 Improvement Plan is an intact duplication of the 1998-2000 plan. There is little evidence that the previous plans were enacted with any faithfulness or understanding, indicating that the current plan is similarly lacking in the clarity and specificity as well as commitment and dedication to make it a useful guide for school change.

Key Question 2: Are the Conditions in Place for the Successful Implementation of the Improvement Plan(s)?

A. Does the school have effective leadership and sound management?

The current school leader has not fostered a positive school climate, and there is evidence that communication is less than thorough within the school community. The Principal has imposed strict requirements on teachers and has made changes in school activities that have produced negative responses among students and parents. Support for the principal and faith in her leadership is limited.

The Superintendent confirmed that the appointment of the current Johnson Principal in 1998 was in response to the school's need for more firm direction. The current Principal has enacted significant reorganization and reassignment among the teaching staff, resulting in some discord and disappointment among faculty. In the two years since that reorganization took place, relationships between the Principal and staff have thawed somewhat, but teacher survey responses indicate lingering lack of support for the leader's efforts.

Interview statements and survey responses reveal a faculty deeply divided in their belief and confidence in the leaders' ability to bring about positive change in student achievement. In interviews and on surveys, teachers report their concerns are not heard and their points of view are not valued. Recently, teachers worked through the Springfield Teachers Association to express strongly held concerns to the Superintendent. A focal point of teachers' worries relates to classroom behavior as it affects theirs and their student's safety. One instance cited by a teacher describes a class of 13 special needs students with no inclusion aide. Daily outbursts are the norm in this class, with the outbursts sometimes becoming violent. Teachers are overwhelmed with the challenges of their students, and perceive little leadership support in finding viable solutions to these challenges. Written replies to teachers' concerns from the principal repeat her belief that teachers must solve the discipline problem by managing classroom behavior more effectively. Teachers agree that they need more effective strategies, but see no opportunity to learn or practice new methods under the current leadership. The discrepancy between the faculty and the principal on the causes and needs of students and teachers highlights increasing divisions among factions within the staff and between staff and the principal.

Parents are similarly divided in their awareness of the school's improvement efforts, indicating a gap in communication between the school leader and the parents. From the sample of parents available to the Review team, it appears that the Principal communicates with selected parents but does not provide helpful information widely to the larger community. One parent illustrated the impact of lack of information about school programs when she described her child's report that some of her friends are in the "dumb class." The parent was unable to determine why students would classify their groups this way and felt uninformed about actions the school may have taken to reorganize student groups. According to parents interviewed during the visit, the PTO has had limited effectiveness in serving as a communication vehicle. Many parents expressed a desire for more timely information about school practices and policies.

Not unexpectedly, students complain that the Principal has eliminated activities that promoted school spirit. They were unsure whether this loss of school spirit contributed to increased student absences, but they complained wholeheartedly about the fights among students and the problems teachers have with some student's behavior. Parents also cited instances where student behavior has been an ongoing problem, but they credit the school leadership with attempting to deal with discipline. The school's efforts have not been widely successful, however, and the Principal lists few plans for changing the climate and atmosphere to address student and parent concerns.

The Principal is supported by three Vice Principals, each of whom takes responsibility for a "floor" of the building, roughly aligned with primary, intermediate and middle grades classes. The Vice Principals are not included in the Site Council and were not represented on the SCDM team that developed the Rebecca Johnson Improvement Plan. Student discipline is the primary responsibility of the Vice Principals, with little indication of any other instructional or programmatic leadership duties. They did not describe significant involvement in defining the direction of school improvement efforts.

In general, the current school leadership has been ineffective in communicating school purposes and activities in a manner that would elicit support and understanding for those efforts. There is little commitment for the hard work of reform from many teachers, and students describe a school climate that is not likely to inspire extra effort towards high standards for academic performance.

B. Is there evidence that the school's faculty supports the planned improvement efforts?

From the conversations and reports available to the Panel Review team, it appears that faculty support the broad goals of the school improvement plan. While they are generally supportive of the plan's goals, they do not have a clear understanding of their roles and responsibilities in implementing the plan beyond general terms and phrases describing student-focused teaching.

Even among personnel involved in the development of the plan and in designing the strategies to achieve the plan, there is a clear lack of understanding of the specific actions teachers must take to reach the plan's outcomes. As noted previously, most of the 2000-2003 Johnson Improvement Plan repeats strategies listed in the 1998-2000 plan, so for most teachers there are no new reasons to expect change. In on-site interviews, teachers expressed genuine hope that the school would improve, but few could cite reasons why such improvement would be likely. They appear to make genuine attempts to follow the directives from the school leaders, but they face major barriers due to the percentage of challenging students in their classes.

Unfortunately, the Panel Review team was unable to assess the classroom climate at the school. Students were regrouped and reassigned for the panel visit, so the sample of classes visited during the day were unrepresentative of typical intact class groups. Teachers' concerns about disruptions and the distribution of special needs students could not be verified because of the reorganization during the visit.

Evidence of a lack of confidence in the school leaders' capacity undermines the likelihood that the plan will lead to its intended goals. Many faculty members respect the principal's role in defining the process for development of the improvement plan but there is little evidence of enthusiasm among staff to implement the plan's strategies. Teachers express sincere hope that the plan will help students reach achievement goals, but they communicate little confidence in their ability to overcome the daily challenges they and their students face.

C. Is the school receiving adequate guidance and support from district leadership?

The district has provided personnel, professional development, and financial support to the Rebecca Johnson K-8 School to support its improvement initiatives. District academic coordinators serve as primary curriculum contacts. Adoption of the principles of the Coalition of Essential Schools and the assignment of regular consultant visits to assist teachers in examining their own practice is highly praised by the faculty as supporting their work. The district supports the "walk through" strategy district-wide, under the guidance of a team of consultants from the University of Pittsburgh, as a tool to monitor classroom practices in each building.

In addition to existing Academic Coordinators, the district has committed to hiring Curriculum Resource Teachers in English/Language Arts and Mathematics to serve as resources in each middle school. These teachers will have a reduced teaching load in exchange for time spent mentoring their colleagues in areas to be defined. Another promising commitment from the district is support of Service Teams, a reorganized collection of existing staff designed to focus their attention on a subset of students for whom they have responsibility to monitor progress and prescribe support services. Rebecca Johnson did not to participate in the current year of this program, its pilot year, but the district will require involvement beginning next year.

The effectiveness of the district's support depends on the alignment between the support efforts and the school's needs. The Superintendent identifies communication within the school and between the school and the larger community as a primary need, an aspect of the school for which the district is providing no particular program. When asked what Rebecca Johnson needs to improve student academic performance, the Superintendent spoke about the school's schedule, explaining that student time in academic classes needed restructuring to support academic growth. Few school personnel mentioned student scheduling as a factor affecting student performance. The Panel members questioned whether the differences between the key factors identified by the district staff and those identified by the school dilutes the school's focus on important areas for change.

In general, the district provides important support for the school's efforts, extending the time commitment of district staff specifically to Rebecca Johnson to assist with its particular challenges. The district's efforts may be insufficient, however, if they are not based on accurate analyses of student and staff needs.

Conclusion

The Rebecca Johnson K-8 School Improvement Plan is lacking the detail and clarity to provide guidance for the school's improvement efforts. The plan's definition of student academic needs is based on an insufficient analysis of the school's programs and practices. Important differences in the needs identified by the Principal, the staff, and the district are likely to lead to unfocused attempts to reach academic performance goals. Lack of confidence in the school leader's ability to bring about change diminishes the capacity of the school to implement its plans. Communication between the school and parents has been insufficient to encourage commitment of parent support for the school's work. At the present time, the conditions are not in place to provide for the school to improve student achievement.

Appendix A

Team Members

Helene Bettencourt, Panel Coordinator, MA Department of Elementary and Secondary Education, Office of Accountability and Targeted Assistance

Dr. Karen Laba, Panel Chairperson, Project Manager, SchoolWorks, Beverly, MA

Terri Caffelle, Curriculum Specialist, Shrewsbury Public Schools, Shrewsbury, MA

Janice Johnson, Quadrant Manager, Worcester Public Schools, Worcester, MA

Patricia Poska, Lead Instructional Facilitator, Lynn Public Schools, Lynn, MA

Appendix B

Evaluating School Performance
Detailed Schedule for Review Panel School Site Visit

The times specified on the following schedule may be adjusted slightly to align with the daily schedule and practices in each of the schools being reviewed.

Please inform all school faculty and students that Review Panel members will be visiting a cross-section of classrooms during the site visit. The selection of classrooms will be determined mutually by the Panel Review Coordinator and the Principal using the staff directory information provided by the school. All faculty members are asked to be prepared to accommodate a visitor on the morning of the site visit. Panel members will make every effort to minimize the disruption of planned classroom activities.