ARCHIVED INFORMATION Horizontal line

The Massachusetts Board of Elementary and Secondary Education

District Accountability Reviews - Process and Next Steps

To:
Members of the Board of Elementary and Secondary Education
From:
Mitchell Chester, Ed.D., Commissioner
Date:
February 24, 2011

Background

At the February 28 meeting of the Board of Elementary and Secondary Education, we will build on the reports the Board has received previously about District and School Accountability by focusing on the Level 3 and 4 district accountability review process. This month's report is designed to give Board members context about the district review process and ways the Department is using the review process to inform prevention work at Level 3 and intervention work at Level 4. No action is required at the February meeting.

At the March 22 Board meeting I will present the reviews completed this year of each of the four districts identified by the Board as "underperforming"/Level 4: Gill-Montague, Holyoke, Randolph and Southbridge. I will also present the semi-annual report by the monitor I appointed to oversee the Recovery Plan for the Fall River Public Schools, and a draft of the "road map" we are developing to guide Board and Department work when the Board identifies the Commonwealth's first Level 5 district(s) under the regulations adopted last spring. I will ask the Board at the April meeting to act on recommendations I will make for the future accountability level for each of these districts.

District Accountability Reviews

Mass. Gen. Laws Chapter 15, Section 55A, directs the Department to "review and report on the efforts of schools, charter schools and school districts, including regional school districts, to improve the academic achievement of their students." The law, which was enacted in 2008, transferred to the Department the district review responsibilities of the former Office of Educational Quality and Accountability (EQA) and Educational Management Audit Council (EMAC). Among other things, the statute directs the Department to "perform not less than 40 school district audits annually, not less than 75 per cent of which shall be in districts whose students achieve at low levels either in absolute terms or relative to districts that educate similar student populations." As we have discussed with the Board previously, the current appropriation for this purpose is not sufficient to accomplish what we need to do in this area, and we have identified it as a priority for funding in the FY2012 budget.

The Department's Center for School and District Accountability, under the direction of Eva Mitchell, leads the district review process. Consistent with our charge under the statute, we have conducted four types of accountability reviews:

All four types of reviews have been grounded in an examination of district performance on six standards, details of which can be found in Attachment #1:

The District Review Process

District reviews are conducted by teams of independent educational experts who are selected and trained by the Center for District and School Accountability. The training is designed to deepen reviewers' understanding of the six District Standards, the methods of qualitative and quantitative analysis used during the course of reviews, the conduct of classroom observations using the Learning Walk criteria, and the guidelines for writing high-priority evidence-based findings and recommendations.

Each review begins with a two-day document and data review at ESE. The Office of Planning and Research to Close Proficiency Gaps (OPR) presents to each team a detailed data analysis for the district and schools under review. DESE data presented to the team (and also made available to districts) includes student demographic, behavioral, performance and proficiency gap trends, as well as staffing, financial, and educator data. The team reviews and discusses the data as well as documents provided by DESE and by the district, to begin learning about the district's systems and outcomes and to develop questions for district staff and stakeholders during the onsite visit.

The document review is followed by a four-day onsite visit to the district and its schools. The review team typically conducts at least 35 hours of interviews with a variety of stakeholders including: district and school leaders and staff, school committee members, teachers' union leadership, municipal officials, and parents. A typical visit will also consist of at least 50 classroom observations using the standardized observation protocol. The onsite visit concludes with the review team's presentation of emerging themes to district leaders, and if available, the Regional Assistance Director from the District and School Assistance Center (DSAC).

After the onsite visit, the review team drafts findings for each standard, which they carefully review with each other during a team meeting at DESE and use to develop recommendations. The draft recommendations are refined and reviewed during a final meeting at ESE. A technical writer ensures the draft reports meet standards for the quality of evidence, internal consistency, accuracy, and adherence to writing guidelines. Draft report findings are sent to the district for review and factual correction before the report and recommendations are finalized and made public. See Attachment #3 for an overview of the review process.

A specifically designed review protocol guides each type of review. See Attachment #4 for the key questions that guide each type of review. The review protocol is provided to districts in advance of the review to help them prepare and to understand the purposes and process for the review; the review protocol also provides clarity and transparency about the parameters of the review for the public as well as for the members of the review team. A review coordinator offers a pre-visit to each district to help the superintendent and staff further understand the review process, answer questions they may have, and address any concerns or needs that arise.

The Center has continued the rigor of the district review process formerly conducted under the auspices of the Educational Management Audit Council (EMAC). At the same time, the Center has added much-needed components, notably:

To ensure that stakeholders in the community can gain perspective about the strengths and weaknesses of district performance, senior Department staff members have presented in person at open school committee meetings the findings and recommendations from the district reviews for Level 4 districts. Plans are being put in place to do the same for Level 3 district reports.

Findings from Level 3 District Reviews

The Center worked with an independent research analyst1 to examine the findings in each of eight Level 3 district reviews completed in 2010 to ascertain trends and help identify potential implications for policy and practice. The report, Level 3 Districts: key themes and challenges, identified three cross-cutting challenges that districts face:

In addition, the analysis uncovered relevant findings about Level 3 district challenges in each of the six district standards. See Attachment #7 for the complete report.

Implications of the trends emerging in Level 3 Districts

Based on the cross-cutting trends and findings, the research analyst identified eight "questions to consider" as DESE establishes its priorities and strategies for accountability, intervention and assistance moving forward:

  1. How can the soon-to-be-developed DESE model educator evaluation system provide incentives for leaders and teachers to improve the consistency and quality of the taught curriculum?
  2. How can the regional DSACs support more districts to use the most effective ESE-developed tools and protocols to build instructional quality and improve the consistency of instruction, e.g., coaching models, learning walks?
  3. How can the DSACs help districts share their best practices for building instructional capacity in schools and classrooms?
  4. How can DESE help districts ensure that their systems and practices actively support schools in meeting student needs?
  5. How can DESE give districts feedback based on the question: "How, and for what purpose, is the district designing systems to meet such needs as communication, refining curriculum and assessments, identifying and sharing best practices and removing practices that are not working, teaming and professional learning, monitoring and supervision, or evaluating administrator and teacher performance?
  6. How might DESE make districts aware of systems of support that provide greater capacity to respond to funding challenges and how might DESE support districts to develop them?
  7. How can DESE and DSACs encourage and support districts to make more extensive use of regional approaches to secure and/or deliver services in order to a) identify and eliminate inefficiencies, b) achieve economies of scale and/or c) meet needs for which, alone, the district lacks sufficient capacity and/or technical expertise?
  8. Is DESE's district review process robust enough in the areas of student support and financial management to detect the impact of tightening resources on districts' capacities to respond systemically to non-academic barriers to learning?

These questions raised by the analyst have been helpful to Department staff engaged in accountability, intervention and assistance work. In some cases, the questions confirmed issues already identified and for which plans and/or activities were already underway to address them. In other cases, the questions are pushing staff members to consider altering or adopting new strategies.

Below is a sampling of work underway to address one or more of the questions posed.

Leadership and Governance

Curriculum and Instruction

Assessment

Human Resources and Professional Development

Student Support

Financial and Asset Management

The Accelerated Improvement Planning and Monitoring Process

The trends emerging in Level 3 districts along with the review of the 2009 District Plans for School Intervention and the Level 4 District Reviews being completed this month suggest that many districts lack capacity to generate, implement and/or monitor sound district improvement plans that hold promise for rapidly accelerating the improvement in teaching that is essential if student achievement is to increase dramatically. As a result, the Centers for District and School Accountability and Targeted Assistance have collaborated to design the Accelerated Improvement Plan process. Described in some detail in Attachment #8, the Plan is being implemented this winter and spring in all four Level 4 districts. The planning template will be made available next fall through the regional DSACs to Level 3 districts.

Conclusion

The Center for School and District Accountability has refined the district review process in ways that are making the findings more useful both to district leaders and Department staff. The superintendent of a Level 3 district recently described the district review for his district as "stunningly informative." Staff from several of the Department's centers, notably the Center for Targeted Assistance, the Center for Research and Evaluation, and the Center for Finance, are contributing to the analysis of district efforts. At the same time, they and others are beginning to make systematic use of findings from the district reviews to inform their on-going work with the districts. These are all much-needed and welcome developments for the Department as we intensify our accountability, assistance and intervention efforts.

Deputy Commissioner Karla Baehr and Eva Mitchell, Director of our Center for School and District Accountability, will be present to offer highlights of the work to date and respond to Board members' questions.

Enclosures:

Download PDF Document
Attachment #1: District Standards and Indicators
Download PDF Document
Attachment #2: District Reviews to Date
Download Word Document
Attachment #3: District Review Process
Download PDF Document
Attachment #4: District Review Protocol
Download PDF Document
Attachment #5: Sample Finding
Download PDF Document
Attachment #6: Sample Recommendations
Download PDF Document
Attachment #7: Level 3 Summary Analysis
Download PDF Document
Attachment #8: Accelerated Improvement Plan