ARCHIVED INFORMATION Horizontal line

The Massachusetts Board of Elementary and Secondary Education

Update on District Accountability Reviews

To:
Members of the Board of Elementary and Secondary Education
From:
Mitchell D. Chester, Ed.D., Commissioner
Date:
September 10, 2010

The purpose of this memo is to provide members of the Board with background about the Department's work in the area of district accountability reviews, a responsibility assumed by the Department of Elementary and Secondary Education in 2008-2009.

Background

In August 2008, the Governor signed legislation that restored to the Department responsibility for district accountability. Under the new law, the Department was expected to complete 15 district reviews in 2008-2009 and 40 district reviews thereafter. The former Educational Quality Audit (EQA) process overseen by the Education Management Council (EMAC) was funded at the level of $3,000,000 when it was fully operational and responsible for district and charter school accountability reviews. Fiscal stress has led to reductions in funding. For 2010-11, funding is less than $1,000,000 which must pay for both district and charter school accountability reviews. The Department has completed all required charter school accountability reviews and half of the district accountability reviews envisioned by the law. To partially compensate for the lack of resources to complete full, on-site accountability reviews in 40 districts each year, the Department has developed for every district (and school) the District Analysis and Review Tool (DART), a potent new tool for accountability and assistance.

Overview of District Reviews conducted between January 2009 and January 2010

The Center for District and School Accountability (CDSA) in the Department of Elementary and Secondary Education conducts district reviews under Chapter 15, Section 55A of the Massachusetts General Laws. CDSA will have conducted 43 reviews in 39 districts from January 2009 through January 2011. The vast majority of reviews have been conducted in Level 3 and 4 districts.

In school year 2008-09, 15 district reviews were conducted in the following districts in this order:

  1. Boston
  2. Lawrence
  3. Springfield
  4. Brockton
  5. Lowell
  6. Worcester
  7. Holyoke
  8. New Bedford
  9. Fall River
  10. Agawam
  11. Quincy
  12. Chelsea
  13. Taunton
  14. Lexington
  15. Westwood

In school year 2009-10, 18 reviews were conducted in the following districts in this order:

  1. Dedham
  2. Malden
  3. Watertown
  4. Lowell
  5. Peabody
  6. Westborough
  7. Lynn
  8. Revere
  9. Worcester
  10. Orange
  11. Gardner
  12. Dracut
  13. Saugus
  14. Holbrook
  15. Haverhill
  16. Pittsfield
  17. Westfield
  18. Southbridge

In school year 2010-11, 20 district reviews are planned.
Reviews have been scheduled in this order:

  1. New Bedford
  2. Nantucket
  3. Winchendon
  4. Salem
  5. Webster
  6. Gill Montague
  7. Randolph
  8. Holyoke
  9. Chicopee

11 more district reviews will be conducted in selected districts with relatively high achievement compared to their peers.

The Review Process: overview of six types of district accountability review

All reviews are evidence-based and data-driven, consisting of a pre-review of district data and documents and a multi-day on-site visit with interviews, school visits and classroom observations. Review teams consist of independent specialists with expertise and experience in each of the standards. All reports include a description of the district context and background, demographic and student achievement data, and findings and recommendations.

The Center has designed 6 types of review processes to explore the practices of districts and schools with relatively low achievement relative to their peers as well as those with relatively high achievement relative to other districts that educate similar student populations, in accordance with the requirements of Chapter 15, Section 55A.

Three types of reviews focus on districts and schools with lower relative achievement. The purpose of these reviews is to provide an analysis of districts systems and practices and the impact to the quality of teaching and learning in the schools, and recommendations for district improvement, DESE assistance, and DESE intervention.

Three types of reviews focus on districts and schools with higher relative achievement. The purpose of these reviews is to explore how well district systems and practices support groups of students for whom a proficiency gap exists. Districts and schools with exemplary practices identified through the review process may serve as models for and provide support to other districts and schools.

Each of these reviews has a distinct purpose, set of focus areas, and a specially-designed methodology. To that end, comprehensive review protocols were developed to guide review team members and the district before, during, and after the review process; and, to ensure that reviews are fair and transparent.

A summary of each review type follows.

1. Level 4 District Review

Purpose:
Districts placed in Level 4 under 603 CMR 2.05(1), approved by the Board of Elementary and Secondary Education (Board) in April, 2010-or previously declared "underperforming" by the Board and placed on turnaround plans-will be reviewed periodically as determined by ESE. The purpose of this review of Level 4 districts is to provide the Department and the Board with information allowing them to assess the extent to which the district has strengthened its systems since its placement in Level 4 or the implementation of its turnaround plan, in order to determine future DESE assistance and intervention requirements.

Key Questions:
Four overarching key questions guide the work of the review team in these reviews.

  1. How has the district addressed the issues that placed it in Level 4?
  2. Is student achievement on the rise?
  3. Do the district and schools have strong systems and practices in place?
  4. Has the district built the capacity to maintain continuous improvement on its own, without continued assistance from DESE targeted to the district?

Methodology:
The review uses former district review reports, the district's turnaround plan, an analysis of the district's current systems and practices, and district and student data in order to assess the district's progress and its capacity to sustain improvements. To focus the analysis, reviews will collect evidence for each of the Key Questions. To answer Key Question 3, reviews will collect evidence for each of the six standards to be reviewed: Leadership and Governance, Curriculum and Instruction, Assessment, Human Resources and Professional Development, Student Support, and Financial and Asset Management. Team members will preview selected district documents and DESE data and reports before conducting a four-day site visit to the district and schools.

4 districts selected:

  1. Southbridge (spring 2010)
  2. Gill-Montague Regional School District (fall 2010)
  3. Holyoke Public Schools (fall 2010)
  4. Randolph Public Schools (fall 2010)

2. Level 3 District Reviews

Purpose:
Per Chapter 15, Section 55A of the Massachusetts General Laws. This review is focused on "districts whose students achieve at low levels either in absolute terms or relative to districts that educate similar populations." Districts subject to review in the 2010-2011 school year were districts in Level 3 of DESE's framework2 for district accountability and assistance in each of the state's six regions: Greater Boston, Berkshires, Northeast, Southeast, Central, and Pioneer Valley. The eight districts with the lowest aggregate performance and least movement in Composite Performance Index (CPI) in their regions were chosen from among those districts that were not exempt under Chapter 15, Section 55A, because another comprehensive review had been completed or was scheduled to take place within nine months of the planned reviews.

Methodology:
To focus the analysis, reviews collect evidence for each of the six standards: Leadership and Governance, Curriculum and Instruction, Assessment, Human Resources and Professional Development, Student Support, and Financial and Asset Management. The reviews seek to identify those systems and practices that may be impeding rapid improvement as well as those that are most likely to be contributing to positive results. Team members preview selected district documents and DESE data and reports before conducting a two-day site visit in the district and a two-day site visit to schools.

14 districts selected:

  1. Dracut (2010)
  2. Gardner (2010)
  3. Haverhill (2010)
  4. Holbrook (2010)
  5. Orange (2010)
  6. Pittsfield (2010)
  7. Westfield (2010)
  8. Saugus (2010)
  9. Chicopee (2011)
  10. 1Nantucket (2011)
  11. New Bedford (2011)
  12. Salem (2011)
  13. Winchendon (2011)
  14. Webster (2011)

3. District Plan for School Intervention review

Purpose:
The purpose of the DPSI review was to understand district efforts to support school intervention, as well as the strategic decisions made to support ongoing school improvement. It also sought to understand the impact of the Massachusetts Department of Elementary and Secondary Education (DESE) support efforts. These reviews also met statutory accountability requirements.

Methodology:
The review process was designed around the District Plan for School Intervention (DPSI), approved by the Board of Elementary and Secondary Education in June 2008. The DPSI, which has served as the guiding document to support and hold accountable Commonwealth Priority Schools (CPSs), is unique to each district and its schools. The DPSI served as the foundation for the review process, ensuring that each district's unique priorities, current improvement strategies, and key decisions are central to the review process. In addition, the review considered other key documents, processes, and initiatives that have been central to the development and implementation of district intervention strategies and Department support efforts in recent years. These include, for example: The District Leadership Report on the Essential Conditions, the State Review Panel report and Memorandum of Understanding (MOU) between the district and the state.

Key Questions:

8 districts selected:

  1. Boston (2009)
  2. Lawrence (2009)
  3. Springfield (2009)
  4. Brockton (2009)
  5. Lowell (2009)
  6. Worcester (2009)
  7. Holyoke (2009)
  8. New Bedford (2009)

4. District Systems and Practices: Special Education

Purpose:
The first set of District Systems and Practices reviews were conducted in May and June 2009, in Agawam, Chelsea, Lexington, Quincy, Taunton, and Westwood, districts where data points to responsive and flexible school systems that are effective in supporting all learners, particularly students with disabilities, or where there is an interest in making these systems more effective.

Key Questions:
Three overarching key questions guided the work of the review team.

Methodology:
To focus the analysis, the review collect evidence in three critical domains: (I) Leadership, (II) Curriculum Delivery, and (III) Human Resource Management and Professional Development. The reviews seek to identify those systems and practices that are most likely to be contributing to positive results, as well as those that may be impeding rapid improvement. Practices that are a part of these systems were identified from three sources: Educational Quality and Accountability standards, Program Quality Assurance Comprehensive Program Review criteria, and the 10 "essential conditions" in 603 CMR 2.03(6)(e). The three domains, organized by system with component practices, are detailed in Appendix F. Four team members consisted of independent consultants with expertise in district and school leadership, governance, and financial management (to respond to domain I); curriculum, instruction, and assessment (to respond to domain II); human resource management and professional development (to respond to domain III); and special education (to collect evidence across all three domains; see indicators in italics under each domain in Appendix F).

6 districts selected:

  1. Agawam (2009)
  2. Quincy (2009)
  3. Chelsea (2009)
  4. Taunton (2009)
  5. Lexington (2009)
  6. Westwood (2009)

The review process identified two districts with deeply-embedded, widespread strong practices that serve students with disabilities effectively: Agawam and Westwood. We are exploring ways to disseminate information about these districts' practices in meaningful ways.

5. District Systems and Practices: Limited English Proficient Students

Purpose:
Spring 2010 reviews aimed to identify district and school factors contributing to relatively high growth for limited English proficient (LEP) student performance in identified schools, to provide recommendations for improvement on district and school levels to maintain or accelerate the growth in student achievement, and to promote the dissemination of promising practices among Massachusetts public schools. The review was also a part of DESE's program to recognize schools as "distinguished schools" under section 1117(b) of the federal Elementary and Secondary Education Act, which allows states to use Title I funds to reward schools that significantly close proficiency gaps.

Methodology:
To focus the analysis, reviews explored five areas: Leadership and Governance, Curriculum and Instruction, Assessment, Human Resources and Professional Development, and Student Support. The reviews seek to identify those systems and practices that are most likely to be contributing to positive results, as well as those that may be impeding rapid improvement. Systems and practices that are likely to be contributing to positive results were identified from the DESE's District Standards and Indicators and from a draft report of the English Language Learners Sub-Committee of the Massachusetts Board of Elementary and Secondary Education's Committee on the Proficiency Gap3 .

11 districts selected:

  1. Dedham (2010)
  2. Malden (2010)
  3. Watertown (2010)
  4. Lowell (2010)
  5. Peabody (2010)
  6. Westborough (2010)
  7. Lynn (2010)
  8. Revere (2010)
  9. Worcester (2010)
  10. Pittsfield (2010)
  11. Westfield (2010)

6. District Systems and Practices: Students in Poverty

Purpose:
Spring 2011 reviews will identify district and school factors contributing to relatively high growth for low-income (FRL/Y) student performance in selected schools, to provide recommendations for improvement on district and school levels to maintain or accelerate the growth in student achievement, and to promote the dissemination of promising practices among Massachusetts public schools. This review will also be part of DESE's program to recognize schools as "distinguished schools," allowing districts and schools subject to review with Title I Academic Achievement funds.

Methodology:
Systems and practices that are likely to be contributing to positive results will be identified from among the DESE's District Standards and Indicators and from research on effective practices for students in poverty.

Districts selected: To be determined in October after the release and analysis of 2010 MCAS data

The District Analysis and Review Tool (DART)

Funding reductions have made it impossible for the Center to conduct 40 district reviews as envisioned in the law for last year or in the current year. At its peak, the operating budget for EQA was $3.1 million. The FY11 budget for the Center for School and District Accountability is $900,000.

To help compensate for the Department's inability to complete a full complement of district reviews, the Center for District and School Accountability collaborated with the Office of Planning and Research to Close Proficiency Gaps to develop DART (The District Analysis and Review Tool), a publicly-available annual snapshot of key trends for every district and school that will be used as part of the Department's accountability responsibilities.

The genesis for DART was the report from a number of district leaders that a potent aspect of the former "EQA" district review process was the report's presentation of district data in new ways. They reported that the charts helped them see trends and comparisons with other "like" districts that had not been apparent previously. They regretted that they had had to wait for an "EQA Review" to learn about these ways of looking at data.

That insight led the Department to develop the DART for every district every year, so no district would have to wait for a district review to gain access to potent data. It is an annual "report card" showing a snapshot of important indicators of performance and process over a five year period. In that sense, it is a powerful tool for accountability. Just as importantly, though, it is a powerful tool for assistance because it is interactive: district and school teams can look at their trends and compare them with the trends in any other district or school in the Commonwealth. They can identify "positive outliers" to learn about practices that can inform their own work. So, too, can the Department.

The Department introduced DART in winter 2010 through regional presentations to district leaders conducted through the District and School Assistance Centers (DSAC). The Office of Planning and Research to Close Proficiency Gaps collected feedback and made modifications. DART is available on the DESE website and will be updated as new data is made available.

Next Steps

During our November Board meeting, I expect to report on the findings from the reviews of Level 3 districts, districts reviewed for practices supporting students with disabilities, and districts reviewed for practices supporting English language learners. The summary and analysis of the review findings is being prepared by an independent third party and will include "lessons learned" from the best practices reviews, as well as policy implications from all of the reviews.

1 In other words, as Level 3 is now defined, districts with one or more schools that score in the lowest 20 percent statewide of schools serving common grade levels pursuant to 603 CMR 2.05(2)(a).

2 Please see definition of framework in Appendix G.

3 Halting the Race to the Bottom: Urgent Interventions for the Improvement of the Education of English Language Learners in Massachusetts and Selected Districts, December 2009