Assessment/Accountability
School and District Accountability

State law and regulations establish an accountability system that includes all public schools and districts in the state.

Beginning in Cycle III (school years 2002/03 and 2003/04), K-2 schools with no MCAS results will be rated based on the grade 3 MCAS results of the schools their "graduates" attend. Any school with a sample size below the minimum required for ratings will be linked with and receive AYP determinations on the basis of test results of the schools their students attend in subsequent grades.

Evidence: Statute and regulations; Cycle II State Summary Report; Cycle II notice to K-2 schools.

All schools are rated on performance and improvement using the same criteria, which are: ELA and mathematics proficiency indexes based on MCAS results, plus (beginning with Cycle III) attendance and graduation rate indicators. The definition and determination of AYP is integrated into our state accountability system.

Evidence: Statute and regulations; Cycle II explanatory materials (transmittal folder / power-point slides)

Accountability system is based on MCAS results. MCAS is a custom designed assessment program based on MA student learning standards set out in State Curriculum Frameworks. MCAS assessments report student results in four performance categories: Advanced, Proficient, Needs Improvement, Failing (HS ) / Warning (elementary and middle grades). Our performance levels are equivalent to those used in reporting NAEP results.

Evidence: MCAS explanatory materials

MCAS tests are administered in the spring (April and May) to permit assessment of the full year of student attainment at the tested grade level. Tests include essay, open response and multiple choice items. Open response items are scored over the summer, with teachers participating in the scoring process. The timeline for reporting MCAS results to schools, parents and the public has been accelerated over past two years by more than 60 days, from late November to mid-September.

Beginning in 2003, the Massachusetts Department of Elementary and Secondary Education will render preliminary AYP determinations and notify schools and districts of those determinations before the end of August of each year.

We will require districts, upon receipt of this notice, to notify the parents of all students who are assigned to a school that has been preliminarily identified for improvement, corrective action or restructuring of their school choice option. Parent notification will, under this plan, take place no later than the first week of each school year, in time for alternative school assignments to be arranged if requested.

Final school and district accountability reports and AYP determinations will be issued within 60 days after test results are returned. Once final accountability ratings and AYP determinations are issued we will revise the list of schools identified for improvement, corrective action or restructuring to reflect any additions or deletions resulting from analysis of the final determinations. Districts, upon receipt of final accountability results, will notify parents of the final results and make mid-year choice available in any cases where the preliminary AYP finding did not identify a school that, based on final results, is identified for improvement, corrective action or restructuring. In cases where a school was preliminarily identified but does not appear on the final list of schools identified for improvement, the school and its district will be so informed and the school will be relieved of prospective requirements. Any school choice commitments (i.e., transportation costs) that were made based on the preliminary identification will be honored for the balance of the school year.

Evidence: MCAS release schedule 2002; SDAS Press release 2002 announcing accountability system results.

School and district MCAS reports, as well as individual student reports are generated annually. Extensive information, including aggregate and disaggregated student assessment results and school and district performance ratings for every MA school and district is reported on ESE website. Downloadable, print-ready versions of LEA and school report cards are being developed. The target date for release for the first version of these NCLB compliant reports is March 15, 2003.

Beginning in 2003, student subgroup information, as well as aggregate information, will be included in all report cards. Teacher data will be produced locally for inclusion in 2003 and 2004 district and school report cards, and submitted in the aggregate for inclusion in state report cards. ESE is planning the development of a new teacher data warehouse. By 2005 we plan to have the data collection and storage systems in place to permit reliable State reporting of all teacher data we are required by federal law to include in annual state, district and school report cards.

Evidence: Sample MCAS reports; Sample Web-version school and district profiles; mock-up of new report card format

State law provides for sanctions when school or districts are determined to be under-performing. Sanctions range from required improvement planning with State oversight, to removal of school principal and reassignment of staff for schools and district receivership when under- performance is chronic. Recognition and rewards are also part of the MA school and district accountability system. Schools with exemplary improvement may be named Compass Schools. Compass schools receive $10,000 to assist with ongoing improvement initiatives and cover costs related to participation in effective practice dissemination activities. Privately funded cash awards are also given each year to principals of the most improved schools.

Evidence: G.L. c. 69, sec. 1J an 1K; 603 CMR 2.00 Regulations on Underperforming Schools and Districts; 2002 Pathways To Improved Student Performance report (listing 2001 and 2002 Compass Schools)

All students enrolled in public schools and those being educated in private schools at public expense are required to participate in the MCAS. Most students participate by taking the standard form of MCAS tests. A variety of accommodations are available to permit most students with disabilities to participate in the standard form of the test. Students with severe and complex disabilities for whom participation in the standard form of the MCAS tests is not feasible or educationally appropriate participate in the MCAS Alternate Assessment program.

With the exception of students who are medically excused, beginning in 2003 all students enrolled at the time of testing will be expected to participate in MCAS assessments. Make up sessions are given for students absent from school on scheduled testing dates. Results for all students except those participating in Alternate Assessment will be included in determining a school, district or subgroup proficiency index.

Beginning in Cycle III (2002 - 2004) we plan to include Alternate Assessment results in the school and district accountability system by use of an "Index of Alternate Assessment Attainment."

Evidence: MCAS and SDAS explanatory materials; Memo from Commissioner re: LEP testing and RFR for new ELP test design and administration

Effective in 2001, the ESE collects individual student data from local districts through our electronic Student Information Management System ("SIMS"). Districts are required to report student enrollment as of October 1 of each year. For purposes of the school and district accountability system, a student is considered to be enrolled in a school for a full year if he or she is enrolled as of October 1 of any school year and remains enrolled at that school up to and including the dates of MCAS test administration in the spring of that school year.

Evidence: MCAS and SDAS explanatory materials

From 1998 - 2002, test results for all students who took the MCAS tests at a particular school were included in determining the accountability system ratings and AYP determinations for that school.Beginning in Cycle III, schools will only be accountable for the spring MCAS test results of students who, according to SIMS records, were enrolled in their school on October 1 of that school year. District accountability results will reflect performance of ALL students who took the MCAS tests while enrolled in the district's schools, regardless of enrollment date in the district.

Evidence: Law and regulations; Memo from Director of Data Collection, Analysis and Reporting

Effective for Cycle II (evaluating performance during the 2000/01 and 2001/02 school years) our target date for accomplishing the objective of all students becoming proficient and advanced was moved up to 2014. All schools are rated, and AYP determinations are made, based on an analysis of the performance and improvement schools and districts demonstrate toward achieving this goal. School and district performance is assessed using a proficiency index that measures the extent to which students have achieved or are progressing toward proficiency in ELA/reading and mathematics. Separate determinations are made for each subject. Improvement is rated by measuring change over time relative to accomplishment of the objective of getting all students proficient or advanced in ELA/reading and mathematics by 2014.

Evidence: Cycle II explanatory materials.

Our AYP determination methodology was revised for Cycle II to incorporate the new NCLB target timeline. School and district AYP for the 2000/01 and 2001/02 school years was determined by comparing performance, for students in the aggregate, to State targets established by the NCLB formula, then comparing improvement to the rate of progress required to accomplish the goal of all students becoming proficient in ELA and mathematics by the year 2014. Beginning in Cycle III this methodology will also be used to determine AYP for subgroups.

Evidence: Cycle II explanatory materials.

We have calculated our starting points for ELA and math using NCLB prescribed methodology (rank ordered lists for ELA and mathematics with determination of performance level of school representing 20th percentile of state enrollment. 20th percentile schools were higher performing in both ELA and mathematics than our lowest performing subgroup).

Our State starting point for ELA is 39.7% proficient and advanced, which equates to a proficiency index of 70.7. For mathematics our starting point is 19.5% proficient and advanced, which equates to a proficiency index of 53. We used our State's NCLB "starting points" as our Cycle II targets for determining AYP for the years 2001 and 2002.

Evidence: State Starting Point Determination slides; Cycle II explanatory materials.

We have established State targets for ELA and math performance, progressing in equal increments from for each two-year period from 2002 - 2014 from our state starting points for ELA and Mathematics to our goals, by 2014, of having all students performing at proficient and advanced levels in these subjects. To make AYP districts, schools and subgroups must demonstrate student performance above the State target for that time period or show improvement at a rate that, projected forward, puts the school "on target" for getting all student to proficiency or above by 2014. . At mid-cycle the improvement goal for determining AYP will be 2/3 of improvement target for that two-year cycle.

Specific performance targets for attendance and graduation rate for each rating cycle between 2002 and 2014 will be set by the MA Board of Education in the spring of 2002 after consultation and review by practitioners and the public. We plan to propose chronic absence by individual students as the measure for which attendance targets will be set. On an interim basis, until we have the data required to calculate a four-year graduation rate, we plan to propose goals related to annual graduation rate (which, effective with the class of 2003, requires passing rate on MCAS tests) and dropout rates.

Evidence: State Starting Point Determination slides; Commissioner's memo to Board of Ed 12/10/02

We have established State targets for ELA and math performance, expressed as a school, district, or subgroup proficiency index target, for each two-year cycle from 2002 - 2014. A proficiency index of 100 indicates that 100% of students are performing at proficient or advanced levels. The MA performance targets for 2004 - 2014 are:

Performance targets for attendance and graduation rates will be set by Board during 2002.

Evidence: State Starting Point Determination slides; Commissioner's memo to Board of Ed 12/10/02

We issue school performance ratings every other year, in even numbered years.

Beginning in the fall of 2003 we will make mid-cycle AYP determinations for schools identified for improvement at the end of the last rating cycle (Cycle II). Those results will be included in the 2003-2004 school year state, district and school report cards. Schools that did not make AYP in 2003 will be so notified and informed of the consequences of that determination.

In 2004, we plan to issue end of cycle ratings and AYP determinations for 2003 and 2004 for all schools, districts and subgroups. Beginning in 2005 and continuing in odd numbered years thereafter we plan to issue mid-cycle AYP determinations for all schools, districts and subgroups. For mid-cycle determinations we plan to review progress in the first year of the rating period toward achievement of the targets for that cycle. AYP determinations will be made by comparing mid-cycle performance to the State target for each indicator. For schools below the target we will determine the adequacy of the rate of improvement demonstrated relative to meeting the goal of all students becoming proficient or advanced by 2014 and determine whether the NCLB safe-harbor criteria are met. On MCAS indicators, to make AYP at mid-cycle schools will be expected to show improvement equal to or greater than 2/3 of their target for the cycle.

Evidence: Planned SDAS Changes for Cycle III, Associate Commissioner Dow

Since 2001 we have reported subgroup results on MCAS tests. Using our new Student Information Management System, we are able to match student data with test results and tabulate results for all required subgroups. Subgroup ratings and AYP determinations for special education students, limited English proficient students, economically disadvantaged students, and students in racial and ethnic minority groups will be included in school and district accountability system reports beginning in 2003.

Evidence: Sample MCAS reports; Memo from General Counsel confirming Board vote 12/18/02

Using our new Student Information Management System, we are able to match student data with test results and tabulate results for all required subgroups. For each school and district, subgroup ratings and AYP determinations will be issued using the same reporting rules used to determine ratings and AYP determinations for students in the aggregate.

Evidence: Sample MCAS reports; Memo from General Counsel confirming Board vote 12/18/02

Most students with disabilities participate in MCAS testing, with accommodations if required. MCAS results for all disabled students who take standard MCAS tests are included in determination of a school, district or subgroup's proficiency index. Severely intellectually/cognitively impaired students for whom the standard MCAS test is not appropriate participate in the MCAS Alternate Assessment program. We plan to include the results of Alternate Assessments in the accountability system beginning in 2003 through introduction of an "index of alternative assessment attainment."

Evidence: Memo from Associate Commissioner Dow re: Planned SDAS Changes for Cycle III

Beginning in 2003 all LEP students will participate in the MCAS assessment program, either by taking the standard form of the test with accommodations as appropriate or by taking an alternative assessment that is aligned with state learning standards. Test results for those taking the standard MCAS test with or without accommodations will be included in determining a school, district, or subgroup proficiency index. LEP students that participate, beginning in 2004, in the MCAS program by taking an alternative version of the ELA test specifically designed to measure language and content acquisition by students with limited English proficiency will have their results included in school and district accountability determinations beginning in 2005 using by introduction of an index designed specifically to measure attainment using that assessment tool.

Evidence: MCAS explanatory materials; 11/19/02 Memo from Commissioner Driscoll to Board of Education re: plans for testing LEP students

To achieve reliability and validity of accountability decisions, our minimum sample size for reporting performance is an average of 20 students/year for each 2 year rating period, and no fewer than 15 students in any one year. We issue improvement ratings, using a standard error band of 2.5 points, for all schools and districts that have an average of 50 students/year and no fewer than 40 students in any one year. For schools with an average of 20 or more but fewer than 50 students/year, we use a custom determined error-band of up to 4.5 points to determine improvement ratings and only issue ratings when a 95% confidence level can be achieved. The minimum sample size rules apply to each year for which data is included and will apply, beginning in Cycle III, to our calculation of sub-group ratings and AYP determinations.

Evidence: Cycle II explanatory materials.

To ensure confidentiality of individual student information we do not report results for groups less than 10.

Evidence: MCAS explanatory materials.

For Cycles I and II, the MA school and district accountability system had student performance results on state MCAS tests as its only indicators. In conformance with NCLB, an additional indicator is being added [graduation rate for High Schools and attendance at elementary and middle school levels] for Cycle III.

Evidence: SDAS Framework and Cycle II explanatory materials

At its December 2002 meeting the MA Board of Education approved modification of the MA SDAS to include graduation rates as an additional indicator for high schools. We will begin reporting graduation rates using the NCLB definition in 2005, using SIMS data from 2001 - 2005. The rate will be calculated as follows:

# Graduates (with regular diploma) who completed high school in four years

Divided by

[# Graduates (same as above) + # of 9th grade dropouts/retentions + # 10th grade dropouts/retentions + # 11th grade dropouts/retentions + # 12th grade dropouts/retentions + # students who complete 12th grade without a regular diploma]

The data for each class will be tracked going forward from 9th grade. Dropouts are defined as students who leave school prior to graduation for reasons other than transfer to another school. Students who are retained in grade, and thus leave their original class, will not count toward the number of graduates, but will be included in the denominator as members of the original class.

In the interim, until SIMS data to perform the calculation set out above becomes available, we plan to define and use an alternative measure, based on schools' reported aggregate dropout and graduation data. The interim measure, for the class of 2003 and 2004, will be the percentage of students who took the 10th grade MCAS, who graduated with a competency determination (MCAS graduation standard) two years later. Students that transfer in or out of the school after the 10th grade test administration will not be included in the denominator or numerator.

# Graduates (with regular diploma) who took the 10th grade MCAS two years prior

(# of students enrolled at the time of the 10th grade test two years prior) - (# students who transferred in or out of the class since the 10th grade test)

Evidence: Memo from General Counsel confirming Board vote; Memo from Director of Data Collection, Analysis and Reporting explaining graduation rate calculation method.

At its December 2002 meeting the MA Board of Education approved modification of the MA SDAS to include attendance as an additional indicator for elementary and middle schools. State goals and school and district performance and improvement standards for this indicator will be decided in 2003. We plan to propose a standard and interim goals and annual improvement objectives related to reduction in the instance of chronic student absence rather than average attendance.

Evidence: Memo from General Counsel confirming Board vote.

Beginning in 2001, we now collect student data on an individual basis through the MA Student Information Management System (SIMS). The reliability of data reporting has been greatly enhanced through implementation of this system.

Evidence: SIMS Data Standards Handbook (excerpt)

Yes. During Cycle I we reported results separately for ELA, math, and science, then combined data to create an "Overall" rating. For Cycle II, we dropped the overall ratings and only reported subject specific ratings and AYP determinations. Going forward, we will issue separate ratings and AYP determinations for ELA and mathematics.

Evidence: Cycle II explanatory material

We use two years of data combined to increase reliability. Use of proficiency index in Cycle II increased reliability of decisions compared to criteria for Cycle I performance ratings. Proficiency index determinations are more stable than findings based simply on percent proficient and advanced.

Evidence: Cycle II explanatory materials

Our assessment system results have been validated using comparisons with other standardized tests and review of alignment with State standards. These results are the primary indicators on which AYP determinations are made. We use minimum sample size rules to ensure the reliability of AYP findings. We look at both absolute performance and improvement to ensure that valid judgement are being rendered with respect to the adequacy of progress by a school toward achieving state and federal student performance goals.

Evidence: MCAS technical documents and Achieve report published on ESE MCAS website

The proficiency index system allows us to incorporate results from new test administrations without disruption of accountability decisions. We require two years of baseline data, and two years in the rating period for determination of improvement.

Evidence: Cycle II explanatory materials.

We have reliable enrollment data from the MCAS testing program and from our new SIMS system. We routinely calculate and report participation rates in MCAS testing program. We began reporting participation rates on Cycle II accountability system reports.

Participation rates on the MCAS are calculated as follows:

# of students with test results
(includes those students absent without a medically documented excuse, given a minimum score)
# of students enrolled at the time of test administration
Students participating in the alternate assessment are included in the numerator and denominator.

Evidence: MCAS reports;

In Cycle II we began reporting participation rates on school rating reports. For Cycle III the 95% participation rate rule will be applied to making AYP decisions.

Evidence: Sample Cycle II school rating report