Family & Community Alternative Learning
Massachusetts Charter Schools

Achievement Comparison Study: An Analysis of 2001-2005 MCAS Performance

Executive Summary

Prepared by the Massachusetts Department of Elementary and Secondary Education - August 2006

In the fall of 2005, the Massachusetts Department of Elementary and Secondary Education contracted with the National Center for the Improvement of Educational Assessment (Center for Assessment) to conduct a comparative analysis of the Massachusetts Comprehensive Assessment System (MCAS) performance of Massachusetts charter schools and their counterpart sending districts. The Center for Assessment is an independent, non-profit organization with extensive experience in conducting statistical analyses of student performance data. The Center has previously conducted studies for over 25 states, the Council of Chief State School Officers, the US Department of Education, and other organizations.

The Massachusetts study consisted of two phases. The first phase consisted of direct comparisons of 2001 to 2005 MCAS results in English Language Arts and Mathematics between individual charter schools and their comparison sending districts (CSD). All fifty-six charter schools operating in Massachusetts in the 2004-2005 school year, regardless of each school's years of operation, were considered for the study. Of these, four were excluded-three because their enrollment was dispersed across large geographic regions without a statistically significant number of students attending from a CSD, and one because it had no students in tested grades. Of the fifty-two remaining schools, six were assigned multiple CSDs because they drew a significant portion of their students from more than one school district.

Major findings of the first phase of the study include:

• When there is a statistically significant difference in MCAS performance, it is much more likely to favor the charter school than the CSD.

• In both English Language Arts and Mathematics, at least 30 percent of the charter schools performed statistically significantly higher than their CSD in each year with the exception of 2001. In 2001, 19% of the charter schools performed statistically significantly higher than their CSD in English Language Arts and 26% in Mathematics.

• The percentage of charter schools performing higher than their CSD each year has remained fairly constant in English Language Arts and Mathematics while the number of charter schools and the number of students tested in charter schools has increased.

• The percentage of charter schools performing lower than their CSD has declined to approximately 10 percent in Mathematics and dropped below 10 percent in English Language Arts.

Similar patterns existed for all demographic subgroups, with the likelihood of the significant difference favoring the charter school being most prevalent for the African American, Hispanic, and Low Income subgroups.

Because eighteen charter schools were located within the city of Boston, the aggregate performance of those schools was also compared to the Boston Public Schools. In this comparison, the analysis showed that:

• Charter school students in Boston as a combined cohort have performed statistically significantly higher than students enrolled in the Boston Public Schools each year from 2001 to 2005 in both English Language Arts and Mathematics, except there was no statistically significant difference in performance in English Language Arts in 2001.

• Among the African American, Hispanic, Low Income, and Special Education subgroups, charter school performance was statistically significantly higher than the CSD in each year since 2002 in both content areas.

• Charter school students in the White subgroup performed statistically significantly higher than their counterparts in the Boston CSD in 3 of the 5 years in Mathematics and 4 of the 5 years in English Language Arts.

The analyses in the first phase compared static measures of student achievement in a charter school to its CSD. A key factor in evaluating school performance, however, is growth in the achievement of individual students over time. The second phase of the study used a statistical technique called hierarchical linear modeling to examine changes in individual student test scores for continuously enrolled students over time.

Results from the analysis of the data in this second phase include:

In English Language Arts:

• Statewide, for all schools and districts, average growth in MCAS scores across grades and years from 2001 to 2005 was modest: 1.10 scaled score points per year in English language arts.

• Ten charter schools and no CSDs had growth scores greater than 2.1 points (or more than 1 point above the state average).

• Six charter schools had growth scores that exceeded their CSD by 1 or more points, while six charter schools had growth scores lower than their CSD by 1 or more points.

In Mathematics:

• Statewide, for all charter and non-charter schools, average growth in MCAS across grades and years from 2001 to 2005 was modest: 1.51 scaled score points per year in mathematics.

• Ten charter schools and one CSD had growth scores greater than 2.51 (or more than 1 point above the state average).

• Fourteen charter schools had growth scores that exceeded their CSD by 1 or more points, while five charter schools had growth scores lower than their CSD by 1 or more points.

Several important points need to be kept in mind when interpreting the results of this study:

• The study uses student performance on the MCAS English language arts and mathematics tests as the measure of school and district performance. Although these are important measures, used for both the state competency determination and assessment of adequate yearly progress, there are many other dimensions of school and district performance.

• The study does not attempt to explain the reasons for any performance differences between charter schools and their CSDs.

• The demographic characteristics of students attending a particular charter school may differ from the demographic characteristics of the students in the CSD.