In this memorandum I will review the background on my recommendations; comment on some of the concerns and issues raised; and provide a detailed outline of my proposed path forward.
We will incorporate into an upcoming procurement for a new MCAS contract1 the services needed to develop next-generation English language arts (ELA) and mathematics assessments, to be administered in all schools beginning in the spring of 2017. In order to expedite the development process and minimize costs, we will maximize the use of existing PARCC development, as well as MCAS test items, as appropriate. These will be augmented by additional test items developed to meet our needs. We remain committed to a policy of transparency with regard to releasing test items, as we currently do with MCAS.
Because of the time required to conduct a procurement for a new MCAS testing contractor, spring 2016 will need to be a transitional year for grades 3-8. Districts that administered PARCC in spring 2015 will administer PARCC again, and will again have the option to select the computer-based or paper-based versions. Districts that administered MCAS in spring 2015 will administer MCAS again, unless the district affirmatively elects to switch to PARCC (either computer-based or paper-based). The MCAS tests will be augmented with a limited number of PARCC test items to facilitate statewide comparisons and to provide teachers and students in MCAS districts with some initial exposure to these types of questions.
We will convene technical advisory committees representing Massachusetts K-12 teachers, higher education faculty, and assessment experts to advise on the content and test administration policies of the next-generation assessments. Among the policies to be reviewed are the content and length of our tests; the scheduling of test administration windows; our testing policies for students with disabilities and English language learners; and the requirements for the new high school competency determination.2 We will also discuss the timing for reinstituting a history and social science test.
As an adjunct to the test development process, we will convene review panels comprised of Massachusetts K-12 teachers and higher education faculty to review the current ELA and mathematics curriculum frameworks and identify any modifications or additions to ensure that the Commonwealth's standards match those of the most aspirational education systems in the world, thus representing a course of study that best prepares students for the 21st century.
We will commit to computer-based testing for our state assessments. A paper-based option will be made available through the spring 2018 administration, with a goal of implementing computer-based testing statewide by spring 2019. We will work with districts to help them identify funding sources for the needed technology.
As we did in spring 2015, districts administering PARCC in grades 3-8 for the first time in spring 2016 will be held harmless for any negative changes in their school and district accountability levels. In spring 2017, when we return to a single test for all districts, every district will be subject to accountability level adjustments.
For ELA and mathematics assessments at the high school level in spring 2016, we will offer only the current MCAS grade 10 tests, in order to focus our efforts on the new test development work. We will consult with our technical advisory committees to propose a broader range of high school testing options beginning in spring 2017. Our current MCAS graduation requirement will remain unchanged at least through the Class of 2019.
We will work to ensure that the new PARCC consortium memorandum of understanding, currently under development, fully protects our ability to use PARCC intellectual property in future Massachusetts-based tests.
We expect to remain an active member of the PARCC consortium. I anticipate that continued membership will give us access to high quality assessment research and new test items, with the costs shared among the participating states. Membership also will provide us with useful multi-state data comparisons. Because we will be contracting with our own testing vendor, we will have the flexibility to leave the consortium at any time that membership is no longer of added value to Massachusetts.
I agree that testing by itself does not improve instruction — but it provides essential information to support those improvement efforts.
The 2001 reauthorization of the federal Elementary and Secondary Education Act (popularly called "No Child Left Behind") added a federal mandate for annual statewide testing. Congress is currently considering proposals for a new reauthorization of this law, some of which reduce the federal testing requirement. If and when a new federal law is passed, it will give us an additional opportunity to review and reflect on our state testing program.
The Commonwealth has a constitutional obligation to ensure that all students have the opportunity to receive an adequate education.8 MCAS results are one of several sources of information the Department and the Board use to identify schools and districts that require some additional assistance or intervention from the state.
High quality assessments send important signals about the kinds of curriculum and instruction, teaching and learning that are reflected in the standards.
Teachers and administrators are provided with detailed analyses of student test results, offering useful information on what parts of their curriculum are effective and where instruction needs to be strengthened.
Test results also allow us to identify higher performing schools and districts and spotlight effective practices.
Parents deserve objective feedback on their children's progress through elementary and secondary school grades. When students are performing below their grade-level expectations, we hope that their MCAS score reports will prompt constructive conversations among parents, teachers, and guidance counselors.
Passing the tenth grade MCAS tests is one of the requirements for a student to receive a Massachusetts high school diploma. Before education reform, too many students, especially in our larger and poorer cities, were receiving diplomas without having even a basic foundation of skills and knowledge.
Finally, test scores help us to demonstrate our achievements and our progress to the Legislature and to the public at large. We spend more than $16 billion a year on K-12 public education in the Commonwealth. We have an obligation to demonstrate to the taxpayers that we are spending that money effectively.
"Our tests don't measure everything." I agree that we want our schools to foster many skills that are not easily measured on standardized statewide tests, for example, creativity or working with others cooperatively. But I also believe that English language arts and mathematics are foundational for success in all other areas. If our schools are not teaching students to be literate and numerate, they are failing those students, regardless of what other successes they may be having.
"Testing takes up too much time." This has been a very widely expressed concern, not only from the public but from educators as well. We have an obligation to ensure that the time required to administer state tests is the minimum necessary to obtain the information we need. But concern over "too much testing" also reflects on assessments selected by districts themselves, as well as classroom time spent in preparing for tests. Research indicates that the value of these activities varies widely. The Department has been studying the amount of time spent in districts on statewide assessments, and we will continue to be vigilant in this area as we encourage and assist districts in evaluating the usefulness of their own testing programs.
"Statewide tests put too much pressure on students." For students, MCAS is a "high stakes" test only in tenth grade, where it is part of the high school graduation requirement. There are no high stakes for students taking the test in the lower grades, so if these students are feeling undue pressure, it seems likely that it is coming from their teachers, principals, and parents. I understand that some educators feel anxiety when we ask how well their schools are performing, but we should expect that they are not sharing those anxieties with their students.
"Our tests are too difficult for students with disabilities and English language learners." We offer a range of accommodations, special tests, and testing policies for these students to reflect their unique needs. We will continue to work with the advocates for these groups to ensure that our testing program is fair. But I do not want to return to the days when we had low aspirations and expectations for these students.
"Testing in some subjects forces schools to deemphasize others." We currently administer statewide tests in English language arts, mathematics, and science. The 1993 education reform law also calls for tests in history and social science, foreign languages, and the arts. Adding additional tests is feasible but pushes against the concerns over too much testing time. There does appear to be considerable interest in reinstating the history and social science assessment, and I expect that we will have more discussion with the field on this topic in the months ahead.
"Private testing companies could misuse confidential student data." We have contracted with private testing companies for more than two decades to help administer our large-scale assessments, including MCAS. All use of confidential student data is subject to federal and state data privacy laws, and we make every effort to ensure that our contractors use best practices in data security. There is no evidence that any of our current testing contractors have misused confidential data, and it is unlikely that they would stay in business very long if they did.
In 2008, the Department began planning for a next-generation MCAS to replace the existing, ten-year-old tests. Data from our state higher education system regarding the high number of students requiring remedial courses pointed out the need for more rigorous assessments at the high school level to signal readiness for post-secondary work. At all grades, we wanted to provide added focus on critical thinking skills as well as factual knowledge, and we wanted to provide richer feedback to students and teachers on areas of strength and weakness. We wanted to explore options for a computer-based assessment, and we knew that changes would be needed to reflect the new ELA and mathematics frameworks then under development.
Budget constraints arising out of the Great Recession of the mid-2000s ended this effort before it got very far. But then the U.S. Department of Education offered funding from the American Recovery and Reinvestment Act to states that were willing to work together in partnership to develop state-of-the-art assessments. Two such multi-state consortia were established and funded: the Smarter Balanced Assessment Consortium (SBAC) and the PARCC consortium. Massachusetts was one of the founding members of the PARCC consortium. Our participation in this partnership offered the opportunity to pool our expertise with other states, share the costs of test development, and realize economies of scale in test administration.9
The governing board of the consortium is comprised of the chief state school officer of each member state. I was selected by my colleagues to chair the governing board meetings. Each state also provides the time and expertise of state agency staff, educators from the field, and higher education faculty, to participate in various leadership groups, advisory committees, and test development activities. Staff from our Student Assessment Services office have devoted a substantial amount of time to the PARCC project over the past five years.10
Test Content and Administration
Our current MCAS assessment includes ELA and mathematics tests in grades 3 through 8 and grade 10. PARCC also has ELA and mathematics tests in grades 3 through 8, but has a broader range of high school tests. There are ELA tests for grades 9, 10, and 11, and course-specific mathematics tests for algebra I, algebra II, and geometry.11
The content and design of the PARCC test items have proved to be of very high quality. The material is well aligned to the common core state standards and provides a richer assessment of reasoning and critical thinking skills than MCAS. Feedback on test content was generally positive from educators who were familiar with both tests. There is, however, room for improvement. There were some isolated instances of test questions that had editing errors or that simply could have been written more clearly (or using vocabulary more appropriate to the grade level). This is not an uncommon occurrence in the initial development of a new test; similar problems cropped up in the first years of our MCAS administration. We also noted that some of the PARCC tests did not have as good a balance in the difficulty of questions as we would like.
The use of time limits, in comparison to the untimed MCAS test, pleased many people because it helped to reduce the amount of time students spent in the test session. Others felt that it was a problem for some students. In general, a timed test with reasonably generous time limits is to be preferred. Whether the PARCC time limits meet that standard or require further adjustment is worth additional study.
The move to computer-based testing (CBT) probably occasioned more comment than the actual content of the test. Last spring's administration demonstrated the significant value of CBT. Test items can include richer and more engaging content and a greater range of accessibility features; tests can be scored more quickly and at a lower cost; and CBT reflects the reality that students in the 21st century are doing more keyboarding than handwriting. We also learned that there is a significant learning curve for test administrators in setting up and administering a computer-based test, but districts that did so in both 2014 and 2015 reported that the process was much smoother the second time. The Pearson testing platform performed extraordinarily well, handling millions of users with only scattered problems. Less satisfactory was the performance of the Pearson call center in handling those scattered problems; improvements are being implemented for 2016.
Until all schools have the necessary technology to administer a CBT, we will need to offer a paper version. But we need to help schools get that technology as soon as possible, not just for assessment but to support more individualized and creative instruction and learning. Today's students need to be technologically literate if they want to succeed in college or the workforce. Schools that do not make the effort to upgrade their technology will find themselves losing students to other schools and districts.
Reporting of Results
PARCC student results are reported in five performance bands, compared to four for MCAS. The standards for each performance band are set by the consortium, allowing for potentially useful comparisons of data among the participating states. In contrast, each state determines how the results will be used in its accountability systems. For example, in states such as Massachusetts that require high school students to pass a state test for graduation, the passing score would also be set by the state.
PARCC is developing an expanded set of reporting tools for use by teachers and administrators. These are intended to provide extensive and useful data to inform curriculum and instruction. Because the complete suite of reports has not yet been made available, we cannot evaluate their usefulness at this time.
In terms of reporting timeliness, first year results were delayed, as expected, due to the standard-setting process. Results in future years will be available earlier; however, the goal of having results by the end of the school year is not likely to be met in the near term. This is due to the decision to combine the two testing windows into a single window. Open-ended and essay questions, which take the longest to score, will now be given later in the year.
Additional Diagnostic Assessment Tools
In addition to the summative annual assessments that have been the focus of our efforts, the PARCC project also includes the development of diagnostic assessment tools that districts will be able to purchase for their own use on a voluntary basis. These tools have not yet been released, and the potential costs have not yet been determined. Because it is too soon to gauge the value of and level of interest in these tools, their availability is not a significant factor in my evaluation.
The total cost of our statewide assessment system is a small fraction of our total K-12 education spending (less than two-tenths of one percent), so I would argue that our decision should be based primarily on the quality of the assessments, not by transitional increases or decreases in that cost as we migrate to the next-generation tests. That said, the per pupil cost of the PARCC assessments is lower than our current MCAS costs, because: (a) the development costs were heavily subsidized by federal and foundation grants; (b) computer-based testing is less expensive to deliver and score; and (c) joining with other states provides economies of scale.12 All testing contracts are subject to periodic cost increases when they are re-bid. The current MCAS testing contract is in its last year; the PARCC testing contract runs through June 2018.13
The development costs for next-generation MCAS ELA and mathematics tests are difficult to project without conducting an actual procurement. Costs will depend in part on the length of the tests; the degree to which existing PARCC and MCAS items can be used; and the speed with which we move to all computer-based testing. Combining the new ELA and mathematics tests in the same contract with the MCAS science and legacy grade 10 tests will provide some economies. We can expect an incremental annual cost of several million dollars, to be applied for three or four years. Savings from even a partial move to computer-based testing will help to offset the development costs.
Once the procurement is conducted, we will be able to provide the Governor and the Legislature with accurate cost information to inform the state budget development.
Governance and Sustainability
Many of the concerns expressed about the PARCC assessment have focused more on the governance structure of the consortium and on its future prospects.
In addition to Massachusetts, the following are currently active members of the consortium: Colorado, District of Columbia, Illinois, Maryland, New Jersey, New Mexico, and Rhode Island.
Aside from Massachusetts, the other members have all committed to using PARCC as their state assessment and are clearly interested in continuing the enterprise. The memorandum of understanding that governs the consortium is scheduled to be renewed at the end of this calendar year; discussions are already underway on needed changes to update and improve the governance structure. In the event that the consortium disbanded for any reason at any time in the future, a process is in place to designate a third party to take over and manage the consortium's intellectual property (test items, scoring rubrics, standards, etc.) for the benefit of the members.
With respect to the consortium's decision making, policies are now set by the governing board, and I would expect that some form of that arrangement will continue. Because Massachusetts has had a leadership role in the consortium, there have been relatively few instances where we disagreed with a policy decision. Nevertheless, we do need to acknowledge that we are only one state with one vote, and there are no guarantees that the other states will always move in the direction that we think is appropriate.
The consortium has engaged a consultant, Bellwether Partners, to study and advise it on its structure going forward. A major focus is the development of options for states (both member and non-member) to access and use the PARCC test content without needing to give the complete assessment or needing to use the designated PARCC testing contractor. A number of states in addition to Massachusetts, as well as other educational entities, are interested in these options. I expect the consortium to issue a statement shortly in which the members express their support for this new direction.