District/School Administration Administration
The Massachusetts Board of Elementary and Secondary Education

Mathematics Subtest for Elementary and Special Education Teachers:
Determination of Passing Score and Proposed Amendment to Educator Licensure Regulations for Transition Period

Introduction and Overview

In April 2007, the Board of Education (as it was then called) adopted amendments to the Regulations for Educator Licensure and Preparation Program Approval for the purpose of strengthening preparation and licensing of elementary and special education teachers to teach mathematics. As a result, the regulations now specify the subject matter knowledge requirements in mathematics, focusing on outcomes rather than on a list of arts and sciences coursework. In December 2007, the Board approved and endorsed Guidelines for the Mathematical Preparation of Elementary Teachers and commended their use in all preparation programs for elementary and K-8 special education teachers. These actions are significant steps to improve students' proficiency in mathematics.

The 2007 regulations are now reflected in the Massachusetts Tests for Educator Licensure (MTEL). As of 2009, the revised General Curriculum test includes a separately scored mathematics subtest and a multi-subject subtest. At this month's meeting, I am presenting my decision for implementing a passing score (cut score) for each subtest. I am asking the Board to approve an emergency amendment to the regulations in order to implement the transition plan discussed below.

Massachusetts and most other states have long used multi-subject tests to determine whether candidates for licenses at the elementary level are sufficiently knowledgeable to be licensed and begin teaching in grades 1 to 6. Massachusetts is, however, the first state to include a separate requirement for passing a mathematics component of an elementary test for licensure.

The new test was administered for the first time on March 7, 2009. A panel of higher education faculty and public school educators was convened on March 25, 2009 to review and rate the General Curriculum test items in order to provide information for the Commissioner to use in establishing each subtest cut score. For the General Curriculum multi-subject subtest of language arts, history & social science, and science & technology engineering, we will implement the panel-recommended cut score. As applied to the examinees from the first test administration on March 7, just over one-half of the test-takers will pass the multi-subject subtest.

The mathematics subtest presents several policy challenges that I want to discuss with the Board, as well as my recommended approach for addressing them. The test is fair but demanding, as can be seen by the sample items in the Addendum. The panel-recommended cut score for the mathematics subtest was responsive to the Board's charge to ensure that candidates seeking to enter elementary school teaching in Massachusetts have a solid mathematical content knowledge base. Notwithstanding, implementing the panel-recommended cut score would result in only a 27% pass rate for individuals who took the math subtest on March 7.

The General Curriculum test is a requirement for both prospective elementary school teachers and special education teacher candidates. While there is currently an oversupply of elementary school teachers statewide, special education is an area of persistent and significant shortage. I believe it is important to uphold the recommended standard, and to do so in a manner that is sensitive to the need to build the capacity of aspiring educators and the institutions that prepare them.

Accordingly, after a thorough internal review, I have determined to:

1. Set the cut score for both the General Curriculum multi-subject and math subtests at the panel-recommended levels and determine that candidates who score at or above those levels have passed the subtest(s) and have met the knowledge requirements for their licenses. Candidates who pass one but not both subtests will not have to retake the subtest that they pass.

2. Establish a transition phase for this new and demanding math subtest. For a limited time (i.e., three years) we will deem those who earn a scaled score of at least 227-239 to have passed the subtest for the purpose of their first stage of licensure.

   These teachers, however, will have to retake and earn a scaled score of 240 or above on the math subtest in order to renew the license or move up to the next stage of licensure. Only upon meeting the panel's recommended cut score will they have fully met the knowledge requirements of the license.

   N.B. By statute, the preliminary and initial licenses are valid for five years of employment. Thus, such teachers would have five years in which to achieve the panel's recommended cut score and demonstrate that they have met the knowledge requirements of the license.

3. At the end of three years (beginning in July 2012), all candidates for the elementary and special education licenses must score 240 or above.

Regulatory Amendment

To allow for such a transition period, it is necessary for the Board to amend the Regulations for Educator Licensure and Preparation Program Approval (603 CMR 7.00) in order to create the new category for candidates in # 2, above. Our proposed amendment to the regulations is as follows:

Between March 7, 2009 and June 30, 2012, candidates for the following preliminary and initial licenses who earn a scaled score of at least 227-239 on the Mathematics portion of the General Curriculum test: Elementary, Teacher of Students with Moderate Disabilities, Teacher of Students with Severe Disabilities, Teacher of the Deaf and Hard-of-Hearing and Teacher of the Visually Impaired, will be deemed to have passed the Mathematics portion of the General Curriculum test. All candidates who are licensed under this provision must earn a scaled score of 240 or above on the Mathematics portion of the General Curriculum test in order to be eligible for the next stage of licensure or to renew their initial license.

We are presenting it as an emergency regulation, in order for the Board to be able to adopt the amendment at our May 19 meeting and have it take effect immediately. Given that the delay in test score release has already created considerable concern among candidates and preparing institutions, we want to act on this issue as soon as possible.

The emergency amendment, if passed, would be effective for three months, during which time we must solicit and review public comment and then bring the regulation back to the Board for a final vote. The proposed regulation would sunset after three years (June 30, 2012). As a practical matter, this means that candidates in category # 2 who are licensed in 2009 would have until the end of their fifth year of employment to meet the math standard, and any candidate who scores in this range in subsequent test administrations before June 30, 2012 would likewise have a five-year period of employment to meet the panel recommended cut score. This aligns with the statutory requirements of our licensure framework. Any candidate who applies in July 2012 or thereafter, however, would have to earn a score of 240 or above on the math subtest.

Finally, I am directing staff at the Department to continue to work with preparing institutions to help them prepare candidates to meet this new standard. We will also review whether the Department can sponsor special institutes or other technical assistance for candidates and preparing institutions as we work together to develop statewide capacity to meet this challenging new mathematics standard.

Additional Background

Individuals seeking licensure as an elementary or special education teacher at the elementary level in Massachusetts are expected to demonstrate proficiency in reading, writing and multiple subject matter areas. The Board requires candidates to pass the MTEL Communication and Literacy Skills, Foundations of Reading, and General Curriculum tests as part of the licensure requirements. The previous General Curriculum test, administered for the last time in November 2008, addressed the content areas of Language Arts, Mathematics, History, Social Science, Science and Child Development.

In December 2006, the Board supported the Commissioner's proposal to establish a separately scored mathematics subtest for aspiring teachers at the elementary level. The new approach came in response to mounting evidence, both statewide and nationally, that elementary teachers are the front line of mathematics education. They prepare all students for the secondary grades, for college, and for careers that require increasingly demanding levels of mathematical skill and thinking. Our students' math achievement is ahead of the nation but below that of their top performing international peers. It will not rise until K-12 mathematics teaching and learning improves substantially-starting with elementary school.

At its January 2007 meeting, the Board voted to solicit public comment on proposed amendments to the Regulations for Educator Licensure and Preparation Program Approval (603 CMR 7.06 (7) (b) 2. a.-c.). These amendments were designed to strengthen mathematics preparation of elementary and special education teachers at the elementary level and to clarify the breadth and depth of mathematics knowledge topics addressed on the General Curriculum test.

Overall, the comments we received on the proposed amendments were supportive of the need to strengthen the mathematics preparation of teachers at the elementary level by specifying the knowledge standards that candidates are expected to demonstrate and expanding the mathematics subject matter knowledge requirements for preparation and a separately scored General Curriculum mathematics subtest.

Based on the comments, the Department made some revisions to the proposed amendments. The Board discussed and adopted the amendments to the regulations on April 24, 2007. Following is the text of the regulation (emphasis added):

2. Mathematics.
   1. Basic principles and concepts important for teaching elementary-school mathematics in the following areas.
      1. Number and operations (the foundation of areas ii-iv)
      2. Functions and algebra
      3. Geometry and measurement
      4. Statistics and probability
   2. Candidates shall demonstrate that they possess both fundamental computation skills and comprehensive, in-depth understanding of K-8 mathematics. They must demonstrate not only that they know how to do elementary mathematics, but that they understand and can explain to students, in multiple ways, why it makes sense.
   3. The Commissioner, in consultation with the Chancellor of Higher Education, shall issue guidelines for the scope and depth of knowledge expected in mathematics, described in a. and b. above.

The regulations directed the Commissioner, in consultation with the Chancellor of Higher Education, to issue guidelines for the scope and depth of knowledge expected in elementary mathematics in order to assist Massachusetts teacher preparation programs in revising how they prepare candidates for licenses at the elementary level, including K-8 special education teachers. The completed publication, Guidelines for the Mathematical Preparation of Elementary Teachers, was distributed to the Board in August 2007. The Board endorsed the guidelines in December 2007. The document was also distributed to all Massachusetts educator preparation programs and posted on the Department's website.

The Guidelines for the Mathematical Preparation of Elementary Teachers articulate the scope and depth of mathematics knowledge—both skills and understanding—expected of elementary teachers. The guidelines require candidates to delve more deeply into the underlying structures of mathematics than previously. They require mathematics and teacher preparation program faculty to substantially rethink and redesign their courses. [http://www.doe.mass.edu/edprep/pd.html]

General Curriculum Test Development

Test development activities for the General Curriculum test, beginning early in 2007, involved the participation of public school educators and higher education faculty across the state. The two-year period from the Board's adoption of the new regulations to the first administration of the new test were intended to provide candidates, educator preparation program faculty and others with time to prepare for the new requirement. Below is a summary of the test development process.

Test Objectives Development (completed June 2007)

The General Curriculum test objectives, which are broad, meaningful statements of subject matter knowledge and skills that are important for entry-level teaching in Massachusetts public schools, define the test. The test objectives were prepared based on several sources, including the educator licensing regulations and student learning standards, with extensive input from and review by educators. The draft test objectives were reviewed by bias review and content advisory committees of Massachusetts higher education faculty and public school educators from across the state.

Content Validation Survey (Fall 2007)

A Content Validation Survey was conducted to provide empirical and confirmatory evidence in support of the validity of the General Curriculum test objectives. The Content Validation Survey, which is sometimes referred to as a job analysis survey, validates the test objectives that form the basis of the test content by ascertaining that job incumbents and educator experts (i.e., Massachusetts public school educators, arts and sciences faculty, and educator preparation faculty) consider the content of each test objective important for entry-level teaching. Participants provided ratings to indicate the importance of each objective and its set of descriptive statements, which provide examples of content that may be included on the General Curriculum test. Respondents also provided information about how well the set of objectives, as a whole, represent the subject matter knowledge required for entry-level teaching in the state. The survey results confirmed each test objective as valid.

Item Development and Review (Winter 2008)

Following the Content Validation Survey, test items, or questions, were developed based on the validated test objectives and other relevant materials from the content advisory committee. Draft test items were reviewed, revised as necessary, and approved by the committee and by the Department.

Pilot Testing (Spring/Summer 2008)

The items for the new General Curriculum test were pilot tested to gather information about their quality and technical characteristics. The pilot test provided teacher candidates and other individuals at Massachusetts institutions the opportunity to respond to test items that may be included on future test forms.

Practice Test (June 2008)

The new test objectives and a full-length General Curriculum practice test were posted on the MTEL website to further assist candidates and institutions in preparing for the content and structure of the new test. The practice test includes the test directions, answer sheets, multiple-choice test items and answer keys, open-response (written) assignments, and sample weak and strong open-response items. The answer key provided with the practice test identifies the specific objective addressed in each test question.

Test Administration (March 2009)

The new General Curriculum test was first administered on March 7, 2009. A total of 905 examinees took the multi-subject subtest only, and a total of 680 examinees took the mathematics subtest only. A total of 596 examinees took both of the subtests. As with the Communication and Literacy Skills Test, examinees have the option of taking one or both General Curriculum subtests per test date. The choice is made at the time of registration.

Qualifying Score Conference (March 2009)

Following the first administration of the test, a panel of 25 Massachusetts higher education faculty and public school educators (about one- third of whom participated on the content advisory committee) participated in qualifying score activities in which they reviewed results of the March 7 test administration and provided the Department with information to use in setting the passing standards for the test. The qualifying score activities began with an extensive orientation and training session, followed by a test simulation task during which panel members independently took the test that was administered on March 7 and then provided a rating for each test item to indicate the percentage of just acceptably qualified entry-level educators who would answer the item correctly

Following the independent item ratings, the panel engaged in additional discussion about the level of knowledge and skills required in Massachusetts of entry-level educators as well as factors that could influence examinee performance on the test. The final task for the panel was to complete a second round of item ratings, in which they had the opportunity to change their initial ratings if they chose to do so.

Following the Qualifying Score Conference, the state's contractor for the MTEL program, Evaluation Systems group of Pearson, calculated the recommended qualifying score for each General Curriculum subtest based on the panel's item ratings. The panel-recommended qualifying score was calculated as the sum of the individual item medians, and the information was submitted to the Commissioner to use in establishing the test cut scores. This process is consistent with the modified-Angoff method that has historically been used in making MTEL qualifying score recommendations.

At each stage of the General Curriculum test development process, two meetings of the content advisory committee and one of the qualifying score committee, participants were provided with copies of the Guidelines for the Mathematical Preparation of Elementary Teachers in order to ensure they understood our expectations for the level of competence we believe is required to establish a solid foundation in mathematics for all students.

The panel's test item ratings and recommended passing score are consistent with the high expectations highlighted in the Guidelines. The qualifying score derived from their item ratings is beyond the capabilities of nearly three-fourths of the current candidates for the elementary and related special education licenses. For those who have been concerned about the weak mathematical content knowledge base of many elementary and special education teachers, the low results were anticipated as a possible outcome. In light of the challenge inherent in transforming the quality of mathematics preparation within our elementary and special education teaching force, I believe it is appropriate to transition to this goal over a reasonable period of time. We are adopting this approach in our proposed amendment.

The proposed approach reflects an attempt to balance higher expectations for mathematics knowledge and skills with the understanding that fundamentally changing the depth of mathematics competence among beginning teachers is a significant and ambitious undertaking that will take some time. During this transition stage we will expect to see pass rates for the mathematics subtest that are lower than for other MTEL tests as educator preparation programs and candidates work toward meeting the new mathematics standard.

This is a challenging new standard for our state's aspiring educators and preparing institutions to meet. I am confident that the approach we are recommending upholds our commitment to high standards while allowing a reasonable transition. This initiative will result in better prepared elementary and special education teachers, and more importantly, students who are provided a strong early foundation in mathematics.

Attachments:

	Sample General Curriculum Mathematics Subtest Items
	MTEL General Curriculum, Mathematics Subtest Committee Members Motion