[Massachusetts Comprehensive Assessment System (MCAS) Logo] Guide to Interpreting the Spring 2007 MCAS Reports for Schools and Districts September 2007 Massachusetts Department of Education [Massachusetts Department of Education Star Logo] This document was prepared by the Massachusetts Department of Education. Jeffrey Nellhaus, Acting Commissioner of Education © 2007 Massachusetts Department of Education Permission is hereby granted to copy any or all parts of this document for non-commercial educational purposes. Please credit the "Massachusetts Department of Education." Massachusetts Department of Education 350 Main Street Malden, MA 02148-5023 Phone 781-338-3000 TTY: N.E.T. Relay 800-439-2370 www.doe.mass.edu [Massachusetts State Seal] Commissioner’s Foreword Dear Colleagues: The Massachusetts Comprehensive Assessment System (MCAS) tests are an important part of our state’s effort to improve teaching and learning across the Commonwealth. It is essential that all teachers, administrators, and parents/guardians understand the MCAS testing program, as all have essential roles to play in helping students to succeed. This Guide explains test results and other information contained within the spring 2007 Test Item Analysis Roster, Test Item Analysis Report Summary, School Report, and District Report. Complete results for the high school Science and Technology/Engineering tests are reported for the first time in 2007. This Guide also provides information on the standard-setting process that took place during summer 2007 for the four high school Science and Technology/Engineering tests (Biology, Chemistry, Introductory Physics, and Technology/Engineering) and the grade 3 Mathematics test. We hope that you will find the Guide useful as you examine the 2007 results for your school and/or district. If you have questions after reviewing this Guide and your reports, please contact the Student Assessment Services office at 781-338-3625. Sincerely, Jeffrey Nellhaus Acting Commissioner of Education Table of Contents I. Document Purpose...........................................................................1 II. Reporting of the 2007 MCAS Test Results...................................................2 III. General Guidelines for Interpretation and Use of MCAS Reports...........................4 Use of MCAS Results..........................................................................4 Measurement of Change from Year to Year......................................................4 Common and Matrix-Sampled Item Results......................................................4 Composite Performance Index (CPI) Points.....................................................5 Conversion of Raw Scores to Scaled Scores.....................................................6 Comparisons of Results.......................................................................6 References to LEP Students...................................................................7 State Regulations Regarding Confidentiality of Student Records.............................7 IV. History of MCAS Standard Setting..........................................................8 1998-2003....................................................................................8 2006..........................................................................................8 2007........................................................................................9 V. Understanding the Test Item Analysis Roster and the Test Item Analysis Report Summary.....................................................14 Overview......................................................................................14 Test Item Analysis Roster....................................................................16 Test Item Analysis Report Summary.......................................................23 VI. Understanding the School Report and District Report.................................27 Overview.....................................................................................27 A. Student Participation Report........................................................29 B. Distribution of Results by Composite Performance Index Points.....................32 C. Subject Area Subscores on Standard Test.............................................34 D. Performance Level Results...............................................................36 E. Three-Year Comparison of School Performance Level Results........................37 F. Performance Level and Preliminary Composite Performance Index for Subgroups of Students...............................................................39 G. Competency Determination Results (Grade 10 Only)......................................40 VII. Understanding the 2007 High School Science and Technology/Engineering Participation and Performance Reports.................................................41 Appendix A: Raw Score to Scaled Score Conversion Tables......................................43 Appendix B: Calculating Threshold Scores (Cut Points)........................................57 Appendix C: MCAS-Related References..........................................................59 I. Document Purpose The purpose of this document is to support local educators’ use of district and school reports of results from the spring 2007 administration of the Massachusetts Comprehensive Assessment System (MCAS) tests. This document describes and explains the standard setting that took place in summer 2007, and provides guidance on interpreting the following MCAS reports for the spring 2007 tests: * Test Item Analysis Roster * Test Item Analysis Report Summary * School Report * District Report In their efforts to better serve the academic needs of all students, local educators can use the important information in these 2007 MCAS reports to improve curriculum and instruction. In addition, this document can help school and district personnel communicate with parents and members of their communities about MCAS test results. II. Reporting of the 2007 MCAS Test Results The table below shows the MCAS tests and pilot tests administered in spring 2007, and how each test’s results are reported. Notes for the table appear on page 3. Table 1. MCAS Results Reported in 2007 Grade Subject Area Raw Scaled Performance Test Item Score Score Level Analysis 3 ELA Reading Comprehension^1 * [gray box] * * Mathematics^1 * [gray box] * * 4 ELA Composition and Reading Comprehension * * * * Mathematics * * * * 5 ELA Reading Comprehension * * * * Mathematics * * * * Science and Technology/Engineering * * * * History and Social Science Pilot Test^2 * [gray box] [gray box] * 6 ELA Reading Comprehension * * * * Mathematics * * * * 7 ELA Composition and Reading Comprehension * * * * Mathematics * * * * History and Social Science Pilot Test^2 * [gray box] [gray box] * 8 ELA Reading Comprehension * * * * Mathematics * * * * Science and Technology/Engineering * * * * High School High School Science and Technology/Engineering^3 Biology * * * * Chemistry Introductory Physics Technology/Engineering 10 ELA Composition and Reading Comprehension * * * * Mathematics * * * * High School U.S. History Pilot Test^2 * [gray box] [gray box] * * = data reported [gray box] = data not reported NOTES to TABLE 1: 1. Results from the MCAS grade 3 ELA Reading Comprehension and Mathematics tests are reported by performance level and raw score; no scaled scores are reported for grade 3 tests. Note that the grade 3 Reading test and the ELA Language and Literature tests for all other grade levels were renamed ELA Reading Comprehension tests in 2007. 2. A pilot administration of History and Social Science tests for grades 5 and 7 and a high school U.S. History test took place in spring 2007. Only Test Item Analysis Rosters and Test Item Analysis Summary Reports are issued for these pilot tests; these reports supply valuable information to schools and districts regarding the alignment of curricula to the History and Social Science Curriculum Framework learning standards. 3. Results for the four end-of-course high school Science and Technology/Engineering tests are fully reported for the first time this year. III. General Guidelines for Interpretation and Use of MCAS Reports USE OF MCAS RESULTS Results contained in the 2007 MCAS reports illustrate the extent to which students and schools are achieving the learning standards contained in the Massachusetts Curriculum Frameworks. The Frameworks are available on the Department Web site at www.doe.mass.edu/frameworks/current.html. Improvement of Student Performance Teachers and parents can use individual student results and school-level results to focus on areas in which students need assistance to improve performance. Alignment of Curriculum and Instruction Teachers and administrators can use school- and district-level results to evaluate the effectiveness of local curriculum and instruction in aligning with the standards defined in the Frameworks. MEASUREMENT OF CHANGE FROM YEAR TO YEAR Beginning in 2007, the Department modified the statistical technique that it uses to ensure the stability of the measurement of MCAS performance standards from year to year. From 1999 to 2006 the Department used a procedure known as the Fixed Common Item Parameters (FCIP) method to ensure that scaled scores were comparable from one year to the next. This means, for example, that a student scoring 220 on the grade 4 Mathematics test in 2005 had approximately the same performance as a student scoring 220 in 2006. In the winter of 2006, the Department’s own studies and independently published academic findings showed that adjustments to the technique were warranted to enhance the precision of the measurement of changes in performance from one year to another. After a thorough review, the Department’s Technical Advisory Committee recommended that the procedure be modified, and the Department accepted that recommendation. This procedure-referred to as FCIP2-represents a slight modification to the Fixed Common Item Parameters method. Because 2007 grade 10 results are pre-equated, this modification does not affect grade 10 results. The modified procedure will be applied to the grade 10 English Language Arts and Mathematics results and high school Science and Technology/Engineering results for the first time in 2008. Internal and external analyses reveal that the 2007 threshold scores did not shift uniformly—some shifted downward, some did not change, while others shifted upward—as a result of the adoption of the modified procedure. Therefore threshold score changes observed this fall are not solely due to the modified procedure. Internal and external analyses have affirmed that the adoption of this modified procedure has improved the measurement of change in student performance since 2006, as intended. Please note that the adoption of this method does not in any way inflate or deflate actual student performance. Instead, this method simply captures more precisely actual changes in student performance from year to year. Complete information about the adoption of the update to the Fixed Common Item Parameters method will be provided in the 2007 MCAS Technical Report to be published in Fall 2007. COMMON AND MATRIX-SAMPLED ITEM RESULTS All MCAS student-level scaled score (raw score at grade 3) and performance level results are based on common items only. The Test Item Analysis Roster and Test Item Analysis Report Summary report results for only common items. In addition to scaled score (raw score at grade 3) and performance level results based on common items, the School Report and District Report contain aggregated subscores for specific content area strands. These school- and district-level subscores include student responses to both common and matrix-sampled MCAS items. COMPOSITE PERFORMANCE INDEX (CPI) POINTS Composite Performance Index (CPI) points are awarded to schools and districts for each student tested. October 2006 enrollment data were used to determine school CPI points; spring 2007 enrollment data were used to determine district and state CPI points. CPI points are assigned for each student according to the tables below. Standard MCAS Tests CPI points for students who took the standard MCAS tests are awarded to schools and districts based on performance level categories and scaled scores (raw scores at grade 3). TABLE 2. ASSIGNMENT OF CPI POINTS FOR STANDARD MCAS TESTS GRADE 3 MCAS Performance Level Reading Mathematics CPI Points per Student Comprehension Raw Score Raw Score Above Proficient 45-48 37-40 100 Proficient 37-44 30-36 100 Needs Improvement-High 29-36 26-29 75 Needs Improvement-Low 22-28 22-25 50 Warning-High 11-21 8-21 25 Warning-Low 0-10 0-7 0 TABLE 3. ASSIGNMENT OF CPI POINTS FOR STANDARD MCAS TESTS GRADES 4-8 AND HIGH SCHOOL MCAS Performance Level Scaled Score CPI Points per Student Advanced 260-280 100 Proficient 240-258 100 Needs Improvement-High 230-238 75 Needs Improvement-Low 220-228 50 Warning/Failing*-High 210-218 25 Warning/Failing*-Low 200-208 0 *The MCAS performance level of Failing is used only for high school test results. Alternate Assessments Since no scaled scores are reported for students taking the Alternate Assessment, CPI points for these students are awarded based on Alternate Assessment performance level categories only. TABLE 4. ASSIGNMENT OF CPI POINTS FOR ALTERNATE ASSESSMENT GRADE 3 Alternate Assessment CPI Points Standard MCAS Performance Level per Student Performance Level Above Proficient (P+A) 100 Above Proficient Proficient (P_A) 100 Proficient Needs Improvement (NIA) 100 Needs Improvement Progressing (PRG) 100 Warning Emerging (EMG) 75 Warning Awareness (AWR) 50 Warning Incomplete Portfolio (INP) 25 Warning TABLE 5. ASSIGNMENT OF CPI POINTS FOR ALTERNATE ASSESSMENT GRADES 4-8 AND HIGH SCHOOL Alternate Assessment CPI Points Standard MCAS Performance Level per Student Performance Level Advanced (A_A) 100 Advanced Proficient (P_A) 100 Proficient Needs Improvement (NIA) 100 Needs Improvement Progressing (PRG) 100 Warning/Failing* Emerging (EMG) 75 Warning/Failing* Awareness (AWR) 50 Warning/Failing* Incomplete Portfolio (INP) 25 Warning/Failing* *The standard MCAS performance level of Failing is used only for high school test results. CONVERSION OF RAW SCORES TO SCALED SCORES Scaled scores are derived from raw scores (based on common items only) using a data analysis process known as scaling; the scaling process is analogous to converting temperature from degrees Fahrenheit to degrees Celsius. Appendix A contains the 2007 MCAS Raw Score to Scaled Score Conversion Tables for each test administered in spring 2007. Appendix B contains an explanation of how threshold scores for each performance level are determined. Further information about MCAS scaling methods can be found in the MCAS 2006 Technical Report, which is posted at www.mcasservicecenter.com/iEnrollmentMCAS/TechReport.htm. COMPARISONS OF RESULTS Performance levels are the most appropriate measure to use when comparing MCAS results among students, schools, and districts. Results from the 2007 MCAS tests should be evaluated in conjunction with results from MCAS tests administered from 1998 through 2006. As with any evaluation, school and district test results are most meaningful when compared with other indicators and when examined over several years for long-term trends in performance. That being said, it is important that decision-makers-teachers, administrators, and policymakers-recognize that some apparent differences over time in scaled scores may not be statistically or educationally significant. Additionally, changes in MCAS policies can make data comparisons over time slightly inaccurate. In particular, two policies enacted in 2006 make it difficult to compare results from 2006 and 2007 with results from previous years. The Department’s Profiles Web site (http://profiles.doe.mass.edu/mcas.aspx) provides a tool that adjusts data from 2003-2005, making results from these years comparable with results from 2006 and 2007. More information regarding cautions for score interpretations over time is available on page 35 ("Caution: Longitudinal Comparisons"). REFERENCES TO LEP STUDENTS Limited English Proficient (LEP) students have been designated as LEP in the Student Information Management System (SIMS) because they cannot perform ordinary classwork in English. In reports for schools and districts, the code of LEP has one of two meanings, depending on its context: 1. Results Reporting When reporting results, the code LEP refers to those limited English proficient students who participated in MCAS testing and who are not first-year LEP students. In addition, the related code of FLEP (Formerly Limited English Proficient) refers to those students who are not currently classified as LEP but who were classified as LEP at any time within the past three years. 2. Performance Level Coding When the code LEP appears as a performance level designation, it refers to a limited English proficient student in his or her first year of enrollment in U.S. schools. Students who were reported as first-year LEP students on MCAS answer booklets were not ordinarily included in results other than AYP participation rate results, provided they participated in the Massachusetts English Proficiency Assessment (MEPA). However, if SIMS enrollment records reported that such a student attended schools in Massachusetts on or before March 1, 2006, the student was counted as tested, received a scaled score, and was included in all performance level results. STATE REGULATIONS REGARDING CONFIDENTIALITY OF STUDENT RECORDS The Massachusetts Student Record Regulations (603 CMR 23.00) and the federal Family Educational Rights and Privacy Act (FERPA) restrict access to individual student results, including those provided in MCAS Test Item Analysis Rosters and Parent/Guardian Reports. A student’s temporary records contain information that does not appear in the student’s transcript. These temporary records generally cannot be examined by anyone other than the student, his or her parents or guardians, and authorized school personnel, unless the parent/guardian or student gives consent. A student’s temporary records are maintained for at least five years after he or she transfers, graduates, or withdraws from the school system. Superintendents and principals are responsible for ensuring the privacy and security of all student records. In accordance with state and federal regulations, authorized school personnel shall have access to the records of students to whom they are providing services only when such access is required for the performance of their official duties. For more information on this topic, refer to the Massachusetts Department of Education publication, Student Records: Questions, Answers, and Guidelines. IV. History of MCAS Standard Setting 1998-2003 The MCAS tests have been administered to students in Massachusetts since 1998. In 1998, English Language Arts (ELA) and Mathematics were the subjects assessed at grades 4, 8, and 10. In subsequent years, additional grades and subject areas were tested. Following the initial administration of each new test, performance standards were set. Table 6 displays the years in which standards were set for various subject area tests in grades 3 through 8 and 10 from 1998-2003 and the standard-setting method used for each test. TABLE 6. STANDARD-SETTING YEARS AND METHODS 1998-2003 Grade Subject Area Test Year Standard-Setting Method 3 Reading 2001 Bookmark 4 Mathematics 1998 Body of Work 4 ELA 1998/2001 Body of Work 5 Science and Technology/Engineering 2003 Body of Work 6 Mathematics 2001 Body of Work 7 ELA 2001 Body of Work 8 Mathematics 1998 Body of Work 8 Science and Technology/Engineering 2003 Body of Work 10 ELA 1998 Body of Work 10 Mathematics 1998 Body of Work 2006 As part of NCLB requirements, in 2006, tests for several new grades and subject areas were added to MCAS; performance standards for these tests were established using the Body of Work method. The new grades and subject areas assessed in 2006 "filled in the gaps" so that all students in grades 3 through 8 and 10 were assessed and reported in both ELA and Mathematics. Additionally, when initial standards for grade 3 Reading were set in 2001, only three performance levels were established; therefore, to be consistent with the other grades and in compliance with NCLB, a fourth performance level was added to grade 3 Reading and was also applied to grade 3 Mathematics during 2006 standard setting. The new performance level, Above Proficient, is the highest level students can achieve in grade 3; in other grades the highest performance level is Advanced. The Above Proficient cut point established in 2006 for the grade 3 Mathematics test was reset in 2007 (see page 9). Standards were set in 2006 for the following grades and subject area tests: * Grade 3 Reading * Grade 3 Mathematics * Grade 5 ELA * Grade 5 Mathematics * Grade 6 ELA * Grade 7 Mathematics * Grade 8 ELA 2007 In 2007, high school Science and Technology/Engineering was operationally assessed for the first time, through four separate end-of-course tests: * Biology * Chemistry * Introductory Physics * Technology/Engineering In summer 2007, standards were set for these four tests, and the Proficient/Above Proficient cut point was reset for the grade 3 Mathematics test. Standard setting for 2007 is described below in the following sections: * Performance Level Descriptors - Grade 3 Mathematics - High School Science and Technology/Engineering * Standard-Setting Activities - Grade 3 Mathematics - High School Science and Technology/Engineering * Application of the Body of Work (BOW) Standard-Setting Method - Content of Student Papers - Selection of Student Papers * The Summer 2007 Standard-Setting Meetings - Grade 3 Mathematics - High School Science and Technology/Engineering - Analysis During Standard Setting Additional information regarding standard setting is available in the Department publication 2006 MCAS Technical Report, which is posted at www.mcasservicecenter.com/iEnrollmentMCAS/TechReport.htm. Performance Level Descriptors Grade 3 Mathematics Grade 3 Mathematics standards validation and standard-setting decisions were based on the same performance level descriptors used during the 2006 standard-setting meetings. High School Science and Technology/Engineering For the high school tests in Biology, Chemistry, Introductory Physics, and Technology/Engineering, content-specific performance level descriptors were developed by science content staff at the Massachusetts Department of Education. In developing the first draft of these descriptors, Department staff referenced the depth of the existing content-specific descriptors for grades 5 and 8 Science and Technology/Engineering, and used the general MCAS Science and Technology/Engineering performance level descriptors as anchors. The 2007 common items for the four tests were mapped to the content-specific performance level descriptors as an initial validation of the clarity and accuracy of the descriptors. The descriptors were then further reviewed and validated by a group of Science and Technology/Engineering educators familiar with both the content standards and the MCAS high school Science and Technology/Engineering items. After review and validation, the content-specific performance level descriptors were used during the standard-setting process. Standard-Setting Activities Grade 3 Mathematics MCAS cut points for the grade 3 Mathematics test were set in the summer of 2006; however, since the 2006 test was designed to reflect only three performance levels (Warning, Needs Improvement, and Proficient), the 2006 standard-setting panelists were unable to establish a meaningful third cut to distinguish between Proficient and Above Proficient. To address this issue, the 2007 test was constructed to include more cognitively demanding items that reflected the measurement characteristics necessary to identify those students performing at the Above Proficient performance level. Since the cut points previously established for the two lower cuts (between Warning/Needs Improvement and Needs Improvement/Proficient) were not problematic and were judged to be reasonable, they were presented to the panelists as starting cuts. The panelists’ task in 2007 was to validate the two currently existing cut points and establish the top cut point. A modification of the Body of Work (BOW) method was used to implement this combined standards-validation/standard-setting process, which is discussed in more detail below and on page 11. High School Science and Technology/Engineering Unlike grade 3 Mathematics, for the high school Science and Technology/Engineering tests (Biology, Chemistry, Introductory Physics, and Technology/Engineering), starting cut points were not calculated for any set of performance levels (Failing/Needs Improvement, Needs Improvement/Proficient, Proficient/Advanced). As with grade 3 Mathematics, a modification of the BOW method was used. A discussion of this process follows. Application of the Body of Work (BOW) Standard-Setting Method The BOW methodology was used to set standards in Massachusetts in both 2006 and 2007. Specifically, each standard-setting panel reviewed thoroughly and discussed sets of student work ("papers"). Ultimately, the panelists recommended cut scores based on their evaluations of these papers. Content of Student Papers The papers for each student included a "multiple-choice display" that provided the following information for each multiple-choice item on the test: * the stem of the item * the correct response * the difficulty level of the item * an indicator of whether the student’s response was correct or incorrect Each student’s papers also included his or her responses to open-response and (for grade 3 Mathematics) short-answer items. The scores awarded to open-response and short-answer items were not indicated on the papers, so panelists focused on the quality of students’ responses rather than the scores that were awarded. Selection of Student Papers Because cut scores are based on panelists’ classifications of student papers, the selection of those papers was a crucial part of the preparation for standard setting. Student papers were selected in advance of standard setting, based on the following criteria: * the relationship between open-response scores and multiple-choice/short-answer scores: Student papers that had low open-response scores and high multiple-choice or short-answer scores, or vice versa, were not selected. * the consistency among open-response scores: Selection was based on a cohesive representation of student work at a raw score point; student papers that received, for example, a score of 4 on one open-response item and scores of 1 on other open-response items were not selected. * exclusion of eleventh- and twelfth-grade student responses: The selection of the initial pool of papers explicitly excluded any student identified as being in eleventh or twelfth grade. An index was assigned for each student based on these criteria. Student papers were ordered first by total raw score, and then by the selection index. The goal for selection of responses was to include papers at all score points, from chance level (i.e., the score a student would be expected to receive by simply guessing) to the highest possible score. To achieve this goal, three papers were identified for each score point and submitted for review by the panelists. Grade 3 Mathematics For grade 3 Mathematics, because panelists were simply validating the two lower cut points, fewer papers were required for the bottom two performance levels. Therefore, three papers with total raw scores at approximately the middle of the score range were selected for each of the two lower performance levels. This procedure was consistent with the one used successfully in 2006 to validate the two lower cut points for grade 3 Reading. Grade 3 Mathematics responses at the Above Proficient category were selected to be evenly distributed across the score range from the lowest Above Proficient score to the highest possible score in the Above Proficient category. High School Science and Technology/Engineering For high school Science and Technology/Engineering, selected papers were evenly distributed across the range of raw scores, from the chance level to the highest possible score. Calibration Papers Part of the selection process included the identification of a set of calibration papers for each standard-setting subject area, to be used for panelist training activities. Potential calibration papers were selected using the criteria listed in the bulleted section above. Final calibration papers were selected by identifying papers that exemplified each of the performance level descriptors; as an added precaution, two papers were identified that aligned with the Proficient performance level. The Summer 2007 Standard-Setting Meetings The summer 2007 standard-setting meetings started with an orientation session. The orientation provided panelists with an understanding of the purpose of the meeting and the procedures to be followed in setting performance levels. After the orientation, panelists dispersed into rooms according to subject area (grade 3 Mathematics, Biology, Chemistry, Introductory Physics, and Technology/ Engineering). Grade 3 Mathematics The grade 3 Mathematics standards-validation/standard-setting meeting lasted two days. Tasks were performed in the order listed below. Day 1 1. Mathematics-Specific Orientation A Mathematics-specific orientation followed the general orientation. It included a review of the 2006 standard setting and an introduction to the combination standards-validation/standard-setting task. 2. Review of Assessment The panelists took the grade 3 Mathematics test, then reviewed their responses to open-response and short-answer items relative to the scoring guides. The MC item map was then discussed, and participants had the opportunity to individually review each MC item and to document the knowledge, skills, and abilities needed to answer it. The MC items were then discussed by the panelist group, focusing on the skills and knowledge needed to correctly answer each item. 3. Review of Performance Level Descriptors Panelists received training on the performance level descriptors and became familiar with the skills and knowledge required for a student to achieve each performance level. 4. Calibration Exercise The panelists learned to apply their understanding of the performance level descriptors to five bodies of work (the calibration set). Grade 3 Mathematics calibration papers had been previously selected for each performance category: Warning (one paper), Needs Improvement (one paper), Proficient (two papers), and Above Proficient (one paper). For the Warning and Needs Improvement categories, calibration papers were selected based on the existing starting cut points. Since there was no starting cut for Proficient/Above Proficient, the calibration papers for those levels were selected by Department and contractor staff. Panelists reviewed the calibration set and discussed the characteristics of each paper as it related to the appropriate performance level descriptors. The meeting facilitator pointed out characteristics of each paper that were instrumental in its classification. Day 2 5. Round 1 Ratings Panelists reviewed all bodies of work, noting the performance level to which each paper was initially assigned. For the two lower performance levels, panelists individually agreed with, or reassigned, each paper. For the two upper performance levels, panelists provided their initial individual classifications. Panelists then completed their first-round individual ratings. After analysis (see below), the panelists were given classifications of papers based on ratings from the entire panel. 6. Round 2 Ratings The panelists discussed the appropriateness of the first-round classifications and their individual ratings. On the basis of those discussions, panelists revised their categorizations and submitted their final categorizations of papers. High School Science and Technology/Engineering A separate three-day standard-setting meeting was held for each high school Science and Technology/Engineering test (Biology, Chemistry, Introductory Physics, and Technology/Engineering). Following a general orientation, tasks were completed in the following order. Day 1 1. Review of Assessment The panelists took the test for which they were to set standards, then reviewed their responses to open-response items relative to the scoring guides. The MC item map was then discussed, and participants had the opportunity to individually review each MC item and to document the knowledge, skills, and abilities needed to answer it. The MC items were then discussed by the panelist group, focusing on the skills and knowledge needed to correctly answer each item. 2. Review of Performance Level Descriptors Panelists received training on the performance level descriptors and became familiar with the skills and knowledge required for a student to achieve each performance level. 3. Calibration Exercise Panelists learned to apply their understanding of the performance level descriptors to specific bodies of student work (the calibration set). The characteristics of each calibration paper were discussed, focusing on how the responses corresponded to the performance level descriptors. This was followed by a discussion of the calibration sets. Day 2 4. Round 1 Ratings Panelists reviewed all bodies of work, then completed their first-round individual ratings. After analysis (see "Analysis During Standard Setting" below), the panelists were given initial classifications of papers based on ratings from the entire panel. 5. Round 2 Ratings Panelists discussed the appropriateness of the initial classifications of papers and their individual ratings. On the basis of those discussions, panelists revised their categorizations of papers where needed. Day 3 6. Round 3 Ratings Panelists were given second-round classifications of papers based on ratings from the entire panel. They again discussed the appropriateness of the classifications and their individual ratings. On the basis of those discussions, panelists revised their categorizations and submitted their final categorizations of papers. Analysis During Standard Setting After each round of ratings, individual cut points were calculated for each panelist, using logistic regression. The threshold for each cut point was modeled as the score on the raw score scale at which students had an equal probability (p = 0.5) of being classified as either above or below the cut. V. Understanding the Test Item Analysis Roster and the Test Item Analysis Report Summary OVERVIEW The Test Item Analysis Roster and Test Item Analysis Report Summary provide results from all MCAS tests administered in spring 2007, including pilot tests. The information provided in these reports can be used to * identify strengths and weaknesses in subject area knowledge and skills exhibited by students * evaluate and plan curricula in the classroom, school, and district * summarize and discuss results with parents and the public (see "State Regulations Regarding Confidentiality of Student Records" on page 7) The Test Item Analysis Roster Test Item Analysis Rosters show student-level results for individual MCAS common items. Each school receives a separate Test Item Analysis Roster for each subject area and grade tested in the school. For each test, the Roster shows how each student * answered each common multiple-choice question * performed on each common open-response question * performed on each short-answer question, where applicable * performed on the English Language Arts Composition (writing prompt), where applicable The Roster is designed to be used in conjunction with the publication Release of Spring 2007 MCAS Test Items, which can be found on the Department Web site at www.doe.mass.edu/mcas/testitems.html. The Release of Spring 2007 MCAS Test Items shows all the questions on which Test Item Analysis Rosters are based, providing educators with a detailed picture of each student’s MCAS performance. This picture should be used in combination with the student’s classroom performance to develop an overview of his or her academic achievement. The Test Item Analysis Report Summary The Test Item Analysis Report Summary shows overall performance at the school or district level, and at the state level, for individual common items. Each school and district receives a separate Test Item Analysis Report Summary for each subject area and grade tested in the school or district. Rosters and Summaries for Specific Tests Grades 4, 7, and 10 English Language Arts (ELA) Rosters and Summaries differ slightly from these reports for other tests, since they include results for the ELA Composition, as described on the following pages. High school Science and Technology/Engineering results are not included in the grade 10 ELA and Mathematics reports. The Biology, Chemistry, Introductory Physics, and Technology/Engineering tests were administered to students in grades 9 and 10. Each test is reported separately, with most results for both grades combined in reported data. The exception is the Science Participation Summary Report, which reports data separately for each grade and subject. ELA and Mathematics retest results are reported only at the student level (in Test Item Analysis Reports); retest results are not aggregated at school or district levels. Test Item Analysis Roster [Sample report of Test Item Analysis Roster] TEST ITEM ANALYSIS ROSTER Each Test Item Analysis Roster has three main sections: 1. Item Information 2. Individual Student Results 3. Legend The number of pages in the Roster depends on the total number of students in the tested grade. 1. Item Information [Screen shot sample of Item Information depicting Item number, Item type, and Reporting category] Section 1 of the Roster (as indicated on page 16) contains four types of information for each common item, listed vertically: * item number * item type * correct multiple-choice (MC) answer, if applicable * reporting category Section 1 also shows the total raw score points possible for the reported test. Item Number Item Number corresponds to the number of the question as it appears in the Release of Spring 2007 MCAS Test Items publication. For grades 4, 7, and 10 ELA only, this information is blank for the first two columns, which provide ELA Composition data. Item Type Item Type indicates whether the item was * multiple-choice (MC) * open-response (OR) * (for Mathematics only) short-answer (SA) * (for grades 4, 7, and 10 ELA only) ELA Composition writing prompt ("WP" appears in the first two columns) See page 18 ("Individual Student Results") for further information about interpreting Item Type data. Correct MC Answer Correct MC Answer provides the letter option (A, B, C, or D) for the correct answer for multiple-choice questions. Reporting Category Each Reporting Category is a strand of the Curriculum Framework for the tested subject area (e.g., Reading and Literature is a strand of the English Language Arts Framework and an MCAS reporting category). For grades 4, 7, and 10 ELA, only the first two columns relate to performance on the composition. Reporting category codes are explained in the Legend at the end of the Test Item Analysis Roster (see page 26). Total Possible Points The number shown in the fourth-from-last column for Total Poss. Points is the sum of the maximum raw score points for all common questions on the test: 1 point for each MC question + 4 points for each OR question (2 points for grade 3 Mathematics OR questions) + 1 point for each SA question (Mathematics only) + 20 points for each WP (grades 4, 7, and 10 ELA only: includes points for both WP columns) 2. Individual Student Results [Screen shot sample of Body Roster depicting Individual Student Results] Individual student results for each common test item are provided in section 2 (as indicated on page 16) of the Roster. Results are displayed in a separate row for each student enrolled in the school or district on the first day of testing. Results for each question are provided using various symbols, letters, and numbers, based on Item Type, as further explained below. For students who took the Alternate Assessment, all items are automatically left blank. For all other students, regardless of question type, a blank space indicates the student did not answer the question and a raw score of 0 was assigned for that question. Each student’s summary scores are provided in the final four columns of his or her row. Multiple-Choice Each multiple-choice question received a score of 0 for an incorrect answer or 1 for a correct answer. The following symbols and letters are used to indicate MC question results: * a + sign indicates the student answered the question correctly * a letter (A, B, C, or D) indicates which incorrect option the student selected * an asterisk (*) indicates the student selected more than one option and received a score of 0 on the question Open-Response Each open-response question received a score of 0-4 (except for grade 3 Mathematics open-response questions, each of which received a score of 0-2). Short-Answer Short-answer questions appeared on Mathematics tests only. Each short-answer question received a score of 0 for an incorrect answer or 1 for a correct answer. Writing Prompt Only grades 4, 7, and 10 English Language Arts Test Item Analysis Rosters show scores for ELA Compositions in the first two writing prompt (WP) columns. The first column shows the student’s score for topic development (e.g., writing development, organization, use of detail, variety in sentence structure and language). The second column shows the student’s score for standard English writing conventions (e.g., grammar, punctuation, mechanics of writing). Each student’s composition was scored independently by two scorers. Each scorer gave the composition two separate scores: * a score of 1-6 for topic development (CT) * a score of 1-4 for the student’s use of standard English writing conventions (CC) The student’s score as shown in each of these two columns (CT and CC) represents the sum of the points awarded by both scorers in each area (up to 12 points for CT; up to 8 points for CC). The Legend on the last page of the Roster displays the characteristics of student writing for each score for topic development and standard English writing conventions. Any student who wrote a totally illegible composition, wrote an off-topic response, or wrote in a language other than English was given a notation of NS (not scorable) and received a score of 0. Total Raw Score The student’s total raw score was calculated by adding one point for each MC question to the points awarded for OR and (if applicable) SA questions, and (if applicable) to writing prompt CT and CC points. Scaled Score The student’s total raw score is translated to a scaled score, reported on a scale ranging from 200-280. No scaled scores are reported for grade 3 students; this column is left blank on grade 3 Rosters. Composite Performance Index (CPI) Points (Comp. Perf. Ind. Pts.) The number in this column for each row indicates the number of CPI points awarded to the school or district based on that student’s results. Chapter III of this Guide provides further information about the award of CPI points. Performance (Perf.) Level The student’s performance level is reported as a code; performance level codes are listed and defined in the Legend of the Roster, and are also explained on the next page. Standard Performance Levels Standard Performance Level Codes Grade 3 Grades 4-8 Grade 10 P+ = Above Proficient A = Advanced A = Advanced P = Proficient P = Proficient P = Proficient NI = Needs Improvement NI = Needs Improvement NI = Needs Improvement W = Warning W = Warning F = Failing Alternate Assessment Performance Levels Seven performance levels are reported for students who participated in the MCAS Alternate Assessment, as shown below. The top three levels are equivalent to the standard MCAS performance levels with the same names. The next three (PRG, EMG, and AWR) fall within the standard MCAS Warning (Failing for grade 10) performance level. The final alternate assessment performance level, INP, corresponds to the standard MCAS "not tested" code (INC, listed under 'Exceptions" on page 21). Alternate Assessment Codes A_A (P+A for grade 3) = Advanced (Above Proficient for grade 3) P_A = Proficient NIA = Needs Improvement PRG = Progressing EMG = Emerging AWR = Awareness INP = Incomplete Portfolio Certificate of Mastery Performance Levels For students who were enrolled in grade 11 or 12 and who participated in the grade 10 ELA or Mathematics test to attempt to qualify for a scholarship, the following Certificate of Mastery Codes appear in the Performance Level column. Certificate of Mastery results are excluded from school and district summaries. Certificate of Mastery Codes A_M = Advanced P_M = Proficient NIM = Needs Improvement F_M = Failing Exceptions There are limited circumstances in which scaled score and performance level results are not provided for a student; instead, in the Performance Level column, one of the following three-letter codes appears. These codes are also listed and defined on the Legend on the last page of the Roster. For further information regarding these codes, see pages 30-31 of this Guide. Not Tested Codes LEP = First-year limited English proficient TRN = Transferred ABS = Absent INC = Incomplete or invalid test results MED = Medically documented absence VAB = Voided answer booklet (appears in Preliminary Roster reports only) DUP = Multiple answer booklets submitted for this student (appears in Preliminary Roster reports only) 3. Legend The Legend on the last page of each Test Item Analysis Roster provides a quick reference for interpreting the codes that appear throughout the Roster. The graphic on the following page shows the Legend from a grade 8 Mathematics Test Item Analysis Roster. For grades 4, 7, and 10 English Language Arts (ELA) Test Item Analysis Rosters only, the bottom section of the Legend also shows characteristics of student writing corresponding to each ELA Composition topic development (CT) and standard English writing conventions (CC) score. [Test item analysis roster: Legend Mathematics] [Test Item Analysis Report Summary] TEST ITEM ANALYSIS REPORT SUMMARY The Test Item Analysis Report Summary for each test (by grade and subject area) shows student results, aggregated at the school and/or district level(s), and at the state level, for each common test item. Each row of the Summary provides data for one common item. Each Summary has four main sections, as indicated on pages 24-26: 1. Item Information 2. Average Item Score 3. Percent of Total Student Responses 4. Legend 1. Item Information [Screen shot of item information] The first three columns of the Summary show the Item Number, Item Type, and Reporting Category of the common item. This information correlates directly to the information provided on the Test Item Analysis Roster for the same grade and subject area (see pages 17 and 18 of this Guide for more details about these three categories). The fourth column of section 1 lists the Framework learning standard tested by that item. 2. Average Item Score [Screen shot of average item score] The Average Item Score section of each Summary shows aggregated results for each item. School Summaries include aggregated school, district, and state results for each item; district Summaries include aggregated district and state results for each item. These statistics allow educators to compare the performance of students in a school to students in the district and across Massachusetts, and to identify strengths and weaknesses in local curriculum and instruction. For multiple-choice and (on Mathematics Summaries only) short-answer questions, which are both scored 0 or 1, the aggregate score shown is equivalent to the percentage of students who answered the question correctly. For open-response questions (scored 0-4, or 0-2 in grade 3 Mathematics) and writing prompts (scored 2-20), the average student score for the question is shown. The total at the bottom of each column is the sum of the average individual item scores for the entire test (excluding the ELA writing prompt for grades 4, 7, and 10 ELA Summaries). 3. Percent of Total Student Responses [Screen shot of Percent of Total Student Responses] The six columns in section 3 (as indicated on page 23) show correct answers for multiple-choice questions (in the far right column), as well as two kinds of aggregated results: * For each multiple-choice question, the columns show the percentage of students who chose each answer option (blank, A, B, C, or D). By looking at items on which students performed well, educators can identify areas of strength in their curriculum and instruction. By focusing on items on which students made similar mistakes, educators can identify possible misconceptions held by students in the school, and can plan to address the issues by strengthening curriculum and instruction. * For open-response and (on Mathematics Summaries only) short-answer questions, the columns show the percentage of students at each score point (blank/0, 1, 2, 3, or 4 for open-response questions; blank/0 or 1 for short-answer questions). These data can be used in conjunction with item rubrics and sample student work (available on the Department Web site at www.doe.mass.edu/mcas/student) to inform changes in curriculum and improve instruction. 4. Legend [School test item analysis roster: Legend Mathematics] The Legend on the last page of each Test Item Analysis Report Summary provides a quick reference for interpreting the codes that appear throughout the Summary. VI. Understanding the School Report and District Report OVERVIEW Each School Report and District Report provides aggregated MCAS results for the district and state; each School Report also provides aggregated results for that school. A separate Report was produced for each grade level (3-8 and 10) tested in spring 2007 and for high school Science and Technology/Engineering.^1 The guidelines given in this chapter may be used to interpret both the School Report and the District Report, which are nearly identical in format. All examples shown throughout this chapter refer to the School Report template. Organization of the School Report The first few pages of each School Report provide general information and definitions to help interpret the report, including a table of contents, performance level definitions, score ranges, student status definitions that relate to subgroup statistics (see section F below), and explanations regarding students who were not tested. Also provided are charts that define how Composite Performance Index (CPI) points are awarded for each test included in the Report. These introductory pages are followed by the reports of spring 2007 MCAS test results listed below: A. Student Participation Report B. Distribution of Results by Composite Performance Index Points C. Subject Area Subscores on Standard Test D. Performance Level Results E. Three-Year Comparison of School Performance Level Results (only 2006-2007 results are available for grades 3, 5, and 7 Mathematics and grades 5, 6, and 8 English Language Arts) F. Performance Level and Preliminary Composite Performance Index for Subgroups of Students G. Competency Determination Results (grade 10 only) Each of these reports is described in further detail on the following pages. ^1 Results for grades 5 and 7 History and Social Science and high school U.S. History pilot tests are not included in School Reports or District Reports in 2007; they are included only in Test Item Analysis Rosters and Test Item Analysis Summary Reports. Further, ELA and Mathematics Retest results are not aggregated at school or district levels and do not appear in School Reports or District Reports; retest results are reported only at the student level (in Test Item Analysis Roster Reports). Identification [Screen shot of identification] A box in the upper right corner of each page of the School Report shows the school name, its eight-digit identification number, the district name, and the grade level for which the report has been generated. Minimum Number of Students Needed to Generate Reports In order to ensure confidentiality of individual student results and discourage generalizations about school performance based on very small populations, the Department requires that a minimum of ten students per reporting area must be tested to report school-level results in any of the following areas: * overall school performance level and scaled score results * school results disaggregated by subgroup * subject area subscore results^2 Consequently, schools with a very small number of students enrolled in a tested grade may not show results in some sections of their School Report. If a school tested fewer than ten students in a particular grade, no School Report is generated for that grade; however, results for these students are included in district- and state-level results reported in District Reports and the State Report. ^2 Multiple forms are created for each MCAS test. Matrix-sampled items are distributed across all forms. A school must administer all forms of the test for accurate computation of subject area subscores. If a school did not administer all forms of a test, the subject area subscore pages are left blank in the school’s Report. A. Student Participation Report [Screen shot of student participation report] The pages titled Student Participation Report show-in separate school, district, and state tables-how many enrolled students were included or not included in performance level results. Each table contains eight rows that report data based on student subgroup. The report has five basic sections: 1. Student Subgroups 2. Number of Students Enrolled 3. Adequate Yearly Progress Participation Rate (AYP Part. Rate) 4. Included in Performance Level Results 5. Not Included in Performance Level Results Each section is described in more detail below. 1. Student Subgroups Section 1 lists the subgroups into which data for the school’s total student population are subdivided. The total population is divided into three Student Status subgroups; it is separately divided into seven Racial/Ethnic subgroups. The three Student Status subgroups are defined as follows: * Students with Disabilities (Sw/D). Students with Disabilities are defined here as students who have an Individualized Education Program (IEP), as reported in SIMS. * Limited or Formerly Limited English Proficient (LEP/FLEP). The LEP/FLEP subgroup combines students who were LEP at the time of testing with those who have been identified as LEP in any of the past three school years. * Low-Income. A student in this subgroup meets ANY ONE of the following SIMS definitions of low- income: - The student’s family is eligible for free or reduced price lunch. - The student’s family receives Transitional Aid to Needy Families benefits. - The student’s family is eligible for food stamps. 2. Number of Students Enrolled Section 2 shows, for each student subgroup, the Number of Students Enrolled in the tested grade on the first day of testing. The total number of students enrolled includes both those Included in Performance Level Results (section 4) and those Not Included in Performance Level Results (section 5). 3. Adequate Yearly Progress Participation Rate (AYP Part. Rate) Section 3 shows the AYP Participation Rate for each subgroup. The number that appears on the Total row is the number of students tested in spring 2007 (on either the standard MCAS or on the MCAS Alternate Assessment) divided by the number of students enrolled on the dates of test administration. First-year LEP students who participated in the MEPA are included in the calculation, as participants (for the ELA test only). All students who have an ABS or MED test status are included in the calculations, as non-participants. Students with an INC test status are not included in the calculations. At the school level, outplaced students are not included either, unless the school was the only one in its district. 4. Included in Performance Level Results Section 4 divides subgroup results into students who were tested on the standard MCAS tests (Tested Standard) and those who participated in the Alternate Assessment (Tested Alternate Assessment). The numbers of Students with Disabilities (Sw/D) for Tested Standard include students who participated in the standard test with or without accommodations. At grade 10 only, the numbers for Limited English Proficient or Formerly Limited English Proficient (LEP/FLEP) students include those students who took the Spanish-language version of the grade 10 Mathematics test as well as those who took the English-language version of that test. (A Spanish-language version was not available for any other MCAS test.) Students who were reported as first-year LEP students on MCAS answer booklets were not ordinarily included in results other than AYP participation rate results (see section 5 below). Exceptions occurred when SIMS enrollment records indicated students had attended schools in Massachusetts before March 1, 2006. In this case, the students were counted as tested; they received scaled scores and were included in School Report performance level results. 5. Not Included in Performance Level Results Section 5 shows the number of students in each subgroup who did not participate in the spring 2007 MCAS tests. The six School Report categories for students who were Not Included in Performance Level Results are defined below. * First-Year LEP (LEP). LEP students in their first year of enrollment in U.S. schools receive a performance level of LEP. First-year LEP results are not included in scaled score, performance level, or CPI results, but are included in AYP participation rate results, provided that the students participated as required in MEPA. (See section 4 above for the exception to this policy.) * Transferred (TRN). Any student who transferred into or out of a school during the testing window and who did not complete all of the test sessions at that school received a performance level of TRN. These results are not included in the determination of scaled score, performance level, CPI, or AYP participation rate results. * Absent (ABS). Any student who was absent during the testing period, and who did not receive a medically documented absence as described below, received a performance level of ABS. These results are not included in the determination of scaled score, performance level, or CPI results, but are counted against the AYP participation rate. * Incomplete Test Results (INC). Any student whose test results were incomplete or invalid received a performance level of INC. These results are not included in the determination of scaled score, performance level, CPI, or AYP participation rate results. If, for instance, a grade 3 student takes the Mathematics test but no scorable ELA answer booklet was returned, the student’s ELA results will appear as INC. However, that student’s Mathematics results will be reported as usual and included in school- and district-level results. * Medically Documented Absence (MED). Any student who obtained a written statement from a licensed medical professional, stating both the medical reason that prevented him or her from taking the test and the date(s) for which he or she was absent due to the medical condition, received a performance level of MED. These results are not included in the determination of scaled score, performance level, or CPI results, but are counted against the AYP participation rate. * Enrolled after October 1st and tested. Any student who enrolled in a school after October 1, 2006, was required to participate in MCAS testing but was not included in most school-level MCAS or accountability reports because the student was not enrolled for a full academic year. However, these students are included in school AYP participation rates and in district-level summaries. Further, results for these students are also included in school-level results for districts with only one school, such as charter and regional vocational districts. B. Distribution of Results by Composite Performance Index Points [Screen shot of Distribution of Results by Composite Performance Index Points] This page of the Report is designed to help schools see how their students are clustered within each performance level and Composite Performance Index (CPI) point interval for each subject area. It can also be used to illustrate how any given school or subgroup with similar aggregate CPI points may have very different percentages of students at each performance level. This results from different distributions of individual scores within CPI point intervals. 1. Performance Levels Section 1 provides a separate row for each applicable standard and MCAS Alternate Assessment performance level at the grade level of the Report. It also provides a row showing Total CPI Points and one showing Total Students. 2. CPI Points The column in section 2 illustrates how CPI points correspond to performance levels and, for standard tests, to scaled scores (except at grade 3, for which raw scores are reported instead). The number of CPI points shown is awarded for each student whose results fall within one of the performance level categories listed within that CPI point interval. 3. Student Performance by Subject Area Section 3 shows student numbers and percents in each performance level for the school, district, and state. Each subject area’s results are shown in a separate table; tables are presented side by side on the same page. 4. Totals The total number of CPI points awarded, based on the numbers of students in the column above and multiplied by the corresponding CPI point intervals, is shown for the school, district, and state on the top shaded row of section 4. The total number of students counted toward the school, district, or state CPI is shown on the bottom shaded row of section 4. C. Subject Area Subscores on Standard Test [Screen shot of Subject Area Subscores on Standard Test] This section of the School Report provides information on student performance in each reporting category for the tested subjects.^3 Each subject area’s results are presented in a separate table. Each reporting category is a strand of the Curriculum Framework for the tested subject area (e.g., Reading and Literature is a content strand of the English Language Arts Framework and an MCAS reporting category). Subject area subscores are the only reported MCAS scores based on both common and matrix-sampled items. Multiple forms are created for each MCAS test; unlike common items, matrix-sampled items differ across test forms and are used to report performance on a broader range of specific curriculum goals than is possible using common items only. A school must administer all forms of a test in a grade for accurate computation of subject area subscores. If a school did not administer all forms of a test in a grade, the subject area subscore pages are left blank in the school’s Report for that grade. This situation often arises in schools with small student populations. For each reporting category and item type used within each tested subject area, the subscore pages show the following: * the number of possible points for common items, the number of possible points for matrix items, and the total number of possible points * the average total number of points attained and the percentage of total possible number of points attained at the school, district, and state levels^4 * the average total number of points attained and the percentage of total possible number of points attained for students at the state level in each of the standard MCAS performance levels for the tested grade The following cautions apply only to raw scores, not to performance level results or scaled scores, both of which are statistically equated from year to year. Refer to the 2006 MCAS Technical Report for more information. ^3 Alternate assessment results are not included in subject area subscores. ^4 Because schools may have had different numbers of students taking each test form, average points were determined using a two-step process. First, the average raw score was determined for each unique item. Second, the sum of these average raw scores was calculated. CAUTION: Raw Scores versus Scaled Scores. The numbers for possible subscore points are based on raw score points, not on the scaled scores used elsewhere throughout the Report (except at grade 3, where only raw scores are reported). Since subject area subscores combine results for both common and matrix-sampled items, total points for a subject area may differ on these subscore pages from total raw points reported elsewhere in this Report or in other MCAS reports, which report raw scores for common items only. CAUTION: Longitudinal Comparisons. Comparisons of performance on subject area subscores across years must be made with caution, because the number of items contributing to each subscore is relatively small, and because the difficulty of items may vary somewhat from year to year. The pool of matrix-sampled items varies from year to year as items move from the matrix-sampled sections to the common sections of the tests to replace publicly released common items. D. Performance Level Results [Performance Level Results] Performance level results for the school, district, and state indicate, for each subject area tested, the percent of students performing at each of the four standard MCAS performance levels for the tested grade level.^5 These percentages are based on the performance of all students who took either the standard MCAS tests or the MCAS Alternate Assessment.^6 Performance level results do not include the results of students in the following categories, for whom performance level results were not generated: * Absent (ABS) * Medically Documented Absence (MED) * First-Year LEP (LEP) * Incomplete (INC) * Transferred (TRN) See pages 30-31 of this Guide for explanations of these five categories. At the school level, the charts include only students who were enrolled on October 1, 2006, unless the school is the only school in its district. ^5 District-level and state-level results include performance level results for students in out-of-district placements. ^6 Alternate assessment performance level results are matched with standard MCAS performance levels in this section. See page 6. E. Three-Year Comparison of School Performance Level Results [Three-Year Comparison of School Performance Level Results] This section of the 2007 School Report shows MCAS results from the spring 2005, 2006, and 2007 administrations. A separate page is generated for each subject area tested. These data, along with results from other past MCAS tests, can be used to make longitudinal comparisons of results from tests that were operational in all three years (see page 35 of this Guide for cautions concerning comparisons). It is important to note that 2005 results have been adjusted to accommodate policy changes made in 2006. Only 2006 and 2007 results are shown for the following tests, which were new in 2006: * Grade 3 Mathematics * Grade 5 English Language Arts and Mathematics * Grade 6 English Language Arts * Grade 7 Mathematics * Grade 8 English Language Arts In addition, only 2007 results are shown for high school Science and Technology/Engineering. For grade 3 tests, no scaled scores are generated. Consequently, the table shown on the right side of the sample on the previous page does not appear in the grade 3 reports. Additionally, since the grade 3 performance level of Above Proficient was used for the first time in 2006, the Proficient performance level data in this section of the grade 3 English Language Arts School Report includes both Proficient and Above Proficient grade 3 student results, to facilitate comparison of data across years. Performance Level Results include students who took either the standard MCAS test or MCAS-Alt. Distribution of Results on Standard Test by Scaled Score Interval includes only students who took the standard MCAS test. F. Performance Level and Preliminary Composite Performance Index for Subgroups of Students [Screen shot of Performance Level and Preliminary Composite Performance Index for Subgroups of Students] The information on these pages of the School Report is used for two primary purposes: * to determine how effectively school programs are serving various student populations * to promote fair comparisons among schools and among districts by allowing them to compare the performances of similar student groups A separate page is generated for each subject tested. Each page shows performance level results at school, district, and state levels for students in ten different AYP Subgroups and in seven Other Subgroups, as listed in the graphic on this page. In addition, total number of students is reported, along with average total percent for each performance level, and average number of CPI points awarded, for school, district, and state. Note that, on these pages of the Report, all references to LEP students apply to current LEP students who participated in testing and who generated results. G. Competency Determination Results (Grade 10 Only) [Screen shot of Competency Determination Results (Grade 10 Only)] Students must pass both the grade 10 English Language Arts test and the grade 10 Mathematics test to earn a Competency Determination. This section of the grade 10 Report shows the number and percent of grade 10 students in the school, district, and state who have * passed both tests * passed only one test * not yet passed either test School-level results are presented in both bar graph and numeric table formats. Data for "Not Yet Passed English Language Arts or Mathematics" includes results for students who were in grade 10 in spring 2007 but did not participate in grade 10 MCAS testing. Competency Determination results do not include results for students in the classes of 2007 and 2008 who participated in retests. Reports of results for these students are issued separately twice a year. VII. Understanding the 2007 High School Science and Technology/Engineering Participation and Performance Reports In spring 2007, schools were required to offer a high school MCAS Science and Technology/Engineering (STE) test to grades 9 and 10 students who were enrolled during the 2006-2007 school year in a course in biology, chemistry, introductory physics, or technology/engineering that was aligned to the Massachusetts Curriculum Frameworks. School, district, and state results for the high school STE tests were reported for the first time this year. There are several unique characteristics of the high school STE tests that must be kept in mind when interpreting these reports. 1. Participation Requirements Students who were in grade 10 during the spring 2007 MCAS administration were required to take one of the four high school STE tests if they were enrolled in a corresponding course in their school. Students who were in grade 9 during spring 2007 were not required to take an STE test. They were eligible to participate (and the Department recommended that they participate) if they were enrolled in a corresponding course, but they were allowed to opt out, with the understanding that they would miss an opportunity to meet the new high school STE graduation requirement and would be required to take a test in grade 10. The impact of this difference in participation requirements is that grade 10 students participated in the tests because they were required to do so. There was no extrinsic benefit obvious to all students. On the other hand, the grade 9 students who participated may have done so because there was a potential extrinsic benefit: If they performed at the Needs Improvement level or higher, they would meet high school STE graduation requirement. The Science Participation Summary Reports provide information regarding the grade-level and demographic characteristics of students who participated in each of the four grade 10 STE tests. The participation columns specify, by subgroup, the number and percentage of students who participated in each high school STE test at grades 9 and 10. The percentages are based on March SIMS enrollment data. It is important to keep in mind that, given this information, it is not possible to distinguish the number of grade 9 students who opted out of testing from the number who were taking science courses that did not correspond to one of the four end-of-course STE tests. 2. Variations in Course Offerings and Course-Taking Patterns by School and District Schools and districts often offer more than one science course or sequence of courses (e.g., core, honors, AP). Students in one school who plan, for example, to follow an AP or honors science sequence might be more likely to take biology in grade 9 and chemistry in grade 10. In another school, students following a core sequence might be more likely to take introductory physics in grade 9 and biology in grade 10. For this reason, the MCAS high school biology results for one school or district may represent the performance of grade 9 honors students, while biology results for another school or district may reflect the performance of students who are not following an honors sequence. Because of these variations in course offerings and course-taking patterns, please use caution when comparing district-to-district and school-to-school results by test. 3. Effect of Competency Determination Requirements on Student Motivation Members of the class of 2010, who were in grade 9 in spring 2007, are the first students required to pass one of the four high school STE tests in order to earn a Competency Determination (CD) and be eligible for a high school diploma. This requirement most likely had an effect on student motivation. Our data show that the percentage of students who passed both the grade 10 ELA and Mathematics MCAS tests on their first attempt rose from 50% in 2000 (the year before performance on the tests was linked to high school graduation) to 68% in 2001 (the first year that performance on the tests determined eligibility for the CD). Since it is likely that student motivation affects performance and that the motivation of grade 9 students (for whom passing the test was linked to earning a CD) differed from the motivation of grade 10 students (for whom the test had no consequences), it is reasonable to expect that in general the performance of grade 9 students on the 2007 STE tests will differ from the performance of grade 10 students. Consequently, please proceed with caution when comparing performance on these tests across grade levels. 4. Comparability The high school Biology, Chemistry, Introductory Physics, and Technology/Engineering tests were developed independently to assess student performance on the learning standards for their respective content areas in the Massachusetts Science and Technology/Engineering Curriculum Framework. While external analyses have established that the four tests have important technical similarities^1 and performance level descriptors for each of the four tests are similar in scope and rigor, the content assessed by each of the four tests is distinct. Therefore comparisons of student performance across these tests should be made cautiously. ^1 The University of Massachusetts, Amherst, study titled "Psychometric Analyses of the 2006 MCAS High School Science Tests," will be available on the MCAS Service Center Web site the week of October 1. Appendix A: Raw Score to Scaled Score Conversion Tables TABLE 1. 2007 MCAS GRADE 4 RAW SCORE TO SCALED SCORE CONVERSIONS Grade 4 English Language Arts Mathematics Raw Score Scaled Score Scaled Score 0 200 200 1 200 200 2 202 202 3 202 202 4 202 204 5 202 204 6 204 204 7 204 206 8 204 206 9 204 206 10 206 208 11 206 210 12 206 210 13 206 212 14 208 212 15 208 214 16 208 214 17 208 216 18 210 216 19 210 216 20 210 216 21 210 218 22 212 218 23 212 218 24 212 220 25 214 220 26 214 220 27 214 222 28 214 222 29 216 224 30 216 226 31 216 226 32 216 228 33 218 228 TABLE 1. 2007 MCAS GRADE 4 RAW SCORE TO SCALED SCORE CONVERSIONS (CONTINUED) Grade 4 English Language Arts Mathematics Raw Score Scaled Score Scaled Score 34 218 230 35 218 232 36 218 232 37 220 234 38 220 236 39 220 236 40 220 238 41 222 240 42 224 242 43 226 244 44 226 248 45 228 250 46 230 254 47 230 256 48 232 260 49 234 262 50 236 266 51 238 268 52 240 274 53 240 280 54 242 280 55 244 56 246 57 248 58 250 59 252 60 254 61 258 62 260 63 264 64 266 65 270 66 276 67 280 68 280 69 280 70 280 71 280 72 280 TABLE 2. 2007 MCAS GRADE 5 RAW SCORE TO SCALED SCORE CONVERSIONS Grade 5 English Language Arts Mathematics Science and Technology/ Raw Score Scaled Score Scaled Score Engineering Scaled Score 0 200 200 200 1 200 200 200 2 202 202 202 3 202 202 202 4 204 204 204 5 204 204 204 6 206 204 204 7 206 206 206 8 208 206 206 9 208 208 208 10 208 210 208 11 210 210 208 12 210 212 210 13 212 212 210 14 212 214 212 15 214 214 212 16 214 214 214 17 216 216 214 18 216 216 216 19 218 216 216 20 218 218 216 21 218 218 218 22 220 218 218 23 220 218 218 24 222 220 220 25 224 220 220 26 224 220 222 27 226 222 224 28 228 224 226 29 230 224 228 30 230 226 230 31 232 228 232 32 234 230 232 33 236 232 234 34 236 232 236 35 238 234 238 36 240 236 240 37 242 238 242 38 244 240 246 39 246 240 248 40 246 242 250 41 248 244 252 42 252 246 256 43 254 248 258 44 256 250 260 45 258 252 262 46 262 256 264 47 264 258 266 48 268 260 270 49 272 262 272 50 276 264 276 51 280 266 280 52 280 268 280 53 274 280 54 280 280 TABLE 3. 2007 MCAS GRADE 6 RAW SCORE TO SCALED SCORE CONVERSIONS Grade 6 English Language Arts Mathematics Raw Score Scaled Score Scaled Score 0 200 200 1 200 200 2 202 202 3 202 202 4 204 202 5 204 204 6 206 204 7 206 206 8 206 206 9 208 208 10 208 210 11 210 210 12 210 212 13 212 212 14 212 212 15 214 214 16 214 214 17 216 214 18 216 216 19 216 216 20 218 216 21 218 218 22 220 218 23 220 218 24 222 218 25 222 218 26 224 220 27 226 220 28 228 220 29 230 222 30 232 224 31 234 226 32 236 228 33 236 230 34 238 232 35 240 234 36 242 236 37 242 238 38 244 240 39 246 242 40 246 244 41 248 246 42 250 248 43 252 250 44 254 252 45 256 256 46 258 258 47 262 260 48 266 262 49 270 264 50 276 266 51 280 268 52 280 272 53 276 54 280 TABLE 4. 2007 MCAS GRADE 7 RAW SCORE TO SCALED SCORE CONVERSIONS Grade 7 English Language Arts Mathematics Raw Score Scaled Score Scaled Score 0 200 200 1 200 200 2 202 202 3 202 202 4 202 202 5 202 204 6 204 204 7 204 206 8 204 206 9 206 208 10 206 210 11 206 210 12 206 212 13 208 212 14 208 212 15 208 214 16 210 214 17 210 214 18 210 216 19 210 216 20 212 216 21 212 218 22 212 218 23 214 218 24 214 218 25 214 218 26 216 220 27 216 220 28 216 220 29 216 222 30 218 224 31 218 226 32 218 228 33 218 230 34 220 232 35 220 234 36 220 236 37 222 238 38 224 240 39 224 242 40 226 242 41 228 244 42 228 246 43 230 248 44 232 250 45 234 252 46 234 254 47 236 256 48 238 258 49 240 260 50 240 262 51 242 264 52 242 266 53 244 272 54 246 280 55 246 56 248 57 250 58 252 59 252 60 254 61 256 62 258 63 260 64 264 65 266 66 270 67 272 68 278 69 280 70 280 71 280 72 280 TABLE 5. 2007 MCAS GRADE 8 RAW SCORE TO SCALED SCORE CONVERSIONS Grade 8 English Language Arts Mathematics Science and Technology/ Raw Score Scaled Score Scaled Score Engineering Scaled Score 0 200 200 200 1 200 200 200 2 202 202 202 3 202 202 202 4 204 204 204 5 204 204 204 6 206 204 204 7 206 208 206 8 208 210 206 9 208 212 208 10 210 212 210 11 210 214 210 12 212 214 212 13 212 214 212 14 214 216 212 15 214 216 214 16 216 216 214 17 216 216 216 18 218 218 216 19 218 218 216 20 220 218 216 21 220 218 218 22 222 218 218 23 224 220 218 24 226 220 220 25 226 220 220 26 228 222 220 27 230 222 222 28 232 224 224 29 234 226 226 30 234 228 228 31 236 230 228 32 238 232 230 33 240 232 232 34 240 234 234 35 242 236 236 36 244 238 238 37 244 240 238 38 246 242 240 39 246 242 242 40 248 244 244 41 250 246 244 42 250 248 246 43 252 250 248 44 254 252 250 45 256 254 252 46 258 256 254 47 262 258 256 48 264 260 258 49 268 262 262 50 272 264 264 51 278 266 270 52 280 268 276 53 274 280 54 280 280 TABLE 6. 2007 MCAS GRADE 10 RAW SCORE TO SCALED SCORE CONVERSIONS Grade 10 English Language Arts Mathematics Raw Score Scaled Score Scaled Score 0 200 200 1 200 200 2 202 202 3 202 202 4 202 204 5 202 204 6 204 206 7 204 206 8 204 208 9 206 212 10 206 214 11 206 214 12 206 216 13 208 216 14 210 218 15 210 218 16 212 218 17 212 218 18 212 218 19 214 220 20 214 220 21 214 220 22 216 220 23 216 222 24 216 224 25 216 226 26 218 228 27 218 228 28 218 230 29 218 232 30 218 234 31 218 236 32 220 238 33 220 238 34 220 240 35 220 242 36 222 244 37 224 246 38 224 246 39 226 248 40 228 250 41 230 252 42 230 254 43 232 254 44 234 256 45 234 258 46 236 260 47 238 260 48 240 262 49 240 262 50 242 262 51 244 264 52 244 264 53 246 266 54 248 266 55 248 268 56 250 268 57 252 270 58 254 272 59 256 278 60 256 280 61 260 62 260 63 262 64 264 65 266 66 268 67 270 68 274 69 276 70 280 71 280 72 280 TABLE 7. 2007 MCAS HIGH SCHOOL RAW SCORE TO SCALED SCORE CONVERSIONS High School Biology Chemistry Introductory Technology/ Raw Score Scaled Score Scaled Score Physics Engineering Scaled Score Scaled Score 0 200 200 200 200 1 200 200 200 202 2 202 202 202 204 3 202 202 202 206 4 204 204 204 208 5 204 204 204 210 6 206 204 206 210 7 206 206 206 212 8 206 206 208 212 9 208 206 208 212 10 208 210 210 214 11 210 212 212 214 12 212 212 214 214 13 214 214 214 214 14 214 214 216 216 15 216 216 216 216 16 216 216 218 216 17 216 216 218 216 18 218 216 218 216 19 218 218 218 218 20 218 218 220 218 21 220 218 220 218 22 220 218 220 218 23 220 218 222 218 24 222 220 224 220 25 224 220 226 220 26 226 220 228 220 27 228 222 230 222 28 228 224 230 224 29 230 226 232 226 30 232 228 234 228 31 234 230 236 230 32 236 232 238 230 33 238 234 238 232 34 238 236 240 234 35 240 238 242 236 36 242 240 242 238 37 242 242 244 240 38 244 244 244 240 39 246 246 246 242 40 246 246 248 244 41 248 248 248 244 42 248 250 250 246 43 250 252 250 246 44 250 254 252 248 45 252 256 254 248 46 254 258 254 250 47 254 258 256 252 48 256 260 258 254 49 258 262 260 254 50 260 262 260 256 51 260 264 262 258 52 262 264 262 260 53 264 266 264 262 54 264 266 264 264 55 266 268 266 266 56 268 270 268 268 57 270 272 270 270 58 274 276 274 274 59 278 280 280 280 60 280 280 280 280 Appendix B: Calculating Threshold Scores (Cut Points) The threshold score is the minimum raw score (number of points) that a student must earn on an MCAS test for her or his performance to be classified at a particular performance level. For example, 19 is the minimum raw score that a student must earn on the 2007 MCAS grade 10 Mathematics test for the student’s performance to be classified at the Needs Improvement level. Because all common MCAS test items are released each year, the test items on the 2007 tests are unique and may vary in difficulty from items on previous tests. To maintain standards and allow comparisons over time, threshold scores representing those standards may change to reflect differences in item difficulty across years. Therefore, MCAS tests are linked across years through a statistical process called equating. Through the equating process, adjustments in the threshold scores for the 2007 MCAS tests maintain the original MCAS standards. THRESHOLD SCORES, MCAS TESTS OF SPRING 2007 Threshold Scores for Performance Levels Grade Subject Area Maximum Advanced Proficient: Needs Score^1 (Above minimum Improvement: Proficient): score needed minimum score minimum score needed needed 3 ELA Reading Comprehension 48 45 37 22 Mathematics 40 37 30 22 4 ELA Composition and Reading Comprehension 72 62 52 37 Mathematics 54 48 41 24 5 ELA Composition and Reading Comprehension 52 46 36 22 Mathematics 54 48 38 24 Science and Technology/Engineering 54 44 36 24 6 ELA Reading Comprehension 52 47 35 22 Mathematics 54 47 38 26 7 ELA Composition and Reading Comprehension 72 63 49 34 Mathematics 54 49 38 26 8 ELA Reading Comprehension 52 47 33 20 Mathematics 54 48 37 23 Science and Technology/Engineering 54 49 38 24 10 ELA Composition and Reading Comprehension 72 61 48 32 Mathematics 60 46 34 19 High School Biology 60 50 35 21 Chemistry 60 48 36 24 Introductory Physics 60 49 34 20 Technology/Engineering 60 52 37 24 ^1 The maximum score is the total number of raw score points that a student could earn in the subject area if he/she answered all of the multiple-choice questions correctly (one point per question) and received the highest possible score for all open-response questions (four points; two points for grade 3 Mathematics), short-answer questions in Mathematics only (one point), and the ELA Composition at grades 4, 7, and 10 (twenty points). Appendix C: MCAS-Related References The resources below are all available on the Department Web site, at the specific locations listed. MASSACHUSETTS CURRICULUM FRAMEWORKS * English Language Arts Curriculum Framework * Mathematics Curriculum Framework * Science and Technology/Engineering Curriculum Framework * History and Social Science Curriculum Framework Each Framework describes what students should know and be able to do in that subject area at various grade spans. All Frameworks are available at www.doe.mass.edu/frameworks/current.html. OTHER RESOURCES Release of Spring 2007 MCAS Test Items contains all of the common test questions that were used as the basis for student results in the 2007 MCAS reports. Available at www.doe.mass.edu/mcas/testitems.html. Requirements for the Participation of Students with Disabilities in MCAS (Spring 2007 Update) outlines the requirements for participation of students with disabilities and describes allowable test accommodations. Available at www.doe.mass.edu/mcas/participation. Requirements for the Participation of Students with Limited English Proficiency in MCAS, MEPA-R/W, and MELA-O (Spring 2007 Update) outlines the requirements for participation of LEP students. Available at www.doe.mass.edu/mcas/participation. 2006 MCAS Technical Report provides specific technical details about the 2006 MCAS administration and reporting of results. Available at www.mcasservicecenter.com/iEnrollmentMCAS/TechReport.htm. The Massachusetts School and District Accountability System provides an overview of the state’s accountability system. Available at www.doe.mass.edu/sda. Student Records: Questions, Answers, and Guidelines (2002) contains information about the Massachusetts Student Record Regulations. Available at www.doe.mass.edu/mcas/lawsregs.advisories.html. Page 28 of 28 Page 3 of 28