[MCAS logo] Guide to Interpreting the Spring 2009 MCAS Reports for Schools and Districts September 2009 Massachusetts Comprehensive Assessment System [MCAS logo] This document was prepared by the Massachusetts Department of Elementary and Secondary Education Mitchell D. Chester, Ed.D. Commissioner of Elementary and Secondary Education © 2009 Massachusetts Department of Elementary and Secondary Education Permission is hereby granted to copy for non-commercial educational purposes any or all parts of this document. Please credit the “Massachusetts Department of Elementary and Secondary Education.” Massachusetts Department of Elementary and Secondary Education 75 Pleasant Street, Malden, MA 02148-4906 Phone 781-338-3000 TTY: N.E.T. Relay 800-439-2370 www.doe.mass.edu [Massachusetts Sigillum Reipublicae logo] Table of Contents I. Document Purpose 1 II. Reporting of the 2009 MCAS Test Results 2 III. General Guidelines for Interpretation and Use of MCAS Reports 3 Use of MCAS Results 3 Comparing Scores from Year to Year and Test to Test 3 Common and Matrix-Sampled Item Results 3 Composite Performance Index (CPI) Points 4 Conversion of Raw Scores to Scaled Scores 5 Comparisons of Results 5 References to LEP Students 6 State Regulations Regarding Confidentiality of Student Records 6 IV. History of MCAS Standard Setting 7 V. Understanding the Test Item Analysis Roster and the Test Item Analysis Report Summary 8 Overview 8 Test Item Analysis Roster 10 Test Item Analysis Report Summary 17 VI. Understanding the School Report and District Report 21 Overview 21 A. Student Participation Report 23 B. Distribution of Results by Composite Performance Index Points 26 C. Subject Area Subscores on Standard Test 28 D. Performance Level Results 30 E. Three-Year Comparison of School Performance Level Results 31 F. Performance Level and CPI for Subgroups of Students 32 G. Competency Determination Results (Grade 10 Only) 33 VII. Understanding the 2009 High School Science and Technology/Engineering Participation and Performance Reports 34 Appendix A: Raw Score to Scaled Score Conversion Tables 35 Appendix B: Calculating Threshold Scores (Cut Scores) 49 Appendix C: MCAS-Related References 51 I. Document Purpose The purpose of this document is to support local educators’ use of district and school reports of results from the spring 2009 administration of the Massachusetts Comprehensive Assessment System (MCAS) tests. This document provides general information about MCAS results, standards, and reports, with specific guidance on interpreting the following MCAS reports for the spring 2009 tests:  Test Item Analysis Roster • Test Item Analysis Report Summary • School Report • District Report In their efforts to better serve the academic needs of all students, local educators can use the information in these 2009 MCAS reports to improve curriculum and instruction. In addition, this document can help school and district personnel communicate with parents and members of their communities about MCAS test results. II. Reporting of the 2009 MCAS Test Results The table below shows the MCAS tests administered in spring 2009, and how each test’s results are reported. [Table 1. Spring 2009 MCAS tests and Results Reported] * = data reported 1. Results from the MCAS grade 3 ELA Reading Comprehension and Mathematics tests are reported by performance level and raw score; no scaled scores are reported for grade 3 tests. III. General Guidelines for Interpretation and Use of MCAS Reports USE OF MCAS RESULTS Results contained in the 2009 MCAS reports illustrate the extent to which students and schools are achieving the learning standards contained in the Massachusetts curriculum frameworks. The frameworks are available on the Department of Elementary and Secondary Education’s (“the Department”) website at www.doe.mass.edu/frameworks/current.html. Improvement of Student Performance Teachers and parents can use individual student results and school-level results to focus on areas in which students need assistance to improve performance. Alignment of Curriculum and Instruction Teachers and administrators can use school- and district-level results to evaluate the effectiveness of local curriculum and instruction in aligning with the standards defined in the frameworks. COMPARING SCORES FROM YEAR TO YEAR AND TEST TO TEST Because MCAS tests are different each year, it is necessary to link the tests through a process called equating. Equating is used to determine the cut scores on each subsequent MCAS test that will maintain the performance standards established on the original test. If the original and subsequent tests were identical in difficulty, the cut scores would be the same on each test. However, it is very challenging to develop two tests that are exactly equal in difficulty. Each subsequent test is likely to be slightly easier or slightly more difficult than the original test. If a subsequent test is easier, the cut scores needed to move from one performance level to the next will be slightly higher. If a subsequent test is more difficult than the original, the cut scores needed to move from one performance level to the next will be slightly lower. For the current MCAS program, tests are equated each year using an anchor test that is embedded within the actual MCAS test. The items on the anchor test do not count toward a student’s score on the test. These items are only used to link performance standards on the original and subsequent MCAS tests. Items on the anchor test are not released to the public after each test administration so that they can be used again to link future tests. The anchor test measures the same content and skills measured by the actual MCAS test. Because of the critical importance of maintaining the same performance standards across years, the Department takes extra precautions to ensure the accuracy of the equating process. Each year, psychometricians from the Department’s assessment contractor and the Center for Educational Assessment at UMASS Amherst independently and simultaneously equate the MCAS tests. The Department, with assistance as needed from its national Technical Advisory Committee, analyzes the results of the two independent equating analyses prior to reporting any MCAS test results. COMMON AND MATRIX-SAMPLED ITEM RESULTS All MCAS student-level scaled score (raw score at grade 3) and performance level results are based on common items only. Each student is administered all common items. The Test Item Analysis Roster and Test Item Analysis Report Summary report results for common items only. In addition to scaled score (raw score at grade 3) and performance level results based on common items, the School Report and District Report contain aggregated subscores for specific content area strands. These school- and district-level subscores include student responses to both common and matrix-sampled MCAS items. COMPOSITE PERFORMANCE INDEX (CPI) POINTS Composite Performance Index (CPI) points are assigned to schools and districts for each student tested for use in Adequate Yearly Progress (AYP) reporting. October 2008 enrollment data were used in the determination of school CPI points; spring 2009 enrollment data were used in the determination of district and state CPI points. CPI points are assigned for each student according to the tables below. Standard MCAS Tests CPI points for students who took the standard MCAS tests are assigned to schools and districts based on performance level categories and scaled scores (raw scores at grade 3). TABLE 2. ASSIGNMENT OF CPI POINTS FOR STANDARD MCAS TESTS GRADE 3 [TABLE 2. ASSIGNMENT OF CPI POINTS FOR Standard MCAS Tests GRADE 3] TABLE 3. ASSIGNMENT OF CPI POINTS FOR STANDARD MCAS TESTS GRADES 4–8 AND HIGH SCHOOL [Table 3. Assignment Of Cpi Points For Standard Mcas Tests Grades 4–8 and High School] Alternate Assessments Since no scaled scores are reported for students taking the Alternate Assessment, CPI points for these students are awarded based on Alternate Assessment performance level categories only. TABLE 4. ASSIGNMENT OF CPI POINTS FOR ALTERNATE ASSESSMENTS GRADE 3 [TABLE 4. ASSIGNMENT OF CPI POINTS FOR ALTERNATE ASSESSMENTS GRADE 3] TABLE 5. ASSIGNMENT OF CPI POINTS FOR ALTERNATE ASSESSMENTS GRADES 4–8 AND HIGH SCHOOL [TABLE 5. ASSIGNMENT OF CPI POINTS FOR ALTERNATE ASSESSMENTS GRADES 4–8 AND HIGH SCHOOL] CONVERSION OF RAW SCORES TO SCALED SCORES Scaled scores are derived from raw scores (based on common items only) using a data analysis process known as scaling; the scaling process is analogous to converting temperature from degrees Fahrenheit to degrees Celsius. Conversion of raw scores to scaled scores is done each year to enable the reporting of results according to the same scale each year and to ensure that the standard for attaining each performance level remains true to the original standards. Appendix A contains the 2009 MCAS Raw Score to Scaled Score Conversion Tables for each test administered in spring 2009. Appendix B contains an explanation of how threshold scores for each performance level are determined. Further information about MCAS scaling methods can be found in the MCAS 2008 Technical Report, which can be accessed by going to www.mcasservicecenter.com and clicking “MCAS.” COMPARISONS OF RESULTS Performance levels are the most appropriate measure to use when comparing MCAS results among students, schools, and districts. Results from the 2009 MCAS tests should be evaluated in conjunction with results from MCAS tests administered from 1998 through 2008. As with any evaluation, school and district test results are most meaningful when compared with other indicators and when examined over several years for long-term trends in performance. More information regarding score interpretations over time is available on page 3 (“Comparing Scores from Year to Year and Test to Test”). REFERENCES TO LEP STUDENTS Limited English Proficient (LEP) students have been designated as LEP in the Student Information Management System (SIMS). In reports for schools and districts, the meaning of “LEP” depends on its context. 1. Performance Level Coding When the code LEP appears as a performance level designation, it refers to a limited English proficient student in his or her first year of enrollment in U.S. schools. First-year LEP students who participated in the Massachusetts English Proficiency Assessment (MEPA) are not ordinarily included in results other than AYP participation rates. The Department uses SIMS enrollment records to confirm first-year status. 2. Results Reporting In reports of results, the code LEP refers to those limited English proficient students who participated in MCAS testing and who are not first-year LEP students. (The code FLEP [Formerly Limited English Proficient] refers to those students who are not currently classified as LEP but who were classified as LEP at any time within the past three years.) STATE REGULATIONS REGARDING CONFIDENTIALITY OF STUDENT RECORDS The Massachusetts Student Record Regulations (603 CMR 23.00) and the federal Family Educational Rights and Privacy Act (FERPA) restrict access to individual student results, including those provided in MCAS Test Item Analysis Rosters and Parent/Guardian Reports. A student’s temporary record contains information that does not appear in the student’s transcript. This temporary record generally cannot be examined by anyone other than the student, his or her parents or guardians, and authorized school personnel, unless the parent/guardian or student gives consent. A student’s temporary record is maintained for at least five years after he or she transfers, graduates, or withdraws from the school system. Superintendents and principals are responsible for ensuring the privacy and security of all student records. In accordance with state and federal regulations, authorized school personnel shall have access to the records of students to whom they are providing services only when such access is required for the performance of their official duties. For more information on this topic, refer to the Massachusetts Department of Elementary and Secondary Education publication Student Records: Questions, Answers, and Guidelines. IV. History of MCAS Standard Setting The MCAS tests were first administered to students in Massachusetts in 1998. At that time, English Language Arts (ELA) and Mathematics were the subjects assessed at grades 4, 8, and 10. The No Child Left Behind (NCLB) Act required that students be tested in grades 3 through 8 and 10 by 2006. In order to meet these requirements, it was necessary to add tests for the grades and subjects that were not previously included. From 2001 to 2006, additional grades and subject areas were tested. Following the initial administration of each new test, performance standards were set. NCLB also required that four performance levels be reported. Therefore, in 2006, a fourth performance level (Above Proficient) was established for grade 3 Reading. A meaningful third performance level threshold score was reset between Proficient and Above Proficient for grade 3 Mathematics in summer 2007. The high school Science and Technology/Engineering (STE) subjects were operationally assessed for the first time in 2007. Table 6 displays the years (1998–2007) in which standards were set for various subjects by grade, and the standard-setting method used. TABLE 6. STANDARD-SETTING HISTORY 1998–2007 Grade Content Year Method 3 Reading 2001/2006 Bookmark/Body of Work 3 Mathematics 2006/2007 Body of Work 4 ELA 1998/2001 Body of Work 4 Mathematics 1998 Body of Work 5 Language & Literature 2006 Body of Work 5 Mathematics 2006 Body of Work 5 Science and Technology/Engineering 2003 Body of Work 6 Language & Literature 2006 Body of Work 6 Mathematics 2001 Body of Work 7 ELA 2001 Body of Work 7 Mathematics 2006 Body of Work 8 ELA 1998/2006 Body of Work 8 Mathematics 1998 Body of Work 8 Science and Technology/Engineering 2003 Body of Work 10 ELA 1998 Body of Work 10 Mathematics 1998 Body of Work High School Science and Technology/Engineering 2007 Body of Work V. Understanding the Test Item Analysis Roster and the Test Item Analysis Report Summary OVERVIEW The Test Item Analysis Roster and Test Item Analysis Report Summary provide results from all MCAS tests administered in spring 2009. The information provided in these reports can be used to • identify strengths and weaknesses in subject area knowledge and skills exhibited by students • evaluate and plan curricula in the classroom, school, and district • summarize and discuss results with parents and the public (see “State Regulations Regarding Confidentiality of Student Records” on page 6) Beginning in 2009, the Department will release approximately 50% of common items annually for all grades 3–8 MCAS tests, instead of all common items. The items released will be representative of all reporting categories assessed. No change will be made to the Department’s current practice of releasing all common test items for ELA, Mathematics, Biology, and Introductory Physics at the high school level. No items will be released in 2009 for high school Chemistry and Technology/Engineering. The Test Item Analysis Roster The Test Item Analysis Roster shows student-level results for individual MCAS common items. Each school receives a separate Test Item Analysis Roster for each subject area and grade tested in the school. For each test, the roster shows how each student • answered each released multiple-choice question and performed on each unreleased multiple-choice question • performed on each common open-response question • performed on each short-answer question, where applicable • performed on the English Language Arts Composition (writing prompt), where applicable The roster is designed to be used in conjunction with the publication Release of Spring 2009 MCAS Test Items, which can be found on the Department website at www.doe.mass.edu/mcas/testitems.html. The Release of Spring 2009 MCAS Test Items shows the questions labeled “released” in Test Item Analysis Rosters, providing educators with a detailed picture of each student’s MCAS performance. This picture should be used in combination with the student’s classroom performance to develop an overview of his or her academic achievement. The Test Item Analysis Report Summary The Test Item Analysis Report Summary shows overall performance at the school or district level, and at the state level, for individual common items. Each school and district receives a separate Test Item Analysis Report Summary for each subject area and grade tested in the school or district. Rosters and Summaries for Specific Tests Grades 4, 7, and 10 English Language Arts (ELA) rosters and summaries differ slightly from these reports for other tests, since they include results for the ELA Composition, as described on the following pages. There are four high school Science and Technology/Engineering (STE) Test Item Analysis Rosters and Test Item Analysis Report Summaries, one for each of the four STE tests: Biology, Chemistry, Introductory Physics, and Technology/Engineering. These rosters and summaries combine results from students in grades 9 and 10. Specifically, the rosters include grade 9 students who participated in an STE test in 2009 and all grade 10 students (based on SIMS). The summaries include grade 9 and grade 10 students who participated in STE in 2009. In addition to these rosters and summaries, the Roster of Class of 2011 Students’ Performance Levels through Spring 2009 is also provided. This roster summarizes ELA, Mathematics, and STE results for all members of the class of 2011 (as determined by June 2009 SIMS). The information on the roster is based on the highest MCAS results each student has earned to date in high school. The columns at the far right of the roster indicate whether a student is included in summary performance results for each subject at both the school and district levels; inclusion is based on SIMS records. The policies for inclusion are outlined on the legend page of the roster. ELA and Mathematics Retest results are reported only at the student level (in Test Item Analysis Reports); retest results are not aggregated at school or district levels. Test Item Analysis Roster [Sample test item analysis roster] TEST ITEM ANALYSIS ROSTER Each Test Item Analysis Roster has three main sections: 1. Item Information 2. Individual Student Results 3. Legend The number of pages in the roster depends on the total number of students in the tested grade. 1. Item Information [sample of section 1 of test item analysis roster] Section 1 of the roster (as shown on page 10 and above) contains four types of information for each common item, listed vertically: • item number • item type • correct multiple-choice (MC) answer for released items only, if applicable • reporting category Section 1 also shows the total raw score points possible for the reported test. Item Number Item Number corresponds to the number of the question as it appears in the Release of Spring 2009 MCAS Test Items publication. For grades 4, 7, and 10 ELA only, this information is blank for the first two columns, which provide ELA Composition data. Item Type Item Type indicates whether the item was • multiple-choice (MC) • open-response (OR) • short-answer (SA) for Mathematics only • ELA Composition writing prompt (“WP” appears in the first two columns) for grades 4, 7, and 10 ELA only See “Individual Student Results” on page 12 for further information about interpreting Item Type data. Correct MC Answer Correct MC Answer provides the letter option (A, B, C, or D) for the correct answer for released multiple-choice questions. No correct answer is provided for unreleased multiple-choice items. Reporting Category Each Reporting Category is a strand of the curriculum framework for the tested subject area (e.g., Reading and Literature is a strand of the English Language Arts Curriculum Framework and an MCAS reporting category). For grades 4, 7, and 10 ELA, only the first two columns relate to performance on the composition. Reporting category codes are explained in the Legend at the end of the Test Item Analysis Roster (see page 16). Total Possible Points The number shown in the fourth-from-last column for Total Poss. Points (see page 10) is the sum of the maximum raw score points for all common questions on the test: 1 point for each MC question + 4 points for each OR question (2 points for grade 3 Mathematics OR questions) + 1 point for each SA question (Mathematics only) + 20 points for each WP (grades 4, 7, and 10 ELA only: includes points for both WP columns) Test Status Codes* New for 2009, each student will receive one of the following Test Status Codes: T = Tested NTA = Not Tested Absent NTL = Not Tested First-Year Limited English Proficient NTO = Not Tested Other NTM = Not Tested Medically Documented Absent 2. Individual Student Results [sample of individual student results] Individual student results for each common test item are provided in section 2 (as indicated on page 10) of the roster. Results are displayed in a separate row for each student enrolled in the school or district on the first day of testing. Results for each question are provided using various symbols, letters, and numbers, based on Item Type, as further explained below. For students who took the Alternate Assessment, results for all items are automatically left blank. For all other students, regardless of question type, a blank space indicates the student did not answer the question and a raw score of 0 was assigned for that question. Each student’s summary scores are provided in the final four columns of his or her row. *New in Spring 2009: Students with a test status of NTA, NTL, NTO, or NTM who completed enough of the test to earn a performance level of Needs Improvement or higher receive scaled scores and performance levels but are not included in performance level or CPI aggregations. Multiple-Choice Each multiple-choice question received a score of 0 for an incorrect answer or 1 for a correct answer. The following symbols and letters are used to indicate MC question results: • A + sign indicates the student answered the question correctly. • A letter (A, B, C, or D) indicates which incorrect option the student selected for a released question. • An asterisk (*) indicates the student selected more than one option and received a score of 0 on the question. • A – sign indicates the student selected an incorrect option for an unreleased question. Open-Response Each open-response question received a score of 0–4 (except for grade 3 Mathematics open-response questions, each of which received a score of 0–2). Short-Answer Short-answer questions appeared on Mathematics tests only. Each short-answer question received a score of 0 for an incorrect answer or 1 for a correct answer. Writing Prompt Only grades 4, 7, and 10 English Language Arts Test Item Analysis Rosters show scores for the ELA Composition in the first two writing prompt (WP) columns. The first column shows the student’s score for topic development (e.g., idea development, organization, use of detail, variety in sentence structure and language). The second column shows the student’s score for standard English writing conventions (e.g., grammar, punctuation, mechanics of writing). Each student’s composition was scored independently by two scorers. Each scorer gave the composition two separate scores: • a score of 1–6 for topic development (CT) • a score of 1–4 for the student’s use of standard English writing conventions (CC) The student’s score shown in each of these two columns (CT and CC) represents the sum of the points awarded by both scorers in each area (up to 12 points for CT; up to 8 points for CC). The Legend on the last page of the roster displays the characteristics of student writing for each score for topic development and standard English writing conventions. New in 2009, any student who wrote a totally illegible composition, wrote an off-topic response, or wrote in a language other than English was given one of the following codes listed below and received a score of 0. blank space = no composition written (0 points earned) IE = insufficient evidence (0 points earned) LO = language other than English (0 points earned) OT = off topic (0 points earned) PS = plot summary (0 points earned) VI = visible but incomprehensible (0 points earned) Total Raw Score The student’s total raw score was calculated by adding one point for each correct MC response to the points awarded for OR and (if applicable) SA responses, and (if applicable) to writing CT and CC points. Scaled Score The student’s total raw score is translated to a scaled score and reported on a scale ranging from 200–280. No scaled scores are reported for grade 3 students; this column does not appear on grade 3 rosters. Composite Performance Index (CPI) Points (Comp. Perf. Ind. Pts.) The number in this column indicates the number of CPI points assigned to the school or district based on that student’s results. Chapter III of this guide provides further information about the assignment of CPI points. Performance (Perf.) Level The student’s performance level is reported as a code; performance level codes are listed and defined in the legend of the roster, and are also explained below. Standard Performance Levels Standard Performance Level Codes Grade 3 Grades 4–8 High School P+ = Above Proficient A = Advanced A = Advanced P = Proficient P = Proficient P = Proficient NI = Needs Improvement NI = Needs Improvement NI = Needs Improvement W = Warning W = Warning F = Failing Alternate Assessment Performance Levels Seven performance levels are reported for students who participated in the MCAS Alternate Assessment, as shown below. The top three levels are equivalent to the standard MCAS performance levels with the same names. The next three (PRG, EMG, and AWR) fall within the standard MCAS Warning (Failing for grade 10) performance level. The final alternate assessment performance level, INP, corresponds to a standard MCAS “not tested” code (INC, listed under “Exceptions” on page 15). Alternate Assessment Codes A_A (P+A for grade 3) = Advanced (Above Proficient for grade 3) P_A = Proficient NIA = Needs Improvement PRG = Progressing EMG = Emerging AWR = Awareness INP = Incomplete Portfolio Certificate of Mastery Performance Levels For students who were enrolled in grade 11 or 12 and who participated in the grade 10 ELA or Mathematics test to attempt to qualify for a scholarship, the following Certificate of Mastery Codes appear in the Performance Level column. Certificate of Mastery results are excluded from school and district summaries. Certificate of Mastery Codes A_M = Advanced P_M = Proficient NIM = Needs Improvement F_M = Failing Exceptions There are limited circumstances in which scaled score and performance level results are not provided for a student; instead, in the Performance Level column, one of the following three-letter codes appears. These codes are also listed and defined in the legend on the last page of the roster. For further information regarding these codes, see pages 24–25 of this guide. Not Tested Codes ABS = Absent MED = Absent—medically documented INC = Incomplete or invalid test results LEP = First-year limited English proficient TRN = Transferred VAB = Void answer booklet (appears in preliminary roster reports only) DUP = Multiple answer booklets submitted for this student (appears in preliminary roster reports only) PRE = Repeating grade 10 students who previously participated in grade 10 ELA or Mathematics tests PAS = Previously passed (high school STE only) PRF = Previously failed (high school STE only) New in 2009: ABS, MED, LEP, and TRN students will receive scaled scores and performance levels if their raw score is at or above the Needs Improvement threshold. These results are provided for individual diagnostic purposes. For high school students, results will count toward meeting the Competency Determination requirement. The corresponding test status—NTA, NTM, NTL, and NTO—indicates that the students are not included in performance level or CPI aggregations. 3. Legend The legend on the last page of each Test Item Analysis Roster provides a quick reference for interpreting the codes that appear throughout the Roster. The graphic on the following page shows the legend from a grade 8 Mathematics Test Item Analysis Roster. For grades 4, 7, and 10 English Language Arts (ELA) Test Item Analysis Rosters only, the bottom section of the legend also shows characteristics of student writing corresponding to each ELA Composition topic development (CT) and standard English writing conventions (CC) score. [Final test item analysis roster legend] Test Item Analysis Report Summary [Test item analysis report summary] TEST ITEM ANALYSIS REPORT SUMMARY The Test Item Analysis Report Summary for each test (by grade and subject area) shows student results aggregated at the school and/or district level(s), and at the state level, for each common test item. Each row of the summary provides data for one common item. Each summary has four main sections, as described on pages 18–20: 1. Item Information 2. Average Item Score 3. Percentage of Total Student Responses 4. Legend 1. Item Information [Item information ] The first three columns of the summary show the Item Number, Item Type, and Reporting Category of the common item. This information correlates directly to the information provided on the Test Item Analysis Roster for the same grade and subject area (see pages 11 and 12 of this guide for more details about these three categories). The fourth column of section 1 lists the framework learning standard tested by that item. 2. Average Item Score [Average item score] The Average Item Score section of each summary shows aggregated results for each item. School summaries include aggregated school, district, and state results for each item; district summaries include aggregated district and state results for each item. These statistics allow educators to compare the performance of students in a school to students in the district and across Massachusetts, and to identify strengths and weaknesses in local curriculum and instruction. For multiple-choice and (on Mathematics summaries only) short-answer questions, which are both scored 0 or 1, the aggregate score shown is equivalent to the percentage of students who answered the question correctly. For open-response questions (scored 0–4, or 0–2 in grade 3 Mathematics) and writing prompts (scored 2–20), the average student score for the question is shown. The total at the bottom of each column is the sum of the average individual item scores for the entire test. 3. Percentage of Total Student Responses [Percentage of total student responses] The six columns in section 3 (as indicated on page 17) show correct answers for the released multiple-choice questions (in the far right column), as well as two kinds of aggregated results: • For each released multiple-choice question, the columns show the percentage of students who chose each answer option (blank, A, B, C, or D). By looking at items on which students performed well, educators can identify areas of strength in their curriculum and instruction. By focusing on items on which students made similar mistakes, educators can identify possible misconceptions held by students in the school, and can plan to address the issues by strengthening curriculum and instruction. • For released open-response and short-answer questions (on Mathematics summaries only), the columns show the percentage of students at each score point (blank/0, 1, 2, 3, or 4 for open-response questions [blank/0, 1, or 2 for grade 3 Mathematics]; blank/0 or 1 for short-answer questions). These data can be used in conjunction with item rubrics and sample student work (available on the Department website at www.doe.mass.edu/mcas/student) to inform changes in curriculum and improve instruction. For each unreleased question, the six columns in section 3 are blank. 4. Legend [Test item analysis report legend] The legend is on the last page of each Test Item Analysis Report Summary. VI. Understanding the School Report and District Report OVERVIEW Each School Report and District Report provides aggregated MCAS results for the district and state; each School Report also provides aggregated results for that school. A separate report was produced for each grade level (3–8 and 10) tested in spring 2009 and for each of the four high school Science and Technology/Engineering tests.^1 The guidelines given in this chapter may be used to interpret both the School Report and the District Report, which are nearly identical in format. All examples shown throughout this chapter refer to the School Report template. Organization of the School Report The first few pages of each School Report provide general information and definitions to help interpret the report, including a table of contents, performance level definitions, score ranges, student status definitions that relate to subgroup statistics (see section F below), and explanations regarding students who were not tested. Also provided are charts that define how Composite Performance Index (CPI) points are awarded for each test included in the Report. These introductory pages are followed by reports of spring 2009 MCAS test results. These reports include: A. Student Participation Report B. Distribution of Results by Composite Performance Index Points C. Subject Area Subscores on Standard Test D. Performance Level Results E. Three-Year Comparison of School Performance Level Results (only spring 2008 and 2009 results are available for high school Science and Technology/Engineering comparisons) F. Performance Level and Composite Performance Index for Subgroups of Students G. Competency Determination Results (Grade 10 Only) Each of these reports is described in further detail on the following pages. ^1 ELA and Mathematics Retest results are not aggregated at school or district levels and do not appear in School Reports or District Reports; retest results are reported only at the student level (in Test Item Analysis Rosters). Identification [sample identification box] A box in the upper right corner of each page of the School Report shows the school name, its eight-digit identification number, the district name, and the grade level for which the report has been generated. Minimum Number of Students Needed to Generate Reports In order to ensure confidentiality of individual student results and discourage generalizations about school performance based on very small populations, the Department requires that a minimum of 10 students per reporting area must be tested to report school-level results in any of the following areas: • overall school performance level and scaled score results • school results disaggregated by subgroup • subject area subscore results^2 Consequently, schools with a very small number of students enrolled in a tested grade may not show results in some sections of their School Report. If a school tested fewer than 10 students in a particular grade, no School Report is generated for that grade; however, results for these students are included in district- and state-level results reported in District Reports and the State Report. ^2 Multiple forms are created for each MCAS test. Matrix-sampled items are distributed across all forms. A school must administer all forms of the test for accurate computation of subject area subscores. If a school did not administer all forms of a test, the subject area subscore pages are left blank in the school’s Report. A. Student Participation Report [Student participation report] The pages titled Student Participation Report show—in separate school, district, and state tables—how many enrolled students were included or not included in performance level results. Each table contains 10 rows that report data based on student subgroup. The report has five basic sections: 1. Student Subgroups 2. Number of Students Enrolled 3. Adequate Yearly Progress Participation Rate (AYP Part. Rate) 4. Included in Performance Level Results 5. Not Included in Performance Level Results Each section is described in more detail below. 1. Student Subgroups Section 1 lists the subgroups into which data for the school’s total student population are subdivided. The total population is divided into three Student Status subgroups; it is separately divided into seven Racial/Ethnic subgroups. The three Student Status subgroups are defined as follows: • Students with Disabilities (Sw/D). Students with Disabilities are defined here as students who have an Individualized Education Program (IEP), as reported in SIMS. • Limited or Formerly Limited English Proficient (LEP/FLEP). The LEP/FLEP subgroup combines students who were LEP at the time of testing with those who have been identified as LEP in any of the past three school years. • Low-Income. A student in this subgroup meets ANY ONE of the following SIMS definitions of low-income: - The student’s family is eligible for free or reduced price lunch. - The student’s family receives Transitional Aid to Needy Families benefits. - The student’s family is eligible for food stamps. 2. Number of Students Enrolled Section 2 shows, for each student subgroup, the Number of Students Enrolled in the tested grade during the testing window. The total number of students enrolled includes both those Included in Performance Level Results (section 4) and those Not Included in Performance Level Results (section 5). 3. Adequate Yearly Progress Participation Rate (AYP Part. Rate) Section 3 shows the AYP Participation Rate for each subgroup. The number that appears on the Total row is the number of students tested in spring 2009 (either on the standard MCAS or on the MCAS Alternate Assessment) divided by the number of students enrolled on the dates of test administration. First-year LEP students who participated in the MEPA are included in the calculation, as participants (for the ELA test only). All students who have an ABS or MED test status are included in the calculations, as non-participants. Students with an INC test status are not included in the calculations. At the school level, outplaced students are not included either, unless the school is the only one in its district. 4. Included in Performance Level Results Section 4 divides subgroup results into students who were tested on the standard MCAS tests (Tested Standard) and those who participated in the Alternate Assessment (Tested Alternate Assessment). The numbers of Students with Disabilities (Sw/D) for Tested Standard include students who participated in the standard test with or without accommodations. At grade 10 only, the numbers for Limited English Proficient or Formerly Limited English Proficient (LEP/FLEP) students include those students who took the Spanish-language version of the grade 10 Mathematics test as well as those who took the English-language version of that test. (A Spanish-language version was not available for any other MCAS test.) Students who were reported as first-year LEP students on MCAS answer booklets were not ordinarily included in results other than AYP participation rate results (see section 5 below). Exceptions occurred when SIMS enrollment records indicated students had attended schools in Massachusetts before March 1, 2008. In this case, the students were not in fact first-year LEP and were therefore counted as tested; they received scaled scores and were included in School Report performance level results. 5. Not Included in Performance Level Results Section 5 shows the number of students in each subgroup who did not participate in the spring 2009 MCAS tests. The seven School Report categories for students who were Not Included in Performance Level Results are defined below. • Absent (ABS). Any student who was absent during the testing period, and who did not receive a medically documented absence as described below, received a performance level of ABS. These results are not included in the determination of scaled score, performance level, or CPI results, but are counted against the AYP participation rate. • Absent—Medically Documented (MED). Any student who obtained a written statement from a licensed medical professional, stating both the medical reason that prevented him or her from taking the test and the date(s) for which he or she was absent due to the medical condition, received a performance level of MED. These results are not included in the determination of scaled score, performance level, or CPI results, but are counted against the AYP participation rate. • Previously Participated (PRE)(grade 10 only). Repeating grade 10 students who previously participated in grade 10 ELA and/or Mathematics tests received a performance level of PRE. • Incomplete Test Results (INC). Any student whose test results were incomplete or invalid received a performance level of INC. These results are not included in the determination of scaled score, performance level, CPI, or AYP participation rate results. If, for instance, a grade 3 student took the Mathematics test but no scorable ELA answer booklet was returned, the student’s ELA results will appear as INC. However, that student’s Mathematics results will be reported as usual and included in school- and district-level results. • First-Year LEP (LEP). LEP students in their first year of enrollment in U.S. schools receive a performance level of LEP. First-year LEP results are not included in scaled score, performance level, or CPI results, but are included in AYP participation rate results, provided that the students participated as required in MEPA. (See section 4 for the exception to this policy.) • Transferred (TRN). Any student who transferred into or out of a school during the testing window and who did not complete all of the test sessions at that school received a performance level of TRN. These results are not included in the determination of scaled score, performance level, CPI, or AYP participation rate results. • Tested and Enrolled after October 1st. Any student who enrolled in a school after October 1, 2008, was required to participate in MCAS testing but was not included in most school-level MCAS or accountability reports because the student was not enrolled for a full academic year. However, these students are included in school AYP participation rates and in district-level summaries. Further, results for these students are also included in school-level results for districts with only one school, such as charter and regional vocational districts. B. Distribution of Results by Composite Performance Index Points [Distribution of results by composite performance index points ] This page of the report is designed to help schools see how their students are clustered within each performance level and Composite Performance Index (CPI) point interval for each subject area. It can also be used to illustrate how any given school or subgroup with similar aggregate CPI points may have very different percentages of students at each performance level. This results from different distributions of individual scores within CPI point intervals. 1. Performance Levels Arranged by CPI Points Section 1 provides a separate row for each applicable standard MCAS and MCAS Alternate Assessment performance level at the grade level of the report. It also provides rows showing Total CPI Points,Total Students, and Average CPI Points. 2. CPI Points Section 2 illustrates how CPI points correspond to performance levels and, for standard tests, to scaled scores (except at grade 3, for which raw scores are reported instead). The number of CPI points shown is assigned for each student whose results fall within one of the performance level categories listed within that CPI point interval. 3. Student Performance by Subject Area Section 3 shows student numbers and percentages in each performance level for the school, district, and state. Each subject area’s results are shown in a separate table; tables are presented side by side on the same page. 4. Totals The total number of CPI points awarded, based on the numbers of students in the column above and multiplied by the corresponding CPI point intervals, is shown for the school, district, and state on the top shaded row of section 4. The total number of students counted toward the school, district, or state CPI is shown on the middle shaded row of section 4. The average number of CPI points is equal to the Total CPI Points divided by the Total Students and is shown on the bottom shaded row. C. Subject Area Subscores on Standard Test [subject area subscores on standard test] This section of the School Report provides information on student performance in each reporting category for the tested subjects.^3 Each subject area’s results are presented in a separate table. Each reporting category is a strand of the curriculum framework for the tested subject area (e.g., Reading and Literature is a content strand of the English Language Arts Curriculum Framework and an MCAS reporting category). Subject area subscores are given as raw score points and are the only reported MCAS scores based on both common and matrix-sampled items. Multiple forms are created for each MCAS test; unlike common items, matrix-sampled items differ across test forms and are used to report performance on a broader range of specific curriculum goals than is possible using common items only. A school must administer all forms of a test in a grade for accurate computation of subject area subscores. If a school did not administer all forms of a test in a grade, the subject area subscore pages are left blank in the school’s report for that grade. This situation often arises in schools with small student populations. For each reporting category and item type used within each tested subject area, the subscore pages show the following (in raw score points): • the number of possible points for common items, the number of possible points for matrix items, and the total number of possible points • the average total number of points attained and the percentage of total possible number of points attained at the school, district, and state levels^4 • the average total number of points attained and the percentage of total possible number of points attained for students at the state level in each of the standard MCAS performance levels for the tested grade The following cautions apply only to raw scores, not to performance level results or scaled scores, both of which are statistically equated from year to year. Refer to the 2008 MCAS Technical Report for more information. ^3 Alternate assessment results are not included in subject area subscores. ^4 Because schools may have had different numbers of students taking each test form, average points were determined using a two-step process. First, the average raw score was determined for each unique item. Second, the sum of these average raw scores was calculated. CAUTION: Raw Scores versus Scaled Scores. The numbers for possible subscore points are based on raw score points, not on the scaled scores used elsewhere throughout the report (except at grade 3, where only raw scores are reported). Since subject area subscores combine results for both common and matrix-sampled items, total points for a subject area may differ on these subscore pages from total raw points reported elsewhere in this report or in other MCAS reports, which report raw scores for common items only. CAUTION: Longitudinal Comparisons. Comparisons of performance on subject area subscores across years must be made with caution, because the number of items contributing to each subscore is relatively small, and because the difficulty of items may vary somewhat from year to year. The pool of matrix-sampled items varies from year to year as items move from the matrix-sampled sections to the common sections of the tests to replace publicly released common items. D. Performance Level Results [Performance level results] Performance level results for the school, district, and state indicate, for each subject area tested, the percentage of students performing at each of the four standard MCAS performance levels for the tested grade level.^5 These percentages are based on the performance of all students who took either the standard MCAS tests or the MCAS Alternate Assessment.^6 Performance level results do not include the results of students whose test status is the following: • NTA: absent • NTM: absent—medically documented • NTL: first-year LEP • NTO: transferred, incomplete, previously participated See pages 24–25 of this guide for more information. At the school level, the charts include only students who were enrolled on October 1, 2008, unless the school is the only school in its district. ^5 District-level and state-level results include performance level results for students in out-of-district placements. ^6 Alternate assessment performance level results are matched with standard MCAS performance levels in this section. See page 5. E. Three-Year Comparison of School Performance Level Results [Three-year comparison of school performance level results] This section of the 2009 School Report shows MCAS results from the spring 2007, 2008, and 2009 administrations. A separate page is generated for each subject area tested. These data, along with results from other past MCAS tests, can be used to make longitudinal comparisons of results from tests that were operational in all three years. This page for STE in the grade 10 School Report or District Report includes results from 2008 and 2009 only. For grade 3 tests, no scaled scores are generated. Consequently, the table shown on the right side of the sample on this page does not appear in the grade 3 reports. Performance Level Results include students who took either the standard MCAS test or MCAS-Alt. Distribution of Results on Standard Test by Scaled Score Interval includes only students who took the standard MCAS test. F. Performance Level and CPI for Subgroups of Students [Performance level and CPI for subgroups of students] The information on these pages of the School Report is used for two primary purposes: • to determine how effectively school programs are serving various student populations • to promote fair comparisons among schools and among districts by allowing them to compare the performances of similar student groups A separate page is generated for each subject tested. Each page shows performance level results at school, district, and state levels for students in eight different AYP Subgroups and in nine Other Subgroups, as listed in the graphic on this page. In addition, the total number of students is reported, along with average percentage of students at each performance level, and average number of CPI points assigned, for school, district, and state. Note that on these pages of the report, all references to LEP students apply to current LEP students who participated in testing and who generated results. G. Competency Determination Results (Grade 10 Only) [Competency determination results] Beginning with the class of 2010, students must reach or exceed the Proficient performance level on the grade 10 English Language Arts and Mathematics tests or retests, and they must also reach or exceed the Needs Improvement performance level in one of the four high school Science and Technology/Engineering tests, in order to earn a Competency Determination (CD). This section of the grade 10 report shows the number and percentage of class of 2011 students who • have attained a CD in all three subjects (percentage only) • have attained a CD in each subject • need to complete an Educational Proficiency Plan (EPP) in English Language Arts and/or Mathematics These CD results are cumulative and include results for students who earned their CD in prior MCAS tests. For complete information on the EPP requirements, including regulations, Commissioner’s updates, frequently asked questions, and resources for planning and implementation see www.doe.mass.edu/hsreform/epp. VII. Understanding the 2009 High School Science and Technology/Engineering Participation and Performance Reports In spring 2009, schools were required to offer a high school MCAS Science and Technology/Engineering (STE) test to students in grades 9 and 10 who were enrolled during the 2008–2009 school year in a course in biology, chemistry, introductory physics, or technology/engineering that was aligned to the Massachusetts curriculum frameworks. There are several unique characteristics of the high school STE tests to consider when interpreting these reports. 1. Participation Requirements Students who were in grade 9 during spring 2009 were not required to take an STE test. They were eligible to participate (and the Department strongly recommended that they participate) if they were enrolled in a corresponding course, but they were allowed to opt out, with the understanding that they would miss an opportunity to meet the high school STE graduation requirement and would be required to take a test in grade 10. Students who were in grade 10 during the spring 2009 MCAS administration were required to take one of the four high school STE tests if they did not particpate in 2008. The Science Participation Summary Reports provide information regarding the grade-level and demographic characteristics of students who participated in each of the four high school STE tests. The participation columns specify, by subgroup, the number and percentage of students who participated in each high school STE test at grades 9 and 10. The percentages are based on March and June 2009 SIMS enrollment data. It is important to keep in mind that, given this information, it is not possible to distinguish the number of grade 9 students who opted out of testing from the number who were taking science courses that did not correspond to one of the four end-of-course STE tests. 2. Variations in Course Offerings and Course-Taking Patterns by School and District Schools and districts often offer more than one science course or sequence of courses (e.g., core, honors, AP). Students in one school who plan, for example, to follow an AP or honors science sequence might be more likely to take biology in grade 9 and chemistry in grade 10. In another school, students following a core sequence might be more likely to take introductory physics in grade 9 and biology in grade 10. For this reason, the MCAS high school biology results for one school or district may represent the performance of grade 9 honors students, while biology results for another school or district may reflect the performance of students who are not following an honors sequence. Because of these variations in course offerings and course-taking patterns, please use caution when comparing district-to-district and school-to-school test results by subject. 3. Comparability The high school Biology, Chemistry, Introductory Physics, and Technology/Engineering tests were developed independently to assess student performance on the learning standards for their respective content areas in the Massachusetts Science and Technology/Engineering Curriculum Framework. While external analyses have established that the four tests have important technical similarities^1 and performance level descriptors for each of the four tests are similar in scope and rigor, the content assessed by each of the four tests is distinct. Therefore, comparisons of student performance across these tests should be made cautiously. ^1 The University of Massachusetts, Amherst, study titled “Psychometric Analyses of the 2006 MCAS High School Science Tests” is available at www.mcasservicecenter.com (click “MCAS”). Appendix A: Raw Score to Scaled Score Conversion Tables Table 1. 2009 MCAS Grade 4 Raw Score to Scaled Score Conversions [Table 1. 2009 MCAS Grade 4 Raw Score to Scaled Score Conversions] Table 1. 2009 MCAS Grade 4 Raw Score to Scaled Score Conversions (Continued) [Table 1. 2009 MCAS Grade 4 Raw Score to Scaled Score Conversions (Continued)] Table 2. 2009 MCAS Grade 5 Raw Score to Scaled Score Conversions [Table 2. 2009 MCAS Grade 5 Raw Score to Scaled Score Conversions] Table 2. 2009 MCAS Grade 5 Raw Score to Scaled Score Conversions (Continued) [Table 2. 2009 MCAS Grade 5 Raw Score to Scaled Score Conversions (Continued)] Table 3. 2009 MCAS Grade 6 Raw Score to Scaled Score Conversions [Table 3. 2009 MCAS Grade 6 Raw Score to Scaled Score Conversions] Table 3. 2009 MCAS Grade 6 Raw Score to Scaled Score Conversions (Continued) [Table 3. 2009 MCAS Grade 6 Raw Score to Scaled Score Conversions (Continued)] Table 4. 2009 MCAS Grade 7 Raw Score to Scaled Score Conversions [Table 4. 2009 MCAS Grade 7 Raw Score to Scaled Score Conversions] Table 4. 2009 MCAS Grade 7 Raw Score to Scaled Score Conversions (Continued) [Table 4. 2009 MCAS Grade 7 Raw Score to Scaled Score Conversions (Continued)] Table 5. 2009 MCAS Grade 8 Raw Score to Scaled Score Conversions [Table 5. 2009 MCAS Grade 8 Raw Score to Scaled Score Conversions] Table 5. 2009 MCAS Grade 8 Raw Score to Scaled Score Conversions (Continued) [Table 5. 2009 MCAS Grade 8 Raw Score to Scaled Score Conversions (Continued)] Table 6. 2009 MCAS Grade 10 Raw Score to Scaled Score Conversions [Table 6. 2009 MCAS Grade 10 Raw Score to Scaled Score Conversions] Table 6. 2009 MCAS Grade 10 Raw Score to Scaled Score Conversions (Continued) [Table 6. 2009 MCAS Grade 10 Raw Score to Scaled Score Conversions (Continued)] Table 7. 2009 MCAS High School STE Raw Score to Scaled Score Conversions [Table 7. 2009 MCAS High School STE Raw Score to Scaled Score Conversions] Table 7. 2009 MCAS High School STE Raw Score to Scaled Score Conversions (Continued) [Table 7. 2009 MCAS High School STE Raw Score to Scaled Score Conversions (Continued)] Appendix B: Calculating Threshold Scores (Cut Scores) A threshold score is the minimum raw score (number of points) that a student must earn on an MCAS test for his or her performance to be classified at a particular performance level. For example, according to the following table, 24 is the minimum raw score that a student must earn on the 2009 MCAS grade 8 Mathematics test for the student’s performance to be classified at the Needs Improvement level. To maintain standards and allow comparisons over time, threshold scores representing those standards may change to reflect differences in item difficulty across years. Therefore, MCAS tests are linked across years through a statistical process called equating. Through the equating process, adjustments in the threshold scores for the 2009 MCAS tests maintain the original MCAS standards. THRESHOLD SCORES, MCAS TESTS OF SPRING 2009 [THRESHOLD SCORES, MCAS TESTS OF SPRING 2009] ^1 The maximum score is the total number of raw score points that a student could earn in the subject area if he or she answered all of the multiple-choice questions correctly (one point per question) and received the highest possible score for each open-response question (four points; two points for grade 3 Mathematics), short-answer questions in Mathematics only (one point), and the ELA Composition in grades 4, 7, and 10 (20 points). ^2 Above Proficient is the highest performance level for grade 3. Appendix C: MCAS-Related References The resources below are available on the Department website, at the specific locations listed. MASSACHUSETTS CURRICULUM FRAMEWORKS • English Language Arts Curriculum Framework • Mathematics Curriculum Framework • Science and Technology/Engineering Curriculum Framework • History and Social Science Curriculum Framework Each framework describes what students should know and be able to do in that subject area at various grade levels. All frameworks are available at www.doe.mass.edu/frameworks/current.html. OTHER RESOURCES Release of Spring 2009 MCAS Test Items contains approximately 50% of the common test questions that were used as the basis for grades 3–8 student results and 100% of the grade 10 ELA and Mathematics and high school Biology and Introductory Physics common test questions that were used as the basis for student results in the 2009 MCAS reports. No high school Chemistry or Technology/Engineering test questions were released in 2009. Available at www.doe.mass.edu/mcas/testitems.html. Requirements for the Participation of Students with Disabilities in MCAS (Spring 2009 Update) outlines the requirements for participation of students with disabilities and describes allowable test accommodations. Available at www.doe.mass.edu/mcas/participation/sped.pdf. Requirements for the Participation of Students with Limited English Proficiency in MCAS and MEPA (Spring 2009 Update) outlines the requirements for participation of LEP students. Available at www.doe.mass.edu/mcas/participation/lep.pdf. 2008 MCAS Technical Report provides technical details about the 2008 MCAS administration and reporting of results. Available at www.mcasservicecenter.com (click “MCAS”). Student Records: Questions, Answers, and Guidelines contains information about the Massachusetts Student Record Regulations. Available at www.doe.mass.edu/lawsregs/advisory/cmr23qanda.html. Page 21 of 22