Students & Families Adult Education Adult & Community Learning Services
Adult and Community Learning Services

Assessment Update, March 2009

Use of the REEP Writing Assessment in FY10

We have received questions about whether ABE programs will be using the REEP Writing Assessment beginning July 1, FY10. To clarify, the REEP has not been approved for use after FY10. The US Department of Education, however, has given Massachusetts a grace period-FY10-in order to put into place an alternative standardized ESOL writing assessment. Please continue to secure REEP certification and recertification for your teachers in FY09 and FY10 until you receive final word from ACLS that the REEP Writing Assessment is discontinued.

REEP Writing Assessment Administrators Annual Recertification Due April 17

This is a reminder that all REEP Writing Assessment Test Administrators must recertify by April 17. All information about the REEP recertification process is listed in last month's Assessment Update in the ACLS Monthly Mailing: http://www.doe.mass.edu/acls/mailings/2009/0213.html

BEST Plus Administrator Annual Recertification Due March 31

This is a reminder that all BEST Plus test administrators must recertify by March 31. All information about the recertification process is listed in last month's Assessment Update in the Monthly Mailing: http://www.doe.mass.edu/acls/mailings/2009/0213.html.

March MAPT Score Report Update to the ABE field

This update gives information about work being done to enable the MAPT for Math to provide more information to ABE learners and teachers. Eventually the same will be done with the MAPT for Reading. This update was written by Drey Martone, Center for Educational Assessment, UMASS Amherst.

Six focus groups were held in Fall 2008, during which participants were asked to provide feedback on draft MAPT for Math score reports. There were 66 participants across the six focus group locations (detailed in Table 1). Participant experience ranged from full-time reading and math teachers, part-time teachers, teachers from the County Houses of Correction, counselors, SMARTT data support people, data analysts, System for Adult Basic Education Support (SABES) personnel, assessment administrators, program directors, and ACLS administration.

In the focus groups, participants were asked to review and comment on draft score reports for the MAPT. These draft score reports were created to display data at the class and individual learner levels, and reported performance data for content areas, benchmarks (aligning with the ABE Math Framework), and cognitive levels. Participants were asked to discuss how the reports could be understood and utilized. Feedback was also sought about what additional information was desired in these reports.

There were 758 comments coded across all six focus groups with an average of 126 comments per focus group. The comments were analyzed and five main themes emerged (Table 2). First, we analyzed specific points of confusion the participants had with the sample score reports. Second, we explored the interpretations and concerns participants expressed when using the sample score reports. Third, we analyzed suggestions participants had to modify the sample score reports. Fourth, we detailed comments and concerns the participants had with accessing the score reports. And fifth, we explored the theme of how participants currently use and would like to use MAPT. The results for each theme are discussed in more detail in the full report.

The final activity in the focus groups, involved having participants consider what the final score page displayed to students after completion of the MAPT tests in OWL should look like. At present, students are given their MAPT score, test level, and ABE Curriculum Frameworks level. In this activity, five different options for the final score page were presented to the participants. These options were:

• maintaining the current page,
• including the table without reference to the Frameworks,
• including the table without reference to the Frameworks or the MAPT level,
• presenting the table with the score ranges only, presenting the score only with no table,
• thanking the student but not showing a score.

The results of the voting are summarized in Table 3. The majority of the participants voted to maintain the layout of the current score page as it is. However, the votes were fairly evenly distributed across all five options.

Based on these results no changes have been made to the current MAPT score report the students see in OWL.

Summary

The focus group participants appreciated the effort ACLS made to produce the sample score reports. The proposed score reports addressed many of the concerns the participants voiced regarding the lack of diagnostic information for the MAPT. However, it was clear from this study that additional steps will be required to truly meet the needs of the ABE practitioners. It is important that next steps regarding the MAPT score reports account for the range of opinions and concerns voiced by the participants while also addressing the issue of the validity and reliability of the reported information.

From this study we saw there is not a clear consensus across ABE practitioners about what types of information from the MAPT would be helpful. Some participants thought the cognitive level information was not needed while others thought it was critical to understanding the possible learning issues of the students. Some participants thought the benchmark level information was too detailed while others thought it was an important way to relate information to what is happening in the classroom. Some participants wanted to only look at the individual level reports while others thought the class reports or a region/statewide analysis was the necessary level to view the results.

The diversity of opinions was clearly captured in the vote regarding the format of the final score report table in OWL. The votes were practically evenly split across the five options. Across each of these different issues the views of the participants clearly reflect the diversity within the ABE field.

Given this diversity, it will be important moving forward to design a flexible reporting tool to allow practitioners to see their data in a way that is meaningful for them, and provides some flexibility to meet their various needs. However, doing so will be difficult because of the adaptive nature of the MAPT. Our challenge is to give teachers flexibility in accessing data related to their students' performance on the MAPT in a way that (a) gives the teachers and students the information they would find useful, and (b) acknowledges the fact that students took different sets of items that differed in difficulty.

Next Steps

We are working with the data team at ACLS to design reports that address the concerns discussed through the focus group meetings. We want the reports to present reliable and valid instructionally useful information. We will be following up with a small group of ABE practitioners from the focus group to discuss the proposed reports in more detail. Then we plan to test the functionality of the reports with the sites this small group of ABE practitioners work with. We will continue to provide updates to the field as the process progresses. We plan to have some type of more detailed score report available by September 1st for the MAPT for Math. If you have any questions or would like more information please don't hesitate to contact Drey Martone at dreymartone@educ.umass.edu.

Questions? Please contact Jane Schwerdtfeger at janes@doe.mass.edu