Family & Community Adult Education Adult & Community Learning Services
Adult and Community Learning Services

Massachusetts' Procedures for Using the BEST Plus Scoring Refresher Toolkit

December 2006

Who Should Read these Procedures:

Directors, and the staff person chosen to facilitate a group Scoring Refresher (explained shortly), and any individuals who will complete the Refresher by themselves must read these procedures. In particular, facilitators and Directors will want to review the Planning Worksheet/Checklist and Needs Assessment in Section V of the Toolkit. Completing the Planning Worksheet and Needs Assessment will inform planning for staff development activities.

Overview

The BEST Plus Scoring Refresher Toolkit outlines a number of ways that it may be used by states across the US using the BEST Plus assessment. Many states have somewhat different assessment requirements, and therefore this flexibility is necessary in the Toolkit. The following gives guidance on how the Toolkit should be used by practitioners in Massachusetts. Please read these procedures before beginning the Refresher or Scoring Activity.

The Requirements

First, thank you to all who have taken part in assessing students this year. While the assessment process has become more routine, we know it continues to take time and careful attention on your part. It is so important that we continue to administer and score the standardized assessments correctly. If we are to maintain accurate scores statewide, then it is vital to make sure all administrators and scorers are following the same test protocols. A significant part of this process is ensuring that all BEST Plus test administrators are scoring strictly according to the BEST Plus rubric and re-familiarizing themselves with the scoring process (known as recalibration) each time before they test.

To ensure scoring accuracy in FY07, all BEST Plus test administrators need to be recertified and receive notification that they successfully accomplished the recertification process. This involves:

1. Taking part in a Scoring Refresher, and

2. Successfully completing a 30-minute Scoring Activity to receive recertification as a BEST Plus test administrator

The Materials

Along with this memo, you should have:

• A 3-ring binder entitled, "BEST Plus Scoring Refresher Toolkit," with two accompanying DVDs or VHS videos (one Toolkit per site that uses the BEST Plus, or if the number of students tested with BEST Plus is small, some sites may share a Toolkit)

• A white booklet entitled, "BEST Plus Scoring Refresher Workbook, Test Administrator's Edition" (one Workbook for each BEST Plus test administrator)

The Toolkit is intended for use by trained BEST Plus test administrators to review and recalibrate their scoring in order to ensure reliable assessment data. It also provides a structured forum for test administrators to share their insights and questions about administering and scoring the BEST Plus.

1. The Scoring Refresher

The Toolkit and Workbook may be used by an individual test administrator or by a large or small group led by a facilitator. If possible, participating in a group is the preferred method, since test administrators will benefit from each other's questions and experience. If a group Scoring Refresher is done, a facilitator is required. If the Refresher is done individually, he/she needs to read the Toolkit and accompanying facilitator's edition of the Workbook prior to the Refresher.

• The Toolkit or DVD/Videos should not be taken home by staff and is the property of the program.

The Refresher Facilitator's Role

Facilitators need to be experienced with using BEST Plus and in working with the adult ESL student population. Section II, page 4 in the Toolkit explains in detail the characteristics of an ideal facilitator:

• Comfortable and confident with the scoring rubric and all three language components tested (Listening Comprehension, Language Complexity, and Communication)

• Able to use the scoring rubric to justify scores used as benchmarks on the video

• Experienced with facilitating discussions and be able to summarize information presented in the Scoring Refresher Video

• Empathetic and able to judge when participants need a break or a question answered

• Available to review all Scoring Refresher materials carefully before facilitating a Refresher session

Facilitators may be

• The staff person who is in charge of assessment at your site
• An on-staff certified BEST Plus trainer
• A staff member who is experienced in facilitating discussions
• A certified BEST Plus trainer hired by the program to facilitate

Facilitators must read the Toolkit and facilitator's edition of the Workbook prior to leading the Scoring Refresher. Both contain necessary information for the facilitator. The Refresher will take approximately 2.5 hours to complete, and all BEST Plus test administrators are required to participate in it.

Things to keep in mind (These are excerpts from the Toolkit):

• It must be made clear to test administrators that although their personal ideas and concepts of language proficiency and assessment can differ from those represented in BEST Plus, these differences must be set aside for the purposes of BEST Plus scoring accuracy. (Section II, page 4).

• The facilitator or program administrator should keep the Toolkit materials in a secure place. (Section II, page 4).

When to use the Toolkit:

Since there has been concern about the relative accuracy of some BEST Plus scores this year, BEST Plus test administrators are strongly encouraged to review the video right away. Doing so will help to increase confidence in test administrators' scoring, and help them score as accurately as possible. Please remember that the impact of any potential decrease in test scores between future and previous BEST Plus testing will be shared by the field as a whole. ACLS will be closely monitoring learning gains and their impact on performance standards.

• Additional information and considerations for when to score are in Section III, page 2-3. A helpful flow chart is on page 3.

Please note: Programs may wish-and are encouraged if possible-to do the Scoring Refresher now, before mid-year testing takes place. Since the required Scoring Activity (# 2, discussed below) must be done within a week after the Refresher, program staff may wish to schedule another Scoring Refresher later to occur within a week of completing the Scoring Activity.

How to use the Toolkit:

• Facilitators must follow the procedures outlined in the Toolkit for conducting the Scoring Refresher.

• Individuals using the Refresher by themselves must follow the Agenda for self-study in Section V-9.

• The Scoring Refresher will generally be about 2.5 hours, but should not be more than 3 hours long. The Refresher can be separated into two sessions, but don't break the Refresher into sessions further apart than over two days.

• Facilitators must use the Needs Assessment, Self Assessment, Refresher Planning Worksheet and Checklist materials in Section V, pages 3-8.

• All test administrators must complete the Self Assessment in Section V, page 3-4 in the Toolkit prior to the refresher. (The facilitator needs to copy and give it to all test administrators.)

• A detailed agenda for a facilitator-led Scoring Refresher session is in Section III, page 3.

• Additional very helpful materials for facilitating the Scoring Refresher are in Sections III and V.

Using the DVD or VHS Video:

• Since you will be using the Toolkit's DVD or VHS video in conjunction with the Workbook, there are a number of times where test administrators will want to pause, back up, or fast forward through the video. You will need the capabilities to do this with either the VHS or DVD, and you may wish to practice using those features before the Refresher if necessary.

Important: The Toolkit instructions state that you may read the sections of the Workbook either now or later while watching the video. There is a great deal of important information that is written in the Workbook, however, that is not gone into as much detail in the video. Printed explanations provide a great deal more information-and often essential information-that is not always stated in the video. Test administrators must read the entire Workbook at the same time as you are going through the video. Please pay particular attention to the "Summary" sections in the Workbook. Stop the video and read the corresponding section in the Workbook before moving on.

2. The Scoring Activity for Recertification:

In FY07, all Best Plus test administrators must be recertified by completing the Scoring Refresher and Scoring Activity. Up to a week after the Scoring Refresher, BEST Plus test administrators (including the facilitator) are required to watch and score responses to a 30-minute Scoring Activity. For more information, see Section IV of the Toolkit. All BEST Plus test administrators need to complete the Scoring Activity by Friday, March 30, 2007. (It may be completed anytime between now and March 30, also).

Test administrators trained after February 1, however, will not have enough time to complete the Scoring Activity before March 30. Those receiving their initial BEST Plus training after February 1, 2007 need to complete the Scoring Refresher and the Scoring Activity within the first two months and after their ten practice administrations following their training. After successfully completing the initial training, the practice administrations, the Refresher and Scoring Activity, they will receive their BEST Plus test administrator certificate, which will be good through April 30, 2007. After that, all test administrators for this Fiscal Year will be on the same schedule for recertification.

Doing the Scoring Activity:

• The Refresher must be completed one week or less before doing the Scoring Activity.

• Use the red scoring video (located in the front of the Toolkit) and the two perforated sheets in Appendix D of the Workbook and complete Scoring Activity 1.

• Test administrators need to watch the Scoring Activity on site at the program and must strictly adhere to proper test procedure protocols (e.g., do not confer with each other during the test)

• Do not stop, pause or rewind the video at any time or allow for discussion of any items.

Prior to taking the Scoring Activity 1, please read and the Scoring Activity Instructions found in Section IV, pages 2-3, and then follow the instructions.

After Completing Scoring Activity One:

Submit the scores by mail or fax to your SABES Regional Center between now and Friday, March 30, 2007. Mailing addresses and fax numbers are on the SABES web site: http://www.sabes.org/staff.htm

Those who do not return the Scoring Activity by this date will not be able to administer the BEST Plus until re-taking the initial BEST Plus training, Refresher and Scoring Activity.

Scoring the Activity

By April 30, your Regional SABES Curriculum and Assessment Coordinator will review your work and inform you of your status. Scores will fall into three categories, based on the scoring system developed by BEST Plus test developers: "Recommended," "Acceptable," or "Unacceptable." An explanation of scoring is found in Section IV, pages 4-5, 19 and 25. (Please note that pages 6-18 have been removed from the Toolkits.)

Staff successfully completing the Scoring Activity with a "Recommended" or "Acceptable" score will receive a letter, a copy of the score activity feedback sheet, and a certificate that is good though April 30, 2008. Staff that receive a borderline "Acceptable" or "Unacceptable" score in any one of the three rubric areas will have an opportunity to receive remediation in order to continue to administer the BEST Plus. More information will follow on the remediation process. This process of recertification will now be in place for the end of April each year.

As previously mentioned, the Scoring Activity must be done within a week after the Refresher. However, program staff may decide to go through the Refresher once soon after receiving it, and then schedule the Scoring Activity some time later before the end of March. This is acceptable so long as another Refresher precedes the Scoring Activity by one week maximum, so please plan accordingly.

Questions?

If you have any questions, please feel free to contact your Regional SABES Curriculum and Assessment Coordinator (you can find their contact information at http://www.sabes.org/staff.htm) or Jane Schwerdtfeger at janes@doe.mass.edu.

Thank you again for your continued hard work.

Jane Schwerdtfeger
Curriculum and Assessment Development Specialist
Adult and Community Learning Services
Massachusetts Department of Elementary and Secondary Education