Family & Community Adult Education Adult & Community Learning Services
Adult and Community Learning Services

Assessment Policies and Procedures Update-Q&A # 1

August 9, 2002

We have received lots of questions in response to the new Assessment Policies and Procedures Manual. We hope that this update will help clarify any information that isn't spelled out sufficiently in the Manual. (If you don't already have a copy of the Assessment Policies and Procedures Manual, get one! You can find it on www.doe.mass.edu/acls/assessment) These updates will also be listed on the Department of Elementary and Secondary Education's ACLS website at the above address, and will go out every so often on the SABES listserv as well. If you have questions in the meantime, please forward them to Jane Schwerdtfeger at janes@doe.mass.edu

QUESTIONS

What assessment tool are we to use for class placement? At our program, we currently use the initial assessment used for reporting for class placement. Can you clarify the policies for when the initial (pre-test) for reporting purposes should be given?

Here are some suggested guidelines for Intake and Class Placement. Since each program has different needs and structures in place, following are four different scenarios for how intake, class placement and assessment might occur, and the advantages of each.

1. Create a welcoming, comfortable environment for the student.
2. Explain the structure and expectations of the program. If this is an ESOL program, whenever possible, include a staff member who can answer questions in the native language for students.
3. Provide opportunities to talk to students about any concerns they have about the program and any barriers to successful attendance. Offer support in overcoming those barriers.
4. When making decisions on class placement, you need to consider the range of classes that you offer. This will vary from program to program. For example, some programs have separate literacy classes for ESOL students with low literacy, while others do not make that distinction.
5. Informal assessments are fine for class placement. To get a fuller picture of the student, assess in several skill areas. For example, for ESOL students this could include an oral interview, a writing sample, and some graded reading selections. For ABE students, reading, writing, and math could be covered. Once students are in class, be prepared to change their class levels based on teacher recommendation as appropriate.
6. Rather than giving a long battery of tests, which could alienate a new student, give short exercises and discuss the results with the student after each part. For example, give a short writing sample and analyze it, then give a reading selection and go over the answers. With a graded series, you can stop whenever the person is having difficulty.

Advantages of using an informal process for placement and following up with the initial tests for educational gain 2-4 weeks later:

1. The student is more comfortable in the environment and more likely to persist in school.
2. The student does not have to take a long battery of tests before starting class.
3. You can go over the answers to the assessments and give the student some diagnostic information.
4. The initial tests for educational gain will be more accurate if they are not given on the first encounter. Test anxiety will be decreased.

Sample sequences for intake, placement, and educational gain assessment:

Scenario One

1. Intake/orientation
2. Informal assessment for placement
3. Goal-setting
4. Class placement
5. Initial test for educational gain (approximately 2-4 weeks after intake)

Advantages of Scenario One:

• More diagnostic information for students and teachers
• Less threatening environment before class placement

Scenario Two

1. Intake/orientation
2. Informal assessment for placement
3. Class placement
4. Goal setting as part of a classroom activity
5. Initial test for educational gain (approximately 2-4 weeks after intake)

Advantages of Scenario Two:

• Goal setting as a classroom activity will involve the teacher directly in understanding the students' needs and interests and to inform curriculum accordingly
• Goal setting in the classroom will allow students to think of additional possibilities as they hear other students

BUT: Initial goal-setting may best be done initially in a confidential environment

Scenario Three

1. Intake
2. Informal assessment for placement
3. Waiting list
4. Orientation
5. Goal setting
6. Initial test for educational gain (could be many weeks after intake because student is on a waiting list but shortly before class placement)
7. Class placement

Advantages of Scenario Three:

• Allows program to place students on a waiting list according to their class level, so they can be called when space opens up in that level.

All three scenarios satisfy ESE requirements:

1. The initial test for educational gain should not be given on intake.
2. The initial test for educational gain should be given shortly before or after a student starts class.
3. The initial test for educational gain should be given only when a student is comfortable in the environment of the learning center, i.e. after an intake and substantial orientation process or after class placement.
4. The initial test for educational gain should be given approximately two weeks after intake and approximately 4 weeks after class placement.

Please note:We understand that programs might already have a process in place that uses the educational gain pre-test assessment as a class placement test. In this case, we will waive this requirement through October 31, 2002, but thereafter programs will need to implement procedures for testing within approximately two to four weeks of intake.

Additional information:

You can use the TABE Locator or the BEST Literacy test as an element of your placement assessment. You may also use any home-grown or center-developed placement tests. Your local SABES Center is working on putting together a list of placement tools, and will have copies to look at. Another resource for placement tools for your program are the past volumes of Adventures in Assessment. They often will evaluate different assessments in each issue. The SABES web site and Regional SABES sites are places to see copies of Adventures in Assessment.

As noted above in the scenarios, we recommend the initial test for educational gain be given approximately two to four weeks after intake, once the student is in a comfortable environment and oriented to the program. Since every program's class schedules vary, it is hard to prescribe a more exact time frame. Programs will want the student to have gone through an intake, goal-setting and orientation process before the pre-test is administered to show educational gain. They don't necessarily have to have been in class for 2-4 weeks, so long as they aren't taking the test without some orientation and context to the program's learning environment. For intensive programs, four weeks may be way too long to have a student in class prior to the pre-test.

This year, we will be looking at various goal-setting procedures and orientations that programs currently provide to guide us in making recommendations for the future. We have received programs' assessment plans and realize that programs have many different ways of approaching intake and placement that work for them. We know you have procedures already in place, and we will work with programs to develop models that will work for a variety of programs in the future.

How many times do two trained staff need to score tests together to achieve inter-rater reliability?

In the Assessment Policies and Procedures Manual (p. 21), we mentioned that in order to have reliable testing scores for the BEST oral interview section statewide must achieve inter-rater reliability (uniformity in our scoring across practitioners throughout the state). When working together for the first time, any group of scorers needs some practice and requires some refresher work at the beginning, mid-point and end-of-year assessments to maintain their uniformity of scoring. This calibration will help ensure that test scores will have the same relative meaning for all staff who are scoring assessments across the state. Please read the section in the Manual on page 21 that discusses this fully.

To achieve inter-rater reliability among scorers, we discussed in the Manual whether two program staff should administer 20 tests together at first, or 10, as the Director's Council requested. We checked with the Center for Applied Linguistics (who put out the test), and there was some ambiguity in their answer; the test developers suggested 20 administrations for the initial report of educational gain, while the trainers suggested 10. We were inclined to go with the developer's recommendation, but the standardized data is not extensive and not up to date. Therefore, for the first year, we will split the difference, and lessen the initial administration to 15 tests which two trained and certified staff need to score together. The mid-point and end-of-year scoring will remain at 10 each for two staff working together.

Who can administer the tests? Can it be done by a staff person who is a clerk at a program?

This answer depends on which test is being administered. The REEP test is designed to be (and should be) administered by the students' teacher, though two other trained and certified staff persons will score each test. In the case of the BEST, it is unlikely that staff can administer this test without prior ESOL teaching experience and training. It may be most applicable to a staff person administering the TABE test.

Clerks may administer the TABE if they go through the training and receive a certificate of proficiency. Staff scoring the REEP need to be staff who have experience in teaching writing skills and will be familiar with using a rubric like the REEP's. Those administering the BEST Oral Interview test must have training in ESOL and a background in teaching ESOL. In the case of REEP and BEST, staff administering and scoring the tests need to attend a training and receive a certificate of proficiency also.

Can I discuss the results of the TABE, BEST and REEP tests used to report educational gain with my students?

You can discuss in general how the student did, and what level the test reflects that the student is at. You must not discuss the individual answers to questions with students, as then they will know the answers and may pass those on to other students. It will invalidate the test results overall to discuss the answers with some or all students.

If a program uses the TABE Locator for placement, it is problematic since the Locator doesn't generate a GLE. Any suggestions?

The Locator score is based on the number of correct test items. When you determine which test to administer to the student (either E, M, D, or A) based on their score using the Norms Book, you will see the GE (Grade Equivalent) number also listed in the norms chart. Please refer to pp 15-16 in the Assessment Policies and Procedures manual for more information. (This initial judgement can also be fine-tuned if your program tracks the number of correct responses students get on the Locator and how they do in the different class levels.)

If students who pass the GED don't come back for the post-test, how can we get credit and report on their educational gain?

If you look at Table 4 in the federal reports, then you will see that students who pass the GED test are counted as completing the level they were placed in. (Please refer to the footnote on Table 4 for more information). This really applies to low and high ASE levels (GLE 9-12). Beginning July 1, 2002, programs will be collecting students' Social Security Numbers so that data matching can be done. If a student does pass the GED but doesn't come back to the program to report it, the program will still receive the information through data-matching (and be able to report the educational gain).

If students' return rate is poor and students don't return after their initial test to take the post-test, I'm concerned the program will be penalized for a poor "track record" in reporting educational gain. Can programs test more often than every four months? Can it be reduced to a certain number of hours instead?

Programs should keep to the policy of testing every four months. If a program's situation requires that a student be tested sooner than four months, then you may re-test if a student has had 75 hours of instruction. Programs should not test any sooner, since it is likely that there may not be any educational gain to capture through the assessment. This adjustment may help those with Corrections and Homeless student populations, where it is difficult to post-test, and the timing of when a student leaves class is often not under a program's control. ACLS knows that this is a problem, and programs will not be penalized because of this phenomenon, since they all will be confronted with the same challenge.

Regarding Placement: if a program has ABE students who are at level 0-1.9 and has been using their pre-approved assessment from last year (in this care the DAR), what happens if the student moves out of the 0-1.9 level? Can we then start giving the student the TABE during the same year?

Programs need to use one assessment per student during the fiscal year-the initial, mid-year and end-of-year assessments used for reporting purposes must use the same test. If a student starts with the DAR at the beginning of the year, the program must continue testing with the DAR the rest of the year. During the next fiscal year, the student may be tested using the TABE.

I noticed in the TABE Complete Battery test for Applied Math that the instructions say that students may use calculators if they wish. Won't this invalidate the test results, and be unfair to students who don't use a calculator?

If you look at the problems in the Applied Math section, most are word problems, or ask the student to apply knowledge of math that would not be aided by using a calculator. (For example, in the TABE Computation section of the Complete Battery, which isn't required for programs to use, use of a calculator is prohibited). We also double-checked with TABE staff, and they agreed that students may use a calculator, or not, as they wish in the Applied Math section without invalidating the test results.

I noticed in the TABE Locator that the Math test items are all computation questions. Since the Math test section we'd administer from the Complete Battery is on Applied Math, will the Locator give an accurate level for placement?

Math computation is the best skill area for determining placement in a math class. If one has the basics, then one can work on applied math problems, and understand math conceptually. Placing students according to their foundation, computation skills, is appropriate, since most students will have to work on applied math skills in class. While not everyone in class will have the same basic skills, they'll have a common denominator of general basic skills.

By its nature, a placement test gives you a broad picture of a student's skills. It is not meant to capture the full range of skills a person has. In taking a test, students are more comfortable with computation questions rather than with word problems. In developing a standardized test, the least intimidating test is advantageous.

Addition to the Assessment Policy and Procedures Manual:

In the Assessment Manual, section 1.8.1 (page 12) the list of exceptions should include the following: ADP/EDP students with NO academic instruction do not need to be tested with the TABE.