Family & Community Adult Education Adult & Community Learning Services
Adult and Community Learning Services

Draft Review of the Massachusetts ABE Teachers' License Model

Thank you for your thorough review of the ABE Teachers' License Model. As we have discussed, we acknowledge that the development of the Massachusetts model will be on-going and once the Department receives more sample portfolios and has trained more review panel members, we will then be able to adjust the model. Below is an outline of some of the areas on which we are currently working.

1. The Department is developing a preservice performance assessment for preparation programs in order to ensure that sufficient levels of proficiency in the standards demonstrated within an approved program are met.

2. The Department will draft definitions for the terms used to give guidance on writing explanatory statements (e.g., convincing, genuine grasp).

3. National Evaluation Systems (NES) has been hired to develop the subject matter test for the ABE teacher's license. NES and Technical Advisory Committee will document the methods used to set passing standards on the test and will document the continuing technical quality of the tests. In terms of the validity of the ComLit test and other, previously developed subject matter tests, enclosed is a copy of the report. In the report you'll note the Committee's opinions that a licensure test, "…is not intended to be a measure of, or a predictor of, performance or effectiveness. It is simply a measure of the knowledge and skills in the domain being assessed…the Massachusetts Tests for Educator Licensure are not designed as either diagnostic or employment tests. Using licensure tests for either of these two purposes is to misuse these tests…Thus, the professional standards do not consider it necessary, or even appropriate to validate scores on licensure tests against an external criterion measure of success on the job." (p. 2)

4. The Department intends to provide guidance on the teaching and learning content and develop training guidance for demonstration observers.

5. As with other sections of the licensure model, the training process will need to be updated, which is also dependent upon receiving more sample portfolios. However, some areas on which we can provide immediate clarification are listed below.

   1. How panel members are chosen: Panelists may be eligible to participate if they meet the following criteria:

      1. They have completed a nomination form (see enclosed ABE Panel Review Nomination Form);
      2. They have been nominated by a current or previous supervisor;
      3. They have been nominated by at least two ABE teachers, (Note: the supervisor does not count toward this requirement); and
      4. They must meet one of the following sets of qualifications:
         1. Five years of ABE teaching experience; or
         2. Hold the ABE Teacher's License at the professional level; or
         3. Three years of ABE teaching experience and two years of ABE supervisory experience; or
         4. Three years of teacher supervisory experience within a teacher preparation program and beginning 2006, have a minimum of three years of teacher preparation supervisory experience and two years of ABE teaching experience. In reviewing the nomination forms, the Department is seeking panel members who meet or exceed the minimum criteria and are recognized as model teachers and leaders in the field by their peers and their supervisors. The Department gives preference to teachers who have had a range of ABE teaching experiences in a range of ABE contexts and will eventually give preference to teachers who have obtained the professional level of the ABE license. In addition, the Department will make every effort to select panelists from the available pool of nominees who represent the range of ABE contexts and the geographic diversity within the Commonwealth.

   2. Rationale for requirements for members, how long they serve, how they are retrained: Panelists will serve for a minimum of one year. The need for retraining will be determined by periodic calibration (see items h and i below) and whether significant changes are made in iterations of the guidelines.

   3. Provide an orientation to the licensure process: The trainees are provided with an orientation to the licensure process as outlined on page 11 of the training manual. Additionally, they are provided with a reference binder that includes the Regulations, Guidelines, scoring rubrics and other materials they may need to consult during portfolio evaluations.

   4. Why expert judgments are preferred and "correct": Expert judgments are made by individuals who have a minimum of 10 years of ABE teaching experience and 10 years of experience supervising ABE teachers in addition to significant experience designing professional development for ABE teachers. In the future, expert reviewers will also be required to be former review panel members, which means they will have been trained and periodically normed (calibrated), and have obtained the professional level of the ABE license. Eventually, we will add the additional requirement of years of participation on the review panel.

   5. Representativeness of examples used to train scorers and method/rationale for choosing these particular training samples: Six candidates from the existing applicant pool were recruited early last summer to participate in piloting the guidelines. These individuals were chosen based on the diversity of their abilities, ABE teaching contexts, geographical locations (it is the opinion of the field that different regions of the state require different teaching skills and materials), quality of teaching (using workbooks solely v. designing specialized lessons), and ability to articulate experience (e.g., both well and not well). The diversity and samplings of abilities was to provide trainees with a range in quality of portfolios.

   6. Focus less on what panel members want to get out of the training and what their feelings about the performances: We will delete this from the training.

   7. Method for ensuring that every panel member in every session over time is trained to a common operational definition of the terms of evaluation: Since our training and norming process is in the early stages of development, we are still refining how this will work.

   8. Method for checking on the application of the training in live scoring, how calibration and quality assurance will be established after the training is over, what process is used to decide when panelists are ready to begin scoring "live" cases or when deemed unacceptable: During the training, trainers calibrate panelists to the experts and each other. If a panelist is not scoring within an acceptable range, the trainers will attempt to determine the reason(s) for discrepancies through group discussion and, if warranted, one-on-one discussion. If warranted, the panelists will be asked not to complete the training at this time and to participate in a retraining at a later date. If, after retraining, the panelist is still not scoring within the accepted range, the panelist may be denied participation in the review panel. Panelists are determined to be ready to score "live" cases if they have been calibrated within an acceptable range by the end of the training. As a double check, at the end of the training review panel members each reviewed two complete portfolios for licensure.

   9. How panelists are calibrated over time: The Department will periodically circulate one or two licensure portfolio(s) among teams so that each review panelist has reviewed the same portfolio(s) to determine whether panelists are still scoring within an acceptable range. If panelists are not scoring within an acceptable range, the same procedures as described above will be employed. The frequency of the calibration will be determined by the volume of portfolio reviews (the more reviews, the more frequent the calibration); however, our initial thinking is that this calibration should happen at least once per year.

   10. Define approaching, meeting, not meeting, exceeding; and how trainees make the change between the 4 point and the 2 point scoring system: The four point scoring system is used during the training to help promote discussion of candidates' responses to the prompts in the rubrics. A score of "Not Meeting or "Meeting "is awarded when there is, respectively, either no clear evidence of having addressed the content of the evaluator question or of adequately addressing the content of the evaluator question. A score of "Approaching" is awarded when responses do not fully address the content of the evaluator question. A score of "Exceeding" is awarded when candidates' response has not only adequately addressed the content of the evaluator question but also shows exceptional knowledge, even to the point that the candidate should be considered a master teacher or is, or should be, publishing his/her work. In the two point scoring system, "Approaching" and "Exceeding" are not options; candidates either "Meets" or "Not Meets" the evaluator question. Please be advised that we have since modified the scoring rubrics to accommodate more notations by evaluators and rationales for their scores, which will allow for more thoughtful discussion during the review panel meetings.

   11. Process for reaching expert judgments and why the mean response was chosen: If at least two of the three evaluations on an Evaluators' Question were in agreement, then that evaluation became the "expert standard". If the three evaluations differed, then the middle evaluation became the "expert standard" (p. 103 from the Training Guide).

6. We also need to articulate how trainees are taught to meet the standards, the differences between what they saw and what they made of what they saw and what the experts saw and what they made of it, how they come to consensus on the performance of the standards, how panelists are taught to analyze the expert commentary and compare their own judgments to it, how the particular elements of evidence cited by the experts are used to point out elements that must always be examined in every evaluation session, and how differences from the expert judgment are coached to understand and adopt the expert standard.

7. Once we receive more sample portfolio we will be able to better articulate and provide examples to address the following areas:

   1. What constitutes the evaluative standard;
   2. How the standard will be applied to their portfolio materials, including common errors and omissions qualified people might make and should avoid;
   3. Define "how much" of this "what" a successful candidate must achieve;
   4. Performance to the standards and rubrics;
   5. Illustrations of the breadth of relevant contexts;
   6. What the samples illustrate and why one example is acceptable and another not;
   7. How candidates should combine the explanatory statements and documentary evidence, what is important to evaluators, what is irrelevant;
   8. What is good v. weak evidence and what additional evidence is convincing and credible;
   9. What evaluators look for in operational terms for both documentation and the teaching demonstration;
   10. How to choose standard-by-standard approach v. case-study approach and the value of each approach;
   11. Differences in evidence that each method can produce and in the evaluation methods for each; and
   12. Guidance on what good or good enough practice looks like, and what doesn't meet the standards.