Family & Community Adult Education Adult & Community Learning Services
Adult and Community Learning Services

Performance Standards Task Force

Introduction

In March 1998, the Massachusetts Department of Elementary and Secondary Education Adult and Community Learning Services (the Department) solicited applications from the field of adult basic education professionals to participate in task forces to seek input for two important documents which would guide policy and services in adult basic education for the next five years: a five year state plan for the delivery of services, and an open and competitive Request for Proposals to fund those services. Both of these initiatives are as a result of the federal Workforce Investment Act, recently enacted to replace the Adult Education Act.

One task force was entitled The Adult Basic Education Program and Funding Criteria Task Force. The second task force, whose report is contained herein, was entitled The Adult Basic Education Performance Standards Task Force.

Selection of Task Force Members

Applications were solicited, in writing, from adult basic education professionals representing a cross section of programs, services and experiences. In selecting members, the Department ensured that the following categories were represented:

• location (i.e., ban, suburban and rural);
• size (i.e., small, medium and large);
• services (i.e., ABE, ESOL, ASE);
• constituencies (i.e., community based organizations, local educational agencies, community colleges, corrections, business, labor unions); and
• service delivery (i.e., classroom based, volunteer tutoring).

Meeting Schedule and Location

Each task force convened ten meetings during the period from November 1998 to April 1999. Meetings were held at centrally located facilities, the majority of meetings being at Assabet Valley Regional High School in Marlboro and the Regional Employment Board in Worcester.

Meeting Protocol

The Department was responsible for disseminating relevant materials and information to task force members prior to each meeting. Some of the materials were prepared by the Department while other materials were reprints of research conducted by other states, universities or practitioners. When appropriate, guest facilitators or presenters were invited.

To underscore the importance and extensive scope of the work, members were urged to prepare in advance since it would be difficult for members to fully participate and contribute to the discussions without having done so. Due to time constraints, not all materials were specifically discussed at the meetings, but they were used as background information for discussion.

Members were also encouraged to discuss the materials with their colleagues in order to seek additional input to bring to the discussions at the meetings.

Most task force decisions were made by consensus. When appropriate, a task force could decide to take a formal vote on an issue. When there was either not enough information or the topic was too broad to make an informed and responsible decision within the allotted ten meeting format of the task force, the group could table the resolution of the discussion for a later date pending further collection of information or research.

Feedback From Non Task Force Members

In order to give the task forces reports the broadest possible audience and exposure, draft reports were mailed for comment to directors of ABE programs funded by the Department, SABES Centers, Regional Employment Boards, Service Delivery Areas, Career Centers, Workforce Investment Act Partner Agencies and other interested organizations and individuals. Draft reports were also mailed to task force members for comments regarding content, clarity and accuracy.

Follow Up

Both the task forces and the Department believed that additional follow up would be necessary to further develop and implement recommendations and make revisions to policy based on gained experience. The Department has made a commitment to continue the work of the task forces, resolve any outstanding issues and translate recommendations into specific guidelines, possibly through a continuation of group work and/or collaboration with the Mass. ABE Directors' Council.

Statement of Purpose

The Task Force reviewed and discussed various Department initiatives and policies currently in force that had been developed to lead to quality services for students. The varied content was grouped into the following overall categories.

1. Opportunities to Learn Standards
   These standards are requirements for funding by the Department and are connected to the ABE rates system (i.e.; instructional intensity, educational counseling, paid preparation time, program and staff development, educational materials, and sufficient resources for administration, recruitment, intake, assessment, translation, evaluation and follow-up). Specific recommendations regarding these standards were within the purview of the Program and Funding Criteria Task Force.

2. Student Participation
   Indicators for student participation include enrollment levels (programs must currently fill 95% of funded slots), instructional hours (programs must currently provide 60% of their proposed instructional hours) and retention (the state average for instructional hours per student is 110 per year). The Task Force debated the merits and disadvantages of maintaining or changing these statistical benchmarks and raised the following issues:

   1. Are these benchmarks (enrollment, instructional hours, average instructional hours) sufficient to measure student participation? Should we be measuring other indicators (i.e. student satisfaction, affective gains results of a program's internal evaluation)?
   2. Are the current benchmarks for student participation reasonable?
   3. Measuring the benchmarks alone is insufficient to create a true picture of the quality of instruction or the quality of program services.
   4. Program factors such as teacher to student ratio, the percentage of learning disabled students at a program, and counseling can affect student participation and need to be included in setting the benchmarks and evaluating a program's performance.
   5. Before changing benchmarks based on currently collected data, more statistical information is needed, and the reliability and integrity of the data must be reasonably ascertained.
   6. Inflexible statistical benchmarks fail to take into account extenuating program circumstances and difficult student populations, discourage programs from experimenting with innovative and perhaps risky initiatives, and generally inhibit program creativity especially with regard to program design and activities.
   7. Programs should not be penalized for "successful termination" or for students who "stop out" temporarily and return after some undetermined period of time.
   8. Benchmarks for student participation should not provide a perverse incentive for programs to "cream" students thereby denying students the opportunity to learn and have access to services.
   9. If program funding will be based even, in part, on numerical benchmarks, the statistical analysis must be conducted by specially trained individuals to ensure data accuracy, reliability, integrity, and meaningful and valid analysis.
   10. Programs need training in analyzing their own data as well as more frequent feedback on submitted statistics in order to make more timely program changes to improve.
   11. The need to open slots to relieve a long waiting list and any resulting attendance policy may adversely affect a program's retention statistics.
   
   
3. Learning Gains
   

   The K 12 Curriculum Frameworks, articulating content standards and effective practices for teaching and learning, are in the process of being adapted for adult basic education. The question of how to effectively and appropriately measure learning gains (i.e., GED, Equipped for the Future, portfolio assessment, informal assessment, standardized tests) was the basis for the Task Force's discussion on setting guiding principles for the development of performance standards (see subsequent section on the Task Force's recommendations for guiding principles). The issues raised included:

   1. How can "small learning gains" best be captured?
   2. What is the incentive for adults to take standardized tests that do not lead to a credential (i.e., the GED)?
   3. What methods offer the most flexibility for measuring varied learning gains, and meet the challenge of viewing student performance in a one-year window when measurable and meaningful progress can actually take much longer? Should learning gains be measured at intermediate levels, completion levels or both?
   4. What policy pressures will there be to "prove" learning gains in the current environment and under the current federal legislation?
   5. How closely does curriculum correspond to standardized test content?
   6. Current standardized tests and assessments (i.e., TABE and CASAS) are limiting and unsatisfactory in what they measure, and do not reflect actual learning gains or what is being taught in the classroom. Should Massachusetts develop its own standardized test based on the curriculum frameworks, given the lirnitations of standardized tests and the resources which would be required to develop one.
   7. How many hours of instruction must a student have at a program before the student's goal achievement can be attributed to participation in the program? How long after a student leaves a program can it be credited with the attainment of that goal (i.e., a student who receives a GED after leaving the program)?
   
   

4. Professional Standards
   Professional standards for teacher preparation (i.e., certification) and ongoing staff development will be articulated to help further professionalize the field of adult education. A separate task force on teacher certification will be convened to recommend a voluntary certification process and criteria in response to recently passed state legislation that mandated establishing a voluntary certification system.

As a result of the above discussions, the Task Force recommended the following statement of purpose to guide the development of the performance standards for ABE programs:

Roster of Task Force Members

Task Force Recommendation #1: Purpose Statement

The primary purpose of establishing performance standards is to improve program performance to ensure that students receive the very best services that they need and deserve.

Guiding Principles

After considerable discussion, the Task Force and the Department mutually agreed that there was insufficient time remaining in the scheduled meetings and insufficient data available from the Department's database system for the Task Force to establish new quantitative standards for opportunities to learn, student participation, learning gains and professional standards. Therefore, the Task Force was re focused to identify what types of standards are most appropriate and what principles should guide the Department's future development of quantitative benchmarks and standards as more data becomes available.

Contested Ground: Performance Accountability in Adult Basic Education (NCSALL Report #1) by Juliet Merrifield of the Center for Literacy Studies at the University of Tennessee was a research brief used by the Task Force as the basis for much of the discussion of guiding principles. As a result, the Task Force's recommendations are being presented in the following categories based on conceptual structures outlined by Ms. Merrifield in her research brief:

1. Agreement on performance
2. Build capacity to perform and be accountable
3. Mutual accountability
4. Create new tools to measure performance

Task Force Recommendation #2: Guiding Principles

The Task Force recommends the following principles to guide the Department in the development of program performance standards.

1. Agreement on performance
   1. Performance standards must reflect high expectations for students and programs.
   2. Performance standards must be responsive to students and other stakeholders.
   3. Performance standards must be clear, but sufficiently broad and flexible tosupport meeting the diverse needs of students and communities.
   4. Performance standards must include how we measure program "environment" or"design" and its relationship to learner outcomes.
   5. Program performance must be reviewed in a contextual and holistic mannerthat is responsive to student and program diversity.
   6. Program performance must be analyzed across multiple standards since any one indicator is inadequate to accurately and appropriately capture performance.
   7. All stakeholders must be able to "own" performance data and receive special training in data analysis to receive the full value of honest and reliable data entry, and reporting.
   
   
2. Build capacity to Perform and be accountable
   1. Accountability is an ongoing process that should lead to building capacity. Accountability expectations should be commensurate with capacity and resources.
   2. The Department must commit to and invest in building the capacity of programs to meet high standards, including providing adequate and appropriate technical assistance and funding.
   3. If a program's performance is unsatisfactory, particularly in the area of student performance, and the Department is considering placing a program on some form of conditional funding or probation, there should minimally be a preliminary site visit, a team based assessment of local program design and service delivery issues, peer-to-peer assistance to help determine remedial action, and technical assistance and support to develop and implement a plan for corrective action. The purpose of probation is to strengthen a program, not to take away funding or be punitive.
   
   
3. Develop mutual accountability
   1. Accountability is a mutually shared responsibility among all stakeholders.
   2. Mutual accountability requires that there is ongoing, timely and effective feedback between the Department and programs to promote continuous improvement and ensure early intervention, if necessary.
   
   
4. Create tools to measure performance
   1. In creating tools to measure performance, programs must honor and respect student confidentiality.
   2. Assessment tools and practices must be minimally intrusive to students' lives and minimize barriers to participation for all students, especially undocumented students.
   3. Assessment tools and practices must address the demands and needs of different stakeholders.
   4. In order for there to be consistency in what is measured and verified across programs, there must be an alignment among learning gains (curriculum frameworks), content, practices, assessment and what is being taught in programs.
   5. The Department and programs must build an accountability system on the goals that students set for themselves, their aspirations, and characteristics that affect participation (i.e., single parents, working adults).
   6. Students must have the ability to add, change and/or modify their desired goals and outcomes over time.
   7. Assessment tools and practices must measure performance across several performance areas, be more than information based content and include other non content based abilities (i.e., problem solving and learning how to learn).
   8. Assessment tools and practices must be able to measure and verify progress toward performance standards, including student goal achievement.
   9. Native language literacy issues must be addressed in both curriculum and assessment.
   10. Some areas that should be measured include student participation (retention, student enrollment, hours of instruction).
   11. Students should receive periodic certificates of achievement to honor their accomplishments.
   
   

Compelling Rationale

The Task Force believed that the following compelling rationale was a critically important foundation for developing guiding principles. The purpose of the compelling rationale is to balance diverse needs, quality, flexibility and fairness while ensuring the integrity of performance and professional standards.

Task Force Recommendation #3: Compelling Rationale

The Task Force recommends the following compelling rationale to guide the Department is the development of program performance standards:

The Task Force believes that the following principles are critical to the ability of the Department to fairly and effectively evaluate the performance of any of its funded programs:

1. In setting performance and professional standards of any kind, it is impossible for either the Department or the Task Force to be able to foresee any and all possible extenuating or mitigating circumstances that could adversely affect a program's performance or its ability to meet recommended or established guidelines and/or standards.

2. Performance and professional standards of any kind must be fair and reasonable, and promote quality services while granting programs the flexibility to be innovative, experiment, task risks, embrace challenges and address unforeseen circumstances and/or obstacles. Future best demonstrated practices can naturally evolve from granting programs the latitude and flexibility to be creative and solve problems.

3. Programs have a responsibility at all times to strive to meet the needs of the community and their students, to provide quality services, and to meet performance and professional standards.

4. Assessing program performance requires more than solely reviewing and relying on selected numerical statistics or program date which, viewed out of context, will not always accurately reflect a program's true performance and/or relevant mitigating circumstances which would warrant important consideration.

Therefore, the Task Force recommends that if, at any time and for any reason, any aspect of a program, including but not limited to its proposed design, practices, operations, outcomes, statistics, data or performance, deviate from established or recommended performance and professional standards, the program should be allowed to submit to the Department a written compelling rationale to explain and/or justify the area(s) of concern or deviation. Further, in keeping with the principles stated above and to fairly and effectively implement this recommendation, the Task Force recommends that:

1. Grant readers and Department staff should receive quality training in diverse program designs andpractices, best demonstrated practices, current research and theory, and how to effectively and fairly evaluate a program's compelling rationale.

2. A compelling rationale can be submitted as part of a program's competitive grant submission,re funding package, or documents submitted relevant to probation or any other disciplinary process which could result in a decision to withhold, cut or terminate funding.

3. If a compelling rationale is submitted as part of a program's competitive grant submission, as part of a probation or other disciplinary process or as part of a decision to withhold or terminate funding, the process for submitting, approving or denying a compelling rationale must be clearly articulated in writing by the Department to programs. Programs should receive written guideline information and technical assistance on all issues relevant to the submission and approval of a compelling rationale, including but not limited to:

   1. the established and/or recommended performance and professional standards programs should meet;
   2. clear and concise criteria that the Department will use in evaluating a compelling rationale;
   3. circumstances which warrant that a compelling rationale should or can be submitted during a particular evaluative process (i.e., during a grant process, program evaluation, probation process, at any time);
   4. the process used when considering if a compelling rationale can be approved and who does so;
   5. if the compelling rationale is not submitted as part of a competitive grant process which will have a published grant ward notification date, how soon the program will receive a written decision on its submitted compelling rationale;
   6. if the compelling rationale is not approved, the specific reasons why;
   7. if disapproval of the compelling rationale is a significant factor in either a program's being placed on probation or losing funding and a grievance or appeal procedure does not currently exist, one must be established;
   8. diverse program designs;
   9. best demonstrated practices;
   10. access to current relevant research and theory;
   11. how to effectively write a compelling rationale.
   
   

4. The above guidelines should be developed with input from significant interested parties, including but not limited to the Task Force, the Mass. ABE Directors' Council, the Mass. Coalition for Adult Education and Department funded programs.

5. As a means of submitting additional program information, a compelling rationale can be submitted by a program at any time for inclusion in its Department program file whether or not the program currently deviates form the performance and professional standards or formally submits the compelling rationale to the Department for approval.

6. A compelling rationale is not a substitute for operating and autonomously managing a professional program that provides high quality services, nor is it intended to encourage programs to circumvent the performance and professional standards. A compelling rationale is intended to encourage the exercise of sound judgment that is neither random nor arbitrary.