Adult and Community Learning Services (ACLS)

Measurable Skills Gain (MSG) Performance Methodology with attachments

January 23, 2019

Dear Program Directors,

I am writing to inform you that ACLS has revised the Measurable Skills Gain (MSG) performance methodology. We made the changes for two reasons. First, there was considerable concern among stakeholders, including program directors, that there was too much time — about seven months — between the end of the program year and when we could post MSG results. Second, DESE leadership and staff noted the risk that Massachusetts could fall short of our federal MSG target even if all programs met their individual weighted targets because those targets were based on past performance rather than the federal expectations.

In addition to the summary below, there are four attachments:

(1) a table comparing the revisions to the original MSG

(2) the revised FY18 target calculator

(3) FY18 MSG program performance

(4) the FY19 target calculator

ACLS anticipates posting preliminary targets in January 2019 after fall pre-test data have been entered.

Two important features of the initial methodology will remain:

• Targets based on weighted averages: ACLS understands that students complete the NRS at different rates and we will continue to use weighted averages to determine program targets. ACLS multiplies the number of students who pre-test into each level by each level's expected completion rate. Those values are summed, divided by the total number of students, and then converted in to a percentage that is the program's MSG target.

• Multiple outcomes: The MSG methodology accounts for students that achieve multiple outcomes in a Period of Participation (PoP). Programs will get one credit (i.e., 1.0) for the first MSG outcome earned; they will get partial credit (i.e., 0.1) for each additional outcome earned beyond the initial. If a student in a program achieves an initial outcome and two additional outcomes, then the program's credit for that student is 1.2. For example, a student that advances two EFLs and attains a high school equivalency credential would result in 1.2 MSG toward the program's overall performance.

There are some important differences:

• First, program targets will be derived from the federal targets that the Office of Career, Technical and Adult Education (OCTAE) assigns to states. The previous methodology determined your program targets based on five years of statewide averages. However, even if all programs met those targets, the state might not meet the federal target. ACLS will use the federal targets as the basis for your targets. In addition, the targets will be adjusted upward because of the opportunity for multiple outcomes for individual students. ACLS will use the most recent fiscal year MSG to calculate how much to adjust the twelve targets. OCTAE has raised Massachusetts' target each year since the start of WIOA. As OCTAE raises Massachusetts' targets, ACLS will raise program targets.

• Second, there will be separate ABE and ESOL targets. Programs that provide both services will have two targets.

• Third, MSG will be determined based on outcomes achieved during the fiscal year. The original methodology included high school equivalency attainment and post-secondary enrollment achieved in the six months after a fiscal year ends. That approach meant that ACLS would not be able to post MSG results until late January, nearly seven months after the end of the program year. ACLS will publish MSG results in late July or early August and provide all stakeholders with program performance in a timely manner. As part of the shift in focus from processes to learner outcomes, ACLS is considering incentive funding that will reward programs that exceed their MSG targets. Constituent feedback clearly stated that for incentives to work, ACLS would have to release increases as early in the fiscal year as possible.

• Finally, ACLS will incorporate Periods of Participation when determining a program's MSG. If a student has a break in participation of more than 90 days and then returns, that student will be counted twice. Although extremely unlikely, it is possible for a student to have as many as four PoPs in a fiscal year (in FY18, 183 of 18,457 or 1% of MA students had more than one PoP). LACES provides clear views of PoPs in the student record assessment tab and the dashboard student alerts show which students are eligible for post-testing. See PoP Data Collection in LACES for more information. ACLS is updating the FY19 Assessment Policy Manual and planning webinars that will include the impact of PoPs.

The performance standards page on the ACLS website will be updated shortly to reflect these changes and will have a more detailed version of the FY18 MSG program performance worksheet.

Sincerely,

Wyvonne Stevens-Carter
Acting State Adult Education Director