Shortly after completing my recent post about Credential As You Go, I read an article about Education Quality Outcomes Standards or EQOS. EQOS is a non-profit organization that was recently acquired by another non-profit organization, Jobs for the Future.
EQOS has established five key metrics of quality assurance that are:
Their initiative began when the U.S. Department of Education launched the EQUIP program that allowed non-traditional providers of education to partner with accredited colleges and universities and access federal student aid for their program’s students. The five key metrics of quality assurance are to be considered as short- and medium-term metrics. EQOS recommends that all entities providing credentials should group their students in 12-month cohorts for reporting purposes.
Learning, according to EQOS, is “a far more complex issue than any other suggested measure in these standards.” I agree. EQOS notes that some elements that indicate learning will vary more depending on the program than measures such as completion. Because of the variance, EQOS offers general guiding principles for measuring learning rather than the exact measure. These principles are for each institution to provide:
- Clearly articulated and transparent learning outcomes and goals with statements of what students can do with the knowledge and skills.
- Clearly documented learning outcomes with pre- and post-program assessments that demonstrate gains in learning and post-program mastery
- Independent, objective, and externally validated assessments that someone with no financial relationship to the provider reviewed to verify the claims of achievement.
- Assessor tests for reliability to offset the fact that human graders will inevitably have inconsistent marks.
- Ensure learners are who they say they are and the work is original to include robust verification of a student’s identity upfront and periodically during and at the end of a program.
Completion rates provide a useful measure of success but should not be the sole measure of success. EQOS recognizes the issues related to part-time students, transfers in, transfers out, and has provided suggestions to reduce some of those recognition problems. The following definitions are provided to assist providers with their reporting.
- Measure Definition – The percentage of learners who entered a given program in the same starting cycle and graduated from that program within the stated, anticipated time to completion.
- Completer Definition – Institutions should only count students if they graduated from their programs. There should be different standards for full-time and part-time students. The full-time student anticipated time to completion should be based on the school’s expectations. The part-time student should be set based on the flexibility of students to attend part-time and a reasonable target for their completion.
Completion measurement period and update cycle standards expect that providers will update their completion rates once each year. To encourage number smoothing, these standards would combine up to three years’ worth of cohort completion rates.
Exclusions from the completion count are worrisome if there are too many. It’s also a potential sign of manipulation. Allowable exclusions include death, permanent disability, and those employed before completion. Circumstances that justify moving a student to a subsequent cohort include military deployment, pregnancy/parental leave, religious mission, and disability. Unacceptable exclusions are students not looking to finish.
Transfer students are one of the biggest challenges to accurate completion rates in traditional higher education. EQOS provides several methods for recognizing transfer students in a cohort as well as removing them from a cohort. I happen to believe that most of these are applicable to higher education degrees and not to short term certificates but am glad for the clarity.
Placement rates provide a direct measurement as to whether a program was successful in aiding learners with their overall goals, whether it is a job after completing the program or admission to a higher-level program (think bachelor’s or master’s programs from a certificate or two-year program).
EQOS proposes measurement definitions to include all students eligible to receive a credential over the past 36 months and the percentage who within 180 days of completing the program:
- Obtained new employment/placement in an occupation/academic program;
- Continued employment at a higher title or substantially higher salary;
- Obtained or continued self-employment or contract work;
- Did not obtain or continue employment, contract work, or self-employment; or
- Placed into another outcome relevant to the program mission or focus (as determined and reported by the program).
Any exclusions would follow those previously outlined and would be reported in a footnote.
EQOS notes that there is a significant difference between the first two bullets. If individuals who advance in their field are the largest percentage of reported outcomes, individuals not in that field may want to consider a different program.
It is also noted by EQOS that having providers claim that a placement is related to the training presents many subjective challenges. Their recommended offset is that providers report the five most common titles for students upon graduating and the percentage of employed graduates that hold these titles.
The proposed standards recommend reporting placement rates within 90- and 180-days post completion noting that some programs may want to report rates within shorter or longer time periods. If a period longer than 180 days is desired, EQOS recommends providing evidence as to why that is the optimal period and recommends reporting the actual data at 180 days as well. Programs should not make promotional claims about placement rates without stating the timeframe that is being measured.
All students who earned or were eligible to receive a credential in the past 36 months and were out of the program for at least 90 or 180 days should be included in the denominator. The “were eligible” inclusion is designed to keep entities from delaying the awarding of credentials in order to reduce the denominator.
EQOS provides guidance on how a learner may be considered as a successful placement and maps it in three situations. For those completers who are newly employed, the jobs obtained would have to employ the completer at least 32 hours a week or more and he/she would have to be paid. For completers who are advancing within existing employment, they would have to occupy positions guaranteeing 32 hours or more a week of employment, be paid, and have advancement through either a title change or a pay raise beyond a cost-of-living adjustment. For completers who are self-employed or who are paid contractors, they would have to be engaged 32 hours a week or more, be paid, and have a duration of five weeks or longer in the position.
In the interest of conservatism, non-responders should not be counted as a successful placement. They should also not be excluded from the denominator. If desired, the non-response rate could be reported as a separate figure. Exclusions should be rare and should follow previous guidance.
Earnings are a critical measure. According to EQOS, they provide the answer to the question, “will my program pay off?” They also serve as a check on completion percentages. If a program has a high completion rate but low earnings, there may not be value in it. Similarly, high earnings may explain lower completion rates.
EQOS recommends looking at standards through four distinct measures:
- Absolute earnings
- Percent of students earning above a minimum threshold
- Change in earnings over time
- Earnings relative to net price paid
According to EQOS, having multiple measures helps protect against certain issues. Looking only at absolute earnings may be a disadvantage for programs that lead to lower paying professions. But these occupations may be a step up from where the learner was prior to completing the program. From this data, outside rankings could construct a ROI measure for interested parties. The ideal standard of evidence would be to obtain earnings data from a government database or state unemployment agencies.
EQOS formed a few measurement definitions for this section. For absolute earnings, they state that this should reflect the median annual earnings of all individuals in the cohort and/or program graduates within the first full calendar year and after the fifth full calendar year post-graduation. This data should include individuals with $0 earnings. These data should be annualized if reported as quarterly data. These measures seem reasonable to me.
Earnings threshold is defined as the percentage of program graduates whose earnings within the first full calendar year and after the fifth full calendar year post-graduation are more than 200 percent of the federal poverty level for a one-person household in the continental U.S. EQOS states that this measure assumes a one-person household regardless of actual circumstances and uses 200 percent because it more closely approximates the earnings of a high school graduate. These measures seem reasonable to me.
Earnings change over time is defined as the median increase in earnings for program graduates within the first full calendar year and after the fifth full calendar year post-graduation, measured as the change in income from the last full calendar year prior to entering a program. The key part of this measure according to EQOS is obtaining prior data on earnings which must be done through a data collection process that occurs at the time of entry. Their logic is based on a theory that obtaining data years after a learner graduates is unlikely to yield accurate data.
Earnings relative to net price paid is define as the typical annual earnings of a graduate within the first full calendar year and after the fifth full calendar year post-graduation expressed as a percentage of the typical amount paid for a program.
These proposed standards recommend two measurement windows to judge earnings success. The first is the earnings within the first full calendar year post completion. The second is the earnings in the fifth full calendar year after completing a program. EQOS believes that the second measure is the one more suited for accountability purposes. At the same time, they recognize that obtaining data from learners five years after completing a program may be difficult and could be less relevant for short term programs.
As with the earlier measures, it is contemplated that earnings data would be updated once each year for the specific cohorts. Providers with multiple programs would have to choose the same data for all its programs.
Stakeholder Satisfaction and Confirmation of Purpose provides insight from students and employers that metrics alone cannot capture according to EQOS. Feedback can help see if the reasons students attend a given program match with the mission of that program. This measure also brings employer validation into the quality assurance process.
EQOS suggests that a benchmark for satisfaction be established by asking students to complete an intake survey before beginning a program. The purpose would be to understand why they are attending and what their expectations for the program are.
After the program is completed, EQOS recommends that two surveys be completed: one by the completer and the other by the completer’s employer post-graduation. The surveys should follow a similar design to the net promoter score. EQOS provides an example of the survey questions.
The EQOS quality assurance framework and metrics are far more detailed than the bare framework proposed by Credential As You Go. At the same time, many of these metrics and ideas are similar to either metrics proposed or utilized by the Department of Education for College Scorecard and Gainful Employment. If this framework was proposed for all degree granting institutions in the U.S., I would eagerly support it although I wouldn’t give it much chance of being implemented anytime soon.
Short term programs, especially microcredentials, are too short and too inexpensive to justify building a system of quality assurance like this. When I began reading the EQOS recommendations, I felt like there was a possibility they could be used for all programs. I don’t believe that now. It’s insightful that the Lumina funding for Credential As You Go began after EQOS proposed their framework. The project managers for Credential As You Go would have had access to the EQOS framework and measures and if they believed them to be appropriate for microcredentials, could have used them or cited them as examples that should be considered. I compliment EQOS on the design of a system that I believe could be appropriate for all degree granting institutions. Instituting a system like this for microcredentials would increase the costs of those programs, something that no potential student in the U.S. is interested in seeing right now.