Sunday’s session for presidents and chancellors had the following topics:  Vision and Change at BYU-Idaho: A Model for America’s Colleges and Universities, Information Technology:  Seize the Day, and a luncheon at which Terry Hartle, SVP of Government and Public Affairs of ACE spoke about the pending Department of Education regulations regarding Credit Hours, State Regulation, Gainful Employment, Accreditation, and Misrepresentation.  Later in the day, Yale’s President Richard Levin spoke about “Why Colleges and Universities Matter.”  I also attended a session hosted by Stan Ikenberry, former president of the University of Illinois and ACE, and George Kuh, Professor Emeritus of Higher Education at Indiana University Bloomington and the founding director of the Center for Postsecondary Research and the National Survey of Student Engagement (NSSE), regarding assessment and ways in which institutions implement it.

Having the conference in Washington provided some benefits.  Eduardo Ochoa, Assistant Secretary of Postsecondary Education at the U.S. Department of Education was an unscheduled speaker at the luncheon and provided a few comments regarding the administration’s position regarding higher education and reminded the group that he had served as a provost at Sonoma State University.  He also stated that he was unable to provide a statement about three of the issues because of a lawsuit against the Department.  Terry Hartle’s major points were that the industry can regulate itself and does not need increased federal regulation at a time when there are many changes occurring as well as innovations required in order to remain competitive.

Rick Levin’s lecture provided three main points:  (1) the basic research principle of our universities is a driver of our nation’s growth and healthcare improvements, (2) the diverse array of higher education institutions provides an education to a broad workforce, and (3) our colleges are the principal avenue of upward mobility for our citizens.  President Levin is an economist and unabashedly stated that his lecture had an economic focus although his sincere comment that “the most profound consequence of higher education is that it improves the soul” was not economically based.

As one of a number of presidents invited to attend the session on assessment, I was pleasantly surprised about the depth of discussion.  Our comments were recorded by a researcher for a project sponsored by the National Institute for Learning Outcomes Assessment (NILOA) and all of us participating received the prerequisite Institutional Review Board (IRB) disclosure.  Assessment has been an interest of the accrediting bodies for the last ten years or so, but usually is not a topic about which college and university presidents are conversant.  The group representing a variety of institutions ranging from public research universities to private liberal arts colleges to an online university.  All the presidents in attendance were focused on the measurement of learning outcomes at their institution.

Many years ago I took a course in oral history at Duke.  Interviewing people who participated in a historical event years after the event took place gave me an appreciation for the fact that reporters write about an event from their perspective which may not be the perspective of the participants.  While some of the lectures and panels at this year’s annual meeting have been discussed in articles published by The Chronicle of Higher Education and Inside Higher Ed, those articles are related to specific topics.  As a somewhat regular attendee of ACE’s annual meetings, I observed a difference through the statements of the speakers, the questions of the participants, and the general tone of the conversations at the social events.  Usually, the major determinants of change in any sector are the market or the government or both.   This year may be the year where a need for change is finally recognized by the entire sector.

Student retention has been an issue in higher education since the late 1800’s.  Some of the early research in the area began in the 1930’s but the volume of research studies increased substantially in the 1960’s through the present era.  Early research focused on psychological reasons for students to drop out of college but most of the literature since the late 1970’s have focused on sociological issues.

While there are many significant contributors to the research of student retention (William Spady, Alexander Astin,  Vincent Tinto, John M. Braxton, George Kuh, Ernest Pascarella, etc.), Vincent Tinto’s Leaving College: Rethinking the Causes and Cures of Student Attrition provides a fairly thorough overview of many research studies (note:  the second edition of this book was published in 1994 and new copies are difficult to find).

Tinto is given credit for developing the first theoretical construct of predicting student retention.  Three major areas of his theory involve the importance of the background characteristics of college students, the social integration of students with their college, and the academic integration of students with their professors and programs.  Subsequent research studies have focused on some of these areas and/or attempted to prove or disprove the original construct.  Colleges and universities use some of the studies as the basis for their internal retention research or external explanations of their graduation rate.

Continually, studies contribute to the theory that background characteristics of students are most influential for their persistence in college.  Among the data included in background characteristics studies are SAT and ACT scores, high school Grade Point Average, educational background of parents, and socioeconomic status of the student’s family.  Students with above average profiles in all of these areas usually graduate from college while students with profiles below the averages are less likely to attend or graduate from college.

At American Military University (AMU) and American Public University (APU), our students are primarily working adults with an average age of 30.  While there were notable studies of student retention at adult-serving institutions, I found very few that studied persistence at online universities or studied students who completed their programs entirely online.  While we knew the relative percentage of our students who graduate, we had not analyzed our student database for characteristics that may be significant indicators for a student’s graduation success.

Whenever institutions examine student retention, the analysis has to include the outcomes for students over a certain period of time.  Given that graduate students have already obtained an undergraduate degree, we eliminated them from the analysis.  We also needed to cover a reasonable period of time that would allow students to graduate, remain active, or disenroll from school.  We decided to examine statistics from approximately 20,500 students who were pursuing an undergraduate degree and who completed at least one course in 2007.  The students were grouped into three groups based on their academic standing as of December 31, 2009.  Those groupings were:  students who graduated, students who were still actively taking at least one course per year, and students who had disenrolled as of the end of 2009.

The analysis was fruitful in that it provided me with results that I could convey to our executive team for future studies and continued analysis.  One important finding was that the average length of time it took a student from that group to earn a bachelor’s degree at AMU or APU was 6.7 years.  We already knew that most of our students were engaged full-time with their military or civilian careers, so the length of time to complete a four-year degree would take longer than someone who was enrolled as a full-time student.  However, more than 85 percent of our students transfer in academic credits from a previously attended institution or from workplace training where the learning had been evaluated for academic credit.  Factoring in the average number of credit hours transferred in to our institution with the average number of courses taken in a year, the study results indicate that many successful adult students complete their academic journey over a period of time approaching 10 years.  That 10-year period may be longer if career or family obligations interrupt the flow of taking classes.

We utilized a forward regression model to determine if there were independent variables that might predict whether an individual student would disenroll from either AMU or APU.   It was not all too surprising that students from the group who transferred no credits were most likely to disenroll (our paper was listed as 31084 and will be available as part of the Printed Proceedings book available at www.digital-factory.net/aace).  Given that 121 credit hours (40 courses) are required for most bachelor’s programs and the average student at AMU/APU takes 12 credits (4 courses) per year, a 10-year commitment may be easier to walk away from in the early stages than after a substantial amount of credits have been earned.  Supporting this regression analysis outcome is the fact that 40.3 percent of the students in the group who disenrolled did so after taking only 2 classes and 65.3 percent disenrolled after taking 4 or fewer classes.  Decreasing the percentage of students who disenroll early has been a focus at American Public University System (APUS) for nearly a decade and continues.  Many on-ground programs have similar issues with new students leaving early and have programs dedicated to the first year student experience in order to improve the student experience and more fully integrate them into the social and academic culture.  Social and academic integration in online learning environments is more difficult, but not impossible and we will continue to investigate ways in which we can meet the needs of new students in their first classes.

Academic institutions like APUS that participate in the Federal Student Aid program are required to submit vast quantities of student data to the National Center for Education Statistics (NCES) through the Institutional Postsecondary Education Data System (IPEDS).  That data is made available to the public and to researchers.  The Department of Education reports an institution’s official graduate rate as the number of first-time, full-time freshmen students who graduate within 150 percent of the normal time to complete a degree program (three years for a two year degree and six years for a four year degree).  If an institution does not have any first-time, full-time freshmen, it will have a graduation rate of zero.  Many institutions that serve a large population of adult students will have very low numbers of first-time, full-time freshmen.  A useful source of data to determine the relative percentage of part-time students at an institution is to divide the FTE (full-time equivalent) students reported in a given year by the unduplicated headcount reported by the institution for the same year.  The lower the decimal calculated, the higher the number of part-time students will be who attend that institution.  If the majority of an institution’s students are part-time, the Department collects data on completions, but does not collect data about the length of time it took to complete a degree or the year in which the students who completed a degree matriculated at the institution.

Enrollments at colleges and universities that offer online programs have soared in recent years for many reasons, one of which is the convenience of working online from work or home versus commuting to a physical location for a class that may be held at times inconvenient for a working adult.  However, the consumer-friendly data that is collected by NCES relates to students who plan to be first-time, full-time freshmen and does not reflect the success rates of adult-students who matriculate and graduate from these programs.  Given that the department tracks unduplicated student headcounts and completions each year, adding two fields relating to unduplicated new students and drops for each year would provide a cohort-tracking system that could provide an aggregate completion/graduation rate for part-time students.  Many institutions serving part-time students allow seven years for those students to complete a two-year degree and up to ten years for a four-year degree.  The pace at which students complete those degrees varies based upon professional and family requirements that may conflict at times with the number of courses that a student can take.  One of the desired outcomes of our research was to provide a benchmark for similar studies as well as a suggested guideline for tracking persistence in adult-serving online degree programs.  As more and more adults return to college to complete their degrees online, finding a consistent system for measuring and reporting their progress will become more important.

For those of us working for regionally accredited, market-driven institutions, the survey confirms that we utilize assessment and assessment tools in many more ways than traditional research institutions.  I cannot speak for all for-profit institutions, but we embraced assessment as a tool when we realized its value in diagnosing what worked and what didn’t as the online learning field continued to evolve through improvements in pedagogy and technology.  A group of like-minded, for profit and non-profit institutions joined together to form Transparency by Design (TBD), an initiative to publish learning outcomes in a common reporting format.  As we continue to utilize assessment for quality improvement, our faculty will gain the knowledge of what works better for online teaching and our students will benefit through better designed and better instructed classes and programs.  Organizations like NILOA and TBD will share best practices with the goal of providing better outcomes for students.

I think you will enjoy reading the results of the survey as summarized by Dr. Stephens.   I look forward to seeing future surveys that indicate that progress in the utilization of assessment tools is being made by all institutions of higher education.

Over the past decade, calls for assessment and accountability have increased as the educational community has become more vocal about the need to be more systematic in assessing student performance.  This is evidenced by: 1) regional and national meetings that focus on assessment; 2) accountability initiatives such as the Voluntary System of Accountability (VSA), University and College Accountability Network (U-CAN), and Transparency by Design; and 3) the recent sharp increase of assessment tools and organizations that focus on the assessment of student learning outcomes.

To better understand the dynamics of student learning outcomes assessment in higher education institutions, the National Institute for Learning Outcomes Assessment (NILOA) was launched in 2008 to assist institutions and others in discovering and adopting promising practices in the assessment of college student learning outcomes. The vision of the NILOA is to discover and disseminate ways that schools can productively use assessment data internally to inform and strengthen undergraduate education, and externally to communicate with policy makers, families and other stakeholders.  The NILOA project is based at the University of Illinois and Indiana University. Stan Ikenberry and George Kuh serve as co-principal investigators, and Peter Ewell serves as a Senior Scholar. The initiative is guided by a National Advisory Panel and supported by foundations including Lumina Foundation for Education, the Carnegie Corporation of New York, and the Teagle Foundation.

To advance their mission and as their first big project, the NILOA surveyed provosts or chief academic officers at all regionally accredited, undergraduate-degree-granting, two and four year, public, private, and for-profit institutions in the U.S. about the assessment activities conducted at their institutions.  In the spring of 2009, the questionnaire was administered to 2809 institutions.  There was a 53% response rate with 1518 schools responding.  

Eight observations were made that are discussed in the report, “More Than You Think, Less Than We Need: Learning Outcomes Assessment in American Higher Education”:

1. Most institutions have identified a common set of learning outcomes that apply to all students.  About three quarters of all institutions have stated student learning outcomes for their undergraduate students.  Larger, research intensive institutions were less likely than associate and bachelor’s degree colleges to have common learning outcomes for all undergraduate students.  Schools range from 65% (doctoral universities) to 81% (Carnegie classified other institutions) when reporting on whether they have a set of common learning outcomes for all undergraduate students. 

2. Most institutions use a combination of institution-level and program-level assess-ment approaches.   The majority (92%) of all schools use at least one institutional level assessment tool.  Two-thirds of all schools use three or more tools.  For profit institutions on average use more institutional level assessment tools than all other types of institutions.  That is, more than half (55%) of for-profit schools use five or more institution-level approaches.  These institutional level assessments include surveys, rubrics, interviews, external judges, and portfolios. 

3. The most common uses of assessment data relate to accreditation.  When asked how campuses are actually using the results from their outcomes assessment processes, all schools report that they are primarily using the results to prepare for program and institutional accreditation.  Among all schools, the least common uses for assessment data are for making daily resource decisions, admissions and transfer policies, and faculty/staff performance.  It is interesting to note that for-profit schools reported the most frequent use of assessment data in all 22 categories.  As noted earlier, not only do for profit schools administer more institutional level measures of assessment, they also report the most frequent use of assessment data.  The researchers hypothesize that the business models of the for-profits, desire to achieve accreditation, and the need to address questions about legitimacy may lead for-profit schools to more actively collect, report, and use assessment results.

4. Assessment approaches and uses of assessment results vary systematically by institu¬tional selectivity.  On average, less competitive institutions are more likely to use standardized measures while more competitive institutions are more likely to use locally developed instruments.  For example, half of the least competitive institutions administer general education tests compared with only one-fifth of the most competitive institutions.  When examining the use of assessment data by selectivity of the institution, the most competitive schools collect information at rates similar to their less selective counterparts, but they report that they don’t use it as often.  That is with one exception, reporting to their governing boards.  The survey results demonstrate that the most competitive institutions are least likely to use assessment data for revising learning goals, responding to calls for accountability, informing strategic planning, improving instructional performance, evaluating programs, allocating resources, and reporting to the public.

5. Assessment is driven more by accreditation and a commitment to improve than external pressures from government or employers. When asked what is driving the assessment movement in higher education, all schools report that the biggest influences are the expectations of regional and specialized accreditation agencies  and the institution’s commitment to improvement.  The least influential drivers are national calls for accountability and mandates from trustees and state coordinating boards.  When the differences among public, private, and for-profit are examined, for-profit schools indicate that every one of the factors (program and institutional accreditation, institutional commitment to improvement, faculty/staff interest, national calls, governing and coordinating board mandates, and institutional associations) was influential in driving assessment activity.  The researchers again suggest that there is a sharper focus on learning outcomes assessment by the for-profit sector. 

6. Most institutions conduct learning outcomes assessment on a shoestring: 20% have no assessment staff and 65% have two or fewer.  The researchers describe the investment in assessment staff on higher education campuses as “relatively modest”.  For example, only 25% of provosts reported having more than one FTE person assigned to assessment.  Almost half (47%) of doctoral institutions reported having one or more staff, while only one-fifth (19%) of community college and other associate degree granting schools had at least one person focused on assessment. 

7. Gaining faculty involvement and support remains a major challenge. Campuses would also like more assessment expertise, resources, and tools.  When asked about what schools need to be more effective in student learning outcomes assessment, the two greatest needs expressed by more than three-fifths of all institutions were more: 1) faculty engagement – 66% of the schools state this would be helpful in assessing learning outcomes; and 2) expertise in assessment – 61% of the schools stated it would be helpful.  Almost half of all provosts said they need more resources for learning outcomes assess¬ment, and this resource pinch appears greatest on smaller campuses.

8. Most institutions plan to continue learning outcomes assessment despite budgetary challenges.  Provosts were asked if there were upcoming changes for their institutional resources currently committed to assessment.  A small amount (one-fifth) of provosts indicated that a decrease in institutional support was possible whereas more than one-half of all schools predicted that the current recession would not affect their assessment activities.  More respondents from public institutions were uncertain about financial support compared to their counterparts at private schools.

The results from this survey state much of what we already know, and that is accreditation remains the primary vehicle for driving the learning outcomes assessment process in American higher education.   Accreditation agencies hold campuses accountable for demonstrating student learning outcomes evidence and using this evidence for continuous improvement of courses, programs, and the institution.  So while quality assurance and assessment activities are being conducted at our college and university campuses, the researchers assert that we are far from where we need to be with our progress in learning outcomes assessment.  That is, conducting assessment because accreditation agencies “mandate” these practices is not healthy for institutions.  Institutions should regularly conduct assessment activities with the intent to inform decisions and for continuous improvement of teaching and learning.  The following recommendations are made to advance progress on higher education learning outcomes assessment:

• Presidents, provosts, and other academic leaders must make quality assurance an institu¬tional priority.
• Governing board members must ensure their institution has a system of academic quality control supported by the assessment of student learning and the use of those results for continuous improvement.
• Faculty members must systematically collect data about student learning, carefully examine and discuss these results with colleagues, and use this information to improve student outcomes.
• Assessment and institutional research personnel should revisit the rationale for using various tools and approaches to be sure they yield the kind of information that your institution needs to respond to improvement and accountability mandates.
• Student affairs staff must share their perspectives on the student experience by partici-pating on the campus assessment committee and self-study committees.
• Faculty developers must become familiar with the campus assessment activities and results and use this information in designing professional development opportunities for faculty, student affairs professionals, librarians, and others who work with students.
• Prospective students and parents should ask to see learning outcomes information about students who attend the institutions they are considering. If it is not publicly accessible on an institution website, ask someone in the institution’s admissions office for data about how their students perform on different kinds of measures.
• Higher education associations must keep learning outcomes assessment on their agenda.
• Statewide planning and coordinating boards must confirm that all institutions under their scope of influence have effective internal systems of academic quality control supported by assessment data that conform to the expectations of both regional and specialized accreditation bodies.
• Accrediting groups must not let up on efforts to promote assessment and the use of student learning outcomes. Sharpen accreditation standards as they are applied to (a) collecting institution- and program-level data about student performance, (b) using assessment results to improve student performance and institutional quality, and (c) making assess¬ment results available internally and externally. In all of these areas, hold institutions accountable.
• Foundations should keep learning outcomes assessment on their funding agendas. Devote more attention to programs and incentives that encourage institutions to use outcomes data productively. Encourage accrediting groups, both regional and specialized, to be vehicles for campus change that is constructive and attainable.

It is good to hear the collective voice of our campus leaders in the national conversation on learning outcomes assessment.  With all of the conversations that are generated among various groups on this topic, our provosts and chief academic officers are often overlooked groups who have an important voice and obvious stake in the process.  Their voice is a significant contribution to the current ongoing national conversation on learning outcomes assessment in higher education.  Kudos to NILOA for this exploration work that is necessary to providing a deeper understanding of the issues that our higher education institutions face in assessment.  By thoroughly understanding these issues as reported through the eyes of provosts, the educational community can more adequately address the call to be more systematic in assessing student learning outcomes.