On October 26, 2009, the National Institute for Learning Outcomes Assessment (NILOA) issued the results of its first annual survey of Provosts and Chief Academic Officers. When I read the press release and skimmed through the survey, I asked Dr. Jennifer Stephens, our Associate Vice President and Dean of Assessment, to provide me with a guest article describing the survey and the significant findings.
For those of us working for regionally accredited, market-driven institutions, the survey confirms that we utilize assessment and assessment tools in many more ways than traditional research institutions. I cannot speak for all for-profit institutions, but we embraced assessment as a tool when we realized its value in diagnosing what worked and what didn’t as the online learning field continued to evolve through improvements in pedagogy and technology. A group of like-minded, for profit and non-profit institutions joined together to form Transparency by Design (TBD), an initiative to publish learning outcomes in a common reporting format. As we continue to utilize assessment for quality improvement, our faculty will gain the knowledge of what works better for online teaching and our students will benefit through better designed and better instructed classes and programs. Organizations like NILOA and TBD will share best practices with the goal of providing better outcomes for students.
I think you will enjoy reading the results of the survey as summarized by Dr. Stephens. I look forward to seeing future surveys that indicate that progress in the utilization of assessment tools is being made by all institutions of higher education.
Over the past decade, calls for assessment and accountability have increased as the educational community has become more vocal about the need to be more systematic in assessing student performance. This is evidenced by: 1) regional and national meetings that focus on assessment; 2) accountability initiatives such as the Voluntary System of Accountability (VSA), University and College Accountability Network (U-CAN), and Transparency by Design; and 3) the recent sharp increase of assessment tools and organizations that focus on the assessment of student learning outcomes.
To better understand the dynamics of student learning outcomes assessment in higher education institutions, the National Institute for Learning Outcomes Assessment (NILOA) was launched in 2008 to assist institutions and others in discovering and adopting promising practices in the assessment of college student learning outcomes. The vision of the NILOA is to discover and disseminate ways that schools can productively use assessment data internally to inform and strengthen undergraduate education, and externally to communicate with policy makers, families and other stakeholders. The NILOA project is based at the University of Illinois and Indiana University. Stan Ikenberry and George Kuh serve as co-principal investigators, and Peter Ewell serves as a Senior Scholar. The initiative is guided by a National Advisory Panel and supported by foundations including Lumina Foundation for Education, the Carnegie Corporation of New York, and the Teagle Foundation.
To advance their mission and as their first big project, the NILOA surveyed provosts or chief academic officers at all regionally accredited, undergraduate-degree-granting, two and four year, public, private, and for-profit institutions in the U.S. about the assessment activities conducted at their institutions. In the spring of 2009, the questionnaire was administered to 2809 institutions. There was a 53% response rate with 1518 schools responding.
Eight observations were made that are discussed in the report, “More Than You Think, Less Than We Need: Learning Outcomes Assessment in American Higher Education”:
1. Most institutions have identified a common set of learning outcomes that apply to all students. About three quarters of all institutions have stated student learning outcomes for their undergraduate students. Larger, research intensive institutions were less likely than associate and bachelor’s degree colleges to have common learning outcomes for all undergraduate students. Schools range from 65% (doctoral universities) to 81% (Carnegie classified other institutions) when reporting on whether they have a set of common learning outcomes for all undergraduate students.
2. Most institutions use a combination of institution-level and program-level assess-ment approaches. The majority (92%) of all schools use at least one institutional level assessment tool. Two-thirds of all schools use three or more tools. For profit institutions on average use more institutional level assessment tools than all other types of institutions. That is, more than half (55%) of for-profit schools use five or more institution-level approaches. These institutional level assessments include surveys, rubrics, interviews, external judges, and portfolios.
3. The most common uses of assessment data relate to accreditation. When asked how campuses are actually using the results from their outcomes assessment processes, all schools report that they are primarily using the results to prepare for program and institutional accreditation. Among all schools, the least common uses for assessment data are for making daily resource decisions, admissions and transfer policies, and faculty/staff performance. It is interesting to note that for-profit schools reported the most frequent use of assessment data in all 22 categories. As noted earlier, not only do for profit schools administer more institutional level measures of assessment, they also report the most frequent use of assessment data. The researchers hypothesize that the business models of the for-profits, desire to achieve accreditation, and the need to address questions about legitimacy may lead for-profit schools to more actively collect, report, and use assessment results.
4. Assessment approaches and uses of assessment results vary systematically by institu¬tional selectivity. On average, less competitive institutions are more likely to use standardized measures while more competitive institutions are more likely to use locally developed instruments. For example, half of the least competitive institutions administer general education tests compared with only one-fifth of the most competitive institutions. When examining the use of assessment data by selectivity of the institution, the most competitive schools collect information at rates similar to their less selective counterparts, but they report that they don’t use it as often. That is with one exception, reporting to their governing boards. The survey results demonstrate that the most competitive institutions are least likely to use assessment data for revising learning goals, responding to calls for accountability, informing strategic planning, improving instructional performance, evaluating programs, allocating resources, and reporting to the public.
5. Assessment is driven more by accreditation and a commitment to improve than external pressures from government or employers. When asked what is driving the assessment movement in higher education, all schools report that the biggest influences are the expectations of regional and specialized accreditation agencies and the institution’s commitment to improvement. The least influential drivers are national calls for accountability and mandates from trustees and state coordinating boards. When the differences among public, private, and for-profit are examined, for-profit schools indicate that every one of the factors (program and institutional accreditation, institutional commitment to improvement, faculty/staff interest, national calls, governing and coordinating board mandates, and institutional associations) was influential in driving assessment activity. The researchers again suggest that there is a sharper focus on learning outcomes assessment by the for-profit sector.
6. Most institutions conduct learning outcomes assessment on a shoestring: 20% have no assessment staff and 65% have two or fewer. The researchers describe the investment in assessment staff on higher education campuses as “relatively modest”. For example, only 25% of provosts reported having more than one FTE person assigned to assessment. Almost half (47%) of doctoral institutions reported having one or more staff, while only one-fifth (19%) of community college and other associate degree granting schools had at least one person focused on assessment.
7. Gaining faculty involvement and support remains a major challenge. Campuses would also like more assessment expertise, resources, and tools. When asked about what schools need to be more effective in student learning outcomes assessment, the two greatest needs expressed by more than three-fifths of all institutions were more: 1) faculty engagement – 66% of the schools state this would be helpful in assessing learning outcomes; and 2) expertise in assessment – 61% of the schools stated it would be helpful. Almost half of all provosts said they need more resources for learning outcomes assess¬ment, and this resource pinch appears greatest on smaller campuses.
8. Most institutions plan to continue learning outcomes assessment despite budgetary challenges. Provosts were asked if there were upcoming changes for their institutional resources currently committed to assessment. A small amount (one-fifth) of provosts indicated that a decrease in institutional support was possible whereas more than one-half of all schools predicted that the current recession would not affect their assessment activities. More respondents from public institutions were uncertain about financial support compared to their counterparts at private schools.
The results from this survey state much of what we already know, and that is accreditation remains the primary vehicle for driving the learning outcomes assessment process in American higher education. Accreditation agencies hold campuses accountable for demonstrating student learning outcomes evidence and using this evidence for continuous improvement of courses, programs, and the institution. So while quality assurance and assessment activities are being conducted at our college and university campuses, the researchers assert that we are far from where we need to be with our progress in learning outcomes assessment. That is, conducting assessment because accreditation agencies “mandate” these practices is not healthy for institutions. Institutions should regularly conduct assessment activities with the intent to inform decisions and for continuous improvement of teaching and learning. The following recommendations are made to advance progress on higher education learning outcomes assessment:
• Presidents, provosts, and other academic leaders must make quality assurance an institu¬tional priority.
• Governing board members must ensure their institution has a system of academic quality control supported by the assessment of student learning and the use of those results for continuous improvement.
• Faculty members must systematically collect data about student learning, carefully examine and discuss these results with colleagues, and use this information to improve student outcomes.
• Assessment and institutional research personnel should revisit the rationale for using various tools and approaches to be sure they yield the kind of information that your institution needs to respond to improvement and accountability mandates.
• Student affairs staff must share their perspectives on the student experience by partici-pating on the campus assessment committee and self-study committees.
• Faculty developers must become familiar with the campus assessment activities and results and use this information in designing professional development opportunities for faculty, student affairs professionals, librarians, and others who work with students.
• Prospective students and parents should ask to see learning outcomes information about students who attend the institutions they are considering. If it is not publicly accessible on an institution website, ask someone in the institution’s admissions office for data about how their students perform on different kinds of measures.
• Higher education associations must keep learning outcomes assessment on their agenda.
• Statewide planning and coordinating boards must confirm that all institutions under their scope of influence have effective internal systems of academic quality control supported by assessment data that conform to the expectations of both regional and specialized accreditation bodies.
• Accrediting groups must not let up on efforts to promote assessment and the use of student learning outcomes. Sharpen accreditation standards as they are applied to (a) collecting institution- and program-level data about student performance, (b) using assessment results to improve student performance and institutional quality, and (c) making assess¬ment results available internally and externally. In all of these areas, hold institutions accountable.
• Foundations should keep learning outcomes assessment on their funding agendas. Devote more attention to programs and incentives that encourage institutions to use outcomes data productively. Encourage accrediting groups, both regional and specialized, to be vehicles for campus change that is constructive and attainable.
It is good to hear the collective voice of our campus leaders in the national conversation on learning outcomes assessment. With all of the conversations that are generated among various groups on this topic, our provosts and chief academic officers are often overlooked groups who have an important voice and obvious stake in the process. Their voice is a significant contribution to the current ongoing national conversation on learning outcomes assessment in higher education. Kudos to NILOA for this exploration work that is necessary to providing a deeper understanding of the issues that our higher education institutions face in assessment. By thoroughly understanding these issues as reported through the eyes of provosts, the educational community can more adequately address the call to be more systematic in assessing student learning outcomes.
