Since APUS courses are offered wholly online with no time for face-to-face instruction, we developed a number of math instructional videos using Camtasia tablet software and embedded them in our classrooms to supplement the instructional materials.  Later, we decided to make our math videos available to everyone on our American Public University iTunesU site  and our APUS Youtube channel.  Comments to the individual videos, primarily in the form of thank you’s, demonstrate the usefulness and the need for technology like this.  More recently, we partnered with McDonogh School to establish a website, www.campusmath.com, to offer primarily math videos to the public for an elementary school through high school curriculum.  While I can’t speak on behalf of McDonogh School, I think that both of our institutions are aligned with the belief that math skills need to be improved and providing access to these videos to teachers, students, and parents may contribute to improved skills without providing the teachers and professors inside of a physical or electronic classroom.

The article about the Khan Academy in Wired mentions a teacher at Santa Rita Elementary in California who uses the videos from the Khan Academy to replace some of her lectures and then spends more time in class working on problem sets.  Teachers using the Khan Academy videos and problem sets have access to a dashboard that lets them see exactly where the student is stuck on a particular concept or problem.  Increasing the amount of time spent in class solving problem sets provides the teacher with more one-on-one time with the students who need assistance.  Contributing Editor Clive Thompson mentions the 1984 Benjamin Bloom metastudy that measured the effectiveness of one-on-one tutoring versus general classroom instruction (two standard deviations more effective for one-on-one instruction).  Obviously, schools cannot afford to teach everyone one-on-one but technology is capable of assisting teachers and students to improve learning outcomes.  What prompted Khan to record his videos was the discovery that viewing videos over and over again in private is less embarrassing for the student than admitting in a one-on-one session that they still don’t understand the material.  Judging from the comments posted by students on our Youtube videos, Khan’s discovery is on track.

Based on the increasing frequency of its use in the classroom, technology continues to advance at a pace faster than the implementation capabilities of many of America’s K-12 schools, colleges, and universities.  At the same time, there are a number of educational innovators who are willing to experiment.  As Clayton Christensen points out in Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns, technology will ultimately provide educators with the ability to teach students with multiple learning modalities the intended lessons without impeding the highly talented individuals from learning more.  It is my hope that few institutions and regulators impede the innovation process.  (To read my review of Christensen’s Disrupting Class, see my August 2008 blog article.)

Sunday’s session for presidents and chancellors had the following topics:  Vision and Change at BYU-Idaho: A Model for America’s Colleges and Universities, Information Technology:  Seize the Day, and a luncheon at which Terry Hartle, SVP of Government and Public Affairs of ACE spoke about the pending Department of Education regulations regarding Credit Hours, State Regulation, Gainful Employment, Accreditation, and Misrepresentation.  Later in the day, Yale’s President Richard Levin spoke about “Why Colleges and Universities Matter.”  I also attended a session hosted by Stan Ikenberry, former president of the University of Illinois and ACE, and George Kuh, Professor Emeritus of Higher Education at Indiana University Bloomington and the founding director of the Center for Postsecondary Research and the National Survey of Student Engagement (NSSE), regarding assessment and ways in which institutions implement it.

Having the conference in Washington provided some benefits.  Eduardo Ochoa, Assistant Secretary of Postsecondary Education at the U.S. Department of Education was an unscheduled speaker at the luncheon and provided a few comments regarding the administration’s position regarding higher education and reminded the group that he had served as a provost at Sonoma State University.  He also stated that he was unable to provide a statement about three of the issues because of a lawsuit against the Department.  Terry Hartle’s major points were that the industry can regulate itself and does not need increased federal regulation at a time when there are many changes occurring as well as innovations required in order to remain competitive.

Rick Levin’s lecture provided three main points:  (1) the basic research principle of our universities is a driver of our nation’s growth and healthcare improvements, (2) the diverse array of higher education institutions provides an education to a broad workforce, and (3) our colleges are the principal avenue of upward mobility for our citizens.  President Levin is an economist and unabashedly stated that his lecture had an economic focus although his sincere comment that “the most profound consequence of higher education is that it improves the soul” was not economically based.

As one of a number of presidents invited to attend the session on assessment, I was pleasantly surprised about the depth of discussion.  Our comments were recorded by a researcher for a project sponsored by the National Institute for Learning Outcomes Assessment (NILOA) and all of us participating received the prerequisite Institutional Review Board (IRB) disclosure.  Assessment has been an interest of the accrediting bodies for the last ten years or so, but usually is not a topic about which college and university presidents are conversant.  The group representing a variety of institutions ranging from public research universities to private liberal arts colleges to an online university.  All the presidents in attendance were focused on the measurement of learning outcomes at their institution.

Many years ago I took a course in oral history at Duke.  Interviewing people who participated in a historical event years after the event took place gave me an appreciation for the fact that reporters write about an event from their perspective which may not be the perspective of the participants.  While some of the lectures and panels at this year’s annual meeting have been discussed in articles published by The Chronicle of Higher Education and Inside Higher Ed, those articles are related to specific topics.  As a somewhat regular attendee of ACE’s annual meetings, I observed a difference through the statements of the speakers, the questions of the participants, and the general tone of the conversations at the social events.  Usually, the major determinants of change in any sector are the market or the government or both.   This year may be the year where a need for change is finally recognized by the entire sector.

For those of us working for regionally accredited, market-driven institutions, the survey confirms that we utilize assessment and assessment tools in many more ways than traditional research institutions.  I cannot speak for all for-profit institutions, but we embraced assessment as a tool when we realized its value in diagnosing what worked and what didn’t as the online learning field continued to evolve through improvements in pedagogy and technology.  A group of like-minded, for profit and non-profit institutions joined together to form Transparency by Design (TBD), an initiative to publish learning outcomes in a common reporting format.  As we continue to utilize assessment for quality improvement, our faculty will gain the knowledge of what works better for online teaching and our students will benefit through better designed and better instructed classes and programs.  Organizations like NILOA and TBD will share best practices with the goal of providing better outcomes for students.

I think you will enjoy reading the results of the survey as summarized by Dr. Stephens.   I look forward to seeing future surveys that indicate that progress in the utilization of assessment tools is being made by all institutions of higher education.

Over the past decade, calls for assessment and accountability have increased as the educational community has become more vocal about the need to be more systematic in assessing student performance.  This is evidenced by: 1) regional and national meetings that focus on assessment; 2) accountability initiatives such as the Voluntary System of Accountability (VSA), University and College Accountability Network (U-CAN), and Transparency by Design; and 3) the recent sharp increase of assessment tools and organizations that focus on the assessment of student learning outcomes.

To better understand the dynamics of student learning outcomes assessment in higher education institutions, the National Institute for Learning Outcomes Assessment (NILOA) was launched in 2008 to assist institutions and others in discovering and adopting promising practices in the assessment of college student learning outcomes. The vision of the NILOA is to discover and disseminate ways that schools can productively use assessment data internally to inform and strengthen undergraduate education, and externally to communicate with policy makers, families and other stakeholders.  The NILOA project is based at the University of Illinois and Indiana University. Stan Ikenberry and George Kuh serve as co-principal investigators, and Peter Ewell serves as a Senior Scholar. The initiative is guided by a National Advisory Panel and supported by foundations including Lumina Foundation for Education, the Carnegie Corporation of New York, and the Teagle Foundation.

To advance their mission and as their first big project, the NILOA surveyed provosts or chief academic officers at all regionally accredited, undergraduate-degree-granting, two and four year, public, private, and for-profit institutions in the U.S. about the assessment activities conducted at their institutions.  In the spring of 2009, the questionnaire was administered to 2809 institutions.  There was a 53% response rate with 1518 schools responding.  

Eight observations were made that are discussed in the report, “More Than You Think, Less Than We Need: Learning Outcomes Assessment in American Higher Education”:

1. Most institutions have identified a common set of learning outcomes that apply to all students.  About three quarters of all institutions have stated student learning outcomes for their undergraduate students.  Larger, research intensive institutions were less likely than associate and bachelor’s degree colleges to have common learning outcomes for all undergraduate students.  Schools range from 65% (doctoral universities) to 81% (Carnegie classified other institutions) when reporting on whether they have a set of common learning outcomes for all undergraduate students. 

2. Most institutions use a combination of institution-level and program-level assess-ment approaches.   The majority (92%) of all schools use at least one institutional level assessment tool.  Two-thirds of all schools use three or more tools.  For profit institutions on average use more institutional level assessment tools than all other types of institutions.  That is, more than half (55%) of for-profit schools use five or more institution-level approaches.  These institutional level assessments include surveys, rubrics, interviews, external judges, and portfolios. 

3. The most common uses of assessment data relate to accreditation.  When asked how campuses are actually using the results from their outcomes assessment processes, all schools report that they are primarily using the results to prepare for program and institutional accreditation.  Among all schools, the least common uses for assessment data are for making daily resource decisions, admissions and transfer policies, and faculty/staff performance.  It is interesting to note that for-profit schools reported the most frequent use of assessment data in all 22 categories.  As noted earlier, not only do for profit schools administer more institutional level measures of assessment, they also report the most frequent use of assessment data.  The researchers hypothesize that the business models of the for-profits, desire to achieve accreditation, and the need to address questions about legitimacy may lead for-profit schools to more actively collect, report, and use assessment results.

4. Assessment approaches and uses of assessment results vary systematically by institu¬tional selectivity.  On average, less competitive institutions are more likely to use standardized measures while more competitive institutions are more likely to use locally developed instruments.  For example, half of the least competitive institutions administer general education tests compared with only one-fifth of the most competitive institutions.  When examining the use of assessment data by selectivity of the institution, the most competitive schools collect information at rates similar to their less selective counterparts, but they report that they don’t use it as often.  That is with one exception, reporting to their governing boards.  The survey results demonstrate that the most competitive institutions are least likely to use assessment data for revising learning goals, responding to calls for accountability, informing strategic planning, improving instructional performance, evaluating programs, allocating resources, and reporting to the public.

5. Assessment is driven more by accreditation and a commitment to improve than external pressures from government or employers. When asked what is driving the assessment movement in higher education, all schools report that the biggest influences are the expectations of regional and specialized accreditation agencies  and the institution’s commitment to improvement.  The least influential drivers are national calls for accountability and mandates from trustees and state coordinating boards.  When the differences among public, private, and for-profit are examined, for-profit schools indicate that every one of the factors (program and institutional accreditation, institutional commitment to improvement, faculty/staff interest, national calls, governing and coordinating board mandates, and institutional associations) was influential in driving assessment activity.  The researchers again suggest that there is a sharper focus on learning outcomes assessment by the for-profit sector. 

6. Most institutions conduct learning outcomes assessment on a shoestring: 20% have no assessment staff and 65% have two or fewer.  The researchers describe the investment in assessment staff on higher education campuses as “relatively modest”.  For example, only 25% of provosts reported having more than one FTE person assigned to assessment.  Almost half (47%) of doctoral institutions reported having one or more staff, while only one-fifth (19%) of community college and other associate degree granting schools had at least one person focused on assessment. 

7. Gaining faculty involvement and support remains a major challenge. Campuses would also like more assessment expertise, resources, and tools.  When asked about what schools need to be more effective in student learning outcomes assessment, the two greatest needs expressed by more than three-fifths of all institutions were more: 1) faculty engagement – 66% of the schools state this would be helpful in assessing learning outcomes; and 2) expertise in assessment – 61% of the schools stated it would be helpful.  Almost half of all provosts said they need more resources for learning outcomes assess¬ment, and this resource pinch appears greatest on smaller campuses.

8. Most institutions plan to continue learning outcomes assessment despite budgetary challenges.  Provosts were asked if there were upcoming changes for their institutional resources currently committed to assessment.  A small amount (one-fifth) of provosts indicated that a decrease in institutional support was possible whereas more than one-half of all schools predicted that the current recession would not affect their assessment activities.  More respondents from public institutions were uncertain about financial support compared to their counterparts at private schools.

The results from this survey state much of what we already know, and that is accreditation remains the primary vehicle for driving the learning outcomes assessment process in American higher education.   Accreditation agencies hold campuses accountable for demonstrating student learning outcomes evidence and using this evidence for continuous improvement of courses, programs, and the institution.  So while quality assurance and assessment activities are being conducted at our college and university campuses, the researchers assert that we are far from where we need to be with our progress in learning outcomes assessment.  That is, conducting assessment because accreditation agencies “mandate” these practices is not healthy for institutions.  Institutions should regularly conduct assessment activities with the intent to inform decisions and for continuous improvement of teaching and learning.  The following recommendations are made to advance progress on higher education learning outcomes assessment:

• Presidents, provosts, and other academic leaders must make quality assurance an institu¬tional priority.
• Governing board members must ensure their institution has a system of academic quality control supported by the assessment of student learning and the use of those results for continuous improvement.
• Faculty members must systematically collect data about student learning, carefully examine and discuss these results with colleagues, and use this information to improve student outcomes.
• Assessment and institutional research personnel should revisit the rationale for using various tools and approaches to be sure they yield the kind of information that your institution needs to respond to improvement and accountability mandates.
• Student affairs staff must share their perspectives on the student experience by partici-pating on the campus assessment committee and self-study committees.
• Faculty developers must become familiar with the campus assessment activities and results and use this information in designing professional development opportunities for faculty, student affairs professionals, librarians, and others who work with students.
• Prospective students and parents should ask to see learning outcomes information about students who attend the institutions they are considering. If it is not publicly accessible on an institution website, ask someone in the institution’s admissions office for data about how their students perform on different kinds of measures.
• Higher education associations must keep learning outcomes assessment on their agenda.
• Statewide planning and coordinating boards must confirm that all institutions under their scope of influence have effective internal systems of academic quality control supported by assessment data that conform to the expectations of both regional and specialized accreditation bodies.
• Accrediting groups must not let up on efforts to promote assessment and the use of student learning outcomes. Sharpen accreditation standards as they are applied to (a) collecting institution- and program-level data about student performance, (b) using assessment results to improve student performance and institutional quality, and (c) making assess¬ment results available internally and externally. In all of these areas, hold institutions accountable.
• Foundations should keep learning outcomes assessment on their funding agendas. Devote more attention to programs and incentives that encourage institutions to use outcomes data productively. Encourage accrediting groups, both regional and specialized, to be vehicles for campus change that is constructive and attainable.

It is good to hear the collective voice of our campus leaders in the national conversation on learning outcomes assessment.  With all of the conversations that are generated among various groups on this topic, our provosts and chief academic officers are often overlooked groups who have an important voice and obvious stake in the process.  Their voice is a significant contribution to the current ongoing national conversation on learning outcomes assessment in higher education.  Kudos to NILOA for this exploration work that is necessary to providing a deeper understanding of the issues that our higher education institutions face in assessment.  By thoroughly understanding these issues as reported through the eyes of provosts, the educational community can more adequately address the call to be more systematic in assessing student learning outcomes.