Engaging Online Faculty & Administrators in the Assessment Process

By Dr. Jennifer Stephens Helm, Vice President and Dean of Institutional Research and Assessment, American Public University System

Foreword by Wally Boston:

The Quality Matters organization developed a very famous rubric for evaluating the quality of online courses, and online educators throughout higher education utilize this rubric. Earlier this year, the organization published, Assuring Quality in Online Education: Practices and Processes at the Teaching, Resource, and Program Levels which, according to the foreword, “…provides a comprehensive overview of quality assurance, a snapshot of current practices and proven recommendations for raising standards of quality in online education.”

I’m proud to congratulate Dr. Jennifer Stephens Helm on being one of several APUS contributing authors to the volume, having co-authored a chapter on “Engaging online faculty and administrators in the assessment process” with APUS Executive Vice President and Provost Karan Powell and APUS Director of Assessment Julie Atwood. Dr. Stephens Helm addresses the topic in the following guest commentary.

—Wally Boston

Engaging faculty and administrators in ongoing assessment processes at a distance learning institution can pose unique challenges given that most faculty members work remotely. Many strategies are employed to engage them in an ongoing assessment process, which ideally leads to a more cohesive academic community. While the following examples represent some of the specific strategies used to promote a culture of assessment and continuous improvement at American Public University System (APUS), similar strategies can also be more generally implemented at other schools operating in an online environment.

Leverage Program Review as a Tool for Continuous Improvement

APUS conducts a system-wide interdepartmental program review of its academic programs, which brings together key stakeholders and members of the institution. The primary aim of a program review at APUS is to assess student learning and to address the questions: What does this program aim for students to learn? Are students achieving these aims through the program?

To address these questions, the objectives of a program review are to:

1. Assess and validate program content through benchmarking the program against similar programs at other universities, obtaining the feedback of outside evaluators, and through discussion with program advisory councils.
2. Evaluate student learning through review of graded end-of-course and program assignments, student evaluations, and learning outcomes assessments and data.
3. Build common ground across the University through engaging the community in the program review process by obtaining perspectives on all dimensions of the program including admissions, student services, faculty, enrollment management, marketing, institutional research, learning outcomes assessment, registrar, library and educational resources, faculty development, cross-department academic leadership, and others.
4. Provide strategic direction for continuous improvement of the program content, faculty, and learning experience through the completion of a strategic, three-year program development, enhancement, and implementation plan.

Promote Accountability through the Analysis of Student Learning Outcomes Data

The integration of student learning outcomes data into the program review process is a key part of the review, enabling internal and external reviewers to effectively evaluate program dimensions during each phase. Direct and indirect evidence of student achievement, coupled with information about other dimensions of program and institutional effectiveness,is integrated into the review process. This information includes, but is not limited to student learning data, standardized testing results, student satisfaction ratings, book costs, program growth, enrollment history, average class sizes, course completion rates, withdrawal rates, faculty credentials, and faculty performance metrics.

Inform Internal and External Stakeholders on the Achievement of Student Learning Outcomes

APUS’s Learning Outcomes Assessment website is used to inform external stakeholders on the achievement of outcomes. For example, this website provides national testing results, allowing the public to compare institution and national norm scores. In addition, the website provides information on assessment resources, learning outcomes, and updates on ongoing assessment initiatives.

Establish a Reciprocal Relationship among Stakeholders and the Institution on the Sharing of Data

As a fully online institution, we have the ability to collect and review an extensive amount of student data, which we recognize as integral to the continuous improvement of quality online programs. Our challenge is to create a balance between the need for accountability in teaching and learning, and stakeholder comfort with data collection.

We disseminate data and share information to stakeholders on several standard benchmarking instruments, including:

• National Survey of Student Engagement (NSSE): Measures how engaged students feel about the value of their education.
• The Community of Inquiry (COI): Measures the dimensions of social presence, cognitive presence, and teaching presence in the online classroom.
• Educational Testing Service (ETS) Major Field Test: Compares student achievement to national benchmarks to measure end-of-program knowledge in major fields of study.
• ETS Proficiency Profile: This measures how our graduating students compare nationally in areas such as quantitative reasoning and critical thinking.

Incorporate Assessment Data into Daily Decision-Making Processes

APUS regularly provides evidence to the public on the success of achieving outcomes, the soundness of the operation, and overall effectiveness of the institution. Similarly, the institution listens, involves, and responds to public feedback. We also use feedback from constituents for day-to-day activities and long-range endeavors. In addition to shaping the curriculum, student feedback and assessment data have led to a number of policies, procedures, systems, services, and features to accommodate the interests and needs of our constituents. This best practice promotes the continuous improvement of teaching and learning, and helps us to ensure academic and instructional quality at all levels of the institution.

About the Author

Dr. Jennifer Stephens Helm is Vice President and Dean of Institutional Research and Assessment at American Public University System. She has an extensive background in learning outcomes assessment where she has consulted with schools, school districts, state departments of education, and school reform organizations. She has taught at both K-12 and higher education levels. She has presented, and was published at, both national and international conferences in the areas of school reform, distance learning, faculty development, institutional research, and assessment. She holds a Ph.D. in Educational Psychology with an emphasis on learning, development, and instruction from Texas A&M University and a Bachelor of Arts degree in Psychology from the University of North Texas.