Assessing the Quality of Online Instruction: Integrating Instructional Quality and Web Usability Assessments

G. Walz,Janet E. Wall

Abstract:Chapter 49 in Measuring Up: Assessment Issues for Teachers, Counselors, and Administrators

Education,Computer Science

What problem does this paper attempt to address?

Case Study of Well Deigned Web-based Course

LI yan

DOI: https://doi.org/10.3969/j.issn.1672-0008.2011.01.013

2011-01-01

Abstract:In recent years,Chinese universities made a big step in construction of Web-based courses.However,such problems as gap between theory and practice were quite obvious in Chinese practice of Web-based course design.This study aims to use a case study to explore the key factors that characterize high quality Web-based course.The selected course,named <Advanced Methods in Distance Education>,was taught by Dr.Dooley,who is from Texas A&M University.Characteristic analysis of the course found that such key aspects as management platform,instructional design,interaction design,performance assessment,and learning support contribute to its success.
Effective Online Instructional and Assessment Strategies

Jorge Gaytan,Beryl C. McEwen

DOI: https://doi.org/10.1080/08923640701341653

2007-09-13

American Journal of Distance Education

Abstract:The purpose of this study was to better understand the instructional and assessment strategies that are most effective in the online learning environment. Faculty and students identified several strategies for maintaining instructional quality in the online environment, including the importance of using a variety of instructional methods to appeal to various learning styles and building an interactive and cohesive learning environment that includes group work. Online assessment strategies include having a wide variety of clearly explained assignments on a regular basis and providing meaningful and timely feedback to students regarding the quality of their work. Effective assessment techniques include projects, portfolios, self-assessments, peer evaluations, and weekly assignments with immediate feedback. The role of meaningful feedback cannot be overemphasized.

English Else
Quality Assessment of Online Graduate Curriculum:the Experience of the United States

Chuan-yi WANG,Shi-yu WANG,Hui-fen YAN

DOI: https://doi.org/10.3969/j.issn.2095-1663.2015.06.021

2015-01-01

Abstract:While it breaks the limitations of time and space,online graduate education may still have a problem in terms of course quality not up to face-to-face teaching.To achieve the aims of online education and ensure the quality of online curricu-lum,schools in the United States implement effective assessment procedures,including traditional evaluations,skill perform-ance assessments and e-portfolios-based assessments.
Identifying factors underlying the quality of online teaching effectiveness: An exploratory study

Arthur W. Bangert

DOI: https://doi.org/10.1007/bf03032699

2006-03-01

Journal of Computing in Higher Education

Abstract:TRADITIONALLY CAMPUS-BASED COURSES rely on student evaluations to provide instructors with feedback about their teaching effectiveness. However, current student evaluations of teaching instruments do not adequately assess many of the essential constructivist-based teaching practices recommended for quality online learning experiences. One of the best known summaries of research-based instructional practices is the widely disseminatedSeven Principles of Effective Teaching authored by Chickering and Gamson (1987). The majority of learner-centered instructional practices which comprise the Seven Principles framework are clearly focused on constructivist-based teaching practices. This study was an initial effort toward the development of a student evaluation of online teaching instrument based on the Seven Principles framework. Four hundred and eighty-nine students enrolled in WebCT courses at Montana State University completed the 26 item instrument. TheStudent Evaluation of Online Teaching Effectiveness (SEOTE) was found to be highly reliable and yielded four interpretable factors. The four factors were interpreted as Student-Faculty Interaction, Active Learning, Time on Task, and Cooperation Among Students.

education & educational research
Are they the same? Comparing the instructional quality of online and face‐to‐face graduate education courses

Andrew Topper

DOI: https://doi.org/10.1080/02602930601117233

2007-12-01

Abstract:As instructors adopt web‐based learning environments, they must consider how students’ evaluations of learning reflect the overall quality of instruction. Traditional course evaluations are used for faculty retention, tenure and promotion decisions, but also provide instructors with valuable information on the quality of their instruction. This study looks at response rates and compares instructional quality, using student course evaluations along with additional data from online and face‐to‐face graduate education courses, to evaluate the effectiveness of instruction. A statistical analysis of students’ course evaluations showed no significant difference in instructional quality based on the format used. Together with comparisons of student work, these results provide additional evidence in support of the finding of no significant difference between formats in the area of instructional quality.

education & educational research
Implementation Issues on the Quality Evaluation of Web-Based Courses and Digital Learning Resources

Xia Teng,Brandon Muramatsu,Jian Wei Zhang,Joseph G. Tront,Flora McMartin,Alice Agogino

DOI: https://doi.org/10.1007/978-3-540-27859-7_49

2004-01-01

Abstract:The emergence of the Web as a delivery mechanism for education has led a number of organizations to develop and implement quality evaluation criteria for digital learning resources and Web-based courses. The Chinese E-Learning Technology Standardization Committee is developing a specification for evaluating Web-based courses. This paper introduces the background on the standard, CELTS-22, and its guidelines for use. Further, this paper explores critical implementation issues through a case study of two similar evaluation criteria and systems that are used by NEEDS - A Digital Library for Engineering Education and MERLOT - the Multimedia Educational Resource for learning and Online Teaching in the United States.
COURSE ASSESSMENT PRACTICES AND STUDENT LEARNING STRATEGIES IN ONLINE COURSES

Bridget D. Arend

DOI: https://doi.org/10.24059/olj.v11i4.1712

2019-02-11

Online Learning

Abstract:Perhaps the most promising and understudied aspect of online education is course assessment. Course assessment is important because it has a strong impact on learning and is an indicator of the quality of learning occurring in a class. In the online environment, methods of assessment can be very different. However, the online education literature is currently lacking empirical data about the general status of assessment practices or how those practices relate to student learning. This article lays the groundwork for future studies by providing a description of formative and summative assessment and learning strategies in 60 online courses and suggesting some ways that assessment practices lead to different types of learning. In this study, instructors appear to follow effective practice by using multiple and alternativeassessment methods, dispersing grades over time, and providing timely and frequent feedback to students. Students report focusing on relatively more complex learning strategies, such as elaboration and critical thinking over rehearsal. However, online instructors need to ensure that assessments are used strategically and that feedback is productive and able to be acted upon by students.
Online Course Design

Sally J. Baldwin,Yu-Hui Ching

DOI: https://doi.org/10.19173/irrodl.v20i3.4283

2019-01-08

The International Review of Research in Open and Distributed Learning

Abstract:The evaluation of online courses is an important step in providing quality online courses. There are a variety of national and statewide evaluation tools used to help guide instructors and course designers of online courses (e.g., Quality Matters, OSCQR). This paper discusses a newly released course evaluation instrument from Canvas, the second largest learning management system (LMS) used by higher education institutions in the United States. The characteristics and unique features of the Canvas Course Evaluation Checklist (CCEC) are discussed. The CCEC is also compared to established national and statewide evaluation instruments. This review is helpful for those interested in online course design and developments in the field of online education.
Evaluating the quality of interaction in asynchronous discussion forums in fully online courses

Dip Nandi,Margaret Hamilton,James Harland

DOI: https://doi.org/10.1080/01587919.2012.667957

2012-05-01

Distance Education

Abstract:Fully online courses are becoming progressively more popular because of their “anytime anywhere” learning flexibility. One of the ways students interact with each other and with the instructors within fully online learning environments is via asynchronous discussion forums. However, student engagement in online discussion forums does not always take place automatically and there is a lack of clarity about the ideal role of the instructors in them. In this article, we report on our research on the quality of discussion in fully online courses through analysis of discussion forum activities. We have conducted our research on two large fully online subjects for computing students over two consecutive semesters and used a grounded theoretic approach for data analysis. Our results reveal what students and instructors consider as quality interaction in fully online courses. We also propose two frameworks based on our findings that can be used to ensure effective online interaction.

education & educational research
Assessing Quality of Teaching from Different Perspectives: Measurement Invariance across Teachers and Classes

Georg Krammer,Barbara Pflanzl,Gerlinde Lenske,Johannes Mayr

DOI: https://doi.org/10.1080/10627197.2020.1858785

2020-12-18

Educational Assessment

Abstract:Comparing teachers' self-assessment to classes' assessment of quality of teaching can offer insights for educational research and be a valuable resource for teachers' continuous professional development. However, the quality of teaching needs to be measured in the same way across perspectives for this comparison to be meaningful. We used data from 622 teachers self-assessing aspects of quality of teaching and of their classes (12229 students) assessing the same aspects. Perspectives were compared with measurement invariance analyses. Teachers and classes agreed on the average level of instructional clarity, and disagreed over teacher-student relationship and performance monitoring, suggesting that mean differences across perspectives may not be as consistent as the literature claims. Results showed a nonuniform measurement bias for only one item of instructional clarity, while measurement of the other aspects was directly comparable. We conclude the viability of comparing teachers' and classes' perspectives of aspects of quality of teaching.
Online vs. offline course evaluation revisited: testing the invariance of a course evaluation questionnaire using a multigroup confirmatory factor analysis framework

Ellen Laupper,Lars Balzer,Jean-Louis Berger

DOI: https://doi.org/10.1007/s11092-020-09336-6

2020-11-01

Abstract:Abstract Survey-based formats of assessing teaching quality in higher education are widely used and will likely continue to be used by higher education institutions around the world as various global trends contributing to their widespread use further evolve. Although the use of mobile devices for course evaluation continues to grow, there remain some unresolved aspects of the classic paper and web-based modes of evaluation. In the current study, the multigroup confirmatory factor analysis approach (MGCFA), an accepted methodological approach in general mixed-method survey research, was chosen to address some of the methodological issues when comparing these two evaluation modes. By randomly assigning one of the two modes to 33 continuing training courses at a Swiss higher education institution, this study tested whether the two different modes of assessing teaching quality yield the same results. The practical implications for course evaluation practice in institutions of higher education as well as the implications and limitations of the chosen methodological approach are discussed.
A Case Study of Usability Testing on an Asynchronous E-Learning Platform

Yuqing Guo,Jiazhen Wang,Joi Moore,Mingzhuo Liu,Hsin-Liang Chen

DOI: https://doi.org/10.1109/jcpc.2009.5420093

2009-01-01

Abstract:An effective and efficient e-Learning platform should hide systems' complexity and provide an easy and flexible interaction operation. Poor interface design of e-Learning platform is a main factor impacting on students' interactive with system and learning outcomes. The purpose of this study is to evaluate a non-degree e-Learning platform with Performance Analysis and User perception methods to investigate the system's efficiency, effectiveness and satisfaction. In this study, efficiency and effectiveness are evaluated by Time on Task and Task Completion score. Satisfaction is evaluated by Questionnaire and participants' feedback. Based on these data, this article analyzed the main interface design problem that frustrate learners and distract them from learning task, and given some advice for interface design in e-Learning platform.
Evaluation of Traditional Classroom Teaching Methods Versus Course Delivery Via the World Wide Web

M Ryan,K H Carlton,N S Ali,Marilyn Ryan,Kay Hodson Carlton,Nagia S Ali

DOI: https://doi.org/10.3928/0148-4834-19990901-08

1999-09-01

Journal of Nursing Education

Abstract:Higher education is moving with deliberate speed to an electronic classroom. Much has been published on faculty experiences with World Wide Web (WWW) course delivery. However, little research exists on the evaluation of these methods. The purpose of this study was to evaluate students' perceptions of two approaches to teaching: classroom and WWW modules. Classroom methods were rated significantly higher in relation to content, interaction, participation, faculty preparation, and communication. Technical skills were rated higher for WWW modules. Critical thinking and time allotted for assignments were not significantly different between classroom and WWW instruction. Open-ended comments were rich and supported both positive and negative aspects of classroom and WWW-based modules. Implications call for creativity in course development, course redesign and orientation, active communication with students, support for technical problems, faculty development, and university-wide planning through partnerships.

nursing
Measuring the Effectiveness of Learning Resources Via Student Interaction with Online Learning Modules

Zhongzhou Chen,Matthew Guthrie

DOI: https://doi.org/10.48550/arXiv.1903.08003

2019-03-19

Abstract:We present a new method for measuring the effectiveness of online learning resources, through the analysis of time-stamped log data of students' interaction with a sequence of online learning modules created based on the concept of mastery learning. Each module was designed to assess students' mastery of one topic before and after interacting with the learning resources in the module. In addition, analysis of log data provides information on students' test-taking effort when completing the assessment, and learning effort when interacting with the learning resources. Combining these three measurements provides accurate information on the quality of each online learning module, as well as detailed suggestions for future improvements. The results from data collected from all 10 modules are presented in a sequence of sunburst charts, an intuitive visual representation designed to allow the average instructor to quickly grasp the key outcomes of the data analysis, and identify less effective modules for future improvements. Online learning modules can be implemented much more frequently than either clinical experiments or classroom assessments, and can provide more interpretable data than most existing online courses, making it a valuable tool for quickly assessing the effectiveness of online learning resources.

Physics Education
On the Evaluating Standard for the Implementations of Web-Based Courses in Higher Education

陈刚,张建伟,李海霞

DOI: https://doi.org/10.3969/j.issn.1001-4519.2003.05.018

2003-01-01

Abstract:This article proposes a standard for evaluating the implementations of web - based courses. The standard includes fifteen items falling into four basie dimensions: learning resources, instructors, learning and teaching processes, and quality monitoring. This standard can be used to help the quality assurance and evaluation of web-based courses.
Perceptions of instructional quality: impact on acceptance and use of an online learning environment

Charlotte Larmuseau,Piet Desmet,Fien Depaepe

DOI: https://doi.org/10.1080/10494820.2018.1509874

2018-08-13

Interactive Learning Environments

Abstract:Given the growing use of online learning environments in higher education, it is important to further unravel how students’ use is influenced by their perceptions towards these learning environments. This study includes the perceived quality of the instructional design based on the First Principles of Instruction of Merrill and students’ acceptance based on the constructs perceived usefulness and perceived ease of use of the technology acceptance model (TAM). The aim of this study is twofold: a first aim is to investigate the influence of the perceived instructional quality on students’ acceptance and the second aim is to investigate the impact of students’ acceptance and the perceived instructional quality on the quantity (i.e. course activity) and quality (i.e. course performance) of use. In this study, a Moodle-based online learning environment for learning French as a foreign language was studied. Participants were 161 university students. Structural equation modeling (SEM) indicates that the perceived instructional quality has a significant positive influence on students’ acceptance. Furthermore, students’ perceived instructional quality has a positive influence on the quality, but not on the quantity of use, whereas students’ acceptance of the online learning environment has no impact on the use of the learning environment.

education & educational research
In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs)

Patrick Lowenthal,Charles Hodges

DOI: https://doi.org/10.19173/irrodl.v16i5.2348

2015-09-29

The International Review of Research in Open and Distributed Learning

Abstract:The concept of the massive, open, online course (MOOC) is not new, but high-profile initiatives have moved MOOCs into the forefront of higher education news over the past few years. Members of institutions of higher education have mixed feelings about MOOCs, ranging from those who want to offer college credit for the successful completion of MOOCs to those who fear MOOCs are the end of the university as we know it. We set forth to investigate the quality of MOOCs by using the Quality Matters quality control framework. In this article, we present the results of our inquiry, with a specific focus on the implications the results have on day-to-day practice of designing online courses.
Toward Measuring Instructional Interactions “At-Scale”

Lindsay Clare Matsumura,Helen E. Garnier,Sharon Cadman Slater,Melissa D. Boston

DOI: https://doi.org/10.1080/10627190802602541

2008-12-15

Educational Assessment

Abstract:This study explores two approaches to directly measuring the quality of instruction: teachers' assignments with student work and focused lesson observations. The technical quality and potential feasibility of these approaches for measuring instruction in large numbers of classrooms are compared within two different content areas (reading comprehension and mathematics). Generalizability and decision studies determined the optimal number of observations and assignments needed to obtain a reliable measure of a teacher's practice, and the association of these direct measures of instructional quality with student achievement was estimated. For both content areas, four assignments assessed by two raters yielded a reliable estimate of quality and as few as two observations yielded a reliable estimate of quality when teachers complied with the requirements of the research. The quality of observed instruction and teachers' assignments differentially predicted gains in students' achievement on the Stanford Achievement Test within each content area. The implications for measuring instruction “at-scale” in different content areas are discussed.

English Else
How do online course design features influence student performance?

Shanna Smith Jaggars,Di Xu

DOI: https://doi.org/10.1016/j.compedu.2016.01.014

2016-04-01

Abstract:Given the rapid growth in online coursework within higher education, it is important to establish and validate quality standards for these courses. While many online learning quality rubrics do exist, thus far there has been little empirical evidence establishing a clear link between specific course design features and concrete, student-level course outcomes. In the current study, the authors develop an online course design assessment rubric that includes four areas, and explore the impact of each area on student end-of-semester performance in 23 online courses at two community colleges. The results indicate that the quality of interpersonal interaction within a course relates positively and significantly to student grades. Additional analyses based on course observation and interview data suggest that frequent and effective student–instructor interaction creates an online environment that encourages students to commit themselves to the course and perform at a stronger academic level.

education & educational research,computer science, interdisciplinary applications
Linking Assessment for Learning, Improvement and Accountability

Hamish Coates,Tricia Seifert

DOI: https://doi.org/10.1080/13538322.2011.554308

2011-01-01

Quality in Higher Education

Abstract:Assessing student learning in ways that offer students, institutions and systems with sound information is a considerable challenge for higher education. Despite demonstrable progress, a disconnect remains between methods used for assessing learning, improvement and accountability. This muddled situation is illustrated through a review of contemporary practice in the USA, Canada, Australia, the UK and New Zealand. To advance research and practice, this paper contributes a structured set of criteria that practitioners can use to evaluate whether an assessment functions effectively across different levels of practice. These technical and practical criteria touch on issues such as sampling, the viability of aggregation, reporting and cost-benefit considerations. The paper tests these criteria by considering their capacity to drive multi-level improvement in education quality. It is contended that the criteria provide perspective and structure for reconciling the quantitative assessment of higher education quality at different levels of practice.

Assessing the Quality of Online Instruction: Integrating Instructional Quality and Web Usability Assessments

Case Study of Well Deigned Web-based Course

Effective Online Instructional and Assessment Strategies

Quality Assessment of Online Graduate Curriculum:the Experience of the United States

Identifying factors underlying the quality of online teaching effectiveness: An exploratory study

Are they the same? Comparing the instructional quality of online and face‐to‐face graduate education courses

Implementation Issues on the Quality Evaluation of Web-Based Courses and Digital Learning Resources

COURSE ASSESSMENT PRACTICES AND STUDENT LEARNING STRATEGIES IN ONLINE COURSES

Online Course Design

Evaluating the quality of interaction in asynchronous discussion forums in fully online courses

Assessing Quality of Teaching from Different Perspectives: Measurement Invariance across Teachers and Classes

Online vs. offline course evaluation revisited: testing the invariance of a course evaluation questionnaire using a multigroup confirmatory factor analysis framework

A Case Study of Usability Testing on an Asynchronous E-Learning Platform

Evaluation of Traditional Classroom Teaching Methods Versus Course Delivery Via the World Wide Web

Measuring the Effectiveness of Learning Resources Via Student Interaction with Online Learning Modules

On the Evaluating Standard for the Implementations of Web-Based Courses in Higher Education

Perceptions of instructional quality: impact on acceptance and use of an online learning environment

In Search of Quality: Using Quality Matters to Analyze the Quality of Massive, Open, Online Courses (MOOCs)

Toward Measuring Instructional Interactions “At-Scale”

How do online course design features influence student performance?

Linking Assessment for Learning, Improvement and Accountability