A framework for evaluating the user experience of digital moderation systems in the South African secondary school environment

Abstract

The continued emphasis on education quality amidst the accelerating pace of technological developments, which create new opportunities, expectations, and challenges in the teaching and learning environment, requires evidence-based, robust, regulatory frameworks for monitoring standards. Innovative and dynamic approaches are required to quality assure assessment processes (moderation). The reviewed literature provided scant evidence of theorization on the concept of digital moderation (eModeration) and little empirical evidence on systems used in secondary schools. This deficiency in the literature in terms of digital moderation is problematic since it leaves educators, managers, and researchers without evidence-based guidance on how eModeration systems should be designed or evaluated. This deficit provides the rationale for an investigation into the components of a framework to evaluate the user experience of an eModeration system. This study draws on the extant eModeration literature and theories of technology acceptance, Information Systems success (IS) models, and constructs from the field of Human-Computer Interaction (HCI) to create a theoretical framework that integrates constructs identified from the different literature streams to evaluate an eModeration system's user experience. A Design Science Research (DSR) approach guided the design, development, and evaluation of an eModeration evaluation framework. A Participatory Action Design Research (PADRE) approach was used to position the user within the iterative DSR cycles as a means of knowledge acquisition. Participatory Design (PD) was positioned as a data collection strategy during requirements gathering and the generation of design ideas for an eModeration prototype system. Qualitative and quantitative data collection was used to record perceptions of individuals interacting with the prototype. The theoretical contribution is the literature-based framework underpinning this study. This theoretical framework was used as input in determining the components of an eModeration evaluation framework, which is the main contribution of this study. This research has practical value in guiding the design of evaluation criteria for the user experience of an eModeration system for implementation in secondary schools.