Abstract Details

User-Centred Game Based Learning: The Role Of Working Memory Performance During Multimodal Interaction

The study of education and instructional design within Game-Based Learning Environments (GBLEs) is an effective means in observing the impacts of learning motivation and cognitive performance on learning processes (Ang et al., 2007; Wen-Hao Huang, 2011). From such observations, GBLE guidelines aiming to reduce overall cognitive load and to mitigate potential constraints on learning processes continue to be developed (Reeves et al., 2004; Kiili, 2005; Turk, 2014). In this presentation, the authors will propose guidelines specifically focused on the influences of working memory (WM) within a multimodal context when delivering game-based educational content. Of particular interest is cross-modal interaction in WM between concurrent auditory stimuli and visual-based tasks. From this, the primary research question is concerned with assessing the value of ‘WM-aware’ guidelines in determining the design of auditory cues, and if such cues can be minimally disruptive to visual-based learning tasks in GBLE contexts.

Working memory (WM) is a brain system that is crucial to cognitive processing and learning mechanisms, and is largely reliant on multisensory integration (Baddeley, 1974; 2015). WM plays an integral role in learning processes, and is reliant on the use of multiple sensory modalities simultaneously. To facilitate learning, modern GBLEs therefore inherently employ multimodal content. In such contexts, the WM system enables end-users to perform various cognitive tasks, by providing a platform for the temporary storage and manipulation of auditory and visual content. Pre-attentive information processing occurs continuously within WM, both during banal, every-day activities, and during complex cognitive tasks. However, the WM system is put to work at full capacity during certain tasks that are fundamental to learning processes. Examples of these tasks include reading, writing or simple algebraic and visuo-spatial problem solving. The effectiveness of the GBLE is therefore reliant, in part, on the end-user’s WM performance and capacity. WM constraints, therefore, are a primary determining factor in the user’s/learner’s ability to interpret multiple streams of information simultaneously, or even one stream of information that is being handled by several modalities. In this presentation, the authors highlight the importance of WM within the context of GBLEs, and put forward design guidelines based on a systematic review of WM research literature

Game-based learning employs a number of interactive, multimodal techniques for delivering information, accepting input from users, and for notifying users about errors. Research in multimodal interaction and cognitive processing therefore have significant roles to play in the design of GBLEs. Effective human communication is often the central theme in multimodal interface design, with a focus on how concurrent multisensory perceptual systems function and interact. The goals of such research are often aimed at improving accuracy in digital systems through refining information presentation so that it is more reliably compatible with the human perceptual system. Early research in human communication and learning often focussed on speech intelligibility and language processing, (Sumby and Pollack, 1954), while more recent trends have highlighted multimodal interaction (Warschauer, 2007).

Furthermore, Huang (2011), highlights the importance of learning motivation within modern GBLEs that integrate multimodal interaction and feedback systems. In that paper, Huang suggests that excessive motivational support within GBLEs could potentially overwhelm a learner’s cognitive processing capacity, due to overly complex and highly interactive multimodal interfaces. A comprehensive review by Angadi and Reddy (2019) outlines analysis techniques for evaluating the user-experience (UX) in modern online multimodal applications, and details approaches in user-sentiment analysis during the presentation of multimodal content. In line with these approaches, the authors suggest integrating a mechanism for testing WM-aware guidelines facilitated through a form of user-feedback analysis during multimodal interaction. Such a mechanism would emphasise a personalised UX of a given GBLE – a feature that correlates with the highly individualised characteristics of WM itself.


  • Angadi, S. and Reddy, R.V.S., 2019. Survey on Sentiment Analysis from Affective Multimodal Content. In Smart Intelligent Computing and Applications (pp. 599-607). Springer, Singapore.
  • Ang, C.S., Zaphiris, P. and Mahmood, S., 2007. A model of cognitive loads in massively multiplayer online role playing games. Interacting with computers, 19(2), pp.167-179.
  • Baddeley, A.D. and Hitch, G., 1974. Working memory. In Psychology of learning and motivation (Vol. 8, pp. 47-89). Academic press.
  • Baddeley, A. D. (2015). “Working Memory” in A. Baddeley, Michael W. Eysenck & Mickael C. Anderson (Eds.), Memory, Ch. 3. Psychology Press, NY.
  • Huang, W.H., 2011. Evaluating learners’ motivational and cognitive processing in an online game-based learning environment. Computers in Human Behavior, 27(2), pp.694-704.
  • Kiili, K., 2005. Digital game-based learning: Towards an experiential gaming model. The Internet and higher education, 8(1), pp.13-24.
  • Reeves, L.M., Lai, J., Larson, J.A., Oviatt, S., Balaji, T.S., Buisine, S., Collings, P., Cohen, P., Kraal, B., Martin, J.C. and McTear, M., 2004. Guidelines for multimodal user interface design. Communications of the ACM, 47(1), pp.57-59.
  • Sumby, W.H. and Pollack, I., 1954. Visual contribution to speech intelligibility in noise. The journal of the acoustical society of america, 26(2), pp.212-215.
  • Turk, M., 2014. Multimodal interaction: A review. Pattern Recognition Letters, 36, pp.189-195.
  • Warschauer, M., 2007. The paradoxical future of digital learning. Learning Inquiry, 1(1), pp.41-49.

Back to list of abstracts