Modes of Meaning

Multimodal Media & 4E+ Cognition in Tech-Enhanced Learning

Abstract

This presentation proposes an approach to designing technology-enhanced learning (TEL) through the strategic integration of diverse multimodal media forms within a framework informed by the 4E+ view of cognition. The 4E+ cognition framework emphasises the embodied, embedded, enactive, and extended nature of cognition, suggesting that cognition is not solely confined to the brain but extends into the environment while involving the body's interactions with that environment (Carney, 2020; Jianhui , 2019; Menary, 2010; Newen, et al., 2018).  

In this theoretical context, our study explores how the combination of various modes of media, such as immersive technologies, digital interactive elements, real-world analogue creations, audio, sound, images, videos, animations, text, and the surrounding environment can be orchestrated to create sensorially rich, and more meaningful learning experiences (Gilakjani, et al., 2011; Philippe, et al., 2020; Sankey, et al., 2010). For example, mixed reality (XR) learning design combines immersive media forms to support multi-sensory and expanded cognitive learning (Philippe et al., 2020; Rakkolainen et al., 2021; Villalobos & Videla, 2023). Other relevant approaches include gamification and transmedia storytelling methods (Doumanis et al., 2019; Perry, 2020). By leveraging different modalities, educators can design learning materials that engage learners with different sensory activations and presentation methods (Bouchey et al., 2021). This approach can cater to the 4E+ view of cognition, and subsequently enhancing knowledge acquisition and retention.

Examples from our own practice and research (such as the Explora: Chile es Mar, Pipi’s World and O-Tū-Kapua XR learning experiences), as well as current educational examples (Bouchey et al., 2021; Philippe, et al., 2020), demonstrate how multimodal media integration facilitates deeper engagement, critical thinking, and a more holistic understanding of complex concepts. Furthermore, we discuss practical strategies for educators to implement these principles in their TEL design, highlighting the potential of aligning multimodal design choices with the 4E+ cognitive framework. 

Ultimately, we advocate for a shift towards a more inclusive and effective approach to technology-enhanced learning - one that embraces the diversity of human cognitive processes and leverages multimodal media to communicate meaningful knowledge in ways that resonate with learners' cognitive structures and experiences. Multimodal methods, when aligned with the distributed 4E+ view of cognition, can make TEL appeal and resonate on deeper levels to engage across various sensory, environmental and communication modes. This type of approach acknowledges the diversity of ways that humans process and understand phenomena, and how more effective learning can occur when multiple ways of knowing are engaged and communicated to. Furthermore, through this method, inclusivity can be heightened for students with diverse cultural, neurological or other backgrounds (Anis & Khan, 2023; Boivin & CohenMiller, 2022).  

Emerging research shows the potential of the 4E+ approach to meet the needs of learning in 21^st century technological environments (Videla & Veloz, 2023; Villalobos & Videla, 2023). This presentation contributes to the literature by examining TEL design through a multimodal media lens. It highlights how the holistic 4E+ framework can more effectively and meaningfully engage students than computational, monomodal and bimodal uses of technology in educational settings.

References

Anis, M., & Khan, R. (2023). Integrating Multimodal Approaches in English Language Teaching for Inclusive Education: A Pedagogical Exploration.

Boivin, A. C. N., & CohenMiller, A. (2022). INCLUSION AND EQUITY WITH MULTIMODALITY DURING COVID-19. Keep Calm, Teach On: Education Responding to a Pandemic, 87.

Bouchey, B., Castek, J., & Thygeson, J. (2021). Multimodal learning. Innovative Learning Environments in STEM Higher Education: Opportunities, Challenges, and Looking Forward, 35-54.

Carney, J. (2020). Thinking avant la lettre: A Review of 4E Cognition. Evolutionary studies in imaginative culture, 4(1), 77-90. 

Doumanis, I., Economou, D., Sim, G. R., & Porter, S. (2019). The impact of multimodal collaborative virtual environments on learning: A gamified online debate. Computers & Education, 130, 121-138.

Gilakjani, A. P., Ismail, H. N., & Ahmadi, S. M. (2011). The effect of multimodal learning models on language teaching and learning. Theory & Practice in Language Studies, 1(10).

Jianhui, L. (2019). Transcranial Theory of Mind: A New Revolution of Cognitive Science. International Journal of Philosophy, 7(2), 66-71.

Menary, R. (2010). Introduction to the special issue on 4E cognition. Phenomenology and the Cognitive Sciences, 9, 459-463.

Newen, A., De Bruin, L., & Gallagher, S. (Eds.). (2018). The Oxford handbook of 4E cognition. Oxford University Press.

Perry, M. S. (2020). Multimodal Engagement through a Transmedia Storytelling Project for Undergraduate Students. Gema Online Journal of Language Studies, 20(3).

Philippe, S., Souchet, A. D., Lameras, P., Petridis, P., Caporal, J., Coldeboeuf, G., & Duzan, H. (2020). Multimodal teaching, learning and training in virtual reality: a review and case study. Virtual Reality & Intelligent Hardware, 2(5), 421-442.

Rakkolainen, I., Farooq, A., Kangas, J., Hakulinen, J., Rantala, J., Turunen, M., & Raisamo, R. (2021). Technologies for multimodal interaction in extended reality—a scoping review. Multimodal Technologies and Interaction, 5(12), 81.

Sankey, M., Birch, D., & Gardiner, M. W. (2010). Engaging students through multimodal learning environments: The journey continues. Proceedings of the 27th Australasian Society for Computers in Learning in Tertiary Education, 852-863.

Videla, R., & Veloz, T. (2023). The 4E approach applied to education in the 21st century. Constructivist Foundations, 18(2), 153-157.

Villalobos, M., & Videla, R. (2023). The roots and blossoms of 4E cognition in Chile: Introduction to the Special Issue on 4E cognition in Chile. Adaptive Behavior, 31(5), 397-404.