Computer Supported Education: 15th International Conference, CSEDU 2023, Prague, Czech Republic, April 21–23, 2023, Revised Selected Papers (Communications in Computer and Information Science) 303153655X, 9783031536557

Table of contents :
Preface
Organization
Contents
Towards a Software Architecture to Provide Hybrid Recommendations for Smart Campuses
1 Introduction
2 Smart Campus
3 Recommendation Systems
3.1 Collaborative Filtering
3.2 Content-Based Filtering
3.3 Hybrid Filtering
3.4 Contextual-Sensitive Filtering
3.5 Knowledge-Based Filtering
3.6 Ontology-Based Filtering
4 Hybrid Recommender System for Personalized Recommendations
4.1 The SmartC Software Architecture
4.2 Content-Based Recommender
4.3 Collaborative Recommender
5 Results
5.1 Developed Prototypes
5.2 Evaluation of the Recommendation Algorithm
6 Conclusion
References
Encouraging Grading: Per Aspera Ad A-Stars
1 Introduction
2 Related Work
2.1 Growth Mindset
2.2 Formative Assessment
2.3 Types of Feedback, Focus on the Process
2.4 Struggling Students
3 Research Context
3.1 Tools Used: Plussa, OpenDSA and Gitlab
3.2 Auto-Graders for Assignment
3.3 Method and Data Collection
4 Results and Discussion
4.1 Auto Grading
4.2 Manual Grading
4.3 Peer-Reviews
4.4 Comparisons
4.5 Learning Analytics
4.6 Fairness Aspects
4.7 Most Useful Graders
4.8 Submission Count
4.9 Growing with Each Submission and Grade
5 Conclusions
6 Further Studies
References
An Approach for Mapping Declarative Knowledge Training Task Types to Gameplay Categories
1 Introduction
2 Elements for Training Game Activities
2.1 Task Types for DK Training
2.2 Gameplay Categories for Dungeon-Like Games
3 Activity Generation: A Mapping Need
3.1 Research Question
3.2 Related Work
3.3 Research Positioning and Objectives
4 Mapping Approach Development
4.1 Identification of the Pivot
4.2 Mapping Task Types onto Gameplay Categories
5 A Systematic Mapping Approach
5.1 Proposed Mapping Approach
5.2 Relations Between Task Types and Gameplay Categories
5.3 Evaluation of the Relations
6 A Formal Modelling of the Mappings
7 Conclusion and Perspectives
References
Designing Declarative Knowledge Training Games: An Analysis Framework Based on the Roguelite Genre
1 Introduction
2 Research Context: The AdapTABLES Project
2.1 Declarative Knowledge Training
2.2 Different Tasks Objectives and Parameters
3 A Relevant Game Genre for Training Purposes
3.1 The Roguelite Genre
3.2 Adequacy of Declarative Knowledge Training with Roguelite Genre
3.3 Targeted Adaptations
3.4 Research Question
4 State of the Art
5 Analysis Framework for a Roguelite Learning Game
6 Framework Application: AdapTABLES Project
6.1 First Analysis
6.2 First Prototype
6.3 Experiment Feedback
6.4 Second Analysis
6.5 Second Prototype
7 Conclusion
References
Cultivating Higher Order Competencies: Complex Thinking in Latin American University Context
1 Introduction
2 Perspectives from Related Works
3 Methodology
4 Results
5 Discussion and Conclusion
References
Correlation Among Competences Involved in Digital Problem-Solving Activities with Upper Secondary School Students
1 Introduction
2 Theoretical Background
2.1 Problem Solving with an ACE
2.2 The Digital Math Training Project
3 Research Questions
4 Methodology
5 Results
5.1 Analysis of Case Study 1
5.2 Analysis of Case Study 2
5.3 Analysis of the Evaluations of All Students
6 Analysis of Students’ Answers to the Final Questionnaire
7 Conclusions
References
Regulatory Strategies for Novice Programming Students
1 Introduction
2 Literature Review
3 Research Method
3.1 Procedures
3.2 Participants
3.3 Instruments
4 Results
4.1 Validity and Reliability of the Questionnaire Applied to the Students
4.2 Students Perceptions of Their Use of Regulatory Strategies
4.3 Evidence-Based Scripts for Students Regulation
5 Discussions
6 Conclusions
References
From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-Choice Questions for Programming Classes in Higher Education
1 Introduction
2 Background
2.1 Motivation
2.2 Related Work
3 Dataset
4 Experiments
4.1 Models
4.2 Experimental Design
5 Results
5.1 Quantitative Analysis
5.2 Qualitative Analysis
6 Implications for Teaching Practice
7 Limitations and Threats to Validity
8 Conclusions and Future Work
References
Author Index

Citation preview

Bruce M. McLaren James Uhomoibhi Jelena Jovanovic Irene-Angelica Chounta (Eds.)

Communications in Computer and Information Science

2052

Computer Supported Education 15th International Conference, CSEDU 2023 Prague, Czech Republic, April 21–23, 2023 Revised Selected Papers

Communications in Computer and Information Science Editorial Board Members Joaquim Filipe , Polytechnic Institute of Setúbal, Setúbal, Portugal Ashish Ghosh , Indian Statistical Institute, Kolkata, India Raquel Oliveira Prates , Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil Lizhu Zhou, Tsinghua University, Beijing, China

2052

Rationale The CCIS series is devoted to the publication of proceedings of computer science conferences. Its aim is to efficiently disseminate original research results in informatics in printed and electronic form. While the focus is on publication of peer-reviewed full papers presenting mature work, inclusion of reviewed short papers reporting on work in progress is welcome, too. Besides globally relevant meetings with internationally representative program committees guaranteeing a strict peer-reviewing and paper selection process, conferences run by societies or of high regional or national relevance are also considered for publication. Topics The topical scope of CCIS spans the entire spectrum of informatics ranging from foundational topics in the theory of computing to information and communications science and technology and a broad variety of interdisciplinary application fields. Information for Volume Editors and Authors Publication in CCIS is free of charge. No royalties are paid, however, we offer registered conference participants temporary free access to the online version of the conference proceedings on SpringerLink (http://link.springer.com) by means of an http referrer from the conference website and/or a number of complimentary printed copies, as specified in the official acceptance email of the event. CCIS proceedings can be published in time for distribution at conferences or as postproceedings, and delivered in the form of printed books and/or electronically as USBs and/or e-content licenses for accessing proceedings at SpringerLink. Furthermore, CCIS proceedings are included in the CCIS electronic book series hosted in the SpringerLink digital library at http://link.springer.com/bookseries/7899. Conferences publishing in CCIS are allowed to use Online Conference Service (OCS) for managing the whole proceedings lifecycle (from submission and reviewing to preparing for publication) free of charge. Publication process The language of publication is exclusively English. Authors publishing in CCIS have to sign the Springer CCIS copyright transfer form, however, they are free to use their material published in CCIS for substantially changed, more elaborate subsequent publications elsewhere. For the preparation of the camera-ready papers/files, authors have to strictly adhere to the Springer CCIS Authors’ Instructions and are strongly encouraged to use the CCIS LaTeX style files or templates. Abstracting/Indexing CCIS is abstracted/indexed in DBLP, Google Scholar, EI-Compendex, Mathematical Reviews, SCImago, Scopus. CCIS volumes are also submitted for the inclusion in ISI Proceedings. How to start To start the evaluation of your proposal for inclusion in the CCIS series, please send an e-mail to [email protected].

Bruce M. McLaren · James Uhomoibhi · Jelena Jovanovic · Irene-Angelica Chounta Editors

Computer Supported Education 15th International Conference, CSEDU 2023 Prague, Czech Republic, April 21–23, 2023 Revised Selected Papers

Editors Bruce M. McLaren Carnegie Mellon University Pittsburgh, PA, USA

James Uhomoibhi University of Ulster Newtownabbey, UK

Jelena Jovanovic University of Belgrade Belgrade, Serbia

Irene-Angelica Chounta University of Duisburg-Essen Duisburg, Germany

ISSN 1865-0929 ISSN 1865-0937 (electronic) Communications in Computer and Information Science ISBN 978-3-031-53655-7 ISBN 978-3-031-53656-4 (eBook) https://doi.org/10.1007/978-3-031-53656-4 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Paper in this product is recyclable.

Preface

The present book includes extended and revised versions of a set of selected papers from the 15th International Conference on Computer Supported Education (CSEDU 2023), held in Prague, Czech Republic, from 21 to 23 April, 2023. CSEDU 2023 received 162 paper submissions from 42 countries, of which 5% were included in this book. The papers were selected by the event chairs and their selection was based on a number of criteria that included the evaluations and comments provided by the program committee members, the session chairs’ assessment of the paper presentation, as well as the program chairs’ overall insight into all the papers included in the technical program. The authors of selected papers were invited to submit revised and extended versions of their papers having at least 30% innovative material. CSEDU, the International Conference on Computer Supported Education, is a yearly meeting place for presenting and discussing new educational tools and environments, best practices and case studies on innovative technology-based learning strategies, and institutional policies on computer supported education including open and distance education. CSEDU provides an overview of current technologies as well as upcoming trends, and promotes discussion about the pedagogical potential of new educational technologies in the academic and corporate world. CSEDU seeks papers and posters describing educational technology research; academic or business case-studies; or advanced prototypes, systems, tools, and techniques. The papers included in this book contribute to the understanding of relevant trends in the current research on Computer Supported Education, with a particular focus on: eLearning Platforms, Portals, Feedback and Learning Support, Game-Based and Simulation-Based Learning, Active Learning, Tools to Assess Learning, Learning with AI Systems, Higher-Order Thinking Skills, Flipped Classroom, Faculty Development, and Constructivism and Social Constructivism. We would like to thank all the authors for their contributions as well as the reviewers whose thoughtful evaluations and comments helped to ensure the quality of this publication. April 2023

Bruce M. McLaren James Uhomoibhi Jelena Jovanovic Irene-Angelica Chounta

Organization

Conference Co-chairs James Uhomoibhi Bruce McLaren

Ulster University, UK Carnegie Mellon University, USA

Program Co-chairs Jelena Jovanovic Irene-Angelica Chounta

University of Belgrade, Serbia University of Duisburg-Essen, Germany

Program Committee Eleftheria Alexandri António Andrade Francisco Arcega Juan Ignacio Asensio Breno Azevedo Ralph-Johan Back Jorge Barbosa João Barros Patrícia Bassani Zane Berge Jesús Berrocoso Huseyin Bicen Andreas Bollin Curtis J. Bonk Ivana Bosnic Federico Botella Karima Boussaha Patrice Bouvier Krysia Broda Manuel Caeiro Rodríguez Renza Campagni Pasquina Campanella

Hellenic Open University, Greece Universidade Católica Portuguesa, Portugal Universidad de Zaragoza, Spain University of Valladolid, Spain Instituto Federal de Educação, Ciência e Tecnologia Fluminense, Brazil Åbo Akademi University, Finland UNISINOS, Brazil Polytechnic Institute of Beja, Portugal Universidade Feevale, Brazil University of Maryland, Baltimore County, USA University of Extremadura, Spain Near East University, Cyprus Klagenfurt University, Austria Indiana University Bloomington, USA University of Zagreb, Croatia Miguel Hernandez University of Elche, Spain University of Oum El Bouaghi, Algeria LDLC VR Studio, France Imperial College London, UK University of Vigo, Spain Università di Firenze, Italy University of Bari “Aldo Moro”, Italy

viii

Organization

Sanja Candrlic Guanliang Chen António Coelho Manuel Perez Cota John Cuthell Rogério da Silva Sergiu Dascalu Luis De-La-fuente-valentín Christian Della Nicoletta Di Blas Tania Di Mascio Yannis Dimitriadis Danail Dochev

Toby Dragon Nour El Mawas Gijsbert Erkens Larbi Esmahi João Esteves Ramon Fabregat Gesa Si Fan Richard Ferdig Débora Nice Ferrari Barbosa Giuseppe Fiorentino Francisco García Peñalvo Isabela Gasparini Henrique Gil Apostolos Gkamas Anabela Gomes Maria João Gomes Ana González Marcos Christiane Gresse von Wangenheim Christian Guetl David Guralnick Roger Hadgraft Thorsten Händler Peter Hastings

University of Rijeka, Croatia Monash University, Australia Universidade do Porto, Portugal Universidade de Vigo, Spain Virtual Learning, UK University of Leicester, UK University of Nevada, Reno, USA Universidad Internacional de la Rioja, Spain University of Glasgow Singapore, Singapore Politecnico di Milano, Italy University of L’Aquila, Italy University of Valladolid, Spain Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Bulgaria Ithaca College, USA Université de Lorraine, France Utrecht University, The Netherlands Athabasca University, Canada University of Minho, Portugal Universitat de Girona, Spain University of Tasmania, Australia Kent State University, USA Feevale University, Brazil University of Pisa, Italy Salamanca University, Spain Universidade do Estado de Santa Catarina, Brazil Instituto Politécnico de Castelo Branco, Portugal University Ecclesiastical Academy of Vella of Ioannina, Greece Instituto Superior de Engenharia de Coimbra (Coimbra Polytechnic - ISEC), Portugal Universidade do Minho, Portugal Universidad de la Rioja, Spain Federal University of Santa Catarina, Brazil Graz University of Technology, Austria Kaleidoscope Learning, USA University of Technology Sydney, Australia Ferdinand Porsche Mobile University of Applied Sciences (FERNFH), Austria DePaul University, USA

Organization

Antonio Hervás Jorge Tomayess Issa Ivan Ivanov Malinka Ivanova M. J. C. S. Reis Stéphanie Jean-Daubias M.-Carmen Juan Michail Kalogiannakis Charalampos Karagiannidis Ilias Karasavvidis Vasilis Komis Vitomir Kovanovic Lam-For Kwok Eitel Lauría Borislav Lazarov Chien-Sing Lee Marie Lefevre Andreas Lingnau Luca Andrea Ludovico Veronika Makarova Ivana Marenzi Verónica Marín Lindsay Marshall Scheila Martins Bruce Maxim Madeth May Elvis Mazzoni Marco Mesiti José Carlos Metrôlho Laurent Moccozet Rafael Morales Gamboa António Moreira Jerzy Moscinski Maria Moundridou Antao Moura Antoanela Naaji Tomohiro Nagashima Minoru Nakayama

ix

Universitat Politècnica de València, Spain Curtin University, Australia SUNY Empire State University, USA Technical University of Sofia, Bulgaria University of Trás-os-Montes e Alto Douro, Portugal Université Claude Bernard Lyon 1, LIRIS, France Universitat Politècnica de València, Spain University of Crete, Greece University of Thessaly, Greece University of Thessaly, Greece University of Patras, Greece University of South Australia, Australia HKCT Institute of Higher Education, China Marist College, USA Institute of Mathematics and Informatics, Bulgarian Academy of Sciences, Bulgaria Sunway University, Malaysia Université Claude Bernard Lyon 1, France German University of Applied Sciences, Germany Università degli Studi di Milano, Italy University of Saskatchewan, Canada Leibniz University Hannover, Germany University of Córdoba, Spain Newcastle University, UK Arden University, UK University of Michigan-Dearborn, USA Le Mans Université, France University of Bologna, Italy University of Milano, Italy Instituto Politécnico de Castelo Branco, Portugal University of Geneva, Switzerland University of Guadalajara, Mexico Universidade de Aveiro, Portugal Silesian University of Technology, Poland School of Pedagogical and Technological Education (ASPETE), Greece Federal Universisty of Campina Grande, Brazil Vasile Goldis Western University of Arad, Romania Saarland University, Germany Tokyo Institute of Technology, Japan

x

Organization

Mai Neo Omid Noroozi Fátima Nunes Dade Nurjanah Ebba Ossiannilsson José Palma Paolo Paolini Stamatios Papadakis Kyparisia Papanikolaou Abelardo Pardo Emanuel Peres Paula Peres Donatella Persico Andreas Pester Elvira Popescu Francesca Pozzi Augustin Prodan Yannis Psaromiligkos Fernando Ribeiro Marco Ronchetti Leon Rothkrantz Razvan Rughinis Jesus Salinas Juan M. Santos Georg Schneider Wolfgang Schreiner Maria Serna Sabine Seufert Ramesh C. Sharma Pei Hwa Siew Jane Sinclair Eliza Stefanova Claudia Steinberger Katsuaki Suzuki Nestori Syynimaa Qing Tan Seng Chee Tan

Multimedia University, Malaysia Wageningen University and Research, The Netherlands Universidade de São Paulo, Brazil Telkom University, Indonesia Swedish Association for Distance Education, Sweden Escola Superior de Tecnologia de Setúbal, Portugal Politecnico di Milano, Italy University of Crete, Greece School of Educational and Technological Education (ASPETE), Greece University of South Australia, Australia University of Trás-os-Montes e Alto Douro/Inesc-Tec, Portugal ISCAP, Portugal CNR - Italian National Research Council, Italy The British University in Egypt, Egypt University of Craiova, Romania CNR - Italian National Research Council, Italy Iuliu Hatieganu University, Romania University of West Attica, Greece Instituto Politécnico de Castelo Branco, Portugal University of Trento, Italy Delft University of Technology, The Netherlands University “Politehnica” of Bucharest, Romania University of the Balearic Islands, Spain University of Vigo, Spain Trier University of Applied Sciences, Germany Johannes Kepler University Linz, Austria BarcelonaTech, Spain University of St. Gallen, Switzerland Dr. B. R. Ambedkar University Delhi, India Universiti Tunku Abdul Rahman, Malaysia University of Warwick, UK Sofia University, Bulgaria University of Klagenfurt, Austria Kumamoto University, Japan University of Jyväskylä, Finland Athabasca University, Canada National Institute of Education, Nanyang Technological University, Singapore

Organization

Steven Tanimoto Dirk Tempelaar Uwe Terton Aristides Vagelatos Michael Vallance Leo van Moergestel Carlos Vaz de Carvalho Giuliano Vivanet Aurora Vizcaino Alf Wang Stelios Xinogalos Diego Zapata-Rivera Thomas Zarouchas Meina Zhu

University of Washington, USA Maastricht University School of Business and Economics, The Netherlands Southern Cross University, Australia Computer Technology Institute, Greece Future University Hakodate, Japan HU University of Applied Sciences Utrecht, The Netherlands ISEP, Portugal University of Cagliari, Italy Escuela Superior de Informatica, Spain Norwegian University of Science and Technology, Norway University of Macedonia, Greece Educational Testing Service, USA Computer Technology Institute and Press “Diophantus”, Greece Wayne State University, USA

Additional Reviewers Leila Bergamasco Federica Caruso Vinh Le Alwyn Lee Rafael Testa

Centro Universitário FEI, Brazil University of L’Aquila, Italy University of Nevada, Reno, USA National Institute of Education, Nanyang Technological University, Singapore University of São Paulo, Brazil

Invited Speakers Art Graesser Barbara Wasson Vania Dimitrova

xi

University of Memphis, USA University of Bergen, Norway University of Leeds, UK

Contents

Towards a Software Architecture to Provide Hybrid Recommendations for Smart Campuses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Martin Hideki Mensch Maruyama, Luan Willig Silveira, Ana Paula Militz Dorneles, Gabriel Vieira Casanova, Renan Bordignon Poy, Elvandi da Silva Júnior, José Palazzo M. de Oliveira, and Vinícius Maran Encouraging Grading: Per Aspera Ad A-Stars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pia Niemelä, Jenni Hukkanen, Mikko Nurminen, and Jukka Huhtamäki

1

23

An Approach for Mapping Declarative Knowledge Training Task Types to Gameplay Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bérénice Lemoine, Pierre Laforcade, and Sébastien George

47

Designing Declarative Knowledge Training Games: An Analysis Framework Based on the Roguelite Genre . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Bérénice Lemoine, Pierre Laforcade, and Sébastien George

69

Cultivating Higher Order Competencies: Complex Thinking in Latin American University Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jorge Sanabria-Z, María Soledad Ramírez-Montoya, Francisco José García-Peñalvo, and Marco Cruz-Sandoval

93

Correlation Among Competences Involved in Digital Problem-Solving Activities with Upper Secondary School Students . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Alice Barana, Cecilia Fissore, Anna Lepre, and Marina Marchisio Regulatory Strategies for Novice Programming Students . . . . . . . . . . . . . . . . . . . . 136 Deller James Ferreira, Dirson Santos Campos, and Anderson Cavalcante Gonçalves From GPT-3 to GPT-4: On the Evolving Efficacy of LLMs to Answer Multiple-Choice Questions for Programming Classes in Higher Education . . . . . 160 Jaromir Savelka, Arav Agarwal, Christopher Bogart, and Majd Sakr Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

Towards a Software Architecture to Provide Hybrid Recommendations for Smart Campuses Martin Hideki Mensch Maruyama1 , Luan Willig Silveira1 , Ana Paula Militz Dorneles1 , Gabriel Vieira Casanova1 , Renan Bordignon Poy1 , Elvandi da Silva J´unior1 , Jos´e Palazzo M. de Oliveira2 , and Vin´ıcius Maran1(B) 1

Laboratory of Ubiquitous, Mobile and Applied Computing (LUMAC), Polytechnic School, Federal University of Santa Maria, Avenue Roraima, 1000 Santa Maria, Brazil {apdorneles,rbpoy}@inf.ufsm.br, [email protected], [email protected] 2 Informatics Institute, Federal University of Rio Grande do Sul, Porto Alegre, Brazil [email protected]

Abstract. Currently, several initiatives have been proposed in order to offer solutions for intelligent university campus environments. It can be said that smart university campuses are a subdomain of the smart cities domain, with some similar problems, but with specificities. Recommender systems identify and suggest relevant information to the user, recognizing their potential interests through specialized algorithms and presenting resources that align with these interests. In the context of intelligent university campuses, recommender systems have been used to define which systems and technologies should be implemented. From this scenario, the objective of this article is to present a software architecture, called SmartC, structured in different services, to provide the essential infrastructure for the application of several recommender systems and a variety of types of items. The services and layers of the architecture were defined especially for intelligent university campuses and divided into three distinct sections: the access environment, the recommendations management environment and the persistence layer. The recommendation algorithms integrated to this architecture are considered hybrids, since they incorporate two types of filtering: content-based filtering and collaborative filtering. When users request new recommendations, the type of filtering will be switched, ensuring that new features are suggested with each system call and avoiding throttling. The developed prototype was evaluated from real item data and showed significant accuracy in the recommendation process. Keywords: Recommendation system · Filtering techniques · Hybrid filtering · Collaborative filtering · Content-based filtering · Smart campus · University · Personalized recommendation · Educational resources

1 Introduction Education is undoubtedly one of the essential pillars for the development of a society, as it is through it that new knowledge is generated, transmitted and allows new technologies and teaching-learning tools to be developed. Today, there is a great interest in the c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024  B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 1–22, 2024. https://doi.org/10.1007/978-3-031-53656-4_1

2

M. H. M. Maruyama et al.

development of digital tools aimed at the academic field, since many students are looking for more flexible learning methods that give them the freedom to learn wherever and whenever they want outside the dependencies of traditional classrooms [21]. In recent years, so-called smart campuses have attracted the attention of universities and fostered the development of research related to education with technologies such as, for example, IoT (Internet of Things), big data, cloud and mobile applications in order to improve the quality of teaching and ensure better use of the resources available at the university by its users [43]. The idea behind smart campuses is to use state-of-the-art technologies and information capture systems (sensors, actuators) to generate a large volume of data, manage and treat it, and make decisions to then return information to the user of interest. that may contribute to your academic performance [38]. Along with academic development and the search for new learning tools by students, there is also research on recommendation systems that, especially during the period of the COVID-19 pandemic when the population had to remain isolated at home, received a lot of attention not only by industry but also by universities and schools, which were forced to adopt distance learning and used these systems to provide students with relevant information for their learning [39]. These systems have the function of filtering information from a certain database based on the profile of a certain user or group of users and returning to him information that meets his interests, that is, the system will recommend to the user what he most likely you are looking within a given platform [24]. Based on these concepts, some authors present and propose the integration between intelligent campuses and recommender systems to improve parameters or services within a university. In [26], the authors present the concept of Social Internet of Things (SIoT), which would be a network of systems with IoT technology capable of establishing connections between them and interacting with users and various devices in a more socially efficient way, and propose the development of a personalized service recommendation framework within a given community. In [14], a system model is proposed that makes use of ZigBee technology, which makes wireless communication between devices on a given network to transmit information and perform a certain action, along with an algorithm to collect information and to evaluate the resources in a university to improve the performance and quality of teaching and learning. Also in [43], a platform is developed for evaluating the performance of teaching and learning in smart campuses that uses big data together with the AHP method to determine the degree of performance and the degree of gray correlation to generate a teaching model and thus have a more relevant evaluation of teachers. In [28] it is proposed the development of a recommendation system on a mobile platform of resources aimed at Arab users in a personalized way using neural networks as a way to help users in the decision-making process and in the performance of their learning, while also take into account information about its context and level of approval of the recommendations to improve the efficiency of the algorithm.

Towards a Software Architecture to Provide Hybrid Recommendations

3

In the related work, the proposals sought to integrate recommendation systems or frameworks to improve the performance of some type of service offered by universities or else the quality of teaching. However, the development of an online platform that indicates resources to students is little explored in the literature. Therefore, this work proposes the creation of an online platform aimed at intelligent university campuses that recommends different types of items related to education for users based on their interests and interactions with the system. In addition, the recommendation system to be developed has two types of filtering to ensure greater volatility and not generate repetitive or redundant information, making it a hybrid system, and will be able to make personalized recommendations to better meet the preferences of each user. In this context, this work presents the definition of a software architecture, called SmartC, which aims to provide subsidies for the integration of different types of recommenders, which applied in different contexts, seek to recommend items considering the context of intelligent university campuses. This article is an extension of the proposal presented in [27], which presented the integration process of two recommenders. In this article, this integration was incorporated into the proposal of a software architecture, with a definition of support services. This work is divided into 6 sections: Sect. 2 presents the various smart campus concepts found in the literature, its components, its architecture and recent research developed; Sect. 3 brings the concept of recommender systems, their applications, the types of systems and their filtering, and systems applied to intelligent campuses; Sect. 4 presents the personalized recommendation system model developed in this work, its functionalities, the algorithms and techniques used and return examples; Sect. 5 shows the results obtained from the system’s recommendations, the evaluation of the algorithm’s precision and comparison with other systems and types of filtering; finally, and Sect. 6 summarizes the main points presented throughout the work and also future expectations for implementation in the system.

2 Smart Campus Technology is extremely relevant and it is one of the causes of the advancement and development of societies, and also responsible for shaping people’s behavior and making their lives more convenient. Over the past few years, universities have begun to adapt their infrastructure to accompany the gradual change in profile on the part of students who are looking for more dynamic, convenient ways of learning that are not limited to traditional classrooms, and thus allow that the student has access to quality education and learning tools and boost the development of new technologies [11]. Smart campuses have attracted the attention of researchers and universities around the world in recent decades, these domains encompass both the physical structure of a university and its virtual environments. In the literature, a universal definition for “smart campus” [5] is not found, however, some authors bring their own definitions and concepts: – Dong et al. [11]: “An educational environment that is penetrated with enabling technologies for smart services to enhance educational performance while meeting

4

M. H. M. Maruyama et al.

stakeholders’ interests, with broad interactions with other interdisciplinary domains in the smart city context”; – AbuAlnaaj et al. [1]: “an integration of three fundamental axes which includes: Data acquisition using IoT, Data centralization and use of big data for the management and analysis”; – Caram´es et al. [13]: “Refers to the hardware and software required to provide advanced intelligent context-aware services and applications to university students and staff”; – Imbar et al. [19]: “Considered a small smart city that acts within the context of smart cities, which offer intelligent services and applications to their citizens to improve their quality of life”. Figure 1 presents the different branches and applications of the smart campus through a mind map. For a university to become an intelligent campus it is necessary, according to Imbar et al. [19], that it is able to use knowledge to resolve conflicts of interest of stakeholders and that it knows how to use resources and data provided by users to contribute to the integrity of the system, that is, it is necessary to implement a model or smart campus framework at the university. In general, smart campuses have common characteristics in relation to the services and tools offered within a university to ensure a good use of spaces, Zhang et al. [46] presents the following applications in his work (Fig. 2): – Smart Learning. This domain represents the use of technologies to promote quality teaching and learning through tools that allow the student to obtain resources, items, teaching materials, academic information and educational activities and thus build a good knowledge base and develop their skills, reaching better results in your learning. Examples of applications can be cited [1]: intelligent classrooms, use of virtual reality, collaborative research and virtual libraries. Chan et al. [6] brings information about the integration of smart libraries in smart campuses and the most diverse technologies used already reported within this application. Villegas et al. [38] propose a model for identifying and evaluating variables through the analysis of student data using artificial intelligence in the decision-making process to improve learning. – Smart Living. This domain encompasses the most diverse services and spaces within a smart campus that, like a smart city, has leisure centers, food courts, health care, convenience stores, transportation services and location. Chotbenjamaporn et al. [8] developed an application aimed at navigation and location within a campus while recommending the fastest routes between two locations considering all available transport options and also the air quality during the journey. – Smart Environment. This domain addresses the resources present in a campus environment such as energy, water, cleaning and organization. Examples of applications include: monitoring of water, electricity, air quality, temperature, waste management and intelligent disposal. Yang et al. [44] propose the development of a real-time smart energy monitoring system using ‘big data processing’ and cloud computing to collect data on electricity usage in campus facilities. – Smart Management. This domain justifies the use of technologies for managing activities on campus to ensure its normal operation, full use of resources and available services. Examples of applications include: managing security and protection,

Towards a Software Architecture to Provide Hybrid Recommendations

Fig. 1. Smart Campus subfields [27].

5

6

M. H. M. Maruyama et al.

Fig. 2. Features in a smart campus.

managing the use of spaces, classrooms, laboratories, online platforms, scheduling activities, monitoring people and cars on campus, and managing people in projects and activities. Jabbar et al. [20] developed a parking management system based on IoT (Internet of Things) devices integrated into the Raspberry Pi to help campus users find places to park their vehicles.

3 Recommendation Systems The decision-making process is strongly linked to human beings, since people are subjected to different choices and options at all times, and it is also very influenced by variables related to the context in which a given individual is exposed. For example, an entrepreneur often must decide which business propositions are most beneficial for his company taking into account his profits and costs, whereas a farmer may have to decide which type of seed to plant based on climate, value of product market and soil fertility, or else a student may be undecided about which higher education course to choose and so organizes their options considering proximity to campus, quality of teaching and their tastes. Many of these choices can be difficult to process and take time to arrive at an adequate solution, but when it comes to options within a virtual environment, there are tools that can help the user to arrive at the best option. Introduced in the early 90s, recommender systems began to be used and studied for the most diverse applications within industry, e-commerce and education. The motivation for the development of these systems and related research was due to the need to find a way to allow people to be able to find information relevant to them amid the massive amounts of new data that were introduced daily on the most diverse platforms on

Towards a Software Architecture to Provide Hybrid Recommendations

7

the Internet. at that time due to its popularization around the world. In addition, another very common problem is the heterogeneity of the data, that is, the information inserted in the network does not need to be in a standard format, much less use a specific language or structure, making the selection and extraction of information a task. extremely laborious and time-consuming [3]. Jussi Karlgren was one of the pioneers in research on this topic in his work ‘An Algebra for Recommendations’ [23], where he aimed to create a document recommendation algorithm based on the similarity between them and on users’ responses, bringing information about explicit and implicit feedback and also about algebra for determining the distance between certain features. David Goldberg et al. in his work he developed Tapestry, a system that filtered various documents and emails and returned them to the user based on his interests and possible new resources that are relevant to him. [16]. Paul Resnick and Hal R. Varian were the first to use the term “recommender systems” in their work ‘Recommender Systems’ [32], where they argue that this term would have a broader approach than the one used until then, “collaborative filtering”. Recommender systems are extremely convenient and widely used tools on platforms, websites, online stores, applications and services, responsible for searching for information, analyzing and filtering data using filtering techniques and algorithms, and returning a list of resources for a given user based on in parameters related to him, such as interests, search history, interactions, profile, similarity with other users and evaluations, in order to contribute to the resolution of his conflicts of interest, optimize the time of the task that he would have to search for relevant information and recommend resources that may align with your interests [24]. These systems are present in the most diverse applications such as, for example, streaming platforms (Netlfix, Youtube, Disney+, Spotify), product sales sites (eBay, Amazon, Facebook Marketplace), educational platforms (Coursera, Udemy, edX), academic information sites (digital libraries, Google Scholar), social networks (Facebook, Twitter, Instagram, Linkedin), delivery apps (UberEats), relationship apps, game sites, health services and tourism services. However, the same recommendation system cannot necessarily be applied in all contexts and, because of this, each platform or application develops its recommendation system based on the types of data that are used and, for that, they use filtering techniques and algorithms responsible for performing the refinement of information and providing users with good recommendations. Among the best known filtering techniques today, we can mention: collaborative filtering, content-based filtering, hybrid filtering, contextsensitive filtering, knowledge-based filtering and ontology-based filtering. 3.1 Collaborative Filtering Collaborative filtering is a filtering technique widely used in the most diverse types of recommender systems, it is based on searching for information based on user interactions with resources available in a system such as, for example, history, implicit feedback or evaluation of certain items and also on similarities between [12] users. This type of filtering is used to analyze the behavior of a group of users with similar tastes or profiles to then determine the best type of resource to recommend for them, however, this technique is also very susceptible to the cold-start problem that happens when recommendations are generated from a database with little information about users, which reduces the performance and accuracy of the algorithm [22].

8

M. H. M. Maruyama et al.

In the literature, several works make use of collaborative filtering. Zhihua et al. in his work [9] makes use of collaborative filtering together with Pearson’s correlation coefficients and time to determine the similarity of users over time to generate more accurate personalized recommendations. Zheng et al. in turn, in his work [47] uses “variational inference”, a technique used in machine learning to approximate complex probability distributions in simpler and easier to be optimized distributions, to optimize problems encountered by recommendation algorithms with filtering collaboration caused by large-scale databases. Wang [41] proposes the design of an educational platform that is composed of courses from different institutions and universities and recommends these courses to users using the Django framework and also an algorithm with collaborative filtering. 3.2

Content-Based Filtering

Content-based filtering is based on the similarity between the characteristics of items and the characteristics of a user’s profile, that is, it works directly linked to a user’s interests. This type of filtering is more versatile due to the fact that the user only needs to change his parameters and tastes and the algorithm will be able to adapt to his new preferences [12]. Lops et al. [25] in their work present detailed information about content-based filtering, its development over the years, research trends and applications. Singer et al. [4] presents the application of a recommendation system that uses content-based filtering on a platform called Decide Madrid, a platform aimed at the participation of the population of the city of Madrid in electronic polls and debates, in order to recommend to themes of relevance to each user to encourage them to participate in the city’s decisions. Xiao et al. [42] describe a design for a personalized course recommendation system for students using content-based filtering to create links between user interests and resource content. 3.3

Hybrid Filtering

A recommender system has its algorithm considered hybrid when two or more filtering techniques are used [36]. This filtering is widely used in most algorithms because it allows certain disadvantages of one type of filtering to be overcome by the advantages of another type, which makes the algorithm more efficient and also allows for greater versatility and variety in recommendations. One of the most popular hybrid systems is the one that uses collaborative and content-based filtering [26], the same one adopted in this work. Paradarami et al. [30] developed a hybrid recommendation system made up of artificial neural networks trained with textual data obtained by collaborative and contentbased filtering for the prediction of ratings. Zhang et al. [45] propose the creation of the iDoctor recommendation system that recommends health professionals based on assessments of a person’s health status and also their emotional state using techniques such as matrix factorization, topic model and textual analysis of feelings. Gulzar et al. [17] propose a hybrid recommendation system that can be integrated with another system aimed at online learning to improve its efficiency, facilitate access to information

Towards a Software Architecture to Provide Hybrid Recommendations

9

and recommend resources in a personalized way to users. Wan and Niu [40], similarly, propose a hybrid recommendation system aimed at the context of online education that generates more personalized and diversified recommendations by using a student influence model to acquire information about the user and allow Collaborative filtering does not face the cold-start problem. 3.4 Contextual-Sensitive Filtering Algorithms that use contextual-sensitive filtering take into account information about the context in which a given user or group of users is inserted, such as location, time and social relationships to generate recommendations [36]. This type of filtering is often addressed in works that involve or propose frameworks or platforms aimed at tourism. Dennouni et al. [10] proposes the use of contextualsensitive filtering in mobile tourism applications for tourists without the need to create a profile for recommending points of interest (POIs) based on their current location, the places they have been previously and in your assessments. Renjith et al. [31] conducts extensive research on the most diverse works related to recommendation systems that use contextual-sensitive filtering in the context of travel and tourism. In turn, Moreira et al. [29] uses contextual-sensitive filtering in his work with Recurrent Neural Networks (RNN) to create a hybrid news recommendation algorithm that manages to overcome problems such as the short life of recommended items and the lack of users with longterm profiles. 3.5 Knowledge-Based Filtering Knowledge-based filtering is a filtering technique that stores data related to users and items and uses this knowledge to search for information, interactions and explicit feedback are collected from the user and constantly updated so that the algorithm can generate more accurate recommendations as the need [36]. Dong et al. [11] proposes the development of a recommendation system aimed at the design of personalized textile products that uses ‘knowledge-based filtering’ to store complex, structured and unstructured information about design. Agarwal et al. [2] present a ‘knowledge-based recommendation system’ to be used in MOOCs (Massive Open Online Courses) educational platforms based on the user’s learning style. Tarus et al. [34] propose a ‘hybrid knowledge-based recommender system’ for recommending educational learning resources for students and which is also based on ontologies to compute similarities and make predictions of the evaluations of a given user. 3.6 Ontology-Based Filtering Ontology is a study dating from the time of Plato and Aristotle that seeks to understand the existence of being by having as object of study a structure of the world and the entities and types of entities that exist in it. When applied to recommender systems, it explicitly describes concepts by representing them as a [36] knowledge base. In addition, this type of system, when applied in the field of education, helps users learn by

10

M. H. M. Maruyama et al.

being aware of the domain of knowledge that each one has, improving the quality and accuracy of [22] recommendations. Joy et al. [22] proposes the development of a framework for e-learning based on ontologies capable of generating personalized recommendations of learning contents to users, bypassing the cold-start problem. Ibrahim et al. [18], similarly, proposes a framework for personalized course recommendation for students that uses ontologies to identify the most influential factors that lead a person to choose a particular course. George and Lal [15] present information on the use of ontologies in recommender systems aimed at e-learning between 2010 and 2019, identify common points of research addressed in the most diverse works and discuss techniques used in these systems to calculate the similarity of users based on their interests.

4 Hybrid Recommender System for Personalized Recommendations In this section we present the definition of the proposed software architecture and the definition of the recommendation strategies applied to it. 4.1

The SmartC Software Architecture

The SmartC platform was designed to be a software architecture, structured in distinct services, to provide the essential infrastructure for the application of several recommender systems and a variety of item types. The services and architecture layers were defined especially for smart campuses, as illustrated in Fig. 3. This structure can be subdivided into three distinct sections, each one playing a crucial role in the operation of the SmartC system: the access environment, the recommendations management environment and the persistence layer, as mentioned by Silva Lopes et al. [33]. The access environment serves as the visible interface for users, where recommendations are presented and users can interact with available resources. In addition, this is where users can set their preferences, explore other university portals and perform more available actions, all through devices such as computers or mobile devices. The recommendations management environment, on the other hand, is a sandbox, accessible only to developers. This is the operational core of the system, housing in this space all the codes and functionalities of the system. Here, new codes are introduced, edited and corrected as needed. Furthermore, this environment plays a vital role in customizing the recommendations for each user, collecting and processing their information and subsequently sending requests to the database so that the data can be processed by the recommendation algorithms. Finally, the persistence layer encompasses the database, which is responsible for storing system information in tables. This includes feature descriptions, user preferences, items interacted with, recommendation history, ratings, and more. This data is kept in an organized way in this layer and, when necessary, is passed to the development management environment to start the filtering and recommendation process.

Towards a Software Architecture to Provide Hybrid Recommendations

11

Fig. 3. SmartC software architecture definition [27].

The recommendation algorithm developed on the platform aims to offer users personalized recommendations of educational resources, based on topics of interest provided by users. This system is characterized as hybrid, since it incorporates two types of filtering: content-based filtering and collaborative filtering. Given the availability of

12

M. H. M. Maruyama et al.

these two types of filtering, it was decided to switch between them whenever users request new recommendations. This ensures that new resources are suggested at each system call, avoiding access limitations and enriching the user experience. The following sections explain each filter and its usage in more detail. 4.2

Content-Based Recommender

Algorithms that adopt content-based filtering (CBF), recommend items considering the specific interests of the user [35]. This type of filtering also searches for items that are similar to each other, that is, based on the description of a given resource, the system will look for other resources with similar characteristics to be recommended. So, because the CBF focuses on the user’s interests and the item’s characteristics, it is able to provide accurate recommendations right from the start, without relying on extensive history. The recommendation is also dynamic, adjusting to changes in user preferences [12]. A practical example is a movie and series streaming platform that uses CBF, where the suggestions reflect the content most consumed by the user or seen recently. As for privacy, this model is advantageous, as CBF does not require users to share or make public their preferences. Simply interact with the system for your inclinations to be identified, processed and securely stored [35]. In the context of this study, the application of the CBF operates as follows: first, the user’s interests registered in the system are cataloged in associative tables, correlating the user and their topics of interest. Based on this mapping, all resources that match these topics are identified. If they have not yet been processed, the texts associated with these resources are converted into strings and subjected to a textual filtering method called “bag-of-words (BoW)”. The technique consists of extracting relevant keywords from a text, quantifying their occurrences and compiling them into a list, thus forming a profile for each resource. This method is then applied to resources already marked as favorites by the user. Subsequently, a textual similarity comparison is made between each resource and the user’s favorites, using cosine similarity. In the end, the 25 resources with the greatest affinity with the user’s favorites are recommended. 4.3

Collaborative Recommender

Collaborative Filtering (CF) is a technology widely recognized and implemented in the field of recommender systems. Its techniques have extensive applications, mainly in sectors such as electronic commerce and social networks, as mentioned by Chen et al. [7]. The main premise of a recommendation algorithm that uses CF is to propose items of possible interest to the user, based on their interactions with other users or on the relationships between the items themselves, as mentioned by Zheng et al. [47]. The FC operating mechanism operates on evaluation matrices. In these systems, users detail the relevance of a given resource, expressing their satisfaction or dissatisfaction with a recommendation. The algorithms, in turn, process and analyze this information to provide new suggestions, as highlighted by ValdiviezoDiaz et al. [37]. However, systems that employ CF can face significant challenges, such as problems with big data, information sparsity and cold-start issues, which compromise the quality and accuracy of [7] recommendations. To overcome such obstacles, the integration of

Towards a Software Architecture to Provide Hybrid Recommendations

13

other filtering techniques is often used, such as clustering, Singular Value Decomposition (SVD), Probability Matrix Factorization (PMF), recommendation with social trust ensemble (RSTE) and Social rating Matrix Factorization (SocialMF). When we look at FC approaches, we realize that it can be divided into two: modelbased and memory-based [37]. In the model-based approach, a dataset is used to build a model that will make recommendations. These model user ratings for specific items, based on factors representative of the characteristics of those users and items. Matrix Factoring (MF) stands out as the most popular implementation of this approach and is recognized for its superior performance and accuracy. On the other hand, in the memory-based approach, the information for recommendation is extracted directly from the evaluation matrices. Algorithms that employ this approach can be categorized into user-based CF, which compares users with similar preferences based on ratings of identical items, and item-based CF, which suggests a list of items similar to those the user has already interacted with or rated [35]. In our study, we developed an algorithm that uses CF in a particular way. Similar to CBF, the user indicates their interests to the system, the system searches for other users with aligned interests and creates an individual list. Combining the topics of both, each common interest adds 1 to the total. In the end, the algorithm selects the resources to be suggested to the most similar users, randomizes them and returns to the original user. It is crucial to point out that without the user’s prior indication of their interests, the systems, whether CBF or CF, will not be fully effective. Specifically for CBF, if the user has not favorited any resources, recommendations will be based solely on their interests. To conclude, it is worth mentioning that the prototyped platform was developed using the following technologies: Angular for the frontend, Python, Flask, Scikit and Surprise! for back-end functionality, and PostgreSQL as the database management system.

5 Results This section presents: (i) a demonstration of the prototypes developed in use by users, and (ii) the evaluation process of the recommendation algorithms used in the SmartC software platform. 5.1 Developed Prototypes The developed software architecture provides a website wich contains a home page, shown in Fig. 4, which offers users the possibility to select topics of interest. The functionality allows the system to make personalized recommendations, based on topics selected by the user, as illustrated in Fig. 5. This optimizes the user experience by delivering relevant content efficiently. On the recommendations page, users have the option to favorite, remove, rate, or view more information about each resource. When selecting the option to bookmark a resource, it will be added to the user’s favorites page, as illustrated in Fig. 6. This functionality gives users greater control over recommended resources and allows them to easily organize and access the content of interest to them.

14

M. H. M. Maruyama et al.

Fig. 4. Home page, with the selection of areas of interest by the user.

Fig. 5. Interface with items recommended to the user. The user is allowed to interact with the items through access or evaluation of the recommendation made.

Towards a Software Architecture to Provide Hybrid Recommendations

15

Fig. 6. User favorited item interface.

Figure 7 represents the resource visualization page, in which the user has access to complete information, such as name, photo, summary, description, among other relevant details. This page provides a comprehensive view of the resource in question, allowing users to obtain all the necessary details for a better understanding and evaluation of the available content.

Fig. 7. Preview interface for a specific resource.

In addition, the system has a page dedicated to useful links related to the university, such as the official website of the institution, the student portal and the Moodle platform. This section provides users with easy access to essential university resources, contributing to a more complete and convenient experience within the platform.

16

5.2

M. H. M. Maruyama et al.

Evaluation of the Recommendation Algorithm

The collaborative filtering process is the idea that if person A agrees with person B on an issue, A is more likely to have the same opinion as B on a different issue than the opinion of a random person. In the example in Fig. 8, this similarity of opinions is mapped across the ratings given to resource 273. Collaborative filtering operates, fundamentally, in the collection and analysis of a large amount of information about the behavior, activities or opinions of a group of people. By identifying similarities between users, the system is able to predict the interests of particular users, even if they have not yet interacted with certain features or information. In this scenario, if user 1 has not yet rated or interacted with resources 275 and 274, but users 195, 315, and 213, who have demonstrated similar tastes to user 1, have given favorable ratings to these resources, then there is a higher probability that user 1 also appreciates these features. Therefore, the suggestion is made based on the evaluations and interactions of its ‘neighbors’ in the user network.

Fig. 8. Recommendation algorithm graph.

Within this context, a controlled scenario was created to test the effectiveness and accuracy of the collaborative filtering algorithm. The use of learning resources, described in Table 1, served as a basis for users’ fictitious interactions. These resources can cover a range of topics, from exact sciences to humanities, and can be articles, videos, tutorials, among other educational materials.

Towards a Software Architecture to Provide Hybrid Recommendations

17

Table 1. Learning Resources Partition (summarized due to restricted space). id topic 1

Engineering

2

Mathematics

3

Science

4

Art

5

Music

6

Sports

7

History

8

Geography

9

Literature

10 Philosophy

Table 2 provides a detailed view of the various user profiles, highlighting their areas of preference and interest. These profiles represent a diversity of inclinations, with a particular focus on users with an engineering bent. By incorporating complementary preferences, we seek to simulate the complexity and diversity of the nature of real users, who rarely have a single field of interest. Within this simulated environment, attention is particularly focused on user 1. Its exclusive inclination towards the engineering area allows a focused analysis of the effectiveness of the recommender system. The challenge, in this case, is to verify if the algorithm is sensitive enough to identify and prioritize such specific interest, filtering out other influences and guaranteeing efficient recommendations. This analysis will serve to evaluate the accuracy of the system in tuning with the individual needs of the users. Table 2. Topics of interest associated with users (summarized due to restricted space). interest topics Art Art, Engineering Science Science, Engineering Engineering Engineering, Mathematics Engineering, Music Sports Philosophy Geography History Literature Mathematics Music

user id 49 77 76 55 1 73 94 90 40 52 66 38 56 46

18

M. H. M. Maruyama et al.

Analysis of the results targeted at user 1, as shown in Table 3, offers valuable insights into the efficiency of the algorithm. The descending order of the suggestions by relevance indicates that the algorithm effectively captures and prioritizes the topics of greatest interest to the user. as reflected by the prevalence of the “engineering” theme at the top of the table. The observed correlation between the indicated preferences and the generated recommendations is direct evidence of the accuracy of the algorithm. This accuracy is particularly crucial in learning environments, where content relevance can directly impact instructional effectiveness and learning retention. Table 3. Forecasting Recommendation Reviews. user id topic 1

Engineering

1

Art

1

Music

1

Science

1

Mathematics

1

Sports

1

Philosophy

1

History

1

Literature

1

Geography

Subsequently, the list evolves into topics that, although not the primary focus of user 1, were positively evaluated by other users with converging interest profiles. This feature emphasizes the algorithm’s ability to perform efficient collaborative filtering, crossing information from several users, to provide recommendations that are both relevant and diverse. This aspect is very important, since, in recommendation systems, it is essential to strike a balance between meeting the direct expectations of the user and introducing new content that expands his area of knowledge, enhancing the assisted learning process. The ongoing challenge, however, is to balance this range of data with the necessary customization to meet individual user needs. After all, each user is unique, with a specific combination of interests, needs, and learning objectives. The ideal recommendation, therefore, should be both a reflection of the collective and adapted to the individual. The effectiveness of any recommendation algorithm will be determined by its ability to improve the user experience, presenting new content that aligns both with the user’s current interests and potential. In this perspective, collaborative filtering stands out, offering a balance between what is already known by the user and what is innovative.

Towards a Software Architecture to Provide Hybrid Recommendations

19

6 Conclusion Smart campuses have become a goal to be achieved by universities around the world, in view of the qualities and benefits offered in the field of a campus that uses cuttingedge technologies to provide numerous tools, products and services to people in their midst. and help them resolve their conflicts of interest. Along with this, recommender systems also prove to be of great importance in the decision-making process, in improving student learning, in improving the quality of teaching and in providing personalized resources in other applications. Therefore, in this work, a hybrid recommendation system was developed that uses collaborative and content-based filtering techniques to recommend educational resources of the most diverse types to users based on their interests, in order to encourage the development of research and projects, bring new quality information, relevant and similar to the user’s interests, and develop the domain of smart learning and education within smart campuses. As future objectives, it is expected to be able to implement more algorithms and filtering techniques in the system in order to increase its accuracy and have the ability to be even more volatile, perform optimizations in the code in order to further reduce the processing time of the recommendations. and creating sub-categories for topics of interest to allow more relationships between resources to be established. With this, it is expected that this platform can provide the personalized support necessary for students based on their interests and bring them relevant educational information. Acknowledgements. This research is supported by CNPq/MCTI/FNDCT n. 18/2021 grant n. 405973/ 2021-7. The research by Jos´e Palazzo M. de Oliveira is partially supported by CNPq grant 306695/2022-7 PQ-SR. The reasearch by Vin´ıcius Maran is partially supported by CNPq grant 306356/2020-1, CNPq PIBIC program , Fundac¸a˜ o de Amparo a Pesquisa do Estado do Rio Grande do Sul (FAPERGS), grant n. 21/2551- 0000693-5 and FAPERGS PROBIC program.

References 1. Abualnaaj, K., Ahmed, V., Saboor, S.: A strategic framework for smart campus (2020) 2. Agarwal, A., Mishra, D.S., Kolekar, S.V.: Knowledge-based recommendation system using semantic web rules based on learning styles for moocs. Cogent Eng. 9 (2022). https://doi. org/10.1080/23311916.2021.2022568 3. Benfares, C., Idrissi, Y.E.B.E., Abouabdellah, A.: Recommendation semantic of services in smart city. vol. Part F129474. Assoc. Comput. Mach. (2017). https://doi.org/10.1145/ 3090354.3090407 4. Cantador, I., Bellog´ın, A., Cort´es-Cediel, M.E., Gil, O.: Personalized recommendations in e-participation: offline experiments for the ’decide madrid’ platform. Assoc. Comput. Mach. (2017). https://doi.org/10.1145/3127325.3127330 5. Chagnon-Lessard, N., et al.: Smart campuses: extensive review of the last decade of research and current challenges. IEEE Access 9, 124200–124234 (2021). https://doi.org/10.1109/ ACCESS.2021.3109516 6. Chan, H.C.Y., Chan, L.: Smart library and smart campus. J. Serv. Sci. Manag. 11, 543–564 (2018). https://doi.org/10.4236/jssm.2018.116037 7. Chen, R., Hua, Q., Chang, Y.S., Wang, B., Zhang, L., Kong, X.: A survey of collaborative filtering-based recommender systems: from traditional methods to hybrid methods based

20

8. 9.

10.

11.

12.

13.

14. 15. 16. 17.

18.

19.

20.

21.

22.

23. 24.

M. H. M. Maruyama et al. on social networks. IEEE Access 6, 64301–64320 (2018). https://doi.org/10.1109/ACCESS. 2018.2877208 Chotbenjamaporn, C., et al.: A web-based navigation system for a smart campus with air quality monitoring (2019) Cui, Z., et al.: Personalized recommendation system based on collaborative filtering for IoT scenarios. IEEE Trans. Serv. Comput. 13, 685–695 (2020). https://doi.org/10.1109/TSC. 2020.2964552 Dennouni, N., Peter, Y., Lancieri, L., Slama, Z.: Towards an incremental recommendation of POIs for mobile tourists without profiles. Int. J. Intell. Syst. Appl. 10, 42–52 (2018). https:// doi.org/10.5815/ijisa.2018.10.05 Dong, Z.Y., Zhang, Y., Yip, C., Swift, S., Beswick, K.: Smart campus: definition, framework, technologies, and services. IET Smart Cities 2, 43–54 (2020). https://doi.org/10.1049/ietsmc.2019.0072 Eliyas, S., Ranjana, P.: Recommendation systems: content-based filtering vs collaborative filtering, pp. 1360–1365. Institute of Electrical and Electronics Engineers Inc. (2022). https:// doi.org/10.1109/ICACITE53722.2022.9823730 Fern´andez-Caram´es, T.M., Fraga-Lamas, P.: Towards next generation teaching, learning, and context-aware applications for higher education: a review on blockchain, IoT, fog and edge computing enabled smart campuses and universities. Appl. Sci. (Switzerland) 9, November 2019. https://doi.org/10.3390/app9214479 Gao, M.: Smart campus teaching system based on zigbee wireless sensor network. Alexandria Eng. J. 61, 2625–2635 (2022). https://doi.org/10.1016/j.aej.2021.09.001 George, G., Lal, A.M.: Review of ontology-based recommender systems in e-learning. Comput. Educ. 142 (2019). https://doi.org/10.1016/j.compedu.2019.103642 Goldberg, D., Nichols, D., Oki, B.M.: nnnnnnunnnm a nmm dul filterihg using to weave an information Gulzar, Z., Leema, A.A., Deepak, G.: PCRS: Personalized course recommender system based on hybrid approach, vol. 125, pp. 518–524. Elsevier B.V. (2018). https://doi.org/10. 1016/j.procs.2017.12.067 Ibrahim, M.E., Yang, Y., Ndzi, D.L., Yang, G., Al-Maliki, M.: Ontology-based personalized course recommendation framework. IEEE Access 7, 5180–5199 (2019). https://doi.org/10. 1109/ACCESS.2018.2889635 Imbar, R.V., Supangkat, S.H., Langi, A.Z.: Smart campus model: a literature review. Inst. Electr. Electron. Eng. Inc. November 2020. https://doi.org/10.1109/ICISS50791.2020. 9307570 Jabbar, W.A., Wei, C.W., Azmi, N.A.A.M., Haironnazli, N.A.: An IoT raspberry pi-based parking management system for smart campus[formula presented]. Internet Things (Netherlands) 14, June 2021. https://doi.org/10.1016/j.iot.2021.100387 Jord´an, J., Valero, S., Turr´o, C., Botti, V.: Using a hybrid recommending system for learning videos in flipped classrooms and MOOCs. Electronics (Switzerland) 10, June 2021. https:// doi.org/10.3390/electronics10111226 Joy, J., Raj, N.S., Renumol, V.G.: Ontology-based e-learning content recommender system for addressing the pure cold-start problem. J. Data Inf. Q. 13 (2021). https://doi.org/10.1145/ 3429251 Karlgren, J.: An algebra for recommendations an algebra for recommendations using reader data as a basis for measuring document proximity (1990) Ko, H., Lee, S., Park, Y., Choi, A.: A survey of recommendation systems: Recommendation models, techniques, and application fields. Electronics (Switzerland) 11, January 2022. https://doi.org/10.3390/electronics11010141

Towards a Software Architecture to Provide Hybrid Recommendations

21

25. Lops, P., Jannach, D., Musto, C., Bogers, T., Koolen, M.: Trends in content-based recommendation: Preface to the special issue on recommender systems based on rich item descriptions. User Model. User-Adapt. Interact. 29, 239–249 (2019). https://doi.org/10.1007/s11257-01909231-w 26. Lye, G.X., Cheng, W.K., Tan, T.B., Hung, C.W., Chen, Y.L.: Creating personalized recommendations in a smart community by performing user trajectory analysis through social internet of things deployment. Sensors (Switzerland) 20, April 2020. https://doi.org/10.3390/ s20072098 27. Maruyama, M.H.M., Silveira, L.W., de Oliveira, J.P.M., Gasparini, I., Maran, V.: Hybrid recommender system for educational resources to the smart university campus domain (2023). https://doi.org/10.5220/0011841900003470 28. Meddeb, O., Maraoui, M., Zrigui, M.: Personalized smart learning recommendation system for arabic users in smart campus. International J. Web-Based Learn. Teach. Technol. 16, 1–21 (2021). https://doi.org/10.4018/ijwltt.20211101.oa9 29. Moreira, G.D.S.P., Jannach, D., Cunha, A.M.D.: Contextual hybrid session-based news recommendation with recurrent neural networks. IEEE Access 7, 169185–169203 (2019). https://doi.org/10.1109/ACCESS.2019.2954957 30. Paradarami, T.K., Bastian, N.D., Wightman, J.L.: A hybrid recommender system using artificial neural networks. Exp. Syst. Appl. 83, 300–313 (2017). https://doi.org/10.1016/j.eswa. 2017.04.046 31. Renjith, S., Sreekumar, A., Jathavedan, M.: An extensive study on the evolution of contextaware personalized travel recommender systems. Inf. Process. Manag. 57, January 2020. https://doi.org/10.1016/j.ipm.2019.102078 32. Resnick, P., Varian, H.R.: Recommender systems (1997). http://www.firefly.com 33. da Silva Lopes, A., et al.: Smartufsm: Uma arquitetura de software para suporte a recomendacoes em campus universitarios inteligentes (2022) 34. Tarus, J.K., Niu, Z., Yousif, A.: A hybrid knowledge-based recommender system for elearning based on ontology and sequential pattern mining. Future Gener. Comput. Syst. 72, 37–48 (2017). https://doi.org/10.1016/j.future.2017.02.049 35. Thannimalai, V., Zhang, L.: A content based and collaborative filtering recommender system. IEEE Xplore, pp. 77754–77766 (2021). https://doi.org/10.1109/ICMLC54886.2021. 9737238 36. Uddin, I., Imran, A.S., Muhammad, K., Fayyaz, N., Sajjad, M.: A systematic mapping review on MOOC recommender systems. IEEE Access 9, 118379–118405 (2021). https://doi.org/ 10.1109/ACCESS.2021.3101039 37. Valdiviezo-Diaz, P., Ortega, F., Cobos, E., Lara-Cabrera, R.: A collaborative filtering approach based on na¨ıve bayes classifier. IEEE Access 7, 108581–108592 (2019). https://doi. org/10.1109/ACCESS.2019.2933048 38. Villegas-Ch, W., Arias-Navarrete, A., Palacios-Pacheco, X.: Proposal of an architecture for the integration of a chatbot with artificial intelligence in a smart campus for the improvement of learning. Sustainability (Switzerland) 12, February 2020. https://doi.org/10.3390/ su12041500 39. Villegas-Ch, W., S´anchez-Viteri, S., Rom´an-Ca˜nizares, M.: Academic activities recommendation system for sustainable education in the age of covid-19. Informatics 8, June 2021. https://doi.org/10.3390/informatics8020029 40. Wan, S., Niu, Z.: A hybrid e-learning recommendation approach based on learners’ influence propagation. IEEE Trans. Knowl. Data Eng. 32, 827–840 (2020). https://doi.org/10.1109/ TKDE.2019.2895033 41. Wang, H.: Design and implementation of web online education platform based on user collaborative filtering algorithm, pp. 2911–2918. Association for Computing Machinery, September 2021. https://doi.org/10.1145/3482632.3487539

22

M. H. M. Maruyama et al.

42. Xiao, J., Wang, M., Jiang, B., Li, J.: A personalized recommendation system with combinational algorithm for online learning. J. Ambient Intell. Humanized Comput. 9, 667–677 (2018). https://doi.org/10.1007/s12652-017-0466-8 43. Xu, X., Wang, Y., Yu, S.: Teaching performance evaluation in smart campus. IEEE Access 6, 77754–77766 (2018). https://doi.org/10.1109/ACCESS.2018.2884022 44. Yang, C.T., Chen, S.T., Liu, J.C., Liu, R.H., Chang, C.L.: On construction of an energy monitoring service using big data technology for the smart campus. Cluster Comput. 23, 265–288 (2020). https://doi.org/10.1007/s10586-019-02921-5 45. Zhang, Y., Chen, M., Huang, D., Wu, D., Li, Y.: idoctor: personalized and professionalized medical recommendations based on hybrid matrix factorization. Future Gener. Comput. Syst. 66, 30–35 (2017). https://doi.org/10.1016/j.future.2015.12.001 46. Zhang, Y., Yip, C., Lu, E., Dong, Z.Y.: A systematic review on technologies and applications in smart campus: a human-centered case study. IEEE Access 10, 16134–16149 (2022). https://doi.org/10.1109/ACCESS.2022.3148735 47. Zheng, K., Yang, X., Wang, Y., Wu, Y., Zheng, X.: Collaborative filtering recommendation algorithm based on variational inference. Int. J. Crowd Sci. 4, 31–44 (2020). https://doi.org/ 10.1108/IJCS-10-2019-0030

Encouraging Grading: Per Aspera Ad A-Stars Pia Niemel¨a(B) , Jenni Hukkanen , Mikko Nurminen , and Jukka Huhtam¨aki Faculty of Information Technology and Communication Sciences, Tampere University, P.O. Box 1001, 33014 Tampere, Finland [email protected]

Abstract. The surge in computer science student enrollment in Data Structures and Algorithm course necessitates flexible teaching strategies, accommodating both struggling and proficient learners. This study examines the shift from manual grading to auto-graded and peer-reviewed assessments, investigating student preferences and their impact on growth and improvement. Utilizing data from Plussa LMS and GitLab, auto-graders allow iterative submissions and quick feedback. Initially met with skepticism, peer-review gained acceptance, offering valuable exercises for reviewers and alternative solutions for reviewees. Autograding became the favored approach due to its swift feedback, facilitating iterative improvement. Furthermore, students expressed a preference for a substantial number of submissions, with the most frequently suggested count being 50 submissions. Manual grading, while supported due to its personal feedback, was considered impractical given the course scale. Auto-graders like unit-tests, integration tests, and perftests were well-received, with perftests and visualizations aligning with efficient code learning goals. In conclusion, used methods, such as auto-grading and peer-review, cater to diverse proficiency levels. These approaches encourage ongoing refinement, deepening engagement with challenging subjects, and fostering a growth mindset. Keywords: Learning management system · Next-generation learning environment · Assessment and feedback · Automatic grading · Manual grading · Peer-reviews · Leaderboards · Learning analytics · Growth mindset · The theory of formative assessment · Flipped learning

1 Introduction The adage “You get what you grade” emphasizes the significance of dependable, impartial, and efficient grading methods. To accomplish this, grading must be designed to provide immediate and ongoing feedback to students, allowing them to make adjustments in real-time and improve their performance in a timely manner. However, this approach must also be able to accommodate ever-increasing student populations, which seems like a daunting task. The complexity of the grading task deepens with the introduction of another grading concept, referred to as “Not yet,”. In contrast to offering definitive evaluations of a student’s potential, the phrase “Not yet” embodies a declaration of their dedication to progress. Open-ended “Not yet” is rooted in the principles of the growth mindset [9], c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024  B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 23–46, 2024. https://doi.org/10.1007/978-3-031-53656-4_2

24

P. Niemel¨a et al.

which leaves the door open for future improvements. Dweck elucidates the ideal nature of a learning task as follows: Let’s give students learning tasks that tell them, “You can be as smart as you want to be.” Hence, the growth mindset encapsulates an ongoing prospect for continuous learning and evaluation. This mode of assessment is facilitated by formative assessment, which centers on furnishing feedback and insights to enhance student learning throughout the learning journey. To foster a student’s growth, a nourishing blend of encouraging feedback and evaluative processes is indispensable [5]. This includes incorporating selfreflective practices that empower students to provide self-feedback [39]. The feedback should not only criticize but be constructive and provide means for self-correction in a timely manner [26]. Furthermore, the quicker the feedback is at hand, the greater its perceived benefits [54]. Formative assessment is employed within the realm of flipped learning as well. In return, the concept of flipped learning has gained significant momentum within university environments, promising heightened learning experiences and improved educational outcomes. Notably, the University of Eastern Finland (UEF) has played a pioneering role in the adoption and integration of flipped learning practices. This journey began with physics instruction, initially on a smaller scale with only a handful of students participating [37]. Problem-solving activities were shifted as small group discussions, motivating better preparation, collaboration that fostered more effective learning than traditional lectures. As educators at UEF extended the application of flipped learning across various disciplines such as forestry, biology, education, and medicine, while also conducting research and disseminating success stories highlighting its positive influence on student engagement and achievement [19], other universities began to express growing interest in the approach. Tampere University invited Saarelainen to demonstrate the successful initiation of flipped learning. Notably, the faculty of mathematics has been involved in this endeavor and since 2019, MathFlip research project has collected feedback on flipped instruction in engineering mathematics [20]. The findings so far show that collaborative activities, peer support, and tasks promoting conceptual understanding contributed to deep approach learning [35] and that videos and primetimes have significant influence in learning in flipped intervention groups [25]. Primetimes are employed to deepen the learning; during the primetime sessions, small groups of students can ask questions to the instructor [24]. Following this, students can apply the learned topics in weekly exercises that involve both theoretical analysis and programming. Flipping has been advocated as a teaching method not only in mathematics but also in computer science courses by Tampere University’s internal Flip&Learn project. In contrast to small-scale mathematics classes, computer science classes may be huge, with hundreds or even over thousand participants. The feasibility of implementing this method on such a large scale has raised concerns among the teaching staff. For instance, when dealing with huge courses, it becomes impractical for an instructor to facilitate primetimes with just four students. The studied Data Structures and Algorithms course (DSA-2022, N = 605) is only partially flipped. In a substantial course like the DSA-2022, the concept of primetime

Encouraging Grading

25

finds its most prominent application in the form of multiple parallel discussion sessions, which draw in dozens of participants, particularly in Zoom sessions as opposed to in-premises settings. While the original notion of primetime and its inherent intimacy might not be fully preserved at these scales, this arrangement aligns with the best possible outcome given these magnitudes. Following the principles of flipped learning, instructional content is presented in the form of videos. I Notably, no slides but only archival backup slides do exist. The videos are cut in short clips of 15.. 30 min, because cutting the material in shorter portions has proven to increase student engagement in the earlier studies [3, 40, 47]. To ensure that the most relevant points have been internalized, DSA-2022 expects students to answer multiple choice questions after watching the videos. The programming exercises in DSA-2022 aim to minimize the need for additional support from course personnel. To achieve this, clear and well-structured exercise instructions and feedback should be provided. Before and during the Fall 2022 semester, the course staff dedicated themselves to the task of automating all exercises and assignments for the course. Although some of the weekly exercises had been automated in previous iterations of the course, the current initiative sought to automate the remaining exercises. This included the more substantial final assignments and the examinations. The primary objective was to develop auto-grading tools that could provide students with valuable and constructive feedback, ultimately enhancing their learning and overall academic growth. While advancing the implementation of auto-grading systems, our intention was to base the process on clear evidence and sound pedagogical practices. In pursuit of this, we collected students’ feedback regarding the grading methods employed with pre/post-surveys. This study examines the advantages and disadvantages of various grading methods, with a primary focus on manual and auto grading. The effects of the change are evaluated through monitoring learning outcomes and surveying students about their perceptions of the shift from manual (the previous) to automatic grading (the new). These findings will serve as crucial inputs for refining and enhancing future course implementations. Preliminary results from this research were previously presented in our original publication [27]. This study builds upon that work by conducting a more comprehensive literature review, particularly emphasizing strategies to support struggling students. Additionally, we perform a more in-depth analysis, specifically exploring submission counts and their correlation with students’ grades for a selected exercise. To gain a deeper understanding of how students perceive the integration of auto-graders, we pose the following research questions: 1. Which grading styles students prefer the most and why? 2. Which auto-graders were appreciated the most? 3. Which were the students’ desired submission counts and what were their consequences? 4. How submission counts correlate with students’ grades?

26

P. Niemel¨a et al.

2 Related Work To ensure the quality of instruction in online or hybrid settings, it is essential to draw guidance from established pedagogical frameworks such as the growth mindset [10], the theory of formative assessment [5, 6], and flipped learning [4] when implementing autograding tools for DSA-2022. These frameworks offer insights into how students benefit from feedback, engagement, and self-directed learning, all of which should inform the development and deployment of grading tools. Moreover, integrating formative assessments and active learning strategies, as advocated by flipped learning, can enhance the efficiency of the learning process. The upcoming sections will specifically explore these enumerated pedagogical frameworks and tools with a focus on grading and feedback. The feedback provided should be both encouraging and constructive, catering particularly to students facing challenges, aiding them to gracefully pass the course, and fulfill the learning goals. 2.1

Growth Mindset

Central to identifying a growth mindset is a student’s perspective on challenges. When a challenge is seen as exciting rather than intimidating, it indicates a growth mindset. A student with a growth mindset welcomes the chance to learn something new and difficult. Conversely, a fixed mindset is its opposite [16]. Those with a fixed mindset are limited by feelings of shame and a fear of failure. This fear is rooted in the risk of being exposed as lacking in intelligence. In a fixed mindset, this outcome is seen as tragically final: intelligence and skills are perceived as inherent traits, leaving no room for improvement. However, the growth mindset takes a different stance. It encourages students to embrace challenges, fostering adaptability and resilience while nurturing skill development. Students with a growth mindset possess confidence in their progress and hold positive expectations of attaining their academic objectives, which connects to the concept of self-efficacy [2]. Educators should harness this growth-oriented mindset, enabling students to regulate their own learning and develop their metacognitive abilities. Teachers aiming to design challenging and meaningful learning tasks encounter diverse student responses: students with a growth mindset approach challenges eagerly, while those with a fixed mindset may feel threatened by the need to stretch or take risks. Risk taking should be cherished and rewarded. This can be achieved through appropriate praise and encouragement that focuses on praising the effort, strategies, choices, and persistence students display during the learning process, rather than simply labeling them as “smart” upon success [9]. Dweck together with her research group have developed a reward system, “brain points,” to encourage the development of growth mindset behaviors by directly incentivizing effort, use of strategy, and incremental progress. [10, 32, 33] Results show that brain points are capable of encouraging the growth mindset by increasing persistence, time spent playing, strategy use, and perseverance, especially in low-performing students. By providing incentives for effort and progress, players are more likely to engage with the game and continue to challenge themselves, thus fostering the growth mind-

Encouraging Grading

27

set among players. Students will be encouraged to see challenges as opportunities for growth and development, rather than as a source of frustration. 2.2 Formative Assessment The theory of formative assessment is based on the idea that assessment should be used as a tool for improving both learning and instruction, rather than just evaluating them. Formative assessment takes place throughout the learning process, it also provides feedback not only for students but also for course personnel on what is being learned and what needs to be improved in the course design. For instance, the exercise instructions and feedback provided for students are often refactored in anticipation of a smoother completion of exercises. According to Clark, there are three facets of feedback that hold the potential to influence a learner’s metacognition and self-efficacy. These facets are formative, synchronous, and external/internal feedback [6]. Clark characterizes formative feedback as an assessment approach that fosters self-regulation, synchronous feedback as immediate feedback, and internal feedback as the way learners engage in self-talk. Internal feedback generation is common among self-regulated learners and is often initiated after receiving external feedback initially. Flipped learning (FL) has grown into a popular approach that involves students watching video lectures before engaging in active learning activities such as problemsolving and discussion. Assessment in FL typically includes formative assessments that take place throughout the course. 2.3 Types of Feedback, Focus on the Process Feedback can have a transformative significance for learning and error reduction. Feedback and scores can be provided not only for tasks (Task Feedback, FT) but also for the process (Process Feedback, FP) and self-regulation (Self-Regulation Feedback, FR). The latter includes factors like student persistence, timing of submissions, and goal attainment. Feedback can also be directed towards the student personally, usually involving positive reinforcement that praises the individual rather than solely focusing on their performance (Feedback to Self, FS) [18]. Automated feedback systems can generate FT solely based on the submission, without requiring user information. Other assessment methods, however, necessitate user identification and some level of data storage and profiling. The growth mindset considers both FP and FR, emphasizing the process and rewarding persistence in problemsolving. FS is risky, and its use is not recommended. In the context of the growth mindset, giving feedback to the individual, whether positive or negative, is generally discouraged. Excessive FS is often linked to encouraging poor students in particular. If then a student receives such feedback, their implicit assumption might be that the teacher had low initial expectations of them. Self-determination theory and intrinsic motivation also play a role. The extrinsic motivation, praise and price, fall short in inducing long-term motivation, instead, the student should find the source of intrinsic motivation [8]. In the realm of computer science, testing serves the purpose of FT. Tests even be written in advance, as in test-driven development, precisely measuring the achievement of agreed-upon objectives [13]. Agile team meetings, where issue boards are updated

28

P. Niemel¨a et al.

and burn-down charts are monitored, provide feedback on schedule and goal adherence, resembling FP and FR feedback. Peer accountability and short intervals, e.g., daily meetings, can help students with weak self-regulation meet their goals. The Agile process also recommends retrospectives at project ends, reflecting on successes, failures, and listing long-term development goals, which could be personal. In such cases, feedback takes forms of FP, FR, and sometimes even FS. Flipped learning and formative assessment move the focus from end result to the process, from the direction of FT more to FP. Flipped learning seems to enhance students’ academic achievements and foster their dedication to the learning journey. The student transitions from a passive participant to an active learner, cultivating essential skills such as information selection, collaborative teamwork, critical thinking. Additionally, flipping encourages self-assessment and self-regulation [17]. All forms of instruction, including feedback, should enhance student autonomy rather than fostering dependency. Students must increasingly learn to progress, make informed choices independently or with peers, and reflect on their actions without requiring constant teacher feedback. 2.4

Struggling Students

In particular, the assessment should help struggling students to improve their performance. In recent PISA results from Finland, a notable trend is the significant decline in boys’ performance, while girls’ performance remains consistent and strong [34]. The article, published in 2023 in Helsingin Sanomat, speculates that this discrepancy could be attributed to gaps in reading skills, motivation and commitment to studies in general. One factor to worse performance of boys is allegedly the assessment. According to Harmanen, Senior Education Adviser from the Finnish National Agency for Education, boys have to some extent suffered from an assessment approach that heavily emphasizes the performance in exams, but she empasizes, “It’s really important that gender would not influence assessment.” The assessment should be flexible enough, and not consistently conducted in the same manner, e.g., exams being the only way to demonstrate one’s skills and competencies. All in all, Harmanen emphasizes the importance of pedagogy being diverse overall, “It could include elements like gamification, even competitive aspects.” Furthermore, there is a growing suspicion that the heightened reliance on digital devices for educational purposes links to adverse learning outcomes and negative effects on self-regulation, visible especially in subjects such as mathematics, natural sciences, reading, and collaborative problem-solving. Worries have emerged regarding the students’ capacity to focus, to concentrate in reading and grasp concepts effectively. Specifically, reading predicts good academic success, so investing in it is especially worthwhile [50]. Likewise, Saarinen in her dissertation [38] draws attention to a link between increased digital learning and declining learning outcomes reported in PISA-2015 and −2018 [30, 41], which has sparked discussions and also critics among Finnish educators. Saarinen further emphasizes that the current trend of increased autonomy particularly affects vulnerable groups like underprivileged male students and students from immigrant and low-income families. To address this, Saarinen suggests that fostering

Encouraging Grading

29

supportive and confidential relationships can promote the development of mature character, cooperation, and enhanced self-regulation skills. As a result, the effectiveness of the flipped learning approach, which underscores increased student autonomy and responsibility, may paradoxically present a substantial challenge during its implementation. Rather than experiencing the anticipated sense of empowerment in their learning journey, students who are already struggling might find themselves overwhelmed and uncertain about how to effectively navigate in their studies, and as stated, in Finland, this challenge has a more pronounced impact on male students. Utilizing learning analytics and statistical tools enables the differentiation and adaptation to students’ learning experiences. Effective dropout prevention strategies encompass innovative systems for predicting and addressing dropout risks, along with early interventions and continuous follow-up [53]. Beyond diminishing dropout rates, learning analytics can elevate the quality of education, empowering students to assess and evaluate their own progress [42]. Consequently, introductory computer science courses at the university (CS1) could provide well-structured scaffolding, which has demonstrated its effectiveness in reducing dropout rates [15]. Also moving from task-based FT-type of assessment to assessing the process and fostering the growth mindset is anticipated to have positive effect on motivation and commitment [27]. In conclusion, the assessment also carries a significant implication in terms of commitment on the studies and signaling potential dropouts early enough. The level of support, however, requires a thoughtful approach to avoid excessive hand-holding, which has been linked to diminished learning outcomes [51]. Ultimately, the required level of autonomy should increase gradually, culminating in the ability to confidently embrace self-directed learning towards graduation. In their future professional life, this adeptness at managing autonomy proves invaluable especially given the continuous growth of flexibility and uncertainty in the job market [43].

3 Research Context The studied DSA course provides a comprehensive introduction to data structures and algorithms, from insertion- to merge sort, from data structures sequences and sets and algorithm analysis including the performance implication of data structure selection. Due to COVID-19, DSA course replaced earlier live lectures with video recordings, and on-premises tutoring with online tutoring sessions. Special Q&A sessions were provided for tutoring purposes. In the sessions, teachers answered questions stated by students and went more in detail than in video lectures. Q&A is similar concept to primetime sessions that are utilized in flipped learning [24], except in DSA-2022 participation was voluntary and not rewarded by points. The removal of points resulted in a remarkable decline in participation compared with previous implementations. The exercises and assignments demonstrate how well the content was internalized. Students struggling with the exercises could get help in Teams where teaching assistants give hints and scaffold students in solving the problems. However, while doing the exercises students should get a sufficient amount of feedback from the developed

30

P. Niemel¨a et al.

Fig. 1. Perftests: resource-consuming and resource-savvy versions. The figure is originally published in [27].

auto-graders to be able to fulfill the requirements. The assignments are designed to be completed independently by the students, and successfully completing the assignments is evidence of adequate learning. Assignment 1 and 2 are divided into two phases: compulsory and optional. Students have the option to accept the results from the compulsory phase or choose to progress to higher grades. Additionally, students may choose to forgo Assignment 2 entirely, resulting in a maximum grade of 2. These various grade options can be viewed as an implementation of flipped assessment, which grants students more control over their assessments, allowing them to be more autonomous and focus on areas that align with their strengths and interests [49]. 3.1

Tools Used: Plussa, OpenDSA and Gitlab

Plussa is the learning management system that was used in the course. Originally Plussa has been developed in University of Helsinki [22]. Original LMS is a service-oriented architecture that was designed to be easily extendable. This allows for the addition of new services, such as auto-graders, to enhance its functionality [21, 22]. Open Data Structures and Algorithms (openDSA) exercises are such enhancements, but basically anything is possible: the docker container with an image of developer’s choice will be launched, and the git repository is cloned inside the image and tests will be executed inside this sandbox. OpenDSA exercises are especially designed to support growth via formative assessment. OpenDSA is a open-source community creative-commons project; a broad community of developers was involved in the effort with the incentive of re-using the materials in their own courses. The University of Virginia has been one of the main promoters; in Finland, Aalto University has been active. The project has achieved a wide variety of self-study resources for learning data structures and algorithms, such as interactive visualizations, quizzes, and coding exer-

Encouraging Grading

31

cises to help students learn and practice computer science concepts [12, 44]. The visualizations cover different algorithms [23, 48] and runtime behaviors [46], that is, the illustration of call stack and heap behavior while executing an algorithm, such as recursion (e.g., Annotation editor exercise about recursion). Recursion as well as time complexity are identified as one of the threshold concepts in computer science [55], and visualizations are an apt tool for lowering the threshold [45]. OpenDSA comprehension aids are good for novices, but to advance students’ craftsmanship in software engineering the courses need no toys but real tools for improving the efficiency and quality of code. Submission through GitLab repos and having real unit and integration tests would better future-proof their development as becoming coders [14]. In addition to Plussa and openDSA, Gitlab is a central tool starting from the middle of the course. Besides functioning as a normal version control system, Gitlab is the means to get instructions pulled from the course upstream and, on the other hand, submit code for grading. The course upstream is a Gitlab repository for pulling only. Course personnel maintain the upstream, new instructions and possible file skeletons are released at the beginning of each exercise round. To complete the course, students had to pass weekly exercises, a coursework assignment, and an exam. The manual assessment of assignments is the most resource-consuming task, thus its automation was the first priority. 3.2 Auto-Graders for Assignment In 2022, the automation of the course was leveled up by introducing multiple autograders to check various aspects of students’ code.

Fig. 2. Plussa and Gitlab co-operate during grading. The figure is originally published in [27].

Figure 2 illustrates auto-graders in action. First, students pull instructions from the course-upstream repository. They then commit their code to their own repository, and finally, submit the Gitlab URL in Plussa system, which is responsible for submission and grading. Plussa system is divided into two parts: the Plussa front-end and

32

P. Niemel¨a et al.

Fig. 3. Preferred grading style. The figure is originally published in [27]. Table 1. Summary of statistics for the preferred grading style for the 240 students who answered to both questionnaires. The table is originally published in [27]. average pre average post diff of post and pre std of the diff % of changed opinions manual auto peer-review self-reflection comparison la

4.14 6.12 2.54 3.52 3.45 4.17

4.8 5.96 3.45 3.89 3.86 4.29

0.66 −0.16 0.91 0.37 0.41 0.12

1.66 1.31 1.73 1.91 1.83 1.96

76.25 54.17 73.33 72.50 70.83 75.00

the MOOC grader. The system launches temporary Docker containers that are only started for performing the grading. The grader clones the student’s git repository and executes the grading as instructed in a shell script. Examples of graders include perftest, unit, integration and Valgrind graders. Most of the tests are also given to students, and the respective Plussa graders run the same tests. By running the tests locally, students receive the same feedback as given by the Plussa graders, which decreases the number of needed submissions. This also gives students an idea of how their work will be graded in the LMS. After grading, the points are returned to the Plussa front-end, which stores the grades. The primary goal of Data Structures and Algorithms (DSA) is to learn how to write efficient code. This goal is emphasized by setting more emphasis on performance testing, with the measurement of the time that program takes to run, i.e., its performance. However, measuring performance using a stopwatch can be affected by other processes running on the computer at the same time, thus competing for the same resources. To overcome this issue, an instruction counter is used instead, which measures the number of instructions that are executed by the program. It is not susceptible to interference from other processes. However, the instruction counter requires certain conditions to be met, such as having perf events open on the server. To meet these requirements, a dedicated ESXi server with an instruction counter was setup for this course implementation. In DSA-2022, performance tests were conducted by gradually increasing the amount of data (e.g., N =10, N =100, N =1000,..). The instruction counts were recorded, and then a curve was fitted using Python to estimate the average complexity of the algorithm. The results were illustrated as a graph, showing the average time complexity as a function of the number of data points, see Fig. 1. To ensure accurate results,

Encouraging Grading

33

the input data used for curve fitting should have minimal noise and the tests should be repeated multiple times to average out the effects of randomized data or other sources of noise. 3.3 Method and Data Collection The DSA course is developed on a yearly basis. The method used is a design-based research (DBR) approach, which involves cyclical development with reflective redesign phases [7, 11, 36]. In DSA-2022, this development of grading system is guided by the theory of formative assessment [5, 6] and involves combining educational solutions with empirical interventions and proof. The DBR approach includes four stages: design, development, enactment, and analysis [1, 31, 52]. The cycle represents a course term and the retrospective analysis is used to inform the design of the next implementation. We collected students’ views with two grading related questionnaires during DSA-2022. The first, pre-questionnaire, was carried on in the middle of the course, in module 8 (N=360) before Assignment 1. The second questionnaire was executed as a post questionnaire in module 14, after Assignment 2 (N= 274). 240 students answered both questionnaires. The questionnaires contained both Likert-scale and open-ended questions. First, we scan through the Likert-scale questions about the preferred grading styles. Likert scales are divided in seven levels from ‘Strongly disagree’ (0) to ‘Strongly agree’ (7), and interpret the results by selecting enlightening quotations from students’ responses to open-ended questions. In addition, we utilize the grader feedback and logs in scanning the learning process. The redesign of the course is done based on the results, and adjustments are made, where feasible.

4 Results and Discussion When transferring from resource-intensive manual grading to auto-grading, the main dilemma is how to keep up the good quality of feedback. During this course, alternative ways of giving feedback were experimented, such as auto-graders, peer-reviews, and comparisons with others. Learning analytics and self-reflection were also mentioned, and students were asked to rate these grading methods, yet they were omitted during this implementation. In Fig. 3, the automatic grading gets upvoted the most in both phases, followed by learning analytics and manual grading, but as we can see, in the latter Fig. Assignment 2, the difference between automatic and manual grading is significantly less (in pre-test, 1.98, in the post 1.2, see Table 1). The reasons for automatic grading dropping in are the problems with perftest in Assignment 1, yet the simplified (thus faster) perftests in Assignment 2 managed to compensate for the felt discomfort.

34

4.1

P. Niemel¨a et al.

Auto Grading

Auto-grading was by far the most appreciated grading method. In their comments, students praised the speed of the graders and the immediacy of the feedback. In addition, students greeted with joy the possibility to return their solution “unpolished” multiple times, such as the following respondent, -Also auto graders don’t (hopefully) think you are a complete idiot if you try to submit something that is not yet well polished. The uniformity and fairness of the evaluation was emphasized especially in comparison with peer-reviews that were not trusted that much. To a minor extent, manual grading was attached with trust issues as well, such as: - Personal written feedback would’ve been very nice, but emphasis on the word personal. and - There could also be some unknown bias causing certain students to receive better/worse grades than they deserve. Auto-graders have advantages in terms of transparency and continuous evaluation (characterized by formative assessment) allowing for intermediate inspections and communication with the grader. However, there were issues with Assignment 1, specifically with perftests, which compromised fairness and caused timeouts due to the slowness of visualizing and estimating asymptotic efficiency. These problems were addressed in Assignment 2 by simplifying the perftests, resulting in a smoother deadline. Refactoring of Assignment1 perftests is the top priority. 4.2

Manual Grading

Many students prefer manual grading because of its depth and being personalized. A student pointed out the need for constructive feedback, e.g., hints for better readability and efficiency, and someone highlighting them good coding conventions. (Yet checking of coding conventions can also be achieved with linting tools.) However, sometimes the feedback can degrade as dealing with irrelevances, such as highlighted in the following response, -Manual grading has on previous courses put enormous weight on nitpicking, which was non-existent now. Students were also concerned about putting their effort to the exercises in vain, If a student cannot do code that passes tests but still has SOME code and has used many hours for the work, the grade cannot be 0. There needs to be some manual grading for that pass/fail situation. Like, you could try just to pass (grade 1) OR get a better grade with autograders. For example, Valgrind grader represents such a pass/failure grader: it did not allow any memory leaks for pass, causing ‘passing panic’ among students. While universities must maintain the quality of graduates, the amount of work should not automatically compensate for lack of skill. It seems that students rely on humans being more empathetic than machines in this sense. Multiple responses highlighted the size of the course as a constraint, where manual grading was identified as the method that consumed the most teaching resources, thus it was seen as unrealistic. Additionally, the slowness and an option of returning work only once were seen as downsides of it. 4.3

Peer-Reviews

In the pre-questionnaire, students were very suspicious about the functionality of peerreviews. Ignoring the initial dismay, peer-reviews were introduced in the last mod-

Encouraging Grading

35

ule of the course, and allegedly the strongest objection has softened after the peerreview period. The primary reason for softening was the fact that peer-reviews did not directly contribute to the grade of reviewee. Also seeing others’ solutions was considered enlightening. However, each review is not equally worthy, I think peer-reviewing is a good idea, however I would have wanted to see a solution that was more performant than mine, or alternatively get feedback from an author of such a solution. Peer reviews depend on the reviewer. Currently it’s a pure lottery. If a peer really reads the submission and knows how to give meaningful feedback, peer-reviews are good. I dislike peer reviews as the reviewing is not done by a person that has a clear understanding of the grading in the course and I feel like I am not being graded as fairly as I could have. Students desire feedback from someone whose domain knowledge is equal or better than their own. To address this, the simplest solution is to increase the number of reviews, as this increases the chances of receiving feedback from a knowledgeable reviewer. However, in some courses increasing the number of feedback items a student has to give has led to a perceived decrease in the quality of the feedback. Peerreviews are inherently less equal in quality when compared to automatic grading, as they are subjective. Students should be given incentives to provide high quality feedback to their peers. Additionally the grading of the outcome of the peer-review process should be considered in order to ensure the quality of the review. The quality checking of the reviews calls for innovative ideas. Extending the peer-review process to include students assessing the feedback they have received could be a relatively straight-forward way to improve the process. Students find it difficult to comment on work if they are still beginners or there is a gap in skill levels, but with proper instructions, the task becomes easier. Thus, the conclusion is to consider increasing the number of peer-reviews and put effort into providing clear and useful instructions to aid the process. 4.4 Comparisons In DSA-2022, the comparison meant perftest top-10 leaderboard. The leaderboard listed students’ initials hinting the name but not revealing it fully. Students listed in the leaderboard may take it as an honor. Of course, being omitted from the board may be discouraging, or felt unequal: -Comparison with other students is good but if there is only the 10 best, hundreds of people get left out and don’t see how they compare. So i think that everyone should be displayed on the leaderboards. Leaderboard has its problems either way, whether with only selected people or all included, yet the ethical problems would peak in publishing “bottom boards”. In students’ responses, the leaderboard induced a lot of negative feedback. The following collection itemizes the reasons for strong objection: Making this a competition lets only the top students flex their coding muscles when those who need actual help might feel very stupid. Seeing other peoples progress puts too much competitive pressure on students. The reality is that sometimes we must prioritize our courses and work in order to stay sane

36

P. Niemel¨a et al.

and still learn as much as possible. Seeing other people’s progress to that function only wrecks havoc. Comparing yourself to others can be really harmful to your self-esteem and many students already struggle with mental health and imposter syndrome. In the comments above, competition is considered having a negative effect on one’s motivation, yet opposite opinions exist: I’m very competitive so getting leaderboards is fun and increases my motivation substantially. After reading the internal review I almost wanted to go back to working on prg1 just so I could see if I could get my perfs down to the best ones :D. Taking others’ progress as a positive challenge exemplifies the growth mindset. The information of the leaderboard can be also found as an additional means of guidance: If we were shown some leaderboards during the development, it would have given us some indication that we were on the right path with the project. Leaderboards and comparative statistical analysis can be provided for students as extra means to follow their progress. Preferably, a student should be able to control whether the information is shown or not. 4.5

Learning Analytics

In larger courses, such as DSA-2022, all additional and summarizing tutoring would be welcome. The attempt is to use statistical information at the benefit of students, yet the exact means are to be found. It is also good to be aware of the delicacy of the issue and focus on discrete handling of the data. Most of all, students wish encouragement and scaffolding with hard exercises, and desire learning analytics for personalized exercises: I would be open to learning analytics, if they are user-friendly and give you gentle suggestions and encouragement when you do well. However, at the same time they marvel how personalizing will affect the grading, for example: I don’t know how this would be implemented in practice; would everyone be able to get as many points if different exercises are suggested, would everyone be able to complete the same exercises even if they aren’t suggested? Making grade predictions could be useful. 4.6

Fairness Aspects

Most students confirmed the claim “I have been graded fairly”, yet there was also a group of students strongly disagreeing with it. Responses to the questions “I deserve a better grade that I am getting” and “I am getting a better grade than I deserve” primarily are left-biased, that is, heavily disagreed, yet the latter even more heavily. In accordance with human nature, students would easily accept a better grade, not vice versa. The claim “Satisfied with the feedback” indicates that the feedback provided to the students is well-received, even though there may be room for improvement. The claim “I would have preferred personal written feedback from the course staff” is also agreed, but not as much as the previous question, this means that while students are generally satisfied with the feedback they got, they would have preferred to have personal written feedback from the course staff (Fig. 4).

Encouraging Grading

37

Fig. 4. Fairness of grading. The figure is originally published in [27].

4.7 Most Useful Graders As seen in Fig. 5, unit and integration tests were considered to be the most useful form of testing, with visualizations provided by performance tests also being highly valued. However, the use of Valgrind as a grading tool was met with mixed reactions due to its strict nature as a gatekeeper that must be passed in order to receive a grade. The main issue with Valgrind is that it does not allow for any memory leaks or errors and thus some participants found it to be too decisive. However, it is questionable to remove graders that check the correctness, where the removal would eventually lower the quality of code. To address the difficulties with Valgrind, providing better resources and guides for effectively using the tool is crucial. This could include incorporating Valgrind checks when learning about invalidating pointers and provide a guide for tackling the most common memory leaks and Valgrind errors. This way, developers can learn how to use the tool in a more effective and efficient manner and avoid common pitfalls.

Fig. 5. Usefulness of the auto-graders. The figure is originally published in [27].

38

P. Niemel¨a et al.

4.8

Submission Count

The submission count, which refers to the number of possible submissions to autograders, was initially high at 150 but was later, in Assignment2, decreased to 20 with the ultimate goal being 3. The desired submission count was asked in the second survey, and the results are shown in Fig. 6. Responses that suggest submission count of 50 or less total three quarters. The weighted average is still far from the intended goal of ten. Figure 7 shows the histogram of the submission counts that students used for one particular grader in Assignment 1. The grader is perftest and it was the most heavily used of all the graders. It is still to be noted that 94% of students submitted at most 20 times, even if there are a few students that needed more than 50. In Assignment1, the unit, integration and performance test codes were released in the middle of the submission period; whereas in Assignment 2, the test codes were readily available right from the start. The test codes allowed students to test their code locally, and for Assignment 2 the submission counts were much smaller than in Assignment 1. This suggests that the majority of students are able to test their codes, if test codes are released. However, students desired to have higher submission counts in auto-graders than they actually needed. This may stem from the need to ensure the possibility to test with the grader, if the test codes are not made available or they behave differently in local testing versus in the learning management system. In any case, students must be encouraged to test their codes by themselves rather than testing against ready-made tests.

Fig. 6. Desired submission counts. The figure is originally published in [27].

Fig. 7. Submission counts of the perftest grader. The figure is originally published in [27].

If manual grading had been employed, dealing with such a high volume of submissions and providing extensive feedback would have been unattainable. At our university, a few preceding courses have transitioned from manual to automatic grading [28, 29]. These prior experiences affirm the findings of this study: when students are given the opportunity to refine their code gradually, they embrace this option and learn iteratively through subsequent versions. Another evident yet equally significant outcome is the conservation of course personnel resources. Given the scale of this course (N = 605), automated grading becomes a necessity. The combination of a substantial number of submissions, diverse formative

Encouraging Grading

39

feedback from multiple graders, and the presence of Q&A discussion forums allows saved course resources to be allocated effectively, with human involvement focusing on delivering constructive feedback. However, more emphasis should be placed on fostering social cohesion and collaboration among students during the courses. In the post-COVID era, a growing number of our university’s students have transitioned to remote learning modes. Encouraging their return to campus and re-engagement in team-based work presents a novel challenge. 4.9 Growing with Each Submission and Grade We conducted a detailed analysis of students’ submissions in Assignment 1 to the perftest grader, focusing on those submitted within the deadlines. The histogram of all submission counts for this grader is already shown in Fig. 7. The reason for this investigation is that the grader is compulsory for passing the course, and it is supposed to test the essential skills from the course perspective. Namely, the grader sums up the results of several functions performances tests into points. These points are then being transformed into grades, which are the results of the grader and affect to the grading of the whole course. The grades given by this grader were in the set of {0,1,2,3}, where 0 is fail and the grades of 1–3 passes.

Fig. 8. The histogram of the final grades in perftest grader in Assignment 1.

Fig. 9. The distributions of students’ submission counts needed to reach their final best grade. The distributions are shown for the grades of 2 and 3.

Fig. 10. The distributions of students’ submission counts after reaching their final best grade. The distributions are shown for the grades of 2 and 3.

40

P. Niemel¨a et al.

376 students made submissions to this grader. Table 8 shows the histogram of the final grades. It can be seen that majority of the students got at least the grade of 2, and only few students passing the exercise got the grade of 1 (N = 8). Figure 9 shows the distributions of students’ submission counts which they had used to obtain their final highest grade for the first time. The distributions are shown for the grades of 2 and 3. It can be seen that the grade of 2 was reached with few submissions, and more than 70% of students having the final grade of 2 had reached the grade with their first submission. On average more attempts were needed to reach the grade of 3 than were needed to reach the grade of 2. One students submitted 24 times until he/she reached the grade of 3. The submission counts for the grade of 1 are not shown in Fig. 9, because the final grade of 1 is consisting of only 8 students. Nevertheless, the final grade of 1 was reached at the worst case with 3 submissions.

Fig. 11. Scatter plots for each student’s total submission counts and (a) points at the moment when they reached a grade of 2 or more; (b) their final points for Assignment 1. There are two horizontal lines on the graph. The space between them is for grade 2. Purple dots are for students who got a grade of 2 and kept it. Blue dots are for students who eventually got a grade of 3. (Color figure online)

The distributions of the submission counts after reaching ones own final best grade are shown in Fig. 10. It can be seen that there are students that did not submit or made only few submissions after reaching the grade that become their final grade. In addition, there are in the both grade groups, the grade of 2 and the grade of 3, students that continued their submissions. In the group of having the final grade of 2, they might have aimed for the grade of 3, but were not able to reach that grade at the end. However, in the group of already having the maximum grade, the grade of 3, these submissions were not affecting their course grading anymore, and these submissions were not necessary from the grading perspective. Two students were able to get the maximum 110 points with their first submissions. Out of the 163 students who attained a final grade of 3, 51 students, i.e., 31%, persisted in their submissions and ultimately achieved 110 points. Figure 11 present the scatter plots of each students’ total submission counts. The positions of the points are consistent along the x-axis in both figures. Comparing the scatter plots provides valuable insight into how students progressively improved their

Encouraging Grading

41

scores. Figure 11 (a) shows the students who have recently crossed the threshold for a grade of 2 (≥ 56 points). While the purple dots persist within grade 2, the blue dots are poised to advance to grade 3. Fig. 11 (b) depicts distinct separation among the cohorts based on their final positions. Notably, the grade 3 cohort exhibits higher submission counts, exemplified by a student with 110 points having made 48 submissions. We are particularly interested in the behaviour of blue dots, which show determination in the effort to reach the highest grade. Although both figure pairs, histograms and scatter plots, convey the same information, the histograms provide a more precise representation of submission effort and the contrast between the grade 2 and 3 cohorts. In hindsight, it becomes apparent that the boundaries were erroneously positioned. The distribution resulted in an insufficient number of students attaining grade 1, and conversely, the attainment of the highest grade (3) appears relatively straightforward. Notably, the individuals who dedicated the most effort to achieve their grades are not exclusively those facing academic challenges or poorer grades, since the cohort of grade 3 has the most submission counts. This is highlighted by the observation that the group with a grade of 3 displayed the highest submission counts, with the top score of 110 corresponding to the most substantial number of submissions.

5 Conclusions 1. Which grading styles students prefer the most and less and why? Auto-graders received acclaim for their speed and the opportunity for incremental improvement with each submission. Moreover, students appreciated that they did not need to present themselves in a more favorable light to instructors, as indicated in the following quote: “Also auto graders don’t (hopefully) think you are a complete idiot if you try to submit something that is not yet well polished.” Auto-grading emerged as the most preferred grading method, with substantial support also expressed for manual grading, although students themselves deemed it impractical given the course’s scale. The attractiveness of manual grading lies in its thoroughness and personalized approach. Furthermore, students stressed the significance of receiving constructive feedback and having some flexibility in the grading process. It appears that a few students believe they have room for negotiation regarding their grade when manual grading is in use. They also argue that the effort invested in an exercise should be acknowledged, even if the code doesn’t fully pass all tests. As one student articulated it: “If a student cannot do code that passes tests but still has SOME code and has used many hours for the work, the grade cannot be 0. There needs to be some manual grading for that pass/fail situation.” Peer-review, on the contrary, faced the most criticism initially, but as students became more acquainted with the DSA-2022 peer-review system, their views on it became more nuanced. Students appreciated the opportunity to explore alternative solutions through peer-review. In the peer-review task, students were assigned the responsibility of providing efficiency suggestions to two randomly selected peers. Nonetheless, some students faced challenges in delivering substantial feedback.

42

P. Niemel¨a et al.

Specifically, they voiced frustration when the solutions they were assessing surpassed the quality of their own work. Conversely, a few students highly appreciated the opportunity to learn from superior solutions. On the downside, feedback from uncooperative or disinterested peer reviewers was considered unproductive. 2. Which auto-graders were appreciated the most? Unit-tests, integration tests, and perftests received positive feedback, while opinions on the Valgrind grader were mixed. Perftests, especially when accompanied by visualizations, significantly contribute to achieving the course’s learning objectives, with the focus being on writing efficient code. However, the current implementation of perftest grading needs a thorough overhaul, as it is notably slow and causes congestion. The revised version must be reliable, even when other CPU-intensive processes are running. Ideally, the task of drawing asymptotic efficiency curves and performing curve fitting should be delegated to a web browser rather than a server. This would conserve server resources and enhance the overall user experience. 3. Which were the students’ desired submission counts and what were their consequences? Students expressed a desire for a relatively high number of submissions, with the most common preference being 50 submissions, and a noteworthy portion of students also supported a limit of 150 submissions. In practice, however, such a high submission count should not be necessary, and fortunately, the majority of students did not exhaust their submission limits. Rather than relying heavily on submitting to the Plussa LMS platform, students were encouraged to conduct local testing, which offers several advantages, including faster feedback and alignment with the educational objectives of the CS faculty related to testing one’s code (test-your-code). The initial allowance of 150 submissions for Assignment 1 sent an inaccurate signal. Instead of depending primarily on Plussa LMS, students are encouraged to invest more effort in testing their code locally. Ideally, students should aim to complete the provided test set and even supplement it with their own tests if any specific test cases are missing. 4. How submission counts correlate with students’ grades? The strongest impetus is evident within the grade 3 cohort. This group can be further divided between those who swiftly secure a commendable grade in their initial attempts and those who persevere in their pursuit of the best grades. The highest level of engagement is typically seen in the pursuit of the highest scores, although other factors, such as system testing and exploration of the auto-grader, may also contribute to the high submission counts. In the grade 2 cohort, submissions are notably fewer and distributed more widely across the points spectrum. In conclusion, resolute efforts yield results, leading to grade improvement among students.

6 Further Studies The data collected by Plussa and GitLab is extensive and could be used for learning analytics. The analysis should be made accessible to both teachers and students, with

Encouraging Grading

43

the potential for students to compare their performance to others, although this might create unnecessary competition, yet a safer approach would be for students to compare their performance to their own earlier performance. Plussa graders currently check code quality and conventions, but an additional “self-reflection grader” would be useful in helping students to improve their understanding of their strengths and weaknesses, preferably by providing suggestions for exercises to fill in any gaps, and showing the path to growth. An interesting area of research would be investigating the most effective way to combine automatic grading and the support provided by course personnel. While automatic grading has been shown to be effective and efficient, many students have expressed a need for support from course personnel. The time saved with automatic grading could be used to provide this support and improve teacher-student communication. In the wake of the COVID-19 pandemic, the utilization of Teams for student-peer and student-teacher interactions has surged. However, the current approach to employing these tools is often unstructured and ad-hoc, lacking clear guidelines on their appropriate usage and the allocation of communication responsibilities. This results in an unpredictable and disorganized communication environment, which may fail to fully engage students and promote social cohesion. To address these concerns, a more structured approach is required, which emphasizes the selection of appropriate communication tools and the allocation of specific communication responsibilities to designated individuals. This will facilitate more predictable and organized communication during the course implementation, enhancing the overall learning experience for students. Furthermore, incorporating team-based activities and assignments can provide students with valuable opportunities to hone their collaboration and communication skills, which are highly sought after in the modern workforce.

References 1. Anderson, T., Shattuck, J.: Design-based research: a decade of progress in education research? Educ. Res. 41(1), 16–25 (2012) 2. Bandura, A.: Self-efficacy mechanism in human agency. Am. Psychol. 37(2), 122 (1982) 3. Bergmann, J., Sams, A.: Flip your classroom: reach every student in every class every day. Int. Soc. Technol. Educ. 3, 71–78 (2012) 4. Bergmann, J., Sams, A.: Flipped learning: Gateway to student engagement. International Society for Technology in Education (2014) 5. Black, P., Wiliam, D.: Developing the theory of formative assessment. Educ. Assess. Eval. Account. Form. J. Pers. Eval. Educ. 21(1), 5–31 (2009) 6. Clark, I.: Formative assessment: assessment is for self-regulated learning. Educ. Psychol. Rev. 24(2), 205–249 (2012) 7. Cobb, P., Confrey, J., diSessa, A., Lehrer, R., Schauble, L.: Design experiments in educational research. Educ. Res. 32(1), 9–13 (2003) 8. Deci, E., Ryan, R.: Intrinsic motivation. The Corsini Encyclopedia of Psychology, pp. 1–2 (2010) 9. Dweck, C.S.: Even geniuses work hard. Educ. Leadersh. 68(1), 16–20 (2010) 10. Dweck, C.S., Yeager, D.S.: Mindsets: a view from two eras. Perspect. Psychol. Sci. 14(3), 481–496 (2019)

44

P. Niemel¨a et al.

11. Ericson, B.J., Rogers, K., Parker, M., Morrison, B., Guzdial, M.: Identifying design principles for CS teacher eBooks through design-based research. In: Proceedings of the 2016 ACM Conference on International Computing Education Research, pp. 191–200. ICER 2016, ACM, New York, NY, USA (2016). https://doi.org/10.1145/2960310.2960335, https://doi. acm.org/10.1145/2960310.2960335 12. Fouh, E., Sun, M., Shaffer, C.: Opendsa: a creative commons active-ebook. In: Proceedings of the 43rd ACM technical symposium on Computer Science Education, pp. 721–721 (2012) 13. Grant, P.F.: Formative test-driven development for programming practicals. Ireland J. Higher Educ. 9(2), 1–33 (2017) 14. Haaranen, L., Lehtinen, T.: Teaching git on the side: version control system as a course platform. In: Proceedings of the 2015 ACM Conference on Innovation and Technology in Computer Science Education, pp. 87–92 (2015) 15. Haatainen, S., Lakanen, A.J., Isom¨ott¨onen, V., Lappalainen, V.: A practice for providing additional support in CS1. In: 2013 Learning and Teaching in Computing and Engineering, pp. 178–183. IEEE (2013) 16. Haimovitz, K., Dweck, C.S.: The origins of children’s growth and fixed mindsets: new research and a new proposal. Child Dev. 88(6), 1849–1859 (2017) 17. Hamdam, N., M.P.M.K., Arfstrom, K.: The flipped learning model: a white paper based on the literature review titled a review of flipped learning (2013). https://www.researchgate. net/profile/Noora-Hamdan/publication/338804273 Review of Flipped Learning/links/ 5e2b51bf4585150ee780918c/Review-of-Flipped-Learning.pdf 18. Hattie, J., Timperley, H.: The power of feedback. Rev. Educ. Res. 77(1), 81–112 (2007) 19. Hyypi¨a, M., Sointu, E., Hirsto, L., Valtonen, T.: Key components of learning environments in creating a positive flipped classroom course experience. Int. J. Learn. Teach. Educ. Res. (2019) 20. Isaksson, N.: Oppimisen edist¨aj¨at ja est¨aj¨at: Laadullinen tutkimus opiskelijoiden kokemuksista oppimisesta insin¨oo¨ rimatematiikan kursseilla Tampereen yliopistossa. Master’s thesis (2021) 21. Kantoj¨arvi, J.: Architecture of A+ LMS (2018). https://apluslms.github.io/architecture/ presentation/ 22. Karavirta, V., Ihantola, P., Koskinen, T.: Service-oriented approach to improve interoperability of e-learning systems. In: 2013 IEEE 13th International Conference on Advanced Learning Technologies, pp. 341–345. IEEE (2013) 23. Karavirta, V., Shaffer, C.A.: Creating engaging online learning material with the JSAV JavaScript algorithm visualization library. IEEE Trans. Learn. Technol. 9(2), 171–183 (2015) 24. Koskinen, P., L¨ams¨a, J., Maunuksela, J., H¨am¨al¨ainen, R., Viiri, J.: Primetime learning: collaborative and technology-enhanced studying with genuine teacher presence. Int. J. STEM Educ. 5(1), 20 (2018) 25. Kuokkanen, T., Kangaslampi, R., Hirvonen, J.: Opetusmenetelm¨an ja oppimisteht¨avien yhteys korkeakouluopiskelijoiden matematiikan oppimiseen (2022) 26. Nicol, D.J., Macfarlane-Dick, D.: Formative assessment and self-regulated learning: a model and seven principles of good feedback practice. Stud. High. Educ. 31(2), 199–218 (2006) 27. Niemel¨a, P., Hukkanen, J., Nurminen, M., Huhtam¨aki, J.: Growing grade-by-grade. In: International Conference on Computer Supported Education, CSEDU, pp. 92–102. Science and Technology Publications (SciTePress) (2023) 28. Niemel¨a, P., Nurminen, M.: Rate your mate for food for thought: elsewhere use a grader. In: Proceedings of the 12th International Conference on Computer Supported Education - Volume 2: CSEDU, pp. 422–429. INSTICC, SciTePress (2020). https://doi.org/10.5220/ 0009564204220429

Encouraging Grading

45

29. Nurminen, M., Niemel¨a, P., J¨arvinen, H.M.: Having it all: auto-graders reduce workload yet increase the quantity and quality of feedback. In: SEFI Annual Conference: Blended Learning in Engineering Education: Challenging, Enlightening-and Lasting, pp. 385–393 (2021) 30. OECD: PISA 2015 key findings for Finland (2015). https://www.oecd.org/finland/pisa2015-finland.htm 31. Ørngreen, R.: Reflections on design-based research. In: Abdelnour Nocera, J., Barricelli, B.R., Lopes, A., Campos, P., Clemmensen, T. (eds.) HWID 2015. IAICT, vol. 468, pp. 20– 38. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-27048-7 2 32. O’Rourke, E., Haimovitz, K., Ballweber, C., Dweck, C., Popovi´c, Z.: Brain points: a growth mindset incentive structure boosts persistence in an educational game. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3339–3348 (2014) 33. O’Rourke, E., Peach, E., Dweck, C.S., Popovic, Z.: Brain points: a deeper look at a growth mindset incentive structure for an educational game. In: Haywood, J., Aleven, V., Kay, J., Roll, I. (eds.) Proceedings of the Third ACM Conference on Learning @ Scale, L@S 2016, Edinburgh, Scotland, UK, April 25–26, 2016, pp. 41–50. ACM (2016). https://doi.org/10. 1145/2876034.2876040, https://doi.org/10.1145/2876034.2876040 34. Paananen, V.: Miksi pojat j¨aiv¨at j¨alkeen? (2023). https://www.hs.fi/politiikka/art2000009743899.html 35. R¨am¨o, J., et al.: Engineering students’ approaches to learning in two student-centred instructional models before and during the COVID-19 pandemic. Eur. J. Eng. Educ. 1–27 (2023) 36. Reimann, P.: Design-based research. In: Markauskaite, L., Freebody, P., Irwin, J. (eds.) Methodological Choice and Design. Methodos Series, vol. 9, pp. 37–50. Springer, Dordrecht (2011). https://doi.org/10.1007/978-90-481-8933-5 3 37. Saarelainen, M., Heikkinen, L.: Oppijakeskeisist¨a opiskelutaidoista ev¨aa¨ t fyysikon uralle, vol. 20, No. 2 (2013). https://yliopistopedagogiikka.files.wordpress.com/2013/09/ yliopistopedagogiikka-2-13-saarelainen-heikkinen.pdf 38. Saarinen, A.: The relationships of temperament, character, and depressive symptoms with paranoid ideation. Ph.D. thesis, University of Helsinki (2018) 39. Sadler, D.R.: Formative assessment and the design of instructional systems. Instr. Sci. 18, 119–144 (1989) 40. Saunders, F., Gellen, S., Stannard, J., McAllister-Gibson, C., Simmons, L., Gibson, A.: Educating the Netflix Generation: Evaluating the Impact of Teaching Videos Across a Science and Engineering Faculty (2020) 41. Schleicher, A.: Pisa 2018: Insights and Interpretations. OECD Publishing, Berlin (2019) 42. Sclater, N., Peasgood, A., Mullan, J.: Learning analytics in higher education. JISC, London (2016). Accessed 8 Feb 2017 43. Sennett, R.: The Corrosion of Character: The Personal Consequences of Work in the New Capitalism. WW Norton & Company (2011) 44. Shaffer, C.A., Karavirta, V., Korhonen, A., Naps, T.L.: OpenDSA: beginning a community active-ebook project. In: Proceedings of the 11th Koli Calling International Conference on Computing Education Research, pp. 112–117 (2011) 45. Shaffer, S.C., Rosson, M.B.: Increasing student success by modifying course delivery based on student submission data. Inroads 4(4), 81–86 (2013) 46. Sirki¨a, T.: JSVEE & KELMU: creating and tailoring program animations for computing education. J. Softw. Evol. Process 30(2), e1924 (2018) 47. Slemmons, K., et al.: The impact of video length on learning in a middle-level flipped science setting: implications for diversity inclusion. J. Sci. Educ. Technol. 27(5), 469–479 (2018) 48. Tilanter¨a, A., et al.: Towards automatic advice in visual algorithm simulation (2020) 49. Toivola, M.: K¨aa¨ nteinen arviointi. Edita, Helsinki (2019)

46

P. Niemel¨a et al.

50. Ukkola, A., Mets¨amuuronen, J.: Matematiikan ja a¨ idinkielen taidot alkuopetuksen aikana (2023) 51. Vanderhyde, J.: Scaffolding assignments: how much is just enough? J. Comput. Sci. Coll. 34(3), 55–63 (2019) 52. Wang, F., Hannafin, M.J.: Design-based research and technology-enhanced learning environments. Educ. Tech. Res. Dev. 53(4), 5–23 (2005) 53. Whitehill, J., Williams, J., Lopez, G., Coleman, C., Reich, J.: Beyond prediction: First steps toward automatic intervention in MOOC student stopout. Available at SSRN 2611750 (2015) 54. Wiggins, G.: Seven keys to effective feedback. Feedback 70(1), 10–16 (2012) 55. Zander, C., et al.: Threshold concepts in computer science: a multi-national empirical investigation. In: Threshold Concepts Within the Disciplines, pp. 105–118. Brill (2008)

An Approach for Mapping Declarative Knowledge Training Task Types to Gameplay Categories B´er´enice Lemoine(B) , Pierre Laforcade , and S´ebastien George LIUM Computer Science Laboratory, Le Mans Universit´e, Laval, France {berenice.lemoine,pierre.laforcade, sebastien.george}@univ-lemans.fr Abstract. Training on declarative knowledge (DK) requires repetition, which can quickly become boring for learners. Consequently, games targeting such training must offer a wide variety of activities in order to keep learners-players engaged. Designing such situations remains a challenge because of the inherent entanglement of didactic elements and game elements. This chapter is an extended version of [14], which tackles the need to map training tasks with different gameplays for the design of relevant gameplay-oriented training activities. The proposed approach was identified during the design of a Roguelite-oriented training game for multiplication tables and has intentionally been specified towards a genericness purpose by using domain-independent task types and abstract gameplays. This chapter details the specification of task types (i.e., abstracted from two didactic domains) and abstract gameplays, the method used to identify the approach, and the resulting mappings when applied to our specific context. Keywords: Serious game · DK training · Didactic-game mapping

1 Introduction The design and use of serious games has become a common practice this last decade [4]. However, due to their lack of gameplay, most learning games fail to be seen as real games [16]. Gameplay can be defined as the fun things that can be controlled, decided, and done by players [16]. Although combining real games’ fun and educational content is not easy [16], it is a key component of a good learning game design. Correctly associating game and learning elements is mainly a difficult task because multiple contextdependant (i.e., didactic domain, targeted knowledge or game genre) variables must be considered. Declarative knowledge (DK, i.e., knowledge about facts, laws, statements) are known to require repetition for encouraging their memorization, generalization, and retention [10, 18]. Test-Based Learning (TBL) is defined in cognitive psychology as the idea that the process of retrieving (i.e., remembering) concepts or facts increases their long-term retention. Retrieval practice (i.e., repeated retrieval) is a form of TBL that has been shown to improve long-term retention [18]. In addition, research suggests that the benefits of retrieval practice are not linked to a specific implementation (i.e., c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024  B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 47–68, 2024. https://doi.org/10.1007/978-3-031-53656-4_3

48

B. Lemoine et al.

various tests formats enhance learning) [2]. In our work, training is defined as a form of retrieval practice that consists of providing learners with various forms of questions about facts repeatedly. As learning games aimed at DK training offer repetitive training sessions, a deeper commitment from learners is required. Accordingly, training games should provide a wide variety of activities in terms of situations, game mechanisms or gameplays. Handcrafted design of varied activities limits the scope for variety. A possibility to design a wide variety of activities is the use of automatic content generation. Content generation is a common technique in game development, especially in Roguelike and Roguelite games, to create unique game levels (e.g., different shapes, elements, elements’ position). Accordingly, designers of training games are faced with different stakes, such as the automatic generation of activities and the design of the mechanism enabling the variety of activities. Our previous work, presents Roguelite, a well known and liked game genre, as an adequate genre for DK training [13]. Had`es, Rogue Legacy, Binding of Isaac are famous Roguelites. These games are dungeon-crawler games: players must explore dungeons, i.e., interconnected rooms where actions take place. Roguelites are based on a mechanism called permanent-death which involves repetition, i.e., players must start a new playthrough each time their avatar dies. Other characteristics of Roguelites games are the procedural generation of dungeons with randomized content (→ variety) and the limited retention of unlockable items (e.g., avatars, powerups, equipments). In short, a Roguelite for DK training will successively propose generated dungeon levels to the learner-player, wherein they will be challenged to answer taskoriented questions. Our research interests concern the generation of varied Roguelite activities for DK training. Our aim is to offer a variety of gameplays. The definition of gameplay, fun things that can be controlled, decided, and done by players [16] can be refined as: descriptions of contextualized actions that players can perform to interact with the environment, through their avatar, in order to answer questions. Since training games involve several domain-specific parametrized training tasks, numerous gameplays must be identified for each of them. Identifying how different training tasks can be implemented using these different game concepts requires conceptualizing and addressing a transdisciplinary Technology Enhanced Learning (TEL) problem: how can didactic knowledge be mapped to different gameplays? This challenge emerged during the design of a Roguelite-oriented learning game to train multiplication tables. First, our approach is to address the challenge at a higher level of abstraction (task types instead of domain-specific tasks, and game categories instead of practical gameplays). This enables a more generic, domain-independent approach. In our previous work [14], task types were abstracted from a single didactic domain. In this chapter, task types are abstracted from two didactic domains. Second, the central thrust of our approach is to use a dedicated pivot to help identify the source (task types) and target (game categories) parameters whose values will guide the elicitation of practical mappings. This chapter explains how we identified this approach (method), what it consists of (proposition) and what mappings are obtained in our context (application). We assume that this approach is sufficiently generic and reusable to help multidisciplinary design teams identify and map gameplays for their specific tasks (contribution).

An Approach for Mapping Task Types to Gameplay Categories

49

2 Elements for Training Game Activities Our overall objective is to identify approaches, models and processes to help multidisciplinary teams design activity generators for DK training. But, as previously mentioned, building activities combining fun and educational content is not easy [16]. Prensky [16] proposed a three-step process to create digital game-based learning: “(1) Find or create a game with great gameplay that will engage our audience, (2) Find the learning activities and techniques that will teach what is required (doing each with the other in mind), and then (3) successfully blend the two”. Accordingly, our work began with the identification of training tasks and possible gameplays with the experts. 2.1 Task Types for DK Training An exploratory research (partially presented in [12]) conducted with 2nd to 6th grade teachers and mathematics experts led to the identification of training tasks for multiplication tables training, such as: complete a fact where the result is missing (e.g., 3 × 5 =?), complete a fact where the operand is missing (e.g., 3×? = 15), decide if a fact is correct (e.g., 3× 5 = 15), identify the results of a table (e.g., [5, 6, 9, 12, 8] which are results of table 3?). These tasks also embed specific parameters to determine which facts to take into account, how to construct them and how to answer them. In addition, exchanges with history-geography teachers also led to the specification of training tasks for history-geography facts, such as: place historical dates in chronological order (e.g., World War II, Storming of the Bastille, Treaty of Rome), name and locate countries of the European Union, decide if a fact is correct (e.g., Did World War II happen between 1939–1945?). An observation is that certain tasks appear to be similar in both domains. Therefore, in a perspective of genericness with other domain-related declarative knowledge, we expressed our tasks at a higher level of abstraction. Without being exhaustive, the following four types of task have been defined: 1. Completion: complete a fact that having missing elements (e.g., complete 3×? = 15, reconstitute ?×? =? using elements in [3, 6, 5, 10, 15], complete World War II happened between ? – ?); 2. Order: order facts based on a given heuristic (e.g., chronologically order: World War II, Storming of the Bastille, Treaty of Rome); 3. Identification: attest of the validity or invalidity of one or several facts (e.g., true or false: 3 × 5 = 15?, Did World War I happen between 1915–1919?); 4. Membership Identification: identify elements that share or not a given property (e.g., [3, 5, 9, 12, 14, 21] which are results of the table 3? [France, Spain, England, Switzerland, Italy] which are part of the European Union?); 2.2 Gameplay Categories for Dungeon-Like Games Prensky [16] stated that “Although learning games can fail as real games in many ways, the failure happens mostly commonly in their lack of gameplay”. Consequently, our aim is to provide a wide variety of gameplays for each task. First, some ideas were discussed

50

B. Lemoine et al.

with the teachers. As a result, one constraint emerged: gameplays must be simple, i.e., interactions to answer must be quick. Then, informal interviews were conducted with game designers. The purpose was to gather ideas to design gameplay mock-ups. Moreover, a game prototype with a few gameplays was produced to try out some ideas and gather feedback.

Fig. 1. Example of mock-ups by gameplay categories [14].

After designing the mock-ups, an observation was made: certain gameplays seemed to belong to the same category (e.g., breaking a pot or opening a chest bearing an answer are similar ways of selecting an object). That observation is consistent with the game classification proposed by Djaouti et al. [5], which consists of describing games in terms of gameplay bricks (i.e., categories of actions that can be performed within the games). Consequently, further reflection resulted in the definition of 5 gameplay categories, cf. Fig. 1, in our context (as with the task types, these categories do not claim to be exhaustive): 1. S ELECT: select (e.g., touch, kill, break, open) objects wearing the correct answers, through avatar actions;

An Approach for Mapping Task Types to Gameplay Categories

51

2. M OVE: correctly place objects at specific locations through avatar actions; 3. O RIENT: orient objects (e.g., rotate), through avatar actions, towards the correct answer; 4. P OSITION: move the avatar to the necessary positions for choosing or typing the correct answers; 5. D IRECT R ESPONSE: no action is required through the avatar, learners can directly type down their answer by using an input device (e.g., enter the correct answer through a keyboard).

3 Activity Generation: A Mapping Need 3.1 Research Question As previously mentioned, our overall objective is to design activity generators. Activity generators are software components that automatically create content from structured data. Following Prensky’s process [16], now that the elements have been determined, they must be correctly associated. Therefore, our main question now is: how to determine and specify the relationships between task types and game categories necessary to the design of learning game activities? Indeed, knowledge about these relationships is essential at the design phase to guide the identification of practical gameplays for each specific task, and at the runtime to control the generation process. Our assumption is that answering this question at a higher level of abstraction (task types and abstract gameplays) will enable the reuse of the relationships in various declarative knowledge contexts.

Fig. 2. Illustration of our research question.

This research question, illustrated in Fig. 2, involves precisely answering the following questions: Which abstract gameplays are suitable for which task types? Is the mapping systematic or conditional? If conditional, how to find these conditions? According

52

B. Lemoine et al.

to Tchounikine et al. [21], this is typically a problem of research in TEL engineering falling into the “elaborating powerful abstractions” case where the problem must be addressed from a transdisciplinary perspective. 3.2

Related Work

Previous work have addressed the issue of identifying relationships between educational and game dimensions. First, several works focused on identifying relations between educational and game elements. Prensky [16] is a pioneer who proposed relations between game genres (e.g., action, role-play, adventure), knowledge to be learned (e.g., facts, skills, judgement, behaviour) and learning activities (e.g., questions, experiments, observation). Rapeepisarn et al. [17], proposed an extension of [16] by adding a relation to Chong et al.’s learning styles [3] (i.e., activists, reflectors, theorists, and pragmatists). Likewise, Sherry [19] identified relations between games genres and the six levels of Bloom’s taxonomy [1]. Gosper et McNeill [7] proposed a framework to support the integration of technology in education. Their framework defines relations between learning outcomes (e.g., acquisition of basic facts, automation of skills and concepts), learning processes (e.g., memorization, analogical reasoning), assessment (e.g., self-assessment, peer assessment) and game genres. These works are very interesting from a general design viewpoint of learning games. However, the identified relations are between high-level concepts, and cannot be used at a specification stage to guide the generation of activities. Second, some works attempt to provide relations at a specification level. Dondi et Moretti [6] linked learning objectives (e.g., memorization/repetition/ retention), knowledge types (e.g., factual knowledge), and game genres to high-level features that games should possess (e.g., presence of content engine, assessment engine). However, these high-level features do not describe how the relations are to be implemented in practice. In addition, other works propose a framework to specify relations (i.e., either for analysing existing games or conceiving one). The LM-GM framework [15] supports the transition from learning objectives/practices to game elements through a concept called Serious Game Mechanic (SGM). It defines learning mechanics and game mechanics and uses SGM to associate both concepts. However, the presented mechanics are highlevel ones (e.g., guidance, collaboration, explore) and the relations are not meant to be implemented as such. Furthermore, Hall et al. [8] proposed a framework to guide the designer in specifying the transition from learning content to core-gameplay. It is composed of 5 categories (i.e., goal, choice, action, rules, feedback) in which a series of questions need to be answered from a real-world and a game-world perspective. However, the framework is more oriented towards the general design of the game rather than its implementation. To conclude, existing approaches are more oriented towards defining relations for analysis purposes or to assist in the high-level design of games rather than specifying relations for low-level design purposes. Moreover, these works address specific learning targets or the contexts of specific game genres, as we do.

An Approach for Mapping Task Types to Gameplay Categories

53

3.3 Research Positioning and Objectives Our work seeks to propose an approach to specify relations between declarative knowledge training tasks (i.e., independent of a specific didactic domain) and gameplays from the Roguelite genre. These relations have to meet one condition: their specification must enable their implementation. Training and the assessment of declarative knowledge are well known to be carried out through questionnaires and quizzes. Furthermore, digital quizzes, compared to paper ones, allow interactions with the user that are closer to those found in basic training games (e.g., a multiplication table training game where the correct answers allow the avatar to run faster or jump onto higher platforms). Accordingly, it appears interesting to use exercise types of quiz formats as a pivot in particular because using existing content may reduce subjectivity. Hence, our work intends to propose a systematic mapping approach based on the use of quizzes exercise types as a pivot, cf. Fig. 3. The next sections present the development of our approach, followed by the proposed approach, a proposition for modelling relations, and an application example.

Fig. 3. Idea to map task types onto gameplay categories.

4 Mapping Approach Development The elaboration of our approach required several stages. Initially, an analysis of quizzes design formats was carried out in order to define the types of existing exercises (i.e., our pivot). This work raised the following questions: (1) How can we draw a parallel between the types of tasks and the exercises identified? (2) How can we draw a parallel between the gameplay categories and the exercises identified? As previously mentioned, the interactions offered by each quiz exercise are closer to game interactions. In addition, each concept (i.e., task types, gameplay categories, and exercises) is characterised by its possible response modalities (e.g., enter an answer, choose between multiple propositions). Therefore, our second step consisted of using the exercise types to identify possible criteria and parameters to specify the task types and game categories

54

B. Lemoine et al.

and ease the identification of the mappings. Finally, these valued parameters (from both task types and gameplay categories perspectives) were used to compare and identify matches. 4.1

Identification of the Pivot

Foremost, exercises types of six tools, mostly extracted from Learning Management Systems (LMS), that allow the creation of numerical questionnaires/quizzes were analysed:  the eponymous and proprietary format from the itsLearning (#1) LMS;  GIFT (#2) a mark-up language for describing tests that is associated with the Moodle LMS;  Performance Matters Assessment and Analytics (#3) format associated with the PowerSchool LMS;  NetQuizzPro (#4) a software allowing the creation of questionnaires;  QTI (Question & Test Interoperability specification) (#5) from the IMS global learning consortium that defines a standard format to exchange and store assessment content;  Tactileo – Maskott (#6) format associated with the French pedagogical platform of the same name. Table 1. Exercises by quiz format. (✓ present; ✗ absent; — present but incomplete) [14]. itsLearning GIFT PMAA NetQuizzPro QTI Tactileo – Maskott Alternative

✓

—

—

✗

✗

✗

Multiple Choice

✓

✓

✓

✓

✓

✓

Multiple Resp.

✓

✓

✓

✓

✓

✓

Short Answer

✗

✓

✓

✗

✓

✓

Fill-in

✓

—

✓

✓

✗

✓

Fill-in Choice

✓

—

✗

✓

✓

✓

Reconstruction

✗

✗

✗

✓

✗

✗

Association

—

—

—

—

—

—

Order

✓

✗

✓

✓

✓

✓

G. Choice

—

✗

—

✗

✓

—

G. Identification

✗

✗

✗

✓

✗

✗

G. Association

✗

✗

✗

—

✓

✓

The analysis consisted of determining the different possibilities offered by these formats (cf. Table 1). More precisely, the various possible questions and their parameters were examined and compared. This led us to the definition of 12 different types of exercises useful for DK (i.e., only exercises for which the verification of results can

An Approach for Mapping Task Types to Gameplay Categories

55

be automated). Exercises (of different formats) which shared the same type of statement, the same number of desired answers and for which the interaction of the answers was similar were merged to form a single type of exercise. In addition, some formats combine several exercises into one (e.g., multiple choice and answers were merged into the itsLearning format). In those cases, the exercises were considered to be independent. Moreover, the possibility of having intruders (i.e., elements which should not be associated) has been requested by domain experts, but none of the “Associate” type exercises analysed from the formats offers this possibility. Therefore, in our definition of exercises, it has been considered as a possibility. The types of exercise defined are as follows: ✏ Alternative: choosing one answer between 2 options; ✏ Multiple choice: choosing one answer between X (i.e., X  2) options; ✏ Multiple responses: choosing Y (i.e., zero or more) answers between X (i.e., X  2) options; ✏ Short answer: enter the correct answer. Multiple form of answers can be accepted, e.g., for example, How much is 3 times 5? as two possible answers, which are 15 and fifteen; ✏ Fill-in-the-blanks: enter for each gap of a text the wanted “short” answer; ✏ Fill-in-the-blanks choices: choose for each gap of a text the correct answer from a list. Each gap can have an associated list of options, or one list can be associated to all gaps; ✏ Reconstruction: reassemble each significant element of an information; ✏ Associate–Group: associate elements from a list or multiple lists together. The association can be done by pairs, or not. The elements can be associated with zero to several other ones; ✏ Order: replace a set of information in the correct order (i.e., following a heuristic); ✏ Graphic choice: point or locate X (i.e., X  1) elements on a picture. ✏ Graphic identification: write the correct label for each area-to-complete of a picture; ✏ Graphic association: associate the correct labels to X areas of a picture. As a reminder, these types of exercises aim to deal with declarative knowledge in general. As a result, some exercises offer a more visual approach that could be useful in the context of geographical facts for example. These exercises are characterised by several parameters, cf. Table 2, such as: their interactions, their response modality (i.e., input or choice), their statement type (i.e., format of the question asked), the number of answers desired, and the number of propositions presented (i.e., if the response modality of a concrete task of this type is “Choice”). Through our analysis, 6 types of interactions were identified: – Select Y From X (i.e., the learner must select Y answers from a set of X values); – Y (Select 1 from X1 to XY ) (i.e., the learner must make a selection of one answer from each set of proposals); – a variant is Y (Select 1 from X) (i.e., the learner must select Y answers, one by one, from a set of proposals); – Write X (i.e., the learner has to enter X answers); – Order X (i.e., the learner must order X elements correctly);

56

B. Lemoine et al. Table 2. Characterisation of the exercises [14]. Number Statement Response

Number

of Facts

Types

Modality

Answers of Choices

Alternative

1

Classic

Choice

1

2

Select 1 from 2

Multiple Choice

1

Classic

Choice

1

2 to ∞

Select 1 from X

Multiple Resp.

1

Classic

Choice

0 to ∞

2 to ∞

Select Y from X

Short Answer

1

Classic

Input

1

0

Write 1

Fill-in

1

Fill-in

Input

1 to ∞

0

Write Y

Fill-in Choice

1

Fill-in

Choice

1 to ∞

2 to ∞

Reconstruction

1

Fill-in

Choice

2 to ∞

2 to ∞

Match Y with X 1-to-1

Interactions

Y (Select 1 from X) Y (Select 1 from X1 to XY )

Choice

2 to ∞

4 to ∞

Match Y with X 1-to-1

Choice

1 to ∞

2 to ∞

Order Y

Graphic

Choice

1 to ∞

2 to ∞

Point Y or Locate Y

Graphic

Input

1 to ∞

0

Write Y

Graphic

Choice

1 to ∞

1 to ∞

Match Y with X 1-to-1

2 to ∞

Classic

Order

2 to ∞

Classic

G. Choice

1 to ∞

G. Identification

1 to ∞

G. Association

1 to ∞

Association

Number

Fill-in

Match Y with X

– Point X or Locate X (i.e., the learner must point X elements on a picture or locate them); – Match Y with X 1-to-1 or Match Y with X (i.e., the learner must associate elements from Y with those from X by pairs or not). In addition, 3 types of statement were found: 1) classic statement (i.e., a text question that can be accompanied by an image), 2) graphic statement (i.e., a classic statement accompanied by a graphic component with which interactions are required to answer), and 3) fill-in statement (i.e., a classic statement with incorporated fill-in areas). It is important to note that none of the formats allows for all possible forms of exercise. 4.2

Mapping Task Types onto Gameplay Categories

Having specified the pivot, the remaining questions are: How to map (1) task types onto exercises and (2) game categories onto exercises? Our main idea involves using parameters that characterise each concept (i.e., task types, game categories and exercises) to map them. Task Types to Exercises. Like exercises, task types are characterised by several parameters: the number of facts targeted by the task, the types of statements allowed for such a task, the response modalities, the number of desired responses, and the number of propositions presented (i.e., when the response modality for a concrete task of this type is “Choice”). For example, the Identification type is defined as follows: 1 to ∞ can be targeted, only classic statements are allowed, both response modalities can be used (i.e., input and choice), the number of desired answer is equal to the number of facts targeted, and at least 2 propositions must be presented when the modality is Choice.

An Approach for Mapping Task Types to Gameplay Categories

57

It is important to note that the assignment of parameter values is not an easy task. Let’s take a task T1 which consists of completing a fact having two missing elements (e.g., ?×? = 12, i.e., number of facts = 1 and number of expected answer = 2). At first sight, it could seem possible to carry out T1 through the Input response modality. However, presenting a statement in the context of declarative knowledge, such as “?×? = 12”, does not give enough information about the fact to work with (i.e., is it 3 × 4 or 6 × 2). As another example, for a T1-like task, depending on how the choices are presented, it is possible to choose one or two answers. If the set of propositions represents numbers, such as [3, 5, 7, 4], two answers must be chosen. However, if each proposition is presented as a multiplication (without the result), such as [3 × 4; 4 × 5; 6×3], then only one answer is required. Table 3 presents our task types characterisation. Thus, except for the interactions parameter, task types and exercises are characterized by the same parameters. Consequently, the mapping between task types and exercises consists of comparing the values of their common parameters. For example, Identification is mapped to Short answer because of the specification of Short answer, i.e., {number of facts = 1; type of statement = classic; modality = input; number of desired answers = 1}, is a possible configuration of a concrete task of the type Identification (i.e., the parameter values are included into those of the type Identification). This gives questions such as “Did World War II happened between 1914–1918?” and “Is 2 × 5 equal to 12?”. Completion is mapped to Fill-in-the-blanks choices exercise specified as {number of facts = 1; type of statement = fill-in; modality = choice; number of desired answers = [1 − ∞]; number of choices = [2 − ∞]}. This gives questions such as “ times 5 equals 15”. Table 3. Characterisation of the task types. Number of Statement Facts

Types

Response

Number of

Number of

Modalities

Answers

Choices

Input 1 Completion 2 to ∞

0 2 to ∞

Choice

Graphic

Choice

∞

Fill-in

Input

Nb Facts

0

Choice

 Nb Facts

2 to ∞

Choice

Nb Facts

Nb Facts

Order

2 to ∞

Classic

Identification

1 to ∞

Classic

Membership

1 to ∞

Identification

1

Classic

Fill-in

Input Choice

Classic

Input

Graphic

Choice

Nb Facts 2 to ∞

2 to ∞

0 2 to ∞ 0 2 to ∞

58

B. Lemoine et al.

Thanks to these mappings, each task type can also be described by its possible interactions (i.e., the interactions of the exercises corresponding to the task type considering the number of requested answers). As an example, the Short answer exercise type asks the learner to write down one short answer to a textual question. Thus, its interaction parameter has for value Write 1. A task of the Identification type could ask questions such as “Did World War II happen between 1914–1918?”, leaving the learner to write down True or T. Therefore, Identification has Write 1 for possible interaction. As a result of all mappings, Membership Identification can be achieved through the following interactions: Y Write Y (i.e., Y ∈ [2, ∞]), Point Y, Select Y from X, Match Y with X 1-to-1. Gameplay Categories to Exercises. Each game category represents gameplays that are similar in terms of the actions to be performed, such as opening the right chest, choosing the right pot, passing through the right bridge which belong to the S ELECT category. Therefore, the common parameters of these gameplays (e.g., number of facts interrogated, number of possible answers) represent those of the category itself. After analysis, these categories were characterised using the following parameters: the interactions, the response modality (i.e., input or choice), the statement type (i.e., format of the question asked), the number of facts targeted, the number of answers desired, and the number of propositions presented (i.e., if the response modality of a concrete task of this type is “Choice”). These parameters are similar to those used for the exercises. They represent a minimal and relevant set of parameters that allow us to discriminate the different categories and gameplays. For example, the S ELECT category is characterised as follows: 1 to many facts can be targeted, both classic and fill-in statement types are allowed, choice is the only possible response modality, 1 to many answers can be desired, and two interactions (i.e., Select Y from X, and Y (Select 1 from X1 to XY )) are possible. However, during the characterisation phase, it became apparent that the possible interactions and the statement type changed depending on whether one or more responses were desired. Therefore, to simplify the mappings, each category allowing one or more possible responses was divided into two sub-categories: single (i.e., only one possible response) and multiple (i.e., from two to several possible responses). As a result, our 5 categories became 9. Table 4 presents the game categories through their characterisation. Afterwards, the mappings consisted of directly comparing the values of the parameters.

An Approach for Mapping Task Types to Gameplay Categories

59

Table 4. Characterisation of the gameplay categories ((S) = Single, (M) = Multiple) [14]. Number Statement Response of Facts

Types

1

Graphic

Number

Modality Answers

Number of Choices

Classic S ELECT ( S )

Choice

1

2 to ∞

Choice

2 to ∞

2 to ∞

Fill-in Classic S ELECT ( M )

1 to ∞

Graphic

Graphic

Point Y or Locate Y Y (Select 1 from X1 to XY )

Fill-in 1

Select 1 from X Point 1 or Locate 1 Select Y from X

Classic M OVE ( S )

Interactions

Select 1 from X Choice

1

2 to ∞

Fill-in

Point 1 or Locate 1 Match 1 with 1 Match Y with X

Classic M OVE ( M )

1 to ∞

Graphic

Point Y or Locate Y Choice

2 to ∞

2 to ∞

Fill-in

Select Y from X Y (Select 1 from X1 to XY ) Order X

O RIENT ( S )

1

O RIENT ( M )

1 to ∞

Classic Fill-in Classic Fill-in Fill-in

P OSITION ( S )

1

Classic Graphic

Choice

1

2 to ∞

Select 1 from X

Choice

2 to ∞

2 to ∞

Y (Select 1 from X1 to XY )

Input Choice

1

Y (Select 1 from X)

0

Write 1

2 to ∞

Select 1 from X Point 1 or Locate 1

P OSITION ( M )

1

Graphic

Input

2 to ∞

0

Write Y

D IRECT R ESP.

1

Classic

Input

1

0

Write 1

Task Types to Gameplay Categories. From there, all the necessary information was available to answer our main question: Which type of task is suitable for which category of game? What are the conditions? Consequently, the final step consisted of comparing the task types and categories on the basis of their parameter values (i.e., comparing Table 3 with Table 4). During this stage we observed that 4 parameters represented mapping conditions based on their values: statement type, number of facts targeted, number of expected answers, and the response modality. Therefore, the obtained relationships are 6-uplets composed as follows: (, [, , . . . ], , , , [, , . . . ]). In conclusion, this section presented the process followed to map the task types for declarative knowledge training to the gameplay categories for the Roguelite video game genre. Next section will present the results obtained.

5 A Systematic Mapping Approach This research resulted in two contributions: (1) an approach for mapping designers’ own task types to their own game categories, and (2) mappings between our task types and our gameplay categories.

60

5.1

B. Lemoine et al.

Proposed Mapping Approach

The proposed mapping approach is a two to five-steps approach, illustrated in Fig. 4. It is composed of two initial steps: 1. abstraction of the concrete tasks using the types of tasks presented (e.g., a task “associate the right date with the historical event” becomes complete a fact with a missing element) or by creating new task types; 2. association of the gameplay to one of the categories presented or to a new gameplay category.

Fig. 4. Proposed Mapping Approach.

At this point, there are four possible states: new task types and categories have been created, only new task types have been created, only new game categories have been created, or nothing has been created. According to the state, the instructions below must be followed: 1. If new task types and new gameplay categories were created: (a) Characterise the task types using the six parameters defined above (i.e., number of facts, types of statements, response modalities, number of desired responses, number of propositions, and interactions). In a sub-step, map task types and quiz exercises (cf. Table 2) to define the values of the interactions parameter. (b) Characterise the gameplay categories using the same parameters.

An Approach for Mapping Task Types to Gameplay Categories

61

(c) Finally, compare both tables (i.e., characterisation) through their values. As a reminder, the values of the Statement Type, Number of Facts, Number of Answers, and Response Modality parameters are possible conditions of the relations. 2. If only new task types were created, then realise step 1a and step 1c. 3. If only new gameplay categories were created, then realise steps 1b and 1c. 4. If no new elements have been created, the work is already done, cf. Fig. 5. Let’s take as example a task type T1 characterised as {number of facts = 1; type of statement = classic; modality = input or choice; number of desired answers = 1}, and a gameplay category C1 = {number of facts = [1 − ∞]; type of statement = classic or fill-in; modality = choice; number of desired answers = [1 − ∞]}. In this case, only one relationship would result: (T1, Classic, 1, 1, Choice, C1). 5.2 Relations Between Task Types and Gameplay Categories The process presented in Sect. 4 resulted in the definition of several conditional relationships between the task types and gameplay categories. Figure 5 presents the resulting relations. As examples, the task type Order has a unique relationship: (Order, [Classic, Fill-in], [2 – ∞], Nb Facts, Choice, [M OVE ( M )]). The task type Identification has four relationships, including: (Identification, Classic, 1, Nb Facts, Input, [P OSITION ( S ), D IRECT R ESPONSE]) and (Identification, Classic, [2 – ∞], Nb Facts, Choice, [S ELECT ( M ), M OVE ( M ), O RIENT ( M )]) 5.3 Evaluation of the Relations For the purpose of gathering feedback on the gameplay mock-ups (i.e., identifying relevant gameplays and game elements), the members (around 10 people) of our user group (from the AdapTABLES project) were asked to complete a survey presenting possible gameplays for each type of task. The experiment was also an opportunity to validate certain mappings (i.e., relationships for which the categories have existing gameplay mock-ups).

62

B. Lemoine et al.

Fig. 5. Conditional relations between task types and gameplay categories.

During the exploratory research [12], five domain-specific configurable tasks were identified for multiplication tables training: 1. Completion 1 (of the type Completion): complete an incomplete multiplication fact that has one missing element (e.g., 3×? = 15, 15 =? × 5, 3 × 5 =?); It is characterised by: {number of facts = 1; type of statement = classic or fill-in; modality = input or choice; number of desired answers = 1} 2. Completion 2 (of the type Completion): complete an incomplete multiplication fact that has two missing elements (e.g., ?×? = 15 with a set of given choices [3, 6, 5, 10], ? × 5 =? or 3×? =? also with sets of given choices); It is characterised by: {number of facts = 1; type of statement = classic or fill-in; modality = choice; number of desired answers = 2} 3. Reconstruction (of the type Completion): reconstitute a multiplication fact (e.g., ?×? =? with a set of given choices [3, 6, 5, 10, 15]); It is characterised by: {number of facts = 1; type of statement = classic or fill-in; modality = choice; number of desired answers = 3} 4. Identification (of the type Identification): identify the accuracy or inaccuracy of one or several multiplication facts (e.g., 3 × 5 = 15, true or false?); It is characterised by: {number of facts = [1 − ∞]; type of statement = classic; modality = input or choice; number of desired answers = [1 − ∞]} 5. (Non-)Membership Identification (of the type Membership Identification): identify the elements that are results of a given multiplication table (e.g., [3, 5, 9, 12, 14, 21] which are results of the table 3?); It is characterised by: {number of facts = [1 − ∞]; type of statement = classic; modality = input or choice; number of desired answers = [2 − ∞]}. The experiment was realised in the context of multiplication tables training. Therefore, none of the relations where the statement type was Graphic were evaluated. Furthermore, the tasks defined for the multiplication tables do not require completion tar-

An Approach for Mapping Task Types to Gameplay Categories

63

geting several facts or ordering. However, for every other relationship, gameplay mockups have been defined for each category and each task type. The survey consisted of presenting the experts with an image of a gameplay for a given task, a description of the gameplay, and a question on the relevance of the gameplay (a comment box was at their disposal). Let’s take the example of a gameplay consisting of selecting the right pot among several pots having propositions (i.e., S ELECT with Choice) to answer a textual question of the type “3×? = 15” (i.e., Completion 1). If this gameplay is validated by the survey, then so is the relationship (Completion, Classic, 1, 1, Choice, S ELECT). According to the results, the mappings were relevant. Negative comments were about didactic issues or a lack of precision. As an example, M OVE gameplays that require objects to be placed on the correct answer were rejected because the selected answer was hidden by the object, thus impacting learners’ thinking. This is a cognitive issue, unrelated to the game mechanic, that can be corrected by displaying the value above the object, or by displaying the chosen value inside the statement at the correct position with another colour. Figure 6 illustrates both solutions: the statue pushed on the left tile hides the associated ‘5’ value, but 1) the value appears on top of the statue, 2) the value appears now in purple inside the room’s statement.

Fig. 6. Examples of possible solutions [14].

Furthermore, the gameplay mock-ups for the O RIENT category relied (at the time) on an object lantern where the avatar had to orient the light towards the answer. This gameplay received mixed reviews because of the lack of cognitive meaning of the object (i.e., light is emitted in every direction). In order to reach a set of satisfactory gameplays, we carried out a focus group in which we discussed the disagreements and proposed solutions to the problems observed (e.g., use of statues rather than lanterns).

6 A Formal Modelling of the Mappings Overall, the aim of our research concerns the design of generators of Roguelite-oriented game activities for declarative knowledge training. Our approach’s originality lies in the use of Model-Driven Engineering (MDE) [9] to depict and structure any data required by the generators.

64

B. Lemoine et al.

Fig. 7. Metamodel describing the conditional relation structure (unique=single).

Basically, the idea involves specifying the data (i.e., domain-dependent tasks, task types, game categories, gameplays, game elements, and relationships) in different interconnected models that are consistent with a dedicated metamodel (i.e., model whose instances are models), that we also specify. Consequently, each activity generation could be seen as a model transformation according to MDE. Currently, a generator has been developed for training multiplication tables. This generator uses models and metamodels that we specified using the Eclipse Modeling Framework (EMF) [20], and generates XMI files (i.e., EMF imposed format) accurately describing every element composing an activity (i.e., a dungeon, e.g., rooms, rooms order, game elements and their positions in each room, questioned facts). In order to have a universal, portable, and easily deployable structure, the XMI files are then translated into XML files using the Epsilon Framework [11]. Figure 7 presents the EMF metamodel used by the generator to depict the relations structure. As defined earlier, the relations are between a task type (i.e., attribute task) and a set of gameplay categories (i.e., attribute gameplays). These relations are bound by a condition featuring authorized statement types (i.e., attribute statementTypes), the number of facts targeted (i.e., attribute nbFacts), the number of expected answers (i.e., attribute nbExpectedAnswers) and the answer modality (i.e., attribute answerModality). EBoundary describes the possible values for the number of expected answers, so that the generation algorithm can interpret them. Domain-dependent tasks, as well as the specification of concrete gameplays and their implementation in game elements, are not shown in the figure. Another part of the metamodel (not displayed here) describes tasks by their attribute type (i.e., ETaskType) and gameplays by their attribute category (i.e., GPCategory).

An Approach for Mapping Task Types to Gameplay Categories

65

Figure 8 presents the instantiation, through EMF, of the metamodel with the relations defined in Fig. 5. At runtime, the generator loads and interprets this model to select the permitted gameplays based on the learner’s context (i.e., the tasks the learner has to perform). For a given domain-dependent task, the main steps performed by the mapping algorithm are as follows: 1. get the associated task type of the targeted task; 2. collect every relation from the mapping model related to this task type; 3. restrict the collected relations to those whose associated condition is satisfied (compare the statement, number of facts, number of expected answers, and response modality values of the condition and the original task); 4. collect the gameplay categories of the remaining relations. Therefore, by following our defined mappings (cf., Fig. 5), multiplication tables training tasks (cf., Sect. 5.3) will be associated by the algorithm with: – Completion 1 is compatible with P OSITION ( S ), S ELECT ( S ), M OVE ( S ), O RIENT ( S ), and D IRECT R ESPONSE; – Completion 2 and Reconstruction are compatible with S ELECT ( M ), M OVE ( M ), O RIENT; – Identification is compatible with P OSITION ( S ), S ELECT ( S , M ), M OVE ( S , M ), O RIENT ( S , M ), and D IRECT R ESPONSE; – (Non-)Membership Identification is compatible with S ELECT ( M ), M OVE ( M ), O RI ENT ( M );

Fig. 8. Tree-based EMF model view of our model describing the defined relations.

Then, the generation algorithm handles further steps such as: selecting the gameplays that implement the allowed categories, filtering them according to other infor-

66

B. Lemoine et al.

mation (e.g., gameplays unlocked and available to the learner), instantiating the game elements composing the chosen gameplay, and so on. Currently, a game prototype0 has been developed that uses this generator every time a new game level is requested. This prototype acts as an interpreter that reads a given XML file and transforms its contents into a playable level (i.e., a dungeon). Figure 9 presents screenshots of dungeon rooms and playable gameplays.

Fig. 9. Examples of playable gameplays.

7 Conclusion and Perspectives In conclusion, this chapter has outlined a systematic approach for mapping task types onto gameplay categories in the context of declarative knowledge training. The origi-

An Approach for Mapping Task Types to Gameplay Categories

67

nality of this work lies in two aspects: 1) the proposed approach is based on the use of a pivot (i.e., exercises extracted from numerical questionnaires tools); 2) it is oriented towards the automation of learning game activity design (i.e., generation) and therefore specifies fine-grained relationships. This chapter also presented a formal modelling of relationships to enable their interpretation by activity generators. The task types and gameplay categories to be mapped were defined with the help of experts (i.e., teachers, didactic experts, game designers) and do not claim to be exhaustive. Gameplay categories are defined to offer a wide variety of activities, while task types are designed to meet the needs expressed by our experts. As a result, these task types could be refined according to other didactic domains, and the game categories proposed could be extended. Furthermore, the parameters used in the approach are subjective in the sense that they represent those that are necessary in our opinion. As a result, these parameters could be argued according to different viewpoints. In addition, not every relationship was evaluated, but only those used for multiplication table training. Furthermore, a generator for multiplication was developed that uses the presented metamodel and model to generate activities. In addition, this generator is currently being used in a game prototype that interprets the generated XML files (i.e., detailed description activities) to provide learners-players with playable dungeons. In future work, we plan to implement a history-geography fact generator and evaluate relationships that have not yet been assessed.

References 1. Bloom, B.S.: Taxonomy of Educational Objectives: The Classification of Educational Goals. Cognitive Domain (1956) 2. Brame, C.J., Biel, R.: Test-enhanced learning: the potential for testing to promote greater learning in undergraduate science courses. CBE Life Sci. Educ. 14(2) (2015). https://doi. org/10.1187/cbe.14-11-0208 3. Chong, Y., Wong, M., Thomson Fredrik, E.: The impact of learning styles on the effectiveness of digital games in education. In: Proceedings of the Symposium on Information Technology in Education, KDU College, Patailing Java, Malaysia (2005) 4. Codish, D., Ravid, G.: Detecting playfulness in educational gamification through behavior patterns. IBM J. Res. Dev. 59(6), 1–14 (2015). https://doi.org/10.1147/JRD.2015.2459651 5. Djaouti, D., Alvarez, J., Jessel, J.P., Methel, G., Molinier, P.: A gameplay definition through videogame classification. Int. J. Comput. Games Technol., 1–7 (2008). https://doi.org/10. 1155/2008/470350 6. Dondi, C., Moretti, M.: A methodological proposal for learning games selection and quality assessment. Br. J. Edu. Technol. 38(3), 502–512 (2007). https://doi.org/10.1111/j.14678535.2007.00713.x 7. Gosper, M., McNeill, M.: Implementing game-based learning: the MAPLET framework as a guide to learner-centred design and assessment. In: Ifenthaler, D., Eseryel, D., Ge, X. (eds.) Assessment in Game-Based Learning, pp. 217–233. Springer, New York (2012). https://doi. org/10.1007/978-1-4614-3546-4 12 8. Hall, J.V., Wyeth, P.A., Johnson, D.: Instructional objectives to core-gameplay: a serious game design technique. In: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-human Interaction in Play, Toronto Ontario Canada, pp. 121–130. ACM, October 2014. https://doi.org/10.1145/2658537.2658696

68

B. Lemoine et al.

9. Kent, S.: Model Driven Engineering. In: Butler, M., Petre, L., Sere, K. (eds.) Integrated Formal Methods, pp. 286–298. Springer, Heidelberg (2002). https://doi.org/10.1007/3-54047884-1 10. Kim, J.W., Ritter, F.E., Koubek, R.J.: An integrated theory for improved skill acquisition and retention in the three stages of learning. Theor. Issues Ergon. Sci. 14(1), 22–37 (2013). https://doi.org/10.1080/1464536X.2011.573008 11. Kolovos, D., Rose, L., Paige, R., Garcia-Dominguez, A.: The Epsilon Book. Eclipse (2010) 12. Laforcade, P., Mottier, E., Jolivet, S., Lemoine, B.: Expressing adaptations to take into account in generator-based exercisers: an exploratory study about multiplication facts. In: 14th International Conference on Computer Supported Education. Online Streaming, France, April 2022. https://doi.org/10.5220/0011033100003182 13. Lemoine, B., Laforcade, P., George, S.: An analysis framework for designing declarative knowledge training games using roguelite genre. In: Proceedings of the 15th International Conference on Computer Supported Education, CSEDU 2023, Prague, Czech Republic, 21–23 April 2023, vol. 2, pp. 276–287. SCITEPRESS (2023). https://doi.org/10.5220/ 0011840200003470 14. Lemoine, B., Laforcade, P., George, S.: Mapping task types and gameplay categories in the context of declarative knowledge training. In: Proceedings of the 15th International Conference on Computer Supported Education, CSEDU 2023, Prague, Czech Republic, 21–23 April 2023, vol. 2, pp. 264–275. SCITEPRESS (2023). https://doi.org/10.5220/ 0011840100003470 15. Lim, T., et al.: The LM-GM Framework for Serious Games Analysis. University of Pittsburgh, Pittsburgh (2013) 16. Prensky, M.: Computer Games and Learning: Digital Game-Based Learning. Handbook of Computer Game Studies (2005) 17. Rapeepisarn, K., Wong, K.W., Fung, C.C., Khine, M.S.: The relationship between game genres, learning techniques and learning styles in educational computer games. In: Pan, Z., Zhang, X., El Rhalibi, A., Woo, W., Li, Y. (eds.) Edutainment 2008. LNCS, vol. 5093, pp. 497–508. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-69736-7 53 18. Roediger, H.L., Pyc, M.A.: Inexpensive techniques to improve education: applying cognitive psychology to enhance educational practice. J. Appl. Res. Mem. Cogn. 1(4), 242–248 (2012). https://doi.org/10.1016/j.jarmac.2012.09.002 19. Sherry, J.L.: Matching computer game genres to educational outcomes. In: Teaching and Learning with Technology, pp. 234–246. Routledge (2010) 20. Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF: Eclipse Modeling Framework. Addison-Wesley Professional, Boston (2008) 21. Tchounikine, P., Mørch, A.I., Bannon, L.J.: A computer science perspective on technologyenhanced learning research. In: Pan, Z., Zhang, X., El Rhalibi, A., Woo, W., Li, Y. (eds.) Technology-Enhanced Learning. LNCS, vol. 5093, pp. 275–288. Springer, Dordrecht (2009). https://doi.org/10.1007/978-1-4020-9827-7 16

Designing Declarative Knowledge Training Games: An Analysis Framework Based on the Roguelite Genre B´er´enice Lemoine(B) , Pierre Laforcade , and S´ebastien George LIUM Computer Science Laboratory, Le Mans Universit´e, Laval, France {berenice.lemoine,pierre.laforcade, sebastien.george}@univ-lemans.fr Abstract. Designing learning games for the retention of declarative knowledge is a way to provide learners with a large variety of adapted training situations. Such training situations can be considered as game activities built upon questioned facts. Learners must then face various game situations wherein interactive elements and rules are a means to read and answer specific questions about these facts. This chapter is an extended version of [18]. We propose Roguelite as a relevant game genre for declarative knowledge training. Indeed, its core design principles tackle the needs of variety and challenging training situations. Additionally, we propose an analysis framework to help teachers and game developers in identifying the key elements to design training games. This framework includes a set of questions to consider during the preliminary design of any training game for declarative knowledge. We identified and used this proposal in a specific research context about the training of multiplication tables. Following an iterative and prototype-centered approach, we illustrate two iterations about applying the analysis framework to guide the design and development of playable prototypes. Keywords: Declarative knowledge · Serious game · Analysis · Training

1 Introduction Declarative knowledge is part of the knowledge necessary to perform a task. Anderson and Lebiere [2] define it as knowledge of “things we are aware we know and can usually describe to others”. It includes factual information, such as multiplication tables, historical dates or geographical information. Repetition is necessary to encourage the memorization, generalization, and retention of declarative knowledge [15, 25]. Retrieval practice, a concept in cognitive psychology, suggests that the act of recalling and retrieving concepts or facts enhances their long-term retention. Retrieval practice includes low-stakes and no-stakes writing prompts, brief quizzes, flashcards, etc. Research has demonstrated that this learning strategy can significantly improve longterm retention [5, 25]. However, it is important to avoid making repetitive or redundant serious games, as well as those that present an imbalanced challenge relative to the players’ skills, as these can lead to boredom [29]. To reduce this feeling of repetition, serious games focused on declarative knowledge should offer a wide range of tailored training activities. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2024  B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 69–92, 2024. https://doi.org/10.1007/978-3-031-53656-4_4

70

B. Lemoine et al.

Many existing serious games designed to train multiplication tables (an example of declarative knowledge) simply present questions for the players to answer, often accompanied by game mechanics such as time pressure, rewards, scores, and currencies. While there are a few exceptions that introduce advanced interactions and gameplay elements, such as platform-game mechanics where players control an avatar that must move and jump to make choices, these gameplay features can sometimes conflict with the training objectives, leading to failures despite correct answers. Overall, training games often lack engaging, long-term activities. Insufficient gameplay can quickly bore students and reduce their motivation, frequency, and duration of training sessions. It therefore seems important to allocate as much importance to gameplay as to educational content when designing learning game activities [21]. In this article, we propose to highlight the Roguelite genre as a potential solution to declarative knowledge training. This genre is built upon several game design principles for which we discuss their compatibility with the requirements for effective and tailored training of declarative knowledge. Our proposal introduces an analysis framework that guides the design of Roguelite-oriented training games. This framework includes practical steps to be followed in each iteration of a prototyping-based design approach. The objective is to address design needs from both training and game dimensions, covering aspects such as technology (i.e., information required for the game engine and generation algorithm), game mechanics (i.e., how the core mechanics of the Roguelite genre operate), and game structure (i.e., the rules of the game). This framework is not confined to a particular declarative knowledge domain, nor is it designed exclusively for a specific target audience of learners. Nevertheless, to illustrate our proposal, we apply this analysis framework and discuss its implementation during two design iterations within the AdapTABLES project, a specific research context. The structure of this chapter is as follows: Section 2 provides an introduction to our research context, including the AdapTABLES project and its focus on declarative knowledge related to multiplication tables. Additionally, Sect. 3 introduces the Roguelite genre in video games and explores the suitability of Roguelites for designing training games. Following a concise overview of the current state-of-the-art in Sect. 4, Sect. 5 presents a two-dimensional analysis framework, encompassing both gaming and training dimensions. This framework has been applied twice within our project’s context. Finally, Sect. 6 presents the two applications of the framework and discusses the feedback gathered from evaluating a prototype aligned with the initial analysis.

2 Research Context: The AdapTABLES Project The project aims to design and develop a serious game dedicated to the individual training of multiplication tables, targeting students ranging from grades 2 to 6. From a teacher perspective, the training game to design will be adapted, prior to its use, to reflect how teachers consider the training: for example progress, difficulty, source facts to consider, etc. This training structure can be set up for the entire classroom’s students, for a group, or for individuals having specific needs. From a student perspective, the training game will follow the learners’ progress, proposing facts according to their previous training sessions and results. From a player

Designing Declarative Knowledge Training Games

71

perspective, the training game will offer game levels that take into account their preferences. From a game perspective, a same training task should be tackled through different gameplays with different game elements. Finally, at runtime, the training game will have to generate varied training game activities, adapted, both in terms of gameplay and educational content, to the teachers and learners-players perspectives. We followed an iterative co-design and prototyping approach, involving teachers, didacticians of mathematics, and game experts during the design and evaluation phases. At first, two initial steps were necessary: 1) specifying the knowledge to be trained, and 2) choosing a game genre that suits the training of declarative knowledge. These contextual elements are necessary to start designing at a high level the main key concepts and rules of the training game. 2.1 Declarative Knowledge Training An exploratory research [17] has been conducted with the help of a user group composed of teachers and mathematics experts. The objective was to specify the adaptations to take into account when considering the training of multiplication tables from a teacher perspective: what to consider (i.e., source and targets of the adaptations) and how to realize these adaptations. The main two results are: a model of the training organization into training paths, and the specification of five detailed training tasks. A training path, see Fig. 1, is represented by a set of objectives ordered by prerequisite relationships. An objective (e.g., “Work on the Table 2”) is broken down into progressive levels of difficulty. Each level is itself broken down into training tasks (e.g., “Level 1: Completion 1 with search for the result, Identification by choice of the correct facts”). A task is defined by its type and parameters. The levels’ achievements are considered from both a percent of encountered facts and a percent of achievement to reach.

Fig. 1. Knowledge Structure [18].

2.2 Different Tasks Objectives and Parameters The teachers have identified different questions that served as a basis for identifying five distinct tasks:

72

B. Lemoine et al.

– Completion 1: i.e., complete an incomplete fact that has one missing element (e.g., 3×? = 15, 15 =? × 5, 3 × 5 = ?); – Completion 2: i.e., complete an incomplete fact that has two missing elements (e.g., ?×? = 15 with a set of given choices [3, 6, 5, 10], ? × 5 = ? or 3×? = ? also with sets of given choices); – Reconstruction: i.e., replace, in the correct order, all important elements of a fact (e.g., ?×? = ? with a set of given choices [3, 6, 5, 10, 15]); – Identification: i.e., identify the correctness or incorrectness of one or several facts (e.g., 3 × 5 = 15, true or false?); – Membership Identification: i.e., identify the elements that share or do not share a given property (e.g., [3, 5, 9, 14, 21] which are results of table 3?). The training tasks are defined based on teachers’ opinions and preferences, allowing them to choose and configure these tasks for each {objective, level} combination. For instance, Table 1 presents the parameters for the Completion 1 task, including the targeted multiplication tables, the position of the multiplicand and the result, the range of multipliers (minimum and maximum), the elements to search for, the order of the questions, the response modality, and the maximum response time. Table 1. Examples of parameters for the Completion 1 training task [18].

Adaptable element

Possible Values

Targeted Table(s)

From 1 to 12

Multiplicand Position

Left ∨ Right

Result Position

Left ∨ Right

Multiplier Interval

Integer Min/Max in [1, 12]

Element to search

Result ∨ Multiplicand ∨ Operand

Questions Order

Ascending ⊕ Descending ⊕ Random

Response Modality

Choice between propositions ∨ Input

Max Response Time

Time in seconds

Some Examples 1 × 2, 1 × 3, 1 × 4.. ∨ 2 × 1, 3 × 1, 4 × 1.. 1×2=2∨2=1×2 [1, 5] ∨ [5, 10] ∨ [1, 12] 1×?=2∨?×2=2∨ 1×2=?

Since we are using a prototyping design approach, the design process is conducted incrementally. Consequently, each new iteration incorporates additional information compared to the previous one. In this article, we focus on presenting the initial two iterations of our design, which do not yet encompass the complete knowledge structuring as outlined. The primary prototypes employ a parameterization approach, wherein teachers are required to provide the parameter values (as indicated in Table 1) for each learner within the game.

Designing Declarative Knowledge Training Games

73

3 A Relevant Game Genre for Training Purposes 3.1 The Roguelite Genre Over the past decade, the Roguelike and Roguelite video game genres have experienced a significant surge in popularity. These genres originated from the groundbreaking game that introduced this style of gameplay, Rogue [31]. Rogue was a turn-based dungeon crawler where players had to navigate through levels of a dungeon, battling enemies, collecting items, and progressing further. The Berlin Interpretation [10] defined eight key factors that characterize Roguelikes, including: – Random generation: Each playthrough is unique due to the random or semi-random generation of levels, enemy placements, item locations, and environmental conditions. This randomness adds an element of surprise and unpredictability, requiring players to adapt their strategies on the fly rather than relying on memorized patterns. Procedural generation is typically employed to avoid unwinnable situations. – Permanent death: When the player’s avatar dies, all progress is lost, and they must start the game from the beginning. There is no carryover of progress between runs. While many Roguelike games adhere to these eight key aspects, some games deviate from certain elements. As a result, these games have been referred to as “roguelike-like” or “roguelite”. The Roguelite genre emerged as a way to differentiate these games from traditional Roguelikes. Roguelites introduce macro-level objectives by allowing players to carry over certain items or upgrades between attempts. This persistent progression system enables players to gradually become stronger across multiple playthroughs, increasing their chances of success in subsequent runs. Some well-known commercial Roguelite games include Hades, Enter the Gungeon, The Binding of Isaac, Rogue Legacy, Children of Morta, and Dead Cells. These games offer diverse gameplays, game styles, lores, and features, as well as permanent elements that contribute to achieving cross-run objectives (e.g., weapons, currencies, upgrades, characters, etc.). Collectible resources, for example, can persist between deaths and be used to unlock permanent upgrades, enhancing the player’s chances of success. Failure is then an integral part of Roguelites. Players often face new mechanics, traps, challenging enemies, bosses, and various features that require learning, resulting in multiple failures and deaths before achieving their first complete playthrough or run. Despite the repeated losses, Roguelites typically feature fast restart times, swiftly immersing players back into the action. With each subsequent run, players gain a deeper understanding of the game’s underlying mechanics, enabling them to progress further. Replayability is also another significant aspect of Roguelites. Every run offers a distinct experience. Players can adapt their strategies and objectives based on the dynamically generated environments. The ever-changing nature of levels and encounters enhances the replayability of Roguelite games, as no two runs are identical. Additionally, many Roguelites incorporate replay value beyond simply reaching the game’s end. This can take the form of a new game+ mode (e.g., Rogue Legacy) or a storyline that requires defeating the final boss multiple times (e.g., Hades or Dead Cells). In both cases, players embark on a new, more challenging playthrough, relinquishing their

74

B. Lemoine et al.

progress towards the primary game objectives while retaining their persistent items and upgrades. When playing a Roguelite game, several moments can be distinguished (Fig. 2): – A play session is a temporal session that begins when the player starts to play and ends when they stop playing. According to the duration of a play session, one or several runs can be concerned. – A run or playthrough encompasses a complete game experience, starting from the game’s beginning and ending either at game over or upon reaching the conclusion of the game (often involving a final boss battle integrated into the game’s storyline). A run can be initiated within the current play session or be a continuation from a previous session if the game supports saving the player’s progress. – A game level session corresponds to the completion of a specific level within the game. Depending on the game, players may encounter a single large level to complete or multiple levels. Procedural generation can be applied to one or multiple game levels simultaneously. For example, in Rogue Legacy, a run involves the completion of a single large game level, structured into four areas (castle, forest, dungeon, and tower), all generated at the beginning of the run. In cases where a run consists of a sequence of consecutive game levels, each level is generated one after the other, often with an increasing level of difficulty.

Fig. 2. Different times when considering playing a Roguelite game [18].

3.2

Adequacy of Declarative Knowledge Training with Roguelite Genre

The training of declarative knowledge requires learners-players to accomplish multiples training sessions, repeating the training activity, but with adapted content to train and varied ways in presenting the activity. The procedural generation feature can be leveraged to generate diverse training scenarios and content, allowing for the training of declarative knowledge in a more engaging and varied manner. The Permadeath Mechanics can be utilized to reinforce the importance of knowledge retention and encourage players to learn from their incorrect answers or lack of knowledge. It can motivate players to actively acquire and retain declarative knowledge. The Difficulty Progression feature also consists in delivering a gradual increase in difficulty as players progress into the generated levels. This natural

Designing Declarative Knowledge Training Games

75

difficulty curve can be leveraged in serious games to provide a scaffolded learning experience. The game can start with simpler declarative knowledge challenges and gradually introduce more complex concepts as players improve, ensuring a suitable learning pace and maintaining engagement. The Replayability characteristic aligns well with the training of declarative knowledge, as repeated exposure and practice are key factors in knowledge retention. By designing a serious game with Roguelite elements, learners can revisit and reinforce their declarative knowledge through multiple playthroughs. The Roguelite genre, with its challenging gameplay, strategic decision-making, and unpredictable nature, can provide an immersive and motivating learning environment. By integrating declarative knowledge challenges into its gameplay, such training games could enhance learner motivation, focus, and overall learning outcomes. Overall, the Roguelite genre offers a range of features and mechanics that can be effectively harnessed to design serious games for training declarative knowledge. Its procedural generation, permadeath mechanics, difficulty progression, replayability, and ability to captivate and motivate players make it a relevant and promising choice for creating engaging and effective training experiences. Therefore, Roguelite seems to be a suitable genre for declarative knowledge training, where the training game activities generated are game levels. 3.3 Targeted Adaptations The adaptation of generated game and training activities is not straightforward. It requires to be characterized from the game perspective as well as the learning perspective. Adaptation is often characterized by three concepts [32]: – the source (i.e., to what do we adapt?); – the target (i.e., what is adapted?); – the pathways (i.e., what methods are used to adapt the target to the source?). Foremost, in our context, the adaptation targets generated game level (e.g. dungeons composed of interconnected rooms) and their elements (i.e., what is adapted). Therefore, it takes place during the generation of an activity (i.e., when it is adapted). In the spectrum of adaptation defined by [23], our targeted adaptation can be positioned in-between adaptivity and adaptability, as it uses user’s data previously collected to automatically generate an activity that is adapted to the user. In the literature, the gaming adaptations are mostly based on players/personalities profiles [9, 22, 30] or players characteristics, such as age and genre. In our context, adaptation from a game perspective seeks to take into account player preferences to choose the game elements (i.e., source). The main idea is to represent preferences as game elements that can be activated/deactivated by the player. From the learning perspective, our intention is to use knowledge of the learner (e.g., actual level, previous mistakes) from his/her learning path (source) to adapt the dungeons’ difficulty in terms of educational content. Since the adaptation is an integral part of the generation, in our context, this article does not dissociate them (i.e., the generation criterion includes adaptation, see Sect. 5).

76

B. Lemoine et al.

3.4

Research Question

Designing a Roguelite oriented game for training declarative knowledge requires to answer several design questions: What is generated? How and when does the avatar die? What are the consequences? What varies? What indicates a progression? And so on. Furthermore, these questions need to be answered from both an educational and a game perspective. Moreover, in a prototyping design approach, the answers may change from one prototype to another. Therefore, our research question is: How can the design of Roguelite-oriented training be facilitated in a prototyping design approach? Our proposal is an analysis framework that helps designers ask the right questions and make their choices explicit during each design iteration.

4 State of the Art Numerous frameworks and methods for games, serious games, and game-based scenario design can be found in the literature. However, most of these frameworks are primarily focused on analyzing existing games [12]. One notable framework proposed by [7] is a learner-centered framework consisting of four dimensions: Representation, Context, Pedagogy, and Learner. This highlevel design approach aids in the design of game-based learning scenarios but does not facilitate the transition from educational content to concrete game elements. Another framework, the DPE framework introduced by [33], extends the Mechanics, Dynamics, Aesthetics (MDA) framework [11] for serious games. The DPE framework is divided into three categories (design, play, experience) and is described by four criteria: Learning, Narrative, Gameplay, and User Experience. Additionally, [26] presents a method involving a series of questions covering various aspects to consider during game design. This method is more generic and not specific to a particular type of knowledge or game genre. [1] describes the GOM II framework, an extension of the Game Object Model (GOM), which considers educational games as compositions of elements described by abstract and concrete interfaces. However, this work is theoretical and focuses on the general design of games rather than their specific implementation. The LM-GM framework proposed by [3] enables the association of Learning Mechanics with Game Mechanics through the use of Serious Game Mechanics (SGM). However, this framework leans more towards game analysis rather than game design. [6] presents the ASTMG conceptual model based on activity theory, aiming to provide a better understanding of the relationships between serious game elements and learning objectives. Similarly, this framework is also more focused on game analysis. Many other framework and methodologies exist [4, 14, 20, 27, 34]. Some of these approaches offer methods that are more closely aligned with specific game genres, such as adventure games or story-oriented games [8]. However, these works are primarily generic and not tailored to a specific type of knowledge or game genre. Considering the specific context of the Roguelite genre, none of the existing frameworks fully meet the requirements. Nonetheless, these frameworks are not mutually

Designing Declarative Knowledge Training Games

77

exclusive and can be used in conjunction with each other. For example, [26] could be used to describe the general design of the game, while ASTMG [6] could be employed to ensure coherence in each prototype. In summary, existing frameworks that share the goal of assisting in the design of learning games focus on different pedagogical objectives, knowledge types, and game genres. None of these frameworks specifically cater to situations where the game genre is already identified due to its relevance to a specific learning objective. Our analysis framework is specifically dedicated to Roguelite games for declarative knowledge retention, making it a highly specific scope. However, multidisciplinary teams also require tailored frameworks to guide them in designing relevant, adapted, and wellbalanced learning games.

5 Analysis Framework for a Roguelite Learning Game Although the design tends to focus on the training and learning dimensions, the game aspect must not be neglected. Indeed, as Prensky noted [24], the main reason for learning game failure lies in their lack of gameplay. To this extent, the proposed framework aims to provide a means of analyzing the design needs of Roguelite-oriented learning games by specifying both dimensions through specific criteria. To design a Roguelite game, the initial step involves defining game mechanics. This includes determining how the game world is generated (e.g., what is generated and how it is generated), when permanent death occurs, and how progression works (e.g., which elements are carried over). The generation mechanism, within this context, encompasses specifying the aspects that should vary during gameplay. Similar to learning, an important concept is the progression of difficulty, where it becomes crucial to define how difficulty increases and when it does so. From both perspectives, these five mechanisms (Generation, Death/Hurt, Variety, Progress, and Difficulty) serve as criteria for analyzing the design requirements. Each criterion comprises a set of questions related to its respective mechanism. Answering these questions is essential for clarifying the design needs of the educational game. The following are the questions for each criterion: 1) Generation Q1. What elements are generated? Q2. When are these elements generated? Q3. Based on what criteria are they generated? (i.e., sources of generation) 2) Death/Hurt Q4. Under what circumstances can the avatar be injured or die? Q5. What are the consequences of being injured or killed? Q6. Where can the avatar sustain injuries or be killed? 3) Variety Q7. Which elements exhibit variation? Q8. How do these elements vary? (i.e., are the variations triggered by player action? Are they random? Is it a combination of both? Are they guided by heuristics?) 4) Progress Q9. What is preserved or carried over between each death? (i.e., which elements?)

78

B. Lemoine et al.

5) Difficulty Q10. What factors contribute to increasing or decreasing the difficulty? Q11. How is the difficulty progression designed? (i.e., if multiple elements affect the difficulty, what is the sequence in which they occur?)

Table 2. Grid for Design Needs Analysis [18]. Educational Game Perspective Perspective

Criteria Q1 Generation Q2 Q3 Q4 Death/Hurt Q5 Q6 Variety

Q7 Q8

Progress

Q9

Difficulty

Q10 Q11

Table 2 provides a structure for conducting a needs analysis. Each criterion is represented by a row, which is further divided into X sub-rows, with each sub-row corresponding to a specific question. The columns represent different dimensions, one for the game aspect and another for the educational aspect. If there is shared information between both dimensions, it can be combined and specified in the related cells. This framework is not restricted to any particular didactic field and can be applied to design training games within the Roguelite genre. The subsequent section illustrates the application of this framework in the context of the AdapTABLES project.

6 Framework Application: AdapTABLES Project This section is divided into five subsections. Subsect. 6.1 provides a detailed analysis focused on the design requirements of the initial prototype. It outlines the necessary considerations and factors to be addressed during the design phase. The first prototype is then described in Subsect. 6.2. This subsection offers an overview of the existing implementation. In the Subsect. 6.3, feedback gathered from real-life conditions is presented. These informal feedback provides insights into how the prototype has been received and the observations made by users during its usage. The design needs analysis for the subsequent prototype is detailed in Subsect. 6.4. It specifies the requirements and improvements that need to be addressed to enhance the design and functionality of the game. The second prototype, currently in development, is presented in the Subsect. 6.5.

Designing Declarative Knowledge Training Games

79

6.1 First Analysis In our iterative process, the first step involved conducting a design needs analysis for the initial prototype. The primary objective was to set aside the knowledge structure and concentrate on a single task, specifically Completion 1 for multiplication tables. This task was manually integrated into the game, ensuring that the information remains persistent across different training sessions. An overview of the design needs analysis for the first prototype is provided in Table 3. Generation. The generated element (Q1) in the game design is a dungeon level comprising various elements, such as rooms, their order, contents, positions, and values. Each room can be categorized as either a question room or a no-question room. A question room is associated with a specific training task defined by the teacher and configured before training. On the other hand, a no-question room is designed for entertainment purposes and includes enemies, traps, and other game elements. The inclusion of non-question-based rooms aims to prevent learners-players from perceiving the game as merely a disguised questionnaire. Following the game flow depicted in Fig. 2, a new dungeon level is generated (Q2) when the player requests it during gameplay. As discussed in Sect. 3.3, our game adaptations are tailored to cater to players’ preferences. To identify these preferences, we examined existing Roguelite games and discovered that many of them incorporate a purchasing mechanism for items such as equipment, upgrades, and skills. Collaborating with game designers, we categorized the preferences into three types: 1) Content, 2) Rules, and 3) Visuals & Audio (Q3). Content preferences encompass additional objects that exist during gameplay or elements that modify the activity’s structure when activated. Examples include extra lives or different dungeon modes, such as linear or labyrinthine layouts. Rules preferences involve elements that impact the players, the avatar, or the Non-Player Characters (NPCs) behavior. Examples of rules preferences are increasing the speed of enemies or introducing a game goal where completing an activity without mistakes earns +10 coins. Regarding the game dimension, the generation process takes into account the three types of preferences mentioned earlier (Q3). For example, if a player has purchased and activated the labyrinthine mode, the generation algorithm considers this preference. To ensure the tracking of learners’ in-game progress, the generator also takes into consideration the last level number and state. If the previous level, let’s say #5, was successfully completed, the next level generated will be #6. However, in the case of death, the next level will be reset to #1. The level number influences various aspects of the dungeon, such as its length, the number of rooms with questions, and the overall dungeon effects (levels above #4 are set in dark mode). In the educational dimension, the parameters for the tasks can vary based on the learners’ level (as discussed in Sect. 2). Therefore, each learner has their own customized setup for Completion 1 task, defined either individually or shared by the teachers. These parameter values are utilized to generate relevant questioned facts associated with the rooms that have questions (Q3). To avoid repeating successful questioned facts, the system takes into account the previously encountered facts and their outcomes. It is important to note that conflicts can arise between the game and training dimensions. For instance, if a learner-player has activated the “labyrinthine” mode while the task setup requires encountering questioned facts in a specific ascending or descending

80

B. Lemoine et al.

order. In such cases, our recommendation is to prioritize the training dimension over the game dimension. Death/Hurt. When the avatar is injured (Q5), it loses a life, and when all lives are depleted, it results in the avatar’s death. The player (via the avatar) can get hurt (Q4) by coming into contact with enemies, falling into pits (game dimension), or by answering questions incorrectly (educational dimension). Additionally, running out of time, which is predetermined in the task parameters, can also result in the avatar getting hurt. Incorrectly answering a question or running out of time can only occur in question rooms (Q6), while encountering the wrong enemy or falling into a pit can happen in both types of rooms. Variety. The game generation algorithm offers a range of choices for different elements. In the context of Roguelite games, variations mainly include selecting different object types, determining their positions, shaping the objects and dungeons, and controlling the quantity of elements present (Q7). Consequently, the positions of decorative objects and the shapes of rooms are selected “randomly” (Q8), while maintaining coherence to ensure that elements are not placed outside the room or inaccessible to the avatar. The gameplay itself represents how learners carry out the tasks. Having only one type of gameplay per task can lead to a sense of repetition. To avoid this, four distinct gameplay variations (Q7) have been identified for the Completion 1 task type: opening chests with the correct answer, passing through doors with the correct answer, touching enemies with the correct answer, or typing in the correct answer. This diversification of gameplay helps prevent monotony and adds variety to the learning experience. In the learning dimension, the facts to be practiced are distinct until each fact has been encountered at least once (Q7). Furthermore, depending on the parameters specified in Table 3, the format of the facts (e.g., missing elements, position of the equals sign, etc.) varies (Q8). While there is an element of randomness involved in the variation of these elements, it is constrained by the game’s preferences, educational considerations (i.e., choices made by teachers), and previous selections made by the algorithm. Progress. Our approach involves implementing a purchase mechanism inspired by Roguelite games. Players have the ability to buy items and subsequently activate or deactivate them as desired. Therefore, game progression can be observed through the elements purchased and the number of coins accumulated (Q9). Some Roguelite games only retain progression when players successfully complete levels or dungeons without the avatar dying. As a result, coins are earned only upon completing a dungeon entirely. These coins can be obtained during the dungeon journey (randomly appearing when opening a correct chest) or at the end of the dungeon based on activated rules (e.g., +10 coins for completing the dungeon level without any mathematical errors). Training progression becomes apparent at the end of a dungeon run, whether by reaching the end or experiencing the avatar’s demise, where statistics are presented showcasing the mistakes made, correct answers given, and areas that require further improvement. These results persist across subsequent runs. Difficulty. In the game dimension, the difficulty increases within a run by progressively increasing the number of rooms with questions (e.g., starting with 5 rooms, then 7, then

Designing Declarative Knowledge Training Games

81

9, then 11, and so on) (Q10). The total number of rooms may still vary due to the generation process, which can randomly include rooms without questions along with the overall structure of the dungeon, such as a tortuous but linear layout or a labyrinthine design (Q11). After successfully completing five levels without losing, a more challenging level is introduced where the player navigates in darkness, with only a torch illuminating the avatar (Q10). In the educational dimension, the difficulty increases based on parameters defined by teachers. As long as these parameters remain unchanged, the questioned facts will consistently reflect the established setup (Q11). These design choices regarding difficulty progression are open to debate and aim to establish an initial version of the game’s difficulty curve, which is subject to further refinement. Table 3. Design Needs of the prototype #1 [18]. Criteria Q1: What? Generation

Q2: When? Q3: Based on? Q4: What?

Death/Hurt

Q5: When? Q6: Where?

Educational Perspective

Game Perspective

One task and one questioned fact per Dungeon + rooms + entry + exit room-with-question When a new game level is required “Completion 1” set-up Previous level number and state Current progress among possible facts Activated game elements or rules Task parameters have priority on activated game elements if conflict Being touched by foes, falling into holes Injuring causes heart lost, no more hearts causes death Question rooms Any room with foes or holes

Incorrect answers or time out

Variety

Q7: What? Q8: How?

Facts Progress and past results

Rooms with gameplay and content Random

Progress

Q9: What?

Success or failure on met questioned facts

Coins collected during successful game levels + purchased elements

Q10: What?

Questioned facts

Q11: How?

In relation with the task parameters

Difficulty

Dungeon level length + dark mode According to previous level number and state

6.2 First Prototype The first prototype of the game was developed using the Unity game engine, employing C# scripts to create a 2D game. It has been exported and deployed as a Web platform WebGL build, allowing for easy accessibility. The game incorporates an HTTP REST API, developed in .Net Core, to store data in a NoSQL MongoDB database. Additionally, a web application teacher dashboard has been created using .NET with Blazor. This dashboard enables teachers to monitor their students’ progress, including the current multiplication parameter settings, achievements related to multiplication facts, and

82

B. Lemoine et al.

purchased game elements. Currently, the available version of the game is in French and can be played using a gamepad or a keyboard. Figure 3 showcases six screenshots from the prototype. Screen 3a displays a portion of the “hub” area, featuring four accessible elements: statistics (“Stats G´en´erales”), progress (“Progression”), educational settings (“R´eglages”), and the purchase panel (“Achats”) for game preferences). The hub area serves as the starting point for each run and is where players can review their progress or manage collectibles, such as purchases and activation/deactivation of game elements. Screen 3b presents a section from the educational settings panel, as outlined in Table 1. Screen 3c showcases a segment from the item purchase panel. Screens 3g, 3e, and 3f illustrate examples of the currently implemented gameplay mechanics within the dungeon levels, including selecting the correct foe, door, and chest, respectively. Finally, Screen 3h displays an example of a room where the player needs to type in their answer. The screenshots provide a visual representation of the prototype, demonstrating various aspects of the game’s interface and gameplay mechanics. 6.3

Experiment Feedback

The design of the prototype underwent three iterations, during which it was tested in real conditions and refined based on feedback from teachers and students. The empirical feedback encompassed various aspects, including ergonomic concerns related to keyboard versus gamepad usage, the overall playability experience, the inclination for replayability, and the motivation to play and practice multiplication tables. Regarding the death/injury criterion, several issues were identified. Firstly, children sometimes made unintended choices due to the current touch-oriented interactions that did not require the use of buttons or keys. Although this gameplay problem is tied to ergonomics, it could lead to a sense of unfairness in the reward/punishment system. Secondly, some rooms had foes positioned randomly in close proximity to the avatar’s entry area. This resulted in children losing hearts without sufficient time to avoid them. Additionally, certain rooms with questions also contained holes to avoid. Teachers pointed out that these game elements could distract children when they should be focused on answering the questions. In terms of the variety criterion, children appreciated the three different gameplays for selecting an answer (door, chest, and foe). However, it was noted that the prototype lacked sufficient variation. The gameplay involving touching the correct foe was considered confusing by both children and teachers. In some rooms, foes needed to be avoided, while in others, players had to guide their avatar to touch the foe corresponding to their chosen answer. This counter-intuitive approach led to teachers suggesting that associating a correct answer with a negative action (i.e., killing foes) should be avoided. The prototype also included rooms with questions that required directly typing in the correct answer on the keyboard. Based on the correctness of the answer, the correct door or chest would open, or all foes would die. An incorrect answer resulted in opening an empty chest, leading to a door that led to a dead end, or having no effect on the touched foe. In all three cases, however, the avatar would be injured. These situations were initially designed to vary the response method while maintaining similar room content but ultimately proved to be confusing.

Designing Declarative Knowledge Training Games

(a) Prototype’s hub

(b) Training settings panel

(c) Gaming settings panel

(d) A no-question room

(e) Door Gameplay

(f) Chest Gameplay

(g) Foe Gameplay

(h) Answer entry

Fig. 3. Screenshots of the features from the first prototype (text in French) [18].

83

84

B. Lemoine et al.

In terms of progression, it was found that collecting coins was unbalanced. Only gameplay modes focused on chests (choosing or typing answers) had a chance to randomly contain ‘+1’ or ‘+3’ coins (or a ‘heart’/life). While certain purchased and activated rules could provide alternative ways to earn coins, early successful dungeon levels might not result in earning any coins. Additionally, the initial items purchased primarily fell under the content category (such as extra hearts), which were preferred over rules. Furthermore, some teachers were unconvinced about allowing learners to freely buy and activate rules that pressured them to act faster, potentially causing stress to answer quickly. As mentioned earlier, the generation process could disable certain activated rules that did not align with the current configuration of the Completion 1 task. This could lead to feelings of confusion when encountering the generated dungeon levels. 6.4

Second Analysis

This second analysis, cf. Table 4, was conducted to inform the design of the upcoming prototype for multiplication tables training. Building upon the feedback received from teachers and learners, we collaborated with game designers to identify solutions and determine further directions for both the game and training dimensions. Similar to the first prototype’s functional scope, the knowledge structure is still not considered, although all five task types (presented in the Subsect. 2.2) are now considered. The prototype will allow manual parameterization of the current training configuration for individual children based on their specific progress, represented by the {objective, level} pair in the learner’s training path (refer to Fig. 1). Generation. The generated element (Q1) remains a dungeon level consisting of organized rooms categorized as either question-free or rooms with one question associated with a specific task type as per the training setup. Each room contains various interactive elements. The generation of a new game level occurs when the player requests it, either from the hub-room to start a new run or after the debriefing screens following a successful dungeon level (Q2). From the game dimension, the generation process continues to take into account the last level number and state (Q3), as well as the features that have been purchased and activated. However, the purchasable and activatable elements have been modified, and these elements are further explained in the following categories. From an educational perspective, each generation considers the learner’s current configuration for all task types (ranging from 1 to 5) and takes into account the previously encountered questioned facts and their results. Death/Hurt. The player continues to experience injury (Q4) when interacting with foes, falling into pits (game dimension), answering questions incorrectly, or running out of time (educational dimension). However, a significant difference is that question rooms will no longer contain traps or game elements that can harm the avatar (Q6). Incurring injuries in question rooms will solely result from providing incorrect answers or exceeding the time limit. The consequences of sustaining an injury remain unchanged (Q5), leading to the loss of a life or resulting in the avatar’s death if there are no more hearts left.

Designing Declarative Knowledge Training Games

85

Table 4. Design Needs of the Future Prototype (Bold describes changes from the first analysis). Criteria Q1: What? Generation

Q2: When? Q3: Based on? Q4: When?

Death/Hurt

Q5: What? Q6: Where? Q7: What?

Variety Q8: How?

Progress

Educational Perspective

Game Perspective

One task and one questioned fact Dungeon structure + rooms per room-with-question When a new game level is required All tasks set-up Previous level number and state Current progress among possible facts Equipped items Task parameters have priority on activated game elements if conflict Being touched by foes, falling into holes Injuring causes heart lost, no more hearts causes death Question rooms Only rooms with no question

Incorrect answers or time out

Different types of rooms, types of gameplays, types of elements Based on the available equipments, Progress and past results gameplays, elements, and in relation to the tasks ⇐⇒ gameplays mappings Facts

Q9: What?

Success or failure on met questioned facts

Q10: What?

Questioned facts

Q11: How?

In relation with the task parameters

Difficulty

Coins collected during successful game levels + purchased items Dungeon level length + curses According to previous level number and state

Variety. In the initial prototype, various types of rooms combined game elements and gameplays. However, to introduce greater room variety (Q7), we have adopted a new approach, as depicted in the conceptual class diagram presented in Fig. 4. This approach involves two key aspects. Firstly, the different types of tasks are mapped to specific gameplay types based on the current task parameters (mapping work presented in [19]). Each gameplay type requires a quantified number of elements possessing the specified ability. Secondly, the types of rooms are defined by their positions, which can accommodate different elements possessing specific abilities (and sizes). Consequently, different types of game elements are associated with the abilities they can manage. These combined elements will play a crucial role in determining the generation of rooms and their respective built-in elements (Q8). Consequently, the purchase mechanic now involves players selecting items that, when equipped (i.e., activated), unlock new types of gameplays that can occur in specific rooms, as long as they align with the associated task. By providing variants of game elements that share certain abilities, game developers can enhance the potential for variations within the game. Progress. In terms of progression, equipment items can now be purchased and retained across multiple runs. The coin mechanism remains, but it will be adjusted so that learners earn one coin for each correct answer. Furthermore, questioned facts encountered

86

B. Lemoine et al.

and their associated results will be saved even after the avatar’s death. As a result, progression can be observed through the availability of equipment items, the number of coins accumulated, and the stored results of facts, which are accessible outside of a game level (Q9). Difficulty. The educational difficulty remains consistent with the first prototype, determined by the parameters of the tasks set by the teachers (Q11). From a gameplay perspective, the progression based on the length of the dungeons is maintained, but a new gameplay mechanism called “curses” is introduced (Q10). The game progression will be structured around different minimum thresholds, with each threshold unlocking curses that may or may not occur during the generated dungeon level (Q11). For instance, if we consider thresholds every three dungeon levels, the generation of level #10 could involve a maximum of three curses or, with some luck, none at all. The inclusion of curses adds an additional layer of challenge and unpredictability to the gameplay, increasing the overall difficulty and providing new obstacles for players to overcome.

Fig. 4. Conceptual class diagram illustrating the domain elements and relationships involved in considering a wide variety of rooms with question. Yellow concepts are to be generated; blue concepts are specifications of the game and didactic elements available; green concepts concern each learner-player. (Color figure online)

Designing Declarative Knowledge Training Games

87

6.5 Second Prototype

(a) New shop

(b) Equipment panel

(c) A no-question room with traps

(d) A question room in dark mode

(e) Completion 1 Gameplay

(f) Reconstitution Gameplay

(g) Identification Gameplay

(h) Membership Id. Gameplay

Fig. 5. Screenshots of the features from the second prototype (some texts are in French).

The second version of the game is being created using the Unity game engine and is accessible through a Web platform WebGL build. However, there has been a change in the game’s structure compared to the first prototype. The software responsible for generating customized and diverse learning activities is no longer integrated within the

88

B. Lemoine et al.

(a) Movable ability

(b) Breakable ability

(c) Catchable ability

(d) Followable ability

(e) Openable ability

(f) Pushable ability

(g) Rotable ability

(h) Standable ability

Fig. 6. Different abilities, gameplays and elements for a same Completion 1 task.

game itself. Instead, it is now being developed separately. This separation allows us to focus on designing the generator as a distinct research subject, while the game interprets the levels generated by the generator. To create the generator component, we are using Java and drawing upon theories from the Model-Driven-Engineering (MDE) field [13]. Additionally, we are using the practical tools provided by the Eclipse Modeling Framework [28] and the Epsilon Framework [16]. We have developed a servlet-based HTTP REST API to facilitate com-

Designing Declarative Knowledge Training Games

89

munication with the generator. The Unity game receives the levels in the form of XML files. As for user input, the game can still be played using either a gamepad or a keyboard. We illustrate the major changes from the first prototype in Fig. 5. Screens 5a shows how learners-players can purchase equipment and activated or deactivated them in 5b. Some items allow to augment the avatar’s hearts. Other items concern the abilities and allow to extend the encountered gameplays. Last items can ease the progression through the potential curses, e.g. the lantern that surround the avatar with a light, making easier the play through the darkest level, or the encompass that unlock the display of a map of the current dungeon level, useful with a labyrinthine dungeon to distinguish rooms already seen from others. Screens 5c and 5d are examples of random traps and dark mode situations that players can met. Four of the five tasks types are then illustrated. Screen 5e is about the Completion 1 task. It uses a statue having the rotable ability to propose 3 choices, validation of a choice being made by triggering the switch. Screen 5f illustrates the use of various floor interrupts to do/undo (selectable ability) make choices of values for reconstituting the fact (Reconstitution task). Screen 5g is also using a rotable ability with a statue to let learners choose if the proposed fact is correct or incorrect (Validation task). Last screen 5h illustrates the Membership identification task. The learner-player have to push to the left all blocks proposing a correct proposition (a result for table 10). Figure 6 illustrates different abilities, and related gameplays and elements, for a same task (Completion 1). This second prototype has not been evaluated yet. However, it was presented during The Science Festival 2023, known as “la fłte de la science” in French, which is an annual event aimed at promoting scientific knowledge and discovery among the public in France. In addition, many experiments are planned in order to collect qualitative feedback from both teachers and learners viewpoints (from different school grades).

7 Conclusion In this article, two main points were covered. Firstly, it discussed the suitability of the Roguelite game genre for training declarative knowledge. Secondly, it introduced a framework for conducting a design needs analysis specifically tailored for Rogueliteoriented learning games. The proposed framework offers an initial understanding of the mechanisms and choices that benefit both the game and training aspects. The core idea is to enable a two-dimensional design approach that considers both the play and learning dimensions separately, while ensuring their compatibility through verification and maintenance. This framework was applied in the context of the AdapTABLES project, which aims to develop a multiplication tables training game. The article describes how the framework was utilized in two iterations. The first advantage of the framework is the traceability of design choices. Indeed, at each iteration, the choices are explained and then summarized. This traceability facilitates the evolution of the design without the risk of involuntarily going backwards. On the other hand, the visual synthesis (cf. Table 2) makes it possible to check that none of the dimensions has been neglected (i.e., presence of an empty box in case of neglect). In a prototyping approach, iterations are essential to fix certain settings. However, it would seem that the use of such a framework (i.e., allowing traceability as well

90

B. Lemoine et al.

as visual verification of the non-neglect of a dimension) could reduce the number of iterations required. Finally, this tool provides a support that can be understood by all stakeholders of the design process. However, the proposed framework takes into account a rather precise context: training or retrieval practice in the context of the Roguelite video game genre. Moreover, the criteria used are those that we consider essential. Consequently, other criteria might be considered essential by other researchers or game designers. In particular, some criteria depending on the application domains might be interesting to add. Looking ahead, we are interested in applying this framework to other fields beyond mathematics. Currently, they are actively working on implementing it for history and geography topics, such as historical dates and countries of the European Union.

References 1. Amory, A.: Game object model version II: a theoretical framework for educational game development. Education Tech. Research Dev. 55(1), 51–77 (2007). https://doi.org/10.1007/ s11423-006-9001-x 2. Anderson, J.R., Lebiere, C.J.: The Atomic Components of Thought, 0 edn. Psychology Press, January 2014. https://doi.org/10.4324/9781315805696. https://www.taylorfrancis. com/books/9781317778318 3. Arnab, S., et al.: Mapping learning and game mechanics for serious games analysis: mapping learning and game mechanics. Br. J. Edu. Technol. 46(2), 391–411 (2015). https://doi.org/ 10.1111/bjet.12113 4. Barbosa, A.F.S., Pereira, P.N.M., Dias, J.A.F.F., Silva, F.G.M.: A new methodology of design and development of serious games. Int. J. Comput. Games Technol. 2014, 1–8 (2014). https:// doi.org/10.1155/2014/817167 5. Brame, C.J., Biel, R.: Test-enhanced learning: the potential for testing to promote greater learning in undergraduate science courses. CBE—Life Sci. Educ. 14(2) (2015). https://doi. org/10.1187/cbe.14-11-0208 6. Carvalho, M.B., et al.: An activity theory-based model for serious games analysis and conceptual design. Comput. Educ. 87, 166–181 (2015). https://doi.org/10.1016/j.compedu.2015. 03.023 7. De Freitas, S., Jarvis, S.: A framework for developing serious games to meet learner needs. In: Interservice/Industry Training, Simulation & Education Conference, I/ITSEC, January 2006 8. De Lope, R.P., Medina-Medina, N., Soldado, R.M., Garcia, A.M., Gutierrez-Vela, F.L.: Designing educational games: key elements and methodological approach. In: 9th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games), pp. 63–70. IEEE, Athens, Greece, September 2017. https://doi.org/10.1109/VS-GAMES.2017. 8055812 9. Goldberg, L.R.: An alternative “description of personality”: the big-five factor structure. J. Pers. Soc. Psychol. 59(6), 1216–1229 (1990) 10. Harris, J.: The Berlin interpretation. In: Exploring Roguelike Games, pp. 37–43. CRC Press, September 2020. https://doi.org/10.1201/9781003053576-9 11. Hunicke, R., LeBlanc, M., Zubek, R.: MDA: a formal approach to game design and game research. In: Proceedings of the AAAI Workshop on Challenges in Game AI, vol. 4. San Jose, CA (2004) 12. Junior, R., Silva, F.: Redefining the MDA framework—the pursuit of a game design ontology. Information 12(10) (2021). https://doi.org/10.3390/info12100395

Designing Declarative Knowledge Training Games

91

13. Kent, S.: Model driven engineering. In: Butler, M., Petre, L., Sere, K. (eds.) IFM 2002. LNCS, vol. 2335, pp. 286–298. Springer, Heidelberg (2002). https://doi.org/10.1007/3-54047884-1 16 14. Kiili, K.: Digital game-based learning: towards an experiential gaming model. Internet High. Educ. 8(1), 13–24 (2005). https://doi.org/10.1016/j.iheduc.2004.12.001 15. Kim, J.W., Ritter, F.E., Koubek, R.J.: An integrated theory for improved skill acquisition and retention in the three stages of learning. Theor. Issues Ergon. Sci. 14(1), 22–37 (2013). https://doi.org/10.1080/1464536X.2011.573008 16. Kolovos, D., Rose, L., Paige, R., Garcia-Dominguez, A.: The Epsilon Book. Eclipse (2010) 17. Laforcade, P., Mottier, E., Jolivet, S., Lemoine, B.: Expressing adaptations to take into account in generator-based exercisers: an exploratory study about multiplication facts. In: 14th International Conference CSEDU. Online Streaming, France, April 2022. https://doi. org/10.5220/0011033100003182 18. Lemoine, B., Laforcade, P., George, S.: An analysis framework for designing declarative knowledge training games using Roguelite genre. In: 15th International Conference on Computer Supported Education, vol. 2, pp. 276–287. SCITEPRESS - Science and Technology Publications, Prague, Czech Republic, April 2023. https://doi.org/10.5220/ 0011840200003470. https://hal.science/hal-04093534 19. Lemoine, B., Laforcade, P., George, S.: Mapping task types and gameplay categories in the context of declarative knowledge training. In: Proceedings of the 15th International Conference on Computer Supported Education, CSEDU 2023, vol. 2, Prague, Czech Republic, 21–23 April 2023, pp. 264–275. SCITEPRESS (2023). https://doi.org/10.5220/ 0011840100003470 20. Marne, B., Wisdom, J., Huynh-Kim-Bang, B., Labat, J.-M.: The six facets of serious game design: a methodology enhanced by our design pattern library. In: Ravenscroft, A., Lindstaedt, S., Kloos, C.D., Hern´andez-Leo, D. (eds.) EC-TEL 2012. LNCS, vol. 7563, pp. 208– 221. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33263-0 17 21. Marty, J.C., Carron, T.: Hints for improving motivation in game-based learning environments. In: Handbook of Research on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches, pp. 530–549. IGI Global (2011) 22. Nacke, L.E., Bateman, C., Mandryk, R.L.: BrainHex: a neurobiological gamer typology survey. Entertainment Comput. 5(1), 55–62 (2014). https://doi.org/10.1016/j.entcom.2013.06. 002 23. Oppermann, R., Rashev, R.: Adaptability and adaptivity in learning systems. Knowl. Transfer 2, 173–179 (1997) 24. Prensky, M.: Computer Games and Learning: Digital Game-Based Learning. Handbook of Computer Game Studies (2005) 25. Roediger, H.L., Pyc, M.A.: Inexpensive techniques to improve education: applying cognitive psychology to enhance educational practice. J. Appl. Res. Mem. Cogn. 1(4), 242–248 (2012). https://doi.org/10.1016/j.jarmac.2012.09.002 26. Schell, J.: The Art of Game Design: A Book of Lenses. CRC Press (2008) 27. Silva, F.G.M.: Practical methodology for the design of educational serious games. Information 11(1), 14 (2019). https://doi.org/10.3390/info11010014 28. Steinberg, D., Budinsky, F., Paternostro, M., Merks, E.: EMF: Eclipse Modeling Framework. Addison-Wesley Professional, Boston, Massachusetts (2008) 29. Streicher, A., Smeddinck, J.D.: Personalized and adaptive serious games. In: D¨orner, R., G¨obel, S., Kickmeier-Rust, M., Masuch, M., Zweig, K. (eds.) Entertainment Computing and Serious Games. LNCS, vol. 9970, pp. 332–377. Springer, Cham (2016). https://doi.org/10. 1007/978-3-319-46152-6 14

92

B. Lemoine et al.

30. Tondello, G.F., Wehbe, R.R., Diamond, L., Busch, M., Marczewski, A., Nacke, L.E.: The gamification user types Hexad scale. In: Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pp. 229–243. ACM, Austin, Texas, USA, October 2016. https://doi.org/10.1145/2967934.2968082 31. Toy, M., Wichman, G., Arnold, K., Lane, J.: Rogue [digital game] (1980) 32. Vandewaetere, M., Desmet, P., Clarebout, G.: The contribution of learner characteristics in the development of computer-based adaptive learning environments. Comput. Hum. Behav. 27(1), 118–130 (2011). https://doi.org/10.1016/j.chb.2010.07.038 33. Winn, B.: Handbook of Research on Effective Electronic Gaming in Education: IGI Global (2009). https://doi.org/10.4018/978-1-59904-808-6 34. Yusoff, A., Crowder, R., Gilbert, L., Wills, G.: A conceptual framework for serious games. In: 9th IEEE International Conference on Advanced Learning Technologies, pp. 21–23. Riga, Latvia, July 2009. https://doi.org/10.1109/ICALT.2009.19

Cultivating Higher Order Competencies: Complex Thinking in Latin American University Context Jorge Sanabria-Z1(B) , María Soledad Ramírez-Montoya1 , Francisco José García-Peñalvo2 , and Marco Cruz-Sandoval1 1 Institute for the Future of Education, Tecnologico de Monterrey, Monterrey, Mexico

[email protected] 2 Computer Science Department, Salamanca University, Salamanca, Spain

Abstract. In the rapidly evolving context of Education 4.0, the urgency to cultivate complex cognitive competencies is increasingly paramount, especially within the multifaceted educational landscape of Latin America. Despite this, there exists a paucity of scholarly inquiry investigating the nuanced perceptions of these competencies across various academic disciplines, genders, and nationalities within the region. To address this research gap, the present study offers an exhaustive multivariate descriptive statistical analysis, examining the perceptions of complex cognitive skills among undergraduate students in Latin America. Drawing upon a sample of 150 students from diverse Latin American nations, the investigation reveals marked disparities in the perception of complex cognitive competencies as a function of gender, academic discipline, and nationality. Specifically, male students consistently reported a higher self-assessment of their complex cognitive abilities compared to their female counterparts, a trend that was observed across multiple nations. Additionally, students enrolled in social science programs exhibited higher self-ratings of their competencies compared to those in technologyoriented disciplines, thereby underscoring the necessity for pedagogical refinements in curriculum design. Conducted under the rigorous ethical supervision of the R4C Interdisciplinary Research Group and the Institute for the Future of Education (IFE) at Tecnologico de Monterrey, this study not only provides invaluable insights for educators seeking to enhance their pedagogical approaches but also establishes a foundation for subsequent research in this critically underexplored area. Keywords: Professional education · Educational innovation · Future of education · Complex thinking · Higher education · Latin America · Descriptive statistical analysis · Interdisciplinary studies · Perception in education

1 Introduction In the scholarly domain of educational psychology and pedagogy, the evaluation of cognitive skills has historically been guided by Bloom’s taxonomy. This foundational framework has evolved, most notably under Anderson’s revision, to split into two primary © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 93–109, 2024. https://doi.org/10.1007/978-3-031-53656-4_5

94

J. Sanabria-Z et al.

dimensions: the Cognitive Process and the Knowledge dimensions (Anderson, 2005). This nuanced structure provides a fertile ground for not only measuring a spectrum of cognitive abilities but also contextualizing them within specific disciplinary frameworks, such as technology-oriented versus non-technology-oriented domains (Nkhoma., 2017; Quan et al., 2017; Poluakan et al., 2019; Chen et al., 2022). As this understanding of cognitive skills continues to evolve, it becomes imperative to apply such frameworks in diverse educational settings, particularly in responding to the challenges and opportunities of the 21st-century workforce. Building on this nuanced understanding of cognitive abilities, higher education institutions are recalibrating their curricula to better serve the needs of both society and the individual learner. As higher education globally grapples with the demands of a rapidly changing job market and societal needs, the emphasis has shifted toward nurturing more complex forms of reasoning and problem-solving (Hämäläinen et al., 2019; Shanta & Wells, 2020; Schofer et al., 2021; Rivas et al., 2022; Crichton et al., 2022). These higherorder skills are increasingly recognized as pivotal in addressing the complex challenges of the 21st century, ranging from technological advancements to social and ethical dilemmas (Alberida et al., 2022). In Latin America, these shifts assume added complexity due to the region’s unique socio-political and economic contexts (Munck et al., 2023). Thus, the application of higher-order thinking skills within the educational landscape, especially in the unique socio-political context of Latin America, is not just an academic exercise but a vital mechanism to prepare students for real-world challenges. In light of this evolving overview, the main objective of this extended article is to investigate further the varied perceptions of Latin American university students regarding their competency in complex reasoning (Morin, 1990). This cognitive skill is analyzed across various academic disciplines and is further contextualized by students’ sociodemographic profiles. By identifying potential gaps and areas of strength in complex reasoning abilities among this cohort, the study aims to inform pedagogical strategies aimed at elevating these crucial competencies. The structure of this article is methodically arranged to facilitate a comprehensive understanding of the subject matter. Following this introduction, a review of related works will survey existing literature focused on the development of higher-order competencies, with an emphasis on the Latin American educational landscape. Subsequently, the research methodology employed in the study will be delineated, providing the foundation for a presentation and analysis of the empirical results. The article concludes with an in-depth discussion, drawing implications for future research and educational practices. It is imperative to note that this article represents an enhanced version of a previous proceedings paper presented at CSEDU 2023, the International Conference on Computer Supported Education (Sanabria-Z., J., et al., 2023). This extended version is particularly enriched by the inclusion of a related works section, particularly exploring the implications of digital literacy on complex reasoning. As digital technologies permeate every facet of modern life (Sousa et al, 2020), their impact on cognitive competencies, such as complex reasoning, becomes increasingly relevant (Wang et al., 2021). The new section discusses higher-order thinking skills, their historical and psychological foundations, the impact of technology and the fourth industrial revolution, as well as the specific focus

Cultivating Higher Order Competencies

95

on complex thinking. Therefore, this inclusion warrants the extended scope and depth of the current submission, aiming to provide a more holistic perspective on the intricate terrain of complex reasoning within the Latin American higher education context.

2 Perspectives from Related Works The investigation into higher-order thinking skills can be traced back to ancient philosophical discussions but achieved marked coherence by the mid-20th century. Researchers like Newman (1990) emphasized that these skills were not merely cognitive abilities but encompassed both deep reservoirs of knowledge and a disposition for insightful problem-solving. Newman posited that these skills needed to be formally assessed to facilitate their further development. Concurrently, the rise of educational psychology added a scientific basis to this philosophical groundwork. Higher-order thinking skills are described as a form of cognition that involves intricate operations such as analysis, synthesis, problem-solving, and evaluation (Prayitno & Titikusumawati, 2018; Kwangmuang et al., 2021). As we consider the elements that constitute higher-order thinking skills, it becomes crucial to examine how contemporary educational frameworks like the revised Bloom’s Taxonomy are effectively capturing and facilitating these intricate cognitive processes. Building on the rich historical and psychological context, the revised Bloom’s Taxonomy serves as an invaluable tool for educational stakeholders, offering structured approaches to both assess and develop higher-order thinking skills in learners. The revised version of Bloom’s Taxonomy has become a pivotal framework in higher education for structuring and assessing these complex cognitive processes (Hadzhikoleva et al., 2019; Maani & Shanti, 2023). Moreover, the revision led to the evolution of new instruments and metrics to assess these skills (Lau et al., 2018; Radmehr & Drake, 2018; Hidayatullah et al. 2022; Baroudi, 2023). This remains an active area of educational research, focusing not only on the skills themselves but also on how they are taught and learned. Importantly, to cultivate these skills effectively, comprehensive approaches are necessary that take into account the perspectives of both educators and students (Murtonen & Salmento, 2019; Singh et al., 2020; Hart et al., 2021; Shanti et al., 2022). As we explore these comprehensive approaches to higher-order thinking within educational settings, it becomes essential to consider how the digital transformations of the fourth industrial revolution are affecting both the teaching and learning of these critical skills. Expanding upon the idea that comprehensive approaches are crucial for nurturing higher-order thinking skills, we turn our attention to the transformative impact of the fourth industrial revolution on educational paradigms. The era of the fourth industrial revolution has had profound implications for educational settings, crystallized in the conceptual framework of Education 4.0 (Schwab, 2017; Oliveira & de SOUZA, 2022). The integration of advanced technologies, such as Artificial Intelligence, Big Data, and Internet of Things, is fundamentally transforming the ways higher-order thinking skills are taught and learned (Yuliati & Lestari, 2018; Kim et al., 2020; Liana et al., 2020; Maulana & Syafa, 2022). Extending this line of thought, Covelli & Roy (2022) highlighted the central role of organized planning and collaborative initiatives in technologybased educational settings, where targeted workshops can be an effective strategy. A

96

J. Sanabria-Z et al.

range of pedagogical models have been developed to include technology as a tool to foster creativity and critical thinking (Swanson & Collins, 2018; Brookfield, 2020; Liang & Fung, 2020; Bereczki & Kárpáti, 2021; Airaj, 2022; Ayu et al., 2023). In both formal and informal educational settings, the advent of technologies has emerged as a focal point. A study by Sanabria-Z et al., (2022) have suggested that the proper use of technology can lead to improved civic engagement and, therefore, the development of complex, higherorder thinking. In addition to that, technology has created avenues for hybrid and online education models, making the investigation into their impact on higher-order thinking skills even more vital (Al-Samarraie et al., 2017; Xie et al., 2020; Yu et al., 2022). As the framework of higher-order thinking evolves in the context of Education 4.0, it becomes critical to explore specialized forms of cognition, such as complex thinking, to understand how they fit into this digitally augmented learning environment. As we confront the challenges and opportunities of the fourth industrial revolution, complex thinking emerges as a pivotal component of higher-order cognitive skills, offering a multifaceted approach to problem-solving and decision-making. Complex thinking is an integral component of higher-order thinking skills that emphasizes holistic, interconnected problem-solving (Ramírez-Montoya et al., 2022.a; Liu et al., 2022; SanabriaZ et al., 2023.b; Romero Rodríguez et al., 2023). Unlike linear or compartmentalized thinking, complex thinking calls for an integrated, systemic approach, as supported by research from Teixeira et al., 2020 and emphasized by the thinker Morin (2022). Complex thinking is particularly vital in our current age of rapid technological and social changes, where problems often require interdisciplinary solutions consistent with digital transformation (Patiño et al, 2023; Farias-Gaytan et al., 2023). Culhane et al. (2018) and Baena-Rojas et al. (2022) argue that the growing emphasis on interdisciplinary and multidisciplinary studies is an implicit recognition of the value of complex thinking. While the subject has received some attention globally, there remains a noticeable scarcity of research focusing on Latin America. Nonetheless, recent studies have emerged in this geographic region that allow us to commence to map the state of complex thinking in various contexts. Research on complex thinking is gaining momentum in Latin America, primarily encouraged by research from an educational perspective. Several studies indicate that complex thinking is a universal skill in higher education, supported by a validated assessment instruments, and gaining academic interest across disciplines (Loaiza et al., 2020; Vázquez-Parra et al., 2022.a; Suárez-Brito et al., 2022; Castillo-Martínez et al., 2023). Recent research expands on the universal role of complex thinking in education, linking it with digital technologies and providing measurable assessments (RamírezMontoya et al., 2022.b; Ponce et al., 2022; George-Reyes et al., 2023; Ibarra-Vazquez et al., 2023). Further studies have identified gaps and opportunities in the integration of complex thinking with social entrepreneurship among Mexican university students, while also validating tools for enhancing related competencies (Cruz-Sandoval et al., 2022; Vázquez-Parra et al., 2022.b; Vázquez-Parra et al., 2023.a). Continuing this trend, new studies are delving into the role of complex thinking in citizen science initiatives, digital game-based learning, and AI-supported educational platforms, further enriching its application and assessment in higher education (Sanabria-Z et al., 2023.b; Alfaro Ponce et al., 2023; Sanabria-Z et al., 2023.c). In another vein, further investigations

Cultivating Higher Order Competencies

97

have explored the relationship between complex thinking and gender as well as cultural education, shedding light on its role in reducing gender-based biases (Cruz-Sandoval et al., 2023.a; Vázquez-Parra et al., 2023.b; Carlos-Arroyo et al., 2023). However, there are philosophical underpinnings of complex thinking in the Latin American context that go beyond educational metrics to explore deeper issues, such as historical identity, decolonization efforts, and global influences (Silva, 2018; Déniz and Marshall, 2018; Halvorsen, 2018; Bollington & Merchant, 2020; Rosset et al., 2021; Pidkurkova, 2022; Ferrer, et al., 2023). Given the relatively unexplored status of complex thinking in Latin America, a meaningful opportunity exists to further investigate how this holistic approach to cognition can be integrated into various educational frameworks across the region.

3 Methodology This research employed a descriptive analytical approach by administering a questionnaire instrument (Loeb et al., 2017). The study was conducted by choosing a sample of convenience that consisted of 150 undergraduates from various countries in Latin America (see Table 1 for details). The sample was equally split with 75 men and 75 women from different academic fields. The data was gathered between August and December 2022. We used Google Forms to distribute a voluntary questionnaire to these students. Given the preliminary nature of this research and the participation of human respondents, the study’s methods were supervised and approved by the interdisciplinary team, R4C, with the added technical assistance from Writing Lab. Both these groups are part of the Institute for the Future of Education (IFE) at Tecnologico de Monterrey. Table 1. Sample data by country and gender. Country

Total

Male

Female

Total

150

75

75

4

2

2

48

18

30

Argentina Chile Colombia

6

3

3

Dominican Republic

6

3

3

58

35

23

Equator Guatemala Mexico

4

2

2

24

12

12

The eComplexity survey, a validated tool, was employed to evaluate participants’ perceived mastery in the area of complex thinking. This instrument is tailored to discern perceptions about the growth in complex thinking competency, covering its unique sub-competencies. With 25 distinct items, it touches on the facets of scientific, critical, innovative, and systemic thinking. Participants provided their insights by ranking

98

J. Sanabria-Z et al.

their perceived achievements using a 5-point Likert scale. A detailed breakdown of this instrument’s items can be found in the research by Cruz-Sandoval et al., 2023.b. After collecting the data, an extensive descriptive statistical examination using the R computation software (R Core Team:, 2017), and complemented with RStudio (RStudio Team, 2022) was performed. This analysis covered basic measures like the arithmetic mean and standard deviation. Graphical visuals, including boxplots and violin plots, further enriched this primary investigation. Moreover, we applied t-test analysis to determine if gender-based mean differences among Latin American university students were significant. It’s worth noting that we endeavored to enable comparisons both between countries and within the same country based on gender, wherever possible.

4 Results Figure 1 provides an initial overview of the study’s results, showcasing the distribution percentages of students’ perceptions of complex thinking competency across Latin American countries. The 10 x 10 grid visually represents these percentages, with each cell representing a 10% segment. Different color codes indicate the range of average values for competency perception. Notably, Chile (10%) and Ecuador (2%) lead in percentages of their populations that associate perception levels between 2 and 3. It’s important to note that these findings pertain to individual countries and don’t generalize to the wider population.

Fig. 1. Overview of results.

To ensure a thorough analysis we carried out a detailed examination, as showed in Table 2. This table presents the mean values and standard deviations (s) concerning the perception of complex thinking competency among Latin American students, separated by gender. Significantly, male students appear to show more advanced progress than females, as indicated by their higher mean values. Additionally, the standard deviation, which shows the spread around the mean, reveals an interesting insight: male students have a tighter dispersion (0.50) compared to females (0.61), suggesting a more consistent perception among them.

Cultivating Higher Order Competencies

99

Table 2. Complex Thinking. Mean and Standard Deviation by Gender. Statistic

Male

Female

Mean

4.02

3.80

Sd

0.50

0.61

To better visualize these findings, Fig. 2 offers a clearer depiction. This graphic sharply highlights the perception of more advanced complex thinking development among male Latin American students. Notably, their scores are closer to the mean value and show less variability compared to their female counterparts.

Fig. 2. Complex Thinking. Bar plot with standard error by gender.

Moving on to Fig. 3, a Kernel density estimation is presented through a smoothed curve. The visualization clearly reveals a higher concentration of perception levels for men around the mean values of 4 to 4.5. In contrast, the density for women is most pronounced around values between 3 and 3.5. To evaluate the presence of significant disparities in the perception of complex thinking development between men and women, a t-test was conducted. As presented in Table 3, the test results indicate clear differences (p < = 0.05) between male and female students in their self-assessed proficiency in this competency in the context of Latin America. To provide a clearer understanding of the analysis by country, Table 4 is presented. This table highlights countries that stand out for their high average values in the perception of complex thinking development, particularly the Dominican Republic, Guatemala, and Ecuador. On the other side of the spectrum, Argentina, Mexico, Chile, and Colombia are recognized for showcasing the peak perception levels regarding the development of this competency. Additionally, the table offers a deeper dive into average perception values, broken down by gender and mapped to individual countries. Aside from the Dominican Republic and Guatemala, a consistent trend is observed: male students

100

J. Sanabria-Z et al.

Fig. 3. Violin Plot. Distribution density. Kernel density. Smoothed histogram of perception in the development of complex thinking by gender. Table 3. Complex Thinking. T-test by gender. t

df

p-value

-2

143

0.02

across various countries perceive themselves as having a more developed capacity for complex thinking. Table 4. Complex Thinking by Gender. Mean values. Male

Female

Country

Mean

Mean

Argentina

3.86

3.30

Chile

4.02

3.67

Colombia

3.95

3.84

Dominican Republic

3.88

4.65

Equator

4.16

4.04

Guatemala

3.72

4.56

Mexico

3.76

3.41

Figure 4 provides a more detailed examination, graphically illustrating the range of mean perceptions and their corresponding fluctuations across countries. Both Guatemala and Ecuador exhibit comparable patterns, a trend mirrored by Colombia and Chile. However, Mexico and Argentina display more moderate mean perceptions. Notably,

Cultivating Higher Order Competencies

101

while the Dominican Republic achieves the highest mean perception, it also demonstrates the most significant variability, as evidenced by its substantial standard deviation.

Fig. 4. Complex Thinking. Mean and Standard Deviation by Country.

In an effort to gain a more profound understanding of students’ perceptions regarding complex thinking, a focused examination was undertaken using country-specific boxplot analysis (see Fig. 5). This method not only clarifies the distribution of data but also accentuates any exceptional values. Clearly, the Dominican Republic, Chile, and Ecuador display a wider range, whereas Argentina and Colombia present a more concentrated spectrum in terms of student perspectives on the advancement of complex thinking.

Fig. 5. Complex Thinking. Boxplot analysis by country.

Figure 6 advances the story by presenting a boxplot analysis categorized by gender across different countries, centered around the perception of complex thinking growth.

102

J. Sanabria-Z et al.

It’s worth highlighting that, even though female perceptions occasionally yield lower mean values, they exhibit less variability compared to their male counterparts, despite the latter holding higher mean values (e.g., Argentina and Colombia).

Fig. 6. Complex Thinking. Box plot by Country and Gender.

Figure 7 gives us a clearer picture of how students think based on their study areas. It seems that social science students feel they’re the best at complex thinking. On the other hand, engineering and technology students don’t feel as confident. Students from the arts, medicine, and natural sciences feel somewhere in between.

Fig. 7. Complex Thinking. Box Plot by discipline.

Cultivating Higher Order Competencies

103

5 Discussion and Conclusion In the context of the fast-paced shifts toward Education 4.0, our study sought to explore the perception of complex thinking competencies among Latin American undergraduate students across various disciplines and genders. Our theoretical framework was anchored in multiple streams of research, including cognitive psychology, pedagogical theories, and cultural studies, which collectively emphasize the importance of complex thinking in modern education. The results corroborate aspects of existing theories while also highlighting new patterns and insights that warrant further exploration. Building on the theoretical proposition that different disciplines cultivate unique cognitive skills, our results reveal pronounced disparities in the perceived levels of complex thinking skills between students from social sciences and those from technology-related fields. This gap may be rooted in curriculum design or pedagogical approaches that emphasize practical skills over conceptual understanding in technological disciplines. Given the growing relevance of technological literacy in Education 4.0, these findings call for a pedagogical reassessment to bridge this gap (Sanabria-Z et al., 2022). The gender disparity in perceived levels of complex thinking is noteworthy. Consistent with the literature on gender differences in educational attainment and selfperception, our data suggests that male students, on average, rate their complex thinking abilities higher than their female counterparts. This aligns with observations that cultural narratives may influence self-perception and actual performance in complex cognitive tasks. The results compel us to reexamine the historical and cultural factors that contribute to these disparities and consider them in the design of future curricula. Our study adds another layer of complexity by highlighting country-specific variations in the perception of complex thinking skills. For instance, the Dominican Republic, Guatemala, and Ecuador exhibited high average perception values, suggesting a more positive self-assessment in those nations. These findings have significant implications for educators in these countries and underscore the need for localized, context-sensitive approaches to teaching and learning. The discussion above integrates the underlying theoretical framework with the practical implications of our findings. Notably, the disparity in perceived complex thinking abilities across disciplines, genders, and countries implies a complex interplay of educational, cultural, and psychological factors. The theoretical underpinnings of our study— rooted in cognitive and pedagogical research—support the necessity of a multi-faceted approach to improving complex thinking skills, customized to these diverse factors. The primary objective of this study was to investigate the perception of complex thinking competencies among undergraduate students in Latin America, taking into account variables such as academic discipline, gender, and country of origin. The key findings indicate significant disparities in these perceptions. Most notably, male students generally rated their complex thinking skills higher than female students. Moreover, students from social sciences appear to perceive themselves as having greater complex thinking abilities compared to those from technology-based disciplines. Country-specific variations were also evident, adding another layer of complexity to the discussion. From a practical standpoint, these findings hold critical implications for educators and curriculum designers. For instance, the apparent gap in perceived complex thinking

104

J. Sanabria-Z et al.

abilities between technological and social science disciplines suggests a need for pedagogical innovation. Integrating complex thinking elements into technical courses could help mitigate this discrepancy. Educators should also be cognizant of gender-based disparities in perception, tailoring instructional strategies to be more inclusive and equally stimulating for both male and female students. On the research front, this study opens up avenues for further exploration into the intricate factors that contribute to the development and perception of complex thinking skills. Future research could focus on investigating how specific curricular interventions affect these perceptions. It could also extend into exploring how socio-cultural elements, perhaps deeply embedded in national educational systems, influence these competencies. Such research would be valuable for educators globally as they grapple with the challenges of Education 4.0. While this study provides valuable insights, it has several limitations. The sample was limited to a selection of countries and therefore does not encompass the full spectrum of Latin American educational contexts. Additionally, the broad categorization of academic disciplines and the cross-sectional design of the study limit the depth of conclusions that can be drawn. Future studies could benefit from a more diverse and comprehensive sample, deeper discipline-specific investigations, and a longitudinal design to track changes in perception over time. Acknowledgments. The authors acknowledge the financial and technical support of Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, Mexico, in producing this work and the financial support from Tecnologico de Monterrey through the “ChallengeBased Research Funding Program 2022”. Project name “OEM4C: Open Educational Model for Complex Thinking” with Fund ID # I001 - IFE001 - C1-T1 – E”.

References Anderson, L.W.: Objectives, evaluation, and the improvement of education. Stud. Educ. Eval. 31(2–3), 102–113 (2005). https://doi.org/10.1016/j.stueduc.2005.05.004 Nkhoma, M., Lam, T., Sriratanaviriyakul, N., Richardson, J., Kam, B., Lau, K.: Unpacking the revised bloom’s taxonomy: developing case-based learning activities. J. Educ. Train. 59, 250– 264 (2017). https://doi.org/10.1108/ET-03-2016-0061 Quan, J., Luo, C., Yang, F., Qiu, H.: Bloom’s taxonomy of educational objectives in information system courses. DEStech Trans. Soc. Sci., Educ. Hum. Sci. (2017). https://doi.org/10.12783/ dtssehs/mess2016/9623 Poluakan, C., Tilaar, A. L., Tuerah, P., Mondolang, A.: Implementation of the revised bloom taxonomy in assessment of physics learning. In: Proceedings of the 1st International Conference on Education, Science, and Technology (ICESTech), pp. not specified (2019) Chen, K., Cheng, X., Zhang, R., Cillo, G., Ragusa, A.: Unveiling the role of cross-cultural and cognitive differences in organizational learning mechanism of technology-acquiring crossborder mergers and acquisitions: evidence from emerging market enterprises. Front. Psychol. 13 (2022). https://doi.org/10.3389/fpsyg.2022.863442 Hämäläinen, R., Wever, B., Nissinen, K., Cincinnato, S.: What makes the difference - PIAAC as a resource for understanding the problem-solving skills of Europe’s higher-education adults. Comput. Educ. 129, 27–36 (2019). https://doi.org/10.1016/J.COMPEDU.2018.10.013

Cultivating Higher Order Competencies

105

Shanta, S., Wells, J.: T/E design based learning: assessing student critical thinking and problem solving abilities. Int. J. Technol. Des. Educ. 32, 267–285 (2020). https://doi.org/10.1007/s10 798-020-09608-8 Schofer, E., Ramirez, F., Meyer, J.: The societal consequences of higher education. Sociol. Educ. 94, 1–19 (2021). https://doi.org/10.1177/0038040720942912 Rivas, S., Saiz, C., Ossa, C.: Metacognitive strategies and development of critical thinking in higher education. Front. Psychol. 13 (2022). https://doi.org/10.3389/fpsyg.2022.913219 Crichton, M., Crichton, H., Colville, G.: Students’ perceptions of problem-based learning in multidisciplinary groups when seeking to solve an engineering grand challenge. J. Probl. Based Learn. High. Educ. 10(1) (2022). https://doi.org/10.54337/ojs.jpblhe.v10i1.6823 Alberida, H., Sari, M., Razak, A., Syamsuriza, S., Rahmi, Y.: Problem solving: a learning model to foster argumentation and critical thinking ability for students with different academic abilities. J. Penelit. Pendidik. IPA 8(3) (2022). https://doi.org/10.29303/jppipa.v8i3.1208 Munck, R., Pinilla, Y., Hodges, R.A., Bartch, C.: Higher education in Latin America and the Caribbean: civic engagement and the democratic mission. Machdohnil Limited (2023). ISBN: 978–1–908689–45–0. (https://bit.ly/3B78Xj4), [Repository] (http://dspaceudual.org/handle/ Rep-UDUAL/1904) Morin, E.: Introduction to complex thought. Le Seuil (1990) Sanabria-Z., J., Ramírez-Montoya, M.S., García-Peñalvo, F.J., Cruz-Sandoval, M.: Complex thinking in interdisciplinarity: an exploratory study in Latin American population. In: Proceedings of the 15th International Conference on Computer Supported Education, vol. 2, pp. 288–295 (2023a). https://hdl.handle.net/11285/650683 Sousa, R., Karimova, B., Gorlov, S.: Digitalization as a new direction in education sphere. In: E3S Web of Conferences (2020). https://doi.org/10.1051/e3sconf/202015909014 Wang, S., et al.: From LSAT: the progress and challenges of complex reasoning. IEEE/ACM Trans. Audio, Speech, Lang. Process. 30, 2201–2216 (2021). https://doi.org/10.1109/taslp.2022.316 4218 Newman, F.M.: Higher order thinking in teaching social studies: a rationale for the assessment of classroom thoughtfulness. J. Curric. Stud. 22, 41–56 (1990) Prayitno, B., S., Titikusumawati, E.: Enhancing students’ higher order thinking skills in science through INSTAD strategy. J. Baltic Sci. Educ. (2018). https://doi.org/10.33225/JBSE/18.17. 1046 Kwangmuang, P., Jarutkamolpong, S., Sangboonraung, W., Daungtod, S.: The development of learning innovation to enhance higher order thinking skills for students in Thailand junior high schools. Heliyon 7 (2021). https://doi.org/10.1016/j.heliyon.2021.e07309 Hadzhikoleva, S., Hadzhikolev, E., Kasakliev, N.: Using peer assessment to enhance higher order thinking skills. TEM J. 8(1), 242–247 (2019). https://doi.org/10.18421/TEM81-34.ISSN:22178309 Maani, D., Shanti, Z.: Technology-enhanced learning in light of bloom’s taxonomy: a studentexperience study of the history of architecture course. Sustainability (2023). https://doi.org/ 10.3390/su15032624 Lau, K., Lam, T., Kam, B., Nkhoma, M., Richardson, J.: Benchmarking higher education programs through alignment analysis based on the revised bloom’s taxonomy. Benchmark.: Int. J. (2018). https://doi.org/10.1108/BIJ-10-2017-0286 Radmehr, F., Drake, M.: An assessment-based model for exploring the solving of mathematical problems: utilizing revised bloom’s taxonomy and facets of metacognition. Stud. Educ. Eval. (2018). https://doi.org/10.1016/J.STUEDUC.2018.02.004 Hidayatullah, A., Yamtinah, S., Masykuri, M.: Development of a two-tier multiple-choice instrument based on higher order thinking skills (HOTS) on acids, bases, and salts. J. Penelit. Pendidik. IPA (2022). https://doi.org/10.29303/jppipa.v8i2.1423

106

J. Sanabria-Z et al.

Baroudi, S.: Exploring teacher education for sustainable development in the UAE. Sustainability (2023). https://doi.org/10.3390/su15031981 Murtonen, M., Salmento, H.: Broadening the theory of scientific thinking for higher education. Redefin. Sci. Think. High. Educ. (2019). https://doi.org/10.1007/978-3-030-24215-2_1 Singh, C., Gopal, R., Ong, E., Singh, T., Mostafa, N., Singh, R.: ESL teachers’ strategies to foster higher-order thinking skills to teach writing. Malays. J. Learn. Instr. (2020). https://doi.org/10. 32890/MJLI2020.17.2.7 Hart, C., Costa, C., D’Souza, D., Kimpton, A., Ljbusic, J.: Exploring higher education students’ critical thinking skills through content analysis. Think. Skills Creat. 41, 100877 (2021). https:// doi.org/10.1016/J.TSC.2021.100877 Shanti, M., Istiyono, E., Munadi, S.: The effectiveness of learning to improve students higher-order thinking skills. Cypriot. J. Educ. Sci. (2022). https://doi.org/10.18844/cjes.v17i8.7780 Schwab, K.: The Fourth Industrial Revolution. Crown Publishing Group, New York (2017) Oliveira, K.K.D.S., de SOUZA, R.A.: Digital transformation towards education 4.0. Inform. Educ. 21(2), 283–309 (2022) Yuliati, S., Lestari, I.: Higher-order thinking skills (HOTS) analysis of students in solving HOTS question in higher education. Perspektif Ilmu Pendidikan 32, 181-188 (2018). https://doi.org/ 10.21009/PIP.322.10 Kim, H., Yi, P., Hong, J.: Students’ academic use of mobile technology and higher-order thinking skills: the role of active engagement. Educ. Sci. (2020). https://doi.org/10.3390/educsci10 030047 Liana, Y., Linuwih, S., Sulhadi, S.: Internet of things based learning media with problem solving approach: its effect on higher order thinking skills. JIPFALBIRUNI 9, 225–239 (2020). https:// doi.org/10.24042/JIPFALBIRUNI.V9I2.6313 Maulana, R., Syafa, S.: The development of hypermedia learning media in an effort to improve higher order thinking skills students in high school economics class X subjects. PEKA 10, 34–44 (2022). https://doi.org/10.25299/peka.2022.vol10(2).10992 Covelli, B., Roy, S.: Emergency remote teaching versus planned remote teaching: narrowing the gap with targeted professional development. High. Learn. Res. Commun. 12(2), 74–91 (2022). https://doi.org/10.18870/hlrc.v12i2.1340 Swanson, H., Collins, A.: How failure is productive in the creative process: refining student explanations through theory-building discussion. Think. Skills Creat. (2018). https://doi.org/ 10.1016/J.TSC.2018.03.005 Brookfield, S.: Teaching for critical thinking. Int. J. Adult Vocat. Educ. Technol. 4, 1–15 (2020). https://doi.org/10.4018/javet.2013010101 Liang, W., Fung, D.: Development and evaluation of a WebQuest-based teaching programme: students’ use of exploratory talk to exercise critical thinking. Int. J. Educ. Res. 104, 101652 (2020). https://doi.org/10.1016/j.ijer.2020.101652 Bereczki, E., Kárpáti, A.: Technology-enhanced creativity: a multiple case study of digital technology-integration expert teachers’ beliefs and practices. Think. Skills Creat. 39, 100791 (2021). https://doi.org/10.1016/J.TSC.2021.100791 Cloud Computing technology and PBL teaching approach for a qualitative education in line with SDG4. Sustainability (2022). https://doi.org/10.3390/su142315766 Ayu, H., Saputro, S., Sarwanto, S., Mulyani, S.: Reshaping technology-based projects and their exploration of creativity. Eurasia J. Math. Sci. Technol. Educ. (2023). https://doi.org/10.29333/ ejmste/12814 Sanabria-Z, J., Alfaro-Ponce, B., González Peña, O.I., Terashima-Marín, H., Ortiz-Bayliss, J.C.: Engagement and social impact in tech-based citizen science initiatives for achieving the SDGs: a systematic literature review with a perspective on complex thinking. Sustainability 14(17), 1–22 (2022). https://doi.org/10.3390/su141710978

Cultivating Higher Order Competencies

107

Su, Y., Cheng, H., Lai, C.: Study of Virtual reality immersive technology enhanced mathematics geometry learning. Front. Psychol. 13 (2022). https://doi.org/10.3389/fpsyg.2022.760418 Al-Samarraie, H., Teng, B., Alzahrani, A., Alalwan, N.: E-learning continuance satisfaction in higher education: a unified perspective from instructors and students. Stud. High. Educ. 44(11), 1946–1964 (2017) Xie, X., Siau, K., Nah, F.: COVID-19 pandemic – online education in the new normal and the next normal. J. Inf. Technol. Case Appl. Res. 22, 175–187 (2020). https://doi.org/10.1080/152 28053.2020.1824884 Yu, Z., Xu, W., Yu, L.: Constructing an online sustainable educational model in COVID-19 pandemic environments. Sustainability (2022). https://doi.org/10.3390/su14063598 Ramírez-Montoya, M.S., Castillo-Martínez, I.M., Sanabria-Z, J., Miranda, J.: Complex thinking in the framework of education 4.0 and open innovation—a systematic literature review. J. Open Innov. Technol. Mark. Complex. 8, 4 (2022a). https://doi.org/10.3390/joitmc8010004 Liu, J., et al.: A systematic review of higher-order thinking by visualizing its structure through HistCite and CiteSpace software. Asia-Pacific Educ. Res. 31, 635–645 (2022). https://doi.org/ 10.1007/s40299-021-00614-5 Sanabria-Z, J., Cruz-Sandoval, M., Moreno-Romo, A., Bosh Gómez, S., Ramírez-Montoya, M.S.: Research foresight in bridging open science and open innovation: overview based on the complex thinking paradigm. Int. J. Innov. Stud. (2023b). https://doi.org/10.1016/j.ijis.2023. 08.002. Retrieved from: https://hdl.handle.net/11285/651088 Romero Rodríguez, J.M., Ramírez-Montoya, M.S., Buenestado-Fernández, M., Lara Lara, F.: Use of ChatGPT at university as a tool for complex thinking: students’ perceived usefulness. J. New Approaches Educ. Res. 12(2), 323–339 (2023). https://doi.org/10.7821/naer.2023.7. 1458. Retrieved from: https://hdl.handle.net/11285/651043 Teixeira de Melo, A., Caves, L.S.D., Dewitt, A., Clutton, E., Macpherson, R., Garnett, P.: Thinking (in) Complexity: (in) Definitions and (mis)Conceptions. Syst. Res. Behav. Sci. 37(1), 154–169 (2020). https://doi.org/10.1002/sres.2612 Morin, E.: Lessons from a century of life. In: Spanish: Paidós, Argentina (2022). ISBN-10, 8449339316; ISBN-13, 978–8449339318 Patiño, A., Ramírez-Montoya, M.S., Ibarra-Vazquez, G.: Trends and research outcomes of technology-based interventions for complex thinking development in higher education: a review of scientific publications. Contemp. Educ. Technol. 15(4), ep447 (2023). https://doi. org/10.30935/cedtech/13416 Farias-Gaytan, S., Aguaded, I., Ramirez-Montoya, M.S.: Digital transformation and digital literacy in the context of complexity within higher education institutions: a systematic literature review. Humanit. Soc. Sci. Commun. 10, 386 (2023). https://doi.org/10.1057/s41599-023-01875-9 Culhane, J., Niewolny, K., Clark, S., Misyak, S.: Exploring the intersections of interdisciplinary teaching, experiential learning, and community engagement: a case study of service learning in practice. Int. J. Teach. Learn. High. Educ. 30(3), 412–422 (2018) Baena-Rojas, J.J., Ramírez-Montoya, M.S., Mazo-Cuervo, D.M., López-Caudana, E.O.: Traits of complex thinking: a Bibliometric review of a disruptive construct in education. J. Intell. 10(37) (2022). https://doi.org/10.3390/jintelligence10030037 Loaiza Zuluaga, Y.E., Gil Duque, G.M., David Narváez, F.A.: A study of critical thinking in higher education students. Latinoamericana de Estudios Educativos 16(2), 256–279 (2020). https:// doi.org/10.17151/rlee.2020.16.2.13 Vázquez-Parra, J.C., Castillo-Martínez, I.M., Ramírez-Montoya, M.S., Millán, A.: Development of the perception of achievement of complex thinking: a disciplinary approach in a Latin American student population. Educ. Sci. 12 (2022a). Art. 289. https://doi.org/10.3390/educsc i12050289. Retrieved from: https://hdl.handle.net/11285/648187

108

J. Sanabria-Z et al.

Suárez-Brito, P., Baena-Rojas, J., López-Caudana, E., Glasserman-Morales, L.: Academic literacy as a component of complex thinking in higher education: a scoping review. Eur. J. Contemp. Educ. 11(3), 931–945 (2022). https://doi.org/10.13187/ejced.2022.3.931. Retrieved from: https://hdl.handle.net/11285/649750 Castillo-Martínez, I.M., Ramírez-Montoya, M.S., Glasserman-Morales, L.D., Millán-Arellano, A.: eComplex: validity and reliability of rubric for assessing reasoning for complexity competence. Qual. Quant. (2023). https://doi.org/10.1007/s11135-023-01697-6. Retrieved from: https://repositorio.tec.mx/handle/11285/650956 Ramírez-Montoya, M.S., McGreal, R., Obiageli Agbu, J.F.: Horizontes Digitales Complejos en el Futuro de la Educación 4.0: Luces desde las Recomendaciones de UNESCO. RIED-Revista Iberoamericana de Educación a Distancia 25(2) (2022b). https://doi.org/ https://doi.org/ 10.5944/ried.25.2.33843. Retrieved from: https://repositorio.tec.mx/handle/11285/648468, https://repositorio.tec.mx/handle/11285/648469 Ponce, P., Ramirez, R., Ramirez-Montoya, M.S., Molina, A., MacCleery, B., Ascanio, M.: From understanding a simple DC motor to developing an electric vehicle AI controller rapid prototype using MATLAB-Simulink, real-time simulation and complex thinking. Front. Educ. 7, 941972 (2022). https://doi.org/10.3389/feduc.2022.941972. Retrieved from: https://hdl.han dle.net/11285/649925 George-Reyes, C.E., Ramírez-Montoya, M.S., López-Caudana, E.O.: Imbrication of the Metaverse in the complexity of education 4.0: approach from an analysis of the literature. Pixel-Bit. Revista de Medios y Educación 66, 199–237 (2023). https://doi.org/10.12795/pixelbit.97337. Retrieved from: https://hdl.handle.net/11285/650030, https://hdl.handle.net/11285/650031 Ibarra-Vazquez, G., Ramírez-Montoya, M.S., Terashima, H.: Gender prediction through complex thinking competence using machine learning. Educ. Inf. Technol. (2023). https://doi.org/10. 1007/s10639-023-11831-4. Retrieved from: https://repositorio.tec.mx/handle/11285/650866 Cruz-Sandoval, M., Vázquez-Parra, J.C., Alonso-Galicia, P.: Student perception of competencies and skills for social entrepreneurship in complex environments: an approach with Mexican university students. Soc. Sci. 11(7) (2022). Art. 314. https://doi.org/10.3390/socsci11070314. Retrieved from: https://repositorio.tec.mx/handle/11285/648721 Vázquez-Parra, J.C., Cruz-Sandoval, M., Carlos-Arroyo, M.: Social entrepreneurship and complex thinking: a Bibliometric study. Sustainability 14(20) (2022b). https://doi.org/10.3390/su1420 13187. Retrieved from: https://hdl.handle.net/11285/649765 Vázquez-Parra, J.C., Carlos-Arroyo, M., Cruz-Sandoval, M.: Social entrepreneurship and complex thinking: validation of methodology for the scaling of the perception of competence achievement. Educ. Sci. 13(2) (2023a). https://doi.org/10.3390/educsci13020186. Retrieved from: https://hdl.handle.net/11285/650172 Sanabria-Z, J.C., Castillo-Martínez, I.M., González-Pérez, L.I., Ramírez-Montoya, M.S.: Complex thinking through a transition design-guided Ideathon: testing an AI platform on the topic of sharing economy. Front. Educ. 8, 1186731 (2023b). https://doi.org/10.3389/feduc.2023.118 6731. Retrieved from: https://hdl.handle.net/11285/650827 Alfaro Ponce, B., Patiño, A., Sanabria-Z, J.: Components of computational thinking in citizen science games and its contribution to reasoning for complexity through digital game based learning: a framework proposal. Cogent Educ. 10(1), 2191751 (2023). https://doi.org/10.1080/ 2331186X.2023.2191751. Retrieved from: https://hdl.handle.net/11285/650836 Cruz-Sandoval, M., Vázquez-Parra, J.C., Carlos-Arroyo, M., del Angel-González, M.V.: Complex thinking and its relevance in professional training: an approach to engineering students in a Mexican university. Int. J. Eng. Pedagogy (iJEP) 13(3), 100–119 (2023). https://doi.org/10. 3991/ijep.v13i3.36885 Vázquez-Parra, J.C., Alfaro-Ponce, B., Guerrero-Escamilla, B., Morales-Maure, L.: Cultural Imaginaries and Complex Thinking. Impact of cultural training on the development of perceived

Cultivating Higher Order Competencies

109

achievement of complex thinking in college students. Soc. Sci. 12(5), 272 (2023b). https://doi. org/10.3390/socsci12050272 Carlos-Arroyo, M., Vázquez-Parra, J.C., Cruz-Sandoval, M., Echaniz-Barrondo, A.: Male chauvinism and complex thinking: a study of Mexican university students. Societies 13, 104 (2023). https://doi.org/10.3390/soc13050104 Silva, J.: Utopías y distopías de nuestra historia: Aproximación historiográfica a lo latinoamericano en el pensamiento social mexicano del siglo XX: Edmundo O’Gorman, Guillermo Bonfil Batalla y Leopoldo Zea. 11 (2018). https://doi.org/10.15848/HH.V11I28.1232 Déniz, J., Marshall, W.: Latin American structuralism and current financialization. Int. J. Polit. Econ. 47, 48–68 (2018). https://doi.org/10.1080/08911916.2018.1449604 Halvorsen, S.: Cartographies of epistemic expropriation: critical reflections on learning from the South. Geoforum (2018). https://doi.org/10.1016/J.GEOFORUM.2018.06.018 Bollington, L., Merchant, P.: Latin American culture and the limits of the human (2020). https:// doi.org/10.2307/j.ctvzgb67d Rosset, P., Barbosa, L., Val, V., McCune, N.: Critical Latin American agroecology as a regionalism from below. Globalizations 19, 635–652 (2021). https://doi.org/10.1080/14747731.2021.192 3353 Pidkurkova, I.: Factor of identity in the process of formation of the Latin American region. The Bulletin of Yaroslav Mudryi National Law University. Series: Philosophy, philosophy of law, political science, sociology (2022). https://doi.org/10.21564/2663-5704.53.259394 Ferrer, H.: Living theology in a pluralistic Latin America: an exploration of ecclesial base communities through the lens of social imaginaries. Religions 14(2) (2023). https://doi.org/10.3390/ rel14020259 Loeb, S., Dynarski, S., McFarland, D., Morris, P., Reardon, S., Reber, S.: Descriptive analysis in education: a guide for researchers. National center for education evaluation and regional assistance (2017). Available from: ED Pubs, P.O. Box 1398, Jessup, MD 20794–1398. Tel: 877–433–7827. Web site: https://ies.ed.gov/ncee/.NCEE2017-4023 Cruz-Sandoval, M., Vázquez-Parra, J.C., CarlosArroyo, M.: Complex thinking and social entrepreneurship: an approach from the methodology of compositional data analysis. HELIYON (2023b). https://doi.org/10.1016/j.heliyon.2023.e13415 R Core Team: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing (2017). https://www.r-project.org/ RStudio Team: RStudio: Integrated Development for R (2022.2.2.485). RStudio, PBC (2022). http://www.rstudio.com/

Correlation Among Competences Involved in Digital Problem-Solving Activities with Upper Secondary School Students Alice Barana , Cecilia Fissore(B)

, Anna Lepre , and Marina Marchisio

Department of Molecular Biotechnology and Health Sciences, University of Turin, Via Nizza 52, 10126 Turin, Italy {alice.barana,cecilia.fissore,anna.lepre, marina.marchisio}@unito.it

Abstract. In today’s increasingly interconnected and technology-driven world, developing strong problem-solving and digital competences is the key to preparing students for a world where technology impacts society, work and education, and to achieving personal and professional fulfilment. The main aim of this paper is to investigate the correlation and the development of problem-solving and digital competences of secondary school students solving problems with an Advanced Computing Environment (ACE). During an online training, students solve a contextualized problem every ten days, collaborating asynchronously with other students. Each solution is evaluated using an evaluation rubric based on five indicators: comprehension, identification of a solution strategy, development of the solution process, argumentation, use of an ACE. The research questions are: (RQ1) “How students develop problem-solving and digital competences during the online training?” (RQ2) “How do the development of problem-solving competence (in its four indicators: comprehension, identification of a solution strategy, development of the solution process, argumentation) and the development of digital competence relate to each other during the online training?”. To answer the research questions, the solutions of 158 grade 12 students to ten problems were analysed. The research methodology consists of three phases: analysis of two case studies; analysis of all students’ grades; analysis of student responses to a final questionnaire. The analyses were carried out considering both the overall evaluations and the evaluations of each indicator. The correlations between the different indicators were also analysed, seeking confirmation of this evolution in the submissions of some exemplary case studies. The results show that solving contextualized problems with the ACE improved students’ problem-solving and digital competences. The results also show a strong correlation between identifying a solving strategy and developing the solving process, and between these two indicators and the effective use of the ACE. Keywords: Advanced computing environment · Digital competence · Mathematics education · Problem solving · Problem-solving competences

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 110–135, 2024. https://doi.org/10.1007/978-3-031-53656-4_6

Correlation Among Competences Involved in Digital Problem

111

1 Introduction This paper presents the extension of an initial research on how to evaluate the development of digital and problem-solving competences [1]. In this study, we have extended the analysis of the development of digital and problem-solving competences by examining a second significant and exemplary case study and by focusing on the correlations between problem-solving and digital competences. We focused on the indicators that characterize problem-solving and digital competences, analyzing their correlations in specific cases and over time. Developing competences that can be used throughout life is an essential need of each individual to respond to the challenges of a world in which technologies influence society, teaching and education, to improve as a person and as a worker, and to foster active citizenship. In the recommendations on the key competences for lifelong learning, the Council of the European Union includes the problem-solving competence and the digital competence [2]. According to the European Parliament and Council [2]: “Competences, such as problem solving, critical thinking, ability to cooperate, creativity, computational thinking, self-regulation, are more essential than ever before in our quickly changing society. They are the tools to make what has been learned work in real time, in order to generate new ideas, new theories, new products, and new knowledge”. The digital competence involves the confident, critical, and responsible use of, and engagement with, digital technologies for learning, at work, and for participation in society. These aspects are also mentioned in the Italian National Guidelines [3], which states that students, at the end of upper secondary school, should be able to apply mathematical concepts to solve problems, also with the help of technologies. Therefore, proposing problemsolving activities with the use of digital technologies is a teaching methodology that responds to institutional objectives. The context of our research is the Digital Math Training (DMT) project, which involves about 3000 Italian upper secondary school students each year. Its main goal is to allow students to develop digital and problem-solving competences by solving contextualized problems with an Advanced Computing Environment (ACE) and collaborating with each other remotely within an integrated Digital Learning Environment (DLE)1 [4, 5]. An ACE, with a special programming language, allows for performing numerical and symbolic computations, plotting two- and three-dimensional static or dynamic graphs and programming interactive components to generalize a solving process. An ACE also allows students to approach a problematic situation in the way that best suits their thinking, to use different kinds of representations in line with the chosen strategy and to display the whole reasoning together with verbal explanation in the same page [6]. Thanks to these potentialities, it is an effective tool to support problem solving and mathematics teaching and learning [7, 8]. A DLE is a “technical solution for supporting learning, teaching and studying activities” [9]. It has been defined by Barana and Marchisio [10] as an ecosystem in which teaching, learning, and the development of competence are fostered in classroom-based, online, hybrid, or blended settings. It is made up of a human component, a technological component, and the interrelations between the two [10]. 1 https://digitalmatetraining.i-learn.unito.it/

112

A. Barana et al.

The main goal of this study is analyzing the development of problem-solving and digital competences of the participants in the DMT project. To meet the research goals, we focused on the 158 12th grade students who participated in the 2021/2022 edition of the project. Their submissions, assessment data, and answers to the final questionnaire were collected and analyzed. This paper is structured as follows. In Sect. 2, the methodology of problem solving and problem solving with an ACE are discussed, followed by a brief presentation of the DMT project. In Sect. 3, the research methodology with which the analysis was carried out is presented. The main results obtained from the analyses are presented in Sect. 4. In Sect. 5 some reflections on the results obtained and possible further developments for the research are presented.

2 Theoretical Background 2.1 Problem Solving with an ACE One of the fundamental competences in Mathematics is the ability to solve problems in real-life situations, which includes the ability to understand the problem, devise a mathematical model, develop the solving process and interpret the obtained solution [11]. The term “problem solving” refers to mathematical tasks which provide intellectual challenges that improve students’ understanding and mathematical development [12]. Problem solving is a real challenge for students. It involves the use of multiple rules, notions, and operations whose choice is a strategic and creative act of the students [13]. Its value lies not only in being able to find the final solution but also in developing ideas, strategies, competences, and attitudes. The focus then shifts from the final solution to the process. Solving problems contextualized in relevant real-world situations activates modeling skills in students and teaches them to recognize how and when to use their knowledge, as well as getting them accustomed to solving problems in real world situations [11, 14]. Teachers should select challenging problems whose mathematical content has already been studied or will soon be; data should be open to offer students a vast range of possibilities to choose from and make decisions about; the tasks should be open to different approaches [10]. Through problem solving it is also possible to develop social and civic competences. For example, by solving problems in small groups, students learn to work in team, to discuss, to support their own opinions and respect those of others, to discuss and present their ideas. Therefore, through problem-solving activities in Mathematics, students acquire ways of thinking, creativity, curiosity, collaborative competences and confidence in unfamiliar situations [6]. Among the best tools to assess problem solving, we can include assessment rubrics. They base the assessment on selected indicators with score scales. Descriptors of the score scales are provided, which explain the reason why a performance was placed in a certain level [15]. Using rubrics allow to compare evaluations of different problems, since the same rubric can be used to assess problems on different mathematical topics. Moreover, through rubric assessment, students are provided with relevant feedback on the problem-solving process, since they receive an evaluation on each indicator [16]. The next level guidance provides students with an idea of what should be achieved and

Correlation Among Competences Involved in Digital Problem

113

what needs to be done to improve. Sharing rubrics with detailed descriptors of the levels is a relevant formative assessment strategy, since it helps students understand the quality criteria [17]. In fact, through rubric assessment they can understand their actual level, the reference level, and in which area they should work more to reach the goals: these are the three main processes of formative assessment identified by Black and Wiliam [17] and by Hattie and Timperley [18]. Thus, feedback provided through rubrics can help them bridge the gap between current and desired performance in problem solving [18]. When students solve a problem, they need not only to apply and repeat knowledge but also formulate new hypotheses. In the field of problem solving, among numerous other studies, the contribution of the Polya’s work “How to solve it” [19] was very important. In this work the importance of problem in mathematical activity is emphasized by pointing out four main phases of solving them: understanding the problem, devising a plan, carrying out the plan, looking back. The looking back phase consists of reviewing and reconsidering the results obtained and the process that led to them. This allows one to consolidate knowledge, better understand the solution and possibly use the result, or the method, for other problems. Generalizing is an important process by which the specifics of a solution are examined and questions such as why it worked are investigated [20]. This process can be compared to the Polya looking back phase and consists of a verification and elaboration stage of invention and creativity. This makes it possible to move from the single case to all possible cases, to extend and readapt the model developed and to consolidate what has been learned through problem solving [21]. Technologies such as ACEs play a fundamental role in problem solving and make it possible to amplify all phases of the process. An important aspect of an ACE for problem solving is the design and programming of interactive components (such as sliders, buttons, checkboxes, text areas, tables, and graphics). They enable to visualize how the results change when the input parameters are changed and, thus, they allow to generalize the solving process of a problem. The use of an ACE for problem solving profoundly affects the entire problem-solving practice and the nature of the problems that can be posed. For example, problems may require difficult pen-and-paper calculations, dynamic explorations, algorithmic solutions to approximate results, and much more. Without having to engage in calculations, students can focus on understanding, exploring, and discussing the solving process and the obtained results. The possibility of combining different types of representation in the same worksheet influences the way students approach problems and their strategic choices, favoring high levels of clarity and understanding [22]. In this way, the ACE is not only a tool, but it becomes an effective methodology that can support problem solving and the learning of Mathematics [6]. Another technology that can enhance problem-solving activities is a DLE. In a DLE, teachers can propose many different types of activities in a single shared environment, in order to make teachers and students interact among them and with the technologies. This aspect is essential in an online teaching context, but it can also enrich the teaching experience in classroom-based, blended or hybrid mode. In a DLE students can create, share, and compare their own work and always keep in touch with each other, exchanging opinions and ideas [10].

114

A. Barana et al.

2.2 The Digital Math Training Project The DMT project was born in 2014 with the aim of developing and strengthening the mathematical, digital, and problem-solving competences of secondary school students. The main part of the project consists of an online training in a DLE. The technological component of the DLE is a Moodle platform integrated with the Maple ACE (https:// www.maplesoft.com/), developed by the Computer Science Department of the University of Turin. The activities of the DMT project are mainly based on the resolution, with the use of an ACE, of non-routine problems, contextualized in the real life, and open to different solving strategies. Students solve the problem individually, collaborating asynchronously online with their peers. Students are also offered training activities and tools that enable self-learning and collaborative learning to understand how to use an ACE and solve problems. The project is open to students from grade 9 to grade 13. The students are divided by grade, then five online courses are designed and set up on the platform. During the online training, a problem is proposed to the students every ten days, for a total of 8 problems. The degree of difficulty of the problems gradually increases during the training. Increasing the difficulty of the problems allows students to be prepared for a final competition. Prizes are assigned to the best performers. All problems include several requests: the first tasks guide students to understand, explore, identify a model and the solving strategy of the problematic situation, the last task requires a generalization of the solution through the creation and programming of interactive components. The problems’ solutions worked out by the students are assessed by tutors according to a rubric designed to evaluate the competences in problem-solving while using an ACE. The rubric is an adaptation of the one proposed by the Italian Ministry of Education to assess the national written exam in Mathematics at the end of Scientific Lyceum, developed by experts in pedagogy and assessment. The rubric has 5 indicators, each of which can be graded with a level from 1 to 4. The first four indicators have been drawn from Polya’s model and refer to the four phases of problem solving; they are the same included in the ministerial rubric. The project’s adaptation mainly involves the fifth indicator, and entails the use of the ACE, which we chose to separate from the other indicators in order to have and be able to provide students with precise information about how the ACE was used to solve the problem. Since the objective of the DMT project is developing problem solving with technologies, it has been considered appropriate to evaluate the improvements also in the use of the ACE in relation to the problem to solve [4, 22]. The five indicators are the following: • comprehension: Analyze the problematic situation, represent, and interpret the data and then turn them into mathematical language (score between 0 and 18); • identification of a solving strategy: Employ solving strategies by modeling the problem and by using the most appropriate strategy (score between 0 and 21); • development of the solving process: Solve the problematic situation consistently, completely, and correctly by applying mathematical rules and by performing the necessary calculations (score between 0 and 21); • argumentation: Explain and comment on the chosen strategy, the key steps of the building process and the consistency of the results (score between 0 and 15); • use of an ACE: Use the ACE commands appropriately and effectively in order to solve the problem (score between 0 and 25).

Correlation Among Competences Involved in Digital Problem

115

A total score (up to 100) is given to each resolution. Finally, each evaluation is integrated with personalized feedback from the tutors related to each indicator, containing advice on how and what to improve. At the end of the training, all participants are asked to fill out a satisfaction questionnaire.

3 Research Questions The aim of this study is to investigate how students’ digital and problem-solving competences change during online training in the DLE. It should be noted that students belong to different schools in different cities and may start the online training with different levels of competence. There may be students who are used to solving problems and using technology, as well as students who have never done problem solving activities or used technology. When solving problems with ACE, these competences are complementary and can influence each other. For example, there may be students who are very good at problem solving but less good at using ACE and vice versa. In order to study the development of these competences in students, we formulated two research questions: • RQ1: How students develop problem-solving and digital competences during the online training? • RQ2: How do the development of problem-solving competence (in its four indicators: comprehension, identification of a solution strategy, development of the solution process, argumentation) and the development of digital competence relate to each other during the online training?

4 Methodology To answer the research questions, the activities of 12th grade participants in the 2021/2022 DMT edition were analyzed. The submissions and all data relating to the assessments obtained by 158 students during the online training were collected from the DLE. The analysis consisted of three phases: • the analysis of how digital and problem-solving competences evolved during the training in two exemplary case studies; • the analysis of all student evaluations from the beginning to the end of the online training; • the analysis of the students’ answers to the final questionnaire. The term “case studies” refers to the learning trajectories of individual learners during online training. The case studies have been studied and proposed with the aim of examining the development of problem solving and digital competences during the online training in some significant and exemplary cases. The average number of submissions was 92 in the first half of the training (first four problems) and 49 in the second half (last four problems). The data collected were organized in a table containing, for each student and for each of the 8 problems, the evaluations relating to the five indicators of the assessment rubric and the total score. We used the scores in the first four indicators to identify the level of problem-solving competence, and the fifth indicator to describe the achieved level of the digital competence.

116

A. Barana et al.

The table was important both for the analysis of the case studies and for the analysis of student assessments (the trend of total grades and the trend of grades of individual indicators). In order to select the case studies, the trend of the total scores obtained by each student for each problem was analysed. Students who showed a general improvement were selected: specifically, students with a low (below 50) or medium (below 70) initial score and a high final score (above 70). Using this method, 6 case studies were selected. All the resolutions provided by these students were analysed in detail in order to select 2 particularly interesting and significant case studies. Analyzing the resolutions of the students, some correspondence was sought between the grades assigned by the tutors and the competences achieved by the students, to analyze in detail how they changed over time. In particular, the case studies presented were chosen because their competences, expressed by the five indicators, showed particularly significant changes during the training. Another factor taken into account was the presence of personalized feedback from tutors, which proved to be effective. All of these investigations made it possible to make a global assessment of the progress of problem-solving and digital competences in the selected case studies. A second level of analysis was quantitative. The analysis focused on all the evaluations over time of the students in the sample, in order to obtain a global vision and a more complete study of the evolution of the students’ competences. In order to be able to effectively evaluate any improvement between an initial and a final phase of the training, it was necessary to examine the students who had actively participated. For this reason, the analysis sample was restricted to students who had solved at least five problems (66 students in total), regardless of which problem they solved. After that, it was necessary to identify a submission that represented the initial level of competences and a final submission that represented the level of competences achieved by participating in the training. As the initial submission, the one related to the second problem was chosen, because the first problem had fewer requests as it did not ask for the generalization of the solution. This further narrowed the sample to students who had solved the second problem (61 students in total). The last assignment completed by each student was chosen as the final submission. For this part of the analysis, we will call: • • • •

“initial problem” the second problem; “final problem” the last submitted problem; “initial grade” the total score that students earned in the initial problem; “final grade” the total score that students earned in the final problem.

The Wilcoxon signed rank test for paired samples was then carried out, to compare the initial and final grades and measure any increases or decreases. The Wilcoxon test was chosen because the data did not represent a normal distribution. The test made it possible to verify whether the difference between the median of the initial grades and that of the final grades was zero. Finally, to confirm the results obtained from the test, the boxplots relating to the initial grades and final grades were created, which made it possible to deepen the study. To further develop the analysis, we moved on to study the development of the competences of the 158 students of the initial sample during the entire training. To carry out this analysis, the trend of all the students’ evaluations in the individual indicators and in

Correlation Among Competences Involved in Digital Problem

117

the total score was studied. We computed the arithmetic average of the grades earned by students in each indicator and of the total score, problem by problem. Since the sample of the population on which the averages were computed varied over time and since the outliers had a great influence on the arithmetic mean, the boxplots related to the grades were created and compared. The analysis of the case studies submissions and the analysis of the averages trend made it possible to observe possible correlations between the indicators. Thus, to further evaluate the correlations between the indicators, correlation matrices between the indicators for the initial problem and the eighth problem were created. To create the matrix related to the first problem, all the grades of the students who solved the first problem were considered (113 students in total). To create the matrix related to the eighth problem, all the grades of the students who solved the eighth problem were considered (25 students in total). To study the correlation among the indicators at the end of the training, it was not possible to consider the problems inherent to the “final grade” variable because the last problem submitted by the students is not the same. Thus, the eighth problem was chosen even if not many students submitted it. The Kendall test was carried out to perform this analysis. This test was chosen because the data did not represent a normal distribution and the sample of students who solved the eighth problem was small. Two types of matrices were created for each of the two problems: the first one displays the shades of colors from red to blue, indicating a specific correlation value according to the scale provided next to the matrix. The second one contains the correlation index values, using the same colors as the previous matrix. This choice was made to facilitate the visual reading of correlations and enable a parallel analysis of the two matrices, making the study more precise. For each pair of variables, the Kendall’s tau statistic was reported in the matrix and the p-value was calculated separately to estimate the significance of the correlation. The students’ answers to the final questionnaire were analyzed to consider the students’ point of view and to draw the final conclusions. 69 students filled out the questionnaire. It included items related to various aspects of the project, such as: degree of appreciation of the use of ACE for problem solving; usefulness of having learned to work with an ACE; difficulties faced in solving problems; usefulness of developing digital and problem-solving competences for their professional future; self-assessment of their mathematical, problem-solving and digital competences at the end of the training. The items were mainly 5-point Likert scale, with 5 as highest value. All analyses were performed using Excel software and R statistical software.

5 Results 5.1 Analysis of Case Study 1 Case Study 1 corresponds to the learning trajectory of Paolo (the name is fictional), a student who obtained a low score (below 50) in the first problem and a high score in the last one. Through the analysis of his submissions, it was possible to observe a general improvement in problem-solving and digital competences.. The student’s path (see Fig. 1) starts with a low initial grade of 33/100, in the first half of the training it remains approximately constant while, subsequently, there is a significant improvement

118

A. Barana et al.

which sees a grade of 93/100 in its last submission. The increase in the degree of difficulty of the problems made it possible to consider these results as particularly significant and to select Paolo’s learning trajectory as a case study. From the graph of the evaluation trends related to the individual indicators (see Fig. 2) it is possible to observe that all the indicators show a significant improvement. They reflect the trend of the total scores: starting from low grades, they show a notable improvement in the second half of the training, with the exception of the “use of an ACE” indicator which shows a progressive even if not linear improvement during the entire training. Like the overall grades, also the evaluations of the single indicators do not show a continuous and linear improvement. Many factors could have influenced this aspect: the non-compulsory nature of the extracurricular project, the progressive increase in the complexity of the problems, school, and personal commitments, the mathematical knowledge possessed by the student.

Fig. 1. Trend of Paolo’s overall grades (case study 1) [1].

The graph in Fig. 2 shows an initial lowering of the scores in the indicators “understanding the problematic situation”, “identifying a solution strategy” and “developing the resolution process”. This may be due to the fact that Paolo is initially not completely accustomed to solving non-routine contextualized problems and finds it difficult to solve problems with increasing difficulty. At the same time, the improvement of these indicators in the last submissions, when the problems have a higher degree of complexity, is particularly relevant. In the submission of the first problem, Paolo does not fully develop the solving process. He carries out some computations without arguing the steps carried out and the strategies chosen and provides a short final answer to one of the questions in the problem. He uses the worksheet as a simple writing sheet and is not familiar with Maple commands yet. In the submission of the second problem, Paolo still does not develop and does not fully discuss the proposed resolution. In this case, however, he tries to use the ACE to create an array of point coordinates and to open packages with more advanced commands (see Fig. 3). Feedback from tutors was effective for his improvement. The feedback for the resolution of the first problem was: “The solution is only partially correct. The use

Correlation Among Competences Involved in Digital Problem

119

Fig. 2. Trend of Paolo’s evaluations (case study 1) by indicators [1].

of Maple and the argumentation are poor but don’t give up, for the next problems it will be better! I suggest you comment more on both the results found and the individual steps”. In the resolution of the second problem, Paolo begins to comment on the chosen strategies, such as: • “I include “plots” to be able to use the “pointplot” command”; • “Imagining that we have an exponential curve, the value that we will have on the tenth day will be around 1600 new cases”; • “If you draw a line between the three points, the new cases on the 19th day will be around 2500”. It is possible to notice how he is still unable to identify a strategy to model the problem and to develop the resolution. He confuses exponential trend and linear trend and demonstrates non to be able to make the best use of the ACE. In fact, Paolo does not obtain a mathematical expression that models the problematic situation and is not able to use the commands to show a graph that adequately describes it and for this reason he tries to “imagine” it. In the submission of the third problem there is an improvement in the use of the ACE and in identifying and implementing solution strategies for modeling the problem. The student is still unable to identify the correct strategies for modeling the problematic situation, however he demonstrates originality and creativity in developing the entire resolution through an interactive component (see Fig. 4). The interactive components consist of an interactive table, a slider and a text area. As the values of the slider vary, the values of the last column of the table and the result of the problem in the text area change. The programming code to create the interactive component (top right in Fig. 4), shows the correct use of the commands to take input data (for example: parameter: = Do(%Slider0)) and return output values (for example: Do(%TextArea0 = parameter)). The code also includes a nested loop. This represents an improvement in programming proficiency. Despite the originality of the problem-solving idea, it is not an effective strategy due to

120

A. Barana et al.

Fig. 3. Solution of the second problem submitted by case study 1 [1].

a poor understanding of the problem situation and the lack of identification of a correct modeling. During the training, Paolo’s digital competences gradually improve, and it is possible to observe an improvement also in the indicators “understanding the problematic situation”, “identifying a solution strategy” and “developing the resolution process” starting from the fifth problem. In the last submission, despite a small error of understanding, he fully develops the resolution and implements an effective strategy, though modeling consistent with the interpretation of the problem. This strategy allows him to solve the problem in a clear, schematic, and effective way by exploiting a piece of code that automatically controls the steps to be carried out in order to obtain the desired result under the conditions required by the problem. For this reason, the student obtained the highest marks for the indicators “identifying a solution strategy” and “developing the solution process” but not in “understanding the problematic situation”. The indicators in fact, as components of problem solving, are closely related but, at the same time, each of them has its own “identity” which characterizes and distinguishes it from the others. During the training, the student’s argumentative competence also improves in the last submission the student discusses the steps taken and the strategies chosen, leading the reader to follow the reasoning made. The results show that solving contextualized problems with the ACE enhanced Paolo’s problem-solving and digital competences.

Correlation Among Competences Involved in Digital Problem

121

Fig. 4. Solution of the third problem submitted by Paolo (case study 1) [1].

5.2 Analysis of Case Study 2 Case Study 2 refers to the learning trajectory of Diego (the name is again fictional), a student who obtained a mid-level score (below 70) in the first problem. Through the analysis of case study 2, it was possible to observe a general improvement in problemsolving and digital competences.. Despite the initial mid-level assessment, there is still a progressive improvement in the total score obtained throughout the training (see Fig. 5). The scores achieved by Diego start with an evaluation of 68/100 and finish with an evaluation of 89/100 in the last submission. The improvement is neither continuous nor linear, and a sharp and sudden worsening is observed in solving the sixth problem. Also in this case, different external factors could have influenced the time and effort devoted to the different problems, the tools and mathematical knowledge employed in solving them. From the analysis of the individual indicators (see Fig. 6), we observe a similar trend to that of the total score. More specifically, it can be observed that there are three indicators of particular interest that reflect significant improvement: “identification of a solving strategy”, “argumentation” and “use of an ACE”. In fact, regarding the “use of an ACE” indicator, it starts with a score of 17/25, progressively improves until reaching the maximum in the fourth problem, and slightly decreases in subsequent submissions. As for the “identification of a solving strategy” indicator, Diego starts with a score of 10/21, improves progressively with the first three submissions, remains fairly consistent in the

122

A. Barana et al.

Fig. 5. Trend of the overall assessments of Diego (case study 2).

Fig. 6. Trend of the evaluations of Diego (case study 2) by indicators.

last ones with a score of 18/21, except for a relapse in the sixth problem, which reflects the overall worsening observed earlier. The “argumentation” indicator follows a similar trend: starting with a score of 8/15, it gradually improves and remains approximately constant in the final submissions with a score of 12/15. From these observations, it is possible to suppose a correlation between the indicators “identification of a solving strategy” and “argumentation”. Employing solving strategies through problem modeling and identifying the most appropriate strategy for resolution can facilitate not only the argumentation of the obtained results but also the explanation of the steps taken and the chosen strategies. This, together with the possibility of using different types of representation, can encourage students to appropriately comment on and justify the problem-solving process. The development of problem-solving and digital competences and their correlations are also evident from the analysis of the submissions. In the submission of the first problem (see Fig. 7), initially, Diego presents his solution on the worksheet as if working on a simple sheet of paper. In fact, he develops the solution with numerous

Correlation Among Competences Involved in Digital Problem

123

steps and a series of manual operations performed without exploiting the potential of the ACE. Once the results are obtained, he provides a brief final answer that states the obtained results without commenting on the chosen strategy: “From the two fractions that express the respective probabilities, we can see how in the game of poker, making a FULL is much more difficult than making a FLUSH. In fact, we can clearly see: 198/833 > 6/4165”. The argumentation in the student’s first submission does not involve explaining the steps taken, the chosen strategies, and the formulas used. However, the argumentation of the entire problem-solving process, as well as the obtained results, is crucial as it allows for a deeper understanding of the problem and the solving procedure. Therefore, it often happens that this aspect is emphasized in feedback from tutors. For example, the feedback for the submission of the first problem of case study 2, was: “…Try to provide more accurate reasoning and choose effective data representations. Good job!”. Regarding Diego’s ability to use Maple in solving the first problem, being a beginner, he uses basic commands but with some errors. However, in this case, there is an attempt to improve. Initially, he does not fully understand the function of variable assignments and the distinction between them and functions. In fact, Diego makes a mistake in naming the variable “P(COLORE)” by creating a function instead, thus being forced to manually perform the calculations to obtain a result. Then he correctly assigns the variable “frazione” and recalls it within a function, avoiding the need to write the calculations manually. Even in this case the student creates a function and not a variable. In the subsequent steps of the solution, he does no longer repeat this mistake, correctly distinguishing between assigning a variable and creating a function. Starting from the submission of the second problem (see Fig. 8) there is a clear change in Diego’s approach to problem-solving. He understands the problem’s data, discusses the solving procedure and the solution obtained, is able to model the problematic situation and represents the model graphically. These observations confirm a possible correlation between the two indicators mentioned earlier. The student also uses and takes advantage of a graph that allows for a better and more complete understanding of the problem and the resolution itself. This shows a deeper comprehension and a better and more efficient identification of solving strategies for the modelling of the problem’s situation. The use of the ACE also improves there is a greater mastery of non-basic commands. For example, the case study 2 takes advantages from commands into plots packages. He uses the “pointplot” command to represent scatterplots and the “display” command to combine different types of graphs in the same graphic window. Furthermore, he employs the “plot” command with various options to enhance and make the graph more accurate. Specifically, he adds labels to the Cartesian axes, clarifying the meaning of the x-axis and y-axis, and he adds a grid, making the plot easier to read and more precise. Despite improvement in the use of the ACE, in the submission of the second problem the student shows that he still does not handle the more advanced programming commands: in fact, he does not succeed in the creation of the interactive component. It is important to notice the link manifested between the different indicators, specifically “comprehension of the problematic situation”, “identification of a solving strategy” and “argumentation”. At the beginning of the resolution, the student comments: “I create the function that expresses the trend of the pandemic and draw a plot of it so once I have a

124

A. Barana et al.

Fig. 7. Solution of the first problem submitted by case study 2.

function that is an exponential, so by substituting the number of 10 days inside the function I can calculate the number of cases on the tenth day. An exponential function has the form “y = 2^x”. Then by applying scaling to bring the exponential up through the points A(1;1000) and B(13;2000) I obtain the function y = 1000·2^(x/13), represented in the plot below”. In this way he explains the procedure and strategy followed, demonstrating that he has understood the problem and found an adequate modelling of the problematic situation. Understanding the problem allows Diego to identify an appropriate solving strategy and support it with a graph that is rich in detail, which in turn underlines a good understanding despite some inaccuracies. It is important to notice the effectiveness of tutors’ feedback for the resolution of the first problem. The feedback was: “The chosen strategy was not the most efficient: it would have been sufficient to use the binomial and calculate the number of favorable cases. Furthermore, it was appropriate to evaluate the approximate numerical value of the fractions, e.g. by using the “evalf” command. Try to argument more carefully and choose effective representations of the data. Good work!”. In fact, in the submission of the second problem it can be seen the attempt of the student to improve “identification of a solving strategy”, “argumentation” and “use of an ACE” indicators. This example shows once again the power of formative assessment, which allows students to use feedback on their work to improve their performance. Starting from the third problem, it can be seen an improvement of the “comprehension of the problematic situation”, “identification of a solving strategy” and “use of an ACE” indicators. In fact, although the grades in the second half of the training are approximately

Correlation Among Competences Involved in Digital Problem

125

Fig. 8. Solution of the second problem submitted by case study 2.

constant, they concern problems with increasing difficulty. Diego, in the resolution of the third problem, starts to create interactive components and shows increasing mastery of the ACE. For this submission, he receives the maximum score in the “use of an ACE” indicator. In the submission of the eighth problem, the student proposes a comprehensive resolution, demonstrating a good understanding of the problematic situation. Starting from the first task of the problem, it is evident that the student carefully explains the procedures and reasoning followed. He correctly identifies the use of trigonometry and applies known models with great mastery, obtaining the correct result. In the second task, Diego attempts to provide a graphical representation, but it does not accurately describe the problematic situation. In the third task, despite a mistake in converting degrees to radians, the student identifies the correct solving strategy, supporting it with a comprehensive explanation. As a result, Diego does not receive the highest evaluation in the “use of an ACE” indicator. However, he still demonstrates improvements in the use of the ACE and in identifying solving strategies. As already emphasized, the indicators are closely interconnected as components of problem-solving. Thus, the analysis of the evolution of problem-solving and digital competences must take all these aspects into consideration. 5.3 Analysis of the Evaluations of All Students For the analysis of the evaluations of all the students, the Wilcoxon test was carried out to evaluate if any improvement occurred between the initial phase and the final phase

126

A. Barana et al.

of the training. The p-value of 0.89 > 0.05 did not allow us to reject the null hypothesis according to which the medians of the initial evaluations and of the final evaluations were equal. This result is satisfactory for the purpose of this research. In fact, since the final grades relate to problems of greater difficulty, the equality or a non-significant difference in the evaluations shows that the students have developed competences to solve problems of greater difficulty, suggesting an improvement in these competences. This result was confirmed by the boxplots related to the initial and final grades (see Fig. 9). Indeed, they show that the medians are the same, with a value of 84/100. This value indicates a high starting level (above 70) which becomes more significant when related to the final submission, reflecting more developed problem-solving and digital competences. In the initial problem, the median is very close to the third quartile indicating a high number of evaluations between 84 and 89, while in the final problem 50% of the grades are distributed symmetrically with respect to the median with evaluations between 74 and 95. These results satisfy expectations: 25% of the grades with a value greater than 84, which initially was between 84 and 89, in the final problem are distributed between 84 and 95, indicating that a greater number of students earned grades greater than 89. At the same time, the first quartile moves from a grade of 72 to a grade of 74, indicating that a greater number of students obtained a grade higher than 74. The greater dispersion found in the final problem compared to the initial problem can be justified by the increase in the difficulty of the problems, which therefore led to a greater variability of the grades. At the same time, however, the dispersion to the right of the median and the increase in the value of the first quartile show a general improvement in students’ competences.

Fig. 9. Boxplot of the overall evaluations related to the initial and final problem [1].

By studying the trend of the average grades during the entire training, it was possible to expand the analysis and obtain a more complete vision of the development of the students’ problem-solving and digital competences. The investigation of average grades showed an overall improvement in problem-solving and digital competences. In fact,

Correlation Among Competences Involved in Digital Problem

127

from the graphs of the trend of the average grades of the individual indicators and of the total ones (see Fig. 10), a slight improvement can be observed for all the indicators, even though not continuous or linear, with a general decrease in the sixth problem. In particular, the “argumentation” indicator shows a more evident improvement, with a more regular trend and a progressive improvement. This aspect indicates that, although it was complex to understand, identify and develop a solution strategy, the students were able to explain and justify their resolution in an even more precise, complete, and pertinent way. In problem six there is a drop in scores on all indicators. The students’ difficulties can also be seen in the discussions in the forum on the platform: • “Hi, I didn’t quite understand the third request”; • “I wanted to ask what the exp function was in the formula that I didn’t understand”; • “Hi, regarding point 2 of the problem, how did you do it (in a very general way)? Did you use a more algebraic or graphical approach? Because graphically it seems to me very complex to visualize, while algebraically I find it more it difficult to find the right commands”. A slight worsening of all indicators in the second problem can also be observed. This aspect is not surprising since the first problem had a lower difficulty as it did not require the generalization phase of the resolution. Analyzing the indicators’ graphs individually and comparing their trends, it can be seen that the “identification of a solving strategy”, “developing the solution process” and “use of a ACE” indicators approximately show the same trend (see Fig. 10): a worsening between the first and the second problem, followed by a slight improvement until the fifth problem, then showing a drop in the sixth problem, a subsequent improvement in the seventh followed by a final worsening in the eighth problem. The “comprehension of the problematic situation” and “argumentation” indicators also show the same trend: a gradual improvement starting from the second problem with a drop in the sixth problem. So, correlations between the “identification of a solving strategy”, “developing the solution process” and “use of an ACE” indicators and between “comprehension of the problematic situation” and “argumentation” indicators can be supposed. This would show that an improvement in “argumentation” indicates a deeper understanding and vice versa. The arithmetic mean may not effectively represent the assessment of students’ performance during training as it is affected by outliers. We decided to analyze the median of the total scores obtained during the training. From the boxplots of the total scores (see Fig. 11) it can be observed, for each problem, a distribution of half of the evaluations approximately symmetrical with respect to the median and a generally reduced width of the interquartile ranges, indicating a concentration of the grades around the median. This indicates that the median provides a good representation of the grades. In the sixth problem, the interquartile range is instead wider, indicating a wider distribution of evaluations. This implies that the median, in this case, is less representative of the evaluations obtained by all the students for that problem. This may be due to the considerations made previously on the difficulty faced by the students in solving that problem. In the seventh problem, however, the median corresponds to an evaluation of 95/100 and is close to the third quartile, indicating many submissions with a very high grade (above 95). This justifies the peak that is also found in the graph of average ratings. The trend

128

A. Barana et al.

of the medians is very similar to that of the average evaluations of the various indicators and of the total. Therefore, it was possible to confirm what emerged from the analysis of the average ratings.

Fig. 10. Graphs of the trends of the average grades of the single indicators [1].

From the analysis of the problem solving and digital competences development during the online training, correlations between various indicators have been observed. Therefore, to further investigate the analysis, we decided to examine the correlation matrix of the indicators in the first and eighth problem created using Kendall’s test. These matrices make it possible to analyze the correlations between the indicators at the beginning and the end of the training and assess their changes. Figure 12 shows the correlation matrix of the first problem and Fig. 13 shows the correlation matrix of the eighth problem. The correlation was studied between the five indicators of the evaluation grid considering the evaluations of the first and eighth problem respectively. The legend used for the indicators used in the matrix is: “C” for “comprehension”, “I” for “identification of a solving strategy”; “S” for “development of the solving process”; “A” for “argumentation”; “M” for “use of an ACE”. For each pair of variables, the p-value of the correlation was calculated, for a total of 20 tests (10 for the first problem and 10 for the eighth problem). The p-values of the tests of the first matrix

Correlation Among Competences Involved in Digital Problem

129

Fig. 11. Boxplot of the overall evaluations of all the students [1].

are all acceptable as all the values are less than 0.0001. The tests of the second matrix gave acceptable p-values except in three cases: • “comprehension”-“argumentation”: p-value = 0.5369 • “identification of a solving strategy”-“argumentation”: p-value = 0.5021 • “development of the solving process”-“argumentation”: p-value = 0.6993 For these pairs of indicators, the correlation value is not statistically significant. The other p-values range from 4.09^{-10} to 0.04222, on average higher than in the first case. This result could be due to the fact that the second sample is much smaller than the first. The indexes with the highest correlation in the first problem are between “identification of a solving strategy” and “developing the solution process” and between “identification of a solving strategy” and “comprehension of the problematic situation” with values of 0.53 and 0.52 respectively (see Fig. 12). These indexes show a moderately positive concordance because the values ± 1 indicate the maximum correlation that can be achieved. These correlations show that the identification of an appropriate strategy using mathematical representations and different registers correlates with a comprehensive, clear, and correct development of the problematic situation. Understanding the problem correctly by analyzing and interpreting it comprehensively correlates with identify a problem-solving strategy that adequately models the problem. The indexes with highest correlation in the eighth problem are between “identification of a solving strategy” and “developing the solution process”, between “use of an ACE” and “identification

130

A. Barana et al.

of a solving strategy” and between “use of an ACE” and “developing the solution process” with values of 0.94, 0.79, 0.74 respectively (see Fig. 13). These values show strong correlations between the indicators. Therefore, the identification and use of appropriate commands, the combination of representation registers, and the creation of interactive components correlate with the development of appropriate problem-solving strategies, supported by models and diagrams, and the development of a comprehensive and relevant problem resolution. These results thus confirm the correlations between the three indicators also observed in the analysis of the trend of the average evaluations. It is interesting to note that from the first to the eighth problem, the correlation index between “identification of a solving strategy” and “developing the solution process” consistently remains the highest, and its value increases along with the training progresses.

Fig. 12. Correlation matrix of the first problem.

Over time, therefore, the relationship between the two indicators becomes stronger and more deeply dependent (with a value of 0.94). These results demonstrate how all the indicators are closely connected and essential for problem solving with an ACE.

6 Analysis of Students’ Answers to the Final Questionnaire The last part of the analysis concerned the students’ answers to the satisfaction questionnaire, to understand their point of view on some aspects of the online training and on the development of their competences. The first question examined was: “In solving problems, which of the following aspects did you have the greatest difficulty with?”. The answers (see Table 1) show that the students had less difficulty developing argumentation (average: 2.70 out of 5), confirming the constant increase in ratings for this indicator. Students experienced a slightly greater difficulty in generalizing the problem, indicated by an average of 3.48. In the other indicators of the evaluation grid, students found approximately the same degree of difficulty: the average values of these answers are between 3.13 and 3.20. These results reflect what was observed in the analysis of the assessments of all students.

Correlation Among Competences Involved in Digital Problem

131

Fig. 13. Correlation matrix of the eighth problem.

Table 1. Students’ answers about the difficulties encountered in solving problems [1].

Interpreting the text

Mean

St. Dev

3.20

1.1

Identifying a solution strategy

3.13

0.87

Completing the resolution process

3.16

1.04

Discussing the solution

2.70

1.06

Generalizing the problem

3.48

1.02

Using Maple

3.17

0.79

The second question examined was: “Please indicate to what extent you think you have acquired the following competences in the online training”. The answers (see Table 2) indicate that, from the students’ point of view, participating in an online training in a DLE and using an ACE for problem solving fostered the development of their mathematical, digital and, above all, problem solving competences. The third question examined was: “Please indicate to what extent you think these competences will be useful in the world of work”. Table 3 shows the results. It is interesting to observe how students are aware that mathematical, problem-solving, and digital competences will be important for their future, even outside the school context. The last question analyzed concerned the school average of the students in mathematics (expressed by the students in a grade from 1 to 10) at the beginning and at the end of the online training. For 30% of the students the average improved, for 67% of the students the average remained unchanged and for 3% of the students the average worsened. In particular, the average decreased only in two students who went respectively from 10 to 9.5 and from 8 to 6. Furthermore, 55% of the students whose average remained unchanged had a high starting average (above 8). For this reason, the results obtained

132

A. Barana et al.

are satisfactory and show a general improvement in the mathematical competence of the students participating in the online training. Table 2. Students’ answers about the development of their competences [1]. Acquired competences

Mean

St. Dev

Mathematical competences

3.12

0.72

Digital competences

3.46

0.70

Problem-solving competences

3.52

0.85

Table 3. Students’ answers about the usefulness of the competences in the world of work [1]. Utility in the world of work

Mean

St. Dev

Mathematical competences

3.60

0.89

Digital competences

4.25

0.72

Problem-solving competences

4.19

0.81

7 Conclusions This research work had the main objective of evaluating the development of secondary school students’ problem-solving and digital competences and the correlations between them. The analysis was conducted on grade 12 students who carried out problem-solving activities with an ACE during an online training in a DLE from the 2021/2022 school year edition of the DMT project. The four indicators used to evaluate problem solving competence are: comprehension; identification of a solving strategy; development of the solving process; argumentation. The fifth and final indicator of the evaluation grid was dedicated to digital competence. To answer the research questions, the evolution of the evaluations obtained by the students during the training was studied, assigned by specially trained tutors following the evaluation grid. The analyses were conducted both globally considering the total assessments and indicator by indicator, also analyzing the correlations between the various indicators and seeking confirmation of this evolution in the submissions of two exemplary case studies. Finally, the responses of the training participants to a questionnaire submitted at the end of the training were examined, to compare the results obtained from the analyses with the students’ point of view. Regarding the first research question, the quantitative analysis of all students’ assessments showed an overall improvement in students’ problem solving and digital competences. The results show an improvement in each indicator, with a more evident improvement in the “argumentation” indicator. As the training progressed, students improved their ability to understand the problem situation, identify an appropriate problem-solving

Correlation Among Competences Involved in Digital Problem

133

strategy, develop a comprehensive and relevant resolution, and particularly, to argue the entire problem-solving process, explaining and justifying the results obtained. The improvement in the “argumentation” indicator may show that the students were not very used to arguing their choices and ideas during the mathematical activities. During the digital problem-solving activities students learned, as can be seen from the case study analysis, to argue the choices made and the results obtained but also to explain (ideally to the tutors or their peers) the processes activated during the resolution and the choices in using the commands of the ACE. This increases the level of understanding, consolidates knowledge, and further develops problem-solving competence. The digital competences of the students also grew during the online training. Increasing proficiency in using the ACE has also impacted problem solving proficiency. The use of an ACE in problem solving has made it possible to support all phases of problem solving, allowing to focus on the resolution process, on exploration and on the results obtained, and to exploit different types of representation in the same environment. The growing difficulty of the problems has also helped to foster the development of problem-solving and digital competences, stimulating the commitment, participation, and training of the students. This also stimulated engagement, participation and challenged students who enjoyed the training problems as challenging, interesting, engaging and useful for understanding mathematics. Regarding the second research question, the results show a general increase in the correlation of problem-solving and digital competences. The correlation between the indicators at the beginning of the training is very different from the correlation between the indicators at the end of the training. In the correlation matrix of the first problem, the greatest correlation was between the indicators “identification of a solving strategy” and “developing the solution process” and between “identification of a solving strategy” and “comprehension of the problematic situation”. In the correlation matrix of the eighth problem, the correlation between the indicators “identification of a solving strategy” and “developing the solution process” significantly increases (from 0.53 to 0.94) and also the correlation between these two indicators and the indicator “use of an ACE”. At the end of the training, the problem-solving competence is much more related to the digital competence of the students. Effectively using the ACE by employing appropriate commands and combining different types of representation correlates the identification and development of correct, creative problem-solving strategies that align better with students’ thinking. The opportunity to choose from different solving methods could encourage students to explore the problem situation, identify and develop appropriate problem-solving strategies, and validate and justify their reasoning, which are fundamental attitudes in the problem-solving process. The analysis of the case studies confirmed the correlation between digital and problem-solving competences. It also showed how these competences grew during the training, even in a different way from student to student. This analysis also showed that the evaluation system had a positive impact on the development of students’ competences. The personalized feedback from the tutors and the comparison of the evaluations obtained with the shared assessment rubric have allowed the students to establish their own level of competence and to understand what and how to improve, which are the three important processes of formative assessment [17].

134

A. Barana et al.

The results show that the problem-solving activities with an ACE by the students and the evaluation system, were effective for the development of problem-solving and digital competences. The analysis of the final questionnaire shows that the students believe that these digital problem-solving activities have favored the development of their problem-solving and digital competences. During the problem-solving activities, they found the same degree of difficulty in most of the indicators of the evaluation grid, except for “argumentation” and “use of an ACE”. In the first case, the perceived difficulty was lower, probably because their improvement in this indicator was more evident. In the second case, the perceived difficulty was higher, probably for the highest complexity of this process. The students really appreciated the feedback from the tutors on their resolutions to understand how to improve in solving problems and in using the ACE. According to students, mathematical competence, digital competence and problem-solving competence are important, both in school and in out-of-school contexts. Since the development of these competences are also part of the institutional objectives, it is desirable to promote these types of activities within the school context. A limitation of this study is the variation in the number of students who submitted their resolutions over the course of the training, because the activities are not mandatory for students. Future research could propose problem-solving activities with an ACE during lessons at school, to carry out the analysis on a sample of students that does not vary over time. It would be interesting to compare the development of problem-solving and digital competences using a control group of the same education level, composed of students who do not participate in the activities. In this way it would be possible to further evaluate the effectiveness of problem-solving activities with an ACE for the development of these competences and their correlations. However, this is not easy because some tasks would be difficult to be implemented without the ACE. This type of project shows how technology can be used naturally in ordinary teaching. It allows teachers to rethink the teaching methods, and at the same time allows students to develop mathematical, digital and problem-solving competences. Acknowledgement. The research was carried out within Indam - Istituto Nazionale di Alta Matematica “Francesco Severi” and the Diderot Program funded by the Bank Foundation “Cassa di Risparmio di Torino”. The authors thank the CRT Foundation for the financial support to the Digital Math Training project line.

References 1. Barana, A., Fissore, C., Lepre, A., Marchisio, M.: Assessment of digital and mathematical problem-solving competences development. In: Jovanovic, J., Chounta, I.-A., Uhomoibhi, J., and McLaren, B. (eds.) Proceedings of the 15th International Conference on Computer Supported Education (CSEDU 2023), pp. 318–329. SCITEPRESS (2023) 2. European Parliament and Council: Council Recommendation of 22 May 2018 on key competences for lifelong learning. Official Journal of the European Union, pp. 1–13 (2018) 3. MIUR: Schema di regolamento recante “Indicazioni nazionali riguardanti gli obiettivi specifici di apprendimento concernenti le attività e gli insegnamenti compresi nei piani degli studi previsti per i percorsi liceali” (2010)

Correlation Among Competences Involved in Digital Problem

135

4. Barana, A., Boetti, G., Marchisio, M.: Self-assessment in the development of mathematical problem-solving skills. Educ. Sci. 12, 81 (2022). https://doi.org/10.3390/educsci12020081 5. Barana, A., Marchisio, M.: From digital mate training experience to alternating school work activities. Mondo Digitale. 15, 63–82 (2016) 6. Barana, A., et al.: The role of an advanced computing environment in teaching and learning mathematics through problem posing and solving. In: Proceedings of the 15th International Scientific Conference eLearning and Software for Education, pp. 11–18. Bucharest (2019). https://doi.org/10.12753/2066-026X-19-070 7. Barana, A., Marchisio, M., Sacchet, M.: Interactive Feedback for learning mathematics in a digital learning environment. Educ. Sci. 11, 279 (2021). https://doi.org/10.3390/educsci11 060279 8. Brancaccio, A., Marchisio, M., Meneghini, C., Pardini, C.: More SMART mathematics and science for teaching and learning. Mondo Digitale. 14, 1–8 (2015) 9. Suhonen, J.: A formative development method for digital learning environments in sparse learning communities (2005). http://epublications.uef.fi/pub/urn_isbn_952-458-6630/index_en.html 10. Barana, A., Marchisio, M.: A model for the analysis of the interactions in a digital learning environment during mathematical activities. In: Csapo, B., Uhomoibhi, J. (eds.) Computer Supported Education: 13th International Conference, CSEDU 2021, Virtual Event, April 23– 25, 2021, Revised Selected Papers, pp. 429–448. Springer International Publishing, Cham (2022). https://doi.org/10.1007/978-3-031-14756-2_21 11. Samo, D.D., Darhim, D., Kartasasmita, B.: Culture-based contextual learning to increase problem-solving ability of first year university student. J. Math. Educ. 9(1), 81–94 (2017). https://doi.org/10.22342/jme.9.1.4125.81-94 12. National Council of Teachers of Mathematics: Executive Summary Principles and standards for school mathematics (2000) 13. D’Amore, B., Pinilla, M.I.F.: Che problema i problemi! L’insegnamento della matematica e delle scienze integrate. 6, 645–664 (2006) 14. Baroni, M., Bonotto, C.: Problem posing e problem solving nella scuola dell’obbligo. Presented at the October 23 (2015) 15. Leong, Y.H., Janjaruporn, R.: Teaching of problem solving in school mathematics classrooms. In: Cho, S.J. (ed.) The Proceedings of the 12th International Congress on Mathematical Education, pp. 645–648. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-126883_79 16. Jonassen, D.H.: Learning to Solve Problems: A Handbook for Designing Problem-Solving Learning Environments. Routledge, New York (2010). https://doi.org/10.4324/978020384 7527 17. Black, P., Wiliam, D.: Developing the theory of formative assessment. Educ. Assess. Eval. Account. 21, 5–31 (2009). https://doi.org/10.1007/s11092-008-9068-5 18. Hattie, J., Timperley, H.: The power of feedback. Rev. Educ. Res. 77, 81–112 (2007). https:// doi.org/10.3102/003465430298487 19. Pólya, G.: How to Solve it. Princeton University, Princeton (1945) 20. Liljedahl, P., Santos-Trigo, M., Malaspina, U., Bruder, R.: Problem Solving in Mathematics Education. Springer, New York (2016) 21. Malara, N.A.: Processi di generalizzazione nell’insegnamento/apprendimento dell’algebra. Annali online formazione docente. 4, 13–35 (2013) 22. Barana, A., Conte, A., Fissore, C., Floris, F., Marchisio, M., Sacchet, M.: The creation of animated graphs to develop computational thinking and support STEM education. In: Gerhard, J., Kotsireas, I. (eds.) Maple in Mathematics Education and Research: Third Maple Conference, MC 2019, Waterloo, Ontario, Canada, October 15–17, 2019, Proceedings, pp. 189–204. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-412586_14

Regulatory Strategies for Novice Programming Students Deller James Ferreira(B) , Dirson Santos Campos , and Anderson Cavalcante Gonçalves Institute of Informatics, Federal University of Goiás, Alameda Palmeiras, Goiânia, Brazil {deller,dirson_campos}@ufg.br

Abstract. Self-regulated learning holds great significance within the realm of introductory computer programming. It refers to the active involvement of students in their academic journey, encompassing motivational, behavioral, metacognitive, and cognitive aspects. In addition, the social regulation of learning plays a crucial role in programming education, wherein students collaborate to regulate their cognition, behavior, motivation, and emotions, fostering temporary coordination with peers or teachers. Unfortunately, current teaching and learning methodologies in programming tend to overlook the development of self-regulation, co-regulation, and shared regulation skills. Therefore, in this research we investigate the extent to which students in introductory programming employ regulation strategies during their programming endeavors. Through an exploratory study involving 198 (one hundred ninety-eight) participants, it was discovered that a significant number of students fail to engage in regulatory strategies while learning introductory programming. Consequently, there is a pressing need to implement strategies that support students’ regulation during the learning process. To address this issue, 8 (eight) instructional scripts have been developed for teachers, incorporating evidence-based principles of programming learning and focusing on fostering regulatory strategies. Keywords: Self-regulation · Co-regulation · Socially shared regulation · Introductory programming

1 Introduction Undergraduate computing courses often encounter significant dropout and failure rates, particularly in introductory programming courses. Programming itself is regarded as a challenging endeavor, as it entails complex problem-solving tasks that impose numerous cognitive demands on students (Loksa and Ko 2016). Introductory programming courses pose various challenges for students, as they must acquire a diverse set of skills encompassing both programming proficiency and an understanding of the program code development process (Falkner et al. 2014). Inefficient utilization of learning strategies, such as self-regulation and shared regulation, may contribute to poor performance in programming learning (Soares 2021). © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024 B. M. McLaren et al. (Eds.): CSEDU 2023, CCIS 2052, pp. 136–159, 2024. https://doi.org/10.1007/978-3-031-53656-4_7

Regulatory Strategies for Novice Programming Students

137

Self-regulated learning is a critical subject in education, encompassing the regulation of student motivation, engagement, cognition, and metacognition. It can be defined as the extent to which students actively participate in their own academic learning, taking into account motivational, behavioral, metacognitive, and cognitive aspects (Pintrich 2000). In a study conducted by Bergin et al. (2005), the relationship between self-regulated learning and performance in introductory programming was investigated, revealing that self-regulated learning serves as a valuable predictor of programming performance. Self-regulation entails various skills, including monitoring one’s processes, reflecting on the effectiveness of the process, monitoring comprehension of crucial concepts, and identifying alternative strategies to solve problems (Loksa 2020). Another significant aspect of programming learning is regulation within collaborative learning. Group regulation comprises two key components: co-regulation and socially shared regulation. Co-regulation refers to the support or management of an individual’s learning regulation by other group members, typically peers of equal status (Hadwin et al. 2018). This aspect becomes particularly crucial during moments of cognitive, motivational, or behavioral challenges, where group members may need to assist in co-regulating each other’s learning. Socially shared regulation, on the other hand, involves the negotiation and sharing of cognitive and metacognitive processes, as well as emotional and motivational states. It entails the development of a collective understanding and experience that emerges through iterative interactions (Järvelä and Järvenoja 2011). Successful collaboration, therefore, relies on multiple individuals who engage in self-regulation by taking responsibility for their own emotions, motivation, cognition, and behavior. They also monitor and support their peers through co-regulation, while simultaneously monitoring the overall progress and dynamics of the group through socially shared regulation. In computer programming, socially shared regulated learning plays a crucial role in enhancing students’ programming skills. It equips students with external resources and abilities, such as seeking social assistance, evaluating others’ ideas, and monitoring tasks, which contribute to their improvement (Tsai 2015). Moreover, in computer science education, it is vital to prepare students for the challenges they will face in their future professional endeavors while providing them with opportunities to develop self-regulation, shared regulation, and co-regulation skills through activities that promote collaborative learning and active engagement (Wang et al. 2013). Despite its importance, self-regulation, co-regulation, and socially shared regulation strategies for programming are not yet well understood by researchers and educators in the field of computer science, making it challenging to effectively develop methods that foster self-regulation and shared regulation learning. The topic of regulation in programming teaching and learning is still relatively new, demanding further research (Soares 2021). While there is a growing interest in regulated learning within computer education research, theories, models, and tools specific to the programming context remain scarce (Prather et al. 2020; Szabo et al. 2019; Malmi et al. 2019). Thus, the objective of this

138

D. J. Ferreira et al.

work is to explore the extent to which introductory programming students employ regulation strategies and to propose instructional scripts aimed at promoting a more efficient utilization of these strategies. The study conducted by Ferreira and Campos (2023) investigated the utilization of regulation strategies among introductory programming students, revealing a deficiency in the application of effective regulatory strategies. Building upon these findings, the present study revisits and expands upon this research, proposing mechanisms to address this issue. The primary objective of this study is to address the gap identified by Ferreira and Campos (2023) in the utilization of regulation strategies among introductory programming students. In particular, the research aims to propose effective support for students’ regulatory strategies in the learning process. The contribution of this work lies in its endeavor to build upon the findings of Ferreira and Campos, thereby expanding and revisiting their research. By proposing innovative mechanisms to rectify the lack of regulatory ability in introductory students, this study strives to offer practical solutions that can significantly impact the learning outcomes of introductory programming students. The importance and timeliness of this work are underscored by the critical need to optimize regulatory strategies in programming education, ensuring that students acquire a solid foundation and proficiency in programming skills. In an era where technological advancements are rapidly evolving, the timely enhancement of programming education is imperative to meet the demands of the ever-changing landscape of the digital world. The research questions that lead the preset work are: RQ1. To what extent students in introductory programming employ regulation strategies during their programming endeavors? RQ2. How can we mitigate a lack of regulatory ability in introductory programming students?

2 Literature Review This section approaches some research regarding investigations on how students’ selfregulation, co-regulation and shared regulation skills influence programming learning and why scripts are adequate for teaching regulatory strategies for students. Two crucial factors associated with student success and retention are student engagement and motivation, both of which are closely tied to emotional and behavioral selfregulation skills. According to Abdullah and Yih (2014), motivation plays a significant role in shaping students’ approach to learning, while engagement involves investing time and effort into learning activities (Crisp et al. 2015). In the case of programming students, inadequate time management skills often lead to complaints about a lack of time for studying or problem-solving (Pereira et al. 2021). While there have been relatively few studies conducted on motivation specifically in programming students (Coto et al. 2022), existing research suggests that inappropriate teaching methods can undermine learners’ motivation during programming instruction. It emphasizes the necessity of incorporating teaching methods that involve self-regulation strategies to enhance motivation (Darabi et al. 2022). Furthermore, the literature highlights cognitive and metacognitive factors contributing to failure in programming courses. One such factor is the challenge of mastering multiple concepts, skills, and computing models necessary for designing, implementing,

Regulatory Strategies for Novice Programming Students

139

and testing programs (Robins et al. 2003). Another recent area of interest in the computer science education research community pertains to the development of knowledge about the problem-solving process as a cognitive obstacle in learning programming (Loksa 2020). Knowledge of the problem-solving process encompasses understanding how to solve programming problems, and utilizing problem-solving strategies is a cognitive selfregulation skill. This process entails various skills, such as interpreting and comprehending programming problems (Wrenn and Krishnamurthi 2019), designing and adapting algorithms, translating these algorithms into programming language notation (Xie et al. 2019), verifying the effectiveness of the implementation through testing (Kaner and Padmanabhan 2007), and debugging programs when they do not perform as intended (Ko et al. 2019). For instance, even if someone has a solid grasp of the fundamentals of a programming language, strong self-regulation skills can help them recognize gaps in their understanding, such as a lack of comprehension regarding how a for loop operates in Python. This recognition prompts them to enhance their understanding before proceeding with writing or reviewing their code. Similarly, when individuals struggle to identify flaws in their programs, self-regulation skills enable them to acknowledge their struggle and seek expert guidance on more productive approaches to diagnose the problem. Programming research consistently demonstrates that self-regulation skills strongly correlate with success in solving computational problems (Falkner et al. 2015). Recent studies, however, indicate that most beginners possess poor self-regulation skills, which are associated with subpar programming outcomes (Hauswirth and Adamoli 2017). An example of a cognitive self-regulation strategy for problem-solving is accurate problem interpretation. When students inaccurately interpret a problem, they are likely to employ ineffective strategies or fail to solve the problem altogether. Studies report that students often struggle to identify and articulate the problem’s objectives, requirements/constraints, and expected outcomes. In other words, students frequently lack self-regulation skills, particularly in terms of task comprehension. Among behavioral self-regulation strategies, effort management, time management, and help-seeking have been found to have a positive correlation with academic outcomes (Daradoumis et al. 2021). Effort management strategies assist students in directing their attention towards the task at hand and utilizing their effort effectively. Through these strategies, students develop skills that enable them to handle setbacks, persevere, and overcome challenges. Consequently, these strategies foster motivation and commitment towards achieving goals, even in the face of difficulties or distractions. Time management strategies enable students to acquire skills related to goal setting, prioritization, planning, self-monitoring, conflict resolution, negotiation, task allocation, and problem-solving. Successful time management empowers students to optimize their use of time, facilitating academic performance, balance, and satisfaction (Daradoumis et al. 2021). In terms of metacognitive strategies, reflective learning plays a crucial role in enhancing students’ awareness of the learning process and its associated challenges (Chang 2019). When students engage in effective self-reflection, they analyze their learning

140

D. J. Ferreira et al.

methods, comprehend the objectives of the learning process, and identify the conditions necessary for success. Reflection also plays a pivotal role in encouraging students to engage in critical thinking about their abilities and contemplate strategies to enhance the learning process. This process fosters awareness of the benefits of lifelong learning and assists in the development of transferrable skills (Chang 2019). The interplay between students’ commitment, self-control, autonomy, and self-discipline enables them to regulate their actions effectively to achieve their learning goals. According to Cheng et al. (2021), computer science students must possess not only technical skills but also collaborative problem-solving and teamwork abilities. Unfortunately, many students enter the job market lacking the necessary skills to meet employer expectations in terms of teamwork and cooperation (Pedrosa et al. 2019). While students may acquire substantial theoretical knowledge throughout their courses, they often lack essential transferable skills, commonly referred to as soft skills, which are rarely addressed in project management education (Groeneveld et al. 2019). The dynamic nature of software development demands that computer scientists possess skills beyond technical expertise, including self-reflection, conflict resolution, communication, teamwork, and creativity. Some authors have highlighted the benefits of collaboration in programming learning activities (Hwang et al. 2012), particularly in terms of motivating students and enhancing their engagement. Collaborative learning sessions enable students to achieve their learning objectives through assignments, group work, and skill-sharing. When comparing collaborative learning with traditional learning, it is important to recognize that in the context of programming, collaboration facilitates the exchange of ideas among students and enables the development of more effective learning processes, skills, and outcomes (Hwang et al. 2008). In terms of problem-solving time, numerous studies (McDowell et al. 2002) indicate that students who engage in group learning require less time to solve programming problems and produce better solutions compared to those who learn individually. Consequently, to succeed in collaborative learning, students need to acquire skills in co-regulation and socially shared regulation. Developing regulatory strategies is vital to helping students achieve success. A regulated learner will, for example, define their goals, organize their resources, then manage their time effectively, and collaborate in an efficient way. Without this fundamental level of metacognition, they cannot direct their knowledge in a useful and constructive way. However, regulation of learning embraces a set of demanding skills that is best learned with teacher support, using regulation strategies. The use of scripts is one of the approaches that can be used by teachers to facilitate student engagement in regulatory processes. In this research, the scripts are treated as a set of steps, which instantiate a sequence of regulatory processes that students must engage in a given educational situation. Script configuration theory (Stegmann et al. 2016) assumes that a student’s internal scripting configuration is influenced by sample scripts provided to students. In any collaborative or individual learning situation, the students’ set of goals and perceived situational characteristics influence how they dynamically configure internal collaboration scripts. Script configuration theory also considers that

Regulatory Strategies for Novice Programming Students

141

students select components of scripts from their dynamic memory that have worked in similar situations in the past.

3 Research Method 3.1 Procedures In order to answer the research question RQ1 (RQ1. To what extent students in introductory programming employ regulation strategies during their programming endeavors?), we applied exploratory study collecting and analyzing data from students’ perceptions of their regulatory skills. Enhancing our comprehension of student learning in educational settings requires acknowledging the significance of students’ constructed realities, as highlighted by Struyven et al. (2005). The experiences students undergo hold substantial value in shaping their understanding and approach to learning. Entwistle (1991) further underscores the impact of the learning environment on students’ learning methodologies. It is revealed that a student’s perception of the learning environment functions as a determining factor in their learning process. Importantly, the lived reality of students serves as an intervening variable, a crucial element that cannot be disregarded. Neglecting this aspect undermines our ability to achieve a comprehensive understanding of student learning, emphasizing the need for its inclusion in both research and educational practices. Collecting and analyzing data regarding students’ perceptions of their regulatory skills becomes important in this context. These insights into how students perceive their own regulatory skills contribute valuable information for educators and researchers. Understanding these perceptions aids in tailoring educational approaches, ensuring they align with students’ lived experiences and learning environments, fostering a more effective and student-centric process. In pursuit of addressing research question RQ2 (RQ2: How can we mitigate a lack of regulatory ability in introductory programming students?), we rely on the script theory and evidence-based principles of programming learning derived from scientific literature. This comprehensive exploration led to the development of 8 instructional scripts. These scripts are meticulously designed to serve as scaffolds, aiding novice programming students in enhancing their regulatory strategies. The potential efficacy of this solution lies in its strategic integration of script theory and evidence-based principles. By drawing on established theoretical frameworks and evidence-backed methodologies, the instructional scripts aim to provide a structured and effective approach to support students in navigating the challenges associated with regulatory abilities in programming. This amalgamation of theoretical insights and practical strategies positions the solution to be not only informative but also actionable, catering to the specific needs of introductory programming students. As a result, the potential impact of these instructional scripts extends beyond mere theoretical discourse, holding promise for tangible improvements in students’ regulatory skills within the realm of programming education. 3.2 Participants The questionnaire was completed by a total of 198 (one hundred ninety-eight) undergraduate students from various disciplines, including computer science, computer engineering, medical physics, physical engineering, statistics, and electrical engineering. The

142

D. J. Ferreira et al.

age range of the respondents varied between 17 and 36 years old, with a predominant age group of 18 to 19 years old. In terms of gender, 86.6% of the students identified as male, 12.5% identified as female, and 0.9% identified as a gender other than male or female. Regarding the respondents’ professional and academic experience with programming skills outside the classroom, the results were as follows: 15.9% reported having no experience, 21.4% had taken extracurricular courses, 7.1% had worked as an intern, 3.6% had participated in scientific research, 0.9% had participated in an extension project, 3.6% had one year of experience, 1.8% had one to three years of experience, 0.9% had more than three years of experience, 0.9% were computer technicians, 0.9% had taken an online course, and 0.9% had programming experience in high school. 3.3 Instruments In this study, we utilized a 5-factor Likert scale (Likert 1932) questionnaire to collect students’ perceptions of their regulatory ability. The questionnaire was designed to gather information regarding students’ perceptions of the utilization of regulatory strategies, specifically focusing on the familiarity and frequency of self-regulation, co-regulation, and shared regulation strategies among introductory programming students. The questionnaire consists of two main parts. The first part aims to assess the students’ use of self-regulation strategies based on their own perceptions. The questions in this section were adapted from the Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich et al. 1983) to align with the context of programming learning. The second part of the questionnaire was developed to check group regulation in programming, again based on the students’ own perceptions. The questions in this section drew inspiration from the Adaptive instrument for Regulation of Emotions (AIRE) (Järvenoja et al. 2013). When designing a questionnaire, two crucial considerations are its validity and reliability. Validity refers to the extent to which the instrument measures what it is intended to measure, while reliability relates to the consistency of the items or questions in measuring the same construct (Prous et al. 2009). To establish the validity and reliability of the questionnaire used in this study, qualitative and quantitative methods were employed. Firstly, six experts in the field of computer science were invited to evaluate the selected questions. Their expertise ensured that the questions were clear, easily understandable, covered relevant aspects of self-regulation, co-regulation, and shared regulation in programming, and were comprehensive. The experts individually evaluated the questionnaire, answering the questions proposed for approximately one hour. This expert evaluation served as a form of questionnaire validation through observation. Secondly, the questionnaire was administered to students enrolled in introductory programming courses to collect data for analysis. This step allowed for the exploration of students’ perceptions and experiences related to self-regulation, co-regulation, and shared regulation in programming. Thirdly, to assess the reliability of the questionnaire, the Cronbach’s alpha statistical test (Cronbach 1951) was applied. This test was used to examine the internal consistency of the questions pertaining to self-regulation, as well as those addressing co-regulation

Regulatory Strategies for Novice Programming Students

143

and shared regulation, respectively. The goal was to determine whether the questions within each construct exhibited consistent measurement properties. Fourthly, drawing on models of self-regulation, co-regulation, and socially shared learning regulation, 8 (eight) scripts were developed for teachers to use for teaching regulatory strategies. The scripts involve strategies embedded evidence-based principles of programming learning. The self-regulation questionnaire is outlined in Table 1, while the co-regulation and socially shared regulation questionnaire can be found in Table 2. Descriptive statistics were employed to analyze the data obtained from the questionnaires. Additionally, the experts provided responses to a set of Yes/No questions, which are summarized in Table 3. Table 1. Self-regulation questions. 1. During the programming course, did I monitor my performance and try to overcome any obstacles? 2. Did I motivate myself to participate in all individual and group programming activities, even when there was not much interest on my part? 3. Did I use as motivation the fact that programming is important for my course and my future profession? 4. Did I seek help from classmates or the teacher when I couldn’t solve a programming problem? 5. Have I found ways to focus on programming even when there are sources of distraction? 6. Have I used time management strategies and managed to finish my programs? 7. Did I use the “divide and conquer” strategy by thinking about each part of the program in different modules? 8. Did I try to remain confident during programming, telling myself that I could do it? 9. When studying introduction to computing, did I look for different sources of information? 10. What study sources did you use? 11. Have I used sketches, diagrams or other types of drawings or sketches to organize my ideas about the logic of programming before coding? 12. Have I thought of different code alternatives for the same computational problem? 13. Did I review the lectures or look for supplementary material when I could not make a program of the practical class? 14. When studying Introduction to Computing, did I set goals for myself to direct my activities in each study period? 15. Did I adapt and match programming patterns when coding my programs? 16. Did I make an effort to participate in the practical classes?

144

D. J. Ferreira et al. Table 2. Co-regulation and socially shared regulation questions.

1. With respect to computational solutions, have I tried to question the teacher and colleagues looking for evidence? 2. Did you use social media and other forms of technology to communicate with classmates? 3. What communication and collaboration technologies did you use during the course? 4. In group projects, did I try to motivate colleagues so that everyone contributed to the construction of the programs? 5. Did I contribute to a good working atmosphere during the joint programming, facing difficulties with good humor? 6. Have I valued colleagues’ code parts and contributed to improvements? 7. Have I treated my colleagues with respect and used positive phrases such as “Very good! Keep it up! Thank you! You’ve helped us a lot now!”? 8. Have I tried to reconcile your goals, priorities and learning style with those of my colleagues? 9. Was the group work organized together, trying to reconcile the preferences of the members? 10. Was any time management strategy used for group projects, such as Kanban or Scrum? 11. Was any tool used to manage collaborative programming, such as Trello or GitHub? 12. Did the group use the “divide and conquer” strategy by thinking about each part of the program in different modules? 13. In group projects, was the commitment of everyone in the group to compliance with the rules and participation in programming activities monitored and action taken if necessary? 14. In group projects, were roles assigned to be played by students during the writing of the program, such as writer, consultant, editor and reviewer? 15. Was any joint programming strategy used, such as the Coding Dojo? 16. In group programming projects, was there reflection on the quality of interactions and group performance, and action taken when necessary? 17. Have group interactions positively influenced my personal performance?

Table 3. Questionnaire for experts’ analysis. Are the questions simple, clear, easy to understand? Do the questions cover relevant aspects of self-regulation, co-regulation and shared regulation in programming? Are the questions parsimonious enough to ignore irrelevant aspects, but do they sufficiently cover self-regulation, co-regulation and shared regulation strategies?

Regulatory Strategies for Novice Programming Students

145

4 Results 4.1 Validity and Reliability of the Questionnaire Applied to the Students The first result pertains to the evaluation of the questionnaire by specialists. The validation process, conducted by the six experts, yielded highly positive outcomes. All six experts responded “Yes” to the three evaluation questions presented to them, as shown in Table 3. The second result focuses on the internal consistency of the questionnaire. Internal consistency examines the reliability of the cumulative scores derived from a Likert scale. It assesses the degree of compatibility and correlation among responses to multiple items within the Likert scale. To assess internal consistency, the Cronbach’s alpha statistical test was applied to the first part of the questionnaire, which pertains to students’ self-regulation, in order to determine the interrelatedness of the questions. Additionally, the Cronbach’s alpha test was applied to the second part of the questionnaire, covering students’ co-regulation and shared regulation, to assess the coherence of the questions. The interpretation of the Cronbach’s alpha coefficient can be found in Table 4. Table 4. Cronbach’s alpha coefficient interpretation (Cronbach 1951). 0.9