Computer Supported Education: 14th International Conference, CSEDU 2022, Virtual Event, April 22–24, 2022, Revised Selected Papers (Communications in Computer and Information Science) 3031405005, 9783031405006

Table of contents :
Preface
Organization
Contents
A Descriptive and Historical Review of STEM + C Research: A Bibliometric Study
1 Introduction
2 Research Background
2.1 A Battle of Definition
2.2 Related Work
3 Methods
3.1 Data
3.2 Analysis
4 Results
4.1 RQ 1. What are the Current Trends in Number of Publications and Citations in STEM + C Research?
4.2 RQ 2. What are the Most Prolific Countries/Regions, Institutions, Resources, and Authors in STEM + C Research?
4.3 RQ 3. What are the Collaboration Patterns in STEM + C Research?
4.4 RQ 4. What are the Most Mentioned Keywords and Their Evolving Trends in STEM + C Research?
4.5 RQ 5. What are the Potential Research Directions Based on Current Literature in STEM + C Research?
5 Discussion
5.1 Trends
5.2 Potential Topics for Future Research
5.3 The Role of CT or CS Education in STEM + C Research
5.4 Limitation
6 Conclusion
References
An Exploratory Test Design and Execution Learning Approach: A Definition of Syllabus and Teaching Plan
1 Introduction
2 Background
2.1 Teaching Exploratory Test Design and Execution
2.2 Curriculum for Supporting
2.3 Active Methodologies
3 Research Methodology
3.1 Step 1: Literature Review
3.2 Step 2: Analysis of Assets from Curricula and Guide for Software Testing
3.3 Step 3: Perception of Professionals Regarding Support Resources for Exploratory Testing
3.4 Step 4: Construction of Syllabus
3.5 Step 5: Definition of Teaching Plan
4 Related Works
5 Syllabus
6 Teaching Plan
7 Evaluation of the Approach
8 Conclusion
Appendix A
References
Understanding Geolocation Data: Learning Scenarios for School Informatics
1 Introduction
2 Learning Scenarios for Understanding Geolocation Data
2.1 Design Methodology
2.2 Evaluation Methodology
3 Exploring and Recording Geolocation Data
3.1 Objectives
3.2 Design of the Learning Scenario
3.3 Results of Internal Quality Evaluation
3.4 Results of External Quality Evaluation
3.5 Discussion
4 Playing and Analyzing a Location-Based Game
4.1 Objectives
4.2 Design of the Learning Scenario
4.3 Results of Internal Quality Evaluation
4.4 Results of External Quality Evaluation
4.5 Discussion
References
Game Design Tools: A Systematic Literature Review: Choice of a Game Design Tool for an Experimentation in the Nursing Field
1 Introduction
2 Related Work
3 The PRISMA Method
3.1 Developing a Review Protocol
3.2 Identification of the Need for a Review
3.3 Specifying the Research Question(s)
3.4 Identification of Research, and Selection of Studies
3.5 Data Extraction
3.6 Data Synthesis, and Reporting the Review
4 Discussion
5 Experimentation
6 Conclusion and Perspectives
References
Development and Evaluation of a Trusted Achievement Record of Accomplishments for Students in Higher Education Using Blockchain
1 Introduction
2 Related Work
3 Conceptual Model Design and Implementation
3.1 DApp Layer
3.2 Blockchain Layer
3.3 Smart Contract
3.4 Transaction and Gas
3.5 Implementation
4 System Use-Cases
4.1 Use-Case: Admin
4.2 Use-Case: University
4.3 Use-Case: Student
4.4 Use-Case: Employer
5 Evaluation
5.1 System Usability Scale (SUS)
5.2 Assessing the Motivation and Utility of Learning
5.3 Transactions Confirmation Time and Transactions Cost
5.4 Comparison of the Proposed System with the CVSS System
6 Conclusion
References
Students' Perceptions of Computer Science and the Role of Gender
1 Introduction
2 Early STEM
3 Theoretical Background
4 Methodology
5 Preliminary Results
5.1 Definitions of Computer Science
5.2 Tasks of Computer Scientists
5.3 Reasons for (not) Pursuing a Career in Computer Science
6 Results
6.1 Students' Interest in CS and the Aim of Working in CS
6.2 Students' Aim of Working in CS and Role Models in CS
6.3 Students' Interest in CS and Role Models in CS
7 Summary
8 Discussion
References
Adaptive Kevin: A Multipurpose AI Assistant for Higher Education
1 Introduction
2 Literature Review
3 Architecture Design
4 Quantitative and Qualitative Assessment
4.1 Quantitative Assessment
4.2 Qualitative Assessment
4.3 Discussion
5 Conclusion and Future Work
References
Comparing Multi-objective GA and PSO for the Pedagogical Activities Sequencing from Bloom's Digital Taxonomy
1 Introduction
2 Related Works
3 Background
3.1 RASI Profile
3.2 Bloom's Taxonomy
3.3 Relationship Between RASI and BT
4 Problem Formulation
4.1 Solution Representation
4.2 Objective Functions
4.3 Multiobjective Binary PSO
4.4 Multiobjective GA
5 Experiments, Results and Discussion
5.1 Students' Profiles
5.2 Sequencing Analysis
6 Conclusions
References
Author Index

Citation preview

James Uhomoibhi (Ed.)

Communications in Computer and Information Science

1817

Computer Supported Education 14th International Conference, CSEDU 2022 Virtual Event, April 22–24, 2022 Revised Selected Papers

Communications in Computer and Information Science Editorial Board Members Joaquim Filipe , Polytechnic Institute of Setúbal, Setúbal, Portugal Ashish Ghosh , Indian Statistical Institute, Kolkata, India Raquel Oliveira Prates , Federal University of Minas Gerais (UFMG), Belo Horizonte, Brazil Lizhu Zhou, Tsinghua University, Beijing, China

1817

Rationale The CCIS series is devoted to the publication of proceedings of computer science conferences. Its aim is to efficiently disseminate original research results in informatics in printed and electronic form. While the focus is on publication of peer-reviewed full papers presenting mature work, inclusion of reviewed short papers reporting on work in progress is welcome, too. Besides globally relevant meetings with internationally representative program committees guaranteeing a strict peer-reviewing and paper selection process, conferences run by societies or of high regional or national relevance are also considered for publication. Topics The topical scope of CCIS spans the entire spectrum of informatics ranging from foundational topics in the theory of computing to information and communications science and technology and a broad variety of interdisciplinary application fields. Information for Volume Editors and Authors Publication in CCIS is free of charge. No royalties are paid, however, we offer registered conference participants temporary free access to the online version of the conference proceedings on SpringerLink (http://link.springer.com) by means of an http referrer from the conference website and/or a number of complimentary printed copies, as specified in the official acceptance email of the event. CCIS proceedings can be published in time for distribution at conferences or as postproceedings, and delivered in the form of printed books and/or electronically as USBs and/or e-content licenses for accessing proceedings at SpringerLink. Furthermore, CCIS proceedings are included in the CCIS electronic book series hosted in the SpringerLink digital library at http://link.springer.com/bookseries/7899. Conferences publishing in CCIS are allowed to use Online Conference Service (OCS) for managing the whole proceedings lifecycle (from submission and reviewing to preparing for publication) free of charge. Publication process The language of publication is exclusively English. Authors publishing in CCIS have to sign the Springer CCIS copyright transfer form, however, they are free to use their material published in CCIS for substantially changed, more elaborate subsequent publications elsewhere. For the preparation of the camera-ready papers/files, authors have to strictly adhere to the Springer CCIS Authors’ Instructions and are strongly encouraged to use the CCIS LaTeX style files or templates. Abstracting/Indexing CCIS is abstracted/indexed in DBLP, Google Scholar, EI-Compendex, Mathematical Reviews, SCImago, Scopus. CCIS volumes are also submitted for the inclusion in ISI Proceedings. How to start To start the evaluation of your proposal for inclusion in the CCIS series, please send an e-mail to [email protected].

James Uhomoibhi Editor

Computer Supported Education 14th International Conference, CSEDU 2022 Virtual Event, April 22–24, 2022 Revised Selected Papers

Editor James Uhomoibhi University of Ulster Newtownabbey, UK

ISSN 1865-0929 ISSN 1865-0937 (electronic) Communications in Computer and Information Science ISBN 978-3-031-40500-6 ISBN 978-3-031-40501-3 (eBook) https://doi.org/10.1007/978-3-031-40501-3 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

The present book includes extended and revised versions of a set of selected papers from the 14th International Conference on Computer Supported Education (CSEDU 2022), which was exceptionally held as an online event, due to COVID-19, from 22 to 24 April 2022. CSEDU 2022 received 181 paper submissions from 38 countries, of which 4% were included in this book. The papers were selected by the event chairs and their selection is based on a number of criteria that include the classifications and comments provided by the program committee members, the session chairs’ assessment and also the program chairs’ global view of all papers included in the technical program. The authors of selected papers were then invited to submit a revised and extended version of their papers having at least 30% innovative material. CSEDU, the International Conference on Computer Supported Education, is a yearly meeting place for presenting and discussing new educational tools and environments, best practices and case studies on innovative technology-based learning strategies, and institutional policies on computer supported education including open and distance education. CSEDU will provide an overview of current technologies as well as upcoming trends, and promote discussion about the pedagogical potential of new educational technologies in the academic and corporate world. CSEDU seeks papers and posters describing educational technology research; academic or business case-studies; or advanced prototypes, systems, tools, and techniques. The papers selected to be included in this book contribute to the understanding of relevant trends of current research in Computer Supported Education, including: Emerging Technologies in Education for Sustainable Development, Instructional Design, Pre-K/K12 Education, Machine Learning, Learning with AI Systems, Higher-Order Thinking Skills, Game-Based and Simulation-Based Learning, Educational Data Mining, Course Design and eLearning Curriculae, and Constructivism and Social Constructivism. We would like to thank all the authors for their contributions and also the reviewers who have helped to ensure the quality of this publication. April 2022

James Uhomoibhi

Organization

Conference Co-chairs Bruce McLaren James Uhomoibhi

Carnegie Mellon University, USA Ulster University, UK

Program Co-chairs Mutlu Cukurova Nikol Rummel Denis Gillet

University College London, UK Ruhr University Bochum, Germany École Polytechnique Fédérale de Lausanne, Switzerland

Program Committee Nelma Albuquerque Eleftheria Alexandri António Andrade Francisco Arcega Juan Ignacio Asensio Breno Azevedo Ralph-Johan Back Jorge Barbosa João Barros Patrícia Bassani Zane Berge Jesús Berrocoso Andreas Bollin Laurent Borgmann Ivana Bosnic Federico Botella Karima Boussaha Patrice Bouvier Krysia Broda

Concepts and Insights, Brazil Hellenic Open University, Greece Universidade Católica Portuguesa, Portugal Universidad de Zaragoza, Spain University of Valladolid, Spain Instituto Federal de Educação, Ciência e Tecnologia Fluminense, Brazil Åbo Akademi University, Finland UNISINOS, Brazil Polytechnic Institute of Beja, Portugal Universidade Feevale, Brazil University of Maryland, Baltimore County, USA University of Extremadura, Spain Klagenfurt University, Austria Koblenz University of Applied Sciences, Germany University of Zagreb, Croatia Miguel Hernández University of Elche, Spain University of Oum El Bouaghi, Algeria LDLC VR Studio, France Imperial College London, UK

viii

Organization

Marina Buzzi Manuel Caeiro Rodríguez Renza Campagni Pasquina Campanella Sanja Candrlic Isabel Chagas António Coelho Gennaro Costagliola Manuel Perez Cota John Cuthell Rogério da Silva Sergiu Dascalu Luis de-la-Fuente-Valentín Christian Della Tania Di Mascio Yannis Dimitriadis Amir Dirin Danail Dochev

Toby Dragon Nour El Mawas Gijsbert Erkens Larbi Esmahi João Esteves Vladimir Estivill Ramon Fabregat Gesa Si Fan Michalis Feidakis Richard Ferdig Rosa Fernandez-Alcala Débora Nice Ferrari Barbosa Giuseppe Fiorentino Judith Gal-Ezer Francisco García Peñalvo Isabela Gasparini Henrique Gil Apostolos Gkamas

CNR, Italy University of Vigo, Spain Università di Firenze, Italy University of Bari “Aldo Moro”, Italy University of Rijeka, Croatia Universidade de Lisboa, Portugal Faculdade de Engenharia da Universidade do Porto, Portugal Università di Salerno, Italy Universidade de Vigo, Spain Virtual Learning, UK University of Leicester, UK University of Nevada, Reno, USA Universidad Internacional de la Rioja, Spain University of Glasgow, Singapore University of L’Aquila, Italy University of Valladolid, Spain Metropolia University of Applied Science, Finland Institute of Information and Communication Technologies, Bulgarian Academy of Sciences, Bulgaria Ithaca College, USA Université de Lille, France Utrecht University, The Netherlands Athabasca University, Canada University of Minho, Portugal Universitat Pompeu Fabra, Spain Universitat de Girona, Spain University of Tasmania, Australia University of West Attica, Greece Kent State University, USA University of Jaen, Spain Feevale University, Brazil University of Pisa, Italy Open University of Israel, Israel Salamanca University, Spain Universidade do Estado de Santa Catarina, Brazil Escola Superior de Educação do Instituto Politécnico de Castelo Branco, Portugal University Ecclesiastical Academy of Vella of Ioannina, Greece

Organization

Anabela Gomes Cristina Gomes Maria João Gomes Ana González Marcos Anandha Gopalan Christiane Gresse von Wangenheim Christian Guetl Raffaella Guida David Guralnick Roger Hadgraft Antonio Hervás Jorge Tomayess Issa Ivan Ivanov Malinka Ivanova M. J. C. S. Reis Hannu-Matti Järvinen Stéphanie Jean-Daubias M.-Carmen Juan Michail Kalogiannakis Atis Kapenieks Charalampos Karagiannidis Ilias Karasavvidis Michael Kerres Lam-for Kwok Eitel Lauría Borislav Lazarov José Leal Chien-Sing Lee Marie Lefevre Maria Beatrice Ligorio Andreas Lingnau Luca Andrea Ludovico Susie Macfarlane Veronika Makarova Maria Marcelino Ivana Marenzi

ix

Instituto Superior de Engenharia de Coimbra, Portugal Instituto Politécnico de Viseu, Portugal Universidade do Minho, Portugal Universidad de la Rioja, Spain Imperial College London, UK Federal University of Santa Catarina, Brazil Graz University of Technology, Austria University of Surrey, UK Kaleidoscope Learning, USA University of Technology Sydney, Australia Universitat Politècnica de València, Spain Curtin University, Australia SUNY Empire State University, USA Technical University of Sofia, Bulgaria University of Trás-os-Montes e Alto Douro, Portugal Tampere University, Finland Université Claude Bernard Lyon 1, LIRIS, France Instituto Ai2, Universitat Politècnica de València, Spain University of Crete, Greece Riga Technical University, Latvia University of Thessaly, Greece University of Thessaly, Greece University of Duisburg-Essen, Germany HKCT Institute of Higher Education, China Marist College, USA Institute of Mathematics and Informatics, Bulgarian Academy of Sciences, Bulgaria University of Porto, Portugal Sunway University, Malaysia University Claude Bernard Lyon 1, France University of Bari, Italy Ruhr West University of Applied Sciences, Germany Università degli Studi di Milano, Italy Deakin University, Australia University of Saskatchewan, Canada UC, Portugal Leibniz University Hannover, Germany

x

Organization

Lindsay Marshall Scheila Martins Bruce Maxim Madeth May Elvis Mazzoni José Carlos Metrôlho Laurent Moccozet Gyöngyvér Molnár Rafael Morales Gamboa António Moreira Jerzy Moscinski Maria Moundridou Antao Moura Antoanela Naaji Ryohei Nakatsu Minoru Nakayama Fátima Nunes Dade Nurjanah Ebba Ossiannilsson José Palma Paolo Paolini Stamatios Papadakis Pantelis Papadopoulos Kyparisia Papanikolaou Iraklis Paraskakis Arnold Pears Emanuel Peres Paula Peres Donatella Persico Alfredo Pina Matthew Poole Elvira Popescu Francesca Pozzi Augustin Prodan Yannis Psaromiligkos Fernando Ramos

Newcastle University, UK Arden University, UK University of Michigan-Dearborn, USA Le Mans Université, France University of Bologna, Italy Instituto Politécnico de Castelo Branco, Portugal University of Geneva, Switzerland University of Szeged, Hungary University of Guadalajara, Mexico Universidade de Aveiro, Portugal Silesian University of Technology, Poland School of Pedagogical and Technological Education (ASPETE), Greece Federal University of Campina Grande, Brazil Vasile Goldis, Western University of Arad, Romania Kyoto University, Japan Tokyo Institute of Technology, Japan Universidade de São Paulo, Brazil Telkom University, Indonesia Swedish Association for Distance Education, Sweden Escola Superior de Tecnologia de Setúbal, Portugal Politecnico di Milano, Italy University of Crete, Greece University of Twente, The Netherlands School of Pedagogical and Technological Education (ASPETE), Greece South-East European Research Centre, Greece KTH Royal Institute of Technology, Sweden University of Trás-os-Montes e Alto Douro/Inesc-Tec, Portugal ISCAP, Portugal CNR - Italian National Research Council, Italy Public University of Navarre, Spain University of Portsmouth, UK University of Craiova, Romania CNR - Italian National Research Council, Italy Iuliu Hatieganu University of Medicine and Pharmacy, Romania University of West Attica, Greece University of Aveiro, Portugal

Organization

Fernando Ribeiro Marco Ronchetti Leon Rothkrantz Razvan Rughinis Rebecca Rutherfoord Demetrios Sampson Eduardo Santos Juan M. Santos Anthony Savidis Georg Schneider Wolfgang Schreiner Maria Serna Sabine Seufert Ramesh C. Sharma Pei Hwa Siew Jane Sinclair Eliza Stefanova Claudia Steinberger Katsuaki Suzuki Nestori Syynimaa Qing Tan Seng Chee Tan Dirk Tempelaar Uwe Terton Aristides Vagelatos Michael Vallance Leo van Moergestel Carlos Vaz de Carvalho Alf Wang Leandro Wives Stelios Xinogalos Diego Zapata-Rivera Thomas Zarouchas Iveta Zolotova

xi

Instituto Politécnico de Castelo Branco, Portugal University of Trento, Italy Delft University of Technology, The Netherlands University “Politehnica” of Bucharest, Romania Kennesaw State University, USA University of Piraeus, Greece University of São Paulo, Brazil University of Vigo, Spain Institute of Computer Science (FORTH) & University of Crete, Greece Trier University of Applied Sciences, Germany Johannes Kepler University Linz, Austria Barcelona Tech, Spain University of St.Gallen, Switzerland Dr. B. R. Ambedkar University Delhi, India Universiti Tunku Abdul Rahman, Malaysia University of Warwick, UK Sofia University, Bulgaria University of Klagenfurt, Austria Kumamoto University, Japan University of Jyväskylä, Finland Athabasca University, Canada National Institute of Education, Nanyang Technological University, Singapore Maastricht University School of Business and Economics, The Netherlands University of the Sunshine Coast, Australia Computer Technology Institute, Greece Future University Hakodate, Japan HU Utrecht University of Applied Sciences, The Netherlands ISEP, Portugal Norwegian University of Science and Technology, Norway Universidade Federal do Rio Grande do Sul, Brazil University of Macedonia, Greece Educational Testing Service, USA Computer Technology Institute and Press “Diophantus”, Greece Technical University in Kosice, Slovak Republic

xii

Organization

Additional Reviewers Martina Asenbrener Katic Federica Caruso Pedro Rito

University of Rijeka, Croatia University of L’Aquila, Italy Polytechnic Institute of Viseu, Portugal

Invited Speakers Ton de Jong Erin Walker Katrien Verbert Cristobal Romero Morales

University of Twente, The Netherlands University of Pittsburgh, USA KU Leuven, Belgium University of Cordoba, Spain

Contents

A Descriptive and Historical Review of STEM + C Research: A Bibliometric Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Hanxiang Du, Wanli Xing, Bo Pei, Yifang Zeng, Jie Lu, and Yuanlin Zhang

1

An Exploratory Test Design and Execution Learning Approach: A Definition of Syllabus and Teaching Plan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Igor Ernesto Ferreira Costa and Sandro Ronaldo Bezerra Oliveira

26

Understanding Geolocation Data: Learning Scenarios for School Informatics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Viera Michaliˇcková and Gabriela Lovászová

51

Game Design Tools: A Systematic Literature Review: Choice of a Game Design Tool for an Experimentation in the Nursing Field . . . . . . . . . . . . . . . . . . . . Sebastian Gajewski, Nour El Mawas, and Jean Heutte

81

Development and Evaluation of a Trusted Achievement Record of Accomplishments for Students in Higher Education Using Blockchain . . . . . . 100 Bakri Awaji, Ellis Solaiman, and Adel Albshri Students’ Perceptions of Computer Science and the Role of Gender . . . . . . . . . . . 125 Sara Hinterplattner Adaptive Kevin: A Multipurpose AI Assistant for Higher Education . . . . . . . . . . 149 Augusto Gonzalez-Bonorino and Eitel J. M. Lauría Comparing Multi-objective GA and PSO for the Pedagogical Activities Sequencing from Bloom’s Digital Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 Denis José Almeida, Newarney Torrezão da Costa, and Márcia Aparecida Fernandes Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

A Descriptive and Historical Review of STEM + C Research: A Bibliometric Study Hanxiang Du1 , Wanli Xing2(B) , Bo Pei4 , Yifang Zeng3 , Jie Lu2 , and Yuanlin Zhang3 1

Western Washington University, Bellingham, WA, USA [email protected] 2 University of Florida, Gainesville, FL, USA [email protected], [email protected] 3 Texas Tech University, Lubbock, TX, USA {y.zhang,yifang.zeng}@ttu.edu 4 University of South Florida, Tampa, FL, USA [email protected]

Abstract. Since Jeannette Wing conceptualized the term “computational thinking” in 2006, calls to prompt computational thinking and computing education have earned a huge number of advocates. Their integration with STEM majors also gained substantial research interests. This work set to overview existing literature in STEM + C. Specifically, we extend our previous research on reviewing the literature to provide more information such as research background, trends in collaboration and potential topics. We searched three academic databases and one scholarly search engine, identifying 202 publications for our analysis. Common bibliometric indicators were used, as well as social network analysis and text mining techniques. Results reported trends in publication and collaboration, topics of interest in the field and their dynamics, methodologies in which how STEM + C was integrated, as well as potential future research directions. This study contributes to the field by systematically examining related literature, providing an overview of the STEM + C field, and introducing data mining techniques to bibliometric analysis. Keywords: Computing education Bibliometric

1

· STEM · Social network analysis ·

Introduction

Computer science is one of the largest growing fields worldwide. The U.S. Bureau of Labor Statistics projects that computing occupations are the number one source of all new wages in the U.S. and make up over half of all projected new jobs in STEM fields [1]. Although students can major or take classes in Computer Science (CS) in college, it is still too late as they may have developed ill-informed attitudes. Therefore, it is critical that CS be introduced at an earlier stage [2]. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2023  J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 1–25, 2023. https://doi.org/10.1007/978-3-031-40501-3_1

2

H. Du et al.

Early exposure to CS can help students develop problem-solving skills, cultivate computational literacy, develop a career path, as well as prepare students to be educated citizens of modern society [3–5]. There has been an increasing interest in prompting CS Education at K-12 schools. One way is to integrate CS with existing science, technology, engineering, mathematics (STEM) courses, which is also known as STEM + C. In relation to teaching practice, CS has been integrated with STEM in various ways: plugged or unplugged, using block-based or text-based programming learning environment, teaching computational thinking (CT) or computational literacy, and so on. National Science Foundation (NSF) interprets STEM + C as an integration of CT to STEM disciplines, rather than computing education [6]. In addition, National Research Council (NRC) reported a number of perspectives on the definition and applicability of CT [7]. Despite the different perspectives on STEM + C or CT, STEM + C remains an active field of research in the past decades. Educators and researchers devote to the field by studying how to integrate CS with STEM courses, how to prompt CS in K-12 schools, and how to assess the intervention. It is worth examining the existing literature and providing an overview of the field. Bibliometric analysis, which uses statistical analysis to systematically extract measurable features from publications , is commonly used for the study of qualitative features and research performance, especially for large quantities of publications [8,9]. It is able to measure the importance of publications, decompose the evolution of a research topic, and identify potential research topics by mining various data such as the number of publications, citations, and h-index. It has been shown as a reliable and useful tool to overview the existing literature of a research field [9–11]. We extend our previous work [11] on examining the literature by including more introduction on the research context, new analysis methods such as social network analysis, and more analysis on collaboration trends and potential topics. Specifically, we applied social network analysis, content analysis, and biliometric analysis to explore existing STEM + C literature, aiming to provide valuable references for researchers. Our work addresses the following research questions (RQ): RQ 1. What are the current trends in number of publications and citations in STEM + C research? RQ 2. What are the most prolific countries/regions, institutions, resources, and authors in STEM + C research? RQ 3. What are the collaboration patterns in STEM + C research? RQ 4. What are the most mentioned keywords and their evolving trends in STEM + C research? RQ 5. What are the potential research directions based on current literature in STEM + C research?

2 2.1

Research Background A Battle of Definition

Before diving into the literature, it is necessary to clarify a few terms: STEM, STEM + C, and CT. However, the goal is not to exhaust various definitions

A Descriptive and Historical Review of STEM + C Research

3

of the terms, but to clarify the scope of our work, as well as to provide our perceptions and rationales for this study. There has been heated discussions on the definition of STEM and CT. STEM, or its alternative versions STEAM and STREAM, stands for a combination of disciplines: Science, Technology, Engineering, Math, Arts, and Reading. However, there is no agreement on the relationship among these subjects. Some argue that STEM is merely four isolated subjects, while others consider STEM as an integrated continuum of multidisciplinary elements [12,13]. Even for those who consider STEM as an integrated field, the discussion on the relationships and conceptual frameworks for learning among science, technology, engineering and mathematics remains unresolved [13,14]. Jeannette Wing [4] argued that CT represents “a universally applicable attitude and skill set everyone, not just computer scientists, would be eager to learn and use” (p.33). To Wing, CT is a fundamental skill that involves problem solving, system design and understanding human behaviour. According to the Royal Society [15], computational thinking is “a way of thinking an working that provides a perspective on the world that is distinct from other disciplines” (p.19), including a number of components such as modelling, decomposition, generalization, designing solutions, testing, and debugging. The work of Barr and Stephenson [3] identified another list of core computational thinking concepts, including data collection, data analysis, data representation, problem decomposition, abstraction, algorithms & procedures, automation, parallelization, and simulation. Despite all these discrepancy on CT’s definition, conceptual framework, and key elements, most literature agreed on one thing: CT is an important skill for K-12 students to develop, and is related to computing or programming practice. In particular, CT is generally considered to be a different skill from programming, while programming is commonly used to teach CT [20]. NSF’s definition on STEM + C (i.e., an integration of CT to STEM disciplines) consists two C components: computing education and CT education. This definition explicitly indicates that computing is a concept different from, but related to CT. Computing education is considered to be a broad term that may include one or more of the following areas: computer science, technology literacy or fluency, and information technology. According to Wing [16], computing is an integrated field involving computer science, information science, computer engineering and information technology. A survey study which examines how computing is taught worldwide also reports that computer science and information technology are normally categorized as computing disciplines [17]. In addition, Psycharis [18] pointed out that “computing” is often used interchangeably with “computation” and “computational” in the context of programming or calculations. So are the terms “computing education” and “computer science education” [19]. The goal of this work is to examine existing literature in STEM + C, providing an overview of the filed using a bibliometric analysis perspective. In this study, we will consider computing education as a broad term that consists of, but is not limited to, computing education. As discussions on CT’s definition and components continue, this work keeps an open mind on related publications

4

H. Du et al.

and their adopted conceptual frameworks. We do not exclude any studies based on a particular definition of computing education or CT. 2.2

Related Work

Several review studies have examined CT, computing education, and their integration from various perspectives. The work of Robins and colleagues [21] reviewed literature relating to the educational study in programming. Their focus is on novice programming and topics related to teaching and learning practice. The study of Grover and Pea [22] examined existing empirical studies to review how CT was prompted in research and teaching practice. In particular, they reviewed a number of definitions on CT and demonstrated how these concepts were related to “computational literacy” or “procedural literacy”. They also investigated research and teaching practice in CT or computing education in past decades. Their focus has involved a large number of studies in computing education, rather than STEM education. Later, Garneli [19] conducted a review study on the educational contexts, efficient instructional tools and practices in K-12 CS education. They examined 47 peer-reviewed articles and made insightful recommendations. Recently, a longitudinal study collected over 500 publications on K-12 CT research from 2012 to 2018. They studied the curriculum content, students’ grade levels, and the way in which computing education was delivered. In relation to assessing CT, Tang and colleagues [23] systematically reviewed 96 journal articles to analyze specific CT assessments with a focus on educational context, assessment type, assessment construct, and validity evidence. A few other studies focused on the CT integration practice in the context of mathematics and science education. Weintrop and colleagues [24] proposed a definition of CT for high school mathematics and science education in the form of a taxonomy. In particular, they reviewed discussions on CT’s definition and its crucial connection to science and mathematics learning, as well as CT-promoting practices at K-12 schools. They also discussed how the definition and taxonomy can be used to prompt CT. Their taxonomy consists of four categories, including data, modeling and simulation, computational problem solving, and systems thinking practices. Focusing on studies prompting CT via mathematics learning, Barcelos and colleagues [25] collected 42 empirical studies with an experimental design specifically aiming at developing CT skills. Their work systematically reviewed these learning interventions from the perspectives of instructional tools and materials, experimental designs, assessments, and reported achievements. Hickmott and colleagues [26] were interested in how CT and mathematical learning were connected. They searched 6 databases and identified 393 peer-reviewed articles on CT in K-12 education. They found that most studies originated from computer science academics rather than education experts. Mathematics is a learning context, whilst teaching focused on programming skills. In addition, mathematics domain concepts such as probability, statistics, measurement or functions were rarely involved. It seems that existing work on STEM and CT mainly focus on either developing conceptual frameworks, or investigated instructional and pedagogical

A Descriptive and Historical Review of STEM + C Research

5

strategies. In their work of CT-STEM, Jona and colleagues [27] actively called educators to bring CT learning activities to students by embedding CT within students’ ongoing STEM coursework. Similarly, Swaid [28] reported an innovative and comprehensive project bringing CT to STEM disciplines. They adopted a CT-based strategy to enforce CT in STEM gate-keeping courses like the introductory level courses of STEM and computer science. Leonard and colleagues [29] reported their pilot study to develop middle school students’ CT strategies using robotics and game design. This is one of the few studies that prompt CT learning in engineering learning environments. Psycharis [18] outlined various research and practices for STEAM integration, and proposed a new framework called Computational STEAM Pedagogy. Although the role of CT, computing education, and their integration was mentioned, they swiftly shift to their focus on the new integration framework. Sengupta and colleagues [30] presented a critical review of literature in educational computing, focusing on current views on CT and arguing for an epistemological transfer. They argued CT, which used to be viewed as a subordinate skill, has become a unique, distinctive and comprehensive skill set. Recently, a few other studies focused on students’ common errors, misconceptions in abstraction, and debugging practices in high school mathematics or science integrated courses [31–33]. However, none of these works focus exclusively on STEM + C using quantitative analysis methods.

3

Methods

In this section, we first introduce how we collected data for the analysis (see Sect. 3.1), then our analysis methods (see Sect. 3.2). 3.1

Data

One big challenge in literature search is to identify key terms. The debate on both STEM and CT or computing education leads to a broad extension and variation of terms. The terms are supposed to maximize the search scope while being efficient. In order to investigate existing literature in STEM + C research, we consider STEM, CT, and computing education as broad terms and do not exclude articles due to definition. Table 1 presents the search terms for title and abstract domain. A search term is a combination of x AND y AND z, where x represents terms for STEM, y for C, and z for education. Term “STEM” leads to a large number of biology and medical articled, hence, was excluded. Term “computer” was excluded from title domain as “computer” will result in a combination of “computer science”, leading to thousands of articles that fall beyond our research focus. Terms “computer science” and “CS” were also excluded due to the same reason.

6

H. Du et al. Table 1. Search terms [11]. Field

Key Terms

x Title

STEM, science, technology, engineering, math, biology, chemistry, physics

y Title

computational thinking, programming, computing

z Abstract learn, course

Figure 1 visualizes the data collection process consisting of two rounds of search. The first round search involved three academic databases: Web of Science, IEEE Xplore, and ACM Digital Library. Web of Science is a well-known database of high-quality academic work and widely used to search articles for systematic review and bibliometric studies. While IEEE Xplore and ACM have a number of conference proceedings dedicated to computing education. We used search terms and conducted a systematic search, and collected 2855 articles. Then, we manually checked these publications. Publications were included in this study if they are: (1). Peer-reviewed journal articles or conference proceedings written in English. (2). Content and topics falling into STEM + C research with a focus on integration practice. Both empirical and theoretical studies were included as long as they meet the criteria. This process resulted in a total of 56 articles.

Fig. 1. Data collection process. [11]

The second-round search is supplementary in nature, aiming to maximize the retrieval and identification of related articles that may have been missed during the first-round search. Although academic databases cover a large number of journals or conference proceedings, the coverage is determined by humans. Google Scholar is a scholarly search engine that connects to the entire Internet, covering more valuable records that cannot be found on Web of Science [34].

A Descriptive and Historical Review of STEM + C Research

7

Due to the computational settings, search results cannot be replicated in Google Scholar even using the same search terms. To ensure a systematic and consistent search process, we only used Google Scholar in the second-round search. During the second-round search, we used both search term combinations and “STEM + C”. In addition, we also conducted forward and backward search based on first-round search results. We checked articles which cited identified articles, as well as those that were cited by the 56 records. It turned out that we have collected much more articles in the second-round search. Eventually, we identified a total of 202 records published by March, 2020. 3.2

Analysis

To answer RQ 1 & RQ2, we used commonly bibliometric indicators, conducted sleeping beauties analysis, and plotted figures to visualize the results. To address RQ 3, we employed social network analysis to create a social network graph to present the collaboration patterns. For RQ 4 & RQ 5, we processed data and developed a topic modeling model to understand the publication topics at document level, and used keywords dynamics to measure the dynamics over time. Bibliometric Indicators. A number of commonly used bibliometric indicators were used to measure the impact of collected articles, as shown in Table 2. Publication count and citation count are used to assess productivity and influence, while h-index is used to measure the level of scientific achievement [35]. We collected authors’ h-index from their Google Scholar page. Table 2. Bibliometric Indicators. Indicators

Definition

Total publication count per country/region

Total number of articles published by one country/region

Total publication count per author

Total number of articles published by one author

h-index

h articles published by an author has at a least h citation

Total citation count per country/region

Total number of citations obtained from articles published by one country/region

Total citation count per author

Total number of citations obtained from articles published by one author

Measuring Recognition: Sleeping Beauties in Science. While citations are widely used to measure the impact of research articles, citation dynamics describe the dissemination trajectories of scholarly articles. Recently, researchers applied the term Sleeping Beauties referring to research articles which were not

8

H. Du et al.

recognized until years later after publication, which is commonly seen in various research fields [38,39]. To identify Sleeping Beauties in science, a parameter-free criterion was proposed to assess the imbalance of citation distribution [39]. Let C be the total number of citations, and ci (i ∈ 1, 2, ..., n) be the number of citations received in the ith year. Gs is an adjustment of Gini coefficient and defined as: Gs = 1 −

2 ∗ [n ∗ c1 + (n − 1) ∗ c2 + ... + cn ] − C , C > 0, C ∗n

(1)

where Gs ∈ (−1, 1]. In general, a higher Gs value indicates a later recognition: most citations occurred in later years. In cases an article receives a citation number of 0, Gs equals 1. Social Network Analysis. Social network is commonly used to represent the relationship of a group of connected actors, such as participants of online communities, students in a traditional classroom, and a group of friends in social media platforms [36,37]. Social network graphs visualize the interactions among a number of connected actors, making it helpful to visualize the collaboration patterns for our study. We considered each author/institution as a node, and drew edges between two nodes if they collaborated. If two nodes have collaborated several times, multiple edges would be added. In this study, we considered two nodes as connected if two authors co-authored one article. Textual Data Processing. Textual data processing is a process which systematically and automatically structure the textual data. As a result, textual data will turn into structured data set and can be used for further analysis using data mining techniques. A number of commonly used textual data processing techniques were used to structure the article abstracts so that we can apply text mining analysis to identify trending topics and their dynamics, as well as potential future research directions. First, removal. Special characters including punctuation, special characters, and non-English terms were removed. Commonly used terms across fields with little information such as “a”, “the”, and “of” were also removed using stopwords package. Second, tokenization. Tokenization is a process dividing a string into several pieces using tokenize package. Tokenization is essential to structure textual data. Third, stemming and lemmmatization. These two techniques are commonly used to reduce inflectional forms of terms. For instance, if we count the frequency of terms’ occurrence, “am” and “is”, “book” and “books”, and “eat” and “eating” will be considered as six different terms. After stemming and lemmmatization, “computer”, “compute”, “computing”, “computation”, “computational” and “computes” will be mapped to the same term “comput”, hence, considered as the same term. Lemmatizer and Porter stemmer packages were used in this process. All analysis was conducted using Python.

A Descriptive and Historical Review of STEM + C Research

9

Measuring Dynamics: Keyword Flow. Based on texutal data mining techniques, keyword flow analysis was used to describe dynamics in trending terms over time [43]. Keywords can be identified either manually based on domain knowledge, or automatically by computation. Then, term frequency of identified keywords is calculated. Eventually, the results are presented in a flow chart providing an overview of keyword dynamics over time. In this work, we defined keywords as terms used repeatedly in abstract, rather than keywords provided by authors. We believe that keywords identified in such a way are more precise in presenting article topics [44]. Keyword flow enables an overview of both popular research directions and their dynamics over time. Topic Modelling. Latent Dirichlet Allocation (LDA)–which is an unsupervised machine learning algorithm to discover the relationship between documents and words in textual data–has been widely used in research scenarios to analyze largesize textual datasets, such as to classify social media posts, recommend scientific articles, and bibliometric analysis [40–42]. LDA topic modelling can generate a group of topics based on given documents, while each topic is represented by a number of words. Then, the model is able to assign topics to each document by representing the document as a probability distribution over topics. Coherence score which measures how well a topic model fits given documents is used to determine the number of topics. In general, a higher coherence score indicates a better fitting model.

4

Results

For this study, we identified a total number of 202 articles published by March, 2020, with a total citation count of 5590. There are 512 distinctive authors of 187 institutions from 29 different countries/regions worldwide. 4.1

RQ 1. What are the Current Trends in Number of Publications and Citations in STEM + C Research?

To address this question, we calculated the number of articles published in each year and the number of citations received by these articles each year, as shown in Fig. 2. A data point of the blue line represents the number of publications within a year, while a dot in the orange dash line is for the number of citations. The identified 202 records were published between 1995 and 2020. Overall, the number of publications were not evenly distributed. We did not identify any publications for several years, leading to a few zero publication data points in the figure. The pattern is different in relation to the number of citations. Even during those years when the number of publications equals zero, the citations kept increasing. There was a significant increase in both the number of publications and citations since 2008, indicating an increasing interest towards STEM + C research field. It is worth to mention that this game change occurred shortly after Wing [4] coined the term computational thinking. This suggests that the

10

H. Du et al.

term “computational thinking” has potentially brought much research interests to STEM + C integration over the years.

Fig. 2. Publication and citation trends [11].

The fact that our data set includes articles published prior to February, 2020 accounts for the significant drop in 2020 in Fig. 2. Nowadays, an increasing number of high schools are offering Advanced Placement CS courses, universities are starting new programs in CS teacher preparation. We believe that both numbers for 2020 will be higher than those of 2019, and STEM + C will continue to be an active research field drawing more and more attention. To further analyze the trends in citation, we calculated the adjusted Gini coefficient (Gs, see 3.2 Measuring Recognition) to measure the imbalance of citation history of identified articles. We identified a highest Gs value of 0.45, and the lowest valus is −0.15. We are interested in articles of great impact in STEM + C research. Table 3 lists articles whose total citation count is one standard deviation higher than the mean (M = 27.98, SD = 83.60) and their Gs values. Based on the Gs value, we believe that most highly-cited publications in STEM + C research received immediate recognition. This also suggests that STEM + C will continue to be an active research field drawing more and more attention in the future.

A Descriptive and Historical Review of STEM + C Research

11

Table 3. Highly-cited articles and their Gs value. Article Title

Total Citation Gs

The learning effects of computer simulations in science education

673

0.21

Thinking like a wolf, a sheep, or a firefly: Learning biology through constructing and testing computational theories— an embodied modeling approach

564

0.24

Defining computational thinking for mathematics and science 406 classrooms

0.38

Computational thinking and tinkering: Exploration of an early childhood robotics curriculum

397

0.39

Development of system thinking skills in the context of earth 321 system education

0.38

Integrating computational thinking with K-12 science education using agent-based computation: A theoretical framework

283

0.37

Computational thinking in K-9 education

253

0.44

Visual programming languages integrated across the 225 curriculum in elementary school: A two year case study using “Scratch” in five schools

0.30

Computational thinking in compulsory education: Towards an agenda for research and practice

209

0.43

A multidisciplinary approach towards computational thinking 166 for science majors

0.22

Designing for deeper learning in a blended computer science course for middle school students

142

0.44

Supporting all learners in school-wide computational thinking: A cross-case qualitative analysis

122

0.34

4.2

RQ 2. What are the Most Prolific Countries/Regions, Institutions, Resources, and Authors in STEM + C Research?

Prolific Countries/Regions. To address this question, we calculated the total publication count per country/region, the total publication count per author, the total citation count per country/region, and the total citation count per author (see Table 2). Table 4 shows the top five countries/regions in terms of total publication count per country/region. Due to the same amount of publications, seven countries/regions are listed. The USA is the most prolific country with 156 publications which is more than 2/3 of the total publications, suggesting its dominate role in STEM + C research worldwide. Canada ranks the second with 17 publications, which is approximately 10% of the total publication count of USA. The two countries account for 85% of published works in the field. The USA also ranks first in terms of citation count, followed by Netherlands with a citation count of 1137.

12

H. Du et al.

However, Netherlands’ total publication count ranks fifth with only four publications, indicating both high quality and great influence of its work. The USA and Netherlands account for over 93% of the total citations, indicating their prominent impact in STEM + C research. Table 4. Top five prolific countries/regions by total publication count. (TP (%) is the percentage of total number of publications, TC (%) is the percentage of total number of citations). Rank Country/Region Total Publication TP (%) Total Citation TC (%) 1

USA

2

Canada

156

77.22

4076

72.92

17

8.42

153

3

2.74

Spain

7

3.47

231

4.13

3

Greece

7

3.47

58

1.04

5

Israel

4

1.98

378

6.76

5

Turkey

4

1.98

3

0.05

5

Netherlands

4

1.98

1137

20.34

Prolific Institutions. We have identified the top prolific institutions, as shown in Fig. 3. Due the same number of publications, 12 institutions are listed. Only one of the 12 institutions is in Canada: University of Calgary, while all the rest are in USA. This also aligns with the dominant role of the USA in prolific countries/regions analysis. Among all the institutions in the USA, Vanderbilt University takes the high ground with a total publication count of 33 followed by Northwestern University with 14 publications. However, Northwestern University has the highest number of total citations followed by Standard University with five publications.

Fig. 3. Top 10 prolific institutions by total publication count.

A Descriptive and Historical Review of STEM + C Research

13

There are two non-university institutions: a nonprofit scientific research institution called SRI International, and a company in education named Looking Glass Ventures, ranking 4th and 8th, respectively. This suggests that STEM + C has drawn interests from both academia and industry. Except for Vanderbilt University and University of Calgary, the total publication count for the other 10 institutions does not vary significantly, suggesting multiple competitive research hot spots in the USA. Prolific Resources. We have also identified the top prolific publication resources (i.e., journals and conference proceedings), as shown in Fig. 4. The most prolific resource is ACM Technical Symposium on Computer Science Education, accounting for about 20% of total publications, followed by Journals of Science Education and Technology. However, Computers & Education, which ranks 7th in terms of total publication count, has the highest citation percentage: more than 25% of total citation count, indicating its high quality of work and broad influence in STEM + C research. In addition, both Education and Information Technologies (9th) and ACM annual conference on Innovation and Technology in Computer Science Education (10th) also have a relatively high total citation percentage and a relatively low total publication percentage.

Fig. 4. Top 10 prolific resources by total publication count.

Prolific Authors. Table 5 presents the top 10 prolific authors in terms of total publication count per author. All affiliation information was collected based on authors’ publications, meaning that some authors have more than one affiliated institution. The most prolific author is Pratim Sengupta with 18 publications, followed by Gautam Biswas (total publications: 17) and Satabdi Basu (total publications: 15). All of them are or were affiliated with Vanderbilt University. Of all authors, Uri Wilensky (total publications: 11) of Northwestern University who ranks 4th has the highest total citation count (1139). His h-index score (52) also ranks the highest. Another author with a 52 h-index score is Gautam Biswas

14

H. Du et al.

of Vanderbilt University. All listed authors have a total citation count over 100, and most have a total citation count over 300, indicating the general high quality and influence of their work. Two authors have two different affiliations through their publications. One of them affiliates with the only institution that is outside of the USA: University of Calgary. All the other authors’ affiliations are also among the most prolific institutions. Most of them are from the top four prolific institutions. As an integrated field, STEM + C is interdisciplinary in nature. Information in authors’ backgrounds also help understand how researchers of various backgrounds can contribute to STEM + C research. Based on identified articles, we categorized authors’ affiliated majors into four categories: Education, Computer Science, STEM and other, as shown in Table 6. Then, we calculated the distribution of researchers’ background, as shown in Fig. 5. About 40% of researchers are from education majors, while around 38% from CS majors. Another 8.8% of authors are in STEM majors. If combining STEM and CS majors, this would be the most populated group. It is interesting to see that most researchers are from CS and STEM majors, rather than education. This provides more empirical evidence to the disciplinary nature of STEM + C research, indicating the importance of domain knowledge in the field. Table 5. Top 10 prolific authors by total publication count. (TP: total publication count; TC: total citation count). TP TC

h-index

Rank Name

Affiliation

1

Pratim Sengupta

Vanderbilt University/University of Calgary 18

530

2

Gautam Biswas

Vanderbilt University

17

461

52

3

Satadbi Basu

Vanderbilt University/ SRI International

15

445

12

4

Uri Wilensky

Northwestern University

11

1139 52

5

John S. Kinnebrew Vanderbilt University

10

447

22

6

Schuchi Grover

Stanford University

9

188

19

7

Amy Farris

Vanderbilt University

8

154

8

7

Amanda Dickes

Vanderbilt University

8

134

7

9

David Weintrop

Northwestern University

7

476

16

10

Douglas Clark

Vanderbilt University

6

317

33

18

Table 6. Authors background categories. Category

Majors

Education Math/science education, teaching and learning, educational technology/psychology, learning sciences CS

CS, software engineering, game science and design

STEM

Mathematics, statistics, physics, chemical/industrial engineering

Other

English, sociology, music, anthropology, social work, media

A Descriptive and Historical Review of STEM + C Research

15

Fig. 5. Background distribution [11].

4.3

RQ 3. What are the Collaboration Patterns in STEM + C Research?

Most identified articles have two or more authors. The number of authors for each publication ranges from 1 to 11, while 25 articles have only one author. Collaboration Among Countries/Regions. Figure 6 presents a collaboration network among countries/regions. Each node represents one country/region, each edge linking two nodes represents collaboration between two countries/regions. A self-loop edge links to one node only, representing a collaboration between two different authors from the same country/node. The width of an edge represents the frequency of a collaboration. The size of a node presents its proportion to collaboration. For instance, if 10 authors of a paper are all from Canada, then the collaboration would be counted as once, which is Canada and Canada. In other words, the node size is not correlated with the number of publications. This explains why Sweden, Lithuania, Finland and Italy have a larger node size than some most prolific ones such as Spain, Greece, and Israel. In general, international collaboration prevails. Among all countries/regions, the USA has the most collaboration partnership with almost half of the countries/regions, formalizing a large international research community. The USA,

16

H. Du et al.

Fig. 6. Collaboration among countries/regions.

Netherlands, Sweden, Lithuania, Italy, and Finland have collaborations with multiple countries/regions. Almost all countries have collaboration partnership with themselves, indicating common domestic collaboration. In addition, nearly one third of the countries/regions do not have international collaboration. Collaboration Among Institutions. Figure 7 shows an institution collaboration network. For visualization purposes, institutions having a publication count larger than two were shown. Self-loops were removed. Most institutions are connected with others, while some are isolated. Since the figure presents only those institutions with a total publication count of two or more, it does not necessarily mean these institutions do not have collaboration with other institutions. Meanwhile, some institutions have more collaborations with each other, forming research communities. For instance, Vanderbilt University, SRI International, Looking Venture Glass, University of Maryland, Stanford University, University of Calgary, and Pennsylvania State University actively participated in collaboration and have developed a closely connected community. More than half of institutions are in the USA.

A Descriptive and Historical Review of STEM + C Research

17

Fig. 7. Collaboration among institutions.

4.4

RQ 4. What are the Most Mentioned Keywords and Their Evolving Trends in STEM + C Research?

To answer this question, we first identified keywords by analyzing article abstracts, then applied keyword flow analysis to visualize the dynamics. We have removed a few terms (use, education, school, learn, and integrate) which have a high frequency yet limited information on the research topic. In addition, the frequency of “computational thinking” includes the number of occurrence for both “CT” and “computational thinking”. Figure 8 presents the identified keywords. “Computational thinking” is the most frequent term among all, suggesting broad research interests in it. A number of computing related-terms also attract much attention, such as “programming”, “modelling”, and “data”. This suggests that various computer science contexts have potentially been applied in

18

H. Du et al.

Fig. 8. Identified keywords based on term frequency [11].

STEM + C research. In addition, terms of disciplines were also identified as keywords: math, science, engineering, and STEM. To better visualize the dynamics of keywords over time, we assigned keywords into two groups and developed two figures: research focus and disciplines Fig. 9. In relation to disciplines, science, math, and STEM are popular choices, while interest in engineering is very limited. This suggests that existing literature has limited understanding in integrating CS with engineering courses. In relation to research focus, “student” is the most frequently used term, while attention for “teacher” is much less, suggesting a potential lack in teacher preparation and professional development. In addition, researchers seemed interested in various aspects of CS knowledge, including “concept”, “data”, “programming”, “modelling”, and “simulation”. Among all these domain, “data” did not receive much attention until recent years, and has the potential to be a new and growing research direction in STEM + C research.

Fig. 9. Keyword flow by (a) research focus, and (b) disciplines.

A Descriptive and Historical Review of STEM + C Research

4.5

19

RQ 5. What are the Potential Research Directions Based on Current Literature in STEM + C Research?

To address this question, we developed a LDA topic model to analyze the main topics of current research. While keyword flow analysis investigates the articles at term level, we hope the topic modelling analysis could bring more insights at document level. To decide the number of topics, we first evaluated the model performance using the coherence score (c v) with a topic number ranging from one to 15, as shown in Fig. 10. The LDA model achieves the highest interpretability when the topic number equals four.

Fig. 10. Coherence score.

For reproduction purposes, we set the model parameter passes at 50 and random state at 1. Figure 11 presents the topic results of our model and their distribution. A few terms are shared by two or three topics, which is anticipated as topics are not mutually exclusive. Terms that are exclusive to one topic provide more information. The LDA model represents each document as a probability distribution over the topics. For a given document, the model tells us the assignment probability on each potential topic. We define the topic for a document as the one with the highest probability, and summarized the following topics: Topic 1 is mathematics/STEM research, Topic 2 is programming learning, Topic 3 is professional and integration development, and Topic 4 is learning outcomes. In relation to distribution, over half of identified articles focused on programming learning, followed by works studying mathematics/STEM. These two topics covered over 90% of identified articles, while less than 1% articles centering on learning outcomes. More efforts are needed to explore these two less studied topics.

20

H. Du et al.

Fig. 11. LDA topic results and their distribution [11].

5

Discussion

Based on 202 identified publications collected from Web of Science, IEEE Xplore, ACM Digital Library and Google Scholar, this work extended our previous work and presented an enriched comprehensive overview of the field by showing publication and citation trends, identifying prolific resources, presenting collaboration patterns, generalizing content-based topics, recognizing research keywords, popular research fields and understudied research topics. This study has both practical and methodological implications. Practically, this study serves as a road map for audience interested in STEM + C research. Methodologically, we introduced machine-learning-based techniques to the analysis, addressing the lack of focus in content. 5.1

Trends

There has been a consistent increase in the number of publications and citations in STEM + C since 2007. The USA has contributed dramatically more publications to STEM + C research and received more citations than any other country/region worldwide. Prolific resource analysis identifies a list of journals and conferences. Several conferences of ACM and IEEE are dedicated to CS education. A number of high quality articles were also published in prestigious educational journals like Computers & Education and Journals of Science Education and Technology. Based on the authors background analysis, almost half of the authors are from CS or STEM, while around 40% are from education. This indicates that STEM + C as an interdisciplinary field has drawn interests from a wide range of majors. This finding is also compatible with a report on CT in K-12 Mathematics classrooms [26] which found that much of the research originates from computer science rather than education. Based on the Sleeping Beauty Gini coefficient analysis, no publication in the field received significantly later recognition. In other words, work of great importance has received immediate recognition so far. Due to the calls to computing education and current

A Descriptive and Historical Review of STEM + C Research

21

trends, we anticipate more publications on this topic in the near future. These findings serve as an introducing map for audiences interested in STEM + C or seeking for a destination to submit their research works. 5.2

Potential Topics for Future Research

We trained an LDA topic model using collected article abstracts to identify topics. According to coherence score evaluating the model performance, four topics were identified: mathematics/STEM research, programming learning, professional and integration development, and learning outcomes. The first two topics covered over 90% of identified articles, while the other two were less discussed. This finding is in accordance with the statement from Lye and Koh [20] that programming is commonly used to teach CT. In addition, mathematics and STEM courses are the most popular choices in terms of integrating CT. More than one third of identified articles were assigned with the topic of mathematics/STEM research, explicitly suggesting the importance of mathematics in STEM + C research. While the LDA topic modelling sees data on the document level, keyword flow analysis provides a dynamic perspective. The term “computational thinking” (including CT) is the single most frequent keyword. In terms of subjects, “science” and “math” received much more attention than “engineering” and “technology”. Meanwhile, several terms are explicitly computing education related, such as “programming”, “modeling”, “simulation” and “data”, which can be viewed as popular computing education components in STEM + C research. It is worth to mention that “data” seems to have received some attention only in the past 5 years. Based on our analysis, more efforts can be spent on data science education at K-12 schools. 5.3

The Role of CT or CS Education in STEM + C Research

CT, as argued by Wing [4], is a skill set which can be obtained from programming practice and has been considered essential to modern citizens of the 21st century, and indispensable to STEM learning. CT has been commonly accepted as an effective strategy to benefit and advance STEM learning in the research community [24,27,28]. It is intuitive to develop CT through programming. Research also suggested that programming is the most commonly used way to teach CT [20]. An alternative way to integrate CT is through computation concepts, modeling, and simulation, especially for young children (e.g., [30]). Works focusing on framework development can be roughly classified into two categories based on grades: K-12 or college level education. For K-12 STEM + C research, researchers aimed to develop an universal framework to integrate CS education with STEM courses in order to advance STEM learning with the help of CT and prepare the next generation to be modern citizens [24,27,30], and to address the inability to offer stand-alone CS or programming classes [24,27]. Programming is not always involved, even when it is, block-based programming

22

H. Du et al.

(e.g., Scratch), instead of text-based programming (e.g., R, Python) environments, are commonly adopted for young students. For college level STEM + C research, framework development tends to design a new stand-alone course with joint efforts from different disciplines. These courses are designed for early years of college education, and mostly involve programming. In relation to course content, college level STEM + C research prefers data manipulation, programming concepts, and simulations. For instance, the study of Swaid [28] aimed to enforce CT in college level STEM gate-keeping courses where programming practices are involved. 5.4

Limitation

Although we have conducted both term-level and document-level content analysis using a data mining method, bibliometric analysis, by its nature, focuses on numbers instead of content. During data collection process, search terms were identified based on our understanding of the field as well as search efficiency. It will be helpful if future work can identify more efficient search terms or mechanisms within the field and explore more databases. In addition, our work did not examine other expressions that are argued similar to CT, such as computational literacy or systems thinking, leading to unidentified related articles.

6

Conclusion

We systematically collected 202 articles in STEM + C research, conducted an enriched bibliometric analysis, presented an enriched overview of STEM + C research by exploring the trends in publication and citation, identifying prolific resources, presenting collaboration patterns, generalizing content-based topics, recognizing research keywords, popular research fields and understudied research topics. This study contributes to the field by systematically examining related literature, providing an overview of the STEM + C field, and introducing data mining techniques to bibliometric analysis. Acknowledgement. This work is supported by the National Science Foundation (NSF) of the United States under grant number 1901704 and 2201393. Any opinions, findings, and conclusions or recommendations expressed in this paper, however, are those of the authors and do not necessarily reflect the views of the NSF.

References 1. U.S. Bureau of Labor Statistics. https://www.bls.gov/ooh/computer-and-inform ation-technology/home.htm. Accessed 30 Nov 2022 2. Carter, E., Blank, G., Walz, J.: Bringing the breadth of computer science to middle schools. In: Proceedings of the 43rd ACM Technical Symposium on Computer Science Education, pp. 203–208. ACM (2012)

A Descriptive and Historical Review of STEM + C Research

23

3. Barr, V., Stephenson, C.: Bringing computational thinking to K-12: what is involved and what is the role of the computer science education community? ACM Inroads 2(1), 48–54 (2011) 4. Wing, J.M.: Computational thinking. Commun. ACM 49(3), 33–35 (2006) 5. Pre-college computer science education: A survey of the field. https://goo.gl/ gmS1Vm. Accessed 5 Oct 2021 6. National Science Foundation. https://www.nsf.gov/funding/pgm summ.jsp?pims id=505006. Accessed 15 June 2020 7. National Research Council: Report of a Workshop on The Scope and Nature of Computational Thinking. National Academies Press (2010) 8. Wallin, J.A.: Bibliometric methods: pitfalls and possibilities. Basic Clin. Pharmacol. Toxicol. 97(5), 261–275 (2005) 9. Agarwal, A., et al.: Bibliometrics: tracking research impact by selecting the appropriate metrics. Asian J. Androl. 18(2), 296 (2016) 10. Campbell, D., et al.: Bibliometrics as a performance measurement tool for research evaluation: the case of research funded by the national cancer institute of Canada. Am. J. Eval. 31(1), 66–83 (2010) 11. Du, H., Xing, W., Pei, B., Zeng, Y., Lu, J., Zhang, Y.: Trends and issues in STEM+ C research: a bibliometric perspective. In: Proceedings of the 14th International Conference on Computer Supported Education, pp. 69–80. Springer, Virtual (2022) 12. Bybee, R.: Advancing STEM education: a 2020 vision. Technol. Eng. Teach. 70(1), 30–35 (2010) 13. Kelley, T.R., Knowles, J.G.: A conceptual framework for integrated STEM education. Int. J. STEM Educ. 3(1), 1–11 (2016). https://doi.org/10.1186/s40594-0160046-z 14. English, L.D.: STEM education K-12: perspectives on integration. Int. J. STEM Educ. 3(1), 1–8 (2016) 15. Royal Society (Great Britain): Shut down or restart?: The way forward for computing in UK schools. Royal Society (2012) 16. Wing, J.M.: Computational thinking and thinking about computing. Philos. Trans. R. Soc. A Mathe. Phys. Eng. Sci. 366(1881), 3717–3725 (2008) 17. Jones, S.P., Mitchell, B., Humphreys, S.: Computing at school in the UK. CACM Report (2013) 18. Psycharis, S.: STEAM in education: a literature review on the role of computational thinking, engineering epistemology and computational science. computational steam pedagogy (CSP). Sci. Cult. 4(2), 51–72 (2018) 19. Garneli, V., Giannakos, M.N., Chorianopoulos, K.: Computing education in K12 schools: a review of the literature. In: IEEE Global Engineering Education Conference, Estonia, pp. 543–551. IEEE (2015) 20. Lye, S.Y., Koh, J.H.L.: Review on teaching and learning of computational thinking through programming: what is next for K-12? Comput. Hum. Behav. 41, 51–61 (2014) 21. Robins, A., Rountree, J., Rountree, N.: Learning and teaching programming: a review and discussion. Comput. Sci. Educ. 13(2), 137–172 (2003) 22. Grover, S., Pea, R., Cooper, S.: Designing for deeper learning in a blended computer science course for middle school students. Comput. Sci. Educ. 25(2), 199–237 (2015) 23. Tang, X., Yin, Y., Lin, Q., Hadad, R., Zhai, X.: Assessing computational thinking: a systematic review of empirical studies. Comput. Educ. 48, 103798 (2020) 24. Weintrop, D., et al.: Defining computational thinking for mathematics and science classrooms. J. Sci. Educ. Technol. 25(1), 127–147 (2016)

24

H. Du et al.

25. Barcelos, T.S., Mu˜ noz-Soto, R., Villarroel, R., Merino, E., Silveira, I.F.: Mathematics learning through computational thinking activities: a systematic literature review. J. Univ. Comput. Sci. 24(7), 815–845 (2018) 26. Hickmott, D., Prieto-Rodriguez, E., Holmes, K.: A scoping review of studies on computational thinking in K-12 mathematics classrooms. Digit. Exp. Math. Educ. 4(1), 48–69 (2018) 27. Jona, K., et al.: Embedding computational thinking in science, technology, engineering, and math (CT-STEM). In: Future Directions in Computer Science Education Summit Meeting, Orlando, FL (2014) 28. Swaid, S.I.: Bringing computational thinking to STEM education. Procedia Manuf. 3, 3657–3662 (2015) 29. Leonard, J., et al.: Using robotics and game design to enhance children’s selfefficacy, STEM attitudes, and computational thinking skills. J. Sci. Educ. Technol. 25(6), 860–876 (2016) 30. Sengupta, P., Kinnebrew, J.S., Basu, S., Biswas, G., Clark, D.: Integrating computational thinking with K-12 science education using agent-based computation: a theoretical framework. Educ. Inf. Technol. 18(2), 351–380 (2013) 31. Hansen, Z., Du, H., Eckel, R., Lugo, J., Xing, W., Zhang, Y.: A preliminary datadriven analysis of common errors encountered by novice answer set programmers. In: Lierler, Y., Morales, J.F., Dodaro, C., Dahl, V., Gebser, M., Tekle, T. (eds.) International Conference on Logic Programming 2022 (ICLP 2022). EPTCS, vol. 364, pp. 12–24 (2022) 32. Du, H., Xing, W., Zhang, Y.: Misconception of abstraction: when to use an example and when to use a variable? In: Proceedings of the 18th ACM International Computing Education Research Conference (ICER), USA, vol. 2, pp. 28–29. ACM (2022) 33. Du, H., Xing, W., Zhang, Y.: A debugging learning trajectory for text-based programming learners. In: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE), USA, vol. 2, p. 645. ACM (2021) 34. Yang, K., Meho, L.I.: Citation analysis: a comparison of Google Scholar, Scopus, and Web of Science. Proc. Am. Soc. Inf. Sci. Technol. 43(1), 1–15 (2006) 35. Song, Y., Chen, X., Hao, T., Liu, Z., Lan, Z.: Exploring two decades of research on classroom dialogue by using bibliometric analysis. Comput. Educ. 137, 12–31 (2019) 36. Haythornthwaite, C.: Building social networks via computer networks: creating and sustaining distributed learning communities. In: Building Virtual Communities: Learning and Change in Cyberspace, pp. 159–190 (2002) 37. Du, H., Xing, W., Zhu, G.: Mining teacher informal online learning networks: insights from massive educational chat tweets. J. Educ. Comput. Res. 61(1), 127– 150 (2022). https://doi.org/10.1177/07356331221103764 38. Van Raan, A.F.: Sleeping beauties in science. Scientometrics 59(3), 467–472 (2004) 39. Li, J., Ye, F.Y.: Distinguishing sleeping beauties in science. Scientometrics 108(2), 821–828 (2016). https://doi.org/10.1007/s11192-016-1977-3 40. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3(Jan), 993–1022 (2003) 41. Iqbal, W., Qadir, J., Tyson, G., Mian, A.N., Hassan, S., Crowcroft, J.: A bibliometric analysis of publications in computer networking research. Scientometrics 119(2), 1121–1155 (2019). https://doi.org/10.1007/s11192-019-03086-z

A Descriptive and Historical Review of STEM + C Research

25

42. Wang, J., Hong, H., Ravitz, J., Hejazi Moghadam, S.: Landscape of K-12 computer science education in the US: perceptions, access, and barriers. In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pp. 645– 650. ACM (2016) 43. Du, H., et al.: Twitter vs news: concern analysis of the 2018 California wildfire event. In: 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), vol. 2, pp. 207–212. IEEE (2019) 44. Zeng, R., Chini, A.: A review of research on embodied energy of buildings using bibliometric analysis. Energy Build. 155, 172–184 (2017)

An Exploratory Test Design and Execution Learning Approach: A Definition of Syllabus and Teaching Plan Igor Ernesto Ferreira Costa

and Sandro Ronaldo Bezerra Oliveira(B)

Graduate Program in Computer Science (PPGCC), Institute of Exact and Natural Sciences (ICEN), Federal University of Pará (UFPA), Belém, PA, Brazil [email protected], [email protected]

Abstract. Exploratory Testing stands out as an alternative used in the industry to meet the needs of agile and/or short-term testing processes. However, professionals in this field use this agile testing approach in an unstructured way, and this may be because they don’t understand for sure how to apply it in a systematic way where they can involve design and execution activities. In this context, this article presents a syllabus and a teaching plan aimed at Exploratory Test Design and Execution activities using active pedagogical interventions (Coding Dojo, Gamification, Problem-Based Learning, Team-Based Learning and Flipped Classroom). This teaching plan is to support students by allowing “learning by doing” in a more interactive way, that is, providing student-centered teaching and learning in order to make such an approach more beneficial to the student, conditioning them to obtain the expected competences and skills in the industry. Keywords: Syllabus · Teaching plan · Test design and execution · Exploratory testing · Active methodologies

1 Introduction Exploratory Testing is a software agile testing approaches has been considered an ideal approach to be applied in projects that require little time to carry out this activity, as well as the need for managers get rapid and continuous feedback from the testing process [1, 2]. However, it is still understood by many professionals as an informal approach, without any structured and organized procedures, thus not supporting test process management activities [3–5]. Testing in agile teams has been considered not only the functional aspect, but several types of testing, with exploratory testing considered suitable to complement other testing approaches, for example, it is commonly used to improve test case coverage with predefined roadmap and system under test model used as a flow map for model-based automated testing [6]. It commonly requires the development of knowledge and practicing of the principles of agile testing, which it values effective communication and cooperation between stakeholders. In addition, testers need to know not only from testing or quality assurance, but © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 26–50, 2023. https://doi.org/10.1007/978-3-031-40501-3_2

An Exploratory Test Design and Execution Learning Approach

27

also from other subjects such as business analysis and coding. These are also factors that encourage this study to use an educational approach that promotes greater interaction among students. However, one of the great challenges faced in Software Engineering education is to meet the need to use teaching methods that make this process more effective [7]. In the educational context, some forms of teaching are prescribed in the literature that may be alternatives to the classical teaching method, and this work focuses on the use of a set of active methodologies. According to [8], active methodologies consist of pedagogical practices that can stimulate greater interaction, communication, learning involving more practical activities and motivation to students, etc. that is, they are student-centered teaching and learning approaches, where the teacher becomes one more facilitator of this process. In addition, the authors noticed that few activities related to the application of Exploratory Testing are carried out in the test design phase from a literature review. Mostly, the application of only execution activities, correlating them to the software development cycle was noticed [9]. For this, the importance of a process well-structured, and when it is aligned with guidelines prescribed in international and/or national (in Brazil) standards, it tends to be carried out systematically. This can make it possible to reach a very significant level of effectiveness in discovering defects, as these documents are organized records of market experiences, uniting theory and practice [10, 11]. This work aims to present in detail a teaching plan directed to the Exploratory Test Design and Execution. This approach is divided into teaching units, aimed at the systematic application of Exploratory Testing and based on constant practices and subpractices in the Test Design and Execution process area, prescribed in TMMi. This work is an extension of [12], where a syllabus is detailed being used as basis the Training Benchmark (RF) for Undergraduate Courses in Computing provided by the SBC and the guidelines contained in Computer Science (CS) Curricula provided by the Association for Machinery and the Institute of Electrical and Electronics Engineer (ACM/IEEE). In this context, this work has the following Research Question (PQ): How to develop an teaching approach that adopts guidelines for Exploratory Test Design and Execution, which develops knowledge and skills in students relevant to the software industry? In addition to this introductory section, this paper is structured as follows: Sect. 2 presents the theoretical foundation, Sect. 3 presents research methodology, Sect. 4 presents some related works, Sect. 5 presents the syllabus, Sect. 6 presents teaching plan, Sect. 7 presents an preliminary evaluation on proposal of this paper, Sect. 8 presents the conclusions and future work.

2 Background In this section presents the main concepts involved in this study. 2.1 Teaching Exploratory Test Design and Execution Exploratory testing is defined as occurrence of learning, test design and test execution simultaneously, that is, tests are not defined in advance in a pre-established test plan,

28

I. E. F. Costa and S. R. B. Oliveira

but are dynamically designed, executed, and modified. The effectiveness of exploratory testing depends on the software engineer’s knowledge, which can be derived from several sources: observation of the behavior of the product during testing, familiarity with the application, the platform used, the failure process, the type of possible faults and failures, the risk associated with a particular product, etc. [13–15]. For the authors [16] and [17], exploratory testing is an agile test approach that can be applied in a targeted way according to the agile test quadrant proposed by Brian Marick, in which tests are subdivided, increasing the participation and subsequent quality of the test professional who performs them. More recently [5] presents a reformulation of the concept of exploratory testing, defining that such an approach to testing consists of evaluating a product by learning about it through exploration and experimentation, including to some degree: questioning, study, modeling, observation, inference, among others. In this context, [5] emphasizes that the exploratory test is a formal and structured approach, exemplified by the analogy with the taxi ride case, where the customer does not request the ride plan because he trusts the intentions and competence of the taxi drivers. The same happens when using exploratory testing, where the tester trusts the exploitation strategies implicitly adopted. A corroborating fact was that [18] identified in their studies that testers implicitly apply many exploration strategies depending on the level of education. In this context, [19] exemplifies that the learning process using the exploratory test approach happens in a cyclical way. This model is called Kolb’s Learning Cycle (as can be seen in Fig. 1), where the tester holds experiences, then makes his inferences and therefore abstracts what he deems relevant and what he managed to perceive about the test object, thus actively experimenting the object. Thus, [20] presented results that show that this Experimental Learning was essential to guide or organize the thinking of testers to detect defects. In addition to this structured way of thinking to interact with the object, it is noteworthy that several documents suggest the use of charter to assist the tester in defining a scope on the object or test area. In this case, the charter is a tool that guides the performance of the test, where it provides the conditions to be covered during a time-boxed test session (pre-established exploration duration). It is important to point out that in several documents there is a suggestion of using charters, for example, in ISO 29119, Syllabus Foundation Level and its extension for agile testing, in Test Maturity Model integration (TMMi) in both versions 1.2 and 1.3, including suggested in the specialized literature by leading scholars in the area, some of which are: [6, 19, 21, 22]. Therefore, it is emphasized that [19] demonstrates how the dynamics of the exploratory test process involving Kolb’s Experimental Learning can be structured (as can be seen in Fig. 2). From the elaboration of the charter, it proceeds with the identification of variables and useful heuristics to observe ways and/or data to explore the object under test, in this circumstance, it allows to conduct an experiment of the system under test (exploration) being carried out careful observations as exploration of the object, finally an interrogation is carried out, and it is quite common to occur in front of the test manager, technical leader or even together with another tester. In this case, questions are

An Exploratory Test Design and Execution Learning Approach

29

Fig. 1. Kolb Learning Cycle [19].

asked about the results, strategies used and other factors deemed relevant for the knowledge of stakeholders. This feedback moment is important, as it allows understanding the level of knowledge acquired and/or identifying the test coverage explored (scope), as well as identifying opportunities for improvement in this test process.

Fig. 2. Structured exploratory testing process involving Kolb’s Experimental Learning [19].

In view of this, it is also mentioned the possibility of using different exploration strategies (navigation through the test object) to make the tests more efficient [18, 19, 23]. In addition, [22] reports that because there are some deficiencies that directly affect the management of test processes, techniques have emerged to mitigate it, for example, Session-Based Test Management (SBTM), Thread-Based Test Management, Risk-Based Test Management (SBTM). In this case, these management techniques establish more systematic procedures to provide a structured approach to the exploratory testing process, managing to address several factors relevant to the effectiveness of the testing process.

30

I. E. F. Costa and S. R. B. Oliveira

2.2 Curriculum for Supporting Software Engineering education continues to evolve constantly with the focus of preparing students to perform their careers in a qualified way, but there are still significant gaps between the contents covered in the university with the techniques and models (frameworks) frequently used in the software industry. In this context, international organizations such as the ACM and IEEE; and Brazilian, such as the Ministry of Education (MEC) and the SBC guide the SE curricula considering the integration of industrial perspectives with alternative teaching approaches to the traditional model [24]. It is emphasized that the latest version of the computing curriculum (CC 2020) strongly supports the use of a teaching and learning approach that offers practical experiences aligned with the industry. It allows also more engagement to students, considering the teacher is a facilitator of this process [25]. SBC - Brazilian Computer Society. SBC has been fundamental in recent decades in relation to the teaching of computing in Brazil, as it has always encouraged discussions on how undergraduate courses should be conducted. From this, the National Curricular Guidelines (DCN) emerged, as well as the “Computer Training References” (RF) for each of the courses contained in the DCN: Computer Science, Computer Engineering, Software Engineering, Graduation in Computing and Information Systems, including a technological degree [26]. For each RF of the courses it contains: presentation, course history, the benefits that the course offers to society, aspects related to the professional training of the course, the profile of the graduate indicating expected skills, training axes, as well as the skills and contents that make up the course, the relationships of the competences described in the RF with the determinations of the DCN, considerations about the realization of internships, complementary activities and course conclusion works, teaching and learning methodologies, the legal requirements foreseen for the course and, finally, thanks to several education and industry professionals who somehow contributed to the construction of this curriculum [26]. There is an emphasis on the methodology of elaboration of the RF for adopting an approach oriented by competences expected from the course graduates related to the contents involved in a given competence. In this case, the RF were structured in order to understand that the expected profile of the graduates determines the general objective of the course, broken down into different training axes. The training axes aim to train graduates in generic competences, thus, in order to achieve each competence, several derived competences are listed, which determine the need to be developed in specific contents [26]. ACM/IEEE - Computer Curricula. ACM and IEEE have gone to great lengths to establish international curriculum guidelines for degree programs in computing in recent decades. Due to the growth and diversification of the computing area, curricular recommendations have also grown, covering Computer Engineering, Information Systems, Information Technology, Software Engineering, Data Science, Artificial Intelligence and Computer Science and among other transversal technological innovations to these subareas. These guidelines are regularly updated in order to keep Information Technology curricula up-to-date and relevant in academia and industry. Examples of courses and

An Exploratory Test Design and Execution Learning Approach

31

programs are presented to provide more concrete guidance related to the curriculum structure and development of numerous institutional contexts [27]. In this way, they established principles for the curriculum of Computing courses that are about skills expected from students. The principles define that the curriculum must be designed to provide flexibility to graduates to work in different sub-areas, that is, students must be prepared for a variety of professions or roles and, above all, have clearly established the skills and knowledge that students need to obtain, while providing greater flexibility in topic selection. For this, in the 2013 curriculum, three levels of knowledge description are established, which are organized into: Core Tier 1, Core Tier 2 and Elective [27]. While the 2020 Computing Curricula (CC) presents new paradigms for computing education, including emphasizing the need to provide teaching and learning that is closely aligned with industry practices. The 2020 CC suggests the use of systematic ways of assessing learning, as well as the possible use of active methodologies in pedagogical practices as a way of trying to encourage or improve student engagement [25]. TMMi - Test Design and Execution. TMMi is structured by process areas being composed of general and specific goals, practices and subpractices, and the work products for each practice. In the Test Design and Execution process area, procedures are proposed that aim to improve the capacity of the test process during the activities of architecture development (design), execution and analysis of tests from the definition of architecture techniques, performing a process structured test execution, as well as managing incidents until their closure [28, 29]. Structured testing consists of the use of test design techniques and is also supported by tools. These techniques are used to derive and select test conditions and cases from requirements analysis and design specifications. In this context, a test case consists of the description of input values, pre-conditions, expected results, post-conditions. This information makes up the test procedures, which are specific test actions. Among such information, it is highlighted that the specific test data required are fundamental to allow the execution of these test procedures in an organized way [28, 29]. All of these Test Design and Execution activities follow the test approach as set out in the test plan. As a result, specific test design techniques are generally defined according to the product level and risks identified during planning. Finally, the activities most directed towards test execution are focused on discovering, reporting and evaluating incidents leading to closure. The reporting of incidents (defects) found can be aided through the use of an incident management system and communication to stakeholders must be formally carried out by established protocols [28, 29].

2.3 Active Methodologies In addition to these documents obtained as inputs, it is important to highlight that the use of pedagogical practices alternative to the traditional way. Among these possible alternatives present in the literature, active methodologies stand out, as they have been widely used and important in the last decade, although there are records that some scholars have already encouraged such teaching for many years. This notorious growth by new pedagogical practices has often been caused by the insertion of computers and Internet

32

I. E. F. Costa and S. R. B. Oliveira

access as an educational resource. Consequently, the need arose to obtain teaching alternatives to provide better learning, generally using more engaging strategies compared to traditional teaching [30]. It is considered that the importance of pedagogical practices centered on the student have been commonly emerging in debates since the last century from the movement called “New School”. In this case, some thinkers of the time, such as William James and John Dewey, already defended a teaching methodology focused on learning through experiences and autonomous student development [31]. This idea of “New School”, preached by John Dewey, is based on “learning by doing” in experiences with educational potential, which is notable in active methodologies. According to [31], the active methodology is characterized by the interrelationship between education, society, culture, politics and school, being developed through active and creative methods centered on the learner’s activity in order to promote learning. For [30], it is a way of conceiving education that presupposes the activity, where the student becomes the protagonist and takes responsibility for his learning process, with the teacher only being a guide in this process. When planning to use active methodologies, some questions permeate the understanding of the real role of the teacher. For this, [32] exposes aspects that better characterize the role of the teacher and the student in an active pedagogy for digital learners (as can be seen in Table 1). Table 1. Comparison of roles between Teacher and Student in active methodologies [32]. Teacher

Student

Don’t speak, ask

Don’t take notes, look for, find

Suggests topics and tools

Search and find solutions

Learn technology with students

Learn about quality and rigor with the teacher

Evaluates student solutions and answers, examining quality and accuracy, contextualization

Refines and improves answers, adding rigor, context, quality

Finally, it is mentioned that there are numerous techniques or methods associated with active methodology. These techniques provide learning through experiences that encourage the development of students’ autonomy and protagonism, as well as create situations that allow arousing curiosity and awareness of reality. Such methods are: flipped classroom, project-based learning, problem-based learning, shared classroom, blended learning, game-based learning (gamification) and so on. It is emphasized that the teacher has the autonomy to adapt or create methods, while respecting the principles of active methodology.

3 Research Methodology To achieve the objectives of this study, the methodological procedures were established as described below.

An Exploratory Test Design and Execution Learning Approach

33

3.1 Step 1: Literature Review First, a literature review was conducted evaluating the existence of evidence to obtain a holistic view of the strengths and gaps on the applicability of the exploratory testing approach in the context of the software industry. This research was restricted to studies published between 01/2001 and 04/2020, as it is understandable that the first time that the term “Exploratory Testing” has been consolidated was from the publication of the book “Lessons Learned in Software Testing: A Context- Driven Approach”, around 2001 [33]. In view of the 20 studies included in the analysis, it was possible to identify a great potential for studying Software Engineering education, with one of the most specific gaps being the non-performance of structured Exploratory Test Design and Execution activities [9]. In this case, the great importance of carrying out an analysis of the curricular guidelines was perceived, as well as investigating software tools, techniques and work products used in the industry to support the construction of a teaching plan, providing relevant practices to the industry, being aligned with the academic curricula. 3.2 Step 2: Analysis of Assets from Curricula and Guide for Software Testing At this step, the organizations and their necessary inputs to be analyzed were established, thus identifying: (i) TMMi for providing a guide describing activities in the Test Design and Execution area, where it presents practices, goals, examples of work products that these activities can be used in an organized way in the industry, (ii) SBC, for providing curricular guidelines in computing specifically in the Brazilian context, using the reference input for the formation of undergraduate courses in computing at SBC, and (iii) ACM/IEEE, as it also makes available subjects related to computing education at an international level, using both inputs: “Computer Science Curricula 2013” and “Computing Curricula 2020” from ACM/IEEE. Therefore, there was the identification and description of each asset contained in the previously pre-established inputs. Thus, each of the documents was analyzed, as well as the structure adopted with the assets. Finally, the correspondence between the identified assets was analyzed. This activity was carried out based on the specific goals and practices described in the TMMi to then analyze the correlation of the assets present in these inputs. In this way, the reference between the assets in the adopted documents was obtained, generating an equivalence structure with the corresponding justifications for each relationship. As a result of this mapping, 13 assets and 110 asset items for Test Design and Execution were identified, interrelated at two levels, namely: i) Training axes (RF-SBC) and knowledge areas (ACM/IEEE) related to Test Design and Execution process area, ii) Derived Content and Competences (RF-SBC), as well as topics and learning outcomes (ACM/IEEE), which relate to the specific goals, specific practices and sub-practices of the TMMi process area, which in this case is the focus of this work [34].

34

I. E. F. Costa and S. R. B. Oliveira

3.3 Step 3: Perception of Professionals Regarding Support Resources for Exploratory Testing Regarding the professionals’ perception of the study area, there was first the identification of this target audience. In this case, these participants were professionals in software testing accredited by a national (Brazilian) and/or international institution, or who obtained professional certification in TMMi with experience in test process improvement. Thus, it is possible to obtain relevant answers to the construction of a teaching plan directed to the Exploratory Test Design and Execution to provide practical content closer to reality [35]. Therefore, the questions were defined based on the Test Design and Execution process area prescribed in the TMMi. This alignment was important due to the fact that TMMi is international and based on the experiences of several professionals. In this way, it allowed to establish adherent questions with the practices commonly used in the industry. Upon establishing the questions, peer review was applied to evaluate the interview questions, and a careful analysis was carried out to certify that there was a coherent relationship with the Test Design and Execution practices prescribed in the TMMi, as well as an evaluation of the public-target and guidelines established for the execution of the interview. Afterwards, the interviews were carried out remotely with each interviewed professional, who participated voluntarily. Finally, in the data analysis stage, there was the interpretation of the collected data being summarized in the form of graphs to facilitate the visualization of the data. Therefore, the results of the interviews were: i) identification of software tools, techniques and work products for Exploratory Test Design, and ii) identification of software tools, techniques and work products for Exploratory Test Execution. In the context of item “i”, the most used software tools to assist design activities were Testlink and Jira, with risk analysis being one of the most cited activities in relation to support in the identification and prioritization of test conditions, including being also useful as a complement to the Exploratory Test. Regarding work products, the test plan and results of previous test runs were most used by professionals. In the context of item “ii”, Mantis and Jira were the software tools most mentioned as being important to assist in the management and execution of tests. As for the execution techniques, the use of Exploratory Testing with manual and automated strategy was observed, while the Incident Report and the Matrix were the work products commonly used by professionals [35]. 3.4 Step 4: Construction of Syllabus In this step of syllabus construction, there was a definition of general competences that the students need to acquire, which it was established from the mapping and based on the competences described in the curricula of the SBC and ACM/IEEE. Thus, there was the organization, structuring and documental record of the teaching and learning approach for the construction of the syllabus. In this case, there was identified 12 competences expected from Exploratory Test Design and Execution, making it possible to establish correlated subjects organized

An Exploratory Test Design and Execution Learning Approach

35

into four teaching units: i) Exploratory test analysis and design, ii) Implementation of exploratory testing procedures, iii) Exploratory test execution, iv) Test and incident process management. Therefore, there was a definition of the structural elements for each teaching unit of syllabus, being: prerequisites, guiding questions, programmatic contents, expected results and learning levels, as well as containing the teaching strategy which is corresponding to the approach to be used in the teaching plan. All this, being adherent to the learning levels established in the teaching unit [12]. Therefore, more details of the study program can be seen in [12]. In addition, it is emphasized that peer review was applied in the mapping of assets, analysis of interview questions, syllabus and teaching plan to ensure from the beginning the adherence of the analyzed inputs until the elaboration of the proposed teaching plan based on active methodologies. In summary, it was a constant activity at each step of this research, in order to also avoid author bias. 3.5 Step 5: Definition of Teaching Plan From the syllabus, it was possible to generate this teaching plan containing established learning strategies, description the dynamics of activities (methodological procedure) and the expected learning level for each of these pedagogical techniques used (active methodologies). This teaching plan was established with a focus on involving a studentcentered approach, where the teacher becomes a facilitator of this process, as an alternative to breaking the paradigm of traditional teaching with massive lectures. Also noteworthy is the identification of possible syllabus contents in the Syllabus Foundation Level provided by the International Software Testing Qualifications Board (ISTQB). This analysis was relevant because it is a study guide commonly used by people seeking the aforementioned certification in software testing. In addition, some active pedagogical practices were identified that could be applied to the teaching and learning of exploratory test design and execution activities as described in the syllabus. In this case, the active interventions planned to be used were: • Dialogued classes with collaborative learning: a teaching strategy that consists of everything that contains a dialogued lecture, with the addition of activities to consolidate learning at the end of the class, • Problem-Based Learning: the teacher works together with the students helping them in their search for knowledge. In this approach, activities are carried out based on the presentation of everyday problems or real cases in the industry so that students can raise possible solutions to these problems, • Flipped Classroom: A pedagogical approach in which direct instruction moves from the group learning space to the individual learning space, and the resulting space is transformed into an interactive and dynamic learning environment in which the educator guides students while they apply the concepts and engage creatively with the subject matter, • Team-Based Learning: a strategy that is characterized by the construction of ideas through group work. This strategy enables greater collaboration and information sharing among students so that they can teach and learn at the same time. It is essential to build mixed teams/groups and that students are engaged with the process,

36

I. E. F. Costa and S. R. B. Oliveira

• Gamification: use of a game element in the educational context of Exploratory Software Test Design and Execution that supports the evaluation of activities, especially in groups, and enables greater engagement of participants. The teaching plan has been planned to cover a total of 60 h divided into 30 synchronous meetings and asynchronously with practical application projects, with the development of activities being monitored at times necessary to understand and assist the team, streamlined process such as mentoring. Faced with the moment of a pandemic experienced, where there are still uncertainties about when to return and what are the appropriate procedures to be adopted to return to face-to-face teaching, it was necessary to resort to a remote teaching strategy. In this way, it was observed that the form of teaching called Education Online (EOL), disseminated by SBC, contains favorable and adherent characteristics to the use of active methodologies through digital technologies. The EOL educational modality [36] is an alternative to face-to-face classes where didactic-pedagogical practices are supported by the use of many technologies in the networked digital environment. It is a form of distance education, but with characteristics and relationships of the teaching-learning process different from conventional Distance Education (DE). EOL is characterized by eight principles: a) knowledge as an “open work”, b) content curation, synthesis and study guides, c) diverse computing environments, d) networked, collaborative learning, e) conversation among all, in interactivity, f) authorial activities inspired by cyberculture practices, g) online teaching mediation for collaboration, h) formative and collaborative evaluation, based on competences.

4 Related Works In this section some related studies are presented that were analyzed by the similarity of the theme that is being presented in this work. In the work of [37] an iterative model for the teaching of Software Engineering (SE) is presented, based on student-focused approaches and practices present in the software industry, presenting significant results that encourage further studies in this context. However, the self focuses on the comprehensive teaching of the Software Engineering subject adapting training practices adopted by the software industry to the academic context in order for students to develop technical skills in SE at the application level. The author builds the proposed framework based on the analysis of the ACM/IEEE curriculum for computing, the CMMI-DEV quality model and a survey applied to students and teachers. The work of [38] presents an approach to the teaching of statistical process control in higher computing courses. It is noteworthy that the author uses the teaching framework proposed by [37] as the basis of his teaching approach, and he also achieves excellent results in the use of active methodologies directed to the specific teaching of statistical process control. The aforementioned author proposes his teaching approach based on the analysis of curricular guidelines for computing, SBC and ACM/IEEE curricula, quality models, as well as a survey applied to teachers, students and software engineers. The work of [39] presents the use of active methodologies aimed at the comprehensive teaching of the subject of software testing, where the applied teaching plan is based on the proposed curriculum from the analysis of national curriculum guidelines,

An Exploratory Test Design and Execution Learning Approach

37

ACM/IEEE and SBC curricula. As with the other studies mentioned above, the results obtained were quite relevant and encourage the use of pedagogical practices that make the student the center of the teaching-learning process in the context of the Software Testing subject. The authors of [40] present an experience report that compares the teaching in the traditional style with the student-centered one being carried out in the Software Engineering course at the Federal University of Pampa. For the second, the Problem-Based Learning methodology and gamification elements are used, although in both cases focused on the teaching of Software Testing topics, the subject also included requirements engineering, verification and validation topics. It is noteworthy that the referred approach was based on the above mentioned course syllabus, and results were achieved that show an effectiveness of teaching using active learning methods, as they favored an encouraging environment, according to students’ reports and teachers’ perceptions. The authors of [20] also presented an experience report on teaching software testing based on gamification/games. Among the testing approaches, it is emphasized that agile testing concepts and session-based test management are learned through Lego-based contexts, and the teaching of exploratory testing through data-based games, and for the teaching of exploratory testing Kolb’s experimental learning cycle structure was adopted (concrete experience, reflective observation, abstraction and active experience). As a result, there are reports that the use of games motivated the students to participate more intensely in the activities, in which the reflection on their actions allowed their self-discovery about the concepts encapsulated in the games. In view of the results, it is essentially emphasized that most students agreed that Experimental Learning created a better understanding of how to conduct exploratory tests. In view of the studies presented, this work has similarities mainly in its use in the analysis of national and international training curricula mentioned in the first three aforementioned works, [37–39]. Regarding these three works, it is mentioned that they served as a basis for structuring and defining the teaching program in line with industry practices, which differs from other works by focusing on the teaching of exploratory testing being applied systematically according to the structure of specific objectives and practices prescribed in the TMMi [28, 29]. Furthermore, it is mentioned that the work of [20] was important to have as a basis the use of Experimental Learning proposed by Kolb.

5 Syllabus It is known in society that there is no single form or even a single model of education but the academic and professional community joins efforts to develop study programs involving topics related to the computing, which may be sufficient to promote skills to students in order to prepare them for the industry. This can be easily noticed in the CS-Curricula, RF-SBC and CC2020, as they present structured curriculum guidelines in line with the theory of the area and practical knowledge in the industry. In this context, Higher Education Institutions (HEIs) remain adherent to these study programs, as they use them as basis for preparing their course [25, 27]. According to [41] the authors it is interesting to involve eight learning components to build a good curriculum,: (1) basic information about the course and contact information,

38

I. E. F. Costa and S. R. B. Oliveira

(2) course purpose, including goals and objectives, (3) instructor teaching philosophy and beliefs, (4) course assignments and schedule, (5) required and optional materials, including textbooks and supplemental reading, such as newspapers, (6) methods of instruction and course delivery, (7) assessment procedures, and (8) learning resources for students. This work addresses the first three topics mentioned by [41], since the authors chose to separate the syllabus aimed mainly at the construction of knowledge units and a teaching plan partially using this program. As previously mentioned, that syllabus is composed by 12 general competences identified from the curricula analyzed, as can be seen in Table 2. After that was defined the structure of the teaching unit to keep a standard for it (as can be seen in Table 3) then the goal and related Basic skill were established for each teaching unit (as can be seen in Table 4). Table 2. General competences adopted [12]. C1. Employ methodologies that aim to ensure quality criteria throughout the exploratory test design and execution step for a computational solution C2. Apply software maintenance and evolution techniques and procedures using the ET approach C3. Manage the exploratory test approach involving basic management aspects (scope, time, quality, communication, risks, people, integration, stakeholders and business value) C4. Apply techniques for structuring application domains characteristics in the exploratory test approach C5. Apply techniques and procedures for identifying and prioritizing test conditions (with a focus on exploratory testing) based on requirements and work products generated during software design C6. Apply software model analysis techniques to enable traceability of test conditions and test data (with a focus on exploratory testing) to requirements and work products C7. Apply theories, models and techniques to design, develop, implement and document exploratory testing for software solutions C8. Apply validation and verification techniques and procedures (static and dynamic) using exploratory testing CG9. Preemptively detect software failures on systems from the exploratory test application C10. Perform integrative testing and analysis of software components using ET in collaboration with customers C11. Conduct exploratory testing using appropriate testing tools focused on the desirable quality attributes specified by the quality assurance team and the customer C12. Plan and drive the process for designing test cases (charters) for an organization using the ET approach

An Exploratory Test Design and Execution Learning Approach

39

Table 3. Generic Construction of the Teaching Unit [12]. Subject Prerequisites These are the subjects or teaching units that can facilitate learning if they are previously attended by students and serve as the basis for the subject addressed in this teaching unit. It is advisable to indicate the reference curriculum that was used Guiding Questions These are questions asked to students during the beginning of each unit, which aim to start the discussion of the topic Programmatic Content (PC) These are the contents to be taught in the curricular unit, in view of the skills foreseen for the didactic unit. Mapping was used to create learning topics Expected Results

Level of Learning

It is what the student must be able to learn and Each of these expected results is associated perform after learning accumulated in the unit, with a certain level of cognitive ability and always of an evolutionary nature knowledge dimension of the revised Bloom’s Taxonomy

Table 4. Syllabus Units. Unit

Goal

Related Basic Skill

Exploratory test analysis and design

Teaching design concepts and of 1, 4, 5, 6, 7, 12 analysis testing techniques of software design for identification and prioritization of test conditions and test data, as well as to present important factors in the definition of a test process strategy exploratory

Implementation of exploratory testing procedures management

Teaching how to elaborate charters, possible exploration strategies, preparation procedures for initial tests (intake test) and drawing up a test execution schedule

Exploratory test execution

Teaching the exploratory test in 2, 8, 9, 10, 11 practice in a systematic way using structuring techniques, including also the strategies of exploration. In addition, teaching how to write the test reports and apply analysis of cause

5, 7, 8, 10, 12

40

I. E. F. Costa and S. R. B. Oliveira

6 Teaching Plan As previously mentioned, this teaching plan is an instance of syllabus being also structured in four teaching unit. Thus, you can see prerequisites, questions, programmatic expected results selected for this teaching plan in Appendix A. Teaching Unit I – Exploratory Test Analysis and Design. For this, collaborative dialogued expository classes will be taught, presenting the basic concepts of levels, testing types and techniques to provide a comprehensive view of Software Testing and favor students to identify more assertively when to apply the exploratory test approach. In addition, exploratory test management techniques were exposed, specifically the SBTM, test design techniques and work products aimed at supporting students in the analysis and elaboration of charters in line with the test objective. These techniques may be approached with fixation exercises in a practical way, obeying the levels of learning and the expected results of a given topic. Thus, there are practical activities on deriving and prioritizing test conditions, identification of test data, as well as the description of test procedures being documented, including the students generated a traceability matrix of these work products. Regarding prioritization, an adaptation of the COORG technique (Acronym for: Classify, Order and Organize) was used, which is inherent to the Product Backlog Building (PBB) [42]. It is noteworthy that these activities will be taught synchronously, where a practical project also started to put even more into practice the concepts learned. In this case, the practical project will be applied asynchronously, however monitored by instant conversation tools and to support the establishment of responsibility on the Kanban board. Another point is that the teacher plays the role of facilitator during the synchronous moment, and acts as a client during the development of the practical project, where he establishes possible goals for delivery. This practical project may be presented in the form of a flipped classroom at the end of the study of this teaching unit. Therefore, the structure of the teaching plan for this programmatic content can be seen in Table 5. Table 5. Structure of Teaching Plan for PC 1.1. Competences

Software Tools

C1, C4, C5, C6, C7, C12 Google Meet, email, instant chat app, Trello, Jira, TestLink, Office Active Technique

Dialogued Class with Collaborative Practice, Problem Based Learning, Team Based Learning, Flipped classroom

Evaluation

Fixation Exercises (practical activity), Practical Project

Teaching Unit II – Implementation of Exploratory Test Procedures. For this, examples of charters will be presented to support the elaboration of new charters through the

An Exploratory Test Design and Execution Learning Approach

41

collaboration of all students. Following this dynamic, there is the activity of defining the verification criteria for initial tests on the main features of the system (intake testing). In this case, they may be divided by group so that they could collaborate with each other in developing this activity. In this context, the concepts and examples of exploration techniques become important to correlate them with the pre-defined charters, as well as to jointly analyze some relevant aspects in the definition of a test schedule. In the elaboration of charters, they also established the exploration techniques and later established an adequate execution schedule for these charters. It is noteworthy that the practical project also took place with the same dynamics described for Exploratory Test Analysis and Design. Therefore, the structure of the teaching plan for this programmatic content can be seen in Table 6. Table 6. Structure of Teaching Plan for PC 1.2 Competences

Software Tools

C5, C7, C8, C10, C12

Google Meet, email, instant chat app, Trello, Jira, TestLink, Office

Active Technique

Dialogued Class with Collaborative Practice, Team Based Learning, Flipped classroom

Evaluation

Fixation Exercises (practical activity), Practical Project

Teaching Unit III – Exploratory Test Execution. For this is involved some practical exploratory testing activities in a collaborative way being streamlined based on Dojo Randori and Dojo Kake (LAB) [43]. In this case, exploratory test application activities took place both in the classroom and outside the classroom, in the latter the students present the strategy used to test a system they used to use. In the classroom (synchronous moment) there are basic activities for the assertive understanding of the use of exploration techniques, the use of these exploration techniques and a technique that allows the structuring of these activities related to execution (SBTM), as well as executing the tests while standing according to the schedule. The dialogued classes with collaborative learning are important to show on the good practices of writing the log or incident record and of analyzing the detected incidents, identifying possible root causes, always maintaining the traceability between test conditions, test procedures and test results. In this context, it emphasizes that the group activity allows the students to practice even more all these procedures gathered in extra-class moments (asynchronous moment). It is noteworthy that the practical project also occurs with the same dynamics described for Exploratory Test Analysis and Design. Therefore, the structure of the teaching plan for this programmatic content can be seen in Table 7. Teaching Unit IV – Test and Incident Process Management. For this will be presented good practices in relation to the review of incident reports. Subsequently, there is a paired activity of reviewing the incident log report using the reports generated when searching for incidents (report generated in the Exploratory Test Execution teaching

42

I. E. F. Costa and S. R. B. Oliveira Table 7. Structure of Teaching Plan for PC 1.3.

Competences

Software Tools

C2, C8, C9, C10, C11

Google Meet, email, instant chat app, Trello, Jira, TestLink, Office

Active Technique

Dialogued Class with Collaborative Practice, Dojo Randori, Dojo Kake, Team Based Learning, Flipped classroom

Evaluation

Fixation Exercises (practical activity), Dojo Randori, Dojo Kake, Practical Project

unit). In addition, the preparation of the test summary report was practiced following best practices for effective communication with stakeholders and providing them with an understanding of possible appropriate actions to correct incidents, as well as directing them to closure. In addition, there was also a dialogue class with collaborative practical activity to discuss opportunities for improvement of some test process flows from real companies. In the second moment, in synchronous class, the groups must carry out an analysis of their process in order to improve it, and make it more adequate or viable (considering good practices) to execute it according to the lessons learned. Therefore, the structure of the teaching plan for this programmatic content can be seen in Table 8. Table 8. Structure of Teaching Plan for PC 1.4. Competences

Software Tools

C2, C3, C12

Google Meet, email, instant chat app, Trello, Jira, TestLink, Office

Active Technique

Dialogued Class with Collaborative Practice, Team Based Learning, Flipped classroom

Evaluation

Fixation Exercises (practical activity); Practical Project

As described for Exploratory Test Analysis and Design, the practical project also took place in this programmatic content with the same dynamics. It is worth mentioning that the software tools, techniques and work products that will be used are listed based on the results of interviews with test professionals [35, 44]. Finally, there are specific feedback classes to collect qualitative data to be analyzed. These moments are important to identify strengths, weaknesses, opportunities and threats regarding software tools, material resources and applied activities, as well as observing the students’ degree of satisfaction and self-evaluation regarding the learning obtained.

An Exploratory Test Design and Execution Learning Approach

43

7 Evaluation of the Approach Both study program and the teaching plan were submitted to a preliminary analysis by experts being carried out through peer review, generating a list of items for adjustments (see Table 9). For each change request, an identifier (ID), a category, the item to be adjusted, a comment justifying the reason for the adjustment and a suggestion for improvement were assigned. In this case, the categories are: • High Technician - TA, indicates that a problem was found in an item that, if not changed, will compromise the considerations, • Low Technician - TB, indicates that a problem was found in an item that it would be convenient to change, • Editorial - E, indicates that a spelling and grammatical error was found or that the text could be improved, • Questioning - Q, indicates that there were doubts as to the content of the considerations, • General - G, indicates that the comment is general regarding considerations. All the adjustments requested during peer review were implemented by the author. It allowed the elaboration of a syllabus, as well as the instantiation of the teaching plan adhering to the curricula. This preliminary evaluation is importance to avoid the author’s bias in the elaboration of the syllabus and teaching plan and being able to make adjustments to avoid problems in its application that may make it unfeasible to be used. Table 9. Adjustment items identified in the peer review [12]. ID

Category

Item

1

E

Teaching Approach

Comment: There are words and grammatical and spelling errors throughout the text Suggestion: Correct these errors and replace the words with more formal texts 2

E

Introduction

Comment: No reference was made to mapping the assets of the curricula Suggestion: Include asset mapping 3

TB

Competences and Teaching Units

Comment: Competences and Teaching Units are comprehensive for any type of test Suggestion: Customize these skills for Exploratory Testing 4

TB

Course Planning

Comment: The origin of the definition of Teaching Units was not specified Suggestion: Define that the Teaching Units maintained compliance with the TMMi 5

E

Description of Teaching Units (continued)

44

I. E. F. Costa and S. R. B. Oliveira Table 9. (continued)

ID

Category

Item

Comment: There is no reference from where the elements that make up the description of the Teaching Units were extracted Suggestion: Reference the base work to describe the teaching units 6

E

Element of Learning Level

Comment: There is no reference from where the Learning Level elements were extracted Suggestion: Reference the Revised Bloom’s Taxonomy 7

TB

Expected Results in Teaching Units

Comment: The expected results of the teaching units do not have an objective detail Suggestion: Be clearer and more objective with the details of the expected results of the teaching units 8

TB

Learning Levels of Teaching Units

Comment: Some learning levels are not in line with expected results Suggestion: Review the alignment between learning levels and expected results for each teaching unit 9

E

Teaching Strategies

Comment: There is no reference on the use of possible software tools commonly used in the industry Suggestion: Reference the work that analyzes tools used in the industry 10

TA

Selection of Pedagogical Practices

Comment: The selection of pedagogical practices used for the delivery of teaching units was not justified Suggestion: Inform the references used for a selection of pedagogical practices 11

TB

Teaching Unit Detail

Comment: Some learning levels detailed in each teaching unit are not in line with their description Suggestion: Review this alignment

8 Conclusion The objective of this work was to present in detail the planned teaching plan from a previously constructed syllabus aimed at the practical application of structured Exploratory Test Design and Execution activities adherent to the TMMi. This teaching plan involves techniques, software tools and work products commonly used by professionals and prescribed in the specialized literature. In addition, it has the propose of using active pedagogical practices as an alternative to the traditional teaching and learning, based on a teaching modality (EOL). In this context, the EOL also proposes interactive, communicative and engaging classroom dynamics through the use of digital technologies, unlike the Distance Education modality.

An Exploratory Test Design and Execution Learning Approach

45

The teaching plan was evaluated by experts, both in research on software engineering education and practical applications in the industry. However, it is planned to apply it in an undergraduate computer science class to validate its effectiveness and efficiency. Regarding the research questions of this study, the answer encourage building a teaching plan using active methodologies, practical activities present in the industrial scenario. It was possible to develop a teaching plan focused on practices (learning by doing) in a more interactive way. The preliminary analysis suggests that it may provide sufficient skills for students to engage in the industry. However, just after applying this teaching plan will be able to measure whether students have acquired the competencies and being noticed improvements that could be make on it. The validation of the approach is already planned, thus in the future, one should qualitatively evaluate with pre-test and post-test, feedback through SWOT (Strengths, Weaknesses, Opportunities, Threats) analysis, evaluation of satisfaction, evaluation of activities regarding their alignment, coherence and feasibility according to established learning levels. In addition, evaluate quantitatively through the grades obtained by students in practical activities. The external validity for this study can be limited to the Brazilian academic context because is used curricula from SBC. Another one is even if the teaching approach is aimed at practice, there is no guarantee that the competences acquired will be reflected in a real environment of a software development team. Thus, the importance of this study in providing the training of students applying concepts in cases near of the reality that can be sufficient for use in the industrial scenario. In addition, it is breaking the paradigm of understanding that exploratory testing is an approach without structure to apply it, as well as this study is relevant to support new studies on it. Therefore, the main future work is to carry out experiments of this teaching plan in the virtual classroom, as soon as possible. For now, it is planned an experiment with an undergraduate class, considering only voluntary student to participate. Acknowledgements. The authors would like to thank the Coordination for the Improvement of Higher Education Personnel (CAPES) in Brazil for the financial support for granting an institutional PhD scholarship.

Appendix A Subject – Exploratory Test Design and Execution Prerequisites ACM/IEEE: (SE) Software Engineering SBC: Software Engineering Guiding Questions (continued)

46

I. E. F. Costa and S. R. B. Oliveira

(continued) Subject – Exploratory Test Design and Execution Q1. How to apply a set of design techniques to identify test conditions and test procedures, both exploratory? Q2. What criteria are suitable for prioritizing test conditions and test procedures, both exploratory? Q3. What aspects are analyzed for the strategic definition of an exploratory testing process? Q4. How to develop test letters (charters) suitable for certain contexts? Q5. How to identify suitable criteria for verification of initial tests? Q6. How to systematically apply exploratory testing considering pre-defined test charts and selected exploration techniques? Q7. How to apply exploratory test session report review best practices? Q8. How to use good practices to communicate incidents to stakeholders? Programmatic Content - PC 1.1 Exploratory Test Analysis and Design 1.1.1. Test fundamentals 1.1.2. Work products suitable for the analysis and identification of test conditions adhering to the test objective 1.1.3. Test design techniques 1.1.4. Analysis aspects for prioritization of test conditions and procedures 1.1.5. Exploratory test management techniques 1.1.6. Agile testing process (focus on exploratory testing) Expected Results

Level of Learning

The student must understand the basics of software quality and testing, including the exploratory testing approach. In addition to understanding the test design techniques to be used

Remember/Factual Understand/Conceptual

The student should be able to establish a list of suitable work products for analysis and identification of test conditions and test procedures

Remember/Factual Understand/Conceptual

The student must be able to identify the test coverage to be Apply/Conceptual achieved with the test design techniques being aligned with Analyze/Conceptual established test conditions Evaluate/Procedural The student must be able to define a suitable test process according to the analysis performed

Apply/Procedural Analyze/Procedural Evaluate/Conceptual

Programmatic Content - PC 1.2 Implementation of Exploratory Test Procedures 1.2.1. Technique and best practices for writing test charts 1.2.2. Identification of initial test verification criteria 1.2.3. Exploration techniques 1.2.4. Test execution schedule development Expected Results

Level of Learning (continued)

An Exploratory Test Design and Execution Learning Approach

47

(continued) Subject – Exploratory Test Design and Execution The student must understand and apply best practices for writing test charts

Understand/Conceptual Apply/Conceptual

The student must be able to analyze and establish verification criteria for initial tests

Understand/Conceptual Analyze/Conceptual Apply/Conceptual

The student must be able to analyze criteria to enable him to develop an appropriate test execution schedule

Apply/Procedural Analyze/Procedural Evaluate/Conceptual Create/Conceptual

Programmatic Content - PC 1.3 Exploratory Test Execution 1.3.1. Practical execution of exploratory testing using structured exploration and SBMT techniques 1.3.2. Application of good incident recording practices 1.3.3. Application of good incident analysis practices 1.3.4. Application of good practices for maintaining work products as per test results Expected Results

Level of Learning

The student must understand and apply exploratory testing adhering to the pre-defined test charts and selected exploration strategy

Understand/Conceptual Apply/Procedural

The student must understand and apply exploratory testing following structured procedures inherent to the SBTM

Understand/Conceptual Apply/Conceptual & Procedural Analyze/Procedural

The student should be able to understand and apply good practices for recording and analyzing incident causes

Remember/Factual Understand/Conceptual Apply/Procedural Analyze/Procedural

Programmatic Content - PC 1.4 Test and Incident Process Management 1.4.1. Practice reviewing reports exploratory testing sessions 1.4.2. Good practices for reporting incidents to stakeholders 1.4.3. Lessons learned analysis for exploratory testing process management Expected Results

Level of Learning

The student should be able to understand and apply good practice in reviewing reports and writing a test summary report

Understand/Conceptual Apply/Conceptual & Procedural

The student must be able to analyze lessons learned to apply appropriate procedures to the management of exploratory testing process

Remember/Factual Understand/Conceptual Apply/Procedural Analyze/Procedural

48

I. E. F. Costa and S. R. B. Oliveira

References 1. Filho, E.D.deS.: Uma Abordagem para Recomendação de Casos de Teste em Projetos Ágeis Baseados em Scrum. Tese. Universidade Federal de Campina Grande – UFCG. Campina Grande (2021) 2. Bhamare, L., Montvelisky, J.: State of Testing: Report 2021 (2021). https://www.practitest. com/assets/pdf/state-of-testing-report-2021.pdf 3. Elgrably, I., Oliveira, S.: The importance of application of agile tests in the software industry: an exploratory approach using interview. In: 14th International Conference on Information Systems & Technology Management – CONTECSI (2017). https://doi.org/10.5748/978859 9693131-14CONTECSI/RF-4651 4. Pfahl, D., Mantila, M., Yin, H., Much, J.: How is exploratory testing used? A state of the practice survey. In: ESEM 2014, Torino, Italy, 18–19 September 2014. Copyright 2014 ACM 978-1-4503-2774-9/14/09.2014 5. Bach, J.: Exploratory Testing. SATISFACE, Software Testing for Serious People (2015). https://www.satisfice.com/exploratory-testing 6. Gregory, J., Crispin, L.: More Agile Testing: Learning Journeys for the Whole Team. Editora Addison Wesley (2015). ISBN 978-0-321-96705-3 7. Valle, P., Barbosa, E., Maldonado, J.: Um Mapeamento Sistemático Sobre Ensino de Teste de Software. Anais do XXVI Simpósio Brasileiro de Informática na Educação CBIE-LACLO (2015). https://doi.org/10.5753/cbie.sbie.2015.7171 8. Prince, M.: Does active learning work? A review of the research. J. Eng. Educ. 93(3), 223–232 (2004) 9. Costa, I., Oliveira, S.: An evidence-based study on automated exploratory testing. In: 17th CONTECSI, Brazil (2020) 10. Crespo, A., Silva, O., Borges, C., Salviano, C., Junior, M., Jino, M.: Uma Metodologia para Teste de Software no Contexto da Melhoria de Processo. In: SBQS (2004) 11. Naik, K., Tripathy, P.: Software Testing and Quality Assurance: Theory and Practice. Wiley, Hoboken (2008) 12. Costa, I., Oliveira, S.: A syllabus to support teaching and learning of exploratory test design and execution. In: 14th International Conference on Computer Supported Education - CSEDU (2022) 13. Bourque, P.E., Fairley, R.: SWEBOK 3.0: Guide to the Software Engineering Body of Knowledge. IEEE Computer Society (2014) 14. Bach, J.: Exploratory testing. In: van Veenendaal, E. (ed.) The Testing Software Engineer, 2nd edn., pp. 253–265. UTN Publisher, Den Bosch (2004) 15. Kaner, C.: A Tutorial in Exploratory Testing. QUEST (2008) 16. Gregory, J., Crispin, L.: Agile Testing: A Practical Guide for Testers and Agile Teams, 1st edn. Editora Addison Wesley (2009). ISBN-13: 978-0-321-53446-0 17. Huttermann, M.: Agile Record: The Magazine for Agile Developers and Agile Testers (2011) 18. Micallef, M., Porter, C., Borg, A.: Do exploratory testers need formal training? An investigation using HCI techniques. In: IEEE. - Ninth International Conference on Software Testing, Verification, and Validation Workshops (2016) 19. Hendrickon, E.: Explore it!: Reduce Risk and Increase Confidence with Exploratory Testing. Pragmatic Bookshelf, 1 edn. (2013). ISBN 978-1937785024 20. Lorincz, B., Iudean, B., Vescan, A.: Experience report on teaching testing through gamification. In: Proceedings of the 3rd International Workshop on Education through Advanced Software Engineering and Artificial Intelligence (EASEAI 2021), 23 August (2021) 21. Ghazi, A.N.: Structuring exploratory testing through test charter design and decision support. Blekinge Institute of Technology Doctoral Dissertation. Doctoral Dissertation in Software Engineering, Sweden (2017)

An Exploratory Test Design and Execution Learning Approach

49

22. Bach, J.: Session-Based Test Management. Software Testing and Quality Engineering Magazine (2000) 23. Whittaker, J.: Exploratory Software Testing: Tips, Tricks, Tour, and Techniques to Guide Test Design, 1st edn. Pearson Education, Inc., London (2010) 24. Cico, O., Jaccheri, L., Nguyen-Duc, A., Zhang, H.: Exploring the intersection between software industry and Software Engineering education - a systematic mapping of Software Engineering Trends. J. Syst. Softw. 172, 110736 (2020) 25. ACM/IEEE: Computing Curricula: Paradigms for Global Computing Education. ACM and IEEE Computer Society, Incorporated, New York (2020) 26. SBC, Sociedade Brasileira de Computação: Referenciais de Formação para os Cursos de Graduação em Computação (2017) 27. ACM/IEEE: Computer Science Curricula 2013. Curriculum Guidelines for Undergraduate Degree Programs in Computer Science. ACM, New York (2013) 28. Van Veenendaal, E.: Test Maturity Model integration – TMMi Release 1.0: Guidelines for Test Process Improvement. Produced by TMMi Foundation (2018) 29. Van Veenendaal, E.: TMMi in the Agile world – TMMI Release 1.3. Produced by TMMi Foundation (2018) 30. Mattar, J., Czeszak, W.: Avaliação em educação a distância. In: Faria, E., Sousa, H., Fernandes, T. (Orgs.) Educação a distância: textos aplicados a situações práticas. João Pessoa, Gráfica São Matheus (2013) 31. Bacich, L., Moran, J.: Metodologias Ativas para uma Educação Inovadora: Uma Abordagem Teórico-Prática. Porto Alegre: Penso Editora (2018) 32. Prensky, M.: Teaching Digital Natives: Partnering for Real Learning. Thousand Oaks, California (2010) 33. Kaner, C., Bach, J., Pettichord, B.: Lessons Learned in Software Testing: A context-Driven Approach. Wiley, New York (2002) 34. Costa, I., Oliveira, S.: An asset mapping in the ACM/IEEE and SBC curriculum to the test design and execution of the TMMi. In: FIE (2021) 35. Costa, I., Oliveira, S.: A study on assets applied in design and execution activities of exploratory test to be used in teaching-learning: a survey application. In: FIE (2021) 36. Pimentel, M., Carvalho, F.S.P.: Princípios da Educação Online: para sua aula não ficar massiva nem maçante! SBC Horizontes (2020). http://horizontes.sbc.org.br/index.php/2020/05/23/pri ncipios-educacao-online. ISSN 2175-9235 37. Portela, C.: Um Modelo Iterativo para o Ensino de Engenharia de Software Baseado em Abordagens Focadas no Aluno e Práticas de Capacitação da Indústria. Tese (Doutorado em Ciência da Computação). Universidade Federal De Pernambuco. Ciência Da Computação, Recife (2017) 38. Furtado, J., Oliveira, S.: A methodology to teaching statistical process control in computer courses. In: 13th International Conference on Evaluation of Novel Approaches to Software Engineering - ENASE (2018) 39. Elgrably, I.S., Oliveira, S.R.B.: Remote teaching and learning of software testing using active methodologies in the COVID-19 pandemic context. In: Proceedings of Frontiers in Education (FIE) Conference (2021) 40. Cheiran, J.F.P., Carvalho, E.L.S., Rodrigues, E.M., Silva, J.P.S.: Problem-based learning to align theory and practice in sofware testing teaching. In: Proceedings of SBES 2017, Fortaleza, CE, Brazil, 20–22 September 2017, 10 pages (2017). https://doi.org/10.1145/3131151.313 1181 41. Harnish, R., et al.: Creating the foundation for a warm classroom climate: best practices in syllabus tone. APS Observer, 24 (2011) 42. Aguiar, F., Caroli, P.:. Product Backlog Building (PBB): Concepção de um Product Backlog Efetivo (2020). http://leanpub.com/pbb

50

I. E. F. Costa and S. R. B. Oliveira

43. Rooksby, J., Hunt, J., Wang, X.: The theory and practice of randori coding Dojos. In: Cantone, G., Marchesi, M. (eds.) XP 2014. LNBIP, vol. 179, pp. 251–259. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-06862-6_18 44. Da Costa, L.C.F., Oliveira, S.R.B.: Um processo de Validação e Verificação em MPS com foco na implementação multi-modelo. In: 17th International Conference on Information Systems and Technology Management - CONTECSI. USP- São Paulo, Brasil (2020). ISSN 2448-1041. https://doi.org/10.5748/17CONTECSI/PSE-6437

Understanding Geolocation Data: Learning Scenarios for School Informatics Viera Michaliˇcková(B)

and Gabriela Lovászová

Constantine the Philosopher University in Nitra, Nitra, Slovakia {vmichalickova,glovaszova}@ukf.sk

Abstract. In this study, we present two learning scenarios for informatics education in a formal school setting, which aim to make pupils understand geolocation data through authentic outdoor experience with mobile technology. The aim of the study is to comprehensively evaluate the quality of proposed scenarios in terms of the quality of the educational resources, tools and learning activities used. The learning activities in the proposed learning scenarios are inquiry-based. They are mainly collaborative and focused on fostering higher-order thinking skills. The learning scenarios were successfully validated by many primary school teachers and were evaluated as useful, effective, and engaging. For misconceptions or failures of pupils (teachers) that were identified by the qualitative analysis of pupils’ products (teachers’ reports), the causes were described and the instructions it the learning scenarios were refined. Keywords: Learning scenario · Informatics education · Geolocation data · Location-based games · Outdoor learning · Inquiry-based learning

1 Introduction This article is an extension of a paper presented at the CSEDU 2022 conference [1] that was focused on solving the problem of building conceptual knowledge of pupils in the field of data representation and tools for their processing at higher levels of Bloom’s taxonomy (BT) using the inquiry based (IB) methodology. In the paper, one teaching scenario focused on working with geolocation data was analyzed with respect to the achieved level of pupils’ knowledge and the effectiveness of the IB methodology. In this study, we present two learning scenarios for Informatics education in a formal school setting, which aim to make pupils understand geolocation data through authentic outdoor experience. The aim of the study is to comprehensively evaluate the quality of proposed scenarios in terms of the quality of the educational resources, tools and learning activities used. The well-designed inquiry-based learning (IBL) enhances learners’ curiosity and motivation, mediates deeper understanding, and fosters critical thinking. Through IBL activities, pupils are taught to think and act like the scientists do. The meta-analysis of many empirical studies presented in [2] points out that whenever learners act like scientists, their teacher should provide them with adequate guidance. The authors of the © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 51–80, 2023. https://doi.org/10.1007/978-3-031-40501-3_3

52

V. Michaliˇcková and G. Lovászová

meta-analysis found out, that the learners who were given guidance acted more skillfully during the task, were more successful in obtaining relevant information from their investigational practices and scored higher on final tests. As we agree on the beneficial implications of the constructivist learning and teaching strategies, we have decided to apply the 5E instructional model of IBL [3] in presented learning scenarios. Since mobile devices have become available in schools, many educational researchers are focusing on finding effective ways of using them in the curriculum. Yakar et al. [4] explain how mobile technology is being adopted in learning nowadays and how it helps learners and educators with the key aspects of constructivism such as interaction, collaboration, and authentic experience. These create an informal atmosphere where social and cognitive constructivism can be implemented more easily to enhance the quality of formal lessons. In [5], a systematic review of using mobile technology in IBL was conducted. The study aimed at examining to what extent the use of mobile technology for IBL supports and limits learners’ agency. The authors developed an analytical framework helpful for designing effective mobile activities that balance learners’ agency with mobile technology. Their classification includes 12 types of mobile activities derived from the direct instruction, access to content, data collection, peer-to-peer interaction, and contextual support type of mobile activities. Furthermore, they identified 6 dimensions for learners’ agency (control over goals, over content, over actions, over strategies, and options for reflection and monitoring). The way the mobile technology is integrated within the proposed learning scenarios corresponds with their recommendations. Our previous research work was focused on identifying the best practices for using mobile technology in formal and informal Informatics education [6]. The use of mobile technology offers the opportunity to extend the learning space of teaching computer science from the computer room to other areas in the school and even outside the school. Many studies provide evidence of the benefits of outdoor education [7, 8], Therefore, our two learning scenarios take place in multiple learning spaces, including indoor and outdoor environments.

2 Learning Scenarios for Understanding Geolocation Data Learning scenarios for understanding geolocation data aims to provide primary school teachers with ideas for lessons in which: • Pupils work with numerical data representing a geographical location (geolocation data) and with text files with recorded geolocation data in which they experience markup language for the first time. • Pupils meaningfully use mobile devices in computing lessons instead of standard computers, working with the operating system of the mobile device. • Pupils use geolocation sensor, transfer data to/from mobile device, and work in the cloud. • Pupils use web and mobile applications to work with geolocation data – digital maps, navigation software, and location-based games. • Pupils gain experience in new areas of use of digital mobile technologies.

Understanding Geolocation Data

53

The scenarios are designed for upper primary school pupils (7th or 8th grade) who already have experience with various other types of data (e.g., text, numbers, graphics, audio, and video). The scenarios are cross-cutting, the objectives map educational standards from several thematic areas of the national curriculum in Slovakia. 2.1 Design Methodology The basis for the methodology of designing innovative teaching scenarios was the formulation of the didactic problem in the form of the desired changes in the teaching of informatics (Table 1). Table 1. Didactic problem. From

To

Learning objectives on the level of lower order thinking skills (LOTS)

Learning objectives on the level of higher-order thinking skills (HOTS)

Specific knowledge and skills

Contextual knowledge and generic skills

Individual working methods

Collaborative methods

Exclusive SW/HW platform

Multiple SW/HW platforms

Exclusive learning space

Multiple learning spaces

The scenarios shall be designed to meet the following requirements: • Content innovation – new types of data (geolocation data, GPX files with geolocation data), new types of applications (web mapping service, route recording application, location-based game), new hardware and software platforms (mobile devices, mobile applications, mobile operating systems, cloud storage services) • Innovation in teaching methods – inquiry-based learning, game-based learning • Innovation in forms of teaching – outdoor learning, collaborative learning When designing the scenarios, we built on the experience of informal teaching during summer informatics (computer science) camps [8]. These were the basis for the preparation of teaching scenarios for formal school education within the framework of the national IT Academy project aimed at innovating primary and secondary school education in STEM subjects [9]. In the scope of the project, we received feedback on the first drafts of the scenarios from teachers in school practice. Based on the feedback, we adjusted the scenarios and learning resources and expanded the methodological instructions for teachers. This was followed by a 2nd round of validation of the scenarios in practice and finalization of the learning scenarios. 2.2 Evaluation Methodology To assess the quality of the learning scenarios, we used an expert evaluation approach inspired by the evaluation model of Kurilovas et al. [11] that specifies the evaluation

54

V. Michaliˇcková and G. Lovászová

criteria in a hierarchical tree structure separately for the learning objects and the learning activities. We first subjected the learning scenarios to an internal quality assessment based on the criteria set for the proposal. To assess the quality in use, the external evaluation by teachers who had taught the scenarios in practice was applied. We obtained feedback from teachers in the form of structured questionnaires and copies of the learners’ outcomes of each activity. The criteria for internal and external quality evaluation of the scenarios are given in Table 2. Table 2. Criteria for evaluation the quality of learning scenarios.

Internal evaluation

Criteria for the learning objects

Criteria for the learning activities

Ease of use, reusability Structure Technical quality Licence

Conformance with goals Interoperability Flexibility Feedback and assessment

External evaluation Sufficiency for the implementation Adequacy for achieving goals Intuitiveness in use Appropriate language

Feasibility within the proposed time Learners’ background, experiences, expectations Learners’ engagement in learning Interaction and collaboration Appropriate teaching methods

3 Exploring and Recording Geolocation Data 3.1 Objectives In this learning scenario, pupils will learn how to acquire geolocation data using a mobile device with a geolocation sensor, record it in a file and process it using a web mapping service. They will find out what kind of data is recorded in the file and how it can be processed on a computer. The motivation for the pupil’s activity is to record own route that forms a nice meaningful picture. This type of activity known as GPS Drawing combines fine art, physical movement outdoors, and digital technology. By implementing this scenario, pupils are expected to acquire the following specific knowledge: • extracting information from data stored in files using appropriate software applications, • deciding on data processing tools, • becoming familiar with geolocation data files and their structure (e.g., GPX) • using a mobile application to record a route, • transferring a data file from a mobile device to a computer, • processing a geolocation data file using web mapping software.

Understanding Geolocation Data

55

The scenario focuses the development of these higher order computational thinking skills: • identify similar properties of parts of the system (pattern in the data structure in a GPX file). • identify which data is relevant in each situation, and which can be neglected (geolocation data vs. technical data in a GPX file) • formulate problems and solve them using digital technology, • assess the suitability of a digital tool to solve the given problem (tools for displaying and processing the GPX file). 3.2 Design of the Learning Scenario Learning scenario is designed for pupils in the 7th or 8th school year. Pupils are already familiar with the concept of geographical coordinates and can find the coordinates of a place on a digital map or find a place on a digital map with given coordinates. Using a mobile device and a suitable app, they can determine their location and navigate to their destination. Pupils work in pairs as collaboration and communication might be helpful while solving a problem. Worksheets for structuring pupils’ ideas and sample GPX files ready to be processed using the digital map are provided. Teachers can also prepare their own files by recording routes in the field, e.g. around the school. To process the routes recorded in the sample GPX files, teachers need to prepare a shared digital map with layers for each pair of pupils. The learning activities in this scenario are planned for a period of 90 min (2 lessons). The inquiry-based approach was chosen to enable pupils to discover the meanings of data being saved in a GPX file on their own as well as find out the way such files can be processed. The inquiry-based 5E instructional model was applied. It consists of five phases: Engaging. Teacher raises these questions: • What type of files do you know? • What information do these files contain? • What software would you use to view and process them? The answers of pupils are written on a board into a 3-column table (Table 3). Pupils use their worksheets to record all the acceptable answers. The teacher guides the activity so that pupils systematize their knowledge of data types and recall file types with which they have age-appropriate experience from Informatics lessons. Exploring. The lesson continues by posing a problem: What does a GPX file contain? Pupils must find out the answers for the following questions: • What software can we use to look into the file? • What type of data is saved in GPX files? • What questions can be answered from analyzing these data?

56

V. Michaliˇcková and G. Lovászová Table 3. Example of the first activity’s solution.

File Type

Contents

Application

jpg

picture

Paint

sb3

Scratch project

Scratch

docx

text document

MS Word

xlsx

table

MS Excel

pptx

presentation

MS Powerpoint

html

web page

Google Chrome

wav

sound

Windows Media Player

Pupils work in pairs. Each pair is given a different GPX file to examine. During the previous activity, pupils recalled their knowledge of various information types, their digital representations as well as tools for their processing. They have realized that different file types are processed by different applications and applications can manage to process or view the contents of various file types. Therefore, when examining a GPX file, they may experiment with opening it in multiple different applications to discover the most appropriate one. The expected findings are that the GPX file can be opened in a text editor, e.g., Notepad, in a web browser, e.g., Google Chrome, in a spreadsheet, e.g., MS Excel, and it contains a structured text. The numeric data in the file is tagged. Pupils should recognize their meaning rather easily (TIME = time, LAT = latitude, LON = longitude, ELE = elevation, TRKPT = track point) either from their English knowledge or the numerical values themselves. They should also note the repetitive structure of data and hypothesize that it is a record of geographical places in a time sequence (times increase), possibly a record of one’s movement while walking or running. At the end of the exploring phase of the lesson, pupils are asked to formulate some questions that can be answered based on the data in their file. Here are some relevant questions that should be resulted by exploration of the GPX file contents: • • • • • •

Where is the route located? When was the route recorded? What is the length of the route? How long did the recording last? What was the maximum, minimum altitude, what about the altitude difference? What was the average speed of movement?

Explaining. Guided by the teacher, pupils try to explain their observations and findings in their own words. The teacher is responsible for connecting pupils’ informal language with formulations based on using the proper terminology. Next, pupils are asked to think about how the questions generated in the previous activity could be answered. Some of the questions related to time and altitude data could be answered using a spreadsheet software (e.g., minimum, maximum altitude, time of

Understanding Geolocation Data

57

movement), some of the questions related to geographical coordinates using a digital map (where the route is located). The next part of the lesson focuses on working with a file of geolocation data using Google Maps. Pupils will be given a link to a shared map with layers prepared for each pair. Pupils are asked to import their GPX file into the appropriate map layer. They should find out where the route is and name it accordingly. The task of renaming the route according to the location and nature of the route develops critical thinking - the ability to interpret data. In the route description, pupils find out the necessary information about the route to answer the questions they formulated in the previous activity. They record their findings on a worksheet. Pupils present their solutions to each other on a shared map. They get examples of different types of routes: routes creating a meaningful image in a structured urban environment, long hikes in nature with high elevation, routes in the open area creating inscriptions (Fig. 1).

Fig. 1. Examples of different types of routes recorded in GPX files.

Teacher encourages students to discuss the use of GPS tracking in everyday life, e.g. for recording sports performances (tourists, runners, cyclists), to record routes traveled by car (business trips, holidays), for map creation purposes (hiking trails, roads, land properties), for creating interesting images, works of art (graphics, advertisements). Extending/Elaborating. After working with GPX files and using a shared Google map, the lesson continues with the outdoor part of the learning scenario. Pupils should apply the acquired knowledge of GPX files and digital map skills to their own data in a mini project. They will design their own image, which they will record outdoors using a mobile device with GPS and publish it in a shared digital map. They still work in pairs. Each pair of pupils has a tablet or a smartphone with Locus Map Free application installed (or some other app providing similar functionalities depending on the targeted mobile platform). The outdoor part of the learning scenario should take approximately 25 min. A school playground or a larger area around the school with no traffic is suitable for the drawing activity. Suggested motifs for drawing are simple one-stroke drawings (e.g., geometric figures and their combinations); outlines of buildings and other urban objects (e.g., floor plan of a school, playground); inscriptions (continuous script font is more suitable than noncontinuous typed font). After returning to school, pupils can view results of their work first on a mobile device. Then, they transfer the GPX file they have just produced to a computer, either wirelessly or by cable. The final task is to import their GPX file into a common map, giving their drawing a proper name.

58

V. Michaliˇcková and G. Lovászová

Evaluating. Finally, pupils evaluate each other’s drawings and vote for the most beautiful picture (originality and aesthetics are considered). The teacher can also project some additional examples of GPS drawings made by professional GPS artists or other groups of pupils. The experience of being involved in the GPS drawing activity is reflected by pupils and all the acquired knowledge is summarized. Pupils fill in the self-assessment cards included in their worksheets. This helps them with reviewing the contents of lessons as well as systematizing the curriculum.

3.3 Results of Internal Quality Evaluation We evaluated the internal quality of the scenario according to the criteria for learning objects and learning activities listed in Sect. 2.2. The set of learning tools and resources used in the scenario includes: • a worksheet containing 3 tasks with space to write answers, a rubric for selfassessment, and a short summary of key knowledge, • GPX working files for exploring the contents and practicing data processing using Google Maps mapping software, • a shared map ready for uploading files to multiple layers, • Locus Map Free route recording software, • handout with instructions on how to record the route, export the data to a GPX file, and upload the file to the cloud. Ease of Use, Reusability. The worksheet and handout were created for the purpose of this scenario. The worksheet helps to structure inquiry-based learning. The handout contains instructions on how to use the mobile application. The guide consists of screenshots from the application in a handy format – one double-sided A4 sized sheet that can be laminated to make it easier to handle when working outdoors. Both documents are in PDF format and can be reprinted and reused. The documents are not editable. In the case of the handout, it is a disadvantage – once the software is updated, the instructions are no longer usable. The scenario includes a link to a sample shared map on Google Maps and instructions for the teacher on how to create a similar map. The map is not reusable, a new shared map needs to be created each time the scenario is applied in education. Locus Map Free software provides several features that are not used in this teaching scenario. The complexity of the application may be the reason for the higher difficulty of use. We address this problem with the instructions in the handout. Structure. The worksheet is structured in a way that follows the structure of the learning scenario and includes a summary of the key knowledge at the end. The handout contains a sequence of screenshots organized in a two-dimensional matrix in a left-to-right and top-to-bottom direction. Sample GPX files with recorded routes were created to represent different types of routes in terms of route length and shape. The variability in terms of regional distribution is low – 8 routes are located in 2 regions. Shared Google map has a ready-made eight-layer structure for uploading eight files.

Understanding Geolocation Data

59

Technical Quality. The technical quality of the documents was ensured by multiple checks and edits. The software is professional, and its technical quality is ensured by updates. License. All resources are openly licensed for use. The worksheet, handout and sample GPX files are created by the authors of the scenario under a CC BY-SA license. Locus Map Free is a free version of the professional mobile application available in the Google Play application store. Google Maps is a freely available web mapping service. A Google account is required to process geolocation data files. The scenario consists of two parts. In the first indoor part, pupils complete 3 activities in the computer classroom. In the second outdoor part, they work with mobile devices outdoors and finally process the recorded data again indoors. Conformance with Goals. The first activity focuses on systematizing pupils’ input knowledge and preparing them for exploring a new type of data. It aims to stimulate pupils’ curiosity and at the same time to give guidance on how they can proceed when exploring the contents of the unknown file in the second activity. The second activity focuses on the objectives of choosing the appropriate tool to display the contents of the file, identifying relevant and irrelevant data, discovering a pattern in the data structure, and formulating problems. The activity develops pupils’ higher order thinking skills – analysis, abstraction, and formulation of questions. The third activity is aimed at answering the questions formulated in the previous activity. Pupils will learn how to process GPX files using a web mapping service. The outdoor part of the scenario focuses on gaining experience of recording a route using a mobile device and transferring the file with recorded data from the mobile device to a computer. At the same time, pupils’ creativity is encouraged by asking them to draw a meaningful picture. The processing of the recorded route in Google Maps fixes the knowledge acquired in the previous activity. Interoperability, Flexibility. All activities in the scenario are interconnected and link various areas of the Informatics curriculum (software and hardware – mobile devices, wired and wireless data transferring; representations and tools – data and information, data processing tools; communication and collaboration – web services, cloud storage). The flexibility of the activities is positively influenced by the chosen teaching methods - inquiry and project-based method, which provide pupils with a high degree of independence. The organizational difficulty of the outdoor part of the scenario can have a negative impact on flexibility. It is necessary to adapt the activity to the weather, the terrain conditions, and the school location. Feedback and Assessment. All activities have clearly defined outputs that can be monitored and assessed. Pupils record the outcomes of the first two activities in a worksheet. The output of the third activity is an uploaded route in a shared map. Solutions to problems related to the data in the GPX file are written in a worksheet. Proficiency with the mobile application and teamwork are assessed by observing pupils during learning activity. The recorded route embedded in the shared map is an objective output that indicates not only the achievement of the learning objectives but also the level of engagement and creativity of the learner. The worksheet includes a self-assessment rubric that allows pupils to reflect on their learning progress.

60

V. Michaliˇcková and G. Lovászová

3.4 Results of External Quality Evaluation The quality in use of the learning scenario A was evaluated by 30 teachers from 30 Slovak primary schools. Teachers had at their disposal lesson plan with detailed methodological instructions for the teacher and the set of learning tools and resources. The lessons were implemented in 7th or 8th grade classes once or repeatedly with different pupils in the school years 2018/2019 and 2019/2020. After the implementation of the instruction, teachers provided us with data of two types: • written structured reflection in the form of evaluation questionnaires in which teachers commented on the objectives of the learning activities, the extent to which they were met, the content’s complexity, the teaching methods and forms, learning tools and resources • pupil outputs comprising completed worksheets and geolocation data recorded by pupils outdoors using mobile devices and processed in Google Maps We analyzed the collected data qualitatively. The teachers’ opinions expressed in the questionnaires were subjected to an in-depth qualitative analysis to better understand the context in which the pupils worked on the activities and to abstract teachers’ underlying views on the usability of the learning scenario in school practice. Pupils’ outputs were evaluated to identify the level of knowledge attained and to categorize the most common errors and misconceptions. Teachers evaluated the learning scenario from two aspects: the quality and the usefulness of the learning tools and resources, and the quality and the effectiveness of the learning activities. Quality of Learning Tools and Resources. The questionnaire asked teachers to comment on whether the teaching tools and resources were adequate to achieve the learning objectives and sufficient to successfully implement the learning scenario. They were also to provide suggestions for supplementation and modification of the learning tools and resources. Teachers rated the tools and resources as “very helpful and well designed”. Learning resources “helped pupils master the subject matter effectively”. Teachers’ recommendations for improving/modifying learning tools and resources were mainly related to three topics: Equipment Use Policy. Teachers did not take a definite point of view on the use of pupils’ own personal mobile devices or the school equipment: “School tablets have a certain advantage, but, in my opinion, learning how to work with your own mobile phone is more important for the pupil.” “I recommend that teachers use mainly school equipment, but on the other hand, it’s good if the pupil tries it out on his/her own device.” Mobile Application. In the first year of evaluating the learning scenario, teachers suggested changing the mobile application for another one due to the inconsistency of the instructions in the handout with the current version of the application and problems with exporting the recorded route to a GPX file. This resulted in an intervention in the form of a change to the recommended mobile application and the preparation of a new handout. Teachers also pointed out the problem of using non-Android devices: “The only issue we had to address in any way was the use of applications on platforms other than Android.”

Understanding Geolocation Data

61

Class Management. The methodological guidelines for teachers do not contain recommendations how to manage resource sharing (files, URLs). Teachers in the feedback provided information on how they solved this problem: “I have posted the handout and a link to the Google map in the Informatics school subject section on the school website.” “We used Google Classroom for easy access to the all required aids.” “I recommend using address shortening via bit.ly to share web addresses”. Quality of Learning Activities. In the questionnaire, teachers rated whether the activities were feasible within the proposed time and whether incorporated learners’ backgrounds, experiences, and expectations. Teachers mostly reported the feasibility of the activities within the proposed time: “The time allocation is sufficient. In case of favorable conditions, it is not a problem to implement in 1 lesson”. Lack of time was reported in the case of organizational and technical problems with the outdoor part of the scenario – moving outdoors, poor GPS signal, and problems with transferring the file to the computer: “It was cloudy and we had no signal.” “A lot of time is taken up by moving and guiding pupils in completing tasks.” One teacher pointed out potential problems with the feasibility of the scenario with younger pupils due to the age restriction on the use of Google accounts: “The scenario assumes the use of Google accounts, which are not used by every school and pupils can only create them after the age of 13 (which they may not yet be eligible for in Year 7).” The time feasibility of the scenario depended heavily on the level of input knowledge and experience of the pupils. Teachers reported that pupils lacked experience of working with a mobile device beyond the routine use of their favorite applications: “Pupils don’t control their smartphones well enough to be able to work comfortably outside of apps, e.g., with files.” “Many pupils have encountered downloading data from mobiles to computers for the first time.” Further, teachers rated whether the activities employed appropriate teaching methods and the extent to which they facilitated interaction and collaboration and engaged learners in learning. Most of the teachers evaluated the teaching methods positively with no suggestions for change. In rare cases, teachers suggested more traditional transmissible teaching methods as a better alternative: “I also had to use the explanatory method, because the pupils had trouble reading the contents of the file at first…. I would have liked to see a change in the nature of some tasks and to focus more on improving the skills in working with a file in the operating system.” Teachers reported that the outdoor activity was a boost to their Informatics lessons and the pupils enjoyed it: “The pupils were so focused on the work and they were enjoying it so much that it was a challenge to get them off the playground.” “Pupils expected to create prettier pictures, but the topic appealed to them and some tried the app later outside of class.” Learning Outcomes. The extent to which lesson objectives were achieved was assessed from pupil outcomes – completed worksheets and recorded routes.

62

V. Michaliˇcková and G. Lovászová

The First Activity. The pupils had to name various types of files and answer the questions what kind of information they contain and what application do we need to view and process them. There were 41 different answers found in worksheets that we have categorized in 12 categories. Most often, pupils mentioned the files containing text, image or a computer presentation, audio/video, tables, compressed files, programming codes, executable or some other system data files, structured data, special files having an educational content as well as various incorrect answers (http, https, gps, w). In most of the answers (80%), the information kind and the related applications were identified correctly. Though, also some mistakes were noticed: • Pupils did not distinguish between the contents of a file and the tool for processing it. They stated e.g., that the file “xls” contains “spreadsheet” and the processing tool is “Excel”, “doc” contains “text editor” and is processed by “OpenOffice.org”. • Pupils stated http, https (mistaken with html) as the file type. • Pupils identified applications with company names: Adobe (instead of Adobe Reader), Google (instead of Google Chrome), Microsoft (instead of MS Word). • Pupils confused operating system and applications: they presented Android, Windows, iOS, Linux as applications for word processing or websites. Most teachers found the difficulty of this task appropriate to prior knowledge of pupils (“In 7th grade, pupils have already experience with various data files.”). However, for some teachers, the activity was difficult, because “pupils do not know lots of files and are not capable of naming it properly”. According to many teachers, pupils did not understand the assignment at first. But after giving a specific example (e.g. a text file having the txt extension contains text data and a text editor, such as Notepad, is applicable for viewing it), they were ready to respond on their own (“First, pupils were a bit confused, but after a short reminding, they continued further without major problems.”). The Second Activity. The pupils had to find out the answers for the following questions: What software do we need for viewing data in GPX files? What data are contained in GPS files? What information can we get from the data? The purpose of this task was to find out what data the unknown file type contains. To do this, pupils should first find out what application they can use to view the contents of the file. In their worksheets, pupils gave more options. They suggested a text editor, a spreadsheet processor, a web browser, even some map software was mentioned (Google Earth, Google Maps, MapSource, OpenStreet Maps). The incorrect answers comprised applications that are not applicable for viewing GPX files at all, e.g. a graphic editor (Paint) or pupils stated something other than an application (e.g. GPS, XML). The second part of the task was focused on higher cognitive processes: to analyse the text and abstract essential data (time, latitude, longitude, altitude), recognize the pattern in the data structure (route as a sequence of points on Earth arranged in time), and formulate problems (questions) which can be solved using the data in the file. The complexity of the task depended on application pupils were trying to use for viewing the GPX file’s contents. In a simple text editor, the text is not structured so the task is more difficult. More sophisticated editors do hide markup language tags that describe the meaning of the data. The most obvious is the display in a web browser – structured

Understanding Geolocation Data

63

with visible tags. In the unstructured view of the file, pupils noticed an irrelevant link to the page of the application that recorded the route, and “this distracted them from the essentials, and they didn’t have enough time to examine their GPX file well.” In the last part of the task, pupils formulated many good questions: “Where does the route start? Where does it end? In which state/city is the route? How long did it take? What was the average speed? What was the lowest and highest altitude? What is the overall ascent/descent? What color will the route be marked?” The most common mistakes of pupils were: • Pupils reported that their file contained data that was not in it. It was primarily the information that could be calculated from the data in the file (average speed, duration, route length, ascent, descent, graphs). But also, such data were mentioned that are not present in the file at all nor they can be calculated from the available data (map, heart rate). • Pupils also asked questions confirming that they did not realize the fact, the route saved in their GPX file had been traveled and recorded in the past (“Where am I? Where to go? When will the route take place? How long will my route take? How fast will I move?”). • Pupils gave vague or ambiguous wording of their questions (“How high are we? What are the coordinates?”) The Third Activity. The pupils had to process the GPX file using a mapping software to answer questions formulated before. Some solutions contained a transcript of the route data as calculated by the map software not corresponding to questions formulated by pupils in previous task. The rest of pupils tried to formulate answers to their own questions. In case of ambiguous questions, the answers were either missing or clarifying the question (“Altitude? - at the beginning 138 m, at the end 134 m”). Approximately half of the pupils did not name their route, or they gave it a neutral title, the one of their work files (e.g. “file1”). This task was focused on the ability to capture the essence of the file’s contents and verbalize it concisely. For pupils, it was either difficult or they did not consider it important. Even teachers did not pay attention to this in their comments. The Fourth Activity. Pupils recorded routes using mobile devices and after returning indoors, they visualized it in a map software. The results varied as for their complexity – from simple random doodles a few meters long to routes having several kilometers representing meaningful pictures (Fig. 2). Teachers reported problems with procedural knowledge in relation with this part of the teaching: “Pupils did not know how to use the mobile application well, they also had difficulty transferring files.” “Surprisingly, many pupils have encountered the need for downloading data from mobile phones to computers for the first time.” “The map creation (routes in Google Maps) has not yet been automated enough.” There were also problems with bad weather and bad GPS signals. Teachers assessed the involvement of pupils mostly positively: “Pupils were so focused on work and started enjoying it, so it was a problem to get them off the field.” “Pupils expected creating nicer pictures, but the topic was appealing to them, and some

64

V. Michaliˇcková and G. Lovászová

Fig. 2. Examples of GPS drawings named by children as cake, bow, modern art.

of them even tried the application later – outside of class.” “Students like to work with mobile devices and applications.” Self-Assessment. Pupils were asked to fill in a self-assessment section in their worksheets. They were indicating the achievement of objectives by approving provided statements: • • • •

I can name at least 3 types of files and applications for their processing. I can name at least 3 types of data that a GPX file contains. I can view the contents of a GPX file in Google Maps. I can find out the length of the route, duration, average speed, ascent, descent from the data in the GPX file. • I can record a route using tablet or smartphone with GPS and the My Tracks or Locus Map Free app. • I can transfer a GPX file from a mobile device to a computer. Most pupils indicated that all goals had been met. However, there were also answers in which the pupils admitted that they had not acquired some knowledge or skills. Most often, they could not name the file types and applications to process them. They also stated problems with recognizing the meaning of data in GPX files. In the second part of the activity, some pupils admitted that they had not learned to use a mobile device to record a route and transfer a file from the mobile device to a computer. The pupils’ problems they indicated in the self-assessment rubric matched the problems reported by teachers. From their point of view, the least problematic was the using of digital map (pupils could view the routes and reading the statistics). Teachers mentioned only some technical issues related to signing into the Google accounts. 3.5 Discussion The results of the analysis of pupils’ outcomes give us a picture of their knowledge level in the field of data types and data processing tools. Pupils in the 7th or 8th grade have the most experience with text documents, followed by pictures, computer presentations, multimedia, and tables. In these categories, pupils usually did formulate the correct answers using the appropriate terminology, which indicates knowledge at high levels of Bloom’s taxonomy – they know the specific data files, understand that a file is a set of data of some kind, know data files of different types as well as the applications suitable for their processing. Their knowledge is not formal, it is based on experience from lessons.

Understanding Geolocation Data

65

Responses in other categories of data types mostly represented knowledge gained through informal learning. In these responses, we noted more frequent cases of misconceptions and uncertainty in terminology. We rate these answers as naive without deeper understanding. The only exceptions were HTML files, which pupils usually named correctly (age-appropriate), even though this is not the content of the formal lower secondary informatics education. In classes with problems reported, the teachers pointed out, that “pupils know how to work with files (delete, move from folder to folder), but to define what a file is, they have a problem with that”. Thus, they have procedural knowledge of working with files, but lack a deeper conceptual knowledge of files and data types. The most challenging task for pupils was to formulate questions (“It was problematic to formulate questions that can be answered using data from the GPX file.”). One teacher explained his evaluation like this: “Finding the answers, which we can find out from the available files, was initially embarrassing – pupils expected “ready-made” information.” When working with mobile devices, we found that although pupils use tablets and smartphones in their personal lives, their technical skills are limited to working with selected applications. These findings confirm the didactic problem addressed by the innovations in our learning scenario. The proposed learning scenario is aimed to improve pupils’ conceptual and contextual knowledge through using higher levels of thinking skills (file data analysis, abstraction of essential data from the file, data structure recognition) as well as the generic skills (critical thinking, question and answer formulation, technical skills, and creativity). The most important indicator of the quality of the scenario is the effectiveness in achieving the learning objectives. This depends to a large extent on the correct implementation of the proposed IB methodology. In the first year of verification, we received feedback from teachers, in which teachers confirmed the achievement of goals, but also those that described problems and failures. We will describe one case of an unsuccessful implementation of the methodology in practice. In the first phase of the 5E model (engaging), the teacher was to involve pupils in researching what data GPX files contains and what software is applicable for displaying and processing it. Pupils were to consider different types of files and applications. The pupils’ involvement was aimed at systematizing their knowledge of informatics, recalling types of files with which they have age-appropriate experience (they have worked with them in class) and preparing a suitable context for discovering a new type of data in a GPX file. In his report, the teacher stated that he considered this learning activity unnecessary because “it did not pursue the main goal, which was geolocation data.” The teacher’s failure to identify with IB method had been reflected in pupils’ work: “Not only were the pupils demotivated at first because they could not move on their own (I had to intervene), but some also eventually skipped the task to complete it at the end of the lesson.” By omitting the engaging phase, conditions were not created for pupils to examine the contents of the GPX file independently, and the problems continued. According to the teacher, “the inquiry method is not very effective in this case, because it is a relatively difficult task, which is also problem for older pupils.” Instead, the teacher suggested a

66

V. Michaliˇcková and G. Lovászová

traditional explanation of “how to open a GPX file than just asking pupils what software we can display it with” that would be more effective. However, in terms of developing conceptual knowledge, pupils would gain additional isolated, specific knowledge without being placed in a more general context. In the exploring phase, after viewing the contents of the file in Notepad, based on the teacher’s instructions, “pupils identified different information in a tangle of different tags and actually guessed it”, so they managed to discover what data is recorded in the file. However, the teacher added a note to his report that formulating the questions we can answer based on the data in the file is a very similar task to finding out what data is in the file, and “pupils could have legitimate objections as to why they should write questions, when they already know what the file contains.” The teacher does not realize that by processing the data, additional data may be obtained that is not in the source file. This task has been included in the methodology to stimulate analytical and synthetic thinking about how much information is contained in the data. Instead, the teacher suggests formulating ready-made assignments for pupils – what pieces of information they should look for in the data. Since pupils did not formulate the questions that would require processing of the file by a map software, the teacher no longer continued in this part of the activity. He went with pupils outdoors to record routes. Data on whether his pupils know how to process a GPX file with a map software and whether they know how to record a route using a mobile device was not provided. After analyzing this case of unsuccessful implementation of IB methodology in practice and based on other teachers’ feedback, we conducted some interventions in the design of the learning scenario. We slightly modified the learning resources for pupils: we refined the wording of the tasks in the worksheet and updated instructions for working with the mobile application in line with technological progress. We expanded the methodical guidelines for teachers: we highlighted the parts that appeared problematic (misunderstood by teachers) and supplemented them with a list of documented pupil errors. As the main contribution of the learning scenario, teachers stated content innovation: • A new type of data – geolocation data: “Despite the fact that pupils now have a lot of information, the topic of the lesson was new to them and obviously interesting to them.” • New applications – digital map, route recording application. • Working with mobile devices: “Pupils had the opportunity to work with a device other than a computer within a informatics lesson, they used a different operating system.” Innovation of the work form: • Outdoor learning: “creating a space for learning outside the classroom using one’s own mobile devices”, “reviving stereotypical lessons”. • Homework: “voluntary and leisure activities”. • Learning in groups: “the team’s work in the exterior was very popular with pupils as well as with me (the teacher)”.

Understanding Geolocation Data

67

Innovative methods: • Discovery: “Structured inquiry research designed in the methodology appropriately guided pupils to solve tasks and meet goals.”, “IBL method motivates pupils and leads them to creativity and thinking.” As an alternation and extension of the scenario, we present the examples of the outcomes of an activity, in which the pupils complete the images created by GPS drawing in a picture editor (Fig. 3). The activity develops imagination, creativity, originality, aesthetic sense. When pupils expect a more beautiful output from GPS drawing, they may find it satisfying to finish the picture in the editor.

Fig. 3. GPS drawings completed in image editor.

4 Playing and Analyzing a Location-Based Game Digital games for mobile devices are attractive for pupils in general. Moreover, locationbased games are played outdoors and so provide a different kind of experience and additional benefits. Besides physical movement in exterior and learning about leisure activities that support a healthy lifestyle, location-based games promote various social interactions while competing or collaborating during the game’s scenario. Playing locationbased games within informatics lessons also provides an opportunity to discuss principles behind applications based on localization technology as well as their limitations. Design of the presented learning scenario draws on our previous experience of playing location-based games with pupils in summer camps and within multiple workshops. During these informal occasions, we noticed, that pupils were highly interested in outdoor activities that are mediated by mobile technology. The collaborative or competitive aspect of the examined learning scenarios were shown to be likely the most significant features of these methodologies [9, 12]. We have also investigated the use of educational location-based games within the informatics curriculum in lower secondary education.

68

V. Michaliˇcková and G. Lovászová

In [13], examples of location-based games aimed at simulating a graph algorithm and using a stack of tasks to make progress in game were described in detail. In this case, we suggest that playing a simple location-based game and consequent reflection of the authentic experience might contribute to deeper understanding of how geolocation data are used by location-based applications as well as foster multiple other important skills (ability to explore new type of applications independently, analytical, and abstract thinking, recognizing and visualizing structures, software modelling, creativity etc.). We propose, that after being involved in a location-based game actively first, thinking of the internal structure of a game and recognizing connections between user interface and computations behind the scenes can be beneficial also for primary school pupils. 4.1 Objectives In this learning scenario we assume some knowledge and skills related to geolocation data that pupils could have acquired e.g. by being involved in activities described in Sect. 3: • activating and deactivating the localization sensor of a mobile device, • understanding the meaning of geographical coordinates and using them to visualize location of places on a digital map, • using simple mobile application to navigate to a specific target or to record geolocation data. The ability to install mobile applications from online stores and transferring files from computer to mobile devices would be very useful as well. The objectives of this learning scenario include some general as well as very specific ones. General Objectives • • • •

exploring new ways of using mobile devices, exploring new features of mobile platforms (e.g. Android), use various software tools (age appropriate), discussing digital technologies and their impact on the society. Specific Objectives

• explaining what a Wherigo game is, what hardware and software is needed for playing Wherigo games, • using a Wherigo player (e.g. the WhereYouGo application for Android or Wherigo application for iOS) to play games, • explaining and visualizing structure of a location-based game, • downloading quality and safe games from the Wherigo community portal, • promoting motivation for designing one’s own location-based game (e.g, by remixing the sample game). 4.2 Design of the Learning Scenario The learning scenario consists of learning activities arranged according to the phases of the inquiry-based learning, namely the 5E mode was applied. The authentic learning

Understanding Geolocation Data

69

through positive experience while playing game with classmates is also an important factor boosting the motivation for next, more demanding activities. Pupils work in pairs while using tablets equipped with GPS receivers during the outdoor part of the scenario or as a group while discussing in the class afterwards. For pupils, worksheets are prepared to guide them throughout the lesson. For teachers, some additional practical information about the Wherigo platform is provided as we anticipate that the topic is probably unknown to most of them. A sample Wherigo game was developed (an adventurous fiction about a teddy bear who is familiar to most Slovak children from a popular television show) to introduce main components and processes comprised in a typical location-based game, including the professional commercial products. Teachers are instructed to play the sample game themselves before implementing the scenario with pupils to be able to guide their pupils properly or to manage possible technical or organizational issues. For diagnostic purposes, we use the self-assessment rubric included in worksheets. Teachers are asked to observe pupils’ actions and record their verbal comments while playing or discussing. For getting feedback from teachers, the questionnaires are used. Pupils’ solutions written in worksheets were also available for further analysis. The learning scenario is meant to be used in a formal instructional context of 2 informatics lessons (90 min) in primary schools (12–14-year-old pupils of the 7th or 8th grades). In following sections, the 5 phases of the inquiry-based model are described: Engaging. The experience of playing any geolocation game can be significantly marred by inclement weather. Pupils’ safety is also important. We therefore always choose terrain that we know well, away from traffic, preferably in the vicinity of the school (schoolyard, playground, park, square, etc.). The lesson should start at the playing area. First, we talk about playing computer games with pupils. We are interested in why they like their favorite games and how much time they spend on them. Specifically, we ask pupils about games for mobile devices. Together with pupils, we should discuss the essential differences between mobile and desktop games: • the smaller screen size may be a disadvantage, but the computing power of modern mobile devices is close to desktop computers, • applications (and games) are available online in app stores, • mobile devices are portable, equipped with a variety of sensors (including the location sensor), support both touch and voice control, provide access to the internet and its services (so often, developers come with original applications as for their purpose or the user interface). Some pupils may mention some location-based games too, probably a commercial one. Nevertheless, we point out all the health-related benefits of playing outdoors and having fun with friends or family members. We introduce Wherigo games, that are available online for free playing and is also rather easy to create them, also for creative pupils or novice programmers. Exploring. This phase is focused on playing a location-based game to gain an authentic experience. We recommend preparing one mobile device for number of 2 or 3 pupils with Wherigo player and sample game already installed, so pupils can immediately learn

70

V. Michaliˇcková and G. Lovászová

about the way of starting a game as well as compare their screens while performing their first experiments together. The sample Wherigo game takes approximately 15 min (zones are deliberately placed close to each other). As the user interface of the Wherigo player is rather intuitive and robust, pupils should be able to explore it on their own with minimal instructions from teachers. However, the initial guidance should include the joint reading of the storyline (to be sure that all pupils understand the goal), navigating to a zone, interacting with a virtual item found inside the zone, using the inventory, and viewing the list of tasks for the first time. From then on, teams should continue on their own. The position of zones used in the sample game are calculated dynamically according to the user’s initial position after starting the game, so the individual teams are likely to move on different paths when starting on different spots. Teacher asks pupils to pay attention to what is happening during the game play (while moving around the playground and on their device’s screen). During the game, pupils look for clues needed for solving the mystery stated in the storyline. In each zone, they take something useful into the inventory or meet a character to talk to. Teacher concludes the outdoor phase of the scenario by reflecting experience of individual teams and asking each team for one of the clues saved in their inventory. To solve the main mystery of the storyline, pupils must reveal the secret place with given geographical coordinates by using a digital map. Without mobile connection, pupils complete the final task after returning to school, in a computer laboratory. Explaining. The learning scenario continues in a computer laboratory, where pupils use mapping software (e.g. Google maps) to visualize the place at the coordinates discovered over the course of the outdoor play (it is the Eiffel Tower in Paris, thanks to the Google Street View, they can look around the tower’s surroundings, though it is not necessary for the game’s goal). During the explanation phase of the learning scenario, pupils are involved into 2 activities. Both are grounded in reflection of the previous outdoor experience. Teacher poses a series of questions. Discussion should end with summarizing and systemizing new knowledge using proper terminology (see Table 4): In Activity 2, pupils think of the games’ course trying to analyze it from a programmer’s point of view. They work individually or in teams using their worksheets. Tasks are formulated so that pupils can train the proper terminology again (Table 5). The reflection on the game is strongly supported through tasks in worksheets and should help pupils with recognizing key components comprising the game as well as getting an idea of how the Wherigo game is being processed in the Wherigo player application. Extending/Elaborating. During this phase of the 5E cycle, pupils are browsing through the Wherigo portal as their teacher presents the way of searching other Wherigo games. There are various types of Wherigo games (puzzles, fictions, tourist guides or sport-like ones). Some of them are bound to specific locations, the other can be played anywhere where the area for playing is large enough. One of the most valuable features of the Wherigo platform is the world-wide community of players and creators of games. For each game, a short description presenting the game is available. If needed, some instructions or warnings useful for successful playing are provided. Players can share their positive or negative experiences by writing a comment. Feedback from community members is very useful for both the creators, and the players of games. We recommend that teachers prepare carefully selected examples of various Wherigo games that are

Understanding Geolocation Data

71

Table 4. Question and answers from the Activity 1. Question

Example of a correct answer

What hardware do we need to play a Wherigo game?

A mobile device (tablet, smartphone, or tourist navigator) with GPS receiver

What software do we need to play a Wherigo A Wherigo player for our mobile platform (the game? mobile application for Android operating system is called WhereYouGo, Wherigo for iOS) Why is it necessary for our mobile device to have a location sensor?

The WhereYouGo application needs to know the actual position of the player to display all information while running the game (available zones, visible content - objects and characters, new tasks, etc.)

Is there anything else we need while playing a Wherigo game?

The game file (with gwc extension), safe terrain, good weather

Table 5. Tasks from worksheets used in Activity 2. Task

Expected solution

Can you remember what zones we visited Pupils write their answers into a 3-column during the game, what we found and who we table. They should recall names of 5 zones, 5 met? items found after entering them as well as a snowman character from the target zone Draw an approximate map with the zones. Also show the order in which you visited the zones during the game

Pupils are expected to use an oriented graph for modeling the sample game’s structure and dynamics (zones are vertices and oriented edges show the path of team’s movement during the game)

What tasks did we have to complete during Pupils write their answers into a 3-column the game? When and how did we solve them? table trying to recall every single task (e.g. to find the X number) and identify the moment of its solution (e.g. the moment of entering the so-called Red Zone) together with the way this task was solved (e.g. by taking the X number into the inventory) Name all items we had in the Inventory at the X, Y and Z numbers needed to calculate end of the game location of the mysterious place; bottle with a message; photo of the teddy-bear character as a special reward for answering the snowman’s question correctly)

72

V. Michaliˇcková and G. Lovászová

situated nearby or might be attractive for their group of pupils. Depending on interests and programming skills of pupils, we can also discuss the possibility of making Wherigo games. For lower secondary education, we found the block based Urwigo builder applicable. By studying simple tutorials or source codes of the other authors’ games, one can create a simple Wherigo game rather easily. Any Wherigo game is a multiplatform game. It runs on various mobile platforms because it is the Wherigo player that interprets the running code. Evaluating. The lesson ends with reviewing its contents. Pupils check those items in the list of educational objectives that they considered as being achieved personally. This self-assessment rubric is the final section of their worksheets. Pupils are also asked to choose one game from Wherigo portal they would like to play later to demonstrate the level of interest in the topic.

4.3 Results of Internal Quality Evaluation The methodical package for teachers includes: • • • • • •

IBL methodology of the leaning scenario with series of learning activities, worksheet document to prepare handouts for pupils, worksheet document with sample solutions, study material about the Wherigo platform, Wherigo game file, document presenting the course of sample Wherigo game from a player’s viewpoint in detail.

When evaluating the quality of individual component in the package, the criteria presented in Sect. 2.1 were considered. We compared the original design intentions with achieved results, both for learning tools and resources and learning activities: Ease of Use. All documents prepared for teachers are well-formatted and clear. The information for teachers is complex and comprehensive. The learning scenario is described respecting the needs of a teacher who is inexperienced in the subject. Wherigo game developed for the outdoor phase of the scenario is playable well. Also, the Wherigo player needed for playing is easy to control, even for first-time users. The storyline of the sample Wherigo game is entertaining and easy to read, tasks for players are manageable in short time. Reusability. The Wherigo game provided in the methodical package is a location-based game, though, it is not bound to any specific location. Just a sufficiently large space must be available in school’s surroundings to play. Wherigo games run on multiple mobile platforms as they are handled by the player applications available in related application stores. Storyline of the sample game is based on a popular TV show for preschoolers. For older pupils, the other storyline might be more attractive. For teachers with programming experience, also source code of the sample game might be interesting to edit the storyline or alternate tasks for pupils.

Understanding Geolocation Data

73

Structure. The Wherigo game used within the scenario, was designed carefully to demonstrate all fundamental components of a typical Wherigo game. The game is wellstructured to make the interaction with user interface easy and quick. The structure of game is age-appropriate, so pupils can think of it and are able to create a simple model. Technical Quality. The Wherigo platform chosen for introducing pupils to locationbased games is verified by many users from the online community and is reliable. The sample game was designed and programmed the way that confirms with standards for Wherigo games. Number and locations of zones, number of items in the inventory and the number of tasks to be completed respects the time restrictions of a formal school setting. License. The methodical package is in the public domain under a CC BY-SA license, teachers can download it from the project’s repository. Wherigo platform is also available for free, both the player apps and builders of games. Conformance with Goals. All learning activities in the scenario are goal oriented. Playing a Wherigo game make pupils ready for subsequent discussion about the geolocation data processing as well as creating the game’s model. The indoor learning activities are focused on fostering higher order thinking skills. As pupils work with mobile hardware and the Wherigo software actively, they get also new skills needed to play similar games on their own. The informative ending with presentation of the Wherigo portal can motivate pupils to try other Wherigo games later, though it would be better to train the procedure of finding and downloading a suitable game practically at school. Interoperability. The learning scenario integrates and extends knowledge from various thematic areas of the informatics curriculum used in Slovak primary schools (software and hardware – mobile devices, data processing; information society – digital technologies in society; representations and tools – data structures; communication and collaboration – community portal). Since the sequence of learning activities is grounded in the IBL model, the scenario is coherent. Flexibility. Though we recommend 90 min (2 lessons) for realizing the whole scenario as is, teacher can adapt to local conditions. In case, the outdoor part of the scenario is realized separately, the in-class activities may need more instructive approach. For teacher capable of remixing the original Wherigo game, editable worksheets would suit better. For successful implementation, satisfactory whether and terrain conditions are needed. Feedback and Assessment. The learning goals are measurable as learning activities result in objective outputs. Teachers are encouraged to reflect the outdoor experience of pupils thoroughly to select optimal level of guidance for indoor part of the scenario. The self-assessment section added to worksheets contains formulations that are brief and concise, so are likely to help pupils with final recapitulation and systemizing of new knowledge. Teams play the location-based game independently, so teacher can observe their activity, respond to technical problems, and assess competence in the skillful use of the mobile device.

74

V. Michaliˇcková and G. Lovászová

4.4 Results of External Quality Evaluation The learning scenario was implemented by 21 primary school teachers in two phases (school years 2018/2019 and 2019/2020), 5 of them repeatedly. Pupils from various grades were involved (73% of implementations were made with 7 or 8 graders, 12% with younger pupils from 5th and 6th grades, and in 15% cases, older pupils from 9th grade were involved). Teachers who implemented the learning scenario provided feedback through a complex questionnaire. Besides this structured reflection on methodical quality of the proposed scenario including authentic observations from lessons, also worksheets of all pupils were analyzed. The qualitative analysis of answers from questionnaires together with worksheets’ analysis resulted in following findings: Quality of Learning Tools and Resources. Most teachers appreciated the learning scenario package as being of a high quality. The instructions provided were sufficient for successful implementation of the scenario. Documents dedicated to Wherigo platform and detailed description of the sample game’s course through series of commented screenshots from the Wherigo player’s user interface were helpful for all teachers, as they had no previous experience with the topic. The language used in methodical material and worksheets was assessed as appropriate. The sample Wherigo game was accepted positively by teachers, they did not report significant technical issues concerned with the game itself, nor controlling the Wherigo player’s user interface. Some suggestions were made to make the final task for pupils more explicit. There were also requests for using bigger font for texts, to make them more readable on the screen of mobile device. Next, we list some authentic quotes from questionnaires: “Thank you for the detailed supplementary material, it helped me a lot.” “This methodology is extremely useful. So, I have no suggestions for modifications.” “The methodology is well worked out; no revision is needed.” “It complements the curriculum in an appropriate way, as it introduces pupils to the new possibilities of using mobile devices.” “We will use it in the future. Great ideas for practical use of geolocation.” “I had no difficulty using the prepared materials. It was necessary to master the new program, but it really was not difficult, so everything went smoothly.” “As I said above, I would modify the wording of the objective of the game, directly in the game, in the form of clearer instructions.” “I would add instructions on what to do if the game gets stuck or doesn’t work properly to the game player manual - instructions on how to restart. The texts were displayed in small letters, harder to read.” Quality and Effectiveness of the Learning Activities. Teachers reflected their experience from the verification process by evaluating various aspects of the learning scenario. Feasibility Within the Proposed Time. As the topic was completely new for all teachers, many of them reported that preparing for the class was time consuming. Teachers with weak digital skills presented a bit frustration as they had difficulties with installing Wherigo software and the game file to mobile devices, especially when the BYOD strategy was applied. The learning scenario was planned for block of 2 informatics lessons. Most teachers stated that the time estimate was reasonable. Three of them had

Understanding Geolocation Data

75

to realize the 2 lessons separately within two different days. Some teachers described that the walk to the playing spot and back took too much time. We noticed also multiple suggestions for adding some extra time for working with Wherigo portal during the elaboration phases as pupils were interested in searching for other games: “The time allocation is sufficient. Even with transfer to the park and back, we managed to do it all beautifully.” “The scenario is feasible within the timeframe, so I have no suggestions for modifications. Of course, the time depends on distance from school where there is a reasonably large space to play the game.” “Due to the smaller number of pupils, we had enough time, but everything was prepared in advance.” “As the pupils were using their own mobile devices, it was not easy to get the game file onto their devices.” Learners’ Background, Experiences, Expectations. Since most pupils were also involved in learning scenario A, they were familiar with concept of geolocation data. Teachers assessed their initial knowledge as sufficient for the learning activities. Pupils were skilled at using a tablet or mobile phone. And were able to install mobile applications from application stores, though transferring files from computer to mobile devices was problematic. The idea of playing a location-based game was accepted by pupils very well as their informatics lessons usually take place in a laboratory, where only desktop computers are used. In case of bad weather conditions or bad quality of GPS signal, some team could not finish the game and were disappointed: “The pupils had knowledge about GPS, its principle and functioning. The pupils were able to work with mobile device without any problems.” “The transfer and preparation of the game was done by teacher, so pupils did not need this skill. The other prerequisites are appropriately chosen.” “Working with phones is not at such a level that they could find the necessary files and copy them on their own.” “Most pupils, not only from the previous lesson but also from real life, had adequate initial knowledge.” “The Wherigo game was very interesting and fun for the pupils. They were learning of principles behind location-based applications while playing a game.” “They find it attractive that they can use mobiles, tablets and even play games and that they don’t just sit at a PC but can also move around.” Learners’ Engagement in Learning. Teachers described the engagement of pupils positively. The game was dynamic and though it was not meant as competition, teams were trying to finish it first. The initial talk about computer games as well as the game play activity was satisfying and enjoyable experience for pupils in general. The subsequent reflection of what was happing while playing the game was difficult for pupils frustrated by technical problems while playing: “Also, girls from 8th grade who were quite sceptic about the storyline were enjoying the game finally.” “Pupils discussed their experience actively, they were proud of finishing the game and saving the teddy bear.” “It was very difficult to make pupils willing to analyze the game’s content when they could not finish is successfully due to bad signal conditions.”

76

V. Michaliˇcková and G. Lovászová

Interaction and Collaboration. In all learning activities, the positive effect of collaboration was observed. During the engagement phase, team were helping each other if needed. Teachers pointed the significance of joint discussion during the explaining phase, especially in cases when pupils have difficulties of recalling some details from previous lesson. The Wherigo platform itself is an example of collaborative platform, as users share their games and experiences from playing. As we have recommended, most teachers divided their pupils into teams of 2 or 3 pupils, but would preferred the 2-member alternative: “Group work, cooperation and being outdoors are a good combination for working with pupils.” “I would recommend working only in groups of 2, so pupils could interact with the device equally. Also the screen size of a mobile device is rather small.” “In teams, pupils complement each other and divide the work.” “Two female pupils were unhappy that application didn’t show them where to go. The boys helped them by showing how to enable the GPS in their tablet.” Appropriate Teaching Methods. In their responses, teachers demonstrated their understanding for teaching methods employed in the learning scenario. The inquiry-based learning activities were assessed as being very beneficial and well-ordered because pupils have chance to explore new software on their own and acquire new knowledge actively in discussion about their authentic positive experience. Collaboration in group was very important also during the indoor part of the learning scenario as it provided additional guidance for pupils. The learning activity focused on visualizing the game’s structure were the most demanding for pupils, but fosters the abstract ana analytical thinking: “Structured exploration, experiential learning and group work are very popular with pupils.” “Experiential learning is a very appropriate method of learning and probably the most effective.” “I liked the forms and methods chosen. Especially, that while filling in the worksheets, both independent and joint work was suggested. Pupils who failed the game did not experience another failure.” “The IBL method was used effectively – pupils discovered principles behind the game on their own. They also recognized the causes of some technical problems and suggested solutions.” “Pupils were surprised, that other teams were moving along different paths. But they found out the reason themselves.” “Without a proposal for modification – certainly with a different composition of the class, a more stimulating discussion would arise.” Learning Outcomes. The general goals concerning the hardware and software skills of pupils were considered fulfilled by all teachers. The user interface of the Wherigo player application was assessed as intuitive enough. In 2 cases, teachers referred that pupils were not able to navigate to zones while playing the game, so they were just observing what the other teams are doing. In this case, the teacher was not competent to help them. Younger pupils have problems to understand the storyline, the teacher attributed it to a lack of reading comprehension skills. The specific goals related to downloading games

Understanding Geolocation Data

77

from the Wherigo was not achieved in most cases as pupils lack the skills of transferring files from computer to mobile devices or more time for training would be in hand. Pupils were able to explain what hardware and software is needed to play a Wherigo game as well as understand the creative contribution and support provided by members of the online Wherigo community. The worksheet tasks and hints in headings of tables were helpful for recognizing that for a location-based game, the geolocation data is the crucial input. When pupils did not remember the layout of the zones and other details of the game, it was due to the large time gap from the previous lesson. When solving tasks in worksheets individually, pupils have difficulties of using the proper terminology and were not able to identify some components or events in the game correctly: • Pupils did not distinguish between tasks displayed in the player in the Tasks list and the questions that they answered in the game while completing the tasks. • Pupils could not correctly identify and name the ‘moment of solution’ of the problem. By solving the problem, some understood “walking” or navigating to the zone by using the compass. • The pupils considered themselves as a figure present in the zone. • Pupils did not use an oriented graph to represent the structure and flow of the game (Fig. 4).

Fig. 4. Two examples of visualizing the game’s structure using a graph. Both pupils also drew the representation of a player and captured their path with oriented edges.

We noticed that almost all cases with misconceptions of pupils were caused by minor or major errors made by teachers when implementing the scenario. Nevertheless, in majority of cases, pupils were able to create a simple model of the location-based game correctly and were motivated to try some Wherigo games also at home. There were also teachers (in 3 cases) that would like to play Wherigo games with pupils within some informal, after school setting or would like to program Wherigo games with pupils as well.

78

V. Michaliˇcková and G. Lovászová

4.5 Discussion The external analysis results showed that teachers were classifying the learning scenario as being of a high quality (the overall learning scenario design as well as the additional study resources and instructions provided in the teacher’s package). After analyzing the research data from the first verification phase, some interventions were made to refine the scenario for the second run to avoid the errors or delays during implementation or explain the sources of misconceptions. Next, we list some of them in short: • When first getting familiar with Wherigo games, we recommend that pupils use the school’s mobile devices. Installing the player and copying the game file will not take much time for the teacher. In addition, the devices will remain ready to play Wherigo games as part of other activities with the pupils in the future. • For most pupils, this is their first experience of playing Wherigo. The start of the game and the introductory part of the game should therefore be given special attention and pupils should be guided accordingly. The text of the story should be read aloud by pupils (one part per team). The font size can be adjusted in the player. After reading the introduction text, teacher should check that each of the pupils knows what mystery they are going to solve and why they must move around the playground for it. • No internet connection is required to play the game in the field. However, the game experience is affected by the quality of GPS signal quality. In case of problems with detecting a player in a zone, we recommend leaving the zone (moving away by about 10–15 m) and trying to return (moving, do not standing on place). At signal outages, it is advisable to save the game, restart it and restart the GPS receiver on the device. • If for organizational reasons it is not possible to implement the learning scenario as a two-hour session, we recommend that the completion of the worksheet is done from the beginning as part of a whole-class discussion, writing the answers on the board. Teachers should guide pupils with good questions. • To make pupils aware of difference between the moment of solution and the way of solving the problem, we can encourage pupils to think about solving each problem as follows: “When we entered this zone, we did this…”. • Drawing a simple model of the game played (an oriented graph with vertices representing zones and edges indicating movements between zones) is an important informatics goal of the lesson. If pupils do not come up with this method of graphical notation on their own, we ask them what information is taken from their picture about the game and what information about the game can and cannot be found out. • The elaboration phase of the learning scenario dedicated to Wherigo portal with games is intended to show that playing Wherigo games is a popular activity, and the authors of games are ordinary people, creative enthusiasts with good ideas, not necessarily professional programmers, some games are location-specific, and others can be played anywhere. The portal can be searched without logging in. The teacher should search for games in advance and explore the site with the pupils. • Homework on designing one’s own game is a voluntary activity for creative pupils and includes planning the location of the zones, their content, writing a short story with tasks for the player (Fig. 5).

Understanding Geolocation Data

79

Fig. 5. Design of a location-based game for Wherigo platform made by a 6th grader. He remixed the sample game by altering all tasks, items and zone captions according to his own.

References 1. Lovászová, G., Michaliˇcková, V.: Exploring geolocation data: an inquiry-based methodology used in lower secondary education. In: Proceedings of the 14th International Conference on Computer Supported Education, vol. 2, pp. 351–360 (2022). https://doi.org/10.5220/001103 6300003182. ISBN 978-989-758-562-3, ISSN 2184-5026 2. Lazonder, A.W., Harmsen, R.: Meta-analysis of inquiry-based learning: effects of guidance. Rev. Educ. Res. 86(3), 681–718 (2016) 3. Bybee, R.W., et al.: The BSCS 5E Instructional Model: Origins and Effectiveness, Colorado Springs, BSCS (2006). 65 p. 4. Yakar, Ü., et al.: From constructivist educational technology to mobile constructivism: how mobile learning serves constructivism? Int. J. Acad. Res. Educ. 6(1), 56–75 (2020). https:// doi.org/10.17985/ijare.818487 5. Suárez, Á., et al.: A review of the types of mobile activities in mobile inquiry-based learning. Comput. Educ. 118, 38–55 (2018). ISSN 0360-1315 6. Lovászová, G., Cápay, M., Michaliˇcková, V.: Learning activities mediated by mobile technology: best practices for informatics education. In: Proceedings of the 8th International Conference on Computer Supported Education (CSEDU 2016), vol. 2, pp. 394–401 (2016) 7. Malone, K., Waite, S.: Student Outcomes and Natural Schooling. Plymouth University, Plymouth (2016). http://www.plymouth.ac.uk/research/oelres-net 8. Waite, S.: Where are we going? International views on purposes, practices and barriers in school-based outdoor learning. Educ. Sci. 10(11), 311 (2020) 9. Cápay, M., Lovászová, G., Michaliˇcková, V.: Learning activities suitable for an ICT-oriented children’s summer camp. Proc. Soc. Behav. Sci. 180, 510–516 (2015). https://doi.org/10. 1016/j.sbspro.2015.02.152. ISSN 1877-0428 ˇ Guniš, J., Klein, D., Kireš, M.: Active learning in STEM 10. Ješková, Z., Lukáˇc, S., Šnajder, L, education with regard to the development of inquiry skills. Educ. Sci. 12(10), 686 (2022) 11. Kurilovas, E., Zilinskiene, I., Ignatova, N.: Evaluation of quality of learning scenarios and their suitability to particular learners’ profiles. In: Proceedings of the 10th European Conference on e-Learning (ECEL 2011), pp. 380–389. Academic Publishing Limited, Brighton (2011)

80

V. Michaliˇcková and G. Lovászová

12. Palmárová, V., Lovászová, G.: Mobile technology used in an adventurous outdoor learning activity: a case study. Probl. Educ. 21st Century Recent Issues Educ. 44(6), 64–71 (2012) 13. Lovászová, G., Palmárová, V.: Location-based games in informatics education. In: Diethelm, I., Mittermeir, R.T. (eds.) ISSEP 2013. LNCS, vol. 7780, pp. 80–90. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-36617-8_7

Game Design Tools: A Systematic Literature Review: Choice of a Game Design Tool for an Experimentation in the Nursing Field Sebastian Gajewski(B) , Nour El Mawas, and Jean Heutte Univ. Lille, ULR 4354 - CIREL - Centre Interuniversitaire de Recherche en Éducation de Lille, 59000 Lille, France {sebastian.gajewski.etu,nour.el-mawas,jean.heutte}@univ-lille.fr

Abstract. Prior studies have demonstrated that game-based learning plays a role in enhancing student learning and increasing student motivation. Game design tools are available today (for free or for sale) and user-friendly even by people without any technical skills. However, these game design tools are plentiful. This complicates choosing a game design tool. This article introduces and compares existing game design tools to help people to choose the one that best suits their needs. We started by identifying nine key criteria that characterize game design tools. We then performed a systematic literature search using the PRISMA method. Of 302 identified studies across five databases (IEEE Xplore, ScienceDirect, Scopus, Springer, Web of Science) and 8 game design tools advised by a digital learning manager, 11 game design tools are explained, compared and discussed. Finally, we used the results of this systematic review to select a game design tool for an experimentation in a nursing school. This research work is dedicated to the Technology Enhanced Learning and Educational community, especially game designers, digital learning managers, teachers, and researchers who are struggling to choose the game design tool that best suits their needs. Keywords: Game design tool · Systematic literature review · PRISMA · Experimentation

1 Introduction An individual will now have many different work opportunities during his or her lifetime. With the university as the starting point and continuing through the professional career with various employment, lifelong learning is emerging as a key benefit. Adults’ needs for education and training change over the course of their life [1]. Prior research works demonstrated the role of game-based learning [2] to increase students’ learning [3] and to improve their motivation. So, this research work is about gamebased learning, more specifically about the game design tools used in those learning activities. Game-based learning includes gameplay-based learning and game design-based learning [4]. In the gameplay-based learning, students play a game to learn. Whereas, in © The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 81–99, 2023. https://doi.org/10.1007/978-3-031-40501-3_4

82

S. Gajewski et al.

the game design-based learning, students learn by designing their own game. In the two cases, there is a need to choose a tool to design the game. In this research work, we present a systematic literature review which is intended for the game designers who will develop games which will be played by students (gameplaybased learning), and for teachers who will choose the game design tool the students will use in their classrooms to design their own games (game design-based learning). This research work is a part of the experimentation of a game co-design method we have developed [5]. In this experimentation, second-year nursing students have codesigned games about liver cirrhosis to help them to learn about this topic. The aim of this experimentation was to assess the effects of this co-design activity on students’ learning and social flow. For this aim, we needed to identify the most suitable game design tool that the secondyear nursing students will use during the learning game design activity. However, it is quite challenging to get a comprehensive picture of the research environment for studies on game design tools in the current GDBL context. For instance, at the time this study was being written, a straightforward, unrestricted Google Scholar search for the keyword “game design tool” produced around 2,900,000 results (early 2022). This research work is dedicated to Technology Enhanced Learning community and pedagogical community, and more specifically to game designers, digital learning managers, teachers, and researchers. This paper is structured as follows. Section 2 identifies criteria that characterize a game design tool through existing research works. Section 3 details the different steps of the PRISMA method. Section 4 discusses the results of this systematic literature review comparing to the identified criteria. Section 5 presents the use of the results of the systematic review in order to choose the suitable game design tool in a nursing experimentation. Finally, Sect. 6 concludes this paper and presents its perspectives.

2 Related Work This section presents important criteria for characterizing a game design tool. These criteria were identified from research works on game design-based learning. In these research works, the authors explain why they have chosen the game design tools they have used. Nine key criteria have been identified: programming language (C1), Tool language (C2), tutorials (C3), scenes and characters (C4), game type (C5), target audiencedesigner (C6), 2D or 3D modelling (C7), prize (C8), and Export (C9). “Whereas programming a video game traditionally required extensive typing in which the smallest syntax error could offset game play altogether, multiple tools rely simply on a “drag-and-drop” approach to coding” [6]. Whereas, Unity uses C#, a programming language using lines of code, in Scratch “users drag and snap command blocks together to build scripts” [7]. Scratch is an interface using “visual code blocks, reducing syntax errors” [7]. Regarding the tool language, some game design tools offer the possibility to select different languages for the game design interface. Others are restricted to English. Some game design tools provide tutorials (manuals, videos, etc.), and sometimes users can choose the tutorial’s language. Sometimes, the tutorial is only

Game Design Tools

83

in English. Some game design tools provide resources, as backgrounds for the scene, and sprites for the characters. If the game design tool does not provide these scenes and characters, users have to draw by themselves and develop character designer skills. Some game design tools are restricted to only one type of game. That is the case of RPG Maker VX Ace which “contains some restrictions, including that the games must be role-playing games” [8]. Other game design tools allow users to choose the type of game they want to develop. For example, Microsoft Kodu allows “users create different types of games (e.g., adventure, arcade, racing)” [9]. Target audience-designer is about the public. Are they professional or novice? For example, RPG Maker VX Ace “requires little programming or game design experience to create games” [8]. Are they adults or children? For example, Scratch is a “user-friendly interface for children” [7]. 2D or 3D modelling is about the visual aspect of the game: 2D or 3D. According to [10] “Compared to 2D environments, the ability to create 3D games in Kodu makes it visually more appealing for young students”. Regarding the prize, some game design tools are freeware. This is the case of Scratch which is “free of charge” [7]. For others, users have to pay a fee for a license. Regarding the export, some game design tools allow the users to upload and share their games in the game design tool’s community website. For example, in Gamestar Mechanic, “Game Alley is where students share their games in an online community” [11]. Regarding the export is about the platforms on which the game will run: Microsoft, Mac OS, iOS, Android, Console, HTML5, etc. In the next section, we present the method followed for the literature review of game design tools.

3 The PRISMA Method This literature review follows the method described by [12], and includes the following steps: developing a review protocol, identification of the need for a review, specifying the research question(s), identification of research, selection of studies, data extraction, data synthesis, and reporting the review. 3.1 Developing a Review Protocol “A review protocol specifies the methods that will be used to undertake a specific systematic review” [12]. The steps used in this literature review are described in the following sections. 3.2 Identification of the Need for a Review In [5, 13], authors have developed a method of game co-design with 11 steps which involves four different actors. In the second step of the method, the game designer identifies the game design tool the more suitable according to the nine criteria identified in the previous section. The purpose of this literature review is to analyze the research works on game design tools from 2010 to 2020 (2020-12-18).

84

S. Gajewski et al.

3.3 Specifying the Research Question(s) The purpose of this study is to (1) identify the game design tools used in the research works, and (2) describe and compare these game design tools according to the nine criteria. 3.4 Identification of Research, and Selection of Studies This section describes the inclusion and exclusion criteria used to form the research corpus of our study. We explain and justify our choice of the search engines and the search terms used. First, our inclusion criteria are the research works on game design tools, published between 2010 and 2020. We consider the five following search engines: IEEE Xplore, ScienceDirect, Scopus, Springer, and Web of Science. We have selected these search engines to find research works that are reliable and up-to-date. On the other hand, the exclusion criteria are the research works which are not written in English, and the literature reviews of game design tools. Our search terms are “game design tool” and its synonyms. The Boolean search in the search engines was written as follows: “game design tool” OR “game design software” OR “game design engine” OR “game design program” OR “game design platform”. 3.5 Data Extraction The selection process of the research works is illustrated in Fig. 1. It has taken place as follows. First, we have exported the results (databases) from each of the five search engines in five different csv-files. Then, we have regrouped the results from the five search engines in a single csv-file. We have sorted the results alphabetically from A to Z, so we could cluster and, later, delete the duplicates. All in all, 302 research works have been identified, distributed as follows: IEEE Xplore (6), ScienceDirect (38), Scopus (112), Springer (120), and Web of Science (26). Forty-three research works were duplicates, and have been deleted. Then, we have looked for each research work to access its abstract, and its keywords. All the research works (titles, abstracts, and keywords) have been regrouped in a single docx-file. One result has been deleted because the paper could not be found. Nine results have been deleted because they did not include any abstract. Then, we have read all the 249 abstracts of the remaining research works. To be considered relevant, the title, the abstract and/or the keywords of a research work should include the name of a game design tool or the research work subject should be about game design. Sixty-two research works mention a game design tool or a study on game design. However, nineteen research works were not available for free. One research work was limited to the abstract. One hundred and eighty-seven research works were off topic. In total, two hundred and seven research works have been rejected.

Game Design Tools

85

Then, we have read the forty-two remaining research works. Eighteen research works were included for the data synthesis. Twenty-four research works were rejected for different reasons: (1) five research works were about tools which are not game design tool, as Microsoft PowerPoint, (2) sixteen research works were about game design tools which are not available anymore or not sharable by their designers, (3) two research works were about modding or about game level design from existing game, reducing creativity, and (4) one research work was rejected because it was a literature review about game design tools.

Fig. 1. PRISMA flow diagram for a systematic review about game design tools from 2010-01-01 to 2020-12-18 [22].

3.6 Data Synthesis, and Reporting the Review After identifying 21 game design tools, of which 18 of them came from studies via databases (five about Scratch, four about Microsoft Kodu, and by the same author, two GameMaker, two Agentsheets, and one for each other), and three are suggested by a digital learning manager, we describe alphabetically each tool according to the nine criteria identified. Agentsheets [15]. This game design tool is a visual block-based environment. It consists in dragging and dropping conditions blocks (i.e., “if…”) and actions blocks (“…then…”) into a script area. Agentsheets is available in different languages, as English, and French. On the website of Agentsheets, it is written: “Programming For Kids” [15]. Agentsheets provides “agents” that can be used as characters or building blocks of the background. Tutorials are available in different languages, as English. French is not available. Agentsheets costs $99.40 annually. However, a trial version of Agentsheets allow users to try the game design tool for free. Agentsheets allows users to design 3D games. An inflatable icon turns the 2D characters or building blocks available in the database or drawn by the users themselves into 3D. Agentsheets allows users to publish their games on the web.

86

S. Gajewski et al.

This tool was used by [16] where the students worked on their project of game design for two weeks. The aim of the research work led by [2016] was to present to teachers how to design games, so that teachers could teach students computer science and Computational Thinking. This tool was also used by [17] where the students worked on their project of game design for seven 2-h weekly sessions. The aim of the research work led by [17] was to compare meaning of game descriptions in Brazilian Portuguese and Visual AgenTalk code. Alice [18]. This game design tool is a “block-based programming environment that makes it easy to […] program simple games in 3D” [18]. Alice provides a list of action blocks that the game designer drags and drops into a script screen to trigger different behaviors. Alice is available in different languages as English. French is not available. Alice is used by teachers at all levels from middle schools (and sometimes even younger) to universities. So, Alice can be used by anyone. Alice provides different scenes and characters which lots of them are inspired from “Alice in Wonderland” because of its name Alice. Different resources, in English, are available on the website of Alice. Alice is a freeware. Alice allows users to develop 3D games. To run and play an Alice game locally in the Alice player, the Alice game needs to be exported from the Alice Integrated Development Environment (IDE). This tool was used by [19] where the students worked on their project of game design. Once Alice has been presented to the students, the students have browsed freely the game design tool for 30 min to discover the game design tool’s functionalities. Then, they have looked at tutorials available on the web site of Alice and on YouTube. Once their games have been designed, they have tested them. Unfortunately, the time spent by the students to design their games is not communicated. The aim of the research work led by [2019] was to study the mental activities of the students when designing a game. Celestory [20]. This game design tool allows users to develop games without coding using visual blocks that users have to drag and drop. The game design interface is in different languages, as English, and French. Celestory provides over 1,500 assets and designs. Tutorial are available on the Celestory website in different languages, as English, and French. A free version of Celestory allows users to develop only one project. Two different commercial versions of the game design tool, whose prizes are either $19 or $109 monthly, depending on options. Celestory allows users to design 2D games. Celestory allows users to develop different types of games (Playing cards, escape game, interactive movie, quiz, etc.). Celestory enables users to develop games to Windows, macOS, Linux, iOS, Android, HTML5. GameMaker [21]. This game design tool allows users to develop games either using its specific code programming language, GameMaker Language (GML), or using a friendly Drag and Drop (DnD™) interface without the need for writing any code. The interface of the game design tool is in different language, as English, and French. GameMaker proposes powerful tools for beginners and professionals alike. GameMaker do no provide neither sprites for the characters, nor tiles for the scenes. A tutorial, the GameMaker Studio 2 Manual, is available on its website in different

Game Design Tools

87

language, as English, and French. A freeware version (with an unlimited time duration) of GameMaker allows users to develop their games and to export them to the GXC platform to share and monetize them. Different commercial versions of the game design tool, whose prizes range from 42.50 e to 679.99 e yearly, depending on option. GameMaker is a “2D Game Development Environment” [21]. GameMaker allows users to develop a wide range of games: shoot them up games, platform games, tycoon games, First Person Shooter game (FPS), etc. GameMaker allows to export the games in Windows, macOS, HTML5, Android, iOS, Sony PlayStation 4 and 5, Microsoft Xbox One and Series S and X, and Nintendo Switch. This tool was used by Baytak and Land [22] where the students worked on their project of game design for 45-min sessions twice a week for eight weeks, i.e. 12 h. The aim of the research work led by [22] was to study how students designed games reflect their understanding of nutrition. This tool was also used by [23] where the students worked on their project of game design during one daily 45-min session, five times a week during three weeks, i.e. 11 h and 15 min. The aim of the research work led by [23] was to compare the potential for fostering creative problem-solving behaviors between two distinctly different problemsolving environments. Gamestar Mechanic [24]. This game design tool “enables students to create their own games without any programming knowledge” [24]. It uses a point and click interface. Gamestar Mechanic is in English. Gamestar Mechanic is “designed for 7- to 14-yearsolds but is open to everyone”. The workshop of Gamestar Mechanic provides different blocks that users drag and drop in a tileset which will be the world of the game. The workshop also provides avatars and enemies, items, like keys to open doors, and “systems”, like a timer or a health meter. Gamestar Mechanic “consists of three sections: The quest, workshop, and game alley” [24]. In the quest, students learn the principles of game (rules, games mechanics, etc.) by playing, and they earn sprites once they have successfully completed missions. “The workshop is where students make their own games” [24] but they can’t design any game as long as they have not earned sprites. “Game Alley is where students share their games in an online community” [24]. Gamestar Mechanic is a freeware. Gamestar mechanic allows users to design 2D games. Gamestar Mechanic allows users to create different types of games: adventure, platform, action, and experimental. Once the game is developed with Gamestar Mechanic, it is uploaded in the Game Alley and shared with the online community. This tool was used by [11] where the students worked on their project of game design for one 45-min session per a day during nine months. The aim of the research work led by [11] was to introduce students to the basics of game design. Microsoft Kodu [25]. This game design tool uses a visual programming language by tiles. Microsoft Kodu is available in different languages as English, and French. Microsoft Kodu is “intuitive, even to the degree to be used easily by children as young as nine or ten” [10]. Microsoft Kodu proposes three different worlds that users can modify, and different objects and sprites. Tips and Resources are available on the website of Microsoft Kodu. They are available only in English. Microsoft Kodu is a freeware.

88

S. Gajewski et al.

Microsoft Kodu allows users to design games in a 3D environment. Microsoft Kodu allows users to design “different types of games (e.g., adventure, arcade, racing)” [9]. The games developed with Microsoft Kodu can be hosted on the web, and shared with the Kodu community. This tool was used by [26] where the students worked on their project of game design for one 3-h session once a week during five week-ends, i.e. 15 h. The aim of the research work led by [26] was to allow students to design games to foster their problem-solving and critical reasoning skills. This tool was also used by [10] where the students worked on their project of game design either during an after-school program (seven 3-h sessions, i.e. 21 h), or during a summer camp (eight 5-h sessions, i.e. 40 h). The aim of the research work led by [10] was to teach students basics of computer programming, game design, and complex problem-solving skills. This tool was also used by [27] where the students worked on their project of game design for five hours a day during ten days, i.e. 50 h. The aim of the research work led by [27] was to teach students problem-solving skills, specifically in system analysis and design, decision-making, and troubleshooting. This tool was also used by [9] where the students worked on their project of game design for one 1-h session per week during ten weeks, i.e. 10 h. The aim of the research work led by [9] was to allow students to learn basics of digital game-design and to practice system design skills such as making flowcharts. RPG Maker VX Ace [28]. This game design tool is “incredibly simple to learn and use, being accessed through a simple point and click interface”, and that users can create a game “without the need for any coding knowledge” [28]. So, [8] have chosen RPG Maker VX Ace in their research work “because it requires little programming or game design experience to create games”. The game design tool interface is in different languages, as English, and French. As written on its website, RPG Maker VX Ace is “simple enough for a child. Powerful enough for a developer” [28]. So, it can be used by anybody. A character generator allows to customize the available characters in the database of the game design tool. The Map Editor allows to create the world of the game by selecting a tileset and by painting the map with the different tiles (scenes and characters). Tutorials, in English, are available on its website. RPG Maker VX Ace costs 64.99 e. However, a trial version of RPG Maker VX Ace allow users to try the game design tool, and all its functions, for free for 30 days. “RPG Maker VX Ace is a game engine designed to make 2D Roleplaying Games” [29]. When a game is developed with RPG Maker VX Ace, a.EXE file is created, and the game can be played on Windows. This tool was used by [8] where the students worked on their project of game design during two consecutive days. The aim of the research work led by [8] was to study how artifacts, as textbooks, world maps, Google, and timelines were used when students design games in a classroom. Scratch [30]. This game design tool is a visual programming environment. So, [7] have chosen Scratch in their research work because “it entails a user-friendly interface for children with visual code blocks, reducing syntax errors”. “Users drag and snap command blocks together to build scripts”. The game design interface is in different

Game Design Tools

89

languages, as English, and French. Scratch “is designed especially for ages 8 to 16” [31]. Scratch provides sprites for the characters, and backdrops for the scenes. Video tutorials, available on its website, are all in English. Some are dubbed, or subtitled in other languages, as French. Scratch is a freeware. Scratch allows to design 2D games. Scratch allows users to develop different types of games: clicker games, platform games, maze game, pong game, etc. The games developed with Scratch can be played locally, or online once the games have been uploaded on the Scratch website. This tool was used by [7] where the students worked on their project of game design during twenty-one 45-min sessions, i.e. 15 h, and 45 min. The aim of the research work led by [7] was to allow the fifth graders to design games about environmental science to learn this topic and to teach it to second graders. This tool was also used by [32] where the students worked on their project of game design for 6-h sessions once a day during four days, i.e. 24 h. The aim of the research work led by [32] was to allow students to design games to teach others about climate change. This tool was also used by [33] where the students worked on their project of game design for nine 2-h sessions, i.e. 18 h. The aim of the research work led by [33] was to allow students to design carnival games to examine the relevance of game design to perspective-taking. This tool was also used by [34] where the students worked on their project of game design for two 1-h sessions a week during six weeks, i.e. 12 h. The aim of the research work led by [34] was to allow students to design games to learn and to teach others about mathematics. This tool was also used by [35] where the students worked on their project of game design for three days. The aim of the research work led by [35] was to allow students to design games to learn and to teach their classmates about geography, chemistry, biology, and social studies. Stagecast Creator [36]. This game design tool is a visual programming environment using a point and click interface. Stagecast Creator is in different language, as English, and French. Stagecast Creator is a program that allows children as young as eight to design their own games. It is “accessible to novice programmers” [37]. Stagecast Creator provides some backgrounds and some sprites. Tutorials, in English, are available in a Stagecast Creator’s directory of the computer. A tutorial, in English, available on YouTube, shows how to design a game. A demo version of Stagecast Creator allows users to design games during 120 days after the tool installation. But the game design tool “hasn’t been available for purchase since 2014”. Stagecast Creator allows users to design 2D games. Stagecast Creator allows user to design different types of games, as action game or adventure game [37]. Stagecast Creator allows users to save the games developed on a local disk or on the internet. This tool was used by [37] where the students worked on their project of game design for 14 months: twice a week during the school year, and every day for three weeks during the summer. Unity [38]. The language that’s used in Unity is called C#. The game design interface is in English. Unity is intended for professionals.

90

S. Gajewski et al.

Unity provides an asset store where users can buy characters, music and sound FX, and many more. So, scenes and characters are not for free. A manual is available in different language, as English. French is not available. For personal use, Unity is free, provided users do not earn more than $100 thousands in the last 12 months. For professional use, Unity isn’t a freeware but a commercial tool, whose prizes range from $399 to $2,400 yearly, depending on option. Unity allows users to design both 2D and 3D games. Unity allows user to design a wide variety of games. Unity allows to export the games in Windows, macOS, Android, iOS, Playstation 3, Playstation Vita, Playstation 4, Xbox 360, Xbox One, Xbox One X, Wii U, Nintendo 3DS, Nintendo Switch, etc. This tool was used by [39] where the girls worked on their project of game design for two half days. The aim of the research work led by [39] was to evaluate the effects of the game design workshop on young girls’ attitudes towards Computer Science. Unreal Engine [40]. This game design tool allows users to design their games either using the C++ code, or a blueprint visual scripting. The game design interface is in English. Unreal engine is intended for professionals. Unreal engine provides templates including the main character with some elements of the game background. A tutorial is available in English in the help tab of the game design interface. Unreal engine is free. If the game succeeds (i.e., the game generates over $1 million), Unreal engine get royalties (5%). Unreal engine is a 3D creation tool. Unreal engine provides users with different game templates (First person, Flying, Puzzle, Rolling, Third person, etc.). Unreal engine enables users to develop games to Windows, Playstation 5, Playstation 4, Xbox Series X, Xbox One, Nintendo Switch, macOS, iOS, Android, Linux, HTML5, etc. VTS Editor [41]. The programming language used in VTS Editor is based on visual command blocks. Users drag and drop command blocks which trigger different actions. The interface of the game design tool is in different languages, as English, and French. In VTS Editor, different characters for the avatars, and different backgrounds for the scenes are available for free. Others are available for purchase. Tutorials are available in different languages, as English, and French. VTS Editor is not free. Different commercial versions of the game design tool exist, whose prizes range from 499 e (only available for education) to 6,228 e (all-inclusive with coaching) yearly. VTS Editor allows users to animate 3D characters. VTS Editor allows users to develop simulation games. VTS Editor allows to export the games in Windows, macOS, Android, iOS, directly from a web page, or from the VTS Perform platform, or from the VTS Player application, or on a Learning Management System (LMS).

Game Design Tools

91

4 Discussion Table 1 compares the different game design tools identified from the systematic literature review conducted with the PRISMA method. If the user doesn’t have any knowledge about programming, he will not use Unity which uses C#, a programming language using lines of code, and if he would like to use GameMaker or Unreal Engine, he will choose their visual programming language. Regarding the language of the game design tool interface, all are in English. Some propose other languages. But if the user speaks only French, he will not choose Alice. Indeed, language barrier could make the use of the game design tool difficult. Regarding the tutorials criteria, all game design tool provides tutorials, explaining and facilitating the use of the game design tools. All are in English. Some propose other languages. But if the user speaks only French, he will not choose Agentsheets, Alice, Gamestar Mechanic, Microsoft Kodu, RPG Maker VX Ace, Stagecast, Unity, or Unreal Engine. Here again, language barrier could make the use of these tutorials difficult. Most of game design tools provides scenes and characters for free. If the cost aspect is not a problem, the user could use Unity which proposes an asset store. Indeed, in Unity, assets are not free. Regarding GameMaker, this game design tool does not provide any asset neither for free, not with charge. With GameMaker, the user will have to ask for a game designer character’s help or draw his characters by himself, probably increasing design time, and decreasing the quality of the graphics. Some game design tools provide the possibility to develop different game types. However, RPG Maker VX Ace is restricted to Roleplaying games; VTS Editor is restricted to simulation games. These two game design tools could decrease creativity by imposing a single game type. The user will choose the game design tool depending on the game type he would like to develop. Some game design tools are restricted to professionals. This is the case of Unity and Unreal Engine which use programming languages using lines of code. Some game design tools are available for the general public. Others are friendly enough allowing young people and children to use them. The choice of the game design tool will depend on the target audience-designer. Some game design tools allow users to develop 3D games (e.g. Unreal Engine). Some are restricted to 2D games (e.g. Scratch). The choice of the game design tool will depend on the look the user would like that the game has. Some game design tools are freeware (e.g. Scratch). Others are commercial tools and users have to pay fees (e.g. RPG Maker VX Ace). According to the budgetary constraints of the user, the user could use some game design tools, but not others. Regarding the export of the game developed, for some game design tools, the user has to upload his game to the community website of the game design tool (e.g. Scratch). For other game design tools, the export possibilities are various: Microsoft, MacOS, iOS, Android, HTML5, console (e.g. Unity).

Drag and drop

Drag and drop

Code (GML) OR Drag and drop

Drag and drop

Visual by tiles

Point and click English, and French

Alice

Celestory

GameMaker

Gamestar Mechanic

Microsoft Kodu

RPG Maker VX Ace

English, and French

English

English, and French

English, and French

English

English, and French

Drag and drop

Agentsheets

Language

Programming language

Criteria Game Design Tool

Tutorials in English

Tips and resources in English

Learning by playing (in English)

English, and French

English, and French

Resources in English

English

Tutorials

Game Type

Different types of games Different types of games Different types of games Different types of games Roleplaying games only

Scenes and characters ✓ ✓ ✓

✗

✓

✓

✓

2D

2D

2D

3D

3D

2D or 3D modelling

For anyone

2D

“Children as young 3D as nine or ten”

“for 7- to 14-years-olds”

For beginners and professionals

-

For anyone

Kids

target audience-designer

Table 1. Comparison of game design tools according to nine criteria [14].

Free for 30 days OR 64,99 e

Free

Free

Free OR Commercial versions

Free OR Commercial versions

Free

Free

Prize

(continued)

Windows

Online

Online

Various

Various

Locally

Web

Export

92 S. Gajewski et al.

Point and click English, and French

Code (C#)

Code (C++) OR visual scripting

Drag and drop

Stagecast Creator

Unity

Unreal Engine

VTS Editor

English, and French

English

English

English, and French

Drag and snap

Scratch

Language

Programming language

Criteria Game Design Tool

English, and French

Tutorial in English

English

Tutorials in English

Videos in English, and in French

Tutorials

Different types of games

✓

Different game templates Simulation games

Templates

✓

Different types of games

Different types of games

✓

asset store Not for free

Game Type

Scenes and characters

Table 1. (continued)

2D

2D or 3D modelling

-

For professionals

For professionals

3D characters in a 2D environment

3D

2D OR 3D

“Children as young 2D as 8”

Eight to 16 years old

target audience-designer

Trial version OR commercial version

Free under conditions

Free OR commercial versions

Demo version available during 120 days

Free

Prize

Various

Various

Various

Locally OR Online

Locally OR Online

Export

Game Design Tools 93

94

S. Gajewski et al.

5 Experimentation In this section, we explain how we have used the results of the systematic literature review to choose a game design tool for an experimentation of game co-design. Indeed, we have experimented a co-design activity at IFsanté, a nursing school in France from April to June 2022 with 110 s-year nursing students. Divided into 21 groups from four up to five students, they have co-designed games to learn about liver cirrhosis. To do this, we have used the game co-design method we have developed [5]. This method involved four different actors (teacher, game designer, students, and researcher), and is composed of 11 steps: (1) Specify the pedagogical objectives, (2) Identify the game design tool, (3) Identify games with similar field, (4) Play games with similar field for inspiration, (5) Deliver learning content to students, (6) Read, watch, listen, understand the learning content, (7) Teach students about how to design a game, (8) Teach students about how to use the game design tool, (9) Co-design the game, (10) Co-implement and co-develop the game, (11) Evaluate the game. The Table 2 illustrates the 11 steps of the method of game co-design and the four actors involved. In this paper, we only focus on the steps two and eight because the paper is about game design tools. Another paper in progress will take the other steps into consideration. Table 2. Method of game co-design [5] Steps

Actions

Actors

1

Specify the pedagogical objectives

T

2

Identify the game design tool

GD

3

Identify games with similar field

GD

4

Play games with similar field for inspiration

S

5

Deliver learning content to students

T

6

Read, watch, listen, understand the learning content

S

7

Teach students about how to design a game

GD

8

Teach students about how to use the game design tool

GD

9

Co-design the game

S

10

Co-implement and co-develop the game

S

11

Evaluate the game

GD-R-S-T

GD: Game Designer/R: Researcher/S: Students/T: Teachers

In the second step of our method of game co-design, the game designer identifies the most suitable game design tool for his needs. For our experimentation, with the help of the teacher who is close to students, the game designer has drawn on the results of the systematic literature review. Thereby, the second-year nursing students have used VTS Editor.

Game Design Tools

95

Why has the game designer chosen VTS Editor? For his needs, that means to allow the second-year nursing students to co-design their own games, he needed a game design tool that could be used by anyone, even by those who have no game design skills. Indeed, it should be noted that no game design course is intended in the French nursing training program. Therefore, the most French nursing students have no game design experience, and thus no game design skills (C1: programming language). Second-year nursing students needed a game design tool with an interface in French language. Indeed, even if nursing students learn English during their studies, the nursing students of the nursing school had a low level in English (C2: language). Linking to the two previous criteria, that means programming language, and language, it was important for the game designer to choose a game design tool that provided tutorials to help the second-year nursing students to use the game design tool (because they don’t have any game design skills), but tutorials in French (since the nursing students of the nursing school had a low level in English (C3: tutorials). Linking to the first criteria, that means programming language, the game designer needed a game design tool that provided scenes and characters. Indeed, mostly French nursing have no game design skills. Therefore, they do not have any game design character skills. Otherwise, drawing characters by themselves would take time; a precious time they could use to learn more about how to use VTS Editor (C4: scenes and characters). The game designer needed a game design tool allowing second-year nursing students to develop simulation games. Indeed, “As nursing students are generally well acquainted with visually realistic game environments, the required standard for such games is high” [42] (C5: game type). The game design tool that second-year nursing students would use should be a game design tool suitable for anyone. Some game design tools, as Unity or Unreal Engine, have been rejected because they are intended to professionals with game design experience and skills (C6: target audience-designer). As noted above, French nursing students have no game design experience, and so no game design skills. For the experimentation of game co-design with second-year nursing students, the game designer needed a game design tool allowing students to create 3D games. Indeed, users “can reproduce real-life situations from the learners’ daily lives and simulate realistic virtual interactions using 3D characters”. Otherwise, as mentioned above, “compared to 2D environments, the ability to create 3D games […] makes it visually more appealing for young students” [10] (C7: 2D or 3D modelling). For economic reason, the game designer has decided to use a freeware game design tool (C8: prize). Finally, co-designing a game is rewarding, but co-designing a game that could be played by others is more rewarding. So, the game designer needed to generate the games co-designed by the second-year nursing students, and to host them on a server or on a Learning Management System (LMS) (C9: export). All these requirements according to the comparison of the 11 game design tools (Table 2) bring the game designer to choose VTS Editor. Once the game design tool has been identified by the game designer (step one), he has taught the students about how to use it (step eight). For our method of game

96

S. Gajewski et al.

co-design, the game designer has chosen VTS Editor, and has taught the second-year nursing students about how to use it. The lesson took place in the second of June and lasted four hours. The day before, at least one student from each group was asked to create a user account, to download, and to install the tool on his own laptop. During the first hour, the game designer has shown how to create, step by step, a short project on VTS Editor while the students, divided in groups, duplicate the same on their own laptops. During the second hour, the students were asked to watch the video tutorials uploaded on iCampus, the LMS of the nursing school. During the third hour, the game designer has created exercises that the second-year nursing students have realized for them to practice using VTS Editor. For example, the second-year nursing students were asked to create scenes, to add blocks (like for example a message, a quiz, etc.), to add characters, to import voices, etc. During the fourth and last hour, the students freely explored VTS Editor to discover all its features. During all the game co-design activity, the second-year nursing students were asked to fill different surveys about motivation [43], knowledge and game design tool usability. Since the focus of this paper is about game design tools, we will present the survey about the usability of the game design tool that the second-year students have filled after the game co-design activity. To assess the VTS Editor usability, we have used the System Usability Scale (SUS) [44, 45]. We have used this scale because (1) it is the most used scale allowing to assess the usability of a game design tool, (2) it is free to use, (3) it is quick to fill (only three to four minutes), and (4) a French version of this scale exists [46]. This scale is a 5-points Likert scale ranging from zero (strongly disagree) to four (strongly agree). Respondents were asked to indicate how much they agree with 10 sentences, as, for example, “I thought the system was easy to use” (item 3). However, in each sentence, the words “the system” had been replaced by “VTS Editor” to make the questionnaire more contextualized. The second-year nursing students have answered the survey at the end of the game co-design activity. To calculate the SUS score, we have to (1) subtract 1 point from each score assigned by the respondent for the items 1, 3, 5, 7, and 9, (2) subtract the score assigned by the respondent for the items 2, 4, 6, 8, and 10 from 5, (3) multiply the sum of the recalculated scores by 2.5 to obtain the overall score of the system usability. System usability scores have a range of 0 to 100 [44]. The more the score is higher, the more the system is user-friendly. The results show that the mean score assigned by the second-year nursing students to the VTS Editor usability was 54.2 out of 100 (sd = 23.0) (See Fig. 2). These results show that learning about how to use VTS Editor should be strengthened. Indeed, the second-year nursing students had only one hour to watch all video tutorials. They had only one hour to realize all the VTS Editor exercises the game designer gave them to do. They had only one hour to freely explore VTS Editor to discover all its features. Only one game designer was available to help the second-year nursing students to use

Game Design Tools

97

Fig. 2. System Usability Scale (SUS) score (Bangor, Kortum, and Miller, 2009), and the mean score assigned by the second-year nursing students to the VTS Editor usability (54.2 /100).

the game design tool. Much more time, many more exercises and many more game designers could help the second-year students using the game design tool.

6 Conclusion and Perspectives The aim of this paper was to identify/describe game design tools, and to compare them in order to help the educational actors to choose the game design tool the more suitable for their needs in a specific context. First, we have identified nine criteria to characterize a game design tool: programming language (C1), Tool language (C2), tutorials (C3), scenes and characters (C4), game type (C5), target audience-designer (C6), 2D or 3D modelling (C7), prize (C8), and Export (C9). Then, we have conducted a systematic literature review following the PRISMA method. From 302 studies identified via five databases, and eight game design tools advised by a digital learning manager, 11 game design tools have been included for the discussion. The discussion concludes that the choice of the game design tool depends on the actors’ needs in educational contexts. Finally, we have presented an experimentation where 110 s-year nursing students have used VTS Editor to co-design games for them to learn about liver cirrhosis. To do this, the game designer has taught the students about how to use VTS Editor. The results suggest that learning about how to use VTS Editor should be strengthened. Further experimentations are needed. The next one is planned between April and June 2023. For this next experimentation, much more time will be devoted to the game design tool’s learning, and many more game designers will be involved. In addition to the surveys the nursing students will be asked to fill, focus groups will be led. The game co-designed by the nursing students will be assessed too.

References 1. El Mawas, N., et al.: Towards a self-regulated learning in a lifelong learning perspective. In: CSEDU 2017: 9th International Conference on Computer Supported Education-Special Session Lifelong Learning, vol. 1, pp. 661–670 (2017) 2. El Mawas, N., Cahier, J.P.: Towards a knowledge-intensive serious game for training emergency medical services. In: ISCRAM (2013)

98

S. Gajewski et al.

3. El Mawas, N., Truchly, P., Podhradský, P., Muntean, C.H.: The effect of educational game on children learning experience in a Slovakian school. In: CSEDU-7th International Conference on Computer Supported Education (2019) 4. Kafai, Y.B.: Playing and making games for learning: Instructionist and constructionist: perspectives for games studies. Games Cult. 1(1), 36–40 (2006) 5. Gajewski, S., El Mawas, N., Heutte, J.: Towards a methodology to co-design a learning game by nursing students. In: Marfisi-Schottman, I., Bellotti, F., Hamon, L., Klemke, R. (eds.) GALA 2020. LNCS, vol. 12517, pp. 273–282. Springer, Cham (2020). https://doi.org/10. 1007/978-3-030-63464-3_26 6. Burke, Q., Kafai, Y.: Decade of game making for learning: from tools to communities. In: Handbook of Digital Games, pp. 689–709 (2014). https://doi.org/10.1002/9781118796443. ch26 7. Baytak, A., Land, S.M.: An investigation of the artifacts and process of constructing computers games about environmental science in a fifth grade classroom. Education Tech. Research Dev. 59, 765–782 (2011) 8. Øygardslia, K., Aarsand, P.: “Move over, I will find Jerusalem”: artifacts in game design in classrooms. Learn. Cult. Soc. Interact. 19, 61–73 (2018) 9. Akcaoglu, M., Green, L.S.: Teaching systems thinking through game design. Education Tech. Research Dev. 67(1), 1–19 (2018). https://doi.org/10.1007/s11423-018-9596-8 10. Akcaoglu, M.: Design and implementation of the game-design and learning program. TechTrends 60(2), 114–123 (2016). https://doi.org/10.1007/s11528-016-0022-y 11. An, Y.-J.: A case study of educational computer game design by middle school students. Education Tech. Research Dev. 64(4), 555–571 (2016). https://doi.org/10.1007/s11423-0169428-7 12. Kitchenham, B., Charters, S.: Guidelines for performing systematic literature reviews in software engineering. Technical report EBSE 2007-001, Keele University and Durham University Joint Report (2007) 13. Gajewski, S., El Mawas, N., Heutte, J.: Vers une méthodologie de co-conception de learning games par les étudiants en soins infirmiers. In: 10e Conférence sur les Environnements Informatiques pour l’Apprentissage Humain, pp. 306–311 (2021) 14. Gajewski, S., El Mawas, N., Heutte, J.: A systematic literature review of game design tools. In: Proceedings of the 14th International Conference on Computer Supported Education Volume 2, pp. 404–414 (2022). https://doi.org/10.5220/0011137800003182. ISBN 978-989758-562-3, ISSN 2184-5026 15. https://agentsheets.com/ 16. Basawapatna, A., Koh, K.H., Repenning, A., Webb, D.C., Marshall, K.S.: Recognizing computational thinking patterns. In: SIGCSE 2011 (2011) 17. de Souza, C.S., Garcia, A.C.B., Slaviero, C., Pinto, H., Repenning, A.: Semiotic traces of computational thinking acquisition. In: Costabile, M.F., Dittrich, Y., Fischer, G., Piccinno, A. (eds.) IS-EUD 2011. LNCS, vol. 6654, pp. 155–170. Springer, Heidelberg (2011). https:// doi.org/10.1007/978-3-642-21530-8_13 18. https://www.alice.org/ 19. Allsop, Y.: A reflective study into children’s cognition when making computer games. Br. J. Edu. Technol. 47, 665–679 (2016) 20. https://www.celestory.io/ 21. https://www.yoyogames.com/en/gamemaker 22. Baytak, A., Land, S.M.: A case study of educational game design by kids and for kids. Procedia Soc. Behav. Sci. 2, 5242–5246 (2010) 23. Buelin-Biesecker, J., Wiebe, E.N.: Can pedagogical strategies affect students’ creativity? Testing a choice-based approach to design and problem-solving in technology, design,

Game Design Tools

24. 25. 26. 27.

28. 29. 30. 31. 32.

33. 34.

35.

36. 37.

38. 39.

40. 41. 42.

43.

44.

45.

46.

99

and engineering education. In: 120th American Society for Engineering Education Annual Conference & Exposition (2013) https://gamestarmechanic.com/ https://www.kodugamelab.com/ Akcaoglu, M., Koehler, M.J.: Cognitive outcomes from the Game-Design and Learning (GDL) after-school program. Comput. Educ. 75, 72–81 (2014) Akcaoglu, M.: Learning problem-solving through making games at the game design and learning summer program. Education Tech. Research Dev. 62(5), 583–600 (2014). https:// doi.org/10.1007/s11423-014-9347-4 https://www.rpgmakerweb.com/products/rpg-maker-vx-ace https://dl.degica.com/rpgmakerweb/tutorials/RPGMakerVXAceTutorial1.pdf https://scratch.mit.edu/ https://scratch.mit.edu/about Puttick, G., Tucker-Raymond, E.: Building systems from scratch: an exploratory study of students learning about climate change. J. Sci. Educ. Technol. 27(4), 306–321 (2018). https:// doi.org/10.1007/s10956-017-9725-x Dishon, G., Kafai, Y.B.: Making more of games: cultivating perspective-taking through game design. Comput. Educ. 148, 103810 (2020) Ke, F., Im, T.: A case study on collective cognition and operation in team-based computer game design by middle-school children. Int. J. Technol. Des. Educ. 24(2), 187–201 (2013). https://doi.org/10.1007/s10798-013-9248-6 Weitze, C.L.: Reflective, creative and computational thinking strategies used when students learn through making games. In: Proceedings of the 11th European Conference on GameBased Learning, pp. 744–753. Academic Conferences and Publishing International (2017) http://acypher.com/creator/ Denner, J., Werner, L., Ortiz, E.: Computer games created by middle school girls: can they be used to measure understanding of computer science concepts? Comput. Educ. 58, 240–249 (2012) https://unity3d.com/get-unity/download Akku¸s Çakır, N., Gass, A., Foster, A., Lee, F.J.: Development of a game-design workshop to promote young girls’ interest towards computing through identity exploration. Comput. Educ. 108, 115–130 (2017). https://doi.org/10.1016/j.compedu.2017.02.002 https://www.unrealengine.com/en-US/ https://seriousfactory.com/en/authoring-software-vts-editor/ Koivisto, J.-M., Multisilta, J., Niemi, H., Katajisto, J., Eriksson, E.: Learning by playing: a cross-sectional descriptive study of nursing students’ experiences of learning clinical reasoning. Nurse Educ. Today 45, 22–28 (2016). https://doi.org/10.1016/j.nedt.2016.06.009 El-Mawas, N., Heutte, J.: A flow measurement instrument to test the students’ motivation in a computer science course. In: 11th International Conference on Computer Supported Education (CSEDU 2019), vol. 1, pp. 495–505. SciTePress Digital Library, Portugal: Setúbal (2019) Brooke: SUS: a ‘quick and dirty’ usability scale. In: Jordan, P.W., Thomas, B., Weerdmeester, B.A., McClelland, I. (eds.) Usability Evaluation in Industry, pp. 189–194. Taylor & Francis, London (1996). http://hell.meiert.org/core/pdf/sus.pdf Bangor, A., Kortum, P., Miller, J.: Determining what individual SUS scores mean: adding an adjective rating scale. J. Usability Stud. 4(3), 114–123. https://uxpajournal.org/determiningwhat-individual-sus-scores-mean-adding-an-adjective-rating-scale/ Gronier, G., Baudet, A.: Psychometric evaluation of the F-SUS: creation and validation of the French version of the System Usability Scale. Int. J. Hum.-Comput. Interact. (2021). https:// doi.org/10.1080/10447318.2021.1898828

Development and Evaluation of a Trusted Achievement Record of Accomplishments for Students in Higher Education Using Blockchain Bakri Awaji1,2(B) , Ellis Solaiman1 , and Adel Albshri1,3 1 Newcastle University, Newcastle upon Tyne, UK {b.h.m.awaji2,ellis.solaiman,a.albshri2}@newcastle.ac.uk 2 Najran University, Najran, Saudi Arabia 3 Jeddah University, Jeddah, Saudi Arabia

Abstract. Verifying the legitimacy of academic accomplishments like CVs and diplomas has become increasingly difficult as the number of institutions participating in the global education market continues to expand. This has resulted in a heightened risk of academic fraud. The distributed ledger technology known as blockchain has the potential to be an instrumental factor in the resolution of this issue. The purpose of this study is to present a blockchain-based achievement record system that generates a record of accomplishments that can be verified. The purpose of the proposed system is to make the process of authenticating and validating certificates more trustworthy, straightforward, and expedient by capitalising on the one-of-a-kind capabilities provided by Blockchain technology (the public Ethereum Blockchain), as well as smart contracts. Here, we will discuss the planning and execution of the system, as well as its components and tools. After that, we conduct a number of studies to evaluate the system in order to measure its usability, effectiveness, performance, and cost. An evaluation using the System Usability Scale (SUS) produced a score of 77.1. We demonstrate through a review of the relevant literature that this system is a significant improvement over legacy systems, both in terms of its friendliness to users and its level of efficiency. In addition to this, we carry out an in-depth cost analysis and have a conversation about the benefits and drawbacks of various alternative blockchain solutions. Keywords: Blockchain · Smart contracts · Ethereum Achievement records · Students · Education

1

· Trust ·

Introduction

University learning records are required for all students in higher education, including those in post secondary institutions working toward a degree [2]. The process of creating these documents and distributing official transcripts to verify the student’s academic achievements is standardised across institutions of higher c The Author(s), under exclusive license to Springer Nature Switzerland AG 2023  J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 100–124, 2023. https://doi.org/10.1007/978-3-031-40501-3_5

Development and Evaluation of a Trusted Achievement Record

101

learning. An official university transcript is used as evidence of a student’s academic performance [2]. Since they provide a way for prospective employers to verify an applicant’s educational background, official transcripts play a crucial role in the employment decision-making process. An applicant’s work portfolio can be used as a tool for employers to assess their abilities and determine whether they are a good fit for the position. Research indicates that work opportunities are significantly enhanced through the provision of adequate achievement record (service-based or project-based) [10,22]. Nonetheless, without a reliable achievement-recording framework, it is difficult to guarantee that transcripts, or work provided in a portfolio, are genuine works by the candidate. For this reason, maintaining accurate learning records can be of tremendous benefit. When applying for jobs, the vast majority of people still rely on time-honored practises such as submitting a resume or curriculum vitae (CV). It is possible for individuals to create their own CVs with the assistance of a number of online resources, and these CVs can be organised and formatted in a variety of ways. When it comes to the process of creating a CV, social networking sites like Facebook and LinkedIn can also be useful platforms. To date, however, no reliable methods have been developed that would allow prospective employers to verify the accomplishments listed on a candidate’s curriculum vitae [8]. According to the findings of the Higher Education Degree Datacheck (2021), about 30 percent of students and graduates fabricated or exaggerated their academic accomplishments. NGA HR services is a well known name in the UK. Their reports showed that 90% of HR managers had seen applicants exaggerated on their job applications [16]. Moreover, there are a number of issues connected to academic credentials and resumes that can lead to mistrust [13]. Weak data continuity is one such issue. Even if a student moves on to a different school, their academic history will typically not change. Due to the fact that each school maintains its own Learning Record Store (LRS), historical data from different campuses cannot be combined. Insufficient information is stored at the current institution to efficiently personalise and monitor the progress of their students, creating a “cold-start” problem [13]. According to studies conducted by [16], there are a few key areas where applicants frequently fabricated or lied on their achievement records. It was discovered that 44% of applicants lied about their accomplishments, and 43% lied about their employment history. 39% also inflated their professional credentials, while 32% inflated their academic credentials. Twenty-seven percent also fabricated membership in a professional organisation, and twenty-four percent gave inaccurate information for references [18]. Since extracurricular accomplishments are not verified on official transcripts, the low standards for recording and validating them are a further cause for concern. Therefore, it is not possible to confirm participation in such activities, such as extracurricular ones, prizes and employability awards, volunteer work, and positions in student union clubs and societies. Most studies that have investigated CV fraud, have found the topic to have a significant negative effect [11]. Blockchain technology has the potential to play a significant role in addressing the issues raised above. Blockchain technology has a number of immutability and security features that have prompted researchers to investigate its potential

102

B. Awaji et al.

applications in fields such as cloud computing, banking, IoT, and education. One significant advantage of blockchain technology is the ability of smart contracts to be programmed to automate data storage and validation processes [10]. A smart contract is an event-condition-action stateful computer programme that can be used on top of blockchain to create a distributed application that can be used by multiple parties who cannot trust each other [2]. The key concepts of smart contracts and their application are discussed in greater detail by [1,14]. This paper aims to introduce a blockchain-based achievement record system that generates verifiable achievement records for higher education students. Using Blockchain technology (the public Ethereum Blockchain) and smart contracts, the proposed system aims to streamline and accelerate the certificate authentication and validation procedure. This paper describes the design and implementation of the system as well as its components and tools. Several studies are then conducted to evaluate the system’s usability, effectiveness, performance, and cost. In this paper, a The following is a summary of contributions: – A conceptual model design of a Blockchain-Based Trusted Achievement Record System. The tools, components, mechanisms, scenarios, use cases and data flow are presented, as well as the design validation process and implementation of a Blockchain-Based Trusted Achievement Record System. – The description of an extensive study to evaluate the system usability using the System Usability Scale (SUS) test. – A detailed study and evaluation to assess the system’s capability to positively impact system users in terms of their motivation to learn, planning for future learning, employment, and providing proof of skills. – An extensive evaluation through analysis of the system performance through two variables, the delay time and the cost of transactions. Also we present a comparison with other existing solutions similar to our own work through a comparison of system capabilities and a comparison between various system evaluation methods. This paper is organised as follow: Sect. 2 summarizes a comprehensive survey of current work related to adopting blockchain for validating academic certificates. Section 3 provides a conceptualized model design of the blockchain-based trusted achievement record system with an explanation of the system development. Section 4 explores the different use-cases of the system. Section 5 presents the evaluation methods used in this project with a presentation of the results and a deep discussion of the results. Finally, the conclusion in Sect. 6.

2

Related Work

Several solutions have been suggested and implemented in response to the highlighted issues; as Table 1 shows. OpenBadge is a standard for digital credentials established by Mozilla and now controlled by the IMS Global Learning Consortium. In OpenBadge, a wallet is created for participants to add certificates as badges. The entity issuing a certificate will have provided authentication of it

Development and Evaluation of a Trusted Achievement Record

103

automatically via the wallet. The certificate’s inclusion in the retained record of the wallet only occurs once the legal authentication procedure has been successfully completed. Therefore, the wallet’s certificate list offers veracity for all entities and individuals. Fundamentally, issues such as damaged or misplaced physical certificates and their associated management and printing expenditure both financially and time-wise have been tackled through OpenBadges. Nevertheless, prospective single-point failures in the service or database of OpenBadges pose safety, security, dependability and transparency issues relating to the platform’s management of the issued certificate database. As a result, entities, particularly state institutions feel that they lack control over the database provided by OpenBadge, and have shown reluctance and scepticism over the platform’s adoption [21]. An alternative program for the issuing and validation of certificates that relies on Blockchain is the cutting-edge, Massachusetts Institute of Technology (MIT) project called Blockcerts [19]. Through Blockcerts, which is made up of decentralised and open mobile apps, tools, and databases, blockchain can be used to make programmes that issue and verify certificates. Documents like practise permits, certificates of education, and criminal records can all be added. Even though the certificate’s issuer and any other entity are not part of the validation process, Blockcerts still makes sure that the certificates are reliable. Blockcerts says that single-point failures will not be a problem as long as Bitcoin is still around. So, Blockerts’ services are always available and can be used. Blockcerts encourages users to have full control over their own privacy, so they are responsible for everything they do. For a certificate to be cancelled, both the entity that issued it and the person who owns it have to agree. This makes it impossible to dispute what happened. Blockcerts makes it easy to get a certificate because each Bitcoin address manages the transactions that make up the certificate. Blockcerts, on the other hand, may be difficult to implement and use because of a number of challenges as described in [17]. CVTrust, Smart Diploma, as well as Block. co are further applications that engage in certificate issuance and validation on the basis of Blockchain. Nevertheless, their adoption approach and technical resolutions are not thoroughly clarified. The characteristics of the Blockchain-based systems that are currently in use for certificate validation are outlined in Table 1 [3]. The comparison between these systems centres on the most important aspects, which include user experience and record sharing as well as accreditation, verification, privacy, and openness. It is generally agreed that the aspects pertaining to user privacy, user accreditation, and user experience are among the most important components of all existing systems. The characteristics pertaining to user experience, privacy, and accreditation are regarded to be essential components of all existing systems, while the remaining features are tacked on in accordance with the goals of the system to be developed. However, the solution that has been provided as a result of this research primarily includes all of the essential aspects, as can be shown in Table 1 [3].

104

3

B. Awaji et al.

Conceptual Model Design and Implementation

A trusted achievement record is a secure system that aims to record and authenticate certificates, key learning activities, and achievements. The system’s conceptual model is designed by gathering important information on stakeholders’ thoughts and outlooks on an achievement record system that uses blockchain and smart contract as described in Awaji et al. [3,6] and [4]. Figure 1 [3] illustrates the overall system design and demonstrates the requirements and components, including a frontend; DApp Layer, and backend; Blockchain Layer, as described in [3,5]. This architectural design contains two ends (frontend and backend) with distinct set dependencies, known as libraries and frameworks. While the frontend acts as a presentation layer that the enduser is introduced to upon entering the site, the backend provides the data and logic which enables the frontend to function. Table 1. Summary Comparison of Various Solutions.

rd gR eco rin Sha

Ver ifyi ng Ce rtifi cat e

Ac ces sib ilit y

Use rE xpe rien ce

cy ren spa Tra n

vac y Pri

Ver ific ati on

Features

Ac cre dit ati on

Systems

OpenBadge

•

⊗

•



•

•

⊗

⊗

Blockcerts

⊗

•

•

•

•

•

•

⊗

Block.Co

•

•

•

⊗

•

•

⊗

⊗

SmartDiploma ⊗

•

•

⊗

•

•

⊗

⊗

CVTRUST

⊗

⊗

•



•

•



⊗

CVSS

•

•

•

•

•

•

⊗

⊗

Our System • • • • •  partially provided, • provided, ⊗ unprovided

•

•

•

Fig. 1. The System Structure [3].

Development and Evaluation of a Trusted Achievement Record

3.1

105

DApp Layer

The frontend is designed to display web pages on PC, Tablets, or Smartphones and contains components that perform different functions in the system. For example, APPLICATION LOGIC component which controls the interfaces of the system and their contents based on the type of user, IDENTITY ACCESS MANAGEMENT component to create a unique ID for each user from the student type, DATA STORE OFF CHAIN component which is an off-chain database to store users data and certificates information in the frontend of system, HASING algorithm to create a unique hash for each uploaded certificate, TRANSACTION MANAGER to initiate and manage the transactions that send the certificate hash and the issuer meta data to the smart contract on the blockchain, WALLET to store and manage account keys, broadcast transactions, send and receive Ethereum tokens, and connect to decentralized applications, API to connect the frontend of the system to the smart contract on the blockchain to store the metadata of universities and certificates’ hashes. The system’s backend relies on the blockchain, a decentralized network of computer nodes that confirm and validate data added to the chain. This process results in digital data blocks being hashed and added to the chain via a cryptographic link. Blockchain-based records are reliably easy and quick to transfer, with students only needing to share a digital address to link future employers to their authenticated credentials. To integrate the blockchain with the frontend of the system, a smart contract has been written using Solidity and deployed on the Ethereum Virtual Machine (EVM) on the blockchain. The smart contract integrates on the front end through the Application Programming Interface (API). Four actors interact with the system, the first actor is the system administrator, responsible for executing the smart contract on the blockchain and the registration of universities on the system. The second actor is the university or learning institution, responsible for the authentication of student records. The third actor is the student, who utilizes the system to create a record of their achievements. The fourth actor is an employer, who utilizes the system to validate the candidate’s certification and assess candidates using their records of achievements. To illustrates the interaction with the system and defines the requirement to describe a particular use of the system, users will interact with the system in different manners. 3.2

Blockchain Layer

A blockchain-based application has been chosen for this system due to its performance and ability to verify education certifications proficiently. There are numerous reasons why blockchain is the most appropriate decision, including because it helps to remove the need for the manual verification of transactions since all necessary information is automatically verified by a decentralised network of computers. This information is also permanently stored in the blockchain, reducing, if not completely removing, the risk of deletion, meaning that additional security services are not needed. Importantly, the falsification or modification of transactions on the blockchain cannot take place. The specific hash system

106

B. Awaji et al.

is used to verify certificates, and no user is capable of modifying this information or uploading a false hash into the network. Users, also known as nodes, can create transactions and then propagate them over the blockchain network. Miners are in charge of maintaining the blockchain network’s ledger by regularly adding new blocks of transactions. Miners pick and execute a number of pending transactions from their pools to build and attach a new block to the ledger and then include them in the block by engaging in a consensus algorithm such as PoW. Subsequently, the created block will be sent to the rest of the network’s nodes. When a new block is created, each node must validate it before adding it to its local copy of the blockchain. The block will be verified and deemed part of the global blockchain ledger if the majority of nodes in the network accept it, append it to their local blockchain copies, and build upon it. The blockchain used for this specific system is Ethereum, an open source blockchain with smart contract capabilities. It also supplies a decentralised virtual machine that can complete the necessary scripts through the use of a system of public nodes. This system is considered to be Turning complete, meaning it can recognise other data sets and is also used as the internal transaction pricing mechanism. Decentralised applications are connected to the smart contract using web3.js, which is an assortment of archives that permit the system to interact with remote or local Ethereum nodes. The smart contract is of primary importance within the system, as it connects the blockchain with the frontend [14,15]. 3.3

Smart Contract

Regarding this specific platform, the use of smart contracts eliminates the need for human management as Application Programming Interface (API) connectivity instructs the smart contract; as in Fig. 2 [3]; to execute actions on the frontend automatically. This reduces the risk of documentation fraud for universities and employers. The primary programming language used when writing smart contracts that run on Ethereum Blockchains is Solidity, a contract-oriented language that is responsible for the secure storage of programming logic during a transaction. Solidity is also a high-level language and is used in the design, writing, an implementation of smart contracts to run on the Ethereum Virtual Machine (EVM), which is held on Ethereum Nodes that are directly linked to the blockchain, that in turn is connected to the frontend. A smart contract allows this platform to register universities and store the relevant document hash values on the blockchain, with Solidity and Truffle Framework working to publish it on the Ethereum Blockchain. It employs automatically when a command is given on the frontend via API connectivity. First, universities are registered on the blockchain using an API with the formulated function add uni. Another formulated function, store hash works to store the relevant document in the smart contract and was generated by the SHA-256 encryption algorithm that subsists in the system’s frontend. The function get hash is used to verify particular documents on the hash through an API.

Development and Evaluation of a Trusted Achievement Record

107

Fig. 2. Achievement Record Smart Contract [3].

3.4

Transaction and Gas

Register University, Store Hash, and Verify Hash are the relevant discussed transactions utilised in this particular system, and work to initiate transactions by applying the data in the transaction, the SHA-256 Hash value [9,24], and the Ethereum Address. Once a transaction has been logged in the blockchain, the details of the transaction, including the asset, price, and ownership, are immediately confirmed within a matter of seconds throughout all nodes, with a verified alteration on one ledger being instantaneously recorded on every other ledger. A specific node on the Ethereum blockchain is used to create a wallet address, which means that, with the aid of an API, it is easy to check the balance of a fee that an admin must pay, and the mining fee is deduced automatically by the Ethereum wallet address [23]. 3.5

Implementation

The frontend of the system is software implemented using HTML5, CSS3, AJAX, Javascript, and Web3js. We used HTML5 to design the frontend application, By using HTML5, we made usable forms. In addition, HTML5 supports crossplatform, is designed to display the application pages on PC, Tablet, and smartphones and keeps CSS better organized. Javascript is used to allow users to interact with the system frontend and to implement the frontend components. AJAX is used in this platform to allow a web page only to reload those portions which have changed, rather than reloading the whole page. This decentralize application is connected to a smart contract using web3js. Web3js is a collection of libraries that allows users to interact with the local or remote Ethereum node using an HTTP connection. The database has been designed to contain two categories of data: public authentication data and private certificate data. The public authentication data is available and released to the blockchain, the student data are stored in MySQL, securely protected and isolated in the intranet. The smart contract in this system is written by Solidity, a high-level and contract-oriented language used to write smart contracts. It is used for designing and implementing smart contracts. It’s designed to run on the Ethereum Virtual Machine (EVM), which is hosted on Ethereum Nodes connected to the blockchain.

108

4

B. Awaji et al.

System Use-Cases

Users will interact with the system in different manners. Users login with the frontend function and then access and distribute their documents if they so choose. Users’ documents are uploaded on the blockchain via the smart contract, with documents also being uploaded by the university for verifiable purposes. As students can access their uploaded documents, they can send them to potential employers, and employers can also verify the document using the document hash to search the system. 4.1

Use-Case: Admin

To begin, admins must log in to the system using the correct login credentials previously supplied to them before being forwarded to the admin dashboard, wherein the menus are laid out. Admins from their home page; as shown in Fig. 4 [3]; can choose from ‘Add University’, ‘University Manage’, or ‘Student Manage’. The ‘Add University’ tab allows admins to add a university into the university database by completing the form. Later, if necessary, the ‘University Manage’ tab allows admins to edit and delete universities. The admin operations presented in Fig. 3 [3]. 4.2

Use-Case: University

The university user must log into the system using correctly supplied credentials, and then will be taken to the university dashboard; as showed in Fig. 6 [3]. The user will be presented with specific menus, including ‘Document List’, ‘Upload Document’, ‘Add Students’, and ‘Manage Students’. By clicking on ‘Document List’, the university user will be able to view a list of the student documents and

Fig. 3. Admin Operations Sequence Diagram [3].

Development and Evaluation of a Trusted Achievement Record

109

Fig. 4. Admin Dashboard [3].

Fig. 5. University Operations Sequence Diagram [3].

clicking on any specific document will produce information about the relevant student. The ‘Upload Document’ tab allows the user to upload a student document by entering their details. If the document type is already available in the system, they can upload the document straight away. If not, they must first add the specific document type. The ‘Add Student’ tab enables the user to add a new student and all of their relevant details into the database, and the ‘Manage Students’ tab allows the university user to see a complete list of students, editing where necessary. University operations are presented in Fig. 5 [3].

110

B. Awaji et al.

Fig. 6. University Dashboard [3].

4.3

Use-Case: Student

The student homepage in Fig. 8 [3] allows university students to register themselves in the system. They are provided with a unique user ID and password, and, once entered correctly, will be redirected to the dashboard. If they enter their details incorrectly, an error message will be displayed, and they will stay on the login page until they enter the correct details. Once successfully logged in, students can see their information on the dashboard and are also presented with a list of certificates that have been uploaded by universities. Using the email system, students can share their documents with different employers, who will receive a link directing them to a document verification page. The student operations presented in Fig. 7 [3].

Fig. 7. Student Operations Sequence Diagram [3].

4.4

Use-Case: Employer

The employer will receive a link via their emails directing them to a document verification page in the system. Once they click on the certificate link, the system

Development and Evaluation of a Trusted Achievement Record

111

Fig. 8. Student Dashboard [3].

Fig. 9. The SUS Scale Obtained [3].

sends a transaction to the smart contract on the blockchain to check if the hash of the selected certificate is stored on the smart contract and returns a message “Verified” if the hash exists in the list in the smart contract storage. Moreover, students can share a token with any potential employer to allow them to access their achievement records on the system.

5

Evaluation

In order to evaluate the effectiveness of our solution, we conducted an experiment to gather feedback from the end-users of the proposed system. Our main objective in this research is to design a user-friendly and trustworthy system for stakeholders. To assess the usability of the system, we utilized the System Usability Scale (SUS) test [7]. In addition, we analyzed the feasibility of the system in terms of learning, employment, planning, and proof of skills. We received responses from 6 universities and had 30 students from those universities participate in the evaluation process. Finally, we evaluated the performance in terms of cost and transaction confirmation time.

112

5.1

B. Awaji et al.

System Usability Scale (SUS)

To assess the usability of our solution, we conducted a System Usability Scale (SUS) test [7]. Developed by John Brooke in 1986, the SUS is a reliable tool widely used for evaluating usability with various products and services, including, but not limited to, hardware, software, and applications. Consequently, it has been referenced in over 1300 articles and publications and is considered an industry standard. Formally, the SUS consists of a 10-item questionnaire that rates the users’s agreement about the usability of a system in terms of ease of use, efficiency, effectiveness, and overall satisfaction on a 5-point Likert scale, ranging from “strongly agree” to “strongly disagree”. Results. The participants evaluated the system’s usability using a five-point Likert scale, with the most common response options ranging from “strongly disagree” to “strongly agree”. 36.67% and 40% of participants indicated that they would use the system more frequently and expressed interest in doing so. In regards to the system’s complexity, 43.33% strongly disagreed and 30% disagreed that it was complex, suggesting that it was easy to use. To understand the participants’ insights about whether they would need technical support to use the system, 30% and 43.3% disagreed and strongly disagreed, respectively. The participants also disagreed that the system was inconsistent and agreed that it was easy to learn. 63.33% and 23.33% strongly disagreed that the system was burdensome, and they felt confident in their ability to use it. The results of the study indicate that the participants found the system to be easy to use without the need for additional learning or experience. They also expressed satisfaction with the system and reported that it was user-friendly. These findings suggest that the system had good usability and was positively received by the participants. Discussion. A system’s effectiveness is typically determined by user feedback on how well it meets their needs, how easy it is to use, and how much time and effort is needed to understand it. In this study, it was found that the system was highly useful and easy to use, with 36.67% and 40% of participants expressing a desire to use it more often. Additionally, 43.33% of participants strongly disagreed and 30% disagreed with the statement that the system was complex. The results also showed that the system was user-friendly and did not require any specific expertise to use, with 30% and 43.33% of participants disagreeing or strongly disagreeing that technical expertise was necessary. Participants reported being satisfied with the system’s usability and efficiency, and it was found that it could be learned quickly without any training or support. The system was not seen as burdensome, and participants felt confident while using it. As a result, participants did not report any difficulties while using the system and said it was easy to learn, as it did not require any specialized expertise or training. This is supported by the system’s high rating on the SUS scale, with a score of 77.1 falling as shown in Fig. 9 [3] within the “good” and “excellent” range and resulting in a grade of B. This indicates the system’s effectiveness and usability.

Development and Evaluation of a Trusted Achievement Record

5.2

113

Assessing the Motivation and Utility of Learning

To assess the effectiveness of the proposed system in terms of achieving a set of objectives, we developed a questionnaire and collected data from 30 students from nine different universities (as shown in Fig. 10 [3]). The results of this questionnaire are depicted in Fig. 11 [3].

Fig. 10. Participants from Students’ Type.

Reliability Test. To evaluate the internal consistency of the statistical data, we used Cronbach’s Alpha, a statistical measure that assesses the consistency of results across all items in a scale. It is commonly used to evaluate the reliability of tests or surveys, and it is typically calculated by comparing the variance of items within a scale to the total variance of the scale. A higher Cronbach’s Alpha value indicates higher internal consistency and therefore stronger reliability. In our case, the Cronbach’s Alpha value of 0.632 indicates that our statistical data has strong internal consistency and can be considered reliable. This is supported by the fact that the value is greater than both the standard threshold of 0.5 and the strong reliability threshold of 0.9. Additionally, a value greater than 0.63 suggests that there is a good level of interrelation between all variables in the data, further supporting its reliability (Table 2). Table 2. Assessing the Motivation and Utility of Learning. Cronbach’s Alpha N of Items 0.632

6

Results and Discussion. According to the results of our survey, the majority of respondents were highly satisfied with the system’s ability to motivate learning, employment planning, and proof of skills, as evidenced by their agreement with the system’s efficiency. Analysis of Fig. 11 shows that the system successfully encouraged students to learn and improve their skills through various

114

B. Awaji et al.

resources beyond their academic courses and modules, with 96.15% of participants agreeing. Additionally, 88.46% of respondents indicated that the system helped them create a future learning plan that aligns with their professional aspirations based on their current progress. This suggests that the system effectively assists students in developing a learning plan. Furthermore, 76.9% of respondents felt that the system facilitated the hiring process and increased students’ connections with potential employers. Most participants, 96.15%, also agreed that the system provided proof of students’ qualifications and skills that was not included in their official transcripts. Half of the respondents were unaware that the system utilized blockchain and smart contract technology, but this did not impact their ability to use the system, as 84.6% of them answered “No” to the question of whether they needed to learn about blockchain in order to use the system effectively. Overall, the results indicate that participants found the system easy to use and user-friendly, despite its integration of blockchain technology. The data also suggests that the system effectively motivates students to improve their skills and learn from various resources, and facilitates connections between stakeholders and employers, which can improve the efficiency of the hiring process. 5.3

Transactions Confirmation Time and Transactions Cost

Transaction confirmation time and cost are important considerations in any blockchain application as they can greatly affect user experience and adoption. This evaluation focuses on two important variables that can impact the performance of blockchain-based systems: Transaction confirmation time and transaction cost. By examining these variables, we aim to evaluate the proposed system in terms performance of blockchain-based systems. These factors are important to consider when evaluating the feasibility and potential success of a blockchain application. Transaction confirmation time, also known as delay time, refers to the amount of time it takes for a transaction to be processed and added to the distributed ledger of a blockchain network. This process begins when a transaction is broadcast to the network and ends when it is included in a block and added to the blockchain. The length of time it takes for a transaction to be confirmed can vary based on a number of factors, including network congestion, the complexity of the transaction, and the fee paid by the sender. Transaction cost, also known as a mining fee or gas fee, refers to the amount paid to process a transaction on the Ethereum blockchain. Gas is the unit that measures the computational effort required to execute specific operations on the Ethereum network, and gas fees are paid in Ethereum’s native currency, Ether (ETH). The cost of a transaction is determined by the amount of gas required to complete the operation and the current market price of gas. Results. We analyzed all executed transactions and found that adding 219 transactions to the blockchain network and confirming their execution took approximately 17 min and 32 s. These transactions were carried out at different

Development and Evaluation of a Trusted Achievement Record

115

times by different universities, with an average confirmation time of 0.24 min per transaction. In terms of transaction cost, the total fee for the 219 transactions was 00.767 Ether, or approximately $657.409 based on the exchange rate at the time of analysis. The average transaction fee was 0.01097 Ether, or approximately $6.1574. The Table 3 [3] provides a summary of the transaction times and costs for the transactions discussed in this study.

Fig. 11. Assessing the Motivation and Utility of Learning by Students.

116

B. Awaji et al.

To obtain more comprehensive results, we analyzed the confirmation time and cost of transactions for six different universities, as shown in Figs. 12, 13 and 14. Figure 12 [3]compares in terms of the number of all executed transactions and total cost for each university. University 1 had 45 transactions, with a total confirmation time of 10 min and 5 s, and a total transaction fee of 0.21992263 Ether. The average transaction time was 14 s, and the average transaction fee was 0.00549 Ether. This pattern is similar for the other universities. The confirmation time of all executed transactions for each university was analyzed, Fig. 13 [3] compares in terms of the number of executed transactions and the average confirmation time for each university. University 1 had 45 transactions, University 2 had 17 transactions, University 3 had 18 transactions, Table 3. Transaction Times and Costs [3]. Total Number of Transactions

219

Total Confirmation Time (MM: SS)

17:32

Average Confirmation Time (MM: SS) 0.24 Total Transactions Fee (Ether)

0.76703077

Average Transactions Fee (Ether)

0.01091671

Total Transactions Fee (USD)

$657.409294

Average Transactions Fee (USD)

$6.15745258

Fig. 12. Number of Transactions and Total Cost (US Dollar $) [3]

Development and Evaluation of a Trusted Achievement Record

117

University 4 had 30 transactions, and Universities 5 and 6 had 28 and 8 transactions, respectively. While University 1 had the highest number of transactions, University 6 took the least amount of time among all the universities. University 2 took 0.009 s and completed 17 transactions, University 3 took 0.091 s and completed 18 transactions, University 4 took 0.025 s and completed 30 transactions, and University 5 took 0.005 s and completed 28 transactions. The cost of all executed transactions for each university was analyzed. Figure 14 [3] compared in terms of the number of executed transactions and cost. University 1 had the highest cost at 0.219 Ether, while University 2 had the lowest cost at 0.051 Ether. University 4 had a total transaction cost of 0.114 Ether, and University 3 had a cost of 0.05 Ether. The number of transactions and total cost in Ether were compared for each university. It was found that University 1 had the highest number of transactions and the highest cost in Ether, while University 2 had the lowest number of transactions but a higher cost in Ether compared to University 4. University 3 and University 6 had similar costs in Ether, but University 3 had a higher number of transactions than University 6. University 2 had the second-lowest number of transactions and the second-lowest cost in Ether.

Fig. 13. Number of Transactions and Average Confirmation Time [3].

The ratio of transactions to cost in US dollars was lowest for University 6, which had 8 transactions and a total cost of $0.068. The number of transactions and average cost varied among the universities. University 1 had the highest number of transactions at 45 and the highest total cost at $125.25. University 6

118

B. Awaji et al.

had the lowest number of transactions at 8 and the lowest total cost at $0.068. The cost for University 1 was higher due to the large number of transactions in a short amount of time. The cost for University 6 was lower due to the small number of transactions. Discussion. The study found that the average time it takes for a transaction to be confirmed on the blockchain varies among the different universities. Additionally, the cost of each transaction on the blockchain also varies. These two factors will discuss in the study. Transaction Confirmation Time. A study analyzed data from 219 transactions and found that the average confirmation times for these transactions varied among different universities. These delays may be caused by congestion on the Ethereum network. The analysis also showed that the efficiency of the system can be evaluated by analyzing the variations in the time it takes for transactions to be added to the blockchain. For example, University 3 had the most transactions (18), but the average confirmation time for these transactions was only 0.09 min. This suggests that the average confirmation time does not necessarily depend on the volume of transactions, but rather on the efficiency of the blockchain system and the level of congestion on the Ethereum blockchain. Additionally, blockchain transactions may be delayed during periods of high traffic, and the gas limit, may also impact confirmation time. Transactions with higher gas limits are more attractive to miners and are therefore more likely to be processed faster, while transactions with lower gas limits may have to wait longer to be completed.

Fig. 14. Number of Transactions and Total Cost (Ether) [3].

Development and Evaluation of a Trusted Achievement Record

119

Transaction Cost. Gas is a unit used to measure the computational effort required to execute certain operations on the Ethereum network. It is used to pay miners for their work in processing transactions and securing the network, as well as to control a transaction’s resources by determining how much computational power is needed to execute it. Gas is kept separate from the Ethereum cryptocurrency (ETH) to protect the system from ETH price volatility and to more accurately reflect the true cost of computation, memory, and storage. Users can specify the gas price for their transactions in Gwei per gas unit, which determines how much they are willing to pay for the resources required to execute their transaction. Higher gas prices generally result in faster confirmation of the transaction, as miners will prioritize transactions with higher gas prices. However, the market determines the price of gas and the cost of computation, so setting a gas price that is too high may not be necessary or cost-effective. The gas limit for proposed system transactions was 40,000 Gwei (0.004 ETH). This value was chosen because it allowed all user transactions during system evaluation to be confirmed in a reasonable amount of time, given the transaction data volume. Lower gas prices may result in slower confirmation for lower priority transactions. The Ethereum Virtual Machine (EVM) measures the cost of computation and storage in terms of Ether-priced gas. Transactions on the Ethereum network specify the gas price in ETH per gas unit, and the total cost of a transaction is calculated by multiplying the gas price by the number of gas units required. The Eq. 1 is used to calculate the transaction fee for a given transaction on the Ethereum network. Transaction fee = totalgasused × gas price paid (in Ether)

(1)

The cost of a transaction on the Ethereum network can vary based on the factors included in the equation used to calculate the transaction fee. These factors include the gas price, which is specified in Gwei per gas unit and determines how much the user is willing to pay for the resources required to execute the transaction, and the number of gas units required, which is determined by the complexity of the operation being performed and the current state of the network. Both of these factors play a role in determining the final cost of a transaction. Total Gas Used. The gas limit for the proposed system for each transactions on the Ethereum network was set at 40,000 Gwei (0.004 ETH) in order to speed up the mining process. This gas limit specifies the maximum amount of resources that can be used for a given transaction. However, it is not necessary to use the full gas limit, as users only need to pay for the computational, bandwidth, and storage resources they actually consume. Any gas not used in a transaction is returned to the user. As a result, the total gas used and the resulting transaction fee can vary with each transaction.

120

B. Awaji et al.

Gas Price Paid (in Ether). Gas fees on the Ethereum network are paid in ETH, the network’s native cryptocurrency. Gas prices are denoted in Gwei, a smaller unit of ETH equal to 0.000000001 ETH, or 10-9 ETH. Conversely, 1 ETH is equal to 1 billion Gwei. The value of ETH can vary significantly on the stock market, which can also affect the cost of a transaction in ETH. For example, during the period in which this study was conducted, the price of ETH was $213.61 in May 2020 and the gas price was 0.004 ETH. This resulted in a transaction fee of 0.004* 213.61 = $0.854. If the price of ETH increased to $2036.55 in February 2021, the transaction fee was 0.004*2036.55 = $8.1462. This demonstrates how the cost of a transaction can change significantly over time due to changes in the value of ETH on the market. It can be difficult to accurately estimate confirmation times for transactions on the Ethereum network, as the transaction fee is not determined until the transaction is confirmed. There are several factors that can affect the confirmation time for a transaction, including the efficiency of the network and the current market price of ETH. Our findings found that confirmation times can be irregular but remain within an acceptable range. However, further research is needed to fully evaluate this limitation and assess the impact of factors such as market prices and network efficiency on the sustainability of the system. Transaction fees are another constraint for the Ethereum system. High fees may discourage users from using the system, as they may be unwilling to pay exorbitant costs for transactions. Additionally, the cost, scalability, and energy consumption of the Ethereum blockchain are issues that may need to be addressed in order to ensure the long-term sustainability of the system. Smart contracts are an important part of the Ethereum system, as they allow for the execution of complex blockchain functions. Ethereum is a permissionless blockchain platform specifically designed to support the creation and deployment of complex smart contracts. However, not all blockchain platforms support smart contracts, and those that do not may not be able to address the issues of cost, scalability, and energy consumption in the same way as Ethereum and other platforms that do support smart contracts. Table 4 [3] illustrates the different characteristics and design decisions of various blockchain platforms, including Ethereum and Hyperledger, which are designed to support rich and complex smart contracts. Hyperledger is an open-source collaborative project that aims to advance permissioned blockchains and provide an infrastructure of different modules and tools for developing blockchain platforms.

Development and Evaluation of a Trusted Achievement Record

121

Table 4. Different Blockchain Platforms and Their Characteristics [3]. Blockchain Platform Network Permission Smart Contract Support Bitcoin

Permissionless

No

Ethereum

Permissionless

Yes

Zcash

Permissionless

No

Litecoin

Permissionless

No

Dash

Permissionless

No

Peercoin

Permissionless

No

Ripple

Permissionless

No

Monero

Permissionless

No

MultiChain

Permissionled

No

Hyperledger

Permissionled

Yes

Using either Ethereum or Hyperledger to build a application-based Blockchain depends on the system requirements and objectives. However, one potential solution to the issue of transaction costs on Ethereum is to use a private Ethereum blockchain, which offers zero transaction fees, higher scalability, and no restrictions. However, transitioning to a private blockchain may require significant modifications to the design of application, including the need for permission to join the controlled blockchain and read its state. Alternatively, Hyperledger Fabric may be worth considering as a solution to the issue of transaction costs. Ultimately, the choice between these platforms will depend on the specific needs of the application. 5.4

Comparison of the Proposed System with the CVSS System

To assess the effectiveness of our proposed system, we compared it with the CVSS system [26] in terms of transaction confirmation time and cost. The CVSS system is an existing system that was used as a point of comparison in this evaluation. Our findings were compared to those presented in the CVSS system’s published paper in order to determine the performance and efficiency of our proposed system. The CVSS system is an approach that utilizes the blockchain technology to issue immutable digital certificates. In a comparison with the CVSS system as shown in Table 5 [3], the proposed system was found to have lower transaction costs, with a cost of $10.76 for deploying a smart contract on the Ethereum blockchain compared to $19 for the CVSS system. The proposed system also had a higher number of transactions, with three times as many as the CVSS system. The mean cost for transactions in the proposed system was $6.16, which was higher than the $0.15 cost per transaction in the CVSS system, but the proposed system had a larger number of transactions over a longer period of time. In terms of transaction confirmation time, the mean time for the CVSS

122

B. Awaji et al. Table 5. Financial Comparison of CVSS and Our System [3]. CVSS Our System Contract Creation Cost

$19

$10.76

Number of Transactions

60

219

Transaction Cost

$0.15

$6.16

Average Transaction Confirmation Time (mm:ss)

00:60

00:24

Total Transactions 05:00 Confirmations Time (mm:ss)

17:32

system was 60 s, while the proposed system had an average confirmation time of 24 s. The total transaction confirmation time for the proposed system, which had 219 transactions, was 17 min, which was deemed acceptable given the number of transactions. The reasons for the differences in transaction costs and confirmation times between the two systems may be due to factors such as Ethereum network congestion and the gas limit.

6

Conclusion

The aim of this work is to illustrate that blockchain technology has a significant amount of untapped potential in terms of certifying and confirming academic accomplishments on a worldwide scale. As a consequence of this, it would be viable and advantageous to a number of different parties. There is a widespread problem with fraudulent credentials, which is detrimental to educational institutions, students, and society as a whole. The options that are now available, such as outdated credential verification systems, are cumbersome and inefficient with both time and money. In addition, they are incapable of providing an effective response to unethical practises, such as fraud committed by educational institutions and certification agencies. When it comes to combating widespread fraud, the record of accomplishments that is provided with the utilisation of Blockchain technology is thorough. In addition, when compared to older systems, this one is a huge improvement, as it is not only more effective but also more user-friendly. In addition, the approach of using Blockchain technology is a solution that may successfully integrate into the ecosystem that is currently used for credential verification. As a result, the purpose of this study is to make a contribution to the current efforts that are being made to combat credential fraud. Finally it is important to note that when building a solution that is based on the blockchain, it is vital to take into consideration the energy consumption as well as the transaction fees that are linked with algorithms such as PoW. The current version of the Trusted Achievement Record System is designed based on model public blockchains and contains models for the most popular public blockchain (Ethereum). However, to tackle its limitations, it is an opportunity

Development and Evaluation of a Trusted Achievement Record

123

for future researchers to refine the Trusted Achievement Record System using private blockchains such as Hyperledger Fabric. This proposed system was developed to be extendable. Therefore, adopting new tasks or services will increase the system’s feasibility. In addition, artificial intelligence (AI) methods could provide more potential use cases to the system, for example, finding the most appropriate student for the job based on the achievement record in the system, where the AI methods can link students’ skills to the job requirements.

References 1. Solaiman, E., Todd, W., Ioannis, S.: Implementation and evaluation of smart contracts using a hybrid on-and off-blockchain architecture. Concurr. Comput. Pract. Exp. 33(1), e5811 (2021) 2. Yumna, H., Khan, M., Ikram, M., Ilyas, S.: Use of blockchain in education: a systematic literature review. In: Asian Conference on Intelligent Information and Database Systems, pp. 191–202 (2019) 3. Awaji, B., Solaiman, E.: Design, implementation, and evaluation of blockchainbased trusted achievement record system for students in higher education. In: Proceedings of the 14th International Conference on Computer Supported Education - Volume 2, pp. 225–237 (2022). https://doi.org/10.5220/0011044200003182. ISBN 978-989-758-562-3. ISSN 2184-5026 4. Awaji, B., Solaiman, E., Albshri, A.: Blockchain-based applications in higher education: a systematic mapping study. In: Proceedings of the 5th International Conference on Information and Education Innovations, pp. 96–104 (2020) 5. Awaji, B., Solaiman, E., Marshall, L.: Blockchain-based trusted achievement record system design. In: Proceedings of the 5th International Conference on Information and Education Innovations, pp. 46–51 (2020) 6. Awaji, B., Solaiman, E., Marshall, L.: Investigating the requirements for building a blockchain-based achievement record system. In: Proceedings of the 5th International Conference on Information and Education Innovations, pp. 56–60 (2020) 7. Brooke, J.: SUS: a quick and dirty usability. J. Usability Eval. Ind. 189(3) (1996) 8. Cappelli, P.: Your approach to hiring is all wrong. Harv. Bus. Rev. 97(3), 48–58 (2019) 9. Shay, J., Simon, W.: SHA-512/256Gueron. In: IEEE Eighth International Conference on Information Technology: New Generations, pp. 354–358 (2011) 10. Han, M., Li, Z., He, J., Wu, D., Xie, Y., Baba, A.: A novel blockchain-based education records verification solution. In: Proceedings of the 19th Annual SIG Conference on Information Technology Education, pp. 178–183 (2018) 11. Henle, A., Dineen, R., Duffy, K.: Assessing intentional resume deception: development and nomological network of a resume fraud measure. J. Bus. Psychol. 34, 87–106 (2019) 12. Higher Education Degree Datacheck (2022). https://hedd.ac.uk/ 13. Jirgensons, M., Kapenieks, J.: Blockchain and the future of digital learning credential assessment and management. J. Teach. Educ. Sustain. 20, 145–156 (2018) 14. Molina-Jimenez, C., et al.: Implementation of smart contracts using hybrid architectures with on and off-blockchain components. In: 2018 IEEE 8th International Symposium on Cloud and Service Computing (SC2), pp. 83–90 (2018)

124

B. Awaji et al.

15. Molina-Jimenez, C., Solaiman, E., Sfyrakis, I., Ng, I., Crowcroft, J.: On and offblockchain enforcement of smart contracts. In: Mencagli, G., et al. (eds.) Euro-Par 2018. LNCS, vol. 11339, pp. 342–354. Springer, Cham (2019). https://doi.org/10. 1007/978-3-030-10549-5 27 16. NGA Human Resources (2022). https://www.ngahr.com/ 17. Nguyen, D., Nguyen-Duc, D., Huynh-Tuong, N., Pham, A.: CVSS: a blockchainized certificate verifying support system. In: Proceedings of the Ninth International Symposium on Information and Communication Technology, pp. 436–442 (2018) 18. How much does your CV lie? (2022). https://www.theukdomain.uk/much-cv-lie/ 19. Schmidt, P.: Blockcerts-an open infrastructure for academic credentials on the blockchain ML Learning (2016) 20. Vidal, F., Gouveia, F., Soares, C.: Analysis of blockchain technology for higher education. In: International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), pp. 28–33 (2019) 21. Virkus, S.: The use of open badges in library and information science education in Estonia. Educ. Inf. 35, 155–172 (2019) 22. Watters, A.: The blockchain for education: an introduction (2022). http:// hackeducation.com/2016/04/07/blockchain-education-guide 23. Werner, S.M., Pritz, P.J., Perez, D.: Step on the Gas? a better approach for recommending the ethereum gas price. In: Pardalos, P., Kotsireas, I., Guo, Y., Knottenbelt, W. (eds.) Mathematical Research for Blockchain Economy. SPBE, pp. 161–177. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-53356-4 10 24. Wood, G.: Ethereum: a secure decentralised generalised transaction ledger. Ethereum Project Yellow Paper 151, 1–32 (2014) 25. Nunnaly, J.C., Bernstein, I.H.: Psychoneric Theory, 2nd edn. McGraw Hill, New York (1978) 26. Nguyen, D.H., Nguyen-Duc, D.N., Huynh-Tuong, N., Pham, H.A.: CVSS: a blockchainized certificate verifying support system. In: The Ninth International Symposium on Information and Communication Technology (SoICT) (2018)

Students’ Perceptions of Computer Science and the Role of Gender Sara Hinterplattner(B) Linz Institute of Technology, Dynatrace Austria, Linz, Austria [email protected] Abstract. Most companies have difficulty recruiting specialists in the field of computer science. The lack of qualified women in this field is particularly striking. Research shows that early exposure to STEM subjects can spark children’s interest and influence their later career choices. However, stereotypical perceptions hinder students in pursuing to work in the field of computer science. In this paper a study in Austrian secondary schools with 188 participating students is described. It investigates fifth graders’ perceptions of computer science before they experience computer science education at school. In particular, the influence of gender will be explored. Results show that students have a very vague picture of what computer science is and what a computer scientist does. Students who identified themselves as female, were less interested in computer science than students who identified themselves as male. (Low) Correlations between the interest in computer science, wanting to work in the field of computer science, and knowing someone working in the field of computer science were found with higher correlations in the group of students who identified themselves as female compared to the group of students who identified themselves as male. Keywords: Perceptions Science education

1

· Computer science · Gender · Computer

Introduction

The 2021 Digital Economy and Society Index showed that 55% of enterprises engaged with recruitment of ICT specialists reported difficulties with filling vacancies, and more than 70% of businesses report difficulties with investing in digital development due to lack of digitally skilled workers [19]. Already in 2004 the final report from a large project founded by the European Union addressing the condition of science and teaching in the European Union demands that “Europe need more scientists!” [18]. In 2015 the European Commission estimated that over a million jobs in STEM professions are expected to be created in the European Union between 2013 and 2025 showing the increasing demand for qualified people in the STEM field [10]. In the 2021 Digital Economy and Society Index, it is also noted that the gender imbalance is high, with only 19% of ICT specialists, and one third of STEM graduate being female. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2023  J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 125–148, 2023. https://doi.org/10.1007/978-3-031-40501-3_6

126

S. Hinterplattner

Bollin et al. (2020) also report that interest in computer science tends to decline among young women [4]. However, there are STEM fields where women are not under-represented, in biological sciences or environmental sciences the gender balance is shifted towards women [12,46]. Potential reasons for these differences are stereotypes of the STEM field, insufficient early experience, or the lack of role models [12,32,45]. This is supported by research by Stout et al. (2011) who showed that female university students could be influenced by role models resulting for example in future career ideas [48]. A lack of specialized educational programs in key areas such as AI and cyber security, and a lack of integration of digital skills in other disciplines increases the difficulties of changing these conditions. Research further shows that the gender gap emerges early, as girls do not even choose schools or studies in the field of computer science because working on computers or coding is unattractive to them. Among young girls, computers and computer science are often perceived as nerdy or geeky [42]. The global technology company Dynatrace is one of the companies that is being affected by the lack of ICT specialists, and the lack of women in the field. With over 60 offices around the world and more than 3600 highly educated employers, the rapidly growing company is developing their platform that “simplifies enterprise cloud complexity and accelerates digital transformation” [17]. At the headquarters in Linz, Austria, the shortage of qualified workers is a felt reality. As the rapid growth of the company have been slowed down by the increasing difficulties to hire, the company has searched even further away to fill its vacancies. As of today, more than 25% of the company’s workforce in Austria located in Graz, Hagenberg, Innsbruck, Klagenfurt, Linz, and Vienna are non-Austrians, having been recruited from nearly 60 different countries with most coming from Germany, Egypt, and India. Even so, the problem remains. To help mitigate this problem in the future, Dynatrace has started initiatives to support families in the company by introducing the “Dynatrace Way of Life” including flexible working hours (flexible schedule to fit family’s needs or personal interests), home office options (choosing how many days to work in the office), supportive and empathetic leadership (letting the lead know how to integrate the work life and home life most effectively), health programs (benefiting from standing desks, healthy food, sport events, mental health workshops among others), diversity initiatives (valuing diversity, empowering women in tech, and supporting internationals), flexible leave policies (enjoying leaves while staying connected with colleagues how it fits best to the schedule), family-friendly environments (parent meetups, working part-time, family days, summer care among others), growing and learning initiatives (attending programs, e-learning courses, conferences, and further training, possibilities for doing research), interesting and fulfilling work goals (deciding the own tasks and having an impact on building important software), one-of-a-kind teams (working with extremely talented and valued people with regular celebrations and initiatives for team building), and STEM projects (youth development initiatives). With the youth development initiatives Dynatrace wants to spark interest in STEM subjects for young children. Aiming to develop early digital skills, and decrease gender equality, the company has conducted different studies. Results of these studies are presented in this paper.

Students’ Perceptions of Computer Science and the Role of Gender

2

127

Early STEM

Young children are curious and want to know how the world works. However, their spontaneous interest in natural phenomena decreases as they grow older [39]. Therefore, the first contact with the STEM field should be as soon as possible, the infant and toddler age offers an ideal condition to present scientific inquiry to children. In this age, the human brain develops rapidly and the natural strong curiosity of the children in exploring the world and the drive towards understanding their surroundings provides possibilities to get to know STEM [7]. Moreover, research shows that the early exposure to STEM related activities supports children’s long term development and achievements withing the field: Early, meaningful experiences of science for young children, for example, have been found to enhance self-belief in their ability to learn science and to promote greater interest in science [40] and that such experiences trigger an appreciation for science and its value to everyday life [22]. Hunting, Mousley, and Perry (2012) highlight that mathematical skills developed at an early age, such as number sense and ordinality, are strong predictors of later academic success [30]. This is supported by research from Duncan et al. (2007) who showed that mathematical knowledge in preschool predicts mathematical achievement into the high school years, and preschool mathematical skills predict later academic achievement more consistently than early reading or attention skills [15]. Besides, the early exposure to STEM can spark children’s interest and influence their future choice of career [33,44,49,50]. As showed in the research of Cheryan et al. (2017) insufficient early experience in the STEM field is a reason for not aiming to work in the STEM field [12]. For this development of STEM engagement for young learners, teachers do play a critical role. Teachers who are confident and enthusiastic about STEM topics, and who engage their students in developmentally tailored STEM activities, pass that excitement to their students. However, many early childhood teachers are not eager and prepared to engage children in rich experiences in domains other than literacy [6,13,16]. In fact, there is widespread anxiety about topics like mathematics among teachers of young children, which correlates with the achievement of their students, particularly girls [3]. Furthermore, many teachers do not know how to adapt STEM instruction to suit the needs of their students [36]. Moreover, teachers experience anxiety, low self-confidence, and gendered assumptions about STEM topics, which can transfer to their children and students [36]. This is also the case for parents [36]. According to a 2012 literature review, parents tend to expect that boys are more gifted in STEM than girls, even when their achievement levels do not differ objectively, and those beliefs are passed along to their children in both implicit and explicit ways in their behaviors [26]. More explicitly, parents’ gendered attitudes toward STEM can be communicated through the opportunities they provide and the activities they encourage. For example, parents are three times more likely to explain science exhibits to their preschool boys than girls when they visit a museum [14] or they tend to purchase more STEM toys for boys than for girls, one of several parental STEM-promoting behaviors that have been linked to children’s STEM involvement several years later [31].

128

S. Hinterplattner

Consequently, children’s earliest experiences with science, technology, engineering, and mathematics might subsidize future engagement and success in these fields [28]. This is not only beneficial for the children themselves, but also for their countries: According to the Organisation for Economic Co-operation and Development there is a common need in Europe to produce higher student achievements in basic mathematics, science and literacy skills and there is still significant under performance, particularly in mathematical skills in all countries being part of the Organisation for Economic Co-operation and Development. If a country wants to support growing technological innovation, then it is important to increase the amount of positive exposures and experiences to STEM fields for young students [1].

3

Theoretical Background

Several authors have discussed the perceptions and misperceptions of computer science among children. Beaubouef and McDowell (2008) investigated common myths and misconceptions about computer science including the nature of the field and the types of activities involved. An example listed is that the definition of “know how to use a computer” is knowing how to surf the internet or to install new peripherals without being aware of the complexity of computer science. The researchers state that overcoming negative myths would help students see computer science as an exciting and fast growing field that leads to many diverse and rewarding careers [2]. Martin (2004) reported on an exercise in an introductory programming course in which students are asked to explain what computer science is as well as draw a computer scientist [35]. In particular, the data from the drawings show that computer science has a “fundamental image problem”, not a single one of the pictures showed attractive, appealing, or normal looking people. This image problem was also shown in other research (e.g. Mercier et al. (2006)). Moreover, students lack a clear understanding of computer science and the tasks of computer scientists [35]. This was already seen earlier, when Greening (1998) for example asked high school students to complete the sentence “Computer Science is mostly about...”. The results showed mainly no responses or answers that were classified as trivial or computer-centric. Mitchell (2009) also reported that few students have a clear notion of computer science [38]. Students’ perceptions of the computer science discipline develop early in their school career and thus better integration between schools and universities is needed. Yardi and Bruckman (2007) conducted interviews with teenagers to find out about the students’ perceptions of computer science and computing related fields and their ambitious to pursue degrees or careers in these disciplines. Their purpose was to better understand why students are not interested in computer science. The results show that the students are mainly not interested in the field of computer science because of stereotypical perceptions, for example that computer science is boring, difficult, tedious, solitary, and lacking a real-world

Students’ Perceptions of Computer Science and the Role of Gender

129

context. This was also the case even that the students were creative, passionate, and enthusiastically engaged in their online environments [51]. Similarly stereotypical descriptions were found by other researchers, for example Cassel et al. (2007) [9]. This lack of understanding what computer science is and what computer scientist do leads to disinterest in computer science [8]. In his study with over 800 high school students, Carter (2006) showed that the students had too less formal classroom experience with computer science and most of them did not know what you are doing when you study computer science [8]. Moreover, research showed that students saw computer science not as an environment for creativity [41], felt unwelcomed in the field [21,34], and did not see themselves pursuing careers in the field of computer science [51]. Hansen et al. (2017) developed the so-called “Draw-A-Computer-ScientistTest” by adapting the “Draw-A-Scientist-Test” [11] to understand how young children perceive computer scientists by analyzing the drawings regarding gender, age, work situation, and work tasks. The findings confirmed the existence of stereotypes meaning the computer scientists in the drawings were male, had a mean age of 25, work alone, predominantly used computers, performed a vague set of tasks (including “working”, “coding”, “making”, “typing”, “doing”, “looking”, “fixing”, and “testing”), and were often scientists who used computers (for example a chemical scientist doing an experiment and having a computer next by). It has to be emphasized that the results of the “Draw-a-Person-Test” showed that students are more likely to represent their own gender [23] in contrast to the experiences with the “Draw-A-Computer-Scientist-Test” [27] where the computer scientist were mainly male no matter which gender the student drawing had. After completing around 12 h of programming instructions the participating students in the study were asked to repeat the “Draw-A-Computer-ScientistTest” showing that more female students drew female computer scientists after the programming classes than before, but also more computer scientist were pictured after the classes bald or wearing glasses [27]. There have been more research regarding the gender differences in misperceptions in computer science. For example, Mercier et al. (2006) performed already before two studies based on surveys, drawings, and interviews to examine sixthand eighth-grade students’ perceptions of knowledgeable computer users and their self-perception as a computer-type person [37]. Both male and female students mostly drew male users, however the sixth graders drew a higher percentage of female computer scientists than the eight graders. Stereotypical features were very common including wearing glasses, lab coats, or pocket protectors, pale complexion, descriptions of “nerds”, descriptions of negative social characteristics, or abnormal body weight and as mentioned before being male. Of the 70 drawings with two or more stereotypical features 62 were male and 66 were wearing glasses. Moreover, most students said that they thought a computertype person existed defining it for example as someone who knows a lot about computers, spends a lot of time with computers, or someone who loves computers. However, most of the students did not believe that they were such a computer-type person [37].

130

S. Hinterplattner

There have been several efforts to remedy this situation and support students in developing perceptions of computer science discipline as more realistic, broader, and with real-world-context. Following this goal, Grover et al. (2014) presented a middle school introductory computer science and programming curriculum designed to increase students’ awareness of computer science as a problem-solving discipline in a real-world context and attempts to address issues of (mis)perceptions of computer science [25]. The authors asked the students about their view of computer science before and after the course. Results show that before the course the responses to what a computer scientist does were mainly “building”, “fixing”, “studying”, or “improving computers”. After the course a positive shift in their perceptions of the discipline could be observed seeing computer science as a problem-solving discipline helping to make lives easier. Yardi and Bruckan (2007) also proposed a new curriculum to teach teenagers core computing principles [51]. The goal of this curriculum is to present computer science as an innovative, creative, and challenging field with authentic, real-world applications. Moreover, they suggested to show the students the diversity of career paths in computer science and highlighted the importance of role models in the field [51]. Bollin et al. (2020) suggested using attractive teaching material and classroom interventions to stimulate realistic interest for girls, such as enacting teamwork or working in an interdisciplinary manner [4]. Interdisciplinary material can strengthen students’ self-confidence and arousing a wide interest [4]. Researchers agree that students’ perceptions of computer science must develop early and was also described in the last section. Moreover, the understanding for computer science must move beyond hardware, software, and programming to evolve a more realistic picture of the field of STEM and computer science leading to a change of the picture of computer science [8,35,51].

4

Methodology

This study focuses on the desire to strengthen children’s conceptions of computer science before they are confronted with it as a subject at school. As seen in the literature review, students’ stereotypes in STEM are very common and are reasons for not choosing a career in the STEM field. Consequently, it is essential to understand students’ stereotypes of computer science to counteract this trend. With this aim, a study with 188 fifth-grade students was conducted before they started having computer science as a subject at school. At the time of the test, the students had not previously been taught other school subjects related to computer science or had any other previous experience that they categorized in the field of computer science. To delve into the students’ perceptions of computer science, a questionnaire consisting of two parts was developed: The first part was dedicated to the students’ perceptions of and experiences with computer science, the second part asked children to draw a picture of a person working in the field of computer science and to answer some questions about the person they had in their mind. This paper shows the results from the first part that contained six questions:

Students’ Perceptions of Computer Science and the Role of Gender

131

four open-ended questions for which a short passage had to be written and two multiple choice questions. None of the questions was mandatory to answer. The questions were asked in German and can be translated as follows: 1. Are you interested in computer science? Choose an option: (1) I am very interested in computer science, (2) I am interested in computer science, (3) I am a little interested in computer science, or (4) I am not interested in computer science. 2. What is computer science? Provide a short answer. 3. What does a person working in computer science do? Provide a short answer. 4. In the future, do you want to work in computer science? Choose an option: (1) yes, (2) no, or (3) maybe. 5. Give the reasons for your answer to question 4. 6. List the people you know that work in computer science. The participants of the study were students from different secondary schools in Austria in different areas. In total, 188 students were willing to fill out the questionnaire. Sixty-eight of the participants identified themselves as female (36.17%), 118 as male (62.77%), and two did not mention their gender identity (1.06%). At the time of the study, all the students were between 9 and 11 years old and attended the fifth grade. This grade was chosen because at the time the survey was conducted students in fifth grade had not yet been taught a subject related to computer science at school. In Austria, computer science education was introduced as a compulsory subject at secondary general school for the first time in school year 1985/86 school [43]. Back then computer science was only taught to ninth-grade students (one teaching unit (50 min) per week) but has later increased to two teaching units per week [47]. Hence, computer science education remained limited to one grade level. Consequently, it was not yet taught to all students continuously or consistently [5]. To adapt the education system to the increasing importance of digitization and to expand the technical infrastructure and better integrate digitization into teacher training, the “Basic Digital Education” curriculum for lower secondary schools was introduced and implemented for the first time in the 2018/19 school year. The implementation of this curriculum was compulsory, but schools could choose to set two to four teaching units per week in total in grade 5 to 8 as well as whether to offer it as an independent subject or integrate it into existing subjects. The curriculum included topics such as social aspects, media design, digital communication, security, technical problem solving, and computational thinking, where students work with algorithms and acquire rudimentary programming skills [20]. Most of the schools offered “Basic Digital Education’ integrated in other subjects, this integration was implemented for example by using the internet for research in science subjects, using word processing software in Language classes, learning touch typing in German classes, or using methods of computational thinking in sports classes. Finally, in the school year 2022/23 the mode of the subject “Basic Digital Education” changed to a mandatory independent subject with four teaching units per week in total in grade 5 to 8 (one

132

S. Hinterplattner

teaching unit per week every year). In preschools or primary schools in Austria there is still no mandatory subject related to computer science. Therefore, the participants of the study have not received any school education in computer science. Moreover, the participants had also not received any additional computer science-related education. Since this study investigates the students’ perceptions of computer science, this was seen as essential. The data analysis was conducted by the author of this paper and three other researchers. Discrepancies between the coders were discussed within the author group to adapt the coding guidelines. To analyze the data, descriptive and inferential statistics as well as content analysis methods were employed. Statistical tests were used to determine if the means of two sets of data were significantly different from each other. SPSS Statistics 25.0 was used to assist in the descriptive and inferential statistics.

5

Preliminary Results

The preliminary results described first findings about experiences of the students with computer science regarding interest, definitions, perceptions, and role models [29]. In the next chapter the detailed results about the gender differences and correlations between the results will be presented. 5.1

Definitions of Computer Science

In the questionnaire, the students had to define computer science using a short passage. An answer was not mandatory, however, all students (N = 188) answered this question. The answer “doing something with a computer/laptop/PC” was mentioned the most often (n = 129, 68.62%). None of the other definitions were as common and are listed here in descending order of frequency: “a subject in school”, “I don’t know”, or “programming” (n = 17, 9.04%), “something with technology” or “ten finger system” (n = 12, 6.38%), “cool”/“fun”/“great” (n = 11, 5.85%), “surfing the Internet” (n = 6, 3.19%), “something with information” (n = 5, 2.66%), “doing something with a device”, or “software” (n = 3, 1.60%), “automation”, “hardware”, “operation”, “processing”, “science”, or “binary” (n = 2, 1.06%), “doing something with a tablet”, “installing”, “looking up things”, “math”, “modifications”, name of a game console, “office”, “smartphone”, “USB stick”, “website”, or “you need IQ” (n = 1, 0.53%). When looking at the frequencies of given answers in the gender groups,“doing something with a computer/laptop/PC” was also the most frequent answers in the group of students who identified themselves as female and in the group of students who identified themselves as male. However, there were some differences between these groups. The terms mentioned by the students who identified themselves as female but not by the students who identified themselves as male included “automation”, “math”, “modifications”, and “website”. But, these definitions were also rarely mentioned (i.e. once or twice) in this group. The

Students’ Perceptions of Computer Science and the Role of Gender

133

terms noted by students who identified themselves as male but not by the students who identified themselves as female included “binary”, “cool”/“fun”/“great”, “doing something with a device”, “installing”, “looking up things”, name of a game console, “office”, “operation”, “science”, “smartphone”, “USB stick”, and “you need IQ”. Again, the definitions were rarely mentioned (i.e. once or twice) in this group except the cluster of positive descriptions (“cool”/“fun”/“great”) noted eleven times. These positive descriptions were not used by students who identified themselves as female, but were used by eleven students who identified themselves as male (9.32%), making it the third most frequent answer for this group. The second most frequent answer given by this group was “I don’t know” (n = 15, 12.71%) in contrast to only two mentions in the group of students who identified themselves as female (0.29%). Here, the second most frequent answers were “a subject in school” and “programming”, with nine mentions each (15.52%). 5.2

Tasks of Computer Scientists

For the question about what a person who works in computer science does, it was possible to answer with a short passage. Although an answer was not mandatory, all the students (N = 188) answered. Similar to the question before (“What is computer science?”), the most frequent answer was “doing something with a computer/laptop/PC” (n = 78, 41.49%) even though this answer was not given as often as for the question before. The second most frequently mentioned aspect was “programming” (n = 35, 18,62%), being mentioned more than twice as often here, than at the previous question (n = 17, 9,04%). However, more different and unique answers were given for this question: “something with technology” (n = 15, 7.98%), “I don’t know” or “writing” (n = 11, 5.85%), “developing” or “installing” (n = 6, 3.19%), “supporting” (n = 5, 2.66%),“software” (n = 4, 2.13%), “calculating”, “doing something with a device”, “explaining”, “looking up things”, or“surfing the Internet”(n = 3, 1.60%), “apps”, “constructing”, “chatting”, “e-mails”, “office”, “operating system”, “repairing”, “robots”, “teacher”, or “website” (n = 2, 1.06%), “appointments”, “architect”, “boss”, “configuring”, “doing something with a screen”, “engineer”, “gambling”, “modifications”, name of a communication platform, name of a search engine, name of a spreadsheet software, name of a word processing software, “problem solving”, “squatting”, or “USB stick” (n = 1, 0.53%). When looking at the frequencies of given answers in the gender groups, the most common answer was again “doing something with a computer/laptop/PC” (45.59% (f) and 34.75% (m)) in both groups. However, there are some differences between the groups. The terms mentioned by the students who identified themselves as male but not by those who identified themselves as female included “app”, “appointments”, “architect”, “boss”, “calculating”, “configuring”, “engineer”, “gambling”, name of a communication platform, name of a search engine, name of a spreadsheet software, name of a word processing software, “office”, “operating system”, “problem solving”, and “website”. But, these definitions were also rarely mentioned (i.e. once, twice, or three times) in this group. Students who identified themselves as female noted only one term that was not mentioned by

134

S. Hinterplattner

the other group: “chatting”, but also her rarely (twice). The second most frequent answer given by the students who identified themselves as female was “I don’t know” (n = 14, 20.59%) followed by “programming” (n = 10, 14.71%). Similarly, “programming” was in the other group the second most frequent answer (n = 24, 8.47%) followed by “I don’t know” (n = 8, 6.78%). 5.3

Reasons for (not) Pursuing a Career in Computer Science

One-hundred-eighty-three reasons for pursuing a career in the field of computer science or not pursuing a career in the field of computer science were given. Twenty-four by the students who wanted to work in the field of computer science in the future (everyone gave one answer), 122 by the students who maybe wanted to work in the field of computer science (everyone gave an answer), and 37 by the students who do not want to work in the field of computer science in the future (from 36 students). The different reasons were depending on the answer given to the question if they want to work as a computer scientist in the future. Nine of the students who were thinking of a career in the field of computer science noted that they were very interested in the field (37.50%), six wanted to learn programming (25%) including three who mentioned programming games (12.50%), five liked “doing things on computers” (20.83%), five noted that computer science was fun (20.83%), and two wrote about having role models in the field (8.33%). Answers that were given only once in this group were that they wanted to learn something new, wanted to support others, wanted to learn computer science, wanted to develop a better search engine, and “were talented at computer science” (4.17% each). Fifty-six of the students answering maybe wanting to work in the field of computer science mentioned that they did not yet know what job they wanted in the future (45.90%), 17 said that computer science was fun (13.93%), 15 noted that they thought that a job in the field of computer science would be interesting (12.30%), 14 already had other plans for their future in mind (11.48%), eight liked “doing things at the computer” (6.56%), and three were sure that they would be good at computer science (2.46%). Answers that were given twice by the students from the “maybe” group were that it would be too much work, that it would be good to have the competency, that programming would be good to know, and that computer science would be very useful because it is “necessary nearly everywhere” (1.64% each). Explanations mentioned once by students from this group included that they liked surfing or having money, that it would be one of their dream jobs, and that they had a role model in the field (0.82% each). Sixteen of the students that did not want to work in computer science in the future already had other plans for their future (43.24%), eleven did not know what they wanted to do (29.73%), and seven mentioned that they were not interested in computer science (18.92%). Answers mentioned only once in this group included that they did not want to sit in front of a computer that much, that they did not know anything about this field, and that they preferred working with people, with animals, or in the sports field (2.70% each). The comparison of the answers by gender identity was omitted due to an insufficient amount of data.

Students’ Perceptions of Computer Science and the Role of Gender

6 6.1

135

Results Students’ Interest in CS and the Aim of Working in CS

To find out more about the relationships between the interest of students in computer science and the aim of working in the field of computer science, correlations were investigated. In the questionnaire, the students were asked if they were interested in computer science. They could answer this question with the following four given options: (1) I am very interested in computer science, (2) I am interested in computer science, (3) I am a little interested in computer science, and (4) I am not interested in computer science. However, the students were not required to provide an answer. For the question about working in computer science in the future, it was possible to choose between three options (yes, maybe, and no) or provide no answer. In total, 182 students (96.81%) answered the question if they were interested in computer science, with 105 saying that they were very interested in computer science (57.69%), 55 that they were interested in computer science (30.22%), 19 that they were a little interested in computer science (10.44%), and three replying that they were not interested in computer science (1.65%). Of the 183 students who answered the question if they want work in the field of computer science in the future (97.34%), 24 stated that they wanted to work in computer science in the future (13.11%), 122 that they maybe wanted to work in computer science (66.67%), and 37 that they did not want to work in computer science (20.22%). The answers to question 1 (“Are you interested in computer science?”) correlate to the answers to question 4 (“In the future, do you want to work in computer science?”), showing that the students who classified themselves as being (very) interested in computer science could more often imagine working in computer science in the future. However, the correlation is low (rS = 0.30328674). Figure 1 provides an overview about the correlation showing that students who said that they were very interested in computer science answered that they could imagine to work in the field of computer science, to maybe work in the field of computer science or not to work in the field of computer science. Students who said that they were interested in computer science answered also that they could imagine to work in the field of computer science, to maybe work in the field of computer science or not to work in the field of computer science. Students who said that they were a little interested in computer science answered that they could imagine to maybe work in the field of computer science and not to work in the field of computer science, but none of them chose the answer that they wanted to work in the field of computer science. Students who said that they were not interested in computer science answered that they could imagine to maybe work in the field of computer science and not to work in the field of computer science, but none of them chose the answer that they wanted to work in the field of computer science.

136

S. Hinterplattner

Fig. 1. Students’ interest in computer science in relation to the aim to work in the computer science in the future (N = 179).

Looking at the answers in the different gender groups, differences appeared. The question about the interest in computer science was answered by 65 students who identified themselves as female (95.59%). Thirty-one of them responded that they were very interested in computer science (47.69%), 22 were interested in computer science (33.85%), nine were a little interested in computer science (13.85%), and three were not interested in computer science (4.62%). Of the 115 students who identified themselves as male and answered this question (97.46%), 74 expressed that they were very interested in computer science (64.34%), 31 were interested in computer science (26.96%), ten were a little interested in computer science (8.70%), and none was not interested in computer science. Figure 2 compares the answers by gender identity. In sum, differences in the answers could be observed. The distribution of the answers was found to be significantly different between the two gender groups. Students who identified themselves as male stated significantly more often that they were (very) interested in computer science than students who identified themselves as female.

Students’ Perceptions of Computer Science and the Role of Gender

137

Fig. 2. Comparison of students’ interest in computer science by gender identity (Nf =65, Nm = 115).

The question about pursuing a career in the field of computer science or not pursuing a career in the field of computer science was answered by 66 students who identified themselves as female (97.06%), eight stated that they wanted to work in computer science in the future (12.12%), 40 that they maybe wanted to work in computer science (60.61%), and 18 that they did not want to work in computer science (27.27%). Of the 115 students who identified themselves as male and answered this question (97.46%), 16 stated that they wanted to work in computer science in the future (13.91%), 80 that they maybe wanted to work in computer science (69.57%), and 19 that they did not want to work in computer science (16.52%). Figure 3 compares the answers by gender, showing no significant differences.

Fig. 3. Comparison of the number of students planning to work in computer science in the future in the gender groups female and male (Nf = 66, Nm = 115).

When looking at the correlation between the two questions in the gender groups, it can be seen that the correlation in the group of students who identified

138

S. Hinterplattner

themselves as female was higher (rS =0.361527503) than in the group of students who identified themselves as male (rS = 0.236364748). An overview can be seen in Fig. 4 showing the differences between the groups of gender identities.

Fig. 4. Students’ interest in computer science in relation to the aim to work in the computer science in the future by gender identity (Nf =65, Nm = 114).

Students’ Perceptions of Computer Science and the Role of Gender

6.2

139

Students’ Aim of Working in CS and Role Models in CS

To find out more about the relationships between the students’ aims of working in the field of computer science and knowing people working in the field of computer science, correlations were investigated. The results of students’ aims of working in the field of computer science were presented in detail in the Subsect. 6.1. The question if the students know someone working in the field of computer science, was possible to answer with a short passage. It was not mandatory to answer this question, however, all the students answered this question (N = 188). Of them, 93 computer science teachers form their schools were noted by 68 students (36.17%), while 64 mentioned that they did not know anyone working in computer science (34.04%). Altogether, 77 family members were mentioned by 48 students (25.53%) including mothers (24 mentions), fathers (23 mentions), uncles (15 mentions), aunts (four mentions), sisters, male cousins (three mentions each), brothers, female cousins (two mentions each), and grandfathers (one mention). Moreover, 14 students mentioned other people that could not be classified as family members or teachers (7.45%): five students mentioned general professions including bank clerk, computer science teacher, electrician, gamer, hacker, manager, modder, (medical) programmer, software developer, and teacher (2.66%) and one noted a famous coding influencer (0.53%). In sum, 119 students did not know anyone working in computer science except the computer science teachers at their schools (63.30%).

Fig. 5. Students’ aims to work in the computer science in the future in relation to knowing people working in the field of computer science (N = 183).

140

S. Hinterplattner

The answers to question 4 (“In the future, do you want to work in computer science?”) correlate to the answers to question 6 (“List the people you know that work in computer science”), showing that the students who know someone working in computer science could more often imagine working in computer science in the future. However, the correlation is very low (rS = 0.209058981). Figure 5 provides an overview about the correlation showing that all possible answer combinations were given. However, there were differences in the frequency. Most of the children who could imagine working in the field of computer science in the future knew someone working in the field of computer science. Most of the people that did not want to work in the field of computer science in the future or did not know it yet, did not know anyone working in the field of computer science. Looking at the answers in the different gender groups, differences appeared. The differences about the aim of working in the field of computer sciences in the gender groups were presented in Sect. 6.1. Answers to question 6 (“List the people you know that work in computer science”) were given by 68 students who identified themselves as female (100%). Of these 68 students 28 students (41.18%) noted 45 computer science teachers from their schools, 21 students (30.88%) mentioned 39 family members, 20 students (29.41%) stated that they did not know anyone working in computer science, six students (8.82%) noted other people that could not be classified as family members or teachers, and one mentioned general professions (1.47%). Of the 118 students who identified themselves as male and answered this question (100%), 43 students (36.44%) mentioned that they did not know anyone working in computer science, 40 students (33.90%) noted 48 computer science teachers from their schools, 27 students (29.66%) listed 38 family members, seven students (5.93%) mentioned other people that could not be classified as family members or teachers, four noted general professions (3.39%), and one mentioned a famous coding influencer (0.85%) (Fig. 6). In sum, 41 students who identified themselves as female did not know any person working in computer science except computer science teachers at their school (60.29%) compared with 78 students who identified themselves as male (66.10%).

Fig. 6. Comparison of the number of mentions of people working in computer science in the gender groups female and male (Nf = 64, Nm = 115).

Students’ Perceptions of Computer Science and the Role of Gender

141

When looking at the correlations between the two questions in the different gender groups, the correlation in the group of students who identified themselves as female was higher (rS =0.323873008) than in the group of students who iden-

Fig. 7. Students’ aims to work in the computer science in the future in relation to knowing people working in the field of computer science by gender identity (Nf = 65, Nm = 116).

142

S. Hinterplattner

tified themselves as male (rS = 0.15463094). An overview can be seen in Fig. 7 showing the differences between the groups of gender identities. 6.3

Students’ Interest in CS and Role Models in CS

To find out more about the relationships between the students’ interest in computer science and knowing people working in the field of computer science, correlations were investigated. The results of students’ interest in computer science were presented in detail in the Subsect. 6.1. The results of students’ knowing people working in the field of computer science were presented in detail in the Subsect. 6.2. The answers to question 1 (“Are you interested in computer science?”) correlate to the answers to question 6 (“List the people you know that work in computer science”), showing that the students who classified themselves as being (very) interested in computer science could list more often people working in the field of computer science (except their teachers). However, the correlation is very low (rS =0.096747352). Figure 8 provides an overview about the correlation showing that all possible answer combinations were given. Figure 8

Fig. 8. Students’ interest in computer science in relation to knowing people working in the field of computer science (N = 182).

When looking at the correlations between the two questions in the different gender groups, the correlation in the group of students who identified themselves

Students’ Perceptions of Computer Science and the Role of Gender

143

as female was higher (rS =0.163929481) than in the group of students who identified themselves as male (rS = 0.076769472). An overview can be seen in Fig. 9 showing the differences between the groups of gender identities.

Fig. 9. Students’ interest in computer science in relation to knowing people working in the field of computer science by gender identity (Nf =64, Nm = 115).

144

7

S. Hinterplattner

Summary

The aim of the study was to investigate students’ perceptions of computer science in Austrian secondary schools before the students experience computer science education. In detail, correlations and differences in the gender groups were explored. The results show that students had a very vague picture of what computer science was and what a computer scientist did before they experienced computer science education in school. Students who identified themselves as female, were less interested in computer science than students who identified themselves as male resulting in significantly different distributions. However, the differences about pursuing a career in the field of computer science were not found to be significantly different between the group of students who identified themselves as female and the group of students who identified themselves as male. The answers about interest in computer science and pursuing a career in the field of computer science showed a low correlation with students that classified themselves as being (very) interested in computer science could more often imagine working in computer science in the future. In the group of students who identified themselves as female, the correlation was higher than in the group of students who identified themselves as male. More than sixty-three percent of the students did not know any person working in the field of computer science other than their own computer science teachers. The answers about pursuing a career in the field of computer science and knowing people working in the field of computer science showed a very low correlation with students knowing someone working in computer science could more often imagine working in the field of computer science in the future. In the group of students who identified themselves as female, the correlation was higher than in the group of students who identified themselves as male. The answers about students’ interest in computer science and knowing people in computer science showed a very low correlation with students that classified themselves as being (very) interested in computer science could list more often people working in the field of computer science. In the group of students who identified themselves as female, the correlation was higher than in the group of students who identified themselves as male.

8

Discussion

The study corresponds to research results of students having a very vague idea about what computer science is and what computer scientists do [2,24,27,35]. Similarly to the findings of Greening (1998) where high school students had to define computer science tasks, the responses to computer scientist tasks were absent or could be classified as trivial or computer-centric [24]. This also agrees to research conducted by Grover et al. (2014) or Hansen et al. (2017) having vague tasks like “building”, “fixing”, “studying” and “improving computers”, or “working”, “coding”, “making”, “typing”, “doing”, “looking”, “fixing” and “testing’ [25,27]. There is a lack of clear understanding of computer science and the tasks of computer scientists as shown by Martin (2004) [35].

Students’ Perceptions of Computer Science and the Role of Gender

145

However, a lack of interest in computer science could not be observed like in the research conducted by Yardi and Bruckman (2007) [51]. Even that the students were missing a clear understanding what computer science is, most of the students were very interested in the field of computer science and more than three quarter being very interested or interested. The findings of this study could not support research that showed that the lack of understanding what computer science is leading to disinterest in computer science [8] or research that students do not see themselves pursuing careers in the field of computer science [51]. However, the career goals were still very vague for the students what could be due to the young age of the participants. The importance of role models shown by researchers [48,51] could be seen in a very low correlation, but for most students there was a lack of role models. Furthermore, this might be a reason for the vague picture of computer science and computer scientists.

References 1. Aronin, S., Floyd, K.K.: Using an Ipad in inclusive preschool classrooms to introduce stem concepts. Teach. Except. Children. 45, 34–39 (2013). https://doi.org/ 10.1177/004005991304500404 2. Beaubouef, T., McDowell, P.: Computer science: Student myths and misconceptions. J. Comput. Sci. Colleges. 23(6), 43–48 (2008) 3. Beilock, S.L., Gunderson, E.A., Ramirez, G., Levine, S.C.: Female teachers’ math anxiety affects girls’ math achievement. Proc. Natl. Acad. Sci. USA. 107, 1860– 1863 (2010). https://doi.org/10.1073/PNAS.0910967107 4. Bollin, A., Kesselbacher, M., Mößlacher, C.: Ready for computing science? A closer look at personality, interests and self-concept of girls and boys at secondary level. In: Kori, K., Laanpere, M. (eds.) ISSEP 2020. LNCS, vol. 12518, pp. 107–118. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63212-0_9 5. Brandhofer, G.: Ein Gegenstand ’ ’Digitale Medienbildung und Informatik “notwendige Bedingung für digitale Kompetenz? Open Online J. Res. Educ. 1, 109–119 (2014) 6. Brenneman, K., Stevenson-Boyd, J., Frede, E.C.: Mathematics and science in preschool: Policy and practice (2009) 7. Buchter, J., Kucskar, M., Oh-Young, C., Welgarz-Ward, J., Gelfer, J.: Supporting stem in early childhood education (2017). https://digitalscholarship.unlv.edu/co_ educ_policy 8. Carter, L.: Why students with an apparent aptitude for computer science don’t choose to major in computer science, pp. 27–31. Association for Computing Machinery (2006). https://doi.org/10.1145/1121341.1121352 9. Cassel, L.B., McGettrick, A., Guzdial, M., Roberts, E.: The current crisis in computing: what are the real issues?, pp. 329–330. Association for Computing Machinery (2007). https://doi.org/10.1145/1227310.1227426 10. Cedefop: Skills, qualifications and jobs in the EU : the making of a perfect match? : evidence from Cedefop’s European skills and jobs survey. Publications Office (2015). https://doi.org/10.2801/606129 11. Chambers, D.W.: Stereotypic images of the scientist: the draw-a-scientist test. Sci. Educ. 67, 255–265 (1983). https://doi.org/10.1002/sce.3730670213

146

S. Hinterplattner

12. Cheryan, S., Ziegler, S.A., Montoya, A.K., Jiang, L.: Why are some stem fields more gender balanced than others? Psychol. Bull. 143, 1–35 (1 2017). https://doi. org/10.1037/BUL0000052 13. Clements, D.H.: Learning and Teaching Early Math : The Learning Trajectories Approach (2nd edn.) (2016) 14. Crowley, K., Callanan, M.A., Tenenbaum, H.R., Allen, E.: Parents explain more often to boys than to girls during shared scientific thinking. Psychol. Sci. 12, 258– 261 (2001). https://doi.org/10.1111/1467-9280.00347 15. Duncan, G.J., et al.: School readiness and later achievement. Psychol. Assoc. 43, 1428–1446 (2007). https://doi.org/10.1037/0012-1649.43.6.1428 16. Duschl, R.A., Schweingruber, H.A., Shouse, A.W.: Taking Science to School: Learning and Teaching Science in Grades K-8. The National Academies Press (2007) 17. Dynatrace Austria GmbH: What is Dynatrace? (2021). https://www.dynatrace. com/support/help/get-started/what-is-dynatrace/. Accessed 09 Nov 2021 18. European Commission: Europe needs more scientists! (2004). https://ec.europa. eu/commission/presscorner/detail/en/IP_04_444 19. European Commission: Human capital and digital skills in the digital economy and society index (2021). https://digital-strategy.ec.europa.eu/en/policies/desihuman-capital 20. Federal Ministry of Education, Science and Research: Verbindliche Übung Digitale Grundbildung - Umsetzung am Schulstandort (2018). https://www.bmbwf.gv.at/ Themen/schule/zrp/dibi/dgb.html 21. Fisher, A., Margolis, J.: Unlocking the clubhouse: the Carnegie Mellon experience. SIGCSE Bull. 34, 79–83 (2002). https://doi.org/10.1145/543812.543836 22. Fleer, M., March, S., Gunstone, D.: Investigations into the engagement of preschool and primary aged children in science, engineering and technology (2006) 23. Goodenough, F.L.: Measurement of Intelligence By Drawings. Harcourt Brace (1926). https://archive.org/details/in.ernet.dli.2015.268362 24. Greening, T.: Computer science: through the eyes of potential students (1998) 25. Grover, S., Pea, R., Cooper, S.: Remedying misperceptions of computer science among middle school students. In: Proceedings of the 45th ACM Technical Symposium on Computer Science Education, pp. 343–348. SIGCSE 2014, Association for Computing Machinery, New York, NY, USA (2014). https://doi.org/10.1145/ 2538862.2538934 26. Gunderson, E.A., Ramirez, G., Levine, S.C., Beilock, S.L.: The role of parents and teachers in the development of gender-related math attitudes. Sex Roles. 66, 153–166 (2012). https://doi.org/10.1007/S11199-011-9996-2 27. Hansen, A.K., et al.: Assessing children’s understanding of the work of computer scientists: The draw-a-computer-scientist test. In: Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, pp. 279–284. Association for Computing Machinery, New York, NY, USA (2017). https://doi. org/10.1145/3017680.3017769 28. Hassan, M.N., Abdullah, A.H., Ismail, N., Suhud, S.N.A., Hamzah, M.H.: Mathematics curriculum framework for early childhood education based on science, technology, engineering and mathematics (stem). Int. Elect. J. Math. Educ. 14, 15–31 (2019) 29. Hinterplattner, S., Rottenhofer, M., Groher, I., Sabitzer, B.: Exploring students’ experiences and perceptions of computer science: a survey of Austrian secondary schools. Int. Conf. Comput. Support. Educ. CSEDU - Proc. 2, 238–247 (2022). https://doi.org/10.5220/0011049000003182

Students’ Perceptions of Computer Science and the Role of Gender

147

30. Hunting, R.P., Mousley, J.A., Perry, B.: A study of rural preschool practitioners’ views on young children’s mathematical thinking. Math. Educ. Res. J. 24(1), 39–57 (2012). https://doi.org/10.1007/S13394-011-0030-3 31. Jacobs, J.E., Bleeker, M.M.: Girls’ and boys’ developing interests in math and science: do parents matter? New directions for child and adolescent development, pp. 5–21 (2004). https://doi.org/10.1002/CD.113 32. Lazowska, E.: Pale and male: 19th century design in a 21st century world. ACM SIGCSE Bull. 34, 11–12 (2002) 33. Maltese, A.V., Tai, R.H.: Eyeballs in the fridge: sources of early interest in science. Int. J. Sci. Educ. 32(5), 669–685 (2010) 34. Margolis, J., Fisher, A.: Unlocking the Clubhouse: Women in Computing. The MIT Press, Cambridge (2002) 35. Martin, C.D.: Draw a computer scientist. In: Working Group Reports from ITiCSE on Innovation and Technology in Computer Science Education, pp. 11–12. ITiCSEWGR 2004, Association for Computing Machinery, New York, NY, USA (2004). https://doi.org/10.1145/1044550.1041628 36. McClure, E., Guernsey, L., Clements, D.H., Bales, S.N., Nichols, J., Kendall-Taylor, N., Levine, M.H.: Stem starts early: Grounding science, technology, engineering, and math education in early childhood (2017) 37. Mercier, E.M., Barron, B., O’Connor, K.M.: Images of self and others as computer users: the role of gender and experience. J. Comput. Assist. Learn. 22(5), 335–348 (2006). https://doi.org/10.1111/j.1365-2729.2006.00182.x 38. Mitchell, A., Purchase, H.C., Hamer, J.: Computing science: What do pupils think? In: Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education, p. 353. ITiCSE 2009, Association for Computing Machinery, New York, NY, USA (2009). https://doi.org/10.1145/ 1562877.1562995 39. Osborne, J., Dillon, J.: Science education in Europe: Critical reflections, vol. 13. The Nuffield Foundation, London (2008) 40. Patrick, H., Mantzicopoulos, P., Samarapungavan, A.: Motivation for learning science in kindergarten: Is there a gender gap and does integrated inquiry and literacy instruction make a difference. J. Res. Sci. Teach. 46, 166–191 (2009). https://doi. org/10.1002/TEA.20276 41. Pfleeger, S.L., Teller, P., Castaneda, S.E., Wilson, M., Lindley, R.: Increasing the enrollment of women in computer science, pp. 386–387. Association for Computing Machinery (2001). https://doi.org/10.1145/364447.364752 42. Rasmussen, B., Håpnes, T.: Gendering technology: young girls negotiating ICT and gender. In: Lie, M., Corneliussen, H. (eds.) He, She and It Revisited: New Perspectives on Gender in the Information Society. Gyldendal (2003) 43. Reiter, Anton: Incorporation of informatics in Austrian education: the project “Computer-Education-Society” in the school year 1984/85. In: Mittermeir, Roland T.. (ed.) ISSEP 2005. LNCS, vol. 3422, pp. 4–19. Springer, Heidelberg (2005). https://doi.org/10.1007/978-3-540-31958-0_2 44. Russell, S.H., Hancock, M.P., McCullough, J.: Benefits of undergraduate research experiences (2007) 45. Schott, G., Selwyn, N.: Examining the “male, antisocial” stereotype of high computer users. J. Educ. Comput. Res. 23, 291–303 (2000). https://doi.org/10.2190/ V98R-5ETX-W9LY-WD3J 46. Schreiner, C., Sjøberg, S.: Science education and youth’s identity construction two incompatible projects? pp. 231–247. Brill|Sense (2019). https://doi.org/10. 1163/9789087901677_020

148

S. Hinterplattner

47. Siller, H.S., Fuchs, K.: Bemerkungen zur Fachdidaktik Informatik. In: 25 Jahre Schulinformatik in Österreich-Zukunft mit Herkunft, pp. 121–125. books@ ocg.at (2010) 48. Stout, J.G., Dasgupta, N., Hunsinger, M., McManus, M.A.: Steming the tide: using ingroup experts to inoculate women’s self-concept in science, technology, engineering, and mathematics (STEM). J. Personal. Soc. Psychol. 100, 255–270 (2011). https://doi.org/10.1037/A0021385 49. Swift, T.M., Watkins, S.E.: An engineering primer for outreach to k-4 education (2004) 50. Tai, R.H., Liu, C.Q., Maltese, A.V., Fan, X.: Planning early for careers in science. Science 312, 1143–1144 (2006). https://doi.org/10.1126/SCIENCE.1128690 51. Yardi, S., Bruckman, A.: What is computing? bridging the gap between teenagers’ perceptions and graduate students’ experiences. In: Proceedings of the Third International Workshop on Computing Education Research, pp. 39–50. ICER ’07, Association for Computing Machinery, New York, NY, USA (2007). https://doi.org/ 10.1145/1288580.1288586

Adaptive Kevin: A Multipurpose AI Assistant for Higher Education Augusto Gonzalez-Bonorino1(B) and Eitel J. M. Lauría2(B) 1

Claremont Graduate University, Claremont, CA 91711, USA [email protected] 2 Marist College, Poughkeepsie, NY 12601, USA [email protected]

Abstract. We present an innovative open domain question-answering (ODQA) intelligent workflow architecture that combines a fine-tuned retriever-reader model with a generative question-answering model with Google search functionality capable of answering questions both within and outside the scope of the domain represented by the documents indexed by the retriever. The retrieverreader architecture acts as the default responder, but if the question is outside of its scope or if its confidence score falls below a predefined threshold, the workflow logic triggers the execution of the generative model that reaches out to Google to formulate an answer. The paper describes the proposed workflow architecture, provides a quantitative evaluation of the retriever-reader performance and demonstrates the system’s flow of execution through several use cases. The paper will be of interest to both researchers and practitioners interested in deploying modern AI-driven ODQA systems. Keywords: Deep learning · Natural language processing · Transformers · AI in higher-education · Open domain question-answering · ELECTRA · Conversational AI · Adaptive architectures · Information retrieval

1 Introduction The discipline of Natural Language Processing (NLP) and statistical learning in general is expanding rapidly. This rapidly progressing technology often redefines what “the state-of-the-art” means in this specific field, in general. While this is great for organizations with the infrastructure required, it can lead to a technology inequality gap. Higher education institutions are among the industries set to benefit the most from adopting these new technologies. Unfortunately, the inherent volatility of the field presents a real challenge for higher-education administrators to implement compliant and flexible AI-driven NLP solutions that can be easily updated as the field advances. Questionanswering (QA) is one of the topics in the natural language processing domain that has gained a considerable amount of attention given its potential to automate tasks without human intervention. Intelligent QA architectures can be applied to improve upon many inefficiencies shared by higher education institutions. Automating the provision Supported by Marist College. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2023  J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 149–159, 2023. https://doi.org/10.1007/978-3-031-40501-3_7

150

A. Gonzalez-Bonorino and E. J. M. Lauría

of information to prospective families via interaction with an intelligent chatbot can help reduce the human capital allocated to handle informative questions about the institution. In addition to administrative tasks, question-answering models can help guide students through the registration process, looking up classes in the courses catalog, and other common tasks for which information can be publicly accessed via the university’s website. But as in all aspects of NLP, the field of question-answering is advancing at recordbreaking speed and it is difficult for industries such as higher education to keep up with the latest technologies. In previous work [1] we presented an end-to-end methodology to guide educational institutions in deploying and evaluating open domain questionanswering (ODQA) architectures1 . In this paper, we propose a new architecture that combines Google search with a pre-trained extractive question-answering model finetuned on custom data. We extend the retriever-reader architecture with a neural search that combines a sparse retriever and a sentence encoder to optimize for both precision and recall. Moreover, we include a decision node that computes a similarity score between the input text query and the custom dataset to identify if there exists a similar question in the knowledge base. If the model was not trained on a similar question, then a generative question-answering model with Google search capabilities is triggered instead of the fine-tuned model. This innovation has two key implications that we will explore in detail in Sect. 3. The paper is organized into the following six sections: the current introduction (Sect. 1) is followed by an outline of the extant literature in the field of questionanswering (Sect. 2). Section 3 depicts in detail the design of the proposed architecture and its implications. Section 4 follows with a description of the experiments performed and their results. Next, Sect. 5 discusses the outcomes of the experiments in connection as a measure of the validity and usefulness of the proposed architecture. Finally, Sect. 6 provides concluding thoughts and a discussion of potential future lines of research.

2 Literature Review It is difficult to include a detailed literature review of open-domain question-answering systems, given the broad set of technologies involved and the fast pace at which some of these technologies have evolved in the last five years. So we have organized this section into several topics that cover the most relevant aspects related to ODQA implementations, highlighting the present research and development in each of them. Question-answering, the subfield of NLP concerned with studying the capabilities of computers to understand and answer questions queried in natural language, is divided into two main approaches: open domain and closed domain. The latter, also known as knowledge-based question answering, was the first paradigm to be developed. It focuses on collecting structured data, often curated by domain experts, and formulating queries on the structured database. One of the first examples of this approach is BASEBALL [2], a question-answering system trained on baseball statistics developed in 1961. This was the preferred approach for question-answering systems until the early 80s, but as 1

The initial research project was coded Kevin. Hence the name in the title of this paper.

Adaptive Kevin: A Multipurpose AI Assistant for Higher Education

151

applications grew in scale the cost and time required to develop high-quality training datasets motivated researchers to explore alternative methods. During that time, advanced statistical methods to process large unstructured databases were developed that made open-domain QA a plausible solution to the mentioned challenges of knowledge-based techniques. Traditional approaches for ODQA consisted of sequence-to-sequence models that leveraged recurrent and convolutional neural networks to infer meaning out of question-document pairs. Research in machine comprehension started with one-directional models such as Long-short-Term-Memory (LSTM) [3, 4] models, then followed by bi-directional architectures [5]. Nevertheless, the inherent sequential nature of these recurrent neural networks limited parallelization within training examples. To overcome this, Vaswani et al. proposed the transformer architecture [6], propelling a new paradigm of question-answering systems based on self-attention layers and highly parallelizable matrix computations that enhanced performance. This improved performance is commonly complemented by a retriever component, preceding the reading comprehension stage, that encodes the corpus of documents in vector representations that can be filtered using similarity with the input text query. Thus, only the most relevant documents are fed to the reader effectively reducing run-time and increasing prediction accuracy. It is this combination of both components in the so-called retriever-reader architecture that has become the dominant paradigm for ODQA. Short after the inception of the transformer architecture, Google’s bidirectional transformer model BERT [7], which also introduced the concept of pretraining, and the public release of the Stanford Question Answering Dataset -SQuAD2 [8], developed specifically for training question-answering models, skyrocketed the resources allocated to the discipline. Nevertheless, the computing power needed to train and run it limited access to this technology. Consequently, the focus shifted from model development to training methods. Transfer learning, a technique developed for computer vision in 2014 [9], allowed to recycle most of the weights learned via training by focusing on general tasks that could, later on, be narrowed down to domain-specific tasks. By solely updating the last few layers of the transformer, a massive model can be repurposed (i.e., fine-tuned) on a variety of tasks such as answering questions about your institution. The architectures reviewed so far have one important characteristic in common, they are discriminative models. Thus, they perform well in understanding the superficial meaning of the input text by exploiting correlations in the data. Nevertheless, this particular feature makes discriminative models particularly sensitive to overfitting if an input question is similar enough to examples the model saw during training. Consequently, incorrect answers may be inferred with high confidence scores. Recently, this limitation gave way to a new area of research that explores generative models for question-answering. Radford et al. [18] were among the first to test Generative PreTraining (GPT), combined with discriminative fine-tuning, for natural language understanding. In 2019, Lewis and Fan [17] proposed a fully generative architecture trained to explain the whole question, instead of just extracting an answer span, using a Bayesian learning approach. Currently, this architecture design has defined the state-of-the-art for question-answering tasks given their capacity to explain and generalize to unseen questions. A word of caution is in order here: generative models like GPT3 [19] and

152

A. Gonzalez-Bonorino and E. J. M. Lauría

Galactica [20], that due to the large corpus of training have been named Large Language Models (LLMs), have recently shown incredible potential in generating original answers to complicated questions. But, they have also demonstrated the importance of human expertise in interpreting the answers. The cutting edge of natural language processing poses interesting ethical challenges in AI explainability and AI-driven biases that should be addressed with tenacious, yet gracious, constructive criticism. In the last decade, the implementation of complex machine learning models trained with large amounts of data has been aided through the use of end-to-end data pipelines that perform the data extraction and transformation (pre-processing) of the data, followed by model training, tuning, testing and final materialization/persistence of the models, in preparation for deployment in a production setting. A number of frameworks have been developed for this purpose. The well-known scikit-learn library [10] provides an API that has been emulated by other machine learning, big data and deep learning frameworks, such as Spark [11], Dask [12], Tensorflow [24] and Pytorch [25], among other software platforms. Hugingface [13] introduced the pipeline abstraction to simplify the process of training models for inference in perceptual problems such as computer vision, natural language, audio processing, and multimodal tasks [14]. In the context of open domain question-answering, Haystack [1, 15], and Cherche [16] provide pipeline objects to simplify information retrieval design. We believe that the use of flexible data pipelines, which have become critical in streamlining the process of taking machine learning models to production -generally referred to as machine learning operations, or MLOps- can also help practitioners develop better and more versatile ODQA systems. Workflows can be developed that combine the best of both discriminative and generative architectures while reducing the risk of incorrect responses as well as deployment and maintenance costs. The workflow proposed in this paper illustrates the validity of this hypothesis in a college setting.

3 Architecture Design This section includes a thorough explanation of the design and motivation underlying the workflow proposed in this paper. The execution flow is determined by two decision nodes. When an input text query is matched with the training knowledge base, a similarity score is computed that determines whether an ODQA generative model or a fine-tuned extractive model will be used to answer the question depending on a predetermined threshold (a workflow design parameter set in advance). If the model has not been trained on a similar question, the generative model will be used to generate original answers with a higher probability of correctly answering the unseen question at a small runtime cost -exploiting the benefits of generative readers described in the literature review section. If the similarity score is above the threshold then the logic embedded in the workflow considers the fine-tuned model as potentially valid QA candidate platform to provide a concise answer at a faster response time. After a response is extracted from the knowledge base, a second decision rule implemented in the workflow assesses the validity of the response based on its confidence score a (a second parameter set in advance as part of the workflow design). If the confidence score is above the threshold, the original extracted answer will be returned. Otherwise, a distilled generative model

Adaptive Kevin: A Multipurpose AI Assistant for Higher Education

153

powered with Google search browses the web and returns a short generated answer as well as the source the answer is based on (this also under the assumption that the information needed to answer the question can be publicly available on the internet). This additional step minimizes the risk of returning an incorrect answer to the user or no answer at all. In parallel, the question that produced a low-confidence answer with the fine-tuned extractive architecture is automatically saved on the knowledge base, paired with the final response returned to the user, to facilitate continuous training and model maintenance. The workflow is depicted in Fig. 1 below. The computational framework implemented in each stage are depicted in Table 1. The remainder of the section illustrates the technology components used at each step: In step 1, the program is initialized and the input query fed to the neural search pipeline that computes the similarity score. To construct and implement the neural search pipeline we made use of Cherche’s pipeline and its union operator. First, the pipeline operator allows us to seamlessly index the corpus of training documents and combine information retrieval algorithms efficiently. Second, the union operator allows us to design a neural search capable of gathering documents retrieved by multiple models while avoiding duplicates. Hence, it is able to prioritize one model or pipeline over another. Our original design follows the recommended guidelines and employs two pipelines: one composed of a BM25 [21] retriever and sentence encoder ranker to optimize precision, and a second one composed solely of a sentence encoder to optimize recall. For the default implementation, we opted for using MPNet base v2 [23]. After the input query is fed to the neural search, it cross-checks it with the previously indexed knowledge base and returns a similarity score to be used in the decision rule. Once the similarity score has been computed, the workflow can take one of two possible paths. We denote step 1a as the instance in which the score is below the threshold and step 1b otherwise. In step 1a, a Huggingface conversational pipeline is instantiated with a multipurpose generative model capable of answering open-ended questions and holding natural conversations with the user. We implement Meta’s distilled version of Blenderbot [22] with only 400M parameters to reduce overall runtime and provide original answers to queries we do not have well structured data for. In step 1b, the fine-tuned extractive model is instantiated and called to predict the span of text most likely to contain the correct answer. The likelihood is expressed as a confidence score, which we use in the following step to determine the next path to take. In step 2, the fine-tuned model prediction from step 1b is fed to a simple logical decision rule that assesses if the confidence is higher or lower than a predetermined threshold. In a college setting, we want to minimize incorrect responses to avoid losing students’ trust and interest in the technology. Hence, we have set a particularly high default threshold of 70% confidence score. We denote step 2a as the instance in which the score is below the threshold and step 2b otherwise. In step 2a, two important actions are taken. First, the question is saved in the knowledge base for continuous training. Second, we employ a combination of a distilled generative model powered with Google search to return an answer as well as the source article the model relied on to generate it. This is an important feature of the architecture because 1) it further reduces the chance of returning incorrect information about the institution; and 2) it prioritizes

154

A. Gonzalez-Bonorino and E. J. M. Lauría

Fig. 1. Proposed Execution Workflow.

transparency and explainability, two qualities of uttermost importance to prevent the erroneous belief of intentional misinformation. In the background, the Google powered question-answering architecture of step 2a calls SerpApi’s (https://github.com/serpapi) to run a Google search for the input query using a secret key and feeds the most relevant document returned by Google to a light generative model that yields a brief answer based on that source document. In step 2b, the workflow logic is fairly confident that the extracted span represents the correct answer and thus it proceeds to return that answer and finalize the execution.

Adaptive Kevin: A Multipurpose AI Assistant for Higher Education

155

Table 1. Computational Framework used for benchmarking the workflow. Workflow Component Computational Framework Ranker

MPNET Base v2

Retriever

BM25L

Similarity Analysis

Cherche: Neural Search

Extractive QA

ELECTRA Large fine-tuned

Conversational

blenderbot 400M distill

Google powered QA

BART + SerpAPI

Index

FAISS Exact Match L2

4 Quantitative and Qualitative Assessment The goal of the proposed open-domain question-answering workflow for highereducation institutions is to provide high-quality, domain-specific answers to a wide range of questions. In order to demonstrate the effectiveness of this workflow, as well as its individual components, we consider two experiments. First, the fine-tuning of the extractive reader comprehension model using the Huggingface Evaluator API. Second, a qualitative assessment of the entire workflow to gauge its overall performance and trace the three possible scenarios to ensure the decision rules work effectively. 4.1 Quantitative Assessment Fine-tuning a custom question-answering model with Huggingface’s can be easily achieved with its Evaluator class to compute the performance of a model on a specific dataset using a specified metric, or set of metrics. In our case, an instance of the Evaluator class is created and initialized with the “question-answering” task. This instance is then used to evaluate the kevin_pipe model on the provided data using the SQUAD metric, with a bootstrap resampling strategy and 30 resamples. The results of this evaluation process are stored in the kevin_eval_results variable. The Evaluator class is designed to work with transformer pipelines out-of-the-box, but it also supports custom pipelines, allowing for a wide range of flexibility in model evaluation. The following snippet of code illustrates this process. t a s k _ e v a l u a t o r = e v a l u a t o r ( ‘ ‘ q u e s t i o n −a n s w e r i n g ’ ’ ) k e v i n _ e v a l _ r e s u l t s = t a s k _ e v a l u a t o r . compute ( model_or_pipeline=kevin_pipe , data =data , m e t r i c = ‘ ‘ squad ’ ’ , strategy =‘‘ bootstrap ’ ’ , n _ r e s a m p l e s =30 )

156

A. Gonzalez-Bonorino and E. J. M. Lauría Table 2. Quantitative assessment results Metric

Score

Confidence Interval Standard Error

Exact Match

87.1

[84.9, 89.18]

1.22

F1 Score

92.94

[91.48, 94.02]

0.77

Total Time (seconds) 2442.62 -

-

Samples/Second

0.41

-

-

Latency (seconds)

2.44

-

-

Table 2 contains evaluation metrics for the proposed question-answering pipeline. The pipeline was assessed using two metrics: exact match and F1 score. The exact match score was 87.1, with a confidence interval ranging from 84.9 to 89.18 and a standard error of 1.22. The F1 score was 92.94, with a confidence interval ranging from 91.48 to 94.02 and a standard error of 0.77. The total time taken to evaluate the pipeline was 2442.62 s, and the average latency for generating each answer was 2.44 s. The pipeline processed 0.41 samples per second on average. By utilizing the L2 method of dense embedding with FAISS Exact Search on a Nvidia Tesla T4 GPU with CUDA 11.2 from Google Collaboratory’s free tier, we found that the average indexing time for all 695 documents was 27 s. The average runtime for the conversational, fine-tuned response, and Google-powered question-answering models was 9.46, 4.85, and 25.57 s, respectively. The fine-tuned response model showed a noteworthy improvement in runtime compared to the other models, highlighting the importance of clean data for increasing the response confidence score for a wider range of student inquiries. These results suggest that the proposed workflow using freely available and open-source resources can achieve decent performance at a very low cost 4.2

Qualitative Assessment

We assessed the proposed workflow using a set of sample queries. For each query, we recorded the confidence scores and final answer output by the workflow. We then compared the answers to the ground truth and assessed the overall performance based on the number of correct answers, response time, and answer quality. These results provide insight into the workflow’s effectiveness and identify areas for improvement. Each possible use case is illustrated next. The chosen questions present different scenarios and trace each of the branches of the workflow. Our own knowledge of Marist College allows us to evaluate the quality of the model’s responses. Scenario 1: The conversational component of the question-answering pipeline is activated when the input query is not similar enough to a question the extractive model has been fine-tuned on. This use case is well suited for handling queries that are not related to the higher-ed institution (Marist College, in our case), or that are not in the form of a question that the extractive model has been trained on. For example, a sample query to test this use case might be: “What is the weather in Chicago today?” Since this query is not related to Marist College and is not in the form of a question that the extractive model has been trained on, it is well suited to trigger the conversational component of the pipeline. The output is

Adaptive Kevin: A Multipurpose AI Assistant for Higher Education

157

Conversation id: c6cd0079-e7e9-4c5f-8e62-5c24a53ca5a1 user » What is the weather in Chicago today? bot » It is a little chilly, but not too bad. How about in your neck of the woods? Scenario 2: The fine-tuned extractive model is used when the input query is similar enough to a question the model has been trained on. This use case is well suited for handling queries that are related to Marist College and are in the form of a question that the extractive model has been trained on. For example, a sample query to test this use case might be: “Where is Marist College located?” Since this query is related to Marist College and is in the form of a question that the extractive model has been trained on, it is well suited to trigger the fine-tuned extractive model. The output is: on the banks of the historic Hudson River and at its Florence, Italy campus Scenario 3: If the confidence score of the extracted answer from the fine-tuned extractive model is not high enough, the question-answering pipeline falls back on a google search powered generative model. This use case is well suited for handling queries that the fine-tuned extractive model is unable to answer with a high enough confidence score. For example, a sample query to test this use case might be: “How much would it cost me to attend Marist?” Since this query may be difficult for the fine-tuned extractive model to answer with a high enough confidence score, but it is information publicly available, it is well-suited to be answered by the google search powered questionanswering system. The output is The annual list price to attend Marist College on a full-time basis for 2020/2021 is $61,795 for all students regardless of their residency. Source: https://www.collegesimply.com/colleges/new-york/marist-college/price/ 4.3 Discussion The goal of this paper is to present a novel workflow that effectively combines several different components to answer queries efficiently and effectively. While we have carefully chosen the components and their respective models, our goal is not to identify the best combination of components, but rather to demonstrate the feasibility and effectiveness of this workflow as a proof of concept. The proposed workflow in this paper is designed to be resilient, reducing the likelihood of the model failing to respond. There are some data quality considerations though that must be addressed in the implementation process: – Data pre-processing in the form of document cleaning is a critical task to ensure the workflow is able to deliver precise answers to the questions formulated. The indexed documents must be thoroughly cleaned and all special characters (e.g. html tags, urls) removed in order for the extractive architecture to be able to return good quality answers. The lack of “clean” text affects the confidence score of the extractive architecture which pushes the logic of the workflow to opt for the Google-powered generative model although a more precise answer may be present in the document store. – The extractive architecture requires a substantive number of question-answer pairs to fine-tune the model. Although there is no absolute guideline as this varies by

158

A. Gonzalez-Bonorino and E. J. M. Lauría

domain, the literature recommends over two thousand pairs for the fine-tuning process to significantly boost the performance of the extractive architecture. This adds some burden to the implementation process, as question-answers pairs cannot be automatically generated, they require human intervention.

5 Conclusion and Future Work In this paper, we have presented our proposed ranker-retriever-reader workflow for question-answering, which exploits both discriminative and generative language models to minimize the likelihood of returning incorrect responses as well as the cost of implementation. We have demonstrated the potential benefits of this technology that fit the needs and resources of higher-education institutions, including more efficient human resource allocation, and reduced burden of administrative tasks. Looking toward the future, there are several directions in which this work can be extended. For example, other use cases can be considered for this technology, such as assisting students with their coursework or providing personalized recommendations. We have also considered investigating the use of other language models or techniques to improve the performance of the system. Furthermore, we would like to study the impact of our proposed workflow on the overall satisfaction of students and faculty. In conclusion, our proposed workflow offers a promising solution for improving the operations of higher education institutions. By leveraging the power of language models, it can provide valuable support to students, faculty and administrators, and help institutions better serve their communities. We believe that this approach offers a promising new direction for question-answering systems in the context of higher education, and we hope that our work will inspire further research in this area. Acknowledgements. The authors would like to thank Ed Presutti and the whole Data Science and Analytics Dept. at Marist College for their collaboration throughout this research project. A special mention goes to Divya Aavula and Pardhasree Chaitanya Tatini who worked on extending the set of questions and answers used to fine-tune the models.

References 1. Gonzalez-Bonorino, A., Lauría, E., Presutti, E.: Implementing open-domain questionanswering in a college setting: an end-to-end methodology and a preliminary exploration. In: Proceedings of the 14th International Conference on Computer Supported Education Volume 2: CSEDU, ISBN 978-989-758-562-3; ISSN 2184-5026, pp. 66–75 (2022). https:// doi.org/10.5220/0011059000003182 2. Green, B. F., Wolf, A. K., Chomsky, C. L., Laughery, K.: Baseball: an automatic questionanswerer. In: IRE-AIEE-ACM 1961 (Western) (1961) 3. Wang, S., Jiang, J.: Learning Natural Language Inference with LSTM (2015). arXiv:1512.08849 4. Wang, S., Jiang, J.: Machine Comprehension Using Match-LSTM and Answer Pointer (2016). arXiv:1608.07905 5. Seo, M., Kembhavi, A., Farhadi, A., Hajishirzi, H.: Bidirectional Attention Flow for Machine Comprehension (2016). arXiv:1611.01603

Adaptive Kevin: A Multipurpose AI Assistant for Higher Education

159

6. Vaswani, A., et al.: Attention Is All You Need (2017). eprint: 1706.03762 7. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019). eprint:1810.04805 8. Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for SQuAD. In: Annual Meeting of the Association for Computational Linguistics (2018) 9. Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2 (NIPS 2014), pp. 3320–3328. MIT Press, Cambridge, MA, USA (2014) 10. Pedregosa, F., et al.: Scikit-learn Machine Learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (011) 11. Zaharia, M.A., Xin, R., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M.J., Ghodsi, A., Gonzalez, J.E., Shenker, S., Stoica, I.: Apache Spark: a unified engine for big data processing. Commun. ACM 59, 56–65 (2016) 12. Rocklin, M.: Dask: Parallel Computation with Blocked algorithms and Task Scheduling (2015) 13. Wolf, T., et al.: HuggingFace’s Transformers: State-of-the-art Natural Language Processing (2019). arXiv:1910.03771 14. Transformers. (n.d.). Huggingface.co. https://huggingface.co/docs/transformers/main. Accessed 8 Dec 2022 15. Pietsch, S., Chan, M.K.: Haystack (version 1.11.0) (2021). https://github.com/deepset-ai/ haystack/ 16. Sourty, R., Moreno, J.G., Tamine, L., Servant, F.: CHERCHE: a new tool to rapidly implement pipelines in information retrieval. In: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval 17. Lewis, M., Fan, A.: Generative question answering: learning to answer the whole question. In: International Conference on Learning Representations (2018) 18. Radford, A., Narasimhan, K.: Improving Language Understanding by Generative PreTraining (2018) 19. Brown, T.B., et al.: Language Models are Few-Shot Learners (2020). arXiv:abs/2005.14165 20. Taylor, R., et al.: Galactica: A Large Language Model for Science (2022). arXiv:abs/2211.09085 21. Robertson, S.E., Zaragoza, H.: The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr. 3, 333–389 (2009) 22. Roller, S., et al.: Recipes for building an open-domain chatbot. In: Conference of the European Chapter of the Association for Computational Linguistics (2020) 23. Song, K., Tan, X., Qin, T., Lu, J., Liu, T.: MPNet: Masked and Permuted Pre-training for Language Understanding (2020). arXiv:abs/2004.09297 24. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., et al.: Tensorflow: a system for large-scale machine learning. In: 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2016), pp. 265–83 (2016) 25. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems 32 [Internet], pp. 8024–8035. Curran Associates, Inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-highperformance-deep-learning-library.pdf

Comparing Multi-objective GA and PSO for the Pedagogical Activities Sequencing from Bloom’s Digital Taxonomy Denis José Almeida1 , Newarney Torrezão da Costa2 , and Márcia Aparecida Fernandes1(B) 1

Computing Department, Federal University of Uberlândia, Uberlândia, Brazil [email protected], [email protected] 2 Instituto Federal Goiano, Campus Iporá, Iporá, Brazil [email protected]

Abstract. Sequencing of pedagogical actions consists of determining action sequences or learning paths for improving or developing the student’s abilities. As the sequence quality is a crucial measure to evaluate the sequencer, the sequencing of pedagogical actions is an optimization problem, and techniques such as the metaheuristics from computational intelligence are suitable for coping with it. This paper formulates the sequencing problem as a multiobjective optimization problem, where the sequences contain actions associated with the Revised Bloom’s Taxonomy, the initial state is the student RASI profile, and the two optimization criteria are the similarity between the student’s profile and the sequence as well as the number of actions in the sequence. The multiobjective algorithms’ bases are genetic algorithms (GA) and particle swarm optimization (PSO) to minimize the aforementioned criteria. Students from higher education institutions were the participants in the experiments. Comparisons between both algorithms included the results found for each criterium and the satisfaction level of students with the sequences. In addition, a group of students received random sequences to compare the effectiveness of such a proposal. The algorithms found similar results among the students and suggested that the proposed approaches are better accepted than the randomized pedagogical sequences. Keywords: Sequencing of pedagogical actions · Pedagogical recommendation · Bloom’s taxonomy · RASI · GA · PSO · Multi-objective optimization

1 Introduction Learning is a continuous and natural process that happens in both methodical situations and everyday activities [19]. According to [5], teaching and learning is a human effort carried out by people to benefit others, and [30] argues that learning is more efficient when students are motivated to learn. In addition, [19] state that learning, when intentional and defined in an institutional context with explicit goals and objectives, is generally supported by structured sequences of instructions designed to facilitate or improve learning and performance. c The Author(s), under exclusive license to Springer Nature Switzerland AG 2023  J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, pp. 160–179, 2023. https://doi.org/10.1007/978-3-031-40501-3_8

Comparing Multi-objective GA and PSO

161

However, as classes are heterogeneous, the students perform better and reach different levels in the learning process, whether adjustments to the content or pedagogical strategies occur. The student cognitive profile is an example of an attribute able to make the learning process student-centered in opposition to the traditional teaching models that depend on course content and are teacher-centered. Moreover, technology plays a crucial role in developing personalized and individualized learning activities (Huang et al., 2019). Recommending tailored activities for the students can be seen as customization of the teaching and learning process in intelligent and adaptive virtual platforms, which considers the student’s cognitive profile or preferences to meet their individual needs, providing stimuli to guide the student, and allowing everyone to learn in their time (Sunaga and Carvalho, 2015). Student performance in Virtual Learning Environments (VLEs) can be improved by recommending teaching strategies customized and individualized. With the support of these environments, it is possible to provide personalized and more appropriate sequences of pedagogical activities for each predominant learning style of students, which is impossible in mass or conventional education (Moran, 2015). Considering such a scenario, a challenge is a search for sequences of activities adapted to a given student model. A set of pedagogical activities, the attributes of the student model, algorithms to perform the search efficiently, and optimality criteria are some requirements to cope with the sequencing problem. As the sequencing problem is also an optimization problem, this paper presents a study that compares two optimization metaheuristics of computational intelligence for sequencing pedagogical activities from Bloom’s Digital Taxonomy (BDT). According to the student profile, it is possible to identify different optimization criteria (objective function). Therefore, the sequencing problem is formulated as a multi-objective optimization problem and algorithms based on Genetic Algorithms (GAs) and Particle Swarm Optimization (PSO) search for sequences of activities. The pedagogical actions1 are those from Bloom’s Taxonomy, and the student model is the cognitive profile of the Revised Approaches to Studying Inventory (RASI). The results of the experiments are promising regarding student satisfaction and sequence quality for both algorithms. The paper is organized as follows. Section 2 presents previous work that used PSO and GA for pedagogical sequencing. Section 3 describes Bloom’s Taxonomy, RASI profiles, and the relationships between these theories. The multi-objective optimization problem formulation for sequencing and the specification of PSO and GA algorithms are in Sect. 4. Experiments, the analysis of the results, and comparisons between the algorithms are presented in Sect. 5. Section 6 contains conclusions and further work.

2 Related Works Meta-heuristics from Evolutionary Computing (EC) and Swarm Intelligence (SI) have been used to cope with pedagogical sequencing since is a hard problem [2], especially when it considers some student characteristics to propose an adaptive sequencing. In 1

In this study, “actions” refer to BT levels, and “activities” to a list of activities from BDT for each BT level.

162

D. J. Almeida et al.

Table 1. Summary of researches using meta-heuristics for the sequencing problem. Source: Adapted from [3] and [8]. Reference Metaheuristic

Student Model

Pedagogical Theories

[10] [6]

Discrete PSO, GA Binary PSO

-

[27]

PSO

[28] [22]

Hibrid PSO (Binary and Discrete) Multiobjective GA

[11]

GA

[1] [17] [18]

GA GA GA

[24]

GA

[8] [4]

Multiobjective GA Multiobjective and Binary PSO AG, Binary PSO, Prey Predator Algorithm, Differential Evolution

Competencies Ability level, expected learning targets, expected learning time of an e-course Objectives, preferences, level of knowledge, learning styles and academic motivations Cognitive classes based on BT Learning time and performance Learning styles, Learning goals, Knowledge level Concept difficulty Knowledge level Knowledge level and pedagogical objectives Learning time and satisfaction Student RASI profile Student RASI profile

[23]

Previous knowledge, time availability, learning preferences based on ILS

-

BT Learning styles BT BT, BDT and RASI profile BT, BDT and RASI profile Learning Style and prior knowledge (ILS), Felder and Silverman Learning Style Model (FSLSM)

this sense, [2] provide an overview of EC approaches — such as Ant Colony Optimization (ACO), Genetic Algorithm (GA), Parallel Memetic Algorithm, and Particle Swarm Optimization (PSO) — for solving the Curriculum Sequencing problem. [10] proposed the automatic sequencing process of learning objects in e-learning content creation. The sequencing problem was transformed into a constraint satisfaction problem, and two optimization agents were designed, developed, and tested: a discrete PSO and a GA. The results showed that both can solve the problem and PSO implementation outperforms GA. In [6], PC2 PSO was proposed to select appropriate e-learning materials for individual learners in a personalized e-course. In this approach, a binary multi-objective PSO was used, considering four different factors as optimization objectives: the learning concept covered, the difficulty level of the e-learning materials, the total learning time required, and the balance among the weights of the learning concepts.

Comparing Multi-objective GA and PSO

163

In [27], a solution was proposed to dynamically adapt the content offered in distance learning courses based on student profiles. Student profiles are generated from personal data collected in virtual learning environments, forums, and social networks. Starting from the identification of the profiles, a PSO-based approach is used to select activities and recommend them in order to be followed in the course. The research conducted by [28] proposed a model for recommending learning paths based on the cognitive classification of Revised Bloom’s Taxonomy and an ontology of learning objects. To determine the most appropriate learning path for the student’s cognitive abilities, the Hybrid PSO method was used, which consists of a Binary PSO to represent the cognitive classes and a Discrete PSO to represent the learning objects of an ontology. Almeida et al. [4] developed a multiobjective and binary PSO for sequencing actions of Bloom’s Taxonomy, where the student model was the predominant cognitive profile given by Revised Approaches to Studying Inventory. A mapping between Bloom’s Taxonomy and the cognitive profile described in [9] was used and two optimization criteria for measuring the sequence quality and length. Similarly, GAs have been applied to optimize action sequences in the pedagogical context. Lin et al. (2016) presented learning map planning using a multi-objective GA, where the objective functions were learning time and student performance. In Dwivedi et al. (2018), GA was applied for curricular sequencing by considering learning styles, knowledge levels, and learning goals as parameters for optimizing learning paths. Curricular sequencing is also addressed in Agbonifo and Olanrewaju (2018), where the learning path optimization in an e-learning environment is performed by a GA, while taking into account the concept difficulty level and the relationship among concepts as objectives to be optimized. Goyal and Rajalakshmi (2018) proposed a method that generates a set of evaluative activities according to BT learning levels, and a GA is used to create test sheets. Hence, the exams presented the degree of knowledge held by the student and optimized their performance. Hssina and Erritali (2019) developed an adaptive e-learning platform for generating learning paths appropriate to student profiles using a GA that considers pedagogical objectives set by the teacher and the student’s knowledge level. A GA for determining learning paths for student groups is addressed in de Miranda et al. (2019), where the optimization criteria were the maximization of student satisfaction and the minimization of time to fulfill activities. da Costa et al. [8, 9] proposed a multiobjective GA for sequencing actions based on RASI and Bloom’s Taxonomy theories. Sequencing was described as a planning problem and GA was an alternative to avoid the combinatorial aspect of such a problem. In the above mentioned studies, the optimization process is related to aspects linked to the curriculum, such as LO or activity restrictions and content requirements. Therefore, other research fronts that investigate the recommendation feasibility from the perspective of the student’s cognitive process are essential. Such a proposition can bring benefits, such as a recommendation process independent of curricular structures, besides focusing the learning control on the student. [23] presented a procedure for generating synthetic data sets to evaluate approaches used in the Adaptive Curriculum Sequencing problem. The generated datasets were

164

D. J. Almeida et al.

used to investigate the contribution of four metaheuristic techniques: Genetic Algorithms (GA), Particle Swarm Optimization (PSO), Prey Predator Algorithm, and a proposed technique based on Differential Evolution. The individual sequencing approach was modeled as a multi-objective problem using information from the students, the learning materials (difficulty, content, and style), and the course (target concepts). Table 1 provides an overview of research addressing the pedagogical sequencing problem using metaheuristics.

3 Background In this section, the Revised Approaches to Studying Inventory and Bloom’s Taxonomy theories are introduced and the mapping [8, 9] created between them is presented. Such mapping is the basis for pedagogical sequencing and allows automatic sequencing of pedagogical actions in a way that is independent of the curriculum and takes into account the learning process. 3.1

RASI Profile

An essential requirement for customizing the sequencing of pedagogical actions is the student’s profile. Several student characteristics can be used. In [25], learning style and VLE’s metadata were used to provide a personalized learning path for the student. In [27], learning style and knowledge level were used to provide a course tailored to the student’s needs. Thus, the student is classified under the surface, strategic or deep dimensions. According to this study, the student classified in the surface category presents a preference for directing the learning process to the requirements of the evaluation. The student whose category is defined as strategic is motivated by personal satisfaction, that is, he or she prioritizes achieving the best results by means of organized study and optimizing time. On the other hand, the student identified in the deep category directs his study toward challenging teaching activities, that is, that aim at researching the meaning of things. The Revised Approaches to Studying Inventory (RASI) defines the student’s cognitive profile from the perspective of three axes (Surface, Strategic, and Deep), as described by [29]. Thus, the student is classified under the surface, strategic or deep dimensions. According to this study, the student classified in the surface category, presents a preference for directing the learning process to the requirements of the evaluation. The student whose category is defined as strategic is motivated by personal satisfaction, that is, he or she prioritizes achieving the best results by means of organized study and optimizing time. On the other hand, the student identified in the deep category directs his study toward challenging teaching activities, that is, that aim at researching the meaning of things.

Comparing Multi-objective GA and PSO

165

The RASI establishes a relationship with the BT since each axes presents an evolution in the student’s cognitive profile, from Lower Order Cognitive Skills (LOCS) to Higher-Order Cognitive Skills (HOCS) just as occurs with the educational objectives in the BT. This characteristic led us to decide by using the RASI as a student model to provide the personalization of pedagogical actions based on the BT. The RASI was developed for use with students in higher education. It is also widely used in several works, such as [13], in which the RASI is one of the dimensions of the Approaches and Study Skills Inventory for Students (ASSIST). Its use can also be seen in [15], whose goal was to identify students’ study approaches to suggest adaptations in the delivery of educational content. The RASI in its original version is composed of 52 objective questions, with answers on a 5-point Likert2 scale. A short version of the RASI consisting of 18 questions is used in [14], which was also used in this work since the chance of student engagement and attention when answering this questionnaire may be increased. 3.2 Bloom’s Taxonomy In [21], BT was extended by the introduction of a second dimension, defining a twodimensional BT composed of Cognitive Process Dimension (CPD) and Knowledge Dimension (KD). Then the taxonomy’s educational objectives are placed in a matrix. CPD has six levels (Remember, Understand, Apply, Analyze, Evaluate, Create) and KD into four levels (Factual, Conceptual, Procedural, and Metacognitive). As the flow through these levels follows a hierarchy from LOCS to HOCS, which is also observed in RASI, it is possible to define pedagogical actions from the BT and RASI perspectives. Thus, we defined 24 pedagogical actions, as shown in Table 2. In Table 2, 24 actions are arranged, one for each educational objective of the BT. These actions follow the hierarchy proposed in the BT in which they develop from actions close to LOCS (concrete actions) to actions close to HOCS (abstract actions), in the order A1, A2, ..., A24. Note also that such a hierarchy allows for supplanting actions according to the student’s needs. In this way, a pedagogical sequence would not necessarily contemplate the 24 proposed actions. Thus, there are 224 sequencing possibilities, which makes manual customization difficult. In this sense, a contribution of this work is the automation of this process, based on the student’s RASI profile. For the sequencing of pedagogical actions (educational objectives), as proposed in this work to be possible, it is essential to associate activities with pedagogical actions. Several works structure activities or digital tools to the educational objectives of the BT. In [26], for each of the levels of CPD in BT, technologies capable of meeting such requirements are listed. In work proposed by [16], digital activities indexed by BT are used to select an optimal set of activities to enhance the learning of a group of students.

2

Likert, R. (1932). A technique for the measurement of attitudes. Archives of psychology.

166

D. J. Almeida et al. Table 2. Pedagogical actions defined according to BT. Source: Adapted from [9]. KD FA CO

PR

ME

CPD Remember A1 A2 A3 A4 Understand A5 A6 A7 A8 Apply A9 A10 A11 A12 Analyze A13 A14 A15 A16 Evaluate A17 A18 A19 A20 Create A21 A22 A23 A24 FA. Factual; CO. Conceptual; PR. Procedural; ME. Metacognitive; A. Action.

In [7], Bloom’s Digital Taxonomy (BDT) was developed to index digital activities to CPD levels to make the pedagogical recommendation based on BT actions feasible. [9] extended this indexing to the KD, making it feasible to use the BDT as support for the recommendation of digital activities from the pedagogical goals structured by the two dimensional BT. In this work, we have chosen to use such a framework because it enables the recommendation from the sequencing of pedagogical actions. 3.3

Relationship Between RASI and BT

Both RASI and BT present a hierarchy based on the evolution of the student’s cognitive level from LOCS to HOCS. [9] proposed a mapping that establishes this relationship based on this principle. Figure 1 shows the influence percentual of each CPD level on each RASI axis. The mapping described in Fig. 1 is essential for the approaches developed in this study since the comparisons between sequences of actions and student RASI profile allow for determining which one is most appropriate to the student. Hence, the goal is an automatic and efficient search for this sequence.

Fig. 1. Mapping RASI versus BT. Source: Adapted from [9].

Comparing Multi-objective GA and PSO

167

4 Problem Formulation A traditional version of an optimization problem is for optimizing (minimize or maximize) an objective function, f(x), where x is an n-sized vector of variables that represent the problem, and a solution is an n-tuple x which values optimize f(x). Approaches based on metaheuristics of EC or SI use solutions collection to find an optimal solution. Consequently, a good solution representation can better describe the problem and facilitate the f(x) calculation. Therefore, this section presents the solution representation and objective functions, two elements of problem formulation shared by PSO and GA. The specific aspects of each of these metaheuristics come next. 4.1 Solution Representation According to BT (Sect. 3), there are 24 possible actions. Hence, a sequence of actions, which is a solution for the problem, must be able to contain all the actions. Then, a solution representation is a binary 24-sized vector as pictured in Fig. 2a, where each bit is associated with a pedagogical action according to Table 2. If an action is in the sequence, the corresponding bit is set to 1, otherwise, 0.

(a) Solution Representation

(b) Recommended Sequence

Fig. 2. A solution and corresponding recommendation sequence with 11 actions. Source: [3].

The recommended sequence is composed of digital activities according to the BDT. Then, for each bit set to 1 in the sequence, a BDT activity is assigned (see Fig. 2b). This attribution was performed according to the mapping presented by [9] between BDT and BT. So, in effect, the recommendation is a sequence of digital activities. 4.2 Objective Functions Optimization problems can consider one or more objective functions, which represent the criteria to be optimized and are directly related to the problem to be solved. These functions can be influenced by independent variables that affect the evaluation of the solutions.

168

D. J. Almeida et al.

Formally, the problem is defined as the minimization of the objective function f (x) as given by Eq. 1, where x is a sequence, F1 measures the similarity between the sequence RASI index, RRASI , and the student’s RASI profile, SRASI , and F2 the sequence size. The weights ωi were adopted to weigh the contribution of each criterium and then the multi-objective aspect was considered as the weighted sum of criteria. 2

f (x) = ∑ ωi Fi (x)

(1)

i=1

F1 checks whether the sequence matches the RASI profile of the student. Therefore, the RASI indices of the sequence must be determined, expressing the strength of each CPD level (Remember, ..., Create) in the sequence weighted by the relevance of that level to each RASI axis (Surface, Strategic and Deep). The index of each RASI axis that makes up RRASI is calculated by the product between the weight of the influence of each CPD level for the RASI axis (Fig. 1) and the number of bits set to 1 in that BT level multiplied by 1/4. The SRASI index is obtained from students’ responses to the RASI questionnaire. Notice in Fig. 1 that the Apply level does not influence the Surface and Deep axes, just as the Evaluate level has no influence on the Deep axis. The RRASI index of the sequence in Fig. 2a is [0.438, 0.484, 0.475], where the value for each axis is Surface = 0.438, Strategic = 0.484, and Deep = 0.475. To illustrate the calculation of the value of each axis, we can take the value for the Surface axis of this sequence, which is the result of the calculation of Eq. 2. Finally, the similarity between the RASI index of the sequence and the RASI profile of the student is given by the Euclidean distance D between RRASI and SRASI . 2 ∗ 0.625 2 ∗ 0.125 1 ∗ 0.000 + + + 4 4 4 (2) 0 ∗ 0.125 4 ∗ 0.000 2 ∗ 0.125 + + 4 4 4 Note that only Euclidean distance is not sufficient to determine whether the sequence is close to the student’s profile with respect to each RASI axis. Then P adds a penalty to F1 (x) for each RASI axis that is violated. Assume that the Deep axis is more relevant to the student and the Surface axis is more relevant to the sequence. This relevance is attributed to each axis according to w1 = 1 for the least relevant axis, w2 = 2 for the intermediate axis, and w3 = 3 for the most relevant axis. At each RASI axis where there is a divergence of relevance between the student and the sequence, the corresponding weight is multiplied by 1/6 of the Euclidean distance. Thus, the penalty is at most the Euclidean distance and consequently, F1 (x) is at most twice the Euclidean distance. If there is no difference in the order of relevance on any of the RASI axes, w1 , w2 , and w3 are set to 0. The objective function F1 and the penalty P are given by Eq. 3 and Eq. 4, respectively. Sur f ace

RRASI

=

F1 (x) = D (SRASI , RRASI ) + P 3

P = ∑ wi ∗ i=1

D 6

(3)

(4)

Comparing Multi-objective GA and PSO

169

Through experiments conducted by [9], the appropriate number of actions for each predominant RASI profile was suggested and it is used in this study as a reference value for the sequence size (re f ): 9 for Surface, 13 for Strategic, and 11 for Deep. Thus, the objective function F2 has the task of optimizing the number of actions that make up the sequence, minimizing the difference between the sequence size and the reference value. Equation 5 defines F2 . ⎧ re f −size(x) ⎪ ⎨ re f −1 , if size(x) < re f (5) F2 (x) = ⎪ ⎩ size(x)− re f 24− re f , otherwise 4.3 Multiobjective Binary PSO The main elements of PSO are the positions and velocities of particles3 . The first version of PSO was defined for real-valued domain problems and the process consists of maintaining a swarm of particles and iteratively updating their positions, xt+1 i , and velocities, t+1 vi , as given by Eq. 6. At each iteration, new velocities are calculated regarding the previous ones, which are weighted by the inertia w. In addition, there are the components t (pt − xt ) and c rt (gt − xt ), which quantify the particle’s performance with c1 r1d 2 2d id id id id respect to its previous performance and its neighborhood, respectively, where. pid is the best position found since the first-time interaction, gid is the best position found so far by that group of particles, c1 and c2 are positive acceleration coefficients, r1 and r2 are random values for controlling the stochastic influence of each component on the general velocity of the particles and are obtained for each time interaction from a uniform distribution ∈ [0,1].  t t t t t t t vt+1 id = wvid + c1 r1d (pid − xid ) + c2 r2d (gid − xid ) (6) t+1 t xt+1 id = xid + vid Since optimization problems with real-valued domains can be converted to binary domains [12], Algorithm 1 describes a version multiobjective and binary of PSO. which is based on a discrete version developed by [20] to work in binary search spaces. In this version, the particles represent positions in binary space, where each element of the position vector can take the values 0 and 1 [12]. The position of the particle changes when any bit of the position vector flips its value from one value to another. In this way, the velocity of a particle can be interpreted as the Hamming distance between its previous and its current position. The new velocity vt+1 id is defined as the probability that a bit is in one state or the other, and its value represents the probability that the bit value is 1. The previous velocity vtid measures the predisposition (or current probability) to choose the next bit value 1. In this probabilistic view, velocity must be normalized to be confined to the interval [0, 1]. This normalization is achieved by using the sigmoid function presented in Eq. 7. 3

According to the original terminology of PSO, the description uses the particle which is a solution representation.

170

D. J. Almeida et al.

Algorithm 1. Multiobjective Binary PSO. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Randomly, create the position vectors, (x), of n particles as in Fig. 2a t =0 for each particle i = 1 to n do Randomly initialize velocity vector, vti Set pti to particle position Calculate f (i) and determine gti end while t < Maxt do for each particle i = 1 to n do vt+1 = wvti + c1 r1t (pti − xti ) + c2 r2t (gti − xti ) i Calculate sig(vt+1 i ) (Eq. 7) Update xt+1 (Eq. 8) i Calculate f (i) = ω1 F1 (i) + ω2 F2 (i) Update pt+1 i end Update gt+1 i t = t +1 end

The parameter Vmax = ±4 was set to limit the particle velocity to the interval [−4, 4] to ensure that there is always the possibility of a bit changing state. sig(vt+1 id ) =

1 1 + e−vid t

(7)

The normalized velocity is now the probability with the d-th bit of the position vector will be set to 1. The position xt+1 id of the particle is changed stochastically by comparing, at each iteration, the result of sig(vt+1 id ) with a random number ρ from a uniform distribution ∈ [0,1], according to Eq. 8. Due to the random number, the new bit position can be changed even if the velocity does not change.  1, if ρd < sig(vt+1 id ) xt+1 (8) id = 0, otherwise 4.4

Multiobjective GA

As an evolutionary algorithm, GA evolves a population of individuals through genetic operators such as selection, crossover, and mutation. An individual is a solution, then is represented according to Fig. 2a. Algorithm 2 describes the multiobjective GA, in which the evolutionary process consists of applying genetic operators to an initial population generating news individuals to make up the next generations until a stop criterium is reached. The genetic operators are critical for the evolution and there are many options for each operator. In Algorithm 2, the tournament method is used to select the individuals that will be part of the next generation. In this process, three individuals

Comparing Multi-objective GA and PSO

171

Algorithm 2. GA. 1 2 3 4 5 6 7 8 9 10 11

P0 = Randomly create n individuals as in Fig. 2a /* Initial population t =1 while t < MAXt do for each individual in Pt−1 do Calculate f (i) = ω1 F1 (i) + ω2 F2 (i) end PSelected = Select from Pt−1 individuals for reproduction according to f (i) PTemp = Apply crossover to PSelected with pc PTemp = Apply mutation to PSelected with pm Pt = Select n best individuals from Pt−1 and PTemp end

are randomly selected, and the best of these is chosen according to a probability pt . The crossover is performed with probability pc in adjacent 4-bit blocks, thus considering the six subcategories of CPD for BT (See Fig. 2. After the new individuals are ranked, the best ranked will make up the next generation population. The mutation, performed randomly in the population and considering a pm probability, inverts a random bit from the individual. Metaheuristics such as these aforementioned described require fine-tuning of parameters. After tests, the most suitable values for the parameters are those in Table 3. Table 3. Parameters. Multiobjective PSO Multiobjective GA n

200

1000

MAXt 100

100

ω1

0.7

0.7

ω2

0.3

0.3

pt

-

0.6

pc

-

0.7

pm

-

0.1

5 Experiments, Results and Discussion The experiments were divided into three phases: i) Application of the RASI questionnaire; ii) Sequencing of pedagogical actions; and iii) Recommendation of pedagogical activities. The participants, higher education students at three educational institutions: Distance learning and presential course of the Federal Institute of Education, Science and Technology and Distance Learning Center of a Federal University at Minas Gerais, and the Federal Institute of Goiás, Brazil, were divided into three groups. The control group (15 participants) received randomly generated sequences, the second one (50 participants) received sequenced activities through Algorithm 1, and the 49 participants of the third one received sequences from Algorithm 2.

172

D. J. Almeida et al. Table 4. Questionnaire questions and their answer groups. Source: [3].

QUESTION

AG∗

Q1.

G1

I consider that the percentage assigned to me on the SURFACE axis should be: Q2. I consider that the percentage assigned to me on the STRATEGIC axis should be: Q3. I believe that the percentage assigned to me on the DEEP axis should be: Q4. I consider that the percentage assigned to me on the SURFACE axis is in line with my learning profile. Q5. I consider that the percentage assigned to me on the STRATEGIC axis is in line with my learning profile. Q6. I consider that the percentage assigned to me on the DEEP axis is in line with my learning profile. Q7. Do you think the number of activities is: The sequence of activities is comfortable to lead you in learning a Q8. new content or subject. Q9. What is the probability that you will complete all the activities in this sequence? Q10. The total number of recommended activities is too many. * Answer Group

G1 G1 G2 G2 G2 G3 G2 G4 G2

Table 5. Questionnaire answer groups. Source: Adapted from [3]. AN∗ G1**

G2**

AN1 Much higher (from 7% more) AN2 Higher (3% to 6% more)

Agree

G3**

Very High (at least 6 more than ideal) Partially Agree High (between 3 and 5 more than ideal) Indifferent Sufficient (up to 2 more or AN3 Equal (up to 2% more or less) fewer than ideal) AN4 Smaller (from 3 to 6% less) Partially Disagree Low (between 3 and 5 less than ideal) Disagree Very Low (at least 6 fewer AN5 Much Smaller (from 7% less) than ideal)

* Answer Number; ** Answer Group

G4** Very High (above 80%) High (between 61% and 80%) Moderated (between 41% and 60%) Low (between 20% and 40%) Very Low (below 20%)

Comparing Multi-objective GA and PSO

173

Fig. 3. Number of participants in the experiment.

5.1 Students’ Profiles As student participation (Fig. 3) was voluntary, 182 participated in Phase i, responding to a free translation of the short version of the RASI questionnaire into Portuguese. After collecting answers, the RASI axes were calculated, and 9% of students were the Surface axis as the predominant, 25% were Strategic and 66% Deep. Only 100 of these participants answered a questionnaire, composed of six objective questions with answers options on a 5-point Likert scale, stating how much they agreed with their RASI indices, as shown in Fig. 4. Table 4 lists the questionnaires of Phases i and iii and Table 5 shows the groups of response options. Of the 182, only 114 students remained for Phases ii and iii.

Fig. 4. Students’ perception of the indices obtained by the RASI profile. Source: [3].

174

D. J. Almeida et al.

In Fig. 4, Q1 to Q6 questions are intended to know about the students’ agreement with their RASI profile obtained from their questionnaire answers. Q1 to Q3 asked the student if the percentage for Surface, Strategic, and Deep, respectively, should be less or more significant. Q4 to Q6 intended to confirm the answers from Q1 to Q3. It was only asked if the student agreed with the percentual of each axis. Most of the answers for Q1 to Q3 were Higher, Equal, or Smaller, with a higher concentration on the Equal answer. This result suggests a certain degree of student awareness while taking the RASI questionnaire. The results show that most of the answers from Q4 to Q6 focused on Agree and Partially Agree, confirming the students’ attention to answer the RASI questionnaire and, therefore, the quality of this questionnaire’s answers. 5.2

Sequencing Analysis

The comparisons between the algorithms considered the following aspects: 1. Activities per CPD Category for each RASI Axes. This first comparison aimed to know if the algorithms found sequences close to the predominant profile of the student by observing Figs. 1 and 5. In other words, if F1 works. Regards the Surface profile, sequences from both algorithms contain activities of Evaluate level, which were unexpected if one observes Fig. 1. On the other hand, GA and PSO did not sequence any activity of Apply, which is a positive result. For the Strategic axes, none of the algorithms sequenced activities from Create, an expected result. GA returned activities from Apply level for Deep and PSO not. Although the algorithms’ results presented negative aspects, one can note that these results were better than random sequences, which did not distinguish by the lack of some levels. 2. Activities in Relevant Levels. The similarity between the sequence and the student’s profile considers all the axes that make up the profile. However, the analysis of the quality of the sequencing can be performed by grouping the CPD levels according to the degree of relevance (High, Moderate, and Low) for each RASI profile. The relevance of the CPD levels for each RASI profile can be seen in Table 6.

Fig. 5. Recommended actions by CPD levels for each RASI profile: (a) GA; (b) PSO; (c) Random - control group.

Comparing Multi-objective GA and PSO

175

Fig. 6 shows the distribution of the activities by the relevance degree. Most of the activities were at CPD levels that are more relevant to the predominant profile for GA and PSO, but the percentual of high relevance was expressive for PSO, which was better than GA for all profiles. Moreover, GA sequences for the Deep profile presented a high percentual of low-relevance activities, the opposite of PSO. Even though the percentual of high and moderate relevances were above 60% for random sequences, activities’ number of high relevance was smaller than GA and PSO. 3. Sequence Size. This item is directly related to F2 and indicated that PSO fulfills the requirement of controlling the activities’ number since the average sequence size for each predominant profile is equal to the reference values, showing the convergence of F2 function. GA converged for sequences where the student profile was Strategic and average sizes were very close to reference values for Surface and Deep profiles. Random sequences were not comparable to the algorithms. 4. Student Satisfaction. Although PSO was not significantly better than GA in Q7, which refers to the student’s perception regards sequence size, this result confirms the previous comparison when PSO overcame GA. In addition, a high percentual (59%) agreed with the sequence size. This item evaluated F2 and proved such an objective function is useful for the problem. Compared to randomly sequenced activities, the results of Q7 for PSO are better since the percentage of participants who consider the number of answers Sufficient is higher than each other answers. Q8 reported the quality of the sequence of activities since the student answered how comfortable the sequence was for learning new content. In Fig. 8, above 80% of the participants agreed or partially agreed with the Q8 statement for PSO and GA. However, the percentual for Agree was high for random sequences, which is a bad result. The Q9 could demonstrate student satisfaction as the sequence quality as the sequence size since the latter can influence answers about whether the students would (or not) complete the sequence. Most of the answers focused on Very High, High, and Moderated for the PSO (80%) and GA (88%). In this case, GA overcame PSO in a significant way, mainly if it also considers the answers for Low and Very Low, where PSO (20%) was higher than GA (12%). These results were better than those for random sequences, which presented 27% for Low and Very Low and less than 80% for other answers. Q10 intended to confirm Q7 as the same question was asked differently. In Q10, there was a statement about the high quantity of activities. Percentual of Agree, Partially Agree and Indifferent for random sequences were 87% against of 79% PSO and 80% GA, but Desagree at random sequences were double of the two algorithms. Again, it can consider GA and PSO tied. The percentage of Agree is lower for the algorithms than for the random sequences.

176

D. J. Almeida et al. Table 6. Relevance of BT levels for each RASI axis. Source: [3]. Surface High Remember Moderate Understand Analyze Create Low Apply Evaluate

Strategic

Deep

Analyze Evaluate Understand Apply

Analyze Evaluate Understand Create

Remember Create

Remember Apply

Fig. 6. Recommended actions by the degree of relevance for each RASI profile: (a) GA; (b) PSO; (c) Random - control group.

Fig. 7. Average sequence size per RASI profile (predominant).

Comparing Multi-objective GA and PSO

177

Fig. 8. Satisfaction questionnaire for activities sequenced by method/algorithm: (a) GA; (b) PSO; (c) Random - control group.

6 Conclusions This paper compared two multiobjective algorithms supported by metaheuristics from computational intelligence for automatically sequencing pedagogical actions. These approaches were suitable if one considers actions from Bloom’s Taxonomy and a student model based on RASI profiles. In this case, the learning process that underlines these theories allowed for determining sequences independent of the content to be learned and showed that the mapping BT X RASI is effective in providing foundations for this problem. The comparisons took into account aspects such as sequence similarity to the student profile, relevance degrees of the actions for the student, sequence size, and student satisfaction. Although multiobjective PSO had overcome GA concerning sequence size, both algorithms demonstrated convergence for the second objective function, which goal was precisely minimizing the sequence size. On the other hand, multiobjective GA performed better than PSO in student satisfaction. In one of the four questions, GA was superior to PSO, and in the other three questions, almost the same results were presented by both. Regards the first and second aspects, the results are similar. Unexpectedly, random sequences were better than GA and PSO in some questions about student satisfaction, but, in general, did not present expressive results related to the sequence similarity and size. This study pointed out that the optimization process was able to find sequences composed of actions that were relevant and in adequate quantity for each student. In addition, students were satisfied with the quantity and quality of the activities returned by the algorithms. A limitation of this study is the discrepancy between the predominant profiles of the students who participated in the experiment, which requires a more in-depth statistical analysis of the data obtained. In future works, we intend to feedback on the reference values of the optimization objectives from the satisfaction survey results and carry out the integration with a Virtual Learning Environment to automate the recommendation process.

178

D. J. Almeida et al.

Acknowledgment. The authors thank the Federal University of Uberlândia, the Goiano Federal Institute, and the Federal Institute of Triângulo Mineiro for supporting this research.

References 1. Agbonifo, O.C., Olanrewaju, A.O.: Genetic algorithm-based curriculum sequencing model for personalized e-learning system. Int. J. Educ. Manag. Eng. 5(8), 27–35 (2018) 2. Al-Muhaideb, S., Menai, M.E.B.: Evolutionary computation approaches to the curriculum sequencing problem. Nat. Comput. 10(2), 891–920 (2011) 3. Almeida, D.J., Fernandes, M.A., da Costa, N.T.: Sequencing and recommending pedagogical activities from bloom’s taxonomy using RASI and multi-objective PSO. In: Cukurova, M., Rummel, N., Gillet, D., McLaren, B.M., Uhomoibhi, J. (eds.) Proceedings of the 14th International Conference on Computer Supported Education, CSEDU 2022, Online Streaming, 22–24 April 2022, vol. 2. pp. 105–116. SCITEPRESS (2022). https://doi.org/10.5220/ 0011090000003182 4. Almeida., D.J., Fernandes., M.A., Torrezão da Costa., N.: Sequencing and recommending pedagogical activities from bloom’s taxonomy using RASi and multi-objective PSO. In: Proceedings of the 14th International Conference on Computer Supported Education - Volume 2: CSEDU, pp. 105–116. INSTICC, SciTePress (2022). https://doi.org/10.5220/ 0011090000003182 5. Brown, M., McCormack, M., Reeves, J., Brook, D.C., Grajek, S., Alexander, B., Bali, M., Bulger, S., Dark, S., Engelbert, N., et al.: 2020 educause horizon report teaching and, learning Technical Report EDUCAUSE (2020) 6. Chu, C.P., Chang, Y.C., Tsai, C.C.: Pc 2 PSO: personalized e-course composition based on particle swarm optimization. Appl. Intell. 34(1), 141–154 (2011) 7. Churches, A.: Bloom’s digital taxonomy (2010) 8. da Costa, N.T., Almeida, D.J., Oliveira, G.P., Fernandes, M.A.: Customized pedagogical recommendation using automated planning for sequencing based on bloom’s taxonomy. Int. J. Distance Educ. Technol. 20, 1–19 (2022) 9. da Costa, N.T., Fernandes, M.A.: Sequenciamento de ações pedagógicas baseadas na taxonomia de bloom usando planejamento automatizado apoiado por algoritmo genético. Revista Brasileira de Informática na Educação 29, 485–501 (2021) 10. De-Marcos, L., Martínez, J.J., Gutiérrez, J.A., Barchino, R., Gutiérrez, J.M.: A new sequencing method in web-based education. In: 2009 IEEE Congress on Evolutionary Computation, pp. 3219–3225. IEEE (2009) 11. Dwivedi, P., Kant, V., Bharadwaj, K.K.: Learning path recommendation based on modified variable length genetic algorithm. Educ. Inf. Technol. 23(2), 819–836 (2018) 12. Engelbrecht, A.P.: Fundamentals of Computational Swarm Intelligence. John Wiley & Sons Inc, Hoboken (2006) 13. Entwistle, N.: Student Learning and Academic Understanding: A Research Perspective with Implications for Teaching. Academic Press, Cambridge (June 2018) 14. Entwistle, N., Tait, H.: Approaches and study skills inventory for students (ASSIST) (incorporating the revised approaches to studying inventory - RASI). Centre for Research on Learning and Instruction, University of Edinburgh, Edinburgh (March 2013) 15. Fusilier, M., Bhuyan, R., Russell, J., Lin, S., Yang, S.: Studying approaches: samples in China, KUWAIT, and USA. J. Appl. Res. Higher Educ. ahead-of-print (2021). https://doi. org/10.1108/JARHE-11-2020-0385

Comparing Multi-objective GA and PSO

179

16. Goštautait˙e, D.: Principal component analysis and bloom taxonomy to personalise learning. In: EDULEARN19 Proceedings 11th International Conference on Education and New Learning Technologies: Palma, Spain. 1–3 July 2019, pp. 2910–2920. IATED Academy (2019) 17. Goyal, M., Rajalakshmi, K.: Personalization of test sheet based on bloom’s taxonomy in e-learning system using genetic algorithm. In: Sa, P.K., Bakshi, S., Hatzilygeroudis, I.K., Sahoo, M.N. (eds.) Recent Findings in Intelligent Computing Techniques. AISC, vol. 708, pp. 409–414. Springer, Singapore (2018). https://doi.org/10.1007/978-981-10-8636-6_42 18. Hssina, B., Erritali, M.: A personalized pedagogical objectives based on a genetic algorithm in an adaptive learning system. Procedia Compu.t Sci. 151, 1152–1157 (2019) 19. Huang, R., Spector, J.M., Yang, J.: Educational Technology: A Primer for the 21st Century. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-6643-7 20. Kennedy, J., Eberhart, R.C.: A discrete binary version of the particle swarm algorithm. In: 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation. vol. 5, pp. 4104–4108. IEEE (1997) 21. Krathwohl, D.R.: A revision of bloom’s taxonomy: an overview. Theory Into Pract. 41(4), 212–218 (2002). https://doi.org/10.1207/s15430421tip4104_2 22. Lin, Y.S., Chang, Y.C., Chu, C.P.: An innovative approach to scheme learning map considering tradeoff multiple objectives. J Educ. Technol. Soc. 19(1), 142–157 (2016) 23. Martins, A.F., Machado, M., Bernardino, H.S., de Souza, J.F.: A comparative analysis of metaheuristics applied to adaptive curriculum sequencing. Soft. Comput. 25(16), 11019– 11034 (2021). https://doi.org/10.1007/s00500-021-05836-9 24. de Miranda, P.B., et al.: Uma abordagem multiobjetivo para recomendação de caminhos de aprendizagem para grupo de usuários. Revista Brasileira de Informática na Educação 27(3) (2019) 25. Pireva, K., Kefalas, P.: A recommender system based on hierarchical clustering for cloud e-learning. In: Intelligent Distributed Computing XI, pp. 235–245 (2017). https://doi.org/10. 1007/978-3-319-66379-1_21 26. Schrock, K.: Bloomin’ Apps: Kathy Schrock’s Guide to Everything (September 2011), https://www.schrockguide.net/bloomin-apps.html 27. Smaili, E.M., Khoudda, C., Sraidi, S., Charaf, M.E.H.: An optimized method for adaptive learning based on PSO algorithm. In: 2020 IEEE 2nd International Conference on Electronics, Control, Optimization and Computer Science (ICECOCS), pp. 1–5. IEEE (2020) 28. Subiyantoro, E., Ashari, A., Suprapto, S.: Learning Path Recommendation using Hybrid Particle Swarm Optimization. Adv. Sci. Technol. Eng. Syst. J. 6(1), 570–576 (2021). https:// doi.org/10.25046/aj060161 29. Tait, H., Entwistle, N.: Identifying students at risk through ineffective study strategies. Higher Educ. 31, 97–116 (1996). https://doi.org/10.1007/BF00129109 30. Woolf, B.P.: Building Intelligent Interactive Tutors: Student-Centered Strategies for Revolutionizing e-Learning. Morgan Kaufmann (2010)

Author Index

A Albshri, Adel 100 Almeida, Denis José 160 Awaji, Bakri 100 C Costa, Igor Ernesto Ferreira D da Costa, Newarney Torrezão Du, Hanxiang 1 E El Mawas, Nour

L Lauría, Eitel J. M. 149 Lovászová, Gabriela 51 Lu, Jie 1

26

160

F Fernandes, Márcia Aparecida G Gajewski, Sebastian 81 Gonzalez-Bonorino, Augusto

125

51

O Oliveira, Sandro Ronaldo Bezerra 26 P Pei, Bo 1

81

H Heutte, Jean 81 Hinterplattner, Sara

M Michaliˇcková, Viera

160

S Solaiman, Ellis 100

149

X Xing, Wanli

1

Z Zeng, Yifang 1 Zhang, Yuanlin 1

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2023 J. Uhomoibhi (Ed.): CSEDU 2022, CCIS 1817, p. 181, 2023. https://doi.org/10.1007/978-3-031-40501-3