Methodologies and Intelligent Systems for Technology Enhanced Learning, 10th International Conference [1st ed.] 9783030525378, 9783030525385

Table of contents :
Front Matter ....Pages i-xiii
Effects of Time in Virtual Reality Learning Environments Linked with Materials Science and Engineering (J. Extremera, D. Vergara, M. P. Rubio, L. P. Dávila, F. De la Prieta)....Pages 1-9
HEMOT®, Helmet for EMOTions: A Web Application for Children on Earthquake-Related Emotional Prevention (Giada Vicentini, Margherita Brondino, Roberto Burro, Daniela Raccanello)....Pages 10-19
Social Video Learning – Creation of a Reflection-Based Course Design in Teacher Education (Eric Tarantini)....Pages 20-30
Engaging Pre-teens in Ideating and Programming Smart Objects Through Play (Rosella Gennari, Maristella Matera, Alessandra Melonio, Mehdi Rizvi, Eftychia Roumelioti)....Pages 31-40
Advanced Placement Physics Exam Performance of High School Graduates in Mexico with the Aid of Online Assignments Designed in Open-EdX (Ernesto M. Hernández, Rubén D. Santiago, José A. Otero, Ma. de Lourdes Quezada-Batalla)....Pages 41-49
Hand Robotics Rehabilitation in Patients with Multiple Sclerosis: A Pilot Study (Marco Tramontano, Laura Casagrande Conti, Niccolò Marziali, Giorgia Agostini, Sara De Angelis, Giovanni Galeoto et al.)....Pages 50-57
When Something Useful Is Also Enjoyable: An Empirical Study on the Intention to Use Web-Based Simulations in Higher Education (Leonardo Caporarello, Federica Cirulli, Federico Magni, Beatrice Manzoni)....Pages 58-67
TEL Adoption in the Riconnessioni Project (Marcello Enea Newman)....Pages 68-73
Cognitive Complexity Analysis of Learning-Related Texts: A Case Study on School Textbooks (Syaamantak Das, Shyamal Kumar Das Mandal, Anupam Basu)....Pages 74-84
Lessons Clustering Using Topics Inferred by Unsupervised Modeling from Textbooks (Matías Altamirano, Abelino Jiménez, Roberto Araya)....Pages 85-94
Automatic Content Analysis of Computer-Supported Collaborative Inquiry-Based Learning Using Deep Networks and Attention Mechanisms (Pablo Uribe, Abelino Jiménez, Roberto Araya, Joni Lämsä, Raija Hämäläinen, Jouni Viiri)....Pages 95-105
The Impact of Personality, Attitude and Visual Decision-Making Dashboard Tools on the Learning Engagement of Economist Students (Liana Stanca, Cristina Felea, Romeo Stanca, Mirela Pintea)....Pages 106-116
HRS-EDU: Architecture to Control Social Robots in Education (John Páez, Enríque González, Maria Impedovo)....Pages 117-126
Authoring Interactive Videos for e-Learning: The ELEVATE Tool Suite (Daniele Dellagiacoma, Paolo Busetta, Artem Gabbasov, Anna Perini, Angelo Susi)....Pages 127-136
Early Detection of Gender Differences in Reading and Writing from a Smartphone-Based Performance Support System for Teachers (Roberto Araya)....Pages 137-146
An Assessment of Students’ Satisfaction in Higher Education (Margarida Figueiredo, Ana Fernandes, Jorge Ribeiro, José Neves, Almeida Dias, Henrique Vicente)....Pages 147-161
Methodological Guidelines to Build Collaborative Serious Games Based on Intelligent Agents (Oscar M. Salazar, Santiago Álvarez, Demetrio A. Ovalle)....Pages 162-173
An Augmented Reality-Based mLearning Approach to Enhance Learning and Teaching: A Case of Study in Guadalajara (Janet Pigueiras, Angel Ruiz-Zafra, Rocio Maciel)....Pages 174-184
Supporting the Construction of Learning Paths in a Competency-Based Informatics Curriculum (Luca Forlizzi, Giovanna Melideo, Cintia Scafa Urbaez Vilchez)....Pages 185-194
Personalized Recommender System Using Learners’ Metacognitive Reading Activities (Lydia Odilinye, Fred Popowich)....Pages 195-205
Technology-Enhanced Learning (TEL) in Anaesthesia: Ultrasound Simulation Training for Innovative Locoregional Anaesthesia (Vincenza Cofini, Pierpaolo Vittorini, Emiliano Petrucci, Stefano di Carlo, Pierfrancesco Fusco, Franco Marinangeli et al.)....Pages 206-215
Profiles in Brain Type in Programming Performance for Non-vocational Courses (Ugo Solitro, Margherita Brondino, Roberto Bonafini, Margherita Pasini)....Pages 216-225
Towards a Gamified Musical Skill Learning Model (MuS-LM): Structural Aspects (Tania Di Mascio, Laura Tarantino, Federica Caruso)....Pages 226-238
Philosophical Approaches to Smart Education and Smart Cities (Javier Teira-Lafuente, Ana B. Gil-González, Ana de Luis Reboredo)....Pages 239-248
Experiences in a Differential Equations Massive Course (Rubén Dario Santiago Acosta, María de Lourdes Quezada Batalla, Ernesto Manuel Hernández Cooper)....Pages 249-258
A Protocol for Simulated Experimentation of Automated Grading Systems (Andrea Sterbini, Marco Temperini, Pierpaolo Vittorini)....Pages 259-270
Back Matter ....Pages 271-272

Citation preview

Advances in Intelligent Systems and Computing 1241

Pierpaolo Vittorini · Tania Di Mascio · Laura Tarantino · Marco Temperini · Rosella Gennari · Fernando De la Prieta   Editors

Methodologies and Intelligent Systems for Technology Enhanced Learning, 10th International Conference

Advances in Intelligent Systems and Computing Volume 1241

Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Sciences, Warsaw, Poland Advisory Editors Nikhil R. Pal, Indian Statistical Institute, Kolkata, India Rafael Bello Perez, Faculty of Mathematics, Physics and Computing, Universidad Central de Las Villas, Santa Clara, Cuba Emilio S. Corchado, University of Salamanca, Salamanca, Spain Hani Hagras, School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK László T. Kóczy, Department of Automation, Széchenyi István University, Gyor, Hungary Vladik Kreinovich, Department of Computer Science, University of Texas at El Paso, El Paso, TX, USA Chin-Teng Lin, Department of Electrical Engineering, National Chiao Tung University, Hsinchu, Taiwan Jie Lu, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, NSW, Australia Patricia Melin, Graduate Program of Computer Science, Tijuana Institute of Technology, Tijuana, Mexico Nadia Nedjah, Department of Electronics Engineering, University of Rio de Janeiro, Rio de Janeiro, Brazil Ngoc Thanh Nguyen , Faculty of Computer Science and Management, Wrocław University of Technology, Wrocław, Poland Jun Wang, Department of Mechanical and Automation Engineering, The Chinese University of Hong Kong, Shatin, Hong Kong

The series “Advances in Intelligent Systems and Computing” contains publications on theory, applications, and design methods of Intelligent Systems and Intelligent Computing. Virtually all disciplines such as engineering, natural sciences, computer and information science, ICT, economics, business, e-commerce, environment, healthcare, life science are covered. The list of topics spans all the areas of modern intelligent systems and computing such as: computational intelligence, soft computing including neural networks, fuzzy systems, evolutionary computing and the fusion of these paradigms, social intelligence, ambient intelligence, computational neuroscience, artificial life, virtual worlds and society, cognitive science and systems, Perception and Vision, DNA and immune based systems, self-organizing and adaptive systems, e-Learning and teaching, human-centered and human-centric computing, recommender systems, intelligent control, robotics and mechatronics including human-machine teaming, knowledge-based paradigms, learning paradigms, machine ethics, intelligent data analysis, knowledge management, intelligent agents, intelligent decision making and support, intelligent network security, trust management, interactive entertainment, Web intelligence and multimedia. The publications within “Advances in Intelligent Systems and Computing” are primarily proceedings of important conferences, symposia and congresses. They cover significant recent developments in the field, both of a foundational and applicable character. An important characteristic feature of the series is the short publication time and world-wide distribution. This permits a rapid and broad dissemination of research results. ** Indexing: The books of this series are submitted to ISI Proceedings, EI-Compendex, DBLP, SCOPUS, Google Scholar and Springerlink **

More information about this series at http://www.springer.com/series/11156

Pierpaolo Vittorini Tania Di Mascio Laura Tarantino Marco Temperini Rosella Gennari Fernando De la Prieta •

•

•

•

•

Editors

Methodologies and Intelligent Systems for Technology Enhanced Learning, 10th International Conference

123

Editors Pierpaolo Vittorini Department of Life, Health and Environmental Sciences University of L’Aquila L’Aquila, Italy

Tania Di Mascio Department of Information Engineering, Computer Science and Mathematics University of L’Aquila L’Aquila, Italy

Laura Tarantino Department of Information Engineering, Computer Science and Mathematics University of L’Aquila L’Aquila, Italy

Marco Temperini Department of Computer, Control, and Management Engineering Sapienza University of Rome Rome, Italy

Rosella Gennari Computer Science Faculty Free University of Bozen-Bolzano Bolzano, Italy

Fernando De la Prieta Department of Computer Science and Automation Control University of Salamanca Salamanca, Spain

ISSN 2194-5357 ISSN 2194-5365 (electronic) Advances in Intelligent Systems and Computing ISBN 978-3-030-52537-8 ISBN 978-3-030-52538-5 (eBook) https://doi.org/10.1007/978-3-030-52538-5 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

Preface

Education is the cornerstone of any society, and it serves as one of the foundations for many of its social values and characteristics. Different methodologies and intelligent technologies are employed for creating technology-enhanced learning (TEL) solutions. TEL solutions are innovative when they are rooted in artificial intelligence, deployed as stand-alone solutions or interconnected to others. They target not only cognitive processes but also motivational, personality, or emotional factors. In particular, recommendation mechanisms enable us tailoring learning to different contexts and people, e.g. by considering their individual traits. The use of learning analytics also helps us augment learning opportunities for learners and educators alike, e.g. learning analytics support self-regulated learning or adaptation of the learning material. Besides technologies, methods help create novel TEL opportunities. Methods come from different fields, such as education, psychology, or medicine, and from diverse communities, such as Makers Communities and Design Communities. Methods and technologies are also used to investigate and enhance learning for “fragile users”, such as children, elderly people, or people with special needs. Both the 10th edition of this conference and its related workshops (i.e. IEETeL, Nursing, SPeL, TEL4FC) contribute to novel research in TEL and expand the topics of the previous editions. The mis4TEL 2020 papers discuss how diverse methods or technologies are employed to create novel approaches to TEL, valuable TEL experiences, or innovative TEL solutions, taking a critical stance and promoting innovation. This volume presents all papers that were accepted for the main track of mis4TEL 2020, while the workshop papers are published in a different volume. All underwent a peer review selection: each paper was assessed by three different reviewers, from an international panel composed of about 40 members of 14 countries. The programme of mis4TEL 2020 counts 26 contributions from diverse countries. The quality of submissions was on average good, with an acceptance rate of approximately 70%.

v

vi

Preface

We thank the sponsor, the Armundia Group (https://www.armundia.com/), the support from national associations (AEPIA, APPIA, CINI, and EurAI), the members of the Local Organisation Team, and the Program Committee Members for their hard work, which was essential for the success of MIS4TEL’20. Pierpaolo Vittorini Tania Di Mascio Laura Tarantino Marco Temperini Rosella Gennari Fernando De la Prieta

Organisation of MIS4TEL 2020

http://www.mis4tel-conference.net/

General Co-chairs Pierpaolo Vittorini Tania Di Mascio

University of L’Aquila, L’Aquila, Italy University of L’Aquila, L’Aquila, Italy

Technical Program Co-chairs Laura Tarantino Marco Temperini

University of L’Aquila, L’Aquila, Italy Sapienza University, Rome, Italy

Paper Co-chairs Rosella Gennari Ricardo Azambuja Silveira Elvira Popescu

Free University of Bozen-Bolzano, Italy Universidade Federal de Santa Catarina, Brazil University of Craiova, România

Proceedings Co-chairs Fernando De la Prieta Ana Belén Gil

University of Salamanca, Spain University of Salamanca, Spain

Publicity Chairs Alessandra Melonio Demetrio Arturo Ovalle Carranza Nestor Dario Duque Mendes

Free University of Bozen-Bolzano, Italy National University of Colombia, Colombia National University of Colombia, Colombia vii

viii

Organisation of MIS4TEL 2020

Workshop Chair Zuzana Kubincova

Comenius University in Bratislava, Slovakia

Program Committee Ana Almeida Juan M. Alberola Ricardo Azambuja Silveira Davide Carneiro Maiga Chang Vincenza Cofini Giovanni De Gasperis Fernando De La Prieta Tania Di Mascio Dalila Duraes Ana Faria Florentino Fdez-Riverola Margarida Figueiredo Rosella Gennari Ana Belén Gil González Jorge Gomez-Sanz Sérgio Gonçalves Vicente Julian Zuzana Kubincová Luigi Laura Matteo Lombardi Constantino Martins Anna Mavroudi Alessandra Melonio Juan-José Mena-Marcos Marcelo Milrad Besim Mustafa Kyparissia Papanikolaou Carlos Pereira Gerlane R. F. Perrier Elvira Popescu Kasper Rodil Sara Rodríguez Juan M. Santos

ISEP-IPP, Portugal Universitat Politècnica de València, Spain Universidade Federal de Santa Catarina, Brazil Polytechnic of Porto, Portugal Athabasca University, Canada University of L’Aquila, Italy University of L’Aquila, Italy University of Salamanca, Spain University of L’Aquila, Italy University of Madrid, Spain ISEP, Portugal University of Vigo, Spain University of Évora, Portugal Free University of Bozen-Bolzano, Italy University of Salamanca, Italy Universidad Complutense de Madrid, Spain University of Minho, Portugal Universitat Politècnica de València, Spain Comenius University in Bratislava, Slovakia International Telematic University Uninettuno, Italy Griffith University, Australia Polytechnic of Porto, Portugal Norwegian University of Science and Technology, Norwegian Free University of Bozen-Bolzano, Italy University of Salamanca, Spain Linnaeus University, Sweden Edge Hill University, UK School of Pedagogical and Technological Education, Greece ISEC, Portugal Universidade Federal Rural de Pernambuco, Brazil University of Craiova, Romania Aalborg University, Denmark University of Salamanca, Spain University of Vigo, Spain

Organisation of MIS4TEL 2020

Olga Santos Antonio J. Sierra Martin Stabauer Andrea Sterbini Laura Tarantino Marco Temperini Sonia Verdugo-Castro Henrique Vicente Pierpaolo Vittorini

ix

aDeNu Research Group, (UNED), Spain University of Seville, Spain Johannes Kepler University Linz, Austria University of Rome, La Sapienza, Italy University of L’Aquila, Italy University of Rome, La Sapienza, Italy University of Salamanca, Spain University of Évora, Portugal University of L’Aquila, Italy

Local Organising Committee Pierpaolo Vittorini Tania Di Mascio Giovanni De Gasperis Federica Caruso Alessandra Galassi

University University University University University

of of of of of

L’Aquila, L’Aquila, L’Aquila, L’Aquila, L’Aquila,

L’Aquila, L’Aquila, L’Aquila, L’Aquila, L’Aquila,

Italy Italy Italy Italy Italy

Organising Committee Juan M. Corchado Rodríguez Fernando De la Prieta Sara Rodríguez González Javier Prieto Tejedor Pablo Chamoso Santos Belén Pérez Lancho Ana Belén Gil González Ana De Luis Reboredo Angélica González Arrieta Emilio S. Corchado Rodríguez Angel Luis Sánchez Lázaro Alfonso González Briones Yeray Mezquita Martín Enrique Goyenechea Javier J. Martín Limorti Alberto Rivas Camacho Ines Sitton Candanedo Elena Hernández Nieves Beatriz Bellido María Alonso

University of Salamanca, Institute, Spain University of Salamanca, University of Salamanca, University of Salamanca, Institute, Spain University of Salamanca, University of Salamanca, University of Salamanca, University of Salamanca, University of Salamanca, University of Salamanca,

Spain, and AIR Spain Spain Spain and AIR Spain Spain Spain Spain Spain Spain

University of Salamanca, Spain University Complutense of Madrid, Spain University of Salamanca, Spain University of Salamanca, Spain, and AIR Institute, Spain University of Salamanca, Spain University of Salamanca, Spain University of Salamanca, Spain University of Salamanca, Spain University of Salamanca, Spain University of Salamanca, Spain

x

Diego Valdeolmillos Roberto Casado Vara Sergio Marquez Jorge Herrera Marta Plaza Hernández Guillermo Hernández González Luis Carlos Martínez de Iturrate Ricardo S. Alonso Rincón Javier Parra Niloufar Shoeibi Zakieh Alizadeh-Sani

Organisation of MIS4TEL 2020

AIR Institute, Spain University of Salamanca, University of Salamanca, University of Salamanca, University of Salamanca, AIR Institute, Spain University of Salamanca, Institute, Spain University of Salamanca, University of Salamanca, University of Salamanca, University of Salamanca,

Spain Spain Spain Spain

Spain, and AIR Spain Spain Spain Spain

Contents

Effects of Time in Virtual Reality Learning Environments Linked with Materials Science and Engineering . . . . . . . . . . . . . . . . . . . . . . . . . J. Extremera, D. Vergara, M. P. Rubio, L. P. Dávila, and F. De la Prieta HEMOT®, Helmet for EMOTions: A Web Application for Children on Earthquake-Related Emotional Prevention . . . . . . . . . . Giada Vicentini, Margherita Brondino, Roberto Burro, and Daniela Raccanello Social Video Learning – Creation of a Reflection-Based Course Design in Teacher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Eric Tarantini Engaging Pre-teens in Ideating and Programming Smart Objects Through Play . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rosella Gennari, Maristella Matera, Alessandra Melonio, Mehdi Rizvi, and Eftychia Roumelioti Advanced Placement Physics Exam Performance of High School Graduates in Mexico with the Aid of Online Assignments Designed in Open-EdX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Ernesto M. Hernández, Rubén D. Santiago, José A. Otero, and Ma. de Lourdes Quezada-Batalla Hand Robotics Rehabilitation in Patients with Multiple Sclerosis: A Pilot Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Marco Tramontano, Laura Casagrande Conti, Niccolò Marziali, Giorgia Agostini, Sara De Angelis, Giovanni Galeoto, and Maria Grazia Grasso

1

10

20

31

41

50

xi

xii

Contents

When Something Useful Is Also Enjoyable: An Empirical Study on the Intention to Use Web-Based Simulations in Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Leonardo Caporarello, Federica Cirulli, Federico Magni, and Beatrice Manzoni TEL Adoption in the Riconnessioni Project . . . . . . . . . . . . . . . . . . . . . . Marcello Enea Newman

58

68

Cognitive Complexity Analysis of Learning-Related Texts: A Case Study on School Textbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . Syaamantak Das, Shyamal Kumar Das Mandal, and Anupam Basu

74

Lessons Clustering Using Topics Inferred by Unsupervised Modeling from Textbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Matías Altamirano, Abelino Jiménez, and Roberto Araya

85

Automatic Content Analysis of Computer-Supported Collaborative Inquiry-Based Learning Using Deep Networks and Attention Mechanisms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Pablo Uribe, Abelino Jiménez, Roberto Araya, Joni Lämsä, Raija Hämäläinen, and Jouni Viiri

95

The Impact of Personality, Attitude and Visual Decision-Making Dashboard Tools on the Learning Engagement of Economist Students . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Liana Stanca, Cristina Felea, Romeo Stanca, and Mirela Pintea HRS-EDU: Architecture to Control Social Robots in Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 John Páez, Enríque González, and Maria Impedovo Authoring Interactive Videos for e-Learning: The ELEVATE Tool Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Daniele Dellagiacoma, Paolo Busetta, Artem Gabbasov, Anna Perini, and Angelo Susi Early Detection of Gender Differences in Reading and Writing from a Smartphone-Based Performance Support System for Teachers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Roberto Araya An Assessment of Students’ Satisfaction in Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Margarida Figueiredo, Ana Fernandes, Jorge Ribeiro, José Neves, Almeida Dias, and Henrique Vicente Methodological Guidelines to Build Collaborative Serious Games Based on Intelligent Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 Oscar M. Salazar, Santiago Álvarez, and Demetrio A. Ovalle

Contents

xiii

An Augmented Reality-Based mLearning Approach to Enhance Learning and Teaching: A Case of Study in Guadalajara . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Janet Pigueiras, Angel Ruiz-Zafra, and Rocio Maciel Supporting the Construction of Learning Paths in a Competency-Based Informatics Curriculum . . . . . . . . . . . . . . . . . . 185 Luca Forlizzi, Giovanna Melideo, and Cintia Scafa Urbaez Vilchez Personalized Recommender System Using Learners’ Metacognitive Reading Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 Lydia Odilinye and Fred Popowich Technology-Enhanced Learning (TEL) in Anaesthesia: Ultrasound Simulation Training for Innovative Locoregional Anaesthesia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206 Vincenza Cofini, Pierpaolo Vittorini, Emiliano Petrucci, Stefano di Carlo, Pierfrancesco Fusco, Franco Marinangeli, and Stefano Necozione Profiles in Brain Type in Programming Performance for Non-vocational Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 Ugo Solitro, Margherita Brondino, Roberto Bonafini, and Margherita Pasini Towards a Gamified Musical Skill Learning Model (MuS-LM): Structural Aspects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 Tania Di Mascio, Laura Tarantino, and Federica Caruso Philosophical Approaches to Smart Education and Smart Cities . . . . . . 239 Javier Teira-Lafuente, Ana B. Gil-González, and Ana de Luis Reboredo Experiences in a Differential Equations Massive Course . . . . . . . . . . . . 249 Rubén Dario Santiago Acosta, María de Lourdes Quezada Batalla, and Ernesto Manuel Hernández Cooper A Protocol for Simulated Experimentation of Automated Grading Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 Andrea Sterbini, Marco Temperini, and Pierpaolo Vittorini Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

Effects of Time in Virtual Reality Learning Environments Linked with Materials Science and Engineering J. Extremera1

, D. Vergara2(&) , M. P. Rubio1 and F. De la Prieta1

, L. P. Dávila3

,

1

3

University of Salamanca, Salamanca, Spain {jamil.extremera,mprc,fer}@usal.es 2 Catholic University of Ávila, Ávila, Spain [email protected] University of California at Merced, Merced, CA, USA [email protected]

Abstract. The increasing presence of virtual reality learning environments (VRLEs) in university classrooms makes it necessary to study what factors influence the effectiveness of this type of teaching tool. In particular, when planning to use a VRLE in class to support the classes, a careful design of the application to achieve a high level of efficiency at the formative level must be carried out. This article discusses key aspects that need to be taken into account during the design of a VRLE that have been determined to be increasingly important for students to achieve a higher level of meaningful learning (and, thanks to it, the knowledge acquired through the use of the VRLE will last in their memory for a longer time) and also feel a greater motivation to use it to: (i) adapt both the level of interactivity as well as the way the VRLE conducts the student through the virtual experiment; and (ii) maintain a look and a handling mode of the VRLE similar to other virtual environments that exist at the present time (e.g. video games). The study carried out and described in this article highlights the effectiveness of using in certain cases a step-by-step guidance protocol to improve long-term learning of concepts under study. In addition, the importance of using modern development tools to achieve a high level of motivation among students is emphasized. Keywords: Virtual reality learning environments engineering
Meaningful learning


Materials science and

1 Introduction The exponential improvement experienced by computer processors in the 21st century is easily observable in all areas of life, having been accompanied by the continuous evolution of all types of hardware (such as memory units, sensors or display devices, to name a few examples). Furthermore, this technological evolution has been followed by the development of many research works that have resulted in new realities hardly imaginable by the general public a few decades ago, such as multi-agent systems [1–3], © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 1–9, 2020. https://doi.org/10.1007/978-3-030-52538-5_1

2

J. Extremera et al.

multi-core processing [4] or advanced image processing [5]. In this context, virtual reality has also experienced a great development and expansion in a multitude of fields, including university education through virtual reality learning environments (VRLEs), which are being used in a large number of disciplines. In the particular case of materials science and engineering (MSE), VRLEs that can be found are focused on helping students to learn about the realization of different tests of materials such as: tensile [6, 7], compression [8], impact [9, 10], hardness [9, 10], and non-destructive [11, 12]. The advantages of using VRLEs to support MSE teaching have been reported in different studies [6, 13, 14], including those describing that: (i) the problem of classroom congestion during training classes is minimized; (ii) experiments whose conduct in a real laboratory would be impossible due to their high price or hazard can be simulated; (iii) detailed visualization of the elements involved in experiments that are often difficult or impossible to see in a real laboratory can be discerned in a VRLE; and (iv) encouragement of students to learn the study subject is a key benefit. The design of a VRLE plays a decisive role in the effectiveness that it will have by improving the teaching-learning process for which it will be used [15–17]. Vergara et al. [15] reported that there is a direct relationship between the design of a virtual teaching tool and the motivation that it generates on the user to continue using it. An important parameter for measuring the effectiveness of a VRLE is the level of meaningful learning that students achieve through its use. Meaningful learning is a concept that refers to the idea that an acquired knowledge is fully understood by an individual, who can connect it with another knowledge previously acquired. The authors of this article have found that not all VRLEs, despite being attractive and motivating, achieve the same degree of meaningful learning. Thus, this article compares different VRLEs designs applied to MSE to elucidate which parameters should be considered to achieve a high level of meaningful learning, being remarkable the use of a guidance protocol to help students to conduct the virtual experiments. The results and conclusions obtained in this study can be taken into account in the creation of VRLEs dedicated to the teaching of various university courses in the sciences and engineering fields.

2 Design Considerations of VRLEs 2.1

Design of Guidance Protocol

The process of creating VRLEs has been described in previous work [18] and consists of the following steps: (i) determine the level of realism necessary to achieve the objectives of the VRLE; (ii) establish the level of interactivity; (iii) choose the software and hardware that best suits the development and use needs of the VRLE; (iv) develop the VRLE, which in turn consists of 3D modeling and interactivity programming; and (v) test the application with a pilot group of users and apply the feedback obtained to modify the VRLE. However, the authors have found that the process described above does not ensure in all cases that students reach an adequate level of meaningful learning. To solve this problem, the authors propose to apply a step-by-step guidance protocol on those VRLE

Effects of Time in Virtual Reality Learning Environments

3

that seeks to train students in conducting experiments in real laboratories. This step-bystep protocol occurs as VRLEs: • Offer a sufficient level of interactivity to carry out the virtual experiment in a motivating and effective way at the formative level. This means that a very low level of interactivity does not allow the user to interact with the VRLE enough to retain knowledge or keep him motivated. On the other hand, a too high level of interactivity can result in the user losing the thread of the experiment, negatively impacting on motivation. • Direct the user what is the next action that shall be taken, as well as provide information about how to do it. • Prevent the user from taking unnecessary actions to perform the experiment or actions that may ruin it. The use of a step-by-step guidance protocol like the one described above allows students to focus on understanding each stage of the experiment without having to invest a large amount of time in learning how to use the VRLE [19]. Figure 1 shows the process of creating a VRLE when a step-by-step guidance protocol is incorporated. In this process (Fig. 1) it can be observed that the level of interactivity that the authors suggest to use depends on the objective of the VRLE, that is: (i) when the objective of the VRLE is to help to understand a theoretical concept, the level of interactivity should be within a range from a step-by-step guidance system (therefore, restricted interactivity) to an open world (plenty of freedom of action); or (ii) when the VRLE is used to teach a laboratory experiment, a step-by-step guidance system should be implemented.

Fig. 1. VRLE creation process considering the implementation of a step-by-step guidance.

4

2.2

J. Extremera et al.

Obsolescence Effects on Students’ Motivation

As noted above, the fourth stage of the process of creating a VRLE consists of two different activities [16, 20]: 3D modeling and interactivity programming. The 3D modeling gathers the activities related to the conceptualization and creation of the 3D elements that form the virtual environment, i.e.: laboratory, instruments, machines, lighting, etc. The virtual environment will then be visualized by the user, either on a computer monitor, on a head-mounted display (HMD) or another type of system as cave automatic virtual environment (CAVE) [21, 22]. There are different programs to carry out the modeling tasks, highlighting among them 3DS Max, Maya, Blender or Cinema 4D. On the other hand, the programming of interactivity consists in providing the 3D environment with the possibility to be manipulated by users, so they can interact with it: grabbing objects, using machines, moving around the scene, etc. This activity is carried out by using specific programs known as game engines, which are tools that were originally developed to program video games. Different game engines can be used to program the interactivity of a VRLE, such as Unreal Engine 4, Unity or CryEngine. Note that this type of VRLE do not incorporate a machine back-end to follow the user activity. Over the last few years, the authors have developed different VRLEs applied to the MSE using different technologies [6, 8, 11, 15, 23–26]. Figure 2a shows the VRLEs created approximately 6 years ago, using development tools of that time. These applications are currently perceived by students as less motivating as they offer an “outdated” look: unrealistic materials and lighting, interactivity restricted to keyboard and mouse, or impossibility to easily adapt the applications to immersive virtual reality. On the other hand, Fig. 2b shows newly created VRLEs using current development tools. The new VRLEs (Fig. 2b) offer an aspect and interactivity in line with the possibilities offered by modern virtual reality technology: realistic-looking environments, lighting based on physical equations, different possibilities of interaction or the possibility of easily adapting applications to be used with immersive virtual reality, among others.

Fig. 2. Virtual reality learning environments: (a) created 6 years ago with former development tools and others (b) created recently with current development tools.

Effects of Time in Virtual Reality Learning Environments

5

3 Meaningful Learning Analysis During the conduct of an experiment in a real MSE laboratory, it is common that a single equipment is being utilized by a large group of students. It is expected that this circumstance will negatively affect the teaching-learning process of the experiment [8], and for this reason, the use of a VRLE is preferred. Consequently, the authors used VRLEs as shown in Fig. 2a, but found that one year after using them, there was a large number of students who did not remember how to perform the experiments that they simulated with VRLEs. Thus, the analysis of the data obtained in this study is intended to elucidate the main factors that explain this fact. 3.1

Methodology

This study was carried out at the Catholic University of Ávila (Spain), during the courses between 2015 and 2020, participating every year approximately 20 MSE students of the degree in mechanical engineering. The methodology used is summarized in the following steps: • The instructor teaches the theory about the operation of the simulated machine in the VRLE to perform the virtual experiment. • The use of a VRLE in the classroom should be under the supervision of the teacher. Also, students can continue using the VRLE without restrictions out of school hours. • Resolution of exercises in groups of 2–3 students, either in VRLE itself or on paper. • One year after the previous three steps have been completed, students answer to technical questionnaires to assess the level of knowledge retained. During the years 2015 to 2018, VRLEs designed without a step-by-step guidance protocol were evaluated; these VRLEs were used a year earlier (i.e. these VRLEs were used in class between 2014 and 2017). Moreover, in 2019 and 2020 VRLEs designed according to Fig. 1 (including a step-by-step guidance protocol) were evaluated and developed with current development tools (Fig. 2b); those latter VRLEs were used subsequently in 2018 and 2019. 3.2

Results

As described above, students were surveyed one year after using VRLEs to assess the degree of knowledge they still retained. Figure 3 shows the data obtained from the surveys filled out by the 120 students who participated in the study (approximately 20 students per course). The resulting bar graph (Fig. 3) shows the average of the questionnaire questions answered correctly each course (accuracy rate), indicating the level of knowledge that students remember about the MSE content they learned a year earlier with the help of VRLEs. Furthermore, Fig. 3 shows that, between 2015 and 2018, were evaluated VRLEs designed without a step-by-step guidance protocol (i.e. developed several years ago) while in 2019 and 2020 were evaluated VRLEs designed according to the scheme of Fig. 1 (i.e. developed more recently, which include a step-by-step guidance protocol).

6

J. Extremera et al.

Fig. 3. Accuracy rate of questionnaires answered by students who used the VRLEs a year earlier as a support to learn fundamental concepts in MSE.

4 Discussion The continuous evolution of information and communications technology (ICT) brings with it an accelerated virtual reality technology improving. This rapid evolution implies that VRLEs, although at the time of their creation are well-valued by students after a few years are perceived as obsolete and less motivating. The experience of the authors indicates that students welcome updates such as those described in this article, which indicates that regular updates of VRLEs using modern development tools favors that these teaching applications can maintain motivation among students (thus achieving that VRLEs do not lose their effectiveness at the formative level over time). However, as discussed below, periodic updates to VRLEs do not ensure by themselves alone a significant improvement in meaningful learning. It is impossible to achieve that a group of students remember 100% of the content learned a year ago. As noted in Fig. 3, the percentage of knowledge retained varies little from course to course between 2015 and 2018. However, in 2019 and 2020 the success rate increased by approximately 30% compared to the period 2015–2018. Considering that the contents taught in the MSE subject and questionnaires used to evaluate the retained knowledge were almost identical every year, this improvement in the level of retained knowledge (30% higher than previous years) can be due to two factors: the use of updated development tools or the implementation of a step-by-step guidance protocol. The technical improvement of VRLEs when they are updated helps students to be motivated to use them and focus on the concepts being taught, which favors some improvement of meaningful learning.

Effects of Time in Virtual Reality Learning Environments

7

However, based on the experience of the authors and previous studies [6, 8, 11, 12, 15, 27, 28], students’ motivation to use this type of teaching resources is usually high. Consequently, updating VRLEs with current development tools cannot be the only factor that explains the increase of 30% in the number of questions correctly answered in the surveys of 2019 and 2020. In fact, in the authors’ opinion, the key factor that explains the improvement in meaningful learning lies in the implementation of the stepby-step guidance protocol (Fig. 1). This is in line with previous studies [29] in which the effectiveness of step-by-step guidance protocols has been tested in teaching tools based on the audio-visual use of e-books for MSE teaching. Nevertheless, further research works should be conducted to measure the level of influence on the meaningful learning of both factors considered in the present paper: the use of recent development tools and the implementation of a guidance protocol. In particular, new studies could be based on fixing one of the factors and varying the other one, considering the same questionnaires described in the present paper to assess the level of meaningful learning. These future research works should: (i) evaluate both a group of VRLEs with guidance and another similar group of VRLEs without guidance protocol–both groups of VRLEs should be developed with the same development tools–; and (ii) compare the evaluation of a group of VRLEs developed with modern tools and another similar group of VRLEs developed with older tools–both groups of VRLE should lack a guidance protocol–.

5 Conclusions The design of a VRLE is directly related to its effectiveness as an educational tool. Periodic updates of a VRLE by using current development tools lead to a modern aspect of this type of application, that helps maintain the motivation of students to use it. Besides, updating a step-by-step protocol in those VRLEs dedicated to simulate laboratory experiments allows students to achieve a higher level of meaningful learning, thereby retaining the acquired knowledge for a longer time.

References 1. Tapia, D.I., Fraile, J.A., Rodríguez, S., Alonso, R.S., Corchado, J.M.: Integrating hardware agents into an enhanced multi-agent architecture for ambient intelligence systems. Inf. Sci. 222, 47–65 (2013) 2. García, E., Rodríguez, S., Martín, B., Zato, C., Pérez, B.: MISIA: middleware infrastructure to simulate intelligent agents. In: De Paz Santana, J.F. (ed.) International Symposium on Distributed Computing and Artificial Intelligence, AISC, vol. 91. Springer, Heidelberg (2011) 3. Rodríguez, S., De la Prieta, F., Tapia, D.I., Corchado, J.M.: Agents and computer vision for processing stereoscopic images. In: Corchado, E., et al. (eds.) Hybrid Artificial Intelligence Systems, HAIS 2010. LNAI, vol. 6077. Springer, Berlin, Heidelberg (2010) 4. Li, T., Sun, S., Bolić, M., Corchado, J.M.: Algorithm design for parallel implementation of the SMC-PHD filter. Signal Process. 119, 115–127 (2016)

8

J. Extremera et al.

5. Chamoso, P., Rivas, A., Martín-Limorti, J.J., Rodríguez, S.: A hash based image matching algorithm for social networks. In: De la Prieta, F., et al. (eds.) Trends in Cyber-Physical Multi-Agent Systems. The PAAMS Collection-15th International Conference, PAAMS 2017. AISC, vol. 619, pp. 183–190. Springer, Cham (2018) 6. Vergara, D., Rubio, M.P., Prieto, F., Lorenzo, M.: Enhancing the teaching-learning of materials mechanical characterization by using virtual reality. J. Mater. Educ. 38(3–4), 63– 74 (2016) 7. Dobrzanski, L.A., Jagiełło, A., Honysz, R.: Virtual tensile test machine as an example of material science virtual laboratory post. J. Achiev. Mater. Manuf. Eng. 27, 207–210 (2008) 8. Vergara, D., Rubio, M.P., Lorenzo, M.: New approach for the teaching of concrete compression tests in large groups of engineering students. J. Prof. Issues. Eng. Educ. Pract. 143(2), 05016009 (2017) 9. Dobrzanski, L.A., Honysz, R.: Building methodology of virtual laboratory posts for materials science virtual laboratory purposes. Arch. Mater. Sci. Eng. 28, 695–700 (2007) 10. Dobrzanski, L.A., Honysz, R.: On the implementation of virtual machines in computer aided education. J. Mater. Educ. 31(1–2), 131–140 (2009) 11. Vergara, D., Rubio, M.P.: The application of didactic virtual tools in the instruction of industrial radiography. J. Mater. Educ. 37(1–2), 17–26 (2015) 12. Vergara, D., Rodríguez-Martín, M., Rubio, M.P., Ferrer-Marín, J., Núñez-García, F.J., Moralejo-Cobo, L.: Technical staff training in ultrasonic non-destructive testing using virtual reality. Dyna 93(2), 150–154 (2018) 13. Omieno, K., Wabwoba, F., Matoke, N.: Virtual reality in education: trends and issues. Int. J. Comput. Technol. 4(1), 38–43 (2013) 14. Martín-Gutiérrez, J., Mora, C.E., Añorbe-Díaz, B., González-Marrero, A.: Virtual technologies trends in education. Eurasia J. Math. Sci. Technol. Educ. 13(2), 469–486 (2017) 15. Vergara, D., Rubio, M.P., Lorenzo, M.: A virtual resource for enhancing the spatial comprehension of crystal lattices. Educ. Sci. 8(4), 153 (2018) 16. Vergara, D., Rubio, M.P., Lorenzo, M., Rodríguez, S.: On the importance of the design of virtual reality learning environments. Adv. Intell. Syst. Comput. 1007, 146–152 (2020) 17. Violante, M.G., Vezzetti, E.: Virtual interactive e-learning application: an evaluation of the student satisfaction. Comput. Appl. Eng. Educ. 23(1), 72–91 (2015) 18. Vergara, D., Rubio, M.P., Lorenzo, M.: On the design of virtual reality learning environments in engineering. Multimodal. Technol. Interact. 1(2), 11 (2017) 19. Rubio, M.P., Vergara, D., Rodríguez, S., Extremera, J.: Virtual reality learning environments in materials engineering: rockwell hardness test. In: Di Mascio, T., Vittorini, P., Gennari, R., Prieta, F., Rodríguez, S., Temperini, M., Azambuja, R., Popescu, E., Lancia, L., Silveira, R. A. (eds.) Methodologies and Intelligent Systems for Technology Enhanced Learning 8th International Conference MIS4TEL 2018. Advances in Intelligent Systems and Computing, pp. 106–113. Springer, Cham (2019) 20. Ren, S., McKenzie, F.D., Chaturvedi, S.K., Prabhakaran, R., Yoon, J., Katsioloudis, P.J., Garcia, H.: Design and comparison of immersive interactive learning and instructional techniques for 3D virtual laboratories. Presence Teleoper. Virtual Environ. 24(2), 93–112 (2015) 21. Wolfartsberger, J.: Analyzing the potential of virtual reality for engineering design review. Autom. Constr. 104, 27–37 (2019) 22. Muhanna, M.A.: Virtual reality and the CAVE: taxonomy, interaction challenges and research directions. J. King Saud Univ. Comput. Inf. Sci. 27(3), 344–361 (2015) 23. Vergara, D., Rubio, M.P., Lorenzo, M.: New virtual application for improving the students’ understanding of ternary phase diagrams. Key Eng. Mater. 572, 578–581 (2014)

Effects of Time in Virtual Reality Learning Environments

9

24. Vergara, D., Rubio, M.P., Lorenzo, M.: Interactive virtual platform for simulating a concrete compression test. Key Eng. Mater. 572, 582–585 (2014) 25. Extremera, J., Vergara, D., Rubio, M.P., Gómez, A.I.: Design of virtual reality learning environments: step-by-step guidance. In: ICERI 2019 Proceedings, pp. 1285–1290. IATED, Valencia (2019) 26. Vergara, D., Sánchez, M., Garcinuño, A., Rubio, M.P., Extremera, J., Gómez, A.I.: Spatial comprehension of crystal lattices through virtual reality applications. In: ICERI 2019 Proceedings, pp. 1291–1295. IATED, Valencia (2019) 27. Meagher, K.A., Doblack, B.N., Ramirez, M., Davila, L.P.: Scalable nanohelices for predictive studies and enhanced 3D visualization. J. Vis. Exp. 93, 51372 (2014) 28. Vergara, D., Lorenzo, M., Rubio, M.P.: Virtual environments in materials science and engineering: The students’ opinion. In: Lim, H. (ed.) Handbook of Research on Recent Developments in Materials Science and Corrosion Engineering Education, 1st ed. pp. 148– 165. IGI Global, Hershey (2015) 29. Flores, C., Matlock, T., Davila, L.P.: Enhancing materials research through innovative 3D environments and interactive manuals for data visualization and analysis. Mater. Res. Soc. Symp. Proc. 1472, 29–38 (2012)

HEMOT®, Helmet for EMOTions: A Web Application for Children on Earthquake-Related Emotional Prevention Giada Vicentini(&)

, Margherita Brondino and Daniela Raccanello

, Roberto Burro

,

Department of Human Sciences, University of Verona, 37129 Verona, Italy {giada.vicentini,margherita.brondino,roberto.burro, daniela.raccanello}@univr.it

Abstract. The integration between technological devices and educational programs can improve students’ learning experience. This is also true when referring to psychological interventions with children to foster emotional competence in relation to disasters. Therefore, within an interdisciplinary project (PrEmT project, Emotional Prevention and Earthquakes in primary school, in Italian Prevenzione Emotiva e Terremoti nella scuola primaria) we developed the web application HEMOT® (Helmet for EMOTions). HEMOT® includes ten levels aimed at promoting knowledge about earthquakes, safety behaviors, emotions, and emotion regulation strategies. We involved 67 second and fourth-graders in a ten-unit training: During each unit they used a level of the web application and participated to traditional activities. The aim of this study was to explore individual differences in perceived usability and performance using HEMOT®, running Latent Class Growth Analyses. We found two different latent groupings for both usability and performance. However, most of the children gradually perceived a decreasing difficulty using HEMOT® and obtained a good performance across the levels. To sum up, this study described two relevant aspects of a web application developed ad hoc to promote earthquake-related emotional prevention with children. On the whole, HEMOT® respects accessibility and usability criteria and its contents are not too demanding for children: It could be considered a useful and fun method to promote earthquake-related emotional prevention. Keywords: Web application
Emotion knowledge strategies
Emotional prevention
Earthquakes


Emotion regulation

1 Introduction Every aspect of human life, including education, has been affected by technological change. Within educational environments, the Information and Communication Technologies (ICT) have an increasingly relevant role, in light of their documented effectiveness for learning achievement [15]. The technology-enhanced learning (TEL) systems represent a challenge for students and teachers: To improve students’ learning experience and achievement, ICT must be integrated with educational strategies [3, 17]. Using mobile devices to promote formal and informal learning is more effective © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 10–19, 2020. https://doi.org/10.1007/978-3-030-52538-5_2

HEMOT®, Helmet for EMOTions: A web application

11

than using traditional teaching methodologies (e.g., pencil-and-paper and desktop computers activities) [15]. The educational programs supported by mobile devices have the advantage to foster students’ motivation, engagement, autonomy, problem solving, and creativity [5, 17]. The benefits of ICT can be reaped also through the conduction of psychological interventions aimed at promoting emotional competences [11]. The emotional competence can be defined as the ability to understand, express, and regulate emotions [4]. The acquisition of this skill occurs gradually from childhood to adolescence and it represents a fundamental basis of the psychological development [4]. However, only a few studies demonstrated the effectiveness of interventions using different hardware and software in promoting abilities related to emotional competence, with both clinical and non-clinical children [11]. As regards preparedness to disasters, some mobile applications focused on earthquakes, without considering specifically emotional competence [11]. Among the applications for android devices, we can distinguish gamebased vs. lecture-based applications [11, 15]. On the one hand, game-based learning applications have the aim to promote earthquake-related knowledge while having fun; these applications address both children and adults and present the contents in a very interactive setting (e.g., dialogue between animated characters). On the other hand, lecture-based applications foster earthquake-related knowledge in a more traditional way, presenting the information with a content-focused methodology (e.g., written text). All these applications are useful to learn characteristics of earthquakes (e.g., typical events and sounds) and safety behaviors, but they treat only marginally emotional preparedness [11]. When developing an application, it is important to take into account its accessibility and usability, and this is particularly true when users are children [6]. Accessibility is an objective property that refers to the matching between the individual’s abilities and the characteristics of the environment: In the technological context, an environment is accessible if it permits an independent fruition to all the users, also with disabilities and impairments [18]. Usability is a more subjective construct which derives from the individual’s interpretation: A usable environment enables users to achieve their goals effectively, efficiently, and with satisfaction [6]. Within psychological research, usability can also be considered as the difficulty experienced using a digital device to complete a task [9]. The reasons underlying usability regard mainly four categories [9, 14]: user, task, tool, and environment. Specifically, the perceived difficulty could relate to user’s behavioral or psychological aspects (e.g., elicited emotions), characteristics of the task (e.g., demanding questions), functioning of the device (e.g., connectivity issues), and/or setting (e.g., confusion) [9]. 1.1

The Development of HEMOT®

This study is part of the pilot phase of an evidence-based interdisciplinary project, i.e., the PrEmT project (Emotional Prevention and Earthquakes in primary school, in Italian Prevenzione Emotiva e Terremoti nella scuola primaria) [11, 12, 16]. The PrEmT project is the first step of the HEMOT® project (Helmet for EMOTions, https://www. hemot.eu), aiming at promoting emotional competence in relation to different types of disasters. Within the PrEmT project we developed ad hoc the web application HEMOT® (patent pending), fostering knowledge concerning earthquakes, emotions, and emotion regulation strategies [11, 16]. Differently from existing earthquake-related

12

G. Vicentini et al.

applications, this one combines behavioral and emotional prevention. HEMOT® addresses both children and adults and it has been created thanks to the collaboration between experts from different disciplines, such as psychology, geology, education, informatics, and illustration. It has both a front-office and a back-office structure. The web application includes ten levels: nine content levels and a final summary level. In its final version, the user will be able to go to the next level only reaching a minimum score. HEMOT® is a server-side software application based on ASP.NET Framework configurated for web servers like Internet Information Services (IIS) and Apache. A MS-sql relational database management system permits to store and retrieve data. Moreover, HEMOT® is built on a Responsive Web Design (RWD) so that the web page adequately fits with different devices, i.e., smartphones, tablets, laptops, and desktop computers. About the language, the current version of this application is in Italian, but its English translation is in progress. 1.2

The Present Study

This study has the aim to describe some steps in the development of the web application HEMOT® used with children to foster earthquake-related emotional preparedness. In particular, we focused on children’s individual differences in usability and performance using HEMOT®. We applied Latent Class Growth Analysis (LCGA) to identify unobserved groupings in longitudinal data. We had two specific aims. We examined differences in usability evaluated for each of the nine content levels of HEMOT® by identifying groups of children characterized by different patterns of perceived difficulty (Aim 1). We then examined differences in performance obtained in the nine content levels of HEMOT® by identifying groups of children characterized by different patterns of correct answers (Aim 2).

2 Method 2.1

Participants

The convenience sample included 28 second-graders (M = 7.70 years, SD = 0.29; 41% females) and 39 fourth-graders (M = 9.61 years, SD = 0.26; 54% females) from a primary school in Northern Italy, with no disabilities, coming from a wide range of socio-economic status. Twenty-three percent of the children had experienced at least one earthquake, without any damage. All the parents signed the informed consent form for the participation of their children. The study was approved by the Ethical Committee of the Department of Human Sciences of University of Verona (protocol n. 134535). 2.2

Material and Procedure

We used a longitudinal design, collecting the data between February and April 2019, during regular school lessons. The children participated to the training conducted within the PrEmT project, aimed at increasing their knowledge on earthquakes (e.g., characteristics and safety behaviors), emotions (e.g., facial expression and emotional

HEMOT®, Helmet for EMOTions: A web application

13

lexicon), and emotion regulation strategies (e.g., coping strategies useful during and right after an earthquake). During the ten units of the training, the participants used HEMOT® through technological devices (i.e., tablets and headphones) and participated to traditional activities (e.g., pencil-and-paper tasks). At the end of each level of HEMOT®, children had to rate its usability. Usability. We operazionalized usability of the web application in terms of difficulty. We asked the children to evaluate the usability of each of the nine content levels of HEMOT® with a single item (i.e., How much difficult was using this app?) on a 5-point scale (1 = not at all, 5 = very much). Web Application. HEMOT® is a web application consisting of nine content levels and a final summary level, about geological and emotional contents. In each level, there is a series of items (ranging from 24 to 48) comprising written sentences read by a digital voice, images and/or sounds (see Table 1), except for the last level. Each item had to be evaluated on a dichotomous response (0 = wrong answer, 1 = right answer). Summing all the levels, the maximum score was 296. Level 1. The title of the first level is “Nature of earthquakes”. It includes 32 written items; possible answers are “Yes” or “No”. Twenty-six items are associated to an image drawn ad hoc representing both internal and external contexts; six items are presented together with a sound. The items describe events that are or are not typical of an earthquake; the correct descriptions of earthquake-related events have been defined on the basis of the Modified Mercalli Intensity scale (MMI). Level 2. The title of the second level is “Safety behaviors”. It includes 36 written items; possible answers are “Yes” or “No”. Each item is associated to an image drawn ad hoc. Twenty-six items refer to behaviors to be implemented during an earthquake; ten items describe behaviors to be implemented right after an earthquake. The right and wrong behaviors have been defined using international guidelines. Level 3. The title of the third level is “Recognition of basic emotions”. It includes 36 written items; possible answers are “Yes” or “No”. Each item is associated to an image drawn ad hoc representing a facial expression (the images have been balanced for gender, ethnicity, and type of emotion). The task requires to recognize the right matching between the written sentence and the corresponding image, focusing on six emotions (i.e., calm, surprise, enjoyment, fear, sadness, and anger). Level 4. The title of the fourth level is “Use of emotional lexicon”. It includes 36 written items; possible answers are “Yes” or “No”. The items are not associated to images or sounds. The task requires to recognize the right matching between words with the same meaning; there are three synonyms for each of the six emotions. Level 5. The title of the fifth level is “Earthquake-related emotions”. It includes 36 written items; possible answers are “Yes” or “No”. Each item is associated to an image drawn ad hoc representing a facial expression (the images have been balanced for gender, ethnicity, and type of emotion). The task requires to recognize the valence of six emotions.

14

G. Vicentini et al.

Level 6. The title of the sixth level is “Intensity of emotions”. It includes 48 written items; possible answers are “Yes” or “No”. Each item is associated to an image drawn ad hoc representing two faces experiencing the same emotion with identical or different intensity (the images have been balanced for gender, ethnicity, type of emotion, and intensity). The task requires to recognize the faces representing the same intensity, for each of the emotions. Level 7. The title of the seventh level is “Intra and inter-personal emotion regulation strategies”. It includes 24 written items; possible answers are “Thinking” or “Doing” in the first part, and “Alone” or “With others” in the second part. Each item is associated to an image drawn ad hoc. The first 12 items refer to emotion regulation strategies to be implemented by thinking (cognitive strategies) or doing (behavioral strategies) something; the second 12 items refer to emotion regulation strategies to be implemented alone (intra-personal strategies) or with other people (inter-personal strategies). The task requires to recognize the right type of each item. Level 8. The title of the eighth level is “Strategies to be effectively used during earthquakes”. It includes 24 written items; possible answers are “Yes” or “No”. Each item is associated to an image drawn ad hoc. The items have been balanced for type of strategy (i.e., cognitive or behavioral), category (according to a classification comprising 12 categories [19]), and emotional elements (i.e., presence or absence of emotional elements in the description). Twelve items refer to adaptive emotion regulation strategies (e.g., support seeking); twelve items refer to maladaptive emotion regulation strategies (e.g., opposition) [10]. The task requires to recognize those strategies useful to reduce fear experienced during an earthquake. Level 9. The title of the ninth level is “Strategies to be effectively used after earthquakes”. It includes 24 written items; possible answers are “Yes” or “No”. Each item is associated to an image drawn ad hoc. As in the previous level, the items have been balanced for type of strategy, category, and emotional elements, and they present both adaptive and maladaptive strategies. The task requires to recognize those strategies useful to reduce fear, sadness, and anger experienced right after an earthquake. Level 10. The title of the tenth level is “Put the pieces back together”. The aim of this last level is to summarize the contents of the previous levels. The task requires to put together four puzzles representing images and written descriptions on: (1) correct safety behaviors during an earthquake, (2) correct safety behaviors right after an earthquake, (3) adaptive emotion regulation strategies during an earthquake, and (4) adaptive emotion regulation strategies right after an earthquake.

HEMOT®, Helmet for EMOTions: A web application

15

Table 1. Examples of items for the nine content levels of HEMOT® (adapted from [13]).

Questions

Examples of items [and right answer]

1. Nature of earthquakes

Does this happen during an earthquake?

The earth shakes. [Yes]

2. Safety behaviors

During an earthquake, is it correct to do this to get safe?

Looking for shelter under a table inside the house. [Yes]

3. Recognition of basic emotions

Does the word describe the face?

Is he happy? [Yes]

4. Use of emotional lexicon

Do the two words describe the same emotion?

Does being irritated mean being angry? [Yes]

5. Earthquakerelated emotions

How do you feel when you experience this emotion?

Do you feel good if you are calm? [Yes]

Do they experience the emotion at the same intensity?

Are they surprised in the same way? [Yes]

7. Intra and interpersonal emotion regulation strategies

To feel better in this way, are you alone or with others?

Paying attention to people who say to stay calm. [With others]

8. Strategies to be effectively used during earthquakes

How can people feel better during an earthquake?

Breathing deeply. [Yes]

9. Strategies to be effectively used after earthquakes

How can people feel better right away after an earthquake?

Talking about how they feel. [Yes]

Levels

6. Intensity of emotions

2.3

Examples of drawings

-

Analysis Procedure

We ran LCGAs using Mplus version 7 [7]. LCGAs use repeated measurements of observed variables as indicators of latent variables that describe specific characteristics of individual’s changes. Intercept and slope are considered two latent variables (called also random coefficients). In particular, the intercept indicates the level of the studied variable when time is equal to zero, and the slope represents the rate of change in the same variable over time [1, 2]. At first we ran growth curve models to define the best baseline model for usability and performance indicators. We compared two growth curves for each studied variable: a first curve with usability and performance measures repeated on the nine units as indicators and intercept and linear slope as higher order latent factors, and a second one adding a quadratic parameter. For performance we standardized all the scores. Then, we choose the best model on the basis of the number of latent classes and the best fitting parameters (linear vs. linear and quadratic). In order to compare the models, we used the information criteria and the fit indices. Furthermore, we considered parsimony and interpretability as relevant criteria, following

16

G. Vicentini et al.

recommendations from the literature (e.g., [8]). We checked the Bootstrap Likelihood Ratio Test (B-LRT). We also assessed: entropy values, to compare the degree of separation among the classes in the models, where scores closer to 1 highlight better fit of the data; the proportions for the latent classes (not less than 1% of total count in a class); and the posterior latent class probabilities (near to 1.00).

3 Results and Discussion 3.1

Children’s Usability of HEMOT®

At first we ran the analysis to compare the linear growth curve model with the one including also a quadratic parameter. The last one resulted the best, v2(36) = 46.41, p = .115; CFI = .91, RMSEA = .065 (.000–.115). We also examined the trajectories of the observed data, and the quadratic growth curve seemed the more appropriate and so it was retained. Then we conducted the analyses to determine the number of latent classes. We compared progressive unconditional models from one to three classes examining them on the basis of different elements. We rejected the three-class model because members of the two classes found in the previous model were mainly confirmed and in the third class only one person was present. B-LRT confirmed the twoclass model (p < .05). Entropy of the two-class solution was good (.84). The two-class model was characterized by a good class proportion (class 1: 31%; class 2: 69%) and good posterior probability estimates (class 1: .96; class 2: .94).

Fig. 1. Linear growth curves on usability of HEMOT® distinguishing two latent classes (Group 1: red; Group 2: blue).

In Fig. 1 we present the trajectories of the two groups. Across the nine content levels, children belonging to Group 1 (red in Fig. 1) gradually perceived as more difficult using HEMOT®, while children belonging to Group 2 (blue in Fig. 1) gradually perceived it as less difficult. In other words, children vary in their perception of

HEMOT®, Helmet for EMOTions: A web application

17

usability, showing two different tendencies. It is worth noting that Group 2 was more numerous. So, for most of the children, a higher use of HEMOT® probably increases familiarity with the web application, resulting in higher usability. However, for children in Group 1 the objectively increasing difficulty of the tasks could have played a more prominent role compared to familiarity. Task is indeed one of the most salient reasons underlying usability [9]. 3.2

Children’s Performance in HEMOT®

Again, we ran a first analysis to compare the linear growth curve model with the one including also a quadratic parameter. The linear one resulted the best model, v2(40) = 39.74, p = .480; CFI = 1.00, RMSEA = .0001 (.000–.083). We also examined the trajectories of the observed data, and the linear growth curve seemed the more appropriate and so it was chosen. The analyses to determine the number of latent classes again highlighted that the two-class model was slightly better than the threeclass one. B-LRT showed that both the models were acceptable, entropy was also the same (.95), but the two-class model was characterized by better posterior probability estimates (two-class model, class 1: .99 and class 2: .98; three-class model, class 1: .94, class 2: .97, and class 3: .99) and a good class proportion (class 1: 23%; class 2: 77%). In Fig. 2 the trajectories of the two groups of children are shown. Across the nine content levels, children belonging to Group 1 (red in Fig. 2) had a lower performance using HEMOT® compared to children belonging to Group 2 (blue in Fig. 2) that had better scores. In other words, children vary in their performance in HEMOT®, showing two different tendencies. It is worth noting that Group 2 was more numerous. So most of the children has a good performance. In addition, both the trajectories are stable across the levels.

Fig. 2. Linear growth curves on performance in HEMOT® distinguishing two latent classes (Group 1: red; Group 2: blue).

18

G. Vicentini et al.

4 Conclusions This study describes the development of HEMOT®, a web application aimed at promoting earthquake-related behavioral and emotional preparedness. Differently from the existing applications, HEMOT® can be considered both lecture-based and game-based [11], since it promotes learning of scientific contents using interactive and fun methodologies. Specifically, the first nine levels are content-focused while the last level is structured as a puzzle game whose main aim is to entertain the users. Two important criteria for the development of a web application are accessibility and usability. As regards accessibility, we inserted within HEMOT® a digital voice reading all the written contents, with the advantage to permit an independent use of HEMOT® also to children with learning disabilities (e.g., dyslexia). In particular, the evaluation of usability of HEMOT® was the first aim of this study. The results showed that most of the children perceived a decreasing difficulty when using the web application, notwithstanding the presence of individual differences. The specific use of the tablet to complete the levels could have played a role as a facilitating factor: In previous research the use of large-screen and touch-screen devices compared for example to mobile phones was an aspect that promoted usability [9]. A second aim of this paper was to explore individual differences in performance. It was found that most of the children had good scores in all the nine levels. This study suffers from some limitations. A first limit concerns the sample size, since the data regard the pilot phase of the PrEmT project. In the future, we could analyze usability and performance of HEMOT® involving a larger sample. Extending the scope of the analyses will be necessary to further consolidate and generalize our findings. Second, we did not take into account the role of some moderators and mediators (e.g., achievement emotions, self-concept, task-value, text comprehension, earthquake experience, disabilities) that could have a key role in explaining the differences in perceived usability and obtained performance [13]: Future research could also examine them. To sum up, our study documented usability and performance associated with a web application specifically devoted to promote earthquake-related emotional preparedness with children.

References 1. Brondino, M., Raccanello, D., Burro, R., Pasini, M.: Positive affect over time and emotion regulation strategies: exploring trajectories with latent growth mixture model analysis. Front. Psychol. (2020). https://doi.org/10.3389/fpsyg.2020.01575 2. Burro, R., Raccanello, D., Pasini, M., Brondino, M.: An estimation of a nonlinear dynamic process using latent class extended mixed models: affect profiles after terrorist attacks. Nonlinear Dyn. Psychol. Life Sci. 22, 35–52 (2018) 3. Daniela, L., Kalniņa, D., Strods, R.: An overview on effectiveness of technology enhanced learning (TEL). Int. J. Knowl. Soc. Res. 8(1), 79–91 (2017). https://doi.org/10.4018/IJKSR. 2017010105 4. Denham, S.A.: Emotional development in young children. Guilford, New York (1998)

HEMOT®, Helmet for EMOTions: A web application

19

5. Fleischer, H.: What is our current understanding of one-to-one computer projects: a systematic narrative research review. Educ. Res. Rev. 7(2), 107–122 (2012). https://doi.org/ 10.1016/j.edurev.2011.11.004 6. Iwarsson, S., Stahl, A.: Accessibility, usability and universal design – positioning and definition of concepts describing person-environment relationships. Disabil. Rehabil. 25(2), 57–66 (2003). https://doi.org/10.1080/dre.25.2.57.66 7. Muthén, L.K., Muthén, B.O.: Mplus Version 7 User’s Guide. Muthén & Muthén, Los Angeles (2012) 8. Nylund, K.L., Asparouhov, T., Muthén, B.O.: Deciding on the number of classes in latent class analysis and growth mixture modeling: a Monte Carlo simulation study. Struct. Equ. Model. 14(4), 535–569 (2007). https://doi.org/10.1080/10705510701575396 9. Raccanello, D., Brondino, M., Pasini, M., Landuzzi, M.G., Scarpanti, D., Vicentini, G., Massaro, M., Burro, R.: The usability of multiple devices for assessment in psychological research: salience of reasons underlying usability. In: Di Mascio, T., Vittorini, P., Gennari, R., De la Prieta, F., Rodriguez, S., Temperini, M., Azambuja Silveira, R., Popescu, E., Lancia, L. (eds.) Advances in Intelligent and Soft Computing, vol. 804, pp. 79–87 (2019). https://doi.org/10.1007/978-3-319-98872-6_10 10. Raccanello, D.; Rocca, E.; Brondino, M.: Disaster-related coping strategies: a meta-analysis on children. Poster Presented in: 19th European Congress of Developmental Psychology, Athens, Greece (2019) 11. Raccanello, D.; Vicentini, G.; Brondino, M.; Burro, R.: Technology-based trainings on emotions: a web application on earthquake-related emotional prevention with children. In: Gennari, R., Vittorini, P., De la Prieta, F., Di Mascio, T., Temperini, M., Azambuja Silveira, R., Ovalle Carranza, D.A. (eds.) Advances in Intelligent and Soft Computing, vol. 1007, pp. 53–61 (2020). https://doi.org/10.1007/978-3-030-23990-9_7 12. Raccanello, D., Vicentini, G., Burro, R.: Children’s psychological representation of earthquakes: Analysis of written definitions and Rasch scaling. Geosciences 9, 208 (2019). https://doi.org/10.3390/geosciences9050208 13. Raccanello, D., Vicentini, G., Florit, E., Burro, R.: Factors promoting learning with a web application on earthquake-related emotional preparedness in primary school. Front. Psychol. 11, 61 (2020). https://doi.org/10.3389/fpsyg.2020.00621 14. Shackel, B.: Usability – context, framework, definition, design and evaluation. Interact. Comput. 21, 339–346 (2009). https://doi.org/10.1016/j.intcom.2009.04.007 15. Sung, Y.T., Chang, K.E., Liu, T.C.: The effects of integrating mobile devices with teaching and learning on students’ learning performance: a meta-analysis and research synthesis. Comput. Educ. 94, 252–275 (2016). https://doi.org/10.1016/j.compedu.2015.11.008 16. Vicentini, G., Rocca, E., Barnaba, V., Dal Corso, E., Burro, R., Raccanello, D.: Uno studio pilota su Prevenzione Emotiva e Terremoti nella scuola primaria (progetto PrEmT): come potenziare le strategie di regolazione delle emozioni [A pilot study on Emotional Prevention and Earthquakes in primary school (PrEmT project): how to enhance emotion regulation strategies]. In: Castelli, L., Marcionetti, J., Plata, A., Ambrosetti, A. (eds.) Well-Being in Education Systems. Conference Abstract Book. Locarno 2019, pp. 217–221. Hogrefe, Bern (2019) 17. Warschauer, M.: A teacher’s place in the digital divide. Yearb. Natl. Soc. Study Educ. 106(2), 147–166 (2007). https://doi.org/10.1111/j.1744-7984.2007.00118.x 18. World Health Organization: ICF. International Classification of Functioning, Disability and Health. WHO, Geneva (2001) 19. Zimmer-Gembeck, M.J., Skinner, E.A.: The development of coping across childhood and adolescence: an integrative review and critique research. Int. J. Behav. Dev. 35(1), 1–17 (2011). https://doi.org/10.1177/0165025410384923

Social Video Learning – Creation of a Reflection-Based Course Design in Teacher Education Eric Tarantini(&) Institute of Business Education and Educational Management, Digital Learning and Corporate Learning, University of St. Gallen, 9000, St. Gallen, Switzerland [email protected]

Abstract. Reflection on teaching practice is a challenging activity. Often, reflection does only take part on a superficial level. The present contribution describes the creation and first testing of a Social Video Learning (SVL) setting by means of Learning Analytics (LA) for a teacher education course. Furthermore, a theoretical foundation for SVL was tried to create to design the course adequately. Key findings are that (I): SVL has the potential to increase the level of depth in reflection processes, (II): A collaborative and learner centered design is important to benefit from SVL, (III): The use of Learning Analytics in SVL scenarios potentially fosters coaching and learning outcomes of the students. Keywords: Social Video Learning
Learning Analytics
Reflection
Teacher education
Experiential learning

1 Introduction The digital transformation changes our daily business and habits fundamentally [2]. In this light, also the education sector and, consequently, the way of teaching and learning faces significant changes [16]. One medium that comes into play when we talk about digitalization of education is video. Technology-Enhanced Learning (TEL) researchers see the potential in Video-Based Learning (VBL) to be an effective learning method which can replace or enhance traditional classroom-based and teacher-led learning approaches [4]. So called “Social Video Learning” (SVL) is a method inspired by the idea of using video annotation in a collaborative setting. Thereby, video-annotation (enrichment with comments and visual elements) is used to enable situational reflection processes [14, 26]. Prior research shows first encouraging results, that an interactive use of video content enhances student motivation and engagement [17]. However, there is still a lack of knowledge on whether the use of video in instructional settings represents an effective way to foster learning. First results in the context of SVL seem promising [24]. Furthermore, Learning Analytics (LA) offers the possibility to facilitate learning and to evaluate learning progress with SVL [21]. The aim of this paper is to create a theoretically founded learner-centered course design. The questions to be discussed in this paper are the following ones: 1. How can

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 20–30, 2020. https://doi.org/10.1007/978-3-030-52538-5_3

Social Video Learning – Creation of a Reflection

21

SVL be theoretically founded to create an effective learning design? 2. How can a SVL setting in combination with LA for a teacher education course be designed? The paper is structured as follows: the first section focusses on the rationale of the research project: 1) what are the benefits of social video learning 2) why is social video learning a suitable tool to foster effective learning and reflection; in the second section, the applied research methodology will be explained; the third section sums up the relevant findings of my study.

2 Rationale of the Research Project What are Benefits of SVL? SVL intends and enables active work with video materials. Specifically, the core of the video-analytical work with SVL lies in the platformbased video annotation. On the platform (we use the edubreak CAMPUS platform; edubreak.de), learners actively watch and work with videos [15, 18, 26]. Comments and visual markings, accurate to the second within the video player, enable reflection in and about the specific situation. Through these annotations, the learner is asked to visually specify the reference point for his interpretation and to formulate it in an understandable way for third parties [4, 27]. Explaining one’s own thoughts, experiences, or insights within the learning process thus ensures a sustainable learning progress. The didactical relevance of this procedure can be enhanced by linking video situations to concrete observation assignments - these help to explicate subjective thinking in the video in a targeted manner [26, 27]. Towards a commentary-based interpretation, one’s mental model becomes visible and perceptible within the framework of SVL. Thus, a “goal-oriented reflection” is the consequence to expand the personal spectrum of action and bring about changes in teaching practice [9, 24].

Fig. 1. Social video learning on the edubreak CAMPUS platform (edubreak Player).

Why do We Use SVL in Teacher Education? The integration of reflection processes in teacher education is an essential success factor for the development of students’ teaching skills [14, 24]. In addition to written and oral feedback, teaching units from students are recorded in the microteaching (core course for the development of teaching

22

E. Tarantini

skills in law and economics for high schools and VET schools at the University of St. Gallen). The independent video watching and reflection of learners represented the lived practice in the original course design. However, there is evidence that knowledge is developed more effectively in groups than individually [4, 25]. The verbalization and explication of observations within reflection processes has a positive effect on teachers’ work related to their own “technical language” [26]. From a lecturers’ perspective, video annotation represents a very clear documentation of the learning process within the teacher education course. This supports coaching processes and enables the use of LA, which will be explained in the next sections.

3 Theoretical Base and Method This section describes the theory building, the methodological approach and how LA is implemented in the project. 3.1

Theoretical Framework and Adaptation to Course Setting

The creation of an effective learning process (i.e. fully understanding something) is linked to real life experience combined with reflection and abstraction. To create an adequate theoretical framework for SVL I worked with Kolbs’ “experiential learning theory” [12]. Kolb stresses the crucial role of practical experience for learning processes [13]. Furthermore, experiential learning is based on the foundation of interdisciplinary and constructivist learning [1, 20]. The core principle is that “learning is a process in which knowledge is created through the transformation of experience. Knowledge therefore results from a combination of the acquisition and transformation of experience” [13]. Experiential learning can take place both field- and classroom-based [20]. According to Kolb and Fry [12] experience-based learning happens cyclically. The steps of this process have been transferred to the microteaching course (Fig. 2).

Fig. 2. Cyclical experiential learning. Own illustration adapted from Kolb [12].

Social Video Learning – Creation of a Reflection

23

The starting point is a concrete, observed or personally made experience. The students observe a teaching unit of a colleague (concrete experience). Subsequently, the video sequence is uploaded on the platform to be annotated and reflected (reflective observation). The concrete observation is used to derive general principles of action for the individual teaching practice through a personal coaching (abstract conceptualization). Finally, one’s teaching activity is carried out, which is to be optimized with the help of the preceding processes and the start of a new cyclical learning process (active experimentation). As a whole, continuous reflection enables effective learning [19]. 3.2

Course Characteristics and Research Method

The microteachings take place every autumn semester on Bachelors’ Level. The students plan a teaching unit in detail (45 min). From this unit, a microteaching sequence (20 min) will be carried out in front of the fellow students (fictious class). The course is split up in two rounds of microteachings, i.e. every student teaches twice a semester. The first round takes place in the first six weeks of the semester, the second round after a break of two weeks in the following six weeks. This structure allows to gather data within the first round, to analyze it by means of LA and consequently improve performance aspects of the students within the second round (Fig. 3).

Fig. 3. Semester structure. Own illustration.

The course was set up by means of Blended Learning. The combination of face-toface and online-learning in a Blended Learning Design allows to effectively promote teaching skills in a team because video annotations can be discussed, clarified and reflected within the classroom setting [6, 24]. The special thing about the course design was, that the SVL part (online) was integrated in the weekly presence setting of four lessons (six times per semester). The students reacted positively to the combination of online feedback through annotations and oral feedback right afterwards. However, it must be emphasized that a detailed evaluation of the established course design is not part of this paper. 3.3

Learning Analytics to Enhance SVL

This section describes an approach of how the learning processes in teacher education can be supported by means of LA in order to optimize coaching/feedback processes and consequently achieve higher learning outcomes. What Benefits Can Result from Learning Analytics? The main purpose of LA lies in the development of methods that harness educational data sets to support a specific learning process. While much of the interest in Big Data and LA is currently focused on

24

E. Tarantini

prediction, reflection (i.e., monitoring and understanding) may in fact become more widely relevant [7, 8, 21, 23]. Especially, as learners take on more responsibility in managing their individual learning processes. Moreover, it is highly challenging to provide personal feedback to a big number of learners in a video-based learning setting [4]. Therefore, effective methods that enable to track learners’ activities and extract conclusions about the learning process in order to support personalized and networked VBL are needed. Chatti et al. [4] state, LA “can play a crucial role in supporting an effective VBL experience. LA that focuses on the perspectives of learners can help to create the basis for effective personalized VBL, through the support of monitoring, awareness, self-reflection, motivation, and feedback processes”. I regard these findings as important for SVL settings as they can be interpreted as a form of “networked VBL”. To sum up, the core idea is to evaluate the gathered data from a learning process in order to support learners more effectively [8, 22]. What Data is Collected and How is it Collected? In the described course setting, data is produced by the students themselves as they annotate microteaching videos of their colleagues within the SVL-process. These annotations are illustrated in the following figure as data points on the timeline of the video player. Each data point stands for one annotation (consisting of the point and a text comment linked to it; see also Fig. 1) (Fig. 4).

Fig. 4. Social video learning player with annotations (edubreak CAMPUS platform).

On the edubreak CAMPUS platform, annotations can be classified with green, yellow or red colors. Depending on the setting, the coding of the annotations can be individualized (e.g. red for critical situations, yellow for discussable and green for successful or positive situations). The annotations are then analyzed in detail with the students. The result is a detailed feedback from the peers and from the lecturer. Furthermore, potential improvements for future teaching can be pointed out very clearly. As we go through two rounds of microteachings, students have the possibility to optimize their performance through SVL and LA. How can LA be Structurally Designed in a Learning Setting? Seufert et al. [21] provided a design framework for LA in teaching settings. It comprises four different procedural steps to be considered. The implications are transferred as well to the microteaching course (Fig. 5):

Social Video Learning – Creation of a Reflection

25

Fig. 5. LA design framework. Own illustration adapted from Seufert et al. [21].

Pedagogic Theory and Learning Design. The base in this paper is a socioconstructivist understanding of learning (experiential learning theory by Kolb, 1975). It is therefore hypothesized, that students’ learning outcomes are enhanced by discussing in groups and, consequently, working with SVL. Objective. For the described course design, the main objective is to enhance teaching competences through effective learning and reflection. Analyzing teaching performance by the use of SVL and LA represents the base for the overall reflection [8]. LA objectives. Greller and Drachsler [10] mainly distinguish between “reflection” and “prediction“as LA objectives. However, “individual learning” and/or “social learning” need to be differentiated as well. Especially from a pedagogical perspective the distinction between social and individual learning is important [22]. SVL can be classified as social-reflective as students actively work with and discuss on video content within their group. Social learning analytics for reflection imply a shift in attention away from summative assessment of individuals towards learning analytics of social activity [3]. LA Stakeholders. Stakeholders in LA activities are those that either are subjects of data analysis services or clients of data analysis services. In the SVL setting, students are clients of data analyses in that specific analyses aim at enhancing their teaching competences. LA Application. The LA application can be represented by technologies, platforms, data sets, and algorithms employed in carrying out analytics activities [10]. In the SVL setting the edubreak CAMPUS platform for video annotation was used. LA Constraints. Rules and regulations concerning privacy and ownership of data, ethical considerations, as well as cultural norms and values are possible constraints [8]. For SVL, questions of data ownership arise as reflection takes part in groups on reallife teaching experiences of the students in the course. However, further research should be conducted to challenge this first approach of designing a course by means of experiential learning and LA.

4 Results and Discussion In this section, the main outcome of this paper, i.e., the Blended Learning Design of the teacher education course is described.

26

4.1

E. Tarantini

Blended Learning Design

Why is a Blended Learning Design Used to Implement SVL? First of all, the course design was constructed with four main steps, which are described in detail as follows. The model was established and discussed with lecturers from the university of St. Gallen. In 2019, a group of eleven students tested the new course design for the first time (Fig. 6).

Fig. 6. Blended learning design for teacher education course. Own illustration.

Planning. At the start of the semester, students (teachers in training) prepare themselves in self-study for their teaching unit in business administration, economics, law or accounting (for the context of a high school or vocational school). The lesson planning includes detailed information about the general conditions, learning objectives, disposition as well as the working materials used within the lesson. Thereby, the students gain a deeper understanding of the complexity of teaching. The planning of the microteaching takes place during the first three weeks of the semester. Coaching. In the coaching session, students receive constructive feedback on their lesson planning from the lecturer. This facilitates their learning process, as they get to know very concretely, where to improve. The lesson planning is critically examined, discussed and adapted with regard to the actual implementation. This discussion between student and lecturer is also intended to enable didactic decisions within the planning to be justified and ultimately implemented. Experiential Learning. The four steps derived from Kolbs’ experiential learning theory represent the core process of the presence setting (see Sect. 4.1). The microteachings take place in front of the fellow students as well as the lecturer and are recorded via smartphone (active experimentation). After the microteaching, students provide feedback first via SVL and orally right afterwards. This allows them to improve their oral feedback competence by gaining a higher sensitivity for critical situations in teaching by using SVL. The addition and implementation of the following elements were meant to enhance learning outcomes of the students. The first impressions are promising: Social Video Learning Supported by Application-Based Live Annotation. As the lecturer starts the “live video”-function within the edubreak App, students have the possibility to annotate via their mobile phone during a lesson held by one of their

Social Video Learning – Creation of a Reflection

27

fellow students (see Fig. 7). The observing students can add time stamps (so-called “tags”) via the application already during the teaching in order to “save” important situations (concrete experience). The edubreak App allows the lecturer to directly upload the video on the edubreak CAMPUS. This saves time and allows consequently to focus on the formulation of the text comments within the edubreak Player (see Fig. 1) (reflective observation). The interpretation of those data points by means of LA happens after the microteachings. The lecturer analyzes the annotations from the students, and identifies optimization potentials within the observed criterions for the microteachings (e.g. media use, interaction with learners, body language etc.). Furthermore, the platform-based video annotation sequence is followed up by a face-toface feedback session in order to reflect further and to enhance the cyclical experiential learning procedure. This analysis sets the base for the next coaching sessions with the students before the second round of microteachings.

Fig. 7. Live-video in the microteaching and user interface on the edubreak App.

Allocation of Observation Criteria to Enhance Reflection Focus. To assure substantial and concrete feedback, students (fictious class) are asked to observe one specific criterion during the teaching unit (e.g. use of media, teacher behavior, etc.) (abstract conceptualization). This creates a sense of responsibility during the observation, annotation and face-to-face feedback session. Furthermore, LA can be used based on the specific criteria to measure students’ progress. Learning Outcomes. Based on the inputs via SVL, the oral feedback (face-to-face) and the personal impressions, a short reflection report (approx. 6 pages) is written. The students identify their major optimization potentials and describe alternative teaching behaviors. This process is supported and facilitated by SVL.

5 Conclusion and Limitations In conclusion, first impressions are that SVL has the potential to improve teaching competences. It supports group-based reflection processes and allows collaborative and active work with video materials. Moreover, LA can play an important role to enhance

28

E. Tarantini

learning outcomes. The allocation of focus criteria for the observation of microteachings showed positive effects. This steering element allowed the definition of clear responsibilities for the annotation work on the SVL platform within the classroom session and with regard to the subsequent oral feedback round. Despite the technological innovation, social aspects like communication, empathy and trust turned out to be crucial elements in order to establish a respectful group- and, in consequence, feedback-culture [5]. Compared to previous course designs, SVL helped students to develop a certain “situational sensitivity” [24, 26]. However, those effects have to be evaluated and verified in detail. But first impressions show, that the students’ ability to reflect in depth on specific situations tend to improve from the first to the second round of microteachings. However, there are some limitations to this research. First of all, the course was carried out with a relatively small group of eleven students at university level. Further studies should therefore focus on creating effective course designs for larger groups and on different school levels. During the course, the trustful and open group culture turned out to be the core factor. In addition, the development of an honest, appreciative and respectful feedback culture within the group may have a direct effect on the quality of SVL. The quality of feedback is essentially determined by the established, familiar level of relationships within the group, an effect also known as “psychological safety” [5, 11]. Moreover, as this paper illustrates a first approach to combine SVL with LA, results have to be critically examined and analyzed to validate the course design. It has to be stressed, that this paper is about the modelling approach of such a setting and not about the evaluation. Future work should consist in the creation, testing and evaluation of other SVLbased education settings. First indications are that the technology brings new and interesting perspectives for reflection in teacher education. Furthermore, it would be interesting to test whether the technology can be used with added value on other educational stages (e.g. high school).

References 1. Bada, S.: Constructivism learning theory. A paradigm for teaching and learning. IOSR J. Res. Method Educ. (IOSR-JRME) 5, 66–70 (2015) 2. Brynjolfsson, E., McAfee, A.: The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. Norton, New York (2014) 3. Buckingham Shum, S., Deakin Crick, R.: Learning dispositions and transferable competencies: Pedagogy, modelling and learning analytics. In: Proceedings 2nd international Conference on Learning Analytics and Knowledge, pp. 324–335. ACM Press, New York (2012) 4. Chatti, M.A., Marinov, M., Sabov, O., Laksono, R., Sofyan, Z., Yousef, A., Schroeder, U.: Video annotation and analytics in course mapper. Smart Learn. Environ. 3, 10 (2016) 5. Edmondson, A.: Psychological Safety and Behavior in Work Teams. Harvard University, Cambridge (1999)

Social Video Learning – Creation of a Reflection

29

6. Ganz, A., Reinmann, G.: Blended Learning in der Lehrerfortbildung. Evaluation einer Fortbildungsinitiative zum Einsatz digitaler Medien im Fachunterricht. Unterrichtswissenschaft 35(2), 169–191 (2007) 7. Gaviria, F., Glahn, C., Drachsler, H., Specht, M., Gesa, R.F.: Activity-based learner-models for learner monitoring and recommendations in Moodle. In: Kloos, C.D., et al. (eds.) Proceedings of the 6th European Conference on Technology-Enhanced Learning, pp. 111– 124. Springer, Heidelberg (2011) 8. Gedrimiene, E., Silvola, A., Pursiainen, J., Rusanen, J., Muukkonen, H.: Learning analytics in education: literature review and case examples from vocational education. Scandin. J. Educ. Res. (2019). https://doi.org/10.1080/00313831.2019.1649718 9. Greif, S.: Coaching und ergebnisorientierte Selbstreflexion. Hagrefe Verlag, Göttingen (2008) 10. Greller, W., Drachsler, H.: Translating learning into numbers: a generic framework for learning analytics. Educ. Technol. Soc. 15(3), 42–57 (2012) 11. Hilzensauer, W.: Videoreflexion 2.0. Zur Rekonstruktion subjektiver Theorien über guten Unterricht. Medienimpulse – Beiträge zur Medienpädagogik (2012) Ausgabe 03 (2010). http://www.medienimpulse.at/articles/view/446 12. Kolb, D.: Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall, Englewood Cliffs (1984) 13. Kolb, D.A., Boyatzis, R., Mainemelis, C.: Experiential Learning Theory. Previous Research and New Directions. Case Western Reserve University, Cleveland (1999) 14. Krammer, K., Reusser, K.: Unterrichtsvideos als Medium der Aus–und Weiterbildung von Lehrpersonen. Beiträge zur Lehrerbildung 23(1), 35–50 (2005) 15. Krüger, M., Steffen, R. Vohle, F.: Videos in der Lehre durch Annotation reflektieren und aktiv diskutieren. In: Csanyi, G., Reichl, F., Steiner, A. (eds.) Digitale Medien – Werkzeuge für exzellente Forschung und Lehre, pp. 198–210. Waxmann Verlag GmbH, Münster (2012) 16. Kumar, B.S., Nivedhitha, D., Chitra Mai, M.R., Perumal, A.: Digital tools for effective learning. Int. Res. J. Eng. Technol. (IRJET) 3, 381–384 (2016) 17. Lee, C., Tsai, C.: An efficient approach to slicing learning video to improve learning effectiveness by considering learner prior knowledge. National University of Taiwan, Taiwan (2018) 18. Meixner, B., Siegel, B., Hölbling, G., Kosch, H., Lehner, F.: SIVA suite – konzeption eines frameworks zur erstellung von interaktiven videos. In: Eibl, M. (ed.) Workshop Audiovisuelle Medien WAM 2009. Aus der Reihe Chemnitzer Informatik–Berichte, pp. 13–20. Technische Universität, Chemnitz (2009) 19. Passarelli, A., Kolb, D.: Using experiential learning theory to promote student learning and development in programs of education abroad. Case Western Reserve University, Cleveland (2011) 20. Schwartz, M.: Best Practices in Experiential Learning. Ryerson University (2012) 21. Seufert, S., Meier, C., Soellner, M., Rietsche, R.: A Pedagogical Perspective on big data and learning analytics. A conceptual model for digital learning support. Universität St.Gallen: Institut für Wirtschaftspädagogik (2019a) 22. Seufert, S., Guggemos, J., Sonderegger, S.: Digital transformation in higher education: augmentation strategies for the use of data analytics and artificial intelligence (AI). ZFHE 15 (1), 81–101 (2019b) 23. Siemens, G., Gasevic, D., Haythornthwaite, C., Dawson, S., Shum, S. B., Ferguson, R., et al.: Open learning analytics: an integrated & modularized platform, 28 July 2011

30

E. Tarantini

24. Tarantini, E.: Social Video Learning Projekt im Didaktischen Transfer der Zusatzausbildung Wirtschaftspädgogik. Planung, Durchführung und Evaluation eines neuen Kursdesigns. Masterarbeit. Universität St.Gallen: Institut für Wirtschaftspädagogik (IWP-HSG) (2016) 25. Trendreport: Vernetzte Gesellschaft. Der Trend hin zur Digitalisierung in der Wirtschaft und Gesellschaft ist ungebrochen. Eine stille Evolution, die Märkte und Macher treibt. In: Handelsblatt (2015). http://trendreport.de/category/ausgaben/032015/. Accessed 28 Oktober 26. Vohle, F., Reinmann, G.: Förderung professioneller Unterrichtskompetenz mit digitalen Medien: Lehren lernen durch Videoannotation. In: Schulz–Zander, R., Eickelmann, B., Moser, H. Niesyto, H., Grell, P. (eds.) Jahrbuch Medienpädagogik, vol. 9, pp. 413–429. Springer, Wiesbaden (2012) 27. Vohle, F.: Lernen 5.0. Fünf Innovationsdimensionen für eine veränderte Lehr-, Lern- und Prüfungskultur mit Social Video Learning. Vortrag SCIL Webinar, 16 October 2019 (2019)

Engaging Pre-teens in Ideating and Programming Smart Objects Through Play Rosella Gennari1(B) , Maristella Matera2 , Alessandra Melonio1 , Mehdi Rizvi1 , and Eftychia Roumelioti1 1

Faculty of Computer Science, Free University of Bozen-Bolzano, Piazza Domenicani 3, Bolzano, Italy [email protected], {alessandra.melonio,srizvi}@unibz.it, [email protected] 2 DEIB, Politecnico di Milano, Via Ponzio, 34/5, 20133 Milan, Italy [email protected] Abstract. Recent research advocates the role of protagonist for children in design: children should participate across different design stages, from ideation to programming, so that they learn through it. However it is not straightforward how to organise a design workshop with children across different design stages, above all in a short time span. This paper tackles such issue. It presents a workshop with children ideating, programming and prototyping smart objects, structured around a card-based board game, and in the time-span of two days. Results are contextual but support that children’s engagement across design stages can be fostered by the game-based structure. Keywords: Children

1

· Game · Design · Making · Smart object · IoT

Introduction

A so-called smart object is a new form of tangible interactive technology [11]: it is a physical object made smart with embedded micro-electronics (e.g., microcontrollers), which have a program that controls inputs (e.g., a button) and outputs (e.g., an LED matrix display) for interacting with the environment [12]. Children have been recently involved in the design of smart objects, with microelectronics for them, e.g., [13]. In line with design thinking, the design of smart objects with children, as the design of any interactive technology, breaks through parts, often alternated: the parts of understanding their context, ideating and reflecting on them, developing by programming besides prototyping, always reflecting on what they make [18]. In this manner, design switches between enlarging possibilities, and converging towards some, this mainly through reflections. This paper picks up the same view on the design of smart objects, focusing on the parts of ideation and programming, with reflections. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 31–40, 2020. https://doi.org/10.1007/978-3-030-52538-5_4

32

R. Gennari et al.

Different research traditions have promoted children’s involvement in design. The Making movement has created different opportunities for children, programming and prototyping, hands-on, smart objects, e.g., [13,19]. On the other hand, different design traditions revolving around child-computer interaction have considered how to best involve children in expressing or exploring design ideas. In the latest years, there has been a convergence of such research streams, promoted in Europe through FabLearn initiatives [2]. Their shared goal is not to design new technology, rather to empower children’s learning through design [2,9]. The workshop, which is the focus of this paper, shared their aim of empowering children’s learning in design. The workshop had a total of 27 children, aged from 11 to 14 years old, males and females. None of the participant children had any related prior experience. The majority of research work with children participating as protagonists in different design stages usually spans several days, if not months. However, that is not often possible, e.g., due to school timetable constraints. Therefore the workshop reported in this paper was organised, in the short span of 2 days, like in design sprint, and, most importantly, around a common structure which could easily move children from one stage to another and keep them engaged: around an evolution of a card-based board game for designing with children [5]. Finally, the paper reports results of this workshop in relation to a dimension not yet explored in other similar work with children as design protagonists: their engagement in the design of smart objects, which is relevant for fostering their learning [3]. Engagement is operationalised and analysed along the behavioural and affective dimensions, in line with education litearature. This paper starts with a brief overview of related work. Then it reports on the main parts of the design of the workshop and focuses on its novel results in relation to children’s engagement. Results are overall positive, indicating that the game-based structure of the workshop engaged children in the design of their own smart objects. The paper concludes by reflecting on the impact of the results on future work concerning design initiatives with children for fostering their engagement in design.

2

Related Work

In the field of child computer interaction as well as of interaction design and children, the work of Druin and colleagues has been influential. In a seminal paper, the roles of user, tester, informant, and design partner were discussed [1]. In the role of design partner, in particular, children are expected to contribute as stakeholders in inter-generational design groups. Lately, the same literature opened a debate concerning children’s participation in design, e.g., their true design possibilities [7,14,16]. Starting from such reflections, the role of children as design protagonists emerged [9]. In the view of Iversen et al., design with children as protagonists should not focus on how to achieve quality new technology, rather, it should be “arranged primarily to help children develop their design competence”. In this tradition,

Engaging Pre-teens in Ideating and Programming

33

design becomes a learning opportunity for children. Different forms of empowerment were further analysed and guidelines concerning them were advanced [10]. Empowerment as learning development has been especially related to children’s skills and competencies “relevant for their future”. The workshop of this paper also considers design as a learning opportunity for children, in relation to smart object design. Children take part in diverse main design stages (as per the recommendations by Iversen et al.) which implement entry conditions for enabling children’s learning and competence empowerment (as per the guidelines of Kinnula and Iivari). In particular, children, firstly are immersed in the design context. Then they ideate their own smart objects. Finally they program and prototype the emerged ideas of smart objects. Differently than in related work, however, the workshop of this paper lasted few days, like in design sprint. Its structured was centred around an evolution of a board game for designing with children, so that this could guide and engage children across design stages. Last but not least, recent work evidences how few studies have been yet dedicated to the topic of clearly analysing factors related to children’s learning in design, when they are given the role of protagonists: “in-depth inquiries on children’s experiences and challenges involved are lacking” [8]. This paper picks up this challenge and focuses on engagement, a factor often considered in association with learning in the education literature. For instance, past game design workshops with children showed how children’s emotions, indicating their engagement, were correlated to children’s quality of products in design, indicating their learning of game design [3]. Similar work shows that their engagement in prototyping enhance different types of learning, e.g., social learning [6]. Engagement, per se, is a complex construct in the education literature. It is often assessed by considering behavioural, affective or cognitive dimensions. In particular, Ocumpaugh and colleagues gave behavioural indicators for assessing engagement of students, mostly related to being on-tasks or off-tasks [15]. Affective engagement instead is often assessed by considering different emotions, or the intensity of liking/disliking an activity [4,17]. This paper considers both the affective and behavioural dimensions of engagement, and assesses it by means of two different instruments, as explained in the remainder.

3

Workshop Design

The design workshop reported in this paper was with children participating voluntarily, and in a specific making environment, as reported below. Research questions, considered in this paper, addressed children’s engagement across design stages. In order to foster it, the workshop was organised around an evolution of a board-game [5] or its cards for inputs and outputs necessary for smart objects: an object is smart if it has, besides a physical object from the environment (e.g., a tree), inputs (e.g., a button) and outputs (e.g., an LED matrix display) which are programmed to interact with the environment.

34

R. Gennari et al.

The organisation of the workshop is explained in the part below for the protocol. See moments of the game-play in Fig. 1.

Fig. 1. Different moments of the game-based design workshop

The last part of this section outlines the data collection instruments for data related to children’s engagement in the workshop. 3.1

Participants and Setting

The workshop had 27 children (33.3% females), from different socio-economic backgrounds. None of them had any prior direct experience in ideating smart objects or programming, albeit they were familiar with the terms. Children were of different ages, with mean age M = 12 years, maximum age max = 14 and minimum age min = 11 years, and standard deviation SD = 1. All children participated on a voluntary basis, and their parents authorised their participation through a written consent form, approved by the ethical committee of one of the authors’ university. Five design researchers also participated in the workshop: three acted as moderators, facilitating the design process; the others acted as technical facilitators, and took care of the data collection instruments, explained below. Postworkshop data processing was performed in aggregated and anonymous form so as to comply with rules on data protection. The workshop was split and replicated in two turns. Children were randomly assigned to a turn, so as to have a moderator for at most 5 children. The workshop was run in a making facility for citizens, with programmable micro-electronics kits for children and a safe working area with further green or recycled prototyping material.

Engaging Pre-teens in Ideating and Programming

3.2

35

Research Questions

The main goal of the research reported in this paper was to analyse children’s engagement in the workshop, a dimension often associated to learning, as explained above. The goal was split into two research questions, one per main stage of the workshop: R.1 Are children engaged in ideating their smart objects? And how so? R.2 Are children engaged in programming their smart objects? And how so? What these stages consisted in is explained next. 3.3

Protocol

The design workshop was split into three stages, all gravitating around the boardgame or its cards, for inputs and outputs, necessary for creating smart objects. The first stage served to give children the basic tools and experience for continuing on the next design stages as independently as possible: this is the so-called understanding stage. The next two are the main stages of the design workshop: ideating with reflections; programming and prototyping, with reflections. All are explained, with the strictly necessary details, in the following. Stage 1: Understanding. In each turn, the workshop started with an “understanding stage”: this took place the morning of the first day, and it consisted of guided exploration and tinkering with programmable micro-controllers, and card-based material for the second stage. Tangible outcomes were programs for the micro-controllers, with the companion cards for inputs and outputs necessary for programming smart objects. Stage 2: Ideating, with Reflections. Then the workshop moved to the second stage, during the afternoon of the first day. Children, in groups of 4 or 5, played with the board-game simulating a context they were familiar with. The game narrative immersed children in the context. Then the game asked them to ideate smart objects of their own. Last, the game gave children reflection lenses for playing the roles of expert designers: in this manner, each child had the opportunity to improve and converge towards a smart object idea by reflecting individually and with others. Tangible outcomes of this stage were children’s card-based dashboards, one per child, which conceptualised an idea of a mart object to carry on in the next stage. Stage 3: Programming and Prototyping, with Reflections. The final stage took place in the morning of the second day, when children, working in small groups, programmed and prototyped ideas of smart objects, conceptualised in dashboards. Again, this stage enabled children to expand on ideas, and converge towards a program and prototype, with reflection stimuli as above. Tangible outcomes were programs and prototypes, by children, of their smart objects.

36

3.4

R. Gennari et al.

Data Collection

Survey. Children’s affective engagement with a novel activity, right after it, can be assessed with specific self-report instruments. These need to phrase questions in a language clear for children and have a visual format, e.g., they could use visual analogue scales, which employ icons or images that children can understand in order to report their emotions or opinions. In this research, children’s overall engagement with the ideation and programming stages, separately, was assessed with a survey for children, administered right after the workshop. These contained two close-format questions. Questions were related to how much children liked ideating smart objects, and how much they liked programming smart objects. Ratings were given with the Smiley-o-meter 5-point Likert scale, ranging from “not at all” (1) to “very much” (5), taken from the Fun tool-kit [17]. An example is in Fig. 2.

Fig. 2. Engagement question related to programming

Observations. Qualitative data come from direct observations, tracked by moderatos in diaries. These were semi-structured with leading questions, concerning behavioural indicators of engagement, in line with [15]. Specifically, during each main stage of the workshop, leading questions for moderators were as follows: – Are children engaged in ideating (Stage 2)? If yes, track example on-task behaviours, e.g., a child stays close to the game board. If not, explain why, and track example off-task behaviours, e.g., a child plays with tokens. – Are children engaged in programming (Stage 3)? If yes, track example on-task behaviours, e.g., a child intervenes asking proper questions. If not, explain why, and track example off-task behaviours, e.g., a child roams without participating in programming.

4

Workshop Results

Data were analysed in relation to the two aforementioned research questions, concerning engagement. SPSS version 25.0 for Windows was used to calculate statistics; the level of significance was set at p < .05. Results of the data analyses are reported below per data collection instrument.

Engaging Pre-teens in Ideating and Programming

4.1

37

Survey Results

Children reported to be generally engaged in ideating as well as in programming: the mean rating for ideation was M = 4.48, with standard deviation 0.7 and 95%CI = [4.76, 4.2]; the mean rating for programming was M = 4.5, with standard deviation 0.51 and 95%-CI = [4.7, 4.3]. See also Fig. 3. Given that data tended to be skewed and positive, a Spearman’s rank-order correlation was run to determine the intensity of the relationship between the 27 children’s engagement ratings for ideation and programming. There was a positive correlation between ideation and programming ratings, which was statistically significant (rs (26) = .62, p = .001). In other words, children who tended to like one stage also tended to like the other.

Fig. 3. Mean engagement ratings for ideation and programming, with related 95%-CIs

4.2

Observation Results

The observation results, generally, confirm the quantitative findings and help interpret them. Relevant excerpts from diaries are reported below per stage. Ideation. Moderators reported similar findings in their diaries concerning the ideation stage. For instance in one group, at the start of the board game for ideating, every child tried to get as many cards as possible, so as to make sure to gain sufficient material to generate at least an idea. Children seemed even more engaged when it was the time to play reflection lenses, and have the roles of design experts: each participant not only appeared engaged in generating her/his ideas, but s/he also was engaged in giving advice about how to make the others’ ideas feasible, and on how to elaborate on other players’ ideas. Even those, who had more difficulties in generating ideas on their own initially, reflected with others on their ideas and collaborated to improve any idea discussed during the game. Only one child showed boredom indicators for the prolongued discussions that others wanted to continue, e.g., he was running for the exit door while another kept on talking.

38

R. Gennari et al.

Self-confidence in one’s abilities seemed also to play a major factor. For instance, in another group, it was noticed that, even though they were all engaged, children who seemed more self-confident in their skills were also those most eager to offer reflection stimuli to others. Interestingly, a child was very keen in making others reflect on the sustainability of their ideas, although this was not a reflection lens the game offered. Programming. As for the programming stage, differences were noticed in engagement, which seemed to depend also on the ability of children to pick up how to program their ideas. For instance, it was noticed in a group that, when the time came to start programming their idea, two children appeared not so engaged probably because they were not able to program the idea by themselves. These were the same children who needed longer to pick up how to ideate a smart object. However then they got more and more engaged by observing how others were able to write their programs; in the end, they tried to learn from what these did. Children were also keen on prototyping to demonstrate their idea to others. Two girls were so excited about their work that they wanted everybody to test their prototype out. Another member of this group was also proud of what he had made: he spent till the last minute in refining the prototype with carton and markers. The other two team members, who were the youngest, were engaged but mostly in programming music, and wanted to make it “better and better”, enriching their program with it till the very last minute. All children, in general, were engaged when they presented their idea; simulating also parts which were not perfectly working, they showed pride in owning and making tangible ideas of smart objects.

5

Discussion and Conclusions

In line with recent debates concerning the role of children in design, this paper considers design as a learning empowerment opportunity for children. It reports on a workshop with 27 11–14 years old children, designing their own smart objects. However, differently than in related work, this workshop took a short time-span of two days, during vacation summer time. During the workshop, children ideated novel smart objets of their own, by exchanging reflection opportunities with others. Then they programmed and prototyped the smart objects, sharing their reflections again with others. The paper considers a dimension often associated in the education literature to learning, which is engagement, and yet not sufficiently investigated in the literature of design with children as protagonists. The paper inspects it from two main angles: the angle of affective engagement, by considering children’s ratings of how much they liked ideating and programming; the angle of behavioural engagement, by considering behavioural on/off-task indicators in design ideation and programming, tracked by moderators.

Engaging Pre-teens in Ideating and Programming

39

Results reported in this paper are overall positive. Children’s ratings indicate that the game-based structure of the workshop engaged children in both design stages: those who liked the ideation stage, tended also to like the programming stage, as correlational analyses show. Moreover, lower bounds of CIs for ratings are all above 4, in a scale with 1 as minimum enjoyment and 5 with maximum. Observations by moderators, written in semi-structured diaries, assessed the behavioural dimension of engagement, and helped contextualise children’s ratings. In particular, it seems that children who felt more self-confident in their abilities were also those more engaged in delivering reflection opportunities to others. Reflection lenses due to consider with children, which engaged some of them, were related to sustainability. This was not part of the game reflection lenses and spontaneously emerged during the ideation stage. Such results give suggestions concerning future design workshops aiming at empowering children’s learning through design, and how to structure the workshops so as to engage children across design stages. Limitations are mainly due to the contextual nature of the reported research. In particular, the duration of the workshop limits its impact on what children could learn. Future editions should consider to extend the workshop duration and change context, and see how the results reported in this paper can be adapted. Overall, the reported results indicate that the workshop structure, gravitating around a card-based board game, engaged children, without any related experience, across different smart object design stages, thereby possibly positively affecting their learning. Future work will analyse their possible correlation.

References 1. Druin, A.: The role of children in the design of new technology. Behav. Inf. Technol. 21(1), 1–25 (2002). https://doi.org/10.1080/01449290110108659 2. Eriksson, E., et al.: Widening the scope of fablearn research: Integrating computational thinking, design and making. In: Proceedings of the FabLearn Europe 2019 Conference, FabLearn Europe 2019, pp. 15:1–15:9. ACM, New York (2019). https://doi.org/10.1145/3335055.3335070 3. Gennari, R., Melonio, A., Raccanello, D., Brondino, M., Dodero, G., Pasini, M., Torello, S.: Children’s emotions and quality of products in participatory game design. Int. J. Hum.-Comput. Stud. 101, 45–61 (2017). http://www.sciencedirect.com/science/article/pii/S1071581917300149 4. Gennari, R., Melonio, A., Rizvi, M.: From TurnTalk to ClassTalk: the emergence of tangibles for class conversations in primary school classrooms. Behav. Inf. Technol. 1–20 (2019). https://doi.org/10.1080/0144929X.2019.1614226 5. Gennari, R., Matera, M., Melonio, A., Roumelioti, E.: SNaP 2: the evolution of a board game for smart nature environments. In: Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts, Barcelona, Spain, 22–25 October 2019, pp. 405–411 (2019). https://doi. org/10.1145/3341215.3356281 6. Gennari, R., Melonio, A., Rizvi, M.: Turn taking with turn-talk in group. Multimed. Tools Appl. 78(10), 13461–13487 (2019). https://doi.org/10.1007/s11042018-7090-2

40

R. Gennari et al.

7. Hourcade, J.P.: Child-Computer Interaction. Self (2015) 8. Iivari, N., Kinnula, M.: Empowering children through design and making: towards protagonist role adoption. In: Proceedings of the 15th Participatory Design Conference: Full Papers–Volume 1, PDC 2018, pp. 16:1–16:12. ACM, New York (2018). https://doi.org/10.1145/3210586.3210600 9. Iversen, O.S., Smith, R.C., Dindler, C.: Child as protagonist: expanding the role of children in participatory design. In: Proceedings of the 2017 Conference on Interaction Design and Children, IDC 2017, pp. 27–37. ACM, New York (2017). https://doi.org/10.1145/3078072.3079725 10. Kinnula, M., Iivari, N., Molin-Juustila, T., Keskitalo, E., Leinonen, T., Mansikkam¨ aki, E., Simil¨ a, M.: Cooperation, combat, or competence building-what do we mean when we are ‘empowering children’ in and through digital technology design? In: Proceedings of the International Conference on Information Systems. AIS (2017) 11. Knowles, B., Beck, S., Finney, J., Devine, J., Lindley, J.: A scenario-based methodology for exploring risks: children and programmable IoT. In: Harrison, S., Bardzell, S., Neustaedter, C., Tatar, D.G. (eds.) Proceedings of the 2019 on Designing Interactive Systems Conference, DIS 2019, San Diego, CA, USA, 23–28 June 2019, pp. 751–761. ACM (2019). https://doi.org/10.1145/3322276.3322315 12. Kortuem, G., Kawsar, F., Sundramoorthy, V., Fitton, D.: Smart objects as building blocks for the internet of things. IEEE Internet Comput. 14(1), 44–51 (2010). https://doi.org/10.1109/MIC.2009.143 13. Lechelt, Z., Rogers, Y., Marquardt, N., Shum, V.: Democratizing children’s engagement with the internet of things through connectUs. In: UbiComp 2016 Adjunct– Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2017). https://uclic.ucl.ac.uk/publications/1189452 14. Netta, I., Marianne, K., Leena, K.: With best intentions: a foucauldian examination on children’s genuine participation in ICT design 28(2), 246–280 (2015). https:// doi.org/10.1108/ITP-12-2013-0223. Accessed 22 Jan 2020 15. Ocumpaugh, J., Baker, R., Rodrigo, M.: Monitoring protocol (BROMP) 2.0 technical & training manual.. Teachers College, New York (2015) 16. Read, J.C., Fitton, D., Sim, G., Horton, M.: How ideas make it through to designs: process and practice. In: Proceedings of the 9th Nordic Conference on HumanComputer Interaction, Gothenburg, Sweden, 23–27 October 2016, p. 16. ACM (2016). https://doi.org/10.1145/2971485.2971560 17. Read, J.C., MacFarlane, S.: Using the fun toolkit and other survey methods to gather opinions in child computer interaction. In: Proceedings of the 2006 Conference on Interaction Design and Children, IDC 2006, pp. 81–88. ACM, New York (2006). https://doi.org/10.1145/1139073.1139096 18. Thoring, K., M¨ uller, R.: Understanding the creative mechanisms of design thinking: an evolutionary approach. In: Proceedings of of the Second Conference on Creativity and Innovation in Design, DESIRE, pp. 137–147 (2011) 19. Wood, G., et al.: Designing for digital playing out. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI 2019), pp. 679:1– 679:15, ACM (2019). https://doi.org/10.1145/3290605.3300909

Advanced Placement Physics Exam Performance of High School Graduates in Mexico with the Aid of Online Assignments Designed in Open-EdX Ernesto M. Hernández(&), Rubén D. Santiago, José A. Otero, and Ma. de Lourdes Quezada-Batalla Instituto Tecnológico Y de Estudios Superiores de Monterrey, 52926 Atizapán de Zaragoza, Estado de México, Mexico [email protected]

Abstract. The advanced placement program (AP) in the United States and Canada has been successful in helping students to gain College credits during the last year of high school. Credits are awarded to high school graduates according to their performance on the AP exam. The content and philosophy of such program is applied to high school graduates in Mexico. During the first year of this work, a group of students was trained through Physics courses with the AP content and principles. The students took the AP exam on Physics before the course, and their scores were slightly improved after the course. The statistical significance of their improvement could not be stablished for this group of students. During the second year of this research, another group of students was exposed to AP content and aided through online assignments, previously designed in Open-EdX with LON-CAPA. The assignments were migrated to the platform Canvas during the last semester of this research. Students were tested through a conceptual exam on force and motion, which has the same spirit as the AP exam, but with a lower level of difficulty. Finally, students are found to improve significantly their performance through the strategy implemented during the second year of this research. Keywords: Advanced placement
Educational innovation
Higher education

1 Introduction The results on the placement exam during the last few years at Tecnológico de Monterrey ITESM-CEM are in agreement with the results reported by the Program for International Student Assessment, (PISA 2018). These results indicate that an average graduate high school student obtains 43 out of 100 possible points on the placement Physics examination. A detailed analysis shows that only 70% of students are able to pass the most basic course in Physics offered by ITESM-CEM. Additionally, only a few number of students are able to transfer fundamental concepts in Physics to other areas of engineering.

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 41–49, 2020. https://doi.org/10.1007/978-3-030-52538-5_5

42

E. M. Hernández et al.

During the 50’s an advanced placement program for high school graduates, was created in the United States (Rothschild 1999). The main goal of this program was to offer high school students the opportunity to gain College credit during the last year of high school. The program consists on developing advanced courses for high schools, in order to improve the student’s performance in College (College Board 2018). These courses are available for almost any area and allows demonstrating College level skills through an advance placement exam. Specifically, the course focused to improve the student’s performance on the advanced placement exam in Physics “AP Physics: Algebra Based” has been designed to develop the understanding of fundamental laws in Physics (College Board 2018). In this work, the contents of the advanced placement program is applied to high school graduates at Tecnológico de Monterrey, Mexico. The goal is to boost the disciplinary skills of students by addressing their conceptual deficiencies. The teaching approach and contents of the course developed in this proposal, intend to change the preconceived idea of using algorithmic processes to find the solution in any Physics problem. This approach is based on the “AP” philosophy, where student learning is enhanced by focusing on the application and identification of the fundamental laws that constrain the behavior of a physical system in different scenarios.

2 Theoretical Framework According to several authors, the content of courses based on the AP program has been partially successful. Ewing and Howell (2015) have shown that students enrolled in AP courses and particularly, those students that are able to obtain a high score on the AP exam, improve their probabilities to gain a higher Grade Point Average (GPA) than students not enrolled in AP courses. According to these authors, research indicates that students with the highest scores on the AP Physics exam can increase their GPA by a factor of 0.21 when compared with students that do not take the AP exam. According to their results, the highest correlation is observed on the AP Chemistry course, with an overall increment of 0.24 in the student’s GPA. In second place, students that take the AP Physics exam show an average improvement of 0.21 in their GPA. In third place, the GPA has shown an improvement of 0.19 for students that take the AP Biology exam, which is equivalent to the performance of students that spend an average time to study of five hours a week (0.18 GPA growth). Additionally, students with moderate scores in the AP exam, improve their GPAs by an average amount of 0.15. Research studies by Mattern, Shaw and Xiong (2009) indicate that AP students with scores higher than 3.0 in the AP exam, have a higher academic performance during the first year of College, than students not enrolled in AP courses. The research of Willingham and Morris (1986) indicates that independently of the score obtained in the AP exam, students that take AP courses, have a higher probability of obtaining a B average during their first year of college. Geiser and Santelices (2004) determined the impact of the time invested in AP and honors courses, as well as the student’s performance on AP exams. These authors found that the number of hours spent in AP courses or honors courses does not have a significant correlation with a student’s performance during the first years of College. However, the authors say that another factor that should

Advanced Placement Physics Exam Performance of High School Graduates

43

be considered in their study is related with the student’s performance in AP exams. Dougherty, Mellor and Jian (2006) confirm the hypothesis that preparing students through the contents of an AP course, is a good indicator that a given high school provides good College training to their students. On the other hand, Klopfenstein and Thomas (2009) determined through a group of sophomore students in Texas that, independently of race or economic status, the AP experience does not increase the probability of success during the first years of College studies. According to these authors, students that take the AP exam show a better performance during the first year of College. The authors state that some case studies where this conclusion is reached, the results are not reliable. Another effort to increase the students’ performance during their freshmen year has been addressed through the Andes system. The intelligent system of Andes has been evaluated on the introductory courses of Physics at the U.S. Navy during each fall semester from 1999 to 2003. According to Van Lehn (2005), the benefits of learning through this system is related with the use of activities with problems that are not required in traditional courses. These type of problems have possibly contributed to improve the performance of students that use Andes. The Andes students have the opportunity to correct their mistakes or omissions while doing their homework and they receive hints in the process, which focus on the Physics laws related to the problem. A control group was constituted where students had to solve traditional problems from textbooks, and where probably the fundamental laws involved in the solutions were not emphasized as much. The evidence is not conclusive on whether the tutoring Andes system helped to enhance students learning or not.

3 Methodology The methodology of AP Physics was applied to an experimental group of 48 high school graduates during the spring and fall semesters of 2018 as shown in Fig. 1. On the same period, the traditional course offered by ITESM-CEM on Introductory Physics, was offered to 48 high school graduates that constitute the control group. The course was designed to train the students during one semester. Then, the AP content was applied to 24 students on the spring semester of 2018 and to another group of 24 students during the fall semester of 2018. The control group consisted of 48 students that already had taken the traditional course of Introductory Physics, and were enrolled in Physics 1101, which covers the basics of Newtonian mechanics for freshmen students. The AP Physics exam was applied to the experimental group at the beginning of the corresponding semester, and to the control group at the beginning of the Physics 1101 course. At the end of the proposed course in this work, where students were exposed to AP content, the AP Physics exam was applied to each student, and their scores were compared with those obtained at the beginning of the course and those obtained by the control group of students. Students were evaluated with the first section of the AP Physics exam: Algebra Based, which contains problems on Newtonian mechanics. Students were given 1.5 h to finish the exam, which is 50% more time that an AP student in the U.S. are allowed for this section. They were evaluated with eight multiple-choice questions and one free response question.

44

E. M. Hernández et al.

Fig. 1. Workflow diagrams for the methodology applied during the periods 2018 and 2019.

On the second stage of this study, the AP Physics content and philosophy was applied to an experimental group consisting of 39 high school graduates, during the fall semester of 2019 as shown in Fig. 1. The course was adapted to the new educational model of ITESM, Tec-21 (2019), where the course is focused on finding approximate solutions for challenging situations or real life problems. The new educational model emphasizes on the conceptual content, where a student is trained to identify the fundamental laws of Physics involved in a particular problem or challenging situation. The new model Tec-21, does not rely on the algorithmic processes for solving problems, but instead its contents and approach are similar to the AP philosophy. Then, the new educational model was helpful to implement the proposed course in this work. Additionally, students from the experimental group were trained through online assignments with the help of hints, which were originally designed through LON-CAPA supported by Open-EdX during the spring semester of 2019, but applied through the Canvas platform during the fall semester of the same year. At the end of the fall semester, the experimental group of students was tested through the conceptual exam of mechanical force and motion. This exam was also applied to a control group of 134 high school graduates, at the beginning of the fall semester. The results from both groups were compared to determine if the improvement shown by the students from the experimental group was statistically significant.

4 Results and Discussion The results obtained by the experimental group on the AP Physics exam during the spring and fall semesters of 2018 are shown in Table 1. Scores are shown in a scale between 0 and 100. A score of 60 is the equivalent of a 3.0 score in the original AP Physics exam.

Table 1. Scores obtained by the experimental group of 48 high school graduates during the spring and fall semesters of 2018. Results obtained before and after the course. Average score Mode Standard deviation Maximum score Before course 20.67 20 11.18 46.67 After course 33.79 30 13.96 68.33

Advanced Placement Physics Exam Performance of High School Graduates

45

The scores obtained by AP students in 2017, College Board (2018) show that 20.3% of the total number of students that took the AP Physics 1 exam, were able to score between 3.0 and 4.0, which corresponds to a score between 60 and 80 in the scale shown in Table 1. Even though, two groups of 24 high school graduates were exposed to AP content during the spring and fall semesters of 2018, the results show that Mexican students are greatly outperformed by U.S. students. Only two students out of 48 that took the exam after the proposed course in this work could obtain an equivalent score between 3.0 and 4.0. Additionally, they were given 50% more time to answer the exam and were only tested on the Mechanics section of this exam. Table 2 shows the scores obtained by the experimental and control groups. The results are used to determine if the students’ improvement is significant. Table 2. Scores obtained by the experimental and control group in 2018. Scores that belong to the experimental group correspond the results of the exam taken after the course. Each group is formed by 48 students. Average score Mode Standard deviation Maximum Score Control 20.37 10 13.15 53.50 Experimental 33.79 30 13.96 68.33

The control group was constituted by 48 freshmen students that already had taken the traditional course on Introductory Physics. On the one hand, the results from the control group are practically the same as the results obtained by high school graduates that took the exam before the AP course proposed in this work. This indicates that Mexican students are not usually exposed to content where the focus is on identifying and applying fundamental laws. These results indicate that Mexican students are only trained on the algorithmic processes involved in solving Physics problems. On the other hand, the results illustrated in Table 1 show an extremely slow progress by the students from the experimental group. There is no sufficient evidence to establish an improved performance through the proposed course. According to the results shown in Tables 1 and 2, the students’ improvement is not statistically significant. Then, a group of students was exposed to AP content during the fall semester of 2019, but they were tested through the conceptual exam of mechanical force and motion. This exam is also focused on evaluating the student’s ability to apply and identify the fundamental laws needed to solve a particular problem. However, its difficulty level is lower than the AP Physics exam. Additionally, the experimental group of students were also exposed to AP content and were assigned online assignments with hints to help them in their training. During the course, students from the experimental group took preparation exams with AP Physics type of problems. During the spring semester of 2019, a course in Open-EdX with LON-CAPA was designed with assignments and exercises to help students on their training for the conceptual exam on mechanical force and motion. Figure 1 shows one type of problems that were assigned to the students of the experimental group. The student has to apply Newton’s second and third law of motion. In addition, the student has to apply

46

E. M. Hernández et al.

the concept of apparent force (normal force) to determine the magnitude, direction and type of friction force acting between the two blocks shown in Fig. 2.

Fig. 2. Homework example designed in Open-EdX

The problem shown in Fig. 2 is designed to test the student on several issues. First, the student has to appeal to his or her imagination in order to understand that friction force is acting upwards in the force diagram of block 1 and apply Newton’s third law to determine that friction is acting downwards on block 2, with the same magnitude and opposite direction. Then, the student must apply Newton’s second law to determine the acceleration of block 1, realizing at the same time that the accelerating force is the normal force, acting leftward and perpendicular to the contact surface between the two blocks. A common mistake in this type of problem is to believe that the accelerating force acting on block 1 is the same as the applied force F over block 2. High school graduates tend to neglect the normal force in this problem, because they are used to think that it only acts perpendicular to the floor. Finally, to determine which type of friction is acting between the blocks, the student must compare the applied force on block 1 (which is the gravitational weight in this case) with the maximum static friction ls N. The students are trained to apply and identify Newton’s laws of motion by constant practice on drawing force diagrams. They are taught in class, that a very important aspect to learn and understand Newton’s third law is to draw a clear force diagram by placing forces with the correct direction and in the proper “spot”, which later helps into applying Newton’s second law and obtain the correct equation of motion for each object.

Advanced Placement Physics Exam Performance of High School Graduates

47

The conceptual exam on mechanical force and motion was applied through Microsoft Forms, and students were allowed a maximum time of two hours to respond 47 multiple-choice questions. The control and experimental groups took an average time of 36 and 57 min to finish the exam, respectively. Figure 2 shows a set of questions that appear on the conceptual exam of mechanical force and motion applied to the high school graduates from the control and experimental group. The control group consisted of 134 students that took this exam before the course. The experimental group consisted of a different sample of students that took the exam after the course. Students from both groups come from different high schools, economic and racial backgrounds. Several questions were designed, where the student has to choose from a set of acceleration vs time plots, the corresponding graph that best matches the specific situation presented to the student. For example, in problem 14, the student has to select the type of plot that corresponds to a car moving to the right (far away from the origin) and at constant speed. Newton’s second law of motion implies that an object moves at constant speed when the net force acting on that object is equal to zero. Therefore, the correct answer through a simple application of Newton’s second law of motion must be option E, which is shown in Fig. 3. As an example, Table 3 shows the percent of correct answers provided by all the students of each group on questions 14 and 15.

Fig. 3. Sample problems on the conceptual exam of force and motion. The student is tested on his or her ability to read and interpret time plots of dynamical variables of motion.

48

E. M. Hernández et al.

Table 3. Percent of students that provided correct answers to questions 14 and 15 on the conceptual exam on mechanical force and motion Question 14 Question 15 Control 5.22 90.30 Experimental 53.85 87.18

The results illustrated in Table 3, show that students from the experimental group have a significant improvement on question 14, whereas the results for question 15 are practically the same. Finally, the average scores obtained from each group on the conceptual exam of force and motion is shown with a scale between 0 and 100 in Table 4. Table 4. Average scores in a scale between 0 and 100 obtained on the conceptual exam of mechanical force and motion. The control group is formed by a sample of 134 students and the experimental group consists of 39 students that were trained through the proposed course during the fall semester of 2019 Average Standard deviation Control 27.47 12.08 Experimental 59.99 24.53

Students from the experimental group seem to show a significant improvement in their performance according to the results shown in Table 4. An analysis of variance was performed where a value of F = 129.2875 with (1,171) degrees of freedom was determined. The statistical study, determines that there is no correlation between the two samples.

5 Conclusions The results obtained in this work suggest that high school graduates in Mexico are poorly prepared for AP content. There is a well stablished culture on advanced placement courses in the U.S. and Canada, where students are thoroughly trained in high schools for the AP exam. According to the results obtained in this work, Mexican students are ill prepared for such exams, because high schools in Mexico tend to focus on the algorithmic aspects of Physics. The scores obtained by the sample of students on the first year of this research, indicate that there is no significant improvement on their performance. The study should be incremental, and during the second year of this research, a different group of students, was trained through a course with AP content and philosophy. This new group of high school graduates received training through online assignments with hints and it was tested through a conceptual exam with a lower level of difficulty. The results obtained on the second year of this research indicate a

Advanced Placement Physics Exam Performance of High School Graduates

49

statistically significant improvement on the performance of students from the experimental group. To the authors’ knowledge, this is the first time that AP content is applied at ITESM-CEM. High school graduates had to unlearn what they are taught in traditional courses, in order to appreciate what Physics is really about. Finally, the authors suggest further studies where AP content will be applied on freshmen students for the courses of Thermodynamics and Electromagnetism. Acknowledgements. “The authors would like to acknowledge the financial support of Writing Lab, TecLabs, Tecnologico de Monterrey, Mexico, in the production of this work”.

References College Board (2018). https://apcentral.collegeboard.org/about-ap/ap-a-glance Dougherty, C., Mellor, L., Jian, S.: The relationship between advanced placement and college graduation. National Center for Educational Accountability, AP Study Series Report 1 (2006) Ewing, M., Howell, J.: Is the relationship between ap participation and academic performance really meaningful? College Board. College Board Research, Research Brief (2015) Geiser, S., Santelices, V.: The role of Advanced Placement and honors courses in college admissions. CSHE Research and Occasional Paper Series, pp. 1–29 (2004) Klopfenstein, K., Thomas, M.K.: The link between advanced placement experience and early college success. South. Econ. J. 75(3), 873–891 (2009) Mattern, D., Shaw J., Xiong X.: The realationship between AP exam performance and college outcomes. college board. College Board Research, Report No. 2009-4 (2009) Rothschild, E.: Four decades of the advanced placement program. Hist. Teach. 32(2), 175–206 (1999) Tec-21: Instituto Tecnológico y de Estudios Superiores de Monterrey (ITESM) (2019). https:// tec.mx/en/tec21 Van Lehn, K., et al.: The andes physics tutoring system: lessons learned. Inte. J. Artif. Intell. Educ. 15(3), 1–47 (2005) Willingham, W.W., Morris, M.: Four years later: a longitudinal study of Advanced Placement students in college. College Board. College Board Research Report No. 86-2 (1986) PISA: Program for International Student Assessment. PISA 2018 (2018). https://www.oecd.org/ pisa/publications/PISA2018_CN_MEX_Spanish.pdf

Hand Robotics Rehabilitation in Patients with Multiple Sclerosis: A Pilot Study Marco Tramontano1(&), Laura Casagrande Conti1, Niccolò Marziali1, Giorgia Agostini1, Sara De Angelis1, Giovanni Galeoto2, and Maria Grazia Grasso1 1

2

Fondazione Santa Lucia IRCCS, Rome, Italy [email protected] Department of Public Health and Infectious Diseases, Sapienza University, Rome, Italy

Abstract. Robot devices may be good candidates for neuromotor rehabilitation of people with Multiple Sclerosis, especially for treating upper extremities function limitations (76% of MS patients). The PABLO®-Tyromotion is a sensor-based device characterized by interactive therapy games with audiovisual feedback. The aim of this study was to evaluate the effects of robotictrained motor rehabilitation as a support of the conventional neurorehabilitation, on increasing upper limbs functions of MS patients. An experimental group that performed the PABLO-Tyromotion training and a control group that performed conventional rehabilitation were compared. PABLO-Tyromotion training consisted of 40 min twelve sessions of upper limb training, three times a week, in addition to the conventional therapy. All patients were evaluated before treatment (T0) and after 4 weeks of training (T1). The results showed substantial improvements in the experimental group, compared to the control group, especially regarding muscular recruitment (such as shoulder and elbow flexextension, forearm pronation and supination, thumb and little-finger op-position) and handgrips strength (such as thumb-index grip, thumb-middle finger grip, tridigital grip). These results underline the effectiveness of robot-assisted treatment in upper limb’s recovery in patients with MS. Keywords: Multiple sclerosis Robotics rehabilitation


Tyromotion
Hand rehabilitation


1 Introduction Multiple sclerosis (MS) is an inflammatory, neurodegenerative, demyelinating disorder of the Central Nervous System characterized by focal lymphocytic infiltration that leads to the damage of myelin and axons [1]. The clinical course of the pathology is unpredictable, different from patient to other and may be change over time [2]. Related to the varying possible distribution of damaged areas, patients with SM present a wide range of neurological symptoms that may compromise many core functions performances [3]. One of the most important function impairments are represented by upper extremities function limitations that © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 50–57, 2020. https://doi.org/10.1007/978-3-030-52538-5_6

Hand Robotics Rehabilitation in Patients with Multiple Sclerosis

51

characterize up to 76% of patients with MS influencing the activities of daily living, both in early stages of the disease and in the later stages and leading to important disabilities [4–7]. Neurorehabilitation often represents the only treatment available to improve some functional symptoms and an intensive multidisciplinary rehabilitation is recommended for all patients with MS [8]. Historically, individuals with MS were advised not to exercise as it was clinically observed that some symptoms, as fatigue, worsened after exercise [9]. However, recent studies shown that regular exercise in MS patients may led to achieve some health benefits [10, 11], including an increase of muscle strength and of mobility an improving of Quality of Life [9, 12–17]. Robot devices are increasingly used in the rehabilitation treatment of subjects with neurological disorders [18, 19] and may be good candidates for neuromotor rehabilitation of subjects with MS, as they allow the design of personalized training protocols and permit quantitative measurement of the motor performances during training [20]. The PABLO® Upper Extremity is a sensor-based device product by Tyromotion for unilateral and bilateral training of upper limb. Thanks to interactive therapy games with audio-visual feedback, it allows to train the movement of shoulders, elbows and writs and to measure the strength of hand functions and the active rage of motions of the upper extremity. Three sensors permit to detect the accelerations of movements in the three dimensions of the space and one sensor detect the hand’s force applied during the movements. Obtained data were registered in an electronic database as quantitative measures and may be used to monitoring individual data and therapy progress. PABLO®-Tyromotion is considered a neurocognitive task-oriented approach of rehabilitation and required an active participation of patients. Based on these considerations, our hypothesis is that a robotic hand training performed with PABLO®-Tyromotion in addition to the conventional neurorehabilitation, may could increase the effect of conventional therapy in upper limbs functions of MS patients.

2 Materials and Methods 2.1

Participants

Twelve patients (6 males, mean age 50, 3 years) with a diagnosis of MS were recruited and enrolled on the basis of consecutive sampling from January 2019 to December 2019 at the Fondazione Santa Lucia (FSL), Institute for Research and Health Care. Participants were randomly assigned to one of two groups: experimental (TYRg) or control group (CTRLg). Demographic characteristics of the sample are reported in Table 1.

52

M. Tramontano et al.

Table 1. Demographic characteristics at baseline. Mean ± standard deviation; M = Male; EDSS = Expanded Disability Status Scale. GROUP TYRg Mean ± DS CTRLg Mean ± DS Age [years] 48.3 ± 9 56.0 ± 6.0 Gender 6M 6M EDSS 7.1 ± 1.3 7.1 ± 1.3

Inclusion criteria were subjects aged between 25 and 80 years with diagnosis of MS and upper limb deficits. Exclusion criteria were: (1) Modified Ashworth Scale (MAS) [21] < 3 at the upper limb; (2) cognitive deficits affecting the ability to understand task instructions (Mini-Mental State Examination < 24 [22]); (3) Medical Research Council (MRC) scale [23] with score 0 or 5; (4) presence of clinically evaluated severe comorbidities; (5) pregnancy; (6) subjects with artificial pacemaker; (7) subjects involved in other studies. 2.2

Experimental Protocol

This was a two-arm single-blind randomized controlled trial. A researcher who was not involved in the intervention sessions assessed the patients’ eligibility to participate based on the inclusion and exclusion criteria and performed the randomization. Block randomization was performed with a computer-generated randomization list using a block size. Allocation concealment was ensured by using an automatic random number generator (www.random.org). The researcher responsible for the randomization process deposited the list in a secure web-based storage. This study was developed by rehabilitation professionals, from Fondazione Santa Lucia in collaboration with ROMA – Rehabilitation & Outcome Measures Assessment Association [24–28]. All patients were evaluated before treatment (T0) and after 4 weeks of training (T1) using the Mini-Mental State Examination (MMSE), Modified Barthel Index (MBI) [29], Rivermead Mobility Index (RMI) [30], Fatigue Severity Scale (FSS) [31], 9 Hole Peg Test (9HPT) [32], Multiple Sclerosis Quality of Life-54 (MSQOL-54) [33], Medical Research Council (MRC) Expanded Disability Status Scale (EDSS) [34] and Tyromotion-PABLO® System in order to assess the exerted force for pinch, lateralthree point and interdigital grips and to assess upper extremities articular motion range (shoulder, elbow and wrist). 2.3

Intervention

TYRg performed twelve sessions of upper limb training with PABLO®-Tyromotion. CTRLg performed twelve session of upper limb sensory-motor training, without robotic support. Both groups performed the training three times a week for 4 weeks. Each session last 40 min and was performed in addition to the conventional therapy.

Hand Robotics Rehabilitation in Patients with Multiple Sclerosis

2.4

53

Statistical Analysis

All the statistical analyses were carried out with the IBM SPSSS Statistic Software version 23, IBM Corp., Armonk, NY, U.S.A. Data were reported in terms of means and standard deviations. The Friedman Test and the Wilcoxon signed ranks test were used for within-subjects comparison for both groups at times T0-T1. The ANOVA was used to analyze TIME  GROUP interaction effect at T0 and T1.

3 Results The statistical analysis of all the single assessments (clinical and functional scales and measurements carried out by the instrument itself) has highlighted substantial improvements in the experimental group (TYRg), compared to the control group (CTRLg), regarding multiple factors. The present study considered numerous indicators, especially those related to the articulation and strength of the upper limb. This allowed to observe in detail the effects of the experimental treatment. The improvement in muscular recruitment in the experimental group’s patients was significant, compared to the control group, especially in: shoulder flexion and extension, elbow flexion and extension, forearm supination and pronation, wrist extension, metacarpophalangeal joints flexion and extension, interphalangeal joints flexion, MCP and IP joints of the thumb flexion and extension, thumb abduction and adduction, thumb and little-finger opposition. Regarding prehension, all 10 types of grips and pinch evaluated with PABLO® showed an improvement of both groups, but there is a substantial improvement in the patients of the experimental group who worked with PABLO®, especially in the strength of thumb-index grip, thumb-middle finger grip and in the tridigital grip, which are the grips with the greatest practical and functional importance in daily activities (such as eating, dressing, manipulating objects). In particular, the flex-extension of the metacarpophalangeal and interphalangeal joints of the thumb have found significant improvements in patient subjected to PABLO® treatment (Table 2).

Table 2. Clinical scales scores. Mean ± standard deviation; EDSS = Expanded Disability Status Scale; 9-HPT = 9 Hole Peg Test; FSS = Fatigue Severity Scale; MBI = Modified Barthel Index; RMI = Rivermead Mobility Index; MSQoL-54 = Multiple Sclerosis Quality of Life-54 TYRg T0 mean ± SD 9-HPT 165.7 ± 141.7 FSS 47.2 ± 9.9 MBI 60.3 ± 30.4 RMI 5.1 ± 4.8 MSQoL-54 142.2 ± 11.7

T1 mean ± SD 152.0 ± 125.8 46.5 ± 9.1 62.0 ± 31.9 5.4 ± 5.2 144.9 ± 10.8

CTRLg T0 mean ± SD 105.5 ± 64.5 40.0 ± 14.7 50.3 ± 27.6 3.6 ± 3.4 144.73 ± 11.6

T1 mean ± SD 105.6 ± 86.9 39.1 ± 13.6 51.4 ± 28.6 4.2 ± 3.9 144.2 ± 10.9

54

M. Tramontano et al.

The statistical analysis of all the single assessments (clinical and functional scales and measurements carried out by the instrument itself) has highlighted substantial improvements in the experimental group (TYRg), compared to the control group (CTRLg), regarding multiple factors. The present study considered numerous indicators, especially those related to the articulation and strength of the upper limb. This allowed to observe in detail the effects of the experimental treatment. The improvement in muscular recruitment in the experimental group’s patients was significant, compared to the control group, especially in: shoulder flexion and extension, elbow flexion and extension, forearm supination and pronation, wrist extension, metacarpophalangeal joints flexion and extension, interphalangeal joints flexion, MCP and IP joints of the thumb flexion and extension, thumb abduction and adduction, thumb and little-finger opposition. Regarding prehension, all 10 types of grips and pinch evaluated with PABLO® showed an improvement of both groups, but there is a substantial improvement in the patients of the experimental group who worked with PABLO®, especially in the strength of thumb-index grip, thumb-middle finger grip and in the tridigital grip, which are the grips with the greatest practical and functional importance in daily activities (such as eating, dressing, manipulating objects). In particular, the flex-extension of the metacarpophalangeal and interphalangeal joints of the thumb have found significant improvements in patient subjected to PABLO® treatment.

4 Discussion This study started with the aim of evaluating the effects of a robotic-trained motor rehabilitation of the upper limb in patients with Multiple Sclerosis (MS). From the comparison of the data obtained from the two groups (experimental and control group) there is no significant improvement in the scales used (EDSS, BMI, Rivermead, MSQOL-54 and FSS). In the 9 Hole Peg Test (9-HPT), both groups have shown improvements, although the experimental group presented better scores, even if this data is not statistically significant. In particular, the best results obtained by the experimental group were the increase in shoulder flexion and extension strength, as well as the interphalangeal joints flexion, metacarpophalangeal and interphalangeal of the thumb flexion and extension, thumb abduction and adduction, thumb and littlefinger opposition and fine hand grip (in total, the TYRg showed a statistically significant improvement in 16 out of 44 components taken into consideration). The control group showed improvements in: shoulder abduction and adduction, elbow flexion, forearm pronation and supination, metacarpophalangeal flex-extension, thumb abduction (in total, CTRLg showed a statistically significant improvement in 8 out of 44 components taken into consideration). Scientific literature shows that subjects with central nervous system lesions have great recovery potential if they follow a repetitive, frequent, intense and oriented rehabilitation in terms of functional recovery [35–38]. To obtain these results, it is useful to take advantage of innovative strategies, such as the use of robot-assisted treatment, because they motivate the patient and get him

Hand Robotics Rehabilitation in Patients with Multiple Sclerosis

55

ready quickly, and it offers a repetitive training that evaluates objectively the achieved progress, after performing exercises that are, at the same time, fun and motivating. Moreover, considering the ease of use and the affordable of the PABLO®-Tyromotion device, it may be possible to propose it in a telerehabilitation program [39], in order to integrate the conventional clinical therapy with a robot-assisted treatment at home. 4.1

Limitation of the Study

This study has certain limitations. First, our sample included patients with different clinical variants of Multiple Sclerosis. Moreover, the numerical sample is limited and not sufficiently numerous, so other studies are required. 4.2

Conclusions

PABLO®-Tyromotion innovative rehabilitation treatment has proven to help increasing functional hand recovery in MS patients. Using the robotic treatment in addition to the conventional neurorehabilitation seems to enhance the improvement of some of the upper extremity functions, compared to the standard therapy. The simplicity and easiness of the treatment, as well as the lack of side effects and the positive results founded in patients who used this type of treatment, support the idea of extending its clinical trial, as a support to the conventional neurorehabilitation. Furthermore, the presence of exercises with an increasing level of intensity and difficulty represent an excellent motivation tool for the patients. Ethics Statement. This single-blind randomized-controlled trial was approved by the Local Ethics Committee of Fondazione Santa Lucia (FSL), all participants or parents gave their written informed consent for participation in the study.

References 1. Compston, A., Coles, A.: Multiple sclerosis. Lancet 372(9648), 1502–1517 (2008) 2. Compston, A., Coles, A.: Multiple sclerosis. Lancet 359(9313), 1221–1231 (2002) 3. Pompa, A., Morone, G., Iosa, M., Pace, L., Catani, S., Casillo, P., Clemenzi, A., et al.: Does robot- assisted gait training improve ambulation in highly disabled multiple sclerosis people? A pilot randomized control trial. Mult. Scler. 23(5), 696–703 (2017) 4. Einarsson, U., Gottberg, K., Von Koch, L., Fredrikson, S., Yitterberg, C., Jin, Y.P., et al.: Cognitive and motor function in people with multiple sclerosis in Stockholm County. Mult. Scler. 12(2), 340–353 (2006) 5. Goodkin, D.E., Hertsgaard, D., Seminary, J.: Upper extremity function in multiple sclerosis: improving assessment sensitivity with box-and-block and nine-hole peg tests. Arch. Phys. Med. Rehabil. 69(10), 850–854 (1988) 6. Johansson, S., Ytterberg, C., Claesson, I.M., et al.: High concurrent presence of disability in multiple sclerosis. J. Neurol. 254(6), 767–773 (2007) 7. Kamm, C.P., Heldner, M.R., Vanbellingen, T., et al.: Limb apraxia in multiple sclerosis: prevalence and impact on manual dexterity and activities of daily living. Arch. Phys. Med. Rehabil. 93(6), 1081–1085 (2012)

56

M. Tramontano et al.

8. Grasso, M.G., Troisi, E., Rizzi, F., Morelli, D., Paolucci, S.: Prognostic factors in multidisciplinary rehabilitation treatment in multiple sclerosis: an outcome study. Mult. Scler. 11(6), 719–724 (2005) 9. Smith, C., Hale, L., Olson, K., Schneiders, A.G.: How does exercise influence fatigue in people with multiple sclerosis? Disabil. Rehabil. 31(9), 685–692 (2009) 10. Slawta, J.N., Wilcox, A.R., McCubbin, J.A., Nalle, D.J., Fox, S.D., Anderson, G.: Health behaviors, body composition, and coronary heart disease risk in women with multiple sclerosis. Arch. Phys. Med. Rehabil. 84(12), 1823 (2003) 11. Campbell, E., Coulter, E.H., Mattison, P.G., Miller, L., Mcfadyen, A., Paul, L.: Physiotherapy rehabilitation for people with progressive multiple sclerosis: a systematic review. Arch. Phys. Med. Rehabil. 97(1), 141–51 (2016) 12. Sevensson, B., Gerdle, B., Elert, J.: Endurance training in patients with multiple sclerosis: five case studies. Phys. Ther. 74(11), 1017–1026 (1994) 13. DeBolt, L.S., McCubbin, J.A.: The effects of home-based resistance exercise on balance, power and mobility in adults with multiple sclerosis. Arch. Phys. Med. Rehabil. 85(2), 290– 297 (2004) 14. Petajan, J.H., Gappmaier, E., White, A.T., Spencer, M.K., Mino, L., Hicks, R.W.: Impact of aerobic training on fitness and quality of life in multiple sclerosis. Ann. Neurol. 39(4), 432– 441 (1996) 15. Mostert, S., Kesselring, J.: Effects of a short-term exercise training program on aerobic fitness, fatigue, health perception and activity level of subjects with multiple sclerosis. Mult. Scler. 8(2), 161–168 (2002) 16. Tramontano, M., Martino Cinnera, A., Manzari, L., Tozzi, F.F., Caltagirone, C., Morone, G., et al.: Vestibular rehabilitation has positive effects on balance, fatigue and activities of daily living in highly disabled multiple sclerosis people: a preliminary randomized controlled trial. Restor. Neurol. Neurosci. 36(6), 709–718 (2018) 17. Motl, R.W., Gosney, J.L.: Effect of exercise training on quality of life in multiple sclerosis: a meta-analysis. Mult. Scler. 14(1), 129–135 (2008) 18. Prange, G.B., Jannink, M.J., Groothius-Oudshoorn, C.G., Hermens, H.J., Ijzerman, M.J.: Systematic review of the effect of robot-aided therapy on recovery of the hemiparetic arm after stroke. J. Rehabil. Res. Dev. 43(2), 171–184 (2006) 19. Morone, G., Paolucci, S., Mattia, D., Pichiorri, F., Tramontano, M., Iosa, M.: The 3Ts of the new millennium neurorehabilitation gym: therapy, technology, translationality. Expert Rev. Med. Dev. 13(9), 785–787 (2016) 20. Carpinella, I., Cattaneo, D., Abuarqub, S., Ferrarin, M.: Robot-based rehabilitation of the upper limbs in multiple sclerosis: feasibility and preliminary results. J. Rehabil. Med. 41(12), 966–970 (2009) 21. Ansari, N.N., Naghdi, S., Arab, T.K., Jalaie, S.: The interrater and intrarater reliability of the modified Ashworth scale in the assessment of muscle spasticity: limb and muscle group effect. NeuroRehabilitation 23(3), 231–237 (2008) 22. Folstein, M.F., Folstein, S.E., McHugh, P.R.: Mini-mental state. A practical method for grading the cognitive state of patients for the clinician. J. Psychiatr. Res. 12(3), 189–198 (1975) 23. Galeoto, G., Formica, M.C., Mercuri, N.B., Santilli, V., Berardi, A.C., Castiglia, S.F.: Evaluation of the psychometric properties of the Barthel Index in an Italian ischemic stroke population in the acute phase: a cross-sectional study. Funct. Neurol. 34(1), 29–34 (2019) 24. Galeoto, G., Iori, F., De Santis, R., Santilli, V., Mollica, R., Marquez, M.A., et al.: The outcome measures for loss of functionality in the activities of daily living of adults after stroke: a systematic review. Top. Stroke Rehabil. 26(3), 236–245 (2019)

Hand Robotics Rehabilitation in Patients with Multiple Sclerosis

57

25. Berardi, A., Biondillo, A., Màrquez, M.A., De Santis, R., Fabbrini, G., Tofani, M., et al.: Validation of the short version of the Van Lieshout Test in an Italian population with cervical spinal cord injuries: a cross-sectional study. Spinal Cord 57(4), 339 (2019) 26. Galeoto, G., Scialpi, A., Grassi, M.L., Berardi, A., Valente, D., Tofani, M., et al.: General sleep disturbance scale: translation, cultural adaptation, and psychometric properties of the Italian version. In: CRANIO® 11, pp. 1–9 (2019) 27. Savona, A., Ferralis, L., Saffioti, M., Tofani, M., Nobilia, M., Culicchia, G., et al.: Evaluation of intra-and inter-rater reliability and concurrent validity of the Italian version of the Jebsen–Taylor Hand Function Test in adults with rheumatoid arthritis. Hand Therapy 24 (2), 48–54 (2019) 28. Berardi, A., Dhrami, L., Tofani, M., Valente, D., Sansoni, J., Galeoto, G.: Cross-cultural adaptation and validation in the Italian population of the wolf motor function test in patients with stroke. Funct. Neurol. 33(4), 229–253 (2018) 29. Castiglia, S.F., Galeoto, G., Lauta, A., Palumbo, A., Tirinelli, F., Viselli, F., et al.: The culturally adapted Italian version of the Barthel Index (IcaBI): assessment of structural validity, inter-rater reliability and responsiveness to clinically relevant improvements in patients ad- mitted to inpatient rehabilitation centers. Funct. Neurol. 22(4), 221–228 (2017) 30. Franchignoni, F., Tesio, L., Benevolo, E., Ottonello, M.: Psychometric properties of the Rivermead Mobility Index in Italian stroke rehabilitation inpatients. Clin. Rehabil. 17(3), 273–282 (2003) 31. Ottonello, M., Pellicciari, L., Giordano, A., Foti, C.: Rasch analysis of the Fatigue Severity Scale in Italian subjects with multiple sclerosis. J. Rehabil. Med. 48(7), 597–603 (2016) 32. Solaro, C., Cattaneo, D., Brichetto, G., Castelli, L., Tacchino, A., Gervasoni, E., et al.: Clinical correlates of 9-hole peg test in a large population of people with multiple sclerosis. Mult. Scler. Relat. Disord. 30, 1–8 (2019) 33. Solari, A., Filippini, G., Mendozzi, L., Ghezzi, A., Cifani, S., Barbieri, E., et al.: Validation of Italian multiple sclerosis quality of life 54 questionnaire. J. Neurol. Neurosurg. Psychiatry 67(2), 158–162 (1999) 34. Kurtzke, J.F.: Rating neurologic impairment in multiple sclerosis: an expanded disability status scale (EDSS). Neurology 33(11), 1444–1452 (1983) 35. Tramontano, M., Bergamini, E., Iosa, M., Belluscio, V., Vannozzi, G., Morone, G.: Vestibular rehabilitation training in patients with subacute stroke: a preliminary randomized controlled trial. NeuroRehabilitation 43(2), 247–254 (2018) 36. Tramontano, M., Dell’Uomo, D., Cinnera, A.M., Luciani, C., Di Lorenzo, C., Marcotulli, M., et al.: Visual-spatial training in patients with sub-acute stroke without neglect: a randomized, single-blind controlled trial. Funct. Neurol. 34(1), 7–13 (2019) 37. Tramontano, M., Bonnì, S., Martino Cinnera, A., Marchetti, F., Caltagirone, C., Koch, G., et al.: Blindfolded Balance Training in Patients with Parkinson’s Disease: A Sensory-Motor Strategy to Improve the Gait. Parkinson’s Disease (2016) 38. Bonnì, S., Ponzo, V., Tramontano, M., Martino Cinnera, A., Caltagirone, C., Koch, G., et al.: Neurophysiological and clinical effects of blindfolded balance training (BBT) in Parkinson’s disease patients: a preliminary study. Eur. J. Phys. Rehabil. Med. 55(2), 176– 182 (2019) 39. Bustamante Valles, K., Montes, S., Madrigal Mde, J., Burciaga, A., Martínez, M.E., Johnson, M.J.: Technology-assisted stroke rehabilitation in Mexico: a pilot randomized trial comparing traditional therapy to circuit training in a Robot/technology-assisted therapy gym. J. Neuroeng. Rehabil. 13(1), 83 (2016)

When Something Useful Is Also Enjoyable: An Empirical Study on the Intention to Use Web-Based Simulations in Higher Education Leonardo Caporarello1,2, Federica Cirulli2(&), Federico Magni3, and Beatrice Manzoni1 1

SDA Bocconi School of Management, Bocconi University, 20136 Milan, Italy [email protected] 2 BUILT, Bocconi University, 20136 Milan, Italy {leonardo.caporarello,federica.cirulli}@unibocconi.it 3 Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong, China

Abstract. The adoption of web-based simulations in higher education has been evolving significantly in recent years. One of the aspects determining the success of simulations as a teaching method is students’ acceptance. For this reason, this study aims to test a model of students’ adoption of web-based simulations in a university context. We adopted a revised version of the Technology Acceptance Model (TAM) to investigate how the interaction between perceived usefulness (PU) and perceived enjoyment (PE) can influence students’ intention to use (ITU) simulations through an increase in user satisfaction (US). We collected data on 191 university students using the TAM based questionnaire and tested our hypothesis with a moderated mediation model. The results support the mediating effect of US in predicting the impact on ITU of the interaction of PU and PE. Consequently, we recommend that designers and stakeholders pay attention to not only users’ PU, but also users’ enjoyment during simulations, so as to increase users’ willingness to adopt simulations as a learning method. Keywords: Web-based simulations
TAM model
Perceived enjoyment Perceived usefulness
User satisfaction
Intention to use




1 Introduction Nowadays traditional learning models and methods are losing their primacy in favour of new approaches that allow students to actively engage in learning experiences [1]. Learning models are evolving from face to online and blended. In terms of methods, there is a decrease in instructor-led lectures and an increase in experiential learning, which engages students in a real-world context [2, 3]. Among the most common experiential learning methods are web-based simulations, which allow participants to live a complex real-life situation and to use their skills in a given online scenario [4]. Simulations are considered suitable methods to enhance learning and to improve students’ performance, knowledge retention, and © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 58–67, 2020. https://doi.org/10.1007/978-3-030-52538-5_7

When Something Useful Is Also Enjoyable

59

involvement [5]. Moreover, simulations increase students’ enjoyment, which is a strong motivational driver in the learning process according to the existing literature [6]. Considering the interest towards students’ intention to use these tech-based methods, many studies [7, 8] developed theoretical and empirical models to investigate the factors that either encourage or hinder their adoption, starting from the work of Davis [9] who developed the Technology Acceptance Model (TAM). This model has been applied in various fields, such as online services [10] and mobile phones [11]. More recently, revised TAM models have been applied to target virtual environments [12] as well as acceptance factors in e-learning [13]. Moreover, existing research aims at extending the TAM by studying previously disregarded variables [14]. Specific studies have considered perceived usefulness (PU) as a preeminent factor in the investigation of technologies’ adoption as well as its impact on users’ intention to use (ITU) simulations [9, 15]. Students who perceive a proposed system as useful, would have a stronger ITU, and would be more likely to use it again. Additionally, a positive effect of PU on user satisfaction (US) has been tested in previous studies [16, 17]. This means that when users perceive a technology as useful, they also feel higher levels of satisfaction and, as a consequence, use the technology better [18]. Other studies added perceived enjoyment (PE) as a variable in the TAM model [19]. For example, a study set in Singapore [20] found that PE has a positive impact on Internet usage. Thus, we expect that enjoyment will also be a factor in predicting ITU. For what concerns our study, as web-based simulations are leisure-oriented methods [21], we can expect that students’ PE will be important to assess their ITU. While an enjoyable simulation should evoke a positive disposition towards it, a not enjoyable one may be perceived as not valuable to be used [22]. Even though the influence of PU on ITU, as well as the impact of PU on US, have been well validated in previous research [9, 11–13, 15], the study of the interaction between PU and PE and their impact on the students’ ITU has not been studied in the extant literature, neither directly, nor mediated by US. Nowadays, in the field of learning it is crucial to understand users’ intention to adopt specific teaching methods [5]. Given that, as far as we know, the interaction between PU and PE and the mediating role of US have not been investigated yet, our study offers an alternative framework for examining how enjoyment and usefulness perceptions interact in predicting ITU through US. The paper’s structure is as follows. In the next section, we review the relevant literature and set out our hypothesis. Afterwards, we present the methodology used to test the hypothesis, including the study context and sample, followed by the findings section. Finally, we discuss the results and their interpretation together with concluding ideas, limitations, and proposals for future research.

60

L. Caporarello et al.

2 Theoretical Framework 2.1

Interaction Between PU and PE and Web-Based Simulations Adoption

The TAM has dominated the research landscape as the most commonly adopted model to explain behavioural intentions about the use of technologies [9]. TAM theory revolves around two constructs. The first one is perceived ease of use (PEU), which refers to the extent to which a person believes that using technology would be free from effort [23]. The second construct is PU, which refers to the extent to which an individual believes that adopting a particular technology will enhance her/his specific job results [9]. Venkatesh and Davis [24] found that the perception of usefulness remains a significant determinant of ITU over time. In particular, PU is a predictor of ITU because individuals who perceive a tool as useful will have a stronger intention to use it. So, ITU can be considered a measure of the likelihood that a person will employ a specific tech-based learning method [25]. The TAM model has been extended to comprise many variables, like self-efficacy, subjective norms, and facilitating conditions of technology use [9, 26]. Another important construct that has been subsequently added to the model is US, defined as the net feeling of pleasure or displeasure that results from aggregating all the benefits that a person hopes to receive from interacting with technology [27]. Alternatively, Seddon et al. [28] considered US as a subjective evaluation of the various consequences (individual, organizational, social) evaluated on a pleasant-unpleasant continuum. In their research on the factors and beliefs that affect the constant use of simulations, PU is considered as a main predictor of US [29]. Similarly, Mann and Sahni [30] empirically validated a direct and positive relationship between PU and students’ e-satisfaction. In their view, the PU of a service establishes a positive and satisfactory perception towards it. In addition, Wixom and Todd [31] proposed a theoretical model that linked US and ITU. Their model built a bridge from design and decision implementation to system characteristics (a core element of US), as well as to decisions about technology acceptance (a core element of ITU). In line with this, DeLone and McLean [32] showed that US positively influences ITU. More recent analyses integrated the PE variable in the TAM [33, 34]. This happened because enjoyment is fundamental for tech-based learning methods to succeed. In this perspective, PE is defined as “the degree to which the activity of using technology is perceived to be enjoyed in its own, right apart from any performance consequences that may be anticipated” [35]. Students who expressed enjoyment towards a tech-based method experienced an overall progression in their conceptual understanding [36]. For this reason, enjoyment represents one of the main aspects that determines the adoption of simulations [37]. As a consequence, one of the objectives of the actual research is to determine the chain of influence between PE and ITU [38]. In this regard, a study carried out by Wong and Huang [39] showed that PE significantly influences the intention to use Mlearning systems. Other studies [40, 41] included PE in a revised version of TAM to

When Something Useful Is Also Enjoyable

61

predict user acceptance and adoption of a specific source, finding a positive effect of PE on the usage of a determined system [42]. As a result of the great debate about TAM for its ability to predict ITU, Dishaw and Strong [24] suggested some improvements. In particular, they pointed out the need to further explore the interactions among the variables that directly affect ITU. Others [30, 33] suggested to develop further analyses on PE to achieve greater clarity in the relationship about PU and ITU. The causal relationship existing between PU and ITU has been investigated by various studies [24, 43, 44]. Conversely, the indirect effect of the interaction of PU and PE is under researched. Specifically, to the best of our knowledge, the effect of the interaction between PE and PU on ITU, further mediated by US has not been investigated yet. For this reason, we answer the call of previous researchers for an analysis of the dynamics linking PU, PE, and ITU by explaining participants’ ITU through the interaction of PE with PU. From the discussion above, the following research hypothesis is proposed: Hypothesis: PE moderates the positive relation between PU and ITU mediated by US, such that this relationship is stronger when PE is higher rather than lower.

The theoretical model is presented in Fig. 1.

Fig. 1. Research model.

3 Research Method 3.1

Sample and Data Collection

We collected data at the individual level between September 2018 and September 2019 on a sample of students at an Italian university. Participation to the study was voluntary and encouraged by students’ instructors ensuring confidentiality. A total of 191 participants completed the questionnaire via an online survey. As far as our sample is concerned, 56,69% of the respondents were male, 63,35% were Italians, 91,1% were students from management courses and 73% were undergraduate belonging to Gen Z so well-accustomed with the use of technology [45]. A sensible part of them has low experience in web-based management simulations because they are at the beginning of their university studies. All the scale items are reported below, and were measured with a seven-point agreement Likert scale (‘‘strongly agree to strongly disagree’’).

62

L. Caporarello et al.

Intention to Use (ITU). We measured ITU with the 3-item scale developed by Balog and Pribeanu [19], with sample item “I will recommend to other colleagues to use management simulations”. User Satisfaction (US). We measured US with the 4-item scale developed by Alshare et al. [46], with sample item “Overall, the performance of the management simulation was good”. Perceived Usefulness (PU). We measured PU with the 4-item scale developed by Davis [9], with sample item “Using the management simulation enabled me to accomplish tasks more quickly”. Perceived Enjoyment (PE). We measured PE with the 4-item scale developed by Wang et al. [23], with sample item “Using the management simulation was exciting”. We further added participants’ age and gender as control variables.

4 Findings Means, standard deviations, and correlations are shown in Table 1. Table 1. Means, standard deviations, and bivariate correlationsa Variables M SD Age Gender 1. Age 19 1.25 2. Gender .40 .49 .15* 3. PU 5.23 1.27 −.07 −.08 4. PE 5.80 1.15 −.15* −.14 5. US 5.79 .96 −.22* −.03 6. ITU 5.92 1.07 −.14 −.08 a Notes: n = 191. Cronbach’s alphas parentheses. *p < 0.05 **p < 0.01

PU

PE

US

ITU

(.91) .60** (.90) .62** .64** (.88) .50** .60** .63** (.90) are on the diagonal in

According to the results obtained by the correlation analysis and to create the mediation models [47], we used linear regression models on SPSS and the PROCESS macro to test our hypothesis. The relationship between the independent variable PU and the dependent variable ITU is hypothesized to be an indirect effect that exists due to the influence of US. On SPSS, we used linear regression models and the PROCESS macro to test our hypothesis. In particular, we ran three different statistical models. In Model 1, we regressed our mediator US on the control variables, the independent variable PU, the moderator PE, and their interaction and we found that the interaction impacted US (B = 0.05, P < 0.05). In Model 2, we regressed the same covariates on our dependent variable ITU. Then, in Model 3 we added US as a covariate to test our mediation, and we found that US impacts ITU (B = 0.42, P < 0.001). Results are reported in Table 2.

When Something Useful Is Also Enjoyable

63

Table 2. Linear regression results (standardised coefficients) Variables US Model 1 Age −.10** Gender .13 PU −.01 PE .10 PUxPE .05* US – Notes: n = 191 *p < 0.05 **p < 0.01 ***p < 0.001

ITU Model 2 Model 3 −.05 −.00 .03 −.02 .10 .10 .30 .26 .02 −.00 – .42***

We found a significant effect on US of the interaction between PU and PE, but we did not find a direct effect of either of them, inconsistently with the existing literature. Suspecting an issue caused by multicollinearity [48], we ran two further regression models excluding the interaction term to test the direct effects of PU and PE on the mediator and the dependent variable. Consistently with the existing literature, we found a positive effect of both PU and PE on US (b = .29, p < 0.000; and b = .33, p < 0.000 respectively) and on ITU (b = .22, p < 0.000; and b = .40, p < 0.000 respectively) [9, 19–23]. Finally, regressions values are showed in Fig. 2. The effect of PU on US is −0.01, the effect of PU and PE on US is 0.05, the effect of US on ITU is 0.42* and the effect of PU on ITU is 0.10.

Fig. 2. Research model: results. Note: n = 191 ***p < 0.000

5 Discussion and Conclusions This study’s findings add further evidence on the adaptability and applicability of TAM in explaining the intention of students to use web-based simulations. Consistently with our hypothesis, our results show that PU and PE interact in predicting ITU via US in the domain of web-based simulations, such that a higher level of enjoyment strengthens the impact of the perception of usefulness on users’ satisfaction, which in turns increases their ITU.

64

L. Caporarello et al.

Moreover, PU and US have been confirmed to be antecedents of the intention to adopt web-based simulations [24, 25]. Indeed, our study further supports the argument that PU is a main predictor of ITU. This means that users will have a stronger intention to adopt a specific learning method, the more they perceive its usefulness. Concerning the mediating role of US, we showed that when students’ satisfaction in using a learning system increases, they show a positive tendency to adopt it again. This means that satisfaction, which is predicted by the interaction of usefulness perceptions and enjoyment, positively impacts the further adoption of web-based learning methods [49]. Nowadays, a great importance has been given to the use of simulations for their capacity to offer experiential learning opportunities [50]. In this perspective, our results give information about the predictors of the use of web-based simulations, shading a light on the relation between PU and ITU, moderated by PE, and further mediated by US. In terms of practical implications, our results imply that it is necessary for instructional designers and practitioners to make effort in turning web-based simulations into useful, pleasant and satisfactory learning methods, if they aim to spread their usage. Instructional designers and instructors should look at enjoyment as a powerful tool to unlock the potential of PU. The simulation per se has to be enjoyable, yet without losing sight of its actual usefulness, and its impact on users’ knowledge and skills acquisition. Also, UX designers should reflect on how to make the navigation experience fun by, for example, working on the interface and the graphics, also bearing in mind the potential of gamification, but without disregarding the PU, as the two components of usefulness and enjoyment work together. Our results may have limited generalizability due to the specificity of our sample. In particular, age could be a limitation considering that our participants are mainly students born in the 1990s. Future research could consider more age-diverse samples of respondents including different levels of digital literacy to test the robustness of the current results. Finally, in order to increase the predictive power of the model, future research could be directed to incorporate additional variables, such as users’ familiarity in adopting new web-based methods.

References 1. Vinagre, M.: Developing teachers’ telecollaborative competences in online experiential learning. System 64(2), 34–45 (2017) 2. Chakravorty, S., Hales, D.: Sustainability of process improvements: an application of the experiential learning model (ELM). Int. J. Prod. Res. 55(17), 4931–4947 (2017) 3. Caporarello, L., Cirulli F., Manzoni B.: Designing a self-regulated online learning course using innovative methods: a case study. In: Gennari, R., et al. (eds.) Methodologies and Intelligent Systems for Technology Enhanced Learning, 9th International Conference. MIS4TEL 2019. Advances in Intelligent Systems and Computing, vol. 1007. Springer, Cham (2020) 4. Prentice, M., Garcia, R.: Service learning: the next generation in education. Commun. Coll. J. Res. Pract. 24(1), 19–26 (2000)

When Something Useful Is Also Enjoyable

65

5. Druckman, D., Ebner, N.: Onstage, or behind the scenes? Relative learning benefits of simulation role-play and design. Simul. Gaming 39(4), 465–497 (2008) 6. Shellman, S.M., Turan, K.: Do simulations enhance student learning? An empirical evaluation of an IR simulation. J. Polit. Sci. Educ. 2(1), 19–32 (2006) 7. Sánchez-Prieto, J.C., Hernández-García, Á., García-Peñalvo, F.J., Chaparro-Peláez, J., Olmos-Migueláñez, S.: Break the walls! Second Order barriers and the acceptance of mLearning by first-year pre-service teachers. Comput. Hum. Behav. 95, 158–167 (2019) 8. Kim, S., O’Rourke, J.: Effectiveness of online simulation training: measuring faculty knowledge, perceptions, and intention to adopt. Nurs. Educ. Today 51, 102–107 (2017) 9. Davis, F.: Perceived usefulness, perceived ease of use and user acceptance of information technology. MIS Q. 13(3), 319–340 (1989) 10. Ervasti, M., Helaakoski, H.: Case study of application-based mobile service acceptance and development in Finland. Int. J. Inf. Technol. Manag. 9, 243–259 (2010) 11. Basaglia, S., Caporarello, L., Magni, M., Pennarola, P.: Individual adoption of convergent mobile phone in Italy. RMS 3(1), 1–18 (2009) 12. Raaij, E.M.V., Schepers, J.J.L.: The acceptance and use of a virtual learning environment in China. Comput. Educ. 50(3), 838–852 (2006) 13. Cheung, R., Vogel, D.: Predicting user acceptance of collaborative technologies: an extension of the technology acceptance model for e-learning. Comput. Educ. 63(1), 160–175 (2013) 14. Venkatesh, V., Bala, H.: Technology acceptance model 3 and a research agenda on interventions. Decis. Sci. 39(2), 273–315 (2008) 15. Martin, R.G.: Factors affecting the usefulness of social networking in e-learning at German University of Technology in Oman. Int. J. e-Educ. e-Bus. e-Manag. e-Learn. 2(6), 498–502 (2012) 16. Venkatesh, V.: Determinants of perceived ease of use: integrating control, intrinsic motivation, and emotion into the technology acceptance model. Inf. Syst. Res. 11(4), 342– 365 (2000) 17. Lee, D.Y., Lehto, M.R.: User acceptance of YouTube for procedural learning: an extension of the technology acceptance model. Comput. Educ. 61(1), 193–208 (2013) 18. Ahuja, M.K., Thatcher, J.B.: Moving beyond intentions and toward the theory of trying: effects of work environment and gender on post-adoption information technology use. MIS Q. 29(3), 427–459 (2005) 19. Balog, A., Pribeanu, C.: The role of perceived enjoyment in the students’ acceptance of an AR teaching platform: a structural equation modeling approach. Stud. Inf. Control 19(3), 319–330 (2010) 20. Teo, T.S.H., Lim, R.Y.C.: Intrinsic and extrinsic motivation in internet usage. OMEGA: Int. J. Manag. Sci. 27(1), 25–37 (1999) 21. Imlig-Iten, N., Petko, D.: Comparing serious games and educational simulations: effects on enjoyment, deep thinking, interest and cognitive learning gains. Simul. Gaming 49(4), 401– 422 (2018) 22. Rieber, L.P., Noah, D.: Games, simulations, and visual metaphors in education: antagonism between enjoyment and learning. Educ. Media Int. 45(2), 77–92 (2008) 23. Wang, Y., Lin, H.H., Liao, Y.W.: Investigating the individual difference antecedents of perceived enjoyment in students’ use of blogging. BJET 43(1), 139–152 (2012) 24. Venkatesh, V., Davis, F.D.: A model of the perceived ease of use development and test. Decis. Sci. 27(3), 451–481 (1996) 25. Dishaw, M.T., Strong, D.M.: Extending the technology acceptance model with tasktechnology fit constructs. Inf. Manag. 36(1), 9–21 (1999)

66

L. Caporarello et al.

26. Keller, C.: Technology acceptance in academic organisations: implementation of virtual learning environment. In: Proceedings of the 14th European Conference on Information Systems, Gothenburg, Sweden (2006) 27. Kim, S.S., Malhotra, N.K.: A longitudinal model of continued IS use: an integrative view of four mechanisms underlying postadoption phenomena. Manag. Sci. 51(5), 741–755 (2005) 28. Seddon, P.B., Kiew, M.Y.: A partial test and development of the DeLone and McLean model of IS success. In: Proceedings of the International Conference on Information Systems, pp. 99–110. Association for Information Systems, Atlanta (1994) 29. Hsu, M.H., Chiu, C.M.: A predicting electronic service continuance with a decomposed theory of planned behaviour. Behav. Inf. Technol. 23(5), 359–373 (2004) 30. Mann, B.J.S., Sahni, S.K.: Inter-relationship of website interactivity and customer outcomes: building trust in Internet Banking website. Glob. Bus. Rev. 12(1), 99–115 (2011) 31. Wixom, B.H., Todd, P.A.: A theoretical integration of user satisfaction and technology acceptance. Inf. Syst. Res. 16(1), 85–102 (2005) 32. Delone, W.H., Mclean, E.R.: Measuring e-commerce success: applying the DeLone & McLean information systems success model. Int. J. Electron. Commer. 9(1), 31–47 (2004) 33. Shernoff, D.J., Csikszentmihalyi, M., Schneider, B., Shernoff, E.S.: Student engagement in high school classrooms from the perspective of flow theory. Sch. Psychol. Q. 18(2), 158–176 (2003) 34. Goetz, T., Nathan, C., Hall, B., Anne, C., Frenzel, A., Pekrun, R.: A hierarchical conceptualization of enjoyment in students. Learn. Instr. 16, 323–338 (2006) 35. Bagozzi, R.P., Davis, F.D., Warshaw, P.R.: Development and test of a theory of technological learning and usage. Hum. Relat. 45(7), 660–686 (1992) 36. Scherer, R., Siddiq, F., Teo, T.: Becoming more specific: measuring and modeling teachers’ perceived usefulness of ICT in the context of teaching and learning. Comput. Educ. 88, 202– 214 (2015) 37. Skalski, P., Tamborini, R., Shelton, A., Buncher, M., Lindmark, P.: Mapping the road to fun: Natural video game controllers, presence, and game enjoyment. New Media Soc. 13(2), 224–242 (2011) 38. Chatzoglou, P.D., Sarigiannidis, L., Vraimaki, E., Diamantidis, A.: Investigating greek employees’ intention to use web-based training. Comput. Educ. 53(3), 877–889 (2009) 39. Wong, W.T., Huang, N.T.N.: The effects of elearning system service quality and users’ acceptance on organizational learning. Int. J. Bus. Inf. 6(2), 205–225 (2015) 40. Zhou, T.: Understanding users’ initial trust in mobile banking: an elaboration likelihood perspective. Comput. Hum. Behav. 28(4), 1518–1525 (2012) 41. Wu, W.H., Wu, Y.C.J., Chen, C.Y., Kao, H.Y., Lin, C.H., Huang, S.H.: Review of trends from mobile learning studies: a meta-analysis. Comput. Educ. 59(2), 817–827 (2012) 42. Igbaria, M., Guimaraes, T., Davis, G.B.: Testing the determinants of microcomputer usage via structural equation model. J. Manag. Inf. Syst. 11(4), 87–114 (1995) 43. Moon, J.W., Kim, Y.G.: Extending the TAM for a world-wide-web context. Inf. Manag. 38(4), 217–230 (2001) 44. Chang, C.-H., Yan, C.-F., Tseng, J.-S.: Perceived convenience in an extended technology acceptance model: mobile technology and English learning for college students. Aust. J. Educ. Technol. 28(5), 809–826 (2012) 45. Twenge, J.M.: iGen: why today’s super-connected kids are growing up less rebellious, more tolerant, less happy–and completely unprepared for adulthood (and what this means for the rest of us). Unabridged, New York (2017) 46. Alshare, K., Mesak, H., Grandon, E., Badri, M.: Examining the moderating role of national culture on an extended technology acceptance model. J. Glob. Inf. Technol. Manag. 14(3), 27–53 (2011)

When Something Useful Is Also Enjoyable

67

47. Preacher, K.J., Hayes, A.F.: Asymptotic and resampling strategies for assessing and comparing indirect effects in multiple mediator models. Beauvoir Res. Methods 3(3), 879– 891 (2008) 48. Gwowen, S.: Clarifying the role of mean centring in multicollinearity of interaction effects. Br. J. Math. Stat. Psychol. 64(3), 462–477 (2011) 49. Gelderman, M.: The relation between user satisfaction, usage of information systems and performance. Inf. Manag. 34(1), 1–11 (1998) 50. Gatti, L., Ulrich, M., Seele, P.: Education for sustainable development through business simulation games: an exploratory study of sustainability gamification and its effects on students’ learning outcomes. J. Clean. Prod. 207, 667–678 (2019)

TEL Adoption in the Riconnessioni Project Marcello Enea Newman(&) Fondazione per la Scuola della Compagnia di San Paolo, Piazza Bernini 5, 10138 Turin, Italy [email protected]

Abstract. The following paper provides an introduction to the challenges and obstacles to TEL (Technology Enhanced Learning) adoption in the context of the Italian school system and subsequently presents an overview of the Riconnessioni project: a 3-year initiative to innovate Turin’s primary and lowersecondary school system through an increase in TEL adoption and use enabled by state of the art infrastructure and a citywide teacher training programme. The lessons learned from the Riconnessioni project shed light on some of the obstacles preventing widespread adoption of TEL and, on the other hand, strategies than can ensure teacher motivation and engagement in TEL. Keywords: TEL


EdTech
Education

1 EdTech and the Italian School System The education system is remarkably resistant to technological innovation. Schools in Italy today, for the most part, use the same technology, in the same way, as their colleagues teaching a century ago. This technology consists in basic classroom supplies, textbooks, notebooks, a blackboard, the architectural structure of the classroom itself. The way this technology is used has also changed very little in the past century: the teacher stands at the front of the class, presenting information to children who are trained to absorb it. This lack of innovation would be justified if the Italian education system were performing as well as we would like it to, or if there were no available and trusted innovations to adopt to tackle underperformance. Before focusing on these innovations, and how they could be useful, we must briefly lay out the main challenges that Italian schools are facing today. 1.1

Educational Inequality

The Italian educational system is unable to contrast growing inequality in society at large. Of the children whose parents only received a primary school education, only 10% will attain a university degree and only 36% will graduate from upper-secondary school. In families with at least one parent with a university degree, on the other hand, 76% of children will also attain a degree. Lastly, Italy has one of the highest early dropout rates in Europe (OECD 2017).

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 68–73, 2020. https://doi.org/10.1007/978-3-030-52538-5_8

TEL Adoption in the Riconnessioni Project

1.2

69

Obsolete Pedagogy

Although Italian teachers and policymakers nominally embrace child-centred, hands-on workshop-style teaching, the reality of pedagogical practice in the classroom remains overwhelmingly traditionalistic (Castoldi 2017), focusing on the transmission of knowledge and rote learning. This pedagogy is particularly unsuited for preparing students for a world in constant flux where they will have to learn to be independent learners throughout their whole lives. Interestingly, the same is true of teacher training, which, even when focusing on pedagogical innovation, is often imparted through teacher-centred lectures, with limited opportunities for experimentation and collaboration with peers. 1.3

Unequal Access to Knowledge

With only 11.2% of schools equipped with a fast internet connection (AGCOM 2019), the majority of Italian schools are unable to implement internet-based technological solutions or use the internet in their teaching practices (or in their management processes). Considering how important digital literacy skills have progressively become for citizenship and workplace readiness, it is paramount that schools have access to adequate connectivity. Furthermore, if access to connectivity, and consequently access to knowledge, are not guaranteed by schools, we will see a growing gap between who can afford it at home and who cannot. The Italian school system is underperforming. It is failing to foster the capacity for lifelong learning in students and failing to provide equal access to knowledge. 1.4

The Case for EdTech

Much thought has been devoted to understanding the potential of educational technologies in both academia and independent think tanks and social innovation organizations in the third sector. Nesta, the London-based, self-described “Innovation Foundation”, has released a series of publications in the past years pertaining to this topic. In the case studies presented in Nesta’s recent report Making the most of technology in education (Baker et al. 2019), including the Riconnessioni project, which we will focus on later, EdTech is being successfully used around the world to address challenges similar to those faced by Italy. Presented benefits include “innovating teaching and learning, improving outcomes for students”, and “being used to make our education systems fairer, vastly increasing access to information and higher-quality learning opportunities for the most disadvantaged. We also see technology improving the efficacy of our school systems”. Though “it is not hard to find cautionary tales that highlight how investment in EdTech does not necessarily imply improvement”, EdTech, particularly due to the rapid acceleration of its technological sophistication, could be a valuable tool for improving educational outcomes in Italy across the board. Building on this analysis, it seems that the main obstacles preventing fruitful innovation and improvement of the Italian school system are obsolete teaching practices caused by inadequate training, and inadequate connectivity infrastructure.

70

M. E. Newman

2 The Riconnessioni Model To face these challenges, Compagnia di San Paolo in collaboration with Fondazione per la Scuola has developed Riconnessioni, a model for social innovation which is adaptable to different territories, to innovate and improve education through infrastructural interventions and professional development. Riconnessioni is a holistic model characterized by a systemic approach: its aim is to bring together all the actors in the educational ecosystem to design, develop, and realize an educational paradigm for the XXI century. As such, Riconnessioni represents the cornerstone of a wide network comprised of public and private partners, teachers, headteachers, heads of administrative staff, parents, charities, foundations, educational publishers, universities, research institutes and international organizations, with whom the project collaborates. Riconnessioni is a flexible model, with specifics which may change from territory to territory and a central core which must stay the same. The main features of the model are: • Next generation school connectivity - Riconnessioni supports schools to ensure they are equipped with the connectivity they necessitate to innovate. Building from state of the art standards, Riconnessioni provides, promotes, and disseminates a connectivity provision which is built to last. • Learner-centric professional development and design of curricular activities Riconnessioni offers five main professional development pathways to schools: – Laboratorio Riconnessioni: a professional development pathway on teacher professionalism, the fostering of communities of practice, and putting teachers in the driver’s seat of EdTech selection and implementation. Teachers are also introduced to the European Commission’s DigComp digital skills frameworks for citizens, teachers, and schools. – Computational Thinking: a professional development pathway on the teaching of problem posing and solving, coding, and robotics, aimed at integrating computational thinking in curricular activities across the board. – Being Digital: a professional development pathway designed to provide critical instruments to understand the digital revolution and gain awareness of related opportunities and threats. The course includes a specific focus on open software, intellectual property, and privacy. – Digital Content Creation: a professional development pathway on digital tools and strategies to foster the creation of digital content in the context of projectbased learning activities in a wide range of curricular subjects. – Innovative Teaching and Inclusion: a professional development pathway on child-centred teaching as a tool for social, linguistic and cognitive inclusion. The pathway is designed to foster collaboration inside and between schools, as well as between schools and the wider community, to boost attainment and wellbeing for all students. The five professional development pathways are attended by one teacher per pathway per school building. These teachers later return to school and train their

TEL Adoption in the Riconnessioni Project

71

colleagues on the aspects of each course that they deem most relevant to their particular contexts. Trickle-down training empowers teachers to have an active role in selecting and disseminating innovations which they have reasons to find convincing. Methodologically, the offered professional development opportunities are characterized by a non-prescriptive approach, capacity building, and peer to peer learning. The key principles of Riconnessioni’s professional development methodology are: • Learner-centricity: learner-centric pedagogy must be taught in a learner-centric way. Aside from a small number of lectures, teachers learn through project-based learning, collaborative learning, and hands-on activities. • Capacity building: teachers are accompanied and supported by facilitators as they independently explore and experiment with educational innovations: this will help them build the self-efficacy to continue experimenting once the professional development is over. Whenever a new innovation is introduced, teachers are encouraged to try it out with a small group of colleagues, asking other teachers for help if necessary and looking for tutorials on the internet. • Collaboration: professional development represents a rare and precious opportunity for contrasting and comparing experiences and learning from peers. Teachers have more to learn from each other than from expert trainers or facilitators. Rules and habits around collaboration, practiced during professional development, can later enable the emergence of communities of practice inside and between schools. • Low-tech: the innovations presented by Riconnessioni are free and, when possible, cross-platform. Schools differ in terms of connectivity, amount and type of devices, and teachers’ attitudes towards technology. For this reason, Riconnessioni selects technology which can work in all schools, for all teachers. Simple technologies are easier to implement and are less likely to distract from pedagogy and the design of rich learning experiences. • Design of curricular activities: to ensure that innovations may find a place in teachers’ practice, they must be experimented with in the context of the curricular subjects which they teach in school. All professional development pathways include frequent lesson planning exercises. These moments allow teachers to identify and share ideas on how particular innovations can enable more effective and engaging teaching. • Trickle-down training: the teachers that attend professional development later become trainers for their colleagues in school. To support them in this endeavour, which may be daunting for some teachers, all pathways end with a module in which teachers design the training they will provide to their colleagues. This way, the content introduced through the project can adapt to each teacher’s needs. • School engagement - When they join Riconnessioni, schools sign a “commitment to innovative teaching”, which details their and Riconnessioni’s responsibilities. By signing this contract, schools commit to actively experimenting with innovative teaching methods and ensure that at least 60% of all teaching staff participates directly or indirectly (by attending trickle-down training) in Riconnessioni’s professional development initiatives. • Monitoring and evaluation - The model comes with a monitoring and evaluation framework which has the aim to gather and elaborate, throughout the rollout of the

72

M. E. Newman

project, all information relevant to the realization of the project’s interventions and its results. An impact evaluation is not available at this time, as the project itself is still running. The monitoring and evaluation framework’s objectives can be described as follows: – Timely reporting of project outputs – Identification of criticalities and coherence between planned and realized outputs – Evaluate impact, focusing on behaviour-change at all levels of participating schools (teachers, headteachers, management), and on skill development (students and teachers). Impact evaluation is both quantitative and qualitative and is carried out in collaboration with universities. Lastly, monitoring and evaluation includes a digital competency test which is taken by students in participating schools on a yearly basis. 2.1

Riconnessioni in Turin

Riconnessioni has been developed and rolled out in Turin by Fondazione per la Scuola della Compagnia di San Paolo. The project involves 350 school buildings, of which 220 connected to fiber-optic broadband (from 1 to 10 GBPS) thanks to a partnership with Open Fiber. The project has involved 98,000 students and engaged more than 1500 teachers in the 5 main professional development pathways. These teachers have gone on to train more than 5000 colleagues in their respective schools. Beyond the 5 main courses, the project launched a wide catalogue of professional development opportunities through partnerships with a variety of actors in the educational ecosystem including DeAgostini Scuola, Giunti Scuola, Pearson Italia, FME Education, Fondazione Torino Wireless, Google Education, Fondazione Paideia, Centro Internazionale di Studi Primo Levi, and Polo del ‘900. Riconnessioni Torino has involved all primary and lower-secondary schools of Turin and its metropolitan area. 2.2

Riconnessioni in Cuneo

In September 2019, Fondazione CRC, a bank foundation based in the city of Cuneo, adopted and localized the Riconnessioni model for the Cuneo province. In this iteration, all lower-secondary schools of the province will be involved in the infrastructural and professional development aspects laid out in the model.

3 Conclusions TEL (Technology Enhanced Learning) adoption may present unique challenges which are different from place to place. At the same time, some of the strategies adopted by the Riconnessioni project may offer insight which could be valuable to organizations operating in settings other than government-run primary and lower-secondary schools. In particular, the model features 3 characteristics which have guaranteed its success with schools:

TEL Adoption in the Riconnessioni Project

73

• Infrastructure: the project identified that a lack of connectivity infrastructure constituted the main obstacle to TEL adoption in Italian primary and lower-secondary schools, and that very few problems relating to TEL in schools could be solved without providing suitable infrastructure. • Professional development: the project identified teachers as the main change agents who determine the character of teaching and learning in schools. TEL adoption can only go as far as the digital competencies of the adopters. Consequently, the model was designed to provide teachers with a variety of training opportunities, designed by and for teachers, paying close attention to the unique challenges and risks faced by them when trying new approaches in the classroom. Also, the courses focus on the ways in which EdTech can offer solutions to problems relating to underperformance which teachers believe are important. • Non-prescriptive approach to EdTech selection and adoption: the professional development offered by Riconnessioni is structured to accommodate and encourage different points of view on which EdTech should be adopted and why. This nonprescriptive-approach has allowed Riconnessioni to be appreciated by teachers with different mind-sets working in different schools which serve different student populations.

References OECD: Education Policy Outlook: Italy, OECD Publishing, Paris (2017) Castoldi, M.: Oltre la retorica del cambiamento. Rivista dell’istruzione n.3/2017, Maggioli Editore, Rimini (2017) AGCOM: Educare Digitale, AGCOM (2019). https://www.agcom.it/documents/10179/ 14037496/Studio-Ricerca+28-02-2019/af1e36a5-e866-4027-ab30-5670803a60c2?version=1. 0. Accessed 04 Feb 2020 Baker, T., Tricarico, L., Bielli, S.: Making the Most of Technology in Education. Nesta, London (2019)

Cognitive Complexity Analysis of Learning-Related Texts: A Case Study on School Textbooks Syaamantak Das1(B) , Shyamal Kumar Das Mandal1 , and Anupam Basu2 1

Centre for Educational Technology, Indian Institute of Technology Kharagpur, Kharagpur, India [email protected], [email protected] 2 Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, Kharagpur, India [email protected] Abstract. The proposed work analyzes the cognitive complexity of given text-based learning material. Cognitive complexity refers to the thinking skills that are required to process the information which is present in the content of the text. A standard methodology to identify cognitive complexity is the use of Bloom’s Taxonomy. However, as observed from the experiments that some of the action verbs are often present in multiple cognitive levels causing ambiguity about the true sense of cognition. To overcome this drawback, signal words of informational text structure has been used as an added feature. Based on both cognitive action verbs of Bloom’s Taxonomy and signal words together, a computational approach using the SVM model has been used for an experiment on the NCERT dataset. It was observed that using signal words as an additional feature has significantly improved the classification task as compared to using only Bloom’s Taxonomy action verbs. Keywords: Text analysis · Cognitive science Informational text structure

1

· Bloom’s Taxonomy ·

Introduction

The cognitive domain of Bloom’s Taxonomy [4] divides both Learning Objectives (LO) and learning outcomes into multiple cognitive levels according to the degree of thinking skills required to complete them successfully. The six levels of Bloom’s taxonomy are Knowledge, Comprehension, Application, Analysis, Synthesis, and Evaluation, ranked from lowest to highest in terms of cognitive thinking. Bloom’s Taxonomy has also been applied in classifying texts such as questions and learning materials in order to understand the cognitive difficulty of questions [25] or depth of students’ learning present in the materials. Thus it is essential to identify the cognitive complexity of learning material in order to analyze whether it is suitable for the given LO and whether it will be sufficient to satisfy the learning outcome. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 74–84, 2020. https://doi.org/10.1007/978-3-030-52538-5_9

Cognitive Complexity of Texts

75

This study focus on automatic cognitive complexity analysis of texts in Learning materials for school level education. Textbooks are one of the major sources of learning materials in schools. As seen in previous studies [6] on text categorization based on Bloom’s taxonomy, this study attempts to classify the cognitive levels reflected in the information present in the text content. This is done by using a pre-defined word list, which is often used for word matching where the text if containing the matched word, is classified into the category of the word. At first, a given set of Bloom’s Taxonomy keywords obtained from a compilation of previous studies was used to identify the cognitive levels present in the text. It was observed that a text contains multiple action verbs of Bloom’s Taxonomy, which may belong to multiple cognitive levels. This created ambiguity about the actual cognitive information level that is present in the text. To overcome this, another approach of using the informational text structure was used. Informational text structure refers to the organization of information within a written text [9] and provides readers an idea about the text’s information type [3]. This feature helps readers to understand what main idea and details a text might present in a topic. Informational text structures can be analyzed using the signal words or cue words contained within the text [21]. Signal words are present in these text patterns, which can help in the identification of the information of the text structure. The five most common informational text structure are: (1) Description (2) Procedure/Sequence (3) Comparison/Contrast (4) Cause–Effect explanation and (5) Problem–Solution presentation. To computationally solve this problem, a machine learning-based algorithm, Support vector machine (SVM) is used. SVM is an algorithm that determines the best decision boundary between vectors that belong to a given group (or category) and vectors that do not belong to it. The rationale behind choosing SVM was based on the work of Qiao [20], which showed that SVM reached the highest macro F1 score when all classifier-features are combined. The advantage of this method will be that it will no longer be dependent on human expertise to select the appropriate learning material. Instead, the system itself will be able to analyze documents and provide a list based on the relevance of the cognitive information structure present in the text content. The paper’s organization is as follows. Section 2 explains the literature review on Bloom’s Taxonomy action verbs and informational text structure in detail. Section 3 describes the SVM algorithm. Section 4 gives details of the NCERT dataset. Section 5 shows the experiments conducted and the results obtained. Section 6 provides observation and discussion. The future work and conclusion are narrated in Sect. 7.

2 2.1

Literature Review Cognitive Domain of Bloom’s Taxonomy and Action Verbs

Bloom’s Taxonomy [4] is a widely used methodology to describe Learning Objectives (LO) in educational documents such as setting up of curriculum, performing the objectives-based evaluation on learner’ s achievement, and for aligning

76

S. Das et al.

curriculum and assessment [16]. Bloom’s Taxonomy describes six levels of cognitive development (Knowledge, Comprehension (Understand), Application, Analysis, Evaluation, Synthesis (Create)), ranging from simple remembering to complex and critical reasoning abilities [7,24]. Knowledge and Comprehension were believed to be more concrete and straightforward than Synthesis and Evaluation, which were more complex and abstract [13]. Claudia Stanny’s work [23] provided a major compilation of these action verbs. The paper compiled 30 different Bloom’s Taxonomy action verbs (BTAV) data set and prepared a list of 176 action verbs, as shown in Table 2. The paper raised an important question - can unambiguous recommendations be made about which verbs correspond to specific levels of cognitive skills by the structure of Bloom’ s Taxonomy? As stated by Lee [16], that despite the practicality and simplicity of the model, Bloom’s Taxonomy is criticized for generalized and uni-dimensional domains of knowledge and skills that could not clearly explain the levels. Moreover, the levels of cognitive demands in the analysis of instructional objectives for students’ learning and assessment plans also remain ambiguous. Also, although the revised Bloom’s Taxonomy [12] tries to overcome the generalization of cognitive dimensions, the paper makes a statement - “there is still the challenge of identifying the level of thinking”. 2.2

Informational Text Structure and Signal Words

Informational text structure refers to how information is organized in a document. There can be multiple types of text structure [10], especially informational text structure. School-level textbooks are an excellent example of informational text structure [19]. There are five types of informational text structure [17,18], which are as follows: 1. Description – This type of text structure gives an idea about the features, attributes, and examples. 2. Procedure/Sequence – This type of text structure gives a sequential and/or chronological occurrence of events or specific things. 3. Comparison/Contrast – This type of text structure shows a comparison between two or more things. This structure shows how similar as well as how different the subjects are. 4. Cause-Effect explanation – This type of text structure depicts ideas, events, or facts as causes and the results as the effects over a given period. 5. Problem–Solution presentation – This describes a problem and gives answers to the problem. This structure can contain other text structures also. An important feature that helps readers to understand the main points of the text is the use of signal words. Signal words are the words in a text that suggest its informational structure [8]. It was observed from the experiments that the signal words do help in identifying the cognitive level when paired with action verbs of Bloom’s Taxonomy. The list of signal words is shown in Table 1.

Cognitive Complexity of Texts

77

Table 1. List of informational text structure and their signal words respectively. Desc.Description; Proc./Seq. - Procedure/Sequence; Comp./Cont. - Comparison/Contrast; PSP - Problem solution presentation Desc.

Proc./Seq.

Comp./Cont.

Cause–Effect

PSP

above

after

alike

accordingly

Dilemma is

an example

afterward

also

as a consequence for the given question

appears to be

at last

although

behind

at the same time as as opposed to

One solution can be The problem is

belongs to

before

as well as

because

Question is

characteristics

during

both

because of

response

defined as

eventually

comparatively

consequently

solution

for example

finally

compared with

due to

The puzzle is

for instance

first

different from

effect of

The result would be

identified as

first of all

Either

for

to fix the problem

Imagine that

following

however

for this reason

To solve this

including

immediately

is a characteristic of In the first place

3

as a result of as illustrated by

in common

hence

-

in comparison

if

-

is a feature of

initially

in contrast

in conclusion

is like

last

in the same way

in order

-

looks like

later

instead of

is caused by

-

most important

meanwhile

just as

leads to

refers to

next

just like

reasons why

-

such as

not long after

less than

so that

-

to illustrate

now

like

therefore

-

-

preceding

likewise

thus

-

-

previously

much as

-

-

-

recently

nevertheless

-

-

-

second

on the other hand -

-

-

since

opposite

-

-

-

soon

same as

-

-

-

then

similar

-

-

-

third

similar too

-

-

-

to begin with

similarity

-

-

-

when

though

-

-

-

whenever

unlike

-

-

-

-

whereas

-

-

-

-

yet

-

-

-

-

or

-

-

Methodology of SVM Algorithm for Text Classification

In machine learning, Support Vector Machines (SVMs) are supervised learning models used to differentiate between data for classification purposes. An SVM is a discriminative classifier formally defined by a separating hyperplane. Given a set of training examples, each marked as belonging to one of the categories. An SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier. The hyperplane optimizes a linear discriminant model representing the perpendicular distance between the datasets.

78

4

S. Das et al.

Details of NCERT Dataset

As observed from previous work by Prater [19], which says that school textbooks are good examples of informational text structure, the proposed research work used school textbooks as the dataset. The NCERT textbook dataset was used previously for other textbook researches [1,2]. The text data sets are from the tenth-grade National Council of Educational Research and Training (NCERT) textbooks of Economics, Geography, and Science (Biology). NCERT is an autonomous organization of the Government of India, responsible for textbook publishing for the Central Board of Secondary Education (CBSE) from one to twelfth grade, in India. Online textbooks are available for free from its website called “epathshala” (meaning e-school). Each book was divided as per the chapters present in them, and only the text content data has been used for experiment purposes. Images (pictures, graphics, graphs, charts, etc.) were not considered for analysis. There are altogether 18 chapters (five from Economics, seven from geography and six from Science (Biology)). The chapters were divided into 160 text data samples. The average number of words for text data is 410 (Economics), 300 (Geography), and 374 (Biology). The statistics of the text data set are shown in Table 3. The text passages were prepared, keeping 300 words as a threshold based on previous work of Lea [15]. It is observed from the data set, that there are multiple distributions of signal words of more than one informational text structure in a text document. Also, there were nine texts with no cognitive action verbs. 4.1

Inter-annotator Agreement for Gold-Standard Data

For preparing the gold standard data set, all the collected text passages were manually annotated. To manually annotate the informational text structure and cognitive level, each of the passages was presented to three annotators and was instructed to rate the text in both the cognitive levels and text structure in 1–5 points MOS scale (1 is lowest). The annotators were trainee graduate teachers having a Master’s degree level of education. Based on the annotator rating, if any two annotators agreed to a particular cognitive level with the highest rating, then that cognitive level was assigned to the given text. If the rating of the annotators is equal for all the six cognitive levels, then all the six cognitive levels are considered equally present in the text. After collecting the result from the annotator, the Fleiss Kappa method has been used for determining the inter-annotator agreement [5,22]. Fleiss kappa is a statistical method for calculating the reliability of agreement between a fixed number of annotators when assigning categorical ratings or classifying items. The kappa value obtained is 70.83% (for Economics), 61.11% (for Geography), and 82.22% (for Biology), which are “substantial agreement” for Economics and Geography, and “almost perfect agreement” for Biology as per Landis and Koch [14].

Cognitive Complexity of Texts

5 5.1

79

Experiments Experiment with Bloom’s Taxonomy Action Verbs

Dataset: For this experiment, the dataset of Bloom’s Taxonomy action verb list provided by Stanny [23] was used to count the frequency occurrences in the NCERT text dataset. Stanny compiled 30 different BTAV data set and prepared a list of 177 BTAV. The article of Stanny states it has 176 action verbs. But actually, it had 177 action verbs. The complete list of action verbs is shown in Table 2. Results: The result showed that there are 16 instances of Bloom’s Taxonomy action verbs, which are overlapping in multiple cognitive levels. These Bloom’s Taxonomy action verbs occurred together in the text passages of NCERT books. And each of these words belongs to multiple levels of Bloom’s Taxonomy cognitive domain. Table 3 shows the list of action verbs and their corresponding overlapping cognitive levels. 5.2

Experiment with Signal Words of Informational Text Structure

Dataset: For this experiment, signal words from Table 1 was used to count the frequency of occurrences in the NCERT text data content. Results. In the experiment with signal words of Informational text structure, it was observed that the informational text structures – Description, Comparisoncontrast, and Cause-effect had distinctive cognitive patterns. The description showed several occurrences in the Knowledge level, and Cause-effect showed several occurrences in the Evaluation level. Comparison-contrast had almost similar occurrences in both the Application and Analysis level as the Cognition required for comparison and contrast is equally an application and analysis task. The signal words could not show distinctive cognitive patterns in the case of procedure text structure. There was no prediction for problem solution presentation text structure. Figure 1 shows the distribution of cognitive levels across Informational text structures. 5.3

Experiment with Support Vector Machine

For SVM, an custom classifier was created using Monkeylearn’s API [11]. The N-grams range consisted of Unigram, bigram ad Trigrams. Stopwords like i, me, my, myself, we, our, ours, ourselves etc. were filtered. WH words (how, what, when, where, which, who, why) were not considered as BTAV. Words were normalized to their base form (lemma) (e.g., compared became compare). SVM draws a hyperplane that divides a space into two subspaces: one subspace that contains vectors that belong to a group and another subspace that contains vectors that do not belong to that group. Those vectors are representations of

80

S. Das et al.

Table 2. Bloom’s Taxonomy action verbs of Stanny’s [23] with occurrence frequency accross 30 dataset denoted by f. Knowledge f

Understand

f

Apply

f

arrange

6

articulate

4

act

19 analyze

24 appraise

22 arrange

22

choose

4

associate

4

adapt

4

11 argue

12 assemble

14

cite

17 characterize

4

apply

22 break

8

arrange

5

7

copy

4

4

back/back up 5

7

assess

17 choose

define

21 clarify

5

calculate

10 calculate

9

attach

4

describe

14 classify

18 change

9

19 choose

cite

Analyze

appraise break down categorize

f

Evaluate

f

Create

categorize collect

10 combine

f

7 9 14

draw

5

compare

11 choose

11 classify

10 compare

18 compile

7

duplicate

7

contrast

7

6

compare

24 conclude

13 compose

19

conclude

6

29

classify

identify

20 convert

13 complete

5

8

construct

indicate

4

12 compute

10 contrast

19 core

6

create

19

label

21 demonstrate

6

13 correlate

5

4

design

24

develop

18

defend

construct

contrast counsel

list

27 describe

22 demonstrate

20 criticize

11 create

4

locate

10 differentiate

8

develop

4

debate

8

criticize

11 devise

match

14 discuss

21 discover

8

deduce

6

critique

14 estimate

5

memorize

10 distinguish

12 dramatize

16 detect

7

decide

4

4

name

22 estimate

11 employ

16 diagnose

4

defend

order

5

28 experiment

6

diagram

12 describe

explain

15 explain

8

4

facilitate

4

4

formulate

18

6

generalize

7

generate

11

outline

11 express

17 explain

5

differentiate 20 design

quote

7

extend

11 generalize

5

discover

read

4

extrapolate

5

4

discriminate 11 discriminate 9

identify

4

determine

evaluate

13

recall

24 generalize

11 illustrate

18 dissect

6

recite

12 give

4

implement

4

21 evaluate

16 improve

5

recognize

14 giveexamples 8

interpret

15 divide

12 explain

9

4

record

13 identify

14 interview

6

4

invent

10

relate

11 illustrate

9

manipulate

10 examine

18 invent

8

make

6

modify

distinguish evaluate

4

estimate

grade

15 hypothesize 8 integrate

repeat

20 indicate

8

12 experiment

9

judge

25 manage

reproduce

11 infer

15 operate

17 figure

4

manage

15 modify

10

review

4

5

4

group

4

mediate

9

21

select

16 interpret

17 paint

4

identify

7

prepare

12 originate

9

state

23 locate

10 practice

15 illustrate

8

probe

4

plan

21

tabulate

4

match

7

predict

9

14 rate

5

predict

tell

4

observe

5

prepare

11 inspect

8

rearrange

19 prepare

12

underline

7

organize

5

produce

13 inventory

9

reconcile

12 produce

13

write

5

paraphrase

22 relate

12 investigate

7

release

6

propose

9

predict

12 schedule

11 order

5

rewrite

4

rate

21

recognize

11 select

4

6

select

5

rearrange

8

relate

7

13 outline

10 set up

15 reconstruct 9

report

10 simulate

5

12 supervise

9

relate

8

represent

4

17 predict

4

synthesize

16 reorganize

9

restate

15 solve

19 prioritize

4

test

8

revise

12

review

15 translate

5

12 value

7

rewrite

7

rewrite

12 use

25 relate

17 verify

9

role-play

4

select

7

4

select

12 weigh

5

set up

9

summarize

20 write

5

separate

10

specify

5

interpolate

organize

show sketch

utilize

infer

organize point out

question

organize

8

8

tell

7

solve

8

summarize

7

translate

21

subdivide

10

synthesize

4

survey

7

tell/tellwhy 5

test

14

write

17

Cognitive Complexity of Texts

81

Table 3. List of overlapping action verbs and their cognitive levels Bloom’s Taxonomy action verbs

Overlapping cognitive levels

describe, identify, locate, recognize interpret discover, experiment, relate criticize distinguish, infer, classify estimate construct, modify compare

Knowledge, Comprehension Comprehension, Application Application, Analysis Analysis, Evaluation Comprehension, Analysis Comprehension, Evaluation Application, Synthesis/Create Analysis, Evaluation

Fig. 1. Distribution of cognitive levels - Informational text structure

the training texts and a group is the label associated with the texts. A list of cognitive action verbs and signal words based on Table 1 and 2 were given as input to the system. The training data consisted of two columns - Text content and Label (both cognitive and informational text structure). Out of 160 texts, 146 were used for training. The rest 14 was for testing. Results of Classifier Accuracy. Precision measures the exactness of a classifier. Higher precision means less false positives, while a lower precision means more false positives. Recall measures the completeness, or sensitivity, of a classifier. Higher recall means fewer false negatives, while lower recall means more false negatives. The classifier gave an overall accuracy of 81% and F1 score of 84%. The Precision and Recall for each cognitive level is shown in Table 4. As seen if a cognitive level has low precision (Synthesis), that means that texts from other cognitive levels are getting confused with the level in question. If a cognitive level has low recall (Comprehension), that means that texts from that cognitive level are getting predicted for other cognitive levels.

82

S. Das et al. Table 4. Precision and recall result of cognitive levels

6

Cognitive level

Precision Recall

Knowledge Comprehension Application Analysis Evaluation Synthesis

91 75 86 50 95 33

53 33 86 35 97 40

Observation and Discussion

Using both action verbs of Bloom’s Taxonomy cognitive domain and signal words of informational text structure captured evidence and patterns based on which cognitive domains of text can be inferred. When compared with existing works, the results showed a better performance in terms of accuracy and F1 score. It was observed that even for text data that did not contain any action verbs, the algorithm was able to predict a Cognitive level based on the training dataset. An observation that is made from the results is that most text belongs to any one of Bloom’s Taxonomy cognitive level, even if the action verbs are not present. E.g., the growth of industrialization, as well as agriculture, is needed for a sustainable economy. Here although no Bloom’s Taxonomy action verb is present, the comparison of industrialization and agriculture can be identified based on the signal words “as well as”, which can be mapped into Analysis level of Bloom’s cognitive domain. As observed that all text data contained multiple levels of informational text structure, it was true that most texts of higher cognitive level (Analysis, Evaluation) do contain action verbs of lower cognitive level (Knowledge, Comprehension), which reduces the discriminating feature of the classifier. The classifier predicts the most probable class(es) for each text sample. When a category is wrongly predicted, a false positive is made. Moreover, a false negative is also created because a true prediction is not made. As observed from the data set that the number of signal words is sparse, and some text contains very few signal words. Such documents are difficult to predict, and false negative is generated for such text samples.

7

Future Work and Conclusion

There are two contributions to this study. First, it provides a list of Bloom’s Taxonomy action verbs that are overlapping over cognitive domains, as observed over the NCERT dataset. Second, it explores the idea of using informational text structure as a feature and combines this with Bloom’s Taxonomy action verbs for identification of the cognitive level of a given text. The experiments were conducted across three different subjects of school-level education, and the

Cognitive Complexity of Texts

83

results provided a solution for the problem of automatic analysis of cognitive classification. However, the limitation of this study is the size of the annotated dataset and also constructing reliable annotations.

References 1. Agrawal, R., Gollapudi, S., Kannan, A., Kenthapadi, K.: Data mining for improving textbooks. In: ACM SIGKDD Explorations Newsletter. Citeseer (2011) 2. Agrawal, R., Gollapudi, S., Kannan, A., Kenthapadi, K.: Study navigator: an algorithmically generated aid for learning from electronic textbooks. J. Educ. Data Mining 6(1), 53–75 (2014) 3. Akhondi, M., Malayeri, F.A., Samad, A.A.: How to teach expository text structure to facilitate reading comprehension. Read. Teach. 64(5), 368–372 (2011) 4. Bloom, B.S., et al.: Taxonomy of Educational Objectives. Cognitive Domain, vol. 1, pp. 20–24. McKay, New York (1956) 5. Bobicev, V., Sokolova, M.: Inter-annotator agreement in sentiment analysis: machine learning perspective. In: RANLP, pp. 97–102 (2017) 6. Chang, W.C., Chung, M.S.: Automatic applying bloom’s taxonomy to classify and analysis the cognition level of English question items. In: 2009 Joint Conferences on Pervasive Computing (JCPC), pp. 727–734. IEEE (2009) 7. DeWaelsche, S.A.: Critical thinking, questioning and student engagement in korean university english courses. Linguist. Educ. 32, 131–147 (2015) 8. Goldman, S.R., Rakestraw, J.A.: Structural aspects of constructing meaning from text. In: Handbook of Reading Research, vol. 3, no. 1, pp. 311–335 (2000) 9. Hearst, M.A.: Tilebars: visualization of term distribution information in full text information access. In: CHI, vol. 95, pp. 59–66 (1995) 10. Hearst, M.A.: Texttiling: segmenting text into multi-paragraph subtopic passages. Comput. Linguist. 23(1), 33–64 (1997) 11. Kaur, A., Chopra, D.: Comparison of text mining tools. In: 2016 5th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO), pp. 186–192. IEEE (2016) 12. Krathwohl, D.R.: A revision of bloom’s taxonomy: an overview. Theory Pract. 41(4), 212–218 (2002) 13. Krathwohl, D.R., Anderson, L.W.: Merlin C. Wittrock and the revision of bloom’s taxonomy. Educ. Psychol. 45(1), 64–65 (2010) 14. Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33, 159–174 (1977) 15. Lea, M.R., Street, B., et al.: Writing as academic literacies: understanding textual practices in higher education. In: Writing: Texts, Processes and Practices, pp. 62– 81 (1999) 16. Lee, Y.J., Kim, M., Jin, Q., Yoon, H.G., Matsubara, K.: Revised bloom’s taxonomy – the swiss army knife in curriculum research. In: East-Asian Primary Science Curricula, pp. 11–16. Springer, Heidelberg (2017) 17. Meyer, B.J.: Prose analysis: purposes, procedures, and problems (1985) 18. Piccolo, J.A.: Expository text structure: teaching and learning strategies. Read. Teach. 40(9), 838–847 (1987) 19. Prater, M.A.: Teaching Students with High-Incidence Disabilities: Strategies for Diverse Classrooms. Sage Publications, Thousand Oaks (2017)

84

S. Das et al.

20. Qiao, C., Hu, X.: Text classification for cognitive domains: a case using lexical, syntactic and semantic features. J. Inf. Sci. 45(4), 516–528 (2019) 21. Roehling, J.V., Hebert, M., Nelson, J.R., Bohaty, J.J.: Text structure strategies for improving expository reading comprehension. Read. Teach. 71(1), 71–82 (2017) 22. Shang, L., Lu, Z., Li, H.: Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364 (2015) 23. Stanny, C.: Reevaluating bloom’s taxonomy: what measurable verbs can and cannot say about student learning. Educ. Sci. 6(4), 37 (2016) 24. Swart, A.J., Daneti, M.: Analyzing learning outcomes for electronic fundamentals using bloom’s taxonomy. In: 2019 IEEE Global Engineering Education Conference (EDUCON), pp. 39–44. IEEE (2019) 25. Yahya, A.A., Toukal, Z., Osman, A.: Bloom’s taxonomy based classification for item bank questions using support vector machines. In: Modern Advances in Intelligent Systems and Tools, pp. 135–140. Springer, Heidelberg (2012)

Lessons Clustering Using Topics Inferred by Unsupervised Modeling from Textbooks Mat´ıas Altamirano(B) , Abelino Jim´enez, and Roberto Araya Center for Advanced Research in Education, Institute of Education, Universidad de Chile, Periodista Jos´e Carrasco Tapia 75, Santiago, Chile [email protected] Abstract. Analyzing the content and quality of teacher and students’ talk has been an active area of educational research. In this context, the importance of temporal analysis of teaching learning events has been growing up. Previous work has proposed a method that automatically describes teacher’s talk using an unsupervised machine learning model to infer topics from school textbooks. To describe teacher talk, the machine learning method measures the appearance of the inferred topics throughout each lesson. We propose a clustering method based on a modification of the method described above. The modification consists in considering super topics (Content, Administration/Feedback, Other), which will describe teacher talk more generally. Then, we cluster using ‘K-means’ with the Dynamic Time Warping metric since the lessons are dynamic phenomena that occur over time. Finally, we propose a way to visualize the center of the clusters to analyze them. We apply the proposed method to a collection of natural science lesson transcriptions, and we analyze and discuss the clusters obtained. Keywords: Clustering Learning analytics

1

· Visualization · Natural language processing ·

Introduction

When teachers analyze discourse in their own classrooms, academic achievement improves [1]. Therefore, giving them new ways to do it has been an active area of educational research [2]. This is the reason why it is important to study teacher’s behaviour. Nowadays many researchers have underlined the importance of temporal analysis of teaching-learning events [3,4]. With this, analyzing the differences and similarities about the evolution of the different lessons has great relevance. Search for differences and similarities in classroom talk has been a relevant area in educational research as seen in a number of studies that have been conducted since the 1960s [5,6]. For example, Yuka Koizumi [7] studies the similarities and differences in teachers’ questioning in German and Japanese mathematics classrooms analyzing consecutive mathematics classes taught by experienced c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 85–94, 2020. https://doi.org/10.1007/978-3-030-52538-5_10

86

M. Altamirano et al.

teachers in Germany and Japan. All these studies are done manually, which is expensive and slow, which motivates us to look for a more efficient way to do it. Educational Data Mining is an emerging interdisciplinary research area that deals with the development of methods to explore data originating in an educational context [2]. One of the most popular Data Mining methods is clustering. In clustering, the goal is to find data points that naturally group together, splitting the full data set into a set of clusters. Clustering is particularly useful in cases where the most common categories within the data set are not known in advance. Clusters can be created at several different possible grain-sizes: for example, schools could be clustered together or students could be clustered together, to investigate similarities and differences between schools or students correspondingly [8]. As we can see, clustering is a method with which we can find differences and similarities in a certain group of data automatically. In this work, we present a clustering method based on an automatic description of classroom talks using topic models, which is an automatic method to summarize text and one of the most popular models for natural language processing. Topic models have been applied in a variety of contexts, being preferred in cases where interpretability and speed are priority characteristics [9]. We rely on previous work which successfully described a class through a topic model trained with school textbooks in order to find a characteristic set of topics that can be used to describe the lessons transcriptions [10], which is one of the many works that the ‘Center for Advanced Research in Education’ has done in this area. For example, [11] which develops tools for ‘text mining’ of teacher speech with VSM techniques, and SmartSpeech [12] which is a mobile application with which you can transcribe visualize teacher speech. The main difference with the previous work [10] is that we look for a more general description of the teacher talk. This is why to describe the lesson, we manually generate super-topics from the topics given by this topic model. Using this lesson description we apply one of the most classic clustering methods that is ‘Kmeans’, with the particularity that we will use the metric called Dynamic Time Warping which is one of the algorithms for measuring the similarity between two temporal sequences, which may vary in speed [13]. Classical metrics (Euclidean, Manhattan) compare two time series by comparing the ith point of one time series with the ith point of the other. This produces a poor similarity score. DTW gives a non-linear (elastic) alignment between two-time series. Simply, it looks for the best alignment between the two-time series. This produces a more intuitive similarity measure. We show that this approach can be used to find differences and similarities about the evolution of the different lessons by looking at the center of the clusters. For this, a method of visualizing the centers of the clusters is proposed so that it is possible to analyze the evolution of the lessons. It is important to note that much of the work consists in achieving a good visualization of the results. As in [14], this method of class analysis quickly and anonymously reveals what is happening in class, and makes it possible to classify classes that can be used for large-scale examinations of teaching practices. It is also a tool to create

Lessons Clustering

87

a systematically classified inventory of classes that goes beyond detecting the teacher’s speech [14], incorporating pedagogical aspects present in the speech. Our research questions are the following: (1) What types of classroom talk can we differentiate with an automatic lesson descriptors?, (2) Are these differences systematic across all over the class duration?. This paper is organized as follows. Section 2 describes the proposed approach and methodology. Section 3 describes the obtained clusters and finally, Sect. 4 discusses the results, shows the conclusion and future work.

2 2.1

Method Data Collection

We collect 195 audio recordings of natural sciences lessons from 4 different teachers, which were collected and transcribed using an Automatic Speech Recognition. We use a mobile application called SmartSpeech, developed by ‘Center for Advanced Research in Education’ together with the universities of Juvaskyla and Tampere of Finland [12]. Beside, we collect 11 natural science textbooks since they must be aligned with the content of the classroom lessons transcriptions. 2.2

Data Analysis

Figure 1 outlines the four-stage method to obtain different clusters and analyze them. Each of these stages will be explained in more detail later.

Fig. 1. Stages to obtain lesson categories

88

M. Altamirano et al.

1. Data Preprocessing: In order to improve the performance of the topic model used is that we replicate the processing done in [10] since we rely on her work to describe the lessons. First of all, it is necessary to digitize the text from the pages of school textbooks because they can be collected in several formats. Since school textbooks have pages that do not contribute to the training of topic models we select only the textbook pages where content or exercises are presented within a context. In the same line, only the textbook pages with more than 20 words were kept. On the other hand, lessons transcriptions are assumed to be already in a digital text format and ready to be preprocessed. From now on the preprocessing is the same for both the school textbooks and for the classroom lesson transcriptions. Before applying general text preprocessing techniques we replace labels for every group of words that work as synonyms (e.g. A regular expression of any number is replaced by NUMBER). We use the Nltk Python library [15], which is a text processing library, with support for several languages, including Spanish, to remove stopwords and the stemming process. 2. Training an Unsupervised Topic Model: In this work we use a Latent Dirichlet Allocation model (LDA) [16] which is one of the most popular topic models, that characterizes each document as a mixture of topics, more precisely, LDA characterizes each document as probability distributions over the LDAtopics, this is why is useful to describe teacher talks. In the same way, LDA topics are defined as probability distributions over the vocabulary (all the words included in the training data). To train the LDA model to obtain 60 topics from the preprocessed text-book pages, we used the Gensim Python library [17]. 3. Description of Teacher Talk Using the Topic Model: As we said before LDA characterizes each document as probability distributions over the topics, which means that there is a vector of 60 dimensions for each document, where each dimension is the probability of belonging to a topic. To describe the development of topics during lessons, we split the lesson transcriptions into intervals of 10 lines each, and then for each interval, we tracked the topic probabilities. Therefore, we get a multivariate time series of 60 dimensions that characterizes the evolution of the lesson. However, working with a multivariate time series of 60 dimensions has high computational costs and also since we are looking for a more general description of the teacher talk, having a 60 dimension representation is overmuch. Therefore, from the topics obtained, 3 super topics are chosen which are: content, administration/feedback and others. We get these super topics adding the probability of all the including topics. Thus, the vector of super topics is also a probability distribution; therefore, the sum of all super topic probabilities is one. 4. Lessons Clustering: Clustering is the task of grouping a set of objects in such a way that objects in the same group (called a cluster) are more similar

Lessons Clustering

89

(in some sense or another) to each other than to those in other groups (clusters). In this research, we use a K-means clustering, which is one of the simplest and most popular unsupervised machine learning algorithms. K-means algorithm identifies k number of centroids, and then allocates every data point to the nearest cluster while keeping the centroids as small as possible. The ‘means’ in the K-means refers to averaging of the data; that is, finding the centroid. Since we are working with time series, we used the Tslearn Python library [18] to train the K-means model. To answer the question ‘What types of classroom talk can we differentiate with an automatic lesson descriptor?’, we visualize the centroids of the clusters contrasting the behavior of the three super topics for each cluster.

3

Results

3.1

Description of a Lesson

Table 1 shows an example of some of the 60 topics that the model obtained from the collection of school textbooks. Each column represents a topic. The header has the topic ID and in parenthesis there are labels manually added. The rows contain the words with higher belonging degrees to each topic. Table 1. Example of six topics and the 5-top words for each Topic 2 (Question)

Topic 11 (Energy)

Topic 16 (Gas)

Topic 28 (Calculation)

Topic 29 (Dynamic)

Topic 60 (Activity)

QUESTION Energy SYMBOLS

Heat

NUMBER

Force

Activity

NUMBER

Electron

Gas

PERCENTAGE Movement SYMBOL

Develop

How

Molecule

Charge

EQUAL SYMBOL

Body

do

Next

Transformation Pressure

CONSTANT

Law

Will

A NAME

Production

Value

Exert

Presentation

Volume

The table shows that some topics include purely content words, for example topic 29 which contain words like ‘Force’, ‘Movement’. Others include words related to the evolution of the lesson for example topic 2 which is composed of words like ‘QUESTION SYMBOLS’, ‘How’. Considering the above is that we generate 3 super topics, which will be content, administration/feedback and other, adding the probability of belonging to the topics included in each super topic. Figure 2 shows an example of a super topic representation of a lesson, obtained from the 60-topic LDA model. Blue line represents the evolution of the content topics, Orange line represents the administrative/feedback topics and the green one the remaining ones. The lesson observed is maintained with a greater probability in the administration/feedback topics.

90

M. Altamirano et al.

Fig. 2. Example of an automatic teacher talk descriptor which has a greater probability in the administration/feedback topics

3.2

Clustering

Figures 3, 4a, 4b, 4c and 4d show the center of the clusters, which is the average of the lessons within each cluster. Clearly, there are 5 types of lessons that are formed automatically. Figure 3 shows a type that only talks about topics related to the evolution of the class, Fig. 4d shows a type of lesson in which content topics are mainly spoken, on the other hand. Figure 4a displays a lesson that starts with administration/feedback topics and then goes between content and administration/feedback. Figure 4b shows a type of lesson that in the middle of the lesson there is a higher probability of talking about the evolution of the class but at other times there is no certainty. Finally, Fig. 4c displays a kind of lesson that the first half relates to content and the other half to the evolution of the class.

Fig. 3. Center cluster 2

Now it is important to check if the lessons within these clusters are really represented by the center of the clusters. First, the lessons of each of the clusters are shown to have a visual comparison between the prototype delivered by the center of the cluster and the lessons within them (Figs. 5 and 6). In the previous figures it can be seen that effectively the lessons belonging to each cluster evolve in the same way as the center, answering the second research

Lessons Clustering

(a) Center cluster 1.1

(b) Center cluster 1.2

(c) Center cluster 1.3

(d) Center cluster 1.4

91

Fig. 4. Center clusters 1

Fig. 5. Lessons and center of cluster 2

question in a positive way. Besides, it can be seen that most of the lessons are in cluster 2 followed by cluster 1.4. Finally, Fig. 7 shows a random example of a lesson which easily we can classify as a lesson belonging to cluster 1.1. This gives us an intuition that it is possible to agree on this classification with human raters, which can be very interesting since these lesson clusters can have a more in deep educational interpretation.

92

M. Altamirano et al.

(a) Cluster 1.1

(b) Cluster 1.2

(c) Cluster 1.3

(d) Cluster 1.4

Fig. 6. Lessons and center of clusters 1

Fig. 7. Example of a lesson belonging to cluster 1.1

4

Discussion and Conclusion

In this study we presented a method to automatically obtain a summarized description of teacher talk and used it as features to clustering lessons. The method uses a topic model to compute a set of topics from school textbooks. Then, with these topics, super topics are generated manually. After that, the set of super topics is used to describe teacher talk from classroom lesson transcriptions. Afterward, lessons are clustered using the K-means algorithm and the cluster centers are visualized. The main assumption behind the proposed method is that each teacher has a way of doing their classes which should be shown through the description provided by a topic model trained with school textbooks. We apply the proposed method using a set of Chilean natural science school textbooks and one hundred ninety-five lessons of four different teachers. We obtain 5 clusters after the K-means algorithm. The cluster centers are

Lessons Clustering

93

interpretable and reveal information about the lesson evolution. Each of the centers of the clusters show us a different kind of class (Figs. 3, 4a, 4b, 4c and 4d). Using this we answer the first research question. Besides, we compare the evolution of the lessons belonging to the same clusters to conclude that the center of the cluster is really representative of the lessons in it. All this was possible thanks to the visualization of the center of the clusters that we proposed. In general, the teachers’ lesson belongs to cluster 2. There could be various explanations: cluster 2 could be a very common kind of lesson which focuses on the dynamic of the class more than the pure content, or maybe as the model just has a finite number of topics could not catch all the content moment of the lesson. In addition, it is important to note that these results not only generate natural clusters of lessons, they also provide an automatic mechanism to decide which cluster a class belongs to. These results are very promising and practical as it allows class types to be automatically classified and this has great value for Directors and Stakeholders. Since, for example, if the district Superintendent introduces a new teaching technique or teacher’s course, one could immediately see if there are changes in the membership composition of the clusters. In this study, the proposed method was applied to Chilean lessons, but it can be applied to lessons of different languages and contents. In the future, we expect to relate the kind of lesson with students learning gains. Acknowledgements. Support from ANID/ PIA/ Basal Funds for Centers of Excellence FB0003 is gratefully acknowledged.

References 1. Rymes, B.: Classroom Discourse Analysis: A tool for Critical Reflection. Routledge, Abingdon (2015) 2. Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 40(6), 601–618 (2010) 3. Knight, S., Wise, A.F., Chen, B.: Time for change: why learning analytics needs temporal analysis. J. Learn. Anal. 4(3), 7–17 (2017) 4. Mercer, N.: The seeds of time: why classroom dialogue needs a temporal analysis. J. Learn. Sci. 17(1), 33–59 (2008) 5. Carlsen, W.S.: Questioning in classrooms: a sociolinguistic perspective. Rev. Educ. Res. 61(2), 157–178 (1991) 6. Dantonio, M., Beisenherz, P.C.: Learning to Question, Questioning to Learn: Developing Effective Teacher Questioning Practices. Allyn & Bacon, Boston (2001) 7. Koizumi, Y.: Similarities and differences in teachers’ questioning in German and Japanese mathematics classrooms. ZDM Math. Educ. 45(1), 47–59 (2013) 8. Amershi, S., Conati, C.: Automatic recognition of learner groups in exploratory learning environments. In: International Conference on Intelligent Tutoring Systems. Springer, Heidelberg (2006) 9. Boyd-Graber, J., Hu,Y., Mimmo, D.: Applications of topic models. Found. R Inf. Retrieval 11(2–3), 143–296 (2017) Trends

94

M. Altamirano et al.

10. Espinoza, C., L¨ ams¨ a, J., Araya, R., H¨ am¨ al¨ ainen, R., Jim´enez, A., Gormaz, R., Viiri, J.: Automatic content analysis in collaborative inquiry-based learning. In: ESERA 2019 (2019) 11. Araya, R., Plana, F., Dartnell, P., Soto-Andrade, J., Luci, G., Salinas, E., Araya, M.: Estimation of teacher practices based on text transcripts of teacher speech using a support vector machine algorithm. Br. J. Educ. Technol. 43(6), 837–846 (2012) 12. Caballero, D., Araya, R., Kronholm, H., Viiri, J., Mansikkaniemi, A., Lehesvuori, S., Kurimo, M.: ASR in classroom today: automatic visualization of conceptual network in science classrooms. In: European Conference on Technology Enhanced Learning, pp. 541–544. Springer, Cham (2017) 13. Senin, P.: Dynamic time warping algorithm review. Information and Computer Science Department University of Hawaii at Manoa Honolulu, USA, vol. 855, no. 1–23, p. 40 (2008) 14. Owens, M.T., Seidel, S.B., Wong, M., Bejines, T.E., Lietz, S., Perez, J.R., Balukjian, B.: Classroom sound can be used to classify teaching practices in college science courses. Proc. Nat. Acad. Sci. 114(12), 3085–3090 (2017) 15. Bird, S., Klein, E., Loper, E.: Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media Inc, Sebastopol (2009) 16. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3(Jan), 993–1022 (2003) 17. Rehurek, R., Sojka, P.: Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks (2010) 18. Tavenard, R.: tslearn: a machine learning toolkit dedicated to time-series data (2017). https://github.com/rtavenar/tslearn

Automatic Content Analysis of Computer-Supported Collaborative Inquiry-Based Learning Using Deep Networks and Attention Mechanisms Pablo Uribe1(&), Abelino Jiménez1(&), Roberto Araya1(&), Joni Lämsä2(&), Raija Hämäläinen2(&), and Jouni Viiri3(&) 1

Center for Advanced Research in Education, Institute of Education, Universidad de Chile, Periodista José Carrasco Tapia 75, Santiago, Chile [email protected], [email protected], [email protected] 2 Department of Education, University of Jyväskylä, PO Box 35, 40014 Jyväskylä, Finland {joni.lamsa,raija.h.hamalainen}@jyu.fi 3 Department of Teacher Education, University of Jyväskylä, PO Box 35, 40014 Jyväskylä, Finland [email protected]

Abstract. Computer-supported collaborative inquiry-based learning (CSCIL) represents a form of active learning in which students jointly pose questions and investigate them in technology-enhanced settings. Scaffolds can enhance CSCIL processes so that students can complete more challenging problems than they could without scaffolds. Scaffolding CSCIL, however, would optimally adapt to the needs of a specific context, group, and stage of the group’s learning process. In CSCIL, the stage of the learning process can be characterized by the inquirybased learning (IBL) phase (orientation, conceptualization, investigation, conclusion, and discussion). In this presentation, we illustrate the potential of automatic content analysis to find the different IBL phases from authentic groups’ face-to-face CSCIL processes to advance the adaptive scaffolding. We obtain vector representations from words using a well-known feature engineering technique called Word Embedding. Subsequently, the classification task is done by a neural network that incorporates an attention layer. The results presented in this work show that the proposed best performing model adds interpretability and achieves a 58.92% accuracy, which represents a 6% improvement compared to our previous work, which was based on topicmodels. Keywords: Inquiry based learning
Deep neural networks
Natural language processing

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 95–105, 2020. https://doi.org/10.1007/978-3-030-52538-5_11

96

P. Uribe et al.

1 Introduction Scholars widely agree that lecture-based teaching should be complemented with more active learning methods to support the development of skills and knowledge that students graduating from science, technology, engineering, and mathematics (STEM) domains need [1, 2]. In this respect, the potential of computer-supported collaborative inquiry-based learning (CSCIL) has been known for a long time [3], and it still is a popular pedagogical approach to enhance skills and knowledge beneficial for future STEM professionals [4]. In short, CSCIL is a technologically facilitated and mediated process in which a group of students follows the practices of scientists to acquire scientific knowledge, learn scientific content, and better understand the nature of science [5]. CSCIL emphasizes the student’s active role in the learning process; students are encouraged to explore the material, ask questions, and share ideas with each other so that technological advancements can increase the success of learning even more [6]. CSCIL is not an unambiguous pedagogical method or model, and there is no unified theory of CSCIL. Pedaste et al. [7], however, have made a synthesis of the various inquiry-based learning (IBL) models. They provided a framework in which the essential aspects of IBL are captured with the help of five phases—orientation, conceptualization, investigation, conclusion, and discussion. In the orientation phase, students should identify the main concepts and variables of the problem and become familiar with the needed technological resources. In the conceptualization phase, students should determine the dependent and independent variables as well as propose research questions or hypotheses that they start to investigate. In the investigation phase, students should plan their data collection procedure, implement the procedure, and analyze and interpret the data. In the conclusion phase, students should offer and evaluate solutions to their questions or hypotheses. In the discussion phase, students should elaborate on their findings and conclusions as well as reflect their CSCIL. Even though collaboration and technological resources themselves can assist students in IBL, research has shown that other scaffolds are also needed to achieve the benefits of CSCIL [8]. It is also known that the needs for scaffolds are different in the different IBL phases [5]. Thus, before designing and implementing the scaffolds, there is a need to study CSCIL with particular focus on the IBL phases. To study CSCIL, researchers can conduct content analysis [9, 10], for example, so that they code the transcribed students’ conversations to the different IBL phases (orientation, conceptualization, investigation, conclusion, and discussion) [7]. Currently, researchers conduct content analysis procedures mostly manually. The human-driven content analysis of large data sets, however, is time-consuming. Moreover, the validity of inferences from the data depends on the consistency of the coding procedures [11], which is why the inter-coder and intra-coder reliability are subject to intense methodological research efforts over long years [12]. The development of an automatic content analyser could have significant implications concerning the scaffolding CSCIL. First, automation allows large-scale analyses. Second, it might enable the real-time monitoring of several groups when they engage in CSCIL. The real-time information about groups’ ongoing IBL phase could be useful so that technological learning environments or teachers could adapt scaffolds based on each group’s needs.

Automatic Content Analysis

97

The present work introduces an automatic content analysis method for utterance classification so that the IBL phase can be automatically captured from CSCIL processes taking place in face-to-face interaction in an authentic higher education setting. Our method shows the potential of computer-driven analysis to address the current challenges of manual content analysis, namely insufficiency of the time resources and issues concerning reliability. We address the following research question: How similar are the results of the manual and proposed automatic content analysis?

2 Related Work The present work focuses on the automatization of the IBL phase coding necessary for all further analysis. Therefore, this work contributes by automatizing a time-consuming process of researchers’ work, so researchers can focus on interpreting results and designing optimal scaffolds. A previous work done by our team presented in [13] had the same objectives and was used as a guide-line for this research. To improve the performance, the methodology presented here focuses on two points: Feature Engineering. The previous work was based on a Latent Dirichlet Allocation (LDA) topic model [14]. Topic models are statistical models that are used to find groups of words, called topics, that usually appear together in large document collections [13]. This model was trained with scientific literature (physics textbooks) to generate features from the utterances, representing them as a distribution of a fixed list of 60 learned topics. However, the LDA model training process did not include natural dialogic language sources that are present in common social interactions (such as groups’ conversations), which are however difficult to obtain. Nevertheless, even with these limitations the results of the previous work were promising. Alternatively, in this work we used a Word Embedding model. This procedure consists in the assignment of high dimensional vectors to words in a way that preserves the syntactic and semantic relationships between them, and is one of the most fundamental techniques in natural language processing [15]. When trained on large enough corpora, this model admits vector representations for a big number of words, even for typical of dialogic language. In this work, we evaluate a word embedding model already trained on a mass scale corpus (provided by the TurkuNLP project [16]), to obtain a numerical representation of utterances as sequences of vectors. Classification Algorithm. The preceding study used Support Vector Machines (SVMs) [17] trained with the hand-labelled transcriptions from the groups’ conversations to classify each utterance. Instead, we used a deep neural network with an embedding and an attention layer, which are widely applied in the Natural Language Processing tasks [18]. When needed, the incorporation of an embedding layer makes possible the adjustment of the word embedding vectors during the training process. On the other hand, attention layers are a standard part of the deep learning toolkit, contributing to impressive results in various tasks. In fact, a standard neural network consists of a series of non-linear transformation layers, where each layer produces a fixed-dimensional hidden representation. For tasks with large input spaces, this paradigm makes it hard to control the interaction between components. However, an

98

P. Uribe et al.

attention network maintains a set of hidden representations that scale with the size of the source by performing a soft-selection over these representations, as explained in [19]. In this work we implement two attention mechanisms. The first is called simple attention, that soft-selects important words from each utterance in a general manner for all categories. The second operates in a category-specific way, so each IBL phase performs the soft-selection according to their own nature. This mechanism is called differentiated attention. Further details will be discussed in the next section.

3 Methodology In this study, we analysed 55 students in an introductory university physics course on thermodynamics. The participants were divided into eleven groups of five students and each group worked with a shared laptop computer. The students were asked to collaboratively solve thermodynamics problems in a technology-enhanced learning environment while their conversations were screen-captured and audio recorded. In this section, we present the procedures when implementing our model for automatic content analysis.1 3.1

Data Set

Our data set was built by manually transcribing each group’s talk while they solved an inquiry problem. The transcriptions are in Finnish. Each group said, on average, 180 utterances, summing up to 1980 for the whole data set. These utterances include, on average, 11 words. The utterances were manually labelled by using theory-driven content analysis [20], i.e. each utterance was coded to one of the IBL phase presented by [7]. One of the researchers coded all the utterances while another researcher outside of this study independently coded 20% of the utterances. The inter-rater agreement was 67.7%, after which the disagreements were discussed and resolved. 3.2

Data Pre-processing

Text data is pre-processed to transform it into a simpler form so algorithms can perform better. First, raw digits are converted to words (example: ‘92’ is turned into ‘ninetytwo’). Second, punctuation marks are removed, except from questions marks that are considered as new words. Later, a tokenization is done considering only the top 2000 most frequent words (out of 3500 different words found in the transcriptions). Finally, utterances are transformed to the same fixed length of 20 words. As a consequence, a total of 254 utterances are truncated, and 1700 are padded with the token 0. 3.3

Feature Engineering

In this work, the main input for the model is the current utterance represented as a sequence of tokens. This sequence will be later transformed into a sequence of 1

Further details are available at https://github.com/pabloveazul/CIBL.

Automatic Content Analysis

99

word-vectors by the embedding layer explained later. Additionally, we considered as input for our model the previous and the next utterances’ token-sequence representations, as well as the number of words of the current utterance and its relative position in the respective group work session. 3.4

The Neural Network Classifier

In this work we have replaced the precedent SVM with a neural network composed of different layers. Distinct configurations of these layers give rise to numerous models with different architectures that are evaluated later. The main layers are explained below2: Embedding Layer. Word embedding vectors are obtained from the TurkuNLP project, where an already trained word embedding model is available for public use. These vector representations are obtained using a word2vec model [21] trained on the Finnish Internet Parsebank (FIB), a mass-scale corpus with automatic syntactic analysis that currently includes about 3.7 billion tokens [16]. This layer turns tokens into word embedding vectors. We then represent utterances as ordered sequences of wordvectors. Mathematically, each word token m in the vocabulary is associated with an ddimensional embedding EmbðmÞ 2 Rd : If m is represented by its one-hot vector m, EmbðmÞ corresponds to the column Em of an embedding matrix E 2 Rd  RjV j . Here jV j is the size of the vocabulary V (jV j = 2000) and d is the dimension of the word embedding (d = 200). Then, the embedding layer output of a sequence of 60 words u ¼ ½m1 ; . . .; m60  corresponding to the previous, current and next utterance (each one of 20 words) is given by: e ¼ ½Em1 ; . . .; Em60 . The matrix weights Ei;j are initialized with the weights given by the TurkuNLP project, and can be adjusted through backpropagation during the training process (if they are set to trainable) or remain constant (if they are set to static). The embedding of the token 0 is the null vector. Attention Layer. To enhance relevant words for the classification task, a simple attention mechanism is built. A Single Layer Perceptron (SLP) with a single output is applied to every temporal slice of the encoded sequence (i.e. to each embedding of e). The full output of this layer is called the attention weight vector: w ¼ ½w1 ; . . .; w60 . Mathematically, the attention weight of each encoded word et is given by: wt ¼ SLPðet Þ ¼ aT et þ b

ð1Þ

Here, a is a vector with the same dimension as et and b 2 R. These are the parameters of the SLP which are learnt through backpropagation. Later, a softmax layer is applied time-wise (i.e. word-wise) to obtain probabilities proportional to the exponentials of the weight numbers. The output
 is then interpreted as an attention-probability vector attðeÞ ¼ ½attðeÞ1 ; . . .; att eÞ60 : attðeÞt ¼ softmaxðwÞt ¼ eða

2

T

et þ bÞ

All vectors and matrix written in bold, scalars in light.

=

X60

eð a i¼1

T

ei þ bÞ



ð2Þ

100

P. Uribe et al.

This attention-probability vector is then multiplied element-wise with the vector e ¼ ½e1 ; . . .; e60  of encoded words to finally obtain a weighted sequence ½attðeÞ1 e1 ; . . .;
 att eÞ60 e60 . Each attention probability can be interpreted as the importance of each word in the utterance. This attention mechanism is called simple attention. However, one may think that the importance of words may vary depending on the final classification. For example, numbers within an utterance should be the important features for the investigation phase, whether concept words should be respectively for the conceptualization phase. This leads to another possible architecture for the attention layer, where each category is connected with one independent attention mechanism. This means each category c is associated with one SLP (noted SLPc ) with a single output, and no parameters are shared between these SLPs. Mathematically this is: wct ¼ SLPc ðet Þ ¼ ðac ÞT et þ bc

ð3Þ

Where ðac Þ and bc correspond to the parameters of the SLP associated to the category c. Similarly, the attention-probability of a given category c and a given encoded word et is then computed as: c T

attc ðeÞt ¼ oftmaxðwc Þt ¼ eða Þ

et þ b c

=

X60

c T

eð a Þ i¼1

ei þ b c



ð4Þ

The output of this layer are the five weighted sequences: ½attc ðeÞ1 e1 ; . . .;
 attc eÞ60 e60 . This attention mechanism is named differentiated attention. Sum Layer. A simple sum is applied timewise over all the vectors from the previous P layer to obtain a context vector s ¼ 60 t¼1 ut , where u ¼ ½u1 ; . . .; u60  is the previous output sequence. When the differentiated attention mechanism is present, this sum is done P independently on each weighted sequence of each category c: sc ¼ 60 t ¼ 1 attc ðeÞt et . Multi-layer Perceptron. After the input corresponding to the utterances is processed by the previous layers, the final sum s is concatenated with the inputs corresponding to the number of words of the utterance (n) and its relative position in the CSCIL-process (r) into a single vector ½s; n; r . This vector is fed into an MLP with one hidden layer, and an output (y) of dimension 5 (one for each IBL phase) with softmax activation function to obtain a probability distribution over the different phases as a final prediction ^y. This prediction is interpreted as the probability of the utterance to belong to each one of the phases. y ¼ MLPð½s; n; r Þ

ð5Þ

^y ¼ softmaxðyÞ

ð6Þ

Automatic Content Analysis

101

In the case of the differentiated attention mechanism, each sum sc is concatenated with n and r into a single vector ½sc ; n; r  and fed into a category-specific MLP. Each MLP (noted MLPc ) is independent and has a hidden layer and a single output yc . A softmax layer is applied then to the concatenation vector of these outputs to obtain the final prediction ^y. This is:

3.5

yc ¼ MLPc ð½sc ; n; r Þ

ð7Þ

^y ¼ softmaxð½y1 ; y2 ; y3 ; y4 ; y5 Þ

ð8Þ

Model Evaluation

To evaluate each model, we independently split the training and testing sets with all possible combinations of nine and two group transcriptions respectively. Models are trained through backpropagation using an ADAM optimizer [22] and cross-entropy loss function. Then, the average accuracy over all the splitting combinations is considered as the performance of each model.

4 Results Several models with different layers are evaluated. A table with the description and the results of the model evaluation process are shown in the Table 1 (the best results in bold): A comparison between the manual and automatic classification of the best performing model (MLP + SEDA) is presented through a confusion matrix. Each component Ci;j in the matrix is the average percentage of utterances in the test set that were manually coded to IBL phase i and automatically coded to IBL phase j: A comparison table between the previous LDA model [6], the MLP + SEDA and a human coder regarding the precision for each phase is shown below (Table 3): Attention probabilities of the differentiated attention mechanism incorporated to this model let us understand what type of words are relevant for each IBL phase. In Figs. 1. and 2. we graphically represent the attention probabilities for each word in the previous, current and next utterance respectively. In the x-axis each word of each utterance is written according to the previous order. Blank spaces correspond to the null word vectors added by the padding process. The y-axis represents the attention probability. Each coloured line corresponds to a different category (IBL phase): Figures 1 and 2 show that the attention probability corresponding to null vectors is close to 0 for every IBL phase. Also, the attention probabilities effectively vary between the different IBL phases, so that peak values are found for a different type of words in each case.

102

P. Uribe et al.

Table 1. Model description and evaluation results. The confidence interval is calculated with the 95% confidence level. (Pre-Trained Trainable: initiated with TurkuNLP weights and trained during training process, Pre-Trained Static: initiated with TurkuNLP weights and not trainable, No: no addition of the corresponding layer. Model names MLP: Multilayer Perceptron, SE: Static Embedding Layer, TE: Trainable Embedding Layer, SA: Simple Attention, DA: Differentiated Attention) Model name MLP + SE MLP + SESA MLP + SEDA MLP + TE MLP + TESA

Embedding layer Pre-trained static Pre-trained static Pre-trained static Pre-trained trainable Pre-trainedtrainable

Attention layer No

Mean accuracy on test set 54:43  0:90

Mean accuracy on training set 76:46  2:02

Simple

57:38  1:02

61:87  0:99

Differentiated

58:92  0:96

64:57  1:34

No

54:33  1:07

85:87  2:78

Simple

56:03  0:98

69:50  1:98

Fig. 1. MLP + SEDA correctly classifying the utterance 500 into the investigation phase. Attention probabilities corresponding to the investigation phase are high for numbers (‘nine’, ‘two’, ‘hundred’, ‘thousand’).

Automatic Content Analysis

103

Fig. 2. MLP + SEDA misclassifying the utterance 12 into the orientation phase (real: discussion). Attention probabilities of the conceptualization phase are high for concept words ‘molekyylien’ (molecules) and ‘jakaumaa’ (distribution).

Table 2. Mean Confusion Matrix of the best performing model (Model 3). For each real phase, the highest percentage between the phase predictions is in bold. Real Phase

Predicted orientation Orientation 17 Conceptualization 2.4 Investigation 2.6 Conclusion 0.04 Discussion 6

Predicted conceptualization 0.87 2.1 0.48 0.25 0.78

Predicted investigation 1.7 1.4 16 0.52 4.2

Predicted conclusion 0.12 0.07 0.02 0.2 0.4

Predicted discussion 5.6 5 5.9 2.7 24

Table 3. Precision of different IBL phase utterance classifiers. For each phase, the highest precision between the automatic models is in bold. Classifier

Orientation precision (%) LDA 50 MLP + SEDA 60 Human Coder 71

Conceptualization precision (%) 49 47 55

Investigation Conclusion precision (%) precision (%) 68 49 67 24 76 45

Discussion precision (%) 51 55 64

5 Discussion and Conclusion This work was our second attempt to automatize the content analysis in an authentic CSCIL context. We compared the results with the ones discussed in [13], which we have exceeded using the new models by 6%, achieving a 58.9% average accuracy (the

104

P. Uribe et al.

previous model has a 52.9% average accuracy). Nevertheless, there is still a challenge to improve the automatic content analysis models to attain human-level performance (67% accuracy). Additionally, most of the issues found for the previous results remain present for the ones obtained in this work: precision is still notably higher for the investigation phase. On the other hand, despite improving the precision for the orientation and discussion phase, the precision of all the other IBL phases decreased. To improve the precision of the conceptualization and conclusion phases (see Table 2), we will gather more data as we now constrained ourselves to a thermodynamics problem. Regarding the methodology, we replaced the previous SVM classifier with a deep network that can model more complex functions. Also, it incorporates an embedding layer that associates high-dimensional features to words, which can be trained through backpropagation. Therefore, the previous feature engineering manual process based on an LDA model is replaced by this procedure. Additionally, the pre-trained embedding model given by the Turku University NLP project incorporates face-to-face Finnish vocabulary that was only partially contained previously in the physics textbooks. For instance, adding an attention layer helps not only to improve performance but also to obtain more interpretable results by analysing the attention weights. For each IBL phase, words with high weights may be interpreted as the key elements for the classification task within each utterance. The automatized identification of the IBL phase from students’ face-to-face conversation could be used for adaptive scaffolding purpose. Even though the idea of the adaptive scaffolding is not a new one [23], the work is still in progress [24]. Our results may provide input for this development of systems that allow technological learning environments or teachers monitor in real-time (through a dashboard in their smartphones or notebooks) the IBL phase of several groups’ CSCIL processes. These systems may include other applications, such as giving quick feedback to teachers regarding their speech when they provide support to students, as presented in [25]. Acknowledgements. Support from ANID/PIA/Basal Funds for Centres of Excellence FB0003 is gratefully acknowledged. We would like to express our gratitude for the financial aid provided by the Academy of Finland [grant numbers 292466 and 318095, the Multidisciplinary Research on Learning and Teaching profiles I and II of JYU].

References 1. Arthurs, L.A., Kreager, B.Z.: An integrative review of in-class activities that enable active learning in college science classroom settings. Int. J. Sci. Educ. 39(15), 2073–2091 (2017) 2. Freeman, S., Eddy, S.L., McDonoug, M., Smith, M.K., Okoroafor, N., Jordt, H., Wenderoth, M.P.: Active learning increases student performance in science, engineering, and mathematics. Proc. Natl. Acad. Sci. 111(23), 8410–8415 (2014) 3. Lipponen, L., Hakkarainen, K.: Developing culture of inquiry in computer-supported collaborative learning. In: International Society of the Learning Sciences, Toronto, Ontario, Canada, pp. 171–175(1997) 4. Jeong, H., Hmelo-Silver, C.E., Jo, K.: Ten years of computer-supported collaborative learning: a meta-analysis of CSCL in STEM education during 2005–2014. Educ. Res. Rev. 28, 100284 (2019)

Automatic Content Analysis

105

5. Bell, T., Urhahne, D., Schanze, S., Ploetzner, R.: Collaborative inquiry learning: models, tools, and challenges. Int. J. Sci. Educ. 32(3), 349–377 (2010) 6. Pedaste, M., Leijen, Ä., Saks, K., De Jong, T., Gillet, D.: How to link pedagogy, technology and STEM learning? In: The Proceedings of the 25th International Conference on Computers in Education, pp. 579–586, January 2017 7. Pedaste, M., Mäeots, M., Siiman, L.A., De Jong, T., Van Riesen, S.A., Kamp, E.T., Manoli, C.C., Zacharia, Z.C., Tsourlidaki, E.: Phases of inquiry-based learning: definitions and the inquiry cycle. Educ. Res. Rev. 14, 47–61 (2015) 8. Alfieri, L., Brooks, P.J., Aldrich, N.J., Tenenbaum, H.R.: Does discovery-based instruction enhance learning? J. Educ. Psychol. 103(1), 1–18 (2011) 9. Lämsä, J., Hämäläinen, R., Koskinen, P., Viiri, J.: Visualising the temporal aspects of collaborative inquiry-based learning processes in technology-enhanced physics learning. Int. J. Sci. Educ. 40(14), 1697–1717 (2018) 10. Lämsä, J., Hämäläinen, R., Koskinen, P., Viiri, J., Mannonen, J.: The potential of temporal analysis: combining log data and lag sequential analysis to investigate temporal differences between scaffolded and non-scaffolded group inquiry-based learning processes. Comput. Educ. 143, 103674 (2020) 11. Weber, R.P.: Basic Content Analysis, vol. 49. Sage, Beverly Hills (1990) 12. Krippendorff, K.: Content Analysis: An Introduction to its Methodology. Sage publications, Beverly Hills (2018) 13. Espinoza, C., Lämsä, J., Araya, R., Hämäläinen, R., Jiménez, A., Gormaz, R., Viiri, J.: Automatic content analysis in collaborative inquiry based learning. In: ESERA (2019) 14. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res 3(Jan), 993–1022 (2003) 15. Tshitoyan, V., Dagdelen, J., Weston, L., Dunn, A., Rong, Z., Kononova, O., Persson, K.A., Ceder, G., Jain, A.: Unsupervised word embeddings capture latent knowledge from materials science literature. Nature 571(7763), 95–98 (2019) 16. Turku NLP page, Finnish NLP. http://turkunlp.org/finnish_nlp.html#treebank. Accessed 29 Jan 2020 17. Burges, C.J.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Disc. 2(2), 121–167 (1998) 18. Hu, D.: An introductory survey on attention mechanisms in NLP problems. In: Bi, Y., Bhatia, R., Kapoor, S. (eds.) Intelligent Systems and Applications IntelliSys 2019. Advances in Intelligent Systems and Computing, vol. 1038, pp. 432–448. Springer, Cham (2020) 19. Kim, Y., Denton, C., Hoang, L., Rush, A.M.: Structured attention networks. arXiv preprint arXiv:1702.00887 (2017) 20. Neuendorf, K.A.: The Content Analysis Guidebook. Sage, Los Angeles (2016) 21. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013) 22. Kingma, D.P., Ba, J.A.: A method for stochastic optimization. arXiv preprint arXiv:1412. 6980 (2014) 23. Kollar, I., Fischer, F., Slotta, J.D.: Internal and external scripts in computer supported collaborative inquiry learning. Learn Instr. 17(6), 708–721 (2007) 24. de Jong, T.: Moving towards engaged learning in STEM domains; there is no simple answer, but clearly a road ahead. J. Comput. Assist. Learn. 35(2), 153–167 (2019) 25. Caballero, D., et al.: ASR in classroom today: automatic visualization of conceptual network in science classrooms. In: Lavoué, É., Drachsler, H., Verbert, K., Broisin, J., PérezSanagustín, M. (eds.) Data Driven Approaches in Digital Education EC-TEL 2017, LNCS, vol. 10474, pp. 541–544. Springer, Cham (2017)

The Impact of Personality, Attitude and Visual Decision-Making Dashboard Tools on the Learning Engagement of Economist Students Liana Stanca1(&), Cristina Felea2, Romeo Stanca1, and Mirela Pintea1 1

Business Information Systems Department, Babes-Bolyai University, TH. Mihaly Str., 400591 Cluj-Napoca, Romania {liana.stanca,mirela.pintea}@econ.ubbcluj.ro, [email protected] 2 Faculty of Letters, Babes-Bolyai University, Horea Str., 400591 Cluj-Napoca, Romania [email protected]

Abstract. At present, big data has a relevant impact on the field of economics practice and education. European strategic documents and research regarding new skills emphasize measures that higher education needs to take for preparing a more skilled labor force for Industry 4.0. The paper draws on specialist interdisciplinary literature on IT/economics knowledge, skills, as well as personality, attitudes, and behavior needed by economists to match the requirements of the labor market. The study aims to identify the profile of an economics student who needs new tools for processing and understanding big data. A learning scenario is set for the study of economics by including a teamteaching approach (IT, economics) and visual decision-making tools (dashboards) to facilitate the training of economists and improving their decisionmaking capacity. A survey was conducted to study student profiles based on personality factor (Cattell’s personality test), their involvement in the learning process and the acceptance of the “new” in their education. Findings confirm studies which emphasize that human values can be used to assess the quality of the workforce and that the introduction of visualization tools in student training may increase their confidence and skills for becoming successful employees in the Big Data era. Results also show that the new teaching approach may generate positive reactions and better learning outcomes. Keywords: Personality
Attitude
Visual decision-making dashboard Adaptive learning
Knowledge economy




1 Introduction Information systems and information technology have become, at the beginning of the 21st century, the most important factors for the functioning of world economy. Knowledge economy dominated by the concept of knowledge management induces modern business and management approaches in which the concept of “value” is © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 106–116, 2020. https://doi.org/10.1007/978-3-030-52538-5_12

The Impact of Personality, Attitude and Visual Decision-Making

107

redefined [18]. In this economy, the use of resources is transferred from strategic capital resources to strategic resources in the form of information, knowledge, creative thinking and innovation. Skills and knowledge are critical factors for production [37]. Businesses can gain competitive advantage by implementing continuous innovations, while skills and managerial knowledge are at the heart of this innovation process, so the whole activity is redefined, so that everything revolves around the notions of skills and knowledge. The Europe 2020 Strategy and the Partnership for 21st Century Skills requires students to be educated in the field of digital competences [32] under the influence of social and economic change. According to studies, it is necessary to revise the curriculum so that economists are provided with knowledge and skills needed to conduct IT expertise and data analysis, being able to guide corporate strategy by creating a link between business activities and IT functions supporting these activities [19]. The “new” economist is a person who can disseminate and exchange knowledge to inspire innovative behaviors, new products and services for their profession in Big Data era [8, 9]. So, it is necessary to identify and formalize the knowledge to train these professionals by creating a framework for teaching/learning economics that combines knowledge sharing, knowledge management on entrepreneurial, smart and collaborative dimensions. The “intelligent attitude” dimension is introduced as teaching analytics [17] combined with skills, attitudes and personalities, which are represented in our study by theories and practices such as data analysis, agile decision making, critical visual data analysis, mining process, and design thinking. The paper is structured in four main parts starting with the literature review on elearning and of theories concerning psychological and social profiles, so as to create a theoretical research framework; then are presented the impact of large data on the field of economic from education to practice, the economic theory teaching scenario and statistical analysis to obtain the economic student’s profile in the big data era. The last part describes the methods, stages and results of our research, anchored in psychological and social theories, accompanied by the statistical interpretation of the obtained results. The paper ends with the conclusions of our research.

2 Literature Review The present study draws on specialist literature from different fields which have been reunited in studies related to the instruction of economist (accounting) students [38]. The economic education needs to include the teaching of visualization tools for students to be able to integrate easily on the labor market [26]. Economic practices, in their turn, are being reconsidered in the knowledge economy [3, 20]. According to [15], knowledge includes an organizational culture, abilities, reputation, intuition, and a codified theory that influence human behavior and thinking. Starting from this, the authors considered that the introduction of visual decision-making tools in economics courses is a step towards improving students’ and future specialists’ intuitive and rational decision-making styles. Additionally, some studies sustain that economics higher education needs to accept that it is of vital importance that future specialists adopt visual tools and understand visual metaphors [1, 38]. Thus, several papers

108

L. Stanca et al.

demonstrate that software used to analyze large data volumes (data extraction tools) as well as sophisticated data viewing tools increase the ability of people to understand the story behind data in both empirical or real-life applications [7, 24]. This is supported by the idea that teachers must provide information in economic area so as to create a scenario that includes the following dimensions: the role of decision analysis in risk analysis, which procedures should be performed, the 100% test implications on the population, whether external data should be used, the role of internal auditors in using decision analysis and interpreting the consequences of using decision analysis [22]. Consequently, students need to know the techniques and methods of analyzing economic data and social media based on which they can model predictions about the factors that influence a business client [21]. These models will allow economists to plan and assess business risk. According to [23], visual modeling of accounting data analysis is done on three dimensions: specific visualization of data representation space, interaction with data and identification of visual models and taking decisions. Currently, however, less emphasis is put on visualization research and, implicitly, interaction in visualization [14]. The interaction in visualization is the catalyst of users’ dialog with data and, ultimately, of their understanding of these data, but it is not easy to quantify. Interaction is a concept that is difficult to design, quantify and evaluate, which are critical actions in the decision-making process based on visualization tools (dashboards) [11]. Additionally, our view of the 21st century skills agrees with [31], according to whom personality determines the various levels of success characterizing specialists in economics. This research attempts to outline an educational solution that eliminates this gap by proposing a training framework for economists including IT tools (dashboards) that will allow economists to induce rational and intuitive decisionmaking styles by preparing them as moderators of the relationship between knowledge management and organizational performance. Our teaching-learning approach also draws on [10] and [7], who state that economists need to make informed decisions. According to [10], “knowledge” is a mix of contextual information, expertise and value that leads to innovation and clean experience. Data analysis and reporting are said to be the objectives of the IT system and, in line with current employers’ requirements, the curriculum should expose students to popular analytical and visual tools so as to increase the accountability of the economic decision [7]. Thus, in the authors’ view, it is inevitable that the teaching and learning process in the economic field expand its boundaries so as to include theories and practices of the big data, visual analytics, and decision dashboards [4]. The authors also consider it important to study student engagement in relation to student personality.

3 Learning Scenario - Personality, Attitude and Visual Decision-Making Dashboard Tools In the following, a model for the reorganization of Big Data courses for economics students is provided, in line with the need identified in [26]. It presents measures higher education needs to take for preparing a more skilled labor force for Industry 4.0. Literature on education and the formation of intellectual capital in the academic environment emphasizes that the practices governing education in the knowledge

The Impact of Personality, Attitude and Visual Decision-Making

109

economy are described by concepts and metaphors such as participation, co-creation and becoming, representing the skills of the 21st century [37]. Yet, literature does not offer clear solutions to train the perfectly qualified workforce. In this context, the creation of psychosocial profiles of economist employees is necessary. Some authors argue that personality is more important than intelligence in the case of intellectual capital [31]. In our view, personality determines the different levels of success that characterize knowledge in the knowledge economy. This paper aims to study the relationship between the personality factors, the level of student involvement in the learning process and the degree of acceptance of the “new” in their jobs, thus becoming successful employees on the labor market in Big Data era. Additionally, the study will contribute to the literature on economics education by identifying the profile of students in economic field analyzed from the perspective of soft skills measured by test scores. Our scenario draws on the research of [35], which demonstrates that cognitive skills can shape and refine the personality of the employees. The research framework is designed so as to define the economist student’s profile based on the International 5 Personality Pool survey [6]. In this experiment students are observed for 14 weeks during a course of Data Bases Theory and Programming Language in the economic field. The teaching environment is complex, and is designed and developed to allow the development of cognitive and digital competences in accordance with the views of [33]. The introduction of Moodle for teaching economics allowed teachers to focus on the development of soft skills needed in the 21st century. Theories of adult learning mention providing accurate and timely feedback to students. Student evaluation involved practical tests followed by a project, then a written assessment over the 14 weeks, focusing on how they interact in teams and where they carry out their tasks. At the end of the course a 33-item questionnaire was created using Cattell 16 PF (The Sixteen Personality Factor Questionnaire) [6], which measures the personality of an individual by Five Big personality factors: Extraversion (E), Conscientiousness (C), Emotional Stability (ES), and Intellect (I). The answers to the 33 questions were presented on a 5-point Likert scale [6], in relation to the individual’s potential to integrate on the labor market. Based on this tool, the research aims to identify personality traits that can predict students’ natural inclusion. The data analysis is expected to result in a profile prototype that will be validated over time. The identified profile is needed within the academic environment and labor market [28]. In the former, the prototype can inform the process of adapting curriculum and the teaching/learning activity to the requirements of the labor market.

4 Research Methodology The study was carried out on 100 students of the Faculty of Economics and Business Administration during the academic year 2018–2019. The course was held by an IT teacher and an economics teacher. The technical information is presented by the IT teacher assisted by the economics teacher, who will help with explanations of the economic phenomena; as a result, data analysis specific for data mining can be more easily understood. We believe that the correct understanding of economic phenomena

110

L. Stanca et al.

makes it easier to understand data mining processes. We introduced a visual representation tool (for visualizing business transactions, flowcharts and data-flow diagrams) of economic phenomena in order to check whether they are easier to understand by students. This method is based on the results from [14, 25], where visualization can be considered synonymous with dashboards. We designed the course materials according to [25], in which performance dashboards are defined as all-inclusive imaging techniques, with clear evidence of their positive impact on business. They are means to enable specialists to understand data stream that is always at hand and have the roles of collecting, summarizing and presenting information from several sources [37]. The learning outcomes allow students to understand the role and usage of dashboards in command, control and agile economic decision-making. Study hypotheses: 1. Economics students according to personality factor are open to accept/reject the new representations of concept, design and usage of visualization tools adapted to decision theory. 2. The affective (cognitive) states of students manifested in accepting/rejecting the new (related to the introduction of IT dimension through visualization tools adapted to decision theory) vary in proportion with personality factor. The present paper belongs to the field of empirical (exploratory) research, which contains scientific reflections on measuring the profile of students in management as future employees. In order to test the statistical hypothesis of research, we used SPSS 12 statistics, Statistics 7.0. The research method was the survey and the data collection tool was the questionnaire based on Cattell’s personality test, known as the Personality Item or Cattell’s P.F.Q [6]. The answers were given on a 5-point Likert scale. After collecting data, we chose the appropriate statistical method as follows: firstly, we applied data analysis to reveal the characteristics of study participants; secondly, we tested whether the elements that make up the personality measurement tool have a single dimension, a single latent factor. This basic rule is known as unidimensionality and is related to the second hypothesis, namely the local independence of the elements [27]. The performance of a subject in relation to a test element is predicted and explained by the existence of a latent factor, and the relation between this performance and latent factors that represent its basis is described by a monotonous and increasing function called the element response function or the characteristic curve of the element [29]. Such an analysis will estimate the viability of the instrument by measuring the internal consistency of elements insofar as they correlate well. The viability of our instrument has been calculated in accordance with Spearmen’s theory [27]. Interpretation of the results was carried out according to the literature [30], namely a scale or test has a better fidelity when fidelity coefficient is higher than 0.70 [27]. At this stage, we decided if the instrument used has an average degree of confidence or consistency, so that its results are the same over time and can be used in scenarios similar to those discussed in this article. The study continued with cluster analysis based on [13] and [36].

The Impact of Personality, Attitude and Visual Decision-Making

111

5 Findings and Discussion Descriptive analysis enabled us to create a course participant profile. We chose to predict information related to the characteristics of the study participants both textually and graphically. Student characteristics are: out of 100 participants, 84% are female, 16% are male; 98% are 18–25 years old, 2% 25–30 years old; 88.70% like to use Moodle platform, 11.70% do not. As to their learning style, 30% learn mostly visually; 3% mostly auditory; 42% mostly by writing and reading; 24% rely on learning through demonstrations, simulations, case studies, practical tasks; the remaining 1% is mixed. Next, we performed a reliability-validity test of the instrument meant to measure the economics students’ profile as future employees. The test was found to be sensitive to the following measured characteristics: alpha Cronbach = 0.752, Mean = 98.959, Std.Dv. = 10.683 and average inter-element correspondence = 0.740, so the number of test items corresponds and an increase of these items is not needed. Then we tested the hypothesis of unilateral characteristic of the study (there are no differences between students and the effect of interaction with the tool) in the ANOVA test. (F = 89.079, p value = 0.000) confirm that there are differences between students’ responses according to their personality and transversal skills. We applied a visual analysis method aimed at clustering the answers. The results confirm those obtained by [33], according to whom there are various personality patterns within a collectivity and these need to be known so as to understand subjects’ reactions when introducing the ‘new’. In addition, [2] and [3] argue that, according to the differences in personality, personal interests and degree of cognitive development, learners accept the ‘new’ in different ways. In this context, we tested the attitude stated by students when introducing the dynamic and visual dimensions of decision theory. The analysis of the answers revealed that 63% are confused, 53% Bored, 38% Frustrated, 33% Motivated and 30% Happy to learn the IT dimension of their potential future job. Most of those who declared themselves Confused are also Bored and Frustrated, eliminating the Happy and Motivated states. Ten percent of those with bored, frustrated and confused attitudes declared to be motivated to learn to eliminate their three uncomfortable states, which identified a group ready to adapt to the new. In conclusion, most students show resistance to change. The analysis continued with identifying students’ attitudes (Frustrated, Confused, Bored, Motivated, and Happy) within the five personality types resulting from our study. In terms of attitudes towards introduction of the IT dimension in the decisional process, the results are as follows: the Intellectual is: Frustrated 5.50%, Confused 23.50%, Bored 48%, Happy 18%, Motivated 5%; the Emotionally stable is: Frustrated 9.30%, Confused 38.70%, Bored 26.70%, Happy 13.70%, Motivate 11.70%;the Conscientious is: Frustrated 10.70%, Confused 36%, Bored 30.70%, Happy 14.30%, Motivate 8.30%; the Extravert is: Frustrated 48%, Confused 41%, Bored 8%, Happy 1%, Motivated 2%; the Agreeable is: Frustrated 48%, Confused 41%, Bored 8%, Happy 1%, Motivated 2%. To sum up, Agreeable and Extravert personality types are the most frustrated, bored, confused, thus showing resistance to the new while the Intellectual accepts with greatest ease the new, followed by the Emotionally stable and the Conscientious. The Intellectual, Conscientious, and Emotionally stable are most

112

L. Stanca et al.

adaptable to change because they are inclined towards assimilation of knowledge while the Agreeable and Extravert are soft-oriented. So, hypothesis 2 is validated as the affective (cognitive) states of the students vary depending on the personality factor and determine the attitude related to acceptance/resistance to the new, in our case learning IT through visualization tools adapted to economic decisional theory. The statistical analysis continued with the application of MacQueen’s K-mean method, perfected by Diday’s dynamic clouds method [13]. The implementation of this method has generated the k-mean algorithm based on [36] aiming to generate clusters with similar student behavior in relation to the acceptance of visual tools. In the first stage (the aggregation stage), we tested the hypothesis: are there differences between subjects regarding their acceptance of visual tools as appropriate in increasing the understanding of decision-making process? The results show that students can be grouped into two clusters based on the set of studied attributes. These clusters are characterized by the fact that the inter-class inertia value significantly exceeds the value of the in-class inertia: The set of attributes representative of clustering (homogeneous and well-defined groups of objects) are: Declared learning styles (F = 13.583, p = 0.04), Full of ideas (F = 26.3, p = 0.001, G-square = 27.26), Empathic (F = 15.38, p = 15.55, p = 0.0004), Ordered (F = 17.898, p = 0.001, G-square = 20.778), I can create my own schedule (F = 48.717, p = 0.0000, G-square = 60.469), Make people feel at ease (F = 26.831; p = 0.0001; G-square = 29.062), I understand abstract ideas (F = 32,3305; G-square = 21.778 p-value = 0,000003). Generalizing this result would require a multi-generational study following the same teaching/learning scenario (Table 1). To conclude, cluster analysis lead us to a set of attributes which characterize student behavior in terms of acceptance of IT in the teaching of economics in the digital era. The results are optimistic and demonstrate that, by introducing team teaching of economics and IT (tools for visualizing economic phenomena), a small number of students are likely to cope with the new realities of the labor market. At the end of the course, they will have the necessary knowledge to understand the role and use of dashboards in command, control and agile economic decision-making, thus most likely making future successful employees. An interesting find for our communication age is the high percentage of introverted students, with low levels of sociability, who tend to focus on their inner self rather than outside, towards the social environment. According to neurobiological personality theory, this particular trait is beneficial in the labor market, because introverts need less incentive to reach an appropriate level of excitement [12]. Introverted people are attracted by tasks that place more emphasis on specialist competencies and accept the new for meaningful career progression. On the other hand, there are a considerable number of extrovert students, who may exhibit levels of under-excitement during programming tasks because of their stronger need for social interactions, which show that they would be more appropriate in jobs involving social interaction [14]. Extroverts are attracted by tasks related to organizational culture, team building and less to technical skills. Extroverted individuals described by conscientious and open attitudes fit EU documents on education 3.0, characterized by participation, co-creation, becoming, while introverts have adaptability issues with these ideas and principles. Findings suggest a possible solution to bridge the gap between knowledge creation

The Impact of Personality, Attitude and Visual Decision-Making

113

Table 1. Cluster analysis- results. Cluster 1 is characterized by values 2–3 for declared attributes and includes students who have low to average acceptance of the introduction of visualization tools. The profile of these students is: grades 7–8 (out of 10) for the discipline economics and 5–6 for IT. The features: the soul of the party, use difficult words, do not have a good imagination, make people feel at ease, difficulty in understanding abstract ideas, not empathic, with their own schedule, often forget to put things back, last to laugh at jokes; learning styles are writing and reading, learning through demonstrations, simulations, case studies, do not like working in teams and acting strangely under stress So: student behavior is fluctuating, and the identified attributes are a little contradictory, they show variable appreciation, the grades in economics are average, and in IT are just above the fail limit. This particular grouping and students’ unstable behavior can be explained by the fact that their interest in the subject and the field of study is rather low

Cluster 2 is characterized by values 3–5 for declared attributes and includes students who have an average-to-high acceptance of the introduction of visualization tools. Their grades are 8–10 for economics and 7–10 for IT. They features: excellent ideas; quickly understand things; they are full of ideas; pay attention to details; respond immediately; orderly; demanding in their work; do not mind being in the middle of the party; communicative; interested in people; empathic; make time for others; feel the emotions of others; make jokes all the time, like music and like reading books; learning style is mostly visual, auditory and projects So: student behavior is constant; they seem to accept the introduction of IT tools for explaining managerial phenomena because they are aware of living in an era which demands this. Hence, even if they do not like IT in particular, they have an attitude of acceptance, as shown by their average to high interest in the subject (the field of study)

process and organizational performance by introducing dashboards as decision-making tools. These tools embedded into the learning process may increase student confidence according to [5, 16, 34], who emphasized the importance of acquiring skills in using graphs and methods of viewing data in reporting. Our study has some limitations. Firstly, as regards the research tool, data was collected at the end of the academic year in a very short period of time (5 days). In the future, we intend to collect data over a longer period of time, with pre-and postsurveys. Then the answers may have been influenced by the bonus for the semester evaluation. We believe total anonymity would have given more precise results. Also, the target groups have distinct profiles (especially in terms of age, educational environment, learning experience, motivation) that have not been sufficiently analyzed.

6 Conclusions Our research sustains specialist literature that provides arguments and tries to find solutions to make economists’ literacy in the field of data easier, by identifying the means to induce non-technical people’s computational, statistical, and skeptical thinking alongside the necessary business skills, human resources skills and knowledge

114

L. Stanca et al.

of the organizational context with less resistance to the new. In this context, we designed a research scenario for the introduction of transdisciplinary materials in economics as a possible solution, and argue that teaching/learning economics in the Big Data era can be facilitated by the use of IT visualization tools (dashboards). An encouraging finding is that the introduction of new approaches in the teaching process of economic disciplines generates positive reactions on a part of the subjects considered, which determined better learning outcomes and a higher percentage of students who passed the exam. A long-term goal would be to identify student profiles based on an interdisciplinary approach (cultural, educational, demographic, language-level) that would allow us to personalize the collaborative learning process into more efficient working groups. This would help to build a relationship-based model of methods from game theory, which would help to optimize the learning process and bring increased professional, social and personal results to all stakeholders.

References 1. Association to Advance Collegiate Schools of Business (AACSB): Information Technology Skills and Knowledge for Accounting Graduates: Standard A7 (2014). www.aacsb.edu// media/AACSB/Pub-lications/white-papers/accounting-accreditation-standard-7.ashx 2. Bayne, S.: Smoothness and striation in digital learning spaces. E-learn. Digit. Med. 1(2), 302–316 (2004) 3. Bayne, R.: Psychological Types at Work: An MBTI Perspective: Engage Learning EMEA (2004) 4. Bouquin, H.: Le controle de gestion: 5eme edn. Gestion PUF (2001) 5. Blocher, E., Moffie, R.P., Zmud, R.W.: Report format and task complexity: interaction in risk judgments. Account. Organ. Soc. 11(6), 457–470 (1986) 6. Cattell, R.B., Eber, H.W., Tatsuoka, M.M.: Handbook for the Sixteen Personality Factor Questionnaire (16PF). Institute for Personality and Ability Testing, Champaign (1970) 7. Capriotti, R.J.: Big Data: bringing big changes to accounting. Pennsylvania CPA J. 85(2), 36–38 (2014) 8. Coyne, E.M., Coyne, J.G., Walker, K.B.: Big data information governance by accountants. Int. J. Account. Inf. Manage. 26(1), 153–170 (2018) 9. Duffy, J.: Something funny is happening on the way to knowledge management. Inf. Manage. 34(4), 64–70 (2000) 10. Davenport, T.H., Prusak, L.: Working Knowledge: How Organizations Manage What They Know. Harvard Business Press, Boston (1998) 11. Dilla, W., Janvrin, D.V., Raschke, D.: Interactive data visualization: new directions for accounting information systems research. J. Inf. Syst. 24(2), 1–37 (2010) 12. Earley, C.E.: Data analytics in auditing: opportunities and challenges. Bus. Horiz. 58(5), 493–500 (2015) 13. Enachescu, D.: Data maining, Metode si aplicatii. Editura Academiei Romane, Bucuresti (2009). ISBN:978-973-27-1798-1

The Impact of Personality, Attitude and Visual Decision-Making

115

14. Elmqvist, N., Moere, A.V., Jetter, H.-C., Cernea, D., Reiterer, H., Jankun-Kelly, T.: Fluid interaction for information visualization. Inf. Vis. 10(4), 327–340 (2011). https://doi.org/10. 1177/1473871611413180 15. Hall, R., Andriani, P.: Managing knowledge associated with innovation. J. Bus. Res. 56(2), 145–152 (2003) 16. Hirsch, B., Seubert, A., Sohn, M.: Visualisation of data in management accounting reports: how supplementary graphs improve every-day management judgments. J. Appl. Account. Res. 16(2), 221–239 (2015) 17. Greller, W., Drachsler, H.: Translating learning into numbers: a generic framework for learning analytics. Educ. Technol. Soc. 15(3), 42–57 (2012) 18. Papulová, Z., Mokroš, M.: Importance of Managerial Skills and Knowledge in Management for Small Entrepreneurs, E-leader, Prague, pp. 1–8 (2007) 19. Pritchett, P.: New Work Habits for a Radically Changing World: 13 Ground Rules for Job Success in the Information Age, vol. 3, pp. 15–17. Pdtchett & Associates, Dallas (1994) 20. Vila, L.E., Cabrer, B., Pavía, J.M.: On the relationship between knowledge creation and economic performance. Technol. Econ. Dev. Econ. 21(4), 539–556 (2015). https://doi.org/ 10.3846/20294913.2013.876687 21. Vandervelde, S.D., Chen, Y., Leitch, R.A.: Auditors’ cross-sectional and temporal analysis of account relations in identifying financial statement misstatements. Audit. J. Pract. Theor. 27(2), 79–107 (2008) 22. Wang, T., Cuthbertson, R.: Eight issues on audit data analytics we would like researched. J. Inf. Syst. 29(1), 155–162 (2015) 23. Vrejoiu, M., Zamfir, M.C., Florian, V: Vizualizarea Datelor Masive si Visual Analytics. Abordări Şi Tendinţe. Revista Română de Informatică şi Automatică 27(2), 1–12 (2017) 24. Horn, M.: KAIST Doesn’t Wait For Change In Korea, Pioneers ‘Education 3.0’ Forbes Magazine. Accessed 19 August 2014 25. Yigitbasioglu, O., Velcu, O.: A review of dashboards in performance management: implications for design and research: Int. J. Account. Inf. Syst. 13, 41–59 (2012) 26. Scholz, T.M.: Big Data in Organizations and the Role of Human Resource Management: A Complex Systems Theory-Based Conceptualization. Peter Lang International Academic Publishers, Frankfurt a. M. (2017) 27. Opariuc, C.D.: Analiza Componentelor Principale pentru date Categoriale (CATPCA) (2013). rpru.files.wordpress.com/2013/04/vol10_2_2012.pdf 28. UBB_NTTDATA: The psycho-social profile of IT professionals, What expectations do IT professionals have? (2018).ro.nttdata.com/news/2018/02/the-psychosocial-profile-of-itprofessional-conducted-by-ubb-cluj-and-ntt-data-romania-what-expectat 29. Hambleton, R.K., Swaminathan, H., Rogers, H.J.: Fundamentals of Item Response Theory. Sage Press, Newbury Park (1991) 30. Popa, M.: Statistica pentru psihologi. Teorie și aplicații SPSS, Colecția Collegium, Psihologie (2008). ISBN: 978-973-46-1045-7 31. Myers, I.B.: The Myers-Briggs Type Indicator: Manual. Consulting Psychologists Press, Palo Alto (1962). https://doi.org/10.1037/14404-000 32. Vuorikari, R., Punie, Y., Gomez, S.C. and Van Den Brande, G.: DigComp 2.0: The digital competence framework for citizens. Update phase 1: The conceptual reference model. No. JRC101254. Joint Research Centre (Seville site) (2016) 33. Tlili, A., Essalmi, F., Jemni, M., Chen, N.S.: Role of personality in computer based learning. Comput. Hum. Behav. 64, 805–813 (2016)

116

L. Stanca et al.

34. Martinsons, M., Davison, R., Tse, D.: The balanced scorecard: a foundation for the strategic management of information systems. Decis. Support Syst. 25(1), 71–88 (1999) 35. Kuncel, N.R., Hezlett, S.A.: Fact and fiction in cognitive ability testing for admissions and hiring decisions. Curr. Dir. Psychol. Sci. 19, 339–345 (2010) 36. Ralambondrainy, H.: A conceptual version of the k-means algorithm. Pattern Recogn. Lett. 16(11), 1147–1157 (1995) 37. Stanca, L., Felea, C., Stanca, R., Pintea, M.: The impact of visualization tools on the learning engagement of accounting students. In: Advances in Intelligent Systems and Computing, vol. 1008. Springer, Cham (2020) 38. Schwartz, N.: Making the invisible visible: practical applications of visual metaphors in teaching and learning accounting. J. Vis. Literacy, 1–23 (2020). https://doi.org/10.1080/ 1051144x.2020.1737906

HRS-EDU: Architecture to Control Social Robots in Education John P´ aez1(B) , Enr´ıque Gonz´ alez2 , and Maria Impedovo3 1

Universidad Distrital Francisco Jos´e de Caldas, Bogot´ a, Colombia [email protected] 2 Pontificia Universidad Javeriana, Bogot´ a, Colombia [email protected] 3 ADEF, Aix-Marseille University, Marseille, France [email protected]

Abstract. The assertive actions done by social robots in education have to consider cognitive and emotional aspects of students. The document presents the HRS-EDU architecture to control the social robot behavior in learning environments. Concepts such as psychological theory of flow, scaffolding educational strategy and software agents were considered to design the BDI agent architecture (Beliefs, Desires, Intentions). The computational implementation was made in BDI-BESA which is an integrated development environment designed by the SIRP research group of the Pontificia Universidad Javeriana and integrated into the Baxter robot. To validate the architecture, three phases were implemented: remote control of the robot while the subjects solve the Jumper problem, the subjects try to solve the problem without the robot support and finally, the robot gives autonomous support during the problemsolving process. The validation phases were developed with 15 children ranged between 10–14 years old and who came from three schools placed in Bogot´ a. Through mixed methods research strategy, the events of Baxter, the reactions of the subjects, and the performance of the architecture were analyzed. The results suggest that the design of BDI goals for the control of the robot for the actions that give emotional and cognitive support to the subjects, encourages the learning of problem-solving strategies. Keywords: Cognitive robots · Education Scaffolding strategy · Software agents

1

· Human-robot interaction ·

Introduction

To develop skills to solve problems is a requirement of the knowledge society. While technological devices contribute to solving problems, humans should not lose the ability to solve problems because that ability supports cognitive processes on which intelligence emerges. To foster the skills development, the educative system has been using technological tools as external representational systems. Teachers according to the learner skills mainly regulate the assertive use c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 117–126, 2020. https://doi.org/10.1007/978-3-030-52538-5_13

118

J. P´ aez et al.

of tools during the learning process. This process is well-known as scaffolding. Because of the technological characteristics, social robots can carry out the scaffolding without take over the teacher role, just like an mediational tool. In order to achieve the assertive robot-behavior, this document presents the artificial architecture named HRS-EDU, which control a social robot according to the scaffolding paradigm. In education, social robots are mediational tools that shape thinking during learning process as assimilation and adaptation of new concepts. The anthropomorphic, cognitive and emotional characteristics of social robots are generating interest in the scientific community to acknowledge conditions as cognitive convergence, emotional convergence, physical interaction capacity, and physical intervention to integrate them into learning environments. Therefore, designing tools with the conditions mentioned above, contributes in the field of knowledge to design tools with both emotional and cognitive scaffolding capacity with positive effects on learning. As a contribution, the document presents the HRS-EDU architecture as a framework to control cognitive, emotional, physical actions of robots in learning situations [1]. The concept is based on the epistemological and ontological value of tools as mediational tools. According the HRS-EDU, the robots assertive actions are designed based on the BDI philosophical paradigm. Having said that, Beliefs means the knowledge and emotional state of the learner, Desires means possibilities of intervention based on the theory of flow, and Intentions for executing the plans according to the cognitive, emotional and physical resources of the robot. To develop skills to solve problems is a requirement of the knowledge society. While technological devices contribute to solving problems, humans should not lose the ability to solve problems because that ability supports cognitive processes on which intelligence emerges. To foster the development of problemsolving skills, the educative system has been using technological tools as external representational systems. According to the learner skills, teachers regulate the assertive use of tools during the learning process and it is well known as scaffolding. Social robots can carry out the scaffolding without take over the teacher role, just like an assistive tool because their technological characteristics. In order to achieve the assertive robot-behavior, this document presents the artificial architecture named HRS-EDU, which control a social robot according to the scaffolding paradigm.

2

Tools for Enhancing Learning

The purpose of this section is to reflect on the role that tools - like social robots - play in the learning process. We consider the interaction with tools in the more general framework of artefacts in the sense that all human activities are mediated by artefacts made of signs and tools. The term ‘artefact’ is an aspect of the material world that people use when interacting with their physical and social environment [2]. Artefacts are considered both material objects (tools, images,

HRS

119

drawings, etc.) and ideas (meanings, values, norms, etc.) [3]. Artefacts allow mediation between subject and the world. They expand and augment cognitive possibilities [4] and, at the same time, the use of a particular tool changes and constrains the cognition [5]. Culture provides human beings with the tools and resources to mediate thinking. These resources and tools, some stable and others moving with time, are always implicated in the way people think and develop. A long and historical discussion is around artefact. People transform the external environment in order to use it as an instrument. In the same time, the tool transforms the entire development of the activities. For the Russian psychologist, the role of culture in the development of higher mental functions: for the author “culture does not create anything, it only modifies the natural environment to conform it to their objectives”. Thus, the way we organize the external world culture plays a central role in the development of higher mental functions. As proposed by [7], a French philosopher, human activity is largely designed to establish relations with artefacts and it is the nature of the relations that we establish which them defines the nature of the artefacts that we build. Each object of design becomes a crystallization of the experience about the mediated relationship between humans and world [8]. This relationship is not fixed but, in the same time, historically changing and requires more knowledge that one single person can possess for a complex design. Adopting a new tool in the educational setting imply a process of appropriation. The notion of appropriation is employed when the user begins to use the artefact in his/her environment until a fruitful utilization. [9] have defined the appropriation in terms of process by which a technology or a particular technological artefact is adopted and shaped in use. The process of appropriation also includes aspects concerning the mutual influence between the technology and the users [11], with a simultaneous transformation process including the learner and the tool. The process of appropriation shows the rich and dynamic relationship between the user and the tool. With a focus on artefacts and tools, in current educational literature, increasing attention is being paid to theoretical approaches which emphasize the meaning of interactions between subject and materiality [4] in learning. The perspective of materiality concerns how tools are part of human actions, bringing in the notion of social interaction between artefacts and humans: in the socio-material perspective, the material, human and social dimensions are inseparable [12]. Considering this approach, it is interesting the convergence of different models and solutions that can be adapted and expressed in new pedagogical configuration, as the social robots allows to do in the educational setting.

3

Robots as Tools in Education

The use of anthropomorphic robots for enhancing the learning is recent. Their use has evolved in three ways. First, robots to learn to program. Second, robots as learning objects. Third, social robots as collaborative agents. The last ones are

120

J. P´ aez et al.

a great innovation in technology in education. They evolve to adapt their cognitive, emotional and social behavior of the learner’s characteristics. In another hand, the social robots’ characteristics are oriented in three topics: cognitive convergence [13,14], emotional convergence [15,16] and capacity for physical interaction with learners [17,18]. Cognitive convergence considers aspects such as information processing theory, knowledge domain and user mental models. Emotional convergence considers aspects such as recognition and expression of emotional states related to learning states. Finally, but not least, there is the physical convergence that considers the development of physical intervention mechanisms during the development of the learning task in a similar way to the learner intervention alternatives. The design of architectures for educationaloriented robots integrates psychological, pedagogical and didactic requirements.

4

Methodology

The previous sections presented educative benefits about robots as tools for enhancing the learning. The next section describes the design process of the architecture and validation experience. Theoretical issues as the psychological flow theory, educative scaffolding strategy, and practical reasoning theory were considered to design the robot actions. The architecture was implemented in the Baxter robot through the multi-agent platform named BDI-BESA JAVA [3,21,22] and Robot Operative System ROS. The process of design and validation were implemented in three phases to identify the theoretical support of Desires and Intentions in learning contexts, implement the Desires and Intentions in the architecture, and validate the architecture. 4.1

Theoretical Proposal

The assertive behavior of robot needs to consider aspects such as the psychological flow theory [19], the scaffolding strategy and learner’s characteristics. The Flow Theory main idea is fostering the flow state based on the relation between two variables: learner’ skills and challenge’s problem. According the Fig. 1, h means the learner’ skills which are necessary to solve the problem, d means the problem complexity and f represent the learner’s emotional state. In this context, desires are plausible actions in order to give assertive support to the learner. Each goal implies the evaluation of two mathematical functions: activation and contribution. The first one, based on the beliefs’ module data, evaluates the minimal conditions required to start a goal associated to the scaffolding process. The second one, based on the activation function value, defines that what extent this achievement of the goal contributes to the learning process. In order to control the robot according to the Flow Model, HRS-EDU architecture take the values from h, d, f and selects a scaffolding goal (Fig. 1). Each goal has an associated plan which is applied by the action performed by a Baxter robot. The selection process of the scaffolding goals is developed in the Desires and Intentions module. The scaffolding is a methodology to support the learning process when the

HRS

121

d f dr

F

h

d

fi

di

A

0

B

hj

C

h

Fig. 1. The psychological theory of flow was proposed by Mihaly Csikszentmihalyi. The theory suggests a relationship between skills and emotions which are associated with learning and task challenge. Throughout learning, the learners’ skills change depending on the task’s challenge. Thus, the goal of the scaffolding strategy is to avoid boredom or anxiety states. Flow Theory shows a relation among three variables: challenge di , skills h1 and emotionei .

object of study exceeds the skills of the learner. The steps of this methodology are three: diagnose needs, perform interventions and evaluate performance. The needs consider three aspects cognitive, meta-cognitive and emotional. The scaffolding’s interventions are grouped into four strategies: direct instruction, guided instruction, cooperative work, Individual work, [8]. The Learner’s characteristic are based on two aspects: knowledge of cognitive activity and control of cognitive activity. The first one considers three variables: learner, task, and strategy to solve the problem. The second one considers three variables: planning, supervision, and evaluation. The aspects mentioned above were useful to design the Desires and Intentions of HRS-EDU architecture. The next section gives a brief of HRS-EDU characteristics [1].

5

HRS-EDU Architecture

The intelligent control architecture was designed based on the BDI multiagent paradigm. HRS-EDU is composed by four modules: sensors, believes, desires and intentions, and action. The sensors module recognizes three data sets: body language, speech, and task state. To get the data, two Kinect cameras and a Real Sense Camera were used. The believe module takes the sensorial data to evaluate two learner’s characteristics: cognitive state and emotional state. To evaluate the learner’s cognitive state a Diffuse Cognitive Binary Tree (DCBT) was developed. To evaluate the learner’s emotional state a Machine Learning model based on K-Means algorithm was implemented. The desire and intention module evaluates the belief’s data in order to start or stop the scaffolding educative process through the robot’s goals. The robot’s goals, which were designed according to the scaffolding and psychological flow theories, are divided in five groups: cognitive control, challenge control, emotional control, Mean-End Analysis knowledge,

122

J. P´ aez et al.

and Life’ signals. The action module develops the robot’s plans to give assertive support [23]. A connection between HRS-EDU and ROS (Robot Operative System) is established. HRS-EDU architecture was developed in JAVA and four components were used: BESA, RationalBESA, RobotACT, and rosbridge [20]. 5.1

Desires and Intentions to Control the Robot Behavior

The Desires and Intentions module controls the robot behavior. It selects the goals of the agent that became active and execute the actions to support the learning process. The design and specification of the goals was based on experimental sessions in real learning settings with children. This process involve two steps: learners solving the problem alone, learners solving the problem with Baxter controlled as a puppet by an expert. The first experience was done to identify the Desires and Intentions goals through the cognitive and emotional needs of 7 students while they were solving a transformation problem. Once the Desires and Intentions goals were identified, the second experience analyzed the behavior of 8 students while they were solving the problem with Baxter’s support the Wizard of Oz technique was used. As a result of this two experiences, the BDI goals, which control the robot behavior, were arranged in six sets. 1. Goals for fostering the problem-solving skills: these goals consider two aspects: operational knowledge of the problem and thoughtful thinking about of the strategy that is being used to solve the problem. 2. Goals for cognitive support: these goals involve three types actions: planning, monitoring, and evaluation which are focused to foster the shift of the robot’s responsibility towards the learners in the learning context. 3. Goals for emotional support: the learner’s emotions show the relationship between skills and challenge. The IHR-EDU architecture looking for preserve the learner’s emotional state through three goals: recognize success, recognize failure and promote confidence. 4. Goals to control the problem’s challenge: the problem’s challenge is to solve the transformation problem the Jumper. Through artificial vision, every change of problem status is saved and a graph of the student’s movements is created. Through the Dijkstra’s algorithm, the challenge for solving the problem was estimated. 5. Goals for signs of life: the HRI-EDU architecture uses the signs of life to contribute to the recognition of the robot as an expectant agent that support the learning process. The life signs presented by the robot are breathing, evoke thought, blinking, and whistle. 6. Goal for immediate help: the goal gives support when the learner claims it. Request for help means to acknowledge weakness to solve the problem. For example, in the goal 6 mentioned above, the condition for activating the goal is the BDI belief associated with the child’s request for help through voice. So, since Baxter has a graph of the problem space and the space traveled by the student, he decides on a scaffolding action to promote learning. The learning action can be done through movement, emotional expression or verbal instructions.

HRS

6

123

Validation of the HRS-EDU Architecture

Finally, the identified Desires and Intentions goals presented in the last section were implemented in the HRS-EDU architecture and tested with 15 students. After that, the Human-Robot Scaffolding phenomena was analyzed through Mixed-Method research. Figure 2 shows the experimental scenario.

Fig. 2. Research experience of the Human-Robot Scaffolding. Robot gives emotional and cognitive support to students while them are solving a problem. Baxter does interventions through movements of the blocks and gives prompts or cues about both the problem-solving process and strategy.

The HRI-EDU architecture was implemented in the Baxter robot, an anthropomorphic two-arms industrial robot made by the Rethink Robotic company. This robot is suitable to interact in a safe way with humans and can express emotions with its face, movements and postures. The learning task is the Jumper game: it is composed by two groups of blocks organized on a table. The game’s objective is to exchange the blocks position of both groups. The movements are regulated by two rules: blocks can only jump one position to its left or right side, and blocks can only jump over another blocks belonging to the other group. The game ends when all the blocks of both groups have been exchanged. The collected data were used both to understand the Human-Robot Scaffolding phenomena and the robot’s behavior. The last experiment was carry out at the robotic laboratory with children ranged 12 to 15 years old which come from three schools in Bogot´ a city. Two cameras were used to record the robot’s behavior and the learners behavior. The categories to analyze the experiment were time, problem state, robot role, learner’s skills, control of the challenge, life’s signals, cognitive control, learner’s emotional state, and learner’s attention. The ideal number of steps to solve the problem is 24, the average number of movements for solving the problem was 37 steps using the HRS-EDU to control the robot and the average number of movements for solving the problem was

124

J. P´ aez et al.

55 steps without Baxter support. As for the goals for cognitive control, 73.12% of the subjects’ attention is oriented towards the problem board and 15.82% towards the screen. Regarding the goals for challenge control, the robot support allowed the subject to advance in the solution 78.57% and back down to correct the strategy in 21.43%. As for the goals of life signs, during the experiences 60.41% - Smile, 13.54% - Blink, 12.5% - Breathe, 13.54% - Whistle. Based on the experimental results, learners consider both the robot’s clues and robot’s hints to solve the problem. The learner’s attention to the robot are joined to the learner’s flow state. Learner usually pay attention to the robot when they had emotional states either boredom or anxiety. According to the data of the 15 subjects analyzed, the subjects did not make mistakes in the application of rules. The relevance of the actions measures how appropriate the movements of the subjects are when they must choose between different possibilities to change states during the trajectory towards the goal. The best continuous options measure the tendency of the subject to repeatedly perform the movements with the best option. The percentage is 76%. The average skill is 84%. The Jumper game is not a trivial learning problem. It has a high level of desertion. When Baxter supports the solving process learners make less mistakes and take more time to analyze and planning the strategy to tackle the problem. The light blue region means the path to solve the problem. In Figs. 3b and c, the points of different colors represents the nodes visited by the learners. The Fig. 3c shows a significant difference when learners solve the problem without the Baxter support which considers the Desires and Intentions goals (Fig. 3). A video-fragment which resume the research process is available in the video https://youtu.be/qbohCjBIwYc.

Fig. 3. Figure shows the problem-space of the game. The graph is represented by black nodes and edges. It means all possible movements to change the game state along of the problem-solving process. The blue paths means the students trajectory to solve the problem. a) The ideal path to solve the problem with minimal movements. b) The path to solve the problem with the Baxter support. c) The path to solve the problem without Baxter support.

HRS

7

125

Conclusions

The configuration of the module of desires and intentions integrated in a novel way three aspects: the five sets of goals (skills development, emotional control, challenge control, signs of life, cognitive control), the hierarchy of goals (frustration control, highlight critical conditions, maintain direction, reduce degrees of freedom, maintain attention) and the means of intervention (feedback, advice, instruction, demonstration, questions). The novel articulation of the three aspects, promotes an assertive behavior of the robot to assist the subject during learning. As evidenced in the qualitative and quantitative analysis, the subjects attend the instructions aimed at solving the problem and the instructions themselves correspond to the cognitive and emotional needs of the subject. This indicates that the conditions modeled to promote the scaffolding strategy with technological devices such as Baxter, are affected, and according to the results obtained in the analysis, avoid emotional states of anxiety and boredom during the solution of the problem. We consider that the experience presented is a starting point to investigate aspects such as the relationship between body language and learning, the recognition of unconventional emotional expressions linked to the state of flow during learning, the exploration of multimodal fusion methods that combine emotional traits. and cognitive to better estimate the student’s needs and finally, but not least, refine the implementation of the theory of BDI agents for controlling the educative agent. Acknowledgements. The authors acknowledge the support of the Universidad Distrital Francisco Jos´e de Caldas, Pontificia Universidad Javeriana, and the Laboratory of ADEF of Aix-Marseille University.

References 1. Rodriguez, J.J.P., Guerrero, E.G.: Human-robot scaffolding: a novel perspective to use robots such as learning tools. In: 2017 18th International Conference on Advanced Robotics (ICAR), pp. 426–431. IEEE, July 2017 2. Cole, M.: Cultural Psychology: A Once and Future Discipline. Harvard University Press, Cambridge (1996) 3. Wartofsky, M.W.: Models: Representation and the Scientific Understanding, vol. 48. Springer, Dordrecht (2012) 4. Clark, A.: Transforming Children’s Spaces: Children’s and Adults’ Participation in Designing Learning Environments. Routledge, London (2010) 5. Ackermann, E.: Experiences of artifacts: people’s appropriation, objects’ affordances. In: Keyworks in Radical Constructivism, pp. 249–259 (2007) 6. Vygotsky, L.S.: Mind in society. In: Cole, M., John-Steiner, V., Scribner, S., Souberman, E. (eds.) (1978) 7. Gilbert, S.: Du mode d’existence des objets techniques. Aubier, Paris (1958) 8. Kuutti, K.: Activity Theory as a Potential Framework for Human-computer Interaction Research. Activity Theory and Human-computer Interaction, Context and Consciousness, p. 1744 (1996)

126

J. P´ aez et al.

9. Agre, P.E.: Plans and Situated Actions: The Problem of Human-machine Communication: Lucy A. Suchman, (Cambridge University Press, Cambridge, 1987), 203+ xii P. (1990) 10. Jones, A., Issroff, K.: Motivation and Mobile Devices: Exploring the Role of Appropriation and Coping Strategies (2007) 11. Overdijk, M., van Diggelen, W.: Appropriation of a shared workspace: organizing principles and their application. Int. J. Comput. Support. Collaborative Learn. 3(2), 165–192 (2008) 12. Orlikowski, W.J., Scott, S.V.: 10 sociomateriality: challenging the separation of technology, work and organization. Acad. Manage. Ann. 2(1), 433–474 (2008) 13. Keren, G., Fridin, M.: Kindergarten Social Assistive Robot (KindSAR) for children’s geometric thinking and metacognitive development in preschool education: a pilot study. Comput. Hum. Behav. 35, 400–412 (2014) 14. Mirolli, M., Parisi, D.: Towards a Vygotskyan cognitive robotics: the role of language as a cognitive tool. New Ideas Psychol. 29(3), 298–311 (2011) 15. Kishi, T., Kojima, T., Endo, N., Destephe, M., Otani, T., Jamone, L., Kryczka, P., Trovato, G., Hashimoto, K., Cosentino, S., Takanishi, A.: Impression survey of the emotion expression humanoid robot with mental model based dynamic emotions. In: 2013 IEEE International Conference on Robotics and Automation, pp. 1663– 1668. IEEE, May 2013 16. Petrovica, S., Anohina-Naumeca, A., Ekenel, H.K.: Emotion recognition in affective tutoring systems: collection of ground-truth data. Procedia Comput. Sci. 104, 437– 444 (2017) 17. Kwak, S.S., Kim, Y., Kim, E., Shin, C., Cho, K.: What makes people empathize with an emotional robot?: the impact of agency and physical embodiment on human empathy for a robot. In: 2013 IEEE RO-MAN, pp. 180–185. IEEE, August 2013 18. Leyzberg, D., Spaulding, S., Scassellati, B.: Personalizing robot tutors to individuals’ learning differences. In: Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction, pp. 423–430. ACM, March 2014 19. Csikszentmihalyi, M.: Toward a psychology of optimal experience. In: Flow and the Foundations of Positive Psychology, pp. 209–226. Springer, Dordrecht (2014) 20. P´ aez Rodr´ıguez, J.J.: Human-Robot Scaffolding: Arquitectura BDI para el desarrollo de habilidades de soluci´ on de problemas 21. Bravo, F.A., Gonz´ alez, A.M., Gonzalez, E.: Design of a multi-agent architecture for implementing educational drama techniques using robot actors. In: International Conference in Methodologies and intelligent Systems for Technology Enhanced Learning, pp. 172–180. Springer, Cham, June 2018 ´ 22. Angel, R., Gonz´ alez, E., Gonz´ alez, A.: MSSIN: agent based social simulation model with an intelligent approach. In: 2013 8th Computing Colombian Conference (8CCC), pp. 1–6. IEEE, August 2013 23. Gonz´ alez, E., P´ aez, J., Luis-Ferreira, F., Sarraipa, J., Gon¸calves, R.: Human-robot scaffolding, an architecture to support the learning process. In: Iberian Robotics Conference, pp. 528–541. Springer, Cham, November 2019 24. Jones, A., Issroff, K.: Learning technologies: affective and social issues. In: Contemporary Perspectives in E-Learning Research, pp. 208–220. Routledge, London (2006)

Authoring Interactive Videos for e-Learning: The ELEVATE Tool Suite Daniele Dellagiacoma1(B) , Paolo Busetta1 , Artem Gabbasov2 , Anna Perini2 , and Angelo Susi2 1

Delta Informatica SpA, Trento, Italy {daniele.dellagiacoma,paolo.busetta}@deltainformatica.eu 2 Fondazione Bruno Kessler (FBK), Trento, Italy {agabbasov,perini,susi}@fbk.eu

Abstract. Interactive videos are becoming common as e-Learning tools, applied to different knowledge areas and for different teaching objectives. Active research concerns the definition of methodologies and authoring tools for teachers and instructional designers for the creation of interactive videos, as well as studies on how to evaluate their effectiveness in distance learning. In the context of the ELEVATE project, we are developing a tool suite that offers a novel environment for authoring and management, including a virtual reality component for video production. This suite is expected to be mainly adopted within blended online courses for professional (adult) training on emergency management procedures. This paper presents the architecture of the ELEVATE Tool Suite and gives an overview of how it can be used, pointing out the role of different stakeholders.

Keywords: e-Learning Virtual reality

1

· Video-based learning · Interactive videos ·

Introduction

The application of educational videos in e-Learning has developed enormously, especially in the context of Massive Open Online Courses (MOOCs) and in flipped classrooms [15]. Different types of educational videos are used, depending on the specific teaching objectives, for example how-to videos, when the teacher aims at letting the student consolidate knowledge on how to perform a procedure, or live registrations, documentaries, and others [14]. Educational videos can have different characteristics and can offer a different degree of interaction to students during a session. Indeed, interactivity mechanisms offered by online educational videos range from basic video-player operations, i.e. stopping, pausing, and replaying the video, to the inclusion of links or 3D-objects that can support free exploration during a session, to videos with quizzes that can engage students in taking decisions [9]. c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 127–136, 2020. https://doi.org/10.1007/978-3-030-52538-5_14

128

D. Dellagiacoma et al.

In the context of the ELEVATE (E-LEarning with Virtual interAcTive Experience)1 project, we focus on interactive videos for training professionals on the execution of critical procedures (e.g. patient reanimation in the healthcare domain or emergency management in civil protection), to be used in distance learning. The main objective of the project is to develop a flexible software system, called ELEVATE Tool Suite, for authoring and managing such interactive videos. Interactivity is achieved through quizzes that are presented to the trainee in-between video playing, keeping her attentive [14], and influencing the flow of the exercise. Consequently, the trainee is allowed to explore different video sequences. Furthermore, leveraging on the author’s previous experience on the application of virtual reality (VR) to training [2,5], the ELEVATE Tool Suite supports the use of VR for video generation. The rest of this paper is organised as follows. In Sect. 2 we analyse the ELEVATE’s application domain, including stakeholders’ needs and provide details on the tool suite. In Sect. 3 we give an overview of how ELEVATE can support authoring and managing interactive video-based courses that are delivered on Moodle. We discuss related work in Sect. 4, then conclude and outline ongoing developments in Sect. 5.

2

Application Domain and Tool Suite

The application domain considered in the ELEVATE project concerns continuous education and training of adults on emergency management procedures. In such context, specific educational methodologies are adopted, which recognize a central role to practical experience that trainees have to develop in dedicated simulation sessions performed in realistic settings. Distance learning is also considered, but mainly for teaching descriptive knowledge regarding the domain of interest. Exploring how to exploit distance learning also for teaching procedural knowledge, as an effective complement to simulations in realistic settings, is the underlying motivation of the ELEVATE project. The analysis of the needs of the main stakeholders involved, conducted with the help of a project partner and by analyzing relevant domain literature2 , resulted into the identification of challenging aspects such as: C1 Given that creating effective educational material, on a given subject, requires specialised skills, which are different from those necessary for producing a “good” video [14], how can we take this into account while defining the ELEVATE authoring process? C2 While VR can help to generate videos of emergency scenarios with (artificial) people, thus avoiding problems due to regulation restrictions (e.g. privacy) or danger for lives and things, how can we integrate it in the ELEVATE authoring process in a sustainable way? 1 2

The ELEVATE project - https://elevate.deltainformatica.eu/, is a research and innovation project, proposed and coordinated by Delta Informatica. See for instance, https://www.erc.edu/.

Authoring Interactive Videos for e-Learning

129

C3 How can we exploit data collected from the logs of trainees’ sessions to improve an interactive video? C4 How can we evaluate the effectiveness of the ELEVATE interactive videos for training on emergency procedures? A deeper analysis of the characteristics of the intended users of the platform, as well as previous work highlighting the diverse skills needed to design VR objects appropriate as educational material (e.g. [6]) and educational filmmaking [14], motivate us to define different authoring roles for the users of the suite, namely instructional designer (called also exercise designer in the context of the project), and director. Each of these roles requires specialized background skills. If they are fulfilled by two people (or more, since there may be several directors working on different clips), the tool suite has also to manage their collaboration during the authoring process. The design and implementation of the ELEVATE Tool Suite described below reflect how we addressed C1, C2, and partly C3, while C4 will be discussed in future works. 2.1

The ELEVATE Tool Suite Components

The architecture of the suite is depicted in the Unified Modeling Language (UML) diagram in Fig. 1, in terms of its main components and their relationships. Its core component is a Web-based graphical story-editor called Exercise Design Tool (EDT), used by the exercise designer to create an exercise by defining its structure and by associating the appropriate video clips. An example of an exercise structure is shown in Fig. 3, which contains a snapshot of the EDT graphical user interface (main window).

Fig. 1. The ELEVATE components

130

D. Dellagiacoma et al.

The exercise structure is a directed graph, which may contain cycles. Its nodes (“fragments”) are placeholders for video clips; the edges outgoing from a node (“choices”) represent options that are offered to a trainee at that point in the story, possibly according to contextual parameters. The destination of an edge is the point from which the story will resume if that edge is followed, i.e. if the trainee takes that choice. Thus, a path through a story graph represents a view of the video or, equivalently, a sequence of choices. While editing a fragment, the exercise designer can browse the media library and see if it contains an appropriate clip, otherwise she creates a video recording request, which is central to the collaboration with the director, as discussed later. The media library allows the uploading of clips, the editing of their titles and descriptions, and their “tagging”. Tags are used both to support searches and, as it will be presented later, for run-time condition evaluation on choices. The video clips stored into the media library may have been recorded externally, e.g. by shooting in real-life settings or by exporting from a multimedia authoring system, or by using another tool of the suite called ELEVATE Video Editor (EVE). This is a VR system developed with the popular Unity game engine3 . EVE offers a virtual environment to create situations that could be difficult to be replicated in the real world for many reasons, e.g., dangers in fire emergencies involving crowds, or privacy restrictions on real-life shooting. A scene in EVE is created by loading a 3D scenario and populating it with a variety of artificial actors (so-called NPCs, Non-Player Characters) and graphical assets. The director gives NPCs a simple script to follow; typically, this is a sequence of goals to be achieved by the underlying goal-oriented multi-agent framework. Once the script is ready, the scene can be recorded by letting the NPCs behave autonomously in the virtual environment as if they were actors on a stage. EVE allows the saving of the state of this virtual stage at any time, from which to restart to create another video. Once all required clips are available in the media library, the exercise can be published online on an ELEVATE server and made available for use. A published exercise is composed of standard multimedia files and ELEVATE-specific metadata containing its structure, conditions and other parameters. An ELEVATE Web-based Video player interprets the exercise’s metadata, controls the interactive video progress and stores data about the interactions of the student with the video, including choices, the number of visits to a fragment, timing and so on. This video player is compatible with any Web-based LMS supporting the SCORM standard (part of the IEEE 1484 series for e-Learning Technologies), so exercises can be embedded in any online course. Another ELEVATE tool, called Exercise Management Tool (EMT), allows course organizers to manage published exercises and analyze the statistics gathered by the video player. EMT, as well as all tools of the ELEVATE suite, manages authentication, group and role-based authorization of its users so that proper protection of the trainees’ exercise data can be put in place. Note that 3

https://unity.com/.

Authoring Interactive Videos for e-Learning

131

trainee authentication and trainee data, apart from those collected by the video player, are handled by the LMS and thus outside of the scope of ELEVATE. Ongoing work aims at extending the ELEVATE Tool Suite, so to address the above mentioned additional requirements.

3

The ELEVATE Tool Suite at Use

The ELEVATE Tool Suite aims at supporting a collaborative authoring process and continuous improvement of the educational material based on the analysis of session logs. An overview of these processes and the main roles involved are sketched in Fig. 2. We describe them in the rest of this section, whereas a case study is illustrated in [4].

Fig. 2. The ELEVATE workflow

3.1

Collaborative Authoring

Before designing a story structure, the exercise designer must pinpoint the goals of the exercise and the criteria used to judge their achievement or failure by the trainees. As can be seen in Fig. 3, EDT supports defining the exercise story structure by adding and editing fragments that compose it. Each fragment includes a video clip that has to be selected from the media library, and the choices that link it to the following fragments. The fragments without any outgoing choice represent the end-points of the interactive-video exercise. Moreover, the details of the currently selected fragment are displayed on the left, where the corresponding video clip and its choices can be seen.

132

D. Dellagiacoma et al.

Fig. 3. Snapshot of the EDT graphical user interface

A simplified logic language, where the set of true facts are represented as simple labels (“tags”), allows the designer to make choices conditional, i.e., to enable or disable them while a trainee does the exercise. Tags can be explicitly asserted or retracted by the designer as side-effects of choices, but many are set automatically by the run-time engine (e.g., concerning the time elapsed during exercise execution) or inherited from the execution environment and video clips. In the current version, there is no restriction nor semantics associated with the naming of tags; future work will investigate the use of an ontology and related extensions of the run-time language e.g. to help the designer when choosing names and to generalize conditions. As mentioned above, if a suitable clip cannot be found in the media library, the exercise designer creates a video recording request, which links back to the exercise and to the specific fragment to be satisfied. Video recording requests are handed to the director, who decides which tool to use (EVE or otherwise) and additional under-specified aspects (e.g., camera perspective, objects on the scene, and so on). Once ready, the director uploads a recorded video to the media library and closes the related video recording request. The exercise designer either approves the video or rejects it, reopening the recording request with comments explaining the reason for the refusal. Conversely, the director can provide feedback to the exercise designer before recording; furthermore, if a request cannot be satisfied for any reason related to the exercise, e.g. the instructions on the fragment are not clear, the request is put into an “error” state. It is left to the exercise designer to change the request as appropriate and resubmit it to the director. The video recording request cycle just described may cause other effects. For instance, feedback from the director may lead the exercise designer to rethink the exercise and readjust the entire story graph. A director using EVE may discover that the available 3D assets are inadequate or similarly that the behaviour of the

Authoring Interactive Videos for e-Learning

133

NPCs, controlled via the EVE scripting language, is wrong or goals or actions missing, consequently requiring changes to the underlying AI support. In the current ELEVATE use cases, the development team works in close contact with a director, so issues can be addressed immediately. In a future commercial phase, the envisaged business model foresees the creation of a store of 3D assets and behavioural models continuously enriched by a community of developers and users, as commonplace in multimedia creation tools (consider, e.g., Unity’s Asset Store or the numerous resources for Microsoft PowerPoint). Once all video recording requests have been satisfied, the exercise designer generates the full interactive video with a quick and fully automated EDT operation; the result can be previewed with the video player launched by EDT itself. Typically, before online publication, this video is subject to a further phase of quality control, e.g. by a domain expert or by an instructional designer creating its containing course (the “reviewer” in Fig. 2). At the moment, this review step is not supported by a dedicated tool, since it is expected that exercise designers and reviewers work in close contact (often they are the same person), thus an informal process is enough. The design/video clip recording/interactive video generation/review cycle is repeated until reviewers are satisfied. Eventually, the exercise designer publishes the exercise on a server running the ELEVATE suite. The exercise metadata and the video player can be downloaded as a SCORM package, uploaded to any SCORM-supporting LMS, and embedded in any online course. The ELEVATE video player streams the clips directly from the server and stores the trainee’s interactions on the same server, together with LMS-specific trainee identifiers. 3.2

Exercise Evolution

As mentioned above, EMT allows authorized course organizers and trainers to analyze the collected data and possibly take actions. Some data are specifically meant to evaluate how an exercise is used; e.g., a so-called “heatmap” shows the frequency of use of paths over a certain number of sessions, and may be useful to understand whether certain choices are never or always taken and consequently provide feedback to the exercise designer for revisions. Within a single exercise session, the choice of a certain path, specific key performance indicators (e.g., the time needed by the trainee to take her decision) or patterns of behaviours (e.g., a combination of choices and timings) may be used to evaluate performance and build a trainee profile. In turn, the latter can be represented as tags and thus it can influence which choices are offered, thus dynamically adapting the exercise.

4

Related Work

Different aspects of video-based learning are actively addressed by researchers. Concerning the work presented in this paper, worth to be mentioned is research on the evaluation of the effectiveness of educational video on cognitive processes

134

D. Dellagiacoma et al.

during training and more generally in e-Learning, which puts in evidence the relevance of the ELEVATE project objective. Empirical study research approaches have been used to study the impact of videos during training sessions in the healthcare domain, such as [1], where an empirical study on cognitive and metacognitive processes during pediatric training with and without the use video cases is reported. A comprehensive review of research work on educational video in e-Learning is presented in [7], together with a discussion on research aspects that would need further attention. For example, the authors pointed out the lack of studies on the instructor perspective, e.g. design challenges they encounter, as well as the need of further studies on the quality of video podcasts with the pedagogical strategies in mind, and finally investigating more on individual differences in the use and impact of video in e-Learning, and the opportunities that can be derived. Our aim in ELEVATE is to take advantage of these recommendations, especially in the definition of a tool supported authoring process, which could enable a seamless improvement of the educational material leveraging on the analysis of sessions’ log. Focusing on interactive videos in e-Learning, several studies analysed the positive impact of different levels of interactivity of educational videos, by measuring students’ learning results, as well as perceived usefulness by the students. Zhang et al. [16] present an empirical study with 138 undergraduate students to evaluate learning with and without video, as well as learning with interactive videos vs. learning with non-interactive video. The results showed that “the interactive video with random content access may help students enhance understanding of the material and achieve better performance, while non-interactive video may have little effect”. The same outcome was observed in terms of benefits perceived by the students. The study presented in [3] features the interface involving the students to identify the safety risks by clicking on the corresponding objects while the video is playing. The majority (75%) of students agreed that “the interactive video had enhanced their learning experience”. Another study presents an environment allowing to insert quizzes into already existing videos, with 70% of the students having reported that the environment “helped them to some extent, or more to deepen their understanding of the learning materials” [8]. The study presented in [10] extends this opportunity with quizzes and hints integrated directly to the video. The additional hints, as well as questions together with input forms, are shown on top of a frame of a paused video. The hints can be textual shown in a separate area; hand-drawn in-place; or involving an embedded object (a link to another video). The “evaluation of 18 participants indicated that they learned more efficiently with the L.IVE integrated approach than with baseline, showing 20% score gain with the former”[10]. Even integrating interactivity limited to “stopping, replaying, reversing or changing speed to adapt the pace of the video demonstration” showed in [11] the decrease of time needed to acquire the necessary skills while considering possible risks of cognitive overloads. The authors conclude that “combining a procedural learning

Authoring Interactive Videos for e-Learning

135

task with a highly intuitive interactive interface constitutes an valuable application field for interactive dynamic visualisations, where learners may adapt the presentation to their individual needs at low cognitive costs”. In a recent study involving both teachers and students, Wijnker et al. [14], analysed the needs of teachers to understand how to select or create videos with the right characteristics, depending on their teaching objective. Among other results of their study, they found that quiz-based interactive videos attract more interest from students. These works seem to support our choice of using quiz-based interactive videos for online training on emergency management procedures and motivated us to design an empirical study with trainees for a project use case that will be executed as part of our future work in the ELEVATE project. Other related work relevant to ELEVATE concerns the lessons learned in using VR for educational purposes, especially from the perspective of the instructional designer, e.g. [6,12,13]. Some authors note that, to enrich their classes with videos (regardless of whether they are interactive) [14] and especially with VR [6], the instructional designers need additional film-making or VR design skills. To address this problem, in ELEVATE we define a collaborative tool-supported authoring process where the exercise designer and director play different roles.

5

Conclusion and Future Work

In this paper, we described the ELEVATE Tool Suite, which supports instructional designers to create quiz-based interactive videos for training on emergency management procedures by distance learning. We presented its architecture and the functionalities offered to its users, namely the exercise designer, the director, the exercise reviewer and, ultimately, the trainee (student). Among its novel features, we pointed out the possibility to exploit VR to generate video clips for dangerous scenarios, and the possibility for the exercise designer to analyse session logs for eliciting requirements for the improvement of an interactive video exercise. While VR is essential for creating video clips for exercises on dangerous scenarios, the potential complexity of its use has to be taken into account, as recognised also in other works (e.g. [6]). We addressed this challenge by defining a collaborative authoring process that involves different roles in a guided structured way. A detailed methodology for the creation of interactive video-based exercises using the ELEVATE Tool Suite is currently under definition. It will include, among others, recommendations on how to exploit the analysis of data from sessions’ logs to improve interactive videos. Among the future research activities worth to be mentioned, is the definition of personalisation mechanisms that can build on a dynamic student model, which can be updated upon a new training session. Moreover, we plan to evaluate the effectiveness of the ELEVATE interactive videos by executing an empirical study with trainees in a project use case.

136

D. Dellagiacoma et al.

Acknowledgements. This work is part of the ELEVATE research project, which is funded by Provincia Autonoma di Trento, L.P. 6/1999.

References 1. Balslev, T., De Grave, W.S., Muijtjens, A.M., Scherpbier, A.: Comparison of text and video cases in a postgraduate problem-based learning format. Med. Educ. 39(11), 1086–1092 (2005) 2. Busetta, P., Dragoni, M.: Composing cognitive agents from behavioural models in presto. In: Proceedings of the 16th Workshop “From Objects to Agents”, Naples, Italy, 17–19 June (2015). http://ceur-ws.org/Vol-1382/paper13.pdf 3. Cherrett, T., Wills, G., Price, J., Maynard, S., Dror, I.E.: Making training more cognitively effective: making videos interactive. Br. J. Educ. Technol. 40(6), 1124– 1134 (2009) 4. Dellagiacoma, D., Busetta, P., Gabbasov, A., Perini, A., Susi, A.: Authoring interactive-video exercises with ELEVATE: the NLS procedure case study. In: 10th International Conference in MIS4TEL, NURSING Workshop. Springer (2020) 5. Dragoni, M., Ghidini, C., Busetta, P., Fruet, M., Pedrotti, M.: Using ontologies for modeling virtual reality scenarios. In: Proceedings of the 2th European Semantic Web Conference, ESWC 2015, LNCS, vol. 9088, pp. 575–590. Springer, Cham (2015) 6. Ewais, A., De Troyer, O.: Authoring adaptive 3D virtual learning environments. Int. J. Virtual Pers. Learn. Environ. 5(1), 1–19 (2014) 7. Kay, R.H.: Exploring the use of video podcasts in education: a comprehensive review of the literature. Comput. Hum. Behav. 28(3), 820–831 (2012) 8. Kohen-Vacs, D., Milrad, M., Ronen, M., Jansen, M.: Evaluation of enhanced educational experiences using interactive videos and web technologies: pedagogical and architectural considerations. Smart Learn. Environ. 3(1), 1–19 (2016) 9. Kol˚ as, L.: Application of interactive videos in education. In: 2015 International Conference on Information Technology Based Higher Education and Training (ITHET), pp. 1–6. IEEE (2015) 10. Monserrat, T.J.K.P., Li, Y., Zhao, S., Cao, X.: L.IVE: an integrated interactive video-based learning environment. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3399–3402 (2014) 11. Schwan, S., Riempp, R.: The cognitive benefits of interactive videos: learning to tie nautical knots. Learn. Instr. 14(3), 293–305 (2004) 12. Vaughan, N., Gabrys, B., Dubey, V.N.: An overview of self-adaptive technologies within virtual reality training. Comput. Sci. Rev. 22, 65–87 (2016) 13. Vergara, D., Extremera, J., Rubio, M.P., D´ avila, L.P.: Meaningful learning through virtual reality learning environments: a case study in materials engineering. Appl. Sci. 9(21), 4625 (2019) 14. Wijnker, W., Bakker, A., van Gog, T., Drijvers, P.: Educational videos from a film theory perspective: relating teacher aims to video characteristics. Br. J. Educ. Technol. 50(6), 3175–3197 (2019) 15. Yousef, A.M.F., Chatti, M.A., Schroeder, U.: The state of video-based learning: a review and future perspectives. Int. J. Adv. Life Sci 6(3/4), 122–135 (2014) 16. Zhang, D., Zhou, L., Briggs, R.O., Nunamaker Jr., J.F.: Instructional video in e-learning: assessing the impact of interactive video on learning effectiveness. Inf. Manage. 43(1), 15–27 (2006)

Early Detection of Gender Differences in Reading and Writing from a Smartphone-Based Performance Support System for Teachers Roberto Araya(&) Center for Advanced Research in Education, Institute of Education, Universidad de Chile, Periodista José Carrasco Tapia Nº 75, Santiago, Chile [email protected]

Abstract. Illiteracy costs the world economy more than USD 1 trillion annually in direct costs. On the other hand, smartphones are now ubiquitous all over the world, and they are already having a huge impact on several areas of the economy. In this paper, we analyze the implementation of ConectaIdeas Express, a smartphone-based app that supports teachers to teach reading and writing to first graders. The app was adopted by 1,235 low SES schools, most of them with poor technological infrastructure and unreliable internet. All training was done through a weekly email and one-minute videos. We collected the assessments of 568,524 students’ answers. To the best of our knowledge, this is the largest dataset of first graders’ assessments ever reported. Doing learning analytics we found important gender differences in reading, writing and oral communication. The biggest gap was in writing, and particularly in students from rural schools, and also with male teachers. The biggest gender gap in writing was in two specific Learning Objectives of the curriculum. This information provides very accurate feedback to curriculum developers in order to redesign textbooks, assessments and materials, and to teachers and stakeholders to help adjust teaching practices. Keywords: Learning Analytics support


Curriculum design
Smartphone-based

1 Introduction Learning to read is a vital component of the human right to education [1], and reading is now critical in the workplace. Improving literacy skills is a key step to overcome poverty. Illiterate people earn 30%–42% less than their literate counterparts, and illiteracy costs the global economy more than USD 1 trillion annually in direct costs alone, and close to a hundred billion in Latin America [2]. However there are still several challenges in teaching to read and write. There are two different pedagogic strategies that for decades have produced the ‘Reading Wars’. According to [3], research in psychological science has provided the evidence of the effectiveness of one strategy over the other, but this research has been slow to make © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 137–146, 2020. https://doi.org/10.1007/978-3-030-52538-5_15

138

R. Araya

inroads into educational policy and practice. For example, in 2017 the U.K. Department for Education [4], estimated that the proportion of pupils in U.K. achieving the expected standards in writing is just 68%. One of the challenges with reading and writing is the learning gap between girls and boys. On one hand, there is the Hyde’s Gender Similarities Hypothesis [5], which argues that most gender differences are small or trivial. For example, [6] from an opportunity sample of 116 children (52 male) ranging in age from 5:0 to 6:7 years recruited from six UK schools, found that no associations between phonological skills and writing were moderated by gender. However, other studies report a completely different picture. [7] found a developmental progression from initially small gender differences in Grade 4 toward larger effects as students progress through schooling. In an interview [8], one of the authors commented that the “common thinking is that boys and girls in grade school start with the same cognitive ability, but this research suggests otherwise”, and that “our research found that girls generally exhibit better reading and writing ability than boys as early as the fourth grade.” “It appears that the gender gap for writing tasks has been greatly underestimated, and that despite our best efforts with changes in teaching methods does not appear to be reducing over time.” In reading tests at school, girls tend to be ahead of boys, in all age groups and in all countries. However, in young adults, [9] there is no longer any gender difference, although women tend to outperform men at ‘prose literacy’ (continuous texts) while men tend to do better than women at ‘document literacy’ (discontinuous texts). [9] explore four possible explanations that have been suggested: cognitive differences between the genders, the feminization of school, the teaching methods used and gender differences in reading frequency and reading motivation. However, after analyzing the results of Nordic counties on the international tests PIRLS 2011, PISA 2009, PIAAC 2012, they conclude that the magnitude of the observed gender differences appears to be associated with certain assessment features including text type, item format, aspects of reading and implementation. For example, PIRLS has 50% of fiction text and PISA has 15%, while PIACC has 0%. PISA measures reading on a mix of fiction and nonfiction texts and formats that makes PISA a girl-friendly reading assessment, while PIAAC use shorter non-fiction texts. Thus monitoring teaching materials, practices, and types of questions in test are critical to improve students’ learning and reduce gender gaps. One option to improve teaching is the use of technology. However the use of computers in schools is still not widespread. For example, [10] reports that in UK only 2 in 5 (39.3%) teachers said they had access to desktop computers, and far fewer to tablets such as Amazon Fire (14.6%), Chromebooks (11.0%) and ereaders (6.4%). This means that strategies based on computers have still big infrastructure barriers to overcome. Another option is smartphones. They are already making having a huge impact on several areas of the economy. The mobile phone technology not only has created a positive impact on several areas but is already contributing in poverty reduction [11]. Additionally, smartphones are now ubiquitous all over the world. A recent Gallup World Poll [12] shows that 83% of adults in developing economies have a mobile phone as of 2018. Farmers use mobile phones to research crop prices and marketing opportunities. For example, the arrival of mobile coverage has been linked to higher

Early Detection of Gender Differences in Reading and Writing

139

consumption and reduced poverty in rural Peru. A recent 2019 World Bank Report [13], shows that people are increasingly using smartphones, tablets, and other portable electronic devices to work, organize their finances, secure and heat their homes, and have fun. Therefore, a natural option is to use smartphones in schools. However, people have serious concerns about the impact of smartphones on children [14]. According to a 2019 BBC press note [15], smartphones have become an integral part of our daily lives, but people think they should be left outside the classroom. For example, the minister for school standards in England, told the BBC he believes schools should ban their pupils from bringing in smartphones. A third option is a performance support system [16] for teachers and educational administrators included on their smartphones. Almost all of them use smartphones every day. Moreover, [14] in advanced or emerging economies, men and women generally use technology – including smartphones, the internet and social media – at similar rates. In this paper we analyze an experience where smartphones were used by teachers and other adults, but not students. They were used to assess students with weekly formative items, to observe teacher practices, to monitor students’ progress in the curriculum, and to engage parents’ participation and support from home. We are interested in providing early feedback to teachers on the impact of their strategies, and to administrators and curriculum and material developers on the impact of their designs. In this work we focus in early detection of gender differences in learning, detecting where they are generated and identifying the particular components of the curriculum that produce the biggest gaps.

2 Methods During the second semester of 2019, a total of 1,235 low Socioeconomic Status (SES) Chilean schools voluntarily adopted ConectaIdeas Express, a smartphone-based performance support system in order to support teaching to read to first graders [17]. The adoption was very rapid. At the end of the semester, the app was being used by almost 50% of the schools that were effectively using the textbook to teach reading offered by the Ministry of Education. ConectaIdeas Express app was implemented to support that textbook. Out of this total, 1,022 first grade classes, with 988 teachers used the app to assess 30,158 students. Teachers used exit tickets in order to assess students. These one-question tickets correspond to formative assessments that allow teachers to quickly know how well their students understand the material they are learning. For this reason, towards at the end of the session, students answered the exit ticket orally or by writing in a paper. Later, each teacher inputted the information on their smartphone. ConectaIdeas Express displays the list of students and the possible tickets. The tickets were designed by the Ministry of Education and shown in the textbook. Each ticket is associated to a specific Learning Objective (LO) of the national curriculum. Teachers can also create their own ticket, different to those proposed on the textbook. In that case they have to associate the Learning Objective to their ticket. However, almost all the tickets used were the official ones that came in the textbook. Once teachers have access to Internet they can upload the information to the cloud. ConectaIdeas Express has also

140

R. Araya

other options for classroom observation and sharing videos of pedagogic strategies with peers and parents, but we won’t analyze that information here. Most of the students were from urban schools, 554 classes with 27,911 students were from urban schools, and 138 classes with 2,525 students were from rural schools. At the end of the year, 568,524 students’ answers were evaluated and uploaded by the teachers, 293,180 from boys and 275,344 from girls. To the best of our knowledge this is the first time ever so many first graders answers have been digitally registered and particularly with an association to students’ gender as well as to curriculum components in order to make a gender difference study. Teachers that used the app took an online survey, a total of 86 teachers answered, 90.5% of them found it easy to use. Moreover, 70% of them agreed that it takes them less than 5 min to assess all of their students (30 students per class, on average). These assessments were provided during the last minutes of the session. The teacher, principal and district administrators have access to different reports and to a geographic map with the schools and to an associated tree showing the class coverage and performance on the three strands and on all the 26 learning objectives of the curriculum. Most of the schools that adopted the app serve a population of low SES students from all over the country. They represent 15% of the students of the country at that grade level. 922 of the teachers were female teachers and 66 were male teachers. Most of the schools have poor technological infrastructure and unreliable internet, particularly the rural schools. All training was done during the second semester by a weekly email with tips on how to install and use the app, and with links to one-minute videos and to a geographic map with the state of the curriculum coverage of each school. On average, each teacher used the app for 22 sessions. This means that it was used approximately twice a week.

3 Results Out of the 568,524 students’ answers, each one assessed by the class teacher, 82.2% were correct. There was a slight difference between the 434 classes from public schools and the 589 classes from charter schools. 81.5% of answers of students from public schools were correct, and 82.6% of answers of student from charter schools were correct. 61.5% of the answers correspond to reading questions, 22.6% to writing questions, and 15.9% to oral communication questions. 83.3% of the answers were correct in the reading exit tickets, 83.2% in the oral communications exit tickets, but only 78.9% were correct in the writing exit tickets. This result with lower performance in writing agrees with other studies [9]. Students’ answers assessed by male teachers obtained 79.1% correct answers whereas students’ answers assessed by female teacher were 82.5% correct. These differences are statistically significant. Learning Objective LO-13 “Experiment with writing to communicate facts, ideas and feelings”, obtained significantly lower assessment than the rest of the seven more frequently assessed Learning Objectives. The results confirm a gender gap, but now in first grade. As in other studies but with older students, girls perform better than boys. 84.3% of the girls’ answers were correct, but only 80.5% of the boys. This is a gap of 3.8% points or equivalently a

Early Detection of Gender Differences in Reading and Writing

141

Cohen’s d of 0.10 standard deviations. This gap is much lower than the one in 4th grade for Nordic countries with the PIRLS test [9], but it coincides well with the result of performing a linear extrapolation to first grade with the data of the gender gap for fourth grade on the PIRLS test and the gender gap for 9th to 10th grade on the PISA test. What is new is that these results correspond to a great scale study with first graders. Assessments in reading, and particularly in writing and oral communication, with students of first grade, are very difficult to obtain in such large scale. We found that the gender gap occurs in all three strands: reading, writing and oral communication, as shown in Fig. 1. Additionally, we found that the biggest difference is in writing with 81.6% of correct answers in girls but 76.5% of correct answers in boys. This is a gap of 5.1% points, or equivalently a Cohen’s d of 0.13 standard deviations.

Fig. 1. Percentage of correct answers of boys and girls in the three strands, reading, writing and oral comprehension.

There are 26 different Learning Objectives (LO) in the Chilean curriculum. 12 LO are part of the reading strand, 4 belong to the writing strand, and 10 belong to the oral communication strand. The seven LO with greater number of answers by the students are shown in Table 1. Three of them belong to reading (LO-3, LO-8, and LO-6), 3 to writing (LO-13, LO-14, and LO-16) and one belongs to oral communication (LO-18). The number of answers that corresponds to these seven LO is 381,548 answers. They represent 67.1% of all answers. As shown in Fig. 2, of these seven LO, the biggest gender gap was in LO-13 “experiment with writing to communicate facts, ideas and feelings” where girls obtained 5.5% points of correct answers more than boys, this means a Cohen’s d of 0.14 standard deviations, and in LO-14 “write complete sentences to convey messages’’, where girls obtained 5.1 more percentage points of correct answers than boys, which corresponds to a Cohen’s d of 0.13 standard deviations. This gap is higher in rural schools, as shown in Fig. 3. Girls obtained 85.3% correct answers whereas boys only 75.6% of correct answers. This is 9.7% points of difference, or equivalently a Cohen’s d of 0.26 standard deviations. The gap is also high in the

142

R. Araya

Table 1. The seven Learning Objectives (LO) of the curriculum with more students’ answers. LO code LO-3 LO-8 LO-18 LO-16 LO-14 LO-6 LO-13

LO description No of answers Identify the sounds that make up the words 167,928 Demonstrate understanding of stories with familiar topics 61,480 Understand oral texts to get info and develop curiosity 51,513 Include new vocabulary from heard/read texts into writing 47,779 Write complete sentences to convey messages 38,827 Understand texts applying reading comprehension strategies 37,897 Experiment with writing to communicate facts, ideas, feelings 37,604

Fig. 2. Percentage of correct answers on the 7 LO with most answers, ordered from the LO with most answers at the extreme left to the LO with the 7th most answers at the extreme right.

lowest SES schools. In that case the gender gap is 5.7% points, which corresponds to a Cohen’s d of 0.15 standard deviations. We also found that there is a higher gender gap when the teacher is male. The gap is higher in writing. As shown in Fig. 4, with male teachers, boys obtain 8.7% points less than girls. This is a Cohen’s d of 0.23 standard deviations. As shown in Fig. 5 the biggest gap is in LO-13 “experiment with writing to communicate facts, ideas and feelings”. Boys obtain 12.1% points less than girls. This is a Cohen’s d of 0.32 standard deviations, whereas, when the teacher is female the gap is only 5.3% points. This result is opposed to a possible interpretation of the feminization of the school hypothesis, where the presence of female teachers could make boys less interested in language than girls. Here, on the contrary, we found that when the teacher is male the gap is greater. Towards the end of the semester, the analysis of the accumulated information of student responses allowed us to detect that there was a very important Learning Objective without student responses. It was the LO dedicated to reading comprehension: LO-10 “read independently and understand simple non-literary texts”. There was

Early Detection of Gender Differences in Reading and Writing

143

Fig. 3. For urban and rural, boys and girls, percentage of correct answers on the 7 LO with most answers, ordered from left to right according to number of answers.

Fig. 4. For girls and boys, and female and male teachers, percentage of correct answers on the three strands of the curriculum.

another reading comprehension LO, the OA-8 “demonstrating understanding of stories with familiar topics.” However, it is only LO-10 that ensures that reading is independently and autonomously carried out by the student, and that they are also topics not known beforehand, avoiding confusing understanding with prior knowledge. That is, the analysis allowed detecting a design problem, since none of the 120 official exit tickets were associated with LO-10. As a result of this fact detected, and given the importance of reading comprehension and having a diagnosis of the level of school readiness achieved in schools, the Ministry of Education designed 4 exit tickets for OA-10, and proposed to the schools that they take it voluntarily to determine the level or reading comprehension reached in their classes. The proposed day was November 28, but the schools could adjust the date to their convenience. The proposed tickets

144

R. Araya

Fig. 5. For girls and boys, and female and male teachers, percentage of correct answers on the 7 LO with most answers, ordered from left to right according to number of answers.

required reading instructions to color some images of a story that the teacher initially read. While the tickets required the student to read a short text with instructions, it also required oral comprehension of a story the teacher read. From November 26 until the end of the semester in mid-December, 19,064 responses were received. The girls obtained a better result. However, the gap was 3.4% points, or equivalently a Cohen’s d of 0.09 standard deviations. The gap on this test was lower than the gender gap in reading that is 4.3% points, or 0.11 standard deviations. This smaller gap may be due to the form of the exit ticket question, which contained short texts, not fiction, and with only instructions for painting.

4 Conclusion and Discussion During the second semester of 2019, 1,235 low SES schools voluntarily adopted a smartphone-based teacher support system that brought a unique opportunity to perform Learning Analytics to study a critical gender gap in literacy education. This is a well know gap extensively studied from 4th grade to adulthood. However, difficulties to gather data from first graders have restricted the study of the phenomenon on that grade. To the best of our knowledge, this is the first time that such a number of evaluations of first grade students are reported. This is very valuable because there is no clear agreement on the existence and dimension of the gap for that grade level. Teachers assessed and registered 568,524 answers from 30,158 students belonging to 1,022 courses, and then uploaded them to the cloud using the ConectaIdeas Express app. All the questions were designed by the Ministry of Education and have an associated Learning Objective described on the National curriculum. This is a set of 120 questions, called exit tickets, whose purpose was to apply one of them at the end of a session as a quick feedback to the teacher. On average, each teacher selected 22 questions from this set throughout the semester. We have to consider that this is not a

Early Detection of Gender Differences in Reading and Writing

145

standardized test, since different classes probably answered a different sample of the 120 questions and different number of questions. From this data we have found that in first grade there is a gender gap, but it is smaller than the one know in fourth obtained from tests as the PIRLS test and in ninth grade in the PISA test. Moreover this gap is present in the three strands of the language subject; reading, writing and oral communication. We also found that this gap is bigger in students from the lowest SES and it is particularly bigger in rural schools. Additionally, somehow surprisingly, the gender gap is higher with male teachers than with female teachers. The biggest gender gap in writing was in two specific Learning Objectives of the curriculum. On one critical Learning Objective of the writing strand with male teacher, the gap, in terms of Cohen’s d, was 0.32 standards deviations. The analysis makes it possible to suggest redesigns of curricula, teaching sequences, class schedules, instructional material and ways to ask evaluation questions. During the semester the analysis helped to detect a design problem with the exit tickets and a special test was prepared. According to U.K. former Prime Minister Gordon Brown [18], in 2011 we were already less than five years from the target date and there were still 67 million primary school age children and not in school, “millions more were sitting in classrooms receiving an education of such abysmal quality that it will do little to enhance their life chances.” In a 2020 report [19] authors conclude, “reading achievement isn’t improving. Too many students—particularly students who are living in poverty or are of color—enter grade 3 unable to read or unable to read as well as they should.” One of the critical problems is the gender gap. Contrary to the Hyde’s Gender Similarities Hypothesis we found a gender gap in first grade. We found that the gap is bigger in writing. Similarly to what is found in higher grades. We have also found no bases for the feminization hypothesis, since the gap is bigger with male teachers than with female teachers. The analysis seems to back up the hypothesis on the type of questions. In a final test with a slightly different type of question the gap was reduced a little. A major reduction may require adjusting textbooks, materials and teaching practices. For example, [20] suggests that courses need “to be reconsidered and re-structured genderspecifically”. This work illustrates how Learning Analytics can help discover critical educational patterns that are essential to redesign the curriculum, instructional material and teaching practices in order to help augment students’ learning outcomes. Acknowledgements. Support from ANID/ PIA/ Basal Funds for Centers of Excellence FB0003 is gratefully acknowledged.

References 1. Sánchez, G., Frandell, T.: Literacy from a right to education perspective. UNESCO (2013) 2. World Literacy Foundation. The economic and social cost of illiteracy: A snapshot of illiteracy in a global context (2015). https://worldliteracyfoundation.org/wp-content/uploads/ 2015/02/WLF-FINAL-ECONOMIC-REPORT.pdf 3. Castles, A., Rastle, K., Nation, K.: Ending the reading wars: reading acquisition from novice to expert. Psychol. Sci. Public Interest 19(1), 5–51 (2018)

146

R. Araya

4. U.K. Department for Education. Statistics on national curriculum assessments at key stage 1 and phonics screening check results (2017) 5. Hyde, J.: The gender similarities hypothesis. Am. Psychol. 60(6), 581–592 (2005) 6. Adams, M., Simmons, F.: Exploring individual and gender differences in early writing performance. Read. Writ. 32, 235–263 (2019) 7. Reilly, D., Neumann, D.L., Andrews, G.: Gender differences in reading and writing achievement: evidence from the national assessment of educational progress (NAEP). Am. Psychol. 74(4), 445–458 (2019) 8. Sciencedaily, American girls read and write better than boys. https://www.sciencedaily.com/ releases/2018/09/180920102135.htm. Accessed 04 Feb 2020 9. Solheim, O., Lundetræ, K.: Can test construction account for varying gender differences in international reading achievement tests of children, adolescents and young adults? – A study based on Nordic results in PIRLS, PISA and PIAAC. Assess. Educ. Principles Policy Pract. 25(1), 107–126 (2018) 10. Picton, I.: Teachers’ use of technology to support literacy in 2018. In: The National Literacy Trust (2019) 11. Londhea, B., Radhakrishnanb, S., Divekar, R.: Socio economic impact of mobile phones on the bottom of pyramid population- a pilot study. Procedia Econ. Finan. 11, 620–625 (2014) 12. Klapper, L.: Mobile phones are key to economic development. Are women missing out? Brookings https://www.brookings.edu/blog/future-development/2019/04/10/mobile-phonesare-key-to-economic-development-are-women-missing-out/. Accessed 04 Feb 2020 13. World bank report. The changing nature of work. http://documents.worldbank.org/curated/ en/816281518818814423/pdf/2019-WDR-Report.pdf. Accessed 04 Feb 2020 14. Silver, L.: Smartphone Ownership Is Growing Rapidly Around the World, but Not Always Equally. Pew Research Center (2019) 15. BBC News Homepage, Smartphones in school: Ban, restrict or allow? https://www.bbc.com/ news/education-47101875. Accessed 04 Feb 2020 16. Araya, R.: Teacher training, mentoring or performance support systems? In: Nazir, S., Teperi, A.M., Polak-Sopińska, A. (eds.) Advances in Human Factors in Training, Education, and Learning Sciences. AHFE 2018. Advances in Intelligent Systems and Computing, vol. 785, pp. 306–315. Springer, Cham (2018) 17. Araya, R.: Mobile Performance Support System for Teachers and Parents Teaching First Graders to Read. (submitted) 18. Brown, G.: Education for All: Beating Poverty, Unlocking Prosperity. UNESCO (2011) 19. Adams, M., Fillmore, L., Goldenberg, C., Oakhill, J., Paige, D., Rasinski, T., Shanahan, T.: Comparing Reading Research to Program Design: An Examination of Teachers College Units of Study (2020) 20. Boettger, H.: Early gender diversity in reading and writing: research and didactical consequences. Training Lang. Cult. 1(2), 42–57 (2017)

An Assessment of Students’ Satisfaction in Higher Education Margarida Figueiredo1 , Ana Fernandes2 , Jorge Ribeiro3 , José Neves4,5(&) , Almeida Dias5 , and Henrique Vicente4,6 1

3

Departamento de Química, Escola de Ciências e Tecnologia, Centro de Investigação em Educação e Psicologia, Universidade de Évora, Évora, Portugal [email protected] 2 Departamento de Química, Escola de Ciências e Tecnologia, Universidade de Évora, Évora, Portugal [email protected] Instituto Politécnico de Viana do Castelo, Rua da Escola Industrial e Comercial de Nun’Álvares, 4900-347 Viana Do Castelo, Portugal [email protected] 4 Centro Algoritmi, Universidade do Minho, Braga, Portugal [email protected] 5 Instituto Politécnico de Saúde do Norte, CESPU, Gandra, Portugal [email protected] 6 Departamento de Química, Escola de Ciências e Tecnologia, REQUIMTE/LAQV, Universidade de Évora, Évora, Portugal [email protected]

Abstract. Student’s Satisfaction (SS) with a particular subject may impact the learning process, being the figure of attentiveness of the utmost importance over time, and also a very difficult undertaking to accomplish. To go forward with such exercise, a workable methodology for problem solving had to be built and tested. It is based on a thermodynamic approach to Knowledge Representation and Reasoning, which is the ultimate goal of SS assessment when working on a particular topic. Keywords: Students’ Satisfaction
Thermodynamics
Entropy
Higher Education
General Chemistry
Knowledge Representation and Reasoning Logic Programming
Artificial Neural Networks




1 Introduction On the one hand, Higher Education institutions serve and enrich the countries in many ways, and their role in modern societies is diverse. In a century in which technologybased industries are gaining in importance, higher education institutions offer many and varied courses in science and technology. In addition, such institutions are strongly committed to ensuring the quality of education through the introduction of assessment and accreditation mechanisms. On the other hand, in courses of Sciences, Technology and Engineering, a subject in General Chemistry must be taken in the first year of © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 147–161, 2020. https://doi.org/10.1007/978-3-030-52538-5_16

148

M. Figueiredo et al.

study. This subject is intended to impart the knowledge required for later studies and to train future specialists with sufficient knowledge. However, General Chemistry is viewed by students as a difficult subject, which affects the role they should play in student education [1, 2]. In order to promote the quality of education, continuous efforts are required to modernize the didactic processes and to organize pedagogy and curricula, as these factors determine the quality of learning and the satisfaction of the students. Indeed, student satisfaction is a complex phenomenon that considers various factors, some of which depend on the subject, others on the teachers and even on the institutions [3, 4]. Therefore, their rating is important as it can help improve the quality of teaching. The paper goal is the evaluation of the SS, being its content expressed as follows, viz. After the introduction, the basics used in this work are discussed, namely the notion of entropy, the use of logic programming to represent and think of knowledge [5, 6] and a computer approach based on artificial neural networks [7, 8]. Section 3 presents the new problem-solving method, while Sect. 4 shows how data is processed and how it relates to concept of entropy [9]. Finally, conclusions are drawn and future work outlined.

2 Fundamentals 2.1

A Thermodynamic Approach to Knowledge Representation and Reasoning

The problem-solving methodology presented in this article is based on Thermodynamics and aims to describe the practices of Knowledge Representation and Reasoning (KRR) as a process of energy degradation [6, 9, 10]. In order to explain the basic rules of the proposed approach, the First and Second Law of Thermodynamics are considered, attending that one’s system moves from state to state over time. The former one, also known as the Energy Saving Law, states that the total energy of an isolated system is constant, i.e., cannot change over time. This means that energy can be converted, but cannot be generated or destroyed. The latter deals with Entropy, a property that quantifies the orderly state of a system and its evolution. These characteristics fit the proposed vision of KRR practices, as this has to be understood as a process of energy degradation. Indeed, it is believed that a data element is in an entropic state, the energy of which can be decomposed and used in sense of degradation, but never used in the sense of destruction, viz. • exergy, sometimes called available energy or more precisely available work, is that part of the energy that can be arbitrarily used by a system after a transfer operation, or in other words, giving a measure of its entropy. In Fig. 2 (Sect. 4.1) it is given by the gray colored areas; • vagueness, i.e., the corresponding energy values that may or may not have been consumed. In Fig. 2 are given by the gray colored areas with spheres; and • anergy, that stands for an energetic potential that was not yet consumed, being therefore available, i.e., all of energy that is not exergy. In Fig. 2 it is given by the white colored areas [6, 9, 10].

An Assessment of Students’ Satisfaction in Higher Education

149

On the other hand, there are many approaches to KRR using the epitome of Logic Programming (LP), namely in the areas of Model Theory and Proof Theory. In this article, the Proof Theoretical methodology for problem solving was adopted and expressed as an extension of the LP language [5]. Under this setting a LP will be grounded on a finite set of clauses in the form, viz.

Program 1. The Archetype of a Logic Program The first clause denotes predicate’s closure, “,” designates “logical and”, while “?” is a domain atom denoting “falsity”, the pi, qj, and p are classical ground literals, i.e., either positive atoms or atoms preceded by the classical negation sign ¬ [5]. Indeed, ¬ stands for a strong declaration that speaks for itself, while not denotes negation-by-failure, i.e., a failure in proving a certain statement since it was not declared in an explicit way. According to this way of thinking, a set of abducibles are present in every program [11]. In this work are given in the form of exceptions to the extensions of the predicates that make the program, i.e., clauses of the form, viz. exceptionp1 ;


; exceptionpj ð0  j  kÞ;

being k an integer number

that denote data, information or knowledge that cannot be ruled out. On the other hand, clauses of the type, viz. ?ðp1 ;


; pn ; not q1 ;


; not qm Þðn; m  0Þ are invariants that make it possible to specify the context under which the universe of discourse should be understood [6, 10]. 2.2

Artificial Neural Networks

Artificial Neural Networks (ANNs) are computing tools inspired by studies of the human brain and nervous system. In fact, they are mathematical models that simulate such systems as are understood today. ANNs were first introduced in 1943 [12], and significant developments occurred until 1969 (e.g., the emergence of single-layer perceptron [13]). However, some limitations (e.g., the fact that a single-layer perceptron cannot solve the

150

M. Figueiredo et al.

XOR problem) led to a decline in interest in ANNs between 1969 and 1986. The abovementioned disadvantages were remedied in the 1980s and research on ANNs again increased the emergence of the back-propagation algorithm in 1986 [14] and the posterior stimulation through the development of numerous fast gradient-based variants (e.g., RPROP) [15]. One of the main characteristics of ANNs is their ability to learn. In fact, it is important to note that ANNs are not traditional computer programs. You learn from examples through a process called training, in which ANNs organize themselves to adjust an internal set of parameters (i.e. weights) that are used to collect the information contained in the data [16]. Compared to traditional methods, ANNs treat inaccurate and/or incomplete data, give approximate results, and are less prone to outliers. In addition, it is not necessary to accept restrictions or to know the relationships between variables from the outset. Multi-Layer Perceptron (MLP) is one of the most widespread ANN architectures in which neurons are layered and only forward connections exist [16]. The MLP design is typically trial and error using an upward approach, starting with an initial architecture that is adjusted to minimize the internal error (e.g., Mean Square Error) [7, 16].

3 Methods This study was carried out at a university in southern Portugal. A total of 198 students participated in this study. The ages of the participants ranged from 18 to 32 years (average age 19.7 years), with 55% women and 45% men. The questionnaire consists of three sections, the first of which contains general questions (e.g. age, gender, course, place of residence), while the second contains information on General Chemistry, teachers and infrastructures. Finally, in the third section, students are asked to express their degree of satisfaction with this subject. The WEKA software was used to implement ANNs as stated above while maintaining the standard software parameters [17]. To ensure statistical significance of the results, 25 tests were carried out in all situations. In each simulation, the database was randomly divided into two mutually exclusive partitions, i.e., the training and test sets.

4 Case Study 4.1

A Thermodynamic Approach to Data Acquisition and Processing

In order to collect information about students’ satisfaction regarding General Chemistry the participants were asked to tick the option(s) that reflects their opinions regarding each statement. If the respondent tick more than one option, he/she is also asked to indicate the evolution trend of his/her answer, i.e., increasing trend (Strongly Disagree ! Strongly Agree) or the opposite (Strongly Agree ! Strongly Disagree) as shown in Fig. 1. The answer options were confined to the following scale, viz. ð4Þ Strongly Agree; ð3Þ Agree; ð2Þ Disagree; ð1Þ Strongly Disagree

An Assessment of Students’ Satisfaction in Higher Education

151

Fig. 1. A fragment of the answers of student #1 to the second part of the questionnaire

The statements under consideration were organized into three questionnaires, namely (Subject Related Statements – Five Items (SRS – 5), Teacher Related Statements – Four Items (TRS – 4), and Infrastructures Related Statements – Three Items (IRS – 3). The former one comprises the statements, viz. S1 – The objectives of the subject are clearly defined; S2 – The syllabus is adequate to my previous knowledge; S3 – The syllabus is important for my future professional practice; S4 – The teaching methodologies are in line with the objectives of the themes being covered; and S5 – The assessment methodologies are adequate. The second questionnaire encompasses the statements, viz. S6 – The taught; S7 – The S8 – The S9 – The

teaching team reveals capability and knowledge about the subjects teaching team explains the subjects with clearness and thoroughness; teaching team is available to answer questions; and teaching team reveals ability to stimulate students.

Finally, the third one includes the statements, viz. S10 – The rooms for the theoretical classes have the necessary conditions; S11 – The spaces for practical classes possess the materials and equipment appropriated; and S12 – The recommended bibliography and information sources are available and easily accessible.

152

M. Figueiredo et al.

In order to quantify the qualitative information and make the process intelligible, complete calculation details for Subject Related Statements - Five Items (SRS – 5) are provided. Table 1 shows the results regarding student #1 answers to the SRS – 5 questionnaire. For example, the answer to S4 was strongly agree (4) ! agree (3), i.e., student #1 indicated that he/she strongly agree (4) with statement S4, but does not reject the possibility that the answer could be agree (3) in certain situations. It shows a trend in the development of the his/her degree of satisfaction with a variation in entropy, i.e., in this case there is a deterioration in the student’s satisfaction once the entropy increased. Otherwise, the answer of student #1 to S1, disagree (2) ! agree (3), shows a trend in the student’s satisfaction with a decrease in entropy, i.e., there is an increase of the student’s satisfaction. For S2 and S5 the answers were strongly disagree (1) and agree (3), respectively, a fact that speaks for itself, while for S3 no options were indicated that indicate a vague situation, i.e., the value of the energy consumed is unknown, although it is known that it is in the bandwidth is the interval 0… 1. Figures 2 and 3 describe such responses regarding the different forms of energy, i.e., exergy, vagueness and anergy. Bearing in mind the fact that the markings on the axis correspond to one of the possible scaling options, the student’s satisfaction behaves better when the entropy decreases, which is the case with S1, as shown in Table 2 for the Best/Worst Case Scenarios (BCS/WCS). Table 1. Student #1 answers to SRS – 5 questionnaire.

Statements S1 S2 S3 S4 S5

decreasing trend (4) (3) (2) (1)

Scale increasing trend (2) (3) (4) × ×

× ×

×

× ×

Leading to

Fig. 2

vagueness

Leading to

An Assessment of Students’ Satisfaction in Higher Education 1

(1)

π

(1) (2)

(2)

S5

(3)

(3)

1

1 π

(1)

(1) (2)

(2)

S1

(1)

S1

(3)

(3)

(4)

(4)

S2

O O

O

(3)

O

(4) O

S4

S2

S5

S3

(2)

O

S3

(1)

(3)

(3)

O O

O

(2)

S3

(4)

S4

π

(1)

(4)

S2

S5

S5

(2)

(2)

(3)

(3)

(4)

(4)

S4

O O O O O O (4) O

O

S4

Leading to

(1)

(2)

O

O

O

S1

S4

S3

O

1

π

(1)

O

(3)

O O

(4)

(4)

(1)

O O

(2)

S2

O

O O O O O

1

π

153

S5

S3

S1

S1

S2

Leading to

Fig. 3

Fig. 2. Estimate energy consumption for each statement sentence in relation to student #1’s answers to the SRS – 5 questionnaire. The dark, gray and dashed areas stand for exergy, vagueness and anergy

1

(1)

π

(2)

S5

1

(1)

(1)

(2)

(2)

(3)

(3)

S1

1

π

S5

(1)

(2)

(2)

(2)

(3)

(3)

(4)

(4)

(3)

(3)

π

(1)

S1

S5

(1)

S1

O

(4)

(4)

O O O O O

(4) O

S4

S2

S4

O O O O O

(4)

S2

S4

S2

O

S3

S3

Leading to

Table 2

S3

Leading to

Fig. 3. A global view of the energy consumed by student #1 when answering to SRS – 5 questionnaire. The dark, gray and dashed areas stand for exergy, vagueness and anergy, respectively.

154

M. Figueiredo et al.

Table 2. Evaluation of the student’s #1 Best and Worst case scenarios or entropic when answering the SRS – 5 questionnaire.

An Assessment of Students’ Satisfaction in Higher Education

155

The data collected above may now be structured in terms of the extent of predicate subject related statements (srs – 5) in the form, viz. srs  5 : EXergy; VAgueness; StudentsSatisfactionAssessment; Qualityof Information ! fTrue; Falseg a construct that speaks for itself, whose extent and formal description follows (Table 3 and Program 2). Table 3. The extent of the srs – 5’s predicate from student #1 answers to SRS – 5 questionnaire. Questionnaire SRS – 5

Ex BCS 0.31

VA BCS 0.29

SSA BCS 0.80

QoI BCS 0.40

EX WCS 0.61

VA WCS 0

SSA WCS 0.79

QoI WCS 0.39

Program 2. The extent of the srs – 5 predicate for the Best case scenario. The evaluation of Students Satisfaction Assessment (SSA) and Quality of Information (QoI) for the different items that make the srs – 5 questionnaire are now given in the form, viz. pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi • SSA is figured out using SSA ¼ 1  ES2 (Fig. 4), where ES stands for the exergy’s that may have been consumed in the Best case scenario (i.e., ES ¼ exergy þ vagueness), a value that ranges in the interval 0…1. SSA ¼

qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi 1  ð0:31 þ 0:29Þ2 ¼ 0:80

156

M. Figueiredo et al.

• QoI is evaluated in the form, viz. QoI ¼ 1  ðexergy þ vaguenessÞ=Interval lengthð¼ 1Þ ¼ 1  ð0:31 þ 0:29Þ ¼ 0:40

1 SSA

? 0

ES

1

SSA evaluation.

Fig. 4.

Table 4. Answers of student #1 to the TRS – 4 and IRS – 3 questionnaires

Questionnaires Statements

TRS – 4

IRS – 3

S6 S7 S8 S9 S10 S11 S12 Leading to

Scale increasing trend

decreasing trend (4)

(3)

×

×

(2)

(1)

(2)

(3) ×

(4)

× ×

× × × Table 5

Leading to

vagueness

An Assessment of Students’ Satisfaction in Higher Education

157

To complement Table 1, Table 4 and Table 5 are related to the student’ #1 answers to TRS – 4 and IRS – 3 questionnaires. Table 5. The srs – 5, trs – 4 and irs – 3 predicates’ scopes obtained according to the student #1 answers to the SRS – 5, TRS – 4 and IRS – 3 questionnaires. Questionnaires SRS – 5 TRS – 4 IRS – 3

4.2

Exergy BCS 0.31 0.13 0.29

Vague BCS 0.29 0.28 0.02

SSA BCS 0.80 0.91 0.95

QoI BCS 0.40 0.59 0.69

Exergy WCS 0.61 0.43 0.35

Vague WCS 0 0 0

SSA WCS 0.79 0.90 0.94

QoI WCS 0.39 0.57 0.65

Evaluation of Students’ Satisfaction – a Computational Logic Approach

The Formal Framework. Computational Logic is the use of computers to establish facts in a logical formalism. The topic dates from the 19th century and tries to understand the nature of mathematical thinking. In this subsection a mathematicallogical program is presented, with the help of which one’s perception of the individuals or assemblies with respect to a particular subject is evaluated and how the organization as a whole is assessed, i.e., it measures the impact of the individuals’ psychosocial threats on the organization through logical reasoning (Program #3, further down). Indeed, contextual programming is not new, but an unusual field that can help find or work out data. In addition, as often the Artificial Intelligence and Machine Learning is currently widely practiced, there is no way to ask the computer why this was the case. It is known how to teach an algorithm to identify a horse in a photo, but you cannot ask why it is a horse. This was the main reason why the system of information or knowledge representation and reasoning was changed. The focus is not on knowing the absolute value of a variable, but on quantifying the associated evolutionary process that has resulted in a certain variable taking on a certain value. Indeed, this framework provides the basis for training an ANN to assess the levels of trust (LoT)) and sustainability (LoS) of Program #3 to Student Satisfaction Assessment (SSA).

158

M. Figueiredo et al.

Program 3. The make-up of the LP or Knowledge Base to establish Student # 1 Satisfaction in the Best case scenario. where ¬ denotes strong negation and not negation-by-failure. Artificial Neural Network Training and Testing Procedures. It is now possible to consider the data sets to train and test an ANN [7, 8] (Fig. 5), viz.

An Assessment of Students’ Satisfaction in Higher Education

159

• The input in the form of the nursing students’ degrees of Adaptation (extent of predicate srs – 5), Anxiety (extent of predicate trs – 4) and Anxiety Trait (extent of predicate irs – 3); and • The output in terms of an evaluationt of Program #3 Levels of Trust (LoT) and Sustainability (LoS) to assess SS, values that range in the interval 0…1. In present work a cohort of 198 students was enrolled, and the training set was gotten by clarifying the theorem, viz. 8ðEX1 ; VA1 ; SSA1 ; QoI1 ;


; EX3 ; VA3 ; SSA3 ; QoI3 Þ; ðsrs  5ðEX1 ; VA1 ; SSA1 ; QoI1 Þ;


; irs  3ðEX3 ; VA3 ; SSA3 ; QoI3 ÞÞ in every possible way, i.e., generate all different possible sequences that combine the terms or clauses of the extents of predicates srs – 5, trs – 4 and irs – 3, viz. ffsrs  5ðEX1 ; VA1 ; SSA1 ; QoI1 Þ; trs  4ðEX2 ; VA2 ; SSA2 ; QoI2 Þ; irs  3ðEX3 ; VA3 ; SSA3 ; QoI3 Þg;


g   ffsrs  5ð0:31; 0:29; 0:80; 0:40Þ; trs  4ð0:13; 0:28; 0:91; 0:59Þ; irs  3ð0:29; 0:02; 0:95; 0:69Þg;


g

0.89

0.56 Level of Sustainability (LoS )

Level of Trust (LoT) Output Layer Hidden Layer

Bias

Input Layer

Bias

srs - 5

trs - 4

0.02

0.29

0.28

0.13

0.31

0.29

Pre-processing Layer

irs - 3

Fig. 5. An abstract view of the topology of the ANN to assess Program #3 Levels of Trust and Sustainability.

160

M. Figueiredo et al.

With regard to LoT and LoS, they may be weighed in the mold, viz. ffðSSAsrs5 þ SSAtrs4 þ SSAirs3 Þ=3g;


gLoT  ffð0:80 þ 0:91 þ 0:95Þ=3 ¼ 0:89g;


gLoT and, viz. ffðQoIsrs5 þ QoItrs4 þ QoIirs3 Þ=3g;


gLoS  ffð0:40 þ 0:59 þ 0:69Þ=3 ¼ 0:56g;


gLoS

5 Conclusions and Future Work General Chemistry is one of the foremost subjects in Science and Technology university courses. However, it is imperative that students acknowledge the importance of the teaching content in their current education and for their future professional performance. To meet this challenge, Student’s Satisfaction (SS) when considering General Chemistry must be as high as possible. However, it is difficult to assess SS since it is a subjective matter and deals with different variables with complex relationships among them. Therefore, and in order to assess SS in Higher Education, a data acquisition and evaluation model was developed and experienced in practice. The focus was on the data processing assignment, being the data collected through questionnaires. Future work will consider new basics, namely the relationships with colleagues, teachers, pedagogical practices, schedules or assessment plans, just to name a few. Acknowledgments. This work has been supported by FCT – Fundação para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020.

References 1. Coe, R., Searle, J., Barmby, P., Jones, K., Higgins, S.: Relative difficulty of examinations in different subjects, Report for SCORE – Science Community Supporting Education (2008) 2. Rodríguez, R.M., Corona, L.B., Ibáñez, M.V.: Cooperative learning in the implementation of teaching chemistry (didactic instrumentation) in engineering in México. Proc. – Soc. Behav. Sci. 174, 2920–2925 (2015) 3. Osma, I., Radid, M.: Analysis of the students’ judgments on the quality of teaching received: case of chemistry students at the faculty of sciences Ben M’sik. Proc. – Soc. Behav. Sci. 197, 2223–2228 (2015) 4. Ďurišová, M., Kucharčíková, A., Tokarčíková, E.: Assessment of higher education teaching outcomes (quality of higher education). Proc. – Soc. Behav. Sci. 174, 2497–2502 (2015) 5. Neves, J.: A logic interpreter to handle time and negation in logic databases. In: Muller, R., Pottmyer, J. (eds.) Proceedings of the 1984 Annual Conference of the ACM on the 5th Generation Challenge, pp. 50–54. ACM, New York (1984)

An Assessment of Students’ Satisfaction in Higher Education

161

6. Fernandes, B., Vicente, H., Ribeiro, J., Capita, A., Analide, C., Neves J.: Fully informed vulnerable road users – simpler, maybe better. In: Proceedings of the 21st International Conference on Information Integration and Web-based Applications & Services (iiWAS2019), pp. 600–604. Association for Computing Machinery, New York (2020) 7. Cortez, P., Rocha, M., Neves, J.: Evolving time series forecasting ARMA models. J. Heuristics 10, 415–429 (2004) 8. Fernández-Delgado, M., Cernadas, E., Barro, S., Ribeiro, J., Neves, J.: Direct Kernel Perceptron (DKP): ultra-fast kernel ELM-based classification with non-iterative closed-form weight calculation. J. Neural Netw. 50, 60–71 (2014) 9. Wenterodt, T., Herwig, H.: The entropic potential concept: a new way to look at energy transfer operations. Entropy 16, 2071–2084 (2014) 10. Neves, J., Maia, N., Marreiros, G., Neves, M., Fernandes, A., Ribeiro, J., Araújo, I., Araú-jo, N., Ávidos, L., Ferraz, F., Capita, A., Lori, N., Alves, V., Vicente, N.: Entropy and organizational performance. In: Pérez García, H., Sánchez González, L., Castejón Limas, M., Quintián Pardo, H., Corchado Rodríguez, E. (eds.) Hybrid Artificial Intelligent Systems. LNCS, vol. 11734, pp. 206–217. Springer, Cham (2019) 11. Kakas, A., Kowalski, R., Toni, F.: The role of abduction in logic programming. In: Gabbay, D., Hogger, C., Robinson, I. (eds.) Handbook of Logic in Artificial Intelligence and Logic Programming, vol. 5, pp. 235–324. Oxford University Press, Oxford (1998) 12. McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5, 115–133 (1943) 13. Rosenblatt, F.: The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65, 386–408 (1958) 14. Rumelhart, D., Hinton, G., Williams, R.: Learning internal representation by error propagation. In: Rumelhart, D., McClelland, J. (eds.) Parallel Distributed Processing: Explorations in the Microstructures of Cognition, vol. 1, pp. 318–362. MIT Press, Cambridge (1986) 15. Riedmiller, M.: Advanced supervised learning in multilayer perceptrons – from backpropagation to adaptive learning algorithms. Comput. Stand. Interfaces 16, 265–278 (1994) 16. Haykin, S.: Neural Networks and Learning Machines, 3rd edn. Prentice Hall, New York (2009) 17. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. SIGKDD Explor. 11(1), 10–18 (2009)

Methodological Guidelines to Build Collaborative Serious Games Based on Intelligent Agents Oscar M. Salazar, Santiago Álvarez, and Demetrio A. Ovalle(&) Departamento de Ciencias de la Computación y de la Decisión, Facultad de Minas, Universidad Nacional de Colombia, Carrera 80 No. 65-223, Medellín, Colombia {omsalazaro,salvarezl,dovalle}@unal.edu.co

Abstract. A serious game can be defined as an e-learning system that aims to motivate and entertain students while achieving learning objectives related to problem solving, specific topics, or skills development, always focused on improving learning. The aim of this paper is to offer methodological guidelines to build collaborative serious games based on intelligent agents along with awareness services in order to support virtual learning processes. This methodological guide offers a breakdown of the main activities needed to build this kind of systems based on three components: the knowledge representation of the domain to which the collaborative serious game is oriented, the development of the MAS, and finally the description and integration of awareness services. The preliminary validation of methodological guidelines through a prototype development shows that the integration of these kinds of technologies fosters the knowledge acquisition in a playful way by the students while learning in a collaborative way. Keywords: Serious games
Computer Supported Collaborative Learning Methodological guidelines
Multi-Agent Systems
Awareness Services




1 Introduction In the last decade, there has been a growth in the execution of collaborative activities along with digital learning in classrooms, due to the intention of teachers to enhance the interaction between students, and diversifying the processes of acquisition and generation of knowledge by taking advantage of collaborative learning environments. Computer Supported Collaborative Learning (CSCL) is a computational approach allowing students –being organized into groups– to work together for a common goal and discuss from different points of view which fosters not only to improve social skills like communication or teamwork on them but their learning processes. A Serious Game can be defined as an e-learning system that aims to motivate and entertain students while achieving learning objectives related to problem solving, specific topics, or skills development, always focused on improving learning. Although Serious Games have already been integrated within learning environments however, very few of them address collaboration learning tasks. In addition, there is no effective mechanism to © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 162–173, 2020. https://doi.org/10.1007/978-3-030-52538-5_17

Methodological Guidelines to Build Collaborative Serious Games

163

integrate early detection and diagnosis of learning failures, with a model that fosters collaboration, evaluates, and motivates students to interact at different stages of the learning process. The aim of this paper is to propose methodological guidelines to build Collaborative Serious Games based on Intelligent Agents along with Awareness Services in order to support virtual learning processes. This guide offers a breakdown of the main activities necessary to build this kind of systems, based on three components: first, the knowledge representation of the domain to which the collaborative serious game is oriented; second, the development of the MAS, and third, the description and integration of Awareness Services. The rest of the paper is organized as follows: while Sect. 2 presents the conceptual framework, Sect. 3 exhibits Related Works regarding to our research. Next, Sect. 4 describes the methodological guidelines proposal. Finally, Sect. 5 presents the conclusions and future work.

2 Conceptual Framework Following are the main concepts and fields related with this research. A Serious Game can be defined as a system that seeks firstly to motivate and entertain students while having an educational character to achieve learning objectives, solve problems, learn a specific topic or develop skills, always attempting to improve learning. Serious games are being built at different levels of education both in primary and secondary schools as well as in universities worldwide. According to [1] the use of serious games involves also private companies and concerns several fields and professions such as aerospace (flight simulators), health (simulation of surgical operations, emergencies), finance (training compliance and financial issues learning), trading (simulation of fictitious companies), and tourism (hospitality training). In addition, it is important to highlight that Serious Games have not been only used for individual learning processes, but for Collaborative Learning Environments obtaining satisfactory results [2]. The concept of Awareness Services is used in collaborative learning environments the reason is that if two students are using distributed schemes of computer-supported collaborative work (CSCW) they generally cannot see or hear, and neither feel the presence and perceive each other actions. In this kind of CSCW environments, these awareness skills are quite limited. The context-awareness has thus become one of major issues when designing pedagogical computer systems in order to reduce the need of meta-cognitive efforts for collaborating on distributed computer environments [3]. Gaver highlights the importance of providing context-awareness information with the purpose of help people to change the individually work role to work in groups [4]. In this fashion, Dourish and Bellotti [5] apply this characteristic within shared learning environments and define the awareness as a shared understanding of each other activities, thus providing a context for their own activity. A Multi-Agent Systems (MAS) is an organized society composed of semiautonomous agents that interact with each other, either to collaborate in the solution of a set of problems or in the achievement of a series of individual or collective goals.

164

O. M. Salazar et al.

The MAS principles have shown adequate potential in developing learning systems since the nature of teaching-learning problems is more easily faced through a cooperative approach. However, much remains to be done in this field, as mentioned in [6]: “While our understanding of learning agents and multi-agent systems has advanced significantly, most applications are still simple on scaled-down domains, and, in fact, most methods do not scale up to the real world”. According to Berners-Lee [7] “The Semantic Web is an extension of the current Web in which information has a well-defined meaning, it is understandable by computers and where people can work cooperatively and collaboratively”. From this new paradigm, ontologies appear as the means to represent knowledge on the Web in a way that is made readable and usable by computers. “An ontology is the result of selecting a domain and apply the same method to obtain a formal representation containing the concepts and relationships that exist among them” [8].

3 Related Works This section presents some related works with the research field, and compares them in order to identify their strengths and weaknesses. González-González et al. [9] propose an intelligent rehabilitation system based on a video game named TANGO: H (from Tangible Goals: Health) which can be used to support the rehabilitation process of hospitalized children. The system consists of two elements: an intelligent platform along with the video game and an exercise design tool. The smart platform includes a recommendation system that analyzes user interactions, along with user history, to select new gamified exercises for the user. One of the main contributions of this work focuses on defining a recommendation system based on different levels of difficulty and user skills to offer the possibility of providing the user with a personalized game mode based on their own history and preferences. According to the authors, one of the main problems of rehabilitation is that therapy sessions can become boring since they usually consist of repetitions of the same exercises over and over again. Therefore, games can help improve motivation and make patients work harder [10, 11]. Salazar et al. [12] built the EOLo system which is based on a serious game model compatible with mobile computing which aims to improve the learning of specific topics related to Adaptive Virtual Courses (CVAs). The model incorporates three main modules: the web-based backend, the mobile application (serious game) and finally, the instructional design module in charge of game content management. EOLo’s, prototype was validated using a case study applied to English learning at the secondary level. The results show that students feel motivated with the use of EOLo thanks to the healthy competition offered by the game during the development of learning activities. In addition, the system allows the detection of learning failures to generate proactive feedback based on the recommendation of educational resources. It should be noted that students can use their mobile devices anywhere and at any time in order to carry out learning activities developed by the teacher through CVAs adapted to the needs and preferences of the students.

Methodological Guidelines to Build Collaborative Serious Games

165

Allal-Chérif and Bidan [13] analyze the relationship between MOOCs (Massive Open Online Courses) and serious games, two approaches to improve learning. Authors attempt to identify the areas of influence in collaborative open training serious games developed by large firms for a significant cost and made available for free to the public and to students according to the same principles as MOOCs. In fact, MOOCs are much more than online courses: these interactive websites are real social networks designed to bring together people who are interested in the same subject [14]. Three serious games from L’Oréal, IBM, and Thales were considered and results obtained from this study established that exist real influence on students according to the following five dimensions: relations, culture, knowledge, innovation, and desire. This model is then discussed and tested on eight other serious games from major industrial companies such as General Electric, Nestlé, and Cisco. Authors conclude that Serious Games represent a good contribution if they are embedded in training programs and recruitment processes. Considering the research works previously reviewed very few of them use collaborative learning approaches and there are no methodological guidelines to build those kinds of systems. In addition, one of the improvements proposed in our approach in order to enhance CSCL systems based on MAS is the integration of awareness services specifically created for monitoring teamwork processes during the execution of the game. Consequently, the collaborative game system gives the students, teams, and teachers real time information that helps them to improve their performance.

4 Methodological Guidelines Proposal We propose a methodological guideless for the conceptualization, design, and development of serious games using awareness services and intelligent agents. This methodological guide presents a breakdown of the activities necessary for the construction of this kind of systems, based on three components: the knowledge representation of the domain to which the Serious Game is oriented, the MAS development and the description and integration of Awareness Services. This methodological guide proposes three steps of development for the implementation of the components. In addition, each step comprises in turn a series of activities which will be detailed below. Step 1. System specification and analysis • Identification of collaborative activities involved • Detail and specification of a serious collaborative game model. • Specification process and development of the ontological model. Step 2. Architecture design and development • Design and description of awareness services involved in the development of collaborative activities. • Multi-agent architecture for the deployment of the collaborative based game system. Step 3. Component implementation and system validation • System implementation. • System validation.

166

O. M. Salazar et al.

Next, each of the steps that comprises the methodological guide is presented, detailing each of the activities from a real case study applied to an undergraduate course. 4.1

Step 1. System Specification and Analysis

This step is intended to capture the main concepts that must be considered to develop collaborative serious games, carrying out the development of the steps presented below. Identification of Collaborative Activities Involved The collaborative activities involved in the Serious Game can be deployed through a flow diagram as shown in Fig. 1.

Fig. 1. Collaborative serious game process flow diagram

Methodological Guidelines to Build Collaborative Serious Games

167

Detail and Specification of a Collaborative Serious Game Model Table 1 specifies in detail all the elements that comprise the collaborative serious game model based on the flow diagram deployed on Fig. 1. Specification Process and Development of the Ontological Model This model includes the specification, design and implementation of a specific domain ontology that represents the knowledge related to the serious game and the topics of the course to be developed, user profiles and contextual information and the Learning Objects (LO) that support the recommendation of educational contents. Main concepts involve within the Collaborative Serious Game Ontology and their description are the following [17]: • Learning Activity Exercise: It corresponds to a work or task to be developed that is framed within a topic of the course. • Cognitive Feature: It corresponds to a knowledge characteristic of a student. In this case, this characteristic may be a failure or a strength. • Team: It corresponds to the group of students who work to achieve the same objective. • Student: It is the individual in the classroom who is learning, sharing and generating knowledge in a learning environment. • Evaluation: It corresponds to an exam that is performed to validate the knowledge acquired by a student or a team of students. • Track of Answer: It is the track of answers provided by the students in the framework of a multiple-choice evaluation. • Educational Objective: It corresponds to the educational goals that must be achieved by the students in the scope of a course. • Team Profile: It is a description of the generalized cognitive characteristics in the team. Basically, it translates into the strengths and weaknesses of a team based on weights of individual characteristics. • Question: It is the statement of the question that is presented in the scope of an evaluation. • Resource: It is an LO or a tutor that can help students learn, understand or clarify a concept related to an activity. • Reply: It corresponds to the options provided to solve or answer each of the questions. 4.2

Step 2. Architecture Design and Development

At this step the main models that must be considered for the subsequent implementation of the collaborative Serious Game are developed based on the concepts defined in the previous step. The models are described below. Design and Description of Awareness Services to Support the Development of Collaborative Activities The following awareness services allow teachers, students, and teams involved in the game to know their status during their execution. This allows the model to provide the necessary awareness to keep the teamwork cohesive, interested, motivated and alert.

168

O. M. Salazar et al. Table 1. Description of steps deployed in diagram of Fig. 1

Step BDPR BD-R BDOAS EQ

PR1 PR2 CS1 S1

S2 S3 S4 S5 S6 S7

S8

S9 S10 ESRO ESRR CS2 CS3 CS4

Description Question Database: stores questions previously built by teams Response Database: stores each of the selected responses issued during the execution of the game Learning Object (LO) Repository. It is an external repository where LO are stored to make recommendations based on failures detected Each team performs a collaborative construction of high-level multiple-choice questions. These questions will be accompanied with answer, thus providing some incorrect options and one correct one The teacher makes a categorization of questions based on difficulty levels in order to deliver badges according to defined rules within the game The teacher starts the question step The system checks if there are questions that have already not been played in any of the rounds The system randomly selects a question and a student from the team where the question was asked (it will be a different student than the one who created the question and will be known hereinafter as a challenging student) and a random student from a different team (designated as challenged student) The system initiates a game round between the challenging student and the challenged student The system assigns a point to the team to which the challenged student belongs The system does not assign or subtract points from either team The system subtracts a point for the team to which the challenging student belongs The system makes the assignment of grades to each team according to the position table, score and badges (recognitions) obtained The system generates in adaptive way a cumulative individual assessment for each student. Note: The adaptation of the assessment is made based on the following criteria: (1) Questions failed by the student; (2) Questions not answered by the team; (3) Questions that were not played, that is, that did not appear in any of the rounds of the game (if applicable) The system grades the individual cumulative assessment of each student and depending on the results, recommends educational resources associated with their own failures detected The system deploys statistics concerning the game performance obtained by students and teams to both the teacher and the students The system assigns a badge (recognition) to the team to which the challenged student belongs The challenged student must answer the question correctly The challenging student must answer the question correctly The system verifies if the challenged student has answered the question correctly The system verifies if the challenging student has answered the question correctly The system verifies if the challenged student deserves to receive a badge (recognition) checking that he/she meets the conditions to receive it

Methodological Guidelines to Build Collaborative Serious Games

169

– Display of rounds and shifts. This awareness service makes the presentation of visual data about the state of execution of a serious game concerning the shifts, rounds, students, and teams involved in each turn. – Table of team positions. It presents for both teacher and student a view of the realtime positions of each team participating in the game. – Alarms and notifications. This awareness service has proactive behavior which allows it to generate alarms and notifications from the performance of students and teams during the execution of the serious game. These notifications or alarms are presented to the users due to the expiration of response times in the game turns, due to the delay in the assessments, or as motivation alert to the students about a possible change within table position produced by the success or failure in his/her answer delivered. – Badge (recognition) table. This awareness service is very relevant since it has a direct impact on the positive emotions and motivations of students and teams. Consequently, the granting of badges (recognitions) motivates the teams to obtain them, and therefore to have a better performance during the execution of collaborative activities through the Serious Game. – Execution Statistics. This service presents to the teacher a view in order to visualize the status and history of the activities that have been carried out during the game, with the objective that the teacher can know and continuously monitor the status of the student’s teams. – Accessible learning resources to be recommended. This service has two associated functions; first, introduces students to recommended accessible learning resources, and second, presents to the teacher the information associated with the resources that have been visualized by each student. Multi-agent Architecture for the Deployment of the Collaborative System In order to provide the model with characteristics of adaptability, distribution of tasks and proactivity, both in the processes of recommendation of educational resources, and during the execution of the proposed model for the development of collaborative activities, it is decided to incorporate intelligent agents framed in a MAS that facilitates the design and implementation of proactive and adaptive tasks. In that sense, the model corresponding to Fig. 2 shows the analysis overview diagram of the Serious Game application, which is the result of the application of the Prometheus methodology [15]. This methodology allows the design of MAS from detailed processes, practical advice, and a variety of artifacts that evolve with modeling iterations. Prometheus methodology was chosen to design the MAS, since it supports the development of intelligent agents by non-expert users. It is a practical, complete and detailed methodology. In the same way, Prometheus methodology provides everything that is needed for defining and designing agents and it also consider goals and plans to develop robust and flexible agents [16]. Prometheus consists of three design phases that are developed further on: first, the system specification phase, second, the architectural design phase, and third, the detailed design phase.

170

O. M. Salazar et al.

Fig. 2. Serious game multi-agent analysis overview diagram

The agents that comprise the collaborative model are the following: • Student Agent, responsible for representing the student in the field of the platform, as well as managing their profile. It is an interface agent, which manages all interactions (perceptions and/or actions) between the system and the student. • Teacher Agent, in charge of representing the teacher within the platform, supports him in the instructional design of the activities and the grading of reports. • Statistics Manager Agent, responsible for presenting execution statistics with the objective of generating awareness from the collection of context information. • Ontological Agent: administers the specific domain ontology proposed by Álvarez et al. [17], that is, it is responsible for populating the entities and making inferences requested by other agents. These inferences are generated from the use of consultations and are related to the detection of cognitive failures, structure of activities, resources, timely information of students and teams. • Evaluation Agent: deploys the evaluations to the students, is responsible for the execution of the game, transfers the information and stores the results in the database (Database) of the system. • Team Manager Agent, responsible for the communication and transfer of information with each of the Student Agents comprising a specific team. In fact, this agent knows the students that belong to each team. • Recommender Agent, in charge of delivering information requested from the Team Manager Agent. On the other hand, it is responsible for recommending learning resources through the consumption of a Web service exposed by an external system of learning resources management. This system receives the keywords inferred by the Ontological Agent when cognitive failures are detected.

Methodological Guidelines to Build Collaborative Serious Games

4.3

171

Step 3. Implementation and Validation of the Collaborative Game System

In order to implement and validate the model based on the methodological guidelines, a serious game prototype was built as shown in Fig. 3. To build this prototype different technologies were used such as MySQL engine that was used for the database assembly. In addition, the Frontend Laravel Version 5.6 framework was used along with the implementation of Blade views for the game development. This Frontend communicates through REST Web services exposed through an API developed using JAVA through JADE (Java Agent Development Framework). It is important to highlight that JADE is a FIPA-compliant to develop each of the agents within the MAS.

Fig. 3. Collaborative serious game interface

To carry out the validation process, a case study was constructed and deployed in a controlled environment, specifically within the Artificial Intelligence course taught at the School of Mining of the National University of Colombia - Medellin branch. The case study had a teacher, 15 students divided into 3 teams which interacted with the prototype and followed the flow of the serious collaborative game until it was completed. In addition, the students constructed a total of 18 high-level questions and were assigned educational objectives to them by the teacher. Finally, an LO (Learning Object) repository was simulated within the system and was populated with 25 real LO with topics associated with the questions previously created.

172

O. M. Salazar et al.

Subsequently, we have considered five different approaches to validate the collaborative serious game based on the survey of student’s perception who participated in the case study. These approaches are: (1) the global validation of the model and prototype implemented; (2) the validation of the effectiveness in the detection of failures and allocation of educational resources recommended to each student; (3) the validation of the functionality and level of adaptation of the final cumulative assessment generated for each student; (4) the validation of the awareness services implemented; (5) the validation of the students’ perception of the knowledge gained through the use of the game which was compared to the use of a traditional learning methodology. The results obtained from the five approaches considered showed that the incorporation of this type of technologies favors the acquisition of knowledge in a playful way by the students, thus awakening their interest and thus providing a possible solution to the current needs regarding the utilization of active methods for teachinglearning processes.

5 Conclusions and Future Work It is important to highlight that the proposed methodological guide allowed the materialization of a series of iterative steps for the specification, analysis, design, implementation, and validation for development of collaborative serious games based on intelligent agents and awareness services. In fact, these guidelines allow us to provide mechanisms and tools that motivate collaboration and participation among teamwork in learning activities, while generating a positive interdependence that makes the participation and involvement of each student in order to succeed the challenges of serious game. In addition, the methodological guide allows the integration of different tools for fault detection, content selection, and the generation of awareness services that enrich the MAS. In addition, the integration of the elements specified in each of the stages, contributes to generate a more enjoyable collaborative environment for learning. As future work we plan to integrate other awareness services which can improve the performance of each teamwork much more, as well as enhance the level of learning, acquisition, and transfer of knowledge. In that sense, it is expected to include an awareness service that exhibits similar characteristics to that presented by Matazi et al. in [18] which is capable of calculating indicators such as: number of messages, number of discussions, number of connections made, number of interactions with teammates, number of contributions made and, therefore, be able to evaluate the degree of collaboration, the degree of presence, and the level of participation of each student in their team. Acknowledgments. This research was partially funded by the doctoral scholarship offered to Oscar Salazar by COLCIENCIAS (Colombian Department for Science and Technology Development) through grant # 761 “Convocatoria Nacional de Jóvenes Investigadores e Innovadores 2016”.

Methodological Guidelines to Build Collaborative Serious Games

173

References 1. Mclain, M.L.: Collaborative game based learning of post-disaster management. In: Proceedings International Conference in Technology for Education (T4E), pp. 80–87 (2016) 2. El-Ghouli, L., Khoukhi, F.: Contributions of serious games on adaptive learning systems. In: Proceedings International Conference on Intelligent Systems: Theories and Applications (SITA) (2016) 3. Palfreyman, K., Rodden, T.: A protocol for users awareness on the world wide web. In: Proceedings of CSCW 1996, USA, pp. 130–139 (1996) 4. Gaver, W.: Sound support for collaboration. In: Proceedings of the ESCW 1091, pp. 293– 308 (1991) 5. Dourish, P., Bellotti, V.: Awareness and coordination in shared workspaces. In: Proceedings of ACM Conference on Computer Supported Cooperative Work (CSCW 1992). ACM Press, Toronto (1992) 6. Kudenko, D., Kazakov, D., Alonso, E.: Adaptive Agents and Multi-Agent Systems III. Adaptation and Multi-Agent Learning, vol. 4865 (2008) 7. Berners-Lee, T., Hendler, J.: Publishing on the semantic web. Nature 410(6832), 1023–1024 (2001) 8. Tramullas, J., Sánchez-Casabón, J., Garrido-Picazo, P.: An evaluation based on the digital library user: an experience with greenstone software. Proc. Soc. Behav. Sci. 73, 167–174 (2013) 9. González-González, C.S., Toledo-Delgado, P.A., Muñoz-Cruz, V., Torres-Carrion, P.V.: Serious games for rehabilitation: gestural interaction in personalized gamified exercises through a recommender system. J. Biomed. Inf. 97, 103266 (2019) 10. Proffitt, R., Sevick, M., Chang, C.-Y., Lange, B.: User-centered design of a controller-free game for hand rehabilitation. Games Health J. 4(4), 259–264 (2015) 11. Rego, P.A., Moreira, P.M., Reis, L.P.: A serious games framework for health rehabilitation. Int. J. Healthc. Inf. Syst. Inf. 9(3), 1–21 (2014) 12. Salazar, O.M., Álvarez, S., Ovalle, D.: EOLo: a serious mobile game to support learning processes. In: Methodologies and Intelligent Systems for Technology Enhanced Learning, pp. 118–125 (2017) 13. Allal-Chérif, O., Bidan, M.: Collaborative open training with serious games: relations, culture, knowledge, innovation, and desire. J. Innov. Knowl. 2, 31–38 (2017) 14. Cooper, S., Sahami, M.: Reflections on Stanford’s MOOCs. Commun. ACM 56(2), 28–30 (2013) 15. Lhafiane, F., Elbyed, A., Bouchoum, M.: Multi-agent system architecture oriented prometheus methodology design for reverse logistics. Int. J. Comput. Electr. Autom. Control Inf. Eng. 9(8), 1914–1920 (2015) 16. Giorgini, P., Henderson-Sellers, B.: Agent-Oriented Methodologies. IGI Global (2005) 17. Álvarez, S., Salazar, O.M., Ovalle, D.A.: Modelo basado en agentes para la detección de fallas cognitivas en entornos de aprendizaje colaborativo. Inf. Tecnol. 29(5), 289–298 (2018) 18. Matazi, I., Bennane, A., Messoussi, R., Touahni, R., Oumaira, I., Korchiyne, R.: Multi-agent system based on fuzzy logic for E-learning collaborative system. In: International Symposium on Advanced Electrical and Communication Technologies, ISAECT 2018 Proceedings, pp. 1–7 (2018)

An Augmented Reality-Based mLearning Approach to Enhance Learning and Teaching: A Case of Study in Guadalajara Janet Pigueiras1,2(B) , Angel Ruiz-Zafra2 , and Rocio Maciel1 1

University of Guadalajara (CUCEA), Guadalajara, Jalisco, Mexico [email protected], [email protected] 2 University of C´ adiz, C´ adiz, Spain [email protected]

Abstract. Augmented Reality (AR) is one of the most widespread technologies used in education, and many researches have been published proving its effectiveness. In recent times, the use of AR has been fostered through the use of mobile devices (smartphones), giving rise to what is called mLearning (mobile learning), with the aim to enhance the educational experience at all levels. Most of AR-based mLearning systems have been developed from a software development perspective, without the involvement of educational stakeholders to provide the content to show to students. On the other hand, many other researches propose solutions to cover specific educational requirements, but they completely forget about the acceptance of end-users (e.g. students). This paper presents an approach to assist in the development of AR-based mobile learning systems. The approach presents a methodology to guide the procedure to deploy an AR-based system, where different stakeholders are involved. In addition, a contribution of the approach is a survey to validate the user acceptance of AR-based mobile applications. To support our approach, most educational know-how and needs to improve in education through ICT were gathered during this research alongside the Polytechnic School “Ing. Jorge Matute Remus” (Guadalajara, Mexico). For this educational center we have developed cuceAR, an AR-based mLearning prototype application to enhance teaching and learning as an example and a contribution of the approach. Keywords: Education · Augmented Reality · Mobile learning Survey · Technology Acceptance Model · Methodology

1

·

Introduction

Education, as a cornerstone in the society, has been during the last years and decades a hot topic in ICT (Information and Communication Technology), due to its benefits and the promising enhancement of educational parameters through the use of the technology [1]. Many ICT-based solutions have emerged during last c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 174–184, 2020. https://doi.org/10.1007/978-3-030-52538-5_18

An AR-Based mLearning Approach to Enhance Learning and Teaching

175

years [2], and several organisations have focused their attention to ICT applied in education [3]. One of the most widespread used technologies is Augmented Reality (henceforth AR), which has been widely accepted in society since its inception in several areas, not only education [4]. AR enables the combination of the real world with virtual objects, overlapping multimedia information in real time [5]. Along with AR, a trendy topic in ICT-based education proposals is mobile learning (also called m-learning or mLearning), that is, the use of mobile devices to enhance the educational experience at all levels [6]. Many AR-based mobile learning researches have been carried out in education showing their benefits against traditional learning environments in many different countries [7]. In Mexico, like in many countries in the world, several studies have been conducted to check the quality of education at all levels. For instance, the Secretariat of Public Education (SEP ) in Mexico warned about loss of quality education during the period 2016–2017, with a failure rate of 4.9% in middle school and 13.7% in high school [8]. Other studies, like the conducted by Rodriguez et al. [9], point out the reason as the lack of individual study habits of the student, as well as the lack of teaching strategies by teachers. Thus, the benefits from the use of mobile learning and AR technologies can significantly enhance the quality education in Mexico, as shown through numerous related researches in other countries [10,11]. However, the development of AR-based mLearning solutions is not a simple task, requiring the involvement of educational staff to target the educational requirements, the development of mobile applications that cover these requirements and support required functionalities and techniques and tools to ensure the end-user acceptance of the applications, with the aim to ensure AR-based solutions are suitable for education compared to traditional educational approaches. In this paper we present an AR-based mobile learning approach with the aim to enhance learning environments. This approach presents 1) a methodology to address the study of educational environments where AR-based mobile learning technologies can improve quality of education; 2) cuceAR, an AR-based mobile application to cover educational needs developed for a specific educational center; and 3) a survey for the validation of AR-based mobile applications through TAM (Technology Acceptance Model ). We have developed our approach using the know-how about educational requirements obtained in the Polytechnic School “Ing. Jorge Matute Remus” in Guadalajara, Mexico (henceforth Polytechnic), which served as case of study. The remainder of this paper is organized as follows. Section 2 presents the related work. Section 3 introduces the approach and presents the methodology proposed to address the startup of AR-based mobile learning solutions. Section 4 shows cuceAR, an AR-based mobile app to cover the needs and enhance the education in the educational center used as case of study. Section 5 introduces the survey proposed for the validation of AR-based mobile applications. Finally, Sect. 6 summarizes the conclusions and further work.

176

2

J. Pigueiras et al.

Related Work

During the last years many researches and studies have been published showing the benefits of the use of AR in mobile learning applications against traditional educational approaches. For instance, the survey presented in [12] describes 68 researches where AR-based application have been used for educational purposes with promising results (against traditional approaches). About the application of AR-based mobile learning approaches in the Mexican education system, one of the most highlighted project is the presented in [13], where an AR-based mobile application was used in-the-wild with a set of students with satisfactory outcomes. In addition, in the literature there is a wide variety of AR-based mLearning researches applied to education at different academic levels [14,15]. During our research around 30 studies have been reviewed. However, for space limitation, only some of them have been cited in this paper. To summarize, these researches exhibit the following common drawbacks: 1. No Requirement Analysis Phase. Many researches and AR-based applications for educational use are developed without the involvement of educational staff, who has the know-how about educational needs. 2. Level-Oriented. Elementary and university are the main educational levels addressed, leaving the upper middle level behind. 3. Static Educational Content. AR-based mobile applications are developed to display specific and static content, hindering the customization of content for students. 4. No User Acceptance Testing. Most researches usually focus their attention in development and software engineering issues, forgetting about the user acceptance of the application. Thus, in this paper we address these drawbacks, proposing an AR-based mLearning approach based on different elements, which are described in the rest of this paper.

3 3.1

Approach to Enhance Learning and Teaching Through Augmented Reality and Mobile Computing Introduction

The enhancement of teaching and learning in education through ICT is a complex task where many stakeholders and technologies are involved. Educational staff as well as students are the know-how carriers, which can detect the issues or aspects to enhance in educational tasks, while technology could be the way to provide solutions for these educational issues/aspects. Due to the different challenges presented in this development process (from know-how to a ready-to-use AR-based mLearning system), we present in this paper an approach with the aim to help in this development process. The approach has three different elements:

An AR-Based mLearning Approach to Enhance Learning and Teaching

177

1. Methodology. A set of stages identifying in each one the stakeholders involved, the required input and the expected outcomes. 2. AR-Based Mobile Application. As an example of AR-based technology, we present cuceAR, an application prototype developed according to educational needs detected in a specific educational institution,the Polytechnic. 3. Survey to Evaluate User Acceptance. Finally, a survey based on TAM (Technology Acceptance Model ) is proposed as tool to validate AR-based mobile applications for education, in terms of user acceptance. This approach has been developed using the know-how obtained from the Polytechnic, where educational staff and students have contributed identifying educational needs. In the rest of this section is presented the methodology. The AR-based mobile application is presented in Sect. 4 and the survey is explained in Sect. 5. 3.2

Methodology for the Development of AR-Based mLearning Systems

Figure 1 shows an overview of the methodology, illustrating the different stages, the stakeholders and elements involved, as well as the required input and outcome of each stage. Although this methodology has been illustrated as a sequential process to clarify the design, the methodology has been envisaged as a sequential, linear and iterative approach, where the different stages can be carried out repeatedly.

Fig. 1. Methodology for the development of AR-based mLearning systems

178

J. Pigueiras et al.

Furthermore, the methodology involves the different stakeholders through the design of the system to define the proper needs. The methodology also focuses on a user-centered design (UCD) approach, where the different goals are oriented to satisfy user needs and also end-users (students) have been involved through different stages of the methodology. An example of UCD for enhancement learning can be checked in [16]. During our research, we carried out teaching assistant tasks in the Polytechnic, interacting with the different stakeholders related to education (educational staff and students) in order to detect the aspects to improve in their daily educational tasks. This knowledge helped us to comprehend educational needs and how the technology can cover them, helping in the design of this methodology. The explanation of the different stages of the methodology, within the framework of this collaboration, is shortly described as follows. The methodology starts with the research about the state of education in Mexico. Through this stage, and using academic researches and official reports (such as those briefly described in Sect. 2), we got a big picture about the education issues in Mexico. In order to find proposals to enhance the education and solve these educational issues, we carried out a systematic review of the technologies available in the second stage, focusing in AR as core technology of our proposal. In this stage, we did review many papers (some of them described in Sect. 2) as well as commercial AR-based technologies that can be applied to education (libraries, toolkits), with the aim to determine benefits provided by AR technology in the education as well as the most suitable software to build AR-based mobile applications. In the third stage, few months were required carrying out teaching assistant tasks in the Polytechnic. Using the big picture obtained in the first stage of the methodology as background, during this stage 1) we did hold several meetings with the educational staff; 2) we reviewed the syllabus of the telecommunications studies; and 3) through several talks with students we got their impressions about the current state of education and their educational needs. As a result of this stage we achieved two main goals: 1) get the point of view of professors (as educators) as well as the students about the use of ICT and AR-based solutions and how beneficial it can be as a support tool in the classroom; and 2) a set of desired features to support educational needs such as the real-time access to information, localized access to any data source and the proper dissemination of concepts and artifacts that are not available in the facility resources, among others. The fourth stage focuses on the development of the AR-based mLearning system by developers/programmers, using the right choice of the AR technology (stage two outcome) according to the aspects to improve in the educational center (stage three outcome). In this particular case, we have developed a specific AR-mobile, cuceAR, which is explained in Sect. 4. The user (student) acceptance of the technology used to enhance the learning and teaching is quite important. In this way, we propose a survey based on TAM

An AR-Based mLearning Approach to Enhance Learning and Teaching

179

(Technology Acceptance Model ) to validate the user acceptance of AR-mobile applications, which is depicted and explained in Sect. 5. Finally, once the application has been validated (user acceptance testing), the entire AR-based mLearning solution should be deployed into the educational center (sixth stage).

4

cuceAR: An AR-Based Mobile Application to Improve Education

As outcome of the fourth stage of the methodology, and in our particular case at the Polytechnic, we have developed cuceAR, an AR-based mobile application that aims to be a support tool for students in telecommunication studies, allowing the access and search of additional information about their academic activities (Fig. 2).

Fig. 2. cuceAR Mobile application (a) Subjects/Courses (b) Measurement Tools (c) Electronic Circuits

Through cuceAR the students are able to access the available content for the different subjects of the syllabus (listed in Fig. 2a). For each subject there are terms (words) predefined by the professor. These terms, detected through AR marker-based technology (Fig. 2b and 2c), allow students to get additional information to solve doubts and expand knowledge about a specific topic, displaying Youtube Videos, Wikipedia entries, online documents, etc.

180

J. Pigueiras et al.

cuceAR, as an educational resource, support the needs detected for this case in the Polytechnic, but also could be applied in any institution with the same requirements. So far, cuceAR is in an early stage of development (prototype), however, the following technological guidelines will be considered during the development life cycle: – Service-Oriented Design. The entire system will be developed following a service-oriented design to ensure their extensibility, where new services could be added at anytime to support new functionalities. – Cloud-Oriented. The entire system will be supported by cloud technology, ensuring the scalability of the system and enabling the access to the information in real-time, anywhere and anytime. – User Authentication and Identification System. Through an identification system it could be possible to track the terms searched by a specific student. This aggregated data could be very helpful to educational staff. – Cross-Platform Software. cuceAR will be developed for iOS as well as web platform to widespread its use. These technological guidelines in the development life cycle bring many benefits, such as scalability, extensibility and real-time access to information, among others. Thanks to these benefits and the technology used, many different key functionalities could be supported in order to enhance teaching and learning, highlighting: – Delocalized Use. The application could be used anywhere, indoor (classroom) as well as in outdoor learning environments, fostering its use and the interest in learning educational-related concepts by students. – Common Technologies to Foster Education. Vast majority of people have a smartphone device with Internet connection, including students. This situation could be exploited to foster the motivation of the student through a friendly mobile application. – Learner-Centred Approach. The application proposes new ways of interaction in classroom, since it breaks with the traditional classroom environment where only the teacher teaches, reducing the gap between professor and students. – Real-Time Interactive Learning. Through cloud technologies and services could be possible access to information in real time. – Retention of Knowledge. The student can use the application to expand and retain their knowledge through multimedia content, making the classes more interactive. – New Resources. Through the application could be possible access to new information not available in the facilities (e.g. oscilloscope, devices, assembly process). To sum up, the benefits provided by cuceAR can enhance teaching and learning in educational centers, specially in the Polytechnic, where the application is oriented to.

An AR-Based mLearning Approach to Enhance Learning and Teaching

5

181

Survey to Validate User Experience in AR-Based mLearning Applications

As an outcome of the fifth stage, we propose a survey to measure user acceptance of AR-based mobile applications by the students. The survey has been designed following TAM (Technology Acceptance Model ), which can predict and explain efficiently the intention and behavior of users related to the acceptance of a particular information system [17]. TAM is oriented to the use of questionnaires to measure the ease of use and usefulness of a technological tool. According to the literature, the elaboration of the questionnaires should reflect the validation of four variables: 1) perception of usefulness, 2) perception of ease of use, 3) attitude toward use and 4) the behavioral intention to use [18]. The survey, presented in Table 1, is organized in four blocks, one per each variable to be validated. Each block contains several easy-to-answer questions where each one is rated using Likert scale [19], which values may vary from 1 (totally disagree with the statement) to 5 (strongly agree with the statement). Table 1. TAM-based survey for user-acceptance validation of AR-based mobile apps Question

Score

Verification of the perceived usefulness variable The application would facilitate a better understanding of concepts and techniques I think the application is an useful tool to be used in class Through the use of the application I can improve my learning process and performance in the different subjects Verification of the perceived ease of use variable The application is easy to use and does not require prior technical knowledge Learning to use the application is easy, it doesn’t take much time Verification of the attitude toward using variable The application makes learning more attractive and stimulating I got bored using the application Verification of the behavioral intention to use variable The use of applications based on augmented reality will make me feel more motivated in class Forthcoming semesters I would like to use the application in the classroom I would like to use the application as a support tool for self-study hours at home

Furthermore, the application of the survey proposed require the population sampling. To calculate the proper sample size for the survey we suggest the use

182

J. Pigueiras et al.

of the Cochran equation, illustrated in Eq. 1a; used for large populations (greater than 10.000 subjects) or an unknown total number of subjects [20]. Z 2 pq e2 n0

n0 = n=

1+

(n0 −1) N

(1a) (1b)

In the Cochran equation, Z 2 is the critical Z value, found in statistical tables which contain the area under the normal curve (also called confidence level ), p is the estimated proportion of an attribute that is present in the population, q is 1-p and, finally, e is the desired level of precision. On the other hand, if our study requires small population samples (less than 10.000 subjects), we suggest the formula presented in 1b to estimate the proper number of subjects required in the study [21]. The formula is based on the original Cochran Eq. 1a, where n is the sample size and N is the population size; and n0 as sample size can be adjusted using Cochran Eq. 1a. In our particular case, in the Polytechnic, there are around 200 students enrolled in the telecommunication career. So, the right formula to calculate the most suitable number for the population sample would be the number Eq. 1b.

6

Conclusions

Augmented reality (AR) gradually comes into education boosting systems and applications developed to support educational tasks, fostering the use of ARbased applications by educational staff and students with the aim to enhance parameters of quality in education. In this paper we describe an approach to enhance learning and teaching through AR-based mLearning technologies. In this approach we have proposed a methodology, where through several stages the different concerns related to the improvement of quality of education are analysed. In addition, as part of the approach and contribution of the paper, a survey to validate user acceptance of AR-based mobile applications has been presented. We have collaborated with the the Polytechnic to get the know-how required to propose this approach. As outcome, we have developed cuceAR, an AR-based mobile application to provide required functionalities to cover educational needs. As for future work, we foresee to complete the development of cuceAR, develop it for other platforms (e.g. iOS) and deploy the entire system to be ready for end-users. Secondly, we will focus in the improvement of the survey to cover usability and usefulness, for instance, adding new questions based on the ISO 9241-11 (Ergonomics of human-system interaction). Finally, we would like to conduct studies in several educational centers, in order to validate our approach.

An AR-Based mLearning Approach to Enhance Learning and Teaching

183

References 1. Pheeraphan, N.: Enhancement of the 21st century skills for thai higher education by integration of ICT in classroom. Proc. Soc. Behav. Sci. 103(2013), 365–373 (2013) 2. Gosper, M., Woo, K., Muir, H., Dudley, C., Nakazawa, K.: Selecting ICT based solutions for quality learning and sustainable practice. Australas. J. Educ. Technol. 23(2), 227–247 (2007) 3. Haddad, W., Jurich, S.: Ict for education: potential and potency. Technologies for education: Potential, parameters and prospects. UNESCO and Academy for Educational Development, pp. 28–40 (2002) 4. Van Krevelen, D., Poelman, R.: A survey of augmented reality technologies, applications and limitations. Int. J. Virtual Reality 9(2), 1–20 (2010) 5. FitzGerald, E., Ferguson, R., Adams, A., Gaved, M., Mor, Y., Thomas, R.: Augmented reality and mobile learning: the state of the art. Int. J. Mob. Blended Learn. (IJMBL) 5(4), 43–58 (2013) 6. Sharples, M., Taylor, J., Vavoula, G.: Towards a theory of mobile learning. Proc. mLearn 1, 1–9 (2005) 7. P´erez-L´ opez, D., Contero, M.: Delivering educational multimedia contents through an augmented reality application: a case study on its impact on knowledge acquisition and retention. Turk. Online J. Educ. Technol. TOJET 12(4), 19–28 (2013) 8. de Educaci´ on P´ ublica, S.: Principales cifras del sistema educativo nacional 20162017 (2017). https://tinyurl.com/ssvucuf 9. Rodr´ıguez, M.A.G., Arteaga, H.U., Altieri, I.M.S., Ulloa, S.M.H.: Estrategias para disminuir el rezago de reprobaci´ on en los programas educativos de la unidad acad´emica de contadur´ıa y administraci´ on de la universidad aut´ onoma de nayarit. EDUCATECONCIENCIA, 21(22) (2019) 10. Arcos-Vega, J.L., Marentes, R., et al.: Information and communication technologies (ICT) and their relation to academic results indicators in state public universities in Mexico. High. Educ. Stud. 7(2), 1–6 (2017) 11. Diegmann, P., Schmidt-Kraepelin, M., Eynden, S., Basten, D.: Benefits of augmented reality in educational environments-a systematic literature review. Benefits 3(6), 1542–1556 (2015) 12. Ak¸cayır, M., Ak¸cayır, G.: Advantages and challenges associated with augmented reality for education: a systematic review of the literature. Educ. Res. Rev. 20, 1–11 (2017) 13. Amaya, P.P., S´ anchez, J.R., DeMoss, V.G., Carre´ on, A.M.: Aplicaci´ on de realidad aumentada en la ense˜ nanza de la f´ısica. Cultura Cient´ıfica y Tecnol´ ogica, 51 (2016) 14. Chen, C.H., Chou, Y.Y., Huang, C.Y.: An augmented-reality-based concept map to support mobile learning for science. Asia Pac. Educ. Res. 25(4), 567–578 (2016) 15. Chao, W.H., Chang, R.C.: Using augmented reality to enhance and engage students in learning mathematics. Adv. Soc. Sci. Res. J. 5(12) (2018) 16. Di Mascio, T., Gennari, R., Melonio, A., Tarantino, L.: Supporting children in mastering temporal relations of stories: the terence learning approach. Int. J. Distance Educ. Technol. (IJDET) 14(1), 44–63 (2016) 17. Davis, F.D.: Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. 13(3), 319–340 (1989) 18. Chang, C.C., Yan, C.F., Tseng, J.S.: Perceived convenience in an extended technology acceptance model: mobile technology and English learning for college students. Australas. J. Educ. Technol. 28(5) (2012)

184

J. Pigueiras et al.

19. Likert, R.: A technique for the measurement of attitudes. Archives of psychology (1932) 20. Cochran, W.: Sampling Techniques. Wiley, New York (1953). Statistical Surveys E, Grebenik and CA Moser (1963) 21. Israel, G.D.: Determining sample size (1992)

Supporting the Construction of Learning Paths in a Competency-Based Informatics Curriculum Luca Forlizzi(B) , Giovanna Melideo, and Cintia Scafa Urbaez Vilchez Department of Information Engineering, Computer Science and Mathematics, University of L’Aquila, Via Vetoio loc. Coppito, 67100 L’Aquila, Italy {luca.forlizzi,giovanna.melideo}@univaq.it [email protected] Abstract. The design of learning activity and paths in informatics is made difficult on the one hand by the absence of a well-established tradition of teaching computer science and on the other by the fact that in many countries the official curricular recommendations are moving towards a competency-based model, increasing the responsibility of teachers. Motivated by the current state of informatics education in Italy, this work presents a system realized to support the research and classification of teaching activities in order to better select the most appropriate ones for the achievement of predetermined learning objectives. A wizard is also provided that allows teachers to use selected activities to design articulated learning paths. The system is designed to be used in the context of a community of practice involving teachers and experts in informatics education, in which innovative learning activities and paths are proposed, validated and shared. Keywords: Informatics education · Competency-based learning · Classification of learning activities · Design of learning paths · Teachers support

1

Introduction

The pervasiveness of informatics1 and information and communication technologies increasingly influences teaching, scientific research, professions and many areas of daily life, making it necessary to include its teaching in training processes. Knowledge of the fundamentals of informatics has a dual role in teaching: on the one hand, a cultural and educational role of basic scientific discipline (alongside mathematics and sciences); on the other hand a role of conceptual tool which provides an additional point of view, complementary to that of other subject areas, to analyze and deal with situations and phenomena. Therefore, 1

This paper uses the term “informatics”, that is more common in continental Europe than “Computer Science”.

c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 185–194, 2020. https://doi.org/10.1007/978-3-030-52538-5_19

186

L. Forlizzi et al.

the study and understanding of the concepts, methodologies and skills of informatics as an independent subject area contribute to forming and enriching the cultural, technical and scientific background of each person [8,15,16]. Several countries have integrated informatics teaching into their education policies starting from primary school. The USA launched in 2015 the “Computer Science for All” initiative aiming to insert informatics into school education on a par with other scientific and technological disciplines [14]. In the United Kingdom “Computing” is a compulsory subjects at all school levels since 2014 [2,4]. In Italy, the objective to introduce computational thinking and coding into primary and lower secondary schools is explicitly stated in the school reform approved by the Italian Parliament in 2015 [9], whose operational plan for what concerns digital technology is defined in the associated National Digital School Plan [11] (for a more comprehensive description of the state of informatics education in Italy, see [1]). Nevertheless, although coding practice is a fundamental activity to learn informatics, it represents only a first step towards a solid scientific education in informatics that must start from the first years of school. The National Interuniversity Consortium for Informatics (CINI), the main point of reference for the Italian academic research in the fields of Computer Science, Computer Engineering, and Information Technology, has therefore made a proposal (hereafter referred to as the CINI Proposal) meant to contribute to the development of informatics education in the Italian school [3]. The proposal is the outcome of a long process, which has also benefited from important contributions of pedagogists and experienced school teachers who took part in the discussion, and can currently be considered the most authoritative reference for instructional design regarding informatics in the context of the Italian school system. However, the operational possibility of schools to use the indications contained in the proposal as an effective tool to plan concrete learning paths on fundamental principles of informatics is severely limited for two main reasons: – In primary and lower secondary schools, informatics is not yet formally established as autonomous subject area; it has never before been taught at these school levels and it is often confused with digital literacy2 , which is included in other subject areas. Hence the implementation of the curricular recommendations is to a large extent responsibility of each school that, in accordance with the degree of autonomy introduced by the reform, must make time to devote to informatics from the already crowded school calendar. – The majority of primary and lower secondary school teachers did not receive formal education on the principles of informatics and indeed, often did not even do scientific studies. Moreover, there is no tradition of or experience with teaching informatics, hence the teachers do not have methodological materials, and the students do not have textbooks. Previous aspects make the introduction of this new subject area more difficult. Starting from these premises, in [7] the authors presented several actions to 2

The term Digital Literacy denotes practical skills in the use of computers, software and other digital tools every citizen should be familiar with, to live in the 21st century, as described in more detail, for instance, in [13].

Supporting the Construction of Learning Paths

187

contrast the critical points previously highlighted. These actions subsequently materialized in a new teaching support system aimed at designing innovative courses for the informatics education in line with the learning objectives outlined in the CINI Proposal, which is the subject of the present work. The system allows not only to catalogue activities (new or available on the internet) with respect to the prior knowledge needed to carry out them and to a set of learning objectives, but also to guide teachers in selecting the activities and designing learning paths appropriate to their own school context. It is fully configurable and therefore adaptable with respect to the legislative evolution of the Italian school system, which could hopefully include the foundations of informatics as an independent subject area in the primary and secondary school curriculum, by defining appropriate learning goals and objectives. The system can be easily applied to other school subjects by taking care of the choice of appropriate content areas and learning objectives. The current state of informatics education in Italy has been one of the main sources of motivation for this work, and therefore Sect. 2 provides a brief description of it together with some other background information. Sections 3 and 4 describe, respectively, the management of learning activities and paths; Sect. 5 describe the validation process implemented within the community of practice; finally, Sect. 6 concludes the paper and outlines possible future developments.

2

Background

This section provides some information that helps to put the work in perspective. The recent trends in pre-tertiary education focus the curriculum design on skills and competences to be acquired in broad areas (for instance, see [5]). In Italy, since their latest issue [10], the curricular recommendations for the primary and lower secondary schools organize, for each subject area, disciplinary knowledge in a set of content areas. For each content area, the recommendations define a set of learning objectives, structured in two levels of details. The learning objectives indicate the main knowledge and competences that students are expected to acquire at certain milestones. In such a competency-based context, the construction of learning paths or complete curricula is guided by the preliminary choice of a set of well-defined learning objectives, structured according to dependency relationships. The CINI proposal for a national informatics curriculum in the italian school, although not an official MIUR document, was suitably organized in accordance with this competency-based model. Five content areas are identified: algorithms, programming, data and information, digital creativity, digital awareness. See [6] for an extended summary of the proposed curriculum that highlights the key underlying motivations, and outlines a possible strategy to ensure that its implementation in schools can be effective. Unlike other subject areas, informatics is not yet a consolidated presence in schools, especially at the primary and lower secondary level. Therefore, there is no teaching tradition that has consolidated contents, methods, and teaching materials of informatics education in schools. This puts most teachers, who rarely

188

L. Forlizzi et al.

have specific disciplinary skills, in serious difficulty. Conversely, a large quantity of ready-to-use resources to teach informatics are available on the internet, most for free. Most of them are the product of computer enthusiasts or passionate teachers who have seized the opportunity offered by the Internet to spread the results of their informatics education experiments. This abundance of resources is an opportunity that schools could take to alleviate teachers’ difficulties. However, there are serious difficulties to overcome. First, finding appropriate activities is more difficult than it may seem because the quality of online resources varies widely. Moreover, many resources are not designed for school contexts: they require resources and make unrealistic assumptions in the school environment. A second difficulty is that most of the online resources are made up of single didactic activities, aimed at specific learning objectives and not included in coherent and articulated learning paths with respect to an overall didactic planning. The construction of articulated and coherent learning paths with respect to a set of learning objectives is an arduous task, which requires extensive knowledge of informatics and methodologically correct approaches [12]. Therefore in order for the schools to take advantage of the available resources, it is necessary to provide teachers with tools to assist them in selecting, adapting and integrating these resources with respect to their own school context, also by making use of the advice of experts in informatics education. This is the goal of the system described in the following sections. Indeed, as explained below, teachers are guided step by step, through the system, in the research of activities validated by experts in informatics education and in the design of learning paths coherent with respect to a set of prerequisites already met and to a set of learning objectives to be achieved.

3

Resource Catalogue

The system presented in this article, is aimed to help educational planning in a competency-based curricular context. The kernel of the system is a catalogue of learning activities classified with respect to the competency-based curriculum determined by a set of pre-established learning objectives. The set of learning objectives is partitioned in several content areas. A detailed analysis was carried out, that highlighted that most of the learning activities available on the internet do not explicit the content areas, the specific learning objectives to achieve and the set of prerequisite objectives necessary to carry them out. Therefore, a lot of careful work has been focused on determining and recording for each reviewed activity the following information: – the type of activity, distinguishing between “unplugged” and “plugged” activity, that is an activity which can either be used alone or with other digital devices; – the content areas activity; – the estimated activity duration;

Supporting the Construction of Learning Paths

189

– a set of learning objectives associated with the activity, consistent with the content areas; – possible preliminary activities to carry out before the new activity could be introduced; – recommended grade to carry out the activity; – possible interdisciplinary connections with other curricular subjects; – the list of teaching materials needed to carry out the activity; – possible additional support material. The information above allows users to filter single learning activities and design through the system wizard appropriate learning paths according to the current educational framework. The most important kind of information, given the competency-based setting, is the association of activities with learning objectives, both those whose achievement is a prerequisite to the fruition of the activity and those that should be achieved by means of the activity (see Fig. 1). The accuracy of this association is crucial for the effectiveness of the system, therefore it is highly advisable that it be carried out by competent experts, as discussed in Sect. 5. The system allows users to search the activity catalogue by filtering with respect to the learning objectives one intends to achieve. In particular, a web interface provides users with a guided procedure to match learning activities with the set of learning objectives that those activities should allow to reach. The procedure works as follows: 1. users choose a content area so as to retrieve a set of learning objectives; 2. users have to select one or more objectives in order to discover which are the activities mapped into those objectives; 3. finally, users obtain a set of activities and the other useful information about them listed above. The tool is currently being tested taking as reference the curriculum defined by the CINI proposal. At this time more than 70 online activities have been collected and reviewed, drawn from two sources of widely recognized educational and scientific value: “Programma il Futuro” 3 and “Computer Science Unplugged” 4 .

4

Learning Paths Design

Building good learning paths is one of the most complex stages of teaching planning, even when well-defined, good quality teaching activities are available. In a competency-based curricular framework is even more difficult than in a traditional, knowledge-driven, setting, as it must be aimed at achieving multiple 3

4

“Programma il Futuro” is a initiative put in place by MIUR and CINI to promote informatics education in Italy, by offering ready-to-use teaching resources, many of which are the Italian translation of materials designed for the Code.org organization. https://csunplugged.org/en/.

190

L. Forlizzi et al.

Fig. 1. Mapping between activities and learning objectives in the system.

objectives, which it is often not correct to force in a fixed sequence defined in the same way for all learners. Some sources of teaching materials, such as Code.org, propose several structured learning paths, but these might not be adequate for all educational contexts or teachers’ needs. Furthermore, it is unlikely

Supporting the Construction of Learning Paths

191

that they will be able to cover exactly the set of learning objectives prescribed by a given curriculum: it is possible that they omit some learning objectives or that they deepen some topics more than what is intended by a curriculum, shifting its focus. Therefore the design of appropriate learning paths is a task that most teachers cannot avoid. Having available, as is usually the case for informatics teaching, ready-to-use teaching activities, designing a learning path means arranging activities in a correct temporal order with respect to learning outcomes, dependencies between activities, and total duration of the learning path. To aid teachers, the system experiments with the idea of employing a pair of graphs as easy-to-understand conceptual and practical tools: – a graph of dependencies among learning objectives, that expresses the (partial) precedence order in which they were reached; – a graph of dependencies among activities, that expresses precedence constraints. A precedence constraint between two teaching activities may be due to different reasons. One reason may be that the piece of knowledge associated with one activity is a prerequisite to that associated with the other. Note that a dependency constraint between learning objectives determines dependency constraints among activities related to those objectives. Another reason for the existence of a precedence constraint between two activities is operational: several activities use tools and devices whose usage has been introduced by other activities, even in the absence of a dependence between the relative learning objectives. A learning path is considered feasible if it is consistent with the desired learning objectives and the dependency constraints among activities i.e., if activities are carried out according to the correct temporal order derived from the precedence order among objectives and activities defined in the two graphs. A correct definition of “prerequisites”, i.e. activity requirements, allows to select a sample of activities of interest, filtering them by grade, learning topics/outcomes, time duration, interdisciplinary links and so on. This can facilitate the design or validation process of paths, even with a growing number of reviewed and classified activities in the dependency graph. The definition of prerequisites allows for focusing on the activities sample dependency graph, i.e. the subgraph induced by the activities in the considered sample, which becomes more tractable the more accurate are the requirements of the learning path. This remark assumes more relevance when automating the learning path creation process. The system contains a wizard for designing learning paths. The user proceeds by first selecting a grade and one or more content areas, in order to retrieve the corresponding set of learning objectives. Then, the user chooses those objectives that the learners have already achieved and the ones that are intended to be achieved in the new learning path. Based on the selection, a set of suitable activities is automatically extracted from the catalog and proposed to the user, who can review and choose according to their needs. The system applies topological ordering to the set of chosen activities so as to present them in a way that respects the defined precedence constraints. Finally, the user can modify the

192

L. Forlizzi et al.

proposed order by dragging and dropping activities from the computed path. During this phase, the system checks dependency constraints and notifies the user only when a modification on the proposed path has produced a violation. It is worth pointing out that this wizard allows users to autonomously design learning paths in informatics even without having a strong scientific background.

5

User Roles and Functionalities

The effectiveness of a didactic planning support system depends crucially on the quality of the resources made available to teachers. Therefore these resources must be prepared by qualified experts in informatics education. On the other hand, teachers certainly find more useful a system that does not limit itself to offering already prepared resources (activities, learning paths), but which allows the insertion and construction of new resources. It is possible to balance these needs by allowing user roles with different capabilities. The system allows to manage users with three different access roles: administrators, experts and teachers, distinguished according to the content they have access to and the operations allowed to them. The main idea is that all the learning objectives, activities and paths that the system makes publicly available must have been previously validated. Each user can access all public material, but in relation to the creation of new materials, he has capabilities that depend on his role in the system. Teachers can create new activities or new paths using the wizard described in Sect. 4, but these new resources are not automatically published and made available to other users. Teachers willing to share resources they produce with other users, could submit an evaluation request. The request evaluation task is assigned to administrators: they use experts’ opinions in order to decide whether to publish the new resources or not. In this regard, the system allows administrators to select a set of reviewers from a list of experts, which are in charge of the review process. The chosen reviewers that receive a submission request notification have a fixed amount of time to answer with a feedback, that could be positive or negative, associated with a text comment. At the end administrators examine the received feedbacks to make a final choice. In detail, with regards to features and related permissions, the system is organized as follows: – Content areas and learning objectives: The consultation is open to all users while classification and management are restricted to administrators only. – Learning activities: The consultation is open to all users, while the constant updating of the database is entrusted only to experts and administrators: each activity in the classification phase must be accompanied by mapping with prerequisites, cultural areas and learning objectives, plan didactic and other operational information relating to recommended age, estimated time of development, interdisciplinary links with other subjects, supplementary material. Teachers can keep the history of the activities displayed in the

Supporting the Construction of Learning Paths

193

database and possibly forward to the administrator a request for evaluation of a new activity. – Learning paths: The consultation of published learning paths and the guided procedure for defining new paths, starting from the prerequisites already met and the objectives to be achieved, are open to all users. The publication of new learning paths is entrusted to experts and administrators. A teacher can save the displayed public paths or those generated with the wizard in his private space; if necessary, he can send administrators a request to evaluate a new learning path. – Administrators can delegate the task of evaluating new learning activities and paths to one or more experts. In addition to the expert evaluation procedure described above, the system also allows teachers’ opinions to be taken into consideration. Every teacher is allowed to express public comments and personal feedbacks about consulted teaching resources, which could in turn be used by other teachers to filter activities.

6

Conclusions and Future Works

This paper describes a system to support didactic planning in the context of a competency-based informatics curriculum. The goal is helping teachers reach autonomy in selecting and validating existing learning paths, or in designing new ones according to their scholar context. The system has been presented to an academic working group on informatics education in Italian schools, raising interest. Some members of the working group volunteered as expert user to help populate the system’s resource catalog. An experimental evaluation of the system will shortly be carried out with a group of primary school teachers, in order to validate its use and to check whether it is necessary to provide the system with more functions and didactic material. As already mentioned, the tool can be configured for a different set of learning objectives and it can be adapted for a different competence-based international or curricular school context. Acknowledgements. This work was partially developed within the framework of the Informatics and School National Laboratory of CINI (National Interuniversity Consortium for Informatics). The authors are thankful to Paolo Tramontozzi and to Stefano Florio for implementing several features of the system as part of their bachelor’s thesis.

References 1. Bellettini, C., Lonati, V., Malchiodi, D., Monga, M., Morpurgo, A., Torelli, M., Zecca, L.: Informatics education in italian secondary schools. TOCE 14(2), 15:1– 15:6 (2014) 2. Computing at School Working Group: Computer science: A curriculum for schools (2012). www.computingatschool.org.uk/data/uploads/ComputingCurric.pdf

194

L. Forlizzi et al.

3. Consorzio interuniversitario nazionale per l’informatica (CINI): Proposal for a national informatics curriculum in the Italian school (2017). https://www. consorzio-cini.it/images/PROPOSAL-Informatics-curriculum-Italian-school.pdf 4. Department for Education: National Curriculum for England: Computing programme of study. Technical report (2013). https://www.gov.uk/government/ publications/national-curriculum-in-england-computing-programmes-of-study/ national-curriculum-in-england-computing-programmes-of-study 5. European Parliament: Recommendation 2006/962/ec of the European parliament and of the council of 18 dec. 2006 on key competences for lifelong learning (2006) 6. Forlizzi, L., Lodi, M., Lonati, V., Mirolo, C., Monga, M., Montresor, A., Morpurgo, A., Nardelli, E.: A core informatics curriculum for Italian compulsory education. In: Pozdniakov, S.N., Dagiene, V. (eds.) Informatics in Schools Fundamentals of Computer Science and Software Engineering ISSEP 2018. LNCS, vol. 11169, pp. 141–153. Springer, Cham (2018) 7. Forlizzi, L., Melideo, G., Rosa, G., Scafa Urbaez Vilchez, C.: Dalla proposta di indicazioni nazionali per l’insegnamento dell’informatica ai percorsi formativi: Strumenti operativi per la scuola primaria. In: Didamatica 2019 Informatica per la Didattica, Atti del Convegno, pp. 105–114 (2019) 8. Hromkovic, J.: Contributing to general education by teaching informatics. In: Mittermeir, R.T. (ed.) Informatics Education - The Bridge between Using and Understanding Computers, ISSEP 2006. LNCS, vol. 4226, pp. 25–37. Springer, Cham (2006) 9. Legge 13 luglio 2015, n. 107: Riforma del sistema nazionale di istruzione e formazione e delega per il riordino delle disposizioni legislative vigenti (2015). https:// www.gazzettaufficiale.it/eli/id/2015/07/15/15G00122/sg 10. MIUR: Indicazioni nazionali per il curricolo della scuola dell’infanzia e del primo ciclo d’istruzione (2012). http://www.indicazioninazionali.it/wp-content/uploads/ 2018/08/Indicazioni_Annali_Definitivo.pdf 11. MIUR: Piano nazionale scuola digitale (pnsg) (2015). http://www.istruzione.it/ scuola_digitale/allegati/Materiali/pnsd-layout-30.10-WEB.pdf 12. Rich, K.M., Strickland, C., Binkowski, T.A., Moran, C., Franklin, D.: K-8 learning trajectories derived from research literature: sequence, repetition, conditionals. Inroads 9(1), 46–55 (2018) 13. The Committee on European Computing Education (CECE): Informatics education in Europe: Are we all in the same boat? Technical report (2017) 14. The White House: Report on the initiative “computer science for all”. Technical report (2016). https://www.whitehouse.gov/blog/2016/01/30/computer-scienceall 15. Wing, J.M.: Computational thinking. Commun. ACM 49(3), 33–35 (2006) 16. Wing, J.M.: Computational thinking: What and why? The Link Magazine (2011)

Personalized Recommender System Using Learners’ Metacognitive Reading Activities Lydia Odilinye(B) and Fred Popowich Simon Fraser University, Burnaby, BC, Canada {lodiliny,popowich}@sfu.ca Abstract. Learning, an active cognitive activity, differs from one learner to another, suggesting the need for personalized learning. The development of personalized recommender systems typically involves a learner model component, which is used to capture and store the personal information, preferences and other characteristics of the learner. While reading, learners engage in number of metacognitive activities e.g. text marking/highlights. These metacognitive interactions could serve as useful information for the learner model, to achieve personalization. The recommender system developed is integrated with nStudy, an online learning platform that provides a number of annotation tools (e.g. highlighting, tags) that support metacognitive activities. A user study was conducted to evaluate the effectiveness of using the highlights (a metacognitive activity) a learner makes while reading, as a preference elicitation method for the learner model. The findings show that the learner generated metacognitive activities while reading serve as an appropriate input mechanism to guide personalized learning recommendations. Keywords: Personalized recommender system activities · Technology enhanced learning

1

· Metacognitive reading

Introduction

Teaching and learning are activities that humans have performed throughout time. These activities have been influenced by advances in technology, and digital systems of today facilitate interactivity which presents content (e.g. text, image) in a variety of media, and can respond to the learners. Technology Enhanced Learning (TEL) according to Duval et al. (2017), harnesses the power of interactivity provided by digital systems and has the potential to enhance what is learned, how we learn and teach. TEL therefore can be described as the application of information and communication technologies to support and enhance all forms of teaching and learning activities (Kirkwood and Price 2014). With the rapidly increasing amount of learning materials and resources available online, it is becoming more difficult for learners to find appropriate information or learning material to satisfy their needs. Many studies report information c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 195–205, 2020. https://doi.org/10.1007/978-3-030-52538-5_20

196

L. Odilinye and F. Popowich

overload as one of the main problems that learners encounter in online learning and when searching for the “right” information to satisfy their needs (Manouselis et al. 2011). Searching for relevant information is considered a pivotal activity in teaching and learning (Drachsler 2015). Therefore, in the context of TEL, a recommendation system (technology) is considered a resourceful software tool that could be used to identify interesting learning materials from a large pool of resources. Also, recommendation systems are able to reduce the burden of information overload by recommending the “right” information at the right time and in the right format (media) of the learner’s interest. Learning is an active cognitive activity that differs from one learner to another (Shishehchi et al. 2011); each learner has individual needs and particular requirements. Some learners may be highly self-motivated and learn by exploring while other students prefer some specific guidance in a structured way. Therefore, the development of a personalized learning recommender system that would cater to the peculiarity of each learner in TEL is considered important. Personalized learning describes the search for, and the recommendation of, potential learning activities that are the most suitable to the individual learner (or learner group) (Drachsler 2015). Personalized learning is said to occur when the learning activities has been designed to fit the needs, goals, talents, and interests of the learners (Klasnja-Milicevic 2011). With respect to recommendation in the TEL environment, the concept of metacognition can be captured by the strategic selection of recommended learning materials and the strategic processing of the selected materials (Zhou and Xu 2012). There exists a number of metacognitive activities a learner could engage in while learning, however, we examined the metacognitive activities related to reading comprehension. This is because it constitutes the most common context where learning occurs (Zhou and Xu 2012). Based on the metacognitive activities associated with reading, recommendations could be made to facilitate comprehension, recall and deeper text processing. Some metacognitive reading strategies include: organization, note taking, underlining/highlighting are considered important learning strategies that are focused on the reading activity, and are essential to the learning process. Furthermore, they could help a learner find connections within a body of new information; pay attention to, encode new material and provide external storage of information for later studies; determine portions of a body of text that are important to learn and what is trivial respectively (Ormrod 2012). These metacognitive and learning strategies also support the notion of personalization, because the strategies allow the learners to take control over the learning process. In this study, we developed a personalized learning recommender system which makes use of the information from the metacognitive activities that the learner engages in while reading is used to build the learner model and guide recommendations.

2

Related Work

The basic idea behind personalized learning recommendation is the need to provide recommendations that meet the specific learning needs and preferences of

Personalized Recommender System Using Metacognitive Activities

197

the learner, in order to enhance the learning experience. The development of personalized learning recommender systems of necessity includes a learner model, which is used to obtain/infer information about the learner. The learner model is used to capture information about the learner’s characteristics such as the learning goal, learning style, prior knowledge, and the information obtained is used to guide recommendations. Broadly speaking, the information about the learner to create a learner model can be obtained explicitly or implicitly, to achieve personalized learning recommendations. The explicit methods for obtaining information to guide recommendation entails the learners specifying their learning goals, prior knowledge, and preferences through competency analysis, questionnaires, essay writing, among others. Techniques such as concept maps, ontologies, fuzzy logic theory can be used to represent the data to enable the computation of recommendations. Okoye et al. (2012) developed a recommender system that deployed concept maps to represent the learner’s characteristics, which is generated automatically from an essay the learner writes. The system also diagnoses the learner’s incorrect, incomplete and fragmented conceptual knowledge by comparing the learner’s concept map with a reference domain concept map. Recommendations are then generated by the construction of a concept graph for each of the concept maps, and computing the maximum sub-graph as the similarity measure, aimed at addressing the learner’s misconceptions. Shen and Shen (2004) developed a personalized recommender system which made use of ontologies to represent the information about the learner’s competency as well as the learner’s objective (learning objective competency) by competency gap analysis. A domain ontology was also created which involved splitting the learning materials into smaller units (e.g. based on the concepts/topics in the resource), defining the interrelation and dependencies of the concepts. Recommendations are generated by also performing competency gap analysis which compares the learner’s competency and the learning objective competency. Lu (2004) proposed a personalized learning recommender system that recommends suitable learning materials to all learners with different learning style, learning needs and knowledge background of the learners. The system framework made use of questionnaires and tests to obtain the learner information and to identify the learner requirement. Fuzzy matching rules are then applied to discover associations between learner requirements and the learning materials in the repository to generate recommendations. Implicit methods for creating a learner model involves analyzing the (Web) navigational history/traces of the learner. Web usage mining, (which performs mining on web data, particularly data stored in logs managed by the web servers, which provides raw traces of the learners’ navigation and activities on the website) and collaborate filtering techniques have been used for the analysis of the learner information collected implicitly. Khribi et al. (2009) proposed the structure for an automatic personalization approach aiming to provide online automatic recommendations for active learners without requiring their explicit feedback. Recommended learning resources are computed based on the current learner’s recent navigation history, as well as exploiting similarities and

198

L. Odilinye and F. Popowich

differences among learners’ preferences and educational content. The recommendation process begins with an offline mining of the learners’ models based on Web usage mining techniques, which clusters the data on the learners’ web sessions. Self explanation, a metacognitive reading strategy, which describes the process of explaining text to one’s self either orally or in writing is the only metacognitive reading strategy that has been implemented to date for educational purposes in self tutoring platforms. A number of systems have been developed to encourage and initiate self-explanation while reading for a number of domains. Self-Evaluation Coach (SE-Coach) developed by Conati and VanLehn (2000) is a scaffolding tool meant to encourage students to spontaneously self-explain. The system produces simple prompts to initiate self-explanations in terms of domain principles, to encourage deeper understanding. NORMIT, a constraint-based tutor developed by Mitrovic (2003) teaches data normalization and supports self-explanation. The system requires an explanation from users for each action that is performed for the first time or if done. The results of the user study performed by Mitrovic (2003) revealed that self-explanation increased problem solving skills and better conceptual knowledge. These results show the benefits of self-explanation as a powerful and useful metacognitive reading technique to enhance learning. Therefore, we examined the use of other metacognitive reading activities and strategies to facilitate personalized learning and recommendations.

3

Our Approach

To develop a recommender system aimed at providing personalized recommendations that facilitates learning, we examined using the metacognitive activities a learner may engage in while reading as the preference elicitation method that is representative of the learners’ interests and preference. Some of the metacognitive activities a learner may engage in while reading include: creating bookmarks, highlights, note taking, creating tags, among others. We view these activities as a reflective practice that could provide insight to the learner’s comprehension, and could also reveal the information seeking needs of the learner. Therefore, the learners’ interactions serve as input to the recommender system, for appropriate recommendations. Figure 1 below shows the data flow of the system. The recommender system is intended to assist learners’ select appropriate textual documents for task-oriented reading. Task-oriented reading is an activity; where an individual reads to meet a goal (McCrudden and Schraw 2007), which may be provided by an instructor or a self-directed reading goal. Such readings may involve multiple documents. Task-oriented reading involving multiple documents therefore requires the ability to search for and identify relevant resources that facilitate the completion of the reading task. In this context, a recommender system is considered a resourceful tool that could be used to identify relevant documents from a large pool of documents. A recommender system for educational purposes therefore should be tailored to support the learners’ information seeking needs as well as enhance the learners’ learning experience.

Personalized Recommender System Using Metacognitive Activities

199

Fig. 1. Pictorial description of the system and data flow.

The use of the learners’ metacognitive activities as input to the recommender system, to guide recommendations is intended to replace search queries. Typically, the information seeking needs of a learner are expressed using a search query in a search engine. However, search queries do not always return relevant information, due to reasons such as poor queries and natural language ambiguities (Batista 2007). Therefore, a more cognitive approach to identifying the learners’ information needs can be achieved by examining the interactions the learner engages in with the learning materials. The metacognitive activities of the learner (e.g. highlights, notes, comments) usually contain more information than search queries text, this gives the learner the freedom to express his/her information seeking needs in different ways without limitations. Using the metacognitive activities of learners as input to the recommender system could also imply that the information contained in the learners interaction may span multiple topics, which with the use of search queries may require multiple queries from the learner. The learning platform that was used is nStudy (Beaudoin and Winne 2009). nStudy is an online Web application that offers learners a wide array of tools for identifying and operating on information they study. nStudy is designed to provide learners and researchers tools to explore their learning skills, metacognition and self-regulated learning. It provides a wide range of metacognitive reading tools that allows a learner interact with an online document or web page the way they would with a paper version. Some of the tools nStudy provides includes: highlights (text marking), notes taking, bookmarking, and tags. nStudy provides the annotation tools that allows learners interact with an online document as they would with a paper version. The annotation tools allow the learners record, organize, view the documents they read. Some of the annotation tools that nStudy provides are highlights (text marking), tags, notes. For this study, the annotation/metacognitive tools that are used are the highlights and tags. The nStudy interface would also be used to display the reading task, task instructions, and the participants would be able to read the articles as well as make annotations on the articles using the interface. As learners use nStudy’s tools to study information on the Internet, nStudy logs fine-grained, time-stamped trace data of all the activities performed by the learner, which reflects the learners’ cognitive and metacognitive events in

200

L. Odilinye and F. Popowich

self-regulated learning. Some of the activities nStudy logs includes bookmarks, the websites visited, notes created, as well as the information operated on (e.g. text highlighted, tags). To obtain data on the learners’ interaction, the recommender system developed is integrated with nStudy. Given the existing system nStudy that supports metacognitive activities, the recommender system was designed as a plug-in and integrated to the nStudy learning platform (it could also be extended to other online learning platform that support metacognitive reading activities). This allows for real time reception and collection of all the (metacognitive) events, activities and interactions the learner performs while reading. Below, the steps on how to use the system is outlined. 1. Create an account or sign into the nStudy platform 2. Commence the reading/learning activity – Create highlights and tags/perform metacognitive activities – The metacognitive activities are sent to the recommender system 3. Step 3: The learner requests for recommendation – A list of documents is provided based on the metacognitive activities The learner is first required to either create an account or sign in. This step is important because for each learner, the system stores and records all the activities the learner performs while using the system to achieve personalized recommendations. After the login phase, the learner may commence searching for documents, reading and interacting with the documents read. While reading, a learner may make highlights or create tags on nStudy, the details of the portion of text highlighted and/or tag created are sent to the recommender system in real time. Based on the metacognitive activities of the learner the recommender system receives, it is able to make personalized recommendations that are tailored to the learner’s interests. In this case, the feedback obtained from the learner serves as a prompt (search query) that signals to the recommender system the kind of items the learner may be interested in. In general, recommender systems provide recommendations either on request or automatically when certain conditions are satisfied. Given the recommender system and learning platform are designed to support self-regulated learning, in which the learner is in charge of the learning process, the system to provides recommendation when the learner requests for it. That is, to receive recommendations, the learner has to request for it, by clicking the request recommendation button provided in the system interface. The metacognitive activities used to guide recommendation in this system design are the highlights the learner makes while reading. Upon creating at least one highlight, the learner may request for recommendations at any time. The list of recommended articles generated is opened on a new tab which contains documents related to the highlights the learner had made. The new tab displays the recommendations in this format: the title of the document, its URL, a dynamic summary (for each of the documents) and an explanation module – a set of keywords extracted from the highlights the learner created based on which the recommendation was generated. The learner may repeatedly perform step 2 and 3 until they are satisfied with the highlights they have put together to complete the reading task.

Personalized Recommender System Using Metacognitive Activities

4

201

User Study and Experimental Design

A user study was conducted to investigate the suitability and effectiveness of using metacognitive activities (learner generated text markings/highlights) to represent the learners’ characteristics, preferences for the learner model creation and to guide recommendations. The user study entailed an educational task – a reading activity. The educational task required the participants complete two essay-like questions after reading some online news articles. To complete the task, instead of writing an essay, the learner is expected to create an ensemble of highlights in lieu of writing an actual essay to complete the educational task of the study. For reasons such as different writing capabilities of learners in general, we decided not to ask the learner to write actual essays, but to put together highlights from the (recommended) articles they read that would be adequate to complete the tasks. To get started, the learner is presented with the list of starting articles for the task. These randomly chosen articles are the initial documents the learner reads and interacts with to commence reading session. As the learner reads the articles, they perform metacognitive reading activities (text marking, tags). There may be various reasons a learner creates a highlight. Therefore, we adopted the use of tags to allow the learner to specify the reason for the highlight. Also, given the study required participants of the user study to create two types of highlights: highlights (metacognitive activities) to guide recommendations, and an ensemble of highlights to complete the educational task, tags were also used to distinguish between the types of highlights that the participants would create. The experimental design of this study entailed two groups: (i) experimental recommendation (Group A), and (ii) random recommendation (Group B). The participants of the study were 49 undergraduate students, 27 females and 22 males with various disciplinary majors attending a university in Western Canada. Ages ranged from 18 to 26 years (M = 21, SD = 2.67). All participants were recruited via an advertisement posted on campus. The participants were randomly assigned to one of the two groups; Group A had 25 participants while Group B had 24. Both groups received recommendations from the collection of news articles used in the study. However, the random recommendation group received randomly selected recommendations using a random number generator module, while the experimental recommendation group received recommendations based on the learners’ metacognitive activities. Two questionnaires were included in the study, a demographics questionnaire, used to obtain some personal information of the participants (e.g. domain knowledge), and a feedback questionnaire where the participants are asked to evaluate the recommender system. The demographics questionnaire is presented before the task, while the feedback questionnaire is administered after the participant completes the task. Likert scale was used to design questions in the feedback questionnaire, and participants were expected to select a rating on a scale that ranges from “strongly agree” to “strongly disagree.”

202

5

L. Odilinye and F. Popowich

Results and Findings

To analyze the data collected, we adopted the user-centric framework for evaluating recommender systems by Knijnenburg et al. (2012). The framework provides insight into the relationships between the general concepts that play a role in the user experience of recommender systems and consists of six interrelated conceptual components. However, we limit the analysis of this study to three components: (1) Objective System Analysis (OSA), (2) Subjective System Analysis (SSA), (3) User experience (EXP). The OSAs are the aspects of the system that are to be evaluated (e.g. algorithm, input mechanism). The SSAs are regarded as the mediating variables that attempt to explain the effects of the OSAs on the user experience. SSAs are measured using questionnaires administered to the participants during or after interacting with the recommender system. The measurements help establish whether the users perceive aspects of the OSA, independently of their evaluation of the aspect. EXP is the user’s self evaluation of the effectiveness of the different aspects of the recommender system and is also measured with questionnaire. The SSA measures the users’ perceived recommendation quality, a subjective measure of the relevance of the recommendations the system provides, which measures the participants’ perception of the quality of the recommendations received, based on the methodology used in the experimental conditions (item ex: the system provided valuable recommendations). The EXP measures two variables: perceived system effectiveness and recommendation choice satisfaction. Perceived system effectiveness evaluates the ability of the recommender system in providing valuable personalized recommendations (item ex: the recommendations fit my preferences), while recommendation choice satisfaction measure the usefulness the appropriateness of the recommendations for the task (items ex: the recommended articles were appropriate for the task). We first test whether participants in the two groups of the experimental design judge the recommendation quality differently. The mean response to the item measuring this concept is 4.45 for Group A and 2.37 for Group B. The results from independent samples t-test shows that this difference is significant with a large effect size: [t(47) = 3.42, p = .001, r = .419]. Next, to test whether the user experience variables (perceived system effectiveness and recommendation choice satisfaction) are related to the perceived recommendation quality, we performed Pearson correlation tests. The results shows that these concepts are strongly and significantly correlated [r = .891, p < .001] and [r = .813, p < .001]. To determine how the two variables that measure the user experience relate and contribute to each other, correlation and regression analysis were performed. The regression model revealed that for recommendation choice satisfaction variable, perceived recommender system effectiveness is a predictor: [R2 = .461, F(1, 47) = 18.366, p < .001]. Furthermore, the correlation test also showed a positive and significant correlation between the concepts having coefficients of [r = .659, p < .001]. We finally test whether there is a difference in the number of recommendations clicked/read between the two groups. That is, since the participants in Group B received random recommendations, there may be fewer relevant and

Personalized Recommender System Using Metacognitive Activities

203

task-appropriate articles to read and make highlights from, thus the number of articles clicked and read might be fewer compared to Group A participants who receive recommendations that are tailored to the highlights created. The results of the t-test confirms that there is a significant difference between the two groups [t(47) = 5.47, p = 0.01, r = 3.23]; where the average number of recommended articles clicked was 8.86 and 3.74 in Group A and B respectively.

Fig. 2. Graphical presentation of the analysis results.

The results of the data analysis suggests that the learner generated metacognitive activities (in this case marked texts/highlights) are representative of the learners’ preferences and information seeking needs, necessary to build a learner model to achieve, and guide personalized recommendations – which were appropriate to complete the educational task of the study. Using the framework of Knijnenburg et al. (2012), Fig. 2 shows the relationship between the three components examined.

6

Conclusion and Future Work

This study examined the suitability and effectiveness of building a learner model to capture the learners’ preferences (preference elicitation method) from the metacognitive activities performed while reading. A personalized recommender system was developed, and was integrated into a learning platform nStudy which provides annotation tools for learning and supports metacognitive activities. A user study was conducted involving 49 participants to evaluate the system developed. Analyses of the results revealed the suitability and effectiveness of the learner-generated metacognitive activities for the task. Finally, future research could investigate how the integration of other TEL functionalities might be included in a personalized learning recommendation system. Some of the TEL functionalities that may be included to enhance learning are: collaborative learning and question generation module.

204

L. Odilinye and F. Popowich

References Duval, E., Sharples, M., Sutherland, R.: Research themes in technology enhanced learning. In: Duval, E., Sharples, M., Sutherland, R. (eds.) Technology Enhanced Learning: Research Themes, pp. 1–10. Springer, Cham (2017) Kirkwood, A., Price, L.: Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review. Learn. Media Technol. 39(1), 6–36 (2014) Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R.: Recommender systems in technology enhanced learning. In: Rokach, L., Shapira, B., Kantor, P., Ricci, F. (eds.) Recommender Systems Handbook: A Complete Guide for Research Scientists & Practitioners, pp. 387–409. Springer, Cham (2011) Drachsler, H., Verbert, K., Santos, O.C., Manouselis, N.: Panorama of recommender systems to support learning. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook. Springer, Boston (2015) Shishehchi, S., Banihashem, S.Y., MatZin, N.A., Mohd, S.A.: Review of personalized recommendation techniques for learners in e-learning systems. In: International Conference on Semantic Technology and Information Retrieval, Putrajaya, Malaysia, 28–29 June 2011 (2011) Klasnja-Milicevic, A., Vesin, B., Ivanovic, M., Budimac, Z.: E-Learning personalization based on hybrid recommendation strategy and learning style identification. Comput. Educ. 56(3), 885–899 (2011) Zhou, X., Xu, H.: Challenges for educational recommender systems. In: Santos, C.O. (ed.) Educational Recommender Systems and Technologies: Practices and Challenges, pp. 281–289 (2012) Ormrod, J.: Human learning, 6th edn. Pearson Education Inc., Upper Saddle River (2012) Okoye, I., Maull, K., Foster, J., Sumner, T.: Educational recommendation in an informal intentional learning system. In: Santos, O., Boticario, J. (eds.) Educational Recommender Systems and Technologies: Practices and Challenges, pp. 1–23 (2012). https://doi.org/10.4018/978-1-61350-489-5.ch001 Shen, L., Shen, R.: Learning content recommendation service based-on simple sequencing specification. In: Liu, W., et al. (eds.) Lecture notes in Computer Science, pp. 363–370 (2004) Khribi, M.K., Jemni, M., Nasraoui, O.: Automatic recommendations for e-learning personalization based on web usage mining techniques and information retrieval. Educ. Technol. Soc. 12(4), 30–42 (2009) Mitrovic, T.: Supporting self-explanation in a data normalization tutor. In: Supplementary Proceedings of AIED 2003 (2003) Conati, C., Vanlehn, K.: Toward computer-based support of meta-cognitive skills: a computational framework to coach self-explanation. Int. J. Artif. Intell. Educ. 11, 398–415 (2000) McCrudden, M.T., Schraw, G.: Relevance and goal-focusing in text processing. Educ. Psychol. Rev. 19, 113–139 (2007) Batista, H.: 7 Reasons Why Search Engines Don’t Return Relevant Results 100% of the Time (2007). https://moz.com/blog/7-reasons-why-search-engines-dont-returnrelevant-results-100-of-the-time. Accessed March 2019 Beaudoin, L., Winne, P.H.: nStudy: an internet tool to support learning, collaboration and researching learning strategies. In: CELC 2009. http://learningkit.sfu.ca/lucb/ celc-2009/nstudy.pdf

Personalized Recommender System Using Metacognitive Activities

205

Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User-Adap. Inter. 22(4–5), 441–504 (2012). https://doi.org/10.1007/s11257-011-9118-4 Lu, J.: Personalized e-learning material recommender system, pp. 374–379 (2004)

Technology-Enhanced Learning (TEL) in Anaesthesia: Ultrasound Simulation Training for Innovative Locoregional Anaesthesia Vincenza Cofini1(&) , Pierpaolo Vittorini2 , Emiliano Petrucci3, Stefano di Carlo4, Pierfrancesco Fusco3, Franco Marinangeli2,3, and Stefano Necozione1 1

Biostatistics and Epidemiology Unit, Department of Life, Health and Environmental Sciences, University of L’Aquila, L’Aquila, Italy [email protected], [email protected] 2 Department of Life, Health and Environmental Sciences, University of L’Aquila, L’Aquila, Italy {pierpaolo.vittorini,franco.marinageli}@univaq.it 3 Department of Anesthesia and Intensive Care Unit, San Salvatore Academic Hospital of L’Aquila, L’Aquila, Italy [email protected], [email protected] 4 Department of Anesthesia, Resuscitation, Intensive and Pain Care, University of Study of Chieti, Chieti, Italy [email protected]

Abstract. Ultrasound use can improve innovative procedures in locoregional anaesthesia, like myofascial block, increasing the success rate, reducing the risk of complications, with better management of post-operative pain. Locoregional anaesthesia can be considered a useful alternative to general anaesthesia in highrisk patients. Despite being an integral component of anaesthesia practice and being recognized as a milestone in specific training domain, ultrasound training is still not a component of many accredited anaesthesia residency programs. In the context of TEL, many opportunities exist. For instance, the Royal College of Anesthesia and e-Learning for Healthcare has recently produced the E-learning anaesthesia, i.e., a joint initiative containing more than 750 curriculum-based topics, catalogued by subject, in addition to other useful learning resources. In the paper, we investigated the change of implementation on innovative locoregional anaesthesia, after an ultrasound training in locoregional anaesthesia that includes the use of a high-fidelity simulator. At the end of the course, the TEL support to the training, i.e., the mannequin simulation, was rated among the most important teaching methods. Keywords: Ultrasound simulation training Locoregional anaesthesia


Technology-enhanced learning


© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 206–215, 2020. https://doi.org/10.1007/978-3-030-52538-5_21

Technology-Enhanced Learning (TEL) in Anaesthesia

207

1 Introduction Ultrasound (US) is a safe point-of-care imaging modality that is being increasingly used in many fields of anesthesiology and has led to the rapid development of new techniques [1, 2]. Ultrasound refers to the use of sound waves (typically from 2 to 22 MHz) above the frequency of audible by the human ear (from 20 to 20 kHz range). In ultrasonography, the electrical energy is converted into acoustic energy through piezoelectric crystals. Depending on the thickness of the crystals, a different frequency is generated. When a wave generated as above hits a tissue, it can be reflected, refracted, scattered, attenuated or absorbed. The returned waves are then converted back into an electric signal. Tissues that allow the wave to pass easily (e.g., fluids) create a small echo and appear black on the screen, while the others (e.g., bones) create stronger echoes and thus appear white on the screen. Several ultrasound imaging techniques exist, but B-mode, M-mode, and color-Doppler are those most commonly used in anesthesiology. B-mode (brightness) is the main mode of any ultrasound machine: the brightness shown on a screen depends on the intensity of the echo that is received from the corresponding location in the body. M-mode (motion) displays the movement of structures along the axis of the wave, whereas doppler modes detect frequency shifts that are created by sound reflections of a moving target [1]. In recent years, ultrasound has seen rapid development with numerous applications in anaesthesia, intensive care medicine, and pain medicine, increasing efficacy and safety of procedures [1]. Peripheral nerve block and myofascial block are two kinds of procedure used in anaesthesia to block the nociceptive stimuli from surgery, from trauma or in case of chronic and neuropathic pain. Performing a peripheral nerve blocks means to inject local anaesthetic close to a nerve or nervous plexus, avoiding iatrogenic injuries such as vascular puncture, intravascular injection, nerve damages or local anaesthetic systemic toxicity. In recent years the concept of fascia has been increasingly developed as a proper organ system with the function of connection between muscles, nerves and blood vessels [3]. Nerve often lies into these myofascial compartments; a myofascial compartment is a virtual space between muscles and connective tissue that surrounds and divides the muscular layers of the human body. The injection of the anaesthetic solution can expand the myofascial plane in order to block the terminal rami of nerves. Ultrasonography allows real-time imaging of the positions of the targeted nerve, needle, and surrounding vasculature. This can improve the ease of performing the procedure, increases the success rate, and may reduce the risk of complications. The ultrasound endpoint for a successful peripheral nerve block is to visualize the tip of the needle close to the anaesthetic target, in order to inject the anaesthetic solution that it appeared as an anechogenic shadow around the nerves. This may reduce the amount of local anaesthetic, decreasing the onset time of blocks. The ultrasound endpoint for a successful myofascial block is to expand the myofascial compartment muscles, obtaining two hyperechogenic layers, separated by the “anechogenic bubble” of local anaesthetic [4].

208

V. Cofini et al.

Before each injection, a colour Doppler echography can be performed to avoid vessel puncture. The ultrasonography allows to check the integrity and functionality of anatomic structures in which nerves lie; for example, the presence of pleural sliding can be continually controlled to prevent iatrogenic pneumothorax. Under ultrasound guidance it is possible to check real-time the tip of needle close to the anaesthetic target, providing effective and long-acting anaesthesia and analgesia during the all peri-operative period, decreasing the needing of general anaesthesia, in an opioid-sparing strategy [5]. In addition, the proper choice of anaesthetic technique is essential for controlling acute postoperative pain. With ultrasound guidance it is possible to inject a proper the volume of anaesthetic solution, enhancing the spread and longer contact time between anaesthetic drugs and sensitive fibres for a longer sensory blockade that ensures better control of acute postoperative pain. Longer-Lasting analgesia reduces the neuroplastic changes that are responsible for persistent postoperative pain syndrome [6, 7]. It is possible also to combine the use of a peripheral nerve block in combination with a fascial plane block allows a better anesthesiologic plane and antalgic coverage especially in patients with high anaesthetic risk when the type of surgery allows it [8]. In the case of chronic pain, ultrasound guidance has the potential to be a viable alternative to the conventional technique, e.g., when a needling treatment of myofascial trigger points is used. The twitches evoked by dry needling procedure can be detected, improving the accuracy of this technique [9, 10]. The use of ultrasound guidance (USG) can also improve the insertion of intraosseous access (IOA) into the medullary space during resuscitation manoeuvres, especially in the emergency setting of a natural disaster [11, 12]. Despite being an integral component of anaesthesia practice and recognized as a milestone as specific training domain, training and education for different ultrasound modalities is still not a component of many accredited anaesthesia residency programs [13]. In the context of TEL, many opportunities regarding the teaching of ultrasound in anaesthesia actually exist. For instance, the Royal College of Anesthesia and eLearning for Healthcare have recently produced the E-learning anaesthesia, i.e., a joint initiative containing more than 750 curriculum-based topics, catalogued by subject, in addition to other useful learning resources [14]. Moreover, evidence exists that serious games and immersive virtual worlds in medicine can improve on patients’ health (e.g., reduce anxiety and improve compliance with treatments) [15]. In a training context, high-fidelity simulators and educational material in virtual reality may be used for scenario-based teaching, where trainees could develop decision-making skills without the real-life consequences [16, 17]. In the paper, we investigated the change of implementation on innovative locoregional anaesthesia (fascial blocks), after the participation to an ultrasound training in locoregional anaesthesia that includes, in the course, the use of a high-fidelity simulator.

Technology-Enhanced Learning (TEL) in Anaesthesia

209

2 The Study 2.1

Study Design

A single group pre- and post-test research was designed. 2.2

Sampling and Participants

There was no a priori sample size estimate because all of the past attendees of the simulation courses (“Ultrasound in Local Anesthesia with Simulation Systems”, “Ultrasound in Intensive Care with Simulation Systems”, “Advanced Medical Simulation for Anesthesia and Emergency in Obstetrics”, “Communication in anaesthesia”, “ACLS-CRM course”, “Management of Cardiac Arrhythmias during Surgery with Simulation Systems”) operating in the San Salvatore Hospital in Abruzzo Region were invited to anonymously participate in this study. The study was carried out in compliance with the Helsinki Declaration and all participants gave their informed consent. Interviews took place from November 2019 to December 2019. 2.3

Course Description

The simulation courses were held by specialists in anaesthesia and intensive care medicine, experts in the subjects, supported by highly trained technicians in high fidelity simulation rallies. The first day of the course, the loco-regional anaesthetic techniques to block peripheral nerves [Brachial plexus, Sciatic, Femoral blocks] and myofascial planes [Lumbar Interfascia, Iliac fascia and Paravertebral, Pecs, Quadratus Lomborum, i-Pack, Erector Spinae Plane Rectus Sheath Toraco, Serratus Plane blocks] were theoretically shown. The peripheral nerve blocks and myofascial block were shown on a phantom simulator. This device is useful for regional anaesthesia ultrasound training because with this model the clinicians can acquire and interpret sonographic images of nerves and vessels as well as developing the psychomotor skills of guiding needles to simulated nerves and vessels in the human patient. The medical simulation model was configured for a linear ultrasound probe (7.5–10 MHz). This is the first step shown to the students. Then, a chest and abdomen mannequins with parenchymal organs were used to reproduce the texture and resistance of human tissue. They had sufficient ultrasound penetration for a convex probe (3.5–5.0 MHz) with a realistic ultrasound imaging to teach trainees how to identify target structures in a patient. In this way, the students had realistic needle forces during phantom training to improve their haptic senses. The second day, a simulated scenario in anesthesiology was designed: the participants tried to practically perform the anaesthetic procedures on a computerized, breathing mannequin with pulses, blood pressure, electrocardiogram, and other physiologic responses served as the standardized patient to produce highfidelity, reproducible, life-threatening acute situations that are not possible in other evaluation settings. They had to demonstrate to correctly identify the anaesthetic. Two scenarios were developed: peripheral nerve block scenarios and myofascial block scenario. The first scenario was complicated by local anaesthetic systemic toxicology

210

V. Cofini et al.

(LAST); the participants had to correctly manage the symptoms and signs of the systemic absorption of local anaesthetics. Diverse early manifestations of had simulated from perioral paresthesia, confusion, audio-visual disturbances, dysgeusia, agitation, reduced level of consciousness. The scenario rapidly developed to simulate seizures with dysrhythmias, conduction deficits, hypotension, and finally cardiac arrest. In the second scenario, standard locoregional anaesthesia was performed but the anaesthetic target was not achieved. Students had to perform a myofascial block to provide useful anaesthesia for the surgical procedure. Evaluation sessions were videotaped for post hoc debriefing of performance (Fig. 1).

Fig. 1. Phantom simulator

2.4

Measures

Subjects enrolled were investigated using an anonymous self-administered questionnaire “Ultrasound Loco-regional Anesthesia Questionnaire” (ULRA-Questionnaire). ULRA-Questionnaire is a modified version of a pre-post course survey instrument [18] and it included questions regarding participants’ characteristics (clinical practice, the average number of nerve blocks, fascial plane blocks and central venous access in a typical month, perceived confidence level to performing them) pre and post-course, the self-perception about theoretical and practical ultrasound knowledge, the obstacles in ultrasound use in regional anaesthesia and the effectiveness of the teaching methods. In addition, the survey gathered information on demographic variables (age and gender), the place of work (city, regions), ultrasound use in pain therapy, landmark and ultrasound use in innovative blocks and a final question about the agreement to an integrated ultrasound curriculum for undergraduate medical students.

Technology-Enhanced Learning (TEL) in Anaesthesia

2.5

211

Outcomes

The primary outcome was the average number of fascial plane blocks reported per month at follow-up, compared to baseline. Secondary outcomes were the average number of nerve blocks and central venous access reported per month at follow-up, compared to baseline, reported a preference for ultrasound or landmark guidance, obstacles and complications using ultrasound in locoregional anaesthesia, self-perception of ultrasound knowledge and ratings of ultrasound teaching methods. 2.6

Statistical Analysis

Statistical analysis was performed with STATA 14/MP software. Descriptive statistics were calculated for all variables in the study. For the primary outcome, a within-group comparison of the average number of blocks at follow-up versus baseline was performed using Anova Model for repeated measures. A regression model was computed to investigate the impact, if any, of age, gender, follow-up interval in years after course completion, clinical experience in years, on the change in the average number of blocks. The alpha level for all analyses was set to p < 0.05.

3 Results Twenty-two participants were enrolled, 55% of them were males, 27% (6/22) were Medical Resident in Anesthesiology, with mean age 44 ± 12 years. Clinical experience on average was 14 ± 12 years with a follow-up interval in years, on average, after course completion of 2 ± 1.9. The participants reported a good confidence level with ultrasound guidance for fascial plane blocks (4.3 ± 0.8), nerve blocks (4.2 ± 0.8) and central venous access (4.2 ± 0.9), before attending the course. 3.1

Primary Outcome

On the post-course survey, participants reported performing 32 (± 33) fascial plane blocks per month, on average, compared to 15 (± 14) at baseline (F = 14.169; p < 0.001) as reported in Fig. 2. Table 1 reports the fascial plane blocks ultrasound-guided per month, on average, performed by participants. The regression model showed that none of the participants’ baseline characteristics were significant cofactors for the change in the average number of the fascial plane blocks. 3.2

Secondary Outcomes

The management of nerve blocks was not changed comparing follow-up data to baseline data (F = 0.038; p = 0.847) while, on the post-course, participants reported

212

V. Cofini et al.

Fig. 2. Fascial plane blocks ultrasound-guided, on average, pre-post course (error bars = 95% CI)

Table 1. Reported fascial plane blocks Fascial plane blocks Tap block Paravertebral block Pecs block Quadratus Lomborum block Iliac fascia i-Pack block Erector Spinae Plane block Rectus Sheath block Toraco Lumbar Interfascial block Serratus Plane block

Mean 7.8 1.4 8.11 4.3 1.0 1.3 2.8 1.2 0.8 3.2

Std. dev. Minimum 7.3 0 2.7 0 5.5 0 4.6 0 2.3 0 2.6 0 5.8 0 4.3 0 2.3 0 5.6 0

Maximum 30 10 20 15 10 10 25 20 10 20

performing 12 (± 13) central venous access per month, on average, compared to 8 (± 13) at baseline (F = 5.533; p = 0.028). The regression model showed that none of the participants’ baseline characteristics were significant cofactors for the change in the average number of central venous access. At follow-up, fourteen participants (64%), reported that they used ultrasound in pain therapy, all of them (22/22) preferred ultrasound guidance for nerve and fascial plane blocks, compared to landmark guidance, and the obstacle to implementing ultrasound use into clinical practice was time pressure and inadequate skills (Fig. 3). At the end of the course, participants’ rating of self-perception of ultrasound knowledge based on a Likert scale point from 1 to 5 reported was 4 with Interquartile range IQR = 2. Eighteen doctors (72%) agreed or strongly agreed to integrate ultrasound education for undergraduate medical students. Ratings of Teaching Methods for Learning Ultrasound-guided Regional Anesthesia were highest for “live observation” and “teaching by colleagues” as expected, whereas

Technology-Enhanced Learning (TEL) in Anaesthesia

213

Fig. 3. Obstacles to implementing ultrasound use in clinical practice (%)

ratings for “online materials”, “Mannequin simulation”, “Model scanning” and “textbooks” were well appreciated (Table 2). Table 2. Participants’ rating of teaching methods for learning ultrasound-guided regional anesthesia based on perceived effectiveness (Likert scale points from 1 to 5) Teaching methods Lectures Phantom practice Model scanning Online materials Textbooks Mannequin simulation Live observation Teaching by colleagues

Median 3.5 3.5 4 4 4 4 5 5

IQR (Q3-Q1) 2 3 2 2 2 2 2 2

4 Conclusion The ultrasound training had a strong impact on the fascial plane blocks representing the innovative locoregional anaesthesia. The use of ultrasound in pain therapy still appears to be not widespread in our hospital but the skills developed through Ultrasound training show the diffusion of innovative blocks avoiding general anaesthesia with the better management of post-operative pain. With specific respect to the study reported in the paper, the structure of the course required participants - supported by highly trained technicians in high fidelity simulation rallies - to try to practically perform the anaesthetic procedures on a phantom simulator and then on people without needle and anaesthetic injection. At the end of the course, the TEL support to the course, i.e., the mannequin simulation, was rated among

214

V. Cofini et al.

the most important teaching methods, even if live observations and teaching by colleagues remained preferred by the participants. It is worth remarking that the study has a clear limitation that we only observed health operators from one Hospital. Nevertheless, we are planning an extended national survey to extend the validity of our findings.

References 1. Terkawi, A.S., Karakitsos, D., Elbarbary, M., Blaivas, M., Durieux, M.E.: Ultrasound for the anesthesiologists: present and future. Sci. World J. 2013 (2013). 15 pages 2. Fusco, P., Cofini, V., Di Carlo, S., Luciani, A., Scimia, P., Petrucci, E., Behr, A.U., Necozione, S., Colantonio, L.B., Fiore, G., Vergallo, A., Marinangeli, F.: Ultrasonography and Italian anesthesiology: a national cross-sectional study. J Ultrasound 22(1), 77–83 (2019). https://doi.org/10.1007/s40477-018-0334-1 3. Adstrum, S., Hedley, G., Schleip, R., Stecco, C., Yucesoy, C.A.: Defining the fascial system. J. Bodyw. Mov. Ther. 21(1), 173–177 (2017). https://doi.org/10.1016/j.jbmt.2016.11.003 4. Fusco, P., Petrucci, E., Marinangeli, F., Scimia, P.: Block failure or lack of efficacy? The “Double V” sign: a novel sonographic sign for a successful interfascial plane block. Minerva Anestesiol. 85(8), 917–918 (2019). https://doi.org/10.23736/S0375-9393.19.13457-8) 5. Fusco, P., Cofini, V., Petrucci, E., Pizzi, B., Necozione, S., Marinangeli, F.: The anaesthetic and analgesic effects of pectoral nerve and parasternal block combination for patients undergoing breast cancer surgery: a phase II study. Eur. J. Anaesthesiol. 36(10), 798–801 (2019). https://doi.org/10.1097/eja.0000000000001026 6. Fusco, P., Cofini, V., Petrucci, E., Scimia, P., Paladini, G., Behr, A.U., Gobbi, F., Pozone, T., Danelli, G., Di Marco, M., Vicentini, R., Necozione, S., Marinangeli, F.: Uni-lateral paravertebral block compared with subarachnoid anaesthesia for the management of postoperative pain syndrome after inguinal herniorrhaphy: a randomized controlled clinical trial. Pain 157(5), 1105–1113 (2016). https://doi.org/10.1097/j.pain.0000000000000487 7. Fusco, P., Cofini, V., Petrucci, E., Scimia, P., Pozone, T., Paladini, G., Carta, G., Necozione, S., Borghi, B., Marinangeli, F.: Transversus abdominis plane block in the management of acute postoperative pain syndrome after caesarean section: a randomized controlled clinical trial. Pain Physician 19(8), 583–591 (2016) 8. Fusco, P., Di Martino, E., Paladini, G., De Sanctis, F., Di Carlo, S., Scimia, P., Petrucci, E., Marinangeli, F.: Fascial plane blocks and peripheral nerve blocks: two planets not so far apart. Minerva Anestesiol. 85(10), 1139–1140 (2019). https://doi.org/10.23736/S0375-9393. 19.13669-3 9. Finlayson, R.J.: Ultra-sound guidance for trigger point injections gold standard or fool’s gold? Reg. Anesth. Pain Med. 42(3), 279–280 (2017) 10. Fusco, P., Di Carlo, S., Scimia, P., Degan, G., Petrucci, E., Marinangeli, F.: Ultra-soundguided dry needling treatment of myofascial trigger points for piriformis syndrome management: a case series. J. Chiropr. Med. 17(3), 198–200 (2018) 11. Petrucci, E., Cofini, V., Pizzi, B., Di Carlo, S., Necozione, S., Fusco, P., Marinangeli, F.: Ultrasound-guidance for intraosseous access could improve resuscitation maneuvers. A retrospective data report on Italian earthquake victims. Minerva Anestesiol. (2019). https://doi.org/10.23736/s0375-9393.19.14072-2

Technology-Enhanced Learning (TEL) in Anaesthesia

215

12. Blasetti, A.G., Petrucci, E., Cofini, V., Pizzi, B., Scimia, P., Pozone, T., Necozione, S., Fusco, P., Marinangeli, F.: First rescue under the rubble: the medical aid in the first hours after the earthquake in Amatrice (Italy) on August 24, 2016. Pre Hosp. Disaster Med. 33(1), 109–113 (2018). https://doi.org/10.1017/S1049023X17007075 13. Mahmood, F., Matyal, R., Skubas, N., Montealegre-Gallegos, M., Swaminathan, M., Denault, A., Sniecinski, R., Mitchell, J.D., Taylor, M., Haskins, S., Shahul, S., OrenGrinberg, A., Wouters, P., Shook, D., Reeves, S.T.: Perioperative ultrasound training in anesthesiology: a call to action. Anesth. Analg. 122(6), 1794–1804 (2016). https://doi.org/ 10.1213/ane.0000000000001134 14. Kirkpatrick, K., MacKinnon, R.J.: Technology-enhanced learning in anaesthesia and educational theory. Contin. Educ. Anaesth. Crit. Care Pain 12(5), 263–267 (2012). https:// doi.org/10.1093/bjaceaccp/mks027 15. Kato, P.M.: Video games in health care: closing the gap. Rev. Gen. Psychol. 14, 113–121 (2010) 16. Kamel Boulos, M.N., Hetherington, L., Wheeler, S.: Second life: an overview of the potential of 3-D virtual worlds in medical and health education. Health Info Libr. J. 24, 233– 245 (2007) 17. Heitz, C., Brown, A., Johnson, J.E., Fitch, M.T.: Large group high-fidelity simulation enhances medical student learning. Med. Teach. 31, e206–e210 (2009) 18. Mariano, E.R., Harrison, T.K., Kim, T.E., Kan, J., Shum, C., Gaba, D.M., et al.: Evaluation of a standardized program for training practising anesthesiologists in ultrasound-guided regional anaesthesia skills’. Ultrasound Med. 34, 1883–1893 (2015)

Profiles in Brain Type in Programming Performance for Non-vocational Courses Ugo Solitro(&), Margherita Brondino(&), Roberto Bonafini, and Margherita Pasini Department of Computer Science and Department of Human Sciences, University of Verona, Verona, Italy {ugo.solitro,margherita.brondino,roberto.bonafini, margherita.pasini}@univr.it

Abstract. Learning a programming language and solving problems in an algorithmic way is a hard task for many students. A better comprehension of the psychological characteristics involved in this process is needed to reduce these difficulties, implementing teaching methodologies sensitive to these aspects. In this study, we analyse the relationship between cognitive styles, inside the theoretical framework of the empathizing–systemizing (E–S) theory, and performance, also considering the role of sex. In fact, E-S theory states the difference in male and female mind, the first one more empathy-oriented, the second one more oriented at understanding systems. A sample of 56 students attending a course of programming in an Applied Mathematics with a relevant practical activity enhanced with a submission system and supported by a few tutors was involved in the study. We defined profiles of students based on their scores in EQ and SQ, and on their sex, using a cluster analysis. 4 clusters were found: 1. female students with low level of SQ and high level of EQ; 2. female students with high level of both SQ and EQ; 3. male students, with low level of EQ and high level of; 4. male students with low level of both SQ and EQ. A first exploration of the relation of profiles with the learning performance is described. Keywords: Programming
Computer science education Empathy Quotient
Systemizing Quotient
Performance


Brain type

Students’ profiles

1 Introduction The early introduction to programming in non-vocational graduate course is always a hard task. The acquisition of computational competence can be influences by the school background, the teaching methodology, the student attitude and skills, and other personal traits. Some studies are related to the influence of personality profile on computer programming. Karimi and Wagner [12] summarized the main findings of the literature on the influence of personality on computer programming. At last, they found a weak relation between personality and performance in computer programming. However, they take into consideration that all personality factors might affect programming. The influence of each personality factor might be overridden by other aspects such as experience or team working. Furthermore, the relation between personality and © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 216–225, 2020. https://doi.org/10.1007/978-3-030-52538-5_22

Profiles in Brain Type

217

performance might not be linear. For example, referring to the Big Five model [10], both extraverts and introverts could be good programmers or those on low and high scores in agreeableness could be good programmers. They concluded that the influence of personality on performance in programming is not clear. Another set of studies uses the Myers-Briggs Type Indicator (MBTI) [7], a personality inventory indicating different psychological preferences in how people perceive the world and make decisions. This instrument is commonly used and accepted amongst researchers in the software engineering domains [18]. It consists of 16 personality types that combine 4 pairs of personality dimensions: Introvert – Extrovert; Sensing – Intuitive; Thinking – Feeling; Judging – Perceiving. Li et al. [14], using MBTI, found that students whose personality type was “perceiving” outperformed those who were of the “Judging” type, in terms of the programming performance. According to MBTI, while “judging types” are described as tending to live in a planned and orderly way, also liking things decided, “perceiving types” tend to live in a more spontaneous and flexible way, open to new information and possibilities. Li and colleagues [13] also found a strong positive relationship between students’ programming performance and “intuition type”, that is a mental function used for gathering information, paying attention to meanings, patterns, and future possibilities. These results seem to suggest that a personality profile oriented to flexibility and opened to understand events appears much more effective in computer programming. Intuition also significantly correlates with empathy [5], another construct on which this kind of research has focused. Students in engineering programmes have significantly lower empathy than students from other courses [17]. However, controlling for sex, this decreased, suggesting a big role of sex in determining empathy. Moreover, empathy as a soft skill could have positive consequences for the employability of the engineering graduates [1,17]. Thomson and colleagues [19] found that greater levels of empathy were connected with enrolment in social and life sciences university courses, while lower levels of empathy predicted physical sciences enrolment, showing that also the choice of university courses is connected with different level of empathy. Further studies look at psychological personality in terms of systemizing vs empathizing types. The Empathizing-Systemizing (E-S) Theory [2] appears an interesting framework that has been used to explore differences in programming performance for university students. It appears to point out certain predictive potential concerning programming aptitude [9, 21], only recurring to two modes of thought. Nevertheless, only few studies have considered this theoretical framework to analyse performance in programming, with mixed results. So, more research is needed to contribute at increasing the knowledge about this topic. This theory suggests that people can be classified on the basis of two different cognitive styles: empathizing, which is connected with the comprehension of emotional states of other individuals, and systemizing, which allow individuals to predict systems’ behaviour on the basis of the knowledge of the underlying rules. Theory refers to these two different cognitive styles as “brain types”, and states that sex is strictly connected with brain type: the female brain is predominantly empathy-oriented, and the male brain is predominantly oriented at understanding and building systems. Two psychometric instruments had been constructed to measure these two psychological and cognitive styles: The Empathy Quotient (EQ), which assess

218

U. Solitro et al.

how easily an individual understand humans’ emotions, and the Systemizing Quotient (SQ), which assess how easily an individual understand object systems. The E-S Theory classify individuals on the basis of “brain types” (S-type vs E-Type), based on the presence or absence of discrepancies between their scores on E or S. The theory assumes that a high level of SQ should be connected with a good performance in the domains in which this skill is important, such as scientific disciplines, whereas a high level of EQ should be more necessary in other domains in which the comprehension of others is important, for instance humanities and social sciences [19, 20]. Additionally, the link between these two cognitive styles and sex has been broadly explored in the literature, showing that women have generally a higher level of EQ than men. On the other side, men generally show higher scores in SQ [3]. The link between E-S Theory and programming aptitude has been stated by Wray [21], and then resumed by Coles and Phalp [9]. Wray [21] research showed that separately these two measures seems to show poor correlation with performance in programming, whereas the difference between these two quotients seems to be more related with performance: individuals with higher scores on SQ than EQ showed better results than individuals with the opposite pattern. Coles and Phalp [9] found no correlations between brain type and programming performance, even if the brain type was related with the choice of the degree subject. However, they [9] have observed an opposite tendency, compared to Wray research. That is a higher EQ score alone could be a predictor of programming ability, at least for female students. Pasini, Solitro, Brondino e Raccanello [15] found that female students had a higher level of empathy than male ones. However, no differences in performance were found between women and men, even if for female students a higher level of systematizing quotients was related with a higher performance. Borzovs et al. [6] tested the hypothesis that programming aptitude could be predicted based on SQ and EQ, with first-year students, but they could not support the hypothesis. Pasini et al. [16] found a positive correlation between the SQ and programming, even if only for female students: women with high level of SQ perform better than women with low level of SQ, whereas no correlation was found for male students. These controversial results suggest that using these scores (EQ or SQ) alone, or even the simple combination of SQ and EQ, that is the difference between the two scores, fails to represent the complexity of the possible combinations. It should be preferable to evaluate how these two measures can be differently combined together, also considering sex as an important variable. In fact, it seems that sex not only is related to the level of EQ and SQ, but also moderate the relationship between EQ, SQ and performance in programming. A possible way to explore the different combinations of these three variables is the use of Cluster Analysis, a statistical classification technique in which a set of objects, in this case individuals, with similar characteristics are grouped together. The aim of this study was, firstly, to define profiles of students based on their scores in EQ and SQ, and on their sex, using a cluster analysis. In a second step, we aimed to find whether the students’ profiles relate to performance in programming. A better understanding of the relation between the basic skills required to a programmer and the aptitude to face difficulties and deal with them will facilitate the design and application of more effective teaching methodologies and recovery strategies.

Profiles in Brain Type

219

2 Method 2.1

Participants and Context

The sample included 56 undergraduate students (M = 19.54 years, SD = .74; 51,8% female) of the University of Verona, in Northern Italy. Students were enrolled at the first year of bachelor’s degree in Applied Mathematics, a program with a special emphasis on its applications to modelling, computations, finance, economics. The first year includes two compulsory courses related to computational skills: a course in programming (12 credits) and another one in Numerical Analyses (6 credits). This research was conducted involving 29 students in Autumn 2017 (M = 18.97 years, SD = .50; 55,2% female) and 27 students in Autumn 2018 (M = 20.15 years, SD = .36; 40,7% female). The research project is part of a larger project on motivation, creativity and academic performance in the university context. The Programming course is one of crucial teaching (12 ECU out of 60) of the first year of Applied Mathematics: the main goals are learning the basics of a programming language (Java) and developing the ability of analyse, solve (by coding) and evaluate the solution of small and medium size problems. A great importance is given to the practical part, but also to the connection to mathematical themes. Due to well-known failure rate for first year students the didactics activity can take advantage of the support of tutors and experimental teaching techniques. In the two academic years we are analysing, two experimental methodologies have been applied: a lightened version of the eXtreme Apprenticeship (light XA) [23, 24] and, in addition, an adaptation of training practices for IT competitions with the support of an online platform specification (CMS) [22] in the second year. The orientation of the high school curriculum for the most of them (around 3/4) was scientific or technical. In spite of this, many of them (about 3/5) had a poor preparation in informatics and even less of them (roughly 1/3) have some experience in programming. Looking at the results of an entrance survey, their attitude towards programming is cautious, sometimes worried about it although the discipline is included in the fundamental activities of the bachelor’s degree. All the students signed an informed consent form for voluntary participation. 2.2

Measures

Brain Type. To evaluate the brain type, we used the short version of the Empathy Quotient (EQ) and the Systemizing Quotient (SQ) [20]. This version includes a total number of 60 items, 20 for the EQ and 20 for the SQ and 20 fillers. We used the items’ Italian translation from the Autism Research Center [8], when available, and an Italian translation of the residual items proposed by the research group, also in collaboration with some experts on the topic. Given the fact that we need a quick version, we decide to remove the fillers. Respondents have to choose their level of agreement on a 4-point Likert Scale. Scores can range from 1 to 8 for each scale. We had used the mean as score respectively of Empathy and Systemizing Quotient.

220

U. Solitro et al.

Active Participation to the Lab Activities. A score value in the range from 0 to 1 was estimated to assess the active participation to the practical activity in the Programming Laboratory. The score had a limited weight (less than 10%) for the final exam grade. Performance in Programming. In this work we considered four different measures concerning performance. Test 1 assesses the performance in the early period (the first two months of the teaching), an consists in two parts, a theoretical one and a practical one; these two parts are joined together in the Test 1 grade. Finally, the final grade was considered. These four performance indicators are described below: Test 1 – Theoretical Part. The first written Test was structured in two sections. The first one contains 3 or 4 questions (with a short answer required) about the fundamental concepts of programming. The evaluation parameters involved completeness, correctness, synthesis, general rigour and precision of natural language usage. The grade value (TH grade) was standardized in the range from 0 to 1. Test 1 – Practical Part. The second section is about programming. It consisted of 3 or 4 exercises about the characterization of a problem and the coding in Python of its possible solution. The goals were problems and code comprehension of the specification of a problem (with the capability to correct small errors), coding a solution satisfying given rigorous specification of a problem, the characterization of an informally given problem and consequently the coding of the solution. The evaluation parameters were (general) completeness, correctness, synthesis, general rigour and precision of the natural language usage; (characterization) completeness, rigour, mathematical language; (coding) precision, correctness, synthesis, structure, abidance of previously stated conventions and rules. The grade value (PR grade) was standardized in the range from 0 to 1. Test 1 Grade. The overall grade of the test (Test 1 grade) was obtained as a weighted average from TH grade (about 1/3) and PR grade (about 2/3); a correction factor (about 1.1) was applied in order to take in account of some intrinsic limitations of the evaluation method. Grade values were standardized in the range from 0 to 1 (honours were not considered). Final Grade. The final grade (Final grade) was standardized in the range from 0 to 1 (honours are not considered). The value was the result a weighted sum of the written tests (2/3), the laboratory activities (1/3). It was also influenced by the final colloquium where the ongoing tests and project development were discussed. 2.3

Data Analysis

We used two-step cluster analysis in SPSS 25 to segment the students based on sex and scores on SQ and EQ. This is a clustering procedure that can form clusters on the basis of either categorical or continuous data. The clustering algorithm is based on a distance measure that gives reasonably good results even if the assumptions of normality of the distributions for continuous variables is not met. We used the log-likelihood criterion as the distance measure, and specify a fixed number of clusters, exploring different solutions, in particular 3, 4 and 5 clusters. The 2-cluster solution was a priori excluded,

Profiles in Brain Type

221

because it obviously should lead to the simple distinction between male and female students. To quantify the “goodness” of the cluster solution, the silhouette coefficient was use, which is a measure of both cohesion and separation. In a good solution the silhouette measure is close to the maximum value of 1; a value from −1 to .2 is considered a poor solution, from .2 to .5 a fair solution, and from .5 to 1 a good solution [11]. Among the good solutions, we decided for the one that was the most satisfactory for the purpose of a sensible clustering of students considering the three variables. In a second step, four different ANCOVAs were run, considering the clusters as the between-subject factor, and the performance (separately: Test 1 – Theoretical part, Test 1 – Practical part, Test 1 grade, Final grade) as the dependent variable. We controlled for Lab score, using it as a covariate in the models, to remove the effect of the active participation to the practical activity in the programming lab.

3 Results and Discussion 3.1

Descriptive Statistics and Bivariate Correlations

Table 1 describes mean, standard deviation, and bivariate Pearson’s correlations for the considered variables. No significant correlations was found between SQ or EQ scores and performance indicators, except for the negative correlation between EQ and the final grade (r = −.28, p < .05): a high level on EQ is related with a low level in the final grade and vice versa, even if this correlation is low. The correlation between active participation to the lab activities (Lab) and the four indicators of performance is high, with the lowest correlation between Lab and the theoretical part of Test 1. The correlations among performance indicators are all high. We explored whether the correlation pattern changes for male and female students [16], but no significant correlations were found between EQ−SQ and performance indicators, separately for male and female students. EQ was higher than SQ (MEQ = 5.86, SDEQ = .76; MSQ = 5.08, SDSQ = 1.02). Nevertheless, when split by sex, it is possible to note that this result depends on a higher level of EQ for female students (M = 6.19, SD = .64), compared with male students (M = 5.54, SD = .73); at the same time, female students showed a lower level of SQ (M = 4.2, SD = 1.51) compared with male students (M = 5.2, SD = .89). A mixed ANOVA, with brain type as the within-subject factor (EQ vs SQ) and sex as the between-subject factor confirmed the main effect of brain type (F(1,54) = 21.59, p < .001), with a high effect size (g2 = .26), and also the interaction brain type x sex (F(1,54) = 7.73, p = .012), even if with a low effect size (g2 = .08). 3.2

Profile of Students on the Basis of EQ-SQ and Sex

We used two-step cluster analysis in SPSS to segment the students based on sex and scores on SQ and EQ. We choose a 4-cluster solution because it was the most reasonable one, with a silhouette measure of cohesion and separation of 0.5, that indicates a good cluster solution quality. Our participants clustered in four distinguished groups: 1. female students with low level of SQ and high level of EQ (F SQ- EQ+); 2. female

222

U. Solitro et al.

Table 1. Bivariate Pearson’s correlations, Means (M), Standard Deviations (SD) for the studied variables Variable Range 1 2 3 4 5 6 7 1. EQ 1−8 – 2. SQ 1−8 −.12 – 3. Lab 0−1 −.17 .09 – 4. TH grade 0−1 .03 −.12 .48*** – 5. PR grade 0−1 −.21 .14 .58*** .46*** – 6. Test 1 grade 0−1 −.12 .09 .62*** .76*** .92*** – 7. Final grade 18–30 −.28* .10 .61*** .58*** .74*** .79*** – M 5.85 5.08 0.66 0.73 0.57 0.69 24.98 SD 0.76 1.02 0.25 0.19 0.19 0.18 3.14 Note. N = 56. EQ = Empathy Quotient, SQ = Systemizing Quotient, Lab100 = active participation, TH = Test 1 - Theoretical part, PR = Test 1 Practical part; Test 1 = Overall grade of the written test, Final grade = final grade of the exam. *p < .05, **p < .01, ***p < .001

students with high level of both SQ and EQ (F SQ+ EQ+); 3. male students, with low level of EQ and high level of SQ (M SQ+ EQ−); 4. male students with low level of both SQ and EQ (M SQ− EQ−). Figure 1 shows the cluster comparison, which enables to see differences between the 4 clusters compared with each other. Sex, that is a categorical variable, is shown as dot plots, where the size of the dot indicates the most frequent category for each cluster. Continuous variables are displayed as boxplots, which show overall medians and the interquartile ranges. It is interesting to note that both clusters of female students showed high level of EQ; nevertheless, a cluster with female students with high level of SQ was identified, whereas, on the opposite side, no one of the male clusters showed high level of EQ. 3.3

Students’ Profiles and Performance in Programming

The assumptions in ANCOVA are valid in the current study. Results of ANCOVAs – after control showed no significant effects of the clusters on performance, even if, as it is possible to see in Fig. 2, displaying the estimate marginal means after controlling for the active participation to the practical activity in the Programming Lab, the trend is quite clear: the two profiles with high level of SQ, that is F SQ+ ES+ (cluster 2) and M SQ+ EQ− (cluster 3), show the best performance (except for theoretical grade), whereas F SQ− EQ+ (cluster 1), shows the worst performance.

Profiles in Brain Type

223

Fig. 1. Cluster comparison by sex, Empathy Quotient and Systemizing Quotient (cluster 1, green: F SQ− EQ+; cluster 2, light blue: F SQ+ EQ+ ; cluster 3, red: M SQ+ EQ−; cluster 4, blue: M SQ− EQ−).

Fig. 2. Estimate marginal means of the four performance indicators (Test 1 theoretical part, Test 1 practical part, Test 1 grade, finale grade), with the error bars for the 95% C.I. (covariate Lab is evaluated at .667). 1 = F SQ− EQ+; 2 = F SQ+ ES+; 3 = M SQ+ EQ−; 4 = M SQ− EQ−.

4 Conclusions This research, using the theoretical framework of the E-S Theory [2], aims to identify profiles of students, also considering sex, that was often taken into account inside this framework. The hypothesis of the possibility of different profiles arose from the fact that the simple indicator SQ minus EQ, used in some researches to identify S-Type or E-Type profiles, seems a too raw predictor of performance in programming. At the

224

U. Solitro et al.

same time, the use of the single quotients, SQ or EQ, separately, gave not clear results as well [9, 15, 16]. This is maybe due to the fact that the situation is more complex and looking for profiles could be a first step to give some light to this complexity. We found, accordingly with the literature, that only females have high level of empathy. This is in line with other studies in which whilst on average women show stronger empathizing, men show stronger systemizing [4]. None of the two male clusters had an EQ higher than the sample average, whereas the two female clusters had both a high EQ. This high EQ, nevertheless, is not in contrast with a good performance in programming: cluster 2, that is female students with high EQ and high SQ, showed the best performance, except for the final grade. Based on the results, high level of systemizing is related to performance in both male and female. Only when the level of systemizing is low the performance in programming fall. The weaker profile in terms of programming performance seems to be the one with female students with high level of empathizing quotient and low level of systemizing quotient, except for theoretical part. Recognizing this weakness from the beginning of the programming course allows to plan a personalized paths of more effective teaching methodologies and recovery strategies. Anyway, high level of emphasizing might play an important role. Empathizing means the comprehension of the other human beings and in making sense of the world. Comprehension requires to open to accept new information and flexibility. That could result in improving performance programming, as found in case of Perception and Intuition type personality [13, 14]. Due to the relatively small sample size, further studies are needed to replicate and generalize this result. Moreover, if a larger sample of participants was available, it may be possible to better describe and clarify the relation of personality profile on performance in programming.

References 1. Ahmed, F., Capretz, L.F., Bouktif, S., Campbell, P.: Soft skills and software development: a reflection from the software industry. Int. J. Inf. Process Manag. 4(3), 171–191 (2013) 2. Baron-Cohen, S.: The extreme male brain theory of autism. Trends Cogn. Sci. 6(6), 248–254 (2002) 3. Baron-Cohen, S., Wheelwright, S.: The empathy quotient: an investigation of adults with Asperger syndrome or high functioning autism, and normal sex differences. J. Autism Dev. Disord. 34(2), 163–175 (2004) 4. Billington, J., Baron-Cohen, S., Wheelwright, S.: Cognitive style predicts entry into physical sciences and humanities: questionnaire and performance tests of empathy and systemizing. Learn. Individ. Differ. 17(3), 260–268 (2007) 5. Blandin, K., McPeek, R.W., Martin, C.R., Autin, K.: Types of empathy: an experimental study of empathy and Myers-Briggs® types. Center for Applications of Psychological Type, Inc. (2017). https://www.capt.org/journal-psychological-type/jpt-whitepapers.htm. Accessed Feb 2020 6. Borzovs, J., Kozmina, N., Niedrite, L., Solodovnikova, D., Straujums, U., Zuters, J., Klavins, A.: Can SQ and EQ values and their difference indicate programming aptitude to reduce dropout rate? In: European Conference on Advances in Databases and Information Systems, pp. 285–293. Springer, Cham, September 2017

Profiles in Brain Type

225

7. Briggs-Myers, I., Briggs, K.C.: Myers-Briggs Type Indicator (MBTI). Consulting Psychologists Press, Palo Alto (1985) 8. Cambridge Autism Research Centre. Empathy Quotient and Systemizing Quotient. http:// www.autismresearchcentre.com/arc_tests. Accessed Feb 2020 9. Coles, M., Phalp, K.T.: Brain type as a programming aptitude predictor. In: PPIG 2016 – 27th Annual Workshop, Cambridge, UK, 7–10 September 2016 (2016) 10. Digman, J.M.: Personality structure: emergence of the five-factor model. Annu. Rev. Psychol. 41(1), 417–440 (1990) 11. George, D., Mallery, P.: IBM SPSS Statistics 23 Step by Step: A Simple Guide and Reference. Routledge, London (2016) 12. Karimi, Z., Wagner, S.: The influence of personality on programming: a summary of a systematic literature review. University of Stuttgart (2014). http://dx.doi.org/10.18419/opus3243 13. Li, X., Shih, P.C., Daniel, Y.: Effects of intuition and sensing in programming performance using MBTI personality model. In: Proceedings of the 2nd International Conference on Advances in Image Processing, pp. 189–193, June 2018 14. Li, X., Shih, P.C., David, E.: The effect of software programmers’ personality on programming performance. In: 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 209–213. IEEE (2018) 15. Pasini, M., Solitro, U., Brondino, M., Raccanello, D.: The role of the cognitive style in improving the learning to program. In: 27th Annual Workshop of the Psychology of Programming Interest Group, PPIG, pp. 150–155 (2017) 16. Pasini, M., Solitro, U., Brondino, M., Burro, R., Raccanello, D., Zorzi, M.: Psychology of programming: the role of creativity, empathy and systemizing. In: International Conference in Methodologies and intelligent Systems for Techhnology Enhanced Learning, pp. 82–89. Springer, Cham, June 2017 17. Rasoal, C., Danielsson, H., Jungert, T.: Empathy among students in engineering programmes. Eur. J. Eng. Educ. 37(5), 427–435 (2012) 18. Sach, R., Petre, M. Sharp, H.: The Use of MBTI in software engineering. In: 22nd Annual Psychology of Programming Interest Group, September 2010. Universidad Carlos III, Madrid (2010) 19. Thomson, N.D., Wurtzburg, S.J., Centifanti, L.C.: Empathy or science? Empathy explains physical science enrollment for men and women. Learn. Individ. Differ. 40, 115–120 (2015) 20. Wakabayashi, A., Baron-Cohen, S., Wheelwright, S., Goldenfeld, N., Delaney, J., Fine, D., Weil, L.: Development of short forms of the Empathy Quotient (EQ-Short) and the Systemizing Quotient (SQ-Short). Pers. Individ. Differ. 41(5), 929–940 (2006) 21. Wray, S.: SQ minus EQ can predict programming aptitude. In: PPIG, vol. 7, pp. 243–254 (2007) 22. Maggiolo, S., Mascellani, G.: Introducing CMS: a contest management system. Olympiads Inf. 6, 86–99 (2012) 23. Solitro, U., Zorzi, M., Pasini, M., Brondino, M.: A “light” application of blended extreme apprenticeship in teaching programming to students of mathematics. In: Methodologies and Intelligent Systems for Technology Enhanced Learning, pp. 73–80. Springer, Cham (2016) 24. Vihavainen, A., Paksula, M., Luukkainen, M.: Extreme apprenticeship method in teaching programming for beginners. In: Proceedings of the 42nd ACM Technical Symposium on Computer Science Education, pp. 93–98 (2011)

Towards a Gamified Musical Skill Learning Model (MuS-LM): Structural Aspects Tania Di Mascio(&), Laura Tarantino, and Federica Caruso University of L’Aquila, 67100 L’Aquila, Italy {tania.dimascio,laura.tarantino}@univaq.it, [email protected]

Abstract. In this paper, we introduce the CrazySquare project, a research and development project aiming at realizing an ICT support system for musical education inspired by Gordon’s Music Learning Theory, within Italian Middle Schools. The CrazySquare project is designed according to an action research based methodological approach, deejay, so to guarantee the required equal attention to research and development objectives. In this paper we aim at introducing the overall methodological setting of the project and the main structural aspects of the learning model, mirroring the two main concepts of music aptitude and music achievements as introduced by Gordon. Keywords: Action research
Technology-enhanced learning education
Playing musical instrument
Young teenagers


Music

1 Introduction In the last decades, the use of ICT in the field of music education has been increasing at a rapid pace, giving rise to a plethora of solutions. Generally, these tools support the learning of the two statistically most played musical instruments (i.e., guitar and piano), providing practical notions for learning how to play them (e.g., Yousician [28]). In this paper, we focus on guitar-oriented solutions given the specific context of use discussed later on. Existing tools (pure commercial products, scientific literature outcomes and research prototypes become commercial products) can be classified depending on whether they adopt either one of or both the two following approaches: (1) a more experience-oriented approach supporting the instrumental practice and (2) a more lesson-oriented approach including music theory (e.g., ear-training) and “how to play’’ modules (see also Table 2). Most popular solutions follow the first approach. Commercially based solutions are generally SW applications assisting the users during the execution of exercises with the guitar, providing real-time feedback of their performance, thus improving the selfevaluation procedures and the acquisition of the mastery of playing the musical instrument (e.g., SmartMusic [25]). Research-based prototype solutions are mainly based on augmented reality technology: GuitAR [16]) is an augmented reality application that assists guitar students mastering their instrument using a projector phone mounted at the headstock of the guitar, the fretboard and the strings of the guitar are in the field of projection of the phone. GuitarSolo [24] is an augmented reality system: © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 226–238, 2020. https://doi.org/10.1007/978-3-030-52538-5_23

Towards a Gamified Musical Skill Learning Model (MuS-LM)

227

Arduino Mega2560 as the main controller board, an LED fretboard for fingering display, and a smartphone application providing a feature to select desired songs from the content server. All the solutions following the second approach are SW applications implementing lessons trough video-tutorials made by some teachers or musicians who explain in detail how the learners have to properly perform practical exercises (e.g., GuitarTricks [13]). In particular, Rockway [23], originated as a research prototype and evolved as a commercial tool largely used in the Finnish school context, offers an e-learning platform providing support for contact lessons, and conversely, contact lessons provide the terminology necessary for students’ knowledge base. Solutions mixing the two approaches are commercial SW applications that not only provide lessons through video-tutorials but also interactively implement them, proposing “question and answers” modules [18, 28], so to guarantee a gamified learning experience. In particular, Yousician [28] is the most popular solution currently supporting the learning of different musical instruments (e.g. guitar, piano, ukulele, and voice) and proposing suitable learning paths, according to the user performance. Unfortunately, as discussed in [3], existing tools are often not adequately evaluated (especially in the case of commercial tools) and do not offer the high level of music education that is guaranteed, at school, by the expertise of teachers following consolidated music-oriented pedagogical approaches (e.g., [27, 14 and 12]) that agree on the fact that musical education implies the internalization of sounds. In particular, Gordon proposed a music learning theory based on the concept of audiation [11]. Audiation is to music what thinking is to a language: for example, one can give meaning to musical sounds by organizing them into tonal patterns and rhythm patterns in terms of one or more tonalities and m. Tonal patterns and rhythm patterns are to music what words are to language, tonality and m are to music what syntax is to language [12]. A major criticism of the learning approach of commercially-driven tools is that they mainly emphasize imitation at the expense of audiation (e.g., tablature flows and rhythm are punctuated in a karaoke-style as in Yousician [28]). Furthermore [15] observed that although research shows the positive impact of ICT in music education, most of the existing tools have limited effect due to the lack of individualization of the learning process. On the other hand, new technologies may incorporate concepts coming from computer games to boost students’ motivations and interests [7]. In this paper we discuss first results of CrazySquare, an ongoing research and development project aimed at designing a gamified ICT solution conceived as a teacher’s ally in musical teaching activities in Middle School, hence utilized by children in the 10–14 age range (notice that typically schools have to follow governmental directives and recommendations defining musical education skills that must be acquired by children). CrazySquare applies gamification elements for the acquisition of audiation, necessary to play guitar consciously, achieving a long learning follow-up. CrazySquare is conducted according to deejay [8], an action research based methodological approach, so to guarantee equal attention to the research objective (i.e., the definition of MuS-LM, a gamified Musical Skill Learning Model mirroring the two concepts of music aptitude and music achievements as introduced by Gordon in [12]) and the development objective (the implementation of a gamified tool based on MuS-LM). More specifically,

228

T. Di Mascio et al.

here we aim at discussing (1) the overall methodological setting of the project and (2) the main structural aspects of the learning model embedding the concepts of aptitude and achievements, which constitute the first milestone of the project and are the basis for the introduction of gamified elements. More specifically, after presenting the organizational situation in Sect. 2, Sect. 3 discusses the learning model in terms of a set of principles defining the learning space and the admitted learning paths and applying it to the real case of an Italian Middle School. Finally, in Sect. 4 conclusions are drawn.

2 The Organizational Situation CrazySquare is an ongoing project carried out in cooperation by the Middle School Istituto Giovanni Pascoli in Rieti and the University of L’Aquila (Univaq), funder of the project and participating with DEWS1 and DISIM2. The two partners have concurrent yet distinct objectives, the school’s one being more focused on development outcomes and the university one being more focused on research outcomes. 2.1

The Methodological Approach

Action research (AR) [1] guarantees the required equal attention to research and development objectives, for its juxtaposition of action (practice) and research (theory) and its commitment to the production of new knowledge through the seeking of solutions or improvements to “real-life” practical problem situations [2,6]. The common notion of the many forms of AR is the existence of some kind of cyclical process repeated until a satisfactory outcome is achieved [2, 17]. According to Susman and Evered [26], after the establishment of a research-client agreement, five phases are iterated (Fig. 1-(a)): diagnosis corresponds to the identification of primary problems causing the organization’s desire for change and develops a theoretical framework to guide the process; action planning specifies actions that should relieve the organizational problem; action taking implements the planned action; evaluation determines whether the effects of the action were realized and produced the desired results; learning formalizes knowledge gained by the process wrt problem situation and the scientific community. To emphasize AR commitment to the production of scholarly knowledge, [17] proposed a model including two cycles running in tandem (Fig. 1-(b)), one addressing the client’s problem-solving interest and the other one the researcher’s scholarly interest (Table 1 instantiates the tandem elements for CrazySquare). To overcome the lack of direct guidance on “how-to-do” AR, raised up in [17], the deejay methodological framework [8] provides an additional level of detail about the design of the overall process, in the tricky case in which the outcome of the research cycle is the problem-solving method of the real-world problem: a regular structure rules time scheduling (what happens before/while what) and exchange of information to

1

2

The center of excellence on Design methodologies for Embedded controllers, Wireless interconnect and Systems-on-chip. The Department of Information Engineering, Computer Science and Mathematics.

Towards a Gamified Musical Skill Learning Model (MuS-LM)

229

Fig. 1. Models for AR processes: (a) the Cyclical Process Model by Susman and Evered [25], (b) the Tandem Model [17]. Table 1. Elements of the CrazySquare project, specified according to [17]: A is a real world problem situation, P is a real-world example of A that allows the researcher to investigate A, F is a theoretical premise declared by the researcher prior to any intervention in A, MR is the research method, and MPS is the method which is employed to guide the problem solving (PS) intervention. A

P F MR MPS

Issues and challenges in effectively applying a pedagogical approach oriented to musical education skills at middle school coherently with formally pre-defined learning objectives, supporting both in class and home activities Implementing a pedagogical approach oriented to a music/guitar course in Italian middle schools, coherently with the D.M. 201/99 [5] Frameworks from validated musical pedagogical approaches can be effectively blended with consolidated gamification approaches deejay MuS-LM

steer the relationships between the two cycles and the actors involved (Fig. 2-(b) in Sect. 3.2 instantiates this structure for CrazySquare). 2.2

Diagnosis of the Problem

Nowadays, in the Italian schools, musical teaching activities are organized as defined by Ministerial Decree 201/99 [5]. While on the one hand the decree explicitly specifies the musical education skills that must be acquired by children at the end of Middle School (melody, harmony, rhythm, timbre, dynamic, agogic, instrumental skills, and internalization of sounds), on the other hand, it more loosely refers to possible pedagogical approaches. Anyhow, it explicitly judges as methodological effective an adequately controlled adoption of “tools made available by modern technologies’’ [5]. We hence evaluated the technological solutions discussed in the Introduction against requirements and pedagogical objectives indicated by the Ministerial Decree and literature desiderata. The results of this evaluation are summarized in Table 2. Since none of the existing solutions satisfies our set of requirements, we decided to

230

T. Di Mascio et al.

Fig. 2. The CrazySquare deejay process.

conduct a deejay project aimed at both implementing a novel ICT solution and defining a gamified musical skill learning model. The current pedagogical approach followed by the school involved in CrazySquare is inspired by Gordon’s Music Learning Theory [11], which - among others - suggests a skill learning sequence and identifies several types of audiation. In particular, the school method addresses Type 1 (Listening to familiar and unfamiliar music), Type 2 (Reading familiar and unfamiliar music), and Type 4 (Recalling and performing familiar music from memory). More details on the mapping between types of audiation and requirements of the DM can be found in [4], which presents some preliminary results on the CrazySquare stimulation plan. While a past collaboration between the G. Pascoli school and Univaq led to a simple prototype which was a straightforward digitalization of paper and pencil exercises used by teachers [21], the present project has a more ambitious objective to develop an adaptive learning solution including state-of-the-art gamification elements [19]: (1) prizes, rewards, points, badges, levels, leaderboards; (2) immediate feedback, progress bars; (3) peer interaction and collaboration; (4) storytelling; (5) avatar, character upgrades, customization, unlockable content.

3 The deejay-Based Process of CrazySquare As discussed in Sect. 2, CrazySquare follows the deejay methodology [8]. Furthermore, we adhere to the principles of the Canonical Action Research approach (CAR) [6], which defines a set of interdependent principles and associated criteria that researchers can use to ensure rigor and relevance.

Towards a Gamified Musical Skill Learning Model (MuS-LM)

231

Table 2. The table assesses tools for the guitar (pure commercial products, scientific literature outcomes and research prototypes became commercial products) against the CrazySquare requirements: (U1) Clear reference to internalization of sound; (U2) Acquisition of the Musical literacy; (U3) Mastery of playing guitar; (U4) Gamified learning experience, (U5) Adaptive and motivating learning path. Tools Approaches Type (1) Commercial

(2) (1) and (2)

3.1

Name SmartMusic Rocksmith Research GuitarSolo GuitAR Research/Commercial Novaxe Commercial GuitarTricks Research/Commercial Rockway Commercial Yousician MelodiQ

Requirements U1 U2 U3 – – Yes – – Yes – – Yes – – Yes – – Yes – – Yes – Yes Yes – Yes Yes – Yes Yes

U4 – Yes – – – – Yes Yes Yes

U5 – Yes – – – – – Yes Yes

Research-Client Agreement (RCA)

As in any CAR project, the research client agreement (RCA) establishes - among others - focus, boundaries, and objectives of the project, defining roles and responsibilities of participants and measures to evaluate the project results (though for the sake of presentation these aspects are discussed here as pre-project agreements, in practice, they are the results of continuous reflections and refinements throughout the deejay process). The project is being carried out by a team including the two panels specified in Table 3; the team agreed that the focus of the project has to coincide with the boundaries of the diagnosed problem, i.e., we will iterate until the achievement of a validated tool. Teachers stated organizational constraints: Org1 - adherence to the Ministerial Decree 201/99 [5] and Org2 - adherence to the EU GDPR [9]. As to the client commitment, there is a word-of-mouth agreement between the school manager and the deejay panel responsible aimed at regulating project activities, e.g., among others: (1) assuring school commitment into all the stages of the deejay process (2) defining rules and roles for in-class activities (in particular regulating the access of researchers into classes), (3) relieving schools from any cost. As to roles and responsibilities, there was agreement on the fact that members from both panels have to play multiple roles and have multiple responsibilities. In particular, all panels’ members will have the responsibility of evaluating project results, based on subjective and objective qualitative and quantitative measures to evaluate the usability of the system and efficacy of the learning approach. Objectives of the immediate problem situation and of the research side of the project, as well as research questions, where refined throughout the iteration of the cyclical process and are summarized in Table 4. Notice that problem objective PO and research objective RO reflect respectively elements P and A of Table 1. In this paper, we report on results related to RO1.

232

T. Di Mascio et al. Table 3. Panels involved in the CrazySquare team.

School panel Musical instrument teachers School managers

Univaq panel 2 action researchers with background in HCI and computer science 1 junior researcher with background in computer science 1 domain-expert with background in music pedagogy 1 domain-expert with background in psychology 2 software developers

Table 4. Specification of objectives and research questions. Research questions and objectives Q0: Can music teaching be conducted through games? Q1: CONSTRAINT COMPLIANCE – Can a game-based musical pedagogical technique approach be designed so to be compatible with organization constraints? Q2: EFFICACY – Can game design be guided by preciously identified musical skills to be learned? Q3: EFFICIENCY – Are music learning activities playful (i.e., involving children and guaranteeing a correct level of commitment during homework)? RO: Definition of a game-based structured Musical Skill Learning Model (MuS-LM), with intermediate goals: RO1: Definition of the oberall MuS-LM structure mirroring the two main concepts of music aptitude and music achievements as introduced by Gordon; RO2: Enriching the MuS-LM release defined in RO1 with basic game elements (e.g., points, stars, immediate feedback) [19] RO3: Enriching the gamified MuS-LM release defined in RO2 with additional game elements (e.g., storyline, rewards)) [19] PO: Implementing a gamified ICT- tool dedicated to music and guitar learning, with intermediate goals: PO1: Implementing a gamified tool based on MuS-LM (CS-Proto1) PO2: Embedding CS-Proto 1 in a playful environment (CS-Proto2) PO3: Adding to CS-Proto2 a back-end oriented to teaching monitoring (CS-Proto3)

3.2

The Cyclical Process

Figure 2 provides an overview of the overall CrazySquare deejay process by illustrating the relationships between the two cycles in terms of time scheduling (what happens before/while/what) and information exchanged (represented by dashed intercycle arrows). Diagnosis stages were de facto described in Sect. 2.2. As to the other stages, in the following we summarize the first iteration of the cyclical process, which led to the result focus of this paper. On the research side, ActionPlanningR was aimed at designing the research project. In particular, the analysis singled out the following interdependencies between research and problem objectives: the complete achievement of RO1 and RO2 is a prerequisite for the achievement of PO1; the achievement of RO3

Towards a Gamified Musical Skill Learning Model (MuS-LM)

233

is a prerequisite for the achievement of PO2, which in turn precedes PO3. Furthermore, one may notice the structure of the action planning and action taking phases on the two sides: only once the newly design problem solving method MuS-LM (or intermediate versions of it) is available on the research side on the right, the team can start to use it as founding model of the tool in the problem solving side on the left. Conversely, the results of the evaluation of the tool may affect revisions of the model. In the remainder of the section, we focus our attention to the design of the first version of MuS-LM (ActionTakingR stage), conceived so to include all structural elements necessary for the formalization of a skill space/learning path as well as of a gamified interaction to be defined in the next iteration of the deejay design process. The first version of MuS-LM was validated through the evaluation (Evaluation/LearningPS stage) of a first based prototype implementing the data structure (ActionTakingPS) allowed us to state the achievement of RO1 (Evaluation/LearningR). 3.3

MuS-LM: The Learning Space

With the research objective in mind – and therefore in view of a generalizability of results – since the beginning we regarded the specific teaching method adopted by the school as a possible instantiation of a more general model capturing its specifics as well as principles from consolidated music-oriented pedagogical approaches, starting with Gordon. In [12] Gordon underlines the distinction between music aptitude and music achievement, where the former represents one’s potential to learn to audiate whereas the latter represents, among other things, what one has learnt to audiate. Taking into consideration the level of music aptitude is crucial in the design of an adaptive learning solution: students with low music aptitude should not become frustrated by the difficulty of proposed exercises, whereas students with high music aptitude should not become bored by the simplicity of proposed exercises. A learning model should hence be capable of embedding the two concepts of achievement and aptitude. The analysis of pedagogical approaches and of the school method led us to a general founding principle: Principle 0 (P0): Students have to achieve skills including competencies gained through learning paths composed by “experience blocks” of homogenous exercises, starting from which we derived a set of principles defining the learning space and the admitted learning paths. Principle 1 (P1): Students have to achieve “a set S = {S1, …, Sn} of (interdependent) musical skills” (e.g., Gordon’s types of audiation). In the specific problem situation (i.e., the school method), S = {A, B, C, D} where skills (addressing Audiation Types 1, 2, and 4) are defined as follows: • A (Type 1): Perceive and maintain the pulsation for predefined bpm value; • B (Type 2): Recognize and execute by reading a sequence of rhythmic symbols; • C (Type 4): Play with the instrument musical notes, articulating them through a reading of rhythmic symbols; • D (Type 4): Execute change notes and chords at different speeds. Principle 2 (P2): Achieving a musical skill requires to achieve a number of competencies associated to them (singling out different competences within a skill is also

234

T. Di Mascio et al.

necessary to address Principle P3). Therefore, the overall competencies space C derives from S by associating skills with the correspondent sets of competencies. In the specific problem situation, A, B, C, and D are achieved by acquiring competencies of two levels of difficulty (base and advanced), from now on denoted by I and II, leading to the following overall competence space: C ¼



AI ; AII ; BI ; BII ; C I ; C II ; DI ; DII



Principle 3 (P3): The learning method has to specify an overall competency learning path CLP that has to consider the mutual dependencies between skills and competences (it is impossible to teach a skill without some competence of another skill). Each competence is associated to a “competence practice space”, which is “unlocked” or “locked” depending on whether the learner has or has not already achieved all its prerequisite competences. In the specific problem situation teachers specified the following linear learning path: CLP ¼ AI ; AII ; BI ; CI ; DI ; BII ; C II ; DII




Principle 4 (P4): Each competency c 2 C is achieved by traversing a path in its competence practice space. The path EB(c) = (EB1, …, EBn) is composed of “experience blocks” of increasing difficulty. An experience block EBi is “unlocked” or “locked” depending on whether the learner has or has not “passed” experience blocks from EB1 to EBi−1. In the specific problem situation teachers modeled the competence practice spaces as depicted in Table 5. Experience blocks are generally composed of exercises and practical activities that have to take into consideration the learner’s musical aptitude, leading to Principle 5. Principle 5 (P5): Each experience block includes “mandatory” sub-blocks and “optional” sub-blocks that may or may not be disclosed depending on the learner’s performance without preventing the experience achievement and hence without discouraging learners with low musical aptitude. In other words, experience blocks can be passed (and therefore competences can be achieved) with a certain grade. In the specific problem situation the school method singles out two levels of aptitude (“low to medium”, and “medium to high”) and organizes all experience blocks according to the common model in Fig. 3-(a): all sub-blocks correspond to a bunch of exercises (see example in Fig. 3-(b) illustrating main characteristics of the single experience block of competence AII); lighter squares on the left are the mandatory sub-blocks, while darker squares on the right are the optional sub-blocks disclosed by a high performance (P = H). Until the performance remains below a given threshold (P = L) the sub-block has to be repeated, while performances of a medium grade (P = M) unlock mandatory sub-blocks and performances of a high grade (P = H) unblock also optional subblocks.

Towards a Gamified Musical Skill Learning Model (MuS-LM)

235

Table 5. The competency practice spaces.

P’= L

P=L

P=H

P = M,H

P’= L P=H

P=L

P = M,H

(a)

(b)

Fig. 3. Experience blocks in the school method: (a) the general model and (b) the case of AII.

3.4

Validation and Gamification of Mus-LM

The first version of MuS-LM was used as founding model of a first prototype implementing the overall data structure and a preliminary version of the school method. This version of the prototype has been developed for mobile devices under Android OS, using Android Studio 3.2 and the Tarsos-DSP library, a Java music library which proved to satisfy the requirements coming from school method specifics. The prototype allowed the design panels to validate the adequacy of the learning model to embed the school method and to provide a general gamifiable environment. In particular, principles P2 and P3 were judged a promising starting point for narrative structural elements, and principles P4 and P5 were judged easily mappable to

236

T. Di Mascio et al.

customary game mechanics. Furthermore, it was observed that the introduction of different kind of rewards associated also to the mere repetition of (single or blocks of) exercises can boost learners’ motivation to improve their performances (Fig. 4).

Fig. 4. The experience block in the case of DI (exercises from 183 to 202): (a) the structure and (b) a screenshot from the prototype (Goal: Executing with the guitar a Chord Change with different speeds - System Sound Type: Visual Metronome and musical accompaniment - User Input Type: Guitar)

4 Conclusions In this paper we presented activities and results of the ongoing CrazySquare project, aimed to (1) define a gamified musical skill learning model taking care of musical aptitude and musical achievements, and (2) implement a novel ICT-tool satisfying the set of requirements coming from the real case of an Italian Middel School. In particular, we reported on the general action research based methodological setting of the project (which has a value per se, illustrating a valid approach to follow when the outcome of the research cycle is the problem-solving method of the problem cycle) and on the results of the first iteration of the project, i.e., a gamifiable learning model coherent with consolidated musical education approaches and with the specific school method. Next iterations will be focused on the achievements of unfulfilled gamification objectives. Acknowledgement. Authors wish to thank Marco Pennese from Istituto Giovanni Pascoli for the fruitful cooperation throughout the project.

Towards a Gamified Musical Skill Learning Model (MuS-LM)

237

References 1. Baskerville, R.L., Wood-Harper, A.T.: A critical perspective on action research as a method for information systems research. Inf. Technol. J. 11(3), 235–246 (1996). https://doi.org/10. 1177/026839629601100305 2. Baskerville, R.L.: Investigating information systems with action research. CAIS 2(1), 19 (1999). https://doi.org/10.17705/1CAIS.00219 3. Beck, D.: Literature review: music technology. In: Education Kierstin Bible University of Arkansas ETEC 5203-Foundations of Educational Technology (2017) 4. Caruso, F., Di Mascio, T., Pennese, M.: Gamify the audiation: the CrazySquare project. In: 11th International Conference on Computer Supported Education (CSEDU2019), vol. 1, pp. 92–99. (2019). https://doi.org/10.5220/0007764900920099 5. D.M. n.201: Ministerial Decree August 6th 1999, on the subject of Reconditioning and Arrangement of experimental courses in music in Italian Middle Schools (1999). https:// archivio.pubblica.istruzione.it/comitato_musica_new/normativa/allegati/dm0608_99.pdf. Accessed 10 Feb 2020 6. Davidson, R.M., Martinsons, M.G., Kock, N.: Principles of canonical action research. Inf. Syst. J. 14(1), 65–86 (2004). https://doi.org/10.1111/j.1365-2575.2004.00162.x 7. Denis, G., Jouvelot, P.: Motivation-driven educational game design: applying best practices to music education. In: International Conference on Advances in Computer Entertainment Technology ACM SIGCHI 2005, pp. 462–465 (2005) 8. Di Mascio, T., Tarantino, L.: The structured methodological framework “deejay”: foundation and its application to the design of an ASD-oriented AAC tool. In: Smart Innovation, Systems and Technologies, vol. 158, pp. 247–259. Springer (2020). https://doi. org/10.1007/978-981-13-9652-6_22 9. EU general data protection regulation 2016/679 (GDPR), Art. 8 Conditions applicable to child’s consent in relation to information society services. https://gdpr-info.eu/art-8-gdpr/. Accessed 16 Feb 2020 10. FenderPlay. https://www.fender.com/play. Accessed 16 Feb 2020 11. Gordon, E.: Learning Sequences in Music: Skill, Content, and Patterns. Gia Publications, Chicago (1989) 12. Gordon, E.: Learning Sequences in Music: A Contemporary Music Learning Theory. Gia Publications, Chicago (2007) 13. GuitarTricks. https://www.guitartricks.com/. Accessed 16 Feb 2020 14. Jaques-Dalcroze, E.: Rhythm, Music and Education. GP Putnam’s Sons, New York (1921) 15. Konecki, M.: Learning to play musical instruments through dynamically generated lessons in real-time based on adaptive learning system. In: 25th Central European Conference on Information and Intelligent Systems, pp. 124–129 (2014) 16. Löchtefeld, M., Gehring, S., Jung, R., Krüger, A.: guitAR: supporting guitar learning through mobile projection. In: CHI 2011 Extended Abstracts on Human Factors in Computing Systems, pp. 1447–1452 (2011). https://doi.org/10.1145/1979742.1979789 17. McKay, J., Marshall, P.: The dual imperatives of action research. J. Int. Technol. People 14 (1), 46–59 (2001). https://doi.org/10.1108/09593840110384771 18. MelodiQ. https://play.google.com/store/apps/details?id=com.ultimateguitar.assessment&hl=it. Accessed 16 Feb 2020 19. Nah, F., Zeng, Q., Telaprolu, V., Ayyappa, A., Eschenbrenner, B.: Gamification of education: a review of literature. In: 1st International Conference on HCI in Business, pp. 401–409. (2014). 10.1007/978-3-319-07293-7_39 20. Novaxe. https://novaxe.com/. Accessed 16 Feb 2020

238

T. Di Mascio et al.

21. Pennese, M., Pomante, L., Rinaldi, C., Santic, M.: The crazy square: an interactive music learning environment for digital natives. In: 10th International Symposium on Computer Music Multidisciplinary Research (CMMR), pp. 526–533 (2013) 22. Rocksmith. https://www.ubisoft.com/it-it/game/rocksmith/. Accessed 16 Feb 2020 23. Rockway. https://www.rockway.fi/. Accessed 16 Feb 2020 24. Seol, S., Shin, Y., Lee, K.: Learning guitar with an embedded system. Contemp. Eng. Sci. 9 (12), 553–560 (2016). https://doi.org/10.12988/ces.2016.6441 25. SmartMusic. https://www.smartmusic.com/. Accessed 16 Feb 2020 26. Susman, G.I., Evered, R.D.: An assessment of the scientific merits of action research. Adm. Sci. 23, 582–603 (1978). https://doi.org/10.2307/2392581 27. Willems, E.: L’oreille musicale. Éditions Pro musica, London (1946) 28. Yousician. https://yousician.com/. Accessed 16 Feb 2020

Philosophical Approaches to Smart Education and Smart Cities Javier Teira-Lafuente1 , Ana B. Gil-Gonz´ alez2(B) , and Ana de Luis Reboredo2 1 2

Philosophy Faculty, University of Salamanca, Salamanca, Spain [email protected] Department of Computer Science and Automation, Science Faculty, University of Salamanca, Salamanca, Spain {abg,adeluis}@usal.es

Abstract. The impact of technology affects the educational field in an extraordinary way. The effect of technology must be treated from the educational method itself as well as its horizon in the new paradigm of citizen in this smart environment. This work proposes a revision of the immediate future of education and citizenship, as determined by the exponential impact of technology, based on relevant issues of classic philosophy and specially along the history and didactic program of the Trivium (Grammar, Rhetoric and Logic). The paper analyses firstly the perspective of the transformation of current education towards a smart education. This transition is determined by the development of Artificial Intelligence (AI), and the reflection on the competences of the 21st century, as something intrinsically related to the issues of citizenship as well as smart cities; secondly, we review the strategic and methodological proposals in accordance with this transformation, based on the theory of generative learning and on computational and algorithmic thinking; thirdly, from the point of view of contents, we analyse the importance of digital skills and, as a fundamental element of these, programming skills. Our proposal is to recover and update the contents of the Trivium, as a renewing and revitalizing element of the methodology and contents of education in the 21st century. This proposal is based on the assumption of the epistemological unity of these disciplines and on the integral anthropological vision that supports them. Both ideas acquire special relevance in the current context marked by the impact of technology as a determining element of the medium (smart education), of the methodology (generative learning, computational thinking) and of the contents (digital skills and programming). Keywords: Smart education · Logic · Rhetoric · Mindtools · Trivium · Computational thinking · Digital skills · Smart cities

1

Introduction

The immediate future of education is determined by the exponential impact of technology. This transformation of education is leading to what is known c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 239–248, 2020. https://doi.org/10.1007/978-3-030-52538-5_24

240

J. Teira-Lafuente et al.

as smart education. The International Association of Smart Learning Environments (IASLE) has defined smart learning as: “an emerging area alongside other related emerging areas such as smart technology, smart teaching, smart education, smart-e-learning, smart classrooms, smart universities, smart society. The challenging exploitation of smart environments for learning together with new technologies and approaches such as ubiquitous learning and mobile learning could be termed smart learning” [13]. These changes can be analyzed under the classic project of the Trivium perspective. This analysis is made from both points of view; from the philosophical reflections that emerge from its historical revision as well as from the content of the disciplines that compose it (Grammar, Rhetoric and Logic). The origin of the use of the term Trivium usually dates back to the 9th century, when the first textual vestige of the term was found. Trivium means triple way, referring to the first three liberal arts (Grammar, Rhetoric and Logic), which together with the four disciplines of the Quadrivium (Arithmetic, Geometry, Astronomy and Music) constituted the compendium of classical education from Antiquity to the Renaissance. Later, they maintained an unequal presence in the arts faculties of the universities, although they were relocated at different rates within the framework of modern and contemporary education, losing their central position, until they occupied the residual and testimonial place they hold in today’s school systems. Nowadays, in any case, they do not have the character of a block of fundamental and coordinated knowledge that they had in their origin. 1.1

Scenario, Content and Horizon of Education

Technology is called to be the stage, the content and the horizon of education, in the same way that it is called to be the stage, the content and the horizon of human life. The question of the technology-education binomial, therefore, overlaps with that of the technology-human life binomial. This situation, which generates so many doubts in our days, receives a particular light if we look at the meaning of education (paideia) in the Athens of the fifth century B.C., where the origin of the Trivium is usually situated. The classical paideia was conceived as education for life in the polis and, therefore, naturally, as education for life in freedom. That is precisely why the “liberal arts” received such a name, because it was the education proper to free men in the polis. The elements that determine this situation are not very different from those of today, except for an absolutely new element that has to do with the 4th industrial revolution. The city of the future, thus, is prefigured as an smart city, and in a concordant manner, education, in and for the cities of the future, will be smart education. The question obviously is: how does this affect the reality of education? With this basic conceptualization, throughout the following sections, the reminder of this paper is organized as follow. In Sect. 2 we analyze, firstly, the

Philosophical Approaches to Smart World

241

transformation of education in the perspective of Intelligent Education, determined by the development of applications carried out with AI, and the reflection on the competences of the student in the 21st century; in Sect. 3, we review the strategic and methodological proposals in accordance with this transformation, based on the theory of generative learning and on computational and algorithmic thinking; At Sect. 4, from the point of view of contents, we analyse the importance of digital skills and, as a fundamental element of these, programming skills; Finally, Sect. 5 concludes the paper.

2 2.1

Smart Education and Smart City Smart Education and XXI’s Skills

Zhu et al. [25], defined smart education this way: “The essence of smart education is to create intelligent environments by using smart technologies, so that smart pedagogies can be facilitated as to provide personalized learning services and empower learners, and thus talents of wisdom who have better value orientation, higher thinking quality, and stronger conduct ability could be fostered”. According to Coccoli et al. [3], the environments of intelligent education are characterized by their richness, interactivity and changing character, in order to be able to fulfill three objectives: (1) To take advantage of the range of technologies and services available in networks, (2) To enhance the skills and abilities of individuals and (3) To encourage interaction in collaborative environments. The consequences of this novelty take place on two levels, objective and subjective. From an objective point of view, it confronts individuals with situations for which, in principle, they lack tools and conceptual schemes. This objective novelty encompasses the personal sphere, as it is progressively immersed in an environment dominated by artificial intelligence; the work sphere, as it is constantly faced with a variation and complication of work profiles; and the social sphere, as it is subjected to the dynamics of intense social mobility and forms part of a new social and civic space defined by the elements of digital citizenship. Based on these assumptions, education must provide a high level of adaptability through multi-disciplinary workers with a wide range of complementary skills and competencies [22]. This means that learning environments must be transformed by promoting synergies between formal education systems, which by their very nature are inflexible and resistant to change, and the world of industry and organisations in general, both private and public, which are directly affected by technological development in almost real time. Thus, in accordance with the work profile and social needs, the format of education in protocols or reproducible practices must be enriched with a model that, based on the knowledge acquired (knowledge provided), facilitates the citizen of the 21st century to think for himself, connect concepts and create knowledge adapted to new problems [16]. This brings us closer to the second dimension of change. We say that there is a subjective dimension to the consequences involved, since they bring about a situation of

242

J. Teira-Lafuente et al.

insecurity and uncertainty that, in itself, constitutes an obstacle to the effectiveness of individuals’ actions and decisions. This idea is emphasized by Segredo et al. [22] when they state: “Citizens of the future must have full confidence in the tools and technologies involved in a smart environment”. Thus, an adequate training must be an instrument of personal success also subjectively, favouring attitudes and feelings of self-confidence and security. This can be achieved by promoting key skills such as problem solving [15], creativity or resilience [17] and that, in short, allow people to develop and “live effectively at work and leisure time” (Zhu et al. [26]). Technology, therefore, is revealed in this context as a fundamental in education, but not enough. Segredo et al. [22], provide us with a precise and illustrative synthesis of recent reflection on competencies for the 21st century, which outlines its main lines by determining its dimensions, skill levels, main components, basic academic goals, and ICT skills. These works show that technology should be a fundamental tool, but not the ultimate goal. Education should also provide a deep cultural qualification that enhances and values not only skills, but also the attitudes and preferences of citizens [3] with a view to the responsible development of their digital citizenship, their quality of life and, in short, their happiness. 2.2

Smart Cities Philosophical Issues

At least five main blocks of questions emerge from the above considerations, in relation to which a retrospective look at the classical world and the contents of the Trivium can help us to shed light: The Need to Include Digital Skills in a Broad Sense in Educational Design. One of the most striking methodological problems facing contemporary education is defined by the operational and organizational difficulties produced by the multiplication of disciplines, which goes hand in hand with the scientific and academic specialization that inevitably produces the development of knowledge. The need to introduce digital competencies in a broad sense, from the use of applications to programming with code in a strict sense, aggravates the problem of an already existing operational and organizational character, and makes the conceptual problem of the unity or not of the knowledge system very topical. A careful look at the history and components of the Trivium can help us to shed light on these two problems, pointing to an integral solution that starts from their common root. The origin and the definitive impulse of the complete scheme of the “liberal arts” as a systematic, complete and sufficient whole of general culture that would embrace the foundations of what we would today call letters and sciences, is rooted in the idea of unity: epistemological unity, unity of reality, anthropological unity. The disciplines of the Trivium (grammar, rhetoric, logic) and those of the Quadrivium (music, astronomy, geometry and arithmetic) formed a coherent and complete whole (’enkiklios paidea’, encyclopedic cycle of knowledge) that rested on a common epistemological foundation: mathematics [12]. This fundamental idea of a coherent and organized whole is what emerges,

Philosophical Approaches to Smart World

243

for example, in three key moments of prehistory and the history of the Trivium that we will now comment on. The first is Plato’s Philebus (18c), when Socrates invokes the number as the origin of the invention of Grammar, mythically attributed to the god Teuth [19]. The second moment is the commentary of the neo-Platonic Proclo (412–485) on the first book of Euclid’s Elements [12]: “The importance and usefulness of mathematics for the other sciences and arts, we can learn it if we think how mathematics imposes perfection and order to theoretical sciences such as Rhetoric and to all those that are executed through discourse”. The third is Book II of the De Ordine of St. Augustine (354–430), where the origin of all knowledge (and, therefore, also that which is consecrated to the study of “the meaning of words”, that is, the three disciplines of the Trivium) is attributed to the activity of “reason”: “In all these disciplines, wherever numerical proportions were encountered, they shone with more evidence and brilliance of absolute truths in the very realm of thought(...). He studied everything diligently, and realized that his strength and all his power were in the power of numbers” [1]. The Need to Include Communication and Critical Thinking Skills. In all the approaches to 21st century skills analysed, the importance of writing, critical and inventive thinking, communication, problem solving and teamwork skills stands out. This highlights the importance of all the tools that include the cultivation of Rhetoric and communication skills in general, something that stands out in all the literature, from classical antiquity to the present day. Technology Is Essential but It Is Not Enough. Objective economic and employment opportunities are accompanied by subjective challenges in the personal and social civic spheres. This same idea emerges strongly in two of the most famous myths found in Plato’s works, the myth of Prometheus and the myth of Theuth. “The theft of Prometheus is not enough to guarantee full human life. It only serves for human nutrition, so that man becomes a craftsman, a builder or a farmer, but not all professional arts together guarantee human coexistence” [9]. The other great Platonic myth, that points to the insufficiency of technical knowledge, is Theuth’s myth, whose moral is that not everyone who can discover something has also been given to understand the importance and value of their finding. In the same way that for Theuth, the possibility of writing down knowledge meant a revolutionary milestone of human consciousness and evolution, the present era hopes that technology will make it possible, not just an advance without precedents of knowledge, but even a qualitative leap forward in the evolution of the human species [7]. Previous Training of a Basic and Propaedeutic Nature, Including Literary and Humanistic Training, Is Necessary. This idea responds precisely to the idea of paidea as “general culture”, basic and preparatory to any higher specialization. If general culture [12] is defined as a set of knowledge whose acquisition

244

J. Teira-Lafuente et al.

is aimed at the maximum development of the personality, a global and comprehensive design is needed defining a progressive, interdisciplinary and complementary skills curriculum. The cycle of the seven liberal arts, the first part of which is the Trivium, was conceived as a whole of knowledge of a universal, coherent, interdisciplinary nature, made up of complementary disciplines, each of which played a specific role in the global project of training the free citizen of the polis. Furthermore, the liberal arts cycle is a direct continuation of the tradition of the enkyklios paide´ıa, the encyclopedia characteristic of Hellenism, which was joined by Cato (Ad Marcum Filium), Varr´ on (Disciplinarum libri novem) and, in late antiquity, other authors such as Macrobio, Boethius, Marciano Capela and, in the early Middle Ages, Cassiodorus and Isidor of Seville with their etymologies [12].

3

Methodology, Design and Educational Environment

With regard to smart education Zhu et al. [26] identify three essential elements: smart environments, smart pedagogy, and smart learners. In this section we will focus on the last two: pedagogy and learners. In relation to these, the main purpose coincides practically with the purpose of knowledge in the liberal arts: to provide higher thinking quality, and foster stronger conduct ability [22]. This definition has two components. First, the idea of developing intelligence as “higher thinking quality”, which in turn can be assimilated to the idea of the soul’s access through the faculty of reason to the intelligible element of reality. And secondly, ethical formation as “better value orientation”, comparable to the idea of classical formation in the virtues proper to the free man for life in the polis. From the point of view of pedagogy, Segredo et al. [22] highlight three methodological needs: (1) The design of learning processes according to the needs and preferences of the students. (2)The Application of a generative learning model in which, instead of giving priority to the reception of transmitted content, the active role of a student who, supported by the educational potential of the intelligent environment, is the main factor, and (3) The Design of intelligent environments according to a constructivist paradigm [22]. Underlying these three points, for different reasons, the central idea seems the teacher is expendable. The emerging questions are respectively: Can knowledge be effectively transmitted to an individual life if it is not from another individual life? What kind of knowledge can be transmitted without the concurrence of the personal presence and the living word of the teacher? Can something other than the master himself and his living word carry out this majeutic work? Probably it is this tradition of teaching, in which personal presence and the living word is primordial, together with the fact of the natural rigidity and reticence to change of the educational systems, which is at the base of the “technological paradox” of Salomon, to which Segredo et al. [22] refers as “the consistent tendency of the educational system to preserve itself and its practices by the assimilation of new technologies into existing instructional practices”, in such a way that technology is domesticated within the framework of the “prevailing educational philosophy of cultural transmission” [22].

Philosophical Approaches to Smart World

4

245

Digital Skills and Educational Content

In recent years it has become clear that an effective way to improve educational content in general and to introduce digital skills is to incorporate as a basis for the development of thinking in a broad sense the models of computational thinking, algorithmic thinking and programming as such. 4.1

Computational Thinking

Since its first formulation at 2006 by Jeannette Wing [24], definitions of computational thinking have proliferated. Certainly, there are discrepancies in the formulation of its basic components, but its definition attracts a fundamental consensus that becomes visible, for example, in how the International Society for Technology in Education (ISTE) and, likewise, the Computer Science Teachers Association (CSTA), state that the definition of computational thinking should encompass the following dimensions: 1. Formulate problems with a view to their solution by means of computers. 2. Organizing and analyzing data logically. 3. Representing data through abstractions, models, and simulations. 4. Automating solutions through algorithmic thinking, that is, through a series of steps ordered to those solutions. 5. Identify, analyze and implement efficient solutions. 6. Generalize the solution process to a wide range of problems [22]. 4.2

Algorithmic Thinking

Algorithmic thinking implies the following skills: 1. Analyze given problems. 2. Specifying or representing a problem accurately. 3. Finding the basic and appropriate operations (instructions) to solve a given problem. 4. Constructing an algorithm to solve the problem following the given sequence of actions. 5. Think of all possible cases (special or not) of a given problem. 6. Improve the efficiency of an algorithm [22]. 4.3

Learn to Program

Through programming we develop computer thinking, and this is a fundamental dimension of thinking that we put into practice whether we use computers or not. Programming, therefore, teaches thinking in general and also introduces computational thinking, so essential in the technological environment for which education must be prepared. Assuming the importance of programming from the point of view of education in general, which is evident in the reflection on computational and algorithmic thinking, it is now a matter of exposing the reasons why the disciplines of the Trivium are decisive in promoting and optimizing the transformation of educational systems as proposed in [3].

246

4.4

J. Teira-Lafuente et al.

Advantages of Trivium Disciplines for 21st Century Education

Along this section we justify the utility of Rhetoric, Aristotelian Logic and Fred Sommers’ TFL Logic System as Elements of 21st Century Educational Systems. Given that the study of Grammar and Literature has been preserved in current educational systems, our proposal for revitalizing education in the 21st century based on the Trivium program includes Rhetoric and Logic and/or Dialectics, which today we would call informal logic. The justification for Rhetoric has already been noted in Sect. 2.2. In relation to Logic, our proposal takes place in two steps. First, in the form of the natural language of Aristotelian term logic, and second, in the form of the algebraic version of Fred Sommers’ Aristotelian logic (TFL - Term Functor Logic). Reasons for the Introduction of Aristotelian Logic – Being a logic that uses natural language facilitates learning, or otherwise reduces the cognitive load. – It is a logic that, through its basic operations, opens access to the understanding of reality from abstract categories and, therefore, to the operations of formulation, organization, representation, abstraction and generalization typical of computational thinking. – It is a logic that allows an elementary and simple transition to the languages of mathematics, electronic design and computer programming. – The demonstration of the translatability of basic formal structures, from natural language to the language of mathematics and the different logical and computational languages, favours the capacity for abstraction and algorithmic design and creativity. – Its multidisciplinary nature makes it an irreplaceable methodological instrument, with interdisciplinarity, adaptability and flexibility as essential ingredients of citizenship and the labour market of the 21st century. Fred Sommers’ Logic. Known as Term Functor Logic (TFL), the system developed by Sommers and Englebretsen [11] is a formal logical language easily assimilated into natural language, since it emerges as a translation of it. Based on the idea that natural language is the “genuine source of natural logic”, represents the categorical propositions using an arithmetic grammar that deals with syllogistic by using terms rather than first order language elements such as individual variables or quantifiers. TFL provides a complete and simple correct decision method for Aristotelian syllogism described in Table 1. Given this algebraic representation, this plus-minus algebra offers a simple method to decide syllogism [10]. According to this algebra, the four categorical propositions can be represented by simple syntax. Its main advantages are: – Its visible “syntactic naturalness” and the simplicity of its reasoning rules, which intuitively and immediately provide cognitively relevant information and make it a “logic of reasoning in natural language”.

Philosophical Approaches to Smart World

247

– Its direct usefulness from the point of view of logical programming languages [2] and through the programming language TFL+ [4]. All of which adds important advantages in programming issues where natural language interaction with humans is ubiquitous and in areas that aim to simulate human reasoning about possibilities or inductions [5]. Table 1. The four categorical propositions TFL SaP:= − S + P =

− S − (− P) = − (− P) − S = − (−P) − (+S)

SeP:= − S − P =

− S − (+ P) = − P− S =

−P − (+S)

SiP:= + S + P =

+ S − (− P) = + P + S =

+P − (−S)

SoP:= −+S − P = + S − (+ P) = + (− P) + S = + (−P) − (+S)

5

Conclusions

Technology, thus, as we suggest at the beginning, constitute the stage, the content and the horizon of education, in the same way that it constitute the stage, the content and the horizon of human life. The question of the technologyeducation binomial, therefore, is the same of the technology-human life binomial. In this paper we have attempted to offer some remarks in the following conclusions: – The historical outline and project of the Trivium provides a valuable basis for reflection on the challenges raised by the technological revolution in the field of education, from the point of view of aims and skills as well as from the point of view of methodology. – The adoption of the schemes of computational thinking and algorithmic thinking and the introduction of digital skills, with programming skills at their core, is reinforced in its principles and procedures with the introduction of the disciplines of the Trivium, especially Rhetoric and Logic (in the Aristotelian version and in the TFL version of Sommers), in educational curricula and in training programs in general. – Finally, the Trivium disciplines (Grammar, Rhetoric and Logic) constitute an appropriate basis for the design of mind-tools devices in order to reinforce the context of smart-education. Acknowledgments. This research has been partially supported by the Department of Education of the JCyL and the project RTI2018-095390-B-C32 (MCIU/AEI/ FEDER, UE).

References 1. Agust´ın: El Orden. En Obras de S. Agust´ın, vol. I. BAC, Madrid (1946). https:// www.augustinus.it/spagnolo/ordine/index2.htm

248

J. Teira-Lafuente et al.

2. Bergin, T.J., Gibson, R.G. (eds.): History of Programming Languages II. ACM Press, New York (1996) 3. Coccoli, M., Guercio, A., Maresca, P., Stanganelli, L.: Smarter universities: a vision for the fast changing digital era. J. Vis. Lang. Comput. 25(6), 1003–1011 (2014). https://doi.org/10.1016/j.jvlc.2014.09.007 4. Castro-Manzano, J.M., Lozano-Cobos, L.I.: TFLPL: programaci´ on con l´ ogica de t´erminos. Res. Comput. Sci.Res. Comput. Sci. 147, 265–283 (2018) 5. Castro-Manzano, J.M., Lozano-Cobos, L.I., Reyes-Cardenas, P.O.: Programming with term logic. BRAIN Broad Res. Artif. Intell. Neurosci. 9(3), 22–36 (2018) 6. Clever, S., et al.: Ethical analyses of smart city applications. Urban Sci. 2, 96 (2018) 7. Di´eguez, A.: Transhumanismo. Herder, Barcelona (2017) 8. Fl´ orez, C., et al.: El humanismo cient´ıfico. Caja de Ahorros y M. P. de Salamanca, Salamanca (1998) 9. Garc´ıa Castillo, P.: Prometeo o la educaci´ on insuficiente. Campo abierto, n◦ 5 (1988) 10. Sommers, F.: The Logic of Natural Language. Clarendon Library of Logic and Philosophy. Clarendon Press, Oxford. Oxford University Press, New York (1982) 11. Sommers, F., Englebretsen, G.: An Invitation to Formal Reasoning: The Logic of Terms. Ashgate, Farnham (2000) ´ 12. Hadot, I.: Arst lib´eraux et philosophie dans la pens´ee antique. Etudes augustiniennes – CNRS, Paris (1984) 13. IASLE: Smart Learning (n.d.). http://www.iasle.net/index.php/about-us/ background. Accessed 10 Feb 2020 14. Jaeguer, W.: Paidea. Fondo de cultura econ´ omica, Madrid (1983) 15. Jonassen, D.H.: Learning to Solve Problems: A Handbook for Designing ProblemSolving Learning Environments. Taylor & Francis, New York (2010) 16. Kirschner, P., Wopereis, I.G.J.H.: Mindtools for teacher communities: a European perspective. Technol. Pedagog. Educ. 12(1), 105–124 (2003). https://doi.org/10. 1080/14759390300200148 17. NMC Horizon Report para la Educaci´ on Superior (2016). http://research.unir.net/ wp-content/uploads/2016/05/2016-nmc-horizon-report-HE-ES.pdf 18. OECD Publishing: Economic Policy Reforms: Going for Growth, 2009. OECD (2009) 19. Plat´ on: Filebo, en Plat´ on III. Gredos, Madrid (2011) 20. Evangeliou, C.: Aristotle’s Categories and Porphyry, vol. 48. Brill (1988) 21. Porfirio: Isagoge. En Categor´ıas/De interpretatione/Isagoge. Tecnos, Madrid (2012) 22. Segredo, E., Miranda, G., Le´ on, C.: Towards the education of the future: computational thinking as a generative learning mechanism. EKS 18(2), 33–58 (2017) 23. Trilling, B., Fadel, C.: 21st Century Skills: Learning for Life in Our Times. Wiley, Hoboken (2009) 24. Wing, J.M.: Computational thinking. Commun. ACM 49(3), 33–35 (2006). https://doi.org/10.1145/1118178.1118215 25. Zhu, Z.-T., He, B.: Smart education: new frontier of educational informatization. E-Educ. Res. 12, 1–13 (2012) 26. Zhu, Z.-T., Yu, M.-H., Riezebos, P.: A research framework of smart education. Smart Learn. Environ. 3(1) (2016). Article 4. https://doi.org/10.1186/s40561-0160026-2

Experiences in a Differential Equations Massive Course Rubén Dario Santiago Acosta(&), María de Lourdes Quezada Batalla, and Ernesto Manuel Hernández Cooper Tecnológico de Monterret, 52926 Atizapán Edo Mex, Mexico {ruben.dario,lquezada,emcooper}@tec.mx

Abstract. This work describes a differential equations online course based on the solution of learning challenges, through mathematical modelling. This course has been developed on the OpenEdx platform. This course is built in order to provide engineering students with the abilities necessary to analyze and solve ordinary differential equations (ODE). The course consists of five units or modules: 1) first order ODE’s, 2) second order ODE’s, 3) Laplace transform, 4) Numerical methods and 5) Partial differential equations (PDE). On each module, the necessary theory is briefly explained through interactive videos. Besides this, our course includes: an e-Book, computational simulations using applications such as Mathematica and Python, an adaptive trainer for problems and exercises, a comprehensive activity about concepts and an evaluation activity. At the end of each module, the course offers a real life situation or a mathematical challenge to be solved in a collaborative way. The goal of this course is to develop the student’s abilities for mathematical modelling of systems through differential equations. Technology being implemented lets to give an individual following to each student. Finally, we show the results obtained recently by the students that took part in the course of differential equations, and we also show several examples with their answers to the proposed problems. Keywords: Mathematical modelling


Technology
Problem solving

1 Introduction Math courses at Tecnológico de Monterrey are very robust and they seek to develop and boost the mathematical modeling and technological skills of students, besides the improvement in their algorithmic skills. In particular, the course on Ordinary Differential Equations (ODE) covers five different and complex subjects which are difficult to cover with the proper depth within the time frame assigned for the course. Several studies have concluded that students taking this course, are not able to analyze problematic situations within the corresponding context, which is majorly due for two reasons: the poor development of skills for the solution of problems, and the excessive use of algorithmic methods for the solution of differential equations [1]. As a result of this, training in mathematical modeling is lost along with the analysis of complex situations in order to dedicate more time for the study of different algorithms. This is how the possibility of using DE to describe natural phenomena related to Physics, © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 249–258, 2020. https://doi.org/10.1007/978-3-030-52538-5_25

250

R. D. S. Acosta et al.

Economy, Demography, Ecology and other areas of science, is lost. Recent papers [2] pretend to develop the necessary skills for mathematical modeling by using the interactive simulators. However, improvements are not encouraging for the development of mathematical modeling related to real world phenomena. On the other hand, several alternatives to study ODE have been implemented by using technological tools. In [1] proposes to use computational applications that support learning by using graphics, numerical and algebraic computations in order to boost the capability to solve problems through DE. On a recent work, the need to implement tools based on mobile technology and reduce time consuming algorithms in class, has been proposed in order to increase the amount of time used for activities related to mathematical modeling [3]. The students of Tecnológico de Monterrey that take the course on ODE have a poor development of their mathematical modeling and problem solving skills, and only a few of them manage to connect the theory with the real world. We propose a reduction of in class time dedicated to study different algorithms for solving ODE and increase the time dedicated to mathematical modeling through the use of several mobile technologies that allow students to dispose of interactive materials that support learning, anywhere and anytime. The goals of this work are: to determine the effects on the development of mathematical modeling skills for students that take a course of DE based on mobile technology and learning based on challenges, and to study changes on the algorithmic processes learned by those students that use online trainers for solution methods of DE.

2 Theoretical Framework In general, mathematical models follow when the need to provide answers for very specific questions related to real situations, arises naturally; or even when it is necessary to take decisions or when predictions related to natural phenomena is needed. In [4] it is suggested that teaching mathematical modeling in classrooms allows the students to tackle problematic situations of interest and develop their capabilities to explore and represent different phenomena through differential equations. Research on problem solving skills has shown that students have difficulties to translate verbal statements to mathematical language; difficulties which are even greater when problems are related to ODE. In this case, students must interpret the situation provided and determine the ODE that best describes the problem of interest. In general, the implementation of models through differential equations, is not simple, and to build these models properly, requires practice [2, 5, 6]. Other authors sustain that teaching of ODE through mathematical modeling requires the development of theoretical elements and the strategies to build the ODE [7]. Both points of view, coincide with the types of problems that students can find and consider the analysis of their solutions and the awareness about the solution process improves the learning quality. A third point of view called “Models and Modeling” stresses the importance of teaching, focused on mathematical modeling through situations on a given context, for two main reasons. The first is focused on preparing students to solve the type of problems that will face outside the university. The second reason is focused on relating these kind of problems

Experiences in a Differential Equations Massive Course

251

with the subjects that are covered on academic mathematics, even though; this relation is not clear or evident. On this line of research, the interest is centered on students developing flexible and creative ways of thinking which allows them to tackle any situation that comes up [8–10]. There is an agreement on this last posture and on a challenge based learning methodology [11], in this last methodology, the students are presented with real challenges which are supported by learning modules with a great deal of success possibilities, and the students manage to relate mathematics with several phenomena. According to the APOS theory (Actions, Processes, Objects, Schemes), students must evolve from using only procedures and actions until they achieve to develop schemes that allow them to solve problems [12]. In practice, APOS theory is used through learning cycles ACE, which are built by activities (A), in class discussions (C) and exercises (E). A wide diversity of research suggests the use of ACE cycles for mathematical teaching. For example, In [13] this cycle was used on a calculus course, and it is suggested the following: to organize students in working teams, develop all activities on a mathematical laboratory where computational packages are used for mathematical analysis, in class discussions about things learned and finish through individual exercises. On the other hand, several learning materials (software, educational assistants, e-books, support tutorials) have been developed with the specific goal to produce an improvement on mathematical learning. Experiences show that if materials are not designed specifically to a given end and a given population since their success can be reduced [14]. In [15] it is stated that “computational technologies alter the traditional balance between the epistemic and programmatic value of techniques”. In other words, even when technology pretends that students achieve more and better learning, it is necessary to take care of the problems that students have with the mathematical objects of learning. An interacting software, so the students learn a given subject or a system that takes into account the professor expertise, the understanding of students, types of teaching, characteristics of learning and ways of communication, is a good support system for learning. In [16] it is shown that learning styles constitute a key factor to build successful virtual environments. Presently, there is a tendency to build adaptive systems for mathematical learning, which take into account the students characteristics and knowledge. A software that produces semi-adaptive trainers is GenTutor, which has been developed within the Mathematica package and has been used to build algorithmic training systems on calculus courses [17]. There are new tendencies and needs about learning that have been the cause of new paradigms of education. That is why massive open online courses (MOOC) have been designed. In general, these courses use the web, the teaching experience of one or several experts and they use a great deal of learning resources. The technology necessary to develop these courses only demands basic hardware and they are available to the most part of the teaching community. For example, Open-EdX platform has been used by many institutions in order to improve the learning quality of their students. Instructional design to build a MOOC is a fundamental part on the failure or success of the course. In [18] it is recommended a six-phase methodology in order to build the course: (1) to have a clear picture of the goals that students must reach, time, dedication and deadlines. (2) To select and build learning units which must be balanced between time and content. (3) To develop a teaching guide with all things that must be mastered

252

R. D. S. Acosta et al.

by the professor. (4) To create a support teaching guide for the student. (5) To organize materials by taking into account goals and evaluation. (6) To build a guide for the communication between professor and students. In a few words, in order to promote the development of mathematical modeling skills and to improve the algorithmic skills of students arises the need to use the OpenEdX platform, to build and offer a hybrid course on ODE that uses state of the art technological tools. The course is formed by activities for the solution of exercises and interactive problems, practices for computational exploration and complex problems. The course can be consulted at the link https://bit.ly/3dGy1R4.

3 Blended Learning Proposal This work has been developed in two stages, in the first stage, following the methodology presented in [18], the necessary materials are built along with a course on the Open-EdX platform. On the second stage, several groups of students enrolled in the course of ODE were selected, so they study a sample of modules and the impact on their learning skills was analyzed. El course is made of five modules. On the first module, “Analysis of first order differential equations”, the basic concepts are reviewed and some examples applied to several fields are shown. The goal of the second module “Study of second order differential equations” is to analyze situations taken from classical mechanics, where second order differential equations appear naturally. The third module “Understanding of the Laplace transform” is dedicated to the study of a different paradigm for the solution of differential equations and systems of differential equations. On the fourth module about “The use of numerical methods for the solution of ordinary differential equations” programs are developed to solve numerically systems of differential equations and several physical systems are analyzed through the equations of Hamilton. On the last module “Basic concepts on partial differential equations” the classical equations of heat are discussed, wave equation and the Laplace equation. Each module is organized as a usual MOOC, and it displays the presentation, support electronic material, basic theory, exploration practice, examples and interactive exercises, problems and challenges, and finally the module concludes with an activity for self-evaluation. On the theory part of the module, the most important concepts are considered, the needed algorithms and relevant results. This part offers a link to a complementary video where the fundamental concepts are explained. With all this material an electronic book was developed. On the exploration practices, the Mathematica package is used to analyze concepts and solve exercises related with the subject. On the examples and interactive exercises section, typical examples are explained in detail and there is a link with the trainer that offers exercises on the subject. This trainer offers exercises of multiple choice or open questions which are chosen randomly and the students receive instant feedback on the solution process. In Fig. 1, an exploration practice about a physical system (double pendulum) is shown where the idea is to stablish a system of differential equations and their solution for small oscillations and for the general situation.

Experiences in a Differential Equations Massive Course

253

Fig. 1. Exploration practice on systems of differential equations

Table 1. Challenges with differential equations Challenge ¡No classes!

About equilibrium position

Populations dynamics Possible History Interface ice-water

Description This challenge is about the modelation of the h1n1 epidemics that took place in Mexico several years ago. Starting with the available information, students build a differential equation to describe the situation. This challenge is complemented with ethical dilemmas as well Newton’s Second Law is applied to the study of dynamic systems on stable equilibrium and which are perturbed about their equilibrium position Interaction between different species in a lake is modeled The goal is to analyze a possible scenario where an asteroid collides with Earth’s surface The goal is to analyze a possible scenario where an asteroid collides with Earth’s surface The goal is to analyze the one dimensional behavior of the interface between ice-water, a phenomena that finds applications in several areas

254

R. D. S. Acosta et al.

In the problems and challenge part of the module, complex situations are presented to the students, where the use of differential equations is fundamental to the analysis and to offer proposals for the solution. In Table 1, some challenges along with their goals are shown.

4 Methodology The research questions for this project are: the first, the use of blended learning course in differential equations improving the algorithmic competences in students? And second, the use of challenges or complex problems with technology in a differential equation course improve the math competences in the students? The research design is qualitative and quantitative and it is considered a Mooc groups and a Traditional group. In order to perform this research, several groups of four semesters with 241 students from all majors that belong to Engineering in the Campus, were selected. The courses were offered through the learning cycle of Activity-Class-Exercises-Problems, where 1. Students review the theory and perform an exploratory activity of concepts (A) by using the package Mathematica outside the classroom. 2. The subject is briefly discussed in class (C) and a few exercises are done by the students. 3. Outside the classroom, students review the examples and do the interactive exercises before the Open-EdX evaluation. Later, they perform the evaluation as many times as they believe is necessary. On each one of these evaluations, different exercises may appear (E). 4. On the next class, students solve a problem or challenge problem, and their solution process is sent to the Google-Classroom platform. The students used the material found on the Open-EdX platform as a support. On the other hand, students had partial evaluations, which are made by the staff which is expert on the subject. Exams presented by the students who took the online course as a support were analyzed through an evaluation list that takes into account the strategy, procedure and answer. These results were compared with those obtained from students that did not take this course. The reports and challenges proposed were analyzed through an evaluation criterion of four levels that considers the construction of the mathematical model, use of technology, use of mathematical language and solution analysis. Finally, a survey on the student’s opinion about the online course and the proposed activities, was performed.

5 Results In order to analyze the results obtained from the exams applied to the students, the questions were grouped in a sequential fashion. A sample of 32 students who did not take the online course and a group of 241 students who took the course were selected.

Experiences in a Differential Equations Massive Course

255

Each question was assigned a grade point of 0, 1, 2 or 3 by considering the strategy (correct or incorrect), procedure (adequate or inadequate), answer (right or wrong). In Fig. 2, the results of both groups per question and the overall result on the specific subject of Laplace Transform are shown. It is observed that in general, students that use the online course (M: MOOC) have better results in their procedures compared to those students who did not take the course (T: Traditional). Remarkably, questions related to problems with context (Aplic: Aplications) are best developed by the students who took the online course. This is because the online course was provided with tools for the study of differential equations related to mechanical systems. On questions regarding the algorithmic part of the Laplace Transform (LT), inverse transform (ILT), solution of differential equations (DE-1: First Order Differential Equations, DE-2: Second Order Differential Equations) and Differential Equations Applications (App-1 for first order and App2 for second order) the results are better in the Mooc Groups.

Fig. 2. Results taken from the exam on Laplace transform

The challenges were tackled on teams of four students through a written report and an oral presentation with the corresponding defense of the proposed solution; a rubric is used. In Fig. 3, the results taken from reports of students, belonging to four of the proposed challenge situations, are shown. In general, a proper description of the model (SolS: Solution Strategy) and use of technology (Tech: Technology), is present; however, a deeper analysis of the proposed solution is lacking (SolA: Solution Analysis). Other considered aspects are Strategy (St), Conclusions (C), and Use of Mathematics (U-Ma) Finally, students answered a survey about their perception on the course. The results of this survey are shown in Fig. 4. Note that students consider that exercises from the platform are adequate (Alg: Use of Algorithms), but they need an instruction manual to capture their answers (Guide). Besides this, they consider that the exploration practices (Math: Mathematica) were useful, the activities were extremely complex (Prob: problems; Th: Thinking activities), the teamwork (Tw) needs to improve and proper support (Supp) was not provided.

256

R. D. S. Acosta et al.

Fig. 3. Results of reports.

Fig. 4. Results from the survey about the online course

6 Discussion The obtained results on the pilot test of the hybrid course on differential equations suggest the following: students keep or maintain their algorithmic skills and improve their problem solving skills for more complex situations. This is in agreement with the thesis sustained in [8, 9] since the students get to develop creative ways of solving problems with use of technology. By trying to solve real problems, such as “No Classes”, the students get more involved on the providing ways of solving the problem and even more, use of computational tools allows them to analyze with more depth subtleties within their solutions, which is not what they do in conventional courses; a similar conclusion as that in [2, 5]. Working on this course with a methodology based on learning cycles allows students to use technology in order to better understand the mathematical ideas behind the course, which is evident upon evaluation of their proposed solution for the corresponding challenge problem. This result is in agreement

Experiences in a Differential Equations Massive Course

257

with the conclusion reported in [13], with the suppositions presented in [12] and the conclusions obtained in [7]. Students receive a challenge within a context and the evaluation criteria. Finally, the last stage of the module is focused to the evaluation of students through exercises and randomly selected problems in the Open-EdX platform.

7 Conclusion The study of differential equations is fundamental for engineering students who need to model physical phenomena in more advanced courses. The goal of this work, was find a way to boost the skills of students in two basic competencies: use of technological tools for visualization and analysis and mathematical modeling of complex situations. For this, support from the online course is needed and material developed for this purpose. Results indicate that students show improvement on both competencies and they learned some mathematical tools for later courses. On the other hand, this study suggests that students who use the online trainer are able to develop their algorithmic skills and it is possible to reduce time dedicated to the study of these processes inside the classroom. The use of learning cycles allows the professor to design a course where use of technology and mathematical modeling of complex situations is taken into account, it allows the student to develop mathematical skills just by the repeated use of algorithms. Finally, every activity proposed for this course fit within a model that allows to develop mathematical skills and boosts the technological skills of the students. This is a pragmatic and robust proposal for students and professors. Acknowledgements. The authors would like to acknowledge the technical support of Writing Lab, TecLabs, Tecnológico de Monterrey, Mexico, in the production of this work.

References 1. Santiago, R.: Ecuaciones diferenciales bajo resolución de problemas con apoyo de LearningSpace y Mathematica. Acta Latinoamericana de Matemática Educativa 15(2), 893–898 (2002) 2. Rodríguez, R.: Enseñanza y Aprendizaje de las Ecuaciones Diferenciales a través de Pensamiento Sistémico: hacia la formación de un ingeniero global. Compendio de innovación educativa 2014. Proyectos apoyados por el Fondo NOVUS (2014) 3. Santiago, R., Delgado, D., Quezada, M.: Sistema de apoyo para el aprendizaje de las matemáticas basado en Web. Compendio de innovación educativa 2012. Proyectos apoyados por el Fondo NOVUS (2012) 4. Lehrer, R., Schauble, L.: The development of model-based reasoning. J. Appl. Dev. Psychol. 21(1), 39–48 (2000) 5. Rodríguez, R.: Aprendizaje y Enseñanza de la Modelación: el caso de las ecuaciones diferenciales. RELIME 13(4–1), 191–210 (2010) 6. Rodríguez, R., Rivera, S.: El papel de la tecnología en el proceso de modelación matemática para la enseñanza de las ecuaciones diferenciales. RELIME 19(1), 99–124 (2016)

258

R. D. S. Acosta et al.

7. Trigueros, M.: El uso de la modelación en la enseñanza de las matemáticas. Innovación Educativa 9(46), 75–87 (2009) 8. Lesh, R., Doerr, H.: Foundations of a models and modeling perspective on mathematics teaching, learning and problem solving. In: Lesh, R., Doerr, H. (eds.) Beyond Constructivism: Models and Modeling Perspectives on Mathematics Problem Solving, Learning and Teaching, Mahawah, NJ, USA. Lawrence Erlbaum Associates, Possani (2003) 9. Lesh, R., English, L.: Trends in the evolution of the Models and Modeling perspectives on mathematical learning and problem solving. ZDM Int. J. Math. Educ. 37(6), 487–489 (2005) 10. Lesh, R., Sriraman, B.: Mathematics education as design science. Zentralblatt für Didaktik der Mathematik 37(6), 490–505 (2005) 11. ITESM: El aprendizaje basado en retos (2016). https://goo.gl/dA3ux8 12. Dubinsky, E.: Reflective abstraction in advanced mathematical thinking. In: Tall, D. (ed.) Advanced Mathematical Thinking, pp. 95–123. Kluwer Academic Publishers, Dordrecht (1991) 13. Vizcaino, O.: Evaluación del aprendizaje del cálculo desde una perspectiva constructivista. IPN, México (2004) 14. Rojas, Y., Muñoz, T.: Mentor: Sistema tutorial inteligente para el desarrollo de habilidades en la solución de problemas matemáticos. Revista de Investigación 7(2), 235–246 (2007) 15. Artigue, M.: Tecnología y enseñanza de las matemáticas: desarrollo y aportaciones de la aproximación instrumental. Cuadernos de investigación y formación en educación matemática 8, 13–33 (2011) 16. Fontalvo, H., Iriarte, F., Domínguez, E., Ricardo, C., Ballesteros, B., Muñoz, V., Campo, J.: Diseño de ambientes virtuales de enseñanza-aprendizaje y sistemas hipermedia adaptativos basados en modelos de estilos de aprendizaje. Revista del Instituto de Estudios superiores en Educación, Universidad del Norte (8), 42–61 (2007) 17. Santiago, R., Quezada, L.: GenTutor: un sistema generador de entrenadores adaptativo. Documento interno no publicado. ITESM, México (2013) 18. Zapata-Ros, M.: El diseño instruccional de los MOOC y el de los nuevos cursos abiertos personalizados. Revista de Educación a Distancia (45) (2015)

A Protocol for Simulated Experimentation of Automated Grading Systems Andrea Sterbini1(B) , Marco Temperini1 , and Pierpaolo Vittorini2 1 Sapienza University of Rome, Rome, Italy [email protected], [email protected] 2 University of L’Aquila, L’Aquila, Italy [email protected]

Abstract. Educational systems providing Automated Grading can be very useful for both learner and teacher. After their design and implementation, systems providing automated grading have to be tested and validated, which let the need for real world experiments arise. In this paper we describe an approach to “simulated experiments” that we hope can ease the task of the researchers developing the above mentioned systems. The concept of our proposal is in defining a simulated class by means of statistical distributions of the features that model the student and her/his work in the system. Since such features can be very different from a system to another, we try and keep the description of our framework as general as possible, and define a simulated class by the distribution of the grades that the learners should get (and the system should infer, if correctly working). We show the use of our approach with two systems, that do grading and model the students quite differently. We conclude that our simulation framework can be beneficial in validating an automated grading system, and in allowing to reflect on possible updates of the system, to make its grading more correct.

Keywords: Automated grading

1

· Peer Assessment · Machine learning

Introduction

In this paper we deal with the problem of Automated Grading (AG) of students’ tasks, by proposing a protocol, and its prototype implementation, to support simulated experimentation of grading systems. Several software systems have been (and will be) proposed to support AG, and the researchers producing such systems are constantly confronted with the task of evaluating their system’s approach, and performance, by means of experimentation with real students in a real class setting. Examples of Automated Grading System (AGS) are in [2,6]. Two other AGSs are relevant in this paper, as they are used in our cases of study: OpenAnswer [9], uses Peer Assessment c The Editor(s) (if applicable) and The Author(s), under exclusive license  to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 259–270, 2020. https://doi.org/10.1007/978-3-030-52538-5_26

260

A. Sterbini et al.

to do student modelling, mediated by the addition of limited grading work performed by the teacher. In addition, the system in [5] is an AGS based on the use of features to perform the grading, whereas the features label aspects of the student’s work. Experimenting with AGSs, during the phases of their 1) development, 2) tuning, and 3) validation, entails a remarkable effort: in each experiment a sufficient number of students has to be involved, the learners’ tasks have to be assigned and collected, real grades (teacher’s grades) have to be collected (to avoid exposing the students to possibly inaccurate AG), and at last, the relevant part of these data is fed to the system, so to compare the system’s and the true grading. Here we try and provide a protocol for the generation of simulated classes, and the application of an AGS to them, to perform what we call a simulated experiment. A simulated class is defined by the representation of its students: in principle this is done by defining a set of Student Models (SM) for the individual member of the class, computed based on statistical distribution of the modelling features. On the other hand the model features are usually specific to the individual AGS, so in this paper, for the sake of generality, we obtain their definition implicitly, by determining 1) the set of grades that the students are supposed to get (Real Grades), and 2) other sets of data, specific to the AGS. The aim of this paper is to describe the approach to simulated experimentation of automated grading systems, showing its generality, and seeing how it can help tuning and validating an AGS. In particular, we will apply the protocol to two case studies, involving the use of the AGSs mentioned earlier, one providing AG by teacher mediated Peer Assessment, and the other using a feature-based approach. In the next section we will present the schema of a simulated experiment. In Sect. 3 we will report on related work, about AGSs based on Peer Assessment, or on features quantifying aspects of the student’s work. Section 4 will explain the methods that can be used for the generation of the sample’s real grades of a simulated experiment, while in Sect. 5 we will describe the data generation “specific” to the AGS at hand, in each one of the presented case studies. Remarks about limitations of the presented research, and description of future work will conclude the paper in Sect. 6.

2

Schema of a Simulated Experiment

Here we present a schemata describing the idea of conducting an experiment by applying an AGS to a simulated class of students. The framework supporting the experiment with simulated class allows the teacher to operate according to the functional architecture shown in Fig. 1: 1. set up statistical distributions of both the real grades and of the additional data specific to the AGS at hand. This process can be performed with different approaches, as discussed in Sect. 4. About the additional (specific) data, notice that

A Protocol for Simulated Experimentation of Automated Grading Systems

261

– in our first case study (see Sect. 5.1), using OpenAnswer as AGS, this set of data would be the PA dataset, comprised of all the grades given by peers to other peers: on this dataset the AGS works, inferring the student models and the consequent automated grades; – in our second case study (see Sect. 5.2), this set will be the set of features associated to the work of each student, on whose ground the computation of inferred grades is done. 2. proceed with the actual (experimental) use of the AGS, passing to it the dataset of point 2). As a matter of fact, the results of that would be differently shaped, depending on the AGS: they can be grades inferred by the AGS, or data related to the individual students (the Student Models) that would be in turn used to compute the inferred grades. For the sake of generality, in Fig. 1 we labelled these results as Inferred Student Models. In our experiments, we will compare the effects of a (simulated) change in our class (in the first case study) or in the system (in the second case study), in terms of the inferred grades. In Fig. 1 we summarised under the functional block Comparison the analysis of the experimental data produced at step 2) above. Under the functional step Suggestions, on the other hand, we refer to the reflection activity of the teacher (or researcher) aimed at devising possible updates of the AGS. The aim of this paper is basically to verify how the simulation framework we propose supports the above mentioned assessment and reflection activities. So in Sect. 5 we will try to apply the framework to two case studies, and evaluate how the comparison between AGS results and Real Grades is feasible, and how it can help devising an AGS’s improvement.

Fig. 1. Functional Architecture of the simulative framework

In relation to the implementation of the framework depicted in Fig. 1, we have to consider that the protocol we are proposing is to be applied to a wide

262

A. Sterbini et al.

variety of AG systems, each one based on its own features. Accordingly, it is in the nature of such protocol to be “loosely specified”: to be applied to a specific AG it needs a customisation over the specific features of interest.

3

Background

We are currently unaware of other initiatives to produce simulated experiments of AGSs, so in the following subsections we discuss the merits and applications of Peer Assessment (PA) and Feature-Based Automated Grading (FBAG). 3.1

Peer Assessment

Peer Assessment (PA) [22] implies letting students (peers) grade the work of other students [12]. From a pedagogical point of view, the teacher’s assessment, PA, and self-assessment, are very important to support and train students’ metacognitive skills [15], besides helping improve actual knowledge on the subject matter [3,13]. [27] reported on the actual application of PA, concluding that the use of PA can help improve the learner’s proficiency, as witnessed by 60% of the analysed studies. In an PA based AGS, grading the learners’ work can be based on the peers grading: the software system supporting the execution of peer-evaluation sessions will produce a grade for each task, taking into consideration the grades given bypeers to-peers, and possibly the level of agreement of such grades. Often, though, the teacher’s grading work is used to calibrate the system output: the teacher’s grading provides the system with information about the “correct grade” for a learner’s assignment, and this information could also be used to influence the grades given by the system alone [17,19,26]. Another example of such systems is OpenAnswer (OA) [9], a framework allowing Teacher Mediated PA. We will use OA in one of the case studies of Sect. 5, so we give here its description. OA is a web based system where learners can participate in PA Sessions. In a PA session each learner (peer) is required to 1) answer an open ended question, and 2) grade the answers of other peers (usually 3). The quality of the peer’s answer, and of her/his assessments, concur to define the peer’s Student Model (SM), by means of two stochastic variables: Ki for peer i’s knowledge about the subject matter, and Ji for her/his capability to give sound assessments. Together with other variables, measuring the correctness of one’s answer (Ci ), and the grades one gives to others, the SMs are represented in a Bayesian Network (BN). As earlier mentioned, the teacher is in turn required to grade some answers. Such contributions provide information, that propagates throughout the BN, resulting in a progressively better configuration of the variables, so that, eventually, the grades inferred by OA, on the answers that were not graded directly by the teacher, provide good automated grading.

A Protocol for Simulated Experimentation of Automated Grading Systems

263

Let us consider the following dataset: Subject Surgery Visibility Days 1 A 7 7 2 A 5 7 ... 10 B 16 12 ... 20 C 19 4 The data regards 20 subjects (variable Subject) that underwent three different surgical operations (variable Surgery). We observe the scar visibility (variable Visibility) in terms of ranks ranging from 1 (the best) to 20 (the worst). We also measure the hospital stay (variable Days). You are required to: 1. calculate the mean (with confidence intervals) and the standard deviation of the hospital stay; 2. calculate the absolute and relative frequencies (with confidence intervals) of the surgical operations; 3. verify if the hospital stay can be considered as extracted from a normal distribution; 4. comment on the result; 5. calculate the median, the 25th and 75th percentile of the hospital stay for the different surgical operations; 6. verify if the aspect of the scar is different within the different surgical operations; 7. comment on the result. Submit as solution a text containing the list of R commands with the respective output, as well as your interpretation of the analyses 3 and 6.

Fig. 2. A sample assignment

3.2

Feature-Based Automated Grading

Let us take into account the assignment described in Fig. 2. A solution to such an assignment can be considered as a list of triples containing the command, its output and the possible comment. In the approach discussed in [5], the automated grading relies on comparing such a solution with the correct solution given by a professor. Accordingly, a student may (i) give a correct command returning the correct output; (ii) give a command with a mistake either in the call or in the passed data, such that the returned output is different from that of the professor; (iii) miss the command; (iv) interpret the result in the wight/wrong way. Therefore, in [5], the authors proposed the following procedure. Firstly, count the number of missing commands (i.e., commands that are in the correct solution, but not in the submitted one), the number of commands with a different output (i.e., commands that are in both solutions, but differ in terms of the output)

264

A. Sterbini et al.

and if the i − th comment can be considered correct or not. Then, compute a distance between the two solutions as follows:  wc · (1 − Ci ) (1) d = wm · M + wd · D + i

where wm is the weight assigned to the missing commands, M is the number of missing commands, wd is the weight assigned to the commands with different output, D is the number of commands with different output, wc is the weight assigned to the distance between the comments, and Ci = 1 if the comment is correct or Ci = 0, otherwise. Obviously, the higher the number of missing commands, commands with different output and wrong comments, the larger the distance. The parameters M, D and Ci are the features identified by the system. Finally, the distance is converted into the final grade. In Italy, a grade is a number ranging from 0 to 30, plus a possible additional mention of “cum laude”, customarily considered as 31. An exam is passed with a grade higher or equal to 18. For further details on the results of the application of the approach in real cases, the readers may refer to [4,5,14]. For the scope of the paper, it is worth mentioning that the automated grade predicts with a good agreement the manual grade (R2 = 0.740), with the aforementioned weights fixed to wm = 1, wd = 0.1, wc = 0.1.

4

Simulative Data Generation

To automatically generate the instances of the models, in a realistic manner, we use statistical distributions. To generate the distributions, a threefold approach can be followed. For each parameter of the model: – we empirically choose a “reasonable” statistical distribution (e.g., a binomial for a discrete parameter, a normal for a continuous one) with “reasonable” parameters (e.g., the average grade is 26); – starting from real data collected in previous experiments, we determine which statistical distribution fits such a data best. To this aim, the following process can be followed. First, we determine if the data are discrete or continuous. Then, we select among a list of possible distributions the candidate ones that seem to fit your data. For instance: binomial, Poisson, negative binomial for discrete data; normal, beta, gamma for continuous data. Then, we find the parameters that fit the candidate distributions to our data (e.g., using a maximum likelihood approach). Finally, we can rely on the well-know Akaike’s information criterion (AIC) [1] or the Bayesian information criterion (BIC) [23] to select the distribution that best fit the data. Figure 3 depicts three sample distributions, i.e., a Gamma with shape = 1.5 and rate = 5, a Normal with mean = 0.4 and sd = 0.1, and a Weibull with shape = 4.5 and scale = 0.7. The reader may refer to [7] for further details. It is worth noting that such a process can be strongly supported by existing software (e.g., see [10] for a solution in R [18]) and therefore automatised. In general terms, this task is usually called parametric inference;

A Protocol for Simulated Experimentation of Automated Grading Systems

265

– starting from data collected in an actual scenario, we e.g. calculate the kerneldensity estimation [24] of the distribution. Differently from the previous approach, this task is usually called non-parametric inference.

Fig. 3. Examples of three distributions

A final point that must be taken into account is the presence of missing data. For instance, in a PE session, not all peers evaluate the work of another peer. Also the rate of missing data should be considered by the simulator. The approach described above is valid to estimate the distribution of parameters that are independent. The task of estimating conditional distribution functions (e.g., we want to take into account that the manual and automated grading are related each other, that the evaluations given by a peer depend on his/her ability as defined by the teacher) is a more complex process, which has been largely approached in the scientific literature (e.g., [11]). However, in the paper and in particular in the case studies reported in the next section, we assume – for simplicity – that the distributions for the parameters of the models are independent.

5 5.1

Case Studies Case Study 1

The real data used to infer the distributions comes from a set of diverse assignments carried out both in High School (for courses on “Physics” and “Computer and networks”) and University (“Computer architectures” and “Programming Basics”). The datasets had different grading ranges (0–10, 0–14, or 0–100), so we have normalised the grades to the 0–1 interval. By applying the parametric inference process described above, we obtained: – Real grades: Weibull distribution with scale = 0.81 and shape = 4.5; – Peer evaluations: Gaussian distribution with mean = 0.72 and sd = 0.23, with a percentage of missing evaluations of 81%. (Notice that 81% is a realistic value, as each peer is grading 3 other peers, and receiving 3 grades in turn).

266

A. Sterbini et al.

A class (Class 1) was defined based on these distributions. A second class (Class 2) was then defined after changing the Gaussian distribution (reducing the mean and increasing the standard deviation, resp. 0.68 and 0.33), with the idea of lowering the students’ assessment capability. In previous work [25] we have noticed that a class populated by students with lower proficiency would feed the system with lower reliable information, increasing its prediction error. Hence, what we expect by using OA on the above defined classes would be a better quality of the AG for the first class. However, as we “lowered the students quality” just by a relatively small amount, we should not expect large differences (this is a choice made to see how sensitive to this kind of difference OA can be: the results were encouraging and we kept the choice). We have run an overall of four experiments with the two classes. In particular we have chosen to see OA’s behaviour in two distinct cases of use of the teacher’s grades, as follows: (Class 1, no Real (Class 1, (Class 2, (Class 2,

none) for the first class OA was fed with the PA data alone, adding Grades (no part of RG) in the process. 30% ) OA was fed with PA data, and with 30% of Class 1’s RG. none) same specifications of (Class 1, none), applied to Class 2. 30% ) same specifications of (Class 1, 30%), applied to Class 2.

We measure how well the OA model predicts the inferred teacher grades by computing the average absolute difference between the teacher grades and the predicted grades AvgInferred). We obtain the following inference errors for the four experiments (standard deviation is shown beside each value) (Table 1): Table 1. Prediction errors. Experiment

% of used RG AvgInferred (stdev)

Class 1, none 0

0.10 (0.08)

Class 1, 30% 30%

0.12 (0.06)

Class 2, none 0

0.17 (0.15)

Class 2, 30% 30%

0.12 (0.10)

What we get from the experiments is not completely satisfactory, but can be a source for further reflections. On the one hand the observation mentioned earlier, and the declared expectations, are quite confirmed: When OA is functioning as a traditional PA system, where no teacher’s grade is used to mediate the automated grading, the prediction error is quite larger in Class 2, confirming that the lower quality of information provided by the peers is affecting the system negatively. Moreover, we can take the results associated to a use of 30% of RG as a confirmation of the usefulness of the mediating process allowed by the use

A Protocol for Simulated Experimentation of Automated Grading Systems

267

of teacher’s grades. On the other hand, the fact that when we are letting OA use 30% of RG the results for Class 1 show a slightly higher prediction error, says that something in the system should be tuned, maybe after many further experiments. Additional benefits, for OA developers, come from the following observations. The stochastic variables used in OA, to represent PA data and SMs, can have dependencies; in particular, the value of a variable can depend on the value of another or more than one other variables, whereas the dependency is computed by means of a Conditional Probability Table (CPT). In principle, the better the CPT, the lower the prediction error of the system will be. So a key in making OA better is in the possibility to compute reliable CPTs, basing on data. The scarcity of PA datasets has limited our chances of learning CPTs from data, so the possibility to define simulated classes at will, with widely varied parameters is going (we think) to allow for a real breakthrough: once we have been able to develop a reliable methodology to define CPTs, out of general and particular characteristics of the classes, we expect to be able to configure OA in such a way to reduce the prediction error, and improve the BN-based model. And then turn back to validation of the system on real data. 5.2

Case Study 2

The real data used to infer the distributions of the features used in the Featurebased Automated Grading system comes from one year of exams, from December 2018 to September 2019. As mentioned in the background section, the features used in the case study are: – M : the number of missing commands, – D: the number of commands with different output, and – Ci = 1 if the comment is correct or Ci = 0, otherwise From these features, the automated grading is derived by the system as of Eq. 1. Furthermore, to generate the real grades, we introduced a further feature Nd which measures the normalised difference between the real Rg and the automated Ag grade. Given that a grade can vary in the range [0, 31], Nd is defined as follows: Nd = (Ag − Rg )/62 + 0.5 Accordingly, a value of Nd = 0 means the largest negative difference (i.e., Ag = 0, Rg = 31), a value of Nd = 1 means the largest positive difference (i.e., Ag = 31, Rg = 0). Such a feature then aims at capturing the existing dependence between the automated and the manual grades, without using a conditional distribution function. By applying the parametric inference process mentioned above, we obtained: – M : Poisson distribution with λ = 4.11;

268

A. Sterbini et al.

– D: Poisson distribution with λ = 1.48; – C0 : two categories distribution with relative frequencies f (C0 = 0) = 0.189 and f (C0 = 1) = 0.811; – C1 : two categories distribution with relative frequencies f (C1 = 0) = 0.344 and f (C1 = 1) = 0.656; – Nd : Gaussian distribution with mean = 0.48 and standard deviation (sd) = 0.05. According to our proposal, these distributions represent a class that we consider “similar” to the real classes encountered during the exams. We then introduce a second class, in an hypothetical scenario where we improved the FBAG system in estimating the real grades. The improvement in the FBAG system is “translated” – in the distributions – as a reduction of the sd from 0.05 to 0.03, for Nd . What we expect from the application of FBAG to the two classes is to measure an increased ability of the system to predict the manual grade from the automated one, measured through the R2 . We therefore generated the two classes, as made up of 100 students each. The results show an R2 = 0.695 for the first class. Such a result is similar to the R2 = 0.740 we measured in the real scenario (as reported in Subsect. 3.2), which confirm that the generated class is “similar” to the real one. Furthermore, by applying the FBAG system to the second class, we measured a new R2 = 0.787, thus confirming that a reduction in the standard deviation for Nd , driven by the hypothesised new setting, results in the expected change in the measured outcome.

6

Conclusions

In this paper we dealt with the problem of allowing for simulated experiments involving the use of systems for Automated Grading (AG) of students’ tasks. We proposed a protocol based on the generation of virtual classes from statistical distributions, either established with empirical assumptions or derived from real data taken from already available experiments. The aim of this approach is to reduce the efforts needed to develop, tune, and validate AGS systems. We also showed, through two case studies, that the virtual classes generated through statistical distributions seem to behave like the real classes, and that changing the parameters of the statistical distribution seems to produce the expected effects. The system is currently a prototype implementation, so we had to execute several steps outside of a general comprehensive software environment. However, taking into account that the protocol has to be “loosely specified”, so to be adaptable and customised to different AGs, we are working on its implementation as a RESTful web service [20]. We see basically two limits in the research we presented here. One regards the statistical distributions. So far, the described process do not take into account conditional distribution functions, which can be used to capture the presence of relationships between the data specific to the AGS at hand. Enhancing our

A Protocol for Simulated Experimentation of Automated Grading Systems

269

protocol by deriving conditional distribution functions from dependent observations [8,16] is a first planned improvement. The second limit regards the fact that we did not validate the effects of changing the parameters in real settings. As for this, the authors are setting up specific studies finalised to validate the results achieved from a simulative setting in a real scenario. For instance, the FBAG system is currently used to support formative and summative assessment activities in two courses, one with (usually, on average) higher grades than the other. Two virtual classes will be generated, and the best virtual weights for the distance defined in Eq. (1) will be calculated – through maximum likelihood estimation [21] – on such classes. The real results (i.e., the weights calculated with the real grades and additional real data) will be then compared with the virtual weights to validate – a posteriori – our protocol.

References 1. Akaike, H.: A new look at the statistical model identification. IEEE Trans. Autom. Control. 19(6), 716–723 (1974) 2. Ala-Mutka, K.M.: A survey of automated assessment approaches for programming assignments. Comput. Sci. Educ. 15(2), 83–102 (2005) 3. Anderson, L.W., Krathwohl, D.R., et al.: A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. Allyn and Bacon, Boston (2000) 4. Angelone, A.M., Menini, S., Tonelli, S., Vittorini, P.: Short sentences on R analyses in a health informatics subject, June 2019 5. Angelone, A.M., Vittorini, P.: The automated grading of R code snippets: preliminary results in a course of health informatics. In: Proceedings of the 9th International Conference in Methodologies and Intelligent Systems for Technology Enhanced Learning. Springer (2019) 6. Blumenstein, M., Green, S., Fogelman, S., Nguyen, A., Muthukkumarasamy, V.: Performance analysis of game: a generic automated marking environment. Comput. Educ. 50, 1203–1216 (2008) 7. Burnham, K.P., Anderson, D.R., Burnham, K.P.: Model Selection and Multimodel Inference: A Practical Information-theoretic Approach. Springer, New York (2002) 8. Crowder, M.J.: Maximum likelihood estimation for dependent observations. J. R. Stat. Soc. Ser. B (Methodol.) 38(1), 45–53 (1976). http://doi.wiley.com/10.1111/j.2517-6161.1976.tb01565.x 9. De Marsico, M., Sciarrone, F., Sterbini, A., Temperini, M.: Supporting mediated peer-evaluation to grade answers to open-ended questions. EURASIA J. Math. Sci. Technol. Educ. 13(4), 1085–1106 (2017) 10. Delignette-Muller, M.L., Dutang, C.: fitdistrplus: an R package for fitting distributions. J. Stat. Softw. 64(4), 1–34 (2015) 11. Hall, P., Wolff, R.C.L., Yao, Q.: Methods for estimating a conditional distribution function. J. Am. Stat. Assoc. 94(445), 154 (1999) 12. Kane, L.S., Lawler, E.E.: Methods of peer assessment. Psychol. Bull. 85, 555–586 (1978) 13. Li, L., Liu, X., Steckelberg, A.L.: Assessor or assessee: how student learning improves by giving and receiving peer feedback. Br. J. Educ. Technol. 41, 525–536 (2010)

270

A. Sterbini et al.

14. Menini, S., Tonelli, S., De Gasperis, G., Vittorini, P.: Automated short answer grading: a simple solution for a difficult task. In: Sesta Conferenza Italiana di Linguistica Computazionale (CLiC-it 2019) (2019) 15. Metcalfe, J., Shimamura, A.: Metacognition: Knowing About Knowing. MIT Press, Cambridge (1994) 16. Nelsen, R.B.: An Introduction to Copulas. Springer, New York (2007) 17. Piech, C., Huang, J., Chen, Z., Do, C.B., Ng, A.Y., Koller, D.: Tuned models of peer assessment in MOOCs. In: EDM (2013) 18. R Core Team: R: A Language and Environment for Statistical Computing (2018). https://www.R-project.org/ 19. Reynolds, J., Moskovitz, C.: Calibrated peer review assignments in science courses: are they designed to promote critical thinking and writing skills? J. Coll. Sci. Teach. 38(2), 60–66 (2008) 20. Richardson, L., Ruby, S.: RESTful Web Services. O’Reilly, Sebastopol (2007) 21. Rossi, R.J.: Mathematical Statistics: An Introduction to Likelihood Based Inference. Wiley, Hoboken (2018) 22. Sadler, P., Good, E.: The impact of self- and peer-grading on student learning. Educ. Assess. 11(1), 1–31 (2006) 23. Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6(2), 461–464 (1978) 24. Silverman, B.W.: Density Estimation for Statistics and Data Analysis. Routledge, Abingdon (2018) 25. Sterbini, A., Temperini, M.: Peer-assessment and grading of open answers in a webbased e-learning setting. In: International Conference on Information Technology Based Higher Education and Training (ITHET), pp. 1–6. IEEE (2013) 26. Suen, H.K.: Peer assessment for massive open online courses (MOOCs). Int. Rev. Res. Open Distance Learn. 15(3), 312–327 (2014) 27. Ten´ orio, T., Bittencourt, I.I., Isotani, S., Silva, A.P.: Does peer assessment in online learning environments work? A systematic review of the literature. Comput. Hum. Behav. 64, 94–107 (2016)

Author Index

A Acosta, Rubén Dario Santiago, 249 Agostini, Giorgia, 50 Altamirano, Matías, 85 Álvarez, Santiago, 162 Araya, Roberto, 85, 95, 137 B Basu, Anupam, 74 Bonafini, Roberto, 216 Brondino, Margherita, 10, 216 Burro, Roberto, 10 Busetta, Paolo, 127 C Caporarello, Leonardo, 58 Caruso, Federica, 226 Casagrande Conti, Laura, 50 Cirulli, Federica, 58 Cofini, Vincenza, 206 Cooper, Ernesto Manuel Hernández, 249 D Das Mandal, Shyamal Kumar, 74 Das, Syaamantak, 74 Dávila, L. P., 1 De Angelis, Sara, 50 De la Prieta, F., 1 de Lourdes Quezada-Batalla, Ma., 41 de Lourdes Quezada Batalla, María, 249 de Luis Reboredo, Ana, 239 Dellagiacoma, Daniele, 127 di Carlo, Stefano, 206 Di Mascio, Tania, 226 Dias, Almeida, 147

E Extremera, J., 1 F Felea, Cristina, 106 Fernandes, Ana, 147 Figueiredo, Margarida, 147 Forlizzi, Luca, 185 Fusco, Pierfrancesco, 206 G Gabbasov, Artem, 127 Galeoto, Giovanni, 50 Gennari, Rosella, 31 Gil-González, Ana B., 239 González, Enríque, 117 Grasso, Maria Grazia, 50 H Hämäläinen, Raija, 95 Hernández, Ernesto M., 41 I Impedovo, Maria, 117 J Jiménez, Abelino, 85, 95 L Lämsä, Joni, 95 M Maciel, Rocio, 174 Magni, Federico, 58 Manzoni, Beatrice, 58

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2020 P. Vittorini et al. (Eds.): MIS4TEL 2020, AISC 1241, pp. 271–272, 2020. https://doi.org/10.1007/978-3-030-52538-5

272 Marinangeli, Franco, 206 Marziali, Niccolò, 50 Matera, Maristella, 31 Melideo, Giovanna, 185 Melonio, Alessandra, 31 N Necozione, Stefano, 206 Neves, José, 147 Newman, Marcello Enea, 68 O Odilinye, Lydia, 195 Otero, José A., 41 Ovalle, Demetrio A., 162 P Páez, John, 117 Pasini, Margherita, 216 Perini, Anna, 127 Petrucci, Emiliano, 206 Pigueiras, Janet, 174 Pintea, Mirela, 106 Popowich, Fred, 195 R Raccanello, Daniela, 10 Ribeiro, Jorge, 147 Rizvi, Mehdi, 31 Roumelioti, Eftychia, 31

Author Index Rubio, M. P., 1 Ruiz-Zafra, Angel, 174 S Salazar, Oscar M., 162 Santiago, Rubén D., 41 Solitro, Ugo, 216 Stanca, Liana, 106 Stanca, Romeo, 106 Sterbini, Andrea, 259 Susi, Angelo, 127 T Tarantini, Eric, 20 Tarantino, Laura, 226 Teira-Lafuente, Javier, 239 Temperini, Marco, 259 Tramontano, Marco, 50 U Uribe, Pablo, 95 V Vergara, D., 1 Vicente, Henrique, 147 Vicentini, Giada, 10 Viiri, Jouni, 95 Vilchez, Cintia Scafa Urbaez, 185 Vittorini, Pierpaolo, 206, 259