Teaching Skills with Virtual Humans: Lessons from the Development of the Thinking Head Whiteboard (Cognitive Science and Technology) 9811623112, 9789811623110

Table of contents :
Acknowledgements
About This Book
Contents
1 Introduction
1.1 Recommended Reading
1.2 For Families and Educators
1.3 For Developers and Researchers
2 Why Use Virtual Tutors
3 Designing Effective Teachers
3.1 What Human Tutors Do
3.2 Learning by Teaching
3.3 Game Based Learning
3.4 The Importance of Assessment
3.4.1 Broad Topic Sequence
3.4.2 Method of Instruction
3.4.3 Lesson Sequences
3.4.4 Effective Feedback
3.5 Emotional Aspects of Learning
3.6 Summary
4 Designing for Specific Populations
4.1 Universal Design
4.2 Sensory Difficulties
4.3 Communication Difficulties
4.4 Generalisation to Novel Contexts
4.5 Summary
5 Existing Software Tutors
5.1 Early Childhood
5.2 The School Years
5.2.1 Language and Reading Tutors
5.2.2 Mathematics and Science Tutors
5.2.3 Social Skills Tutors
5.3 Adulthood
5.3.1 Health and Medicine
5.3.2 Job Training
5.3.3 Social Skills
5.3.4 Higher Education
5.3.5 Second Language Learning
5.4 Summary
6 Creating Engaging Embodied Conversational Agents
6.1 Appearance
6.2 Body Language and Speech
6.3 Presence and Interaction
6.4 User Input and Accessibility
6.4.1 Speech Recognition
6.4.2 Gesture Recognition
6.4.3 User Interface Design
6.5 Customisation and Adaptability
6.6 Socially Responsive Agents
6.7 From the Desktop to Virtual Worlds
6.8 Summary
7 Implementing a Social Tutor for Autism
7.1 Traditional Interventions
7.1.1 Story and Comic Style Interventions
7.1.2 Play Based and Peer Group Interventions
7.1.3 Applied Behaviour Analysis
7.1.4 TEACCH Intervention
7.1.5 Video Modelling
7.2 Technology Based Interventions
7.2.1 Robots and Hardware
7.2.2 Virtual Environments and Augmented Reality
7.2.3 General Software
7.2.4 Selecting Curricula
7.3 Assessing Social Skills
7.4 Summary
8 The Thinking Head Whiteboard
8.1 Design Overview
8.2 Lesson Authoring and Customisation
8.3 Technical Features
8.4 A Typical Learner Workflow
8.5 Summary
9 Evaluating the Social Tutor
9.1 Research Methodology
9.2 Aims
9.3 Participants
9.4 Measures
9.4.1 Vineland-II
9.4.2 Content Quiz
9.4.3 Software Log Data
9.5 Intervention Groups
9.6 Procedure
9.7 Results
9.8 Changes in Knowledge
9.9 Changes in Behaviour
9.10 Maintenance of Skills
9.11 Discussion
9.12 Changes in Knowledge
9.13 Changes in Behaviour
9.14 Maintenance of Skills
9.15 Selection of Measurement Tools
9.16 Limitations in Methodology
10 The Future of Virtual Teachers
10.1 Natural Interaction
10.2 Emotional Response and Authenticity
10.3 Reflective Practice
10.4 Intelligent Student Model
10.5 Game-Based Learning and Collaboration
10.6 Summary
Bibliography

Citation preview

Cognitive Science and Technology

Marissa Bond David M. W. Powers Parimala Raghavendra

Teaching Skills with Virtual Humans Lessons from the Development of the Thinking Head Whiteboard

Cognitive Science and Technology Series Editor David M. W. Powers, Adelaide, SA, Australia

This series aims to publish work at the intersection of Computational Intelligence and Cognitive Science that is truly interdisciplinary and meets the standards and conventions of each of the component disciplines, whilst having the flexibility to explore new methodologies and paradigms. Artificial Intelligence was originally founded by Computer Scientists and Psychologists, and tends to have stagnated with a symbolic focus. Computational Intelligence broke away from AI to explore controversial metaphors ranging from neural models and fuzzy models, to evolutionary models and physical models, but tends to stay at the level of metaphor. Cognitive Science formed as the ability to model theories with Computers provided a unifying mechanism for the formalisation and testing of theories from linguistics, psychology and philosophy, but the disciplinary backgrounds of single discipline Cognitive Scientists tends to keep this mechanism at the level of a loose metaphor. User Centric Systems and Human Factors similarly should inform the development of physical or information systems, but too often remain in the focal domains of sociology and psychology, with the engineers and technologists lacking the human factors skills, and the social scientists lacking the technological skills. The key feature is that volumes must conform to the standards of both hard (Computing & Engineering) and social/health sciences (Linguistics, Psychology, Neurology, Philosophy, etc.). All volumes will be reviewed by experts with formal qualifications on both sides of this divide (and an understanding of and history of collaboration across the interdisciplinary nexus). Indexed by SCOPUS

More information about this series at http://www.springer.com/series/11554

Marissa Bond · David M. W. Powers · Parimala Raghavendra

Teaching Skills with Virtual Humans Lessons from the Development of the Thinking Head Whiteboard

Marissa Bond College of Science and Engineering Flinders University Adelaide, SA, Australia

David M. W. Powers Flinders Digital Health Research Centre College of Science and Engineering Flinders University Adelaide, SA, Australia

Parimala Raghavendra Disability and Community Inclusion Unit Caring Futures Institute College of Nursing and Health Sciences Flinders University Adelaide, SA, Australia

ISSN 2195-3988 ISSN 2195-3996 (electronic) Cognitive Science and Technology ISBN 978-981-16-2311-0 ISBN 978-981-16-2312-7 (eBook) https://doi.org/10.1007/978-981-16-2312-7 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd. The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721, Singapore

Acknowledgements

A huge thank you to the young people and families who volunteered to be involved in the evaluation of the Social Tutor software. Without you the evaluation and all the valuable insight it has provided would not have been possible. Thank you also to Autism SA for allowing us to recruit participants through your communication channels. We also wish to acknowledge our colleagues at Flinders University, especially Dr. Richard Leibbrandt, for their ongoing support. Thank you for your insight, wise words, and encouragement throughout this journey. We also extend our gratitude to the authors of the social skills curricula ‘Playing and Learning to Socialise (PALS)’, ‘Social Decision Making/Social Problem Solving (SDM/SPS)’ and ‘Skillstreaming’ for allowing us to include elements of your lesson materials in our software. Having access to the evidence-based, engaging content that you developed undoubtedly contributed to the positive outcomes evaluation participants achieved.

v

About This Book

The face of education is rapidly changing in response to a demand for more flexibility in learning and a plethora of new technologies becoming available. Embodied Conversational Agents (ECAs) are one such technology changing the way we teach and learn. Often referred to as pedagogical agents or virtual tutors when used in an educational setting, ECAs are used to increase both engagement and understanding in educational software settings. Here we explore the range of educational areas ECAs are currently used in, how we can create appealing and effective agents and learning systems, and what the future holds for teaching and learning with ECAs. To assist in this discussion the development and evaluation of the Thinking Head Whiteboard, an ECA-based system used for teaching social skills to children with autism, is presented as a case study.

vii

Contents

1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Recommended Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 For Families and Educators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 For Developers and Researchers . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 2 2

2

Why Use Virtual Tutors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5

3

Designing Effective Teachers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 What Human Tutors Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Learning by Teaching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Game Based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 The Importance of Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Broad Topic Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Method of Instruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Lesson Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Effective Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Emotional Aspects of Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7 7 9 10 12 12 14 15 18 19 20

4

Designing for Specific Populations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Universal Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Sensory Difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Communication Difficulties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Generalisation to Novel Contexts . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23 23 25 26 27 28

5

Existing Software Tutors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1 Early Childhood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 The School Years . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2.1 Language and Reading Tutors . . . . . . . . . . . . . . . . . . . . . . . 5.2.2 Mathematics and Science Tutors . . . . . . . . . . . . . . . . . . . . . 5.2.3 Social Skills Tutors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Adulthood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.1 Health and Medicine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29 29 30 30 34 36 38 38 ix

x

Contents

5.3.2 Job Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.3 Social Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.4 Higher Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3.5 Second Language Learning . . . . . . . . . . . . . . . . . . . . . . . . . . Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

40 41 41 42 43

6

Creating Engaging Embodied Conversational Agents . . . . . . . . . . . . . 6.1 Appearance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Body Language and Speech . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Presence and Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 User Input and Accessibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Speech Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Gesture Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.3 User Interface Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Customisation and Adaptability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Socially Responsive Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 From the Desktop to Virtual Worlds . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

45 45 47 48 50 50 52 53 55 56 58 59

7

Implementing a Social Tutor for Autism . . . . . . . . . . . . . . . . . . . . . . . . . 7.1 Traditional Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Story and Comic Style Interventions . . . . . . . . . . . . . . . . . . 7.1.2 Play Based and Peer Group Interventions . . . . . . . . . . . . . . 7.1.3 Applied Behaviour Analysis . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.4 TEACCH Intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.5 Video Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Technology Based Interventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.1 Robots and Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Virtual Environments and Augmented Reality . . . . . . . . . . 7.2.3 General Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.4 Selecting Curricula . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Assessing Social Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

61 61 61 64 65 66 66 67 68 69 71 73 74 75

8

The Thinking Head Whiteboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.1 Design Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 Lesson Authoring and Customisation . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Technical Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.4 A Typical Learner Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

77 77 78 79 80 81

9

Evaluating the Social Tutor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.1 Research Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2 Aims . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Participants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Vineland-II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

83 83 83 84 84 85

5.4

Contents

xi

9.4.2 Content Quiz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.3 Software Log Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Intervention Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Changes in Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Changes in Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maintenance of Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Changes in Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Changes in Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Maintenance of Skills . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Selection of Measurement Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . Limitations in Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

85 85 86 86 89 89 89 91 94 94 95 95 96 97

10 The Future of Virtual Teachers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.1 Natural Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Emotional Response and Authenticity . . . . . . . . . . . . . . . . . . . . . . . 10.3 Reflective Practice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Intelligent Student Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.5 Game-Based Learning and Collaboration . . . . . . . . . . . . . . . . . . . . 10.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

99 99 100 101 102 103 104

9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 9.13 9.14 9.15 9.16

Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

Chapter 1

Introduction

In the context of education ECAs can take on multiple roles. It is common for the ECA to take on the role of the teacher, facilitating learning through a combination of explicit instruction, feedback and questioning, much like a classroom teacher would (Bosseler and Massaro 2003; McNamara et al. 2004). Another approach entails having the ECA take on the role of peer or collaborator, modelling positive behaviours and skills while interacting with the learner in order to guide the learner towards developing desired knowledge and behaviours in a more natural way (Milne et al. 2013; Tartaro and Cassell 2008). ECAs may also take on the role of another learner, requiring that the human learner teach them about a subject of interest, and in doing so expand and consolidate their own knowledge of that subject (Blair et al. 2007). We are all individuals, and as such have different learning preferences, backgrounds, and areas of need. Considering this, it makes sense to implement ECAs in educational software in the role most suited to the target audience and topic content being addressed. When teaching fact and procedure driven areas, such as mathematics or physics, a teacher ECA may be the most appropriate choice, whereas in socially driven areas, such as learning conversation skills, a collaborative peer may prove more beneficial. Of course, we are not limited to a single ECA in any given scenario, and as such a combination of teacher and peer ECAs may be the optimal solution for engaging and effective learning. While we can hypothesise which approach is most appropriate for a given context based on what works with human teachers, tutors and peers, ultimately it is through evaluation of educational outcomes that we will discover the most suitable option for any given combination of learner and content. A key consideration when developing ECA-based educational software is the ECA itself—its appearance, voice, facial and body gestures, and other mannerisms all combine to create a persona. Existing research shows that for an ECA to be engaging and well accepted, these need to mesh in a culturally realistic manner and come across as natural rather than robotic or contrived (Iacobelli and Cassell 2007). Further, the persona must be appropriate to the role the ECA is playing and be in line with existing human–human social conventions. It is not appropriate to have a

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_1

1

2

1 Introduction

young child-like ECA teaching tertiary level physics, nor an adult ECA play the role of a peer for a school age learner. In this book we provide a variety of examples of ECA-based educational software, spanning both age group and application area, a discussion of issues relating to the design and implementation of both the ECA itself and the system it is embedded in, and a discussion of underlying educational principles. As a case study we also present the Thinking Head Whiteboard, a flexible and customisable ECA-based learning system, and discuss the outcomes of an initial evaluation using this system as a social skills tutor for children with autism. Finally, we look at future directions for ECA-based educational software.

1.1 Recommended Reading This book is designed to be useful for a wide audience, and as such some sections will be more relevant for each individual reader than others. An overview and reading recommendations are provided here to assist you to get the most out of this text.

1.2 For Families and Educators This Chapter and Chapters 2, 3, 4 and 9 are likely to be most relevant, depending on your interests and goals. This Chapter and Chapters 2 and 3 cover the reasons why virtual tutors are useful learning aides, what makes an effective teacher and an effective educational environment, and which elements from these learnings can be used in a virtual setting. Chapter 4 provides an overview of existing virtual tutors, not only for children with autism, but across multiple applications, subjects, and stages of life. Chapter 9 summarises and speculates about what the future may hold for virtual teachers.

1.3 For Developers and Researchers For colleagues who are currently developing virtual tutors or embodied conversational agents for other purposes, Chapters 2 and 3 provide useful insight into what practices effective human teachers use to support their learners, and how we can adapt that into a software context. Chapter 5 addresses specific recommendations about designing Embodied Conversational Agents that are engaging, trustworthy, and appropriate for their intended purpose. Chapters 6 and 7 are specifically aimed at developers who are creating educational software for individuals with autism, and

1.3 For Developers and Researchers

3

these lead into Chapter 8 where we share the details of the evaluation of our own Social Tutor for children with autism, including technical challenges faced, lessons learned, and recommendations for future developers.

Chapter 2

Why Use Virtual Tutors

There are a multitude of reasons why virtual tutors, or pedagogical agents as they are often known, are appealing for use in an educational context. They not only provide an opportunity for one-on-one interaction, but importantly they allow the learner to work through material at their own pace rather than being subjected to the constraints of a classroom, as well as giving learners a way to revisit and consolidate their skills as needed in their own time and without outside pressures. Autonomous virtual tutors allow learners to practice their developing skills independently, relieving some pressure from caregivers, teachers, and other professionals who work with the student, and can be used to complement the educational content being provided in the classroom and other settings. This also allows those working with the learner to focus on the more complex and specific aspects of the learner’s education, while many routine and general points are covered by the virtual tutor. Another advantage is that the virtual tutor will never get tired or impatient, unlike even the most patient human teacher (Massaro 2004). Virtual tutors provide consistent feedback and behaviours, which can help control anxiety in those who feel more at ease in predictable situations (Parsons et al. 2000; South and Rodgers 2017). Additionally, virtual tutors can provide a stress-free learning opportunity as the anxiety connected with interacting with real humans is removed, and the tutor can be programmed to ensure that it only provides positive and guiding feedback, rather than negatively perceived judgement or criticism that other humans may provide. Software-based virtual tutors are highly customisable and can be tailored to suit the individual learner’s needs, an important consideration for any learner, but especially for those with special or complex needs (Ploog et al. 2013). For example, for a learner with autism who finds looking at faces uncomfortable, the virtual human’s appearance could start out very cartoon-like and, as the learner becomes accustomed to it and their confidence grows, the realism and complexity could gradually be increased. Similarly, the lesson content can be modified to meet the individual’s current level of interest and need. For example, generic images can be replaced with those that have special significance to the learner, and any taught phrases can be updated to match those being learned in school or therapist-based interventions the learner is © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_2

5

6

2 Why Use Virtual Tutors

participating in. Being software-based, many different media can be incorporated into the learning material, including line drawings, photos, videos, animations and more, and training with a variety of stimuli is one method that can help to support generalisation of skills to novel situations (McCleery 2015; Stokes and Baer 1977). Furthermore, multiple tutor ‘personas’ with unique appearances and voices can be used to model target behaviours in an effort to improve the likelihood of generalisation to multiple people and situations. The flexibility and customisation that virtual tutors offer make them a cost effective and potentially highly beneficial intervention tool. Not only can the tutor appearance and learning material be customised to suit the individual, but also the educational experience as a whole. A key motivator for developing virtual tutoring software is that over time it can learn about the individual using it and consequently adapt to their needs in a dynamic way, such as only offering complex lessons once simpler prerequisite lessons have been successfully completed or presenting more of the activity types that the learner has been successful with and fewer of those found less beneficial. Case Study: Social Tutor for Autism Virtual tutors are well suited for use with individuals with autism for many reasons, not least of which is the widely acknowledged affinity that many individuals report having with computers and technology (Baron-Cohen et al. 2009; Putnam and Chong 2008). Especially important in the context of a social skills tutor, using a virtual teacher also means that the learner can practice their skills without interfering with others or learning inappropriate responses from incidental people in the learning environment (Gay et al. 2016; Kerr 2002). It is openly acknowledged that nothing should aim to replace genuine interaction with peers and others when learning about social interaction; however, an independent learning tool, such as one incorporating a virtual tutor, can provide a valuable first step in developing these complex social skills. In the context of developing nonverbal skills, animated virtual tutors can be particularly useful as they can model behaviours for the learner, such as facial expressions, body language and gaze behaviours. This is akin to the video modelling technique which has had success with many individuals with autism (Ploog et al. 2013). This benefits learners at all locations on the spectrum, from gifted students who move quickly through their tasks to those who need a little more time and support. Furthermore, the technology required to use virtual tutoring software is becoming more and more accessible and affordable for schools and families, with tablet and mobile computing in particular developing at a rapid pace in recent years (Meder and Wegner 2015; Ploog et al. 2013).

Chapter 3

Designing Effective Teachers

When developing educational content in any context, the educational process that takes place and the resources used must be carefully considered. Existing research has shown that one-on-one tutoring produces greater understanding and a higher level of motivation in students than traditional classroom situations, with students also able to progress through topic content at a faster rate. The average performance of students in a one-on-one tutoring situation was found to be up to 2.3 standard deviations above the average of students in a typical classroom situation (Chi et al. 2001; Graesser et al. 1999), providing strong evidence for the benefits that a personal tutor, be it human or ECA, can provide a student.

3.1 What Human Tutors Do In the case of a virtual tutor, understanding what makes human one-on-one tutoring effective is essential to developing a useful application. As the meta-analysis by Bowman-Perrott et al. (2013) demonstrates, peer tutoring is a well-established evidence-based practice that has been successfully used across a range of subjects, settings and age groups, and is shown to be effective for learners both with and without disabilities. There are two main schools of thought about what makes tutoring effective, one being that it is the tutor’s actions that result in positive learning outcomes, and the second suggesting that it is the student’s ability to construct knowledge and build connections between concepts that results in learning, and that successful tutors facilitate this process (Chi et al. 2001; Hmelo-Silver and Barrows 2015). It was found that even when tutors were restricted from giving explanations and feedback and could only prompt the students, students learned just as effectively, however this approach relies on students having access to the information they need in a format that they can consume, and it is suggested that the effect is due to the students having to take more control of their own learning (Chi et al. 2001). © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_3

7

8

3 Designing Effective Teachers

There are many hypotheses surrounding why a tutor’s actions can lead to positive learning outcomes, some with more evidential support than others. One is that human tutors are thought to engage in continuous diagnostic assessment of their tutee, identifying gaps in mastery, misconceptions and lacking skills. Unfortunately, in practice human tutors rarely engage in this behaviour and often fail to ask questions that could help them unearth this information about their tutees (Putnam 1987; VanLehn 2011). While acknowledged as a core component of teaching ‘best practice’ (Ritter et al. 2007), this is an issue that has been acknowledged for some time. For example, Putnam (1987) observed expert teachers working with students on simple mathematics problems and found that they only appeared to explicitly determine the nature of a difficulty before correcting it 7% of the time. Another hypothesis is that tasks can be individualised for the tutee, however again human tutors are often found to simply work from a curriculum script with only minor deviation, much the same as in a modern classroom where providing differentiated curriculum for learners is expected. Learner control of dialogue and the broader domain knowledge of the tutor are two more areas where tutoring is hypothesised to be at an advantage, as students can ask as many questions as they need to achieve understanding and tutors can explain concepts in depth and in alternative ways, however it has been shown that students rarely take initiative outside of confirming that a statement they make or a behaviour they are performing is correct, and the broader domain knowledge is rarely utilised to advantage (VanLehn 2011). In all of these areas, computer tutoring systems can perform in a similar manner to how human tutors behave in practice, although there is a lot of scope for both human tutors and computer systems to increase the richness and personalisation of their teaching approaches here. A tutor action hypothesis that appears more promising is that of immediate feedback and prompting. This process lets the student know they are on the right track and guides them towards correct understanding (Bowman-Perrott et al. 2013; Chi et al. 2001). In a one-on-one scenario, tutors typically allow the student to continue at their own pace until they get stuck or make a mistake, they then intervene to resolve the issue so the student can continue without losing momentum (VanLehn 2011). In a classroom scenario, the student’s error may not get identified straight away or they may not receive assistance immediately, causing them to stall and lose this momentum, become unnecessarily frustrated or confused, and possibly resulting in the need to backtrack and re-do work. Tutors also typically encourage students to ‘think out loud’ and explain their reasoning as they go, making it easier to identify misunderstandings and facilitating students to become actively engaged with their learning, rather than passive recipients of information (Chi and Wylie 2014; Merrill et al. 1992). There are a number of techniques that can be used in software-based tutoring systems to emulate this behaviour to benefit learners, such as the previously mentioned scaffolding approach, specifically breaking large tasks into smaller subtasks that can be individually assessed with feedback provided to assist the learner in future attempts.

3.1 What Human Tutors Do

9

As is evident from its wide implementation in the social skills interventions already discussed, scaffolding can be a very powerful learning tool, particularly for individuals with autism who need concepts to be explicitly taught. Specifically, scaffolding often involves decomposing complex concepts and tasks into simpler, more manageable subtasks, and within a lesson sequence this often takes on the broad format of introduction, demonstration, then practice, where demonstration for social skills in an autonomous tutoring application could involve multiple agents acting out the interaction to be practiced (Jackson et al. 2010; van de Pol et al. 2015). Tutors encourage learning by guiding students through this process, helping learners to master the subtasks, and gradually working towards the final goal. As opposed to a classroom setting where the learner often passively receives information, a tutoring session encourages students to interact with their new knowledge through predicting, justifying, criticising and otherwise engaging with the material (Chi et al. 2001; Chi and Wylie 2014). Interesting to note is that most tutors lack formal training, and yet tutoring is a very effective educational tool even when feedback is provided by a peer or other non-expert (Chi et al. 2001; Hamer et al. 2015). This suggests that even an imperfect tutor can provide great benefit, and thus indicates that an imperfect virtual tutor can also still be valuable to students. Tutors typically follow a set pattern when working with learners. First, the tutor asks a question, to which the student provides an answer. The tutor provides feedback and performs scaffolding across a number of turns with the student in order to help the learner develop their understanding. Finally, the tutor assesses the learner’s comprehension of the taught content (Chi et al. 2001). This same pattern can be performed by a virtual tutor. Throughout this process, the tutor monitors the learner for confusion and frustration, as deeper learning is achieved when learner misconceptions are addressed immediately. For a virtual tutor to identify when learners are struggling or have misunderstood, several methods can be employed. Basic approaches such as tracking student performance across and within tasks, analysing the mistakes made or tracking the number of times students engage with the same task, are relatively straightforward to integrate into a software tutor. Another approach with some potential is detection of frustration, confusion or boredom through facial expression recognition, emotion in speech, or use of external sensors. While emotion detection is outside of the scope of the current project as it relies on the availability of particular hardware, such as a webcam, microphone or biometric sensors, understanding the emotional aspects of learning and the impact different affective states have on learning outcomes is important when designing educational activities.

3.2 Learning by Teaching In previous sections some examples of teachable ECAs were discussed, namely SimStudent and Betty’s Brain (Ogan et al. 2012). These ECAs employ a ‘learn by teaching’ paradigm where the human student teaches their ECA pupil about the

10

3 Designing Effective Teachers

target content. How well their ECA pupil performs when it undergoes assessment, for example by competing in a quiz against other ECA pupils, provides an indication of how well the human learner understands the target content. Existing research into human–human interaction suggests that tutoring others benefits both parties involved, with the tutor consolidating their own understanding by explaining it to their tutee, taking a deeper responsibility for their own learning and having an improved self-concept. Tutoring involves three phases; preparing to teach, teaching, and recursive feedback. Research shows that individuals who study material with the expectation of teaching it to a tutee outperform those who learn the same material but without the expectation of teaching it, and that individuals who both prepare to teach and actually undertake the teaching perform better still, as they retain the information for longer (Fiorella and Mayer 2013). It is believed that this is because the expectation of teaching causes individuals to engage in generative learning, whereby they make sense of the material by integrating it with their existing knowledge, rather than engaging with the material in a more passive or rote-learning sense (Fiorella and Mayer 2013). By actually teaching the material individuals reinforce and consolidate their knowledge. Current research has shown that these effects hold in virtual environments, and that recursive feedback helps tutors improve even further (Okita et al. 2013). Recursive feedback occurs when the tutor observes their tutee apply what they have been taught, for example by interacting with an examiner (Okita et al. 2013). It was found that tutors who engaged in recursive feedback outperformed those who did not. Okita et al. (2013) suggest that this could be for two key reasons; first, during recursive feedback the tutor is not actively engaged in the teaching process and therefore has more cognitive capacity to assess the performance of their tutee and identify the underlying causes of any issues; second, by observing a tutee being assessed rather than being assessed directly themselves, any ego threats are removed, and the tutor is not affectively impacted by negative feedback (Okita et al. 2013).

3.3 Game Based Learning Play is an important part of learning, especially for children. Game-based learning is likewise thought to have many positive impacts on learning in that it is engaging, intrinsically motivating and increases the positive associations children have with the target subject matter. Computer games often facilitate collaboration and teamwork which are important life skills in their own right, as well as further enhancing individual learning as participants strive to explain their understanding and point of view to their peers, consolidating their own knowledge as a consequence. Existing research in this area contains many examples of game-based learning resulting in positive academic outcomes, however whether game-based learning is superior to traditional classroom methods remains to be seen (Meluso et al. 2012). Here we argue that this is not the goal, and rather computer games for education should be

3.3 Game Based Learning

11

seen as another tool for educators to use to enhance and diversify learning, rather than a replacement for traditional methods. Educational software involving ECAs also achieves many of the same benefits as game-based learning, being engaging, motivating and often providing for collaboration opportunities when human peers and teachers are unavailable to participate. Including ECAs in game based educational software has much potential. Academic self-efficacy is an important skill for life-long learners, with students strong in this skill often choosing to challenge themselves with trickier activities and persevering even when things are difficult (Meluso et al. 2012). Self-efficacy is an important predictor of academic success that can be nurtured, rather than being a static ‘trait’ of an individual. Recent research has begun looking into game-based learning as a means to improve self-efficacy and found positive results. Meluso et al. (2012) conducted an evaluation involving 70 elementary school students with diverse cultural backgrounds who were asked to use the computer game ‘Crystal Island’ to learn about science. They found a significant increase in self-efficacy from preto post-test, along with a significant increase in content knowledge (Meluso et al. 2012). As discussed previously, ECAs with particular personas such as motivators and mentors have also been shown to improve self-efficacy in learners (Baylor and Kim 2004). Combining these approaches could potentially lead to even better outcomes for learners. To investigate the inclusion of game-based features in existing ECA-based educational software, Snow et al. (2013) examined interactions between 40 high school students and the iSTART-ME software previously discussed. Students used the system over 8 sessions and for at least 1 h per session. It was found that students who interacted with the system more frequently and earned more in-game ‘trophies’ from playing optional practice games also displayed significantly higher academic achievement and reported greater enjoyment of the system. This supports the notion that game-based features enhance engagement, and it follows that students who are more engaged will spend more time with the software and ultimately exhibit better outcomes. In the context of games, flow theory is an important consideration. Hsieh et al. (2013) explain that within ‘flow theory’ flow involves being completely focussed on the activity at hand and engaging whole-heartedly with it without self-consciousness. Through their research Hsieh et al. (2013) confirmed that students engaged in higher flow experiences tend to have better learning performance than those in lower flow experiences. To achieve higher flow, feedback and clear goals must be provided and a balance struck between the level of task difficulty and the skills required to successfully complete it. The goal is to have the learner working within the zone of proximal development, where they have the prerequisite skills and knowledge mastered and need only a small amount of appropriate guidance for them to make new connections and acquire the target skill (Wertsch 1984). If they lack the appropriate prerequisites or their ‘flow’ is interrupted or hindered in some other way, concentration and learning can also be interrupted. Flow is an important consideration when designing educational software, but particularly that which is game-like, as it impacts on students’

12

3 Designing Effective Teachers

enjoyment and motivation to not only continue with the current session but return independently in future for more.

3.4 The Importance of Assessment Assessment is a core feature of any educational program, be it a traditional classroom approach or within educational software such as that which is the focus of here. To best meet the needs of learners, assessment should be targeted, purposeful and ongoing, with assessment outcomes explicitly used to inform future learning activities. Here a range of assessment tools and techniques are discussed in the context of developing an ongoing, automated assessment and dynamic lesson sequencing system for a virtual tutor, with examples drawn from existing virtual tutors and other computer-aided learning software. In order to successfully meet the academic needs of learners, first what they already know must be accurately assessed and this information used to make an informed decision about what to teach them next. In a virtual tutor, this process must also be automated so it can be performed continually by the software. Four broad applications of assessment are addressed here—determining the topic sequence, i.e. what large-scale skills need to be taught; determining the method of instruction, i.e. how to teach these skills based on the individual’s needs; determining the lesson content on a smaller scale, i.e. what tasks to present to the learner to help them improve at the current skill; and finally providing effective feedback.

3.4.1 Broad Topic Sequence Developing a curriculum and experimentally validating the content for its educational efficacy is a considerable undertaking, and for many developers may be outside of their area of expertise or simply impractical due to financial and time constraints. An alternative is to incorporate the content of established, experimentally supported curricula, and use that content to guide the higher-level topic sequence that learners are provided with. Case Study: Social Tutor for Autism The Thinking Head Whiteboard draws its content from three established and empirically supported social skills curricula, The ‘Playing and Learning to Socialise’ (PALS) curriculum (Cooper et al. 2003) is aimed at kindergarten aged children and provides fundamental skills, while the ‘Skillstreaming’ (McGinnis and Goldstein 2012) and ‘Social Decision Making/Social Problem

3.4 The Importance of Assessment

13

Solving’ (SDM/SPS) curricula (Butler and Poedubicky 2006) build on this foundation and provide more advanced instruction. Software aimed at individual use on home and school desktop computers must provide self-contained practice and assessment opportunities that do not require a peer or parent to be present. For many academic skills this is straightforward, however for less clear-cut skills such as those involved in social interactions and for skills involving physical processes both practice and assessment can be much more challenging. In a software context, direct observation of the learner performing a skill in a natural situation is typically not viable; for example we do not currently have a straightforward, reliable way to assess whether a learner has successfully performed the skill of ‘greeting someone’, ‘washing hands correctly’ or ‘baking a cake’. Given this, one option is for the learner to self-report, however while this has been shown to be accurate for purposes such as assessing anxiety and depression (Ozsivadjian et al. 2014) great care must be taken when using self-reports for assessing other skills, as discrepancies can exist between what a student knows they should do and what they actually do, and whether a difficulty stems from a skill deficit or a performance deficit greatly influences the educational tasks required to overcome it (Bellack 1983). Instead, opportunities for observing, interacting with and responding to virtual roleplays and scenarios can be used to help increase the level of realism and consequently make differences between skill and performance deficits more easily detectable. Another viable option is to allow educators and caregivers the ability to complete a skills assessment for the learner in question when a new account in the software is created, and then allow them to input updated information as it becomes available over time. This may be particularly beneficial in conjunction with virtual role-plays and scenarios. Case Study: Social Tutor for Autism In the current implementation of the Thinking Head Whiteboard, the three selected curricula are used to provide the broad topic sequence, and students simply choose which topic they feel will be most beneficial to them at the time. In future iterations of the software when more of the curricula content is implemented, a standard test may be incorporated to ensure that students begin at the right difficulty level within the content, and to periodically check their progress and adjust learning activities accordingly. Of course, another alternative is to give learners the reins, and simply provide them with a choice of topics drawn from the overarching curriculum content. To avoid overwhelming learners this can be approached in a hierarchical manner, with more challenging content becoming unlocked as learners demonstrate mastery of more basic concepts. To provide further support, the system could highlight or otherwise

14

3 Designing Effective Teachers

prioritise topics and tasks that it has identified as most relevant to the learner at the given point in time, while still providing users with choice and control over their learning.

3.4.2 Method of Instruction Educational experiences that mesh well with the current knowledge and preferred learning style of the student are known to improve the processing of new knowledge, facilitate a deeper understanding of the content, and generally expedite the learning process (Truong 2016). A number of factors influence a student’s preferred learning style at any given moment, including their pre-existing preferences and their level of experience with the current concept (Truong 2016). For example, it has been shown that inexperienced and experienced learners display different needs, with inexperienced learners gaining more from following worked examples and experienced learners benefitting more from solving problems (Wittwer et al. 2010). It should be noted that care must be taken when considering the use of exploratory educational games such as the one used by Robison et al. (2009) as it has been shown that these typically only benefit students who already have the skills to gain knowledge from this style of task, whereas less skilled learners need more structure (Conati 2002). Case Study: Social Tutor for Autism In terms of learning preferences, it is often found that individuals with autism fare best with visual information over spoken instruction (Knight et al. 2015; Shane et al. 2009). While all learners are individuals, this aligns with the communication difficulties that are a core deficit of autism. An automated mechanism that can detect and implement the appropriate method of instruction for the student’s current situation, much like human educators unconsciously do, would be a valuable component for an autonomous tutoring system. Shute and Towle (2003) present a generic framework for intelligent tutoring systems that takes into account individual learner differences, the learner’s current state of knowledge and best practices for instruction of the learner. It is based on Dick Snow’s aptitude-treatment interaction (ATI) research, which aims to quantify and predict diverse learner profiles to allow for lesson presentation and content to be adapted to the learner’s needs. Content presentation can range from step-bystep, highly structured instruction to exploratory presentation where the student has nearly complete control over the lesson sequence, with different presentation styles suiting different learning strategies. The three elements presented in the framework of Shute and Towle (2003) are the content model, learner model and instructional model, with these elements being used by an adaptive engine to determine what and how content should be presented. Shute and Towle (2003) then propose the use

3.4 The Importance of Assessment

15

of Learning Objects (LOs) to facilitate flexible content presentation. LOs are small, self-contained and reusable components that can be combined into lesson sequences. Each LO should be limited to one of the three types of knowledge: basic knowledge, which includes facts and formulas; procedural knowledge, such as steps and skills; or conceptual knowledge, which covers understanding and theory. Sets of LOs that comprehensively teach a particular skill or knowledge set can then be defined, with relationships between LOs influencing the sequence that tasks are presented in. This approach is very flexible and provides a good framework for any autonomous tutoring system where the goal is to dynamically respond to learner needs.

3.4.3 Lesson Sequences In order to present students with learning tasks suited to their current needs, it is essential to continually assess their state of knowledge. Research suggests that in order to be most effective, assessments should be integrated into the overall learning sequence rather than viewed as a separate activity and used to continually inform and adjust the activities presented to learners (Black 2015). It is often seen that students learn how to complete a task or pass a topic without gaining any deep understanding of the topic material covered (Conati 2002). Providing opportunities for reflection on the processes and concepts involved, for example in self-explanation tasks, ensuring that any reward activities do not distract from the desired lesson outcomes, and implementing robust methods of assessing student knowledge all contribute to combating this issue. Shute and Towle (2003) state that common methods of evaluating student mastery are insufficient, for example simply getting a particular percentage or a certain number of consecutive assessment tasks correct. Instead, Shute and Towle (2003) suggest the use of Bayesian inference networks (BINs) or student mental modelling to provide probabilistic values which can be used to determine gaps or misunderstandings in the learners’ knowledge map. Case Study: Social Tutor for Autism When determining assessment techniques to implement, it is essential to consider the needs and capabilities of your tutoring system’s users. While traditional BIN and LSA based techniques are promising, to successfully implement these requires tasks with open-ended or flexible answers, which in a software environment typically translates to writing paragraph-style answers. This is highly challenging for individuals with autism who experience both language and communication difficulties, making them unsuitable for this population. In many autonomous tutoring applications, a common approach to judging students’ knowledge is to use latent semantic analysis (LSA) techniques to judge

16

3 Designing Effective Teachers

the semantic similarity of student responses to a provided ‘ideal’ response. This is the approach taken in the successful iSTART tutoring system discussed earlier (Jackson et al. 2010). Hu and Xia (2010) also use latent semantic techniques in their automated assessment system and found no significant difference between the grades provided by their system and those provided by teachers, suggesting that this is an educationally valid technique. More recent work by de Klerk et al. (2016) involves the use of multimedia performance-based assessment, where users interact with a virtual lesson and the interaction data is fed into a BIN for assessment. This approach has the potential to mesh well with role-play and scenario driven learning tasks and ensures that learners with less developed reading and writing skills are still able to demonstrate their knowledge effectively. Meyer and Land (2010) recommend the use of speak aloud self-explanations as a reflective practice. Meta-cognitive skills and reflective practice, such as selfexplanations, have been demonstrated to lead to better problem-solving skills and the construction of deeper, more meaningful conceptual connections (Amico et al. 2015; Mitrovic 2001). Such meta-cognitive skills can be nurtured in students to help them improve their ability to learn. Amico et al. (2015) reviewed a year-long drama therapy course for developing social skills in adolescents with autism and emphasised the benefits of reflective practice and having students explore the perspectives of other characters. Nicholas et al. (2015) found that for neurotypical young adults, recall of events that occurred in a virtual world could be enhanced by the use of highly detailed reminiscing involving open-ended questioning by a virtual partner. Mitrovic (2001) conducted a study with university level computer science students to evaluate their self-assessment capabilities. It was found that more able students displayed better understanding of their own educational needs, while less able students abandoned many more practice questions, often citing that the problem was too easy even when evidence suggested otherwise. This suggests that a system that prompts students to consider more carefully the reasons for their difficulties may help to nurture metacognitive skills and improve educational outcomes. Case Study: Social Tutor for Autism To encourage reflective practice, the Thinking Head Whiteboard includes two part ‘homework’ tasks where the first part asks students to plan their homework, and the second part asks them to reflect on how they went and why. Black and William (2009) also emphasise the importance of reflective practice for deep and long-term learning. They suggest that reflection can assist students to make the processes they unconsciously use explicit and concrete, making them easier to understand and implement in future. It is suggested that discussion with peers and others improves the outcomes of reflective practice, in following with Vygotsky’s principle that ideas are initially constructed in social interactions, and then internalised by the learner (Black and Wiliam 2009). Additionally, challenging

3.4 The Importance of Assessment

17

students to identify other situations where they can use the same thinking processes, to compare and contrast ideas, and to critically analyse their ideas, can help learners improve their problem-solving and cognitive skills in general and to apply their skills to other areas. In addition to speak aloud self-explanation, Meyer and Land (2010) recommend conceptual mapping as a method of making misunderstandings and barriers to knowledge observable and hence manageable for educators, and a recent meta-review further supports the use of various graphic organisers for supporting individuals with autism to organise and express their knowledge effectively (Finnegan and Mazin 2016). Concept maps are particularly applicable to autonomous tutoring software as they can be automatically assessed and have demonstrated educational benefits in a range of settings (Finnegan and Mazin 2016; Kinchin 2014; Roberts and Joiner 2007). Existing research by Kinchin et al. (2000) suggests that concept maps allow educators to discover what students really know and how their knowledge is interconnected, rather than trying to make judgements and informed guesses, and emphasises the importance of synthesising and integrating ideas and concepts rather than simply repeating isolated facts. In a review of the existing literature around concept maps, Kinchin (2014) highlights some of the difficulties associated with using and assessing concept maps and provides recommendations to ensure their effective use. Case Study: Social Tutor for Autism While social learning may appear in conflict with the development of a social tutoring program to be used individually, the virtual agent can play the role of a peer and activate these same learning gains, as is attempted in the ‘homework’ activities of the Social Tutor described previously. As part of the learning process and formative assessment, concept maps can be created collaboratively between peers or between the learner and educator. Several concept map types exist, and the type used must be considered carefully in relation to the desired outcome and the target content, as no single dominant method currently exists (Park and Calvo 2008; Watson et al. 2016). Spontaneous maps can be challenging to automatically assess, as students are free to use any terms and connections they wish, however the richness of assessment can be highly beneficial, with map hierarchy indicating knowledge depth and interconnectedness of ideas (Kinchin et al. 2000; Park and Calvo 2008). The simplest concept maps may be in the form of ‘fill in the blanks’, and if terms to fit the blanks are provided, the task of assessment is further simplified (Cline et al. 2010; Park and Calvo 2008). Concept map format can fit anywhere between these two extremes; however care must be taken to ensure the task is sufficiently complex that the outcome is representative of the students’ actual knowledge and not just ‘good guessing’ and yet assessable in a consistent and valid way. Depending on the map type, measures may include raw and weighted counts of connections, node and proposition matching, and measures of congruence and salience, i.e. proportion of valid student propositions over all criterion propositions

18

3 Designing Effective Teachers

and over all student propositions in the population, respectively (Park and Calvo 2008). Cline et al. (2010) developed an automated system for constructing and assessing concept maps known as the Concept Mapping Tool (CMT). The CMT is a webbased tool including GUI front ends for teachers to build criterion concept maps and for students to build their own maps, in the form of directed graphs, which are then compared to produce a grade. CMT uses a rule-based evaluation system to compare the nodes, direction of connections between nodes and other aspects of the map to determine a final grade. The system performs rapidly and thus students are given immediate feedback, which has been repeatedly demonstrated to be beneficial to the learning process (Cline et al. 2010). Students are presented with the central concept, concept nodes and distractor nodes based on the criterion map provided by the teacher and are required to use these to demonstrate their knowledge by providing connections between appropriate concept nodes. This is highly structured, as students cannot provide their own terms for concepts, however it is also flexible as no hint is given to the student regarding the connections between the concept nodes and distractor nodes must be dealt with correctly as well. Case Study: Social Tutor for Autism When used in conjunction with peer group instruction, conceptual mapping has been shown to lead to improvements in social skills (Laushey et al. 2009), believed to be due to it being a very visual medium and thus making clear otherwise abstract ideas. In light of this evidence, several concept map activities have been incorporated into the Thinking Head Whiteboard for assessment purposes.

3.4.4 Effective Feedback Feedback is an essential element of learning in any context. It has been shown that immediate feedback while a student is undertaking a task provides the most benefit and avoids situations where the student solidifies misconceptions rather than accurate understandings, presumably because the student is still engaged in thinking about the concepts and processes at hand (Bowman-Perrott et al. 2013; Crook and Sutherland 2017; Stuart 2004). However, determining how to provide feedback and what kind of feedback to give is of great importance. It was found that having a pedagogical agent interrupt students to provide hints provided no benefit, with experimental data indicating that students did not read the provided hints at all in these situations (Conati and Manske 2009). The content of feedback is likewise essential, as shown in the study by Hattie and Timperley (2007). It was found that simply providing praise, reward or punishment only had a small influence, while feedback suggesting how to

3.4 The Importance of Assessment

19

perform a task better or containing information about the task lead to very significant gains. Feedback must cater to the student’s immediate needs, with task-level feedback addressing misunderstandings about the task or the outcome, and process-related feedback assisting students to use their own error-detection strategies and to choose appropriate strategies to implement, and finally self-regulation feedback, helping students to monitor, determine and review their own practices (Hattie and Timperley 2007). Black and William (2009) emphasise the need for ongoing assessment, as it provides three key functions: establishing what students know now, ascertaining what they need to know, and determining what to do to reach these goals. If this is done regularly, the educational process is managed such that the chances of misunderstandings, repetition of already mastered content, and other difficulties are minimised. Accurately assessing student needs means accurately determining the cause of difficulties that students are encountering. This could be for a range of reasons, including misunderstandings of the language used, the purpose of the task, or the task itself, being misled by an unimportant element of the task, using ineffective strategies, or simply not providing a clear or sufficiently detailed response (Black and Wiliam 2009). In many of these situations it is possible that the student does in fact have the targeted skills or knowledge mastered, but simply misunderstood what was required of them. By implementing ongoing assessment and feedback these difficulties can be detected and rectified in a timely manner, ensuring students do not waste time or inadvertently consolidate inaccurate knowledge or skills.

3.5 Emotional Aspects of Learning Emotion can have a strong impact on learning, so to maximise educational outcomes it is important to understand this relationship. Students experience a wide range of emotions while they are learning, from confusion, frustration, dejection and boredom, to satisfaction, enthusiasm and excitement. Typically, individuals who are anxious, angry or depressed do not retain information effectively or perform well in learning tasks, so it is the role of the tutor to guide learners through these states and into affective states more conducive to learning, as expert human teachers naturally do (Kort et al. 2001; Storbeck et al. 2015). Emotions can be viewed has having an evolutionary function, where even slightly stressful situations and negative emotions can trigger a flight or fight reflex. This results in many physiological changes including an increase in heart rate and blood pressure, and adrenaline being released which causes the brain to switch into a reactive mode rather than a reflective, problem-solving mode (O’Regan 2003; Storbeck et al. 2015; Wolfe 2006). While memory is enhanced at this time, it is not typically an ideal situation in which to be learning new concepts or making new connections (Wolfe 2006). Further, research indicates that positive emotions during learning can reduce cognitive effort and increase working memory (Storbeck et al. 2015) and thus providing students with learning opportunities that are inherently pleasant can also

20

3 Designing Effective Teachers

result in strong retention. Gamification is one approach gaining much attention of late, with a recent review showing that in an educational context inclusion of gamelike aspects or embedding the learning within a game can, when done mindfully, lead to increased motivation, engagement and enjoyment (Hamari et al. 2014). It should be acknowledged that a small degree of frustration or uncertainty can be constructive and may even indicate that a learner is in their zone of proximal development (Vygotsky 1978). The zone of proximal development is defined in the seminal article of Vygotsky (1978) as “the distance between [a learner’s] actual developmental level as determined through independent problem solving and the level of potential development as determined through problem solving under adult guidance or in collaboration with more capable peers” and is considered the ‘sweet spot’ where the balance between what an individual knows and what they need to know is ideal for making a new connection. Some frustration is natural while learning a new skill or concept, for example when a student recognises that they are close to succeeding in a task and becomes motivated to persevere until they do. It is a fine line to tread, with too much frustration being counterproductive and disengaging, however human tutors intuitively step in at the right time to support the learner. Emulating this behaviour remains an active research area for intelligent tutoring software. Emotions can also be seen from a behaviourist viewpoint where the emotions themselves act as reward and punishment, and therefore influence the choices an individual makes. In this context, negative emotions like anxiety behave as punishment and cause the individual to avoid the situation that triggered the negative emotion (O’Regan 2003). The goal is therefore to minimise the occurrence of unnecessary negative emotions and increase the positive emotions. Acknowledging and celebrating learners’ achievements, using positive reinforcement, and providing inherently enjoyable tasks are important ways to maximise the positive emotions associated with learning, while detecting negative emotions and dealing with them constructively, for example noticing frustration and providing guidance, are essential in minimising the negative emotions.

3.6 Summary Learning from the actions of human tutors, it has been shown that immediate feedback and prompting is likely to be a key contributor to the observed improvement one-onone tutoring has over traditional classroom approaches (Bowman-Perrott et al. 2013; Chi et al. 2001), and that even an imperfect tutor can lead to educational benefits, which is great news when developing a virtual ECA-based tutor. Evidence suggests that game-based approaches are also helpful in terms of engagement (Meluso et al. 2012), and that carefully managing the affective aspects of the educational experience can also influence learning outcomes (Storbeck et al. 2015). Further, evidence has shown that ‘learning by teaching’s approaches can be even more effective than traditional tutoring (Fiorella and Mayer 2013; Ogan et al. 2012).

3.6 Summary

21

Research into human–human tutoring has highlighted many benefits for both the tutor and tutee. Research into human-ECA interaction has shown that these benefits hold true even in a virtual environment, making it a promising approach for effective teaching in ECA-based software.

Chapter 4

Designing for Specific Populations

When designing a tutoring system for any population, the needs and characteristics of the specific audience need to be carefully considered. In this chapter we discuss some of the challenges individuals with specific needs may experience and suggest strategies to support these users and lead to positive educational outcomes. The case study software presented here, the Thinking Head Whiteboard, was designed for individuals with autism and thus the specific examples provided here, while presented in a general manner, are largely reflective of this case study.

4.1 Universal Design A very broad set of well-established guidelines for designers in a range of disciplines are the Principles of Universal Design (Connell et al. 1997), with a more recent iteration of these being the notion of Inclusive Design (University of Cambridge 2017). Both of these philosophies emphasise the importance of designing products and services that are accessible to as many users as possible, regardless of any physical or other challenges they may have. The nature of a virtual tutor delivered via a standard desktop computer means that some of the product-focussed principles, such as requirements for sufficient size of and space around equipment and the possibility for presenting information in a tactile manner, are not as relevant in this context, however most are highly applicable. These include considering equitable and flexible use, which leads to providing multiple modes of interaction. In the case of autonomous tutoring software, this could encompass allowing the user to input responses via the keyboard, mouse, touch screen and speech depending on the task and user preference. These guidelines, along with the principle of perceptible information, suggest that not only should input be multimodal, but output should also be provided in a variety of ways, such as visually, textually and aurally. An example of this may be that when asking a question, the system provides the written text, an informative icon, and reads it aloud. The Principles of Universal Design also © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_4

23

24

4 Designing for Specific Populations

stresses the importance of being tolerant to errors (Connell et al. 1997). For example in a virtual tutoring system, preventing users from accidentally clicking an irrelevant button by disabling it when not needed or allowing them to undo their last action and provide a different response can help increase tolerance to errors and reduce their likelihood in the first place. Case Study: Social Tutor for Autism In the case of individuals with autism, providing multiple options for input and output and letting the learner choose which to utilise is important considering that some learners may have abnormally high or low sensory tolerance towards some options, and this notion is supported by outcomes from a recent small scale survey conducted by Fletcher-Watson et al. (2016). Helal, Mokhtari and Abdulrazak (2008) provide guidelines for developing virtual companions, most of which can be applied successfully to developing pedagogical agents. They cite the following as important requirements: adaptability towards the environment and user, availability of multiple options for completing tasks, provision of useful and accurate solutions, proactive offering of services to the user, being tolerant to faults and unexpected inputs and having the ability to adapt its goals and behaviours to suit the user’s needs. Many of these points overlap with those suggested by the Principles of Universal Design (Connell et al. 1997) and Inclusive Design (University of Cambridge 2017). Case Study: Social Tutor for Autism Adaptability is one of the recommendations Helal, Mokhtari and Abdulrazak (2008) provide for creating virtual companions. Often, virtual agents are considered adaptable if they learn from their user and alter their behaviour accordingly. Caution must be taken when applying this principle to individuals with autism who may display abnormal social and verbal behaviours. As the goal of the pedagogical agent in this case is to help minimise undesirable behaviours and maximise desirable ones, having the tutor adapt its behaviour to the user in this way is counterproductive. The more appropriate form of adaptability here is to gather data during interactions with the user and use this to determine their current needs, in turn allowing the system to present lesson content appropriate to the learner’s current level and to cater to their learning style and sensory needs. Scaffolding is an established educational technique with demonstrated usefulness in a wide range of learner groups, from children with special needs (Radford et al. 2015), to high school students (Mulder et al. 2016) and adults (Kyun et al. 2015). Tartaro and Cassell (2006) and Silver and Oakes (2001) both stress the importance of

4.1 Universal Design

25

scaffolding when developing software for individuals with autism, which involves providing the learner with very simple and straightforward learning experiences and tasks initially and, as the learner increases in competence, gradually adding complexity and distractions. Silver and Oakes (2001) note the importance of providing opportunities to repeat tasks in order to reinforce the concepts within them and stress the need to provide timely and accurate feedback so that learners understand where they went wrong, why and what to do next time. Children with autism have difficulty learning from their own mistakes without explicit support, so such feedback is particularly vital for this audience. Both research teams state that providing tasks that are inherently reinforcing and rewarding leads to the richest outcomes, and Tartaro and Cassell (2006) further this by stating that generalisation of behaviours to real situations and novel contexts must also be considered and supported as much as possible. Further, Tartaro and Cassell (2006) emphasise the importance of providing a safe environment for children to practice their emerging skills in. Finally, they highlight that all learners are individuals and thus educational experiences should be customisable to their personal needs and skills. More recently Fletcher-Watson et al. (2016) elicited recommendations directly from young children with autism themselves, their caregivers, and educators, prior to developing an iPad game. Their findings further support the inclusion of customisable features, minimising unnecessary images and background music to avoid fixation and distraction, inclusion of a reward token system, and having the system make no response when incorrect answers are given rather than a negative response. These recommendations are likely to be true for many audiences, not just those with autism. After evaluation of the designed game the researchers found that individuals had differing reward preferences, and that while very young children were happy to continue playing the game even when it was very repetitive, more able children lost interest unless continually challenged. Families reported positive perceptions of the software generally (Fletcher-Watson et al. 2016). In the case of virtual tutors, many of these recommendations can be easily implemented and much can be personalised and adapted to the child including, but not limited to, the appearance of the virtual tutor, input and output modes, and the content and format of the lessons provided.

4.2 Sensory Difficulties Sensory Processing Disorder (SPD) is commonly associated with autism, however it is not exclusive to it, and can also be seen in combination with ADHD, schizophrenia, and other conditions (Sanz-Cervera et al. 2015). Difficulties with sensory integration and tolerance can have a major impact on learners (Robertson and Simmons 2013), and taking these challenges into consideration when designing software can go a long way in delivering a positive educational experience for these individuals, and generally at no detriment to neurotypical learners.

26

4 Designing for Specific Populations

Case Study: Social Tutor for Autism Many individuals with autism report looking at faces difficult, and one cause of this is thought to be the sheer amount of information, visual and social, that is contained in the human face (Jones et al. 2003). An animated pedagogical agent can be advantageous here, as it can be given a very simple and cartoonish appearance initially, and as the learner increases in confidence, the complexity can be increased. This is an example of scaffolding, which in a study by Parsons and Mitchell (2002) was also shown to support generalisation of skills to real social situations. Sensory overload is of particular concern, as individuals with low sensory tolerance may require only minimal exposure to particular stimuli before registering a strong response. To minimise the risk of sensory overload and thus make the tutoring software accessible for a wider range of learners, it is recommended to omit unnecessary material and avoid developing software that is aurally or visually ‘noisy’ (Clark and Choi 2005; Fletcher-Watson et al. 2016). In practical terms, this means avoiding the use of animations, bright colours, or sound effects unless they add significant educational value. Doing so helps keep the interface simple, making it easier for the learner to understand what is required, while minimising possible distractions or fixation (Davis et al. 2005; Fletcher-Watson et al. 2016). Some individuals have low tactile tolerance which could make mouse and keyboard use challenging, while others may have low aural tolerance, making a speech-recognition and text-to-speech interface confronting. Given this, multiple input and output modes should be offered where possible so users can choose which best suits their needs, thus catering for this wide range of challenges and preferences.

4.3 Communication Difficulties Difficulties with communication, both literacy and processing related, are common in many developmental disorders including autism, but can also be displayed by children of varying backgrounds for a wide range of reasons. Thus, unless the focus skill of the virtual tutor is literacy dependent, it makes sense to support learners with low reading, writing and listening skills by providing a variety of alternatives. For example, to support individuals with low reading ability, a visual prompt such as an icon can be provided along with any verbal or written information or instructions given (Knight et al. 2015; Quill 1997; Shane et al. 2009). Icons should be simple and clear, and used wherever they add meaning, without being used excessively and contributing to the sensory overload previously discussed. Provision of multiple input and output modes is also important in the context of the communication difficulties, not just for sensory reasons.

4.3 Communication Difficulties

27

For example, expecting a learner with communication difficulties to write or speak full, grammatically correct sentences when their language skills are not the central focus of the lesson may serve to put learners off and draw attention away from the core skill concepts being taught. Instead, point and click interfaces and other simple interaction modes may be better suited to allowing students to express and explore their knowledge without exacerbating communication barriers. Following this same line of thought, complex or lengthy sentences should be avoided in favour of short, concise sentences. Additionally, self-paced lessons are ideal as they give the learner a sense of control and ownership of the learning process, lowering anxiety and helping with content retention. Case Study: Social Tutor for Autism Learners with autism often miss subtle cues and can become confused or distressed by ambiguity, so instructions should be presented in simple, clear steps and scaffolding used to move learners from simpler concepts to more complex ones as their skills improve (Brown et al. 2001; Parsons et al. 2000; Quirmbach et al. 2008; Silver and Oakes 2001).

4.4 Generalisation to Novel Contexts Difficulty generalising skills and knowledge to new situations is commonly associated with a diagnosis on the autism spectrum, however it can also be present with severe learning difficulties and other challenges (Cromby et al. 1996; McCleery 2015; Webster et al. 2015). In these cases, it is not unusual for learners to improve their skills at a given task in a given environment but fail to exhibit these same improvements in other situations. As with providing support for sensory and communication difficulties, providing support for generalisation can benefit all learners, not only those with a specific need. Drawing on the recommendations from the seminal article of Stokes and Baer (1977) and more recent review by McCleery (2015), a number of practical steps can be taken to encourage generalisation. First is to ensure that learning tasks are embedded in real-world experiences and situations, so that their real-world value can be understood by the learner. Another is to expose the learner to a wide range of situations and media within the tasks, the idea being that if they are able to generalise across these different situations and see the similarities and common cues between them, it will help with generalisation to contexts outside of educational environment. In the case of a virtual tutor, this may mean including videos, line drawings, animations, photos and other varieties of media rather than only exposing the learner to one media type. Additionally, with a virtual tutor, the ability to change their appearance and voice may be beneficial. This is similar to having a student learn

28

4 Designing for Specific Populations

and role-play with a variety of peers instead of just their favoured peer. This approach may improve the chances of generalisation as it avoids having the learner associate the task with the single tutor presenting it and helps them identify common elements across multiple appearances and situations. Additionally, presenting predictable tasks should not mean identical tasks, as learners with generalisation difficulties need to be gently encouraged to be flexible in their thinking. Instead, tasks should follow a predictable pattern, warn the learner before significant changes to the expected occur, but present some differences and alternatives each time they undergo the task.

4.5 Summary When designing virtual tutors, careful consideration should be given to the notion of inclusive design. For learners with specific needs, be it sensory, communication or otherwise, having these needs catered to mean the difference between a positive educational experience and useful learning tool, and frustration and exclusion. Generally, these needs can be met without detriment to other learners who do not share the same difficulties, and in fact this can often lead to better outcomes for these learners as well. Thus, putting the effort in to providing this support can be seen as a win–win situation.

Chapter 5

Existing Software Tutors

From early childhood through to adulthood, ECAs are already being used for a range of education and training applications. In early childhood the focus so far has been on literacy, however social skills are also a promising emergent area. In the school years ECA-based software has been developed to target core curriculum areas such as literacy and numeracy, with a focus on struggling students and special needs learners. Social skills have also been addressed for this age group. In adulthood, ECAs are used both in higher education in areas such as computer science and physics, and on-the-job training teaching new employees procedural skills such as the operation and maintenance of machinery. Here, some notable examples from each life stage are provided and areas for future development discussed.

5.1 Early Childhood While little research has been done into the use of ECAs in early childhood education, it appears to be an area with potential. Ryokai et al. (2003) developed an authorable virtual peer, Sam, designed to undertake collaborative storytelling with young children. Sam models complex linguistic features including quoted speech and spatial and temporal features, and after children interacted with Sam it was found that their own stories improved in these areas. Further, the children improved their critical listening skills. Following this success, it is anticipated that similarly positive outcomes could also be gained in other learning areas.

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_5

29

30

5 Existing Software Tutors

5.2 The School Years The majority of work concerning ECAs in educational settings has been focused on the school years, where large cohorts of learners are readily available to evaluate and contribute ideas towards the development of the ECAs in question. This makes it a very active research area, with many novel proposals and agents in the early stages of development and beyond. In this section we will focus only on ECAs that have been developed to a functional stage and evaluated for their effectiveness or are particularly noteworthy and novel. Virtual tutors are gaining traction in a multitude of areas, including showing promise for application in the complex area of social skills development for children on the autism spectrum. Embodied pedagogical agents can be categorised into two groups, the first where the agent only appears when it is required, either requested or unrequested, and the second, referred to as a peer learning agent or virtual peer, where the agent is always present either as a learning partner or opponent in a task (Sklar & Richards, 2010). Table 5.1 provides an overview of the tutoring applications discussed in detail here. While a diverse set of application areas are discussed here, they provide evidence of the efficacy of virtual tutors for teaching children a variety of skills in different contexts and have led to inspiration for many features of the Social Tutor software developed for the current project, ranging from approaches towards user interaction, underlying pedagogical frameworks, virtual agent presentation and responsiveness, and approaches towards personalisation and automated assessment.

5.2.1 Language and Reading Tutors A number of virtual tutors are available for improving reading and other language skills in children. Project LISTEN, a research project at Carnegie Mellon University, has resulted in the development, deployment and evaluation of an automated reading tutor. This reading tutor, which has been used extensively by several primary schools across America, uses the Sphinx-II speech recogniser to listen as children read aloud, and is able to detect errors and provide spoken and visual feedback immediately. Research has shown the software to be very effective, with children using the software improving their reading significantly faster than peers in a typical classroom setting (Mostow 2005). While it does not incorporate a visually embodied virtual agent, it does harness speech recognition technology to respond to users in a human-like way. While aimed at high school students rather than primary school students, the University of Memphis has also developed a tutor focussed on teaching strategies for reading comprehension. The reading tutor, Interactive Strategy Training for Active Reading and Thinking (iSTART), includes multiple pedagogical agents who interact with the student to teach a modified version of the Self-Explanation Reading Training (SERT) technique for reading comprehension. They encourage the student to use

5.2 The School Years

31

Table 5.1 Examples of existing tutoring software Application

Participant description Embodied agent(s)?

Autonomous?

Evaluation outcomes

Language and reading tutors Project LISTEN Primary school (Mostow 2005) students (neurotypical)

No

Yes

Significantly faster improvements in reading versus typical classroom

iSTART, iSTART-2 (Jacovina et al. 2016; McNamara et al. 2004)

Secondary school students (neurotypical)

Yes—multiple

Yes

Improved comprehension skills versus no training

Baldi and Timo (Bosseler & Massaro, 2003)

Primary school students (autism, hearing impaired)

Yes—single

Yes

Significantly increased vocabulary, maintenance and generalisation to other contexts

Sight Word Pedagogical Agent (Saadatzi et al. 2017)

Young adults (autism, intellectual disability)

Yes—single

Yes

Significantly improved reading of target sight words, maintenance and generalisation to other contexts

Mathematics and science tutors AutoTutor (Graesser et al. 2005)

Secondary school students (neurotypical)

Yes—single

Yes

Improvement of up to one letter grade in physics assessment

Wayang Outpost (Woolf et al. 2010)

Secondary school students (neurotypical—focus on low achievers)

Yes—multiple

Yes

Reduced anxiety and frustration, improvements in mathematics assessment

No

Yes

Significantly higher improvements in Mathematics assessment versus typical classroom

Cognitive Tutor Secondary school (Ritter et al. students 2007) (neurotypical—focus on low achievers)

(continued)

32

5 Existing Software Tutors

Table 5.1 (continued) Application

Participant description Embodied agent(s)?

Autonomous?

Evaluation outcomes

Betty’s Brain (Biswas et al. 2009; Blair et al. 2007)

Primary school students (neurotypical)

Yes—single

Yes

Students who taught the system and received prompts outperformed those without prompts and those building concept maps with system coaching. All conditions lead to learning gains

SimStudent (Ogan et al. 2012)

Middle school students (neurotypical)

Yes—single

Yes

Students who interacted with the virtual agent in a more natural way had higher learning gains

Mindstars Books (Hautala et al. 2018)

Primary school students (neurotypical)

Yes—single

Yes

Specifically designed for very young students (Grade 1). Students displayed learning gains regardless of existing reading and listening skills

ISLA (Mondragon et al. 2016)

Primary school students (autism)

Yes—single

Yes

Students supported by an affective tutor showed better emotion management and higher learning gains than those in a control group

Yes—single

No

Improved gaze and turn-taking behaviour, skills generalised to context with human peers, improved scores on Test of Early Language Development

Social skills tutors Sam (Tartaro and Cassell 2008)

Primary school students (autism)

(continued)

5.2 The School Years

33

Table 5.1 (continued) Application

Participant description Embodied agent(s)?

Autonomous?

Evaluation outcomes

ECHOES (Bernardini et al. 2014)

Primary school students (autism)

Yes—single

Yes

Children enjoyed the system and showed improved social behaviours with the virtual peer. No standardised measures and generalisation not evaluated

Thinking Head Whiteboard (Milne et al. 2010)

Primary school students (autism)

Yes—multiple

Yes

Children displayed improved theoretical knowledge of social skills and found the system easy to use and enjoyable

strategies such as paraphrasing, predicting and elaborating to develop their understanding of the text. In a controlled study it was found that students using iSTART did improve their comprehension skills (McNamara et al. 2004), however after extended software use it was found that student motivation decreased (Jackson et al. 2010). The research group has since developed iSTART-ME and iSTART-2 which incorporate more game-like elements to enhance long term motivation, and which have demonstrated effectiveness with middle school, high school and college aged students (Snow et al. 2016). This continues to be used as an active research platform for investigating effective design and teaching strategies for pedagogical agents. Baldi and his successor Timo are animated virtual tutors developed by Bosseler and Massaro (2003) who are designed to improve the vocabulary of children with hearing impairments. Baldi and Timo have anatomically correct facial muscles, allowing their faces to make highly realistic movements during speech. The software in which Baldi and Timo are embedded provides learners with opportunities to interact with the taught words in different ways, using the words in appropriate contexts, representing them as pictures, typing the words, and other activities. The goal of this is to reinforce the sound and meaning of the words and thus encourage a deeper understanding and higher retention of the learned content. It was found that by watching Baldi as he spoke, children improved their vocabulary significantly more than when they only listened. Following the success of Baldi with hearing impaired children, he was trialled with children with autism for the same purpose. The same gains in vocabulary were found, and a month later the children retained 85% of the learned words. Most significantly, these children with autism were able to use the words they learned in everyday situations, providing evidence that virtual

34

5 Existing Software Tutors

tutors can lead to generalisation and use of learned skills in novel contexts for this learner group. In more recent work Saadatzi et al. (2017) developed a desktop application for teaching sight words to young adults with autism. Three males aged 19–20 with mild to moderate intellectual disability in addition to their diagnosis of autism were involved in the study. The desktop application incorporated a full-bodied pedagogical agent in a virtual classroom and both text-to-speech and automatic speech recognition so that users could engage with the software via the more natural mode of voice rather than typing or tapping. Participants were taught four target words using the software and were trained until they completed three consecutive sessions with 100% success, then their performance without reinforcement was checked via an assessment phase. 8 weeks after the assessment phase it was found that two participants could remember the target words at 100% accuracy, while no data could be collected from the third. Further, it was found that these participants were able to generalise their use of the four target words to a novel environment in the form of their classroom, and a novel stimuli as written words on paper, and the novel change agent of their teacher. No control group was included in the study so comparisons cannot be made to other methods of instruction, however it is very encouraging that the learners were able to improve their performance and apply it to contexts outside of the original software environment.

5.2.2 Mathematics and Science Tutors Along with the iSTART reading tutors, the University of Memphis has developed AutoTutor and its open-source counterpart GnuTutor, which have been used to teach physics concepts to high school students. They incorporate a virtual tutor that asks students questions requiring an answer in sentence format. Natural language processing techniques, such as latent semantic analysis (LSA), are performed on students’ written input to determine if the student understands the content or whether more probing questions or hints are required to assess their understanding. The virtual tutor uses speech synthesis and animation to appear lifelike and interact with the student (Graesser et al., 2005). GnuTutor is an open-source version of AutoTutor that includes the majority of the functionality, however it is highly reliant on the student possessing good written language skills and is therefore unlikely to be appropriate for use with students who experience communication difficulties, such as children with autism. Developed by the University of Massachusetts Amherst to help high school students improve their mathematics skills for the Scholastic Aptitude Test (SAT), Wayang Outpost is an online Flash game that includes several virtual peer tutors (Woolf et al. 2010). Many mathematical problems are presented in the context of the game’s story and are presented using interactivity and animations. Wayang Outpost stores information about the student’s interaction with the system, including hints

5.2 The School Years

35

requested, answers chosen and time taken to answer, and builds this into an individual student model which is in turn used to guide how the virtual tutors react to the student in question. This includes the student’s spatial aptitude, which in turn affects whether visual hints or arithmetic hints are more likely to be given by the tutoring system. Even without the intelligence component activated, it was found that Wayang Outpost resulted in improved results for learners, while including the intelligence and decision-making component boosted these gains even further (Woolf et al. 2010). Wayang Outpost continues to be used as a research platform for investigating student interactions with intelligent tutoring systems, including investigating underlying factors relating to student affect and motivation (Rai et al. 2013) and provides much inspiration to the current project in terms of engagement strategies and student modelling. Another successful high school mathematics tutoring program is Carnegie Mellon University’s Cognitive Tutor (Ritter et al. 2007). In a study comparing a group of students learning by traditional classroom methods with a group of students using the Cognitive Tutor, it was found that those using the Cognitive Tutor comfortably outperformed their classmates in terms of grades and standard testing. It has also been found that using Cognitive Tutor, student attitudes towards learning mathematics improved and that disadvantaged populations also gained significant benefits. While Cognitive Tutor does not use an animated virtual tutor, it does embody a wide range of relevant technologies including monitoring student knowledge and their interactions with the system in order to adjust content accordingly, guiding students in the right direction, and ensuring students only progress when they have sufficiently mastered the prerequisite skills (Ritter et al. 2007). Cognitive Tutor is now commercially available and also continues to be used in research, for example Fancsali et al. (2016) analysed data from Cognitive Tutor to explore how different aspects of the learning environment can impact student outcomes when using intelligent tutoring systems in a classroom setting, finding that the human teacher also has an important role in ensuring their students engage mindfully and purposefully with such software if positive outcomes are to be achieved. A novel approach to pedagogical agents that is gaining interest is that of a teachable pedagogical agent, one that the student must teach concepts to as a means of learning the concepts themselves. This idea is motivated by the observation that many teachers find that they have a better understanding of a concept after they have taught it. In this scenario, the student takes more responsibility for their own learning, a valuable life skill, and tests their understanding by trying to pass on their knowledge to a virtual agent. Betty’s Brain is an example of such an agent (Blair et al. 2007). Using a concept map style interface, students teach Betty concepts by adding nodes and connections between the nodes. Betty can then answer questions using the concept map and can tell students when she detects missing information. Betty’s Brain has been incorporated into a number of appealing game-like fronts, including a quiz where students put their virtual agents against one another to see which has learnt the concepts best (Blair et al. 2007). The Betty’s Brain system was tested with fifth grade students on a task requiring them to develop concept maps about river ecosystems, and then eight weeks later used the same systems but applied to a new topic, the land-based

36

5 Existing Software Tutors

nitrogen cycle. Three versions of Betty’s Brain were tested, one in which students taught the system, one in which they taught the system and received prompts from Betty, and one in which they built a concept map for themselves but with coaching from the system. It was found that students in the first two conditions performed better than in the last, providing evidence that learning by teaching is a valuable technique (Biswas et al. 2009). Another example of this approach is SimStudent (Ogan et al. 2012) which aims to improve learner’s mathematical knowledge by having them help an ECA tutee named Stacy to complete various tasks. Students were asked to use a ‘think-aloud’ approach and narrate their experience as they taught Stacy so that the social aspects of their experience could be investigated. It was found that many students appeared to interact with Stacy as they would with a human peer, for example encouraging her or teasing her, and those who did had higher learning gains than those who did not interact with Stacy in such a social manner (Ogan et al. 2012). Further work in the ‘teachable agents’ domain has shown that students complete more tasks and are more motivated when these agents have their own intrinsic motivation and behave in a more friendly and human way (Borjigin et al. 2015), which mirrors the findings of related work with non-teachable pedagogical agents. Hautala et al. (2018) developed a virtual science tutor for first grade students and found that students not only enjoyed using the system, but they also exhibited significant learning gains, and that these gains occurred even for students with low pre-existing reading and listening skills. Hautala et al. (2018) compared a version of their software where the virtual tutor was voice-only and one where the tutor also had a visible presence, however no significant difference in learning gains, accuracy, or individual preference were found between the face-on and face-off versions of the software. Interest in affective tutoring systems is also growing, where the pedagogical agent detects and responds to users’ emotional state. One particularly relevant example of such a system is ISLA, developed by Mondragon et al. (2016) to provide emotional support to students with autism while teaching them mathematics skills. While still in the prototype stage, a small evaluation of ISLA was conducted with 12 children with autism, aged 6–12 years old, who were randomly allocated into an experimental group with affective support or a control group with no affective support. It was found that the affective support improved participants’ levels of encouragement and decreased frustration and anxiety. Mondragon et al. (2016) intend to conduct a longer term and larger scale evaluation given these promising preliminary results.

5.2.3 Social Skills Tutors The domain knowledge required for the topics discussed previously, such as mathematics and reading, is relatively clear cut and typically has right and wrong answers with set facts and rules that can be followed to reach these outcomes. In contrast, the domain of social interaction presents a bigger challenge. Different cultural backgrounds, locations, and situations can call for a different ‘answer’ in terms of the

5.2 The School Years

37

social actions required. Also, neurotypical individuals learn social skills almost intuitively through their interactions with parents, peers and others throughout their development, making it hard for educators to know exactly what to teach and how to teach it when a student requires explicit instruction in this area. Understandably, there are few functional pedagogical agents in existence that cater to social skills teaching. One such example is Sam, a life-size animated virtual peer, designed to be gender ambiguous so that both boys and girls will relate to it (Tartaro and Cassell 2006). All agents described thus far are considered to be autonomous, in that they require no outside input in order to respond to the learner and are embedded in a standalone program. In contrast, Sam is an authorable virtual peer and requires the researcher to observe the student and choose Sam’s actions from a set of pre-recorded speech segments and scripted gestures. Sam was designed specifically for children with autism and engages them in collaborative story telling. Through this Sam models positive social behaviours including turn-taking, gaze and questioning, and helps learners recognise when they are being given an opportunity to contribute to the dialogue (Tartaro and Cassell 2008). It was found that through interaction with Sam learners were able to significantly improve their Test of Early Language Development scores, and even displayed enhanced social behaviours such as improved gaze, which they then used with their peers. This evidence of generalisation is particularly encouraging, indicating that virtual agents can be beneficial in improving the social skills of children with autism. Another example of a virtual peer for social skills teaching is Andy of the ECHOES program (Bernardini et al. 2014). ECHOES provides an exploratory environment where Andy is presented as an autonomous play partner in a virtual world that the learner can interact with through a large touch screen and eye gaze tracking system. ECHOES is based on strong theoretical underpinnings and best practice principles, with activities based on encouraging cooperation, joint attention and initiation of social behaviours. ECHOES was evaluated via deployment to five school sites across the UK, where a total of 29 children with autism aged 4–14 years old interacted with the software over a six-week period. The 19 children who had the most exposure to ECHOES and completed all pre- and post-tests were used in the evaluation. While some positive trends were observed in this preliminary evaluation, no significant conclusions could be drawn. Anecdotally some very promising events occurred, for example one child who was initially thought to be non-communicative by his teachers and practitioners waved and said ‘Hi Andy!’ in a later session, while others began spontaneously greeting their teacher after they had been practicing with Andy, something they had not previously done. The authors reported that teachers and practitioners also expressed enthusiasm about the platform. ECHOES appears to have promise as a tool for practitioners and specialised classrooms, however due to the requirement for specialised equipment it may not be appropriate for home use or some classrooms yet. The Thinking Head Whiteboard is another example of an autonomous social skills tutor that incorporates ECAs (Milne et al. 2010), and is the focus of the case study presented in this book. This tutoring software utilises existing ECA technology, Head

38

5 Existing Software Tutors

X, which is capable of realistic facial expressions and dynamic speech (Luerssen et al. 2010). Three virtual humans are used in the software, one teacher to explain content and guide the learner and two child peers who model social interactions. Aimed at children with autism who are currently in mainstream schooling, the Thinking Head Whiteboard uses basic automated assessment to respond to student needs, is highly customisable, including the appearance and voices of the realistic virtual humans, and includes the ability for educators and parents to modify existing lesson content and add their own (Milne et al. 2013). A pilot study was conducted using a basic version of this software containing two modules, one focused on dealing with bullying and another focused on conversation skills. With both modules it was found that students displayed significantly better test scores after using the software for a short period of time and according to their survey results found the experience to be a positive one. In the extended study focussed on conversational skills, which is discussed in further detail here in the chapter “The Thinking Head Whiteboard”, participants again showed an improvement in knowledge of social skills and reported having an overall positive experience with the Social Tutor. Virtual tutors are gaining traction in a multitude of areas, with strong evidence of their efficacy in the more clear-cut domains of mathematics and reading, and resulting in some systems such as the Cognitive Tutor being not only well-established experimentally but also now commercially available. Virtual tutors are likewise showing much promise in the more complex domain of social skills development for children with autism, with Sam, Andy and the Thinking Head Whiteboard being prominent examples in this space.

5.3 Adulthood ECAs are increasingly being used to teach adults as well as children, and not just in educational institutions such as Universities, although there are certainly applications in this area. Improving professional skills in the area of medicine, teaching social conventions of other cultures to military personnel, and providing on-the-job training are promising areas for ECA use, as are learning a second language and improving general life skills such as health literacy.

5.3.1 Health and Medicine Bickmore et al. (2010) have investigated the application of ECAs to helping adults to improve their health literacy. Access and usability are major challenges in the area of health education, as those who would benefit most from additional learning often have the lowest ability to access the resources they need. Further, face-to-face discussions with a health provider are still accepted as the best method of conveying health information. An ECA can simulate this face-to-face interaction and is simple to

5.3 Adulthood

39

interact with regardless of user’s reading and writing ability. ECAs can also establish rapport with users and this is hoped to improve adherence rates to implemented health regimes, for example the ECA may display disappointment if the learner fails to eat as healthily as they promised to. As we know, ECAs provide a low-pressure learning environment, present information in a consistent manner and can be accessed by the learner as often and for as long as is necessary for them to fully understand the content being conveyed. Two trials were conducted at the Boston Medical Center to test these ideas, one teaching discharged patients about their after-hospital care plan and another encouraging older adults to walk for exercise more often. In both trials participants found the system easy to use and a positive attitude towards it, with some individuals even commenting that they preferred the ECA over doctors for this task because they could take as much time as they needed to understand the content, compared with doctors who are often pressed for time (Bickmore et al. 2010). In a follow up trial, it was found that participants in the ECA condition of the walking promotion study walked significantly more steps that those in the control group, supporting the notion that ECA use can effectively improve health behaviours (Bickmore et al. 2013). While still in the development phase, Kenny et al. (2007) are developing an ECA to allow studying psychotherapy clinicians to practice their interviewing and diagnostic skills on a patient displaying the characteristics of conduct disorder. The ECA developed for this purpose, Justin, is fully embodied and response to users using both speech and gestures. The system utilises speech recognition technology to allow users to interview Justin naturally, via speaking. A small initial pilot study indicated that users found the system easy to use and sufficiently simulated a real-life situation, however it also highlighted limitations due to the dialogue management system, for example not being able to deal with user speech that contains multiple questions and not recognising some key words, and in the amount of conversation history ‘remembered’ by the ECA (Kenny et al. 2007). While the system is still in the development phase and has some wrinkles to iron out, it was overall well received by the participants of the pilot study and is a promising area for further research. While most of the systems discussed so far display the ECA on a digital screen of some sort, a novel idea is to insert the ECA into the real world using augmented reality. The Virtual Anatomy Assistant does just this, displaying the virtual tutor Ritchie on a life-sized screen behind a real model skeleton (Wiendl et al. 2007). The Anatomy Assistant aims to teach people where organs are located inside of the body. To interact with the system users employ a pointing device to place virtual organs into the desired locations in the real-world skeleton. If they place the organ correctly Ritchie displays happy body language, otherwise he performs a disappointed gesture. An evaluation of the system was conducted during a trade show, where 71 individuals chose to participate in the evaluation and complete the accompanying questionnaire and system interaction. Overall people found the system easy to understand and interact with, although placing organs precisely was a little challenging, and the ECA Ritchie was perceived as entertaining but not as helpful as hoped (Wiendl et al. 2007). This may be due to the fact that Ritchie only responds in a ‘right or wrong’ fashion to the actions users take but does not provide any hints or scaffolding prior to

40

5 Existing Software Tutors

the exercise (Wiendl et al. 2007). This is something to be explored further in future research. Another research group looking at more immersive techniques are Ponder et al. (2003) who have developed a virtual reality simulation for teaching para-medical personnel how to respond in a health emergency. The simulation involves ECAs who are actors in the scenario, both in the roles of patient and responder. This is an area when real on-the-job training is very unforgiving—making a wrong decision could cause serious harm to the patient—thus a virtual method of practice provides a beneficial intermediate between theory and practice. Thus far the system has been evaluated for ease of use and found to be intuitive and straightforward (Ponder et al. 2003), however direct learning outcomes are harder to assess. Chiang et al. (2018) have developed a system for learning physical skills, such as sports, martial arts, or dance. The system incorporates a Microsoft Kinect and displays a virtual teacher on screen alongside a real-time video of the learner. The system analyses the users’ movements in real-time and provides highly specific immediate feedback and guidance. The key benefits of this system over traditional approaches are that it is low-cost, and that it can be used at any time of day or night without a coach needing to be present. The system is a prototype and has not been evaluated against traditional methods or for its applied effectiveness, however it appears a promising approach.

5.3.2 Job Training Another area where ECAs have potential is for conducting on the job training. The Soar Training Expert for Virtual Environments, known simply as Steve, is one early example of such a system (Rickel and Johnson 1999). Steve exists in a 3D replica of the work environment and is used to both demonstrate the procedural tasks involved in operating a piece of machinery and provide feedback for students as they take their turn at the task. To interact with Steve’s virtual world, the learner wears a head mounted display with a microphone. Steve is capable of demonstrating a range of naval operating procedures (Rickel and Johnson 1999). Other more recent examples include the virtual training environment (VTE) by Gallerati et al. (2017), which trains new oil rig employees on what to expect in a typical work environment along with a number of procedural tasks, and the VTE by Goulding et al. (2012) which teaches construction site safety, however unlike ‘Steve’ neither of these include an embodied virtual human. The benefits of approaching on the job training this way is that it provides a safe environment for beginners even when the target machinery is potentially dangerous, it does not require human mentors or colleagues to take time away from their own roles to teach the newcomer basic introductory information, and the learner can repeat tasks as often as required until they feel they have mastered the required procedures.

5.3 Adulthood

41

5.3.3 Social Skills Tactical Iraqi is an immersive 3D video game incorporating many culturally accurate ECAs. It is aimed at teaching military personnel the social conventions of Iraq and to assist them to develop their Arabic language skills (Losh 2005). In a pilot study including 20 enlisted marines, 78% felt that they had acquired a functional ability in Arabic in the target areas after 50 h of training with the system and most also gave it a high subjective rating overall (Johnson 2007). Tactical Iraqi was originally developed at the University of Southern California with funding support from the U.S. military. Since its inception, the project has expanded to include a range of other ‘Tactical Language’ games and simulations including French, Chinese, Danish and Pashto and is now applied not only in military settings but is also used for improving the skills of business professionals, new immigrants and school students (Alelo 2018).

5.3.4 Higher Education One ECA based tutoring system geared towards tertiary students is the web-based Virtual Tutee System (VTS) (Park and Kim 2011). The goal of VTS is to improve the academic reading skills and habits of college level students. Park and Kim take a novel approach here, inspired by the Betty’s Brain system previously discussed (Blair et al. 2007), in that the goal is to have the learner teach the ECA and, in doing so, improve their own skills. There is much evidence to suggest that during peer tutoring, both the tutor and the tutee improve their proficiency in the target skills and those doing the tutoring show higher academic engagement following their teaching role, typically focussing on conceptual understanding of concepts rather than simply rote learning and displaying more sophisticated learning strategies themselves (Park and Kim 2011). A common issue with college level students is that they often skim read their set texts rather than engaging with them at a deeper level. It is hoped that by using VTS students will develop improved reading habits and engagement with their texts. The VTS is geared towards pre-service education students, who will teach their virtual tutee about the content of their course texts. Empirical research still needs to be done to validate the systems’ efficacy. Oscar is an ECA with a cartoon appearance used for teaching the database query language SQL to undergraduate students (Crockett et al. 2011, 2017). Oscar predicts the user’s learning style throughout the educational conversation and dynamically adapts his tutoring style to suit the learner. Oscar builds his student model by applying a set of logic rules to the actions a learner takes in response to teaching cues. For example, a user does not know an answer, is shown a diagram and then displays understanding. Following this, Oscar would increase the ‘visual learner’ weighting in their learner profile and consequently would use more visual techniques, i.e. more diagrams, in future interacts with the user. Further, Oscar is what is known as a goal-orientated agent, and so is not simply there to answer the learner’s questions

42

5 Existing Software Tutors

when they ask, but rather guides the learner towards a particular learning goal, again by applying a set of logical rules. Oscar aims to provide hints and scaffolding much like a human tutor would, rather than just telling the user whether they are right or wrong or simply presenting them with the correct answer immediately. The research goal with Oscar was to implement a fuzzy classification tree to use for predicting student learning styles, which it did successfully. Further research into the original system’s effectiveness discovered that students found the system helpful and achieved an average learning gain of 13% (Latham et al. 2012). An improved learning style prediction mechanism has since been developed (Crockett et al. 2017). Ramírez et al. (2017) developed a virtual tutor for teaching biotechnology skills. The evaluation was run over two years and took place in the subject ‘Biochemistry and Biotechnology’ which was delivered to first year university students. Students controlled their avatar in a virtual laboratory, and a virtual teacher guided their learning. Their task was to uncover the function of a particular gene. The cost of the equipment and materials required to replicate the same activity for a large cohort of undergraduate students in the real world would be prohibitively expensive, and thus the virtual world offers a unique learning opportunity. The students benefitted educationally and enjoyed using the system.

5.3.5 Second Language Learning Demenko et al. (2010) have developed a virtual tutoring system incorporating ECAs, AzAR, which is focused on assisting people to independently learn a second language. The role of ECAs in this context are as model speakers for the target language, providing a visualisation of the movements occurring in the face, lips and other articulators while they speak. For accuracy, the spoken language itself consists of recordings of natural voices rather than dynamic text-to-speech synthesis. In subjective evaluations, a clear majority of respondents felt that AzAR was a good tool for learning pronunciation, however more objective measures will be required to validate its effectiveness for this purpose. Bergmann and Macedonia (2013) investigated the use of ECAs for teaching a foreign language to adults, with a particular focus on the impact that iconic gestures have on memory. An iconic gesture is one whose physical form corresponds with object features of the word being spoken, for example tracing out the shape of a staircase while speaking the word ‘staircase’ helps to cement the sound of the word with the object it labels by combining aural and visual information. While there is much evidence supporting the use of iconic gestures, they have not been widely investigated in the context of foreign language learning or when delivered by an ECA. Bergmann and Macedonia did just this, and found strong evidence that gestures delivered by both a human and an ECA improved both short- and long-term recall. In fact, in most cases ECA delivered gestures outperformed human delivered gestures and led to bigger learning gains, possibly due to the clear manner the ECAs used to deliver their gestures (Bergmann and Macedonia 2013).

5.3 Adulthood

43

ECAs are a promising avenue for second language learning as they convey information in both aural and visual modes, through both speech and gesture. Motivated by this, Wik and Hjalmarsson (2009) developed two systems with different goals, designs and behaviours, however both incorporate ECAs and both are focused on teaching Swedish to foreign students. The first system, Ville, is a virtual teacher who provides feedback and guidance on the learner’s language use and pronunciation. The second system, DEAL, takes a role-play approach to conversation training, but does not provide direct feedback. Instead, DEAL behaves like a human conversation partner, indicating misunderstandings by asking clarifying questions, body language and, in worst case scenarios, through a breakdown of communication. The DEAL and Ville systems are used in conjunction—when a learner successfully converses with DEAL, they may move onto the next learning stage in Ville. The ECAs used in these systems display lip movements that are synchronised with their speech and use non-verbal gestures including head, eye and eyebrow movements to indicate encouragement, turn-taking events and emotional state. Qualitatively, students reported that the Ville system was helpful particularly in improving their pronunciation (Wik 2011; Wik and Hjalmarsson 2009). Second Life has also provided opportunities for developing virtual agent-based learning experiences. Henderson et al. (2018) utilised it for teaching Mandarin Chinese and presented students with two tasks centred around purchasing food. While there were some minor issues, such as students following other students’ avatars instead of working out locations in the virtual world for themselves, overall learners engaged in the appropriate cognitive processes, improved their language skills, and reported having a positive experience.

5.4 Summary As illustrated here, ECAs are being developed for educational purposes in a range of contexts and aimed at people in all walks of life. However, only a handful of the examples discussed have reached a point where they are publicly available. Use of ECAs in education is still a relatively new area of research and development, and we are continually learning more about the underlying concepts such as human–computer interaction and perception, effective teaching methods in this context and the core technologies that go into developing such systems, from knowledge assessment and dynamic lesson delivery, to speech recognition, natural language understanding and 3D modelling and animation in this context. It is a diverse and dynamic research area, with much still to be done.

Chapter 6

Creating Engaging Embodied Conversational Agents

Many factors go into creating an ECA that is engaging, appropriate for the intended purpose and educationally beneficial. The persona of the ECA is a major consideration and covers not only the appearance and voice of the ECA but also their responses and mannerisms. Another factor is perceived availability versus interference of the ECA—should it be a constant presence or just appear when needed? In which case, how do we know whether it is appearing often enough or too often, like the infamous Microsoft Clippit (Picard 2004)? Clearly, a range of issues must be considered, and what works for one learner does not necessarily work for everyone, however existing research in psychology and human–computer interaction can provide us with some guidelines.

6.1 Appearance When designing an appropriate appearance for an ECA it is important to consider both the function it is performing and the audience it is aimed at. Where the ECA is providing information and guidance, a cartoon non-human appearance may be just as effective as that of a lifelike human, however in situations where the ECA is modelling specific behaviours, such as in a social skills focused software, realistic human appearance becomes more important. Existing research suggests that even in non-social educational domains a more realistic agent can lead to better learning outcomes than a cartoon agent (Baylor and Kim 2004). When aiming for this, care must be taken to avoid the ECA falling into the Uncanny Valley. The Uncanny Valley is a phenomenon hypothesised by Mori (1970) who suggests that as human likeness and perceived familiarity increase a nearly human-like point is reached, however due to the perceived imperfections we feel uneasiness and even fear instead of comfort in the familiar. Mori adds that movement magnifies this phenomenon, for example consider a healthy human compared with a corpse or, even more frighteningly, a zombie. Clearly for educational purposes we want an ECA that evokes positive © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_6

45

46

6 Creating Engaging Embodied Conversational Agents

emotions, so it is typically safest to opt for an ECA that is sufficiently realistic to convey the intended information, without attempting to be photorealistic. In terms of the visual identity of the ECA there are many factors to consider, including age, gender, perceived socio-economic status and ethnicity. Again, choices must be made keeping the intended role of the ECA in mind. For an ECA performing as a teacher, an adult presenting a knowledgeable and gently authoritative persona may be most appropriate, whereas for an ECA performing as a peer, a child of the same age as the target group is likely to be better received. In terms of ethnicity, Nass et al. (2001) found strong evidence to support the notion that people perceive others of the same cultural identity as themselves as not only more attractive, but more reliable, trustworthy and knowledgeable, and that this holds true for ECAs as well. Further, users conformed more to the decisions of ECAs with matching ethnicity to their own. This presents clear implications in the design of ECAs for use in educational contexts, as an ECA in a position of authority, such as a teacher, must be seen to have many of these characteristics. Baylor and Kim (2004) also investigated the role of ethnicity in ECAs and found that University students performed better when the ECA taking the role of an ‘expert’ was personified as black rather than white. It was found that students focused on the relevant information and concentrated better (Baylor and Kim 2004). It is suggested that this may be due at least in part to a black ECA being more novel than a white one, with Caucasian ECAs being more pervasive at the time of writing. This suggestion is further supported by additional work by Baylor and Kim (2004) where a similar effect was found in an engineering scenario where a non-traditional female ECA was used rather than the traditional, stereotypical male persona. Moreno and Flowerday (2006) also investigated the impact that ECA appearance can have on learning outcomes, noting that people have repeatedly shown a preference for peers and tutors with similar ethnicity to themselves in a human–human context. When study participants were given a choice of ECA to work with, it was found that non-Caucasian participants would typically choose an agent of the same perceived ethnicity as themselves, although this did not hold true for Caucasian participants. For participants who were given a choice of ECA there was a significant correlation between ethnic similarity of the ECA to themselves and both their learning retention and transfer of problem-solving skills, however for students who were not given a choice of ECA there was no correlation between ethnic similarity and educational outcome measures. Further, it was found that the better retention and transfer outcomes were achieved by students who chose an ECA of a different ethnicity to their own. The authors suggest that this may be due to the fact that the students are focussing their attention on how the ECA represents them, rather than the information and task at hand, and that in this case the ECA becomes a distraction (Moreno and Flowerday 2006). This may be further compounded if the ECA in question is only visually a representation of their ethnicity, without the verbal and non-verbal behaviours that are part and parcel of that particular cultural identity. Evidence shows that users prefer interacting with ECAs who display a verbal style and non-verbal behaviours that are consistent with each other (Nass et al. 2001; Zibrek et al. 2018), and it follows that they should also be consistent with the visual

6.1 Appearance

47

identity of the ECA. However, if this cultural representation could be achieved more accurately it is possible that the results would be reversed, as with human–human interactions.

6.2 Body Language and Speech It is not only our appearance, but also our body language and manner of speaking which express our personalities, and likewise the persona of an ECA. Nass et al. (2001) conducted an experiment where participants were asked to interact with an ECA that displayed either introverted or extroverted style speech, coupled with either introverted or extroverted style body language. They found that users consistently ranked ECAs with matching non-verbal and verbal behaviours as more likeable, helpful and fun to interact with than those with inconsistent behaviours, and even more so when the perceived introversion or extroversion of the ECA matched their own (Nass et al. 2001). The embodiment used in this study was simply a stick figure displaying either an introverted or extroverted pose, and yet users still made judgements about the perceived personality based on this very simple body language. More recent research from Zibrek et al. (2018) not only supports the work of Nass et al. (2001) but expands it, this time utilising a set of five virtual humans ranging in rendering style from basic to realistic, including some deliberately designed to be eerie and unsettling. It was found that users preferred virtual humans who displayed behaviours consistent with their appearance, even where the virtual humans were deliberately designed to appear ‘creepy’. Since people typically display an instinctive distrust of other humans who display mismatched verbal and non-verbal cue (Ekman and Oster 1979) these outcomes are not unexpected, however they further support the idea that people perceive ECAs as social peers and therefore use very similar strategies to make judgements about them, and apply the same rules and expectations as they do with other humans. To further investigate the role of consistent verbal and non-verbal behaviours and ethnicity in human-ECA interactions, Iacobelli and Cassell (2007) developed two ECAs with identical ambiguous appearances, but differing verbal and non-verbal behaviours representative of different ethnicities. One ECA speaks using African American Vernacular English (AAVE) and displays non-verbal behaviours determined from study of African American children, and the other speaks Standard American English (SAE) and uses non-verbal cues determined from study of Caucasian children. 17 AAVE native speakers aged 9–10 years old participated 3.in the study. 83% of children in the SAE condition identified the ECA as Caucasian and 56% in the AAVE condition identified the ECA as African American, however due at least in part to the small sample size the results did not reach significance. This research suggests that ECA ethnic identity can be modelled using speech and behaviour and not just appearance, however further research is needed to better understand the nonverbal and linguistic cues of AAVE speakers in order to strengthen the identity of the AAVE ECA used here. Several researchers have addressed the need for culturally

48

6 Creating Engaging Embodied Conversational Agents

appropriate and diverse ECAs, with many focusing on the more superficial aspects of appearance and voice. As discussed, this on its own is insufficient as people prefer interacting with ECAs that display consistent behaviours. Research into verbal and non-verbal cultural cues moves us closer to developing ethnically authentic ECA personas.

6.3 Presence and Interaction ECAs can be categorised into two groups based on their presence, the first where the agent only appears when it is required, either at the user’s request or in response to some automated cue, and the second where the agent is always present, perhaps as a teacher, peer, or even opponent in a task (Sklar and Richards 2006). In designing educational software, the ECA presence must be a conscious choice, with the benefits and pitfalls of all options weighed up for the given context. Practical considerations such as limited screen real estate, particularly on smart phones or tablet PCs, may lead to an ‘on-demand’ or ‘on-cue’ model being most appropriate. Further, evidence suggests that the most usable interfaces are those that can be modified by the user, with personalisation of the interface increasing user satisfaction (Ventura et al. 2005). One pitfall here is the question of motivation - how do we know the user will chose to access the ECA, and thus gain the benefits of this software feature? For ‘on-demand’ ECAs to be effective we must teach the user not only how to access them but why they would want to, and in following this, must construct the ECA to be of genuine benefit to the user rather than just a novelty. In choosing a style of presence, another aspect that must be considered is the role of the ECA itself. Is the ECA there simply to provide facts and feedback, or is the goal for the user to create an emotional connection with it? Evidence suggests that users who anthropomorphise the ECA and have that emotional tie will display higher engagement and enjoyment, and users who have a positive experience are more likely to return and use the software again without prompting (Wik & Hjalmarsson 2009). Users are more likely to build up a familiarity with an ECA when they are exposed to it more, which leans in favour of an ‘always present’ ECA. For contexts aiming at long term use, this is particularly important. The role that the ECA takes on can also have an impact on independent learning skills. Baylor and Kim (2004) compared the educational impact on students when using ECAs in the roles of ‘expert’, ‘motivator’ and ‘mentor’. It was found that students who worked with the ‘mentor’ and ‘motivator’ ECAs rated their self-regulation and self-efficacy skills significantly higher than those who worked with the ‘expert’ agent. It is thought that this is due to the fact that with these ECAs the learner had to take more control of their own learning, with the ECAs prompting and encouraging the user rather than simply providing answers like the ‘expert’ ECA did. Following from this, the ‘expert’ role is the one most often used in an ‘on-demand’ style interaction where long term relationships between the ECA and user are less important, whereas ‘motivator’ and ‘mentor’ style ECAs are more commonly associated with ‘always present’ ECAs.

6.3 Presence and Interaction

49

Case Study: Social Tutor for Autism On-demand or constant presence is an important question, the answer of which is strongly dependent on the target user group. In the case of the Social Tutor, it has been shown that individuals with autism often use quite different visual processing strategies to those without autism, and can become overloaded by visual stimuli in terms of both detail and movement. In this case, the choice of interaction style is likely to depend on what the rest of the interface looks like. If the interface is already visually complex, having the ECA appear only when needed may be most appropriate, whereas with a simple interface it may be best to have the ECA constantly present rather than having it cause distraction as it appears and disappears. Another essential consideration when deciding between ‘always present’ or ‘ondemand’ style ECA interaction is the issue of distraction. What is more distracting— an avatar constantly popping in and out, or a highly detailed, constantly animated face? This is an oversimplification; however, it is an important question to ask and the answer will depend largely on the target user group, in many cases even the individual, and again the goal of the ECA. If the goal of the ECA is to draw attention to salient points on the screen, then having an animated character appear in the location that you want to draw user attention to would be an effective strategy as the eye is drawn to movement and change. However, if the goal of the ECA is to unobtrusively provide support for the user, this may not be the optimal strategy. As mentioned earlier, often interfaces that can be personalised by the user lead to higher satisfaction ratings, so consider putting the choice of ‘on-demand’ versus ‘always present’ into the user’s hands. Related to the role of the ECA is the depth of knowledge that the ECA possesses, or appears to possess, and the way it communicates with the learner. In an educational context, it is very important not to give inaccurate or misleading information to the user and equally important not to incorrectly tell the user that they are right or wrong. Wherever possible we need to give learners detailed feedback, however due to the limitations inherent in existing technology, particularly natural language processing, we sometimes cannot definitively tell whether the answer the user has provided is completely accurate. In these situations, careful wording of ECA dialogue and a demonstration or explanation of the expected response can ensure the learner is not misled. Even when confirming that a user’s answer is correct or not, we should carefully consider the dialogue used (Shneiderman and Plaisant 2005). Existing research shows that users prefer computers to praise and even flatter them rather than simply stating ‘correct’ and moving on and are more willing to continue with tasks when given this style of feedback. However, when users are given negative feedback they prefer a simple, impersonal response such as ‘incorrect, try again’ otherwise it can be seen as patronising, and would damage the emotional connection between ECA and user (Shneiderman and Plaisant 2005). Evidence shows that learning is enhanced when users view their interactions with ECAs in social terms involving

50

6 Creating Engaging Embodied Conversational Agents

reciprocal communication (Moreno et al. 2001). Clearly, the careful management of the human-ECA relationship can have significant impacts on educational outcomes.

6.4 User Input and Accessibility When creating educational software not only must the ECA itself be carefully designed, but also the system surrounding it. The input methods available to the learner and the nature of the expected input, be it open-ended or constrained, all impact not only the usability of the system from the user perspective, but also the assessment methods that can then be utilised and systems that must be put in place to ensure the user is sufficiently supported and provided with accurate feedback. For example, if students are able to type open-ended responses, how do we deal with situations where the system incorrectly understands or only partially interprets that input? The input method is often determined by the nature of the content being taught and the target user group, so the ability to customise systems and adapt to individual students can be highly beneficial. ECA-based learning systems already come in a variety of different formats, with the continuing uptake of mobile devices expected to play a large role in shaping the future of technology driven education. Following on from the notion that social contexts lead to more effective learning (Krämer and Bente 2010), and the prediction that therefore socially responsive agents will be more effective at teaching social skills, an investigation into just how to create such an agent is necessary. While there are a range of technologies which may potentially be of benefit when integrated into a pedagogical agent for teaching social skills to children with autism, to ensure the software is accessible to as many families as possible the Social Tutor must be able to be deployed on the technology already present in homes and schools. For this reason, gesture and speech recognition have some potential as they only require a web camera or microphone, both of which are commonly built into domestic laptops and desktops. These additional technologies could also be used to mitigate some of the other difficulties associated with autism, such as poor fine motor skills and sensory difficulties, as can careful design of the user interface.

6.4.1 Speech Recognition Following the principles of universal design, developers aim to make software that is accessible to as wide an audience as possible. For people experiencing difficulties that prevent them from easily using a keyboard or mouse, speech recognition technology can be beneficial. Speech recognition can also be used to provide novel educational activities, such as practicing reading aloud or directly interacting with a virtual human to practice conversational turn taking in a more realistic manner.

6.4 User Input and Accessibility

51

While speech recognition potentially has a lot to offer a virtual tutoring system, it also brings with it many challenges in terms of implementation. Case Study: Social Tutor for Autism Individuals on the autism spectrum often display poor motor skills, both fine and gross, making it difficult to perform clear and coordinated gestures (Noterdaeme et al. 2010). This can make using a mouse or keyboard quite challenging, since fine motor control is essential. Along with touch screen technology, speech recognition is also a possibility to assist with combating this. Speech recognition systems rely heavily on making predictions about what is likely to be said next. This means that they work best in situations where the dialogue is restricted to a specific domain, rather than being completely open-ended. Thus, the success of the speech recognition system can be improved by structuring activities in such a way that it is possible to present users with a range of answers to select from or where expected responses are very restricted, such as the approach taken in teaching sight words to children with autism by Saadatzi et al. (2017). This approach is only likely to be appropriate in some circumstances and it reduces the realism of tasks such as practicing conversations, which limits its usefulness and applicability to many real-world situations. Another challenge when using speech recognition systems is that they are trained on large datasets from typical speakers of the target language. This means that for speakers with atypical speech properties, be it accent, intonation or phrasing, the system can encounter difficulties and become less accurate. It is not only these types of differences between speakers that cause issues, but also between the speech of children and adults, for example Jokisch et al. (2009) demonstrated differences in speech characteristics across age groups ranging from very young to the elderly. However, as technology improves and our ability to quickly process large amounts of data grows, so does the quality of our speech recognition systems (Huang et al. 2014). One way to overcome the difficulties posed by diverse speakers is to use audiovisual speech recognition instead of relying solely on audio information. Navarathna et al. (2010) developed a speaker independent automatic speech recognition system for use with GPS based navigators inside cars and found that in a noisy environment, the audio-visual approach provided higher accuracy and robustness compared to an audio-only approach, supporting the use of audio-visual data over audio-only data for speech recognition. The downside is that both a microphone and camera are then required to implement this technology. Case Study: Social Tutor for Autism

52

6 Creating Engaging Embodied Conversational Agents

Children with autism experience communication difficulties and may not follow social conventions, therefore saying things seemingly out of context. They can also exhibit atypical speech properties due to their communication difficulties. A study by Hoque (2008) which collected and analysed 100 minutes of one-on-one conversational speech between individuals with autism, down syndrome, and their neurotypical teachers, found that there were notable differences between the groups. Examples include distinct differences in pitch, intensity and energy of utterances, with individuals with autism displaying less use of appropriate intonation. These factors provide an additional challenge for speech recognition systems that rely on matching sound input to a limited selection of expected patterns. Traditionally, the accuracy of a speech recognition system is based on the percentage of words that it correctly interprets. However, Hirschberg et al. (2004) suggest that for a tutoring application, this is inappropriate. Instead, the conceptual understanding of the speech recognition system is important, for example interpreting the utterance ‘show me the trains’ as ‘show me trains’ should not be considered an error. Hirschberg et al. (2004) suggest that inclusion of pragmatic, semantic, lexical and conceptual features may be used to provide more relevant accuracy measures. Rotaru and Litman (2006) investigated the impact of emotional speech on word recognition rates and found that emotional speech often leads to a high error rate, which in turn increases frustration and emotional content of the user’s speech, causing a cycle. Rotaru and Litman (2006) suggest that being able to detect emotional speech and correct for it can help reduce speech recognition errors. Additionally, the authors suggest that in a tutoring application, a low threshold for speech recognition errors is likely to result in better outcomes for learners than a high threshold, as long as the tutoring systems implements a ‘guiding’ rather than ‘punishing’ approach to incorrect and ambiguous answers. In any tutoring application, special care must be taken to balance the risk of giving incorrect feedback versus the risk of unduly frustrating learners and reducing their engagement and motivation. Clearly developing a speech recognition system for a tutoring application can bring with it many additional challenges, on top of the well acknowledged existing challenges of dealing with multiple speakers, noise conditions, slang and filler words, and other scenarios likely to be encountered in a real-world application of the technology (Zeng et al. 2009).

6.4.2 Gesture Recognition Gesture recognition is another alternative input mode that could support some users and provide unique interactive learning experiences in appropriate contexts, for example when learning a physical skill it may be possible to detect if the actions are

6.4 User Input and Accessibility

53

being performed correctly. However, the application and the environment in which the learner will be located does need to be considered. If the computer being used is located in a communal area or classroom, the learner may not feel comfortable interacting with the computer in a way that draws attention to themselves. Performing the gestures for recognition may also be distracting to other students working in the same area. However, if the skills being taught in the tutoring software are gesture-rich, then providing opportunities for learners to practice these skills and gain feedback from the system would be highly beneficial in assisting the learner to apply their skills to real-world situations successfully.

6.4.3 User Interface Design One simple way to cater for those with poor motor skills without drawing undue attention or distracting others working in the same area is simply through thoughtful user interface design. Avoiding interaction that requires typing or otherwise excessive keyboard use, and ensuring on-screen buttons are large enough to be forgiving when tapped, clicked or dragged, can go a long way to making the software accessible. Using this approach means the software can easily be paired with a touch screen or track pad if the user prefers but remains comfortably usable when using a typical desktop computer with a standard mouse. While this alone does not harness the additional educational benefits of speech and gesture recognition, it does assist in making the software more widely accessible, in line with the principles of universal design (Connell et al. 1997; University of Cambridge 2017). Case Study: Social Tutor for Autism Individuals with autism can find gestures and body language confusing as it is, and can struggle with both fine and gross motor skills (Noterdaeme et al. 2010). Thus, introducing a new set of gestures needed to interact with a tutoring system is likely to be an unnecessary challenge for learners. However, recent work has shown that gesture-based games such as those using a Kinect have the potential to support social skills development by providing opportunities for cooperative competition and emotional experiences (Ge and Fan 2017), so it is important to consider the tutoring system’s goals and audience in tandem. When deciding which input methods to make available to users many factors must be considered. First, the input methods chosen need to align with the content being taught. In a reading tutor such as Project LISTEN’s iSTART (Snow et al. 2016), it makes sense to use speech recognition as the primary input method as the goal in both cases is to improve learners’ ability to read aloud and this method allows their performance to be directly assessed. For a mathematics tutor, selecting the correct answer from a list of options or typing in a number may be more appropriate. The

54

6 Creating Engaging Embodied Conversational Agents

input method must also align with the needs of the target user group. For example, young children and learners with poor literacy would severely struggle to type a lengthy answer to a question, however being able to select from a list of visual options or speak their answer to the computer would overcome this difficulty and is equally accessible to those with strong literacy skills. The input method chosen will also have a direct impact on the complexity of the system being implemented, and the difficulty of ensuring that students are always given accurate and useful feedback. While speech recognition has many benefits in terms of accessibility and mirroring human–human interaction more closely than most other input methods, and requiring students to type lengthy explanations to problems allows them to explain their reasoning and provide evidence of deeper understanding, both of these approaches can be very difficult to implement in an open-ended sense as possible combinations of words are essentially infinite, particularly accounting for use of slang, partial sentences, fillers and misspellings. Natural language processing is an active area of research in both of these contexts. In saying that, educational systems that use these technologies attempt to combat these difficulties by limiting the domain of expected responses. In the reading tutors discussed, users are provided with text to read, thus limiting the scope of possible inputs. If an unexpected utterance is detected, it is typically safe to assume that it was read incorrectly, and the learner can be provided with feedback. These systems often include mechanisms to deal with cases where the interpreter is unsure what the learner meant, encouraging them to try again or simply providing the correct response to ensure that the learner is not given incorrect feedback. In contrast, having the learner select the correct response from a list or type in a single number or word is very simple to implement with very little scope for the learner to input a response that the system is unable to deal with. The chosen input method will also have a direct impact on the methods that can be used to assess learner understanding. Constrained responses are easiest to implement from a technical standpoint and are also simplest to assess, as in many cases they can be compared to the desired response and marked as right or wrong, with an overall count or percentage indicating student mastery of the content. However, these methods do not allow students to explain their reasoning nor do they provide enough data for the system to determine the causes behind misunderstandings. It is argued that approaches that allow students to explain their working, such as typing a detailed paragraph-style response, lead to deeper understanding of the content and longer-term retention, and provide a better overall picture of student understanding (Shute and Towle 2003). In these situations a common approach to assessment is to apply latent semantic analysis techniques to judge the similarity of the learner’s input to a provided ‘ideal’ response and thus make an overall judgement of the learner’s level of competency (Jackson et al. 2010). A comparison of the benefits and pitfalls of constrained and open-ended style input methods can be seen in Table 6.1. It is inappropriate to expect students with low literacy skills to type paragraph lengthy responses, just as providing only multiple-choice style questions limits student expression and assessment. Another option that strikes more of a balance between constrained and open-ended input types is the concept map (Park and Calvo

6.4 User Input and Accessibility

55

Table 6.1 Comparison of interaction types Constrained responses e.g. choosing from a list

Open-ended responses e.g. typing a paragraph

Advantages

Simpler to assess for accuracy System can be designed to handle all possible inputs Less reliant on strong writing skills

Allows students to explain reasoning Students can express deeper understanding of content

Disadvantages

Student responses are limited May not provide enough information to identify reasons for misunderstandings

Reliant on strong writing skills Understanding of student input is limited by system domain knowledge System may misinterpret a student response and give incorrect feedback

2008). By providing students with a set of ‘nodes’ to link together the possible input values are constrained to a manageable level, making assessment computationally efficient. To be successful, students must display an understanding of the relationships between the given nodes and be sufficiently confident in their responses to omit any ‘distractor’ nodes that may be present. The ‘nodes’ can potentially contain text, images or even short videos and thus are accessible even to learners with poor literacy. All of the input methods can be used effectively at different times throughout the learning process, however care must be taken to ensure that the methods used align with the content being presented and the needs of the learner.

6.5 Customisation and Adaptability The target user group must always be considered when designing the persona of educational ECAs and the system surrounding them, however even within a user group there will be much individual variation. There are several options for coping with this, including implementing a system that can be customised manually, be it by the student themselves, a parent, or a teacher, and implementing a system that dynamically adapts to the user over time. In terms of manual customisation, the idea of lesson authoring tools that can be used by non-programmers, such as classroom teachers and parents, is gaining ground. Two examples of existing ECA-based educational software with associated lesson authoring abilities are the physics tutor AutoTutor (Susarla et al. 2003) and the social skills tutor Thinking Head Whiteboard (Milne et al. 2013) which will be discussed here in more depth in the chapter “The Thinking Head Whiteboard”. The goal of these tools is to allow for new content to be created that takes advantages of the benefits of these teaching approaches, either in addition to existing content or in an entirely new domain area, or to edit existing content to better suit individual learners. Individuality can also be responded to dynamically by ECA learning systems. A learner profile can be implemented and kept updated through tracking of responses, learning preferences and performance over time. Knowing that a particular student

56

6 Creating Engaging Embodied Conversational Agents

tends to perform better with visual guides rather than textual ones and presenting information in this manner has the potential to greatly assist the learner, just as identifying that a student already understands the basic concepts of a unit and fast tracking them to more challenging content will assist in maintaining engagement as well as improving academic performance. Zhiping et al. (2012) discuss four learner models, namely the stereotypes, overlays, buggy, and constraint-based models. The stereotypes model simply assigns learners a ‘level’, being either a novice, intermediate or expert. This is beneficial in that the level of scaffolding and the depth of instruction can be matched accordingly; however it is too coarse grained to provide much adaptability beyond this. The overlay model represents both knowledge that the learner has already acquired and knowledge that they need to learn, with the learner’s mastery of the domain being judged according to the overlap between the two. This is more flexible than the stereotypes model but still very coarse grained. The buggy model takes an alternative approach and considers user cognitive ability by recording the mistakes that the learner makes when they are solving problems, with the goal being to rectify the process that the learner takes, rather than the focus being on the ‘facts’ that the learner knows. Finally, the constraintbased model represents learner knowledge in the form of ‘if–then’ statements, where a student meeting all of the ‘if’ constraints is judged to understand that portion of knowledge. One of these models alone is unlikely to represent the user’s state of knowledge sufficiently, however taking inspiration from these and implementing a combination model may prove fruitful. To respond to individuality in learners, not only can the content and appearance of the system be manually customised, but a learner profile can be implemented including a model representing their current state of knowledge, their learning preferences, and their past performance. From this information, the software can adapt to individual needs, increasing both engagement with the system and academic performance.

6.6 Socially Responsive Agents Studies have shown that deeper and more meaningful learning occurs in social contexts rather than when working alone (Krämer and Bente 2010). To take advantage of this finding, it is imperative to create a virtual tutor that is sufficiently realistic and relatable so that the student engages with it in a social manner, constructing knowledge collaboratively as with a human peer or tutor (Krämer and Bente 2010) and building rapport with the user (Zhao et al. 2016). As discussed previously, it is likewise important to ensure that the virtual tutor does not impede or disrupt learning, for example interrupting with unwanted hints, as this can negate any positive effects the presence of a virtual tutor may have, and ultimately makes the educational experience much less enjoyable (Conati and Manske 2009).

6.6 Socially Responsive Agents

57

Case Study: Social Tutor for Autism Research exploring whether individuals with autism have a preference for computer generated voices or pre-recorded voice actors is mixed, with some suggestion that verbal children find computer generated voices “too synthetic” (Williams et al. 2004), others finding that non-verbal children actually perform better with computer generated voices than pre-recorded voice actors (Herring et al. 2017), and some finding no significant difference between the two (Ramdoss et al. 2011). Studies have shown that it is primarily the speech capability of a virtual agent that is responsible for motivating the learner and increasing the quality of their learning and problem-solving skills (Krämer and Bente 2010). It was found that agents that could speak led to students having fewer difficulties and being more able to apply their skills to other contexts than when they used a text-based agent. One explanation for this is that it enables students with weak reading skills to engage with the content more easily, allowing them to focus their thinking on the concepts involved in the task rather than the mechanics of reading the material. While pedagogical agents alone were not found to have a general impact on student motivation or learning, likeable agents did lead to some improvements. Interestingly, Tsiourti et al. (2016) found that for the older neurotypical adults in their study, having the virtual human mirror the emotional facial expressions of the user led to it being perceived as more likeable and persuasive. Case Study: Social Tutor for Autism For most people the nonverbal behaviour of an ECA strongly influences their perceptions of it, however there is little existing research to suggest whether this holds true for individuals with autism and other difficulties that impact on their social interactions. While research into measuring rapport with a virtual tutor exists, it focusses on neurotypical users and the techniques have not been validated with individuals on the autism spectrum or with other related difficulties. As Tsiourti et al. (2016) demonstrated, while speech is clearly important, an agent’s nonverbal communication also has an impact on engagement. Schilbach et al. (2006) demonstrated that virtual characters displaying social facial expressions, for example raising their eyebrows or smiling at the participant, caused the same brain regions to activate as they do in human–human interaction. In human–human interaction, nonverbal behaviour has several functions which may also be helpful in a virtual learning environment. For example, teachers model tasks for students, use illustrative gestures such as pointing, use gestures to emphasise important points and guide learner focus, as well as engage in dialogue management and turn-taking

58

6 Creating Engaging Embodied Conversational Agents

cues (Allmendinger 2010; Krämer and Bente 2010). It has been shown that smiling and other feedback cues affect student interest, motivation and learning outcomes, for example encouraging students to continue down a particular train of thought by smiling and nodding assists them to know they are progressing well (Allmendinger 2010; Krämer and Bente 2010). However, it is imperative that nonverbal behaviours appear sufficiently natural, as in human–human interaction they are processed automatically by the limbic system, and this may fail to occur if the behaviours appear odd (Krämer and Bente 2010). Interestingly, research has shown that sometimes incorporating socially accurate behaviours can negatively impact the perceived friendliness of the virtual agent. Hastie et al. (2016) found that incorporating episodic memory into their virtual tutor improved task performance, but decreased user ratings of likeability. They suggest that the virtual tutor may have reminded users of their past mistakes too often, and this was perceived negatively. Thus, it is clear that social features can have both positive and negative effects on the user, and care must be taken when incorporating them to balance these effects in a manner that optimises the end-goals of the system being developed.

6.7 From the Desktop to Virtual Worlds Until relatively recently, ECA-based educational software has been developed for individual users sitting at their home or school computer. Now the focus is changing, with many researchers examining collaborative learning approaches, web-based systems, and applications for mobile devices. These are exciting advancements with much potential, but of course there are accompanying pitfalls. Questions of equality and available technology must be asked, for example what happens if internet access becomes unavailable? What if the learners most at need, often with low socioeconomic backgrounds, do not have access to sufficiently advanced technology to run the programs aimed at them? Like many things, there are often ways to get around these issues, however they are real-world concerns and should be considered by software designers. Often, when people think about web-based learning they think of systems akin to forums where all the information is available but it is up to the learner to navigate through it in an appropriate sequence. This is not the only possibility. As described by Piramuthu (2005), with learning systems of this nature, even browser-based systems that are visually forum-like, adaptivity can occur both at the content presentation level and at the navigation level, ensuring that users are guided to undertake the available activities in a sequence that best suits them as an individual. Further, ECA agents can be included to assist the learner, implemented in a similar manner to ‘tech support’ and ‘marketing’ ECAs that are often found on websites today. Web-based learning does not need to be limited to this style of system. Take for example the ECA-based learning environments developed in Second Life (Okita et al. 2013). This is a much more immersive experience and facilitates collaborative learning with real

6.7 From the Desktop to Virtual Worlds

59

and virtual peers within the same environment. (Virvou and Troussas 2011) provide another example of a web-based ECA learning system with the client-side running as an applet on a web page. The system is aimed at second language learning with adaptability achieved through the use of a student model incorporating information about preferred learning style, capabilities and material covered. Web-based approaches such as those described here have many benefits above and beyond the typical advantages when using ECA-based learning systems in that they have the potential for facilitating collaboration between individuals who are physically remote from one another and they attempt to overcome the disparity in available technologies by being accessible online via a standard web browser. As the popularity of mobile devices such as smart phones and tablet PCs continues to rise, so too does interest in educational software for these environments. Tomlinson and colleagues discuss the concept of ‘embodied mobile agents’, or EMAs, that can migrate between various computational devices (Tomlinson et al. 2006). For example, an EMA could act as a virtual tutor as part of a sophisticated learning system on your desktop PC and then migrate across to your smart phone and continue its role there as part of a simpler ‘revision’ or ‘quiz’ app. Of course, simply having ECA-based software on a mobile device is a benefit as it can then be accessed at any time of the day regardless of where the learner is currently located, encouraging users to engage with it more often.

6.8 Summary Carefully managed appearance, voice and interaction style all contribute to a user’s positive educational experience with ECAs. Nass et al. (2001) stress the importance of appearance, voice and behaviour being in tune with each other in order to create authentic ECA personas, and the positive impact that matching that persona to the user’s own personality can have. Further, their research has shown that if a choice must be made between the ECA persona or matching the user’s personality, consistency in the ECA’s personality has a bigger impact and should be favoured, as users perceive consistent ECAs as more trustworthy and intelligent (Nass et al. 2001). Interaction style, be it ‘on-demand’ or ‘always present’, is also a key factor, however there are no hard and fast rules about what works best. Instead, consideration must be given to the characteristics of the intended user group. Physiologically, when individuals are stressed they become tense and less tolerant, however when they are happy they become relaxed and are likely to be more tolerant and creative (Shneiderman and Plaisant 2005). So, while all efforts are made to create ECAs that are authentic and accurate in their responses, the happier and more engaged we can keep our learners, the more forgiving they are likely to be of unavoidable limitations or faux pas. Finally, designers of virtual tutoring should carefully consider the method of system deployment given today’s ubiquitousness of mobile devices and internet access. Utilising the connectedness that these technologies bring provides many benefits, not

60

6 Creating Engaging Embodied Conversational Agents

least of which is the opportunity to provide engaging and stimulating collaborative learning experiences, or situation and location specific activities.

Chapter 7

Implementing a Social Tutor for Autism

Prior to developing the Social Tutor software discussed in the remainder of this book, an investigation into existing social skills interventions, both traditional and technology-based, was conducted. Technology-based interventions include hardware and software, with some incorporating virtual and augmented reality. Here, a brief overview of some of the more influential and novel interventions are given. As there is a wealth of fascinating, cutting edge interventions under development in this area, only a small selection is included here, namely those that fit particularly well with the objectives of the Social Tutor, are well established in the field with experimental support, have practical elements that could be directly drawn from and implemented in the Social Tutor, or otherwise provide unique insight to assist in this development. Table 7.1 provides an overview of these highlights, with a more detailed discussion of each featured intervention following.

7.1 Traditional Interventions 7.1.1 Story and Comic Style Interventions One theory that attempts to explain some of the social difficulties that children with autism encounter suggests that they lack a fully developed ‘Theory of Mind’. This means they have difficulty understanding that other people have separate thoughts and feelings to themselves. Carol Gray’s Social Stories™ and Comic Strip Conversations (Bock et al. 2001; Gray 2001; Quirmbach et al. 2008) and the thought bubble approach used by Wellman et al. (2002) aim to address this deficit. One of the best known and most influential social skills interventions for children with autism are Carol Gray’s Social Stories™ (2001). These are instructional stories that explain how to behave in particular social situations. They are written following a set of guidelines developed by Carol Gray, that state that sentences should be © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_7

61

62

7 Implementing a Social Tutor for Autism

Table 7.1 Examples of existing interventions for children with autism Intervention title

Summary

Evaluation outcomes

Applied behaviour analysis (Lovaas 1987; Schreibman 2000)

Therapist provides consequences for behaviours, e.g. objects, food and actions that the learner finds reinforcing when desirable behaviours occur

Well established as a highly successful technique, particularly when used for early intervention. Time intensive and can be overwhelming for the participant (Sallows and Graupner 2005)

Social stories™ (Balakrishnan and Alias 2017; Gray 2001; Quirmbach et al. 2008)

Carefully formatted instructional stories explain how to behave in particular social situations

Extensive study by Quirmbach et al. (2008) indicates high effectiveness, however outcomes rely heavily on the quality of the stories used

LEGO therapy and the social use of language programme (SULP) (Owens et al. 2008)

Compared LEGO building tasks as facilitator of social interaction against established peer group intervention SULP which involves stories, modelling and role-play

LEGO therapy reduced maladaptive behaviours, SULP improved social and communication skills. LEGO found to be a natural motivator and helped children interact with peers

Video modelling (Dowrick 2012; Marcus and Wilder 2009)

Children watch videos of themselves (self-modelling) or others (peer-modelling) correctly performing desired behaviours

Strong evidence of effectiveness, especially self-modelling, but may need to be used in conjunction with other techniques to maintain long term behavioural changes (Reichow and Volkmar 2010)

Robots (Huijnen et al. 2016; Kozima et al. 2009; Robins et al. 2005; Scassellati 2005)

Robots can be used as social facilitators for high functioning children with autism and for eliciting verbalisation from children who are rarely verbal

Robots appear to be naturally engaging for children with autism, making them potentially effective for a range of applications. Most existing research is still exploratory

SIDES Cooperative table top game (Piper et al. 2006)

A touch-screen table top game designed to encourage cooperative skill building between up to four players

Initial play testing indicated high engagement and excitement but more structure required to ensure fairness and enforce pro-social behaviours

Traditional interventions

Hardware interventions

(continued)

7.1 Traditional Interventions

63

Table 7.1 (continued) Intervention title

Summary

Evaluation outcomes

Emotion bubbles (Madsen et al. 2008)

Wearable camera system includes automated facial expression detection software that lets users know what the expression on a detected face is via colourful bubbles displayed on screen

Game-like activities were found to be very engaging for users and helped them critically analyse faces for unspoken social cues

Virtual reality fully immerses the learner in a simulated environment, often using specialised helmets or other hardware

Virtual environments can improve some daily living skills, however high functioning individuals do not always behave as they would in the real world making generalisation of skills uncertain for some audiences (Parsons et al. 2005)

Software interventions Virtual reality (Cheng et al. 2015; Herrera et al. 2008; Ke and Im 2013; Parsons et al. 2005)

Augmented reality (Chen et al. Augmented reality provides 2015; Washington et al. 2016) information overlayed on a view of the real external environment, often using wearable technology such as specialised glasses or a helmet

Systems have been shown to increase the percentage of facial expressions correctly recognised and responded to. Generalisation expected to be supported given grounding in the real environment

Group therapy linked software (Beaumont and Sofronoff 2008; Whalen et al. 2010)

Software developed in conjunction with group therapy provides learners with opportunities for linked practice in the form of role-play with peers and real-world experience

Both examples here resulted in improved test scores from pre- to post-intervention. Hybrid approaches show promise in addressing generalisation from intervention to real-world application (Whyte et al., 2015)

Standalone software (Abirached et al. 2011; Hopkins et al. 2011; Silver and Oakes 2001; Sturm et al. 2016)

Standalone software requires no special hardware beyond a home computer making it highly accessible. It is cost effective and can be used often and independently by learners

The examples here indicated high levels of user engagement and led to improvements in target skills, however generalisation to novel contexts was not evaluated

short and explicit and accompanied by simple, informative icons that support understanding. An extensive study evaluating the efficacy of Social Stories was conducted by Quirmbach et al. (2008). It involved forty-five children in a randomised control trial and examined the ability of the stories to elicit, maintain and generalise cooperative behaviours in a game. The results strongly supported the effectiveness of

64

7 Implementing a Social Tutor for Autism

Social Stories™ for this purpose, as all children with average verbal skills or above, as measured by the Verbal Comprehension Index, made significant improvements. Social Stories™ are very visual and provide explicit instructions specific to the situation, suiting the typical learning style of children with autism. Several other smaller studies support these findings (Balakrishnan and Alias 2017; Delano and Snell 2006; Sansosti and Powell-Smith 2008), while other studies suggest that the effectiveness of Social Stories™ can be variable and reliant on a variety of factors, including the quality of the stories themselves and the behaviours they are being applied to, particularly given that untrained individuals are often the ones responsible for implementing the intervention (Lorimer et al. 2002; Reynhout and Carter 2006). In the same vein as Social Stories™, Carol Gray has developed Comic Strip Conversations, which are developed following similar rules to the stories, but in comic strip format. The use of these comic strips has led to similarly positive results. Wellman et al. (2002) also use a pictorial approach but start with a more concrete version and gradually work towards the more abstract images. Initially, Wellman et al. (2002) used dolls with cardboard cut-out thought bubbles above their heads, and gradually reduced the concrete supports. Wellman et al. (2002) concentrated on generalisation of skills to real social situations with peers and demonstrated increased performance with skill transfer to novel contexts. In all of these interventions the focus is on providing visual supports to aid understanding along with clear and concise step-by-step information, as suits the typical learning style of individuals with autism. These same guidelines can also be incorporated into software developed for this user group, whenever visual, written or spoken information is presented.

7.1.2 Play Based and Peer Group Interventions Peer group interventions are used extensively to help develop social skills in children with autism, often in conjunction with other methods and tools such as LEGO® , robots or software such as the Junior Detective game. Children with autism are less likely than their neurotypical peers to initiate social interaction, often play alongside rather than with peers, and typically engage in less sophisticated interaction behaviours. It is thought that an object of mutual interest, for example LEGO® , acts as a facilitator and can help children with autism interact more richly with peers. Peer group interventions range from short but frequent school-based groups, often including neurotypical peers, to longer and less frequent clinical groups, all of which have evidence to support their effectiveness to varying degrees (Owens et al. 2008; Reichow and Volkmar 2010). Less formal, naturalistic approaches that centre on activities and materials that are naturally motivating and reinforcing and occur in the everyday life of the children with autism have been shown to support generalisation of skills. One such intervention is LEGO® therapy. A study by Owens et al. (2008) contrasted two peer

7.1 Traditional Interventions

65

group therapies for 6- to 11-year-old children, LEGO® therapy and the Social Use of Language Programme (SULP). In LEGO® therapy, children in small groups are given roles, and must work together following social rules to build a LEGO® construction. The small group can include neurotypical peers and adults as well. The construction task requires group members to use many social behaviours including joint attention, verbal and nonverbal communication, collaboration and problem-solving skills. LEGO® is particularly suited to this learner group as it is predictable and systematic, fitting with their common preference for consistency. The SULP intervention is used by a number of schools and therapists and begins with stories, then adults modelling desired behaviours, followed by the children practicing these behaviours and playing games within the social group. Finally, activities are performed in new situations to encourage generalisation. Owens et al. (2008) found that the children involved in LEGO® therapy reduced their maladaptive behaviours, and those in the SULP group improved their social and communication skills, with both intervention groups outperforming those in the control group. Since the two therapies appear to target different sets of social skills more research is required, however both did lead to improvements in social behaviour and both were relatively cost effective and easy to implement. The aim of a virtual peer as an intervention is not to replace learning opportunities such as those experienced in real peer to peer play, but to provide a helpful first step in leading to the development of the sophisticated behaviours required for rich, everyday social interactions.

7.1.3 Applied Behaviour Analysis Applied Behaviour Analysis (ABA) is one of the most widely known techniques for reducing undesirable behaviours and increasing preferred behaviours in children with autism (Lovaas, 1987). Traditional ABA involves a therapist providing direct consequences, for example providing objects, food and actions that the learner finds reinforcing when desirable behaviours occur. While traditional ABA is very effective at teaching desirable behaviours to children with autism, problems with generalisation to novel contexts and self-initiation of behaviours were found (Schreibman, 2000). Modern ABA aims to address these issues by incorporating more naturalistic behavioural approaches that use real-world settings and are more child-driven, and this has had demonstrated success (Schreibman, 2000). ABA is highly effective when used as an early intervention technique, with Sallows and Graupner (2005) finding that approximately 48% of children under 5 years old who received the prescribed ABA intervention were successful in mainstream school classrooms by age 7, and many more made significant improvements to their language, intellectual and adaptive skills. There are some shortcomings to this approach, particularly its time-consuming nature. ABA also relies heavily on trained professionals, which quickly becomes expensive. Additionally, it requires the child to interact in an intense fashion with another human being which can be very confronting, at least initially

66

7 Implementing a Social Tutor for Autism

(Hailpern 2007). The ABA approach is known to be effective for many individuals and is widely used for a variety of different applications. Many programs similar to that of Lovaas have been developed and its principles, such as prompting and positive reinforcement, are used in a range of settings (Reichow and Volkmar 2010). These principles are likewise suitable for inclusion in the social tutoring software being developed here.

7.1.4 TEACCH Intervention Panerai et al. (2002) investigated the effectiveness of the Benefits of the Treatment and Education of Autistic and Communication Handicapped Children (TEACCH) programme as compared to a control group who were in typical classrooms with support teachers. They found that students in the TEACCH program made significant gains across the duration of the evaluation. TEACCH provides continuous, structured intervention, has a strong focus on the use of visual aids to make abstract concepts more concrete, and provides for environmental adaptation and training in alternative communication (Panerai et al. 2002). As autism is a pervasive disorder, TEACCH is designed to be used in all aspects of the learner’s life instead of being restricted to specific learning sessions. The use of visual aids, adaptations in the learner’s environment, and the focus on providing more methods of communication are all important general principles that are widely used in a variety of educational situations for children with autism and provide valuable guidance for the educational approach implemented in the current research.

7.1.5 Video Modelling Video modelling is a technique in which the learner is shown a video of someone, possibly a peer or themselves (self-modelling), performing an action that the learner is intended to acquire. Video modelling has many advantages including that minimal expertise or expense is required to implement the intervention, it is repeatable, it can be conducted in a standardised manner, and it is portable. A review by Reichow and Volkmar (2010) into best practices for social skills interventions found numerous studies supporting the effectiveness of video modelling but suggest that video modelling alone may not be sufficient to maintain long term behavioural changes and state that more research is required into exactly what circumstances optimise the effectiveness of video modelling, for example the type of model, such as self, peer or adult. A more recent review by Wong et al. (2015) likewise indicated strong support for the use of video modelling as an evidence-based practice for teaching skills to individuals with autism generally, and work by Dowrick (2012) suggests that the reason self-modelling is successful is that it increases learners’ ability to see their own potential in achieving the target behaviour.

7.1 Traditional Interventions

67

Marcus and Wilder (2009) compared the effectiveness of self-video modelling and peer-video modelling with three children with autism, one four-year-old male, one nine-year-old male and one nine-year-old female. The acquisition task was for the children to learn the sounds and symbols for a set of Greek and Arabic letters. In the self-modelling condition, all three children reached the mastery condition whereas only one child did in the peer-modelling condition. Anecdotally, the authors reported that children enjoyed the self-videos more and even wanted to watch them after the study was concluded. However, this study involved a textual task not a socially oriented one. Sherer et al. (2001) compared self and video modelling for teaching conversation skills to five children, but found no significant difference between the two, with some learners performing better in one condition and some in the other. More recently Sng et al. (2014) reviewed video modelling and scripts for teaching conversation skills specifically and found video modelling to be borderline between questionable and effective as an intervention for this purpose. Thus, video modelling has strong evidence of effectiveness for teaching many different types of skills to individuals with autism, including social skills, but more investigation is required for conversation-related skills specifically. It is hoped that the human-like appearance and behaviour of the virtual characters in the software developed for the current research may capitalise on the same effects that cause video modelling to be so successful, thus enhancing educational outcomes for learners.

7.2 Technology Based Interventions It is often said that individuals with autism have an affinity for computers and technology in general, and both the survey by Putnam and Chong (2008) and the recent investigation of technology usage patterns among adolescents with autism (MacMullin et al., 2016) support this. Consequently, any technology-based intervention is likely to be appealing to young people with autism. Combining this innate interest with educational content is hoped to prove very beneficial for them educationally. In a study by Jacklin and Farr (2005), the impact of computer use in general on the social interactions of children with moderate autism was investigated. The motivation behind this was that using a computer would provide an object of joint attention and would help to lower anxiety levels, making the social interaction more enjoyable and relaxing. When the children were focussed on their computer-based tasks, fewer self-stimulatory behaviours were observed, and they were more willing to interact with their teachers. Even more encouraging was that in a follow up case study better turn taking and on-task behaviour was observed, fewer maladaptive behaviours were present, and the participants displayed improved eye gaze and problem-solving skills. Jacklin and Farr (2005) emphasise the importance of monitoring computer use to ensure that it does not reinforce any obsessive or repetitive behaviours or increase the learner’s social isolation.

68

7 Implementing a Social Tutor for Autism

As there are many technology-based interventions currently available, only a sample is given here. These are restricted to approaches that have been experimentally validated, provide a particularly novel approach, or are directly informative in the development of the Social Tutor and the choice of its platform.

7.2.1 Robots and Hardware While robots are very appealing and motivating for many children with and without autism, extensive research into their efficacy is still somewhat lacking, with most existing research being exploratory in nature (Huijnen et al. 2016). Furthermore, robots can be quite expensive and present numerous drawbacks in terms of their usefulness as social skills interventions. Robots have a set appearance, not being customisable in this sense. This makes generalising any social skills that children with autism may develop while using the robot into a considerable challenge. Additionally, their appearance is typically very dissimilar to a real human. For children who find faces difficult to look at this may be an advantage, making the robot an anxiety free learning tool, but conversely it is likely to make generalisation of skills to a real person difficult. Thus, robots may not be best suited to the purpose of teaching ‘social etiquette’ between socially active individuals, however there is evidence of their potential as social facilitators, helping to break down barriers and make interacting with peers and adults easier for children with autism (Huijnen et al. 2016). Research from the AuRoRA group has demonstrated that robots help to engage high functioning children in social interaction with adults and their peers, and help low functioning children engage in parallel play, an important first step towards socially interactive play (Robins et al. 2005; Werry et al. 2001). Another group of researchers have also investigated the notion of robots as social catalysts, with equally promising results. Scassellati (2005) found that by reacting to participants’ actions, rather than simply following a set script, the number of social behaviours from the participants towards the robot was significantly higher. It was found that even a very simple robot following a set script was potentially useful for encouraging low functioning, rarely vocal children with autism to elicit vocalisations, generating excitement and many utterances from participants (Scassellati, 2005). Another simple, commercially available robot is Keepon (Kozima et al. 2009). Keepon has been carefully designed to ensure that it conveys the potential for social agency and emotional expression while being very simple in appearance, in line with its capabilities, and making it socially accessible for young children with autism. It can be used in both autonomous and authorable mode, and approximately 400 h of interaction data has been collected over the course of four years. Interestingly, Keepon has been shown to elicit social actions from children on the spectrum, including spontaneous shared observation of Keepon’s mental states with a third party such as a caregiver. A more recent example of a social robot is Nao, who is also commercially available and able to act both autonomously and in an authorable ‘Wizard of Oz’ mode (Huskens et al. 2015; Warren et al. 2015). Nao has been used as a mediator in a

7.2 Technology Based Interventions

69

LEGO-based intervention and has also been used as part of an autonomous system designed to model social gestures to children, assess the quality of their imitations, and give feedback. It should be noted that outcomes so far have been mixed, with one suggested explanation being the limited repertoire of social responses Nao can produce. Still, Nao remains a very interesting platform for future research. For a more extensive list of existing social robots see Huijnen et al. (2016). While robots may not currently be suited for teaching rules of social etiquette, there is clearly potential for many other social and language skill benefits to be gained from their use. A few novel hardware-based interventions have also been developed, notably the SIDES cooperative tabletop game and the Emotion Bubbles portable system. The SIDES cooperative tabletop game was developed in close consultation with twelve high school students with autism. The goal was to develop a game that encouraged cooperative skill development without it feeling like an educational game (Piper et al. 2006). A sturdy touch screen big enough for four players to sit around and interact simultaneously is at the heart of the system. As many individuals with autism experience poor fine motor skills, a large touch screen makes it accessible to a wider range of learners. The game itself enforces the rules, making it more predictable than a human ‘referee’ and helping to reduce anxiety while learners are having fun and developing confidence in their social skills. Initial play testing indicated that the system was very motivating and exciting, but perhaps too exciting as players often talked over each other and quieter players were left out. Increased built-in structure is required to encourage more pro-social behaviours (Piper et al. 2006). The Emotion Bubbles system also mentioned combines a small portable computer and a software package that aims to help learners with autism to read facial expressions (Madsen et al. 2008). The computer’s camera can be pointed towards a person’s face, which is then analysed in the software and the ‘emotion bubbles’, represented on-screen as colourful circles, will grow or shrink to indicate which emotion is being represented on the tracked face, and to what extent. A pilot study involving three high school age males evaluated the potential of the system. The participants were asked to point the camera at their partner and try to get them to display particular emotions, using the system as a guide. The results suggest that this technology has much potential, as the boys quickly understood how to use the software and appeared to thoroughly enjoy the experience. The next stage of development for the Emotion Bubbles system is applying it to teach skills that can be used in real social contexts. While interesting lessons can be learned from these robot and hardware-based systems, the need for specialist equipment and its cost can be a barrier to uptake for many families, and having a fixed configuration can be limiting. Thus, the Social Tutor is targeted for use on standard home computers.

7.2.2 Virtual Environments and Augmented Reality Virtual and augmented environments are appealing and motivating for most learners and are thought to promote generalisation to real-world contexts as the learner is

70

7 Implementing a Social Tutor for Autism

either entirely immersed into a simulated environment or is interacting in the real environment with additional information overlaid on some form of display. These technologies provide learners with the opportunity to role-play scenarios in a realistic yet supported environment. However, these approaches also come with limitations. Like the robot and hardware-based approaches, virtual and augmented reality often require specialised, sometimes costly, equipment. For some learners, particularly those with sensory issues, having to wear equipment such as helmets or goggles can also be a major barrier, and while virtual and augmented reality applications can be deployed as three-dimensional worlds on a typical computer or mobile device, the immersive effect is not as strong. Perhaps more concerning, Parsons et al. (2005) found that teenagers with autism behaved differently in the simulated environment than they would in a real environment, and stated that because they knew the environment was not real, they did not feel the need to behave in their normal manner. This suggests that generalisation for this functioning level and age group may not be supported. However, virtual reality has been shown to lead to significant benefits for children with autism for other purposes, such as teaching life skills including finding a seat in a crowded cafe and safely crossing the road (Kerr 2002; Strickland 1998). Herrera et al. (2008) developed a virtual environment that used a scaffolding approach to gradually take children from functional interaction to imaginative play. In this manner, abstract ideas can be made concrete and illustrated clearly. Through use of this virtual environment children improved their skills, with one participant even generalising their skills to another context. Children with autism have difficulty identifying their mistakes and the causes behind them and must be explicitly taught how to deal with new situations. In a situation with peers, making a social mistake can cause severe anxiety and discomfort for the child. Thus, collaborative virtual environments which facilitate role-play between real humans but in a controlled manner may provide a highly beneficial environment for learners with autism to practice their social skills in a less threatening context (Kerr 2002). Software-based learning opportunities make it easy to keep initial scenarios simple and gradually add distractions and complexities as the learner increases their confidence. Kerr emphasises that the purpose of virtual environments as autism interventions is as a valuable tool for developing learners’ social skills repertoires, and must be accompanied by practice in real social situations. These same advantages and caveats apply to the development of the Social Tutor for this research. Existing virtual worlds such as Second Life provide another interesting avenue for investigation, particularly since they are reasonably accessible to families, requiring only the use of a standard computer and not any specialised equipment. Ke and Im (2013) developed a set of social skills focussed Second Life tasks and tested the efficacy of this approach with four primary school aged children. A group of adults with education backgrounds were also recruited for the study, their role being to control characters within Second Life, interacting with the children during their learning tasks as communication partners and facilitators. It was found that in general the participants improved their ability to initiate and maintain social behaviours, and also improved their dispositions towards developing peer friendships

7.2 Technology Based Interventions

71

and engaging in social interactions with others (Ke and Im 2013). While promising, this approach required the involvement of adult mediators, thus learners were not fully self-sufficient, in contrast with the goal of the Social Tutor being developed here. A more recent study by Cheng et al. (2015) involved development of a threedimensional virtual environment to teach various aspects of social understanding, deployed using a head-mounted display. They conducted a preliminary study over six weeks with three participants on the autism spectrum, aged 10–13, and found that the target behaviours improved from baseline to intervention, and improvements were maintained at two, four- and six-weeks post-intervention. While Cheng et al. (2015) did not formally evaluate generalisation to everyday situations, anecdotal evidence suggests that some generalisation did occur, for example one participant increased their efforts to socialise with the researchers, use manners and raised their hand when the virtual character asked a question. While only a preliminary evaluation, it lends support to the idea of virtual environments as promising tools for improving social skills in children on the spectrum. Augmented reality is another interesting technology gaining ground in the area of autism intervention, particularly when paired with wearable or otherwise mobile devices. The recent release of Google Glass has opened up new avenues for researchers, with Washington et al. (2016) harnessing the technology to create a prototype wearable social aid for children with autism. The system uses automated emotion recognition and provides social cues in real-time on the heads-up display. The system can run in a casual mode, or wearers can engage in gamified activities that encourage them to develop their emotion recognition skills. The system also autorecords ‘emotional moments’ throughout the day that can be reviewed by parents and therapists via an Android application. An initial evaluation of the system has been conducted with twenty children with autism and twenty typically developing children, aged 6–17 years old. Children responded well to wearing Google Glass and enjoyed the gamified activities and feedback mechanisms. Interestingly, participants overwhelmingly preferred verbal cues over visual cues, finding the visual cues distracting (Washington et al. 2016). While still early days, the combination of augmented reality with wearable technology holds much potential for social skill development in children with autism. However, for the purposes of the current research, equipment that may present a barrier to uptake for families is undesirable.

7.2.3 General Software A wide range of software targeting many of the difficulties associated with autism are available, for example software has been developed to encourage vocalisation in pre-vocal children (Hailpern et al. 2009) and to encourage development of spoken language in young children at the earlier stages of language acquisition (Lehman 1998), however many of these programs have not been experimentally validated. Some examples of software that have received positive experimental results and

72

7 Implementing a Social Tutor for Autism

focus on social skills for higher functioning individuals include The Junior Detective, Teach Town: Basics, FaceSay and Emotion Trainer (Beaumont and Sofronoff 2008; Hopkins et al. 2011; Silver and Oakes 2001; Whalen et al. 2010). The Junior Detective and TeachTown: Basics are both computer-assisted intervention programs that involve a software use component alongside opportunities to practice skills in real-world role-plays (Beaumont and Sofronoff 2008; Jones et al. 2016; Whalen et al. 2010). Hybrid approaches such as these appear to be promising in addressing the issue of generalisation from intervention to real-world application (Whyte et al. 2015). The Junior Detective computer game was evaluated as part of a sequence of social skills group therapy sessions, where students were given opportunities to role-play the skills taught in the game. It was found that this combination led to significant improvements in the participants’ social skills and their ability to suggest strategies to manage their emotions and those of others. In a follow up session months later, participants had maintained their skills (Beaumont and Sofronoff 2008). The TeachTown: Basics software takes an ABA approach where learners are taught using a discrete trial format and correct responses are reinforced immediately with praise and graphics, and on a variable ratio also rewarded with short, animated games (Jones et al. 2016; Whalen et al. 2010). The TeachTown Connection real-world activities aimed to generalise the skills taught in the software as well as teaching additional skills and utilise principles of Pivotal Response Training. Most students showed significant improvement from pre-test to post-test, including on standardised measures (Whalen et al. 2010). Both TeachTown: Basics and The Junior Detective demonstrate how software can be used as step in the scaffolding process that leads to the development and maintenance of sophisticated social behaviours and problem-solving skills. Emotion Trainer, FaceSay and LIFEisGAME are examples of software designed to teach children how to identify emotions based on the appearance of peoples’ faces (Abirached et al. 2011; Hopkins et al. 2011; Silver and Oakes 2001). Silver and Oakes (2001) developed Emotion Trainer, which presents learners with an image or text description of an emotional face or scene and proves multiple choice buttons for learners to use to indicate which emotion is being depicted. Students are rewarded with a ‘well done’ message and a simple animation for a correct choice and asked to ‘try again’ and given a direct cue for an incorrect choice. While there are five sections of increasing difficulty, the program does not adapt to the user. The Emotion Trainer was evaluated using a randomised control trial in which eleven pairs of children with autism matched by age, school grade and gender participated. One child in each pair used the software while the other child did not. All children who used the software improved their skills, but to varying degrees, compared to those who did not. Additionally, children were able to generalise their skills to a similar paper-based task, but their ability to apply their skills to real social situations was not investigated (Silver and Oakes 2001). While Emotion Trainer offered one primary type of task and had one goal, the FaceSay software offers learners a range of games, with the overall aims being to increase their skills in emotion detection, face detection and social interaction (Hopkins et al. 2011). The games include identifying what object a face was looking

7.2 Technology Based Interventions

73

at, matching the missing facial part to a given face, and matching the expression on a pair of faces. Again, the software did not adapt to the user. It was found that children classified as having ‘low functioning’ autism improved on both emotion recognition and social interactions, while high functioning children improved in these target areas as well as facial recognition (Hopkins et al. 2011). In contrast to FaceSay and Emotion Trainer, LIFEisGAME takes a unique approach to teaching emotion recognition and uses Active Appearance Models to have a virtual character directly mimic the user’s own facial expression in real-time (Abirached et al. 2011). The pilot study presented users with a set of games ranging from observation and recognition, to matching a shown expression with their own face. Users responded well to these games, and the approach was found to be highly motivating. In more recent research, the serious game eMot-iCan has been developed for mobile devices and is designed for teaching and assessing emotion recognition skills (Sturm et al. 2016). The authors suggest that atypical attention patterns may be behind many of the social and communication difficulties experienced by individuals with autism and aim to explicitly teach users what elements to pay attention to in order to read facial expressions. Users are presented with a set of photo or cartoon images and must choose the correct match. Some additionally noteworthy features of this work include that administrators can customise the trials for individual users, and that being designed for a mobile device means that it can be taken to clinics and schools and used in a consistent manner across various environments (Sturm et al. 2016). Pilot results suggest that both administrators and children found their aspects of the software intuitive to use and engaging. Well-designed software certainly appears to be a promising avenue for basic social skills development in children on the autism spectrum, with the added benefits of it not requiring any specialised equipment and typically not presenting any major barriers for individuals with sensory difficulties. Many lessons can be learned from the sample of technology-based interventions provided here, particularly around the importance of scaffolding and insights into the kinds of activities that learners find engaging and useful.

7.2.4 Selecting Curricula While the ideal scenario when producing educational software would be to develop the software and content simultaneously, creating and empirically evaluating a curriculum is a vast undertaking in itself. For this reason, it is often the pragmatic choice to implement content that has already been developed and experimentally validated in a different context. In selecting an existing curriculum for such an application, a range of considerations must be addressed including suitability of activities for a software-based context, content that can be implemented while still maintaining integrity and validity, and overall engagement.

74

7 Implementing a Social Tutor for Autism

In order for a curriculum to be suitable for implementation in educational software, there are several features that are especially desirable and some elements that are difficult to translate into a software context. Inclusion of recommended scripts for the teacher or characters to speak, explicit step-by-step instructions relating to how particular skills should be performed, and a high level of visual material and digital media content such as worksheets, demonstration videos and songs, are all likely to make for engaging content and are straight-forward to translate into a software context with high fidelity, ensuring the validity of the curriculum is maintained. Depending on the level of detail given and the approach taken by the curriculum, role-play can also be a valuable feature that, given the unique nature of software including virtual humans, can to an extent be implemented in this context. At the very least, role-plays can be modelled between the virtual characters and then the learner encouraged to analyse and respond to what they observed. There are a number of features that are commonplace in many educational materials but difficult to reliably implement in a software context given the limitations of current technology. The main examples of such features include anything reliant on the educator observing and responding to the learner in a natural setting, and openended discussion or role-play. In a software context it is imperative to ensure that all possible interaction pathways can be responded to in a meaningful, or at the very least non-confusing and non-counterproductive, manner. While human–computer interaction technologies that could potentially be implemented to simulate these teacher-facilitated activities are available, as discussed previously the technology in many cases is not yet reliable enough to ensure that the student is always provided with correct feedback, and therefore could unintentionally lead to reinforcement of misunderstandings.

7.3 Assessing Social Skills There are many established social skills assessment tools in existence, with two of the most commonly used and recommended being the Matson Evaluation of Social Skills with Youngsters (MESSY) and the Social Skills Rating System (SSRS) (Wilkins 2010). In both MESSY and SSRS, evaluation items are presented in Likertstyle scale and forms exist for the individual, parent and teachers to respond. This process can be automated, and thus incorporated into an autonomous social tutor. MESSY has sound psychometric properties and has been validated for use with individuals with autism, whereas SSRS exhibits some inconsistencies from test to retest and lower inter-rater reliability, thus use of the MESSY assessment tool appears preferable (Wilkins, 2010). Both MESSY and SSRS are valid for use with primary and secondary school age children, and thus are applicable here. The Behavioural Assessment of Social Interaction in Young Children (BASYC) is another tool that may be useful for high level assessment in the social tutoring application, as it is designed to be easy for teachers to administer and thus does not require psychology training to perform, and can be used for goal planning and monitoring

7.3 Assessing Social Skills

75

existing social skills treatments (Gillis et al. 2010). It has been developed to meet the need for an objective measurement system for social behaviours and to inform intervention planning and monitoring, thus its goals marry with the requirements of this study on several levels. BASYC provides a list of interactions as a guide and a checklist of behaviours, so the influence of examiner subjectivity is minimised, and the task of automating assessment is simplified. However, completing this assessment requires behavioural observation in naturalistic, semi-structured settings. While there is potential for BASYC to be adapted to a software environment where a virtual peer behaves as the examiner, experimental evaluation would be required to determine if the assessment maintained its validity in this context. More recently Social Skills Q-Sort (SSQ) has been developed to screen for autism in a school-based setting (Locke et al. 2013). It is designed for use by paraprofessionals and involves sorting a set of one hundred items into nine piles according to those that describe the child most to least. While a unique approach and appropriate for use by non-clinicians, its purpose is more about overall diagnostics and less about developing a profile of strengths and difficulties that could be used to inform the topic sequence of an intervention. Theory of Mind (ToM) techniques are another suggestion for evaluating social awareness in virtual role-plays. In evaluation of the ‘Fear Not!’ educational program for constructively dealing with bullying, Hall et al. (2009) evaluated neurotypical children’s social awareness through ToM questions. Children were presented with bullying scenarios acted out by virtual characters and following this were asked by the ‘victim’ character for advice. At the conclusion of the program, children were provided with a questionnaire asking them to judge how various characters felt at different points throughout the story. Questions were devised by experts in the field and asked learners to make inferences about mental states, emotions and intentions of the characters. Students were asked a combination of short answer and multiple-choice style questions, which were accompanied by visual prompts, such as screen shots, to help them remember the role-play. Hall et al. (2009) found this technique provided valuable insight into the children’s social awareness of the presented situations, however application of this insight was not discussed. Assessing social awareness is a challenge as socially competent adults still often disagree on the interpretation of a social situation, thus there is often no definite distinction between ‘right’ and ‘wrong’ answers, rather answers fall on a continuum from less to more probable. This makes it particularly challenging to implement robustly in an automated manner as is required in this Social Tutor, and makes the implementation of a variety of techniques combined using heuristic rules and scoring thresholds a more viable approach for the current research.

7.4 Summary Creating social tutoring software for individuals with autism is a complex task and care must be taken at each stage of development to ensure best practice principles

76

7 Implementing a Social Tutor for Autism

are followed and that decisions are evidence-based. Lessons and inspiration can be drawn from the wealth of existing related work, ranging from traditional interventions through to technology interventions including robots, extended reality technologies and other software.

Chapter 8

The Thinking Head Whiteboard

To investigate the potential of virtual human-based educational software for teaching social skills to children with autism, the Thinking Head Whiteboard was produced. The Thinking Head Whiteboard provides a mechanism to control multiple instances of the virtual human software ‘Head X’ (Luerssen and Lewis 2009), and is also used to display interactive lesson content, provide automated assessment, dynamic lesson sequences, and feedback. A summary of the system is provided here, with further detail found in Milne et al. (2013) and Milne et al. (2018). The combination of the Thinking Head Whiteboard and the Head X instances it controls will be referred to as the ‘Social Tutor’ from here on.

8.1 Design Overview The Thinking Head Whiteboard was designed specifically with individuals on the autism spectrum in mind. People with autism often experience sensory difficulties, be it difficulties with sensory integration or atypical sensory tolerance. For this reason the Thinking Head Whiteboard is visually simple with low sensory demand, as shown in Fig. 8.1. This approach ensures accessibility to a wide audience, as designing software in this way ensures individuals with autism are supported, while still being perfectly accessible and appropriate for neurotypical individuals and those with other learning needs to use as well. As with most software, it was also designed to be intuitive to use so that learners could engage immediately with their learning. In the case of more complex elements or less obvious features, the virtual humans are available to guide students and take them through the necessary processes step by step. The software also has many unique features behind the scenes including automated assessment, dynamic lesson sequencing, and a three-tier rewards system (see Milne et al. 2018). In response to the developer’s research requirements, the current feature set of the Thinking Head Whiteboard is skewed towards teaching in two areas: social skills and literacy. Thus, it includes options for interactive activities such as drag and drop © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_8

77

78

8 The Thinking Head Whiteboard

Fig. 8.1 The ‘Social Tutor’ consisting of the Thinking Head Whiteboard and three Head X instances representing the teacher “Kate’ and two peers “Anna” and “Jack”

sorting, word grids, highlighting and cloze activities, simple role-plays, question lists, simple speech recognition, flexible cartoon faces, concept maps, drawing, video, and even a very simple RPG game. Given the modular nature of the system it would be straightforward for other software developers to extend the feature set to support niche areas that are not covered by the current selection.

8.2 Lesson Authoring and Customisation The Thinking Head Whiteboard was designed to allow educators and caregivers with little to no programming background to modify and create simple, interactive lesson content for their learners, and thus includes XML-based lesson and curriculum authoring capabilities (see Milne et al. 2013 for technical details). Each XML lesson file is intended to contain one short, focussed interactive activities such as a single drag and drop sorting task, a concept map to be completed, or a ‘Social Story’ role-played by the virtual humans. Lesson files can optionally specify prerequisite activities the learner must complete first. Curriculum XML files group lessons into objectives, and objectives into overarching topics. They also specify minimum achievement thresholds that learners must reach for each lesson and objective to be considered complete, and specify which lessons are compulsory ‘core’ activities and which are ‘extra’ activities intended to support learners who need more practice to reach concept mastery. This structure supports educators to describe constraints ensuring appropriate scaffolding is achieved for learners, while also allowing the lesson sequencing system to mix and match activities in a responsive manner to meet learner needs on the fly.

8.2 Lesson Authoring and Customisation

79

Using simple XML files to define lesson content means that anyone can easily open a file of interest using a text editor—no special tools needed—and adjust the images, text or virtual human speech to suit their learner. For example, if your learner responds better to cartoon images and the lesson provided features photos, you can find your own images and update the lesson files to refer to them. If the lesson provided uses phrases like ‘I’m pleased to meet you’ and you would prefer your learner to use ‘It’s nice to meet you’, you can adjust the lesson accordingly. Customisation does not just end at modifying lesson files. The virtual humans themselves are also defined using XML files, so it is relatively easy to modify their appearance provided you have appropriate 3D models and image files available. There are also customisation controls built into the Thinking Head Whiteboard that allow the learner to adjust the speed of the virtual humans’ speech, toggle screen reading functionality on and off, and other helpful adjustments.

8.3 Technical Features The Thinking Head Whiteboard is a comprehensive educational system with many features including the ability to create multiple accounts to facilitate siblings sharing a home computer, a timer to remind users how much longer they should aim to spend with the software per session, a visual and printable report for caregivers and educators to track how their learner is progressing, encouragement of real-world skills application through ‘homework’ activities, and responsive systems for lesson sequencing, assessment and feedback, and rewards. Existing research has shown that praise, reward or punishment alone has only a minor influence on educational outcomes, and it is instead timely and targeted feedback that benefits learners most (Wisniewski et al. 2020). The automated assessment system therefore has two main functions—within lesson feedback, and assessment on completion. Within each lesson, the software tracks user actions and can provide immediate feedback where appropriate. For example, in a lesson where a topic is being introduced the system might provide feedback any time an incorrect move is made, such as when an item is sorted into the wrong category. For an end of topic task, the system might only provide hints and feedback when explicitly requested by the learner. The feedback also starts out more general and becomes more detailed if the learner makes repeated mistakes. These behaviours are configured partly within the lesson file and partly in the backend code defining the activity type, since the nature of appropriate feedback varies greatly depending on the task at hand. When a learner exits a lesson, either on completion or by choice, the automated assessment system calculates the percentage correctness and percentage accuracy that the learner achieved for that lesson overall. Again, exactly how this is done depends on the type of lesson the user is participating in. The system then updates the overall correctness and accuracy values for the objectives and topics that the lesson belongs to. This information then feeds into the lesson sequencing system.

80

8 The Thinking Head Whiteboard

As previously mentioned, lesson and curriculum XML files provide not only educational content but also constraints for lesson order that ensure appropriate scaffolding of concepts takes place. The lesson sequencing system pulls all these constraints together, compares it with the learner’s current progress and past activity, and presents three lessons for the learner to choose from for their next task. These lessons can be newly unlocked tasks, tasks they have attempted before but not completed, or, in cases where additional practice is needed, lessons drawn from the pool of ‘extra’ content. Each time the learner exits their current lesson, the automated assessment system updates the data, and the lesson sequencing system uses this to present new lesson options to the learner. This assessment information is also fed into the rewards system. The three-tier rewards system not only provides extrinsic motivation for learners to persist with and return to the software each day but also helps students recognise and appreciate the progress they are making. In the first tier of the rewards system, learners earn a ‘gold star’ for each lesson they complete, allowing them to easily see their progress. When students have earned five stars, they can trade them in for a virtual sticker of their choice from the assortment of collections available. For many people collecting items can be quite motivating (Carey, 2008), so this tier is intended as a quick reward without taking too much focus away from the educational content. In the third and final tier of the rewards system, games reinforcing the educational concepts being taught are unlocked at both 50% and 100% completion of each topic. These games are more substantial rewards, reflective of the effort and perseverance learners have displayed to unlock them. For a detailed discussion of the implementation of the lesson sequencing, automated assessment, and rewards systems, please see Milne et al. (2018).

8.4 A Typical Learner Workflow The Social Tutor described here is intended to be used for only ten to fifteen minutes per day, at least three times per week. The goal of this is to reinforce ‘basic’ social skills concepts so that any one-on-one or group therapy intervention time can be spent focused on more nuanced and complicated concepts and skills. The software is therefore also intended to be intuitive and easy to use, and as hands-off as possible for parents. A user flow diagram depicting the learner workflow for a typical session with the software is shown in Fig. 8.2. As can be seen from the diagram, users are expected to complete several lessons during each 15-min session with the tutoring software. The user can exit a lesson at any stage, but if they have not completed enough content or to a high enough accuracy, they may need to repeat the lesson later to progress to the next objective or topic. Users can also return to the topic selection screen at any time, although it is recommended to work through a single topic to maintain momentum and focus. While only three lesson options are provided for the learner to select from at once, learners

8.4 A Typical Learner Workflow

81

Fig. 8.2 A typical learner workflow for a single ‘Social Tutor’ session

can see all previously attempted lessons through the ‘Repeat Previous Lesson’ option in the File menu and can choose to repeat these activities.

8.5 Summary The Thinking Head Whiteboard was developed to investigate the use of virtual humans for teaching social skills to school aged children with autism. The system has a range of features including multi-user support, an automated assessment and feedback system, dynamic lesson sequencing, a three-tier rewards system and support for custom lesson authoring by non-programmers. Next, we discuss the lessons learned from the system’s development and subsequent evaluation.

Chapter 9

Evaluating the Social Tutor

Reviews by Rao et al. (2008) and Neely et al. (2016) identified a range of recommendations to improve the experimental methods used to evaluate social skills interventions. These include but are not limited to inclusion of control groups, sample sizes of over ten participants, use of blinded observer ratings, longitudinal data collection, and explicitly including measurement of generalisation and maintenance of skills in the methodology. Wherever possible, these recommendations have been followed throughout the evaluation of the Social Tutor described in the previous chapter.

9.1 Research Methodology Following the recommendations by Rao et al. (2008), the research design for evaluating the Social Tutor included an experimental group who received ‘social content’ lessons and a control group who used identical software but instead received ‘maze content’ lessons. Pre-test data was collected immediately before participants started using the software, at the conclusion of the software use period, and then again both two and four months later. This study was approved by the Flinders University Social and Behavioural Research Ethics Committee (approval number: 5703). Participants and their caregivers provided informed consent prior to participating.

9.2 Aims The specific aims the software evaluation were to determine whether any changes in knowledge occurred due to use of the Social Tutor, whether any changes in behaviour occurred due to use of the Social Tutor, and if any changes occurred, whether these were maintained after software use ended. © The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_9

83

84

9 Evaluating the Social Tutor

9.3 Participants Participants were between 6 and 12 years old, attended mainstream school, and had an existing diagnosis of Asperger Syndrome (AS) or high functioning autism (HFA) under DSM IV (American Psychiatric Association 2000), or of autism requiring Level 1 support under DSM V (American Psychiatric Association 2013). This combination of inclusion requirements was intended to ensure that the content presented was relevant to participants’ needs, was appropriately matched to their communication and technical skills, and that the participant group was homogenous enough for the data collected to be informative. Three pairs of siblings were recruited, with two sets in the experimental group and one in the control group. Participants were allocated to either the experimental or control condition using a matched-pairs process according to three age ‘buckets’ of 6–8 years, 9–10 years, and 11–12 years. The first participant was allocated to the experimental condition. The next participant was allocated to the control group if they fell into the same ‘bucket’ as the previous participant, or the experimental group if they fell into a different ‘bucket’. This was repeated for all participants. The only exception was in the case of siblings, where all siblings were allocated into the same bucket to avoid issues of jealousy and issues where the blinded observer (i.e. the caregiver) could determine which group their children had been allocated to.

9.4 Measures Given the intention of minimising the burden on families, the selection of measurement tools favoured those that could be completed electronically and independently by the participant and caregiver. However, to ensure an accurate picture of the impact of the Social Tutor software could be gleaned, a multifaceted data collection approach was taken. As generalisation to novel contexts is a known difficulty for individuals with autism, ‘near transfer’ of skills was measured using the in-software content quiz, while ‘far transfer’ of skills to real-world situations was measured using the VinelandII behavioural assessment. Additionally, to provide insight into participant interaction with the software and how much content was covered, the software itself continuously collected interaction data. A known issue with children on the autism spectrum is that they can learn how to ‘do the intervention’ without applying what they are learning to situations outside of the intervention context. Comparison of results from the Vineland-II and content quiz provides insight into whether knowledge gained from using the software is likely to have transferred to behaviour changes outside of the intervention setting, and investigation of log data provides an indication of which components in the software had the most impact or require change.

9.4 Measures

85

9.4.1 Vineland-II The Vineland Adaptive Behaviour Scales, Second Edition (Vineland-II) was selected as the behavioural assessment most appropriate for assessing ‘far transfer’ generalisation of skills to real-world scenarios in the current study due to its fit to the audience and its system of domains and subdomains (Sparrow et al. 2005b, 2005a). These domains provide a level of detail sufficient to allow for detection of subtle behaviour changes in the target population and enable administration of only the areas relevant to the current study, reducing the burden on caregivers and participants. In the current study, the sections included were the Receptive and Expressive subdomains within the broader Communication domain, all of the Socialization domain, and all of the Maladaptive Behaviours domain. The Written subdomain of the Communication domain and the entire Motor Skills domain have been omitted as these areas were not addressed in any way by the content of the Social Tutor. Caregivers were asked to complete the Vineland-II at pre-test and all three posttest data collection points. The Vineland–II was presented as a Google Form, with each statement displayed with a set of multiple-choice radio button answers. At the end of each page a text box was provided so caregivers could clarify or provide detail on their answer for a particular question if they wished. Following data collection, raw scores for the included domains were converted to v–scale scores according to standard Vineland procedure.

9.4.2 Content Quiz The content quiz was designed to directly measure basic and procedural knowledge (Shute and Towle 2003) and consists of four short activities drawn from each topic within the experimental group curriculum, the three topics being greeting, listening and turn-taking, and starting and ending conversations, totalling twelve activities all together. The twelve activities were presented within the software itself and displayed in random order each time the content quiz was run. The content quiz activities were assessed automatically by the software, which then provided a percentage correctness and percentage accuracy score for each question, along with timestamps that could be used to determine the time taken to complete each of the twelve activities.

9.4.3 Software Log Data In addition to the data collection tools already discussed, the Social Tutor software itself continually logged user interactions with the system and saved user progress

86

9 Evaluating the Social Tutor

across learning tasks, providing information about the number of lessons that participants completed, the average amount of time spent on these lessons, and which topic areas and lessons they attempted.

9.5 Intervention Groups The backend software used for both the experimental and control groups was identical, only the lesson content used by the groups was different. This ensured that the experience of all participants was as identical as possible. Experimental group: content varies between lessons, but each set of lessons within a ‘topic’ follows a standard pattern where initial tasks explain and demonstrate the skill, while later tasks encourage learners to identify and apply the skill steps in a variety of contexts. Activity types include drag and drop and Venn Diagram sorting activities, videos and songs, Social StoryTM style activities, virtual roleplays, basic speech recognition tasks, concept maps, and more. Control group: content consists entirely of mazes. The mazes are still grouped into ‘topics’, with the mazes fitting the topic theme. An example from the ‘Spooky and Kooky’ topic can be seen in Fig. 9.1. Mazes increase in difficulty and length as the user progresses, from easy single page mazes through to extra hard mazes with more obstacles, multiple pages, or requiring more than one item per obstacle. To maintain consistency between the intervention groups, homework and reward lessons are also included in both cases. Homework activities for the experimental group encourage learners to apply their skills in the real world using a reflective practice approach, while homework activities for the control group contain no social content and instead ask the student to do tasks like draw pictures, read, or design their own maze. Reward activities likewise are reinforcing for the experimental group and contain no social content for the control group.

9.6 Procedure To reduce the burden on families, the evaluation procedure was designed so that only a single visit to the participants’ home was necessary and all data was collected electronically. A summary of the data collection schedule can be seen in Table 9.1. During the researcher’s visit, a discussion of the research project took place and informed consent was obtained from all individual participants included in the study. Following this, participants were asked to complete the pre-test questionnaire and caregivers to complete the Vineland-II behavioural scale, both electronically. The researcher then installed and tested the software on the family computer, and the participant was provided with a username and password that triggered the correct version of the software to be automatically displayed. The participant was then asked to complete the pre-test content quiz presented by the Social Tutor software and was

9.6 Procedure

87

Fig. 9.1 A control group maze task from the ‘Spooky and Kooky’ themed topic

shown how to navigate and use the software effectively. All pre-test data collection was done with the researcher present to ensure any questions could be answered immediately if required. Following this visit, no more visits from the researcher were compulsory, however families were informed that if they encountered any difficulties with the software or wanted support with data collection, the researcher would be happy to visit and assist them. Families were instructed that, starting the day after the researcher visits, participants should use the software for 10–15 min per day, 3–5 days per week, for 3 weeks. A timer in the software allowed users to self-manage their session times, reducing the burden on caregivers. Three weeks and one day from the researcher visit, the software automatically presented the user with their first post-test content quiz. At the same time, caregivers were asked to repeat the Vineland-II, and both caregivers and participants were asked to complete the post-test questionnaire addressing their experiences and recommendations. At this point the software automatically locked down, and users could not access lesson activities until the study was over. No action was required from participants or their caregivers for the next two months.

88

9 Evaluating the Social Tutor

Table 9.1 Data collection schedule Pre-Test

Software use

Immediate post-test

2 Month follow up post-test

4 Month follow up post-test

Timeline

Day 1

Day 2–22 Ongoing for 3 weeks

Day 23 Day after software use ends

Day 83 2 months after post-test

Day 143 2 months after second post-test

Researcher visit

Yes

No

On request

On request

On request

Caregiver actions

Complete Vineland-II online

Support participant to use software if required

Complete questionnaire & Vineland-II online

Complete Vineland-II online

Complete Vineland-II online

Participant actions

Complete content quiz via software, questionnaire online

Software use 10–15 min a day, 3–5 times a week

Complete content quiz via software, questionnaire online

Complete Complete content quiz content quiz via software } via software }

Researcher actions

Install software Support caregiver to complete Vineland-II, participant to complete quiz and questionnaire

Contact caregiver to remind them to complete assessments

Contact caregiver to remind them to complete assessments

Contact caregiver to remind them to complete assessments. When all data received, provide unlock code, reimburse

} unforeseen circumstances necessitated some participants completing the content quiz at two and four month follow up via Word document

Two months later caregivers were emailed and asked to complete the VinelandII again and to have their participants log into the software and complete the next content quiz. Following this was another two-month break, after which caregivers were once again emailed and reminded to complete their final Vineland-II and have their participants complete a fourth and final content quiz. On completion of the content quiz, the software automatically unlocked the user’s account, and their lesson activities became accessible once again. Once all data was complete and received by the researcher, the family was reimbursed $30 for their participation and were provided with the necessary login details and instructions for them to access the version of the software that they were not initially assigned to.

9.7 Results

89

9.7 Results A total cohort of thirty-one children participated, with sixteen in the experimental group (M = 8.81 years, SD = 1.83) and fifteen in the control group (M = 9.20 years, SD = 2.08). An independent samples t-test found no significant difference in mean age between the two groups (p = 0.497, r = 0.10), and a 2-sample t–test for equality of proportions found no significant difference in gender ratio between the two groups, χ2 (1, N = 31) = 0.51, p = 0.47, indicating that the groups were sufficiently balanced according to both age and gender.

9.8 Changes in Knowledge As previously reported in Milne et al. (2018), analysis of the content quiz data indicated that use of the Social Tutor led to a mean improvement in social skills knowledge for participants in the experimental group (M = 7.36%, SD = 9.05, 95% CI [2.54, 12.19]), with paired t-test with False Discovery Rate (FDR) correction indicating statistical significance (p = 0.01, d = –0.73). However, those in the control group displayed a much lower mean improvement (M = 1.14%, SD = 7.16, 95% CI [–3.41, 5.69]), with paired t-test showing this to be statistically insignificant (p = 0.799, d = 0.08). Post hoc analysis also revealed the presence of response subgroups, with participants grouped as either high (n = 4), average (n = 8) or low (n = 4) responding. Participants identified as ‘high responding’ appear to have made higher gains in social skills knowledge from using the Social Tutor software (Milne et al. 2018).

9.9 Changes in Behaviour Twenty-nine caregivers completed the Vineland-II at both the pre-test and immediate post-test, including 93.8% of those in the experimental group (N = 15, M = 8.93 years, SD = 1.83, 95% CI [7.74, 9.83]) and 93.3% of those in the control group (N = 14, M = 9.21 years, SD = 2.15, 95% CI [7.97, 10.45]). To ensure accuracy and consistency, a computer program was written to calculate the raw score for each subdomain according to the procedure in the Vineland-II Survey Forms Manual (Sparrow et al. 2005b). Following this procedure, the v-scale score for each subdomain was then manually obtained from the provided tables. The next step in the procedure is to calculate a standard score for both the Communication and Socialization domains overall, however for the Communication domain not all subdomains were administered. To allow an approximate standard score for the Communication domain to be obtained for each participant, the mean score for

90

9 Evaluating the Social Tutor

the omitted Written subdomain was substituted in, as per Table 8.15 of the VinelandII Survey Forms Manual (Sparrow et al. 2005b). This was deemed an acceptable approach given that the purpose of administering the Vineland-II was comparison across time points rather than standalone diagnosis. From here, standard procedure was followed to calculate a standard score for both the Communication and Socialization domains, then percentile ranks and adaptive levels were obtained from the tables provided in the Vineland-II manual. Change from pre-test to post-test was calculated for each participant and Wilcoxon rank sum tests were used to compare the control group with the experimental groups, then the pre-test and post-test scores for each group were compared using Wilcoxon signed rank tests, however no domain or subdomain reached significance for comparison between groups or between pre-test and post-test. These tests were repeated using the Vineland-II raw scores to confirm that the scaling procedure had not inadvertently masked any behavioural changes, and again no significant findings were uncovered. The data showed that both the groups performed similarly overall, with both achieving a small positive change from pre-test to post-test in most domains and subdomains. The experimental group outperformed the control group on the Socialization domain overall, the Receptive subdomain of the Communication domain, and the Play and Leisure Time subdomain of the Socialization domain. The data was further broken down into response subgroups, and from Table 9.2 a positive trend can be observed whereby the high responding subgroup (n = 4) outperforms the other subgroups on several domains and subdomains, and the average responding group (n = 8) likewise outperforms the low responding group (n = 4). What is particularly encouraging is the alignment with the content quiz results previously discussed, which may indicate that with more time these improvements in knowledge could also manifest as improvements in target behaviours. Table 9.2 Summary of Vineland-II domain change from pre-test to post-test by response subgroups

High (n = 4)

Average (n = 8)

Low (n = 4)

Communication

Socialization

Combined

v-Sum

v-Sum

v-Sum

Standard

Standard

Standard

M (SD)

1.00 (1.41) 1.50 (2.38) 3.25 (5.44)

6.00 (9.93) 4.25 (6.85)

7.50 (12.29)

95% CI

−1.25, 3.25

−2.29, 5.29

−5.40, 11.90

−9.81, 21.81

−12.05, 27.05

M (SD)

−0.29 (2.06)

−0.57 (3.82)

2.29 (4.03)

4.00 (7.12) 2.00 (4.62)

3.43 (8.04)

95% CI

−2.19, 1.62

−4.11, 2.96

−1.44, 6.01

−2.58, 10.58

−2.27, 6.27

−4.01, 10.86

M (SD)

0.25 (2.22) 0.50 (3.70) −0.25 (1.26)

−0.50 (1.73)

0.00 (2.94)

0.00 (4.69)

95% CI

−3.28, 3.78

−3.26, 2.26

−4.68, 4.68

−7.46, 7.46

−5.38, 6.38

Note Positive values indicate better performance

−2.25, 1.75

−6.65, 15.15

9.9 Changes in Behaviour

91

Table 9.3 Summary of participant data where a change in adaptive level occurred from pre- to post-test Domain

Group

Pre-test level

Post-test level

Communication

Experimental

Low (mild deficit)

Moderately low

+3

Low (mild deficit)

Moderately low

+2

Socialization

Change in standard score

Adequate

Moderately low

−8

Control

Moderately low

Adequate

+4

Low (mild deficit)

Moderately low

Experimental

Moderately low

Adequate

+20

Low (mild deficit)

Moderately low

+12

Moderately low

Adequate

+11

Low (mild deficit)

Moderately low

+7

Adequate

Moderately low

−7

Low (mild deficit)

Moderately low

+7

Low (mild deficit)

Moderately low

+2

Adequate

Moderately low

−5

Control

+4

Next, an analysis was conducted on participants’ overall adaptive levels of functioning for both the Communication and Socialization domains. The lowest level of adaptive functioning obtained was ‘mild deficit’ and the highest was ‘adequate’ performance. This range indicates that the selection criteria for participants was appropriate and included individuals who were high functioning in the context of autism, with nothing more severe than a ‘mild deficit’ obtained, but also low enough functioning in these domains for the software to have the potential to benefit them, with nothing above ‘adequate’ functioning detected. As can be seen in Table 9.3, more experimental group participants than control group participants increased their adaptive level of functioning in the Socialization domain over the intervention period, with the improvements made by the experimental group typically being larger in magnitude. Again, this positive trend in the target domain of Socialization is encouraging, although it must be interpreted cautiously given the lack of statistical significance.

9.10 Maintenance of Skills The change in content quiz correctness scores between the two and four-month follow up tests and the pre-test and immediate post-tests were calculated for both groups, as can be seen in Fig. 9.2. Interestingly, a dip in scores at the two-month post-test is present for both groups, however at the four-month post-test all groups and subgroups demonstrate better performance than they did at pre-test. From Fig. 9.2 it appears that use of the Social Tutor may have led to a levelling effect, where participants who

92

9 Evaluating the Social Tutor

Fig. 9.2 Mean correctness at each data collection period by group and subgroup

scored lowest initially displayed marked improvement at the immediate post-test, and by the final four-month follow up were performing at approximately the same level as the other subgroups and control group. Wilcoxon signed rank tests were used to analyse the Vineland-II longitudinal data across the four data collection points, however no significant interactions were found for any domain or subdomain (all p > 0.05), suggesting that participants made no significant change in observable behaviour across the period of the study. However, as can be seen from the graphed results in Fig. 9.3, at the four-month follow up the majority of participants in both groups scored better than they had at pre-test on all three domains, with the experimental group scoring more favourably than the control group on all domains at this time point. Interestingly, the Vineland-II data at the two-month point displays the same brief decline in skills as the content quiz data. From further inspection of the graphs in Fig. 9.3 it can be seen that for all three domains the average scores at each data point are displaying an encouraging trend, with the Communication domain increasing very slightly over time and the Socialization domain increasing more markedly. For the Maladaptive Behaviours domain, a reduction in score reflects a reduction in problematic behaviours, and thus the downward trend observed here is likewise a positive sign. To further investigate these trends the data was divided into the previously identified high, average and low response level subgroups. The data showed similar trends for the control group and low and average responding subgroups, however some differences were noted for the high responding subgroup, as can be seen in Fig. 9.4. It should be noted that this division resulted in especially small sample sizes for the low (n = 4) and high (n = 4) responding subgroups, hence the large standard error bars in Fig. 9.4, and was performed only to investigate whether any trends worth further exploration may exist.

9.10 Maintenance of Skills

93

Fig. 9.3 Mean v-sum scores at each data collection point for the control (n = 15) and experimental (n = 16) groups for the Communication, Socialization and Maladaptive Behaviours domains of the Vineland-II

For the Communication domain the data indicates that the control group (n = 15) outperformed the high responders (n = 4) at every data collection point, however the opposite is true for both the Maladaptive Behaviours and Socialization domains. For Maladaptive Behaviours high responders display a larger decrease in problematic behaviours than the experimental group overall, indicating a larger improvement. Similarly for Socialization, high responders display a much stronger improvement trend than the experimental group as a whole, which is consistent with the findings from analysis of the content quiz (Milne et al. 2018).

94

9 Evaluating the Social Tutor

Fig. 9.4 Mean v-sum scores at each data collection point for high responders (n = 4) versus control group (n = 15) for the communication, socialization and maladaptive behaviours domains of the Vineland-II

9.11 Discussion The research aims addressed in this research were to determine if any changes to knowledge or behaviour of the target social skills occurred due to interaction with the Social Tutor, and to determine if any such changes were maintained after software use ended.

9.12 Changes in Knowledge Outcomes from analysis of content quiz correctness data indicate that use of the Social Tutor has led to an improvement in social skill knowledge for participants in the experimental group, i.e. near transfer generalisation, and post hoc categorisation

9.12 Changes in Knowledge

95

of participants suggests that a subgroup consisting of approximately a quarter of participants (n = 4) responded particularly well to the software, with the shared characteristics of this group potentially informing future research and development.

9.13 Changes in Behaviour Analysis of the Vineland-II data uncovered some promising trends, however given the similarity of performance between the experimental and control group the apparent improvements may have been heavily influenced by factors other than the use of the Social Tutor software, e.g. chance, a placebo effect in the parent-reported data, increasing maturity across the intervention period, or exposure to a range of interventions at home and school. Thus, it appears that ‘far transfer’ generalisation has not occurred at this time. However, given that the intervention period was only three weeks long and that behaviour changes can be difficult and time-consuming to enact, often including the necessity of breaking entrenched patterns, it is promising to see that the experimental group showed a slightly higher improvement in the Socialization domain than the control group. Perhaps even more notable was the ‘levelling effect’ apparent in Fig. 9.2, where the high responding group who were the lowest performers initially appeared to have ‘caught up’ and were performing approximately on par with all other participants at the four-month follow up. The positive, yet not statistically significant, trend observed in Table 9.2 where the high responding subgroup outperforms all other subgroups may indicate that the Social Tutor is a good candidate for inclusion in a hybrid approach to teaching social skills, such as that taken by Beaumont and Sofronoff (2008) in the Junior Detective program or Whalen et al. (2010) in TeachTown: Basics. In these examples the software is paired with a school or group therapy-based role-play component which provides learners with an explicit opportunity to practice the skills they are developing via the software with their real-world peers in a guided manner. Both programs demonstrated good outcomes for learners. It was hoped that inclusion of virtual humans in the Social Tutor software would facilitate some of the same benefits, which may well be the case and become more apparent with a longer intervention period, however inclusion of more natural interaction methods and full-bodied virtual characters may also help to amplify this effect.

9.14 Maintenance of Skills From the content quiz results it was found that the experimental group not only maintained their post-test improvement but continued their upward trend after the software use period had ended, and this was particularly notable for the high responding subgroup. As anticipated, the control group remained stable at all four data collection points. Longitudinal data from the behavioural assessment showed that the control

96

9 Evaluating the Social Tutor

and experimental groups both performed similarly at all four data collection points with a slight improvement over time, however again the high responding subgroup appeared to not only retain but continue building on their improvement in scores beyond the end of the three-week software use period. These results are encouraging and suggest that content learned with the Social Tutor is maintained beyond the intervention period, a known challenge when designing interventions for individuals on the autism spectrum. Analysis of the longitudinal data for the Vineland-II indicates that at four-month follow up the majority of participants in both groups were displaying improvements across all three domains when compared to their pre-test scores, with the experimental group slightly outperforming the control group on both the Communication and Maladaptive Behaviours domains. From the visual representation presented in Fig. 9.3 it can be seen that for the Communication and Maladaptive Behaviours domains the trajectory of scores is similar for the control and experimental groups, and while for the Socialization domain the trajectory is somewhat different, ultimately at four-month follow up both intervention groups performed at approximately the same level, suggesting that the Social Tutor had little impact on Vineland-II scores overall. The longitudinal data from the content quiz further supports the notion that the high responding subgroup benefitted most from the software, with these participants not only improving the most between pre-test and immediate post-test, but at fourmonth follow up continuing to increase their correctness scores beyond that achieved at immediate post-test. As can be seen in Fig. 9.4, for the high responding subgroup only, this positive trend is also present in the longitudinal data from the Vineland-II behavioural assessment. Given this upwards trend in scores even after Social Tutor use has ended, it appears that other factors may also be at play in the current results. A few likely candidates given the time frame are increasing maturity, exposure to social skills interventions beyond the Social Tutor software itself, and ongoing opportunities to practice their new knowledge in real-world situations and consolidate it. It is possible that the knowledge gained from the Social Tutor may have fed into their learning at other interventions and learning opportunities and helped them learn more rapidly, or may have been applied and practised in real-world scenarios, consolidating and reinforcing the theoretical knowledge they gained with the Social Tutor, which was then reflected in higher content quiz correctness scores at follow up.

9.15 Selection of Measurement Tools While the Vineland-II has been used to measure change in social skills of children with autism following human-led interventions on numerous occasions (Laugeson et al. 2012; Koning et al. 2013; Ng et al. 2016), it has not been used to measure the impact of software like the Social Tutor, making comparisons with existing work difficult. Additionally, little existing research using virtual tutors for autism

9.15 Selection of Measurement Tools

97

has progressed beyond initial pilot studies, and often the authors use bespoke measurement tools that are difficult to directly compare. Here the content quiz and Vineland-II were used in tandem to mitigate these challenges. The Vineland-II was chosen because it was thought to be fine-grained enough to detect subtle changes in behaviour, but also for many pragmatic reasons including the ability to administer only the relevant sections, it not being too time-consuming for caregivers to complete, and it being appropriate for administration by the research team given their diverse skill sets. However, it appears that one possible explanation for the lack of difference found between the experimental and control groups could be that the Vineland-II was not sensitive enough to detect behavioural changes in the specific areas targeted by the Social Tutor software. Other recent work using the Vineland-II to measure social skills changes in children with autism following human-led interventions has since been found that also encountered this issue, with Vineland-II results failing to reach significance even when multiple other measures did (Laugeson et al. 2012; Koning et al. 2013; Ng et al. 2016). Given the Vineland-II results in the current study and the results from human-led intervention studies with comparable participant cohorts, it appears that a measurement tool focused on conversation skills specifically, such as the ability to initiate, maintain and terminate conversations, may be more suitable for detecting generalisation to real-world interactions in future work.

9.16 Limitations in Methodology One limitation of the current study is sample size. While thirty-one participants completed the evaluation, approximately half were in the control group and half in the experimental group. Once these groups are further broken down for finegrained exploratory analysis, it is difficult to claim sufficient statistical power to draw meaningful conclusions beyond observing trends. As this was a pilot study and given the necessity for the researcher to physically visit the family to install the software, a small sample size was appropriate. A power analysis with a more accurate indication of effect size can now be conducted, and thus exists an opportunity to run a future evaluation with a more suitable sample size. One possible avenue for both ensuring increased consistency and improving sample size may be to port the Social Tutor to a mobile device platform, such as making it iPad or Android compatible given the popularity of these platforms or provide it in a self-contained executable file for desktop PCs. Given that all data collection was done electronically, this would mean that participant recruitment would no longer be limited to the physical location surrounding the research team. It would also potentially enable the software to be made more widely available and allow its benefits to be enjoyed by families outside of the current study. The evaluation was also relatively short, with the active software use period being only three weeks. Several indicators suggest this was not long enough, for example some initial upwards trends were observed in the Vineland-II results, but not enough data was obtained to know whether these were genuine trends or due to chance.

98

9 Evaluating the Social Tutor

Alternatively, the scope of the curriculum may be insufficient to lead to the broad change in social functioning required for detection by the Vineland-II. In terms of the measures used, the content quiz was identified as too limited in its possible answers and the Vineland-II as not being sensitive enough, thus for future research these measures should be refined or replaced as appropriate.

Chapter 10

The Future of Virtual Teachers

Technology is continually evolving, and this presents us with new and exciting options for incorporating more realistic learning activities and providing an increasingly dynamic and personalised experience in future virtual teaching systems. The following discussion is strongly informed by the evaluation of the Social Tutor system described in the previous chapter. Some of the factors discussed here emerged directly from user feedback, while others were recognized during development and testing as limiting the potential of the intervention. For example, simply using better quality synthetic voices reduces difficulties with auditory processing and thus makes the software more accessible, while swapping to full-bodied virtual characters has the potential for increasing the educational value of the software, as it facilitates learning about body language and complex gestures. Along with this, speech recognition and natural language processing technology have improved, as have gesture recognition and emotion recognition technologies, and the area of ‘gamification’ in education has likewise boomed. All of these have the potential to increase user engagement and educational outcomes for future virtual human based teaching software.

10.1 Natural Interaction One way to unlock more educational potential relatively easily would be to implement full-bodied virtual characters in place of the current head-and-shoulders-only models. This would allow for more authentic modelling and explanation of nonverbal communication, such as hand gestures and body language, which would be highly beneficial to the target learner group. Complementing this, gesture recognition is not only improving but the hardware required to access it is more widely available, with cameras built in on most mobile devices and laptops. Thus, the virtual characters could model body language more accurately, and they could recognise and respond to the learner’s own gestures as well.

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7_10

99

100

10 The Future of Virtual Teachers

In human-human interaction, nonverbal behaviour has several functions which may also be helpful in a virtual learning environment. For example, teachers model tasks for students, use illustrative gestures such as pointing, use gestures to emphasise important points and guide learner focus, as well as engage in dialogue management and turn-taking cues (Allmendinger 2010; Krämer and Bente 2010). It has been shown that smiling and other feedback cues affect student interest, motivation and learning outcomes, for example encouraging students to continue down a particular train of thought by smiling and nodding assists them to know they are progressing well (Allmendinger 2010; Krämer and Bente 2010). However, it is imperative that nonverbal behaviours appear sufficiently natural, as in human-human interaction they are processed automatically by the limbic system, and this may fail to occur if the behaviours appear odd (Krämer and Bente 2010). It is unknown if these nonverbal cues will impact the target population of the current study given that difficulties with social interaction form a core part of a diagnosis of autism, however the benefits of these technologies in explicitly teaching how to interpret and display body language appropriately remain. Along with gesture-based interaction, social communication with a virtual human should also include speech interaction. Speech recognition and the natural language processing associated with a robust conversational system such as Siri or Alexa have both improved greatly in recent years and continue to do so. This is largely due to these systems’ ability to collect and learn from vast quantities of data, and with processing being done in the cloud rather than locally, the storage capacity and processing capability of the local device is no longer a limiting factor (Aron 2011; Guzman 2016). If these technologies continue to improve and become more accessible, they present many exciting opportunities for authentic interaction and practice of social skills within virtual tutoring software. Being able to speak with the virtual characters instead of merely pressing buttons, and having them respond to natural gestures like waving hello, raises the interaction to a level that is much closer to humanhuman interaction, reducing the need for learners to rely on their reading skills and potentially making it more likely for skills practiced in the software context to be generalised to real-world situations. Of course, care must still be taken to ensure that inappropriate social behaviours are not inadvertently reinforced, such as having an unfriendly hand gesture interpreted as waving hello and encouraged, so there are many challenges in implementing this level of natural human-computer interaction in this context even once the technology itself is robust.

10.2 Emotional Response and Authenticity Students experience a wide range of emotions during learning, from confusion, frustration and boredom, to satisfaction, enthusiasm and excitement. Typically students who are anxious, angry or depressed do not retain information effectively or perform well in learning tasks, so it is the role of the teacher to guide learners through these

10.2 Emotional Response and Authenticity

101

states and into affective states more conducive to learning (Kort et al. 2001; Storbeck et al. 2015). Detecting boredom could be a cue to fast track the student through work that is too easy for them, while detecting frustration could indicate they need additional support or redirecting back to more fundamental activities. Krämer (2006) suggests that implementing a ‘theory of mind’ for the virtual characters may go a long way towards improving their likeability, while also helping the software to make better judgements of the learner’s current state and needs. Existing work has shown that when a virtual character can empathise with the user, the user becomes more interested and displays higher self-efficacy. To do this effectively, the virtual human must be able to detect the learner’s emotion and respond appropriately (Krämer and Bente 2010). As discussed previously, learning is intertwined with emotion, so being able to respond to learner emotion authentically appears to hold promise for improving educational outcomes. Emotion detection also has potential both for increasing rapport between the user and virtual characters, with the virtual characters being able to comment on and respond to the user’s expressions, and for developing unique interactive learning activities. Recent research has seen the development and validation of a number of computer vision based approaches to engagement and emotion detection that appear suitable for future incorporation in the Social Tutor provided a camera is made available, for example see Grafsgaard et al. (2013), Whitehill et al. (2014) and Monkaresi et al. (2017). Activities could cover understanding one’s own emotions, as well as detecting and responding to the emotions of others. Coupled with a mobile device, this could present some very interesting learning opportunities for users to explore the world around them and the people in it.

10.3 Reflective Practice Meta-cognitive skills and reflective practice, such as self-explanations, have been demonstrated to lead to better problem-solving skills and the construction of deeper, more meaningful conceptual connections (Amico et al. 2015; Mitrovic 2001; Nicholas et al. 2015). Mitrovic (2001) conducted a study with university level computer science students to evaluate their self-assessment capabilities. It was found that more able students displayed better understanding of their own educational needs, while less able students abandoned many more practice questions, often citing that the problem was too easy even when evidence suggested otherwise. This suggests that a system that prompts students to consider more carefully the reasons for their difficulties may help to nurture meta-cognitive skills and improve educational outcomes. Black and Wiliam (2009) also emphasise the importance of reflective practice for deep and long-term learning. They suggest that reflection can assist students to make the processes they unconsciously use explicit and concrete, making them easier to understand and implement in future. It is suggested that discussion with peers and others improves the outcomes of reflective practice, in following with

102

10 The Future of Virtual Teachers

Vygotsky’s principle that ideas are initially constructed in social interactions, and then internalised by the learner. Additionally, challenging students to identify other situations where they can use the same thinking processes, to compare and contrast ideas, and to critically analyse their ideas, can help learners improve their problemsolving and cognitive skills and learn to apply their skills elsewhere. While social learning may appear in conflict with the development of a social tutoring program to be used individually, the virtual agent can play the role of a peer and activate these same learning gains. Meyer and Land (2010) recommend speak aloud self-explanations as a reflective practice, which is an approach that could be easily implemented in this context.

10.4 Intelligent Student Model Moving away from methods of customising content and the virtual characters themselves, another way to personalise the user learning experience is to introduce adaptive student models. Wittwer et al. (2010) emphasise the benefits of adapting instruction to the individual learner, including encouraging researchers to consider implementing detection of nonverbal cues. In a virtual tutoring system, adaptive student models could be provided with initial information when the user creates an account, but then be continually fed data about the individual user’s interaction with the system. Inspired by the suggestions of Sansosti (2010), an input mechanism could allow educators and caregivers to provide an initial overview of their learner’s abilities, then learners could periodically be prompted to complete a mini quiz, with strong performance in the quiz resulting in fast tracking the user through their current material and poor performance resulting in the provision of additional support material, much like the approach taken by Jones et al. (2016). These approaches allow students to skip content they have already demonstrated mastery for in other contexts, the initial information combined with ongoing log and quiz data can then be used to determine user preferences and areas of strength and weakness, and from this a more dynamic and targeted user experience is provided. For example, different lesson activities could be offered depending on whether the user is a more visual or more aural learner, or if signs of frustration were building up the software could backtrack and offer simpler lessons or prerequisite lessons as a refresher, before offering the challenging content again. In combination with intermittent ‘pop quizzes’, the learner could also be directly asked about their affective state. Robison et al. (2009) compared the use of task-based and affect-based feedback in an exploratory narrative-based learning environment. When learners interacted with agents in the environment, they were prompted for a self-report of affective state. The agent then provided either a task- or affect-based response, such as a hint or empathy. The user was then prompted with ‘… and you respond’ and used a Likert scale to evaluate the effectiveness and appropriateness of the agent’s response. The approach which combined student characteristics, affect, and situational data was found to be most effective, making accurate predictions

10.4 Intelligent Student Model

103

about appropriate feedback 96% of the time. Including a model such as this in social skills tutoring software may assist to maximise both motivation and learning gains. However, adaptation may be needed given that individuals with autism can have difficulty deciphering their own affective states and this model relies heavily on self-reports. Combining techniques, for example adding affective cues extracted from speech and visual data to the decision-making process, could assist to increase robustness. Case Study: Social Tutor for Autism The Social Tutor discussed previously includes two-part social action-based ‘homework’ tasks where the first part asks students to plan their homework (e.g. to practice greeting someone, who will you greet, where and when), and the second part involves reflecting on how they went and why. These tasks are optional, and in future software iterations more regular prompts for selfreflection within standard learning tasks could also be included to increase the likelihood of improving learners’ meta-cognitive skill set.

10.5 Game-Based Learning and Collaboration Gamification is rapidly becoming not only an accepted feature but even an expected feature in educational software. Hamari et al. (2014) reviewed gamification literature and found that in an educational context inclusion of game-like aspects or embedding the learning within a game can, when done mindfully, lead to increased motivation, engagement and enjoyment. Research indicates that positive emotions can reduce cognitive effort and increase working memory (Storbeck et al. 2015) and thus providing students with learning opportunities that are inherently pleasant can support strong retention. Collaborative story telling has already been shown to lead to improvements in social skills for children with autism (Tartaro and Cassell 2008). The concept of collaboratively learning with the virtual humans rather than simply being taught and supported by them is an interesting one and may also present unique opportunities for improving educational outcomes in an engaging way. Providing social tutoring software as an app for mobile devices rather than desktop computers could also present opportunities for collaborative learning with human peers, for example by utilising the touch screen as a shared surface for simultaneous users in a similar fashion to the approach used in the SIDES tabletop game (Piper et al. 2006). Many games are social by nature, and therefore can provide a base for social skills education that is both engaging and directly relevant to our audience’s interests.

104

10 The Future of Virtual Teachers

10.6 Summary As the state-of-the-art continues to improve, additional human-computer interaction technologies such as gesture, speech and emotion detection and an adaptive student model could improve the realism of interaction within social skills tutoring systems, making it possible to truly harness the power of peer modelling and human-human tutoring to support learners with autism to improve their social skills.

Bibliography

Abirached B, Zhang Y, Aggarwal JK, Tamersoy B, Fernandes T, Carlos J (2011) Improving communication skills of children with ASDs through interaction with virtual characters. In: 2011 IEEE 1st international conference on, serious games and applications for health (SeGAH). Alelo (2018) Alelo. Play. Learn. Communicate. Alelo. Retrieved April from https://www.alelo.com/ products-and-solutions/ Allmendinger K (2010) Social Presence in synchronous virtual learning situations: the role of nonverbal signals displayed by Avatars. Educa Psychol Rev 22(1):41–56 American Psychiatric Association (2000) Diagnostic and statistical manual of mental disorders: DSM-IV-TR (4th ed, text revision ed ed.) American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders: DSM-5 (5th ed.) Amico M, Lalonde C, Snow S (2015) Evaluating the efficacy of drama therapy in teaching social skills to children with autism spectrum disorders. Drama Ther Rev 1(1):21–39. https://doi.org/ 10.1386/dtr.1.1.21_1 Aron J (2011) How innovative is Apple’s new voice assistant, Siri? New Sci 212(2836), 24. https:// doi.org/10.1016/S0262-4079(11)62647-X Balakrishnan S, Alias A (2017) Usage of social stories in encouraging social interaction of children with autism spectrum disorder. J ICSAR, 1(2) Baron-Cohen S, Golan O, Ashwin E (2009) Can emotion recognition be taught to children with autism spectrum conditions? Philosoph Trans R Soc b: Biolog Sci 364(1535):3567–3574. https:// doi.org/10.1098/rstb.2009.0191 Baylor AL, Kim Y (2004) Pedagogical agent design: the impact of agent realism, gender, ethnicity, and instructional role. Intelligent Tutoring Systems Beaumont R, Sofronoff K (2008) A multi-component social skills intervention for children with asperger syndrome: the junior detective training program. J Child Psychol Psychiatry 49(7):743– 753. https://doi.org/10.1111/j.1469-7610.2008.01920.x Bellack A (1983) Recurrent problems in the behavioral assessment of social skill. Behav Res Ther 21(1):29–41 Bergmann K, Macedonia M (2013) A virtual agent as vocabulary trainer: iconic gestures help to improve learners’ memory performance. In: Aylett R, Krenn B, Pelachaud C, Shimodaira H (eds) Intelligent virtual agents, vol 8108. Springer Berlin Heidelberg, pp 139–148. https://doi.org/10. 1007/978-3-642-40415-3_12 Bernardini S, Porayska-Pomsta K, Smith TJ (2014) ECHOES: an intelligent serious game for fostering social communication in children with autism. Inf Sci 264:41–60. https://doi.org/10. 1016/j.ins.2013.10.027

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 M. Bond et al., Teaching Skills with Virtual Humans, Cognitive Science and Technology, https://doi.org/10.1007/978-981-16-2312-7

105

106

Bibliography

Bickmore TW, Pfeifer LM, Byron D, Forsythe S, Henault LE, Jack BW, Silliman R, PaascheOrlow MK (2010) Usability of conversational agents by patients with inadequate health literacy: evidence from two clinical trials. J Health Commun 15(S2):197–210 Bickmore TW, Silliman RA, Nelson K, Cheng DM, Winter M, Henault L, Paasche-Orlow MK (2013) A randomized controlled trial of an automated exercise coach for older adults. J Am Geriatr Soc 61(10):1676–1683. https://doi.org/10.1111/jgs.12449 Biswas G, Roscoe R, Jeong H, Sulcer B (2009) Promoting self-regulated learning skills in agentbased learning environments. Black P (2015) Assessment: friend or foe of pedagogy and learning. Past as Prologue 235 Black P, Wiliam D (2009) Developing the theory of formative assessment. Educ Assess Eval Accountability 21(1):5–31 Blair K, Schwartz D, Biswas G, Leelawong K (2007) Pedagogical agents for learning by teaching: teachable agents. Educ Technol 47(1):56 Bock M, Rogers MF, Myles BS (2001) Using social stories and comic strip conversations to interpret social situations for an adolescent with asperger syndrome. Interv School Clinic 36(5):310–313. https://doi.org/10.1177/105345120103600510 Borjigin A, Miao C, Lim SF, Li S, Shen Z (2015) Teachable agents with intrinsic motivation. In: Conati C, Heffernan N, Mitrovic A, Verdejo MF (eds) Artificial intelligence in education: 17th international conference, AIED 2015, Madrid, Spain, June 22–26, 2015. Proceedings. Springer International Publishing, pp 34–43. https://doi.org/10.1007/978-3-319-19773-9_4 Bosseler A, Massaro D (2003) Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with Autism. J Autism Dev Disorder 33(6). https://home. cc.gatech.edu/autism/uploads/44/bosselermassaro_2003.pdf Bowman-Perrott L, Davis H, Vannest K, Williams L, Greenwood C, Parker R (2013) Academic benefits of peer tutoring: a meta-analytic review of single-case research. School Psychol Rev 42(1):39–55 Brown DJ, Standen PJ, Proctor T, Sterland D (2001) Advanced Design methodologies for the production of virtual learning environments for use by people with learning disabilities. Presence: Teleoper Virtual Environ 10(4):401–415. https://doi.org/10.1162/1054746011470253 Butler LB, Poedubicky V (2006) Social decision making/social problem solving. The educator’s guide to emotional intelligence and academic achievement: social-emotional learning in the classroom, 131 Carey C (2008) Modeling collecting behavior: the role of set completion. J Econom Psychol 29(3):336–347. https://doi.org/10.1016/j.joep.2007.08.002 Chen C-H, Lee IJ, Lin L-Y. (2015) Augmented reality-based self-facial modeling to promote the emotional expression and social skills of adolescents with autism spectrum disorders. Res Dev Dis 36:396–403. https://doi.org/10.1016/j.ridd.2014.10.015 Cheng Y, Huang C-L, Yang C-S (2015) Using a 3D immersive virtual environment system to enhance social understanding and social skills for children with Autism spectrum disorders. Focus on Autism and Other Dev Disab 30(4):222–236. https://doi.org/10.1177/108835761558 3473 Chi M, Siler S, Jeong H, Yamauchi T, Hausmann R (2001) Learning from human tutoring. Cogn Sci 25(4):471–533 Chi MTH, Wylie R (2014) The ICAP framework: linking cognitive engagement to active learning outcomes. Educ Psychol 49(4):219–243. https://doi.org/10.1080/00461520.2014.965823 Chiang H-H, Chen W-M, Chao H-C, Tsai D-L (2018) A virtual tutor movement learning system in eLearning. Multimedia Tools Appl. https://doi.org/10.1007/s11042-018-5922-8 Clark RE, Choi S (2005) Five design principles for experiments on the effects of animated pedagogical agents. J Educ Comput Research 32(3):209–225 Cline B, Brewster C, Fell R (2010) A rule-based system for automatically evaluating student concept maps. Expert Syst Appl 37(3):2282–2291 Conati C (2002) Probabilistic assessment of users emotions in educational games. Appl Artific Intell 16(7):555–575

Bibliography

107

Conati C, Manske M (2009) Eval Adapt Feedback in an educational computer game Cooper J, Goodfellow H, Muhlheim, E, Paske K, Pearson L (2003) PALS social skills program: Playing and Learning to Socialise: Resource Book. Inscript Publishing Crockett K, Latham A, Mclean D, Bandar Z, O’Shea J (2011) On predicting learning styles in conversational intelligent tutoring systems using fuzzy classification trees. In: 2011 ieee international conference on, fuzzy systems (FUZZ) Crockett K, Latham A, Whitton N (2017) On predicting learning styles in conversational intelligent tutoring systems using fuzzy decision trees. Int J Human-Comput Studies 97:98–115. https://doi. org/10.1016/j.ijhcs.2016.08.005 Cromby J, Standen P, Newman J, Tasker H (1996) Successful transfer to the real world of skills practised in a virtual environment by students with severe learning difficulties. In: Proceedings 1st European conference on disability, virtual reality and associated technologies, Maidenhead, UK, Crook C, Sutherland R (2017) Technology and theories of learning. In: Duval E, Sharples M, Sutherland R (eds) Technology enhanced learning: research themes. Springer International Publishing, pp. 11–27. https://doi.org/10.1007/978-3-319-02600-8_2 Davis M, Robins B, Dautenhahn K, Nehaniv C, Powell S (2005) A Comparison of interactive and robotic systems in therapy and education for children with autism. In: Pruski A, Knops H (eds) Assistive technology: from virtuality to reality. IOS Press, pp 5 de Klerk S, Eggen TJHM, Veldkamp BP (2016) A methodology for applying students’ interactive task performance scores from a multimedia-based performance assessment in a Bayesian Network. Comput Human Behav 60:264–279. https://doi.org/10.1016/j.chb.2016.02.071 Delano M, Snell ME (2006) The effects of social stories on the social engagement of children with autism. J Positive Behav Interventions 8(1):29–42. https://doi.org/10.1177/109830070600 80010501 Demenko G, Wagner A, Cylwik N (2010) The use of speech technology in foreign language pronunciation training Dowrick PW (2012) Self modeling: expanding the theories of learning. Psychol Schools 49(1):30– 41. https://doi.org/10.1002/pits.20613 Ekman P, Oster H (1979) Facial expressions of emotion. Annu Rev Psychol 30:29 Fancsali SE, Ritter S, Yudelson M, Sandbothe M, Berman SR (2016) Implementation factors and outcomes for intelligent tutoring systems: a case study of time and efficiency with cognitive tutor algebra. FLAIRS conference Finnegan E, Mazin AL (2016) Strategies for increasing reading comprehension skills in students with autism spectrum disorder: a review of the literature. Educ Treat Child 39(2):187–219 Fiorella L, Mayer RE (2013) The relative benefits of learning by teaching and teaching expectancy. Contemp Educ Psychol 38(4):281–288. https://doi.org/10.1016/j.cedpsych.2013.06.001 Fletcher-Watson S, Pain H, Hammond S, Humphry A, McConachie H (2016) Designing for young children with autism spectrum disorder: a case study of an iPad app. Int J Child-Comput Interact 7:1–14. https://doi.org/10.1016/j.ijcci.2016.03.002 Gallerati P, Bagnato S, Casciaro D, Conte A, Maisano M (2017) Use of serious gaming and virtual reality applications improves students’ learning retention and reduces safety risks and costs associated with training activities. Offshore Mediterranean conference and exhibition, Ravenna, Italy. Gay V, Leijdekkers P, Pooley A (2016) Building social awareness for teens and young adults with autism via gamification. In:Marsh T, Ma M, Oliveira MF, Baalsrud Hauge J, Göbel S (eds) Serious games: second joint international conference, JCSG 2016, Brisbane, QLD, Australia, September 26–27, 2016, Proceedings. Springer International Publishing, pp 116–127. https://doi.org/10. 1007/978-3-319-45841-0_10 Ge Z, Fan L (2017) Social development for children with autism using kinect gesture games: a case study in Suzhou Industrial Park Renai School. In: Cai Y, Goei SL, Trooster W (eds) Simulation and serious games for education. Springer Singapore, pp 113–123. https://doi.org/10.1007/978981-10-0861-0_8

108

Bibliography

Gillis J, Callahan E, Romanczyk R (2010) Assessment of social behavior in children with autism: the development of the Behavioral Assessment of Social Interactions in Young Children. Research in Autism Spectrum Disorders. Goulding J, Nadim W, Petridis P, Alshawi M (2012) Construction industry offsite production: a virtual reality interactive training environment prototype. Adv Eng Inf 26(1):103–116. https:// doi.org/10.1016/j.aei.2011.09.004 Graesser A, Chipman P, Haynes B, Olney A (2005) AutoTutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans Educa 48(4) Graesser A, Wiemer-Hastings K, Wiemer-Hastings P, Kreuz R (1999) AutoTutor: a simulation of a human tutor. Cognitive Syst Res 1(1):35–51 Grafsgaard JF, Wiggins JB, Boyer KE, Wiebe EN, Lester JC (2013) Automatically recognizing facial indicators of frustration: a learning-centric analysis. In: 2013 Humaine association conference on affective computing and intelligent interaction (ACII) Gray C (2001) How to respond to a bullying attempt: what to think, say and do. The Morning News 13(2) Guzman AL (2016) Making AI safe for humans: a conversation with Siri. In: Socialbots and Their Friends. Routledge, pp 85–101 Hailpern J (2007) Encouraging speech and vocalization in children with autistic spectrum disorder. SIGACCESS Access Comput 89:47–52. https://doi.org/10.1145/1328567.1328576 Hailpern J, Karahalios K, Halle J (2009) Creating a spoken impact: encouraging vocalization through audio visual feedback in children with ASD. In: Proceedings of the 27th international conference on human factors in computing systems, Boston, MA, USA Hall L, Woods S, Hall M (2009) Lessons learned using theory of mind methods to investigate user social awareness in virtual role-play. Human Technol 5(1) Hamari J, Koivisto J, Sarsa H (2014) Does gamification work?—a literature review of empirical studies on gamification. In: 2014 47th hawaii international conference on system sciences Hamer J, Purchase H, Luxton-Reilly A, Denny P (2015) A comparison of peer and tutor feedback. Assess Eval Higher Educ 40(1):151–164. https://doi.org/10.1080/02602938.2014.893418 Hastie H, Lim MY, Janarthanam S, Deshmukh A, Aylett R, Foster ME, Hall L (2016) I remember you!: interaction with memory for an empathic virtual robotic tutor. In: Proceedings of the 2016 international conference on autonomous agents and multiagent systems Hattie J, Timperley H (2007) The power of feedback. Rev Educ Res 77(1):81 Hautala J, Baker DL, Keurulainen A, Ronimus M, Richardson U, Cole R (2018) Early science learning with a virtual tutor through multimedia explanations and feedback on spoken questions. Educ Tech Res Dev 66(2):403–428. https://doi.org/10.1007/s11423-017-9558-6 Helal AS, Mokhtari M, Abdulrazak B (Eds) (2008) The engineering handbook of smart technology for aging, disability and independence (1 ed.). Wiley Inc. Henderson M, Henderson L, Grant S, Huang H (2018) Cognitive engagement in virtual worlds language learning. In: Gregory S, Wood D (eds) Authentic virtual world education: facilitating cultural engagement and creativity. Springer Singapore, pp 117–134. https://doi.org/10.1007/ 978-981-10-6382-4_8 Herrera G, Alcantud F, Jordan R, Blanquer A, Labajo G, De Pablo C (2008) Development of symbolic play through the use of virtual reality tools in children with autistic spectrum disorders: two case studies. Autism 12(2):143–157. https://doi.org/10.1177/1362361307086657 Herring P, Kear K, Sheehy K, Jones R (2017) A virtual tutor for children with autism. J Enabling Technol 11(1):19–27.https://doi.org/10.1108/JET-01-2016-0006 Hirschberg J, Litman D, Swerts M (2004) Prosodic and other cues to speech recognition failures. Speech Commun 43(1–2):155–175 Hmelo-Silver CE, Barrows HS (2015) Problem-based learning: goals for learning and strategies for facilitating. In: Walker A, Leary H, Hmelo-Silver CE, Ertmer PA (eds) Essential readings in problem-based learning: exploring and extending the legacy of Howard S. Barrows, pp 69–84

Bibliography

109

Hopkins IM, Gower MW, Perez TA, Smith DS, Amthor FR, Casey Wimsatt F, Biasini FJ (2011) Avatar assistant: improving social skills in students with an ASD through a computerbased intervention. J Autism Dev Disord 41(11):1543–1555. https://doi.org/10.1007/s10803-0111179-z Hoque ME (2008) Analysis of speech properties of neurotypicals and individuals diagnosed with autism and down. In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility, Halifax, Nova Scotia, Canada Hsieh Y-H, Lin Y-C, Hou H-T (2013) Exploring the role of flow experience, learning performance and potential behavior clusters in elementary students’ game-based learning. Interact Learn Env 1–16.https://doi.org/10.1080/10494820.2013.834827 Hu X, Xia H (2010) Automated assessment system for subjective questions based on LSI Huang X, Baker J, Reddy R (2014) A historical perspective of speech recognition. Commun ACM 57(1):94–103. https://doi.org/10.1145/2500887 Huijnen CAGJ, Lexis MAS, Jansens R, de Witte LP (2016) Mapping robots to therapy and educational objectives for children with autism spectrum disorder. J Autism Dev Disord 46(6):2100–2114. https://doi.org/10.1007/s10803-016-2740-6 Huskens B, Palmen A, Van der Werff M, Lourens T, Barakova E (2015) Improving collaborative play between children with autism spectrum disorders and their siblings: the effectiveness of a robot-mediated intervention based on lego® therapy. J Autism Dev Disord 45(11):3746–3755. https://doi.org/10.1007/s10803-014-2326-0 Iacobelli F, Cassell J (2007) Ethnic identity and engagement in embodied conversational agents. In: Pelachaud C, Martin J-C, André E, Chollet G, Karpouzis K, Pelé D (eds) Intelligent virtual agents, vol 4722, pp 57–63. Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-540-749 97-4_6 Jacklin A, Farr W (2005) The computer in the classroom: a medium for enhancing social interaction with young people with autistic spectrum disorders? British J Spec Educ 32(4):202–210. https:// doi.org/10.1111/j.1467-8578.2005.00398.x Jackson G, Guess R, McNamara D (2010a) Assessing cognitively complex strategy use in an untrained domain. Top Cognit Sci 2(1):127–137 Jackson GT, Dempsey KB, McNamara DS (2010) The evolution of an automated reading strategy tutor: from the classroom to a game-enhanced automated system. In: Khine MS, Saleh IM (eds) New science of learning: cognition, computers and collaboration in education. Springer New York, pp 283–306. https://doi.org/10.1007/978-1-4419-5716-0_14 Jacovina ME, Jackson GT, Snow EL, McNamara DS (2016) Timing game-based practice in a reading comprehension strategy tutor. In: International conference on intelligent tutoring systems Johnson WL (2007) Serious use of a serious game for language learning. In: Proceedings of the 2007 conference on artificial intelligence in education: building technology rich learning contexts that work Jokisch O, Hain H, Petrick R, Hoffmann R (2009) Robustness optimization of a speech interface for child-directed embedded language tutoring Jones P, Wilcox C, Simon J (2016) Evidence-based instruction for students with autism spectrum disorder: TeachTown Basics. In: Technology and the treatment of children with autism spectrum disorder. Springer, pp 113–129 Jones RSP, Quigney C, Huws JC (2003) First-hand accounts of sensory perceptual experiences in autism: a qualitative analysis. J Intell Dev Disabil 28(2):112–121. https://doi.org/10.1080/136 6825031000147058 Ke F, Im T (2013) Virtual-reality-based social interaction training for children with high-functioning autism. J Educ Res 106(6):441–461. https://doi.org/10.1080/00220671.2013.832999 Kenny P, Parsons TD, Gratch J, Leuski A, Rizzo AA (2007) Virtual patients for clinical therapist skills training. Intelligent Virtual Agents Kerr SJ (2002) Scaffolding: design issues in single & collaborative virtual environments for social skills learning. In: Proceedings of the workshop on virtual environments 2002, Barcelona, Spain

110

Bibliography

Kinchin I, Hay D, Adams A (2000) How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development. Educ Res 42(1):43–57 Kinchin IM (2014) Concept mapping as a learning tool in higher education: a critical analysis of recent reviews. J Cont Higher Educ 62(1):39–49. https://doi.org/10.1080/07377363.2014.872011 Knight V, Sartini E, Spriggs AD (2015) Evaluating visual activity schedules as evidence-based practice for individuals with autism spectrum disorders. In: J Autism Dev Disord 45(1):157–178. https://doi.org/10.1007/s10803-014-2201-z Kort B, Reilly R, Picard R (2001) An affective model of interplay between emotions and learning: reengineering educational pedagogy-building a learning companion Kozima H, Michalowski MP, Nakagawa C (2009) Keepon. Int J Soc Robot 1(1):3–18 Krämer N, Bente G (2010) Personalizing e-learning. The social effects of pedagogical agents. Educ Psychol Rev 22(1):71–87 Krämer NC (2006) Theory of mind as a theoretical prerequisite to model communication with virtual humans Krämer NC, Bente G (2010) Personalizing e-learning. The social effects of pedagogical agents. Educ Psychol Rev 22(1):71–87 Kyun S, Kim SA, Lee J-K, Lee H (2015) Facilitating adult learning: the effects of scaffolding strategies and self-regulation on discussion participation and performance in online learning. J Korea Soc IT Serv 14(1):115–128 Latham A, Crockett K, McLean D, Edmonds B (2012) A conversational intelligent tutoring system to automatically predict learning styles. Comput Educ 59(1):95–109 Laushey K, Heflin L, Shippen M, Alberto P, Fredrick L (2009) Concept mastery routines to teach social skills to elementary children with high functioning autism. J Autism Dev Disord 39(10):1435–1448 Lehman JF (1998) Toward the use of speech and natural language technology in intervention for a language-disordered population. In: Proceedings of the third international ACM conference on assistive technologies, Marina del Rey, California, United States Locke J, Kasari C, Wood JJ (2013, 2014/02/01) Assessing social skills in early elementary-aged children with autism spectrum disorders. J Psychoeducational Assessm 32(1):62–76. https://doi. org/10.1177/0734282913485543 Lorimer PA, Simpson RL, Smith Myles B, Ganz JB (2002). The use of social stories as a preventative behavioral intervention in a home setting with a child with autism. J Positive Behav Interventions 4(1):53–60. https://doi.org/10.1177/109830070200400109 Losh E (2005) In country with tactical iraqi: Trust, identity, and language learning in a military video game. Digital experience: proceedings of the digital arts and culture conference Lovaas OI (1987) Behavioral treatment and normal educational and intellectual functioning in young autistic children. J Consult Clin Psychol 55(1):3 Luerssen M, Lewis T (2009) Head X: tailorable audiovisual synthesis for ECAs. HCSNet Summerfest 2009, Sydney Luerssen M, Lewis T, Powers D (2010) Head X: customizable audio-visual synthesis for a multipurpose virtual head. In: Australasian joint conference on artificial intelligence MacMullin JA, Lunsky Y, Weiss JA (2016) Plugged in: electronics use in youth and young adults with autism spectrum disorder. Autism 20(1):45–54. https://doi.org/10.1177/1362361314566047 Madsen M, Kaliouby RE, Goodwin M, Picard R (2008) Technology for just-in-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder. In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility, Halifax, Nova Scotia, Canada Marcus A, Wilder D (2009) A comparison of peer video modeling and self-video modeling to teach textual responses in children with autism. J Appl Behav Anal 42(2):335 Massaro DW (2004) Symbiotic value of an embodied agent in language learning. In: Hawaii international conference on system sciences. https://doi.org/10.1109/HICSS.2004.1265333

Bibliography

111

McCleery JP (2015) Comment on technology-based intervention research for individuals on the autism spectrum. J Autism Dev Disord 45(12):3832–3835. https://doi.org/10.1007/s10803-0152627-y McGinnis E, Goldstein AP (2012) Skill streaming the elementary school child: a guide for teaching prosocial skills, 3rd edn. Research Press. McNamara D, Levinstein I, Boonthum C (2004) iSTART: interactive strategy training for active reading and thinking. Behav Res Methods, Inst Comp 36(2):222 Meder AM, Wegner JR (2015) iPads, mobile technologies, and communication applications: a survey of family wants, needs, and preferences. Augmentative Alternat Commun 31(1):27–36. https://doi.org/10.3109/07434618.2014.995223 Meluso A, Zheng M, Spires HA, Lester J (2012) Enhancing 5th graders’ science content knowledge and self-efficacy through game-based learning. Comput Educ 59(2):497–504. https://doi.org/10. 1016/j.compedu.2011.12.019 Merrill DC, Reiser BJ, Ranney M, Trafton JG (1992) Effective tutoring techniques: a comparison of human tutors and intelligent tutoring systems. J Learn Sci 2(3):277–305 Meyer J, Land R (2010) Threshold concepts and troublesome knowledge (5): dynamics of assessment. In: Threshold concepts and transformational learning, pp 61–79 Meyer JHF, Land R (2010) Threshold concepts and troublesome knowledge (5): Dynamics of assessment. In: Threshold concepts and transformational learning, pp 61–79 Milne M, Leibbrandt R, Raghavendra P, Luerssen M, Lewis T, Powers D (2013) Lesson authoring system for creating interactive activities involving virtual humans the thinking head whiteboard. In: 2013 IEEE symposium on, Intelligent agent (IA) Milne M, Luerssen MH, Lewis TW, Leibbrandt RE, Powers DM (2010) Development of a virtual agent based social tutor for children with autism spectrum disorders. The 2010 international joint conference on, neural networks (IJCNN) Milne M, Raghavendra P, Leibbrandt R, Powers DMW (2018) Personalisation and automation in a virtual conversation skills tutor for children with autism. J Multimodal User Interfaces 12(3):257–269. https://doi.org/10.1007/s12193-018-0272-4 Mitrovic A (2001) Investigating students’ self-assessment skills. User Model 247–250 Mondragon AL, Nkambou R, Poirier P (2016) Evaluating the effectiveness of an affective tutoring agent in specialized education. In: Verbert K,. Sharples M, Klobuˇcar T (eds) Adaptive and adaptable learning: 11th European conference on technology enhanced learning, EC-TEL 2016, Lyon, France, September 13–16, 2016, Proceedings. Springer International Publishing, pp. 446–452. https://doi.org/10.1007/978-3-319-45153-4_41 Monkaresi H, Bosch N, Calvo RA, D’Mello SK (2017) Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Trans Affect Comput 8(1):15–28 Moreno R, Flowerday T (2006) Students’ choice of animated pedagogical agents in science learning: a test of the similarity-attraction hypothesis on gender and ethnicity. Contemp Educ Psychol 31(2):186–207. https://doi.org/10.1016/j.cedpsych.2005.05.002 Moreno R, Mayer RE, Spires HA, Lester JC (2001) The case for social agency in computer-based teaching: do students learn more deeply when they interact with animated pedagogical agents? Cognition Inst 19(2):177–213. https://doi.org/10.1207/S1532690XCI1902_02 Mori M (1970) The uncanny valley. Energy 7(4):33–35 Mostow J (2005) Project Listen: a reading tutor that listens (2nd August). Carnegie Mellon University, School of Computer Science. https://www.cs.cmu.edu/~listen/research.html Mulder YG, Bollen L, Jong TD, Lazonder AW (2016) Scaffolding learning by modelling: the effects of partially worked-out models. J Res Sci Teach 53(3):502–523. https://doi.org/10.1002/ tea.21260 Nass C, Isbister K, Lee E-J (2001) Truth is beauty: researching embodied conversational agents. Embodied conversational agents Navarathna R, Dean D, Lucey P, Sridharan S (2010) Audio visual automatic speech recognition in vehicles

112

Bibliography

Neely LC, Ganz JB, Davis JL, Boles MB, Hong ER, Ninci J, Gilliland WD (2016) Generalization and maintenance of functional living skills for individuals with autism spectrum disorder: a review and meta-analysis. Rev J Autism Dev Disord 3(1):37–47. https://doi.org/10.1007/s40489-0150064-7 Nicholas M, Van Bergen P, Richards D (2015) Enhancing learning in a virtual world using highly elaborative reminiscing as a reflective tool. Learn Inst 36(Supplement C):66–75.https://doi.org/ 10.1016/j.learninstruc.2014.12.002 Noterdaeme M, Wriedt E, Höhne C (2010) Asperger’s syndrome and high-functioning autism: language, motor and cognitive profiles. Eur Child Adolesc Psychiatry 19(6):475–481. https://doi. org/10.1007/s00787-009-0057-0 O’Regan K (2003) Emotion and e-learning. J Asynchronous Learn Netw 7(3):78–92 Ogan A, Finkelstein S, Mayfield E, D’Adamo C, Matsuda N, Cassell J (2012) Oh dear stacy!” social interaction, elaboration, and learning with teachable agents. In: Proceedings of the SIGCHI conference on human factors in computing systems Okita SY, Turkay S, Kim M, Murai Y (2013) Learning by teaching with virtual peers and the effects of technological design choices on learning. Comput Educ 63(0):176–196. https://doi.org/10. 1016/j.compedu.2012.12.005 Owens G, Granader Y, Humphrey A, Baron-Cohen S (2008) LEGO® therapy and the social use of language programme: an evaluation of two social skills interventions for children with high functioning autism and asperger syndrome. J Autism Dev Disord 38(10):1944–1957. https://doi. org/10.1007/s10803-008-0590-6 Ozsivadjian A, Hibberd C, Hollocks MJ (2014) Brief report: the use of self-report measures in young people with autism spectrum disorder to access symptoms of anxiety, depression and negative thoughts. J Autism Dev Disord 44(4):969–974. https://doi.org/10.1007/s10803-013-1937-1 Panerai S, Ferrante L, Zingale M (2002) Benefits of the treatment and education of autistic and communication handicapped children (TEACCH) programme as compared with a non-specific approach. J Intellect Disabil Res 46(4):318–327. https://doi.org/10.1046/j.1365-2788.2002.003 88.x Park SW, Kim C (2011) A design framework for a virtual tutee system to promote academic reading engagement in a college classroom. Guest Editorial 17 Park U, Calvo R (2008) Automatic concept map scoring framework using the semantic web technologies Parsons S, Beardon L, Neale H, Reynard G, Eastgate R, Wilson J, Cobb S, Benford S, Mitchell P, Hopkins E (2000) Development of social skills amongst adults with Asperger’s Syndrome using virtual environments: the ‘AS Interactive’ project. 3rd ICDVRAT, University of Reading, Sardinia Italy, 163–170. Parsons S, Mitchell P (2002) The potential of virtual reality in social skills training for people with autistic spectrum disorders. J Intellect Disabil Res 46(5):430–443. https://doi.org/10.1046/ j.1365-2788.2002.00425.x Parsons S, Mitchell P, Leonard A (2005) Do adolescents with autistic spectrum disorders adhere to social conventions in virtual environments? Autism 9(1):95–117. https://doi.org/10.1177/136 2361305049032 Picard RW (2004) Toward Machines with emotional intelligence. ICINCO (Invited Speakers) Piper AM, O’Brien E, Morris MR Winograd T (2006) SIDES: a cooperative tabletop computer game for social skills development. In: Proceedings of the 2006 20th anniversary conference on computer supported cooperative work, Banff, Alberta, Canada Piramuthu S (2005) Knowledge-based web-enabled agents and intelligent tutoring systems. Educ IEEE Trans 48(4):750–756 Ploog BO, Scharf A, Nelson D, Brooks PJ (2013) Use of computer-assisted technologies (cat) to enhance social, communicative, and language development in children with autism spectrum disorders. J Autism Dev Disord 43(2):301–322. https://doi.org/10.1007/s10803-012-1571-3 Ponder M, Herbelin B, Molet T, Schertenlieb S, Ulicny B, Papagiannakis G, Magnenat-Thalmann N, Thalmann D (2003) Immersive VR decision training: telling interactive stories featuring advanced

Bibliography

113

virtual human simulation technologies. In: Proceedings of the workshop on virtual environments 2003 Putnam C, Chong L (2008) Software and technologies designed for people with autism: what do users want? In: Proceedings of the 10th international ACM SIGACCESS conference on computers and accessibility, Halifax, Nova Scotia, Canada Putnam RT (1987) Structuring and adjusting content for students: a study of live and simulated tutoring of addition. Am Educ Res J 24(1):13–48. https://doi.org/10.3102/00028312024001013 Quill KA (1997) Instructional Considerations for young children with autism: the rationale for visually cued instruction. J Autism Dev Disord 27(6):697–714. https://doi.org/10.1023/A:102 5806900162 Quirmbach L, Lincoln A, Feinberg-Gizzo M, Ingersoll B, Andrews S (2008) Social stories: mechanisms of effectiveness in increasing game play skills in children diagnosed with autism spectrum disorder using a pretest posttest repeated measures randomized control group design. J Autism Dev Disord 39(2):299–321. https://doi.org/10.1007/s10803-008-0628-9 Radford J, Bosanquet P, Webster R, Blatchford P (2015) Scaffolding learning for independence: clarifying teacher and teaching assistant roles for children with special educational needs. Learn Inst 36:1–10. https://doi.org/10.1016/j.learninstruc.2014.10.005 Rai D, Beck J, Arroyo I (2013) Causal Modeling to understand the relationship between student attitudes, affect and outcomes. Educ Data Mining 2013 Ramdoss S, Mulloy A, Lang R, O’Reilly M, Sigafoos J, Lancioni G, Didden R, El Zein F (2011) Use of computer-based interventions to improve literacy skills in students with autism spectrum disorders: a systematic review. Res Autism Spectrum Disord 5(4):1306–1318 Ramírez J, Rico M, Riofrío-Luzcando D, Berrocal-Lobo M, de Antonio A (2017) Students’ evaluation of a virtual world for procedural training in a tertiary-education course. J Educ Comput Res 56(1):23–47. https://doi.org/10.1177/0735633117706047 Rao PA, Beidel DC, Murray MJ (2008) Social skills interventions for children with asperger’s syndrome or high-functioning autism: a review and recommendations. J Autism Dev Disord 38:353–361. https://doi.org/10.1007/s10803-007-0402-4 Reichow B, Volkmar F (2010) Social skills interventions for individuals with autism: evaluation for evidence-based practices within a best evidence synthesis framework. J Autism Dev Disord 40(2):149–166. https://doi.org/10.1007/s10803-009-0842-0 Reynhout G, Carter M (2006) Social storiesTM for children with disabilities. J Autism Dev Disord 36(4):445–469. https://doi.org/10.1007/s10803-006-0086-1 Rickel J, Johnson WL (1999) Animated agents for procedural training in virtual reality: Perception, cognition, and motor control. Appl Artif Intell 13(4–5):343–382 Ritter S, Anderson J, Koedinger K, Corbett A (2007) Cognitive tutor: applied research in mathematics education. Psychon Bull Rev 14(2):249 Roberts V, Joiner R (2007) Investigating the efficacy of concept mapping with pupils with autistic spectrum disorder. Br J Spec Educ 34(3):127–135 Robertson AE, Simmons DR (2013) The relationship between sensory sensitivity and autistic traits in the general population. J Autism Dev Disord 43(4):775–784. https://doi.org/10.1007/s10803012-1608-7 Robins B, Dautenhahn K, te Boekhorst R, Billard A (2005) Robotic assistants in therapy and education of children with autism: can a small humanoid robot help encourage social interaction skills? Univ Access Inf Soc 4(2):20. https://homepages.feis.herts.ac.uk/~comqbr/aurora/public ations/UAIS_Journal.PDF Robison J, McQuiggan S, Lester J (2009). Modelling task-based versus affect-based Feedback behavior in pedagogical agents: an inductive approach Rotaru M, Litman D (2006) Dependencies between student state and speech recognition problems in spoken tutoring dialogues. Ryokai K, Vaucelle C, Cassell J (2003) Virtual peers as partners in storytelling and literacy learning. J Comput Ass Learn 19(2):195–208. https://doi.org/10.1046/j.0266-4909.2003.00020.x

114

Bibliography

Saadatzi MN, Pennington RC, Welch KC, Graham JH, Scott RE (2017) The use of an autonomous pedagogical agent and automatic speech recognition for teaching sight words to students with autism spectrum disorder. J Spe Educ Technol 0(0): 0162643417715751. https://doi.org/10.1177/ 0162643417715751 Sallows GO, Graupner TD (2005) Intensive Behavioral treatment for children with autism: fouryear outcome and predictors. Am J Mental Retard 110(6):417–438.https://doi.org/10.1352/08958017(2005)110[417:IBTFCW]2.0.CO;2 Sansosti FJ (2010) Teaching social skills to children with autism spectrum disorders using tiers of support: a guide for school-based professionals. Psychol Schools 47(3):257–281 Sansosti FJ, Powell-Smith KA (2008) Using computer-presented social stories and video models to increase the social communication skills of children with high-functioning autism spectrum disorders. J Positive Behav Intervent 10(3):162–178. https://doi.org/10.1177/1098300708316259 Sanz-Cervera P, Pastor-Cerezuela G, Fernández-Andrés M-I, Tárraga-Mínguez R (2015) Sensory processing in children with autism spectrum disorder: relationship with non-verbal IQ, autism severity and attention deficit/hyperactivity disorder symptomatology. Res Dev Disabil 45–46, 188–201 https://doi.org/10.1016/j.ridd.2015.07.031 Scassellati B (2005) Quantitative metrics of social response for autism diagnosis Schilbach L, Wohlschlaeger A, Kraemer N, Newen A, Shah N, Fink G, Vogeley K (2006) Being with virtual others: neural correlates of social interaction. Neuropsychologia 44(5):718–730 Schreibman L (2000) Intensive behavioral/psychoeducational treatments for autism: research needs and future directions. J Autism Dev Disord 30(5):373–378. https://doi.org/10.1023/a:100553512 0023 Schuller B, Wöllmer M, Moosmayr T, Rigoll G (2009) Recognition of noisy speech: a comparative survey of robust model architecture and feature enhancement. EURASIP J Audio, Speech, Music Proc 2009:1–17. https://doi.org/10.1155/2009/942617 Shane HC, O’Brien M, Sorce J (2009) Use of a visual graphic language system to support communication for persons on the autism spectrum. Perspect. Augmentative Altern Commun 18(4):130–136. https://doi.org/10.1044/aac18.4.130 Sherer M, Pierce K, Paredes S, Kisacky K, Ingersoll B, Schreibman L (2001) Enhancing" conversation skills in children with autism via video technology: which is better, “elf” “or” “Other” as a model? Behav Modif 25(1):140 Shneiderman B, Plaisant C (2005) Designing the user interface 4th, edition. Pearson Addison Wesley, USA Shute V, Towle B (2003) Adaptive e-learning. Educ Psychol 38(2):105–114 Silver M, Oakes P (2001) Evaluation of a new computer intervention to teach people with autism or asperger syndrome to recognize and predict emotions in others. Autism 5(3):299–316. https:// doi.org/10.1177/1362361301005003007 Sklar E, Richards D (2006) The use of agents in human learning systems. In: Proceedings of the fifth international joint conference on autonomous agents and multiagent systems, Hakodate, Japan. Sklar E, Richards D (2010) Agent-based systems for human learners. Knowl Eng Rev 25(02):111– 135 Sng CY, Carter M, Stephenson J (2014). A review of video modelling and scripts in teaching conversational skills to individuals with autism spectrum disorders. Rev J Autism Devel Disord 1(2):110–123. https://doi.org/10.1007/s40489-013-0010-5 Snow E, Jackson GT, Varner L, McNamara D (2013) The impact of system interactions on motivation and performance in a game-based learning environment. In: Stephanidis C (ed) HCI international 2013—posters’ extended abstracts, vol 374. Springer Berlin Heidelberg, pp 103–107 https://doi. org/10.1007/978-3-642-39476-8_22 Snow E, Jacovina M, Jackson GT, McNamara D (2016) iSTART-2: A reading comprehension and strategy instruction tutor. In: Crossley SA, McNamara DS (eds) Adaptive educational technologies for literacy instruction. Routledge. South M, Rodgers J (2017) Sensory, emotional and cognitive contributions to anxiety in autism spectrum disorders. Front Human Neurosci 11:20. https://doi.org/10.3389/fnhum.2017.00020

Bibliography

115

Stokes TF, Baer DM (1977) An implicit technology of generalization. J Appl Behav Anal 10(2):349– 367. https://doi.org/10.1901/jaba.1977.10-349 Storbeck J, Davidson NA, Dahl CF, Blass S, Yung E (2015) Emotion, working memory task demands and individual differences predict behavior, cognitive effort and negative affect. Cogn Emot 29(1):95–117. https://doi.org/10.1080/02699931.2014.904222 Strickland D (1998) Virtual reality for the treatment of autism. In: Riva G (ed) Virtual reality in neuro-psyco-physiology. IOS Press, pp 209 Stuart I (2004) The impact of immediate feedback on student performance: an exploratory study in Singapore. Global Perspect Account Educ 1(1):1 Sturm D, Peppe E, Ploog B (2016) eMot-iCan: design of an assessment game for emotion recognition in players with Autism. In: 2016 IEEE international conference on serious games and applications for health (SeGAH) Susarla S, Adcock A, Van Eck R, Moreno K, Graesser A, Group TR (2003). Development and evaluation of a lesson authoring tool for AutoTutor. In: AIED2003 supplemental proceedings Tartaro A, Cassell J (2006) Authorable virtual peers for autism spectrum disorders. In: Proceedings of the workshop on language-enabled educational technology at the 17th european conference on artificial intelligence (ECAI06), Riva del Garda, Italy. https://articulab.northwestern.edu/pub lications/Tartaro_Cassell_LEET.pdf Tartaro A, Cassell J (2008) Playing with Virtual Peers: Bootstrapping Contingent Discourse in Children with Autism. In: Proceedings of international conference of the learning sciences (ICLS), Utrecht, Netherlands. https://www.soc.northwestern.edu/justine/publications/TartaroCa ssell_ICLS_2008.pdf Tomlinson B, Yau ML, Baumer E (2006) Embodied mobile agents. Proceedings of the fifth international joint conference on Autonomous agents and multiagent systems Truong HM (2016) Integrating learning styles and adaptive e-learning system: current developments, problems and opportunities. Comput in Human Behavior 55:1185–1193. https://doi.org/10.1016/ j.chb.2015.02.014 Tsiourti C, Ben-Moussa M, Quintas J, Loke B, Jochem I, Albuquerque Lopes J (2016) A virtual assistive companion for older adults: design implications for a real-world application. In: SAI intelligent systems conference University of Cambridge (2017) Inclusive design toolkit. inclusive design group. Retrieved 7 July from https://www.inclusivedesigntoolkit.com/ van de Pol J, Volman M, Oort F, Beishuizen J (2015) The effects of scaffolding in the classroom: support contingency and student independent working time in relation to student achievement, task effort and appreciation of support. Instruct. Sci. 43(5):615–641. https://doi.org/10.1007/s11 251-015-9351-z VanLehn K (2011) The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educ Psychol 46(4):197–221. https://doi.org/10.1080/00461520.2011. 611369 Ventura J, Ventura M, Olabe JC (2005) Embodied conversational agents: developing usable agents. SoutheastCon, 2005. Proceedings. IEEE Virvou M, Troussas C (2011) Web-based student modelling for learning multiple languages. In: 2011 international conference on information society (i-Society) Vygotsky L (1978) Interaction between learning and development. Read Dev Child 23(3):34–41 Warren Z, Zheng Z, Das S, Young EM, Swanson A, Weitlauf A, Sarkar N (2015) Brief report: development of a robotic intervention platform for young children with ASD. J Autism Dev Disord 45(12):3870–3876. https://doi.org/10.1007/s10803-014-2334-0 Washington P, Voss C, Haber N, Tanaka S, Daniels J, Feinstein C, Winograd T, Wall D (2016) A Wearable Social Interaction Aid for Children with Autism. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems, Santa Clara, California, USA Watson MK, Pelkey J, Noyes CR, Rodgers MO (2016) Assessing conceptual knowledge using three concept map scoring methods. J Eng Educ 105(1):118–146. https://doi.org/10.1002/jee.20111

116

Bibliography

Webster J, Whitworth A, Morris J (2015) Is it time to stop “fishing”? A review of generalisation following aphasia intervention. Aphasiology 29(11):1240–1264 Wellman HM, Baron-Cohen S, Caswell R, Gomez JC, Swettenham J, Toye E, Lagattuta K (2002) Thought-bubbles help children with autism acquire an alternative to a theory of mind. Autism 6(4):343–363. https://doi.org/10.1177/1362361302006004003 Werry I, Dautenhahn K, Ogden B, Harwin W (2001) Can Social Interaction Skills Be Taught by a Social Agent? The Role of a Robotic Mediator in Autism Therapy. In: Proceedings of the 4th international conference on cognitive technology: instruments of mind Wertsch JV (1984) The zone of proximal development: some conceptual issues. New Dir Child Adolesc Dev 1984(23):7–18 Whalen C, Moss D, Ilan AB, Vaupel M, Fielding P, Macdonald K, Cernich S, Symon J (2010) Efficacy of TeachTown: basics computer-assisted intervention for the Intensive Comprehensive Autism Program in Los Angeles Unified School District. Autism 14(3):179–197. https://doi.org/ 10.1177/1362361310363282 Whitehill J, Serpell Z, Lin Y-C, Foster A, Movellan JR (2014) The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans Affect Comput 5(1):86– 98. https://doi.org/10.1109/TAFFC.2014.2316163 Whyte EM, Smyth JM, Scherf KS (2015) Designing serious game interventions for individuals with autism. J Autism Dev Disord 45(12):3820–3831. https://doi.org/10.1007/s10803-014-2333-1 Wiendl V, Dorfmüller-Ulhaas K, Schulz N, André E (2007) Integrating a virtual agent into the real world: The virtual anatomy assistant Ritchie. Intelligent Virtual Agents, Wik P (2011) The virtual language teacher: models and applications for language learning using embodied conversational agents KTH Royal Institute of Technology Wik P, Hjalmarsson A (2009) Embodied conversational agents in computer assisted language learning. Speech Commun 51(10):1024–1037 Wilkins J (2010) The relationship between social skills and challenging behaviors in children with autism spectrum disorders Williams JH, Massaro DW, Peel NJ, Bosseler A, Suddendorf T (2004) Visual-auditory integration during speech imitation in autism. Res Dev Disabil 25(6):559–575. https://doi.org/10.1016/j.ridd. 2004.01.008 Wisniewski B, Zierer K, Hattie J (2020) The power of feedback revisited: a meta-analysis of educational feedback research [Review]. Front Psychol 10(3087). https://doi.org/10.3389/fpsyg. 2019.03087 Wittwer J, Nückles M, Renkl A (2010) Using a diagnosis-based approach to individualize instructional explanations in computer-mediated communication. Educat Psychol Rev 22(1):9–23 Wolfe P (2006) The role of meaning and emotion in learning. New Direct Adult Continu Educ 110:7 Wong C, Odom SL, Hume KA, Cox AW, Fettig A, Kucharczyk S, Brock ME, Plavnick JB, Fleury VP, Schultz TR (2015) Evidence-based practices for children, youth, and young adults with autism spectrum disorder: a comprehensive review. J Autism Dev Disord 45(7):1951 Woolf B, Arroyo I, Muldner K, Burleson W, Cooper D, Dolan R, Christopherson R (2010) The effect of motivational learning companions on low achieving students and students with disabilities. Intelligent tutoring systems Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE transactions on pattern analysis and machine intelligence, 31:39–58. https://doi.org/10.1109/TPAMI.2008.52 Zhao R, Sinha T, Black AW, Cassell J (2016) Socially-aware virtual agents: automatically assessing dyadic rapport from temporal patterns of behavior. IVA Zhiping L, Yu S, Tianwei X, Yang L (2012) The research of classical learner models in intelligent tutoring systems. In: 2012 7th international conference on comput science & education (ICCSE) Zibrek K, Kokkinara E, Mcdonnell R (2018) The effect of realistic appearance of virtual characters in immersive environments—does the character’s personality play a role? IEEE Trans Visual Comput Graphics 24(4):1681–1690. https://doi.org/10.1109/TVCG.2018.2794638