Handbook of Digital Games and Entertainment Technologies [1 ed.] 9789814560504, 9789814560498

Table of contents :
Front Matter....Pages i-xxv
Front Matter....Pages 1-1
Front Matter....Pages 3-22
Front Matter....Pages 23-45
Front Matter....Pages 47-76
Front Matter....Pages 77-95
Front Matter....Pages 97-97
Front Matter....Pages 99-132
Front Matter....Pages 133-150
Front Matter....Pages 151-171
Front Matter....Pages 173-196
Front Matter....Pages 197-224
Front Matter....Pages 225-250
Back Matter....Pages 251-251
....Pages 253-292

Citation preview

Ryohei Nakatsu Matthias Rauterberg Paolo Ciancarini Editors

Handbook of Digital Games and Entertainment Technologies 1 3Reference

Handbook of Digital Games and Entertainment Technologies

Ryohei Nakatsu • Matthias Rauterberg Paolo Ciancarini Editors

Handbook of Digital Games and Entertainment Technologies With 373 Figures and 36 Tables

Editors Ryohei Nakatsu Design School Kyoto University Kyoto, Japan

Matthias Rauterberg Industrial Design Eindhoven University of Technology Eindhoven, The Netherlands

Paolo Ciancarini Dipto. Scienze dell’Informazione Università di Bologna Bologna, Italy

ISBN 978-981-4560-49-8 ISBN 978-981-4560-50-4 (eBook) ISBN 978-981-4560-51-1 (print and electronic bundle) DOI 10.1007/978-981-4560-50-4 Library of Congress Control Number: 2016940980 # Springer Science+Business Media Singapore 2017 This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, express or implied, with respect to the material contained herein or for any errors or omissions that may have been made. Printed on acid-free paper This Springer imprint is published by Springer Nature The registered company is Springer Science+Business Media Singapore Pte Ltd.

Preface

We are very proud of presenting to the reader this Springer Handbook of Digital Games and Entertainment Technologies. The Handbook covers all aspects of designing and building the most advanced interactive systems and devices for playing and entertaining, such as humanmachine interfaces, networks and robots, artificial intelligence, and interactive television, and includes interdisciplinary studies on serious games, digital art, edutainment, entertainment ethics and sociology, and many more. The scope of each part spans from basic theories to enabling technologies, and from advanced applications to psychological and sociological reflections on those. Entertainment is an essential part of our everyday activities. When we are children we play with our friends and listen to stories told by our relatives: these experiences are the basis of our ability to communicate, discuss, and negotiate with others. Johan Huizinga showed in his book Homo Ludens that playing is the basis of our culture. In the course of the human history, however, somehow entertainment has been thought as a marginal activity of lesser importance with respect to other activities such as education, work, medicine, etc. Thanks to the development of digital information and communication technologies (ICT), recently a plethora of new and interactive entertainment systems and products have emerged: from lean-back consumption to lean-forward interaction. Not only the younger generations but also more mature generations enjoy playing video games, communicating via social networks, and using new enhanced entertainment media like interactive television or immersive virtual reality systems. These new systems and products are blurring the distinction between work and play just as the psychologist Mihaly Csikszentmihalyi indicated in his “Flow Theory.” The entertainment market is huge; the companies offering products in the areas of playing consoles, smart toys, online games, digital music, interactive TVs, movies, robots, etc., are economically very relevant. However, until recently entertainment was not considered as a major research topic in academia. In the first decade of the current twenty-first century, some pioneering researchers including us working in the area of entertainment met several times in specific conferences and agreed to define a new research area called “entertainment computing.” In this new area, we wanted to get together various types of interdisciplinary research. We asked the International Federation on Information Processing (IFIP) to setup a new technical committee v

vi

Preface

focusing on entertainment computing. In 2002, our proposal was accepted and we formed a new group called Specialist Group on Entertainment Computing that in 2006 was upgraded to Technical Committee on Entertainment Computing (IFIP TC14). Members of TC14 have been working in various areas of entertainment computing and have been promoting academic activities in this area. Today, the design of digital games exploiting entertainment technologies has been recognized as an important and attractive topic in academic research. There are many people both in academia and industry who want to know the most recent topics and developments. Therefore we accepted the invitation by Springer to edit this Handbook. We hope that this work will contribute to a prospering development of entertainment computing both in academia and industry. The aim of this Handbook is to serve as a key reference work as it provides the readers with a holistic picture of this interdisciplinary field covering technical issues, aesthetic/design theories, and sociological investigations. The Handbook consists of invited contributions from top class scholars and researchers from several topic areas. Each author was assigned the task to recall the foundations of a specific subject in the field of entertainment computing, to survey the current state of the art in the same field, and finally to sketch the most advanced entertainment applications related to that field. The parts and their editors are the following: 1. Artificial Intelligence and Games (part editor Paolo Ciancarini): Artificial intelligence is a fundamental enabling technology for improving the playing experience in several types of games. This part includes four chapters dealing with algorithms and technologies for solving games, especially based on machine learning from large sets of playing data. 2. Brain-Computer Interfaces and Games (part editor Anton Nijholt): The direct exploitation of the brain activities of players is a radically new way to interact with entertainment products. This part includes six chapters describing how special devices allow to play in novel ways and how they influence the design of new videogames. 3. Entertainment Games (part editor Junichi Hoshino): Digital games are the core of entertainment computing. This part includes four chapters on different types of videogames exploiting a variety of entertainment computing technologies. Especially a survey on digital game industry would give readers the latest and deep insight into this fast moving area. 4. Interactive Storytelling (part editors Marc Cavazza and Michael Young): Storytelling is a very ancient activity; interactive storytelling is based on software which supports a narrative whose storyline is not predetermined. Interactive storytelling fulfills an old dream: the ability of the listener to “enter” the story she is told. This part includes five chapters which display a very interdisciplinary panorama on this subject. 5. Networking in Games (part editor Marco Roccetti): The global availability of the Internet and the widespread diffusion of powerful smartphones and personal

Preface

6.

7.

8.

9.

10.

11.

vii

computers allow millions of people to play anytime everywhere, alone or in very large parties. Entertainment systems need advanced network technologies which connect devices with very different capabilities. This part includes five chapters on the main issues in networking for entertainment. Serious Games (part editor Alessandro De Gloria): Serious games are one of the most promising areas in bridging the gap between enjoyful play and professional use through gamification. This part includes three chapters on serious gaming regarding science, technology, engineering, and mathematics (STEM); corporate identity; and ethics, privacy, and trust. Art and Entertainment (part editors Ryohei Nakatsu and Naoko Tosa): Entertainment computing is a discipline whose aim is to combine technology with other areas such as art, culture, etc. Digital arts are novel forms of expression that we are learning to appreciate. This part includes seven chapters showing various examples on how entertainment computing handle art and culture. Edutainment (part editor Wolfgang Mueller): The combination of edutainment and entertainment technologies – sometimes called “gamification” – offers new possibilities to educators and learners. This part includes two chapters. Entertainment Robots (part editors Hooman Samani and Elham Saadatian): Robots are just starting to coexist with humans in several fields. Playful robotic devices offer new challenges in human-machine interactions and enable new kinds of user experiences that need to be studied with special care. This part includes four chapters. Interactive Television and Online Experiences (part editor Marianna Obrist): Digital technologies enable new ways of interacting with old media: interactive TV is one major example, where the viewer is allowed to participate in the TV experience. This part includes five chapters. Social and Ethical Issues (part editor Matthias Rauterberg): Because entertainment products have not only a technical and economical impact but also an enormous societal impact, this part addresses all related topics. This part includes six chapters on social and ethical aspects regarding positive and negative effects, in particular addiction, emerging media technology, and unconscious emotions.

This Handbook is a work in progress (a living reference work in Springer terms). This means that the authors and the part editors are allowed to update the online version of the papers even before the next edition of the Handbook. We are already planning a new edition, to include the new developments and topics that the exciting field of entertainment computing will study in the next future. Therefore, we will invite additional chapters from recognized experts in such fields. The editors wish to thank all those who contributed to this Handbook, especially all part editors who have collected valuable chapter papers and reviewed them to guarantee the high quality of this Handbook. This Handbook can be cited as follows:

viii

Preface

Ryohei Nakatsu, Matthias Rauterberg, and Paolo Ciancarini (eds.), Handbook of Digital Games and Entertainment Technologies, Springer Singapore, 2016. DOI 10.1007/978-981-4560-52-8. ISBN (online) 978-981-4560-50-4 Ryohei Nakatsu Matthias Rauterberg Paolo Ciancarini

Contents

Volume 1 Part I

......................................

1

1

Game Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Akihiro Kishimoto and Martin Mueller

3

2

General Game Playing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Yngvi Björnsson and Stephan Schiffel

23

3

Monte-Carlo Tree Search in Board Games Mark H. M. Winands

..................

47

4

Physics Simulation Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Jochen Renz and Xiaoyu Ge

77

Part II 5

AI and Games

BCI and Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Action Games, Motor Imagery, and Control Strategies: Toward a Multi-button Controller . . . . . . . . . . . . . . . . . . . . . . . . . Damien Coyle, Jacqueline Stow, Karl. A. McCreadie, Chen Li, Jhonatan Garcia, Jacinta McElligott, and Aine Carroll

97

99

6

Brain-Computer Interface Games: Towards a Framework . . . . . . Hayrettin Gurkok, Anton Nijholt, and Mannes Poel

133

7

Brain-Computer Interfacing and Virtual Reality . . . . . . . . . . . . . . Doron Friedman

151

8

Games for BCI Skill Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reinhold Scherer, Gernot M€uller-Putz, Elisabeth V. C. Friedrich, Viktoria Pammer-Schindler, Karin Wilding, Stephan Keller, and Johanna Pirker

173

ix

x

Contents

9

Towards Serious Games for Improved BCI . . . . . . . . . . . . . . . . . . Brent J. Lance, Jon Touryan, Yu-Kai Wang, Shao-Wei Lu, Chun-Hsiang Chuang, Peter Khooshabeh, Paul Sajda, Amar Marathe, Tzyy-Ping Jung, Chin-Teng Lin, and Kaleb McDowell

197

10

User-Centered BCI Videogame Design . . . . . . . . . . . . . . . . . . . . . . Emilie Loup-Escande, Fabien Lotte, Guillaume Loup, and Anatole Lécuyer

225

Part III 11

Entertainment Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

251

Current Status of Applying Artificial Intelligence in Digital Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Youichiro Miyake

253

12

Hand Gesture Interface for Entertainment Games Kiyoshi Hoshino

............

293

13

Intelligent Character Technologies for Entertainment Games . . . . Hiroshi Mori

313

14

Real-World Game Platform for Lifelong Learning Society . . . . . . Junichi Hoshino

331

Part IV

Interactive Storytelling

............................

347

15

Discourse and Camera Control in Interactive Narratives . . . . . . . Arnav Jhala

349

16

Interactive Storytelling Paradigms and Representations: A Humanities-Based Perspective . . . . . . . . . . . . . . . . . . . . . . . . . . Hartmut Koenitz

361

17

Introduction to Interactive Storytelling Marc Cavazza and R. Michael Young

.....................

377

18

Planning Technologies for Interactive Storytelling . . . . . . . . . . . . . Julie Porteous

393

19

User Interaction for Interactive Storytelling Marc Cavazza and Fred Charles

.................

415

Part V

Networking in Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

429

20

Commodity Video Game Technology in Teletherapy . . . . . . . . . . . Gary Ushaw, Richard Davison, and Graham Morgan

431

21

Network Support for Mobile Gaming . . . . . . . . . . . . . . . . . . . . . . . Armir Bujari, Marco Furini, and Claudio E. Palazzi

459

Contents

xi

22

Opportunistic Networking for Games and Entertainment . . . . . . . Dario Maggiorini, Laura Anna Ripamonti, and Christian Quadri

481

23

QoE and Latency Issues in Networked Games . . . . . . . . . . . . . . . . Jose Saldana and Mirko Suznjevic

509

24

Video Gaming on Ad Hoc Networks: Challenges and Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Nadjib Achir and Khaled Boussetta

Part VI 25

26

27

Serious Games

...................................

A Tangible Serious Game Approach to Science, Technology, Engineering, and Mathematics (STEM) Education . . . . . . . . . . . . Riccardo Berta, Francesco Bellotti, Erik van der Spek, and Thomas Winkler Serious Games and Their Application in Creating Corporate Identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Magdalena Bielenia-Grajewska Ethics, Privacy, and Trust in Serious Games . . . . . . . . . . . . . . . . . Rod McCall and Lynne Baillie

545

569

571

593 611

Volume 2 Part VII 28

29

Art and Entertainment . . . . . . . . . . . . . . . . . . . . . . . . . . . .

641

Computer Music Languages and Systems: The Synergy Between Technology and Creativity . . . . . . . . . . . . . . . . . . . . . . . . Hiroki Nishino and Ryohei Nakatsu

643

Häusliches Gl€ uck: A Case Study on Deception in a Mixed Reality Environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alex Davies and Jeffrey Koh

693

30

Entertainment, Culture, and Media Art . . . . . . . . . . . . . . . . . . . . . Ryohei Nakatsu, Naoko Tosa, Matthias Rauterberg, and Wang Xuan

725

31

Games of Chance: Explorations into Our Animal Selves . . . . . . . . Siddharth Ramakrishnan and Victoria Vesna

777

32

Interface-Centric Art Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Christa Sommerer, Ulrich Brandstätter, and Laurent Mignonneau

805

33

Love and Sex with Robots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Adrian David Cheok, David Levy, Kasun Karunanayaka, and Yukihiro Morisawa

833

xii

34

Contents

Media, Art, and Society: Interface of the Digital Image, Aesthetics, and Culture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alistair D. Swale

Part VIII

Edutainment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

859

881

35

Edutainment in Sport and Health . . . . . . . . . . . . . . . . . . . . . . . . . . Josef Wiemeyer and Lars L. Tremper

883

36

Gamification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Alke Martens and Wolfgang M€uller

909

Part IX

Entertainment Robots

.............................

37

Challenges for Robots Acting on a Stage . . . . . . . . . . . . . . . . . . . . Mayumi Bono, Perla Maiolino, Augustin Lefebvre, Fulvio Mastrogiovanni, and Hiroshi Ishiguro

38

Design and Development of Playful Robotic Interfaces for Affective Telepresence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Elham Saadatian, Hooman Samani, and Ryohei Nakatsu

933 935

979

39

Enrobotment: Toy Robots in the Developing Brain . . . . . . . . . . . . 1011 Irini Giannopulu

40

Manzai Robots: Entertainment Robots as Passive Media Based on Autocreated Manzai Scripts from Web News Articles . . . . . . . 1041 Tomohiro Umetani, Akiyo Nadamoto, and Tatsuya Kitamura

Part X

Interactive TV and Online Video Experiences . . . . . . . . . . .

1069

41

Digital Interactive Television and the Older Generation . . . . . . . . 1071 Mark Rice and Mark Springett

42

Interactive Digital Narratives for iTV and Online Video . . . . . . . . 1097 Hartmut Koenitz and Noam Knoller

43

Place and ITV: Playful Design Strategies Towards Place-Oriented ITV . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1127 Gabriele Ferri

44

Social Interaction Design for Online Video and Television Pablo Cesar and David Geerts

45

The Interactive TV Experience: Where We Came From and Where We Are Going . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1195 Michael J. Darnell

. . . . . . 1157

Contents

Part XI

xiii

Social and Ethical Issues

...........................

1223

46

Addiction and Entertainment Products . . . . . . . . . . . . . . . . . . . . . 1225 Mark D. Griffiths and Halley M. Pontes

47

Applied Entertainment: Positive Uses of Entertainment Media . . . 1247 Jeffrey H. Goldstein

48

Introduction to the Ethics of New and Emerging Science and Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1271 Tsjalling Swierstra

49

Negative Effects of Video Game Play . . . . . . . . . . . . . . . . . . . . . . . 1297 Christopher L. Groves and Craig A. Anderson

50

Unconscious Emotions in Media Content . . . . . . . . . . . . . . . . . . . . 1323 Huang-Ming Chang, Leonid Ivonin, and Matthias Rauterberg Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1353

About the Editors

Paolo Ciancarini is Professor of Computer Science at the University of Bologna since 1992. He got a Ph.D. in Informatics at the University of Pisa in 1988. In Bologna, he lectures on Software Engineering and Software Architecture, and is member of the Faculty of the PhD School in Computer Science. He currently is the President of the Italian Association of University Professors in Computer Science. He is also the Vice-Director of CINI (National InterUniversity Consortium for Informatics), a consortium of 43 universities engaged in national and international research projects. In the period November 2011–June 2013, he has served in the national panel (ANVUR GEV01) for the evaluation of computer science research in Italian universities and research centers (VQR 2004–2010). In the period March 2012–December 2013, he has been a member of the Italian ICT Delegation at the European Union for the 7th ICT Framework Program. His research interests include: coordination languages and models, software architectures and infrastructures, advanced Web technologies, and software engineering for computer games. He has been involved as a site leader in several projects funded by the European Commission and by the Italian Government. He is the author of over 120 scientific papers and books. He is married, has two children, and is a passionate chess player and book collector.

xv

xvi

About the Editors

Prof. Dr. Matthias Rauterberg Eindhoven University of Technology (Netherlands), received a B.S. in Psychology (1978) at the University of Marburg (Germany), a B.A. in Philosophy (1981), a B.S. in Computer Science (1983), an M.S. in Psychology (1981), an M.S. in Computer Science (1986) at the University of Hamburg (Germany), and a Ph.D. in Computer Science/Mathematics (1995) at the University of Zurich (Switzerland). He was a senior lecturer for “usability engineering” in computer science and industrial engineering at the Swiss Federal Institute of Technology (ETH) in Zurich, where later he was heading the Man-Machine Interaction research group (MMI) (1989–1998). Since 1998, he is Fulltime Professor for “Interactive Systems Design” first at IPO – Centre for User System Interaction Research, and later at the Department of Industrial Design at the Eindhoven University of Technology (TU/e, The Netherlands). From 1999 till 2002, he was director of IPO. He was director of the graduate program at the Department of Industrial Design of the TU/e (2012–2014). He was the head of the Designed Intelligence research group (2006–2015). He was the Swiss representative in the IFIP TC13 on “Human Computer Interaction” (1994–2002) and the chairman of the IFIP WG13.1 on “HCI and Education” (1998–2004). He is now the Dutch representative in the IFIP TC14 on “Entertainment Computing” and the founding vice-chair of this TC14 (2006–2012). Since 2012, he is the IFIP TC14 chair (2013–2015). He was appointed as visiting professor at Kwansei Gakuin University (Japan) (2004–2007); he is senior honorary research fellow of Taicang University Science and Technology Park (since 2012) and guest professor at Jiangnan University (Wuxi, China) (2011–2015) and at East China University of Science and Technology (Shanghai, China) (2013–2016). He received the German GI-HCI Award for the best Ph.D. in 1997 and the Swiss Technology Award for the BUILD-IT system in 1998. In 2004, he was nominated as member of the “Cream of Science” in the Netherlands (the 200 top-level Dutch researchers) and among the 10 top-level TU/e scientists. Since 2007, he is holder of the IFIP Silver Core Award. He has over 400 publications in international journals, conference proceedings, books, etc. He acted also as editor and member of the editorial board of several leading international journals. Since 2009, he is co-editor-in-chief of the journal Entertainment Computing (Elsevier). He acts regularly as reviewer for national and international funding bodies, individual selection and departmental assessments committees, and large-scale European funding schemas. He was appointed as member of one of the few expert and evaluation panels for the most esteemed European grant from the European Research Council (2010–2014).

About the Editors

xvii

Ryohei Nakatsu received the B.S., M.S., and Ph.D. degrees in Electronic Engineering from Kyoto University in 1969, 1971, and 1982, respectively. After joining NTT in 1971, he mainly worked on speech recognition technology. In 1994, he joined ATR (Advanced Telecommunications Research Institute) as Director of ATR Media Integration and Communications Research Laboratories. In 2002, he became Professor at School of Science and Technology, Kwansei Gakuin University. Since March 2008 until December 2014, he was Professor at National University of Singapore (NUS) and was Director of Interactive and Digital Media Institute (IDMI) at NUS. In December 2014, he retired from NUS and came back to Japan. Now he is Adjunct Professor of Kyoto University, Kyoto/Japan, and Visiting Professor of Seika University, Kyoto/Japan. Also he has established two start-up companies and now he is serving as CEO of Hexogon Japan and Executive Director of NT & Associates. His research interests include interactive media, entertainment technologies, and communication robot/agent. In 1978, he received Young Engineer Award from the Institute of Electronics, Information and Communication Engineers Japan (IEICE-J), in 1996 the best paper award from the IEEE International Conference on Multimedia, in 1999, 2000, and 2001, Telecom System Award from Telecommunication System Foundation and the best paper award from Virtual Reality Society of Japan, and in 2000 the best paper award from Japanese Society for Artificial Intelligence. Also he received in 2010 IEEE Kansai Section medal, in 2011 IEEE Virtual Reality Service Award, and in 2012 IFIP TC14 Contribution Award. He is a fellow of the IEEE since 2001 and a life fellow since 2014. Also he is a fellow of the Institute of Electronics, Information and Communication Engineers Japan (IEICE-J) since 2001 and Virtual Reality Society of Japan since 2012. Also he is a honorary member of Japanese Society for Artificial Intelligence. He is a member of various academic societies such as IEEE, IEICE-J, Japanese Society for Artificial Intelligence, and others. He was a chair of IFIP Technical Committee on Entertainment Computing (TC14) since 2006 until 2012 and now is an honorary member of IFIP TC14.

Contributors

Nadjib Achir L2TI – Institut Galilée, University Paris 13, Sorbone Paris Cité, Villetaneuse, France Craig A. Anderson Center for the Study of Violence, Department of Psychology, Iowa State University, Ames, IA, USA Lynne Baillie Department of Mathematical and Computer Science, Heriot-Watt University, Edinburgh, UK Francesco Bellotti University of Genoa, Genoa, Italy Riccardo Berta University of Genoa, Genoa, Italy Magdalena Bielenia-Grajewska Intercultural Communication and Neurolinguistics Laboratory, Department of Translation Studies, Institute of English, Faculty of Languages, University of Gdansk, Gdańsk, Poland Yngvi Björnsson School of Computer Menntavegur, Reykjavík, Iceland

Science,

Reykjavik

University,

Mayumi Bono National Institute of Informatics, National Center of Sciences, Tokyo, Japan Khaled Boussetta Urbanet, CITI Insa Lyon / INRIA Grenoble Rhône-Alpes, CITI lab, Villeurbanne, France Ulrich Brandstätter Interface Culture, University of Art and Design Linz, Linz, Austria Armir Bujari Department of Mathematics, University of Padua, Padua, Italy Aine Carroll National Rehabilitation Hospital, Dun Laoghaire, Republic of Ireland Marc Cavazza School of Electronics and Digital Arts, University of Kent, Canterbury, UK Pablo Cesar CWI: Centrum Wiskunde and Informatica, Amsterdam, The Netherlands xix

xx

Contributors

Huang-Ming Chang Department of Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands Fred Charles School of Computing, Teesside University, Middlesbrough, UK Adrian David Cheok Imagineering Institute, Nusajaya, Malaysia City University London, London, UK Chun-Hsiang Chuang Brain Research Center, National Chiao Tung University, Hsinchu, Taiwan Damien Coyle Intelligent Systems Research Centre, Ulster University, Derry, Northern Ireland, UK Michael J. Darnell User Experience Center America – Visual Displays, Samsung Research America, Mountain View, CA, USA Alex Davies Creative Robotics Lab, NIEA, UNSW Art and Design, Sydney, NSW, Australia Richard Davison School of Computing Science, Newcastle University, Newcastle, UK Gabriele Ferri School of Informatics and Computing, Indiana University, Bloomington, IN, USA Doron Friedman The Advanced Reality Lab, The Interdisciplinary Center, Herzliya, Israel Elisabeth V. C. Friedrich Department of Cognitive Science, University of California San Diego, La Jolla, CA, USA Marco Furini University of Modena and Reggio Emilia, Modena, Italy Jhonatan Garcia Intelligent Systems Research Centre, Ulster University, Derry, Northern Ireland, UK Xiaoyu Ge Artificial Intelligence Group, Research School of Computer Science, The Australian National University, ANU College of Engineering and Computer Science, Canberra, Australia David Geerts CUO, iMinds/KU Leuven, Leuven, Belgium Irini Giannopulu Virtual Reality Prism, IHU-A-Brain and Spine Institute (ICM), UPMC, Groupe Hospitalier Pitié-Salpêtrière, Paris, France Jeffrey H. Goldstein Institute for Cultural Inquiry, Utrecht University, Utrecht, The Netherlands Mark D. Griffiths International Gaming Research Unit, Psychology Division, Nottingham Trent University, Nottingham, UK

Contributors

xxi

Christopher L. Groves Center for the Study of Violence, Department of Psychology, Iowa State University, Ames, IA, USA Hayrettin Gurkok Department EWI Research Group, Human Media Interaction (HMI), Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands Junichi Hoshino Graduate School of Systems and Information Engineering¸ Entertainment Computing Laboratory, University of Tsukuba, Tsukuba-shi, Ibaraki, Japan Kiyoshi Hoshino University of Tsukuba, Tsukuba, Ibaraki, Japan Hiroshi Ishiguro Osaka University, Osaka, Japan Leonid Ivonin Department of Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands Arnav Jhala University of California Santa Cruz, Santa Cruz, CA, USA Tzyy-Ping Jung Swartz Center for Computational Neuroscience, University of California San Diego, San Diego, CA, USA Kasun Karunanayaka Imagineering Institute, Nusajaya, Malaysia Stephan Keller Knowledge Technologies Institute, Graz University of Technology, Graz, Austria Peter Khooshabeh Cognitive Sciences Branch, U.S. Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, MD, USA Akihiro Kishimoto IBM Research, Ireland Research Lab, Dublin, Ireland Tatsuya Kitamura Department of Intelligence and Informatics, Faculty of Intelligence and Informatics, Konan University, Kobe, Hyogo, Japan Noam Knoller Interface Studies Group, Amsterdam School for Cultural Analysis (ASCA), University of Amsterdam, Amsterdam, The Netherlands Hartmut Koenitz Department of Entertainment and Media Studies, University of Georgia, Athens, GA, USA Jeffrey Koh Creative Robotics Lab, NIEA, UNSW Art and Design, Sydney, NSW, Australia Brent J. Lance Translational Neuroscience Branch, U.S. Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, MD, USA Anatole Lécuyer INRIA, Centre de Recherche Rennes Bretagne-Atlantique, Campus de Beaulieu, Rennes Cedex, France Augustin Lefebvre Sorbonne Nouvelle University Paris 3, Paris, France David Levy Imagineering Institute, Nusajaya, Malaysia Retro Computers Ltd, Luton, UK

xxii

Contributors

Chen Li Intelligent Systems Research Centre, Ulster University, Derry, Northern Ireland, UK Chin-Teng Lin Brain Research Center, National Chiao Tung University, Hsinchu, Taiwan Fabien Lotte INRIA, Centre de Recherche Bordeaux Sud-Ouest, Talence Cedex, France Guillaume Loup LIUM, Université du Maine, Laval Cedex 9, France Emilie Loup-Escande CRP-CPO (EA7273), Université de Picardie Jules Verne, Amiens, France Shao-Wei Lu Brain Research Center, National Chiao Tung University, Hsinchu, Taiwan Dario Maggiorini Department of Computer Science, University of Milan, Milan, Italy Perla Maiolino Goldsmiths University of London, London, UK Amar Marathe Translational Neuroscience Branch, U.S. Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, MD, USA Alke Martens Institute for Computer Science and Electrical Engineering, University of Rostock, Rostock, Germany Fulvio Mastrogiovanni Department of Informatics, Bioengineering, Robotics, and Systems Engineering, University of Genoa, Genoa, Italy Rod McCall Environmental Research and Innovation, Luxembourg Institute of Science and Technology, Esch-sur-Alzette, Luxembourg Karl. A. McCreadie Intelligent Systems Research Centre, Ulster University, Derry, Northern Ireland, UK Kaleb McDowell Translational Neuroscience Branch, U.S. Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, MD, USA Jacinta McElligott National Rehabilitation Hospital, Dun Laoghaire, Republic of Ireland Laurent Mignonneau Interface Culture, University of Art and Design Linz, Linz, Austria Youichiro Miyake Square Enix Co., Ltd., Tokyo, Japan Graham Morgan School of Computing Science, Newcastle University, Newcastle, UK Hiroshi Mori Graduate school of Engineering, Utsunomiya University, Utsunomiya-City, Tochigi, Japan

Contributors

xxiii

Yukihiro Morisawa Saitama Institute of Technology, Saitama, Japan Martin Mueller University of Alberta, Edmonton, AB, Canada Wolfgang M€ uller Media Education and Visualization Group (MEVIS), University of Education Weingarten, Weingarten, Germany Gernot M€ uller-Putz Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria Akiyo Nadamoto Department of Intelligence and Informatics, Faculty of Intelligence and Informatics, Konan University, Kobe, Hyogo, Japan Ryohei Nakatsu Design School, Kyoto University, Kyoto, Japan Anton Nijholt Department EWI Research Group, Human Media Interaction (HMI), Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands Hiroki Nishino NUS Graduate School for Integrative Sciences and Engineering, National University of Singapore and Graduate School of Media Design, Keio University, Singapore, Singapore Claudio E. Palazzi Department of Mathematics, University of Padua, Padua, Italy Viktoria Pammer-Schindler Know-Center GmbH, Graz, Austria Johanna Pirker Institute of Information Systems and Computer Media, Graz University of Technology, Graz, Austria Mannes Poel Department EWI Research Group, Human Media Interaction (HMI), Faculty of Electrical Engineering, Mathematics and Computer Science, University of Twente, Enschede, The Netherlands Halley M. Pontes International Gaming Research Unit, Psychology Division, Nottingham Trent University, Nottingham, UK Julie Porteous School of Computing, Teesside University, Middlesbrough, UK Christian Quadri Department of Computer Science, University of Milan, Milan, Italy Siddharth Ramakrishnan Neuroscience Program, Department of Biology, University of Puget Sound, Tacoma, WA, USA Matthias Rauterberg Industrial Design, Eindhoven University of Technology, Eindhoven, The Netherlands Jochen Renz Artificial Intelligence Group, Research School of Computer Science, The Australian National University, ANU College of Engineering and Computer Science, Canberra, Australia Mark Rice Institute for Infocomm Research, A*STAR, Singapore, Singapore

xxiv

Contributors

Laura Anna Ripamonti Department of Computer Science, University of Milan, Milan, Italy Elham Saadatian School of Electronics and Computer Science, Interaction, Complexity Group, University of Southampton, Southampton, UK Paul Sajda Department of Biomedical Engineering, Columbia University, New York, NY, USA Jose Saldana Department of Electrical Engineering and Communications EINA, Aragon Institute of Engineering Research (I3A), University of Zaragoza, Zaragoza, Spain Hooman Samani Department of Electrical Engineering, College of Electrical Engineering and Computer Science, National Taipei University, NTUP, Taipei, Taiwan Reinhold Scherer Institute for Knowledge Discovery, Graz University of Technology, Graz, Austria Stephan Schiffel School of Computer Menntavegur, Reykjavík, Iceland

Science,

Reykjavik

University,

Christa Sommerer Interface Culture, University of Art and Design Linz, Linz, Austria Mark Springett Middlesex University, London, UK Jacqueline Stow National Rehabilitation Hospital, Dun Laoghaire, Republic of Ireland Mirko Suznjevic Department of Telecommunications, Faculty of Electrical Engineering and Computing, University of Zagreb, Zagreb, Croatia Alistair D. Swale Screen and Media Studies, School of Arts, Faculty of Arts and Social Sciences, University of Waikato, Hamilton, New Zealand Tsjalling Swierstra Department of Philosophy, Faculty of Arts and Social Sciences, Maastricht University, Maastricht, MD, The Netherlands Naoko Tosa Academic Center for Computing and Media Studies, Kyoto University, Kyoto, Japan Jon Touryan Translational Neuroscience Branch, U.S. Army Research Laboratory, Aberdeen Proving Ground, Aberdeen, MD, USA Lars L. Tremper Institute of Sport Science, Technische Universitaet Darmstadt, Darmstadt, Germany Tomohiro Umetani Department of Intelligence and Informatics, Faculty of Intelligence and Informatics, Konan University, Kobe, Hyogo, Japan Gary Ushaw School of Computing Science, Newcastle University, Newcastle, UK

Contributors

xxv

Erik van der Spek Technische Universiteit Eindhoven, Eindhoven, The Netherlands Victoria Vesna Department of Design Media Arts, Art | Sci Center, University of California Los Angeles, Los Angeles, CA, USA Program in Empowerment Informatics, School of Integrative and Global Majors, University of Tsukuba, Tsukuba, Japan Yu-Kai Wang Brain Research Center, National Chiao Tung University, Hsinchu, Taiwan Josef Wiemeyer Institute of Sport Science, Technische Universitaet Darmstadt, Darmstadt, Germany Karin Wilding Knowledge Technologies Institute, Graz University of Technology, Graz, Austria Mark H. M. Winands Department of Data Science and Knowledge Engineering, Maastricht University, Maastricht, The Netherlands Thomas Winkler University of Luebeck, L€ubeck, Germany Wang Xuan Rolls-Royce, Singapore, Singapore R. Michael Young North Carolina State University, Raleigh, NC, USA

Part I AI and Games

1

Game Solvers Akihiro Kishimoto and Martin Mueller

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Terminology and Definitions on AND/OR Tree and Minimax Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . Algorithms for Game Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The αβ Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Proof-Number Search Variants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Basic Proof-Number Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Depth-First Proof-Number Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reduction of Memory Requirement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . PNS Variants in Directed Acyclic and Cyclic Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Endgame Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Threat-Based Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Early Win/Loss Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Monte Carlo Tree Search Solver . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Probability Propagation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results Accomplished on Solving Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recommended Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4 5 8 9 10 10 12 14 15 16 17 17 18 18 18 19 19 19

A. Kishimoto (*) IBM Research, Ireland Research Lab, Dublin, Ireland e-mail: [email protected] M. Mueller University of Alberta, Edmonton, AB, Canada e-mail: [email protected] # Springer Science+Business Media Singapore 2017 R. Nakatsu et al. (eds.), Handbook of Digital Games and Entertainment Technologies, DOI 10.1007/978-981-4560-50-4_35

3

4

A. Kishimoto and M. Mueller

Abstract

Games have simple, fixed rules as well as clear results such as win, draw, or loss. However, developing algorithms for solving games has been a difficult challenge in Artificial Intelligence, because of the combinatorial complexity that the algorithms must tackle. This chapter presents an overview of successful approaches and results accomplished thus far on game solving. Conducting tree search is a standard way to solve games and game positions. Remarkable progress has been made in developing efficient search algorithms over the last few decades. The chapter describes several standard techniques including αβ search, proof-number search, and endgame databases. Keywords

AND/OR tree • Search • αβ algorithm • Proof-number search • df-pn • Endgame databases

Introduction Researchers have invested significant resources on research in two-player zero-sum games with perfect information. Many popular board games such as chess, checkers, and Go fall into this category, and these games have been used as test beds for testing algorithms in artificial intelligence (AI) research. In this type of zero-sum game, the two players’ goals are strictly opposite: when one player wins, the opponent loses. Perfect information means that all information is available to both players. Game positions are typically represented by a board state and the turn to play. Depending on the game, extra information such as the history of the moves played so far may be needed to play and score the result according to the rules. In a two-player zero-sum game with perfect information, if both players continue playing optimal moves from a position, the final outcome for that position, called the game-theoretic value or value, is either a win for the first player (i.e., a loss for the second player), or a loss for the first player (i.e., a win for the second player), or a draw (if allowed by the rules of the game). Any finite game can be solved in principle since the value of the starting position can be determined by following optimal moves of both players. Allis (1994) defines three levels of solving a game: 1. Ultra-weakly solved. The game-theoretic value of the start position has been determined. 2. Weakly solved. A strategy from the start position has been determined to obtain the game-theoretic value of the start position under reasonable computing resources. 3. Strongly solved. The game-theoretic value and a strategy have been determined for all legal positions under reasonable computing resources.

1

Game Solvers

5

There are often significant differences among these three levels in terms of difficulties of achieving the levels of solving. For example, the game of n  n Hex can be proven to be a win for the first player (Nash 1952). However, winning strategy is only known for n 10 (Pawlewicz and Hayward 2014). For weakly or strongly solving games, the availability of computing resources is restricted to only reasonable ones. In principle, as remarked in Allis (1994), given a large enough amount of time, CPU, and memory resources, games such as chess or Go could be weakly or strongly solved by performing αβ search or retrograde analysis described later in this chapter. In practice, many games are far too large for a brute force approach, and therefore the development of game solvers that work efficiently under the available resources has been an ongoing challenge. This chapter gives an overview of the most popular computational approaches for finding strategies for game positions of interest, that is, for at least weakly solving them. Search algorithms are the core of these approaches. In practice, highperformance game solvers combine game-independent search algorithms with game-specific knowledge. While both game-independent and game-specific approaches are necessary to significantly improve the performance of the solvers, the chapter mainly deals with game-independent search algorithms due to their applicability to many games and even to other domains.

Terminology and Definitions on AND/OR Tree and Minimax Tree Assume that a player p tries to prove a win for a position where p is to play. Then, p must prove that at least one of the legal moves leads to a win. However, if it is the opponent’s turn to play, p must be able to win against all the opponent’s moves. This check can be performed recursively, leading to the concept of an AND/OR tree search. The definitions and terminology for AND/OR tree search introduced in this section follow (Kishimoto et al. 2012). An AND/OR tree is a rooted, finite tree consisting of two types of nodes: OR and AND nodes. OR nodes correspond to positions where the first player is to play and AND nodes to positions where the second player moves next. A directed edge representing a legal move is drawn from node n to node m if that move played at position n leads to position m. All nodes except the root node have a parent. In this chapter, players are assumed to move alternately. Therefore, each child of an OR node is an AND node, and each child of an AND node is an OR node. In addition, the root is assumed to be an OR node with no loss of generality, but it can be an AND node as well in practice. Each node in an AND/OR tree has three possible types of values: win, loss, or unknown. As in Kishimoto et al. (2012), the phrase “a node is x” is short for “a node has value x.” A node of value win/loss indicates that its corresponding position is a sure win/loss for the first player, respectively. For the sake of simplicity, the value of draw is defined to be the value of loss if possible game outcomes are not explicitly defined. Several techniques for dealing with draws are surveyed in

6

A. Kishimoto and M. Mueller

Kishimoto et al. (2012). A node of value unknown indicates that the game-theoretic value of its corresponding position has not yet been proven. To determine its gametheoretic value, such a node must be examined further. Expanding a node is the procedure of generating all children of the node, which represent legal moves, and connecting the node to these children by directed edges. A node with no children is called a terminal node. A terminal node is either a win or a loss, as determined by the rules of the game. An internal node is a node that has at least one child. A leaf node is an unexpanded node with unknown value. A leaf node must be expanded to determine whether it is internal or terminal. AND/OR tree search aims to solve an AND/OR tree, i.e., determine whether the root is a win or a loss. The value of an internal node is calculated from the values of its children. If at least one child of an internal OR node n is a win, then n is also a win. At position n, the first player can play a move that leads to that child and win against the second player. If all children of n are losses, n is a loss since all the legal moves of the first player at position n lead to losing positions. Otherwise, n is unknown. Similarly, an internal AND node n is a loss if at least one of its children is a loss, a win if all its children are wins, and unknown otherwise. A node that is a win is also called a proven node, while a node that has been determined to be a loss is a disproven node. A proof is a computed win, while a disproof is a computed loss. When a node is computed to be a win, a subtree of an AND/OR tree contains a winning strategy for the first player. Such a subtree is called a proof tree and guarantees that node is a win. A proof tree T with root node r is constructed as follows: 1. 2. 3. 4.

T contains r. For each internal OR node of T, T contains at least one child. For each internal AND node of T, T contains all children. All terminal nodes in T are wins.

A disproof tree, which contains a winning strategy for the second player, is defined in an analogous way, by swapping AND and OR in the definition above, and requiring all terminal nodes to be losses. Figure 1 illustrates an example of an AND/OR tree. OR nodes are shown by squares and AND nodes are shown by circles. Values win, loss, and unknown are shown by W, L, and U, respectively. Nodes D, F, H, and K are terminal nodes and nodes G, J, and L are leaf nodes. The other nodes are internal nodes for which values are calculated by propagating back the values of the leaf and terminal nodes. For example, node C is a loss because node F is a loss. Irrespective of the value of G, the second player can win against the first player by selecting a move that leads to F. Node I is unknown because node K is a win and node L is unknown. The second player still has a chance to win against the first player by examining L. By following this back-propagation procedure, the value of the root node A is determined to be a win. A proof tree of A is shown with bold lines. Note that for weakly solving the root, AND/OR tree search can ignore nodes that are not part of the

1

Game Solvers

7

Fig. 1 Example of AND/OR tree

constructed proof tree. For example, in Fig. 1, there is no need to examine the nodes that are not along the bold lines. A high-performance AND/OR tree search algorithm focuses on finding a proof tree as quickly as possible. Assume the standard search space for trees with depth d and branching factor (the number of legal moves at each internal node) b. Also, assume there is only one proof tree in this search space and all terminal nodes are located at depth d. In the worst case, search d examines j k all b terminal nodes to find a proof. In contrast, the proof tree contains d

only b2 terminal nodes. There may be many proof trees, but finding one is sufficient to solve the root. In many games, more than one sequence of moves lead to the same position (e.g., Hex and Othello) and a move sequence may lead to a repeated position (e.g., chess and checkers). In other words, the search space of such games can be represented by a directed acyclic graph (DAG) or a directed cyclic graph (DCG). The notion of AND/OR trees, proof, and disproof trees can be generalized for such graphs. Minimax trees are a generalization of AND/OR trees. Instead of Boolean values, numerical scores are assigned to leaf and terminal nodes. An OR node in such a minimax tree is called a Max node, and an AND node is called a Min node. The scores are assigned by calling an evaluation function that approximates the chance of the first player winning. A larger score indicates that a position is more favorable for the first player. As in AND/OR trees, the score at each internal node of a minimax tree is calculated from the leaf nodes in a bottom-up manner. At an

8

A. Kishimoto and M. Mueller

Fig. 2 Example of minimax tree

internal Max node, the first player aims to maximize its advantage by taking the maximum score of all children of that node. At an internal Min node, the second player aims to minimize the advantage of the first player by calculating the minimum score of the children. Figure 2 illustrates an example of a minimax tree where Max and Min nodes are represented by squares and circles, respectively, and evaluation scores are written inside the squares and circles. The score of the root node A becomes 50 by propagating back the scores of leaf nodes D, E, F, and G. The solution tree which contains a strategy in the minimax framework is defined in a similar way to the proof tree. For details, see e.g., (de Bruin et al. 1994).

Algorithms for Game Solvers This section describes general approaches to solve games or game positions. Forward search explores a tree from the root until it reaches terminal nodes. Depth-first search (DFS) and best-first search (BFS) are standard search methods commonly used in many applications including game solvers. While DFS requires only a small amount of memory, it suffers from a combinatorial explosion of its time complexity when the search space is large. In contrast, while BFS tends to explore much smaller search space than DFS, BFS suffers from a combinatorial explosion of its space complexity caused by storing the explored search space in memory. However, game research has revealed that the issue on BFS large memory requirement can be resolved by preserving only important portions of the search

1

Game Solvers

9

space. In addition, BFS can often be enhanced further by incorporating ideas behind DFS. Backward search is the other approach to search for a solution and deals with the scenario where all terminal nodes can be enumerated with the available computing resources. Backward search starts with terminal nodes and determines the values of positions toward the root. It can be combined with forward search (e.g., Gasser 1996; Schaeffer et al. 2007). The following sections introduce standard forward and backward search algorithms that can be used for game solvers.

The ab Algorithm The αβ algorithm (Knuth and Moore 1975) is a depth-first forward search algorithm commonly used in many two-player game-playing programs. αβ is used to determine the next move to play but can be applied to solve games, for example, by assigning a score of 1 to terminal positions that are wins for the first player, 1 to terminal positions that are not wins, and other heuristic values to undecided positions. Basic αβ examines a minimax tree in a depth-first manner with a fixed depth d to compute the best score of the root node. If d is set deep enough, αβ returns the winner at the root by returning the score of either 1 or -1. αβ preserves a lower bound α and a upper bound β on the score of a minimax tree. During performing search, the scores of α and β are updated and used for pruning subtrees that are irrelevant for calculating the score at the root. By incorporating good move ordering such as (Schaeffer 1989), αβ can reduce the search space to examine by increasing the frequency of pruning subtrees. This enables αβ to search much deeper and contributes to significantly improving the performance of αβ-based game solvers. Many variants and enhancements to αβ have been developed over decades (see the literature review such as (Marsland 1986). Iterative deepening (ID) (Slate and Atkin 1977) is a standard enhancement to αβ. ID carries out a series of shallower depth-first search before performing direct search to depth d. That is, ID first performs depth-first search from the root with d = 1. Then, if ID finds no solution, it performs depth-first search again from the root with d = 2, then d = 3, 4,   , and so on. This procedure is repeated until ID either finds a solution, proves that there is no solution, or exhausts resources. Intuitively, because of extra overhead of reexamining previously examined nodes, αβ combined with ID looks less efficient than basic αβ that performs direct search to depth d. However, αβ with ID is empirically more efficient, because previous search results can be used to improve move ordering. As a result, the cost paid for shallower search becomes a small price in order to significantly increase the chance of pruning subtrees for deeper search. For example, the best move calculated in previous shallower search has a high probability that is also the best in deeper search. Examining this move first therefore reduces a large amount of work. ID is enhanced further by using a transposition table (TT) (Greenblatt et al. 1967; Slate and Atkin 1977), a large cache which stores search results of

10

A. Kishimoto and M. Mueller

previously examined nodes such as scores, flags indicating whether these scores are exact, lower bounds or upper bounds, search depths, best moves for shallower search, etc. The transposition table is usually constructed as a hash table and takes an advantage of the fact that the search space of many games are a graph where more than one path can lead to the same node, a so-called transposition. The transposition table prevents ID from examining the subtree again by merely retrieving and returning a score saved in the TT, when ID encounters a transposition and verifies that the cached score can be reused. In addition, even if the cached score does not result in eliminating the examination of subtrees, such as the case where a cached node has not been explored deep enough, the best move information in the TT can be used to improve move ordering of αβ, significantly reducing search effort.

Proof-Number Search Variants An essential drawback of αβ is that search is limited by fixed depth that causes the minimax tree to grow exponentially with the search depth. The drawback can be alleviated by introducing enhancements such as fractional depth and search extensions (e.g., Campbell et al. 2002; Tsuruoka et al. 2002). However, because they cure the problem of exponential tree growth only partially, αβ-based solvers are still unable to solve positions that require deep search. For example, it is difficult to adjust αβ to solve positions that depend on narrow but deep lines of play, as occur in Go-Moku and checkmating puzzles in chess-like games (Kishimoto et al. 2012). This section describes proof-number search variants that address this problem.

Basic Proof-Number Search Proof-Number Search (PNS) (Allis et al. 1994) is a best-first forward search algorithm. PNS calculates the proof and disproof numbers that estimate the difficulty of solving nodes. Based on the proof and disproof numbers, PNS aims to examine nodes in simplest-first order. As long as a node is considered to be easy because of a low proof or disproof number, PNS keeps exploring its subtree without any bound on the search depth. This characteristic enables PNS to find narrow but deep proofs or disproofs efficiently. Formally, the proof number of a node is defined as the minimum number of leaf nodes in its subtree that must be proven to prove that the node is a win, while the disproof number is the minimum number of such leaf nodes that must be disproven to prove that the node is a loss. The smaller the proof/disproof number is, the easier PNS assumes that it is to prove that a node is a win/loss. Let n be a node with children n1,  ,n k. One proven child suffices to prove a win at an OR node, while all children must be proven to show a win at an AND node

1

Game Solvers

11

Fig. 3 Example of proof and disproof numbers

(and vice versa for disproof). The proof number pn(n) and the disproof number dn (n) of node n are therefore calculated as follows: For a proven terminal node n, pn(n) = 0 and dn(n) = 1. For a disproven terminal node n, pn(n) = 1 and dn(n) = 0. For an unknown leaf node n, pn(n) = dn(n) = 1. For an internal OR node n that has children c1,   , ck, pn(n) = min(pn(c1),   , pn(ck)), dn(n) = dn(c1) +    + dn(ck). 5. For an internal AND node n that has children c1,   ,ck, pn(n) = pn(c1) +    + pn(ck), dn(n) = min(dn(c1) +    + dn(ck)).

1. 2. 3. 4.

Figure 3 shows an example. The proof and disproof numbers of a node are shown inside that node. The proof number is shown above the disproof number. In this Figure, node I is a terminal node, a loss. Nodes D, F, G, H, J, and K are leaf nodes with proof and disproof numbers initialized to 1. The proof and disproof

12

A. Kishimoto and M. Mueller

numbers of internal nodes are calculated by the rule described above. For example, pn(E) = min(pn(I), pn(J), pn(K )) = min(1, 1, 1) = 1 and dn(E) = dn(I ) + dn (J) + dn(K ) = 0 + 1 + 1 = 2. PNS maintains a proof and a disproof number for each node. In the beginning, the AND/OR tree of PNS consists only of the root node and its proof/disproof numbers are initialized to 1. Then, until either the value of the root is determined, or resources are exhausted, PNS repeats the following four steps: 1. Starting from the root, one path in the tree is traversed until PNS finds a leaf node called a most-promising node (MPN) (aka most-proving node). To find a MPN, PNS selects an AND child with the smallest proof number among all children at internal OR nodes, and an OR child with the smallest disproof number at internal AND nodes. Ties are broken arbitrarily. In practice, game dependent knowledge can sometimes be used here. 2. The MPN is expanded by generating all its children and adding new edges from the MPN to them. The MPN becomes an internal node and the children are new leaf nodes. 3. If the MPN turns out to be a terminal node, the proof and disproof numbers of the MPN are set according to the rules of the game. Otherwise, the proof and disproof numbers of the new leaf nodes are initialized to 1. 4. The proof and disproof numbers of the affected nodes are recomputed along the path from the MPN back to the root. Figure 4 illustrates an example of the procedure of PNS. Starting from the root node A, PNS traverses path A ! B ! D and finds MPN D. Then, PNS expands D and generates three children of which proof and disproof numbers are initialized to 1. Next, it updates the proof and disproof numbers of D, B, and then A.

Depth-First Proof-Number Search One inefficiency of basic PNS is that it always propagates back updated proof and disproof numbers from a MPN to the root even if the child with the smallest (dis) proof number remains the same. For example, assume that a MPN is located 100 levels down in the tree from the root. To expand only one leaf node (i.e., MPN), basic PNS must traverse back and forth 100 nodes along the path from the root to the MPN. The depth-first proof-number (df-pn) search algorithm (Nagai 2002) reduces the frequency of reexamining internal nodes. If the search space is a tree, Nagai proves that df-pn is equivalent to PNS in the sense that both algorithms can always select a MPN. This section briefly describes the idea behind df-pn. See (Kishimoto et al. 2012) as well as (Nagai 2002) for precise, detailed descriptions. The node selection scheme of df-pn is still identical to PNS, which makes df-pn explore the search space in a best-first manner. However, df-pn uses two thresholds to explore the search space in a depth-first manner as well: one for the proof number and the other for the disproof number. If both proof and disproof numbers of a node

1

Game Solvers

13

Fig. 4 Example of PNS procedure

n are smaller than the thresholds of proof and disproof numbers, respectively, df-pn continues examining n’s subtree without backtracking to n’s parent. Let thpn(n) and thdn(n) be the thresholds of the proof and disproof numbers at node n, respectively. For example, in Fig. 5, the root OR node A has three children B, C, and D where pn(B) = 4, pn(C) = 8 and pn(D) = 10. B remains on a path to a MPN until pn(B) exceeds pn(C) = 8, the second smallest proof number among all children. Therefore, the proof number threshold for B, thpn(B) = 9, and search can stay in this subtree without updating exact proof numbers until the threshold is reached. When pn(B)  9 = pn(C) + 1, the MPN switches to a node below C in the tree. Thresholds of the disproof number at AND nodes are handled analogously with a disproof threshold. Df-pn can remain in the subtree of the child c1 with smallest disproof number as long as dn(c1) is better than the disproof number of the second best child. For example, in Fig. 5, df-pn sets thdn(E) = dn(F) + 1 = 4 + 1 = 5 to be able to switch as soon as the disproof number of F becomes strictly smaller than E. Thresholds for proof numbers of children at AND nodes and disproof numbers of children at OR nodes are set as illustrated below for AND node B in Fig. 5. Assume that thpn(B) = 9, and B has two children E and F with pn(E) = 3 and pn(F) = 1. E is selected for expansion since dn(E) < dn(F), and search can stay in its subtree until pn(E) + pn(F) reaches the threshold for the parent,

14

A. Kishimoto and M. Mueller

Fig. 5 Illustration of df-pn behavior

thpn(B) = 9. Therefore, for searching E, its threshold thpn(E) is set to thpn(B) – pn(F) = 9–1 = 8.

Reduction of Memory Requirement The baseline PNS and df-pn algorithms store all nodes that they expand in main memory. For difficult problems, they quickly run out of space. More practical algorithms use a fixed-size transposition table (TT) with a replacement strategy (Breuker 1998; Seo et al. 2001; Nagai 2002). This solves the memory problem but adds overhead from re-expanding nodes that have been overwritten in the table. The SmallTreeGC algorithm (Nagai 1999) is an effective replacement strategy used in many high-performance solvers (Nagai 2002; Kishimoto and M€uller 2005; Kishimoto 2010; Kaneko 2010). The TT entry for each node stores the size of the subtree rooted at that node. SmallTreeGC assumes that a small subtree can be reconstructed easily with a small amount of effort even if that subtree is not available in the TT. When the TT becomes full, SmallTreeGC discards a fixed fraction R of the TT entries, starting with those of smallest subtree size. The value of R is determined empirically. According to Kishimoto et al. (2012), Pawlewicz uses the multiple-probe replacement strategy of Beal and Smith (1996) in their Hex df-pn implementation. Their multiple-probing TT replacement strategy implementation is popular with αβ chess

1

Game Solvers

15

programs such as Fruit and Stockfish. The technique probes four consecutive entries in a single hash table, and overwrites a TT entry with smallest subtree size among the four. PN2 and its variants are another approach which reduces the memory requirement of PNS (Allis 1994; Breuker 1998; Winands et al. 2004). The main idea is to use two levels of PNS. The first level, PN1, stores a tree starting from the root node. At a leaf node of PN1, PN2 invokes the second level of PNS, PN2, with limited number of nodes expanded and a separate memory allocation. After PN2 completes search, this PN2 tree is discarded and only the proof and disproof numbers of the root, corresponding to a PN1 leaf node, are passed back to PN1. A hybrid approach was used to solve the game of checkers (Schaeffer et al. 2007). As in PN2, a disk-based first-level search called the front-end manager was based on PNS. It invoked a memory-only second level search called the backend prover. This back-end used df-pn with a TT and SmallTreeGC.

PNS Variants in Directed Acyclic and Cyclic Graphs There are three problems to address when PNS and df-pn searches in DAG or DCG. The overestimation problem results from double-counting proof and disproof numbers of nodes in a DAG that can be reached along multiple paths (Allis et al. 1994). The Graph-History Interaction (GHI) problem (Palay 1983; Campbell 1985), which can occur both in αβ and PNS variants, originates from incorrect handling of cycles in DCG. The infinite loop problem (Kishimoto and M€uller 2003; 2008) refers to the phenomenon that df-pn may loop forever without expanding any new nodes in DCG. Current solutions to these problems are summarized in Kishimoto et al. (2012).

Search Enhancements Although PNS variants are already powerful without domain knowledge, highperformance game solvers incorporate many search enhancements to be able to solve difficult game positions. This section describes some of the well-known methods. See Kishimoto et al. (2012) for comprehensive survey. The proof and disproof numbers of the leaf node are always set to 1 in their original definition. However, in practice, some leaf nodes are easier to (dis)prove than others. With heuristic initialization, leaf nodes are heuristically initialized with proof and disproof numbers (Allis 1994). One example for this is using the number of legal moves at a leaf node, which can be calculated with little overhead in some games (Allis et al. 1994). Another popular approach is using domain-specific heuristic functions (Nagai 2002). One important remark is that the thresholds of df-pn must be increased to reduce the overhead of internal node re-expansions when heuristic initialization is combined with df-pn, as in df-pn+ (Nagai 2002; Kishimoto and M€ uller 2005). If more than one promising sibling exist and the search space does not fit into the TT, df-pn sometimes suffers from thrashing the TT by fast switches between subtrees. The 1 + e trick (Pawlewicz and Lew 2007) increases the threshold when

16

A. Kishimoto and M. Mueller

df-pn selects the best child. This enables df-pn to stay longer in the subtree of one child, without frequently switching to other promising siblings. Given a proven node n, tree simulation (Kawano 1996) performs a quick proof check for an unknown node m that looks similar to n. The similarity of positions is usually defined in a game-specific way. Unlike normal search such as df-pn that generates all legal moves, simulation is restricted to generate moves borrowed from n’s proof tree at each OR node and checks if a proof tree of m can be constructed in a similar way to n’s proof tree. If simulation succeeds, m is proven as well. Otherwise, m’s value remains unknown and normal search is performed. Considering that any search must examine all nodes in m’s proof tree even in an ideal case, simulation makes almost the smallest effort in proving m, therefore, requires much less effort than normal search. Schaeffer et al. (2007)’s PNS-based solver estimates the game-theoretic value of a node with high confidence by using scores from their αβ-search-based gameplaying program. Let s be such a score of node n, and th be a threshold for an iterative PNS algorithm. If s  th, n is regarded as a terminal node with value likely win. Similarly, if s  th, n is considered to be a likely loss. As in iterative deepening, the algorithm starts with a small value of th and gradually increases th after constructing each heuristic proof tree. When the root is solved with th = 1, a true proof tree is constructed, where all leaf nodes are true values, and the proof is complete.

Endgame Databases In many converging games (Allis 1994) such as chess and checkers, the number of pieces on the board decreases as the game progresses. This indicates that it is often feasible to enumerate all endgame positions that occur close to the end of the game. In this case, the game-theoretic values of these endgame positions can be precomputed and saved as endgame databases that map a position to its corresponding game-theoretic value. If the endgame databases contain the gametheoretic values of all the legal positions for a game, that game is strongly solved. As an example, the database for the game of Awari calculated by Romein and Bal (2003) contains the exact scores of all legal Awari positions. Retrograde analysis (Bellman 1965; Thompson 1986) is a standard backward search algorithm to systematically construct endgame databases, starting from terminal nodes and progressing toward the root. Retrograde analysis has been successfully used in many domains (Lake et al. 1994; Schaeffer et al. 2003; Romein and Bal 2003). Additionally, endgame databases contribute to significantly improving the ability of game solvers that employ forward search (Schaeffer et al. 2007), because forward search can obtain the game-theoretic value of a node without deeper search. Assume that the game-theoretic value of a game is either a win, a loss, or a draw, and a repeated position is defined to be a draw by the rule of that game. Then, one way to implement retrograde analysis is summarized as follows:

1

Game Solvers

17

1. Let S be a set containing all positions, that may include unreachable ones from the root. 2. Initialize: assign the game-theoretic value determined by the rules of the game to all terminal nodes and a value of unknown to all other positions. 3. For each node n  S with value unknown, check if n’s value can be determined from its children. For example, OR node n is a win if n has at least one child in S that is a win. 4. Repeat step 3 until the number of nodes with value unknown no longer decreases. 5. Mark the nodes of value unknown as draws since these nodes are either unreachable from the root, or no win or loss can be forced by either player. When constructing a database for a difficult game in practice, retrograde analysis requires large CPU and storage resources. Implementations typically use parallel machines as well as large amounts of both main memory and hard disk space. There are several approaches that achieve efficient parallelism and reduce disk I/O. For example, as in paging of the operating system, retrograde analysis determines which part of the databases should be preserved in the main memory to alleviate the overhead of disk I/O (Lake et al. 1994; Romein and Bal 2003). Schaeffer et al. (2003) not only compress databases space-efficiently but also decompress the databases in real-time so that forward search can use them both time- and memory-efficiently. Techniques to reduce the main memory requirement at the cost of a small computational overhead are presented in Lincke (2002); Romein and Bal (2003). To initiate parallelism, Romein and Bal (2003) present an asynchronous distributed-memory parallel algorithm that overlaps database computation and processor communication via network. In contrast, Lake et al. (1994) split the search space into a set of small slices that can be solved easily and independently. The correctness of databases must be verified, since both software and hardware errors including disk and network errors may occur during a large-scale computation (Schaeffer et al. 2003; Romein and Bal 2003).

Other Approaches This section briefly describes other approaches related to forward search.

Threat-Based Approaches Threats are moves to which a player must reply directly to win, or to avoid an immediate loss. If threats exist in a position, threat-space search (Allis 1994; Allis et al. 1996) considers only threats and safely ignores other legal moves. Threatspace search can significantly reduce the search space to explore, because the number of threats is usually much smaller than that of the legal moves. The idea

18

A. Kishimoto and M. Mueller

behind threat-space search is used in solvers such as Henderson et al. (2009) and Kishimoto and M€uller (2005). Different levels of threat sequences, which are a generalization of direct threats, can be detected by using pass moves. Intuitively, moves are limited by using the information on how many moves a player needs to make in a row so that that player has a forced win. Examples of such methods are λ-search (Thomsen 2000) and generalized threats search (Cazenave 2002). These techniques can be applied to both αβ and df-pn (Nagai 2002; Yoshizoe et al. 2007).

Early Win/Loss Detection Game-specific features can sometimes be used to statically detect whether a player is winning or losing. Such static win/loss detection can significantly reduce the size of a proof tree, as well as the search space explored by forward search. Examples include virtual connections in the game of Hex (Anshelevich 2002; Henderson et al. 2009) and the detection of eye space and potential eye space in the life and death problem in Go (Wolf 1994; Kishimoto and M€ uller 2005). Detecting a dominance relationship between positions can also contribute to recognizing wins or losses early. An important example is the checkmating problem in shogi (Japanese chess) (Seo 1999). For example, assume that forward search proves that node n is a win and then encounters unproven node m that dominates n. Then, m is also a win because the first player can copy the winning strategy from n.

Monte Carlo Tree Search Solver Monte Carlo Tree Search (MCTS) is a forward search algorithm that has achieved remarkable success in playing the game of Go and many other games where accurate evaluation functions are difficult to develop (see the chapter on computer Go, written by Yoshizoe and M€uller). MCTS combines tree search with Monte Carlo simulation that is used to evaluate a leaf node. In addition to Monte Carlo simulation results, if game-theoretic values of wins or losses are available, the MCTS-Solver propagates back these values (Winands et al. 2008). This modification enables MCTS-Solver to solve positions.

Probability Propagation Instead of using proof and disproof numbers as the criteria to explore an AND/OR tree, probability propagation (PP) performs best-first forward search based on a probability of a first player win for each node in an AND/OR tree (Pearl 1984; Stern et al. 2007). Enhancements similar to the ones for PNS can be incorporated into PP,

1

Game Solvers

19

including heuristic initialization (Stern et al. 2007), transposition tables, and two-level search (Saffidine and Cazenave 2013).

Results Accomplished on Solving Games The techniques described thus far contributed to solving many nontrivial, popular games that people actually play, such as Awari (Romein and Bal 2003), checkers (Schaeffer et al. 2007), Connect-Four (see http://tromp.github.io/c4/c4.html and (Allis 1988), and Go-Moku (Allis et al. 1996). Researchers use games with smaller boards as a research test bed to develop new algorithms. Examples of games solved in this way are 5  5 Go (van der Werf et al. 2003), 6  6 Othello (see http://www.tothello.com/), 10  10 Hex (Pawlewicz and Hayward 2014), and 5  6 Amazons (Song and M€uller 2014). In chess, endgame databases are constructed for most positions with 3–7 pieces (see http://chessok.com/?page_id=27966). Puzzle or endgame solvers based on PNS variants have been developed to achieve state-of-the-art performance in many games, such as chess (Breuker 1998), checkers (Schaeffer et al. 2007), Hex (Arneson et al. 2011), life and death in Go (Kishimoto and Muller 2005), and tsume-shogi (Seo et al. 2001; Nagai 2002; Kishimoto 2010). In particular, research on tsume-shogi has achieved remarkable success by testing with many difficult tsume-shogi problems created by human experts. State-of-the-art solvers based on df-pn can solve all the existing hard problems with solution sequences longer than 300 steps, including Microcosmos with 1525 steps. Sophisticated game solvers combine many of the techniques discussed. For example, the checkers solver of Schaeffer et al. (2007) incorporates PNS, df-pn, αβ, endgame databases, and many other game-independent and game-specific enhancements.

Conclusions This chapter presented an overview of the techniques for weakly or strongly solving games or game positions. Game positions can be solved by forward search, backward search, or a combination. As standard algorithms to perform forward search, the chapter introduced αβ and PNS variants. Backward search for building endgame databases uses retrograde analysis. Additionally, the chapter gave an overview of other approaches including threat-space search, MCTS and PP, followed by a summary of the results accomplished thus far on solving games.

Recommended Reading L.V. Allis, A Knowledge-Based Approach of Connect Four: The Game is Over, White to Move Wins. Master’s thesis, Vrije Universiteit Amsterdam, Amsterdam, 1988. Report No. IR-163

20

A. Kishimoto and M. Mueller

L.V. Allis, Searching for Solutions in Games and Artificial Intelligence. PhD thesis, University of Limburg, Maastricht, 1994 L.V. Allis, M. van der Meulen, H.J. van den Herik, Proof-number search. Artif. Intell. 66(1), 91–124 (1994) L.V. Allis, M.P.H. Huntjes, H.J. van den Herik, Go-moku solved by new search techniques. Comput. Intell. 12(1), 7–23 (1996) V. Anshelevich, A hierarchical approach to computer Hex. Artif. Intell. 134(1–2), 101–120 (2002) B. Arneson, R.B. Hayward, P. Henderson, Solving Hex: beyond humans, in Computers and Games 2010, ed. by H.J. van den Herik, H. Iida, A. Plaat. Lecture Notes in Computer Science (LNCS), vol. 6515 (Springer, Berlin, 2011), pp. 1–10 D. Beal, M.C. Smith, Multiple probes of transposition tables. ICCA J. 19(4), 227–233 (1996) R. Bellman, On the application of dynamic programming to the determination of optimal play in chess and checkers. Proc. Natl. Acad. Sci. U. S. A. 53, 244247 (1965) D.M. Breuker, Memory Versus Search in Games. PhD thesis, Universiteit Maastricht, Maastricht, 1998 M. Campbell, The graph-history interaction: on ignoring position history, in Proceedings of the 1985 ACM Annual Conference on the Range of Computing: Mid-80’s Perspective, (ACM, New York, 1985). pp. 278–280 M. Campbell, A.J. Hoane Jr., F. Hsu, Deep Blue. Artif. Intell. 134(1–2), 57–83 (2002) T. Cazenave, A generalized threats search algorithm, in Computers and Games 2002, ed. by J. Schaeffer, M. M€uller, Y. Bjo¨rnsson. Lecture Notes in Computer Science (LNCS) (Springer, Heidelberg, 2002), pp. 75–87 A. de Bruin, W. Pijls, A. Plaat, Solution trees as a basis for game tree search. ICCA J. 17(4), 207–219 (1994) R. Gasser, Solving Nine Men’s Morris, in Games of No Chance, ed. by R.J. Nowakowski. MSRI Publications, vol. 29 (Cambridge University Press, Cambridge, 1996), pp. 101–113 R. Greenblatt, D. Eastlake, S. Croker, The Greenblatt chess program, in Proceedings of the Fall Joint Computer Conference, 1967, pp. 801–810. Reprinted (1988) in Computer Chess Compendium, ed. by D.N.L. Levy (Batsford, London), pp. 56–66 P. Henderson, B. Arneson, R.B. Hayward, Solving 8  8 Hex, in Proceedings of the 21st International Joint Conference on Artificial Intelligence (IJCAI’09), ed. by C. Boutilier (AAAI Press, Pasadena, 2009), pp. 505–510 T. Kaneko, Parallel depth first proof number search, in Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, (AAAI’10), ed. by M. Fox, D. Poole (AAAI Press, Menlo Park, 2010), pp. 95–100 Y. Kawano, Using similar positions to search game trees, in Games of No Chance, ed. by R.J. Nowakowski. MSRI Publications, vol. 29 (Cambridge University Press, Cambridge, 1996), pp. 193–202 A. Kishimoto, Dealing with infinite loops, underestimation, and overestimation of depth-first proof-number search, in Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, (AAAI’10), ed. by M. Fox, D. Poole (AAAI Press, 2010) A. Kishimoto, M. M€uller, About the completeness of depth-first proof-number search, in Computers and Games 2008, ed. by H.J. van den Herik, X. Xu, Z. Ma, M.H.M. Winands. Lecture Notes in Computer Science, vol. 5131 (Springer, Heidelberg, 2008), pp. 146–156 A. Kishimoto, M. M€ uller, Df-pn in Go: an application to the one-eye problem, in Advances in Computer Games 10 (ACG’03): Many Games, Many Challenges, ed. by H.J. van den Herik, H. Iida, E.A. Heinz (Kluwer, Boston, 2003), pp. 125–141 uller, Search versus knowledge for solving life and death problems in Go, in A. Kishimoto, M. M€ Proceedings of the 20th National Conference on Artificial Intelligence (AAAI’05), ed. by M.M. Veloso, S. Kambhampati (AAAI Press/MIT Press, Menlo Park, 2005), pp. 1374–1379 A. Kishimoto, M. Winands, M. M€uller, J.-T. Saito, Game-tree search using proof numbers: the first twenty years. ICGA J. 35(3), 131–156 (2012)

1

Game Solvers

21

D.E. Knuth, R.W. Moore, An analysis of alpha-beta pruning. Artif. Intell. 6(4), 293–326 (1975) R. Lake, J. Schaeffer, P. Lu, Solving large retrograde analysis problems using a network of workstations. Advances in Computer Games 7, 135–162 (1994) T. Lincke, Exploring the Computational Limits of Large Exhaustive Search Problems. PhD thesis, ETH Z€ urich, 2002 T. Marsland, A review of game-tree pruning. ICCA J. 9(1), 3–19 (1986) A. Nagai, A new depth-first-search algorithm for AND/OR trees. Master’s thesis, The University of Tokyo, Tokyo, 1999 A. Nagai, Df-pn Algorithm for Searching AND/OR trees and its Applications. PhD thesis, The University of Tokyo, 2002 J. Nash, Some games and machines for playing them. Technical Report D-1164, Rand Corp., 1952 A.J. Palay, Searching with Probabilities. PhD thesis, Carnagie Mellon University, 1983 J. Pawlewicz, R. Hayward, Scalable parallel depth first proof number search, in Computers and Games (CG 2013), vol. 8427. Lecture Notes in Computer Science (Springer, 2014), pp. 138–150 J. Pawlewicz, L. Lew, Improving depth-first pn-search: 1+ε trick, in Proceedings of the 5th Computers and Games Conference (CG’06), ed. by H.J. van den Herik, P. Ciancarini, H.H.L.M. Donkers. Lecture Notes in Computer Science, vol. 4630 (Springer, Heidelberg, 2007), pp. 160–170 J. Pearl, Heuristics: Intelligent Search Strategies for Computer Problem Solving (AddisonWesley, Reading, 1984) J.W. Romein, H. Bal, Solving the game of Awari using parallel retrograde analysis. IEEE Comput. 36(10), 26–33 (2003) A. Saffidine, T. Cazenave, Developments on product propagation. in Computer Games 2013, vol. 8427. Lecture Notes in Computer Science, 2013, pp. 100–109 J. Schaeffer, The history heuristic and alpha-beta search enhancements in practice. IEEE Trans. Pattern Anal. Mach. Intell. 11(1), 1203–1212 (1989) J. Schaeffer, Y. Bjo¨rnsson, N. Burch, R. Lake, P. Lu, S. Sutphen, Building the checkers 10-piece endgame databases, in Advances in Computer Games 10: Many Games, ed. by H.J. van den Herik, H. Iida, E.A. Heinz. (Kluwer, Boston, 2003), pp. 193–210 J. Schaeffer, N. Burch, Y. Bjo¨rnsson, A. Kishimoto, M. M€ uller, R. Lake, P. Lu, S. Sutphen, Checkers is solved. Science 317(5844), 1518–1522 (2007) M. Seo, On effective utilization of dominance relations in tsume-shogi solving algorithms, in Proceedings of the 8th Game Programming Workshop, 1999, pp. 137–144 (in Japanese) M. Seo, H. Iida, J.W.H.M. Uiterwijk, The PN*-search algorithm: Application to tsume shogi. Artif. Intell. 129(1–2), 253–277 (2001) D.J. Slate, L.R. Atkin, Chapter 4. Chess 4.5 – Northwestern University chess program, in Chess Skill in Man and Machine, ed. by P.W. Frey (Springer, New York, 1977), pp. 82–118 J. Song, M. M€ uller, An enhanced solver for the game of Amazons, 2014, in Accepted for IEEE Transactions on Computational Intelligence and AI in Games (TCIAIG), 12 pp D. Stern, R. Herbrich, T. Graepel, Learning to solve game trees, in Proceedings of the 24th International Conference of Machine Learning (ICML), 2007, pp. 839–846 K. Thompson, Retrograde analysis of certain endgames. ICCA J. 9(3), 131–139 (1986) T. Thomsen, Lambda-search in game trees – with application to Go. ICGA J. 23(4), 203–217 (2000) Y. Tsuruoka, D. Yokoyama, T. Chikayama, Game-tree search algorithm based on realization probability. ICGA J. 25(3), 132–144 (2002) E.C.D. van der Werf, H.J. van den Herik, J.W.H.M. Uiterwijk, Solving Go on small boards. ICGA J. 26(2), 92–107 (2003) M.H.M. Winands, J.W.H.M. Uiterwijk, H.J. van den Herik, An effective two-level proof-number search algorithm. Theor. Comput. Sci 313(3), 511–525 (2004) M.H.M. Winands, Y. Bjo¨rnsson, J.-T. Saito, Monte-Carlo tree search solver, in Proceedings of the 6th Computers and Games Conference (CG’08), ed. by H.J. van den Herik, X. Xu, Z. Ma,

22

A. Kishimoto and M. Mueller

M.H.M. Winands. Lecture Notes in Computer Science, vol. 5131 (Springer, Berlin, 2008), pp. 25–36 T. Wolf, The program GoTools and its computer-generated tsume Go database, in 1st Game Programming Workshop in Japan (Hakone), 1994 K. Yoshizoe, A. Kishimoto, M. M€uller, Lambda depth-first proof number search and its application to Go, in Proceedings of the 20th International Joint Conference on Artificial Intelligence (IJCAI-07), ed. by M.M. Veloso (2007), Morgan Kaafmann, San Francisco, pp. 2404–2409

2

General Game Playing Yngvi Björnsson and Stephan Schiffel

Contents Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Game Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Game Description Language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Reasoning in GDL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Traditional Game-Tree Search in GGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Monte Carlo Tree Search in GGP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Learning Simulation Search Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Move-Average Sampling Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Predicate-Average Sampling Technique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Features-to-Action Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N-Gram Selection Technique and Last Good Reply Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Learning Heuristic Evaluation Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ClunePlayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FluxPlayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . KuhlPlayer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Other Agents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Learning Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . GGP Competitions and Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Competitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Workshops and Online Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Recommended Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

24 25 25 27 28 28 28 29 31 31 32 33 33 35 36 36 37 38 39 39 40 40 41 42 42

Y. Björnsson (*) • S. Schiffel School of Computer Science, Reykjavik University, Menntavegur, Reykjavík, Iceland e-mail: [email protected]; [email protected] # Springer Science+Business Media Singapore 2017 R. Nakatsu et al. (eds.), Handbook of Digital Games and Entertainment Technologies, DOI 10.1007/978-981-4560-50-4_34

23

24

Y. Björnsson and S. Schiffel

Abstract

Game playing is one of the oldest areas of investigation in artificial intelligence (AI) and has been at the forefront of AI research ever since the birth of the first computers, over half a century ago. The research focus was initially on developing general approaches for game playing, but gradually shifted towards building high-performance game-playing systems capable of matching wits with the strongest humans in the world in individual games. To renew interest in more general approaches to computer game playing, the AI community established the International General Game Playing Competition (IGGPC) in 2005, which has run annually ever since. General game playing (GGP) has in the decade since established itself as a fascinating research area, posing numerous interesting research challenges to a wide range of artificial intelligence subdisciplines. In here, we review the progress made in the field so far and highlight mainstay techniques used in contemporary state-of-the-art GGP agents. Keywords

Artificial neural networks (ANN) • Competitions and events • Custom-made interpreters • Game description language (GDL) • Game specifications • Gametree search in • Game-tree search, in GGP • GDL. See Game description language (GDL) • General game playing (GGP) • GGP. See General game playing (GGP) • Goblin and Ogre • KuhlPlayer • Learning heuristic evaluation functions • Learning simulation search control • Magline agent • Monte Carlo Tree Search in GGP • Reasoning

Introduction In General game playing (GGP) the goal is to create intelligent agents that can automatically learn how to skillfully play a wide variety of games, given only the descriptions of the game rules. This requires that the agents learn diverse gameplaying strategies without any game-specific knowledge being provided by their developers. A successful realization of this task poses interesting research challenges for artificial intelligence subdisciplines such as knowledge representation, agentbased reasoning, heuristic search, computational intelligence, and machine learning. The two core components of any game-playing program are search and evaluation. The former provides the ability to think ahead, whereas the latter provides a mechanism for assessing the merits of the game positions that arise during the search. Domain-dependent knowledge plays an important part in both components, in particular for game-position evaluation, understandably, but also for providing effective search guidance. The main challenge faced by general game-playing systems, as opposed to game-playing systems for playing one specific game, is that the relevant game-specific knowledge required for expert-level play, whether for the search or the evaluation, must be effectively discovered during game play. This distinction provides unique challenges for GGP research.

2

General Game Playing

25

In this chapter, we give an overview of the field of GGP. It has been an active research area over the last decade, or ever since the establishment of the International GGP Competition (IGGPC) (Genesereth and Love 2005) revitalized interest into general problem-solving approaches to computer game playing. The field has advanced steadily since and noteworthy research progress been made. However, given the large body of recent research into GGP systems, we out of necessity limit our overview for the most part to research results and techniques that have demonstrated themselves effective in competitive GGP agents. Main results are highlighted and references to more in-depth discussions given were relevant. The chapter is structured as follows. In the next section, we show how games are specified in GGP and describe common approaches for representing and interpreting the game specifications. In the section that follows, we give an overview of the search techniques used in GGP agents for thinking ahead and highlight the unique challenges they face, in particular the need to autonomously learn relevant gamespecific knowledge on the fly. The next three sections discuss in more detail how the agents learn search control, evaluation functions, and game structures, respectively. Finally, before concluding, we review competition results and provide references to useful online resources.

Game Specifications GGP agents are capable of playing many different games. The rule set of a game to play is communicated to the agents in a formal language called game description language (GDL) and the agents then have a built-in mechanism to interpret and reason with the game description, enabling them to play a legitimate game.

Game Description Language The game description language (GDL) (Genesereth et al. 2005; Love et al. 2008) is a declarative first-order logic-based language strongly resembling Datalog (Abiteboul et al. 1995), allowing function constants, negation, and recursion (in a restricted form). Its syntax, however, is adopted from the knowledge interface format (KIF) language (Genesereth et al. 1992). The expressiveness of GDL allows for a wide range of deterministic, perfect information, single- or multiplayer games to be described. By definition, all games are simultaneous-move based; however, turn-based games can be modeled by having the players who do not have the turn return a special no-operation move (noop). Special relations are used to specify the setup of the initial game state, the rules for generating and executing legal moves in a given state, as well as rules for detecting and scoring terminal states. A game state is defined by the set of propositions that are true in that state (using the closed-world assumption, that is, the only true facts are those that are known to be so, others are assumed false). Figure 1 shows an example game written in GDL.

26

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36

Y. Björnsson and S. Schiffel (role xplayer) (role oplayer) (init (init (init (init

(cell 1 1 b)) (init (cell 1 2 b)) (init (cell 1 3 b)) (cell 2 1 b)) (init (cell 2 2 b)) (init (cell 2 3 b)) (cell 3 1 b)) (init (cell 3 2 b)) (init (cell 3 3 b)) (control xplayer))

(