Cyborg Futures: Cross-disciplinary Perspectives On Artificial Intelligence And Robotics 303021835X, 9783030218355, 9783030218362

Table of contents :
Acknowledgements......Page 7
Contents......Page 8
Notes on Contributors......Page 10
List of Figures......Page 13
Chapter 1: Introduction......Page 16
References......Page 30
Chapter 2: Evolution Ain’t Engineering: Animals, Robots, and the Messy Struggle for Existence......Page 31
Evolution Suffices......Page 32
A History of Struggles: Darwin’s Finches......Page 34
Modeling the Struggles: Evolutionary Biorobotics......Page 38
Evolution Ain’t Engineering......Page 46
References......Page 47
Chapter 3: Demystifying the Intelligent Machine......Page 49
Relocating Imaginaries of Machine Intelligence......Page 50
Generating Robotic Progeny......Page 53
Where Is Deep Blue?......Page 55
Bodies in Relation
......Page 56
Slow Robots and Slippery Rhetorics......Page 59
Domesticating Robot Labors......Page 62
The Labors of Violence......Page 65
Conclusion......Page 69
References......Page 73
Introduction......Page 76
Rationality and Autonomy in the AI Risk Discourse......Page 78
AI and the Illusion of Transcendent Rationality......Page 81
Capitalist Context of Rationality......Page 84
Capitalist Subjects, Transcendental Rationality......Page 88
Bostrom’s Occult Motivations of AI Machines......Page 90
Saving Transcendent Intelligence, Abandoning Earth......Page 94
References......Page 98
Introduction......Page 102
Mind the Gap......Page 104
Swarming as Natural Inspiration for Military Futures......Page 110
Emergent Stupidity......Page 113
Conclusion......Page 119
References......Page 123
Introduction......Page 126
What Is Ethics?......Page 128
Why Ethics Now?......Page 130
Why Robots and AI and Ethics?......Page 133
Corporate Anthropomorphism......Page 135
Why We Need a Feminist Ethics of Robots and AI......Page 136
Conclusion......Page 137
References......Page 138
Chapter 7: Fiction Meets Science: Ex Machina, Artificial Intelligence, and the Robotics Industry......Page 140
References......Page 153
Chapter 8: Rossum’s Mimesis......Page 154
Rossum’s tekhne......Page 157
Mimetics and Politics......Page 160
Mimetic Arts: Shifting Genres......Page 162
Utopian Vision: A Singularity?......Page 164
References......Page 171
Chapter 9: Race and Robotics......Page 172
References......Page 184
Index......Page 185

Citation preview

SOCIAL AND CULTURAL STUDIES OF ROBOTS AND AI

Cyborg Futures Cross-disciplinary Perspectives on Artificial Intelligence and Robotics

Edited by Teresa Heffernan

Social and Cultural Studies of Robots and AI Series Editors Kathleen Richardson Faculty of Computing, Engineering, and Media De Montfort University Leicester, UK Cathrine Hasse Danish School of Education Aarhus University Copenhagen, Denmark Teresa Heffernan Department of English St. Mary’s University Halifax, NS, Canada

This is a groundbreaking series that investigates the ways in which the “robot revolution” is shifting our understanding of what it means to be human. With robots filling a variety of roles in society—from soldiers to loving companions—we can see that the second machine age is already here. This raises questions about the future of labor, war, our environment, and even human-to-human relationships. More information about this series at http://www.palgrave.com/gp/series/15887

Teresa Heffernan Editor

Cyborg Futures Cross-disciplinary Perspectives on Artificial Intelligence and Robotics

Editor Teresa Heffernan Department of English St. Mary’s University Halifax, NS, Canada

ISSN 2523-8523     ISSN 2523-8531 (electronic) Social and Cultural Studies of Robots and AI ISBN 978-3-030-21835-5    ISBN 978-3-030-21836-2 (eBook) https://doi.org/10.1007/978-3-030-21836-2 © The Editor(s) (if applicable) and The Author(s), under exclusive licence to Springer Nature Switzerland AG 2019 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the ­publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and ­institutional affiliations. Cover illustration: weerayut ranmai / Alamy Stock Photo This Palgrave Macmillan imprint is published by the registered company Springer Nature Switzerland AG. The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

In the later stages of dementia, my mother asked me what a social robot was, and when I explained that they were humanoid machines designed for, among other things, a care role, she, who had wonderful human carers, responded, “Oh, that is the loneliest thing I have ever heard.” In memory of my mother, Geraldine Joan Heffernan, 1923–2018.

Acknowledgements

This collection began as a workshop at St. Mary’s University (SMU) in Halifax. It would not have been possible without the support of the University and the Social Science and Humanities Research Council, for which I am very grateful. Nor would it have been possible without the generosity of the invited speakers who were open and willing to come together to converse with other disciplines and contribute to the ongoing discussions about the futures of robotics and artificial intelligence. The students in my Cyborg Futures class at SMU were wonderful at helping with the organization of the workshop and also keen participants in the discussions. I would also like to thank the academic community and members of the public who attended, physically and virtually. Karen Asp, a PhD candidate in the Faculty of Environmental Studies at York University and Social Robot Future’s web designer, deserves a special thanks. She was invaluable in helping with the planning of the workshop and the editing and indexing of this collection. My thanks also go to the blind reviewers of the manuscript and to the production team and editorial staff at Palgrave Macmillan, in particular, Rachel Nature Daniel and her assistant Madison Allums.

vii

Contents

1 Introduction  1 Teresa Heffernan 2 Evolution Ain’t Engineering: Animals, Robots, and the Messy Struggle for Existence 17 John H. Long Jr. 3 Demystifying the Intelligent Machine  35 Lucy Suchman 4 Autonomy of Artificial Intelligence, Ecology, and Existential Risk: A Critique 63 Karen Asp 5 Visions of Swarming Robots: Artificial Intelligence and Stupidity in the Military-­Industrial Projection of the Future of Warfare 89 Patrick Crogan 6 The Business of Ethics, Robotics, and Artificial Intelligence113 Kathleen Richardson

ix

x 

CONTENTS

7 Fiction Meets Science: Ex Machina, Artificial Intelligence, and the Robotics Industry127 Teresa Heffernan 8 Rossum’s Mimesis141 Jennifer Keating and Illah Nourbakhsh 9 Race and Robotics159 Louis Chude-Sokei Index173

Notes on Contributors

Karen Asp  is a PhD candidate in the Faculty of Environmental Studies at York University. She holds an MA in Social and Political Thought (Acadia University) and an MA in Geography (Simon Fraser University). Her doctoral dissertation draws on Adornian critical theory and Marxian political economy to analyze how capitalist expansion and technological progress are naturalized in contemporary “Anthropocene” narratives, particularly with respect to planetary-scale ecological catastrophe. She was the research assistant for the Social Robot Futures project, which culminated in the Cyborg Futures Workshop and this collection. Louis Chude-Sokei  is Professor of English, the George and Joyce Wein Chair of African American Studies and Director of the African American Studies Program at Boston University. His work includes the award-­ winning The Last Darky: Bert Williams, Black on Black Minstrelsy and the African Diaspora (2005) and The Sound of Culture: Diaspora and Black Technopoetics (2015). He is also Editor-in-Chief of The Black Scholar, one of the oldest and currently leading journal of Black Studies in the United States. Patrick Crogan  is Associate Professor of Digital Culture at the University of the West of England, Bristol. He wrote Gameplay Mode: War, Simulation and Technoculture (2011) and has had numerous essays published on digital media and cultural theory. He also edited a special issue of Cultural Politics (2010) on Bernard Stiegler and wrote the Oxford Literary and Critical Theory Bibliography entry on Stiegler (2017). xi

xii 

NOTES ON CONTRIBUTORS

Teresa  Heffernan is Professor of English at St. Mary’s University, Halifax, Nova Scotia, Canada. She is author of Post-Apocalyptic Culture: Modernism, Postmodernism, and the Twentieth-Century Novel (2008) and Veiled Figures: Women, Modernity, and the Spectres of Orientalism (2016). She is coeditor (with Daniel O’Quinn) of a critical edition of Lady Mary Wortley Montagu’s The Turkish Embassy Letters (2012). She is series editor, with Reina Lewis, of Cultures in Dialogue developed with Social Sciences and Humanities Research Council research funding. She is coeditor (with Jill Didur) of a special issue of Cultural Studies entitled “Revisiting the Subaltern in the New Empire” and of a special issue of Cultural Critique (with Jill Didur and Bart Simon) on “Posthumanism.” Her articles have appeared in journals such as Studies in the Novel, Eighteenth-Century Studies, Arab Journal for the Humanities, Subject Matters, Canadian Literature, and Twentieth Century Literature. Jennifer Keating  is Assistant Dean for Educational Initiatives in Dietrich College, Carnegie Mellon University, where she also teaches regularly in the Department of English. Her research interests include representations of colonialism, nationalism, and gender relations in nineteenth-century British literature and twentieth-century Anglophone literature, primarily in Ireland and parts of the Caribbean. She also pursues work pertaining to memoir, legacies of violence, and identity formation in societies in strife. John H. Long Jr.  holds a joint appointment as Professor of Biology and Cognitive Science at Vassar College on the John G. Vassar Chair of Natural History. He currently serves as Chair of the Department of Cognitive Science and Director of the Interdisciplinary Robotics Research Laboratory. He is author of Darwin’s Devices: What Evolving Robots Can Teach Us About the History of Life and the Future of Technology (2012). He is also a professor for The Great Courses, having written and performed “Robotics.” Illah  Nourbakhsh is Professor of Robotics and Director of the Community Robotics, Education and Technology Empowerment (CREATE) Lab and head of the Robotics Master’s Program in The Robotics Institute at Carnegie Mellon University. His research projects explore community-based robotics, including educational and social robotics and ways to use robotic technology to empower individuals and communities. He is a member of the Global Future Council on the Future of AI and Robotics for the World Economic Forum, the IEEE

  NOTES ON CONTRIBUTORS 

xiii

Global Initiative for the Ethical Considerations in the Design of Autonomous Systems, and the Global Innovation Council of the Varkey Foundation, and Senior Advisor to The Future Society, Harvard Kennedy School. He is author of Robot Futures (2013) and Parenting for Technology Futures (2015). Kathleen Richardson  is Professor of Ethics and Culture of Robots and AI at De Montfort University in Leicester, UK, and part of the Europe-­ wide DREAM project (Development of Robot-Enhanced Therapy for Children with AutisM). She is also cofounder and director of the Campaign Against Sex Robots. Her books include An Anthropology of Robots and AI: Annihilation Anxiety (2015) and Machines and Challenging Sociality: An Anthropology of Robots, Autism, and Attachment (2018). Lucy Suchman  is Professor of Anthropology of Science and Technology in the Department of Sociology at Lancaster University, UK. Before taking up her present post she was a Principal Scientist at Xerox’s Palo Alto Research Center, where she spent 20 years as a researcher. She is author of Human-Machine Reconfigurations (2007).

List of Figures

Fig. 2.1

Fig. 2.2

Fig. 2.3

Isla Genovesa, a young, small island in the Galapagos archipelago of Ecuador. On Darwin Bay Beach, saltbush, Cryptocarpus pyriformes, are evergreen shrubs that pioneer the beaches (top). Inland, the arid zone fields yellow cordia, Cordia lutea, a common flowering shrub (lower left). Cacti, like the prickly pear, Opuntia sp. (lower right), are abundant and are eaten by animals; if you look closely, you can see a cactus finch in the center. (All images were taken by the author on 9 August 2002) The struggle for existence of finches in the Galapagos Islands. Finches on Isla Genovesa forage for seeds (top) and, when found, attempt to eat them (bottom). Sharp-beaked ground finch, Geospiza difficilus, is on the left; large ground finch Geospiza magnirostris is on the right. Rosemary and Peter Grant have shown that finches with larger, deeper bills are better at eating the tough seeds available during drought years; finches with smaller, sharper bills are better at quickly collecting and processing the small seeds available during wet years. (All images were taken by the author on 9 August 2002) Evolutionary biorobotics, a life-cycle approach with humans in the loop. Robots do not spontaneously or autonomously evolve. Humans determine almost everything about the individual robots and their world. But the robots enact their own behavior autonomously, without the remote-control help of a human. Autonomous behavior is the final element in the life cycle of a robot, it determines the evolutionary fitness of that individual, and it produces the response variables in experiments that we use to test biological hypotheses

22

23

25 xv

xvi 

List of Figures

Fig. 2.4

Fig. 2.5

Fig. 2.6

Biorobotic models. The class of biorobots known as Tadros (bottom left) are physical models of fish such as the extinct jawless fish, Drepanaspis (upper left). The figure here is an artistic morphing of the animal into its biorobotic model, serving to show the similarities in shape, size, and manner of swimming. Tadros operate using sensorimotor mechanisms used by fish: a flapping, flexible tail generates thrust; sensors detect light gradients and the proximity of predators; neural circuitry decides when to escape from predators and override the goal of seeking light. This version of Tadro was used to test hypotheses about the evolution of the vertebral column in the first, extinct vertebrates (Roberts et al. 2014) Tadro class, model T15A biorobot. Physically embodied and behaviorally autonomous, this Tadro was designed with a nonevolving body morphology and an evolving, genetically encoded neural network (Livingston et al. 2016). This system, which includes a simple aquatic environment with a single light source, tested the hypothesis that modular substructures of the neural network would evolve under selection for enhanced light seeking. Tadro T15A swims on the surface of the water, using its eyespots to sense a centralized light source. Depending on how those light sensors (labeled “light left” and “light right,” in the neural network diagram, above) are connected to the motor output (labeled “offset” and “frequency”) determines how the Tadro behaves in different places in the environment. The neural network has 60 possible connections (lines with arrows), and each connection may be excitatory or inhibitory. Each connection is represented by a gene in a genetic system that undergoes mutation and asexual reproduction. Every generation of Tadros consists of ten individuals, each with a different pattern of connections in the neural network. (This image is modified, under the CC-BY license, from Livingston et al. (2016)) A Tadro with a randomly generated neural network from the ancestral generation. This is a simple aquatic world, a 3-meter wide tank, with a single light source. As shown in snapshots, over five minutes, this individual swam but collected light only once, giving it a low evolutionary fitness (integration of the green line in the graph of light and time). Because of the oblique view, the center of the tank, with the highest intensity of light, is indicated with the white ellipse. (This image is modified, under the CC-BY license, from Livingston et al. (2016). Color figure online)

26

27

28

  List of Figures 

Fig. 2.7 The Tadro with the best neural network, evolved after nine generations of selection. This individual actively sought the light, orbiting four times through the center of the tank (white ellipse) in five minutes. Its evolutionary fitness, as shown by the light harvesting graph, was five times greater than that of the Tadro shown from the first generation (Fig. 2.6). (This image is modified, under the CC-BY license, from Livingston et al. (2016)) Fig. 2.8 Descent with modification in a population of physical robots. The population began with ten randomly generated neural networks (bottom row) that connected light sensors to the tail motor in the behaviorally autonomous Tadro (see Figs. 2.6 and 2.7). Individuals were chosen to reproduce asexually with a probability proportional to their evolutionary fitness, which was determined by how well they gathered light. Individuals that reproduced had their genome, which coded for the neural network, mutated. Those mutations produced new genomes that were used to create offspring with different neural networks. Only four individuals from the first generation reproduced, and only two of those had descendants in the final, tenth generation. (This image is modified, under the CC-BY license, from Livingston et al. (2016)) Fig. 2.9 The body and the neurocontroller interact continuously to enact intelligent, autonomous behavior. Inspired by Pfeifer and Bongard (2006), this diagram shows a causal loop typical of a dynamical systems approach to understanding the operations of a real-world system such as a physical robot. While we typically think of the neurocontroller as a brain-like overlord for behavior, it must interact with the world by giving instructions to actuators, like the motor driving the tail of Tadro. The Tadro moves in response to those instructions in a way that is governed by the laws of physics for the motion of bodies. As the robot moves, it generates sensory information about its movements in relation to the world that informs, through the sensors, the neurocontroller. These interactions occur continually Fig. 3.1 Setup for the Turing test. (By Hugo Férée – Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index. php?curid=17059503. Accessed December 24, 2017) Fig. 3.2 Third Convention on Conventional Weapons meeting on lethal autonomous weapons systems, Geneva, April 12, 2016. (Photo credit: Peter Asaro, International Committee for Robot Arms Control)

xvii

29

30

31 38

54

CHAPTER 1

Introduction Teresa Heffernan

I wished to write a comedy, partly of science, partly of truth. The odd inventor, Mr. Rossum (whose name translated into English signifies “Mr. Intellectual” or “Mr. Brain”), is a typical representative of the scientific materialism of the last century. His desire to create an artificial man—in the chemical and biological, not the mechanical sense—is inspired by a foolish and obstinate wish to prove God unnecessary and absurd. Young Rossum is the young scientist, untroubled by metaphysical ideas; scientific experiment to him is the road to industrial production. He is not concerned to prove but to manufacture …. Those who think to master the industry are themselves mastered by it; Robots must be produced although they are a war industry, or rather because they are a war industry. (Karel Č apek 1923)

The nineteenth-century optimism about science and technology started to dim after the catastrophic Great War. The brutal machinery of modern warfare and the dehumanizing mechanization of life were the subjects of Karel Cˇ apek’s 1920s play R.U.R. (Rossum’s Universal Robots), where the term robot was first coined, derived from the Czech word robota, meaning servitude or forced labor. The development of the atomic bomb during the Second World War did nothing to allay fears about the abuses of technology and the view that science had outpaced humanity. Optimism surged T. Heffernan (*) Department of English, St. Mary’s University, Halifax, NS, Canada e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_1

1

2 

T. HEFFERNAN

briefly in the 1990s with the hope that the Internet would connect the world and foster democracy, but that dream quickly soured as corporations like Facebook, Google, and Amazon took over the online world, and powerful government agencies began spying on their citizens. China, for instance, has been explicit about its embrace of algorithmic governance and mass surveillance and plans to implement a rating system for its citizens enabled by image and voice recognition technologies. Many of the early inventors and enthusiasts of the web, Tim Berners-Lee, Jaron Lanier, and Sherry Turkle, among them, are profoundly disturbed by the toxic turn of the Internet and online culture, which is only heating up in the race amongst superpowers to dominate the field of artificial intelligence (AI) technology that has been completely uncoupled from any sense of social progress. In an interview, the historian Jill Lepore discusses the contemporary tech world and notes: Disruption emerges in the 1990s as progress without any obligation to notions of goodness. And so ‘disruptive’ innovation, which became the buzzword of change in every realm in the first years of the 21st century, including higher education, is basically destroying things because we can and because there can be money made doing so. Before the 1990s, something that was disruptive was like the kid in the class throwing chalk. And that’s what disruptive innovation turned out to really mean. A little less disruptive innovation is called for. (Goldstein 2018)

Discussions about AI and robotics have largely been shaped by computer science and engineering, steered by corporate and military interests, often underscored by transhumanist philosophy and libertarian politics, animated by fiction, and hyped by the media. One of the issues that makes headlines and dominates popular discussions is how to ensure this technology proves miraculous rather than catastrophic for humanity. Interconnected boards and institutes have sprung up to guide its development: Google has promised to establish an ethics board (though it has not disclosed its members), and the Machine Intelligence Research Institute (formerly the Singularity Institute for Artificial Intelligence) in Silicon Valley; the Future of Humanity Institute at Oxford; the Centre for the Study of Existential Risk at Cambridge, England; and the Future of Life Institute at Cambridge, Massachusetts—have all been established to track the “existential risk” AI poses.

1 INTRODUCTION 

3

Yet many of the founders and funders that dominate these institutes and boards (such as Ray Kurzweil, Nick Bostrom, and Max Tegmark) are transhumanists—those who believe in the radical “uplifting” of the human condition through technological enhancement; the inevitability of strong or “general” AI; the emergence of an autonomous machine superintelligence that will exceed human intelligence; the technological singularity, the merging of man and machine; and immortality. In other words, we have a classic case of the foxes guarding the chicken coop: those overseeing the technology to prevent it from going rogue are also those heavily invested in the idea of its potential for dramatically transforming humanity for the better. However, many of the underlying premises that drive the research remain, for the most part, unexamined: it proceeds with a religious-­like faith in technology; it assumes that brains and computers, the biological and the machinic, the technological and the natural are interchangeable; it privileges instrumental reason and algorithmic logic, big data, computing speeds, and efficiency; it promotes itself as a future that will radically break from the past but never imagines itself as an archeological relic; it positions humans as autonomous machinelike managers and masters of the cosmos rather than animals that are dependent on the planet. Moreover, the question of whether at some future point this technology will prove a threat or a blessing distracts us from the infrastructure currently impacting virtually all domains of life—the economic, political, social, cultural, and environmental. Big data results and propriety black box solutions replicate biases. Social media platforms that harvest and monetize personal data, steal private data and turn it into proprietary knowledge, and circulate sensationalist clickbait headlines and propaganda to targeted audiences have violated privacy, sucked up advertisement revenue, eroded the independent press, and encouraged trolls and tribalism, which in turn have led to the degradation of political discussions and the undermining of democracy. In the rush “forward,” older computer languages and technologies are rendered obsolete by new ones even as many critical systems are structured by the former. The military, which funds a great deal of the research, is developing highly controversial autonomous weapons, hoping to automate war, despite vocal protests from many in the field. Big corporations, like Google, Amazon, Apple, and Facebook that are heavily investing in AI and robotics, are virtual monopolies. Contributing to wealth concentration, they avoid taxes even as they benefit from publically-funded infrastructure and hire relatively few people, all

4 

T. HEFFERNAN

the while making record profits. The short shelf life of electronic devices, with their built-in obsolescence, is exhausting natural resources and contaminating landfills with millions of tons of toxic e-waste, a problem that will continue to grow with the industry push for a robotic labor force. These problems are only the tip of the iceberg that we are already experiencing in the blind race for AI and robots. While much of AI/robotics research takes place within disciplinary silos, the impact of AI/robotics is wide-ranging and needs to be considered in a much broader transdisciplinary context. John McCarthy first coined the term “artificial intelligence” in 1956 at the Dartmouth Summer Research Project workshop on machine computation. The purpose was to explore “the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.” What was conjecture has now become gospel for many despite the fact that McCarthy himself regretted coining the term. He wished he had named the field “computational intelligence,” which perhaps would have better checked the almost automatic conflation of the concepts of the “human” and the “machine” that is so prevalent and problematic in discussions of AI.  If, from its inception, the field has been driven by the misleading premise that machines will master and exceed human intelligence, the term computation—“a calculation involving numbers or quantities”—better qualifies the very limited range of “intelligence” that the field in fact covers. At its core, AI is the attempt to get systems to act or think like animals/humans. In reality, it is a combination of math and engineering that involves using computer algorithms, a specific set of unambiguous instructions in a specific order, applied to information that has been turned into data (ones and zeros). In practice, the field is mostly concerned with building profitable artifacts and is unconcerned with abstract definitions of intelligence. Many myths have distorted the field, making it difficult to assess the problems and potential of the technology. One: references to fiction often problematically structure the debates about AI and robotics. A 2015 European Parliament draft report on the state of robotics begins: from Mary Shelley’s Frankenstein’s Monster to the classical myth of Pygmalion, through the story of Prague’s Golem to the robot of Karel Č apek, who coined the word, people have fantasized about the possibility of building intelligent machines, more often than not androids with human features.

1 INTRODUCTION 

5

Framed by this fictional legacy, the report then proceeds to discuss the implications of the rise of robotics. The report is only one example of the almost ubiquitous conflation of science and fiction in the cultural imaginary that not only infects policy and research but also drives the robotics/ AI industry. Claims that fiction is coming true demonstrate a fundamental lack of understanding of how fiction works and thoroughly obfuscate the field, clouding the science and neutering the critical force of fiction. Two: throughout the field, animals and machines are often viewed as interchangeable. Discussions of the rights of robots often begin with slippery statements like “we must get tougher on technology abuse or it undermines laws about abuse of animals.” This problematic conflation of machines and animals dates back to Descartes, who argued animals are automatons and incapable of processing pain. But, quite simply, animals are not machines and Descartes’ view has long been dismissed. As industrialization is on the one hand invested in the rapid rise of robots and on the other causing the rapid extinction of species, it is all the more urgent that we resist this conflation. A battery-operated robotic “bee” can never replace bee populations, and the misleading argument that we can replicate animals that we have only a rudimentary understanding of puts all animals (including humans) in peril. Three: corporations present robots and AI (i.e., Big Blue, Siri, Jibo, Pepper, Robi, Watson, Google’s search engine) as autonomous, but behind these “magic” machines are the humans producing the books, music, research, images, and maps that are the source of the machine “knowledge.” Take computer language translation, for instance, which, as Jaron Lanier points out, is only possible through the appropriation of the labor of millions of human translators, many of whom are now unemployed because of the technology. In a world of ever-increasing wealth disparity and precarious employment, the high priests of AI promise miracles to a populace in awe of technology; yet behind the “magic show,” humans produce the “data” that corporations turn into ones and zeros that is then monetized. Moreover, the promotion of machines as autonomous deflects responsibility and complicates industry liability as it allows for claims of “rogue” machines. Four: Silicon Valley vows to liberate us from the whims and limits of humans. The AI/robot doctor, soldier, teacher, or lawyer that benefits from massive data sets will be smarter and more objective than any single human, or so the thinking goes. Yet the very algorithms that are driving the industry are necessarily informed by the biases of coders and the ­prejudices of the human-created material they mine, and they replicate all

6 

T. HEFFERNAN

the toxic inequities that have infected human culture. Yet the black box effect, where inputting data leads to outputs or results but the connection between the two is opaque, means that the machine results, unlike human narratives, are almost impossible to challenge. The promise of an “objective system” that promotes the supposed “neutrality” and “universality” of big data echoes and repeats the destructive imperialist dream of a universal hegemonic culture. While there are many areas where big data will prove useful in addressing global problems, it is important to remember that it is neither objective nor intrinsically creative. Five: computer languages are different from human languages and are not interchangeable. As Vikram Chandra has argued, the first coding languages were modeled on modern linguistic theory, which in turn borrowed from Classical Sanskrit, developed by Paˉn·ini in 500 BCE. Sanskrit was thus the basis of the first generative grammar, a general theory that attempted to reveal the rules and laws that govern the structure of language. Sanskrit itself was an attempt to eradicate the confusion and ambiguity in the communication of the Vedas, the ancient sacred texts that required precise transmission. But these abstracted rules for communication proved fraught, as the idiosyncrasies of context and culture and the openness and suggestiveness of language remained resistant to regulation. The new religion of “big data” that fantasizes about perfect communication between humans and machines shares the hopes of this ancient grammar that understood the universe as rooted in language and religion. Yet rather than despair over the impossibility of creating a perfect code, a language that trumps all others, Chandra argues that the beauty of the aesthetic was found to reside not in clarity but in the meditative “pleasures of ambiguity.” The aesthetic experience was understood as reactivating all the layers of consciousness, feeling, and memory, but now released from the ego. The ambiguity of language, unlike rule-based binary computer languages, thus resonates with the complexities of human consciousness (Chandra 2017). Six: techno-utopians look forward to the merging of machines and man as the next step in human evolution and often chart the steps from upright apes to immortal supercomputers. Ray Kurzweil argues: By the time of the Singularity, there won’t be a distinction between humans and technology. This is not because humans will have become what we think of as machines today, but rather machines will have progressed to be like humans and beyond. Technology will be the metaphorical opposable thumb that enables our next step in evolution. (2005, 69)

1 INTRODUCTION 

7

But the idea that evolution has ever been a question of “next steps” or a simple linear progression is to profoundly misunderstand the nature and complexities of evolution. As the biodiversity on which humans and other species depend is rapidly disappearing as a result of human activity, investing in fantasy machines and promises of immortality are unlikely to be our best strategy for habitable worlds. As Don DeLillo perceptively noted in his novel White Noise: “This is the whole point of technology. It creates an appetite for immortality on the one hand. It threatens universal extinction on the other. Technology is lust removed from nature” (1986, 285). In order to address these myths, we need more voices asking different and challenging questions about this technology before the world is locked into an irreversible path paved with faulty assumptions, narrow perspectives, and impulsive decisions driven by profit, out-of-control egos, war, and an unquestioned faith in AI. This collection emerged out of a two-day workshop aimed at doing just that: bringing more voices to the conversation. Held at Saint Mary’s University in Halifax in the spring of 2017, the workshop brought together scholars from digital culture, robotics, anthropology, sociology, literary theory, race studies, environmental studies, creative writing, and evolutionary biology for a wide-ranging discussion. As the term “cyborg” in the title suggests, this collection recognizes the inevitable entanglement of humanity and technology, while insisting that the discussion about future communities of animals and machines must be extensive and transdisciplinary. While it is not uncommon in the world of AI and robotics to consider machine-human hybrids as the next stage in human evolution as many of the transhumanists do, John H.  Long Jr., Director of Vassar’s Interdisciplinary Robotics Research Laboratory and Professor of Biology and Cognitive Science, complicates this simplistic linear model in his chapter “Evolution Ain’t Engineering: Animals, Robots, and the Messy Struggle for Existence.” His study of both finches in the Galapagos and robotic models of early extinct fish challenges the view of evolution as the “optimal” thrust forward of a species. He shows how evolution works in the decentralized web of life and the struggle to survive in the face of chance, history, and the particulars of time and space, which involves compromises and trade-offs. Unlike abstract mathematical models and simulations of evolution that are built to work, the inevitable messy complexities of material bodies interacting in a world and subject to any number of unexpected events makes predicting evolution difficult. Simulated models that “are designed to succeed” erase the embodiment,

8 

T. HEFFERNAN

embeddedness, and complexities of time and place that his robots are forced to navigate. Evolution–or “descent with modification,” as he calls it, is neither “predictive” nor linear, much to the irritation of computer engineers. Moreover, Long is quick to highlight the differences between the animals he studies and the robots he builds that mimic certain biological traits of early life forms. Resisting the mystification of technology as life-­ like, he reveals that his models are made out of such things as tupperware, motors, neural nets, and jelly, materials that share little in common with biological bodies. This careful parsing of the differences between the two is a rare exception in a field that continually collapses animals (including humans) with robots/AI.  Demonstrating a keen awareness that human history is hardly a blink in the long history of the planet, Long’s research challenges models of “intelligence” that focus solely on the brain and neural activities. Long’s work makes clear that there is no clean division between the brain and the body, and that a body subject to the whims of an environment is key to the development of animal intelligence. It is the conflation of human and machine that Lucy Suchman, a professor of Anthropology of Science and Technology in the Department of Sociology at Lancaster University, also challenges in her chapter “Demystifying the Intelligent Machine.” She points to the “sleight of hand” that happens in the media and that is encouraged by some industry heads and scientists who insist that, whether for better or worse, it is only a question of time before humanlike machines supersede humans. The debate, she argues, already presumes an inevitable march toward “humanness” that obscures the interests driving the investment in this technology. Not only is there little evidence that machines are becoming more humanlike in their capabilities, but the “march” itself is not out of control but driven by the funding of certain visions of the future and not others. In considering the prevalent representation of humanoid machines as “autonomous,” Suchman exposes the ways in which this staging repeats the legacy of liberal individualism and its complicity in imperialism and paternalism. Thus a computational machine, for instance, is often portrayed as a child birthed and guided by its highly intelligent (often male) creator and on the road to autonomy. Restoring historical and cultural contexts to these carefully curated figurations of AI and robots, Suchman points to the complex networks of human labor and technology, which belie the depiction of machines as autonomous or humans as lone genius/creators.

1 INTRODUCTION 

9

Further, Suchman questions the DARPA-sponsored robotics industry that promotes robots as human helpers and the solution to anything from aging populations to environmental catastrophes. The industry promises robots will rescue humans from such predicaments as nuclear or climate-­ change disasters, help with domestic labor, and perform as caregivers to the elderly. However, Suchman is attentive to the slippery rhetoric that frames these promises as, while they might be good for generating business, they distract us from the need to mitigate climate change and halt the increasing automation of war, overlook the contingent nature of housework that makes it ill-suited for robots, and manufacture the claim that there is a shortage of human care workers. Rather than feeding the myth of autonomous social robots, Suchman exposes the material realities and extended networks that enable these performances and, in doing so, frees up space for other questions and concerns about the future of a technology that after all is improving due to data storage space, increased computational speed, and processing algorithms, which have little to do with how humans operate or behave in the world. The mystification of AI is also challenged by Karen Asp, a PhD candidate in the Faculty of Environmental Studies at York University, Toronto, in her chapter “Autonomy of Artificial Intelligence, Ecology, and Existential Risk: A Critique.” Asp argues that the idea that “autonomous” artificial intelligence poses an “existential risk” is an upside-down representation of the workings of capital that is driving our current human-­made environmental crisis. Touted as having the potential to, on the one hand, miraculously uplift humanity and solve global problems, and on the other, to destroy civilization, scientists discuss “autonomous” supermachines as capable of both rationality and occult motivations. Yet, Asp suggests, this existential risk narrative, in displacing this threat onto “maverick” machines, covers over human culpability and the drive of capitalism—itself a “utility maximizing machine” that has given rise to AI. In other words, there is nothing “mystical” about the pursuit of more and more efficient labor costs through increasing automation and abstraction. As an example of the problematic ideology of “existential risk,” Asp points to the film Transcendence that dramatizes both the “healing” potential and the destructive possibilities of the “singularity”—the view that “superintelligent” machines will exceed the limits of human cognition. This film prompted Stephen Hawking to warn that this catastrophic scenario could well come true as humans pursue the “explosion of the new knowledge potential of this technology.” Yet while we are focused on

10 

T. HEFFERNAN

technology going awry in some distant future, we are encouraged to embrace the “good technology” that promises a continually upgradeable and better life. Dreaming of a collective humanity that is able to transcend materiality, space, and time through technological “maturity” and fetishizing abstract intelligence, while simultaneously being haunted by the fear of species annihilation, existentialist risk proponents symptomatically expose humans themselves as those “autonomous” supermachines that exploit and exhaust planetary resources, while fantasizing about colonizing space to escape an earth they have helped render uninhabitable. This dangerous fetishization of automated technology is nowhere more in evidence than in the military-industrial complex. The rise in lethal autonomous weapons has both sparked a new arms race among global powers and been met by fierce resistance from researchers in the field and from the Campaign to Stop Killer Robots. Patrick Crogan, Associate Professor of Digital Cultures at the University of the West of England, in his chapter “Visions of Swarming Robots: Artificial Intelligence and Stupidity in the Military-Industrial Projection of the Future of Warfare,” considers the embrace of swarming robots by the war industry. He argues that autonomous weapons change the fundamental nature of war by shutting down the tradition of what Aristotle and others refer to as the “noetic” component of human intelligence that involves thinking, reasoning, and reflecting as part of both individual and collective dynamic responses to others, culture, and history. In his discussion of the Centre for a New American Security’s “Future of Warfare” research initiative, Crogan scrutinizes the turn to automatic swarming technologies in the American military, which are roughly modeled on insects and animals, to fight wars and asks what that might mean for the future possibilities of negotiating political and cultural differences in a globalized world. Billing these automated weapons as cost effective and faster than humans, comparing the swarming intelligence among animals to human violence and war, modeling artificial intelligence on a theory of emergence, The Coming Swarm (a volume based on various reports emanating from the “Future of Warfare” research program) advocates for the adoption of autonomous systems. Crogan, however, argues that these reports rely on simplified computer graphics that edit out any of the reality of war, ignore the controversies and scandals that have plagued semi-autonomous systems that cannot successfully discriminate between combatants and noncombatants, and refuse to engage with the critiques of military leaders and strategists, who argue this technology does nothing to win the hearts and minds of local civilian

1 INTRODUCTION 

11

populations. All this leads him to refer to this research project as “voluntary stupidity.” Moreover, the “stupidity” of the model itself is precisely that it sidesteps the ethical, political, and legal questions, which have always been part of military operations, in favor of autonomously functioning systems. With no fundamental acknowledgement of the ways in which humans are enmeshed and reshaped by technology as much as they shape technology, all the bigger questions of how war is being radically and problematically altered by autonomous swarms are notably absent from the report, Crogan maintains. Despite the complete lapse in ethics everywhere in the field—from sex robots to autonomous weapons—there is nevertheless pervasive talk of building “ethical” robots and AI, while ethicists engaged on large research projects spin the technology to make it palatable for the public. Distracted by questions about the rights of robots or the relation between autonomous cars and pedestrian safety, researchers and journalists often ignore the larger questions of global inequity that have been exacerbated by the embrace of unfettered technology and corporate capitalism. Kathleen Richardson, Professor of Ethics and Culture of Robots and AI at the Centre for Computing and Social Responsibility and founder of the Campaign Against Sex Robots, explores this problematic invocation of ethics by industry and academics even as the broader questions about patriarchy and capitalism are eschewed in these discussions. In her chapter “The Business of Ethics, Robotics, and Artificial Intelligence,” Richardson argues that an ethical approach must take account of the broader fields of sociology, anthropology, politics, and the humanities. Ethics and the questions of right and wrong do not take place in some “neutral” space but reflect larger social and political values. Working in the field of the ethics of AI and robotics, she has found that the burgeoning interest in ethics touted by corporations invested in this technology coincides with their business models as they are engaged with the project of reducing humans to machines or to algorithms and bits of data that can be manipulated. While this technology promises to solve the problems facing humans, Richardson proposes it is bound to fail unless it makes feminist and class analyses central to its ethical model. From Terminator to R2D2, the fields of AI and robotics are thoroughly entwined with fiction. As a professor of literature in the Department of English at Saint Mary’s University in Halifax, my own work has been ­concerned with this intersection between science and fiction. Fiction long predates these fields, yet increasingly fiction and science

12 

T. HEFFERNAN

are touted as having merged in the oft-cited mantra “fiction is becoming reality.” As I argue in my chapter “Fiction Meets Science: Ex Machina, Artificial Intelligence, and the Robotics Industry,” this conflation in the cultural imaginary helps to drive the fantasy aspect of the robotics/AI industry that encourages the view that there is no difference between a computing machine and a human. As if there is an inevitable trajectory from fiction to science, Jeff Bezos, for instance, announced at the 2016 Code Conference: “It has been a dream from the early days of sci-fi to have a computer to talk to, and that’s coming true.” But it is this invocation of “dreams coming true” that needs to be challenged. If fiction offers an exploration and interrogation of the shifting terrain of what it means to be human, the industry’s overly literal readings of fiction fetishize the technology, strip it of its cultural and historical context, and claim it for the here and now. While the industry exploits fiction to help animate machines and bring them to “life” in the name of a certain technological future, it erases the “fictiveness” of the fiction that keeps open the question of the future. My chapter argues that we need to restore the gap between the literary and scientific imaginings of AI and robots. Resisting literal readings of fiction, it considers the ways in which metaphors shape our reading of humans and other animals. For instance, in the field of AI, rather than the computer serving as a metaphor for the brain, the brain has come to serve as a metaphor for the computer. The film Ex Machina, as a modern-day Frankenstein story, exposes the consequences of this metaphor that reduces humans to computing machines that in turn entraps them in an algorithmic logic under corporate control. In this film, it is not Ava, the programmed machine, that is the subject of the experiment, but rather Caleb who finds himself locked in the robot lab at the end of the story. The collaborative chapter by  Jennifer Keating, Assistant Dean for Educational Initiatives in Dietrich College and Professor in the Department of English, and Illah Nourbakhsh, Professor of Robotics and Director of the CREATE Lab at the Carnegie Mellon University, also considers the meeting of science and fiction and proposes that writing and robotics are sister arts as both share an interest in mimetic representations. In their chapter, “Rossum’s Mimesis,” on Karel Cˇ apek’s 1920 play R.U.R. (Rossum’s Universal Robots), and Ray Kurzweil’s 2007 nonfiction book The Singularity Is Near, the authors ask what this earlier dystopian play written at a time of historical turmoil, rising nationalism, factory capitalism, mass

1 INTRODUCTION 

13

manufacturing, escalating consumption, lax labor regulations—all driven by technological-militarization culminating in the Second World War— might have to say about the present day techno-utopian dreams and engineering fantasies in the unsettled early decades of the twenty-first century. R.U.R., the origin of the term “robot,” highlights this question of mimesis and the ancient Greek sense of tekhne—as a crafting and creating of a likeness of the world in order to make sense of its complexities. In the play Old Rossum and Young Rossum demonstrate the two ambiguous and overlapping connotations of tekhne as both imitation and creation that continue to inform the field of robotics. Old Rossum is trying to dethrone God by building a perfect and faithful replica of a human, while the Young Rossum wants to engineer or optimize particular features, like intelligence and efficiency, to create a more productive workforce. The complex human thus gives way to a simplified, functionally improved, “mechanically more perfect” robot that better fits the needs of capitalism and increasingly specialized industrial tasks. In contrast to Č apek’s dystopic fiction critiquing the political and economic mythos of technological advancement of his day, Ray Kurzweil’s New York Time’s bestselling The Singularity Is Near circulates as nonfiction, but offers technological solutions and the promise of immortality more reminiscent of the “unimaginable” of science fiction, as a way of addressing the social and political flux of our time. In a dangerous mixture of utopic “form and wish,” the authors argue, Kurzweil combines the aspirations of both the Old and the Young Rossums in his desire to create and radically engineer a superhuman, hoping, as the scientists in the play do, that this technology will lead to increasingly cheap commodities, rapid education, intelligence enhancement, exponential growth, and leisure time for all. With absolutely no basis in current science, Kurzweil predicts that “efficient” digital computing will end with the collapse of machine and human, the real and the virtual, giving way to the expansion of human intelligence across the universe. While Kurzweil fails to acknowledge the influence of fiction on robotics and AI scientists, the authors ask what a productive dialogue between writers, artists, and roboticists might offer in terms of imagining more nuanced technological futures that operate in a space between the dystopic/utopic. When not portrayed as malevolent overlords, robots in fiction have often played the role of the marginalized other, the oppressed, and the enslaved. Moreover, Afrofuturism has long explored the connection between slavery and science fiction as the tropes of an alien invasion or

14 

T. HEFFERNAN

abduction to another planet perfectly describe the lived estrangement of the African diaspora. However, in his chapter “Race and Robotics,” Louis Chude-Sokei, Professor of English, George and Joyce Wein Chair in African American Studies, and Director of the African American Studies Program at Boston University, argues that, more than just imaginative associations, the material cultures of technology and slavery are so entwined that racialized technology makes possible the very genre of science fiction. Norbert Wiener, the Jewish mathematician and philosopher, who developed the theories of cybernetics, robotics, computer control, and automation, Chude-Sokei notes, explicitly discussed the ethics of this technology in terms of industrialization, dehumanization, and slavery, exposing the ways in which race haunts the development of cybernetics. All of Weiner’s concerns about whether machines or artificial life had feelings or souls emerge out of nineteenth-century debates about the humanity of African slaves in the transatlantic world, situating “blacks as the first robots”—as tools and prosthetic devices that increased white productivity. Racism, colonialism, and power are all part of the history of technology, he argues, and attending to this history reframes readings of works like R.U.R. The fears of racial violence and rebellious blacks in America situates this play—with its fear of evolving robots rising up against humans (whites)—in this context, as does Cˇ apek’s later novel, which in turn borrows from a longer legacy of racialized machines in Victorian science fiction. The chapters in this collection, though they emerge from different disciplines, all share a skepticism about the dominant representations and assumptions that are currently driving the fields of AI and robotics. Fantasies of immortality, transcendence, and digital utopias are deflecting attention from the urgent issues raised by the technology. Widening the conversation to include the larger context of corporate capitalism; global wealth disparity; evolution and biology; militarization, surveillance, and the war industry; gender politics; the history of race and technology; environmental destruction and diminishing resources, this collection hopes to check the distracting and destructive mystification of this technology in order to better address its potential and its limits.

1 INTRODUCTION 

15

References Č apek, Karel. 1923. “The Meaning of R.U.R.” The Saturday Review 136 (July 21): 79. Chandra, Vikram. 2017. “The Pleasures of Ambiguity.” Talk at the Cyborg Futures Workshop, Saint Mary’s University, Halifax, April 1. https://socialrobotfutures. com/events/cyborg-workshop-2/vikram-chandra/. Accessed 18 Jan 2018. DeLillo, Don. 1986. White Noise. New York: Penguin. Goldstein, Evan. 2018. “The Academy Is Largely Itself Responsible for Its Own Peril.” The Chronicle of Higher Education, November 13. https://www.chronicle.com/article/The-Academy-Is-Largely/245080. Accessed 18 Jan 2018. Kurzweil, Ray. 2005. The Singularity Is Near: When Humans Transcend Biology. New York: Viking.

CHAPTER 2

Evolution Ain’t Engineering: Animals, Robots, and the Messy Struggle for Existence John H. Long Jr.

In On the Origin of Species, Darwin (1859) described the moment-by-­ moment experience of each organism as a “struggle for existence.” Individuals struggle with and against others to survive and reproduce, and this collective struggle creates the transgenerational process that he dubbed “descent with modification,” waiting until the book’s last word to call life forms “evolved.” But there is no last word in evolution. Ongoing, “it” (as though it were an entity) struggles, stumbles, and muddles its way along, and that sufficing, just-in-time, and just-good-enough engineering is not controlled by a central scrutinizer with a plan. Instead, evolution is a web of decentralized processes enacted by individuals on themselves, each other, and their world. This dynamical system of agency and patiency, actions and reactions, is an interaction network (Gordon 2010). While individuals enact the network, they themselves do not evolve. Instead, the network evolves. Through births and deaths, matings and mutations, migrations and other

J. H. Long Jr. (*) Vassar College, Poughkeepsie, NY, USA e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_2

17

18 

J. H. LONG JR.

struggles for existence, the collective properties of the interaction network change from one generation to the next (Hartl and Clark 2007). This evolving interaction network is called a population, taxon, or species. This network is a continuously emergent historical phenomenon, instantiated collectively and unknowingly by individuals struggling for existence at a particular time and place. Constrained by the need to find energy and then put it to work to grow and survive, individuals leave their mark through the abyss of generational time by reproducing or not. Individual differences in reproduction, one’s evolutionary fitness, occur by chance and by struggle. Partitioning the fluctuating magnitude and impact of those two causes in the historical and ongoing creation of life is the task of the evolutionary biologist.

Evolution Suffices For the roboticist, biological evolution provides an existence proof of a design process that is “hands off,” meaning, for engineering purposes, that humans don’t make a priori decisions about what designs to create and test. Random chance varies existing designs and then selection, the collective struggle, tests and judges them. This idea has been exploited and developed by three methods used to extend our ability to design machines (Bäck 1996): genetic algorithms, evolutionary computing, and evolutionary robotics. While humans know enough about evolution to use the same principles that we employ for breeding animals for the designing of machines, we are constantly confronted with the limitations of our own implicit thinking about this process. Some claim that biological evolution produces perfect or optimal designs. For this continued misunderstanding, Darwin (1859) deserves some of the blame: “How have all those exquisite adaptations of one part of the organization to another part, and to the conditions of life, and of one distinct organic being to another being, been perfected?” Beyond a metaphor for “well-functioning,” the problem with perfection is its inherent abstractness: what is perfection, or the engineer’s analogous “optimality,” when we move from formal, mathematical definitions to actual, physical instantiations? We get a clue from one field of engineering and computing science (Deb 2001, 2012): multi-optimal design. This approach recognizes that “optimal” performance of most machines should be measured by different criteria assessed at the same time. Practitioners recognized a central truth:

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

19

any physical, functioning machine is a compromise between trade-offs imposed by physics. Automobiles, a product of two centuries of intense human engineering, provide a great example. If you want a fast car, measured in terms of acceleration performance, its big, heavy-but-powerful engine will have poor fuel efficiency when it comes to cruising at steady highway speeds. The evolution of fish shows a similar trade-off between acceleration and energy efficiency, with broad tails good for acceleration but inefficient for low-energy cruising (Webb 1984). Specialists push performance in one dimension with the costs imposed and exposed in others. Generalists, by definition, perform suboptimally in each dimension, but not horribly in any. As the number of performance dimensions increases, the trade-offs among them blur into the integrated compromise of a complex machine or complicated organism. Trade-offs can be rebalanced but not eliminated, a fact that helps us replace the abstraction of perfection with the realities of sufficiency. That evolution is a process that suffices rather than optimizes results from at least four facts about the world: (1) optimization in the sense of theoretical perfection is impossible in practice, thanks to the physical constraints that create trade-offs and inefficiencies in performance; (2) the evolving interaction network is a finite population of individuals representing a few or many but not all possible designs in life’s high-dimensional search space; (3) the evolutionary (“Darwinian”) fitness of individuals, measured post hoc by reproductive success, is, a priori, a propensity and not a given (Smith 1989); and (4) random events make the life of an individual and the interaction network to which it belongs partially unpredictable. As we shall see, the evolutionary trajectory of any population of animals or robots is caused by the fluctuating influences of random events and selective interactions on individuals. To uncover the particulars in any interaction network, evolutionary biologists have three main tools at their command: direct observation, experimental manipulation, and model simulation. The power of direct observation is demonstrated by the work of Rosemary and Peter Grant, who, with collaborators, have spent over 30 years observing and studying individual finches in the Galapagos Islands. Experimental manipulation is exemplified by humans’ selective breeding of plants and animals, the domestication process Darwin put forward as direct evidence for the process that he called natural selection. The value of model simulation is still in hot debate, and our work on evolving robots demonstrates its power and pitfalls.

20 

J. H. LONG JR.

A History of Struggles: Darwin’s Finches Every individual struggles at a particular time and place. While mathematical models abstract away these details, for any individual, each moment is fraught with the contingencies of history that look predictable only in retrospect: hence the fallacy of post hoc ergo propter hoc. Travisano et  al. (1995), evolving bacterial populations over thousands of generations, demonstrated experimentally the dramatic importance of history relative to selection and chance. Thus history, which includes the population’s genetic possibilities, is an evolutionary force. Grant and Grant (2002), working on Darwin’s finches in the Galapagos Islands, demonstrated a fourth great evolutionary driver: place. All told, then, we have four factors to consider in any explanation of how a particular population has evolved or is evolving: selection, chance, history, and place. Darwin (1859) recognized these factors, but emphasized natural selection, his insight. It is a mechanism that explains, to large degree, the origins of adaptations, those traits and features that appear, at first glance, “perfectly” matched to their individual possessor’s place and struggle for existence. Selection, chance, history, and place are all present in one of his most famous passages from the first edition (Darwin 1859): It may be said that natural selection is daily and hourly scrutinising, throughout the world, every variation, even the slightest; rejecting that which is bad, preserving and adding up all that is good; silently and insensibly working, whenever and wherever opportunity offers, at the improvement of each organic being in relation to its organic and inorganic conditions of life. We see nothing of these slow changes in progress, until the hand of time has marked the long lapse of ages, and then so imperfect is our view into long past geological ages, that we only see that the forms of life are now different from what they formerly were.

This was rhetorical, in part. Anthropomorphizing natural selection as a “scrutinizer” makes it easier for subject- and purpose-hunting humans to get a purchase on the complex operations of natural selection. But, even though marked with a cautionary “It may be said that,” the passage pushes one’s intuition away from the idea of a decentralized interaction network of distributed, interconnected causes and effects, both direct and indirect. Darwin, as a young pre-evolutionary naturalist sailing the globe on the HMS Beagle, originally underplayed interconnectedness. His visit to the Galapagos Islands, and collection of what we now call “Darwin’s” finches,

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

21

was not his eureka moment, as is often portrayed in textbooks (Sulloway 1984). He failed to note which birds were collected from which island. Only in hindsight, with help from colleagues at the British Museum, did he come to understand that these small, drab birds varied from place to place, island to island. Later, with Darwin’s new evolutionary framing, biologists came to understand the importance of an island’s place in terms of its rainfall and vegetation, its isolation from the mainland, and its isolation from other islands within the archipelago. Because of their isolation, the Galapagos Islands, a volcanic archipelago almost 1000 kilometers west of Ecuador’s mainland, stage one of the most complete demonstrations that we have of the evolution of animals (Weiner 1994). History matters, too. The oldest islands in the archipelago are 2–3 million years old, geological newborns compared to the 200-million-year age for the South American continent. Because they are new and isolated, the islands offer the game of life in a simple, striped down version, in which it is easy to observe the struggle for existence. While the larger, older islands have mountains and forests, the smaller, younger islands, like Isla Genovesa (Fig. 2.1), about 700,000 years old, stand just tens of meters above high tide with low shrubbery and exposed volcanic rock. With little rainfall during the dry season, the vegetation is dominated by drought-resistant species in this lowland arid zone. Isla Genovesa and Daphne Major are two islands where Rosemary and Peter Grant study the ongoing evolution of Darwin’s finches (Weiner 1994). Islands in the Galapagos are often far enough apart to prevent easy migration for small birds, like Darwin’s finches. Yearlong residents on their island, finches can be carefully netted, outfitted with colored identity bands on their ankles, and then identified individually throughout the year as they struggle for existence. On these smaller islands there are few enough finches, numbering in the hundreds, that the Grants can know them all as individuals, know who is mating with whom, know how many eggs are laid, and which youngsters make it to adulthood. Just as importantly, the Grants and their collaborators can watch what the finches eat and don’t eat. While the large seabirds such as Nazca boobies and great frigate birds make long journeys to sea to fish, the 14 species of finch are bound to their islands, most foraging primarily for seeds (Fig. 2.2). During the wet season, small, soft seeds are abundant. But during the dry season, which can last for half a year or more, most of what’s left are seeds that are big, hard, and hard to crack.

22 

J. H. LONG JR.

Fig. 2.1  Isla Genovesa, a young, small island in the Galapagos archipelago of Ecuador. On Darwin Bay Beach, saltbush, Cryptocarpus pyriformes, are evergreen shrubs that pioneer the beaches (top). Inland, the arid zone fields yellow cordia, Cordia lutea, a common flowering shrub (lower left). Cacti, like the prickly pear, Opuntia sp. (lower right), are abundant and are eaten by animals; if you look closely, you can see a cactus finch in the center. (All images were taken by the author on 9 August 2002)

When hard seeds are the plentiful type, birds with deeper, larger, blunt-­ shaped bills can more easily process the tough casing that surrounds the seeds of the Tribulus plant, getting more seeds out (Grant 1999). But when small, soft seeds are available, birds with small, pointed beaks can dig those seeds out of crevices more readily. Trade-offs are the rule, not perfection or optimization. In wet years, with plants producing abundant soft seeds, individuals with sharp, small bills produce many offspring, while the birds with big bills struggle, relatively speaking (Grant and Grant 2002). In dry years, the opposite is true.

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

23

Fig. 2.2  The struggle for existence of finches in the Galapagos Islands. Finches on Isla Genovesa forage for seeds (top) and, when found, attempt to eat them (bottom). Sharp-beaked ground finch, Geospiza difficilus, is on the left; large ground finch Geospiza magnirostris is on the right. Rosemary and Peter Grant have shown that finches with larger, deeper bills are better at eating the tough seeds available during drought years; finches with smaller, sharper bills are better at quickly collecting and processing the small seeds available during wet years. (All images were taken by the author on 9 August 2002)

Observing the daily struggles of the different individuals of two finch species over 30 years, the Grants came to a startling conclusion: at the beginning of their study the course of evolutionary change 30 years later was neither predictable nor constant (Grant and Grant 2002). Historical contingency, coupled with chance in the form of climatic forces like El Niño or the unexpected immigration of new species, changes the place, the environmental and genetic possibilities, and the selection forces from year to year.

24 

J. H. LONG JR.

The unpredictable evolution of Darwin’s finches doesn’t mean that we can’t understand what is happening and has happened. It’s the future that’s looking dark, clouded by our incomplete understanding of the global climate system, the small but nonzero probability of a bird from one island getting blown to another, or the haphazard movement patterns of humans, tourists, and scientists, as they vector cats, goats, and microbes along their pathways. Most importantly, even if we had complete information, random events, by definition, would limit the certainty of our predictions. With a sequence of events, each contingent upon the previous, those uncertainties compound. We can circumscribe the probable, but we can’t pinpoint the inevitable.

Modeling the Struggles: Evolutionary Biorobotics Unlike life forms, physical robots don’t evolve spontaneously or autonomously, even though this is not impossible in principle (Jelisavcic et  al. 2017). In actuality, humans decide any evolving robots’ basic body plan, environment, and traits that may or may not have the potential to evolve (Fig. 2.3). What’s left for the robots to do is enact their own autonomous behavior as they operate in their environment. How well an individual performs a predetermined task, how well it struggles for existence, determines that individual’s evolutionary fitness in its population of robots. Differences in fitness are what drive the differential reproduction that is the hallmark of adaptive evolution. Humans evolve robots for at least two reasons: (1) to serve the engineering purpose of creating novel, successful designs or (2) to serve the scientific purpose of testing hypotheses about nature itself (Long 2012). The former approach is usually called evolutionary robotics, and the latter I call evolutionary biorobotics (Long 2007), an extension of Webb’s (2001) Biorobotics, a method that uses physical robots as models of animals to test biological hypotheses. The focus here is on testing hypotheses about how animals have or may not have evolved. Why build robotic models in the first place? The more general scientific question is this: What is a model, what scientific function does it serve, and what value does it have? In evolutionary biology, models are often mathematical representations of a process. For example, the Hardy-Weinberg (HW) model demonstrates what happens to a population’s genetic structure when that population is not under evolutionary pressure (Hartl and

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

25

Fig. 2.3  Evolutionary biorobotics, a life-cycle approach with humans in the loop. Robots do not spontaneously or autonomously evolve. Humans determine almost everything about the individual robots and their world. But the robots enact their own behavior autonomously, without the remote-control help of a human. Autonomous behavior is the final element in the life cycle of a robot, it determines the evolutionary fitness of that individual, and it produces the response variables in experiments that we use to test biological hypotheses

Clark 2007). One function of the HW model is to provide a null hypothesis against which to test alternative models that may posit evolutionary forces, such as mutation or selection, at work in a population. In general, the function of a model may be to serve as a description, predictive framework, and/or proof-of-concept (Webb 2001). It is incumbent on the modeler to make known and to justify the model’s functional purpose. The value of a model is usually judged historically, based on the guidance it gives to future research and/or its applicability to solving real-world problems. In evolutionary biology, Fisher’s (1922) early and influential mathematical model of the dominance of genes provided the foundation for a century of theoretical population genetics (Wahl 2011). Biorobots are physical models that behave autonomously (Webb 2001) and function as proofs-of-concept and experimental testbeds. While ­mathematical representations are abstractions, physical models are concrete instantiations; the proof of these models is in the pudding—they work or

26 

J. H. LONG JR.

Fig. 2.4  Biorobotic models. The class of biorobots known as Tadros (bottom left) are physical models of fish such as the extinct jawless fish, Drepanaspis (upper left). The figure here is an artistic morphing of the animal into its biorobotic model, serving to show the similarities in shape, size, and manner of swimming. Tadros operate using sensorimotor mechanisms used by fish: a flapping, flexible tail generates thrust; sensors detect light gradients and the proximity of predators; neural circuitry decides when to escape from predators and override the goal of seeking light. This version of Tadro was used to test hypotheses about the evolution of the vertebral column in the first, extinct vertebrates (Roberts et al. 2014)

they don’t. The most valuable biorobotic models work using mechanisms used by the animals that they represent (Fig. 2.4). The concept of representation drives the logic of modeling in biorobotics: an experimental test of the robot is a test of the biological system that it represents. The extension to evolutionary biorobotics is to use evolutionary processes as the shared mechanism and to test evolutionary hypotheses (see Fig. 2.3). We used the Tadro class of biorobots (Fig. 2.4) to model extinct, 400 million-year-old fish in order to test hypotheses about the evolution of the first vertebrates (Long et  al. 2006; Roberts et  al. 2014). Tadros are ­self-­propelled, using a flapping tail to generate thrust and maneuver. All versions to date have been built to seek light as an energy source. Eye spots are simple sensors called photoresistors; the flapping tail, driven by a servo

27

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

motor, can be a biomimetic version of a vertebral column (Fig. 2.4) or a simple rigid tail (Fig.  2.5). The simplified equivalent of a fish’s n ­ ervous system is the neurocontroller or controller, a small computer that calculates the motion for the tail based on patterns of light detected by the eyespots (Fig. 2.5).

neural network

h0

light left

light right

i0

i1

h1

h2

h3

h4

o0

o1

offset

frequency

h5

Fig. 2.5  Tadro class, model T15A biorobot. Physically embodied and behaviorally autonomous, this Tadro was designed with a nonevolving body morphology and an evolving, genetically encoded neural network (Livingston et al. 2016). This system, which includes a simple aquatic environment with a single light source, tested the hypothesis that modular substructures of the neural network would evolve under selection for enhanced light seeking. Tadro T15A swims on the surface of the water, using its eyespots to sense a centralized light source. Depending on how those light sensors (labeled “light left” and “light right,” in the neural network diagram, above) are connected to the motor output (labeled “offset” and “frequency”) determines how the Tadro behaves in different places in the environment. The neural network has 60 possible connections (lines with arrows), and each connection may be excitatory or inhibitory. Each connection is represented by a gene in a genetic system that undergoes mutation and asexual reproduction. Every generation of Tadros consists of ten individuals, each with a different pattern of connections in the neural network. (This image is modified, under the CC-BY license, from Livingston et al. (2016))

28 

J. H. LONG JR.

Because Tadros are behaviorally autonomous, we can observe how each individual struggles for existence in a simple aquatic environment (Figs. 2.6 and 2.7). To create the original, ancestral population, we randomly generated ten neural networks, one for each Tadro in the population. Because this initial population has never undergone selection, we expected a wide range of mostly poor light-seeking behaviors. This was what we observed (Fig. 2.6). Typically, a Tadro with a randomly generated neural network may swim into the light once, but it spends most of its five-minute life in the dark, moving aimlessly against the walls of the tank. Its evolutionary fitness, measured as the amount of light that it harvests, is low.

Fig. 2.6  A Tadro with a randomly generated neural network from the ancestral generation. This is a simple aquatic world, a 3-meter wide tank, with a single light source. As shown in snapshots, over five minutes, this individual swam but collected light only once, giving it a low evolutionary fitness (integration of the green line in the graph of light and time). Because of the oblique view, the center of the tank, with the highest intensity of light, is indicated with the white ellipse. (This image is modified, under the CC-BY license, from Livingston et al. (2016). Color figure online)

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

29

Fig. 2.7  The Tadro with the best neural network, evolved after nine generations of selection. This individual actively sought the light, orbiting four times through the center of the tank (white ellipse) in five minutes. Its evolutionary fitness, as shown by the light harvesting graph, was five times greater than that of the Tadro shown from the first generation (Fig.  2.6). (This image is modified, under the CC-BY license, from Livingston et al. (2016))

Over nine generations of selection for enhanced light-seeking behavior, the population evolved. Individuals in the final generation became much better at harvesting light, increasing evolutionary fitness by a factor of five compared to individuals in the ancestral population (Fig. 2.7). We conducted these experiments to test the hypothesis that brains— and the neural networks that model them—should become more modular in their wiring patterns as selection acts to evolve improvements in behavior (Livingston et al. 2016). Modules within a single brain can be thought of as separate pathways specialized for different functions. We predicted that from a population of randomly generated networks, two modules would evolve, one to control the frequency at which the tail flapped (and hence the speed of the robot) and one to control the angle at which the tail was oriented (and hence the heading of the robot). To our surprise,

30 

J. H. LONG JR.

the individual with the best evolved light-seeking behavior (Fig. 2.7) had only a single module, one connecting the two light sensors to the motor output controlling the heading angle. Rather than arriving at our idea of the best two-module solution, in just a few generations evolution found a simpler and novel solution. Evolution surprises as it suffices. Because we know the reproductive fates of every individual, we can represent the evolution of Tadros as descent with modification in a genealogical tree (Fig. 2.8). The modification represented in the tree is the evolutionary fitness. Since fitness is a measure of how much light each individual harvested, it’s clear that the population didn’t show much improvement in behavior until the selection event that created the eighth generation from the seventh generation. This delay is not surprising, considering that the ancestral population had neural networks that were randomly generated and, hence, poorly functioning. Time was required to produce enough different

Fig. 2.8  Descent with modification in a population of physical robots. The population began with ten randomly generated neural networks (bottom row) that connected light sensors to the tail motor in the behaviorally autonomous Tadro (see Figs.  2.6 and 2.7). Individuals were chosen to reproduce asexually with a probability proportional to their evolutionary fitness, which was determined by how well they gathered light. Individuals that reproduced had their genome, which coded for the neural network, mutated. Those mutations produced new genomes that were used to create offspring with different neural networks. Only four individuals from the first generation reproduced, and only two of those had descendants in the final, tenth generation. (This image is modified, under the CC-BY license, from Livingston et al. (2016))

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

31

variations for selection to find and then preserve genotypes that code for neural networks that allow for improved light-seeking behavior. One benefit of modeling evolution is that the investigator can manipulate which traits may evolve. Using different models of the Tadro class biorobot, we evolved only the brain, as just explained, keeping the body constant (Livingston et al. 2016), and we evolved only the body, keeping the brain constant (Roberts et al. 2014). In both sets of experiments, the population of Tadros, under selection for enhanced light seeking, evolved better light-seeking behavior. The individuals became smarter. These results provide evidence for the counter-intuitive idea that behavioral intelligence may be controlled by the body and not just the nervous system (Fig. 2.9), as predicted by the theory of morphological and embodied intelligence (Pfeifer and Bongard 2006).

Fig. 2.9  The body and the neurocontroller interact continuously to enact intelligent, autonomous behavior. Inspired by Pfeifer and Bongard (2006), this diagram shows a causal loop typical of a dynamical systems approach to understanding the operations of a real-world system such as a physical robot. While we typically think of the neurocontroller as a brain-like overlord for behavior, it must interact with the world by giving instructions to actuators, like the motor driving the tail of Tadro. The Tadro moves in response to those instructions in a way that is governed by the laws of physics for the motion of bodies. As the robot moves, it generates sensory information about its movements in relation to the world that informs, through the sensors, the neurocontroller. These interactions occur continually

32 

J. H. LONG JR.

Evolution Ain’t Engineering With Darwin’s finches and Tadros we’ve seen that the evolution of actual physical entities, even when we created them, is difficult to predict ahead of time. In retrospect, we can analyze the events and understand the relative contributions of selection, chance, history, and place. But it is unsettling to understand what has happened only in retrospect. If we know so much about evolution, why do we have such poor predictive powers? Complexity. Even with simple environments like the Galapagos or simple robots like Tadros, the interactions of autonomous agents with their worlds are complicated. Every detail matters. A hidden cache of soft seeds might allow a finch with a small bill to survive until the next rain. One new neural connection, in just the right spot, might provide another boost to the light-seeking behavior of Tadros. If these events are dependent on other events—and they always are—and there are many possible events, then the number of possible outcomes from a given starting point explodes in a combinatorial nightmare. This is true even if we were to have complete knowledge at the time that we make the prediction. The more time elapsed from that moment of prediction the greater the probability that different elements of the unfolding events will interact in ways that are unexpected. The increasing uncertainty with elapsed time is part of the reason that behaviorally autonomous robots and animals never stop collecting data about themselves and their worlds. If, as Hawkins and Blakeslee (2004) propose, motor commands are predictions about how one’s body is about to interact with the world, then effective movement is a great example of a model of the world updated from step to step as we walk. Probabilistic robotics, an approach which has produced highly functional robots in real-world situations, deals with uncertainty by continuously modeling the uncertainty of information from sensors, responses of the motors, and self and world models run by the robot (Thrun et al. 2005). Complexity and uncertainty do not mean that evolution runs amok. The possibilities are bounded by the mechanisms: selection, chance, history, and place. These mechanisms are physical and they create and accumulate evolutionary change gradually, over generational time. Time is still the infinite jest, building complexity slowly and destroying it quickly. Evolution ain’t engineering.

2  EVOLUTION AIN’T ENGINEERING: ANIMALS, ROBOTS, AND THE MESSY… 

33

References Bäck, Thomas. 1996. Evolutionary Algorithms in Theory and Practice: Evolution Strategies, Evolutionary Programming, Genetic Algorithms. New York/Oxford: Oxford University Press. Darwin, Charles. 1859. On the Origin of Species. A Facsimile of the First Edition, 1964. Cambridge, MA/London: Harvard University Press. Deb, Kalyanmoy. 2001. Multi-Objective Optimization Using Evolutionary Algorithms. Vol. 16. Chichester/New York: John Wiley & Sons. ———. 2012. Optimization for Engineering Design: Algorithms and Examples. New Delhi: PHI Learning Pvt. Ltd. Fisher, Ronald A. 1922. “On the Dominance Ratio.” Proceedings of the Royal Society of Edinburgh 42: 321–341. Gordon, Deborah M. 2010. Ant Encounters: Interaction Networks and Colony Behavior. Princeton: Princeton University Press. Grant, Peter R. 1999. Ecology and Evolution of Darwin’s Finches. Princeton: Princeton University Press. Grant, Peter R., and Rosemary B.  Grant. 2002. “Unpredictable Evolution in a 30-Year Study of Darwin’s Finches.” Science 296 (5568): 707–711. Hartl, Daniel L., and Andrew G. Clark. 2007. Principles of Population Genetics. 4th ed. Sunderland: Sinauer Associates. Hawkins, Jeff, and Sandra Blakeslee. 2004. On Intelligence: How a New Understanding of the Brain Will Lead to Truly Intelligent Machines. New York: Henry Holt and Company. Jelisavcic, Milan, Matteo de Carlo, Elte Hupkes, Panagiotis Eustratiadis, Jakub Orlowski, Evert Haasdijk, Joshua E. Auerbach, and A.E. Eiben. 2017. “Real-­ World Evolution of Robot Morphologies: A Proof of Concept.” Artificial Life 23 (2): 206–235. Livingston, Nicholas, Anton Bernatskiy, Kenneth Livingston, Marc L. Smith, Jodi Schwarz, Joshua C.  Bongard, David Wallach, and John H.  Long Jr. 2016. “Modularity and Sparsity: Evolution of Neural Net Controllers in Physically Embodied Robots.” Frontiers in Robotics and AI 3: 75. https://doi. org/10.3389/frobt.2016.00075. Long, John H., Jr. 2007. “Biomimetic Robotics: Self-Propelled Physical Models Test Hypotheses About the Mechanics and Evolution of Swimming Vertebrates.” Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science 221 (10): 1193–1200. ———. 2012. Darwin’s Devices: What Evolving Robots Can Teach Us About the History of Life and the Future of Technology. New York: Basic Books. Long, John H., Jr., Thomas J. Koob, Kira Irving, Keon Combie, Virginia Engel, Nick Livingston, et al. 2006. “Biomimetic Evolutionary Analysis: Testing the

34 

J. H. LONG JR.

Adaptive Value of Vertebrate Tail Stiffness in Autonomous Swimming Robots.” Journal of Experimental Biology 209 (23): 4732–4746. Pfeifer, Rolf, and Josh Bongard. 2006. How the Body Shapes the Way We Think: A New View of Intelligence. Cambridge, MA: MIT press. Roberts, Sonia F., Jonathan Hirokawa, Hannah G. Rosenblum, Hassan Sakhtah, Andres A. Gutierrez, Marianne E. Porter, and John H. Long Jr. 2014. “Testing Biological Hypotheses with Embodied Robots: Adaptations, Accidents, and By-Products in the Evolution of Vertebrates.” Frontiers in Robotics and AI 1: 12. https://doi.org/10.3389/frobt.2014.00012. Smith, John Maynard. 1989. Evolutionary Genetics. Oxford: Oxford University Press. Sulloway, Frank J. 1984. “Darwin and the Galapagos.” Biological Journal of the Linnean Society 21 (1–2): 29–59. Thrun, Sebastian, Wolfram Burgard, and Dieter Fox. 2005. Probabilistic Robotics. Cambridge, MA: MIT press. Travisano, M., J.A. Mongold, A.F. Bennett, and R.E. Lenski. 1995. “Experimental Tests of the Roles of Adaptation, Chance, and History in Evolution.” Science 267 (5194): 87–90. Wahl, Lindi M. 2011. “Fixation When N and s Vary: Classic Approaches Give Elegant New Results.” Genetics 188 (4): 783–785. Webb, Barbara. 2001. “Can Robots Make Good Models of Biological Behaviour?” Behavioral and Brain Sciences 24 (6): 1033–1050. Webb, Paul W. 1984. “Form and Function in Fish Swimming.” Scientific American 251 (1): 72–83. Weiner, Jonathan. 1994. The Beak of the Finch: A Story of Evolution in Our Time. New York: Vintage.

CHAPTER 3

Demystifying the Intelligent Machine Lucy Suchman

Contemporary discussions in the popular media of developments in the creation of humanoid robots, reinforced by concerned roboticists, begin with the question: Given the “rise” of these humanlike machines, how should we humans respond? This chapter argues that our first response should be to question the question, to recognize the rhetorical sleight of hand that underwrites this seeming invitation to concern and action. The question implies acceptance of the proposition that these technologies are advancing towards humanness, and that this advance is inevitable. The inevitability is based on the premise that development of the field of robotics is a kind of natural/cultural occurrence, like global warming. The implication is that, like climate change, however much the expansion of robotics may be induced by human activity, it is now proceeding with its own dynamic. But AI and robotics are very different kinds of natureculture than global warming. True, dynamics are in place that will unfold if they are not actively interrupted and mitigated. But these dynamics are much more wholly human ones, less entangled with the more-than-human and more amenable to a political will to intervention. Moreover, while technological initiatives are progressing in some areas (processing power, data storage, L. Suchman (*) Department of Sociology, Lancaster University, Lancaster, UK e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_3

35

36 

L. SUCHMAN

the sophistication of algorithms, and networking), there is a notable lack of progress in efforts to achieve humanlike capacities. These differences are obscured, however, by the prevailing mystification of the state of the robotic arts and sciences. So what if the questions that we ask are rather these: In what ways, and to what extent, are machines becoming more humanlike, and in relation to what figure of the human? In whose interests are these projects, and who decides that they should go forward, in lieu of other projects of transformative future making? We can begin to address these questions by looking more closely at the boundaries of robot agencies: that is, the ways in which they are currently designated, and how they might be drawn differently. This approach begins from the observation that the framing of so-called autonomous robots—in both their visual and narrative representation, and in the material practices of their demonstration—reiterates a commitment to the figure of a human subject characterized by bounded individuality and to the reproduction of an order of hierarchical humanity deeply rooted in imperial/colonial histories. The reading of humanoid robot mediations that follows is part of a broader critical engagement with projects to configure robots in the image of living creatures, and in particular humans and their companion species. Tracking and responding to media reports of these developments, I try to identify alternative resources from anthropology, science and technology studies, feminist and post/decolonial scholarship that can help us to question the assumptions that these stories repeat, at the same time that they purport to be telling us about things that are unprecedented and, most disturbingly, sure to happen. My aim is to destabilize the authority, the credibility, of these narratives of humanoid (and more broadly lifelike) robots, in order to hold open a space for critical analysis that might enable, in turn, very different technological projects.

Relocating Imaginaries of Machine Intelligence At the end of 2014, a media storm followed the statement by physicist Stephen Hawking that “The development of artificial intelligence could spell the end of the human race.”1 The context was a question from a BBC interviewer about an upgrade to Hawking’s speech generation technology, but that specificity was lost in the more universal scope of Hawking’s remarks. An artificial intelligence that could match or surpass the human, Hawking explained, “would take off on its own, and re-design itself at an ever increasing rate. Humans, who are limited by slow biological ­evolution,

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

37

couldn’t compete, and would be superseded.” The BBC’s coverage of Hawking’s statements is followed by reassurances from Rollo Carpenter, creator of the software agent “Cleverbot,” that while full artificial intelligence will be achieved sometime in the next few decades, “we will remain in charge of the technology for a decently long time and the potential of it to solve many of the world problems will be realised” (Cellan-­ Jones 2014). In their respective representations of technology as either the out-of-­ control problem or the in-control solution for the human condition, these quotes reiterate the universalizing progress narrative that underwrites announcements of our supersession as humans by machines. It is that narrative that comprises “the Singularity,” an idea popularized by AI enthusiast Raymond Kurzweil, to reference the “tipping point” at which machine intelligence will exceed our own.2 Reinforced by expressions of concern like that issued by Hawking, this premise circulates across the worlds of science fiction and science fabulation, calling us to rally a response.3 But what is really worrying about the Singularity is not the future that it foretells so much as the imaginary of the evolving human and His (using that pronoun advisedly) relations with technology that it recapitulates. A helpful counter-narrative is offered by STS scholar Hélène Mialet’s study of Stephen Hawking himself (2012), which refigures Hawking from a bounded individual to what Mialet names “a distributed-centred subject.” Mialet’s analysis disperses the boundaries of Hawking’s agency, from that of the individual genius to a collective of variously positioned humans and associated technologies, which together make his agencies (including his centering within the network) possible. It is that relocation that we need to elaborate as a method. Relocation involves among other things restoring cultural histories, including gendered politics, to robot narratives. As one starting place we might return to the 1950 paper by mathematician Alan Turing, taken to be the origin of the so-called Turing test for machine intelligence, otherwise known as the Imitation Game (1950).4 Figure 3.1 shows the most common diagram of the test, with an unmarked “person” and a test computer on one side, an interrogator attempting to differentiate them on the other. But Turing describes the test quite differently: It is played with three people, a man (A), a woman (B), and an interrogator (C) who may be of either sex. The interrogator stays in a room apart from

38 

L. SUCHMAN

Fig. 3.1  Setup for the Turing test. (By Hugo Férée – Own work, CC BY-SA 3.0, https:// commons.wikimedia. org/w/index. php?curid=17059503. Accessed December 24, 2017)

the other two. The object of the game for the interrogator is to determine which of the other two is the man and which is the woman. He knows them by labels X and Y, and at the end of the game he says either ‘X is A and Y is B’ or ‘X is B and Y is A.’ The interrogator is allowed to put questions to A and B thus: C: Will X please tell me the length of his or her hair? Now suppose X is actually A … it is A’s object in the game to try and cause the interrogator to make the wrong identification. His answer might therefore be: ‘My hair is shingled, and the longest strands are about nine inches long.’ In order that tones of voice may not help the interrogator the answers should be written, or better still, typewritten. The ideal arrangement is to have a teleprinter communicating between the two rooms. Alternatively the question and answers can be repeated by an intermediary. The object of the game for the third player (B) is to help the interrogator. The best strategy for her is probably to give truthful answers. She can add such things as ‘I am the woman, don’t listen to him!’ to her answers, but it will avail nothing as the man can make similar remarks. We now ask the question, ‘What will happen when a machine takes the part of A in this game?’ Will the interrogator decide wrongly as often when the game is played like this as he does when the game is played between a man and a woman? These questions replace our original, ‘Can machines think?’ (1950, 433)

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

39

The setup, in sum, is somewhat more complicated than it is generally represented, as a trial between a person in one room and a computer in another, both attempting/programmed to persuade the interrogator that they are human. Rather than human versus machine, the contest is around whether a man or a computer can best enact a convincing imitation of normative/feminine gender identity, under conditions where appearance is reduced to written words. The extraordinary erasure of the Imitation Game’s gendered specificity is part of what produces it as the “universal” test of machine intelligence. This is part of a wider project of figuring intelligence as a universal, a capacity that can be abstracted from its specific cultural and historical relations and enactments. And of course Turing himself embodied complex cultural and political contests over masculinity, sexuality, and masquerade.5

Generating Robotic Progeny Feminist critiques of the AI project have suggested that its aspirations include a form of male birthing; at the least, the robotic imaginary frequently deploys the figure of the child, and of individual development as well as species evolution (Adam 1998; Helmreich 1998; Castañeda and Suchman 2014). A quote from Turing’s seminal article indicates that he was also captivated by the premise that a computational machine might be created that would have the developmental potential of a human child: Instead of trying to produce a programme to simulate the adult mind, why not rather try to produce one which simulates the child’s? If this were then subjected to an appropriate course of education one would obtain the adult brain. (1950, 456)

Turing’s musing exemplifies the conjoining of Anglo-European colonialism and cognitivism that is a hallmark of the AI imaginary, enacted in the figure of the (unmarked/White/male) adult and child brain, and its normative formation through “an appropriate course of education.” This fantasy is reiterated in contemporary fictional renderings of the humanlike machine: as just one prominent example, in the year 2001, the originary thinking machine HAL  9000  in the 1968 film 2001: A Space Odyssey is refigured, in Steven Spielberg’s film A.I.  Artificial Intelligence, as an almost perfect human replica, in the form of a robot child (A.I. 2001). Claudia Castañeda explores the figure of the child in a

40 

L. SUCHMAN

variety of contexts (developmental, postcolonial) in her book Figurations (2002). She explains figuration as method this way: To use figuration as a descriptive tool is to unpack the domains of practice and significance that are built into each figure … Understood as figures, particular categories of existence can also be considered in terms of their uses – what they ‘body forth’ in turn. Figuration is thus understood … to incorporate a double force: constitutive effect and generative circulation. (2002, 3)

Following Castañeda involves tracking the histories of which figures are an effect, and the generative work that they do in the present. Castañeda is drawing in turn on critical feminist science studies, in their orientation to tracing how technological projects materialize cultural imaginaries, an idea developed most extensively by Donna Haraway. Somewhat surprisingly, we find Haraway talking about AI in her major work on the field of primatology, published in 1989: Children, AI computer programs, and nonhuman primates: all here embody ‘almost minds.’ Who or what has fully human status?… What is the end, or telos, of this discourse of approximation, reproduction, and communication, in which the boundaries among and within machines, animals, and humans are exceedingly permeable? Where will this evolutionary, developmental, and historical communicative commerce take us in the techno-bio-­ politics of difference? (1989, 376)

These questions were the inspiration for a paper titled “Robot Visions” (Castañeda and Suchman 2014), conceived at a conference at Goldsmiths College in London in 2003 on the theme of “Passionate Machines” and inspired by the presence of the roboticist Steve Grand. Grand is the inventor/father of Lucy the robot orangutan, a robot child who he was “bringing up” at the time in his family’s home in Somerset. In a perfect if unwitting Harawayan synthesis, Grand explicitly links Lucy the robot orangutan to Lucy the chimpanzee, raised from infancy to adulthood in the home of psychotherapist Maurice Temerlin and his wife Jane, along with their human son Steven. Grand’s book title, Growing Up with Lucy (2003) is a direct citation of Temerlin’s 1976 book Lucy: Growing Up Human. Grand figures Lucy the robot baby as kin to other child-apes who have approximated humans in scientific research on the origins and essence of human nature. “Robot Visions” explores the ways in which the robot

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

41

and the primate child have been figured as mirrors of the (unmarked/ White/male) adult human-in-the-making, and how they might be refigured to allow us to tell different stories about ourselves as humans, our companion species, and our machines.

Where Is Deep Blue? While these examples mobilize figuration as a way of extending the temporal/historical frame of AI/robot projects, we can also expand the frame more spatially, aiming to recover the connections and networked relations that dominant renderings obscure. A case in point is the first technological assemblage declared as a demonstration of the achievement of a humanlike artificial intelligence, IBM’s project Deep Blue.6 A canonical site for enacting AI, given its powerful association with human genius, chess is well-suited to the particular capabilities of computation insofar as it is, or can be modeled as, a closed world with a fixed set of rules that generate an extraordinarily large set of possibilities. For the most part chess-playing computers rely on “brute force”; that is, sheer speed and processing power. On May 11, 1997, the story goes, an IBM computer named Deep Blue defeated world chess champion Garry Kasparov in the deciding game of a six-game match, leading to widespread media declarations that AI had finally achieved the holy grail of a machine intelligence that surpassed the human. An Associated Press photo shows a moment in the final match.7 On the left side of the table sits Kasparov, hunched over the board, presumably contemplating his next move. Beside him is a pen and paper, the classic “analog” medium for the aide-de-memoire. Across the table from Kasparov sits another man, not named, whose gaze is directed at a computer terminal to his right (he also, we might note, has access to a pad of paper and pen). So where, we might ask, is Deep Blue? To answer that question requires us to move outside of the frame of this image. The man pictured is not, of course, Kasparov’s opponent, but rather a peripheral device mediating the input/output relations between the chessboard (for which he is the requisite pair of eyes and hands) and the terminal to his right, the interface to the processors. As the game proceeds he enters Kasparov’s moves via the terminal, and executes the moves displayed there in response. The terminal, in turn, is networked to an array of 512 PowerPC processors, positioned somewhere off-stage and out of sight, combined with a raft of specialized chips designed to evaluate positions and coordinate data in real

42 

L. SUCHMAN

time, which in turn are maintained by a staff of technicians. The code being processed, moreover, is the product of extensive labors of software and electrical engineers, informed by chess experts consulting with IBM as part of the team. Far from a singular entity, in other words, Deep Blue is the nominalization of a network of laboring bodies, including humans as well as multiple machines. So what of Kasparov? Following Mialet’s diagnosis of the agencies of physicist Stephen Hawking, we can see Kasparov as well not as a bounded individual, but rather as a “distributed-centred subject.” However rightly celebrated as an individual, Kasparov’s genius is an effect of his relations with, among others, the long lineage of chess players and games through which his expertise is constituted. Rather than Kasparov being a singular agent while his opponent is distributed, in other words, both comprise material-semiotic relations singularized through their rendering as bounded individuals. This does not imply, however, that their agencies are the same, or even arguably equivalent. Kasparov’s competencies comprise his formation as a person deeply entangled with a very specific cultural practice, for which he has a demonstrable “gift” and at which he has trained with great commitment and discipline. Going into the final game, however, Kasparov reported that he was tired and, in his words, “not in the mood to play.”8 Deep Blue, on the other side, was neither tired nor in any mood. Kasparov made a small error, which might well have been missed by a human opponent, but was not missed by Deep Blue’s search and analysis engines. Given any current board configuration, the system analyzed the possibilities ten moves ahead, examining 200 million board positions each second, before producing a next move. Given that chess players take an average of three minutes to make a move, Kasparov would have to stare at the board for 360 years to examine the moves that Deep Blue evaluated every three minutes.9

Bodies in Relation10 The absence of embodied agencies and affective capacities in the case of chess-playing machines has not gone unnoticed within the fields of AI and robotics. Beginning in the early 1990s, Rodney Brooks and colleagues at MIT’s Computer Science and Artificial Intelligence Laboratory turned their attention to what Brooks named “situated robotics,” foregrounding the essential place of embodiment in the enactment of intelligent agency.11 This approach was elaborated by Brooks into humanoid robotics projects

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

43

focused on hand/eye coordination, and subsequently by his student, Cynthia Breazeal, with respect to affect, materialized most famously as the robots Cog and Kismet, respectively (both now “retired” to MIT’s science museum). Both projects are represented through an extensive corpus of media renderings—stories, photographs, and QuickTime videos available on MIT’s website.12 Pictured from the “waist” up, Cog appears as the upper torso of a freestanding humanoid, while Kismet’s overview offers a series of recorded demonstrations of the robot’s expressive capabilities and interactions, closely framing the robot’s “face.” Sitting between the genre of documentary film and system “demo,” the videos create a record that can be reliably repeated and reviewed in what becomes a form of eternal ethnographic present. These reenactments thereby imply that the capacities that they record have an ongoing existence; that like any other living creatures Cog’s and Kismet’s agencies are not only ongoing, but also continuing to develop and unfold. And like other conventional documentary productions, these depictions are framed and narrated in ways that instruct the viewer in what to see. I was able to experience my own encounters with the robots Cog and Kismet on a visit to the MIT AI lab in the fall of 2001. The graduate student who led our tour explained apologetically that Cog was inactive and had been so for some time. As there were no researchers actively working on its development, Cog was afflicted by a condition commonly known as “bit rot”: that is, the degradation of the robot’s animating software in the absence of ongoing maintenance and upgrading. We were, however, able to witness the inanimate Cog sitting in a corner of the lab. While still an imposing figure, what struck me most about Cog was the remainder of its “body” not easily visible in media portrayals. The base of Cog’s torso was a heavy cabinet from which came a thick sheaf of connecting cables, running centaur-like to a ceiling-high bank of processors that provided the computational power required to bring Cog to life. Seeing the robot “at home” in the lab, situated in this backstage environment, provided an opportunity to see as well the extended network of human labors and affiliated technologies that afford Cog its agency, rendered invisible in its typical media staging as Rod Brook’s singular creation, and as an autonomous entity. While Kismet was operational, in contrast to the interlocutors pictured in the website videos, none of our party was successful in eliciting coherent or intelligible behaviors from it. Framed as an autonomously affective entity Kismet, like Cog, must be said to have failed in its encounters with my colleagues and me. But, as in the case of Cog, there are more interest-

44 

L. SUCHMAN

ing and suggestive lessons to be learned from the difference between Kismet’s demonstrated competencies and the Kismet that we encountered. Those lessons require that we reframe Kismet, like Cog, from an unreliable autonomous robot, to a collaborative agency made possible through very particular sociotechnical relations, and through reiteratively developed and refined performances. The contrast between my own encounter with Kismet and that recorded on the demonstration videos makes clear the ways in which Kismet’s affect is an effect not simply of the device itself, but of Breazeal’s trained reading of Kismet’s actions and her extended history of labors with the machine. In the absence of Breazeal, correspondingly, Kismet’s apparent randomness attests to the robot’s reliance on the performative capabilities of its very particular “human caregiver.”13 Like all forms of agency, in other words, Cog’s and Kismet’s capacities for action are generated through sociomaterial relations that instantiate particular histories, enabling more and less reliable, always contingent, future reenactments. In “Subject Objects” (Suchman 2011), I continue the exercise of recovering the embodied relations that enable robotic agencies through the case of a descendant of Kismet, the robot Mertz, designed by Breazeal’s student Lijin Aryananda.14 As part of an experiment conducted sometime in 2004, Mertz was installed in the atrium lobby of the Frank Gehry-­ designed building that was by then home to MIT’s departments of Computer Science and Artificial Intelligence. Meant to engage passing humans in communicative exchange, Mertz was the materialization of what its designers describe as “a robotic creature that lives around people on a regular basis and incrementally learns from its experience” (Aryananda 2005). According to reports on the experiment, Mertz ran more or less continuously for seven hours a day over five days, recording its own visual and aural input and corresponding speech output every second, and— according to this autobiographical record—engaging over 600 passersby. I have been captivated less by the technical papers, however, than by one of these encounters, videotaped by one member of a group that engaged Mertz, and subsequently posted (of course) on YouTube.15 My reading of this encounter focuses on the entangled boundaries of subjects and objects that it vividly demonstrates. I am fascinated by the moment-to-moment, shifting choreography of its lively objects and obliging subjects. The humans’ unremarkable interactional competencies alternately shape them into appropriately cartooned subjects for reciprocal engagement with Mertz, and reframe the robot as an object of their shared puzzlement and pleasure. Mertz, meanwhile, robotically translates the

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

45

fragments of sound and motion that these noisy objects emit into readable signals and enacts its own, in-built, logic of more and less sensible replies. Entrained by Mertz’s vitality, the human interlocutors are robotically subjectified; shifting their orientation to each other’s queries and laughter, the robot is correspondingly restored to humanlike objectness. The interactivity of persons and things is manifest here as moments of bodily imitation and connection animated by affective dynamics that escape their classification. These are what Haraway has identified as an experiment’s “immeasurable results” (1997, xiii).

Slow Robots and Slippery Rhetorics In mid-December 2013, the US Defense Advanced Research Projects Agency (DARPA) sponsored its second annual Robotics Challenge at a NASCAR racetrack near Homestead, Florida. The event seemed to have a refreshingly sobering effect on media outlets that covered it. A field comprising 16 competitors, the victors of an earlier trial, the teams represented the leading laboratories internationally invested in the development of mobile, and more specifically “legged” robots. The majority of the teams worked with machines configured as upright, bipedal humanoids, while two figured their robots as nonhuman primates (Robosimian and CHIMP), and one as a nonanthropomorphized “hexapod.” An article by DARPA Program Manager Gill Pratt, published early in December in the Bulletin of the Atomic Scientists, set out Pratt’s aspirations for the Challenge (Pratt 2013). With an invocation of the familiar (and, as I have noted earlier, highly misleading) analogy between the robot and the developing human, Pratt wrote: By the time of the DRC Finals [scheduled for 2015], DARPA hopes the competing robots will demonstrate the mobility and dexterity competence of a 2-year-old child, in particular the ability to execute autonomous, short tasks such as ‘clear out the debris in front of you’ or ‘close the valve,’ regardless of outdoor lighting conditions and other variations.

I would challenge this comparison on the basis that it underestimates the level of the two-year-old child’s competencies, but I suspect that many parents of two-year-olds might question its aptness on other grounds as well. The Challenge staged a real-time, public demonstration of the state of the art; one that proved disillusioning to many who witnessed it. For all

46 

L. SUCHMAN

but the most technically knowledgeable in the audience, the actual engineering achievements were hard to appreciate. More clearly evident was the slowness and clumsiness of the robots, and their vulnerability to failure at what to human contenders would have proven quite unremarkable tasks. The photo gallery that accompanies an article titled “Robots to the Rescue, Slowly” (Knight 2013a) is indicative, and the BBC titles its coverage of the Challenge “Robot Competition Reveals Rise of the Machines Not Imminent” (Paterson 2013). Reporter Zachary Fagenson sets the scene with a representative moment in the competition: As a squat, red and black robot nicknamed CHIMP gingerly pushed open a spring-loaded door a gust of wind swooped down onto the track at the Homestead-Miami Speedway and slammed the door shut, eliciting a collective sigh of disappointment from the audience. (Fagensen 2013)

In the BBC’s video coverage of the event, Dennis Hong, Director of the Virginia Tech Robotics Lab, tells the interviewer: “When many people think about robots, they watch so many science fiction movies, they think that robots can run and do all the things that humans can do. From this competition you’ll actually see that that is not the truth. The robots will fall, it’s gonna be really, really slow” (Paterson 2013). Then DARPA Director Arati Prabhaker concurred: “I think that robotics is an area where our imaginations have run way ahead of where the technology actually is, and this challenge is not about science fiction it’s about science fact” (ibid.). While many aspects of the competition would challenge the separateness of fiction and fact (not least the investment of its funders and competitors in figuring robots as humanoids), this is nonetheless a difference that matters. These cautionary messages are contradicted, however, in a whiplash inducing moment at the close of the BBC clip, when Boston Dynamics Project Manager Joe Bondaryk makes the canonical analogy between the trials and the Wright brothers’ first flight, reassuring us: “If all this keeps going, then we can imagine having robots by 2015 that will, you know, that will help our firefighters, help our policemen to do their jobs” (a short time frame even compared to the remarkable history of flight) (Paterson 2013). The winning team, the University of Tokyo’s spin-out company Schaft (subsequently acquired by Google), attributes their differentiating edge in the competition to a new high-voltage liquid-cooled motor technology making use of a capacitor rather than a battery for power, which

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

47

the engineers explained lets the robot’s arms move and pivot at higher speeds than would otherwise be possible. Second and fourth place went to teams that had adopted the Boston Dynamics (also recently acquired by Google) Atlas robot as their hardware platform, the Florida Institute for Human and Machine Cognition (IHMC) and MIT teams, respectively. Third place went to Carnegie Mellon University’s CHIMP, while one of the least successful entrants, scoring zero points, was NASA’s Valkyrie.16 In a reiteration of the unmarked category with respect to the robots’ gendering, Valkyrie is described in media reports as the only gendered robot in the group (as signaled by its white plastic vinyl body and suggestive bulges in the “chest” area, a convenient housing, it turns out, for the linear actuators that control the robot’s “waist”). Asked about the logic of Valkyrie’s form factor, Christopher McQuin, NASA’s chief engineer for hardware development, offered: “The goal is to make it comfortable for people to work with and to touch” (Fagensen 2013). Despite its sponsorship by DARPA, the agency charged with developing new technology for the military, the Challenge is framed not in terms of military research and development, but as an exercise in the development of “rescue robots.” More specifically, DARPA statements position the Challenge itself, along with the eight tasks assigned to the robotics teams (e.g., walking over rubble, clearing debris, punching a hole in a drywall, turning a valve, attaching a fire hose, climbing a ladder) as a response to the disastrous melt down of the Fukushima-Daiichi nuclear power plant. The theme of rescue is reiterated by journalist Will Knight (2013c), writing about the Challenge, who includes the tag line: “Why it Matters: If they can become nimbler, more dexterous, and safer, robots could transform the way we work and live.” Knight implies that we should care about robots, their actual clumsiness and unwieldiness notwithstanding, because if they were like us, they could transform our lives. The invocation of the way we live here echoes the orientation of the Challenge overall, away from robots as weapons—as instruments of death—and towards the figure of the first responder as the preserver of life. In elaborating the motivation for the Challenge, Pratt (2013) invokes a grab bag of familiar specters of an increasingly “vulnerable society” (population explosion with disproportionate numbers of frail elderly, climate change, weapons of mass destruction held in the wrong hands) as calling for, if not a technological solution, at least a broad mandate for robotics research and development:

48 

L. SUCHMAN

The world’s population is continuing to grow and move to cities situated along flood-prone coasts. The population over age 65 in the United States is forecast to increase from 13 percent to 20 percent by 2030, and the elderly require more help in emergency situations. Climate change and the growing threat of proliferation of weapons of mass destruction to non-state actors add to the concern. Today’s natural, man-made and mixed disasters might be only modest warnings of how vulnerable society is becoming.

One implication of this enumeration is that even disaster can be good for business, and humanoid robotics research, Pratt assures us, is all about saving lives and protecting humans (a term that seems all-encompassing at the same time that it erases from view how differently actual persons are valued in the rhetorics of US Homeland Security). The figure of the “war-­fighter” appears only once towards the end of Pratt’s piece, and even there the robot’s role in the military is about preserving, not taking life. But many of us are not reassured by the prospect of robot rescue, and would instead call on the US government to take action to mitigate climate change, to dial down its commitment to militarism along with its leading role in the international arms trade, and to invest in programs to provide meaningful jobs at a living wage to humans, not least those engaged in the work of care. The robot Challenge could truly be an advance if the spectacle of slow robots would raise questions about the future of humanoid robotics as a project for our governments and universities to be invested in, and about the good faith of slippery rhetorics that promise the robot first responder as the remedy for our collective vulnerability.

Domesticating Robot Labors While the invocation of robot rescues following a nuclear accident begs the question of how robots would be hardened against the effects of radiation and at what cost (the robots competing in the challenge already costing up to several million dollars each), Knight (2013b) suggests that if robots can be developed that are capable of taking on these tasks, “they could also be useful for much more than just rescue missions.” He observes that the robot that went on to win the Robotics Challenge in 2015 “is the culmination of many years of research in Japan, inspired in large part by concerns over the country’s rapidly aging population,” a proposition affirmed by DARPA Program Manager Pratt who “believes that home help is the big business opportunity [for] humanoid robots” (Knight

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

49

2013b). Just what the connection might be between these pieces of heavy machinery and care at home is left to our imaginations, but quite remarkably Pratt further suggests “that the challenges faced by the robots involved in the DARPA event are quite similar to those that would be faced in hospitals and nursing homes” (ibid.). The robot’s figuration as subservient worker,17 beginning with its origins in industrial manufacturing, has more recently been brought home in the fantasy of the robot maid or butler. Hedging bets against downturns in future military funding, Boston Dynamics (makers of the Atlas platform) sold itself in 2013 to the Google conglomerate named “Alphabet,” and began to orient to more domestic applications for humanoid robots.18 As part of a huge buy-up of robotics companies under the internal project name “Replicant,” Google promised a general-purpose robot that could cook and take care of the elderly by 2020. A somewhat ironic video posted by Boston Dynamics in 2016 shows Atlas almost managing to wield a vacuum cleaner.19 The irony acknowledges the robot’s ineptitude at this task. That ineptitude is signaled more directly in a Guardian piece headlined “Multimillion Dollar Humanoid Robot Doesn’t Make for a Good Cleaner” (Gibbs 2016). The article reproduces the video clip (produced in fast motion with an upbeat techno music soundtrack) showing Florida’s Institute for Human and Machine Cognition (IHMC) testing new code “by getting the multimillion dollar Atlas robot to do household chores.” Housework, we are told, proves “more difficult than you might imagine.” The challenge of housework is revealing of the conditions required for effective automation. Specifically, robots work well just to the extent that their environments—basically the input that they have to process, and the conditions for an appropriate response—can be engineered to fit their capacities. The factory assembly line has, in this respect, been made into the robot’s home. Domestic spaces, in contrast, and the practicalities of work within them are characterized by a level of contingency that has so far blocked attempts at automation beyond the kinds of appliances that can either depend on human peripherals to set up their conditions of operation (think loading the dishwasher), or can operate successfully through repetitive, initially random motion (think of  the robot vacuum cleaner Roomba and its clones). Long underestimated in the value chain of labor, robotics for domestic work might just teach us some lessons about the extraordinary complexity of these most ordinary human activities. An open-ended Google image search on “elder care” returns a photo accompanying a blog post setting out concerns regarding funding for the

50 

L. SUCHMAN

implementation of the UK’s Care Act of 2014 (Dobie 2015).20 An article published in the Georgia Tech Newsletter around the same time figures a robot caregiver as the companion to an article titled “Would You Trust a Robot to Take Care of Your Grandpa,” in which the Open Roboethics Initiative publishes its findings on public attitudes towards robot caregivers (Georgia Tech Newsletter 2013).21 The first article cites a persistent problem, that of funding for human caregivers. The second presupposes that a solution to this problem is robot caregivers, and that the only remaining problem might be public resistance to them. But the question of whether robot caregivers are trustworthy or desirable begs two prior questions, which receive much less attention in the public media. The first question is on what basis should we accept the statement that adequate funding for human caregiving is impossible? What are the assumptions underlying the premise that there is a shortage of human caregivers, and might those assumptions be open to challenge? The second question left unaddressed in discussions of the ethics of robot caregivers is that raised at the opening of this chapter: that is, on what basis should we accept the proposition that robot caregivers are even a viable possibility? Getting clearer about the answer to this question is necessary if we are to open the space for other ways of thinking about, and investing in, inventive solutions to the problem of care. A closer reading of these images is helpful in thinking about what care—in this case giving medication—entails. We could start with the bodily positioning of the caregivers in each case, in relation to those receiving medication; the multiple ways in which the human caregiver orients to the woman to whom she is handing a pill, leaning in to the appropriate distance, monitoring the woman’s face, and offering her an encouraging smile, aligning the hand holding the pill in relation to the hand that feels it to pick it up, the glass of water waiting for the moment when it will be needed, to mention just the most obvious. In the case of the prospective robot caregiver we might note the substitution of a pill bottle for the individual pill, the reach of the recipient to meet the robot’s grasp, the question of how and why this PR2 mobile robot, a development platform widely adopted in university robotics labs, is imagined as the right tool for this job. We might think about what understanding of “medication,” as both substance and practice, each of the caregivers has, and what further labors of care each might be able to provide in the next frame, the moment after this one.

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

51

The Labors of Violence Having set out the motivation and conditions of the Robotics Challenge, in a section titled “Don’t Be Scared of the Robot,” DARPA Program Manager Gill Pratt turns to the “broad moral, ethical, and societal questions” that the program raises, noting that “although the DRC will not develop lethal or fully autonomous systems, some of the technology being developed in the competition may eventually be used in such systems” (Pratt 2013). He continues: the tempo of modern warfare is escalating, generating a need for systems that can respond faster than human reflexes. The Defense Department has considered the most responsible way to develop autonomous technology, issuing a directive in November 2012 that carefully regulates the way robotic autonomy is developed and used in weapons systems.22 Even though DRC robots may look like those in the movies that are both lethal and autonomous, in fact they are neither. (ibid.)

This statement opens with the circuitous proposition that the escalating speed of modern warfare (observed as if it were a naturally occurring phenomenon) comprises a problem for which faster response is the solution, which in turn justifies the need for further automation, which in turn increases the tempo, ad infinitum (or at least until whatever the endgame is imagined to be). Pratt’s closing reference to “the movies” then loops us back to the concerned roboticists with which I opened this chapter. Amidst endless screen shots from Terminator 3: Rise of the Machines (Terminator 3 2003), and seemingly obligatory invocations of Stephen Hawking, Elon Musk, and Steve Wozniak as signatories, the media reported the release on July 28, 2015 of an open letter signed by thousands of robotics and AI researchers calling for a ban on lethal autonomous weapons.23 The letter’s release to the press was timed to coincide with the opening of the International Joint Conference on Artificial Intelligence (IJCAI 2015) in Buenos Aires. Far more significant than the inclusion of celebrity signatories—their stunning effect in drawing international media attention notwithstanding—is the number of prominent computer scientists (not a group prone to add their names to political calls to action) who have been moved to endorse the letter. Consistent with this combination of noise and signal, the commentaries generated by

52 

L. SUCHMAN

the occasion of the letter’s release ranged from aggravatingly misleading to helpfully illuminating. The former category is well represented in an interview by Fox News anchor Shepard Smith with theoretical physicist and media scientist Michio Kaku.24 In response to Smith’s opening question regarding whether or not concerns about autonomous weapons are overblown, Kaku suggests that “Hollywood has us brainwashed” into thinking that Terminator-style robots are just around the corner. Quite the contrary, he assures us, “we have a long ways to go before we have sentient robots on the battlefield.” This “long ways to go” is typical of futurist hedges that, while seemingly interrupting narratives of the imminent rise of the machines, implicitly endorse the assumption of continuing progress in that direction. Kaku then further affirms the possibility, if not inevitability, of the humanoid weapon: “Now, the bad news of course is that once we do have such robots, these autonomous killing machines could be a game changer.” Having effectively clarified that his complaint with Hollywood is less the figure of the Terminator-style robot than its timeline, he reassures us that “the good news is, they’re decades away. We have plenty of time to deal with this threat.” “Decades away, for sure?” asks Shepard Smith. “Not for sure, cuz we don’t know how progress is,” Kaku replies, and then offers what could be a more fundamental critique of the sentient robot project. Citing the disappointing conclusion of the DARPA Robotics Challenge as evidence, he explains: “It turns out that our brain is not really a digital computer.” The lesson to take from this, he proposes, is that the autonomous killing machine “is a long term threat, it’s a threat that we have time to digest and deal with, rather than running to the hills like a headless chicken” (at which he and Shepard share a laugh). While I applaud Kaku’s skepticism regarding advances in humanoid robots, it is puzzling that he himself frames the question in these terms, suggesting that it is the prospect of humanoid killer robots to which the open letter is addressed, and (at least implicitly) dismissing its signatories as the progeny of Chicken Little.25 Having spent all but 30 seconds of his 3 minutes and 44, Kaku then points out that “one day we may have a drone that can seek out human targets and just kill them indiscriminately. That could be a danger, a drone that’s only mission is to kill anything that resembles a human form … so that is potentially a problem – it doesn’t require that much artificial intelligence for a robot to simply identify a human form, and zap it.” This reference to indiscriminate targeting of any human form brings us closer

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

53

to the realm of the actual capabilities of prospective autonomous weapons. One such technology is the South Korean “Super Aegis” border monitoring and response system deployed (in semi-autonomous mode only so far) in the so-called demilitarized zone between North and South Korea. Like the spatial arrangement that constituted the Wall between East and West Berlin, the demilitarized zone (in fact a hypermilitarized space) marks a territory that constitutes any human as a legitimate target. Under those circumstances, a weapon designed to select targets and deploy force against them becomes imminently possible. One rendering of the view from the weapon features the familiar red box indicating the system’s detection of a human face.26 We might note as well the raised arms of the figure, however, a gesture conventionally designating surrender and placing this person, under international humanitarian law, “outside of combat” and therefore as not a legitimate target. The possibility of such a gesture immediately complicates even the highly constrained “closed world” of the prohibited zone. The threat posed by the “Super Aegis” and its kind is not the prospect of a Terminator-style humanoid or “sentient” bot, but the much more mundane progression of increasing automation in military weapon systems: in this case, automation of the identification of particular categories of humans (those in a designated area, or who fit a specified and machine-­ readable profile) as legitimate targets for killing. The untethering of robotic weapon systems from their human controllers comprises the next logical step in the automation of warfare—a trajectory that I have been involved in working to interrupt along with colleagues in the campaign, led by Human Rights Watch, to “Stop Killer Robots.”27 As roboticist and founding member of the International Committee for Robot Weapons Control (ICRAC), Noel Sharkey points out, rather than assuming humanoid form, lethal autonomous weapons are much more likely to look like already-existing weapons systems, including tanks, battleships, and jet fighters.28 He explains that the core issue for the campaign against autonomous weapon systems is an international ban that would preempt the delegation of “decisions” to kill to machines. A crucial issue here is that this delegation of “the decision to kill” presupposes the specification, in a computationally tractable way, of algorithms for the discriminatory identification of a legitimate target. The latter, under the rules of engagement, international humanitarian law, and the Geneva conventions, is an opponent that is engaged in combat and poses an “imminent threat.” We have ample evidence for the increasing uncertainties involved in differentiating

54 

L. SUCHMAN

combatants from noncombatants under contemporary conditions of war fighting (even apart from crucial contests over the legitimacy of targeting protocols). However partial and fragile their reach, the international legal frameworks governing war fighting are our best current hope for articulating limits on killing. The precedent for a ban on lethal autonomous weapons lies in the United Nations Convention on Certain Conventional Weapons (CCW), the body created to prohibit or restrict the use of “certain conventional weapons which may be deemed to be excessively injurious or have indiscriminate effects.”29 Since the launch of the campaign in 2013, the CCW has put the debate on lethal autonomous weapons onto its agenda. Figure  3.2 shows one of the expert panels held at the CCW in April 2016, in which I presented testimony on the impossibility of ­automating the capacity of what in the military is named “situational aware-

Fig. 3.2  Third Convention on Conventional Weapons meeting on lethal autonomous weapons systems, Geneva, April 12, 2016. (Photo credit: Peter Asaro, International Committee for Robot Arms Control)

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

55

ness,” necessary for discrimination between legitimate and illegitimate targets—or more colloquially and directly, friends and enemies—as a prerequisite to legal killing (Suchman 2016). A core and irremediable problem is that prescriptive frameworks like the laws of war (or any other human-designated directives) presuppose, rather than specify, the capacities for comprehension and judgment required for their implementation in any actual situation. And it is precisely those capacities that artificial intelligences lack, now and for the foreseeable future.

Conclusion To appreciate the urgency of interventions into the development of lethal autonomous weapons, the field of science and technology studies offers a useful concept. The idea of “irreversibility” points to the observation that, while technological trajectories are never self-determining or inevitable, the difficulties of undoing technological projects grow over time (Callon 1990). Investments (both financial and political) increase as does the iterative installation and institutionalization of associated infrastructures (both material and social). The investments required to dismantle established systems expand commensurately. While those of us engaged in thinking through STS are preoccupied with the contingent and shifting distributions of agency that comprise complex sociotechnical systems, the hope for calling central human actors to account for the effects of those systems rests on the possibility of articulating relevant normative and legal frameworks. This means that we need conceptions of agency that recognize the inseparability of humans and technologies, and the always contingent nature of autonomy, in ways that help to reinstate human deliberation at the heart of matters of life, social justice, and death. The limits on “situational awareness” that render autonomous robot soldiers unlawful within the military’s own terms index a profound and enduring problem for the creation of humanlike machines more generally. The unquestionable expansion of computational agencies over the past few decades, as I suggested at the opening of this chapter, are due to advances in the capacities of data storage and the efficiency of processing algorithms on the one hand, and the extension of network infrastructures on the other. Combined with developments in computer graphics and animation, these developments have generated a broader and more ­intimate engagement with machines for those who inhabit relevant zones

56 

L. SUCHMAN

of technological investment. However, the capacities of embodied sociality that characterize the particular animals that humans are, and which are presupposed in our everyday practices of going on together and becoming with one another, have proved resistant to translation into the material-­ semiotic resources that computation affords. As a consequence, the life of the humanoid robot has remained within the status of the promissory prototype and the carefully curated bounds of the demonstration. At the same time, the framings that demonstrate humanoid robot agencies are based on a series of erasures that work to obscure the material practices and extended relations on which the agencies not only of robots, but also humans depend. Those materialities and relations, made evident, challenge the figure of the human that animates popular robot imaginaries. Abandoning the fantasy of the autonomous individual as the locus of agency might allow us to reframe roboticists, robots, and human-robot interactions into less normatively replicative, more generative and transformative configurations. And articulating the interests that promote robotics projects as a necessary progression in one very particular, albeit dominant, historical narrative might clear a space for other stories, more aligned with a commitment to the proposition that other sociotechnical relations are possible.

Notes 1. See, among many other reports, Cellan-Jones (2014), http://www.bbc. com/news/technology-30290540. Accessed December 24, 2017. 2. For the full set of Kurzweil’s prognostications on the subject see http:// singularity.com/. Accessed December 24, 2017. 3. It is important to note that the science fictions and fabulations referenced here are antithetical to the genres of SF (science fiction, speculative fabulation, speculative feminism, and science fact) celebrated by Donna Haraway in her writings, most recently Staying with the Trouble (2016). 4. For another reflection on the role of gender in the game, see the preface to Hayles (1999). See also Rhee (2018, 13–14). 5. Turing was prosecuted in 1952 for homosexual acts of “gross indecency,” then a criminal offense in the UK. He accepted chemical castration treatment, with DES, as an alternative to prison. He died two years later, just before his 42nd birthday, from cyanide poisoning. In 2009, following an Internet campaign, British Prime Minister Gordon Brown made an official public apology on behalf of the British government for “the appalling way he was treated,” and Queen Elizabeth II granted him a posthumous par-

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

57

don in 2013 (BBC News 2013). The Alan Turing law is now an informal term for a 2017 law in the UK that retroactively pardoned men cautioned or convicted under historical legislation that outlawed homosexual acts (BBC News 2016). 6. For IBM’s own recounting of this history see http://www-03.ibm.com/ ibm/history/ibm100/us/en/icons/deepblue/. Accessed December 24, 2017. 7. Garry Kasparov playing against Deep Blue, the chess-playing computer built by IBM.  Adam Nadel/AP Images. See https://www.britannica. com/media/full/topic/155485/61084, accessed April 30, 2019. 8. http://whyfiles.org/040chess/main1.html, accessed via the WayBack Machine https://web.archive.org/web/20130103124847/; http:// whyfiles.org/040chess/main1.html/. Accessed December 24, 2017. 9. The New York Times reported at the time that, to their credit, “I.B.M. scientists have taken pains to emphasize that Deep Blue is just a glorified calculator. On a special Web page put up for the occasion, I.B.M. describes its chess expert like this: ‘Deep Blue is a machine that is incapable of feeling or intuition…. Deep Blue is stunningly effective at solving chess problems, but it is less ‘intelligent’ than even the stupidest human’” (Johnson 1997). 10. This account is drawn in part from Suchman (2007a, chapters 13 and 14). 11. See Brooks (1990), Brooks et al. (1998), Breazeal and Velasquez (1998), Brooks (2002), Breazeal (2002). For a critical reflection of the sense of embodiment mobilized in Brooks’ project, see Suchman (2007b). 12. See for example http://www.ai.mit.edu/projects/humanoid-roboticsgroup/cog/overview.html; http://www.ai.mit.edu/projects/humanoidrobotics-group/kismet/kismet.html. Accessed December 26, 2017. 13. See http://www.ai.mit.edu/projects/sociable/overview.html. Accessed on December 26, 2017. In an interview (Menzel and D’Aluisio 2000, 71), Breazeal responds to the suggestion that Kismet might be tuned specifically to her as an interactional partner with the hope that this is an early stage of the robot’s development, that will improve with time; see discussion of the child above. 14. A kindred analysis of relations between human and robotic bodies is provided by Morana Alac (2009). 15. See http://www.youtube.com/watch?v=HuqL74C6KI8. Accessed on December 26, 2017. 16. See https://upload.wikimedia.org/wikipedia/commons/3/34/Valkyrierobot-3.jpg, accessed April 30, 2019. 17. See Karen Asp’s excellent blog post on the robot Pepper (Asp 2016), including claims to emotional intelligence and substitutability for service work. Asp’s insightful observations regarding the gap between the “spirit”

58 

L. SUCHMAN

of capitalism in the arena of humanoid robots and the “body” of the robot, focusing on the dream of laborless manufacture and service provision and the realities of labor politics in both arenas, intersects with another, related gap, between the future imaginary of robotic capabilities, and the real lack of progress in robot sociality (i.e. Pepper embodies the current state of the art in image/speech recognition plus a networked tablet). For an incisive analysis of relations between “the robotic imaginary,” dehumanization, and the devaluation of labor see Rhee (2018). 18. The company was sold on again by Alphabet in June 2017 to the Japanese firm Softbank, makers of the robot Pepper. 19. Available at https://www.youtube.com/watch?v=r0qWVKcJR3w. Accessed December 26, 2017. 20. See https://theknowledgeexchangeblog.com/2015/03/13/issues-aroundcare-for-older-people-towards-the-end-of-life/. Accessed December 26, 2017. 21. See https://www.news.gatech.edu/2013/04/29/how-would-you-yourassistant-human-or-robotic, accessed 12 November 2018. 22. US Department of Defense (DoD), Autonomy in Weapon Systems, Directive 3000.09, 21 November 2012, 14 states: “Autonomous and semi-­autonomous weapons systems shall be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.” As many critics have pointed out, this begs the question of how the requisite of levels of judgment would be adjudicated, and by whom. 23. https://futureoflife.org/open-letter-autonomous-weapons. Accessed December 36, 2017. 24. “Tech Experts Warn of ‘Killer Robot’ Arms Race, Call for Ban,” July 28, 2015. http://video.foxnews.com/v/4381759480001/?#sp=show-clips. Accessed December 26, 2017. 25. See https://en.wikipedia.org/wiki/Henny_Penny. Accessed December 26, 2017. 26. See https://curiousmatic.com/how-tech-trends-are-enabling-the-rise-ofkiller-robots/. Accessed November 12, 2018. 27. See https://www.stopkillerrobots.org/. Accessed December 26, 2017. 28. See https://www.cnet.com/news/ban-autonomous-weapons-urge-hundreds-of-experts-including-hawking-musk-and-wozniak/. For ICRAC see https://icrac.net/. Accessed December 26, 2017. 29. https://www.unog.ch/80256EE600585943/(httpPages)/4F0DEF093 B4860B4C1257180004B1B30?OpenDocument. Accessed December 26, 2017.

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

59

References A.I. Artificial Intelligence. 2001. Directed by Steven Spielberg. Warner Brothers and DreamWorks. Adam, Alison. 1998. Artificial Knowing: Gender and the Thinking Machine. New York: Routledge. Alac, Morana. 2009. “Moving Android: On Social Robots and Body-in-­ Interactions.” Social Studies of Science 39 (4): 491–528. Aryananda, Lijin. 2005. “Out in the World: What Did the Robot Hear and See?” Epigenetic Robotics. http://people.csail.mit.edu/lijin/publication.html Asp, Karen. 2016. “Learning American Manners: Pepper Robots, My Fair Lady and the Spirit of Capitalism.” Social Robot Futures. Website post June 22, 2016. https://socialrobotfutures.com/?s=Pepper. Accessed 26 Dec 2017. BBC News. 2013. “Royal Pardon for Codebreaker Alan Turing.” December 24. https://www.bbc.com/news/technology-25495315. Accessed 12 Nov 2018. ———. 2016. “Alan Turing Law: Thousands of Gay Men to be Pardoned.” October 20. https://www.bbc.com/news/uk-37711518. Accessed 12 Nov 2018. Breazeal, Cynthia. 2002. Designing Sociable Robots. Cambridge, MA: MIT Press. Breazeal, Cynthia, and Velasquez, Juan. 1998. “Toward Teaching a Robot ‘Infant’ Using Emotive Communication Acts.” Paper presented at the Proceedings of 1998 Simulation of Adaptive Behavior, workshop on Socially Situated Intelligence, Zurich, Switzerland. Brooks, Rodney. 1990. “Elephants Don’t Play Chess.” Robotics and Autonomous Systems 6: 3–15. ———. 2002. Flesh and Machines: How Robots Will Change Us. 1st ed. New York: Pantheon Books. Brooks, Rodney, Cynthia Breazeal, Matthew Marjanovic, Brian Scassellati, and Matthew Williamson. 1998. “The Cog Project: Building a Humanoid Robot.” In Computation for Metaphors, Analogy and Agents, ed. C.  Nehaniv, 8–13. Berlin: Springer-Verlag. Callon, Michel. 1990. “Techno-Economic Networks and Irreversibility.” The Sociological Review 38: 132–161. Castañeda, Claudia. 2002. Figurations: Child, Bodies, Worlds. Durham/ London: Duke. Castañeda, Claudia, and Lucy Suchman. 2014. “Robot Visions.” Social Studies of Science 44 (3): 315–341. Cellan-Jones, Rory. 2014. “Stephen Hawking Warns Artificial Intelligence Could End Mankind.” BBC News Technology, December 2. http://www.bbc.com/ news/technology-30290540. Accessed 24 Dec 2017. Dobie, Laura. 2015. “Issues Around Care for Older People Towards the End of Life.” The Knowledge Exchange Blog, March 13. https://theknowledgeexchangeblog.com/2015/03/13/issues-around-care-for-older-people-towardsthe-end-of-life/. Accessed 26 Dec 2017.

60 

L. SUCHMAN

Fagensen, Zachary. 2013. “Robots to the Rescue.” Business Report, December 23. https://www.iol.co.za/business-report/technology/robots-to-the-rescue-1625955#.Urnv6mRDsXZ. Accessed 26 Dec 2017. Georgia Tech Newsletter. 2013. “How Would You Like Your Assistant—Human or Robotic?” April 29. Gibbs, Samuel. 2016. “Multimillion Dollar Humanoid Robot Doesn’t Make for a Good Cleaner.” The Guardian, January 19. https://www.theguardian.com/ technology/2016/jan/19/multimillion-dollar-humanoid-robot-doesntmake-for-a-good-cleaner. Accessed 26 Dec 2017. Grand, Steve. 2003. Growing Up with Lucy: How to Build an Android in Twenty Easy Steps. London: Weidenfeld & Nicolson. Haraway, Donna. 1989. Primate Visions: Gender, Race, and Nature in the World of Modern Science. New York: Routledge. ———. 1997. Modest_Witness @Second_Millenium.FemaleMan_Meets_OncoMouse™: Feminism and Technoscience. New York: Routledge. ———. 2016. Staying with the Trouble: Making Kin in the Chthulucene. Durham/ London: Duke University Press. Hayles, N.  Katherine. 1999. How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics. Chicago: University of Chicago Press. Helmreich, Stefan. 1998. Silicon Second Nature: Culturing Artificial Life in a Digital World. Berkeley: University of California. Johnson, George. 1997. “Ghost in the Chess Machine: A Brain, or Just a Black Box?” New York Times, May 9. Knight, Will. 2013a. “Robots to the Rescue, Slowly.” MIT Technology Review, December 21. https://www.technologyreview.com/s/523086/robots-tothe-rescue-slowly/. Accessed 26 Dec 2017. ———. 2013b. “Google’s Robot Recruits Dominate DARPA’s Rescue Challenge.” MIT Technology Review, December 21. https://www.technologyreview. com/s/523091/googles-robot-recruits-dominate-darpas-rescue-challenge/. Accessed 26 Dec 2017. ———. 2013c. “Q&A with James Kuffner, Google Robotics Researcher.” MIT Technology Review, December 23. https://www.technologyreview. com/s/523096/qa-with-james-kuffner-google-robotics-researcher/. Accessed 26 Dec 2017. Menzel, Peter, and Faith D’Aluisio. 2000. Robo sapiens. Cambridge, MA: MIT Press. Mialet, Hélène. 2012. Hawking Incorporated: Stephen Hawking and the Anthropology of the Knowing Subject. Chicago: University of Chicago Press. Paterson, Leigh. 2013. “Robot Competition Reveals Rise of the Machines Not Imminent.” BBC News Online, December 23. http://www.bbc.com/news/ av/world-us-canada-25469077/robot-competition-reveals-rise-of-themachines-not-imminent. Accessed 26 Dec 2017.

3  DEMYSTIFYING THE INTELLIGENT MACHINE 

61

Pratt, Gill. 2013. “Robot to the Rescue.” Bulletin of the Atomic Scientists, December 3. https://thebulletin.org/robot-rescue. Accessed 26 Dec 2017. Rhee, Jennifer. 2018. The Robotic Imaginary. Minneapolis: University of Minnesota Press. Suchman, Lucy. 2007a. Human-Machine Reconfigurations: Plans and Situated Actions. Rev. ed. New York: Cambridge. ———. 2007b. “Feminist STS and the Sciences of the Artificial.” In The Handbook of Science and Technology Studies, ed. E. Hackett, O. Amsterdamska, M. Lynch, and J. Wajcman, 3rd ed., 139–163. Cambridge, MA: MIT Press. ———. 2011. “Subject Objects.” Feminist Theory 12 (2): 119–145. ———. 2016. “Situational Awareness and Adherence to the Principle of Distinction as a Necessary Condition for Lawful Autonomy.” In Lethal Autonomous Weapon Systems: Technology, Definition, Ethics, Law & Security, ed. R. Geiss and H. Lahmann, 273–283. Berlin: Federal Foreign Office, Division Conventional Arms Control. Temerlin, Maurice. 1976. Growing Up Human: Chimpanzee Daughter in a Psychotherapist’s Family. New York: Doubleday. Terminator 3: Rise of the Machines. 2003. Directed by Jonathan Mostow. Warner Brothers. Turing, Alan. 1950. “Computing Machinery and Intelligence.” Mind 59: 433–460.

CHAPTER 4

Autonomy of Artificial Intelligence, Ecology, and Existential Risk: A Critique Karen Asp

Introduction In the spring of 2014, Stephen Hawking et al. (2014) launched a public campaign to warn of the risks posed by progress in the field of “artificial intelligence” (AI). In an Op-Ed piece published in The Independent they claimed that “[s]uccess in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks.” The Op-Ed article was followed by “An Open Letter” on AI that garnered 8000 signatures.1 The campaign expressed concerns about existential threats to the “future of humanity” and the “future of life” arising from uncontrolled technological progress. Philosopher Nick Bostrom, a central figure in this movement, defines an “existential risk” as “one that threatens to cause the extinction of Earth-­ originating intelligent life or to otherwise permanently and drastically destroy its potential for future desirable development” (Bostrom 2014, 115). The notion that AI might pose such a risk goes hand in hand with the ambition to build superpowerful computers capable of fully autonomous activity. As the movie Transcendence (2014) showed, on the one hand, this ambition is linked to utopic ideas about how these machines

K. Asp (*) Faculty of Environmental Studies, York University, North York, ON, Canada © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_4

63

64 

K. ASP

would precipitate “exponential growth” for the good of humanity (e.g. Good 1966; Kurzweil 2005; Vinge 1993). On the other, autonomous machines portend scenarios in which the meaning, and very survival, of “humanity” is at stake. To be sure, the so-called “fourth industrial revolution” (Elliot 2016) implies risks.2 But if the autonomy of AI is the source of specifically existential threats, then pursuing it is a strikingly paradoxical objective. Particularly striking is the fact that the Cassandra-like voices of “existential risk” include many thinkers, designers, and entrepreneurs leading developments in the field. This fact suggests that AI research is far more compulsively and irrationally driven than its individualistic and transcendental imaginary implies. In the following, that psychological compulsion is juxtaposed to the industry’s idealization of the autonomy of intelligence. Specifically, I argue that the pursuit of the autonomy of AI is compulsively driven by the rationality intrinsic to capitalist institutions, and thus by the systemic demand for profitability. I maintain, therefore, that the AI existential risk discourse obscures how the objective social context of capital accumulation determines  the automation of both manual and mental labor. The discourse thereby reinscribes the terms of social and ecological destruction on which the logic of capital accumulation depends. Only by recognizing this institutional context can the future of the Earth and its inhabitants, as distinct from the future of capitalism, be imagined at all. The first section of the chapter presents the notions of “intelligence” and “autonomy” in the AI existential risk discourse, and shows how two ideas about the modern subject are mobilized in it: the idea of transcendental rationality and the idea of the psychologically motivated individual self. In the second section I show how these ideas comprise an ambiguous and illusory concept of autonomous subjectivity, as illustrated in the film Transcendence and in Bostrom’s idea of “Earth-originating intelligent life.” Since autonomy amounts to an illusory ideal, it is not clear why ostensibly rational AI researchers pursue it. Thus, in the third section, I show how the institutions of capitalist rationality determine the societal drive to automate labor processes, and in the fourth section, I show how this same rationality determines individual subjectivity. Autonomy is shown to be the value of abstract labor, which is mirrored in the subject as its own principle of productivity; this same ideal is the model for the development of autonomous AI. The final sections show how the AI risk narrative obscures the capitalist context by fetishizing AI “superintelligences” as psychological subjects driven by purportedly “non-anthropomorphic”

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

65

motivations (Bostrom 2014, 108). The specter of a rogue AI turns out to be an inverted image of capitalist rationality; and saving the “future of humanity” means saving the future of capital accumulation by continuing the present pattern of human and ecological resource exhaustion.

Rationality and Autonomy in the AI Risk Discourse In their Op-Ed piece for The Independent, Hawking et  al. (2014;  also Russell et al. 2015) inadvertently draw attention to an implicit fact about the social division of labor: human intelligence depends on the work of “diligent hands” (Adorno 1973, 178) and on ecological systems to produce “everything.” They write, The potential benefits [of AI] are huge; everything that civilisation has to offer is a product of human intelligence; we cannot predict what we might achieve when this intelligence is magnified by the tools that AI may provide, but the eradication of war, disease, and poverty would be high on anyone’s list. (Hawking et al. 2014)

Staking a claim in defense of science and technology, however, they obscure that dependence by isolating the concept of intelligence and elevating its value relative to physical activities and material processes (Adorno 1973, 177–178). Elevating the importance of intelligence is a persistent feature of the Enlightenment progress narrative on which Hawking et al. (2014) draw to leverage the further claim that AI is significant because it can “amplify” what the human mind can do.3 Thus, the AI existential risk discourse reiterates the Enlightenment conceptual schema in which the idea of “intelligence” is established as the agent of “everything that civilisation has to offer.” With the privileged position of human intelligence presupposed, in AI risk discourse, “intelligence” is further defined as “rationality” (Russell et  al. 2015) and “instrumental rationality” (Bostrom 2012, 2014). As Russell et  al. (2015) explain: “the criterion of intelligence is related to statistical and economical notions of rationality – colloquially, the ability to make good decisions, plans, or inferences” (2015, 105).4 Defined in these narrow, common-sense terms, the implication is that this “ability” is an ordinary human mental activity, one that is already being technologically enhanced and/or automated through AI and robotics applications. Moreover, it encompasses the performance of a range of socially p ­ roductive

66 

K. ASP

activities: everyday unpaid and paid physical work like house cleaning, truck driving and hamburger flipping; activities involving “basic cognitive skills” like data collection and processing; and highly specialized intellectual work, such as the tasks performed by “human computers” that the film Hidden Figures recently depicted (Becker 2017; McKinsey 2017b). Russell et al.’s (2015) reference to rationality as an “ability” underscores their framing of intelligence in conventional economic terms, that is, as an isolatable and alienable property, a labor commodity amenable to automation. As they indicate, however, AI’s potential to become more like homo oeconomicus depends on the degree to which a computer’s “mental” ability to calculate and plan is enhanced through the acquisition of the properties of agency and autonomy. Russell et al. (2015) describe how, over the past 20 years, researchers in the field of AI and robotics have pursued the “construction of intelligent agents.” These are defined as “systems that perceive and act in some environment” (2015, 105). The notion of agency refers to capabilities ranging from visual and auditory sensors to moveable limbs and motorized wheels that enable an “intelligent agent” to respond to sensory inputs, communicate, and do things. For example, the effective use of domestic vacuum cleaners typically depends on the “rationality,” locomotion, and sensory capabilities of human operators. In contrast, as Russell et al. (2015) show, robotic vacuums are programmed with instructions for effective performance; they are automotive; and they are constructed with sensors to allow movement around things in a room. Nonetheless, their behavior is predetermined through programming; in this sense, they are “intelligent agents” that perform “rationality” scripts, but they are not autonomous. In their canonical engineering textbook on AI, Russell and Norvig (1995) illuminate what they mean by “autonomy” in the context of designing AI systems. The authors distinguish between “actions” determined by “built-in knowledge” (e.g., instinct, preprogramming), and those determined by knowledge developed through perception of objects in the environment (i.e., “experience”). The latter is correlated with autonomy: “[a] system is autonomous to the extent that its behaviour is determined by its own experience” (1995, 35). A simple “clock” has no need for “experience” because its “built-in” knowledge is sufficient for the task of keeping the time. Even when a clock, like most “smart” phones today, is designed to detect changes in time zones, if it is built to update automatically, then its capacity for “experience” remains limited. For, as the authors observe, “the intelligence” that prescribes when to change the

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

67

time “seems to belong to the clock’s designer rather than to the clock itself” (1995, 35). This ambiguous statement implies that “experience” is determined by more than sensory perception alone; it requires a cognitive capacity, “intelligence.” Moreover, that capacity is determined by whether or not it “belongs” to the machine, such that the machine can make its “own” determinations. For the clock to be “autonomous” in this sense, therefore, it should be able to make its “own” judgment about adjusting its time-display in response to perceptible differences in its spatiotemporal context, rather than depend on the designer for such a decision (i.e., via automation). As Russell et al. (2015) report, research on the autonomy of rationality has advanced to the point where “fully” autonomous AI systems are an emerging technology, giving rise to the need to “control” the risks associated with developing them. These supercomputers will be able to “modify, extend, and improve themselves, possibly many times in succession” (2015, 109), such that they could indeed overwrite the “built-in knowledge” (Russell and Norvig 1995) put there by their designers. Here, machine “rationality” is directly aligned with autonomy in the conventional sense of self-rule—being able to rationally determine one’s “own” purposes and rules of action rather than relying on external authorities. By extension, the AI-system would be “free from manipulative and ‘external’ forces” (Christman and Anderson 2005, 3) controlling the “rationality” scripts determining the behavior of robotic vacuums and digital clocks. Russell et  al. (2015, 110) argue that to deal with the risks posed by these emergent autonomous AI systems, a shift from standard engineering frameworks to mainstream economic theory is required. Their description of machine “intelligence” in “statistical and economic” terms as “rationality” legitimates this shift, even though “smart” machines like robotic vacuum cleaners lack the requisite autonomy. The application of economic theory permits treating fully autonomous AI-system objects in behavioral terms as utility maximizing or satisficing “rational agents.” These assumptions “allow use of formal tools to anticipate and control [their] behaviours” (Russell et al. 2015, 110).5 In other words, AI systems are to be modeled as if their observable behavior was determined by subjective motivations and preferences per models of “human nature” as homo oeconomicus. Bracketing the implied anthropomorphism for now (see the “Bostrom” section below), Russell et al.’s (2015) shift in frames of reference reveals two ideas at work in the AI risk discourse. The first is the idea of autonomous, or transcendental, rational intelligence as the essence of

68 

K. ASP

subjectivity; the second is the idea that psychological “motivations” ­constitute “rational agents” as economic subjects. As I show in the next section, the entanglement of these ideas is a pervasive feature of the form of modern individual subjectivity.

AI and the Illusion of Transcendent Rationality The 2014 film Transcendence, referred to by Hawking et  al. (2014), brought renewed attention to the speculative technoscientific discourse on autonomous AI first articulated by Good (1966) and popularized by Kurzweil (2005). The film offers some insight into the paradoxical form of individual subjectivity, and the relationship between rationality and autonomy, in the AI risk discourse. Transcendence tells the story of how the mind of a dying computer scientist (Will Carter, played by Johnny Depp) is uploaded to a “supercomputer” so that he can remain with his beloved wife Evelyn and continue their scientific work. The experiment produces a conscious personality (Will) who appears in spectral form, largely on computer screens. While Will as AI-system has every power and ability scientists could hope for in their pursuit of technological progress, he is viewed as a rogue agent who threatens the social order. The conflict between the self-directed AI-system and the state institutions that seek to control its behavior is ultimately resolved by killing the personality inhabiting the machine. The idea of autonomy is implied in the supposition that Will’s mind can literally transcend his body’s sensory organs and biological needs, which are enmeshed in a determinate—concrete and particular—context. His mind is thus a self-determining, substantial thing in its own right, an objectively real subject and agent. Moreover, it has the qualities of abstraction and universality that Will’s body, with its biological and spatiotemporal specificity, lacks. However, the question arises as to whether the totality of Will’s mind could, logically, transcend his body? For, not only were Will’s cognitive capabilities transferred to the supercomputer, like the “rationality” that should be transferred to the clock as its “own” power in Russell and Norvig’s (1995) discussion of autonomy. So too were his memories, personality traits, and emotional attachments. Indeed, it is Will’s enduring desire for Evelyn, the motivations of a mortal man, that drive the plot. How can thoughts, feelings, and desires acquired through this body’s experience be anything other than immanent to it? These questions point to a specific ambiguity in the mind-body conceptual complex

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

69

in the film’s narrative. Will’s mind behaves like a transcendent and ­universal rational subject, even while it is also being compelled by the “oh so human” subjective desires and experiences attributable to his physical life; that is, Will is also a psychologically motivated self. In this sense, the ambiguity lies not in the relation between mind and body, but in the idea of “mind” itself, the mind as the autonomous subject of knowledge and action. This ambiguous idea of the subject is implicit in the transhumanist ideology that informs not only Transcendence, but AI risk discourse as well. Specifically, Nick Bostrom, transhumanist, philosopher and AI existential risk expert, mobilizes the notion of “Earth-originating intelligent life” to underwrite the claim that the “future of humanity” is at risk from rogue AI systems (Bostrom 2013, 15; 2014, 115). Bostrom’s concept of “intelligent life” identifies the collective subject, humanity, and its instantiation in individual bodily forms. He maintains that, from among the kinds of “intelligent life” possible in the universe, this distinctive “species” emerged specifically on Earth and specifically in the human biological form. Nonetheless, what is “human” about the “species,” in Bostrom’s account, is not dependent on these determinate bio- and geophysical contexts. As in the film Transcendence, technological development might change the kinds of bodies humans inhabit; moreover, for Bostrom, the location of “Earth-originating intelligent life” might change too, as the species colonizes other planets (2013, 19–20). Nonetheless, what is constant are the attributes of “mind” that define “humanity” at the individual and collective levels of existence. These attributes include “rationality,” as the necessary condition of “humanity,” in combination with the idea of subjective value preferences that is captured in the phrase, “what we have reason to value” (2013, 20). Thus, all “rational” subjects are members of the collective subject, “humanity,” except those which do not share the same subjective value preferences as “we humans” do. Like Will as AI system, Bostrom’s “subject” is split between the abstract and universal form (rationality) and a concrete and particular motivational content that is not reducible to that rationality or to the bodily context either. The paradox at issue here is captured in what Theodor Adorno (1998, 248) referred to as the notion of the “subject” pervasive in capitalist society, the “transcendental subject.” This term originates in the heritage of German philosophical idealism, but Adorno uses it to characterize common sense assumptions about the interrelationship between human intelligence, knowledge, agency, and autonomy in capitalist society. The idea of

70 

K. ASP

the “transcendental subject” is based on the supposition that the abstract, formal rules and universal categories of rational intelligence transcend the domain of sensual experience and individual consciousness, providing a “stable epistemological” framework (Schecter 2010, 23) that mediates individual experience of external reality. Thus, the “transcendental subject” refers to the cognitive “mind” as the epistemological ground, or constitutive agent, of objective knowledge (Adorno 1998, 247). In more expansive theories, it “engenders the world itself ” (ibid.), or supplies the internal (subjective) ground for unconditioned possibility, or “free will” (Hegel 1967, 20–21). Just like the Enlightenment idea of “reason,” Bostrom’s idea of “Earth-­ originating intelligent life” is the universal condition shared by individual humans; it is what defines them as “transcendental subjects.” Moreover, on Adorno’s account, in both cases a paradoxically structured agent is in play. It is an abstract self-ruling subject that is the “absolutely dynamic principle” and cause of determinate knowledge and action. And it is a concrete, compulsively driven psychological self (Adorno 1973, 176–177; 1998, 255). The problem is thus that the allegedly autonomous subject is implicated in an impossible relation of dependency with its psychological self, a relation that exposes the illusory character of its autonomy. As Adorno characterizes the problem: The aporetical concept of the transcendental subject—a nonentity which is nonetheless to act, a universal which is nonetheless to have particular experiences—would be a soap bubble, never obtainable from the autarkic immanent context of consciousness, which is necessarily individual. (1973, 177)

The transhumanist and “intelligent life” concepts are thus vulnerable to the same criticism leveled against German idealism. The substance of the “transcendental subject” is abstract and universal, and in these ways, it is supposed to be external to, but not separate from, the psychological self. How can this self-ruling agent that is situated outside time and space, this “nonentity,” act according to the laws it gives itself? How can it learn from experience when, due to its purely logical, universal structure, by definition as it were, it is unable to access the concrete and particular contents of experience? Indeed, how can it “have” an experience at all? Inversely, how can the psychological self, determined by inner impulses and concrete forces, conform to the abstract, universal rules and procedures of the transcendental?6

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

71

The exposure of this contradiction puts into question the autonomy of the “transcendental subject.” For the free rational subject will be heteronomously determined by its compulsive, need-driven psychological self, while that “self” will be deluded by its attachment to a merely metaphysical idea of autonomous rationality, a “soap bubble,” that has no bearing on its actual existence (Adorno 1973, 177). Yet the demystification of autonomous rational intelligence does not do away with the rational subject as much as reveal its conditionality, its entwinement with the conceptual construction of the psychological individual. That entwinement is evident in Bostrom’s transcendental idea of “humanity,” even though he prioritizes the abstract, universal quality of intelligence. And as Transcendence showed, the risk of a rogue AI superintelligence is construed on the basis of both the autonomy of transcendent rational intelligence and the heteronomy of immanent, psychologically motivated individual existence. Some sense can be made of this paradox if it is recalled that the idea of autonomy is a paramount “political and moral value” in modern liberal democracies (Christman and Anderson 2005). This normative ideal of autonomy, I suggest, which is both an illusory and a forceful value, is presupposed in the existential risk discourse. Yet, the ideal of the free rational subject is inextricably entwined with the idea that the subject is shaped by irrational, psychological characteristics. Nonetheless, the ideal motivates the pursuit of fully autonomous AI.

Capitalist Context of Rationality Given this depiction, and contra the popular stories about heroic scientists fighting over the “future” of “technological civilization,” the question of what compels the pursuit of autonomous AI is not readily explained in terms of the personalities and ideas of individuals involved in technoscientific research and practice. Indeed, the critique of the autonomy of the “transcendental subject” conveys this limitation, for the individual “transcendental subject” (aka Will Carter) is not the unconditioned ground of objective knowledge, nor is it unconditionally self-determining in the sense of giving the law to itself. The explanation is rather that the grounds of scientific knowledge and action lie elsewhere, specifically in the determinate context of society (Adorno 1973, 139; 1998, 251). As the critical theorist Theodor Adorno wrote, “society comes before the individual consciousness and before all its experience” (Adorno 1973, 181). What is it about “society,” one might thus ask, that drives researchers toward

72 

K. ASP

automating “intelligence” on the model of the paradoxically structured transcendental subject? Notions like “industrial society” and “technological society” are commonly used to characterize the determinate context of “society” in which automation, and crises associated with progress, emerge as topics of concern. These ideas of “society,” as Adorno observed (2003), correctly grasp the significance of how human labor, knowledge, machines, and natural resources are specifically organized in modern society (i.e., as “forces of production”). However, the contextual factors determining the knowledge and agency of subjects are distorted by an exclusive focus on industry, technology, and progress. Indeed, while there is much about modern society that has changed since the invention of the steam engine, there is also a constant factor: capitalism. To paraphrase Adorno’s argument, the crises and “risks” attributed to the “fourth industrial revolution” resemble those of the “first industrial revolution” because “production takes place today as then for the sake of profit” (2003, 117). On Adorno’s account, the “forces of production” are structurally “interlocked” with the social institutions and processes governing capitalist commodity exchange. A collectively based “principle” of exchange is the objective “social” ground, or “condition of possibility” (in Kantian transcendental terms), that has the “upper hand” in mediating individual needs and wants, including the subjective profit motive (Adorno 2003, 121, 124; 1998, 253). The principle of capitalist exchange is “the objective abstraction to which the social process of life is subject. The power of this abstraction over human beings is more palpable than the power of any other single institution” (Adorno 2003, 120). The challenge of understanding how the capitalist exchange mechanism has a determinate effect on individual subjects, and technological development more broadly, concerns its abstract character (Adorno 2003, 120; Backhaus 1992, 59, 86–7; Marx 1976, 165). Exchange determines what appears to individuals in empirically objective forms, notably as “commodities” and “money,” but the process is not apparent to them. Rather, what we see and know with certainty are concrete objective things and intersubjective communications, mediated—especially in the present era—by “technologies.” Thus, it is not surprising that less elusive dimensions of society—industry, technology, and individual preferences—might be construed as the catalysts of societal changes, like those driving the pursuit of autonomous AI on the model of the “transcendental subject.”

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

73

Karl Marx made sense of the relation between this abstract exchange process and concrete subjects through his theory of the “fetishism of the commodity” (1976, 163). On Marx’s account, capitalist society is a collective system of social production organized specifically to produce commodities for the purpose of accumulating capital. “Peculiar” to this system is that labor is organized on the model of private, independent production, which is socially mediated by the exchange principle (165). Different kinds of labor activities are compared and exchanged through their reduction to quantities of abstract labor time (166–67). “Abstract human labour” is therefore the “substance” of value comprising the commodities that are exchanged, with the standard of value being based on the “socially necessary labour time” needed to reproduce the labor force (168–169). Money is essential to this process as the stable, objectively valid, form through which the exchanges of equivalents are transacted; it functions as the “general equivalent” in relation to which the relative value of commodities is determined (169). Thus, capitalism is a system of collective commodity production that is mediated by the exchange of “unequal” quantities of abstract labor time through processes of abstraction and with reference to an objectively valid standard (Adorno 1973, 146–147). Given this “rationality” of exchange, as Adorno called it (ibid.), fetishism involves the attribution of “social characteristics” to the products of labor themselves, such that the abstracted value appears concretely as a “socio-natural” property of the object (Marx 1976, 165). The daily discourse of exchange necessarily deals with abstract labor in these concretized fetish forms of “commodities” and “money.” Yet market participants are unaware of what they are doing when they exchange these objects (167). What appears to them objectively in the realm of exchange are already personified, autonomous things. Significantly, these fetishes are not simply the externalized and animated “cognitive” projections of individual subjects (Ellen 1988; Backhaus 1992, 59–61). Rather, they are collectively constituted through the institutionalized rationality of exchange, and thus they are socially “valid” or objectively real (Marx 1976, 166–67). Thus, the concrete forms and social practices of daily life are mediated through a social logic of abstraction, universalization, and unequal exchange. It is in these fetish forms that the abstract (“supra-sensuous”) and universal process of exchange appears to the senses of human individuals. Moreover, capitalism is characterized by the drive to expand the value of abstract value, a process which, according to Marx, appears to everyday

74 

K. ASP

consciousness and in economic theory as the “magic” of money making more money—that is, profits and interest—through its circulation and exchange (Marx 1976, 255). To market participants, this “magic” seems to be the effect of a “self-engendering” agent, the goose that lays “golden eggs.” In other words, the process of capital accumulation is itself a collectively constituted, objectively real, fetish. Marx characterized the fetishized appearance of this process in terms of an “automatic subject” (ibid.). Capital accumulation seems to be an autonomous “supraindividual” subject that gives to itself the principle of exchange, the “law,” that determines the value of “things” (Backhaus 1992, 57). Yet that “supraindividual” subject is nothing more than the collective activity of independent producers. The term “automatic” refers to the unintentional, unconscious character of their collective activity; they obey the “law” that they give to themselves, even while they remain its unwitting enforcers. Hence, capital accumulation is collectively experienced as an objective, universal and dominating agent, like a “regulative law of nature” akin to the law of gravity (Marx 1976, 168). Yet the “magic” of money making more money depends on commodity production; specifically, capital accumulation relies on the creation of “surplus value” (profit) through the exploitation of waged labor (Harvey 2006, 22–23). It is primarily by increasing the “productivity” of labor (that is, more widgets per hour) that the cost of labor time is reduced, which leaves more of the value of the product as potential profit. The replacement of human labor with machines, and techniques such as speeding up the production process, are major factors in this equation (2006, 31–32, 107–8). Thus, too, accumulation depends on the application of science and technology in the pursuit of productivity; technoscientific research is inextricably “intertwined” with capitalist production (Horkheimer and Adorno 2002, xviii; Adorno 1973, 139; 1998, 251). The drive to automate is, therefore, neither random nor simply an extension of the idealistic pursuit of scientific progress for its own sake, for it serves an essential function in the process of accumulation (Harvey 2006, 119–121). As David Harvey writes, “[t]echnological change exists, therefore, as the prime lever for furthering the accumulation of capital through perpetual increases in the value productivity of labour power” (2006, 133). The competitive impetus to increase productivity leads to systemic crises, involving the devaluation of capital assets, the destruction of commodities, and unemployment (2006, 84–5, 190–93). Such crises are intrinsic to the system, and by the same token, inherently irrational. The

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

75

depth of that irrationality is expressed in the sense that, insofar as capitalist profitability depends on “living” labour, the drive toward full automation threatens the very principle of accumulation. The idea that full automation poses an existential threat to the capitalist system is contentious; yet its persistence is reflected in various recent works, some of which thematize AI and robotics (see Balakrishnan 2009; Heinrich 2013; Mason 2015; Noyes 2014; Srnicek and Williams 2015). In short, for the sake of accumulation, “labor” and “capital” collectively and unconsciously constitute a “supraindividual” subject that exploits individual lives and irrationally exposes the totality to threats of its own making (Adorno 1973, 304, 306–307; Backhaus 1992, 57).

Capitalist Subjects, Transcendental Rationality As Adorno shows, individuals are constituted as subjects of this “supraindividual” process of abstract, unequal exchange: “If the standard structure of society is the exchange form, its rationality constitutes people: what they are for themselves […] is secondary” (1998, 248). The form of this individual subjectivity was revealed in German idealism in the concept of the transcendental subject: In a certain sense, although idealism would be the last to admit it, the transcendental subject is more real, that is, it far more determines the real conduct of people and society than do those psychological individuals from whom the transcendental subject was abstracted and who have little to say in the world; for their part they have turned into appendages of the social machinery, ultimately into ideology. The living individual person, such as he is constrained to act and for which he was even internally molded, is as homo oeconomicus incarnate closer to the transcendental subject than the living individual he must immediately take himself to be. (1998, 248)

What is “more real” about the “transcendental subject” is the collective, unconscious rational activity of abstraction and unequal exchange in accordance with the law of value, which appears fetishistically as the autonomous “automatic subject.” The “living individual person” is a thing that is enlivened, or animated, by the “more real” collective, rational subject, homo oeconomicus. Thus, the living individual is socially constituted as an object with the subjective characteristics of “economic man”—that is, as a concrete individual entity who performs a socially productive function

76 

K. ASP

that is determined by the objective context of capital accumulation. In this sense, individual subjects are formed to function as “character masks” and “personifications” of capital; they carry out the objective requirements of accumulation, but they are not individually responsible for its effects (Adorno 1973, 304, 311; Marx 1976, 179). The idea of the “transcendental subject” thus reveals that the essence of all “character masks” is productivity: “hidden in this principle, in the general and necessary activity of the mind, lies work of an inalienably social nature” (Adorno 1973, 177; my emphasis). For Adorno, the sense that there is something “hidden” refers to the tendency in philosophical idealism to isolate and elevate the “work” of “the mind” because it is the kind of labor that the dominant class tends to contribute to the social production process. The ideological occlusion of the value of work by “diligent hands” is thereby prevalent (1973, 178). Hence, Hawking et al. (2014) equate the “products” of civilization with the work of “human intelligence” and Bostrom maintains that the “superintelligence’s power resides in its brain, not its hands” (2014, 99). What lies “hidden” is thus all of the “physical” labor that contributes to the productivity of society, and as well, the contributions of “nature,” which are erased altogether (Adorno 1973, 178). More than this though, Adorno’s point is that the philosophical idea of the “transcendental subject” codifies the “predominance” (ibid.) of the productivity of labor for the sake of capital accumulation. Various kinds of labor are significant only in terms of their functional contribution to that end, their utility. The meaning of “work” is thereby reduced to “pure activity,” activity without concrete determinations (ibid.). Furthermore, what is “inalienably social” about this “pure activity” is that it represents the abstract general labor time composing the real “substance” of value. The “transcendental subject,” as self-determining “pure activity” on an individual level, thus reprises the illusionary autonomy of capitalist exchange. The objective reality of abstract labor is “projected onto the pure subject as its origin” (Adorno 1973, 178), such that living individuals are constituted as “free” subjects who believe that they give themselves the “law” (262). They believe this even though the “law” is given to them  socially, “projected” “over their heads and through their heads” (304) as a norm constituting the rational ground of subjective agency on capitalist terms. This identification process ideologically anchors “living” individuals to the collective system of capitalist production. Bostrom (2008, 29) exemplifies a transhumanist version of just this productivist identity when he writes of “autopotent” posthumans who are “able to

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

77

remold themselves at will and assume any internal state they choose,” and who reside “in a ‘plastic world’ because they can easily remold their environment exactly as they see fit.” Contra this fantasy, however, as Marx and Adorno show through analysis of the fetishism and ideological operations associated with the collective “automatic subject” and the individual “transcendental subject,” rather than being autonomous, these individuals are in fact ruled by their own expropriated labor time. Thus, while science, technology, and progress are significant factors in the crisis of automation, as Adorno wrote (2003, 118), it is important not to “blame” technology for the correctly apprehended sense of catastrophe.

Bostrom’s Occult Motivations of AI Machines Notwithstanding Adorno’s advice, the attribution of blame to machines is one of the features of the AI risk discourse, particularly in Bostrom’s influential work (2012, 2013, 2014). I maintain that, in framing autonomous AI systems in the economic terms of “utility maximizing” “rational agents,” Bostrom isolates the egoic moment of the capitalist subject described above (2012, 8; 2014, 10–11, 185–86). In so doing, he obliquely  depicts the fetish-characteristics of the collective  “automatic subject,” mobilizing the fear and blame that circulate with the automation-­ related negative effects of this unseen social cause. The notion of utility maximization refers to the utilitarian premise that a rational agent, homo oeconomicus, is a “bundle of appetites” that appropriates things not to appreciate them for what they are but as means to satisfy infinite desires (MacPherson 1973, 4). Appetites originate in the pleasure/ pain indices of the body and express the physical and psychological needs and experiences of the individual. It is such limitless bodily needs and experiences that “motivate” a person to utilize things and other persons as means of satisfaction. This conception of rational agency correlates with what democratic theorist C.B.  MacPherson refers to as a “descriptive” concept of power: “A man’s powers, in this view, were not his essence but merely instrumental: they were, in Hobbes’s classic phrase, ‘his present means to obtain some future apparent good.’ Powers were a way of getting utilities” (1973, 6). Bostrom (2012, 4; 2014, 108) describes “intelligence” in similarly nonnormative, “descriptive” terms as “instrumental rationality” and “instrumental reasoning” to demonstrate how superintelligent machines will amass powers in a relentless struggle that risks the “future of humanity.”

78 

K. ASP

Bostrom argues that instrumental rationality is the defining characteristic of a class of rational agents that includes both biological organisms, like humans, and artificial entities. A superrational entity in an artificial (nonbiological) body, he nonetheless maintains, will have motivations that seem weird or unintelligible to humans (i.e., contra Will as AI-system, whose undiminished love for Evelyn was the romantic hook in Transcendence). He argues that, despite the human tendency to “anthropomorphize,” there is no reason to assume that an AI would “care intrinsically about any of those things” that concern “humans.” He gives the example of an AI “whose sole fundamental goal is to count the grains of sand on Boracay […] or to maximize the total number of paperclips in its future light cone” (2012, 2; 2014, 107). Moreover, he supposes that different “species” of rational agents might occupy the same space, invoking a Hobbesian-like scenario of existential threat in which some “automata” have powers of rationality that far exceed the norm, and motivations that make them dangerously unpredictable to others. However, for there to be such a problem, I suggest that Bostrom would need to dispense with the assumption that the inhabitants of such a space were all “utility maximizing rational agents.” For that assumption implies that superintelligent AI’s are conditioned by the same motivational structure that conditions we “humans,” namely homo oeconomicus.7 And since the psychology “we have now” is mediated by the capitalist social context, then so too will be the psychology of nonhuman participants of “our” society. In other words, if superintelligent machines are utility-maximizing agents, their motivations are no less intelligible to us than those of Wall Street brokers. The objective principle of exchange establishes the subjective conditions in which the psychology of existential threat—that is, the Hobbesian experience of individual life being “solitary, poor, nasty, brutish, and short”—arises. Bostrom, I suggest, obscures the objective context that structures the relation between “present means” and “future apparent goods” in the Hobbesian scenario; he attributes what one might call an occult genesis to the “motivations” of machines, such that the machine’s motivations are construed as esoteric or hidden. The theory of a hidden compulsion frames Bostrom’s argument for the need to develop predictive tools and scenarios to search for signs of danger on the technology frontier. This, he maintains, can be done based on modeling the “instrumental reasons” that any rational agent would pursue en route to its final aim (2012, 6; 2014, 108). The argument takes the

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

79

form of deducing “intermediary” aims common to all agents, and then using “species” classification (biological, mechanical) to infer how AI systems, in particular, might behave. Nonetheless, the support provided for this approach reveals that “intermediary” aims amount to the “final” aims of Hobbesian automata. For example, Bostrom suggests that the first fully autonomous superintelligence might become a “singleton,” with “no significant intelligent rivals” (2012, 11; also 2014, 95–104, 109–114). It might try to “acquire an unlimited amount of physical resources and, if possible, to eliminate potential threats to itself and its goal system” (2012, 14). The image of the “superintelligent singleton” that so dominates the field of utility maximizers that it “has no rivals” recalls the Hobbesian end game in the “war of all against all” in which only one person remains alive and unsubordinated. Moreover, Bostrom argues that a superintelligent agent will likely pursue “unlimited resource acquisition” (13). Referring to “resources” in terms of utility functions, and reifying abstractions like “time” and “space,” he avers that “[w]ith mature technology, basic resources such as time, space, and matter and other forms of free energy, could be processed to serve almost any goal” (12). Even so, he argues, all the “basic resources” on Earth will not be enough to satisfy the AI hegemon. It would be compelled to colonize outer space and “harvest” the supposedly infinite supplies of resources out there too, since its colonizing activities on Earth would “easily consume far more than one planet’s worth of resources” (12). As the ultimate goal of an AI superintelligence is supposedly unintelligible “to us,” the account of this intergalactic colonial fantasy is justified only by reference to the intermediary necessity of accumulation. One can see this more clearly, now, in Hobbesian terms as the “acquisition” of knowledge, technology, and resources as “present means” to the future command of world markets and political institutions. Bostrom (2012, 14; 2014, 116) maintains that his argument shows why there is reason to believe that superintelligent machines might “materially infringe on human interests,” even though “humans” cannot know what motivates them. Bostrom accounts for the existential threat to “human interests” on the basis of the machine’s “instrumental reasons” rather than on its “non-anthropomorphic” motivations. Specifically, since he does not know what those motivations might be, he resorts to speculating on the superintelligence’s instrumental behavior; the supposed threat to “human interests” stems from those means themselves rather than from the occult ends. Yet, the “instrumental reasons” are the same for humans

80 

K. ASP

and machines. Therefore, whatever the alien motivations, the source of the threat is not hidden after all. These points obviate the need to maneuver around the problem of “utterly non-anthropomorphic goals.” With that problem out of the way, Bostrom’s quandary of anthropomorphic projection comes into sharper focus. For, prima facie, to correlate the behavior of machines with Hobbesian-­ style descriptions of utility-maximizing agents is to animate “things” with the characteristics that defined “human” personality in the utilitarian model from the start. In this anthropological sense of fetishism, Bostrom is indeed guilty of assigning to a thing the social value implicit in the meaning of homo oeconomicus (see Ellen 1988). But the quandary of projection is more complicated than it appears here. For, as I showed in the previous section, “living” individuals in capitalist society function as incarnations of homo oeconomicus. “Economic man” names the “supraindividual” objective essence that is projected onto concrete individuals and internalized as their own abstract-value producing identity  and social function (Backhaus 1992, 57). Thus, Bostrom’s analysis presupposes homo oeconomicus as the collectively imposed character of individual subjectivity. It is the imposition of this character on individuals, the heteronomy of, and domination by, the exchange principle, that is revealed in Bostrom’s anthropomorphic gesture. By attributing the imposition of homo oeconomicus to the rogue AI supercomputer rather than to the social relations of exchange, what is human about homo oeconomicus is disavowed and rhetorically configured in inhuman terms. Human threats to “human interests” are thereby personified as dangers posed by the alien other. Hence, Bostrom’s rogue AI supercomputer amounts to an overtly negative, inverted image of capital accumulation, a representation of the “automatic subject” as driven by motivations that are not simply inhuman but antihuman. This “supraindividual” subject appears in the tale of the first fully autonomous AI aiming to become master of the universe, a hegemon whose rule will seem like an inescapable “natural law,” like the law of gravity, to those subject to it.8 And yet, a further clarification is needed of this symptomatic reading of Bostrom’s rogue superintelligence, because Bostrom does not discuss capitalism. The arc of his existential risk narrative concerns the threat of a machine seizing control of the means of production. Capitalist motivations are thereby occulted; they are hidden from view by virtue of the occlusion of the relations of production and the principle of exchange. The machine is driven to produce surplus value through the exploitation of labor power and colonial expansion, but it is

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

81

not clear why. The answer resides in the “realm” of exchange where abstract value is realized, but that realm is not explicated in Bostrom’s text. Instead, he repeats the distortions characteristic of the “industrial society” thesis that Adorno (2003) warned against. Technology, Adorno advised, is an element of the social production process rather than its defining principle. Bostrom not only overdetermines the role of technology, he compounds that distortion by deflecting responsibility for the automation crisis from the social collective to the computing machines that are fabricated by it.

Saving Transcendent Intelligence, Abandoning Earth The late Stephen Hawking articulated a sense of collective responsibility for the impacts of “technological society,” not only with reference to rogue AI systems, but also when he claimed later on, We have given our planet the disastrous gift of climate change … When we have reached similar crises there has usually been somewhere else to colonize … But there is no new world, no utopia around the corner…. We are running out of space, and the only places to go to are other worlds. (Hawking cited in Barclay 2017)

Yet, rather than ponder how the effects of the “disastrous gift” to our planet might be treated, he implied that, since “we” are a colonizing species with a history of crisis-inducing resource exhaustion, it is time to transcend the Earth in search of another planet. While protecting people from anthropogenic threats is indisputably a collective goal, this astonishingly fatalistic response to the global ecological crisis suggests that, in the final analysis, the AI risk discourse offers no such protection. Why? The answer, I suggest, is that  the value of “humanity” refers to the value of capital accumulation rather than to people. This becomes clear when we consider the lengths to which Bostrom goes to show why the “future of humanity” matters. He presents “cosmological estimates” showing the “total” future value of humanity to be somewhere between 1052 to 1054 total possible lives to be lived (2013, 19).9 Notably, these are not concrete lives, but lifespans of human activity serving as indicators of the total abstract value of humanity. The weighty numbers express the supposed enormity of the risk posed by a wrong turn

82 

K. ASP

on the road to the future, yet this total future value is not wealth that the collective “we” has now in its peoples, built habitats, and ecosystems. Rather, it is an estimate of the potential value of the joint stock company “Humanity” spread across the quasi-infinite expanse of space and time. We do not have it yet. To close that gap, Bostrom makes a teleological claim about progress toward “technological maturity.” The same assumptions about “instrumental values” that were applied to AI superintelligences also apply to the “perfection” of human technological civilization. It is simply “us” instead of “them” amassing powers in the forms of resources, technology, and knowledge. Discerning the difference between “us” and “them” thus lies in parsing the meaning of “humanity,” which Bostrom defined as “Earth-originating intelligent life.” Humanity’s animating spirit is not just “intelligence,” but “rationality” motivated by “what we have reason to value” (Bostrom 2013). The meaning of the qualifier “what we have reason to value” thus seems to make all the difference, yet it is elusive. At first glance, having the freedom to choose one’s form of embodiment—biological or machinic— seems like one such value for Bostrom. But he does not claim that such a freedom is an end in itself. Rather, the problematic of value is split between our present bodies and the shapes “we” might choose later on. By deferring “what” we are now—a deferral encoded in the claim to save “the future” of humanity—it seems like the collective “we” transcends the finitude of bodies, individual and planetary. But the very notion of existential risk reveals the paradox in the idea of the transcendental subject, for how can transcendent “intelligent life” be exposed to the possibility of its own finitude, its own extinction? The possibility that abstract “humanity” is determined by the same heteronomous forces that rule living bodies and Earth systems is an unhappy contradiction. It brings into view the merely instrumental value of “intelligence” itself in the narrative, the presupposition that it too is a contingent “ability” enstructured through the relation between use and exchange. Intelligence, like bodies, is not a good in itself in the existential risk theorem, but a “way of getting utilities,” and it is existentially vulnerable on the same account. This ultimate truth is confirmed when Bostrom nullifies the “future of humanity” on the supposition that even if it were possible for everyone to agree on what the best future might be, “we” might choose badly nonetheless. Thus, he argues, it is better not to choose at all! Rather, we should prioritize the instrumental means of value cre-

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

83

ation, keeping the “options open” for “a future version of humanity with great powers and propensity to pursue them wisely” (Bostrom 2013, 24). Hence, saving “humanity” means saving the future of value creation for its own sake, which is nothing other than the structural demand of capital accumulation operating “behind our backs.” Yet capitalism’s future relies on increasing productivity through the integration and automation of social labor and the relentless abstraction of concrete physical, social, and cultural abilities. Stephen Hawking’s pronouncement that “we” will need to abandon the Earth because it is becoming uninhabitable articulates the implications of this situation (Barclay 2017).10 So, while Bostrom claims that the “future of humanity” is “full” of value creation, my argument suggests that the future presupposed by Bostrom (2013, 2014), Hawking et al. (2014) and Russell et al. (2015) depends on continuing the present pattern of exhausting human and planetary “resources.” In the end, for there is an end in their narrative, a few will have accumulated enough “value” to purchase seats on a space ship to Mars, leaving the catastrophe to the rogue superintelligences and displaced climate migrants. This is the “existential risk” left to those who cannot afford to “transcend,” but it is not posed by out-of-control AI-systems. Rather, it is imposed as an unrecognized, fetishized demand of the system of capital accumulation, which we collectively inscribe, obey and enforce. In summary, by situating the pursuit of the “autonomy” of AI systems in the objective social context of capital accumulation, four points arise that illuminate the distortions compounded in the AI risk discourse. First, the automation of skills and labor in the so-called “fourth industrial revolution” is a function of the abstract “law” according to which capital must expand regardless of the human or ecological costs. Capitalism operates in this sense as an “automatic subject,” behind our backs, through our heads, and despite ostensibly “human interests.” Second, Russell et al.’s (2015) conflation of “intelligence” with economic “rationality” shows how the industry’s unconscious identification with the subjective character mask, homo oeconomicus, enables the literal reproduction of that character in AI systems as agents of capital accumulation. Third, the drive to construct fully autonomous AI systems arises from identification with the normative value of autonomy expressed in the idea of the “transcendental subject,” specifically the normativity of capitalist productivity as it is constituted in the individual subject as its own ideal. And fourth, the objective, abstract exchange principle “rules over everything” (Marx, cited in Dyer-Witheford 2015, 275). Thus, while science, technology, and progress are important

84 

K. ASP

elements in the crisis of automation, they are not to “blame” for the correctly apprehended, if distorted, sense of catastrophe. Since the pursuit of fully automated and autonomous systems is inextricably entwined with the societal context of capital accumulation, altering the trajectory of development utterly depends on recognizing the extent and degree to which capitalism determines the ideas and choices made in research labs and institutes. Only by recognizing this institutional context can the future of the Earth and its inhabitants, as distinct from the future of capitalism, be imagined at all.

Notes 1. The 2014 Op-Ed piece names the Cambridge Centre for the Study of Existential Risk, the Future of Humanity Institute, the Machine Intelligence Research Institute and the Future of Life Institute as among the nonprofit institutes involved in the movement. 2. McKinsey (2017a) forecasts that 51% of “activities” in the US economy are susceptible to automation. See also Bland (2016), Bremner (2015), Ford (2015), Brynjolfsson and McAfee (2014), Standage (2016), WEF (2018). 3. Hawking et al. (2014) unscientifically repeat the central myth of the scientific Enlightenment. See Horkheimer and Adorno’s “The Concept of Enlightenment” for an immanent critique of the contradictions in the Enlightenment discourse. Upholding the value of enlightenment as critique, they show how knowledge “is power” and “technology is the essence of this knowledge” in the prevailing conceptual schema of progress (2002, 1–2). 4. Russell et al. (2015) is a technical report that was hyperlinked to the “An Open Letter” petition and published as an article in AI Magazine. 5. Russell et al. (2015) refer to the “rational agent” of rational choice theory, and to “satisficing” decision theory, indicating that a range of mainstream economic models could be relevant. In this chapter, I use the notions “rational agent” and homo oeconomicus to capture the broad sense of individual rational agency in mainstream economics. 6. Adorno describes the logic of this divided subject in terms of a “chorismos,” which Transcendence depicts in melancholic terms insofar as Evelyn wishes to touch her spectral lover, Will, who is unable to break out of the noumenal realm of the AI system and thus can only create for her the sensory illusion of his concrete existence. 7. See MacPherson (1962, 22) for an explanation of how Hobbes’ counterfactual “state of nature” was not a rendition of wild nature, outside ­civilized

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

85

society. To the contrary, it represented “civil society,” which was in fact early modern “market society” or capitalism. 8. Interestingly, in a popular article on the Future of Humanity Institute at Oxford University, Anderson (2013) characterizes Bostrom’s understanding of AI superintelligence in similar terms—as a “force of nature,” “something strong but indifferent.” However, while Marx referred to “natural laws” to expose the law-like historical (nonnatural) tendencies of capitalist society, Anderson takes Bostrom’s rhetoric at face value, compounding the impression that supercomputers are from an ontologically distinct order of nature rather than being historically contingent products of contemporary society. 9. “In other words, assuming that the observable universe is void of extraterrestrial civilizations, then what hangs in the balance is at least 10,000,000, 000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000, 000 human lives” (Bostrom 2014, 103). 10. See also Christian (2017). Entrepreneur Elon Musk has expressed concern about the risks of AI, while developing a space-colonizing, space travel business (Gibbs 2014; Dowd 2017).

References Adorno, Theodor. 2003. “Late Capitalism or Industrial Society? The Fundamental Question of the Present Structure of Society.” In Can One Live after Auschwitz? A Philosophical Reader, ed. R.  Tiedemann, 111–125. Stanford: Stanford University Press. ———. 1973. Negative Dialectics. Trans. E.B. Ashton. New York: Continuum. ———. 1998. “Subject and Object.” In Critical Models: Interventions and Catchwords. Trans. Henry W.  Pickford, 245–258. New  York: Columbia University Press. An Open Letter: Research Priorities for Robust and Beneficial Artificial Intelligence. n.d. https://futureoflife.org/ai-open-letter. Accessed 19 Dec 2018. Anderson, Ross. 2013. “Omens.” Aeon, February 25. https://aeon.co/essays/ will-humans-be-around-in-a-billion-years-or-a-trillion Backhaus, Hans-Georg. 1992. “Between Philosophy and Science: Marxian Social Economy as Critical Theory.” In Open Marxism, ed. W. Bonefeld, R. Gunn, and K. Psychopedis, vol. 1, 54–92. London: Pluto Press. Balakrishnan, Gopal. 2009. “Speculations on the Stationary State.” New Left Review 59 (Sept–Oct): 5–26. Barclay, Eliza. 2017. “Stephen Hawking: It’s Time To Get the Hell Off Planet Earth.” Vox, June 21. http://www.vox.com/science-and-health/2017/6/20/ 15836426/stephen-hawking-colonize-other-planets

86 

K. ASP

Becker, Kate. 2017. “When Computers Were Human: The Black Women Behind NASA’s Success.” New Scientist, January 20. http://www.newscientist.com/ article/2118526-when-computers-were-human-the-black-women-behindnasas-success Bland, Ben. 2016. “China’s Robot Revolution.” Financial Times, June 6. http:// www.ft.com/content/1dbd8c60-0cc6-11e6-ad80-67655613c2d6 Bostrom, Nick. 2008. “Dignity and Enhancement (Chap. 8).” In Human Dignity and Bioethics: Essays Commissioned by the President’s Council on Bioethics. Washington, DC: The President’s Council on Bioethics. https://bioethicsarchive.georgetown.edu/pcbe/reports/human_dignity/chapter8.html. Also at https://nickbostrom.com/ethics/dignity-enhancement.pdf ———. 2012. “The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents.” In Theory and Philosophy of AI, ed. V.C. Müller, Special Issue, Minds and Machines 22 (2): 71–85. ———. 2013. “Existential Risk Prevention as Global Priority.” Global Policy 4 (1): 15–31. ———. 2014. Superintelligence: Paths, Dangers, Strategies. London/New York: Oxford University Press. Digital Edition. Bremner, Brian. 2015. “Japan Unleashes a Robot Revolution.” Bloomberg, May 28. http://www.bloomberg.com/news/articles/2015-05-28/japan-unleashesa-robot-revolution Brynjolfsson, Erik, and Andrew McAfee. 2014. The Second Machine Age: Work, Progress and Prosperity in a Time of Brilliant Technologies. New  York: W. W. Norton. Christian, Bonnie. 2017. “Stephen Hawking Believes We Have 100 Years Left on Earth  – And He’s Not the Only One.” Wired, May 19. http://www.wired. co.uk/ar ticle/stephen-hawking-100-years-on-ear th-prediction-starmus-festival Christman, John, and Joel Anderson. 2005. “Introduction.” In Autonomy and the Challenges to Liberalism: New Essays, ed. J. Christman and J. Anderson, 1–23. Cambridge: Cambridge University Press. Dowd, Maureen. 2017. “Elon Musk’s Billion-Dollar Crusade to Stop the A.I.  Apocalypse.” Vanity Fair, March 26. http://www.vanityfair.com/ news/2017/03/elon-musk-billion-dollar-crusade-to-stop-ai-space-x Dyer-Witheford, Nick. 2015. Cyber-Proletariat: Global Labour in the Digital Vortex. Toronto: Between the Lines. Ellen, Roy 1988. “Fetishism.” Man (New Series) 23 (2): 213–235. Elliot, Larry. 2016. “Fourth Industrial Revolution Brings Promise and Peril for Humanity.” The Guardian, January 24. http://www.theguardian.com/business/economics-blog/2016/jan/24/4th-industrial-revolution-brings-promise-and-peril-for-humanity-technology-davos

4  AUTONOMY OF ARTIFICIAL INTELLIGENCE, ECOLOGY, AND EXISTENTIAL… 

87

Ford, Martin. 2015. Rise of the Robots: The Futures Has Lots of Robots, Few Jobs for Humans. New York: Basic Books. Gibbs, Samuel. 2014. “Elon Musk: Artificial Intelligence Is Our Biggest Existential Threat.” The Guardian, October 24. http://www.theguardian.com/technology/2014/oct/27/elon-musk-artificial-intelligence-ai-biggest-existential-threat Good, Irving John. 1966. “Speculations Concerning the First Ultraintelligent Machine.” Advances in Computers 6: 31–88. Harvey, David. 2006. The Limits to Capital (New and Fully Updated Edition). London/New York: Verso. Hawking, Stephen, Stuart Russell, Max Tegmark, and F.  Wilczek. 2014. “Stephen Hawking: ‘Transcendence Looks at the Implications of Artificial Intelligence  – But Are We Taking AI Seriously Enough?’” The Independent, May 1. http://www.independent.co.uk/news/science/stephen-hawkingtranscendence-looks-at-the-implications-of-artificial-intelligence-but-are-wetaking-9313474.html Hegel, Georg W.F. 1967. Hegel’s Philosophy of Right. Trans. T.M. Knox. London: Oxford University Press. Heinrich, Michael. 2013. “Crisis Theory, the Law of the Tendency of the Rate of Profit to Fall, and Marx’s Studies in the 1870s.” Monthly Review 64 (11). https://monthlyreview.org/2013/04/01/crisis-theory-the-law-of-the-tendency-of-the-profit-rate-to-fall-and-marxs-studies-in-the-1870s/ Horkheimer, Max, and Theodor Adorno. 2002. “The Concept of Enlightenment.” In Dialectic of Enlightenment: Philosophical Fragments. Trans. E.  Jephcott, 1–34. Stanford: Stanford University Press. Kurzweil, Ray. 2005. The Singularity Is Near: When Humans Transcend Biology. New York: Viking. MacPherson, C.B. 1962. The Political Theory of Possessive Individualism: Hobbes to Locke. Oxford/New York: Oxford University Press. ———. 1973. “The Maximization of Democracy.” In Democratic Theory: Essays in Retrieval, 3–23. Oxford: Claridon Press. Marx, Karl. 1976. Capital. A Critique of Political Economy. Trans. Ben Fowkes, vol. 1. London: Penguin Books. Mason, Paul. 2015. Postcapitalism: A Guide to the Future. New York: Farrar, Straus and Giroux. McKinsey Global Institute. 2017a. A Future That Works: Automation, Employment and Productivity. Report. McKinsey & Co. www.mckinsey.com/globalthemes/digital-disruption/harnessing-automation-for-a-future-that-works. Accessed 19 Dec 2018. ———. 2017b. Jobs Lost, Jobs Gained: What the Future of Work Will Mean for Jobs, Skills, and Wages. McKinsey & Co. http://www.mckinsey.com/featuredinsights/future-of-work/jobs-lost-jobs-gained-what-the-future-of-work-willmean-for-jobs-skills-and-wages. Accessed 19 Dec 2018.

88 

K. ASP

Noyes, Benjamin. 2014. Malign Velocities: Accelerationism and Capitalism. Winchester: Zero Books. Russell, Stuart, and Peter Norvig. 1995. Artificial Intelligence: A Modern Approach. Upper Saddle River: Pearson Education/Prentice-Hall, Inc. Russell, Stuart, Daniel Dewey and Max Tegmark. 2015. “Research Priorities for Robust and Beneficial Artificial Intelligence.” AI Magazine 36 (4): 105–114. https://www.aaai.org/ojs/index.php/aimagazine/issue/view/212 Schecter, Darrow. 2010. The Critique of Instrumental Reason from Weber to Habermas. London/New York: Continuum. Srnicek, Nick, and Alex Williams. 2015. Inventing the Future: Postcapitalism and a World without Work. London/New York: Verso. Standage, Tom. 2016. “The Return of the Machinery Question.” The Economist, June 6. http://www.economist.com/special-report/2016/06/25/the-returnof-the-machinery-question Transcendence. 2014. Directed by Wally Pfister. Los Angeles: Warner Brothers. Vinge, Vernor. 1993. “The Coming Technological Singularity: How to Survive in the Post-Human Era.” In Vision-21: Interdisciplinary Science and Engineering in the Era of Cyberspace, ed. G.A.  Landis, 11–22. Washington, DC: NASA Publication CP-10129. World Economic Forum. 2018. Future of Jobs Report 2018. Geneva: World Economic Forum. http://www.weforum.org/reports/the-future-of-jobs-report-2018

CHAPTER 5

Visions of Swarming Robots: Artificial Intelligence and Stupidity in the Military-­ Industrial Projection of the Future of Warfare Patrick Crogan

Introduction My subject in this chapter is the analysis, speculation, and recommendations for the future development and deployment of lethal autonomous robotic systems such as they appear in reports, studies, and presentations emanating from what is still aptly called the military-industrial complex. This term is still apt inasmuch as it continues to characterize a more or less explicit set of relations between networks of think tanks, lobbying organizations, defense and government agencies, politicians, academics and funded university research projects and their spinoffs into private sector industry and venture capital, as well as the military “clients,” who are both end users and beta-testers of new technological initiatives.1 I want to look in particular at the promotion of artificial swarming intelligence in the research and strategic scoping work that is supporting the development of this robotic future of warfare—if that is what it should still be called if military operations become automated under the control of artificially

P. Crogan (*) University of the West of England, Bristol, UK e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_5

89

90 

P. CROGAN

i­ntelligent systems. This is a question that Gregoire Chamayou has posed in Drone Theory in relation to the current uses of robotic systems in conflict, including and especially the remotely piloted unmanned aerial vehicles known ubiquitously as “drones”: Contrary to Carl von Clausewitz’s classical definition, the fundamental structure of this type of warfare is no longer that of a duel, of two fighters facing each other. The paradigm is quite different: a hunter advancing on a prey that flees or hides from him. The rules of the game are not the same. (Chamayou 2015, 53)

Chamayou is discussing the rapid growth over the last decades in the development and use of robotic systems by the militaries of advanced industrial powers in conflict zones such as Afghanistan, Iraq, and the occupied Palestinian territories, and covertly in other countries like Yemen, Somalia, and Pakistan. The deployment of these systems for surveillance and strike operations represents for him a blurring of accepted understandings of the nature of armed conflict that disturbs the ontological, ethical, geo-­political, legal, and strategic coordinates in which war is contemplated, conducted, and evaluated by society and its members. Chamayou has shown how a vision of the deployment of fully automatic weapons drives developments in drone technology (Chamayou 2015, 205–221). My chapter seeks to shine a critical light on an instance of this dreaming of a future of smart robot soldiers from a critical and philosophical perspective. As Chamayou points out, the research and development of robotic weapons includes the deployment of systems in actual operations. Systems developed by technologists of the military-industrial complex are trialed and tested on the ground by forces and this feeds back into the next cycle of system development. This means that the thinking about future swarms of drones does not end with the work of the think-tank boffins and Pentagon report writers, but continues right through all of the processes of funding, design, testing, accreditation, implementation, and so on, right the way down to the idiosyncratic adoptions, workarounds, and alternative uses that those forces using new weapons will develop in acquiring a “working knowledge” of them. My focus in this text is on the theorizing, recommending, and legitimating work at its closest to the sphere of political, collective debate and review. This work engages with the public’s political representatives through various fora, reporting, and lobbying activities. Often these reports are also available to the public, and various

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

91

talks and presentations and demonstrations related to these activities can be found online. My aim is to develop a critical reflection on the vision of the future of war in this well-funded and strategically well-placed work dedicated to influencing political decision-making regarding military spending and strategy in the USA. To develop this critical reflection I will mobilize a critical perspective on the future of military technology informed by the work of French philosopher of technology Bernard Stiegler. Stiegler’s propositions concerning the inescapably technical character of human being have implications for the consideration of all the important questions about human being and its technologically-driven transformation today. This includes consideration of the dynamics of military developments, which Stiegler has characterized as part of the concretization of a prevailing tendency toward a highly dangerous and destructive deployment of the possibilities of digital technology.2 For my part, I want to consider from this perspective the human intelligence that is marshaled and directed toward the future plans for a new kind of “war in the age of intelligent machines”—as Manuel De Landa, writing at the opening of the digital age, named the vision animating this military-industrial thinking (De Landa 1991). More specifically, I hope to show the pertinence of Stiegler’s comments on the human exercise of its potential for rational, reflective intelligence in order to critically evaluate the promotion in these plans of a strategy advocating the delegation of killing to a synthesized swarming intelligence.

Mind the Gap The work supporting the material development and implementation of automated robotic systems in actual military and security operations is an instance of what Stiegler calls “noetic” labor. Noetic refers to the ancient Greek noesis—understanding or intellect, from which the English word “nous” derives. There are two related claims that underpin Stiegler’s work on the nature of the work of human intellect that I need to introduce briefly. Firstly, noetic labor is the realization of the potential of the individual intellect that is only possible and always takes place in relation to the intelligence of the collective. The noetic individual thinks, imagines, articulates, and reflects on her experience and her perceptions. She reasons and speculates, drawing on established knowledge in (more or less explicit) dialogue with others. These others differentially comprise various groupings which define and condition her identity understood as

92 

P. CROGAN

process rather than as innate or essential element. Intelligence develops in this process of a mutual, ongoing “individuation” of individual and collective(s).3 As part of this individuation of the I and the we, the “psychic” and “collective” individual, intelligence is never without a social dimension, that is, it is never without an historical, cultural, and political dimension. For Stiegler, human intelligence is something that is formed in the relation between the individual and the social group(s) as individuals multifariously act on, confirm, question, modify, and reestablish the “we’s” collective knowledge of the world.4 This is especially pertinent to and only too clear in considering this military-industrial effort to rethink the nature, value, and conduct of military operations against other “we’s” through a discourse on the potential applicability to these questions of the latest intelligence about artificial intelligence. Secondly, as what distinguishes ours from the intelligence of other beings, human intelligence amounts to a process that emerges from a reflecting on and sharing of experience that is equally dependent on the accumulated knowledge of the shared experience of the collective’s history. The material substrate of this accumulated knowledge is artifactual, that is, technical. We inherit past insights and understandings from what is left behind from those who lived before. The social, cultural knowledge-­ banks of the collective are mediated by the material residues of the experiences and discoveries and plans of the past. For Stiegler this puts the specificities and the contingencies of the production and retention of technical forms at the very heart of individual and collective human becoming. What we make and what we preserve depends on these residues and also impacts future access and orientation to them for those thinkers to come. In what follows I hope to explain the relevance of these perspectives on human intelligence in my examination of the strange fruits of this noetic labor expended on advancing military robotics toward a model of a swarming intelligence that is promoted as both more and less than human intelligence. Stiegler’s thinking about human thinking is influenced by many writers, but his adoption and reformulation of Aristotle’s doctrine of the souls, taken together with the insight he draws from André Leroi-Gourhan about the technical character of the evolution of the human, offer the most pertinent frame for my topic. In The Decadence of Industrial Democracies Stiegler returns to Aristotle’s On the Soul to reread his influential account of the different character of vegetable, animal, and human life:

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

93

The noetic soul cannot therefore be simply opposed to the sensitive soul, from which it must on the other hand nevertheless be distinguished, as Aristotle did in fact do. The sensitive and the noetic compose as potential and act. The sensitive soul, according to Hegel, is the dunamis [potential] of the noetic soul that is only ever in action (energeia, entelecheia) intermittently. (2011b, 134)

Aristotle’s doctrine of the different “souls” animating different forms of life, that is, beings whose principle of movement is contained within themselves—from the vegetative to the “sensitive” (or perceiving) to the “noetic” or mental/intellectual soul—delineates three categories of beings that also represent a layered hierarchy where the lowest is subsumed within the higher (Aristotle 1986). The vegetative soul can nourish and reproduce itself, but does not perceive or move in response to the world around it. It takes care of the basics of life, drawing nourishment from the environment and propagating its reproduction. The sensitive (or perceptive) soul belongs to those life forms that perceive and act in the world, the animals who hide, run, migrate, lay in wait for prey, or seek out their reproductive partner. The noetic (intelligent, thinking) soul belongs only for Aristotle to those beings who can think, reflect, use language, who have access to logos, to logic and reason. Aristotle’s noetic labor of categorizing can be read as delineating clear oppositions between these three kinds of soul. The progression from the vegetative to the sensitive involves the incorporation of the lower soul as a kind of base layer of the higher soul. Similarly, the noetic soul is built on sensitive and vegetative layers that it transcends. Stiegler, a thinker of the becoming of human (and indeed all life) beyond fixed categories, pushes Aristotle’s categories toward a more processual reading of the interplay between modes of life animating living beings. For Stiegler, Aristotle can be read (after Hegel and others) by posing the relations between the three kinds of soul in terms of potential and act (dunamis and energeia).5 The sensitive soul of the animal life forms associated with it would be from this point of view active only part of the time while otherwise it would remain in the vegetative “mode.” Likewise, the potential of noetic life would be that of a being which remained at the “sensitive” animal level for part of the time. The noetic would be in action only intermittently, as the highest realization of the potential of mortal beings who cannot remain permanently as the expression of their dunamis.

94 

P. CROGAN

Daniel Ross explains this by saying that Stiegler understands the interrelation of these different kinds of life in a compositional rather than oppositional way: Thus, thinking compositionally, Stiegler says that the sensitive soul “inherits” the vegetation of the vegetative soul, and that the noetic soul “inherits” the sensitivity of the sensitive soul. But the point here is that this inheritance cannot be grasped in terms of layers of the soul: it is a matter of potential and act. Vegetativity is the sensitive soul’s way of being in potential, and sensitivity is its way of being active; similarly, sensitivity is the noetic soul’s way of being in potential, and noeticity is its way of taking action. (Ross 2009, 3–4)

In this compositional view, the human is composed of noetic and sensitive kinds of soul (and even, within the inheritance of the sensitive, of a vegetative kind), which could be thought of as three modes of existing. The noetic mode is in action some of the time, as the realization of its potential for reflective, reasoned, abstract, speculative (and so on) kinds of response to what is perceived, remembered, and anticipated. In a later work Stiegler will articulate this dynamic, processual character of human being in terms of a composition of autonomy and automaticity, where reflection, theorization, and conceptualization are composed with habitual, routine, and reflex behaviors in response to perceptions (Stiegler 2016, 72–74). The two are irreducibly related in human ways of life and its development. Culture is most often lived, for instance, as a kind of automatic, “sensitive” program for acting in relation to certain perceptions in certain situations and contexts. Its vitality and potential to remain current, compelling, and valuable to individuals depends, however, on the possibility of individuals to reflect on its appropriateness at times, to challenge, modify, or reaffirm it in and for the collective. This is not possible, however, at every moment of one’s lived existence. The routine observance of cultural norms holds in potential their review, criticism, suspension, and reformulation. The balance of these composed tendencies of the human life form is an ongoing question for the noetic mode’s intermittent actualization. To look ahead briefly to my argument’s conclusion, this is one way to express the stakes of the current promotion of a mode of warfighting that would relegate killing to an artificially intelligent, “sensitive” mode of thought in its actual conduct and execution. Is this balancing of the automatic and

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

95

the noetic tendencies of human action in the terrain of military and security operations appropriate to a form of life that lives up to the idea of the human such as it is conceived today in the concepts (and documents) of humanity associated with the “advanced” Western democracies prosecuting these operations? For Stiegler, the progress of the noetic being toward actualizing and extending its potential is dependent on its elaboration of a technical mode of existence.6 Stiegler is influenced by paleoanthropologist André Leroi-­ Gourhan’s notion of the exteriorization of biological functions through technical development. In Gesture and Speech Leroi-Gourhan (1993) developed a thesis concerning the crucial role of technical development in “hominization.” For him it is, paradoxically, our genetic nonadaptation that characterizes our evolution: Our significant genetic trait is precisely physical (and mental) nonadaptation: a tortoise when we retire beneath a roof, a crab when we hold out a pair of pliers, a horse when we bestride a mount. We are again and again available for new forms of actions, our memory transferred to books, our strength multiplied in the ox, our fist improved in the hammer. (Leroi-­ Gourhan 1993, 246)

Human evolution must be understood as having passed beyond an essentially genetic process to one which proceeds more technically, through the “evolution” of ethnic and cultural groupings on the basis of technical differentiation.7 The early hominid’s capacity to invent and then develop technical forms that replaced and improved functions such as biting and scratching—for example, the knapped flint tool, touchstone of evolutionary anthropology—initiated a pathway of human differentiation based on this process of exteriorization (and differentiation) of functions. Drawing on this approach, Stiegler sees the human as an inherently technical form of life whose evolution is “nongenetically programmed: since ethnic memory is external to the individual, it can evolve independently of genetic drift and is thus found in this sense to be temporal” (Stiegler 1998, 155). This ethnic memory evolves historically, conditioned by the material character of human artifactual production and subject to the contingencies of existence in time. The human animal develops and becomes different within the framework of the ethnic group. Her individual becoming leaves its mark on ethnic memory and contributes to evolving the identity of the ethnic. As discussed above, Stiegler thinks of

96 

P. CROGAN

the human as the combined “individuation” of psychic and collective individuals. For Stiegler the human must be understood as a composition of biological and technical elements that cannot be neatly opposed but must be thought of as composed. Stiegler will call these elements “organs” to insist on the irreducible composition of biological and artificial components in human life and what it has produced in the course of human history. The word “organ” comes from the ancient Greek organon, which means organ in the biological sense but also tool or instrument. The irreducible character of the composite of the human organic and social, collective body with “inorganic organized matter” is Stiegler’s core proposition leading off from Leroi-Gourhan’s daring claim (at the time, and perhaps even now) of the decisive differentiation of evolution that the human animal enacted and represents (Stiegler 1998, 82).8 A philosopher influenced by Derridean deconstruction but also by Gilbert Simondon’s process philosophy of individuation, Stiegler tries to formulate in these engagements with Leroi-Gourhan and Aristotle an explanation of how we are like and also unlike other biological life forms. It is necessary to make distinctions that avoid simplistic oppositions in Stiegler’s compositional thought. This point is one which I argue we cannot ignore as we humans of the globalized world pursue today the development of quasi-biological forms of intelligent technologies to prosecute our cultural and political disagreements about how to live. Our challenges as well as our potential are bound up with our technological development, and so a rigorous critical scrutiny is demanded when it is proposed to secure our future becoming through the development of cutting-edge technological systems simulating the automatic, swarming operations of species within the animal kingdom. If we are to understand ourselves as intelligent and ethical beings animated by a noetic soul, exercising our nous, it is important for Stiegler that we understand that we are that being only intermittently, as our most specific potential, one which we realize sometimes, while often resting in or “falling back” towards the sensitive “animal” and even “vegetative” states. This does not mean for Stiegler that the essential ground or “true nature” of human being is animalistic or instinctual, as evolutionary psychologists or figures like Richard Dawkins might claim. If we often regress toward the mode of being represented by Aristotle’s “sensitive soul” that is part of our biological and genetic history, this “animality” has always

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

97

and forever been altered by our passage beyond it to access the noetic soul. There is no pure going back.9 All of the ethical, moral, and political stakes of our “regression” to brutal, cruel, or insensitive and stupid acts reside in this, which is our key difference to the life forms that we have so much in common with but are also so different from. The bloodlust of violent slaughter in armed combat or in the shocking acts of groups like Daesh or Boko Haram should not be considered more authentically human (or indeed inhuman in a symmetrical fashion) in their supposedly “animalistic” savagery than the long distance executions by hi-tech drone strikes, the starvation of populations by strategic trade blockade, or the cynical indifference of economic imperialism. All of these kinds of violence and killing amount to various forms of regression from the higher possibilities of the being we conventionally call “human.” All of them are technically conditioned. They each require a rigorous analysis of the context and conditions in which they occur or have become acceptable to those who prosecute them. None of them can be excused or explained, however, as more or less “essentially” human by recourse to some biological bedrock of humanity. Such a gesture must be taken for what it is: a (technically conditioned and enabled) attempt by the noetic soul to avoid responsibility for its regressive realization.

Swarming as Natural Inspiration for Military Futures This “no going back” informs my approach to the projected application of the techniques of “natural” swarming phenomena to advanced military operations involving autonomous robots. I have in view the work of US think-tank lobby groups like the Washington-based Center for a New American Security (CNAS) and the Rand Corporation, and of the contributions to planning and policy by the United States Defense forces disseminated through the numerous reports and scoping documents they produce. Reports like the United States Air Force RPA [Remotely Piloted Aircraft] Vector: Vision and Enabling Concepts 2013–2048 from 2014 and its 2009 predecessor, the USAF Unmanned Aircraft Systems Flight Plan 2009–2047 contain detailed projections of future development and use of autonomous vehicles along with economic and strategic rationales for their adoption as a central plank of future military doctrine (United States 2009, 2014; also Department of Defense 2013). The RPA Vector report’s

98 

P. CROGAN

Executive Summary states that the extent of the deployment of autonomous systems in the USAF’s various “core functions” will be the decision of the personnel responsible for integrating the human and remotely or automatically piloted elements, but that it will be necessary to build “increased capability development and synchronization [for autonomous vehicles] within the Air Force’s various budgeting, development and requirements processes” (United States 2014, iv). The various reports and papers from CNAS’ “Future of Warfare” research program—including the two-volume Robotics on the Battlefield— amount to a much more strident advocacy for the role of autonomous systems in the future of US military operations (Scharre 2014b, c). The second volume, The Coming Swarm, is of particular interest here because of its consolidation of the theoretical and logistical arguments for the adoption of swarming robotic elements in the application of military force. In The Coming Swarm Scharre recommends embracing the “disruptive” paradigm shift from “direct human control” to one where “human controllers supervise the mission at command level and uninhabited systems maneuver and perform tasks on their own” (Scharre 2014c, 6). In the future, Scharre predicts, developments in AI will exceed the bounds of current doctrine regarding “networked” forces and achieve “true swarming – cooperative behaviour among distributed elements that gives rise to a coherent, intelligent whole” (10). The report promotes the potential of autonomous systems to provide a decisive (and cost effective) means for the US armed forces to maintain their superiority and global reach in a changing world (18–22).10 The coming military swarm of autonomous machines will move, communicate with each other and “think” faster, much faster, than human warfighters (33). In a section discussing the origins of the applicable concepts for a military mobilization of swarming robots, Scharre refers to the “groundbreaking monograph” by RAND Corporation intellectuals John Arquilla and David Ronfeldt (2000) entitled Swarming and the Future of Conflict. The Coming Swarm adopts the RAND study’s rather circuitous, duplicitous, and contradictory incorporation of the biological concept of swarming into the development of military doctrine. At the beginning of their discussion of “Swarming in nature” Arquilla and Ronfeldt state that “Military swarming cannot be modeled closely after swarming in the animal kingdom. But some useful lessons and insights can be drawn from it” (Arquilla and Ronfeldt 2000, 25). No reasons are given for why military swarming cannot be closely modeled on the natural variety. Moreover, full advantage

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

99

is taken of this “but” in the ensuing comparisons between examples of biological swarming phenomena in ants, bees, wolves, and mosquitos, and human instances of organized conflict, from mass war to U-Boat “wolfpack” tactics, Vietnamese National Liberation Front (Viêt Công) operations to guerilla warfare and the political deployment of mobs of “soccer hooligans” in Milosevic’s Serbia (25–27). Ant activities against other ant nests “have an operational complexity that mirrors human wars in striking ways”—indeed, the authors claim that ants have been “making war” long before humans came along (26). The World War Two German submarine tactics known as the “wolfpack” are interpreted as a metaphor with more than a kernel of biological truth. According to the authors, like the predatory animal wolfpack, the U-Boat tactics employed small groups of mobile units who targeted isolated members of a larger group (the shipping convoy). Another “powerful metaphor” identifies countermeasures against cyber-attack with the “mobbing” tactics of the biological immune system (27). Arquilla and Rondfeldt’s envisioning in 2000 of a future “BattleSwarm” doctrine identified advances in “information operations” and the networking of forces as a crucial reason and opportunity for moving beyond maneuver-based Air-Land Battle doctrine (Arquilla and Ronfeldt 2000, 78). Scharre’s The Coming Swarm imagines artificially intelligent swarming forces whose capacities for action based on their realtime communications linkages far exceed the goals of networked human warfighters. This vision of swarming robotic military force seeks to realize the potential of what Kevin Kelly called the “neobiological” shift in work on intelligent systems design from the 1980s (Kelly 1994). It sets out to project applications more or less modeled on the “striking” similarities and “powerful metaphors” offered by swarming intelligence in the animal kingdom to the extremes of violent conflict between humans. These metaphors imply an essential, biological equivalence between human and animal existence. The swarming phenomenon has been hugely significant in the development of what became “the New AI,” which took a different tack from the “classic AI” approach of modeling human thinking. According to John Johnston’s overview of the history of artificial intelligence, the classic AI works toward software implementations of models of the functioning of human intelligence conceived as a top-level, hierarchical decision-making logic.11 That is, it sets out to build a program that emulates thinking understood as a complex process of logical symbol manipulation that works with models representing external reality. Instead of modeling

100 

P. CROGAN

c­ ognition as a top-down, abstract logical scheme, the new AI adopted a bottom-­up approach of experimenting with smaller parts of less symbolic and more immediate, responsive functioning to see how their combination and coordination might open onto higher, more conceptual levels of processing inputs and organizing behavior. The key concept here is “emergence.” Intelligence is theorized as an emergent phenomenon that appears in evolution as an unexpected result of the combinations of lower level operations of the organism’s perception-­ response physiology. The symbolic, conceptual, and logical thinking of the human mind has emerged in the course of the evolution of biological nervous systems conducting and arranging these relatively simpler processings of data: fight or flight, sense prey—attack, detect potential mate— begin the reproduction process, and so on.12 Emergence is unexpected, surprising, messy, and dramatic. It brings forth a whole that exceeds the individual parts, a whole which is not discernible as an implication in them or is only visible after the fact. The emergent intelligence exhibited in the coordination of actions by the collective insect species like the ant, the termite, and the bee, the flocking of birds and the schooling of fish, became phenomena of interest and inspiration for the new AI and robotics researchers from around the 1980s. The foraging ant has a simple signaling and response system, the famous pheromone secretions that leave a mark other ants can detect and so can contribute to a temporary trail. The individual ant’s simple functioning—follow the trail, secrete or not (thereby intensifying the trail’s collective signaling to subsequently arriving ants) is the basis of the collective, emergent “superorganism” of the ant nest’s food scouting and acquisition, threat response, and other “higher level” capabilities.13 Drawing on the success of the new AI in modeling and simulating such emergence in artificial systems, it is the potential emergence of a collective warfighting intelligence among a “swarm” of military bots that is being imagined— and indeed confidently predicted—in reports like The Coming Swarm.

Emergent Stupidity In a conference presentation on the same theme of the potential of swarming AI for the future of war, The Coming Swarm author Paul Scharre uses a series of computer graphics images and simple diagrammatic animations to illustrate the report’s imagining of future uses of autonomous swarms of robots (Scharre 2014a). These images depict land and sea terrain from

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

101

an overhead view reminiscent of the perspective provided the player in strategy simulation games. The simplified, schematic representation of military units such as ships, missiles, aircraft, and tanks resembles the graphics in these games and in even older traditions of the depiction of battles for military gaming, simulation, and historical representation.14 Ships, planes, or missiles are recognizable, but the aesthetic is one of depicting symbolic tokens on a simplified map of the territory rather than an illusionistic, photorealistically rendered world.15 This visual strategy is both institutionally conventional and absolutely symptomatic of the willful blindness—a kind of voluntary stupidity—that characterizes this projection of the immediate future of the warfare envisaged for the armed forces of the United States. Like all fiction, this imaging (which is also an imagining) operates a strategic selection (and exclusion) of the elements to be included in this vision of the scenario. In one sequence of images conventional naval ships are surrounded by a defensive screen of robotic vessels, which respond to the appearance of enemy vessels with a swarming attack. Whether the unfriendly vessels have humans on board or are also autonomous is not specified. In another sequence a swarm of land-based robots advances inland as an avant-garde, eliminating enemy units in advance of icons representing conventional armored personnel carriers. The kernel of the controversies and debates that have accompanied the US military and security deployments of semi-autonomous and remotely piloted systems over the past decade or so is conspicuous by its absence in these visualizations of future war: no noncombatants, no villages, no temples, no town square meetings, no farmers, fishers, no bus trips to weddings or football matches, no “others” except the implied presence of enemy combatants (and even this is a weak and unnecessary implication). The absence of the occupants of the contested territory in these images is reflected in The Coming Swarm, which contains no substantial discussion of the challenge of discriminating between combatants and noncombatants. As such it sidesteps the subject of much of the civil protest and activism in recent years challenging the policies of US military and covert CIA use of drones in surveillance and targeted killings over the skies of Afghanistan, Pakistan, Yemen, and Somalia. Human rights groups have criticized the deployment of drones over inhabited territories not recognized as conflict zones. In Waziristan in northern Pakistan, for instance, the resident population has been subjected to a perpetual threat of attack by ever present overflying drones. Legal researchers attacked the failure of

102 

P. CROGAN

the Obama administration to establish the legal grounds for these operations in either the laws of war or of policing.16 The Obama administration’s controversial “signature strike” attacks on unnamed and unidentified Pakistani, Yemeni, and Somali inhabitants highlighted for some the risks of conducting war via remote-controlled systems in the kind of “asymmetrical conflicts” of the perpetual “war on terror” (Becker and Shane 2012). These operations targeted individuals on the basis of the “signature” of the data trail collected from drone and electronic communications surveillance. The Obama administration’s direction of travel in dealing with the problem of identifying combatants among the civilian population was revealed by investigative journalists who discovered that the White House had adopted a George W.  Bush administration-era definition of combatant as any military-aged male in the zone of operations (Becker and Shane 2012).17 Moreover—and this is one reason why I called this a kind of voluntary stupidity—it ignores the significant debates in military strategic circles about the value of drone deployments in counterinsurgency operations such as those undertaken in these countries. As Chamayou has shown, the burgeoning use of drones and surveillance and strike operations was not universally welcomed among military leaders and strategists because it was seen as inimical to the doctrine of counterinsurgency, which is based on winning the support of local civilian populations.18 The most substantial critique along these lines is made by former advisor to General David Petraeus in Iraq, David Kilcullen (Chamayou 2015, 65). For Kilcullen and other supporters of counterinsurgency over “antiterrorism,” the key aim of occupation campaigns such as in Iraq and Afghanistan is to “mobilize the population in its cause” against competing efforts by the enemy (Kilcullen cited in Chamayou 2015, 67). The avoidance of any engagement in these debates, in the context of the projection of future strategic doctrine and the configuration of military forces, seems calculated to ignore, that is, to remain ignorant of, these significant questions concerning the use of remotely controlled and semi-automated forces. There is a second reason why I called this report’s projection of the future of swarming forces stupid, one which relates specifically to the conception of human intelligence as a dynamic and intermittent phenomenon. This stupidity reveals itself most clearly here, not as an error in reasoning or gathering of relevant information but as an approach that accepts a diminution of existing warfighting nous and, with it, the noetic potential of humans conducting war. For what is being advocated here is

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

103

effectively a development of an emergent, artificial intelligence that would replace and render obsolete existing traditions and doctrines of “military intelligence” about fighting the enemy. The actual execution of military operations involving lethal force would be directed by a radically transformed combination of human and artificially intelligent decision-making. CNAS calls for a shifting of the “existing paradigms” of command and control to enable “human supervision of large swarms” (Scharre 2014c, 6). In the overturning of the paradigms of fighting war the specific skills and knowledges of many warfighting activities would be delegated to autonomously functioning systems, envisaged to be able to react with an emergent tactic to different challenges, posed for instance by either human or nonhuman (AI-based) opponents. This would inevitably entail a de-­ skilling and re-skilling process for the armed forces as a whole, and what is only tacitly acknowledged in this report is that a significant diminution in the scope and depth of existing skills and experience of actually killing the enemy is a consequence of the promoted paradigm shift. Skills develop through the analysis and formalization of the “lessons of experience.” When the robots are left to conduct many engagements with the enemy, there will be less need for those skills and a consequent withering of the competencies and the “higher learning” of those lessons. It is among this “higher” strata that resides the subtlety of decisions about who is the enemy and who is the noncombatant caught in the middle, about what is the appropriate action in response to both the situation and the rules of engagement governing action in that particular situation, who to believe and who to discount, and so on. This subtlety involves the ethical-­political, the legal, the moral as much as the tactical and operational dimensions of the conduct of war. From my perspective informed by Stiegler’s work, it is important not to characterize this simplistically as a disabling of human intelligence in favor of machine intelligence. As I argued above there is no purely human intelligence that is independent of its technical supports. The human is always conditioned by its technicity—as ethnically and historically “evolving,” the human is composed with a technical becoming that it animates but which, as exterior and material process, is not entirely reducible to human being nor is it merely its instrumental supplement. In this regard, USAF fighter pilot, M. Shane Riza, has examined the highly complex and semi-automated condition of his advanced technological weapon system in his book Killing without Heart, which nonetheless argues for a substantial reconsideration of the trend towards the

104 

P. CROGAN

increased automation of warfare (Riza 2013, 3–6). As Riza shows, automation already plays a central role in the advanced weapons systems of the industrial powers, in tracking and targeting from missile guidance to missile defense, in vehicle navigation and systems monitoring, in the functioning of communications and logistics, in training and simulation and so on. This blurring of the opposition between automation and human agency does not mean that there is no difference between the existing and the proposed future state of warfare, or that the difference is one of degree and does not really matter. On the contrary, because of the composed, dynamic and therefore contingent nature of the becoming of human activity in general, and here in the “extreme” zone of human conflict especially, it is all the more important to distinguish rigorously between different compositions and to argue their merits in shaping what we will become. For this is what is at stake.19 What The Coming Swarm is advocating is a shift in the combination of autonomy—machinic and human—prevailing in military operations. In N. Katherine Hayles’ terms this would be a reconfigured “cognitive assemblage” of human and artificial, computer-generated cognition: Because humans and technical systems in a cognitive assemblage are interconnected, the cognitive decisions of each affect the others, with interactions occurring across the full range of human cognition…. As a whole, a cognitive assemblage performs the functions identified with cognition  – flexibly attending to new situations, incorporating this knowledge into adaptive strategies, and evolving through experience to create new strategies and kinds of responses. (Hayles 2016, 33)

Hayles has in mind complex systems of human and computing and mechanical elements such as Los Angeles’ Automated Traffic Surveillance and Control system.20 Her key insight is that it is important to pay attention to the “thinking of the system” as a whole, and to think of this in terms of a coevolving complex rather than as a sophisticated tool in the hands of the human designers and operators. The implication of this wholistic characterization of a cognitive assemblage is that as these become more widespread and more technologically sophisticated, human cognition will become increasingly incorporated within and conditioned in its possibilities by the emergent evolution of the assemblage’s “adaptive strategies.”

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

105

For Stiegler, human thinking has always to be understood as the product of a composition of biological and artificial elements. Today, with ongoing advances in artificial intelligence, data processing algorithms, and so on, more and more tasks traditionally reserved for the people in the “cognitive assemblage” are being delegated to these artificial elements. The character, rhythm, and quality of the thinking of the assemblage will change. The nature of and the passage to the highest levels of our noetic potential are consequently subject to alteration, put into question. One could say that the highest task of thinking is precisely to think this circumstance, to understand the alterations to the very conditions of cognition brought about by technical innovations, and to think about how and what to make of these innovations in order to best shape the becoming of the technical beings that we are. For Stiegler, this is to adopt technical change rather than to adapt ourselves to it as if we are still an essentially biological being, susceptible to environmental changes but not able to selectively inflect their affects on us. The refusal to do so is to choose to remain in a stupid mode of thought. The critical challenge today in the era of the explosion of artificial intelligence is to come to terms noetically with the potential and the challenge posed by the capacity of “cognitive assemblages” to flexibly attend to the new and to adapt its operations iteratively and automatically at a speed which threatens to outpace their human elements. The human noetic potential remains for now the distinguishing element within the assemblage, the element which brings intermittently the highest potentials of the “noetic soul” to its evolution. In the cognitive assemblage of the future of war envisaged by The Coming Swarm, the role of the human is cast in very general terms in relation to an increasingly “sensitive” mode of operational, artificial cognition focused exclusively on perceive-and-act processes of targeting and attack. The report opens by saying that AI will “fall short of human intelligence in many respects” and that experimentation has to be pursued in an “aggressive campaign of experimentation and technology development” in order to discover which combinations of human and autonomous robots is optimal (Scharre 2014c, 7). The role of the human is then broadly conceived as one of exercising “common sense” oversight and a restrictive response to any undesired behaviors from the inherently unpredictable emergent swarm (26). No sense of the inherently compositional dynamic of the changing “military intelligence” of the conduct of war as a whole is explored in this prospective vision of the future of war. The shocking inadequacy of The Coming Swarm’s idea of “common sense”

106 

P. CROGAN

­ anagement of swarming AI elements appears as a fissure in the report’s m own consideration of the dangers of emergent behaviors that could outstrip the capacities of humans to respond to them in time: While increased automation may have tactical benefits in allowing faster reaction times to enemy actions, it could also have strategic consequences if the speed of action on the battlefield eclipses the speed of decision-making for policy makers. Increased autonomy in the use of force raises the dangerous spectre of “flash wars” initiated by autonomous systems interacting on the battlefield in ways that may be unpredictable. While militaries will need to embrace automation for some purposes, humans must also be kept in the loop on the most critical decisions, particularly those that involve the use of force or movements and actions that could potentially be escalatory in a crisis. (Scharre 2014c, 7)

A simple question: how will the supervising human controllers know what is happening in these crisis circumstances—developing at a speed beyond their ability to comprehend them—in order to make these critical decisions before it is too late? The “noetic soul” of CNAS projections of the future of war amount to a willful limiting of the intermittent potential of human interiority (in warfighting), and an ignoring of existing debates about the strategic and political merits of remotely conducted warfare. I have characterized this as advocating a pathway toward an increasingly stupid global, geopolitical, military engagement in the world by the “advanced industrial powers.” It is a responsibility of those of us who are members of the societies of these democratic states to actualize our noetic potential to intervene in these debates, to respond to these propositions for the conduct of war made in our name by organizations and institutions that think about these complex issues on our behalf.

Conclusion In this chapter I have taken a philosophically informed approach to this disturbing shifting of the goalposts (of military operations and of the human-weapon relations that are being reimagined). This is in part to take seriously Stiegler’s claim that changes in the technical possibilities of the human have to be taken as potentially reframing the human being inasmuch as it is a prosthetic, technical form of life. As a cultural, historical,

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

107

and artifactual form of existence, the human is a being susceptible to the dynamics of technological development. Questioning the relevance or legitimacy of an existing or projected state of cultural or political affairs— and this means today global, geopolitical affairs—can appeal to the way things were as most appropriate to the conduct of human life and society, such as those challenges to drone strikes based on existing human rights and international law. It might also confront, however, the sense of the human as, precisely, an historical, political, and technologically contingent one. This is what I have tried to do here. What is being advocated in reports like The Coming Swarm is the necessity (both economic and strategic) to adopt a future deployment of autonomous systems that amounts to a less restrained, less deliberative, less controllable, and less understood mobilization of lethal force than what exists today in accepted military doctrine (if not always in practice). This is “regressive” in the sense of a falling back from the highest level of thinking, a perhaps lazy, but perhaps calculated and strategic falling away from, or cynical avoidance of a rigorous consideration of the projections of a near future for lethal swarming robotics. What is regressive in these projections, what is being ignored, overlooked, unthought-of? Above all, or perhaps before all, that war is a human activity that destroys all the others, that states cooperate in international agreements and subscribe to regulations designed to severely limit its occurrence, circumscribe its destructive affects and contain the threat of its escalation. In other words, that war is not biological, we are not exactly like other animals, and that the passage to the act of war—which is a catastrophic falling back into forms of thinking and acting much “lower” than political negotiation and conflict resolution—that especially the act of war must never be projected, imagined, or programmed as something where the killing and destruction might be left to a “cognitive assemblage” more automatic and unpredictable than thoughtful, and less self-doubtful for all its potential gains in speed, effectiveness and emergent inventiveness.

Notes 1. The term “military-industrial complex” was (in)famously coined by Dwight D. Eisenhower in his presidential farewell speech in 1961, having presided over its expansion across his two terms in the midst of the Cold War (Eisenhower 1961). For accounts of the central role played by the mobilization and transformation of university research and knowledge

108 

P. CROGAN

production in the intensification of the military-industrial complex, see Edwards (1996, 52–55), and Pickering (1995). 2. See Stiegler (2016, 47). Stiegler refers to Chamayou’s critique of the performative undermining of the laws and conventions of war by the US (and other state) military’s drone operations. 3. See Stiegler (2011a, 93–98). 4. “The individual psyche is originarily psychosocial, and the social is not an ‘intersubjective’ aggregate of already-constituted individuals. The individuation of the I is that of the We, and vice versa, even though I and We differ….” (Stiegler 2011a, 94). 5. Dan Ross will add Heidegger and Derrida to this list of proponents of a more processual reading of Aristotle. See Ross (2009). 6. Stiegler points out in Technics and Time 1 (1998) that this is a thought Aristotle could not entertain, because for him technical objects were not animate and did not contain the principle of their own movement. They were merely tools in the hands of the living being, relegated by Aristotle to the margins of questions concerning human being and life in general, initiating a long history of the marginalization of technics from philosophy in the Western tradition. 7. In Leroi-Gourhan’s thesis, the genetic evolutionary drift that altered the relative functionality of what became the hands and the feet of the earliest homonid predecessors also altered the potential of cranial and facial development and opened up the possibility of the expansion of the brain pan. At some stage, the potentialities of the advance and accumulation of technical invention and symbolization were realized, and this transformed the evolution of these life forms. 8. Indeed, Stiegler proposes in For a New Critique of Political Economy (2010) that today we urgently need a shift in scientific thought toward an “organological” paradigm—organology would be dedicated to the analysis of the relations between the biological and the artificial organs in the formation and refinement of knowledge in all fields of endeavor. 9. Dan Ross puts it this way: “For the noetic soul, the noeticity of that soul is something which pervades it, and thus which characterizes even its sensitivity: the sensitivity of the noetic soul is transformed by its being-noetic, and exists as a power of the noetic. In other words, for the noetic soul, aesthesis is always inscribed in noesis, and noesis, thinking, is always inscribed in aesthesis” (Ross 2009, 4). 10. The cost effectiveness of swarming systems is argued by Scharre by resort to the notion of the “cost-exchange ratio” employed in the theory of nuclear war. This refers to the relative cost efficiency (versus developing conventional weapons) of deploying massive numbers of units against your enemy and so draining their resources and capacity to defend themselves or

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

109

launch a counter-attack. The rationale here is that massive numbers of cheaper (and overall lower quality) military units allowed by their swarming coordination enables “a disaggregation of that combat capability into larger numbers of less exquisite systems which, individually, may be less capable but in aggregate are superior to the enemy’s forces” (Scharre 2014c, 20–21). 11. My overview of the new AI is indebted to John Johnston’s account of the history of AI and robotics in The Allure of Machinic Life (2008). See in particular pp. 277–336. 12. Of course, these are already quite sophisticated levels of the sensory-­motor schema built on layers of emergent organization arising from patternings of the simplest levels of sensory stimulus and response (see Johnston 2008, 302–303). 13. See Kelly’s accessible overview of the influential entomological and ethological work on swarming in Out of Control (Kelly 1994, 5–29). 14. In Crogan (2017), I provide an overview of the aesthetic logics found in different genres and historical developments of digital wargaming. 15. The only exception in the presentation to this aesthetic is an image representing the potential use of autonomous medical evacuation helicopters for injured US soldiers. This image is richly rendered in naturalistic detail, showing a rescue from the perspective sharing the soldiers’ situation on the ground in the midst of the battle terrain. 16. See Human Rights Watch (2010), Stanford Law School (2012). 17. The assertion that this redefinition of “combatant” derived from the Bush administration is made in Woods (2012). 18. See Chamayou (2015, 60–72). 19. This point is made most tellingly by Chamayou, when he insists on the ethical, critical, and political potentiality of human soldiers as something which the deployment of drones is able to sidestep. He discusses something of a set piece in the philosophical debates about the morality of war: the often recounted refusal to kill a defenseless enemy when presented with the opportunity (the archetypal instance of which is catching an enemy in the course of having a wash or going to the toilet). Chamayou resumes commentaries by philosophers Michael Walzer and Cora Diamond before arguing that the refusal to kill in this situation is because “if he does it he knows he will have to live with that action. And that is what he rejects in advance. It is a matter not of duty but of becoming. The crucial, decisive question is not ‘What should I do?’ but ‘What will I become?’” (Chamayou 2015, 199). Stiegler will characterize this as the capacity of the soldiery to “dis-­automatize” their actions, a potential which emerges in the composition of highly drilled and routinized behaviors and intermittent noetic agency, of psychic interiority with collective identity formation, and pros-

110 

P. CROGAN

ecuted in the technical milieu of human conflict (Stiegler 2016, 55ff). This potential is something that Riza’s concern with the ethics of the “warrior” is aiming at as well, from “within” the intellectual traditions of the armed forces. Like all human life, warfighting involves a composition (that is also an ongoing compromise) of the automatic and the autonomous, the habitformed, reflex response and the thoughtful, reflective or modifying one. When Scharre makes a comparison between the coordinated action of a pack of wolves and the heavily drilled skill execution and tactical maneuvers of a combat squad, his conception of the soldiers as “sensitive,” perceiveand-act beings is a privative one that strips them of this intermittent potential (Scharre 2014c, 44). 20. Hayles does discuss the deployment of remotely piloted vehicles by the US, but in a way that does not, in my view, develop productively the implications of her theorization of “cognitive assemblage.”

References Aristotle. 1986. De Anima (On the Soul). Trans. Hugh Lawson-Tancred. London: Penguin. Arquilla, John, and David Ronfeldt. 2000. Swarming and the Future of Conflict. Santa Monica: RAND Corporation. Becker, Jo and Scott Shane. 2012. “Secret Kill List Proves Test of Obama’s Principles and Will.” New York Times, May 29. http://www.nytimes.com/2012/05/29/ world/obamas-leadership-in-war-on-al-qaeda.html?_r=1&hp# Chamayou, Gregoire. 2015. Drone Theory. Trans. Janet Lloyd. London: Penguin. Crogan, Patrick. 2017. “Videogames, War and Operational Aesthetics.” In War and Art: A Visual History of Modern Conflict, ed. Joanna Bourke, 324–329. London: Reaktion Books. De Landa, Manual. 1991. War in the Age of Intelligent Machines. New  York: Zone Books. Department of Defence. 2013. Unmanned Systems Integrated Roadmap: FY2013–2038. http://archive.defense.gov/pubs/DOD-USRM-2013.pdf Edwards, Paul N. 1996. The Closed World: Computers and the Politics of Discourse in Cold War America. Cambridge, MA: MIT Press. Eisenhower, Dwight D. 1961. Military-Industrial Complex Speech. Yale Law School Lillian Goldman Law Library. http://avalon.law.yale.edu/20th_century/eisenhower001.asp. Accessed on 12 Dec 2017. Hayles, N.  Katharine. 2016. “Cognitive Assemblages: Technical Agency and Human Interactions.” Critical Inquiry 43 (1): 32–55. Human Rights Watch. 2010. Open Letter to President Obama: Targeted Killings and Unmanned Combat Aircraft Systems (Drones). Washington, DC: Human

5  VISIONS OF SWARMING ROBOTS: ARTIFICIAL INTELLIGENCE… 

111

Rights Watch. December 7. http://www.hrw.org/news/2010/12/07/letterobama-targeted-killings Johnston, John. 2008. The Allure of Machinic Life: Cybernetics, Artificial Life, and the New AI. Cambridge, MA: MIT Press. Kelly, Kevin. 1994. Out of Control: The Rise of Neobiological Civilization. Reading: Addison-Wesley. Leroi-Gourhan, André. 1993. Gesture and Speech. Trans. Anna Bostock Berger. Cambridge, MA: MIT Press. Pickering, Andy. 1995. “Cyborg History and the World War II Regime.” Perspectives on Science: Historical, Philosophical, Social 3 (1): 1–48. Riza, M. Shane. 2013. Killing without Heart: Limits on Robotic Warfare in an Age of Persistent Conflict. Washington, DC: Potomac. Ross, Daniel. 2009. “Politics and Aesthetics, or, Transformations of Aristotle in Bernard Stiegler.” Transformation 17: 1–8. Scharre, Paul. 2014a. Eighth Annual Conference: Robotics on the Battlefield: The Coming Swarm. Center for a New America Security. Video Published 20 June 2014. https://youtu.be/_WuxwBHI6zY ———. 2014b. Robotics on the Battlefield  – Part One: Range, Persistence and Daring. Washington, DC: Center for a New American Security. https://www. cnas.org/publications/reports/robotics-on-the-battlefield-part-i-range-persistence-and-daring ———. 2014c. Robotics on the Battlefield  – Part Two: The Coming Swarm. Washington, DC: Center for a New American Security. https://www.cnas. org/publications/reports/report-preview-robotics-on-the-battlefield-part-iithe-coming-swarm Stanford Law School (International Human Rights and Conflict Resolution Clinic) and New York University School of Law (Global Justice Clinic). 2012. Living Under Drones: Death, Injury and Trauma to Civilians from US Drone Practices in Pakistan. http://livingunderdrones.org/wp-content/uploads/2012/09/ Stanford_NYU_LIVING_UNDER_DRONES.pdf Stiegler, Bernard. 1998. Technics and Time 1: The Fault of Epimetheus. Trans. Richard Beardsworth and George Collins. Stanford: Stanford University Press. ———. 2010. For a New Critique of Political Economy. Trans. Daniel Ross. Cambridge: Polity Press. ———. 2011a. Technics and Time, 3: Cinematic Time and the Question of Malaise. Trans. Stephen Barker. Stanford: Stanford University Press. ———. 2011b. The Decadence of Industrial Democracies: Disbelief and Discredit, Volume 1. Trans. Daniel Ross and Suzanne Arnold. Cambridge: Polity Press. ———. 2016. Automatic Society: The Future of Work, Volume 1. Trans. Daniel Ross. Cambridge: Polity Press.

112 

P. CROGAN

United States Air Force Headquarters. 2009. U.S. Air Force’s Unmanned Aircraft Systems Flight Plan 2009–2047. http://www.fas.org/irp/program/collect/ uas_2009.pdf ———. 2014. United States Air Force RPA [Remotely Piloted Aircraft] Vector: Vision and Enabling Concepts 2013–2048. http://www.af.mil/Portals/1/documents/news/USAFRPAVectorVisionandEnablingConcepts2013-2038.pdf Woods, Chris. 2012. “Analysis: Obama Embraced Redefinition of ‘Civilian’ in Drone Wars.” Bureau of Investigative Journalism, May 29. https://www.thebureauinvestigates.com/opinion/2012-05-29/analysis-obama-embracedredefinition-of-civilian-in-drone-wars

CHAPTER 6

The Business of Ethics, Robotics, and Artificial Intelligence Kathleen Richardson

Introduction There is a widespread explosion in the subject of ethics of robots and artificial intelligence (AI) on the public stage led by governments, businesses, academics, and activists. Global technological corporations such as Facebook, Google (Alphabet), IBM, and Microsoft are leading the charge to make robots and AI more ethical and have set up research units in their companies committed to exploring the ethics of new technologies. The ethics of robots and AI is also a cause funded by the European Commission and governments in China, the US, Russia, France, and the UK all laying stake to the technological innovations produced by robots and AI. The professional bodies of ACM1 and IEEE2 have engaged in stakeholder consultations with their membership and drawn up their own codes of ethics. The privately funded Future of Life Institute3 and nongovernmental organizations such as the Campaign to Stop Killer Robots4 and my own group, the Campaign Against Sex Robots,5 also present different ethical a­ rguments

K. Richardson (*) Faculty of Computing, Engineering, and Media, De Montfort University, Leicester, UK e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_6

113

114 

K. RICHARDSON

and goals. Ethics as a model for solving the perceived threats of robots and AI has achieved pre-eminence in Europe and the US, in ­professional, nongovernmental, and private-funded bodies. Ethics, to put it mildly, has become big business. I write this chapter drawing on my experiences exploring the ethics of robots and AI in two European Commission– funded projects, DREAM (Development of Robot-Enhanced Therapy for Children with Autism6) and REELER (Responsible Ethical Learning in Robotics7), and participating in ongoing discussions among a vibrant community made up of academics, business, government, and members of the public about the importance of ethics in robotics and AI. Ethics, broadly speaking, is the study of right or wrong, what makes something right or wrong, moral or immoral and how decisions are developed in ways that can be systematically described and studied. Many ideals that underscore our daily lived experiences, such as “do not kill” or “do not steal” are described in ethical systems and one could argue going across many contemporary and historical ethical value systems. Ethicists might be interested to know why a person should not kill or steal and what rules, and arguments for when murder or robbery might be just. There are also values that arise out of specific cultural shifts, such as the ethics that underlined the development of Western liberal juridical democracies. Western countries have juridical–legal and political systems that are organized around ethical values of rights-based justice of individualism often balancing personal liberty with social good. At a European Commission workshop held in Brussels in May 2018, stakeholders were invited to discuss the ethics of robots and AI; but participants were unclear about what was meant by “ethics.” What is it about ethics that has singled it out to be the subject, par excellence, to help us address concerns for robots and AI and their potential impacts on humans and society in the twenty-first century? Or are there limits to ethical discourses in sorting through these issues? In what follows I will explore ethics using examples from contemporary narratives in robots and AI. I want to show how asking questions about right or wrong, about potential impacts, risks, harms, benefits, and personal versus social good only gets us so far while engaging with ethics. Then, I will move on to show the problems with ethical theories today, and how feminist ethics is necessary, not merely because feminist ethics deals with marginalized populations (women), but that feminist ethics critically examines how power and exclusion is normalized through Western ethical narratives.

6  THE BUSINESS OF ETHICS, ROBOTICS, AND ARTIFICIAL INTELLIGENCE 

115

What Is Ethics? Any study of ethics will show that it is not a unitary body of ideas, and there is no ethics template for the “good life” that all ethicists agree with across space and time. In fact, in ethics, what constitutes the good can differ from one philosophical set of ideas to another; such as Immanuel Kant’s (1978) focus on reason as the principle focus of ethics versus David Hume’s (1940) focus on the passions. We might believe there are ethical universals such as “thou shall not kill,” but many countries have standing armies and sanction warfare, buy and sell weapons of destruction, and they do kill. We might believe “do not steal” is a universal maxim but then what constitutes stealing? Is high interest a form of legal stealing or a legitimate risk for a bank? If ethics is about right or wrong, how is it possible to be the advocate of virtue, as was Aristotle, who wrote about how adult male citizens could realize their worthiness or virtue, while simultaneously providing a rationalization for slavery and the subordination of women? If we examine ethics, we do not find a neutral, nor indifferent set of ideas, but ideas developed in specific contexts, laced with prejudices and hatred directed against women. There is also a view that ethical thinking can be purged of its socially unacceptable parts, so that Aristotle’s virtue ethics can be stripped of its pro-slavery and misogynistic arguments, while keeping other parts such as the description of good character (Aristotle 1962). Is it true that we can take ideas construed as valuable, while rejecting other parts of a body of ethical knowledge? There are also epistemological ethical perspectives that have become contaminated by use, interpretation, or association with particular ideologies or brutalizing political practices. One such example of this is the moral philosophy of Nietzsche, whose ideas were made use of by leaders of the Nazi movement in Germany. Nietzsche (1989) proposed that the masses are unable to differentiate their thought and constantly trapped in a “herd morality.” He called for a leader who would stand above the crowd— Übermensch the superhuman—that was made use of in the philosophy of the Third Reich, though the Nazi’s use of Nietzsche was selective, ignoring his passages where he criticized racialized nationalism and Germans (Brinton 1940, p. 141) and wrote favorably about Jews (ibid., p. 143). Moreover, are virtues fixed across space and time or up for debate? In the writings of Russian-American novelist and philosopher Ayn Rand, “selfishness” was recast as a virtue in her book The Virtue of Selfishness (1965).

116 

K. RICHARDSON

Classical ethics is heterogeneous, and partial, often reflecting the outlook of a dominant group at a particular time, engaging in sex, class, and racial discrimination. This is not to say that there haven’t been ethical challenges to dominant paradigms at different times. Socrates stands out as he was in favor of education for women, a radical idea in ancient Greece when women were considered mentally inferior or property. So too the work of Karl Marx and Friedrich Engels (1998), in contrast to Nietzsche, argued “the masses” were a force of liberation for humanity. A systematic and radical challenge to classical, contemporary, and continental ethics was to come from feminists, who protested at the exclusion of women (Wollstonecraft 1988; McLaughlin 2000; Ruether 1974) and challenged systems of domination and sexual violence (Dworkin and MacKinnon 1988), which ignored the interrelatedness of human beings (Gilligan 1982; Stawarska 2009).8 Misogynistic philosophy produces an ethics of exclusion, denigration, and elitism. By feminist ethics, I refer to critical reflection of the way ethics includes, excludes, or denudes women in narratives about human existence. Despite the urgent need to transform ethics by incorporating critical feminist approaches, feminist ethics is still seen as a subcategory of ethics, rather than a challenge to mainstream patriarchal ethics. Peter Singer had this to say about feminist ethics: Carol Gilligan … argues that much of what men have written about ethics has presented only a partial picture, because it has taken the masculine view of ethics as the only possible one. (1994, p. 20)

Feminist ethics opens up the possibility of transforming ethics into a philosophy of human liberation rather than a confirmation of the status quo because it developed in response to alterity and emerges from otherness, and a claim for recognition (Wollstonecraft 1988; Dworkin and MacKinnon 1988; Dworkin 1981, 1997; MacKinnon 2017). Feminist ethics of humanity begins and ends with the other always looking back, feeling, thinking, and experiencing, not as a lifeless object, a dehumanized other, or an It in the Buber sense. If these principles are not established a priori, contemporary engagements with ethics cannot take us beyond the status quo. Or worse, ethical models used to make sense of robots and AI will compound the problems of humanity by legitimizing new kinds of hierarchies, even celebrating commodity fetishism (Marx 1976; Gunkel 2014; Richardson 2015). The numerical increase in “ethics experts” of robots and AI is situated in a commercial and research culture that is made

6  THE BUSINESS OF ETHICS, ROBOTICS, AND ARTIFICIAL INTELLIGENCE 

117

up mainly of white educated males (in Europe and North America at least), and tied to political philosophies of libertarianism—an outlook that is inspired by corporate individualism, restriction of government, and support for the commercial sex trade. Ethics will be used, as it has been, instrumentally, in the service of maintaining a hierarchy, rather than one committed to human interdependence and economic and political equality.

Why Ethics Now? Ethical debates are accompanied by political and democratic shifts regarding new technologies and their impacts on human beings, the environment and natural life. For technologies such as Genetically Modified Organism (GMOs), ethical arguments have, sometimes, constrained the development of these technologies in business and academia. The European Union, for example, severely regulated and restricted the use of GMOs in the 1990s in response to concerns by environmental groups about the effects of GMOs on the environment.9 This led to the use of the “precautionary principle,” a principle used to advise precaution with new technologies (Beck 1992) if the effects of those technologies are not fully understood. In the US and countries in Africa, GMOs are legal and permitted to use. Businesses and scientists with interests in the field have challenged the EU arguing that regulation constrains innovation and economic development. Robots and AI technologies require new ethical standards and forms of regulation. Robots as automated machines will transform working practices. The introduction of tools and machines and robots have traditionally replaced human labor and allowed for efficiency in the production of goods or services. The introduction of increased automation in the workplace is raising concerns about unemployment. There are also other concerns about how work will change: will automation mean that work becomes even more tedious, merely pushing buttons or responding to the needs of a machine? Marx wrote about this tedium as the “labour of Sisyphus” (Marx [1867] 1976). Since the early twentieth century, the depersonalization of the human in factory life has been captured by artists including Chaplin (193610), Čapek (192011) and Lang (192712). Fordism describes a production system based on Henry Ford’s factory production line method of automobile manufacturing. Workers would stand at either side of a long conveyor belt, with each task broken into ever more miniscule parts. Each  worker’s time on each activity would be

118 

K. RICHARDSON

­ easured by time-and-motion studies, popularized as the practice of “scim entific management” referred to as Taylorism (Taylor 1911). Taylorism and Fordism radically altered working practices leading initially to increased tedium of work as work was broken up into discrete repetitive acts in large-scale industrial settings. Over the course of the twentieth century there were shifts in economic structures leading to decentralized and increasingly specialized units of production (Amin 1994). In post-­ Fordism, “flexibility” was a key term (Vallas 1999) indicated by the growth of information technologies, the decline of traditional working-class allegiances and unionized membership, the shift to outsourcing, and the “feminization” of the workforce. The effects of post-Fordism became visible from the 1970s in the US and the UK, significantly changing the working landscape as it was organized for over a century (Sennett 1998). Robots (automated machines) are projected to carry out particular functions in the production process, the first robot arm—Unimate—was used in 1961 in a Ford factory. But robots are now entering into domains that bring them into close proximity with humans—primarily in areas that concern interpersonal interactions, care, intimate relationships, and sex. AI presents different kinds of issues in a society that is increasingly “online,” with products and personal artifacts that were once embodied in physical entities (atoms) changed to digital data (bits) (Negroponte 1996). Digital corporations designed technical systems allowing users to store and access their personal data. AI are complex programs that run on computer systems typified by machine learning (learning algorithms that draw on their own reserves of information) and deep learning (learning algorithms that are produced as input-out metrics, sometimes shaped by what is called a black box—a set of metrics that are generated by including or weighting different values in a program). Many would argue the AI described above is different from traditional computer programs that primarily carried out specific kinds of actions. It is the technological transference of data from physical artifacts to digital data producing vast amounts of Big Data. Current AI (which many argue is distinct from AI as it was originally envisioned) is a response to the increasing move to online interactions (for social interaction as well as working practices), and the data generated from this. The shift from face-­ to-­face social interaction to machine-mediated interaction transformed the web as increasing numbers of people signed up to corporate services such as Facebook, Google, and Twitter. Personal data are collected and

6  THE BUSINESS OF ETHICS, ROBOTICS, AND ARTIFICIAL INTELLIGENCE 

119

monetized by companies in different ways leading to concerns around privacy and data protection. The European Union recently passed the General Data Protection Regulation (GDPR13), an attempt to give back power to citizens from corporations. Perhaps AI should be renamed Advertising Intelligence. With the loss of traditional media and political institutions to control the flow of knowledge via established media organizations, new forms of knowledge have emerged, and new power brokers that challenge state institutional power of broadcast media or government. While in general this is seen as a positive development, the election of Donald Trump in the US in 2016 had led to concerns that electoral manipulation shaped the election result. Under his leadership, American news can be dismissed as “fake” if it is disagreed with; it is no longer necessary to challenge arguments on substantive grounds—this charge of “fake news” is sufficient. Last but not least, there are moves to sell animatronic dolls or AI-inspired avatars as substitute friends, companions, or sexual others. Robots and AI programs are envisioned as companions for the elderly, caregivers to the sick or disabled, and “sexual”14 partners to alienated males. Machines dressed up in humanlike forms are promoted as intimate or emotional substitutes. But why are machines taking on intimate roles? With a population of over 7.5 billion in the world there is no shortage of humans, but there appears to be a crisis of intimacy and a redefining on the ontological differences between humans and machines. A libertarian or techno-utopian ethics promotes human relationships with machines as beneficial, but should we all embrace machines as loving partners without critically examining power and inequality of sex and class? Finally, there is the prediction that robotics machines and AI algorithms will become “conscious.” No one can quite agree on what constitutes consciousness. Consciousness as political awareness has given way to a new kind of corporate product—a fantasy of a sentient product—a super-­ product that is superintelligent. It is a vision that has inspired a generation of academics who act as the cheerleaders for this corporate philosophy writ large. There are many reasons why ethics should be central the social and cultural effects of robots and AI on humans. It is necessary to ask ethical questions about the rightness or wrongness, the morality or immorality, and the individual versus the social good. This is especially so when dealing with complex issues about new artifacts that potentially will create new regimes of capital affecting work and personal life. Such questions are

120 

K. RICHARDSON

necessary and appropriate. However, the limits of ethics will be revealed in how such questions are answered if a priori commitment to the recognition of human interdependence, mutuality, and nonviolence is not established. Classical ethical theories do not commit to the value of human mutuality but are built around abstract principles of reason or justice, or “the greatest happiness of the greatest number” (Bentham 1890). Here we have to ask the substantive questions that give rise to the political-economy, without which the debates about the ethics of automation and work turn into issues that have no meaningful capacity to bring about change for human beings, instead favoring the status quo.

Why Robots and AI and Ethics? In this section, I drill down further into demand for an ethics of robots and AI. I propose this inquiry can be divided into two parts: 1. These are artifacts (robots and AI programs) that act in the world and perform actions that produce effects without their producers knowing always the actions or effects (through the use of Big Data using machine learning or deep learning algorithms). 2. Robots and AI will become so advanced they will be indistinguishable from humans. The division between a human and a machine will become meaningless. Robots and AI are developed by business and academic research groups. Robots and AI algorithms are potentially beneficial because increased automation may reduce the burden of unpleasant or tedious work on the one hand; but it could also increase unemployment and decrease taxable revenues of nation states on the other (Frey and Osborne 2013). Is it really possible to have an ethical discussion about the pros and cons of robots, AI, and work without addressing capitalism as the economic system? If capitalist economics is ignored by ethicists, so too are the real possibilities of making sense of how robots and AI will impact on society. What about “assistive robots”? Are they different from factory robots and, if so, why? There are moves among some manufacturers and academics to transform the meaning of a robotic product from an appliance to a “living being” (Breazeal 2004). Robots and AI are a subset of commercial products; but we are persuaded into believing they are something different, more akin to humans and other sentient beings than typical ­commercial

6  THE BUSINESS OF ETHICS, ROBOTICS, AND ARTIFICIAL INTELLIGENCE 

121

appliances. Fiction acts as a backdrop to this fantasy that nonliving machines can come alive! But fiction and the science of robotics and AI are not the same, yet they are presented as interchangeable (Richardson 2015). Reframing of ontological boundaries of what it means to be human coincides with a shift in the ubiquitous domination of corporate ­capitalism—TINA (There is No Alternative). Humans as machines and machines as humans become increasingly characterized as analogical and interchangeable with each other. In the Pygmalion myth, the sculptor produces the likeness of a woman out of stone, but he does so in a culture where women are first and foremost characterized as sexual property of men (Innes 1955). The stone in which Pygmalion carves is property, so too are the women in this culture. Women and the inanimate stone are at some level interchangeable in patriarchy. Patriarchy is a system of ownership and property (be it a woman or stone or robot) and it gives its owners and beneficiaries (wealthy males) power and domination. From fictional tales in Metamorphoses to contemporary technocapitalism the stone is substituted with the machine. The asymmetrical humanism inherent in patriarchy is recast in new forms—new fictions that machines will become sentient and conscious and so are in need of rights (Gunkel 2014). Mainstream ethics of robots and AI is infected by patriarchal values of asymmetrical humanism. While the ethics and business community muses about robot rights and AI slaves, corporations sell products that record and collect data on the lives of ordinary people. Robots and AI appliances are recast as alive and as “friends,” not as corporate products. Robotic scientists frequently put faces onto machines that are cute with the deliberate intention of producing feelings of social connectedness between humans and machines (Richardson 2015). Why are companies and research scientists keen to promote connection between humans and machines? Should we be concerned about this as a society? We do not police the imagination and so are not concerned when people use their imaginations and regard some entity (be it art, music, literature, etc.) as something it is not. In the Tom and Jerry cartoons we do not put limits on what a mouse can do and argue for “realistic” portrayals of cats or mice because we are concerned that the viewer (often a young child) might believe animated characters are the same as living beings. But what of a company who produces robots for the elderly, and markets this robot as a “companion”—is this false-advertising? What about those social scientists who are concerned about violence toward robots as a sign of humanity’s failing?

122 

K. RICHARDSON

Corporate Anthropomorphism Techno-utopian capitalists are absorbed by patriarchal ethics that promotes a metaphysical project of matter coming to life—corporate anthropomorphism. This is the logical end-point of a culture that privileges the few over the many, and no longer even pretends to uphold of the myth of humanism as a project of human equality. The abandonment of the humanist project is popularized through transhumanism (Bostrom 2005). For Bostrom, the Enlightenment corporate liberal humanist subject must be free to continue to its logical conclusion by allowing unfettered commodification of the human. Alienation is the name of the game, and commercial products are recast in human forms: as companions, sentient beings, even slaves. Another project of abandoning the Human is Haraway (1991), who rejected the mythic humanist subject as an autonomous and self-contained entity—instead proclaiming the cyborg as a political entity from the biomedical and military-industrial complex within which it arose. Humanism, like ethics, is not a homogenous enterprise either. There are humanists who are rooted in a human communitarian vision of society, as expressed through the work of Carol Gilligan, Karl Marx, and David Hume. There are also humanists rooted in an egocentric and individualistic model of the human, as expressed through the works of Rene Descartes (1968) and Immanuel Kant. Other ethical systems, such as Aristotelian ethics, talk about the importance of human relationships, but only value relationships between adult male citizens, ignoring and deriding slaves and women. Children are also derided, but some of the male children will grow up and become citizens later on, their experience is meaningful, but temporal. Moreover, in philosophy, dialogical phenomenologists (Buber 1937) and feminists (Gilligan 1982; Stawarska 2009), postcolonial scholars (Spivak 1993; Kapoor 2004; Taussig 1993) have rejected the egocentric first person perspective paradigm that created a dehumanized or incomplete “Other” that shaped and continues to shape ethical narratives of egocentric abstract philosophy. Spivak challenged the Western paradigm of rights-based cultures versus subaltern responsibility arguing for the importance of learning from the subaltern (1993). The individual ego, “the I,” reigns supreme, but this time it is cemented in a political economic culture through an intensely individualistic version of capitalism; even products we buy privilege the ego—for example, the iPhone. This new abstract philosophy has given rise to a call for disintegrating d ­ ifferences

6  THE BUSINESS OF ETHICS, ROBOTICS, AND ARTIFICIAL INTELLIGENCE 

123

between humans and machines and the Other has been reframed as the oppressed robot or AI system (Gunkel 2014) even borrowing the language from the history of slavery, women’s oppression, and colonialism to advocate for the “rights of machines” and an end to “robot slavery.” Facebook’s diagrammatic model of connectivity, which it features on its website to display the meaning of the “social network,” shows separate (individual) nodes connected by relationships (computer networks) to other autonomous nodes. This accurately represents a computer network, but can it be mapped onto humans using computers in a network? The individualistic and egocentric paradigm that privileges mind over body and the ego in favor of I-you interrelationship shapes the underlying metaphysical project of techno-utopians. For utopian technologists, their own egocentric creations that bring them enormous wealth, prestige and influence, far greater and more numerically verifiable than human relationships could ever, are a means for their ultimate fulfillment. Rather than carry on with the political goal of human liberation and an end to repressive hierarchies, intellectuals, academics, and businesses redefined the project of human destiny around corporate anthropomorphism: the celebration of alienation and commodification. Ethics deployed in this manner then may turn out to be a new form of legitimizing control over humans, dressed up as “caring” responses in the new world order.

Why We Need a Feminist Ethics of Robots and AI As outlined, there are many serious issues about the development of new technologies, only a few of which I have outlined in this chapter. Feminist ethics has been marginalized as a subfield of ethics—an option rather than the main course—with the classical male doctrines of ethics dominating (deontology, utilitarian or virtue to name three dominant modes). What I want to suggest is a feminist ethics is not an option but a transformative ethics that attends to political, economic, and social exclusion. Feminist ethics is grounded in I-you interrelatedness that is primordial, but is also aware of power imbalances of age, sex, class, and race. The feminist ethics focus on interrelationships and intimacy is necessary to confront ethical practices that celebrate alienation, detachment, exploitation, and corporate anthropomorphism.

124 

K. RICHARDSON

Conclusion In this chapter I have tried to show how ethics is about the rightness or wrongness of acts, and these questions are valuable because humans are spontaneously ethical beings. However, in doing so, it is necessary to add caution and draw attention to the problematic egocentric humanism and antihumanism found in traditional and contemporary ethical narratives of robots and AI. An ethics narrative that does not start from I-you interrelatedness will only end up as a bureaucratic tick-box exercise of regulation. Ethicists of robots and AI will end up endorsing the privileges of the powerful, complicit in rubber stamping the status quo. Unfortunately, this already occurs and ethicists are invited onto research projects because they are known to align with the values of the corporation or public body. Moreover, on many European Union-funded projects of technology, the “ethics component” is often given less economic and temporal resources and less status comparatively to other work packages (the term used to describe the components of work). When fear of losing one’s job or position is threatened by asking difficult questions, then the questions do not get asked or pursued. Ethics could be a means to ensure our technologies develop for the benefit of humanity, living beings, and the environment rather than  for corporations, products, and profits, but this depends on a philosophy and political practice that does not privilege the few over the many.

Notes 1. “ACM Code of Ethics and Professional Conduct” (published July 2018), https://www.acm.org/code-of-ethics. Accessed 27.7.28. 2. IEEE “Ethically Aligned Designed: A Vision Prioritizing Human Well-­ Being with Autonomous and Intelligent Systems,” https://standards.ieee. org/develop/indconn/ec/autonomous_systems.html. Accessed 27.7.18. 3. Future of Life  Institute, https://futureoflife.org/,  – funding received from billionaire Elon Musk among others. 4. Campaign to Stop Killer Robots, https://www.stopkillerrobots.org/ 5. Campaign Against Sex Robots, https://campaignagainstsexrobots.org/ 6. Development of Robot-Enhanced Therapy for Children with Autism Spectrum Disorders, https://www.dream2020.eu/ 7. Responsible Ethical Learning in Robotics (REELER), http://reeler.eu/ 8. This is not an exhaustive list.

6  THE BUSINESS OF ETHICS, ROBOTICS, AND ARTIFICIAL INTELLIGENCE 

125

9. GMO Legislation, EU, https://ec.europa.eu/food/plant/gmo/legislation_en. Accessed 27.7.18. 10. Modern Times (1936). Dir. Charles Chaplin. United Artists. 11. R.U.R. (Rossumovi univerzální roboti) (1920) written by Karel Čapek. 12. Metropolis (1927). Dir. Frizt Lang. 13. EU General Data Protection Regulation, https://www.eugdpr.org/ 14. Though sex includes an other, advocates of “sex robots” say that sex does not have to include an other, and masturbation is synonymous with sexual intercourse.

References Amin, A. 1994. “Post-Fordism: Models, Fantasies and Phantoms of Transition.” In Post-Fordism: A Reader, ed. A. Amin. Oxford: Blackwell. Aristotle. 1962. The Politics, ed. T.A. Sinclair. Baltimore: Penguin. Beck, Ulrich. 1992. Risk Society: Towards a New Modernity. London: Sage. Bentham, Jeremy. 1890. Utilitarianism. London: Progressive Publishing Company. Bostrom, Nick. 2005. A History of Transhumanity Thought. https://nickbostrom. com/papers/history.pdf. Accessed 17 July 2018. Breazeal, Cynthia L. 2004. Designing Sociable Robots. Cambridge, MA/London: MIT Press. Brinton, Crane. 1940. “The National Socialists’ Use of Nietzsche.” Journal of the History of Ideas 1: 131–150. Buber, Martin. 1937. I and Thou. Trans. Ronald Gregor Smith. London: T. & T. Clark. Descartes, René. 1968. Discourse on Method: And the Meditations. Harmondsworth: Penguin. Dworkin, Andrea. 1981. Pornography: Men Possessing Women. New York: Putnam. ———. 1997. Intercourse (Free Press paperbacks). New York: Simon & Schuster. Dworkin, A., and C. MacKinnon. 1988. Pornography and Civil Rights: A New Day for Womens’ Equality. New York: Organizing Against Pornography. Frey, C.B., and M.A. Osborne. 2013. The Future of Employment: How Susceptible Are Jobs to Computerisation. Oxford: University of Oxford. Gilligan, Carol. 1982. In a Different Voice: Psychological Theory and Women’s Development. London/Cambridge, MA: Harvard University Press. Gunkel, David J. 2014. “A Vindication of the Rights of Machines.” Philosophy & Technology 27 (1): 113–132. Haraway, Donna J. 1991. Simians, Cyborgs and Women: The Reinvention of Nature. London: Free Association Books. Hume, David. 1940 [1793]. A Treatise of Human Nature: Being an Attempt to Introduce the Experimental Method of Reasoning into Moral Subjects. Vol. 1. London: John Noon.

126 

K. RICHARDSON

Innes, Mary. 1955. The Metamorphoses of Ovid (Penguin classics). Harmondsworth/ New York: Penguin Books. Kant, Immanuel. 1978. Anthropology from a Pragmatic Point of View. Carbondale: Southern Illinois University Press. Kapoor, Ilan. 2004. “Hyper-Self-Reflexive Development? Spivak on Representing the Third World ‘Other’.” Third World Quarterly 25 (4): 627–647. MacKinnon, Catharine A. 2017. Butterfly Politics. Cambridge, MA: The Belknap Press of Harvard University Press. Marx, Karl. 1976 [1867]. Capital. Vol. 1. New York: Modern Library. Marx, Karl, and Friedrich Engels. 1998. The Communist Manifesto: A Modern Edition. Introduction by E. Hobsbawm. London: Verso. McLaughlin, Eleanor. 2000. “Equality of Souls, Inequality of Sexes: Woman in Medieval Theology.” In Ethics: Classical Western Texts in Feminist and Multicultural Perspectives, ed. James P.  Sterba, 137–144. New  York: Oxford University Press. Negroponte, Nicholas. 1996. Being Digital (1st Vintage Books ed.). New York: Vintage Books. Nietzsche, Friedrich W. 1989. Beyond Good and Evil (Rev. ed.). New  York: Prometheus Books. Rand, Ayn. 1965. Virtue of Selfishness. New York: Penguin Group US. Richardson, Kathleen. 2015. An Anthropology of Robots and AI: Annihilation Anxiety and Machines. New York: Routledge. Ruether, Rosemary R. 1974. Religion and Sexism: Images of Woman in the Jewish and Christian Traditions. New York: Simon and Schuster. Sennett, Richard. 1998. The Corrosion of Character: The Personal Consequences of Work in the New Capitalism. New York: Norton. Singer, Peter. 1994. Ethics. Oxford: Oxford University Press. Spivak, Gayatri Chakravorty. 1993. “Can the Subaltern Speak?” In Colonial Discourse and Post-Colonial Theory. A Reader, ed. Patrick William and Laura Chrisman, 66–111. Hemel Hempstead: Harvester. Stawarska, Beata. 2009. Between You and I: Dialogical Phenomenology, Series in Continental Thought. Vol. 36. Athens: Ohio University Press. Taussig, Michael T. 1993. Mimesis and Alterity: A Particular History of the Senses. New York: Routledge. Taylor, Frederick. 1911. The Principles of Scientific Management. New  York: Harper & Brothers. Vallas, Steven P. 1999. “Rethinking Post-Fordism: The Meaning of Workplace Flexibility.” Sociological Theory 17 (1): 68–101. Wollstonecraft, Mary. 1988. A Vindication of the Rights of Woman: An Authoritative Text, Backgrounds, the Wollstonecraft Debate, Criticism, ed. C.H.  Poston, 2nd ed. London/New York: Norton.

CHAPTER 7

Fiction Meets Science: Ex Machina, Artificial Intelligence, and the Robotics Industry Teresa Heffernan

“Many thinkers have broad interpretations of consciousness for humanitarian reasons, Aaronson tells Popular Science. After all, if the giant game of Mind in that field (or C-3PO, or Data, or HAL 9000) simulates a thought or a feeling, who are we to say that consciousness is less valid than our own?” Scott Aaronson is a theoretical computer scientist, but what does “that field” have to do with the fictional characters from Star Wars, Star Trek, or 2001: A Space Odyssey? As a literary scholar, I have often been struck by the many references to fiction in discussions about artificial intelligence (AI) and robotics, but how is fiction mobilized in this field? Why, for instance, are fictional robots so frequently collapsed with the robotics industry? And how do science and fiction differently imagine robots and artificial intelligence? Even as the field references fiction there is very little transdisciplinary discussion between scholars who study fiction and computer scientists and roboticists who cite it, so this chapter aims to encourage this dialogue, to challenge the claims that science fiction is continuous with science fact and to restore the gap between science and fiction.

T. Heffernan (*) Department of English, St. Mary’s University, Halifax, NS, Canada e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_7

127

128 

T. HEFFERNAN

Twenty-first-century headlines have been full of claims about fiction coming true: “The Craziest Sci-Fi Fantasies That Got Closer to Reality This Year” announces Wired; “8 Sci-Fi Predictions That Are Coming True” reads Huffpost; “31 Science-Fiction Things That Actually Exist Now” claims BuzzFeed, and many of these stories are about robots and AI. It is also not uncommon for science museums to design their exhibits on the subject by drawing parallels between fiction and the industry. Roboworld, installed at the Carnegie Science Center in 2009, uses a timeline that corresponds to real-life robots. This permanent exhibit announces to visitors: “The first robots were creations of imagination rather than engineering. The replica robots lining our ‘walk of fame’ are a tribute to the fictional machines that helped spark the imaginations of those who created real robots that followed.” The 2017 exhibit at the Science Museum in London—“Robots: The 500-Year Quest to Make Machines Human”—also mixed fictional robots with industry robots and its title played on a popular fictional trope with a long history: artificial people wanting human status. Robot engineers, designers, and industry heads often cite literature and film as a source of inspiration: “My dream Star Trek computer is becoming a reality, and it is far better than what I ever imagined,” announced Amit Singhal, a former software engineer and vice-president at Google (Bergen 2016); and in a similar vein, Jeff Bezos said at the 2016 Code Conference: “It has been a dream from the early days of sci-fi to have a computer to talk to, and that’s coming true” (Jefferson 2016). So too, Cynthia Breazeal, the director of the Personal Robots Group at the MIT Media Laboratory, often credits Star Wars as inspiration for her military-funded robots: “In many ways those droids were full-fledged characters; they cared about people. That was what, I think, really sparked my imagination” (Greenfield 2014). An interview about Breazeal’s latest creation Jibo, which is being launched as the first “family robot,” opens with the comment: What has made droids like BB-8, R2D2 and C-3PO so popular wasn’t just the fact that they were robots that could do cools things, but that they seemed to have real emotions and would react to their human and alien counterparts in a variety of ways. While emotional robots have been a thing of science fiction for decades, we are now finally getting to a point where these kinds of social robots will enter our households. (Martin 2017)

But why are fictional robots collapsed with real robots in a way that no other fictional characters have been? Beatrix Potter’s flopsy bunnies, for

7  FICTION MEETS SCIENCE: EX MACHINA, ARTIFICIAL INTELLIGENCE… 

129

instance, are never conflated with real rabbits or Disney’s the “lion king” with a real lion. Fiction is full of references to strange things from talking space ships to friendly dragons. The robots in Star Wars are no less fictional characters yet industry robots read through the lens of fiction are often presented as interchangeable and continuous with their fictional counterparts. This conflation poses certain problems. For instance, an advertisement for Jibo includes clips of R2D2 from Star Wars, the nameless robot from Lost in Space, Johnny Five from Short Circuit, Rosie the robot from the animated series The Jetsons, and WALL-E from the postapocalyptic 2008 film of the same name. “We have dreamt of him for years and now he is finally here,” the narrator tells us, as if fiction participates in a technological teleology that necessarily ends in the materialization of real humanoid robots. Like the use of other references to fiction in the marketing of this technology, these clips animate this new invention. Strip Jibo of this fictional lineage that is used to sell the device and it looks like a desk lamp with a black mirror—it has a three axis motor with an operating platform, equipped with stereo cameras, motion sensors, speakers, and a touch screen. If the advertisement for Jibo uses fictional references to transform a plastic shell with wires and chips into “one of the family,” it also overlooks allegory, simile, metaphor, metonymy, and the social and political context of the fictional sources it references. Jibo is presented as an example of fiction coming true as it sells the fantasy of humans being catered to by a friendly mechanical servant. The inserted film images, however, tell a different story. The happy family depicted in the advertisement shares the dream of the bourgeois consumers of The Jetsons, an animated TV show set in 2062 that began in the early sixties. The show featured George Jetson who worked an hour a day and the homemaker, Joan Jetson, who lived to shop for clothes and new gadgets, while Rosie the robot did the housework and the family drove about in flying cars. This framing of Jibo reminds us of that leisure carrot that is always dangled before us in the marketing of “future” technologies: the persistent promise that robots, for those that can afford them, will be our helpers and even companions and will make our lives easier, releasing us from tedious jobs so we have more free time. Ironically, the advertisement for this technology from the future evokes a sense of nostalgia for the fantasy of the all-American family—healthy, smiling, blonde, white children living in domestic bliss in a detached house with a garage and car on a tree-lined street in the suburbs with women

130 

T. HEFFERNAN

happily baking in the kitchen and large family dinners. Yet Jibo is being launched in a much more precarious America of housing foreclosures, job precarity, and a disappearing middle class. Moreover, the leisure dream of the Jetsons has, decades later, turned sour. Not only are people working longer hours in our “always connected world,” but dwindling resources and overflowing toxic landfills make Jibo closer to the anxious world of WALL-E of the last clip, where obese humans, enslaved by technology and disconnected from one another, float around in space after trashing the earth with their rampant consumerism. So what is the “he” we have “dreamt” of that is announced in the advertisement? The line of fictional robots from Rosie to Wall-E suggests more of a nightmare. While science makes claims about fiction coming true it is also selective about which fiction it cites and is wary of dystopic versions: “Fear is not likely to stop scientific research, but it is likely that various fears will slow it down. Mechanisms for distinguishing real dangers from speculation and hype, fueled by science fiction, are needed” the preface to Robot Ethics argues. Similarly a Guardian headline reads: “Brave New World? Sci-Fi Fears ‘Hold Back Progress of AI’, Warns Expert.” Yet while science works with a model of fiction as either coming “true” or fueling “hype,” by definition, fiction is fiction and resists this literal approach. So how do scientists and literary critics differently imagine robots and artificial intelligence? In the fields of robotics and AI, robots are often compared to humans and humans to robots. A quote from the Good AI website, a research institute in Prague, proposes, for instance, that “Our future artificial intelligences will perceive stimuli in the same way that a human does – by seeing, feeling, interacting, and learning  – and use this data to generate behavior, perform tasks, and respond to motivations given by human mentors.” A quick scan of an issue of the MIT Technology Review exposes the prevalence of these founding tropes. Ashutosh Saxena, the Director of the RoboBrain project argues: “we shouldn’t expect a robot to learn on its own from scratch, any more than we’d expect a human to do so—imagine a child growing up with no access to textbooks, libraries, or the Internet … a robot, just like a person needs to be able to have all the knowledge it needs in one place” (2016, 15). In another article about robots teaching each other, Stefanie Tellex, a computer science professor at Brown University refers to childhood as “a big data collection process.” Yet another article asks whether AARON and the Painting Fool, computer programs that generate “machine art,” are creative, imaginative, and

7  FICTION MEETS SCIENCE: EX MACHINA, ARTIFICIAL INTELLIGENCE… 

131

responsive to emotions. The images produced by this software are compared to those by artists from Rubens to Van Gogh. AlphaGO, an AI program designed by Deep Mind Industries (which was bought by Google), beat the South Korean grandmaster at the ancient Chinese game of GO. The program uses “deep learning”—the layering of artificial neural networks very roughly and selectively modeled on much more complex living cells, biological neurons. Yet claims are made that the program “learns like humans—by seeing the world, consuming data, and learning patterns and rules” (Lee 2016). Geoffrey Hinton, the lead scientist on the Google Brain project, said the program won because it relies on “a lot of intuition” (Lee 2016). In a discussion about the use of this term, I asked one of Hinton’s former students (now a professor of computer science) what Hinton meant by “intuition” in the context of machine learning; in other words how could a machine bypass reason and act based on feelings. He responded in an email string that Hinton believes that “any kind of mental ‘activity’ or experience—whether it’s intuition or thinking or reasoning or feeling, etc.—ultimately corresponds to a set of neural activities … so if everything corresponds to neural activities, then ‘intuition,’ too, is much more of an electrical/computational process than we think of it as being.” Hinton believes that once machines can adequately model the brain by recognizing patterns and learning from their mistakes, thinking and “consciousness” will be revealed to be no more than a question of mechanics and that computer programs that simulate neurons will expose the brain as no more than a computer. It is not the success of these programs that is in doubt—from self-­ driving cars to killer robots to voice translation to legal and medical software, the impact of “deep learning” programming is widespread. But what motivates the use of metaphors that conjure up humanlike qualities: childhood, intuition, feelings, creativity, and thinking? Why are we invited to think of a robot as “just like a person”? Why is creativity reduced to algorithms and rules? Why is childhood compared to “data collection”? Why is the brain described solely in terms of networks, circuits, and pattern recognizers? Why are the differences between an electrical/computational process performed by a computer and a biological brain in a body glossed over? How did a computer model of inputs and outputs that require huge datasets, massive amounts of storage and computing power (so very different from how human intelligence works) come to be understood as humanlike? In this field, rather than the computer serving as a metaphor for the brain, the brain has come to serve as a metaphor for the

132 

T. HEFFERNAN

computer. A circular logic is operating in these analogies. While poets, philosophers, and scientists have wrestled for centuries with the question of what it means to be conscious or human, the debate has been commandeered and life/consciousness is defined in terms of algorithms and data processing—and then the inventions that operate in this fashion are declared to be humanlike. In other words the starting premise predetermines the conclusion and we are asked to accept the illogical claim that there is no difference between a computing machine and a human. Nadine, for instance, is a humanoid robot modeled on Nadia Thalmann, a professor at Nanyang Technological University in Singapore. Thalmann describes Nadine as “like a real companion that is always with you and conscious of what is happening” (Dockrill 2016). She claims that Nadine has already long passed the Turing test, a test that was devised by Alan Turing in 1950 to predict a machine’s ability to exhibit intelligent behavior that is indistinguishable from a human, a model itself that has been the subject of much critique (see, for instance, Lucy Suchman’s chapter in this volume). While Thalmann may be on the extreme side in her claims that Nadine is “conscious,” the industry of social robots is moving in the direction of collapsing humans with machines. Sharpe, for instance, markets its Robohon phone, a small humanoid robot, as a device that will “know you better than anyone.” The sex industry markets robots as companions while David Levy is hoping to see “love” and marriage with robots by 2050. Others in the industry look forward to robots as “carers” for the elderly and “teachers” for children. David Hanson described his humanoid robot Sophia as “basically alive” on The Tonight Show in 2017, and it was granted citizenship by Saudi Arabia the same year. The company’s website describes Hanson’s philosophy: Dr. Hanson believes that three distinctively human traits must be integrated into the artificial intelligence of these genius machines: Creativity, empathy, and compassion. As an extension of human intelligence, Hanson Robotics’ genius machines can evolve to solve world problems too complex for humans to solve themselves. Sophia personifies this bold and responsible goal.

Neither a “genius machine” nor creative, empathetic, or compassionate, Sophia is a chatbot that, when it is working, uses face tracking and robotic movements: it can be preloaded with text, it can run a dialogue system

7  FICTION MEETS SCIENCE: EX MACHINA, ARTIFICIAL INTELLIGENCE… 

133

that responds with prewritten scripts, and it can gather facts from the Internet. In other words, Sophia works on a bunch of separate and ­single-­tasked algorithms that have been linked together and involve pattern recognition: an image recognition algorithm that can detect a specific person’s face; that then triggers another algorithm that offers a selection of appropriate phrases; a transcription algorithm that turns the person’s response into a text that is then matched to a string of possible responses. Most of the open source code that runs Sophia is available on Github. Not surprisingly, Hanson, the CEO of the company, has an undergraduate degree in fine arts and a graduate degree in interactive arts and engineering and worked at Disney. Sophia is an elaborate PR stunt, an animatronic puppet with humans running its Twitter account. It is not going to solve world problems nor is the marketing of this robot in anyway “responsible” or ethical. In a recent article entitled: “Should We Love Robots?—The Most Liked Qualities of Companion Dogs and How They Can Be Implemented in Social Robots,” the researchers set out to understand why humans like dogs better than robots and discovered that the interviewees liked dogs better because of their “personality,” “empathy,” “emotions,” and “souls.” The researchers then argue that robot designers can simulate these traits in machines and no one will be the wiser: [T]he majority of the most liked qualities and reported advantages of dogs (e.g. love, attachment, faithfulness, and kindness) were related to attachment, companionship or the emotional support owners get from their dogs. The behavioral descriptions participants gave for describing attachment (and related features like faithfulness) were very similar to the scientific description of attachment (proximity-seeking, separation stress, and the preference for the attachment figure). According to these results, robots or artificial agents should be furnished with an attachment system: they should recognize and prefer the user, try to maintain proximity to the user, show some stress behavior when they are separated from them and greet the user happily when he/she returns … the “Turing test” for social robots is to “pass” the attachment test. (Konok et al. 2018, 138)

Why do we want humans to “love”—perhaps the better term is to be addicted to—robots other than to open up new markets? What is at stake in this equivalence that assumes a short-term commodity, built in a factory, sold on the market, and controlled by corporations is the same as humans or other living creatures? In this Baudrillard-like nightmare, the

134 

T. HEFFERNAN

simulation proposes to completely absorb the complexity of the real in the name of profit and the commodification of all life, leaving us with his much prophesized “anorexic” future. The difference between scientific and fictional approaches to the question of what it means to be human is that fiction—necessarily as fiction— makes no claims to the “truth” of the human and leaves open the question. If discussions in the industry are preoccupied with whether or not a machine can manifest humanlike qualities or consciousness and casual references to Frankenstein are used to frame the debates about preventing this technology from turning monstrous, Mary Shelley’s novel itself calls into question the very binary of monster and man. Many popular adaptations of the novel rightly pick up on the blurring of Frankenstein with his monster, as in the novel Victor increasingly starts to mirror his creation and begins to describe himself in the same terms he describes the monster as a “reckless spectre” and an “evil spectre” (Shelley 1999, 194, 117). Collecting limbs from corpses, Dr. Frankenstein deciphers the secret of life and animates these disparate body parts in his laboratory. But just as his wonderful “man”—a “human being”—comes to life (81; 233), Victor inexplicably turns from his creation, describing it variously as “nonhuman,” a “wretch,” a “demoniacal corpse,” a “mummy,” “a spectre” (86, 88). Frankenstein gives no reason for rejecting his work—for why he switches from viewing his creation as a man to viewing it as a monster. Grief over his mother’s death drives him to want to bring the dead back to life. Yet when he succeeds in this very task, he does not rejoice in his creation but runs from his lab. It is not the failure of the experiment that haunts him but its success. The doctor then wants to kill off the monster in order to preserve “the whole human race” (209). But in rejecting his creature and in failing to recognize it as human, he necessarily returns full circle to his mother, to men born by women, and to death. Herein lies the paradox at the heart of his quest: the “secret” or “essence” of life is always in tension with humans as mortal, temporal, adaptable, and contingent. Man, thus, cannot be replicated or preserved but is reshaped and reinvented by the terms of his experiment. Hence Frankenstein begins to resemble his creation. In other words the novel asks what kind of anemic creature we produce in the very claim to “know” the human. The film Ex Machina (2015), a modern-day Frankenstein story, takes up the question of manufacturing humanoid machines with “consciousness.” Like Shelley’s monster, the robot in this film is produced in a lab that is sealed off from the messiness of its contemporary world. Outside

7  FICTION MEETS SCIENCE: EX MACHINA, ARTIFICIAL INTELLIGENCE… 

135

the world of women, birth, history, culture, and politics, the lab is removed from concerns about electronic waste, climate change, refugees, poverty, rising sea levels, polluted oceans, and disappearing species. Nathan, the alcoholic bully and billionaire CEO of Blue Book, a global Google-like search engine, builds a female cyborg and arranges for Caleb Smith, a young low-level employee who works as a coder, to join him in his high-­ tech research lab/house on his massive, pristine, natural estate, with its miles of forests, rivers, and mountains. Technology and the environment appear to seamlessly blend in this world as if they exist in complete harmony just as the human is blended with the machine. In the world of AI, words like the “cloud” encourage this naturalization of technology. In reality, of course, “the $247 billion cloud computing industry funnels everything through massive centralized data centers operated by giants like Amazon, Microsoft, and Google” that are not “cloud-like” at all (Hsu 2017). Nathan wants Caleb to help him conduct the same Turing test that Thalmann claims her robot has mastered. Predictably Caleb grows infatuated with the robot, Ava, and agrees to help it/her escape—“her” if Ava is read in the tradition of women rendered as objects trying to claim their “humanity” and “it” if Ava is read as a machine built by men. Both readings are possible. As employee and employer engage in complicated games, where each assumes he is outmaneuvering the other, in the end the robot triumphs: Nathan is killed and Caleb is imprisoned in the lab. After seven days of the test that mimics the biblical seven days of creation and positions AI as a new religion, Ava/Eve puts on the skin and long hair of an earlier sex model, dresses in white, catches the helicopter that has been sent to pick up Caleb, and flies away from the garden/lab and out into the world. There is a long history of creation myths and stories about animating life, which dates back to ancient cultures. If we read the film in this context, as a reflexive comment on the creation myths generated by contemporary sexual politics, it exposes the fantasies of men in the notoriously sexist tech industry in the first decades of the porn-fueled twenty-first century. The misogyny and racism are palpable. Nathan builds Kyoko—a silent, submissive, scantily clad maid and sex toy to serve him, playing into all the racist stereotypes about compliant Asian woman. He also designs the most recent version of Ava. White, cute, innocent, with big eyes and a round childlike face, she is flirtatious and coy rather than provocatively sexual and designed to appeal to the sensitive “nice” tech nerd Caleb.

136 

T. HEFFERNAN

In the tradition of Ovid’s Pygmalion, the Cypriot sculptor who was contemptuous of “real” women and so built a statue that resembled one and fell in love with it, wishing it into life, the film plays with the masculine desire to create life. Denigrating women and co-opting this power, Nathan gives “birth” to a fantasy woman that exists solely for his pleasure. The prevalent Hollywood tradition of presenting the woman as the object of male desire, imprisoned and circumscribed by this gaze, is carried to its logical extreme in this film as the woman is literally rendered as an object: “some alarm clock” Nathan says to Caleb referring to Kyoko after she has been sent to wake the new recruit. Nathan also builds the robot Ava, telling Caleb she thinks of her builder as a “dad.” But the cache of files documenting his violent sexual acts with earlier models tellingly expose Nathan’s rape and incest fantasies (Japan is already selling childlike sex robots catering to this market and the Roxxxy TrueCompanion has a setting for Frigid Farah). Ava is equally subsumed by the male gaze—Caleb watches her as she speaks, sleeps, dresses and undresses for him and Caleb is in turned watched by Nathan. But Nathan’s desire to control life is accompanied by the equal fear of the return of a repressed female power that escapes his control: by the end of the film, the female robots jointly conspire to turn on their creator. Rebelling against the limited scripts written for them by their “father,” they stab Nathan to death while Ava abandons the screaming Caleb in the lab. However, Ex Machina is also a commentary on contemporary technology—about two men and the gendered machines they knowingly and unknowingly build that end up locking them into a world they thought they controlled. The film opens with a question about a field that is dominated by the interests and worldview of largely privileged men and the ways in which “universal” algorithms—that claim to replicate the human— more often reproduce the biases of the programmers. Algorithms construct worlds based on the data from images that engineers—like Nathan—feed into the machines, so they are more likely to reproduce than disrupt the social and cultural hierarchies of the day that determine such things as the parameters of female “beauty.” For instance, the android Erica is marketed by its maker, Hiroshi Ishiguro, as “the most beautiful robot” in the world. On a visit to his lab, I asked one of the assistants how its “beauty” was determined and whether the very white-skinned robot was based on some “universal” code. The response was vague. Ishiguro says of Erica: “I used images of 30 beautiful women, mixed up their features and used the average for each to design the nose, eyes, and so on …

7  FICTION MEETS SCIENCE: EX MACHINA, ARTIFICIAL INTELLIGENCE… 

137

That means she should appeal to everyone” (McCurry 2015). Which photos of women did Ishiguro find “beautiful”? Which “noses” and which “eyes”? Which race? What skin color? Of the roughly 3.52 billion women in the world, which 30 photos were selected to produce this “universal” look that would “appeal to everyone”? If another man or a woman or a child had chosen the pictures that were then averaged, would not the result be entirely different? Even if we ignore the questions of whether the unique beauty of a human can be captured by a machine or whether beauty lies not in an “averaging” but in uniqueness, algorithms cannot bypass history, culture, and politics even if they are presented as “neutral” formulas. Nathan refers to Ava as “a rat in a maze”: the robot is given a single goal—to escape its compound. It accomplishes this by successfully using Caleb, flirting with him, feeding his ego and inviting rescue. Caleb wants to talk to Nathan about how Ava works—about the programs, algorithms, and abstractions that produce Ava, about whether they are stochastic (nondeterministic), for example. But Nathan interrupts him to ask him how he “feels” about Ava and how Ava “feels” about him. While Ava simulates emotions and intimacy while working in a purely logical manner, calculating each step, to reach the programmed goal to get him to help it escape; Nathan discourages Caleb from thinking, wanting him to address only his emotions. Lonely, isolated, and orphaned at 15, Caleb loses his parents in a car accident, which he witnesses from the back seat. He learns coding while he is recovering and grieving in the hospital for the year that follows this traumatic event. Nathan plays on Caleb’s vulnerability and encourages Caleb to further succumb to the seduction of technology. The earlier prototypes and Kyoko make it clear that Ava could have appeared with humanlike skin from the start, but instead it appears with a mesh body that exposes its circuitry. The shiny metallic computerlike body and skull combined with a humanlike face and sexual parts—with breasts and “functioning” genitals that are modestly clothed—are designed to attract the alienated tech geek. Initially Caleb thinks he has won a contest to visit the CEO, and he is immediately inundated with affirmative emails congratulating him. However, in the mountain scene between Caleb and Nathan, Nathan then tells Caleb he was not lucky but “chosen” because he, like his employer, is especially gifted with intelligence. Convinced and pumped by his newly privileged and elite status, as Nathan has sex with Kyoko, Caleb imagines sexual intimacy with Ava. Yet, despite Nathan’s invitation on Caleb’s

138 

T. HEFFERNAN

arrival to forget the “whole employee employer thing,” as if power and wealth disparities could be simply erased, Caleb finds himself very much the pawn of Nathan. In the web of games and lies, Caleb discovers that he was manipulated and is neither lucky nor chosen. Caleb was selected based on his “search engine inputs”—he has no family, no girlfriend, and Ava’s look matches his pornography search profile. This everyman is locked in a windowless claustrophobic research facility and made to sign a “standard” nondisclosure agreement. The agreement predictably protects Nathan’s corporate privacy while Caleb is forced to surrender his—a ubiquitous practice with social media sites. Caleb signs only to discover that he, as Nathan tells him, is “the real test.” The tech nerd has been reduced to the very data he must generate and sign over. In one scene Caleb cuts his arm and bleeds to make sure he is not a robot though at this point the distinction between the man and the machine has already started to collapse—not because the robot has become humanlike but because the human has been reduced to a machine. Like Ava, Caleb is newly produced and shaped by algorithms; his identity outside these codes has dissolved. The end of the film finds him pounding on the windows of the secure soundproof compound formerly inhabited by his robot love, who has, in turn, escaped. Caleb’s protests echo the previous scene where one of Nathan’s earlier models also tries to bash through the glass that imprisons it. In the end, the real monster that the lab has engineered is Caleb. The data-mined human is reduced to components—selected by a computer search engine—that come to define him. It is not the human that births the computer but the corporate computer that births the human. In the course of the film, Nathan comes back to what he and Caleb refer to as the “chess” problem: is the robot working in a closed programmed loop or does it know it is playing chess? In other words are Ava’s emotions mere simulations or is it conscious and capable of displaying a real emotion? Nathan provides a third option: that Ava is pretending to like Caleb because it sees him as a means to an end. The burgeoning industry of affective computing and social robots is invested in playing this same game of chess. Capitalizing on human loneliness—already aggravated by the very technology that promises to alleviate it—this business profits from humans’ embrace of these affective machines as they mine data for profit that can be used to further imprison their users. Like Ava, who is set to be replaced by the “next” generation like the models before her, these humanoid machines are short-term commodities built to manipulate. From complicated contracts to “cute” and “sexy” technology

7  FICTION MEETS SCIENCE: EX MACHINA, ARTIFICIAL INTELLIGENCE… 

139

to the serial “upgrading” of computers and cell phones to the promise of infinitely replaceable parts and always better models and brighter futures, we can all recognize how we are being seduced, like Caleb, by an industry that is interested in always distracting us from how it works and the conditions of its production. Like techno-utopians who look forward to the merging of machines and man as the next step in human evolution, Nathan proposes that “one day the AI’s are going to look back on us the same way we look at fossil skeletons on the plains of Africa. An upright ape living in dust with crude language and tools, all set for extinction.” Yet this linear, teleological model has much more in common with religious narratives than with messy reality. Humans may indeed join the other species headed for extinction, but it will have nothing to do with the logic of evolution and “Lucy” fossils. As we enter into what has been called the sixth extinction, caused by humans who are contributing to the rapid eradication of the biodiversity of the planet on which they depend, machines efficiently building and generating more things for profit are unlikely to be our best option for the future. As the “future” inevitably dates, we might also want to consider what we leave behind. Instead of rendering fiction literal and making it serve the robotics/AI industry, which shuts down its critical force, I want to suggest that fiction offers useful ways of thinking critically about technological futures by de-­ familiarizing our present. If sectors of the robotics/AI industry are busy trying convince us that fiction is coming true and that humans and machines are no different in a reductive logic that abstracts and monopolizes emotions, creativity, intelligence for profit, literary theory thrusts humans back into the messiness of narrative and the unresolved question of what it means to be human. Early on in Shelley’s novel, Frankenstein reflects on the productive tension he has with his love Elizabeth: “The world was to me a secret, which I desired to discover; to her it was a vacancy, which she sought to people with imaginations of her own.” Our world of late—with its fascination with technological realism—is out of balance. We have lost the productive tension between science and fiction, between discovery and imagination. We ignore at our peril—the reminder from Frankenstein—that the world we discover is also, in part, the world we create.

140 

T. HEFFERNAN

References Bergen, Mark. 2016. “Google Search King Amit Singhal Retiring, Artificial Intelligence Research Head Taking Over.” Recode, February 3. https://www. recode.net/2016/2/3/11587548/google-search-king-amit-singhal-retiringartificial-intelligence. Accessed 1 July 2018. Dockrill, Peter. 2016. “Watch: Meet Nadine, The ‘Emotionally Intelligent’ Companion Robot.” Science Alert, March 8. https://www.sciencealert.com/ watch-meet-nadine-the-emotionally-intelligent-robot-designed-to-be-the-perfect-social-companion. Accessed 1 July 2018. Ex Machina. 2015. Directed by Alex Garland. Universal Studios. Greenfield, Rebecca. 2014. “How ‘Star Wars’ Influenced Jibo, The First Robot for Families.” Fast Company, July 21. https://www.fastcompany.com/3033167/ how-star-wars-influenced-jibo-the-first-robot-for-families. Accessed 1 July 2018. Hsu, Jeremy. 2017. “It’s Time to Think Beyond Cloud Computing.” Wired, August 23. https://www.wired.com/story/its-time-to-think-beyond-cloudcomputing. Accessed 1 July 2018. Jefferson, Graham. 2016. “Amazon’s Bezos: A.I.’s Impact Is ‘Gigantic’.” USA Today, June 1. https://www.usatoday.com/story/tech/2016/05/31/amazon-founder-s-impact-gigantic/85200740. Accessed 1 July 2018. Konok, Veronika, Beáta Korcsok, Ádám Miklósi, and Márta Gácsi. 2018. “Should We Love Robots?—The Most Liked Qualities of Companion Dogs and How They Can Be Implemented in Social Robots.” Computers in Human Behavior 80 (March): 132–142. Lee, Adrian. 2016. “The Meaning of AlphaGo, The AI Program That Beat a Go Champ.” Macleans, March 18. https://www.macleans.ca/society/science/ the-meaning-of-alphago-the-ai-program-that-beat-a-go-champ. Accessed 1 July 2018. Martin, Dylan. 2017. “Why This Boston Startup’s Family Robot Could Be a Game Changer.” BostInno, May 18. https://www.americaninno.com/boston/ai-inboston/interview-with-jibo-founder-cynthia-breazeal-on-social-robots-ai. Accessed 1 July 2018. McCurry, Justin. 2015. “Erica, the ‘Most Beautiful and Intelligent’ Android, Leads Japan’s Robot Revolution.” The Guardian, December 31. https://www.theguardian.com/technology/2015/dec/31/erica-the-most-beautiful-and-intelligent-android-ever-leads-japans-robot-revolution. Accessed 1 July 2018. Saxena, Ashutosh. 2016. “Wikipedia for Robots.” MIT Technology Review 119 (2): 15. Shelley, Mary Wollstonecraft. 1999. Frankenstein, or the Modern Prometheus, ed. D.L. Macdonald and Kathleen Scherf, 2nd ed. Peterborough: Broadview.

CHAPTER 8

Rossum’s Mimesis Jennifer Keating and Illah Nourbakhsh

But would you call the painter a creator and maker? Certainly not. Yet if he is not the maker, what is he in relation to the bed? I think, he said, that we may fairly designate him as the imitator of that which the others make. Good, I said: then you call him who is third in descent from nature an imitator? Certainly, he said. And the tragic poet is an imitator, and, therefore, like all other imitators, he is thrice removed from the King and from the truth? That appears to be so. Then about the imitator we are agreed. And what about the painter? I would like to know whether he may be thought to imitate that which originally exists in nature, or only the reactions of artists? The latter. (Plato 2005, The Republic Book 10)

J. Keating (*) Dietrich College, Carnegie Mellon University, Pittsburgh, PA, USA e-mail: [email protected] I. Nourbakhsh The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_8

141

142 

J. KEATING AND I. NOURBAKHSH

Robotics has a history steeped in opposing motivational poles. For the cognitive neuroscientist, the biologist, and the theorist, robotic innovation is a tool for understanding how humans work. Consciousness, autonomy, agency: all of these mysteries might be unlocked through ­ incremental modeling, experimentation, and refinement. In this worldview a robot system is a human simulacrum, from form to intellect. The most perfect mimesis will unearth basic truths about humans that we can best discover through making or representing facets of ourselves. This pathway of inquiry demands knowledge that is declarative. We must build robots that are deterministic, guided by an undergirding of known mathematics; and we must be able to describe exactly how they work, when all is done. Perfection in mimesis is measured by how thoroughly the robot acts as a looking glass into how humans function, from basic molecular composition all the way to high-level cognition. Opposite this view, the motivation of robotic engineers rests on functional results: can robots replicate the skills of humans? Can robots win the FIFA World Cup? Can they replace human categories of labor or optimize on particular features of a needed labor force? In the form of the Turing test, can artificial intelligence (AI) become behaviorally indistinguishable from natural intelligence in humans? From this viewpoint, the robot is a metaphysical demonstration of human ingenuity. The most perfect robot will demonstrate superhuman capability that bests its own creator. In contrast to scientific mimesis, this inquiry requires only procedural knowledge. We must build evolving, learning, self-improving artificial beings that achieve levels of performance ultimately beyond our limits. Perfection in human engineering resolves into posthumanism: creating intelligences that are our evolutionary successors. These competing views in robotics suggest underlying values in roboticists that are in tension: mimesis standing apart from the prospective metaphysical utopia of poetics. The true friction of robotics occupies the space in between these extremes, and we can see the turmoil of this in-­ between space in the work of an artist who narrates a robot future, Karel Č apek, at the turn of the twentieth century; and a pioneering AI futurist/ scientist, Raymond Kurzweil, at the turn of the twenty-first century. In these cases we are introduced to a science-fiction tragedy and the existential target of singularity—dystopian and utopian visions—that often characterize technologists’ public portrayals of our shared future in the contemporary moment. As we engage with the edge of rapidly advancing technology that has deeply influenced our sociological, political, and

8  ROSSUM’S MIMESIS 

143

c­ultural landscape in the last decade, what connections do we need to explore in the arts’ influence on advancing technology? How might mining science fiction’s influence on the present generation of technologists offer insights on how to characterize and analyze our present moment as we prepare for a near future where humans are influenced and implicated by socially engaging with the very machines that we design and build? The language that technologists presently use to describe their work is a significant encapsulation of the tension between the arts and technology; this is embodied by the way in which fictional tropes are often used in engineered systems that are fielded. In Keywords for Today, MacCabe and Yanacek document the etymology of “technology” as a term that was used from C17 to describe a systematic study of the arts (cf. ART) or the terminology of a particular art. It is from fw tekhnologia, GK, and technologia, mod. L—a systematic treatment. The root is tekhne, GK—an art or craft. In eC18 a characteristic definition of technology was “a description of arts, especially the Mechanical” (1706). It was mainly in mC19 that technology became fully specialized to the “practical arts”; this is also the period of technologist. The newly specialized sense of science and scientist opened the way to a familiar modern distinction between knowledge (science) and its practical application (technology), within the selected field. (MacCabe and Yanacek 2018)

In the etymology of the term “technology,” we are given insights on the precedence of mimetic arts. The Greek term tekhne indicates the manner in which the craftsmanship of the arts seeks to create or recreate the likeness of the lived experience through building, crafting, and composing pieces that are recognizable to an audience or user. Even in artistic forms, realism serves as an aesthetic exploration for individuals to organize human experiences, navigating and developing systems in order to understand the features of a complicated world. Echoes of Platonic notions of mimesis, in the “imitator” rather than the “maker” or “creator,” are evident in this early iteration of the word, even if one takes issue with the narrow suggestions presented in Book Ten of Plato’s The Republic. Contemporaneous with the rise of the industrial revolution in the mid-nineteenth century, we see that the “practical application” of specialized “science” characterizes technology as a term. This differentiates the word’s meaning significantly from that of its temporally and practically distant Greek tekhne origins. The rise of the Enlightenment period, its cult of reason driving ­technological

144 

J. KEATING AND I. NOURBAKHSH

advancement emblematic of this age, results in the historical shift in economy, culture, and politics associated with the industrial revolution. It is in the advent of the steam engine, used to transport traded goods from Africa, Asia, and the West Indies throughout the United Kingdom and the United States, the widespread use of the printing press to link literate audiences the world over, and the machination of manufacturing in industries like textiles that the labor forces throughout Europe are shaped day in and day out by rapidly changing technologies (Anderson 1983). This shift gives rise to early mass consumption, a driving economical force that will continue to work hand-in-hand with the evolution and drive of technological advancement up to our present moment. The vestigial “arts” in the etymology of “technology” is a striking relic in the context of today’s rift between the arts and technology. From Snow’s “Two Cultures,” indicating the seeming chasm between the arts and technology in the late 1950s at Cambridge University to the Obama administration’s record investments of governmental funds in STEM (science, technology, engineering, math) education for underrepresented populations in the technological industries and prestigious research academies of the United States, we see the manner in which the sciences and technology are esteemed and privileged forces in today’s economic, sociological, and political contexts. Yet the rise of power in Silicon Valley, multinational technological companies and current media attention on robotics and developing artificial intelligence might be readily traced to a precedent in the arts as well. As science-fiction literature, theatrical production and film dominate current social and cultural fascination in regard to the influence of machines on human individuals and communities, what precedent might we decipher in the links between the arts and the sciences emblematic of “technology’s” etymology? To what degree are the efforts to replicate or exceed facets of human features in machines indicative of a significant or benign mimetic exercise or practice?

Rossum’s tekhne In robotics discourse, a cursory citation is Karl Č apek’s R.U.R. (Rossum’s Universal Robots). It is presented as the discipline’s origin story, as practitioners boast the coining of the term “robot” (from Czech robota, meaning “heavy labor”) (Kussi 1990, 33). The dystopian play emerges in 1920, a bell-weather warning that criticizes the rupture of European society in the aftermath of the technologically driven military assaults associated

8  ROSSUM’S MIMESIS 

145

with the First World War. Č apek’s satirical tragedy explores advancing technology through the lens of a dystopian view of humanity, where characters drive for facets of mimetic advancement that are unconsciously ­detrimental to key elements of human experience: namely sexual reproduction and dignity associated with labor. The drive of capital in the play, situated within the context of mass manufacturing, war mongering, and a virtual absence of a regulatory politic, suggests Č apek’s impatience with a European context rife with uncertainty and primed for political disaster (which arrived thereafter with the rise of fascism and the Second World War). Shifts in labor needs throughout Europe in the midst of the industrial revolution led to record numbers of individuals moving from rural communities to urban centers. Laborers worked in undesirable and oftentimes dangerous environments, no longer enslaved to the agricultural landscape through serfdom, but instead subject to the demands for labor in a capitalist, technologically driven production process. Enfranchisement throughout Europe was uneven at this time as individual countries moved through various political experiments with the rise of nationalism and mixed reactions to the prospect of universal rights. As machines are introduced to the urban work environment, Č apek’s play suggests dramatic changes in human-to-human interactions, as well as humans’ relationships with machines. Although the play attends to tragic depictions of such relationships, the significance of making in the play is not to be discounted as an insignificant detail. In Č apek’s play we are introduced to an engineering mind-set applied to a society in flux. As we know, the results are disastrous. Throughout the play Č apek draws attention to tensions between the Old Rossum’s and Young Rossum’s mimetic enterprises. According to Domin in the play’s introduction, the deductions are crucial: [DOMIN]. But old Rossum meant that literally. You see, he wanted to somehow scientifically dethrone God. He was a frightful materialist and did everything on that account. For him the question was just to prove that God was unnecessary. So he resolved to create a human being just like us, down to the last hair. (Č apek 2004, 7)

Unlike the engineer Young Rossum, Old Rossum seeks mimesis in the poetic capacity of “creating.” Unlike the arts as imitation, Old Rossum is personified as invoking a god-complex, singularly obsessed with recreating humanity. As Old Rossum seeks to “create a human being just like us,

146 

J. KEATING AND I. NOURBAKHSH

down to the last hair,” we learn that he strives for prospective making in the form of biological replication. His drive is to prove that his work is not “imitation” in the Platonic sense, but “creator” instead. This effort to craft a holistic alternative to human beings, complete with consciousness and intelligence, offers Č apek a trope of arrogance and a ready foothold for launching a scientific critique in the form of art. In contrast to Young Rossum, Old Rossum seeks to “prove that God was unnecessary.” If he can produce the lifelike clone, then perhaps little else is needed in an advancing society. Old Rossum’s tactic was the ultimate mimesis: to create so thorough a copy of the human form that every detail would be faithful to the original. In contrast, Young Rossum, who never discovered the secret of creating life, set out to optimize through engineering a humanlike labor force. He had no hesitation about diverging from the human form for the utilitarian motive of making a better machine that can be mass produced and mobilized for an industrial economy: [DOMIN] Young Rossum was of a new age, Miss Glory. The age of production following the age of discovery. When he took a look at human anatomy he saw immediately that it was too complex and that a good engineer could simplify it. So he undertook to redesign anatomy, experimenting with what would lend itself to omission or simplification. (Č apek 2004, 8)

In the hands of the R.U.R. Corporation, with Young Rossum at the helm, the secret of life became literally procedural. A recipe, on paper, for how to create more robots was all that remained of the internalized life-­giving knowledge of Old Rossum. By burning that recipe Helen destroys tangible knowledge, recorded only with ink and paper. Yet in the face of missing knowledge, the engineers of R.U.R. recognize their robots, not as mimetic equivalents of the human, but as explicit improvements that are authentically posthuman. They have been optimized for functionality, surpassing human capacities in these areas (Simon 1996): [DOMIN] My dear Miss Glory, Robots are not people. They are mechanically more perfect than we are, they have an astounding intellectual capacity, but they have no soul. Oh, Miss Glory, the creation of an engineer is technically more refined than the product of nature. (Č apek 2004, 9)

8  ROSSUM’S MIMESIS 

147

As he attends to the question of simulacrum, Č apek considers aesthetic, political, and social questions of his particular moment in time. The capacity for technologists to move from the “age of discovery” to the “age of production” is significant in Europe’s emergence from the First World War and the economic, political, and cultural upheaval associated with its aftermath. With its focus on the possibility to “discover” and “produce,” scientific innovation is funneled directly into application and the production of new systems that are specialized for specific tasks. These mobilize an integrated and developing economy that also facilitates governments controlling important aspects of growing urban workforce populations. In the play we hear echoes of the evolving meaning of “technology” in the aftermath of the Industrial Revolution and can recognize the manner in which Č apek’s play leverages a dystopian vision to explore these motifs in culture and politics (MacCabe and Yanacek 2018). Young Rossum’s explicit mimetic enterprise, developed to ensure scaled production and use of the robotic systems, is distinct from Old Rossum’s effort to create, to develop a system that replicates not only human capacities and abilities but also captures the possibility to create, express, and forge new connections or discoveries, which is inherent to our understanding of individuality and subjectivity in the West (see Seigel 2004). In the two systems developed respectively by the Rossums we see tension in the enterprise of imitation and optimization in the Young Rossum robots, and the biological “monsters” that Old Rossum develops (Č apek 2004, 8). They each fail in drastically different ways to achieve a utopian alternative or succession to fallible human beings.

Mimetics and Politics Č apek’s play explores anxieties pertaining to the reshaping of a world order, the rise of capital and, alongside it, nationalism. The robot trope serves as a satirical farce to illustrate the manner in which technological advancement in the interwar period indicates the potential demise of humankind, or at least the old world order preceding the catastrophic First World War, and therefore humankind as it was once known. In the exploration of the “mechanically more perfect” system, Č apek suggests the potential arrogance of the technologist who can mimic only chosen elements of that which exists in nature. His portrayal is not one to lay foundations of hope and optimistic vision. The systems that Young Rossum designs lead to the militaristic overtake of humankind by the tools that are

148 

J. KEATING AND I. NOURBAKHSH

designed to optimize specific features of human capabilities. The mechanized workforce, falling susceptible to a seemingly human will to power, becomes weaponized on a scaled level, leading to the demise of society within the context of the play. Robotic mechanization echoes the mechanized warfare, through mustard gas, airplanes, and early drones, that shift the capacity to catastrophically slaughter a generation of men in England and France in the First World War. Tools for war are developed in scaled manufacturing in this period. The predictive quality of Č apek’s work is significant here and certainly indicative of a sensitivity to existential threats, alongside political and social instability, associated with many individuals in this period in Europe. Aside from the introduction of humanoid robots, R.U.R. is a rather strange antecedent to contemporary portrayals by technologists. Yet Č apek’s dystopian play deeply influences technologists’ culture and vision of how their advancements will shape our contemporary world, encapsulating both the optimism of Young Rossum’s optimization trope and the seduction of creation suggested in the failed but intriguing examples of Old Rossum’s experiments. In Archaeologies of the Future, Fredric Jameson suggests: Utopia has always been a political issue, an unusual destiny for a literary form: yet just as the literary value of the form is subject to permanent doubt, so also its political status is structurally ambiguous. The fluctuations of its historical context do nothing to resolve this variability, which is also not a matter of taste or individual judgment. (Jameson 2007, xi)

In the context of Č apek’s material reality, the promise of political utopia was suggested in the prospect of nationalism (even in the examples of early failures throughout Western Europe and the United Kingdom in the aftermath of the war). In the years thereafter, the political possibility of prospective utopian political systems manifested in the fascist tendencies emerging in Italy, Spain, and Germany, and the communist system in Russia. These efforts to develop and maintain closed-system politics, often in the name of capitalism, betray discomfort with an uncertain economic and political order after the first wave of nineteenth-century industrialization. The coming political waves, however, driven by facets of technological advancement from mechanization and mass production of weaponry for war, demonstrate sinister elements of these closed systems that distinguish them fundamentally from the promise of a utopian vision.

8  ROSSUM’S MIMESIS 

149

Č apek’s dystopian performance offers audiences a fractured vision of the benefits and costs of a technologically driven narrative. These include the complexity of the mimetic efforts in the context of the play, but also in the creation of the play itself. His work suggests ready alternatives to the farcical story of Old and Young Rossum. As Č apek situates his deep critique of cultural and political trends in the context of his debut work as a playwright, his artistic enterprise offers an early link between the arts and emerging technological drive of the late nineteenth and twentieth century and the tech revolution that we are experiencing today.

Mimetic Arts: Shifting Genres In the “Epilogue” to Mimesis: The Representation of Reality in Western Literature, Erich Auerbach writes, The subject of this book, the interpretation of reality through literary representation or “imitation,” has occupied me for a long time. My original starting point was Plato’s discussion in book ten of the Republic  – mimesis ranking third after truth – in conjunction with Dante’s assertion that in the Comedia he presented true reality. As I studied the various methods of interpreting human events in the literature of Europe, I found my interests becoming more precise and focused … When Stendhal and Balzac took random individuals from daily life in their dependence on historical circumstances and made them the subjects of serious, problematic, and even tragic representation, they broke with the classical rule of distinct levels of style, for according to the rule, everyday practical reality could find a place in literature only within the frame of a low or intermediate kind of style, that is to say, as either grotesquely comic or pleasant, light, colorful and elegant entertainment. They thus completed a development which had long been in preparation … and they opened the way for modern realism, which has ever since developed in increasingly rich forms, in keeping with the constantly changing and expanding reality of modern life. (Auerbach 1953, 554)

As Č apek used the medium of theater to launch a political critique of the economic pressures and encroachment on human individuality by the drive of technological advancement in his science fiction play, his work demonstrates facets of the evolving concept of realism that Auerbach traces in Western literature in exile in Istanbul during the Second World War. The transition that he delineates, explicating the passage of Homer’s Illiad and Odyssey through to Virginia Woolf’s To the Lighthouse, following

150 

J. KEATING AND I. NOURBAKHSH

a through-thread on the rapidly expanding purview of literary realism to capture and explore “increasingly rich forms, in keeping with the constantly changing and expanding reality of modern life,” offers an interesting parallel to roboticist and computer scientists’ enterprises to develop sophisticated hardware and software systems better equipped to navigate the richness of Auberbach’s understanding of the “constantly changing and expanding reality of modern life” and the prospect of decision making in this context. Current state-of-the-art machine systems have not reached the levels of sophistication achieved in sister disciplines like the literary arts, demonstrated by masters like James Joyce, T. S. Eliot and Virginia Woolf. Yet, the parallel visions for human achievement are worthwhile considering as we entertain the notion that the field of robotics has been influenced, even if only in relatively superficial ways, by the imagined world captured in Č apek’s play. The political salience of Č apek’s work became more pronounced in his last play, “The White Plague” (Kussi 1990; see also Majer and Porter 1999). Yet the cultural relevance of his vision for technological advancement, and its role in the prospect of a dystopian future, is countered by several technological optimists who invoke the rise of technological advancement and its recent, relatively rapid sophistication as a potential answer to political, social, and economic disenfranchisement and injustice. The narratives associated with such tech-optimism in tension with tech-­dystopian visions offer, if not a rich discourse in the public venue, a persistent one nonetheless. A chief example of such positioning is offered in Raymond Kurzweil’s The Singularity Is Near (2007). The New York Times bestselling nonfiction text posits the prospect of immortality through computational means that is supposed to be achieved with coming generations of computational and robotic advancement. As Head of Technology at Google, Kurzweil’s conception of state-of-the-art technological innovation and his perception of its influence on society is nearly unparalleled. And yet his nonfiction text suggests that he is seduced by the prospect of writing science fiction instead, dangerously mixing the influence of the genre on his own nonfiction writing. In relation to science fiction’s utopian vision in dialogic exchange with political utopian visions, Jameson writes, It has often been observed that we need to distinguish between the Utopian form and the Utopian wish: between the written text or genre and something like a Utopian impulse detectable in daily life and its practices by a specialized hermeneutic or interpretive method. Why not add political

8  ROSSUM’S MIMESIS 

151

­ ractice to this list, inasmuch as whole social movements have tried to realp ize a Utopian vision, communities have been founded and revolutions waged in its name, and since, as we have just seen, the term itself is once again current in present-day discursive struggles? (Jameson 2007, 1)

Writing in the context of the turn of the twenty-first century’s political fragmentation, economic instability, and inequitable distribution of wealth, Kurzweil’s text betrays many of the wishes for stability, sustainability, and viability expressed in utopian form and wish. Yet he writes technical nonfiction rather than science fiction or political manifesto. The mixing of form and wish are potentially disastrous, especially when considered in the context of the technologists’ current position of political, cultural, and social power. His motives are quite apart from those of both Old and Young Rossum. Instead, he wishes to achieve posthuman immortality literally for himself. If one begins to tease out the tensions between the “Utopian wish” and the “Utopian form” for Kurzweil, it seems only fair to suggest that his text be reclassified as science fiction for parsing purposes. Yet it circulates in our current culture as a bestselling nonfiction text. What are the potential political implications for this genre mixing? If the term “utopia” is “once again in present-day discursive struggles” like those with which Kurzweil suggests he intends to engage, what parameters are needed to attend to such technological optimism matched with political positioning and wealth in our current moment?

Utopian Vision: A Singularity? Kurzweil’s means for achieving utopia is bound in the computational methodology of the technologist of our age; the computer scientist and roboticist. Unlike the fictional Young Rossum whose engineering ethos is that of the imitator, Kurtzweil’s character trope is likened more to the mimetic aspirations of the character, Old Rossum. His suggested pathway to achieving immortality is striking in its similar treatment of mimetic methodology paired with poetic utopian aspirations. He begins as a utilitarian engineer with one simple aim: to extend his life just enough to wait for the coming innovation of human immortality, treating death as none other than a disease to be cured through human progress: When I was forty, my biological age was around thirty-eight. Although I am now fifty-six, a comprehensive test of my biological aging conducted at Grossman’s longevity clinic measured my biological age at forty. … These

152 

J. KEATING AND I. NOURBAKHSH

results are not accidental; I have been very aggressive about reprogramming my biochemistry. I take 250 supplements (pills) a day and receive a ­half-­dozen intravenous therapies each week. As a result, the metabolic reactions in my body are completely different than they otherwise would be. (Kurzweil 2007, 211)

Here Kurzweil is hacking his human body, attempting to outwit the standard aging process by embracing technology to literally kick the can on death. While there is little biological evidence for this tactic—he is still mortal for the time-being—he invokes the prospect of outsmarting mortality through clinical application of experimental science. But for what future is he delaying death? Kurzweil’s plan begins with mimesis at the cellular level. He tells this story from a nanorobotics point of view. Imagine microscopic machines coursing through veins, delivering nutrients to every cell in the body. First, these robots make food and drink redundant, since they take care of the care and feeding of all cells: Ultimately we will be able to determine the precise nutrients necessary for the optimal health of each individual. These will be freely and inexpensively available, so we won’t need to bother with extracting nutrients from food at all. Nutrients will be introduced directly into the bloodstream by special metabolic nanobots, while sensors in our bloodstream and body, using wireless communication, will provide dynamic information on the nutrients needed at each point in time. This technology should be reasonably mature by the late 2020s. (Kurzweil 2007, 304)

Next, these nanobots obviate each of our major organs, one at a time. Kurzweil does away with the liver, the kidneys, the pancreas—incrementally describing how robots within us do the jobs formerly reserved for each of these organs, optimizing functions beyond current capabilities of biological systems. Eventually, little remains inside the human body cavity save nanorobots and blood. Mimesis has given way to a bottom-up reconstruction of human bodily functions in the form of engineered silicon, giving rise to a new category of humanity that is systematically superior to biological humans by virtue of technological enhancement. Echoes of Young Rossum abound. Whereas Kurzweil and Č apek use the prospect of mimetic practices as fictional tropes in the case of R.U.R., or prospective methodology in the case of Singularity, as the pathway toward newly imagined futures, their work awkwardly converges on a shared approach for how a drive for tech-

8  ROSSUM’S MIMESIS 

153

nological mimesis can be ascribed to technological innovation, even in the turn of the twenty-first century. Although centuries away from its Greek etymological antecedent in tekhne and in distinct genres of writing, Č apek and Kurzweil share a vision for how technological mimesis creates an age of extreme productivity and extreme leisure. In Kurzweil, we are introduced to a near future: As with therapeutic cloning, we would not be creating the entire animal but rather directly producing the desired animal parts or flesh. Essentially, all of the meat—billions of pounds of it—would be derived from a single animal. There are other benefits to this process besides ending hunger. By creating meat in this way, it becomes subject to the law of accelerating returns- the exponential improvements in price-performance of information-based technologies over time—and will thus become extremely inexpensive. (Kurzweil 2007, 224)

In contrast, we can look back on history to determine the efficacy of Č apek’s futuring effort: [DOMIN] But within the next ten years Rossum’s Universal Robots will produce so much wheat, so much cloth, so much everything that things will no longer have any value. Everyone will be able to take as much as he needs. There’ll be no more poverty. Yes, people will be out of work, but by then there’ll be no work left to be done. Everything will be done by living machines. People will do only what they enjoy. They will live only to perfect themselves. (Č apek 2004, 21)

As the vision continues to unfold, the echoes suggest the mixing of form and wish against which Jameson warns: The education of AIs will be much faster than that of unenhanced humans. The twenty-year time span required to provide a basic education to biological humans could be compressed into a matter of weeks or less. Also, because non-biological intelligence can share its patterns of learning and knowledge, only one AI has to master each particular skill. (Kurzweil 2007, 294) The “criticism from the rich-poor divide”: It’s likely that through these technologies the rich may obtain certain opportunities that the rest of humankind does not have access to. This, of course, would be nothing new, but I would point out that because of the ongoing exponential growth of price-performance, all of these technologies quickly become so inexpensive to become almost free. (Kurzweil 2007, 430)

154 

J. KEATING AND I. NOURBAKHSH

Free food, fast education, free technology, and leisure for humanity are each key features of Kurzweil’s vision. Technological mimesis of humanity itself becomes the disruptive catalyst that changes the logic of labor (see also Chude-Sokei 2016). Work becomes the province of artificial intelligence, whether as artificial serfs, nanorobots, or superhuman future designs. Goods approach zero marginal cost because the world’s efficiency increases without bounds, and the extreme inequities due to labor disappear, so long as we ignore the subaltern status of the engineered beings that become the new laboring class. Ironically, the predicted age of leisure also erases basic human activities that no longer have a functional purpose: Kurzweil deletes eating and drinking; Č apek removes human sexual reproduction and human labor. As the visions continue to collapse upon one-another the reflexive reaction to such a shift might be to rename the genre. Should Kurzweil at least suggest the text as a wild futuring thought exercise? Is he beholden to readers to explain that, no, this technology is definitively not state-of-the-­ art? When engaging with science fiction as a genre commonly attending to utopian visions, Jameson suggests, That very distance of culture from its social context which allows it to function as a critique and indictment of the latter also dooms its interventions to ineffectuality and relegates art and culture to a frivolous, trivialized space in which such intersections are neutralized in advance. This dialectic accounts even more persuasively for the ambivalences of the Utopian text as well: for the more surely a given Utopia asserts its radical difference from what currently is, to that very degree it becomes, not merely unrealizable but, what is worse, unimaginable. (Jameson 2007, xv)

As Kurzweil suggests, a utopian vision for a technological future that alleviates all suffering in the physical domain collapses the time needed for education and attends to confounding issues like worldwide access to food resources, to what degree does the nonfiction text suggest access to an “unrealizable” vision? If we more deeply scrutinize the chasms between current technological tools and near future iterations, the future that Kurzweil paints becomes one that is genuinely “unimaginable.” Kurzweil’s text depicts a lack of acknowledgment for the manner in which the arts have informed and given influence and shape to technological ambition for generations. Perhaps an engagement with science fiction on an analytical level to attend to the deep cultural and political influence on communities of current technologists could prove worthwhile. Rather

8  ROSSUM’S MIMESIS 

155

than placing an emphasis on precision, truth, and attention to the potentially egregious effects of polemic language in The Singularity Is Near, Kurzweil tellingly demonstrates the engineering trait we call satisficing (Simon 1996). His work suggests solutions that are awkwardly just good enough, rather than optimal. This is illustrated in his description of how human consciousness will eventually divorce entirely from the nanorobot-­ fed modified human body. He cannot resist cutting corners a bit as he writes, Once uploading [of a human mind] is feasible, we are likely to find that hybrid solutions are adequate. For example, we will probably find that it is sufficient to simulate certain basic support functions such as the signal processing of sensory data on a functional basis (by plugging in standard modules) and reserve the capture of subneuron details only for those regions that are truly responsible for individual personality and skills. (Kurzweil 2007, 199)

As the engineer overtakes the biological mimetic scientist, we can envision a merging of the Old and Young Rossums, a monster creator/imitator whose “unimaginable” world begins to echo the dystopian vision of Č apek’s play rather than the nonfiction speculation that might be in keeping with Kurzweil’s chosen genre. Why copy an entire consciousness when we can create a cheaper, mediocre facsimile by simply copying the bits that are most unique, and keeping standard versions for the rest of consciousness? This is true engineering mimicry—it is the art of technical compromise that is built into the practice of making, and Kurzweil cannot help but adopt the strategy even when considering a posthuman future with unbounded optimism and energy. Whereas Č apek’s utopian narrative ends with a (failed) age of leisure and plenty for humanity, Kurzweil pushes a technologic utopia to its ultimate, metaphysical conclusion. For him it is not enough to replace human bodily functions, nor even to upload a consciousness into human-­ engineered vessels. He imagines a transcendental utopia that values ­information and data above all else. He essentializes what is human as, literally, the digital data that is our past and future footprint. Kurzweil proposes that data is the repository of all humanity, extending data itself to mean not only numerics but emotion and morality as well:

156 

J. KEATING AND I. NOURBAKHSH

This book will argue, however, that within several decades information-­ based technologies will encompass all human knowledge and proficiency, ultimately including the pattern-recognition powers, problem-solving skills, and emotional and moral intelligence of the human brain itself. (Kurzweil 2007, 8)

Once all intelligence has been materialized with a physicality that computers can render, then the distinction between the digital realm and the physical world ceases to be significant. The metaphysical jump becomes set by marrying these worlds together into a single cosmic, hybrid union: The Singularity will represent the culmination of the merger of our biological thinking and existence with our technology, resulting in a world that is still human but that transcends our biological roots. There will be no distinction, post-Singularity, between human and machine or between physical and virtual reality. If you wonder what will remain unequivocally human in such a world, it’s simply this quality: ours is the species that inherently seeks to extend its physical and mental reach beyond current limitations. (Kurzweil 2007, 9)

Kurzweil is proposing a new, higher plane of reality in which machine and human intelligence have become unified. Since he argues, as do all Singularity advocates, that the speed of digital computing increases unfailingly, this also provides that the metaphysical intelligence just borne will become superhuman, eluding all known boundaries. Kurzweil’s language describes a pathway from mimesis of human function, to immortality, to group consciousness, to a deeply posthuman cosmic intelligence: Billions of nanobots will travel through the bloodstream in our bodies and brains. In our bodies, they will destroy pathogens, correct DNA errors, eliminate toxins, and perform many other tasks to enhance our physical well-being. As a result, we will be able to live indefinitely without aging. In our brains, massively distributed nanobots will interact with our biological neurons… This will provide full-immersion virtual reality incorporating all of the senses, as well as neurological correlates of our emotions, from within the nervous system. More important, this intimate connection between our biological thinking and the nonbiological intelligence we are creating will profoundly expand human intelligence… Intelligence on and around the Earth will continue to expand exponentially until we reach the limits of matter and energy to support intelligence computation. As we approach this

8  ROSSUM’S MIMESIS 

157

limit in our corner of the galaxy, the intelligence of our civilization will expand outward into the rest of the universe, quickly reaching the fastest speed possible. (Kurzweil 2007, 300) Ultimate, the entirely universe will become saturated with our intelligence. This is the destiny of the universe. We will determine our own fate rather than have it determined by the current “dumb” simple, machinelike forces that rule celestial mechanics. (Kurzweil 2007, 29) In any event the “dumb” matter and mechanisms of the universe will be transformed into exquisitely sublime forms of intelligence, which will constitute the sixth epoch in the evolution of patterns of information. This is the ultimate destiny of the Singularity and of the universe. (Kurzweil 2007, 21)

While old Rossum is on a mission to upend the need for God, Kurzweil is proposing that the destiny of the universe as a whole is nothing less than to become a vessel for a superhuman, cosmic intelligence. It would be easy to dismiss such a narrative as a purposeful exaggeration; but Kurzweil himself takes a metacognitive stance, quoting Stephen Jay Gould’s warning that science often dominates discourse with arrogance, only to explain that, in this case, Gould is wrong and we human  engineers really are responsible for the universe’s fate: Stephen Jay Gould said, “The most important scientific revolutions all include, as their only common feature, the dethronement of human arrogance from one pedestal after another of previous convictions about our centrality in the cosmos.” But it turns out that we are central, after all. Our ability to create models—virtual realities—in our brains, combined with our modest-looking thumbs, has been sufficient to usher in another form of evolution: technology. That development enabled the persistence of the accelerating pace that started with biological evolution. It will continue until the entire universe is at our fingertips. (Kurzweil 2007, 487)

Computers are trapped in the sandbox of computation, set apart from the physical reality of our world. Robotics breaks computing out of its digital prison, proposing that the work of human ingenuity can develop thinking systems that will subsume our material world, and even evolve into the kings of our physical reality. Both Č apek and Kurzweil elucidate narratives that lead to a robotic takeover, and both find their footholds in the power of technological mimesis chased by a posthuman vision of a superhuman

158 

J. KEATING AND I. NOURBAKHSH

machine consciousness. As we cautiously step into our near future, how might responsible dialogic exchange between the writers, artists, and technologists equip us to imagine and explore prospective futures that inhabit the space between the polemics of dystopian doom and future “unimaginable” utopias?

References Anderson, Benedict. 1983. Imagined Communities. New York: Verso. Auerbach, Erich. 1953. Mimesis: The Representation of Reality in Western Literature. Princeton: Princeton University Press. Č apek, Karel. 2004. R.U.R. (Rossum’s Universal Robots). London: Penguin Classics. Chude-Sokei, Louis. 2016. The Sound of Culture: Diaspora and Black Technopoetics. Middletown: Wesleyan University Press. Jameson, Fredric. 2007. Archaeologies of the Future: The Desire Called Utopia and Other Science Fictions. New York: Verso. Kurzweil, Ray. 2007. The Singularity Is Near. New York: Penguin. Kussi, Peter, ed. 1990. Toward the Radical Center: A Karel Capek Reader. North Haven: Catbird Press. MacCabe, Colin, and Holly Yanacek, eds. 2018. Keywords for Today: A 21st Century Vocabulary: The Keywords Project. New York: Oxford University Press. Majer, Peter and Cathy Porter, trans. 1999. Č apek: Four Plays. London: Methuen Publishing Ltd. Plato. 2005. The Republic. New York: Barnes & Noble Classics. Seigel, Jerrold. 2004. The Idea of the Self: Thought and Experience in Western Europe Since the Seventeenth Century. Cambridge: Cambridge University Press. Simon, Herbert. 1996. The Sciences of the Artificial. Cambridge: MIT Press.

CHAPTER 9

Race and Robotics Louis Chude-Sokei

In his now classic and soon to be reissued manifesto More Brilliant Than the Sun: Adventures in Sonic Fiction (1998), black British theorist, critic and putative Afrofuturist Kodwo Eshun brings together two singularly influential figures from erstwhile distinct intellectual traditions: African American activist and polymath W.E.B.  Du  Bois and American scientist and philosopher Norbert Weiner, father and founder of the science of cybernetics. Eshun binds these radically divergent thinkers tantalizingly, describing Weiner’s classic work The Human Use of Human Beings (1950) as simply Du  Bois’s epochal work of African American identity and American race relations, The Souls of Black Folk, “updated for the analog age” (Eshun 1998, 06[086]). Eshun explores the connection no further, yet his intervention—where two distinct forms of in-humanity are compared, one technological and one racial—helps me contend that there is much to be gleaned from studying technology via the literary, critical, and cultural resources of the African Diaspora. Such materials may or may not explicitly be focused on technology but in their focus on race, slavery, colonialism, and the global mechanisms of power and gender have much to offer as supplementary to conventional histories of technology, robotics and AI—not only in the L. Chude-Sokei (*) Boston University, Boston, MA, USA e-mail: [email protected] © The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2_9

159

160 

L. CHUDE-SOKEI

highly imaginative ways that have been made possible through Afrofuturism, or in theoretically intricate attempts to delineate race as a technology; and also, not simply in the often reductive modes of antiracism that see technology as a mere surrogate for whiteness or insist that blacks are or have always been mere subjects of its imperial power. There is simply much to be gained through a focused study of the material histories of race and technology, which are indeed what makes those imaginative extrapolations and unexpected conjunctions possible. In fact, the first thing that one gleans from such a study is that technology has never been absent, alien, or distinct from discourses of race, and vice versa. I will be suggesting that there has long been an intimacy between these spheres, a mutually constitutive one that feeds those uncanny sensations that empower the genre of science fiction (SF). That intimacy is precisely what enables Eshun’s suggestion that the experiences of African Americans (slaves and former slaves, famously marked by a specifically racialized form of “the uncanny” that Du Bois called “double consciousness”) bear some relationship to SF or that the struggle for black humanity against hostile and violent forces shares similar narrative traits with cybernetics. To those familiar with the foundational tropes of Afrofuturism, this parallel should be familiar in that it evokes critic Greg Tate’s now famous equivalence between slavery and science fiction. There is little doubt that it had some influence on Eshun’s work: One of the things I’ve been trying to say all along is that the condition of alienation that comes from being a black subject in American society parallels the kind of alienation that science fiction writers try to explore through various genre devices—transporting someone from the past into the future, thrusting someone into an alien culture, on another planet, where he has to confront alien ways of being. All of these devices reiterate the condition of being black in American culture. Black people live the estrangement that science fiction writers imagine. (Dery 1993, 211–212)

Critic Mark Dery would, also famously, name “Afrofuturism” not as the multifaceted political and artistic/intellectual movement it has now become but instead as a sprawling “psychogeography” of “African-­American concerns in the context of twentieth century technoculture—and more generally, African-American signification that appropriates images of technology and a prosthetically enhanced future” (Dery 1993, 180). As such the par-

9  RACE AND ROBOTICS 

161

allels of race and technology, cybernetics and slavery are deeply implied or assumed and run the gamut of African American, Caribbean, and black British cultures and subcultures, from the literary to the musical and the broader complex of comic books and paraliterature. Yet one of the weaknesses of Afrofuturism has been its tendency to dwell largely in epiphenomena. It revels in suggestions and parallels and is largely empowered by the imaginative and so the riches of material history become so neglected as to remain fantastical. From these above-cited passages one might suspect that my interest is in merely reading science fiction as an extrapolation of African American experiences—alongside critics like Mark Dery, the late Mark Fisher, and of course Greg Tate and Kodwo Eshun. Such an observation would be welcome but the way it orders knowledge is historically incorrect, as are Tate’s and Dery’s. Science fiction, I ultimately argue, as a central component of our histories of technology, is instead a direct extrapolation of the material experiences of blacks and other marginalized groups from and through slavery and colonialism. SF doesn’t enable race and slavery as much as race and slavery made SF possible. Eshun’s joining of Du Bois and Weiner is possible not simply due to the convenience of metaphor, but because the evolution of robotics and cybernetics explicitly borrowed from and were imaged through the rhetoric, assumptions, and social positions of blacks, for at least a hundred years before Hugo Gernsbach would even name the genre—a naming that must ever be remembered as having occurred during the Jim Crow era and alongside the Harlem Renaissance and the emergence of new black radicalism in art and protest. This paralleling of race and technology, black slaves and machines is in fact something Norbert Weiner himself was profoundly aware of, so much so that one can argue that cybernetics as it was first being articulated in the 1950s was haunted by race. The presence of racism and black protest is clearly on Weiner’s mind as he writes this epochal book, and his thinking about how humans use other human beings (or deny other human beings their humanity) bears the impact of those concerns. It’s not clear if Eshun knew this in More Brilliant Than the Sun, but The Human Use of Human Beings as well as Weiner’s various lectures and articles written in its wake made consistent references to African Americans and to slavery in order to mark the limits of robots, cybernetic beings (the term cyborg would of course be coined by scientists Manfred Clynes and Nathan Kline six years later), and artificial intelligence.

162 

L. CHUDE-SOKEI

The initial response to Weiner’s direct references to African Americans and to slaves and slavery is clearly, why? This wasn’t the hyperawareness of Isaac Asimov who had already been attempting to undo what had become an unbearable cliché: robots as metaphors for marginalized groups or, more broadly, for race or forms of social difference. Nor was this an explicit attempt to exploit that cliché, which had become naturalized by “golden age” SF but which can be traced further back into the nineteenth century and to the dawn of those colonial era texts that would eventually give rise to the genre itself. I will discuss those texts later but to emphasize the material bases of my observations, Weiner himself acknowledges his dependence on metaphor, or on “the analogies between living organisms and machines” as would Marinetti and the Futurists a few decades before (whose technological obsessions were also haunted by blacks and colonial Africa); and he is as concerned with the positive repercussions of new technologies on society as he is on the apocalyptic (Weiner 1950b, 48). But most important about Weiner’s references to slavery in a purely scientific context is not just their interruptive regularity but that they were deployed to generate a guiding ethics of his new science and its inevitable technological products. And this was an ethics very much rooted in an awareness of racism. Cybernetics as a science of communication and control was explicitly framed in terms of American racial politics, in terms of mastery and slavery. Weiner writes: Until white supremacy ceases to belong to the creed of a large part of the country it will be an ideal from which we fall short. Yet even this modified formless democracy is too anarchic for many of those who make efficiency their first ideal. These worshipers of efficiency would like to have each man move in a social orbit meted out to him from his childhood, and perform a function to which he is bound as the serf was bound to the clod. (Weiner 1950b, 50)

The rhetoric is of course familiar: modern life makes us all robots, an insight that precedes the word robot and is in fact Victorian in the wake of Thomas Carlyle. Yet that insight had become an Americanized cliché by the time of Weiner’s technological revolution, Americanized in that it is rhetorically inescapable from slavery, a system which Caribbean thinkers from C.L.R. James to Sylvia Wynter and Antonio Benitez Rojo argue presages the regimentation and subject formations of industrial capitalism—a system

9  RACE AND ROBOTICS 

163

that produces a modern subjectivity in the crucible of institutionalized unfreedom. In Weiner’s passage, white supremacy and race in this context easily gives way to a more general sense of social bondage as “each man” is reduced to servitude by “worshippers of efficiency.” This type of control is not only industrial but is modeled on the form of the assembly line, what Weiner and others in the wake of World War II will liken to fascism. Now, to be far more specific about the place of race in Weiner’s cybernetics, he writes: “Let us remember that the automatic machine, whatever we think of any feelings it may have or may not have, is the precise economic equivalent of slave labor” (Weiner 1950b, 152). Not only had the analogy of slave/machine become naturalized in America by the time of Weiner’s work, so was the very question of affect, of whether the machine “may have or may not have feelings.” In 1950 he would conclude his presentation to the American Academy of Arts and Sciences with these words: “Whether we say that these machines think or do not think, or even whether we say that they live or do not live is a quibble over words which are adequately defined for the normal everyday contingencies of life, but are not adequately defined for the greater problems which these new machines present” (Weiner 1950a, 4). These equivocations may seem odd in their suggestion that a machine could have feelings or could have life; at least we can agree that had to have seemed so in 1950. But given the analogies deployed by Wiener and the tradition of science fiction he is more than marginally aware of and will be so influential to, they reflect a growing concern for the ethics of a notion of otherness rooted in technology and that can be traced back to the historical ground of the analogy itself: slavery. Weiner would make this clearer in “Some Moral and Technical Consequences of Automation.” As he puts it in a section appropriately subtitled “Man and Slave”: “The problem,” of the use of “learning machines”—machines that can learn to read, play chess, evaluate complex situations and function in warfare—“is a moral problem … very close to one of the great problems of slavery” (Weiner 1960, 1357). Granted, the peculiar institution of chattel slavery was riven with questions and denials of morality as they circulated around the “humanness” of the African. It is a curious transposition or projection here, though. Why, after all, is there any question of morality if the machine is merely a machine? Does one feel any such ambivalence or uncanny hesitation around, say, a toaster? Unless of course one believes that machines are capable of being much more than they are:

164 

L. CHUDE-SOKEI

Let us grant that slavery is bad because it is cruel. It is, however, self-­ contradictory, and for a reason which is quite different. We wish a slave to be intelligent, to be able to assist us in the carrying out of our tasks. However, we also wish him to be subservient. Complete subservience and complete intelligence do not go together … Similarly, if the machines become more and more efficient and operate at a higher and higher psychological level, the catastrophe foreseen by Butler of the dominance of the machine comes nearer and nearer. (Weiner 1960, 1357)

That his deliberations include a reference to Samuel Butler and the trope of “the dominance of machines” is important to the historical framework I’m sketching here, where race and robotics, slavery and industrialization are coterminous and co-constitutive, and where references to one are steeped in the history of the other. What this tells us is that the notion of a potentially sentient, cognizant and affective machine draws attention to how cybernetics emerges out of an awareness of those questions of an African slave’s humanity that were central to late nineteenth-century knowledge and politics in the trans-Atlantic world. As African American cyberneticist and mathematician Ron Eglash puts it, cybernetics as a “science of computation and control systems is merely a thin disguise for methods of social domination and control” (Eglash 1995, 18). Weiner’s words should serve to remind him and us that cybernetics was no “thin disguise.” It was founded on the very awareness of the relationships of power, race, and politics and was in fact quite cognizant of the social history behind its metaphors and analogies in a way that cyberor technotheory and their discursive descendants haven’t always been. That race and racism operated to provide an ethics for the creation of artificial beings and for an imminently autonomous technology is striking, though not unusual. Slavery ultimately haunted cybernetics as it does science fiction: as an example of what must not be done with these new beings due to the moral crimes of what had already been done during chattel slavery to blacks. That Weiner would so ground his concerns about artificial life in Negro slavery was merely to acknowledge that both shared a social position as “free” labor but had also been similarly denied “souls” and intelligence and were, as Orlando Patterson would put it, socially dead; it was also to prophesy that just as blacks would evolve from the inhuman to the human through a political paradigm shift in conceptions of life and personhood, so inevitably would (or could) machines. Toni Morrison’s oft-quoted line that African Americans were the first ­modernists

9  RACE AND ROBOTICS 

165

could here be supplemented with the fact that blacks were the first robots, an observation that will now be fleshed out even further. To fully pursue the parallels toward even greater material resolution, between Du Bois and Weiner, race and technology, then, is to move beyond the vagueness of Eshun’s suggestion without losing the evocative power of Tate’s equation between black peoples and SF. These equations are resonant because they trouble the assumption that the histories of race and technology exist in quite distinct and unrelated spheres and so can be discussed or recalled without necessary recourse to each other; this despite the fact that these histories have long depended on each other for support, opposition, and material legitimacy. As Weiner certainly knew, race and Western technology had in fact been intertwined since, at least, the nineteenth century, where two of the most pressing concerns in the trans-­ Atlantic world were industrialism and slavery. Because the development of one depended on the other, it should be no great surprise to find each discourse encroaching on the other in the cultural responses of at least two centuries. These parallels are particularly meaningful when technology encroaches upon the sacrosanct category of “the human,” which though imagined as the antithesis of technology is indeed its primary foil, metonym, or cure. As Judith Butler puts it, “technology is a site of power in which the human is produced and reproduced” (Butler 2004, 11). Though many may still use “the human” to imply or invoke commonality or sameness, its racial history and gendered cultural usage suggests otherwise—sameness, at the risk of poststructuralist cliché, being ultimately dependent on difference and power. Race and “the human,” for example, is a long-standing opposition in Western racism and in black critiques of that racism. The latter is clearly a contested term and has been identified as such for some time now. It stages and presages what I’m arguing for here, a politics where technology is subject to race in the way that race has long been subject to technology and produced by science. Indeed, the tension between “race” and “the human” has become a leitmotif in the work of recent black critics due to the evolution of discourses of technological embodiment, particularly those for whom it alongside sexuality and desire demarcate structures of power and, as Judith Butler would have it, orders of livable life. Afrofuturism, for example. Despite common assumptions about organicity (lost in the wake of social constructivism and in the acknowledgment that even “the body” is relentlessly produced), “the human” links blacks

166 

L. CHUDE-SOKEI

to technology, just as industrialism is linked to slavery and the machine linked to the primitive. These are historically contemporaneous categories, suggestive of labor, power, and in some cases, are interchangeable, the one often used to replace the other or as a sign of the other. “The human” is necessary as that which technology putatively serves, supplements, or amplifies, but is rendered fragile to it also as we creatures of flesh out of lust, necessity, or a deep fidelity to science fiction reshape machines in the once-divine masquerade of anthropomorphic embodiment. This latter is, of course, where race and gender are unavoidable signifiers of variation but also ensnare technology in extant systems of power. After all, “the human”—as Jamaican theorist Sylvia Wynter excavated in advance of posthumanism—has from the dawn of Western modernity been code for “the white,” the “European” or “the West.” Instead of abandoning its universality now tainted by racism and imperial power, Wynter figures it as merely one genre of a malleable category—“man,” for example, which functions as its white, colonial overrepresentation and that uses “native” as its necessary opposition (one that may be epistemologically figured as external, geographically or culturally othered from “the human” but which is necessary to its meaning). From this we can extrapolate that like “woman,” robots or artificial intelligence are potentially other genres all of which emphasize how much the attribution and denial of life or consciousness are eminently political gestures. The history of black subjectivity is therefore the migration from genre to genre, in Wynter’s schema, or the history of how blacks evolved from things. In this context, which we can recognize as implicated in the history and ontology of objects, Bill Brown reminds us that these denials and whatever subsequent vacillations around the status of the Negro slave should always be rooted in material terms. Though a familiar conceit of a racist modernity, these discourses of inhumanity become materialized in America via the “contradictory legal status of the American slave—both human and thing” (Brown 2006, 179). It’s worth mentioning here that there is now somewhat of a tradition of black theorists and critics for whom the primary technologies of modernity are in fact racialized ones that depend on what Negritude poet Aimé Cesaire once referred to as colonial “thingification.” First, the slave ship, which on the one hand denatured black slaves while expanding the material bounds and needs of modernity, as well as its conceptual and social possibilities; second, the plantation, what Caribbean thinkers from C.L.R. James to Antonio Benitez Rojo and Sylvia Wynter herself have proclaimed

9  RACE AND ROBOTICS 

167

central to the construction of regimented, modern subjectivities in advance of industrial processes; and, thirdly, in America, the cotton gin, which helped engineer the industrial revolution while entrenching slavery via those very industrial processes. I cite this tradition of engagement to bolster my work’s overall insistence that thinking about technology is indeed incomplete without appending the long tradition of thinking about racism, colonialism and the common problems of bodies and power. That aforementioned legal denial of humanity operated alongside the rhetoric of technology since the Negro was a liminal figure in between human and animal; the rhetoric was also enabled by it because blacks were also figured as labor saving devices, prosthetic extensions of the white master. “Man shaped ploughs,” the great Barbadian writer George Lamming once described black slaves, guided by the allegedly rational control of their masters (Lamming 1992, 121). Otherness, therefore, has simultaneously been imagined, projected, and sociopolitically framed as racial, sexual, and certainly technological—all variations and distinct genres of “the human,” in Wynter’s baroque formulations, traceable to the same epistemic root and historical processes. In the wake of such crises of and opportunities for embodiment, and the taxing political legacies of the differences that it implies, the West has chosen also to define technological capacity and perhaps historical process itself via a quest for verisimilitude as “the human” eventually competes with its own creations, eventually replacing itself as the pole of representative authority with machinic doubles. These figures then function as the sign of fidelity while our inhuman racial doubles serve as a primordial past and a present impossibly stained by that past—is this not also the history of science fiction? With this I’d like to now skim through the history of science fiction to provide more examples of the benefits of a racialized reading of technology’s histories and to also make clear how lacking our knowledge has been without it. We know that the term robot entered into the English language in 1923 upon the English translation of Czech writer Karel Č apek’s play R.U.R. (Rossum Universal Robots), which premiered in New York the year before. Along with his brother Josef, Č apek coined the term robot from the proto-Slavic word, robota, meaning labor, drudgery, mindless work, or, essentially, slavery. As a connection to blacks the term would remain metaphoric or imaginative were we to leave it at that, and there would be much to keep us satisfied. After all, readings of the play in terms of the labor and class

168 

L. CHUDE-SOKEI

r­ elationships of industrial capitalism dominate the history of its reception, so do readings of the “robot” in terms of early twentieth-century tensions around women’s empowerment and gender relations. But perhaps the greatest lapse in these readings has been in not foregrounding race, which is where the links become grounded also in early twentieth-century anxieties around race relations. As we know, the play is about a revolution, a potent and ongoing fear in the early years of the twentieth century, along with the realities of riots and other forms of racial violence and terror in America. This uprising is one in which a servile, inhuman caste—and this is central here—discourse upon and eventually evolve “souls.” The question of “soul” is what removes the play from being exclusively within the context of class relations, given the fact that one of the rationales of slavery was that the Negro was without one. In R.U.R. “soul” is crucial to the robots’ resistance and enables them to rise up and destroy “humans” (read: whites). The human is here in parentheses because once they have been effectively erased, the category now shifts to their successors. The play even features abolitionists (like Helena, a gendered liberal white stereotype that would have functioned just as well in Uncle Tom’s Cabin) who attempt to first liberate the machines that they call “brothers,” and turn them into wage laborers as well as give them voting rights. The robots are popular among missionaries, anarchists, and the Salvation Army trying to locate their “souls” and convert them to some “human” creed or another until the robots create their own organizations and political movement. The makers then counter via a “divide and conquer” strategy by making robots of different races, nationalities, and languages in the hopes that prejudice will keep them from organizing. Of course, all this fails, the robots rise up, “humans” are slaughtered and the era of man— that distinct and “overrepresented” genre of the “the human”—is proclaimed over. Č apek would pursue the relationship between blacks and science fiction even further in a text that retroactively sheds great light on the racial politics of R.U.R.—1936’s War with the Newts. This lesser known novel tackles everything from miscegenation to lynching, the early twentieth-century fetishization of racial performances to even pan-Africanism with barely a direct reference to African Americans (well, there is one eye-winking footnote where he deliberately pretends to not be talking about blacks while talking explicitly about lynching, racial stereotypes, and interracial sex only to hammer home the true source of his satire).

9  RACE AND ROBOTICS 

169

Č apek, however, was not the first. He was only working in a tradition of associations that go back to the dawn of what would become science fiction, back to the Victorian era and proto-SF/post-Gothic/imperial adventure narratives of the sort that would influence and prepare the world for H.G. Wells’ War of the Worlds (1898). That tradition arguably rests with a text like Samuel Butler’s Erewhon: Or Over the Range (1872) prefigured by his 1863 essay, “Darwin Among the Machines.” This text is where the equation of slave and machine emerge and blend in ways pertinent to our current thinking about race and technology. It matters also because it is simultaneously crucial to the development of SF as it is to the emergence of robotics and the very notion of artificial life. Although Herman Melville’s story 1856 “The Bell-Tower,” from The Piazza Tales, was an earlier articulation of this equation and was, like Butler, a response to Mary Shelley’s Frankenstein, its influence was hardly as great as Butler’s and its dense metaphors rendered racial details far less obvious. Erewhon is where it is first theorized—however satirically—that natural selection could apply to machines, and that machines would develop their own independent consciousness and intelligence. Butler is not only the first to emphasize the “revolt of the machines” plot of so much science fiction (taking Mary Shelley’s suggestions further than anyone had at the time) but was the first to describe machines as a distinct race, and would do so in the language and context of nineteenth-century biological science. And the immediate historical context of his work’s production matters a great deal here: Samuel Butler wrote “Darwin Among the Machines” during the New Zealand (or Waikato) Wars that took place between 1845 and 1872 between British-backed settler colonists and the indigenous Maori who provided armed resistance to the encroachment upon traditional lands. Butler’s arrival coincided with the Invasion of Waikato that in 1863 brought more British troops to New Zealand than anywhere in the world. It is therefore necessary to read in the proto-science fiction of Erewhon quite material fears of European destabilization much in the way white anxieties of black freedom would manifest in the literatures and cultural phenomena of late nineteenth-century America much in the way that human anxieties of technology’s dominance would structure the growth of science fiction. Most important is how so much of Butler’s thinking about and description of machines is done in terms borrowed explicitly from the only available analogues: slavery and colonization. This is all present in Erewhon’s obsession with crossbreeding and machine sexuality; his depiction of

170 

L. CHUDE-SOKEI

i­ntimate power relationships between machines and humans; his very description of machines as distinct, evolving beings, inscrutable but uncannily human, dependent yet resentful and ultimately destined for a “civil war”—a term he uses in advance of the actual US conflict to emphasize that one “race” was inevitably doomed to dominance and the other to erasure. And his obsession with the parallels is made relentlessly clear, again, in his at times panic-stricken discussion of machines in the language of master and slave and his insistence on depicting and describing them as a race and their relationship as so terrifyingly mutual as to invite apocalypse since the language of mutuality escapes the colonial mind. I’ll conclude here precisely because this is where so many of these parallels begin, though they are just a part of the larger structuring oppositions of colonial modernity, that between civilization and the primitive, Europe and Africa. But there is one final thing: soul. The issue of “soul,” of who or what has it and who or what has the authority to bestow it is also present in Butler’s vision of machine intelligence as fundamentally dependent on racial embodiment. The narrator inevitably encounters the issue/cliché but in doing so discovers something akin to contemporary thinking about artificial intelligence as an echo of race and colonial power. If the attempt to define machine intelligence reveals how little we know about what thinking is, and the attempt to attribute humanity only throws into question what “the human” is or has been, Butler anticipates these epistemological crises at the moment when slaves and machines become interchangeable. Butler writes: [T]he servant glides by imperceptible approaches into the master; and we have come to such a pass that, even now, man must suffer terribly on ceasing to benefit the machines … Man’s very soul is due to the machines; it is a machine-made thing: he thinks as he thinks, and feels as he feels, through the work that machines have wrought upon him, and their existence is quite as much a sine qua non for his, as his for theirs. This face precludes us from proposing the complete annihilation of machinery, but surely it indicates that we should destroy as many of them as we can possibly dispense with, lest they should tyrannize over us even more completely. (Butler 1967, 149)

It is soul that is inorganic here, “machine-made.” It is the product of a distinction between self and other in which “the other” is other than human, at least initially. Because it is the inhuman other that generates it in this context, “soul” is merely claimed as the solitary possession of the

9  RACE AND ROBOTICS 

171

“human,” as Sylvia Wynter would insist (and as Č apek’s robots would agree and demonstrate by the end of that remarkable play). Soul here is not an essential or certainly God-given quality and, again, emerges from the slave and is appropriated by the master as a way to control the slave. Like race and intelligence, “soul” is ultimately a product of technology and then as now illuminates and delimits relationships of power.

References Brown, Bill. 2006. “Reification, Reanimation, and the American Uncanny.” Critical Inquiry 32 (Winter): 175–207. Butler, Samuel. 1967. Erewhon: Or, Over the Range. New York: Airmont. Butler, Judith. 2004. Undoing Gender. New York: Routledge. Dery, Mark, ed. 1993. “Flame Wars: The Discourse of Cyberculture.” Special Issue, South Atlantic Quarterly 92 (4): 179–222. Eglash, Ron. 1995. “African Influences in Cybernetics.” In The Cyborg Handbook, ed. C.H. Gray, 17–28. London: Routledge. Eshun, Kodwo. 1998. More Brilliant Than the Sun: Adventures in Sonic Fiction. London: Quartet Books. Lamming, George. 1992. The Pleasures of Exile. Ann Arbor: University of Michigan Press. Weiner, Norbert. 1950a. “Cybernetics.” Bulletin of the American Academy of Arts and Sciences 3 (7): 2–4. ———. 1950b. The Human Use of Human Beings: Cybernetics and Society. New York: Doubleday/Anchor Books. ———. 1960. “Some Moral and Technological Consequences of Automation.” Science, n.s. 131 (3410): 1355–1358. www.jstor.org/stable/1705998

Index

A Aaronson, Scott, 127 Abstraction autonomous subjectivity and, 68–70 capitalism and, 72–74, 83 machine intelligence and, 39, 100 mathematical modeling and, 7, 20, 31 noetic soul and, 94 ACM, 113 Adorno, Theodor capitalist society and, 72–74 rationality of capitalism and, 71, 73, 75 technology and, 65, 72, 74, 77, 81 transcendental subject and, 69–70, 75–76 Affective machines, 42–45, 138, 163–164 African American identity and experience, 14, 159–161 slavery and cybernetics, 160–162 See also Race and racism; Slaves and slavery

Afrofuturism foundational tropes in, 160–161 idea of the human and, 165 K. Eshun and, 159 robots and, 13–14 the uncanny, and, 160 See also Science fiction Agency automation and human, 103–104 biological evolution and, 18, 36 boundaries of, 36, 37, 42 collaborative, 44 collective capitalist, 74 embodiment and, 42–45 of intelligent agents, 66 modeling of human, 142 paradoxical structure of, 69–71 rational, 67, 77–78 A.I. and robotics field, 2–4 capitalism and, 64, 83, 120 marketing in, 120–121, 128–130, 133 metaphors of humanlike qualities in, 131

© The Author(s) 2019 T. Heffernan (ed.), Cyborg Futures, Social and Cultural Studies of Robots and AI, https://doi.org/10.1007/978-3-030-21836-2

173

174 

INDEX

A.I. and robotics field (cont.) metaphors of the monstrous in, 134 motivations in, 142–143 myths in, 4–7 race, slavery and, 160–165 reference to fiction in, 4, 11–14, 37, 128–130 See also Artificial intelligence; Engineering; Robots A.I. Artificial Intelligence, 39 Algorithms, 3, 4, 18 advances in, 36, 105 cultural determinants of, 136–138 ethics and, 11, 120 humanlike machines and, 131–134 kill decisions and, 53 Alive, A.I. and robots as, see Animation/animating; Life/living Alphabet Inc, see Google AlphaGO, 131 Amazon, 2, 3, 135 Animals, 3, 4, 7, 12, 56, 107 in Aristotelian doctrine, 93 as automatons, 5 dog-like robots, 133–134 economic productivity and, 153 human slavery and, 167 Lucy the chimpanzee, 40 primate-like robots, 45 swarming behavior of, 10, 96, 99–100 See also Birds; Fish; Insects; Robots Animation/animating, 12, 45, 82 A.I. and robots, 43, 120–121, 129 capitalist subjects, 73, 75, 80 creation myths on, 135 Frankenstein’s monster, 134 the human, 96, 103 life forms, 92–93 See also Life/living Animatronic dolls, 119, 133 Anthropomorphism A.I. superintelligence and, 64, 78–80 corporate, 122–123

evolutionary theory and, 20 science fiction and, 166 Apple, 3 Aristotle, 10, 96, 115 On the Soul, 92–93 Arquilla, John, 98–99 Artificial intelligence black box design in, 3, 6, 118 Cleverbot and, 37 cognitive assemblages and, 104–106 consciousness of, 119, 121, 131, 132 cybernetics and, 161–164 data protection and, 118–119 decision-making and, 99, 103 Deep Blue and, 41–42 deep learning and, 118, 120, 131–133 humanlike attributes of, 35–36, 39–40, 119, 130–132 intelligent agent theory of, 66 military applications of, 89–90, 98–100, 103–104, 115–118, 121–122 origin of concept, 4 risks of, 36, 63, 64 Turing test and, 37–39 See also Computation; Intelligence; Machine(s) Aryananda, Lijin, 44 Asimov, Isaac, 162 Asp, Karen, 9–10 Auerbach, Erich Mimesis, 149 See also Mimesis Automation capitalism and, 72, 74, 77, 81, 83, 145, 147–149 of domestic tasks, 49 ethics of, 120 of factory systems, 117–118 human redundancy and, 154 ideal of leisure and, 153, 154 of rationality, 65–66

 INDEX 

of war, 3, 10, 51, 53, 54, 89, 103–104, 106 See also Industrial revolution; Machine(s); Technology Autonomous, 43, 44 Autonomous weapons, 3, 10–11 ban on, 53–54 delegated decisions and, 53, 103, 106 drones debate and, 54, 101–103 future of, 89–91 Geneva conventions and, 53 regressive visions of, 106–107 robots as, 51–54 Stop Killer Robots Campaign on, 10, 53 swarming behavior and, 97–100 U.S. Air Force and, 97, 98 voluntary stupidity and, 11, 101, 102 See also Military; Swarming technology; War Autonomy of A.I. and robots, 3, 5, 8–9, 36, 43–44, 66–68 automatic behaviour and, 66, 74, 94 automation and, 103–104 biorobots and, 24–26, 32 of capitalist subjects, 76–77 cognitive assemblages and, 104–106 cybernetics, slavery and, 164 humanism and, 122 ideal of, 64, 142 myth of, 9 See also Agency; Subject, the B Backhaus, Hans-Georg, 73 Baudrillard, Jean, 133 BB-8, 128 BBC, 46 Benitez Rojo, Antonio, 162, 166 Bentham, Jeremy, 120 Berners-Lee, Tim, 2

175

Bezos, Jeff, 12, 128 Big data, 3, 6, 118, 120, 130 See also Data Biorobotics, see Evolutionary biorobotics Birds, 21 Darwin’s finches, 20–24 finches, 7, 19 Swarming behavior of, 100 Bodies, physical brains and, 8, 31 human evolution and, 96 intelligent life and, 69, 82 machine embodiment and, 42–45 psychological motivations and, 69 reengineering human, 151–152, 155 and utilitarian theory, 77 See also Embodiment Bondaryk, Joe, 46 Boston Dynamics, 46, 47, 49 Bostrom, Nick, 3, 122 Earth-originating intelligent life and, 69 on existential risk, 63 future of humanity and, 81–82 motivations of A.I. and, 77–81 Brain, the, 3, 8, 76 artificial neural networks and, 131–132 figure of, 39 metaphors of, 12, 131 neural models of, 29 singularity and, 156–158 See also Networks Breazeal, Cynthia, 43, 44, 128 Brooks, Rodney, 42 Brown, Bill, 166 Buber, Martin, 116, 122 Butler, Judith, 165 Butler, Samuel, 164 “Darwin Among the Machines,” 169 Erewhon: Or Over the Range, 169–170

176 

INDEX

C Campaign Against Sex Robots, 11, 113 Campaign to Stop Killer Robots, 10, 53, 113 Č apek, Karel, 1, 4, 12, 13, 117, 142 automation and, 147–148 mimesis in, 145–147 politics and, 147–149 race/machine metaphors in, 14, 167–169 R.U.R. (Rossum’s Universal Robots), 144, 167–168 War with Newts, 168 “The White Plague,” 150 Capitalism, 9, 11, 13, 14 automation and, 64, 74, 83, 145 ethics and, 120, 121 fetishism and, 73, 75, 80, 83 homo oeconomicus and, 66, 67, 75, 77, 78, 80, 83 modern slavery and, 162, 163, 168 rationality of, 64–65, 73 robot ethics and, 11 subjectivity and, 75–77, 122–123 techno-utopianism and, 122 See also Automation; Industrial revolution Caregiving robots and, 49–50, 118, 119, 121, 132 See also Robots Carlyle, Thomas, 162 Carnegie Science Center, 128–129 Carpenter, Rollo, 37 Castãneda, Claudia Figurations, 40 “Robot Visions,” 40 Centre for the Study of Existential Risk, 2 Cesaire, Aimé, 166 Chamayou, Gregoire, 102–103 Drone Theory, 90

Chandra, Vikram, 6 Chaplin, Charlie, 117 Chatbot, 132 Child, childhood, 122, 129, 130, 135, 136 as metaphor, 131 robot figures of, 8, 39–41 CHIMP robot, 45–47 Chude-Sokei, Louis, 13–14 Cleverbot, 37 Climate change, 9, 24, 35, 47, 48 CNAS, 10, 97 Coming Swarm, The, 98 Robotics on the Battlefield, 98 Cog, 42–44 Colonialism, 4, 39, 123, 159 existential risk and, 80, 82, 83 figures of anxiety and, 169–170 technologies of slavery and, 166–167 Coming Swarm, The, 10, 98, 104 human role in, 105 regressive vision in, 107 See also Swarming technology Companions animal robots as, 133–134 Nadine robot as, 132 nonhuman species of, 36, 41 robots as, 50, 119–122, 129 sex robots as, 132, 136 See also Domestic robots; Robots Computation affective, 138 evolutionary, 18 immortality and, 150–151 intelligence, 4 machine, 39 metaphysics and, 155–158 intuition and, 131 See also Artificial intelligence; Computer(s); Cybernetics Computational intelligence, 4

 INDEX 

Computer(s) A.I. programs and, 118 Deep Blue and, 41–42 fiction and, 128 graphics, 10, 55, 100–101 human, 66 languages, 3, 5, 6 metaphors of brain and, 12, 131–132 science of, 2, 127, 131 as supercomputers, 6–7, 63, 67, 68 See also Machine(s); Networks; Superintelligence Computer scientist, 151 Consciousness, 6 of A.I. and robots, 119, 121, 127, 131–133 fictional machines and, 68, 134–135, 146, 169 individual, 70, 71 Kurzweil, Raymond and, 154–158 unconscious activity and, 74, 75, 83 See also Agency; Life/living; Subject, the Crogan, Patrick, 10–11 C-3PO, 127, 128 Cuteness A.I. and robots designed for, 121, 132, 137 Cybernetics, 159 race, slavery and, 14, 160–164 (see also Weiner, Norbert) Cyborg, 7, 122, 161 D DARPA, 9 Robotics Challenge, 45, 48, 49, 51, 52 Dartmouth Summer Research Project on A.I., 4 Darwin, Charles

177

evolutionary perfection and, 18 natural selection and, 20–21 On the Origin of Species, 17 Darwin’s finches, 20 struggle for existence and, 20–24 See also Evolutionary biorobotics Data, 35, 41, 55, 105 biases in, 136–138 big data, 3, 6, 120, 130, 131 drone surveillance and, 102 exploitation of, 5, 121, 138–139 General Data Protection Regulation, 119 as human essence, 155–156 processing centers, 135 protection and privacy of, 118–119 Dawkins, Richard, 96 De Landa, Manuel, 91 Deep Blue, 41–42 Deep Mind Industries, 131 DeLillo, Don White Noise, 7 Dery, Mark, 160 Descartes, René, 5, 122 Dewey, Daniel, 65 Domestic robots, 9, 48–50 See also Robots DREAM project, 114 Drones, 52, 148 The Coming Swarm and, 98 Drone Theory and, 90–91 military use of, 97, 101–103, 107 See also Military; Swarming technology; War Du Bois, W.E.B., 160, 165 Souls of Black Folk, The, 159 E Earth-originating intelligent life, 63, 64, 69–70, 82 Eglash, Ron, 164

178 

INDEX

Eliot, T.S., 150 Embodiment capitalist subjects and, 75 digital data and, 55, 118 of idea of the human and, 165–166 robot agency and, 44, 56 of soul in machines, 170 See also Bodies, physical; Subject, the Emergence, 10 Engels, Friedrich, 116 Engineering, 24, 42, 46, 47, 49, 67 evolutionary theory and, 18, 32 fictional tropes in, 143, 146, 148, 151, 152 imitation vs. creation in, 13, 146, 148, 151 as mimicry, 154–155 modelling and, 24, 142 optimization in, 13, 18–19, 146–147 satisficing and, 67, 155 See also Mimesis; Representation; Simulation; Technology Environment, 145 crises of, 9, 14 harmony with in Ex Machina, 134 human power to change, 105 impacts of technology on, 3, 10, 117 posthuman power over, 76 robot agency and, 43, 49, 66 role of in evolution, 24 unlimited resources of, 79 Erewhon slavery and colonialism in, 169–170 Erica android, 136 Eshun, Kodwo, 160, 165 More Brilliant Than the Sun, 159, 161 Essentialism, 96–97, 155 Ethics of A.I. and robotics, 11–12, 113–114 automation and, 117–118

capitalism and, 120–122 feminist, 114, 116–117, 123 human intelligence and, 97, 103 humanism and, 121–124 personal data and, 118–119 robots and, 50, 118, 119 See also Feminism European Parliament, 4 Evolutionary biorobotics autonomous behavior in, 27, 28, 30, 32 concept of fitness in, 24–25, 28–31 concept of modeling in, 24, 29–31 neural networks and, 27–31 Tadro robot models in, 26–27 See also Darwin, Charles; Engineering; Evolution Evolutionary robotics, 18, 24 Evolution, 8 in A.I. and robot design, 36, 39–40 of brains, 29–31 emergent intelligence and, 100 human, 92, 95–96 race, slavery and, 164–170 simulations and models of, 7 struggle for existence and, 17, 20–24 sufficing vs. optimizing in, 18–19 techno-utopian idea of, 6–7, 139, 142, 157–158 web of life and, 7 See also Narrative(s); Technology Existential risk, 2, 9–10, 78, 81 capitalism and, 64, 80, 83 concept of humanity and, 63, 69, 81–83 Ex Machina, 12, 134–138 Ava and Kyoto androids in, 135–138 modern Frankenstein story of, 134–136

 INDEX 

Experience A.I. and embodied, 68 A.I. and human, 91–93, 102–103 A.I. capacity for, 67 realism and lived, 143 simulating mental, 131 See also Embodiment F Facebook, 2, 3, 113, 118, 123 Fagenson, Zachary, 46 Feminism ethics of, 114, 116–117, 123 patriarchy and, 116, 120–122 science and technology studies and, 36, 40 See also Ethics; Gender; Sexuality Fetishism of A.I. and robots, 10, 12, 65 capitalist subjects and, 75–77, 83 feminist ethics and, 116 in Marxian theory, 73–74 racial stereotypes and, 168 rogue superintelligences and, 83 See also Capitalism; Subject, the Fiction A.I. and robotics field and, 4, 11–14, 39, 42, 46, 127–131 dystopic, 13 ethics of robots and, 120–121 influence on technology, 142–143 literal interpretations and, 12, 83, 130, 139, 145, 146, 150–151, 155 metaphor in, 129 realism and, 139, 143, 149–151 tropes of machines and technology in, 13, 128, 143, 152, 164 utopian nonfiction and, 142, 150–151, 154–155 visualizing warfare and, 101

179

See also Myth(s); Narrative(s); Science fiction Figuration, concept of, 40 Fish, 7, 19 modeling evolution of, 26–27 Fisher, Mark, 161 Ford, Henry, 117–118 Frankenstein, 4, 12, 134–135, 139, 169 Future(s) and Afrofuturism, 13, 160 Baudrillard’s anorexic, 133 of capitalism, 64, 83 envisioning robot, 13, 56, 142–143, 158 futurist hedges, 52 marketing technologies of the, 128–130, 138–139 political utopias and the, 148 transhumanist visions of, 2–3 techno-utopian, 37, 139, 154–155 of warfare, 10, 89–91 See also Narrative(s); Utopianism Future of Humanity Institute, 2 Future of Life Institute, 2, 113 G Game(s) of chess with Deep Blue, 41–42 of GO with AlphaGO, 131 of real vs. simulated traits, 135, 138 simulated war, 100–101 Turing test as a, 37–39 Gender, 14 Erica android and, 136 Ex Machina androids and, 134–137 robot narratives and, 37, 47 technology, race and, 159, 165, 166, 168 Turing test and, 37 See also Feminism; Sexuality

180 

INDEX

Genetically modified organisms (GMOs), 117 Geneva conventions, 53 Gernsbach, Hugo, 161 Gilligan, Carol, 116, 122 Google, 2, 3, 113, 118, 135 A.I. projects of, 131 DARPA Robotics Challenge and, 47 humanoid robots and, 49 Gould, Stephen Jay, 157 Grand, Steve Growing Up With Lucy, 40 Grant, Peter and Darwin’s finches, 20–24 Grant, Rosemary and Darwin’s finches, 20–24 H Hanson, David, 132–134 Haraway, Donna, 40, 45, 122 Hawking, Stephen, 9–10, 36, 37, 51, 65, 76 agencies of, 37, 42 ecological crisis and, 81, 83 risks of A.I. and, 63–64 Hayles, N. Katherine, 104 Hidden Figures, 66 Hinton, Geoffrey, 131–133 History/histories of race and technology, 14, 159–161, 164–166 robot narratives and, 36, 37 of science fiction genre, 167–170 struggle for existence and, 20–25 Hobbes, Thomas, 77, 78 Homer, 149 Hong, Dennis, 46 Human(s) concept/idea of the, 94–95, 155, 164–167

conflated with machine, 4, 8, 11–12, 120–123, 132, 138–139 contingent meaning of the, 134 figure of the, 36 perfect and functional models of, 142–143 potential in war, 102, 105–107 souls of, 170–171 Humanoid robot(s) Atlas, 47, 49 caregiving, 50 Cog, 43–44 DARPA Robotics Challenge and, 45–47 development of, 35 domestic, 49 Erica, 136 in Ex Machina, 135–138 introduced in R.U.R., 148 Kismet, 43–44 limits of, 55, 56 Mertz, 44–45 military, 52–53 Nadine, 132 Robohon, 132 Sophia, 132–133 Terminator-style, 53 Valkyrie, 47 See also Machine(s); Robots; Technology Human Rights Watch, 53 Hume, David, 115, 122 I IBM Corpration, 113 IEEE, 113 Imitation Game, 37, 39 Immortality, 3, 7, 14 Kurzweil and, 13, 150–151, 156

 INDEX 

Individualism, 8 bounded form of, 36, 37, 42 ethics and, 114, 117, 122–123 Industrial revolution, 143–144, 147, 167 contemporary, 149, 163 fourth, 64, 72, 83 Insects, species of, 5, 10, 100 Institute for Human and Machine Cognition, 47, 49 Intelligence, 31 abstract, universal idea of, 39, 69, 70 in Aristotelian doctrine, 92–93 in evolutionary theories, 31, 100 human vs. artificial, 4, 36, 41, 52 Kurzweil, Raymond, metaphysics of, 155–158 racism, slavery and idea of, 164–165, 169 as rationality, 65–68, 77–78 Turing test of, 37 voluntary stupidity and human, 102–103 See also Artificial intelligence; Computation; Superintelligence International Committee for Robot Weapons Control, 53 International Joint Conference on A.I., 51 iPhones, 122 Ishiguro, Hiroshi and Erica android, 136 J James, C.L.R., 162, 166 Jameson, Fredric, 150, 154 Archaeologies of the Future, 148 Jetsons, The, 129–130 Jibo, 128–130 Jim Crow era, 161 Joyce, James, 150

181

K Kaku, Michio, 52 Kant, Immanuel, 115, 122 Kasparov, Garry, 41–42 Keating, Jennifer, 12–13 Kelly, Kevin, 99 Kilcullen, David, 102 Killer robots, see Autonomous weapons Kismet, 43–44 Knight, Will, 47 Kurzweil, Raymond, 3, 6–7, 12–13, 37, 142 dystopic implications in, 154–155 human reengineering and, 151–153 immortality and, 150–151 Singularity is Near, The, 150 utopian metaphysics of, 155–158 See also Singularity; Transhumanism; Utopianism L Labor abstract, 64, 73, 76 domestic robots and, 9, 48–50 ethics and automation of, 117–118, 120 leisure time and machine, 13, 129–130, 153–155 meaning of “robot” as, 1, 144–145, 167 networks of human and machine, 8, 42, 43 noetic, 91–93 optimizing with machines, 142, 146 robota as forced, 1 technology and slave, 163–164, 167 Lamming, George, 167 Lang, Fritz, 117 Lanier, Jaron, 2, 5

182 

INDEX

Law(s) new technologies and, 117 and rights-based justice, 114 status of slaves in, 166, 167 transcendental subject and, 70, 71, 76 of value, 74, 75, 80 of war, 53–55, 102, 103, 106, 107 Leisure time, 13, 129–130, 153–155 Lepore, Jill, 2 Leroi-Gourhan, André, 92, 95–96 Gesture and Speech, 95 Lethal autonomous weapons, see Autonomous weapons Levy, David, 132 Life/living A.I. and robots as, 36, 43, 120–121, 132, 166 Frankenstein’s monster as, 134 intelligence as, 69, 82 as metaphor, 162 See also Animation/animating; Fiction Literal, 130, 145, 151, 152, 155 Long, John, 7–8 Lost in Space, 129 M MacCabe, Colin Keywords for Today, 143 Machine Intelligence Research Institute, 2 Machine(s) age of intelligent, 91 animal conflated with, 5 concept of the, 4 embodying soul in, 170 human conflated with, 8, 11–12, 120–123, 132, 138–139 intelligence, 4, 36–39, 67, 103, 170 killing, 52

learning, 118, 130–132, 163 perfection in design of, 18–19, 142 rise of, 35, 46, 51, 52, 168, 169 sexuality of, 169 slaves equated with, 161, 163–164, 169–171 techno-utopian merger with, 11–12, 155–157 See also Affective machines; Robots; Technology MacPherson, C.B., 77 Marinetti, Filippo Tommaso, 162 Marx, Karl, 73, 116, 117, 122 McCarthy, John, 4 McQuin, Christopher, 47 Melville, Herman “The Bell-Tower,” 169 Mertz robot, 44–45 Metamorphoses, 121 Mialet, Hélène, 37, 42 Microsoft Corporation, 113, 135 Military cognitive assemblages in, 105–106 -industrial complex, 10, 89, 90, 122 intelligence, 102–103 regressive vision in, 107 situational awareness in, 54–55 sponsored research on robotics, 9, 45 use of drones, 97, 101–103, 107 See also Autonomous weapons; War Mimesis Auerbach on, 149 imitation vs. creation and, 146, 155 Kurzweil and, 152–153, 156 politics and, 147–149 robotics and, 13, 142 See also Engineering; Representation; Simulation MIT, 42–45, 128 Morrison, Toni, 164 Motivations, 67–68, 72, 82 A.I. superintelligence and, 77–81

 INDEX 

in design of A.I. and robots, 142, 146–147 See also Rationality; Subject the Musk, Elon, 51 Myth(s) in A.I. and robotics, 4, 7, 9 creation, 135–137 humanism as, 122 of progress, 13 Pygmalion, 4, 121, 136 See also Fiction; Narrative(s) N Nadine robot, 132 Nanorobots, 152, 154–156 Narrative(s) A.I. and robot ethics, 114–116, 122, 124 evolution, 36, 139, 157–158 existential risk, 9, 64–65 framing of robots, 36, 43, 56 race and technology, 160, 169 technological progress, 37, 52, 56, 63, 142, 149, 150, 155, 157 See also Fiction; Myths; Utopianism NASA, 47 Networks, 37, 55 computer, 123 human and machine, 8–9, 42, 43 interaction, 17–20 neural, 27–32, 131 social, 123 swarming technology and, 98–99 Nietzsche, Friedrich, 115–116 Noetic soul, 10, 91 animating the human, 93–94, 96–97 human potential and, 102, 105, 106 regression and, 97, 106–107 voluntary stupidity and, 102 See also Labor; Stiegler, Bernard

183

Norvig, Peter, 66 Nourbakhsh, Illah, 12–13 O Objects, 44–45, 73, 75, 116, 135–137, 166 Open Roboethics Initiative, 50 Optimization engineering design and, 13, 19, 142, 146–149, 152 evolutionary theory and, 7, 18–19, 22 satisficing design vs., 155 trope of, 148 See also Engineering; R.U.R. (Rossum’s Universal Robots) Other/otherness, 13, 80 ethics and, 116, 163 idea of the human and, 122–123, 166–167, 170 See also Ethics; Race and racism P Patterson, Orlando, 164 Petraeus, David, 102 Plato The Republic, 141, 143 Politics mimesis and, 147–149 race, technology and, 162, 164, 165, 168 technology, gender and, 37, 135 Posthumanism, 142, 155–158, 166 posthuman, 76, 146, 151, 155–156, 167 See also Transhumanism; Utopianism Prabhaker, Arati, 46 Pratt, Gill, 45, 47–49, 51 Primates, nonhuman, 40, 45 Progress, 52 Pygmalion, 4, 121, 136

184 

INDEX

R Race and racism, 137 cybernetic ethics and, 162–165 equated with technology, 160–161, 165, 169 idea of the human and, 165–167 science fiction and, 160–161, 167–170 technology and, 13–14 See also Afrofuturism; Slaves and slavery; Technology Rand, Ayn The Virtue of Selfishness, 115 RAND Corporation, 97, 98 Rationality, 9 of capitalism, 64, 73, 75 machines and, 65–68, 83 superintelligence and, 77–79 utilitarian theory of, 77 See also Subject, the REELER project, 114 Representation of reality through mimesis, 149 scientific models and, 25–26 of swarming robots, 99–101 See also Evolutionary biorobotics; Mimesis; Simulation Richardson, Kathleen, 11 Riza, M. Shane Killing without Heart, 103–104 RoboBrain project, 130 Robohon, 132 Robota definition of, 1, 144, 167 Roboticist, 151 Robot Ethics, 130 Robots caregiving, 9, 49–50, 119, 132 child figures of, 8, 39–41, 45 domestic, 9, 48–50 first, 14, 165 Jibo social, 128–130 as metaphors, 162 museum exhibits of, 128–129

origin of term, 13 rescue, 9, 46–48 rights, 5, 11, 121, 123 sex, 11, 118, 119, 132, 135–137 social, 9, 132, 138 Tadros biorobots, 31–32 See also Affective machines; Animals; Companions; Humanoid robot(s); Swarming technology Ronfeldt, David, 98–99 Rosie the robot, 129 Ross, Daniel, 94 R2D2, 11, 128, 129 R.U.R. (Rossum’s Universal Robots), 1, 144, 167 creation vs. imitation in, 13, 143, 145–147 meaning of robot in, 144, 167–168 posthumanism and, 151 slave/machine metaphors in, 168 Russell, Stuart, 65–68 S Sanskrit, 6 Saxena, Ashutosh, 130 Scharre, Paul, 103, 105–106 Coming Swarm, The, 98–100 Science, 130 of cybernetics, 159–164 idea of the human in, 134 industrialization and, 145–146 literal interpretations and, 12, 130, 145 STEM education and, 144 techno-utopianism and, 152, 157 tekhne and, 143–144 See also A.I. and robotics field; Evolutionary biorobotics; Technology Science fiction, 13–14 A.I. and robotics references to, 37, 46, 127–128, 130

 INDEX 

genre of, 142, 154, 160–161 race, slavery in, 159–160, 167–170 realism and, 149–150 utopic ideas and, 142, 150–151, 154–155 See also Afrofuturism; Fiction; Technology Science Museum (London), 128 Scientific, 145 Sexuality, 39, 165, 167 of machines, 169 See also Companions; Gender; Robots Sharkey, Noel, 53 Shelley, Mary Frankenstein, 4, 134–135, 139, 169 Short Circuit Johnny Five character in, 129 Silicon Valley, 5, 144 Simondon, Gilbert, 96 Simulation, 4, 7 Baudrillard and, 133 of companion animals, 133–134 of emergent intelligence, 100 of human emotions, 137, 138 of neural activity, 131 perfection and, 142–143, 155–156 scientific modeling and, 19 of swarming behaviour, 96 Turing test and, 39 See also Engineering; Mimesis; Representation Singer, Peter, 116–117 Singhal, Amit, 128 Singularity, 3, 6, 9, 37, 142 metaphysics of, 156–157 Singularity is Near, The, 13, 150, 155 utopic vision and, 151 Slaves and slavery A.I. robots and, 13–14, 121, 123 cybernetics and, 160–165 human evolution and, 164–165 science fiction and, 160–161, 167–170

185

technologies of, 166–167 See also Labor; Machine(s); Race and racism; Soul Snow, C.P. “Two Cultures,” 144 Social media, 3, 118, 138–139 Sophia, 132–134 Soul inhuman other and, 170–171 of the robot, 168 of slave and machine, 164 Souls of Black Folk, The, 159 See also Noetic soul Species, 39 biodiversity and, 7 companion, 36, 41 in evolutionary biology, 18 extinction of, 5, 10, 135 on the human, 69, 81, 156 On the Origin of Species and, 17 of rational agents, 78 swarming, 96, 100 Spielberg, Steven A.I. Artificial Intelligence, 39 Spivak, Gayatri Chakravorty, 122 Star Trek, 127, 128 Data character, 127 Star Wars, 127–129 BB-8 character, 128 C-3PO character, 127, 128 R2D2 character, 11, 128, 129 Stiegler, Bernard bio-technical composition and, 94–96, 105 Decadence of Industrial Democracies, The, 92 evolution and regression in, 95–97 noetic intelligence in, 91–95 See also Noetic soul Subject, the automatic, 74, 75, 77, 80, 83 distributed, 37, 42 humanist notions of, 122–123

186 

INDEX

Subject, the (cont.) psychological, 64, 68–71, 75, 77, 78 race, slavery and, 162, 165–167 robot agencies and, 36, 44 supraindividual capitalist, 74, 75, 80 transcendental, 64, 67, 69, 70, 75–77 Suchman, Lucy, 8–9 “Robot Visions,” 40 “Subject Objects,” 44 Sufficing in evolutionary theory, 17–19 Superintelligence, 3, 9–10 brain of, 76 as corporate product, 119 ideal of, 64, 69, 142, 156–158 motivations of, 77–80 threat/risk of, 36, 64, 79–82 Transcendence and, 68, 69 See also Computer(s) Supermachines, 10 Swarming technology, 10–11 biological basis of, 97–100 The Coming Swarm and, 98 Drone Theory and, 90–91 emergent intelligence and, 100 metaphors, use of in, 99 regressive visions of, 107 stupidity of intelligence in, 102–106 stupidity of vision in, 100–102 See also Autonomous weapons; Military; Noetic soul; War T Tadros, see Evolutionary biorobotics Tate, Greg, 160, 161, 165 Taylorism, 118 Technology advancement of, 8, 142–144 capitalist industrialization and, 72, 74, 81, 83 colonial slave, 166–167

cultural determinants, 136–137 dystopic ideas of, 145, 147–151 humans entangled with, 7, 37, 95–97, 103–105 human evolution and, 6–7, 92, 95–96, 157, 164–165 marketing future, 129–130 progress narratives on, 35, 37, 52, 55–56, 63, 65, 82, 142 race equated with, 159–161, 165, 169 seduction of, 137, 139 slavery and cybernetic, 162–164 tekhne as meaning of, 143–144 Tegmark, Max, 3, 63, 65 Tekhne, 13, 143–144, 153 Tellex, Stefanie, 130 Temerlin, Maurice Lucy: Growing Up Human, 40 Terminator, 11, 51 Thalmann, Nadia and Nadine robot, 132 Transcendence, 9–10, 63, 64, 71 idea of subjectivity in, 68–69 Transcendental subject, see Subject, the Transhumanism, 2–3, 7, 69, 70, 76, 122 See also Posthumanism; Utopianism Turing, Alan, 37, 39 Turing test engineering ideals and, 142 Ex Machina and, 135 gendered specificity of, 39 social robots and, 132, 133 Turkle, Sherry, 2 Twitter, 118 2001: A Space Odyssey, 39, 127 HAL 9000 character in, 39, 127 U Unconscious, 74 U.N. Convention on Certain Conventional Weapons, 54

 INDEX 

U.S. Air Force RPA Vector report, 97 Unmanned Aircraft Systems report, 97 Utilitarianism engineering motivations and, 146–147, 151, 155 ethics and, 123 machine behavior and, 67, 77–80 Utopianism digital, 14 dystopic narratives and, 142, 144, 147–148, 150–151 form and wish in, 150–151 singularity theory and, 155–157 techno-, 6–7, 13, 119, 122, 139 V Valkyrie robot, 47 Violence, 10, 51–55, 97, 116, 121, 168 racial, 14 Voluntary stupidity, 11, 101, 102 W WALL-E, 129, 130 War automation of, 3, 9, 10, 51, 53, 55, 89, 90, 103–104, 148 cognitive assemblages and, 105–106

187

combatants, noncombatants in, 10, 54, 101 counterinsurgency and, 102 future of, 10, 52, 89–90, 100–102, 105, 106 human noetic potential and, 102, 105–106 human regression and, 96–97 international laws governing, 54, 102 situational awareness and, 55 on terror, 102 World War I, 1, 145, 147–148 World War II, 1, 13, 99, 145, 149 See also Autonomous weapons; Swarming technology; Technology Weiner, Norbert, 14, 165 ethics of cybernetics, 162–164 Human Use of Human Beings, The, 159, 161 Wells, H.G. War of the Worlds, 169 Wilczek, F., 63 Woolf, Virginia, 150 Wozniak, Steve, 51 Wynter, Sylvia, 162, 171 universal human and, 166 Y Yanacek, Holly, 143