Materializing Digital Futures: Touch, Movement, Sound and Vision 9781501361258, 9781501361289, 9781501361272

Table of contents :
Cover
Half Title
Title
Copyright
Dedication
Contents
Figures
Contributors
Acknowledgements
Introduction
SECTION one Socio-aesthetics of sound and sight
1 Virtual reality, the chiasm and the doubled body (Angela Ndalianis)
2 Sensing Sims: Atmospheres, aesthetics and the cyborg player (Merlin Seller)
3 Embodied audiovisual experience: The role of sound in contemporary screen and digital media (Darrin Verhagen and Ben Byrne)
4 Volumetric Black: Post-cinematic Blackness (Triton Mobley)
SECTION two Meaning-making in the data-driven era
5 Quantified me: Curatorial lives and the pixelated spectre of self (Toija Cinque)
6 Virtual reality and kinaesthetic connection: Qualities of ‘being there’ (Kim Vincs)
7 Feminist memes: Digital communities, identity performance and resistance from the shadows (Shana MacDonald and Brianna I. W
8 The infinite portrait: A case of post-human authorship (Andrew McIntyre)
SECTION three Touch, body, metal, screen
9 First encounters with robots through embodied observation, imagined narrative and choreography (Amy LaViers)
10 Physical digitality: Making reality visible through multimodal digital affordances for human perception (Luke Heemsbergen,
11 A true feel: Re-embodying the touch sense in the digital fashion experience ( Michela Ornati)
12 What robots learn from performative relationships and interactive performance (Steph Hutchison and John McCormick)
SECTION four Digital futures
13 Smart home: Smart devices and the everyday experiences of the home (Xi Cui)
14 Affect and the digitalization of war (John MacWillie)
15 Automation is a myth (Luke Munn)
16 A triadic typology of material mediation: Ontology, intentionality and vitalism (Renata Lemos Morais)
Index

Citation preview

Materializing Digital Futures

ii

Materializing Digital Futures Touch, Movement, Sound and Vision Edited by Toija Cinque and Jordan Beth Vincent

BLOOMSBURY ACADEMIC Bloomsbury Publishing Inc 1385 Broadway, New York, NY 10018, USA 50 Bedford Square, London, WC1B 3DP, UK 29 Earlsfort Terrace, Dublin 2, Ireland BLOOMSBURY, BLOOMSBURY ACADEMIC and the Diana logo are trademarks of Bloomsbury Publishing Plc First published in the United States of America 2022 Volume Editor’s part of the Work © Toija Cinque and Jordan Beth Vincent, 2022 Each chapter © of Contributors (see p. xii) For legal purposes the Acknowledgements on p. xvii constitute an extension of this copyright page. Cover image: Scanner Sombre (2017) by Introversion Software. © Mark Morris from Introversion Software All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage or retrieval system, without prior permission in writing from the publishers. Bloomsbury Publishing Inc does not have any control over, or responsibility for, any third-party websites referred to or in this book. All internet addresses given in this book were correct at the time of going to press. The author and publisher regret any inconvenience caused if addresses have changed or sites have ceased to exist, but can accept no responsibility for any such changes. Library of Congress Cataloging-in-Publication Data Names: Cinque, Toija, editor. | Vincent, Jordan Beth, editor. Title: Materializing digital futures : touch, movement, sound and vision / edited by Toija Cinque and Jordan Beth Vincent. Description: New York : Bloomsbury Academic, 2022. | Includes bibliographical references and index. | Summary: “Offers a way to re-evaluate deeply evocative futures in which humans and intelligent systems increasingly engage in symbiotically connected experiences via continuous flows of data and information exchanges”– Provided by publisher. Identifiers: LCCN 2021043558 (print) | LCCN 2021043559 (ebook) | ISBN 9781501361258 (hardback) | ISBN 9781501388088 (paperback) | ISBN 9781501361265 (epub) | ISBN 9781501361272 (pdf) | ISBN 9781501361289 (ebook other) Subjects: LCSH: Digital media–Social aspects. | Multimedia communications–Social aspects. | Human-computer interaction. Classification: LCC HM851 .M379 2022 (print) | LCC HM851 (ebook) | DDC 302.23/1–dc23/eng/20211013 LC record available at https://lccn.loc.gov/2021043558 LC ebook record available at https://lccn.loc.gov/2021043559 ISBN: HB: 978-1-5013-6125-8 ePDF: 978-1-5013-6127-2 eBook: 978-1-5013-6126-5 Typeset by Newgen KnowledgeWorks Pvt. Ltd., Chennai, India To find out more about our authors and books visit www.bloomsbury.com and sign up for our newsletters.

To my family You are the constant reminder of gentleness and beauty Marc, Surija and Oliver, my joy and inspiration For all the days I owe you – Toija Thanks to my family for their love and support Special thanks to my husband, Paul, and my children, Alice and Zach – Jordan

vi

CONTENTS

List of Figures  x List of Contributors  xii Acknowledgements  xvii

Introduction: Screen media, artefacts and intra-communication  1 Toija Cinque and Jordan Beth Vincent

Section One  Socio-aesthetics of sound and sight  7 1 Virtual reality, the chiasm and the doubled body  11 Angela Ndalianis

2 Sensing Sims: Atmospheres, aesthetics and the cyborg player  25 Merlin Seller

3 Embodied audiovisual experience: The role of sound in contemporary screen and digital media  45 Darrin Verhagen and Ben Byrne

4 Volumetric Black: Post-cinematic Blackness  61 Triton Mobley

viii

CONTENTS

Section Two  Meaning-making in the data-driven era  81 5 Quantified me: Curatorial lives and the pixelated spectre of self  87 Toija Cinque

6 Virtual reality and kinaesthetic connection: Qualities of ‘being there’  107 Kim Vincs

7 Feminist memes: Digital communities, identity performance and resistance from the shadows  123 Shana MacDonald and Brianna I. Wiens

8 The infinite portrait: A case of post-human authorship  141 Andrew McIntyre

Section Three  Touch, body, metal, screen  163 9 First encounters with robots through embodied observation, imagined narrative and choreography  169 Amy LaViers

10 Physical digitality: Making reality visible through multimodal digital affordances for human perception  187 Luke Heemsbergen, Greg Bowtell and Jordan Beth Vincent

11 A true feel: Re-embodying the touch sense in the digital fashion experience  205 Michela Ornati

12 What robots learn from performative relationships and interactive performance  223 Steph Hutchison and John McCormick

CONTENTS

Section Four  Digital futures  243 13 Smart home: Smart devices and the everyday experiences of the home  249 Xi Cui

14 Affect and the digitalization of war  269 John MacWillie

15 Automation is a myth  289 Luke Munn

16 A triadic typology of material mediation: Ontology, intentionality and vitalism  305 Renata Lemos Morais Index  321

ix

FIGURES

2.1 2.2 5.1 5.2 6.1 6.2 6.3 7.1 .2 7 7.3 7.4 7.5 8.1 8.2 8.3 8.4 8.5 .6 8 9.1 9.2 .3 9 9.4 9.5

Unfinished Swan from Giant Sparrow (2012)   32 Scanner Sombre (2017)  35 Mediation on the network of networks  94 Communication habits  97 The layered design of the space   115 The Crack Up (2014)  116 Choreotopography (2010)  117 Black and white image of a witch and the witch’s ghouls in the background   131 Tweet by soulshine_tarot – Twitter account   133 Image posted on Instagram by illustrator @crucifix.vi   135 Image posted on Instagram on the @thehoodwitch and @bluecollards accounts   136 A post from @thehoodwitch account that accompanies the image in Figure 7.4  137 Portrait d’Edmond de Belamy  142 Memories of Passersby I   144 An approximation of a GAN program  145 Three-dimensional plot of Edward Lorenz’s ‘strange attractor’  148 Successive screenshots displaying the changing imagery produced by the GAN used for Memories of Passersby I  149 Approximation of the aesthetic universe  154 Performance of ‘A Machine’   176 Symbols from the Laban/Bartenieff Movement System commonly used in Motif  179 NAO hardware with movement design  180 Robotic platform  181 Adult learners presenting a robot design and purpose to a larger group  182

FIGURES

12.1 12.2 12.3 12.4

Emergence  226 Pinoke Kneeling  228 Robot Margaret  230 Eve of Dust  232

xi

CONTRIBUTORS

Greg Bowtell is Lecturer in Games and Virtual Reality working in the School of IT at Deakin University, Australia. Greg’s research focus includes game design, immersive learning experience design and evaluation, co-design methods, accessibility and affordance of XR-based applications and experimental systems for education and performance. Greg has a strong industry focus, having worked with a diverse range of industry partners in health, education and creative domains including the Australian Broadcasting Corporation (ABC), City of Melbourne, Peter MacCallum Cancer Centre, Australian Marine Parks and the Australian National Academy of Music. Ben Byrne is Program Manager and Senior Lecturer (Digital Media) at RMIT University, Melbourne, Australia. He is a listener, curator and educator experimenting with digital sound, art and design while living through big  tech and climate crises. He is the founder and curator of Avantwhatever,  a  platform for experimental sonic practice. He recently published ‘Directing Listening: Sound Design Methods from Film to SiteResponsive Sonic Art’ in The Bloomsbury Handbook of Sonic Methodologies (Bloomsbury, 2021). Toija Cinque is Associate Professor in Communication (Digital Media) in the School of Communication and Creative Arts at Deakin University, Australia. Cinque uses research methodologies for creative design drawn from multidiscipline integration including playful learning applications and experiences that address real-world problems from digital literacies, privacy and surveillance, to science communication. Published works include: Digital Media Ecologies (forthcoming), Changing Media Landscapes: Visual Networking (2015), Communication, Digital Media and Everyday Life, 2nd edn (co-authored, 2015) and New Media in Everyday Life (co-authored, 2012). Xi Cui is Associate Professor of Communication Studies at the College of Charleston in South Carolina, USA. His research is concerned with the ritualistic aspects of the media and media uses, and their cultural implications on the social construction of reality. He is particularly interested in how

CONTRIBUTORS

xiii

both media contents and media use construct a sense of normalcy in the times of social or technological disruption. His studies have been published in numerous journals including Journalism, Mass Media & Society and International Journal of Communication. Luke Heemsbergen is an Early Career Researcher in Deakin University, Australia. He engages emerging forms of the sociopolitical visibility afforded by digital communications and studies their impact. This includes substantive focus on the evolving interfaces of the digital-material world like 3D printing and augmented reality. He has published over 15 peer-reviewed papers, a majority within top quartile international journals. His research has been drawn on in national (ABC, Nine) and international (New York Times, Wired) media, and his first monograph is forthcoming in 2021. Steph Hutchison is a choreographer, performer and artist-researcher. Her practice is driven by dance improvisation and collaboration with motion capture, animation, robotics, haptics, extended reality and artificially intelligent performance agents. Steph is currently a dance academic and research ethics advisor at the Queensland University of Technology (QUT), Australia. At QUT Steph teaches contemporary dance, choreographic practice, screen dance and interdisciplinary collaboration. Amy LaViers is Director of the Robotics, Automation and Dance (RAD) Lab. Her research interests focus on the intersection between dance and robotics. She has published scholarly work and presented live choreography and installations at this intersection for over a decade. She is currently writing her first authored manuscript with fellow Certified Movement Analyst Catherine Maguire, tentatively titled Making Meaning with Machines. Shana MacDonald is Associate Professor in Communication Arts at the University of Waterloo, Canada, and the current President of the Film Studies Association of Canada. Her interdisciplinary research examines intersectional feminism within social media, popular culture, cinema, performance and public art. MacDonald is Director of the qcollaborative, a feminist design lab that develops new forms of relationality through technologies of public performance, where she co-runs the online archive Feminists Do Media (Instagram: @aesthetic.resistance). She has published in Feminist Media Histories, Media Theory Journal, Feminist Media Studies and is Lead Editor in the forthcoming book Networked Feminist Activisms (Lexington Press). John McCormick is a technology-based artist with a major interest in movement. John has collaborated on works worldwide, including at International Symposium of Electronic Art (ISEA), ZERO1SJ, SIGGRAPH,

xiv

CONTRIBUTORS

Melbourne Festival, Venice Biennale, Siggraph Asia, Ars Electronica Futurelab, Tokyo International Forum and Art Science Museum Singapore. John was a 2020 Australian Antarctic Arts Fellow. He is currently a lecturer and researcher in the Centre for Transformative Media Technologies at Swinburne University of Technology, Australia, where he investigates artistic practice in mixed reality environments, robotics, artificial intelligence and human action. Andrew McIntyre is a doctoral student in the Film and Television Studies Department at the University of Glasgow, UK. Funded through the UK Arts and Humanities Research Council and the Scottish Graduate School for Arts and Humanities, his research on artificial intelligence art focuses on authorship theory, post-humanist philosophy and the media theory of Vilém Flusser. John MacWillie is Principal Researcher at the Leviathan Project, a study of the US national security establishment (armed forces, intelligence agencies and law enforcement). He retired from the graduate media studies program at California State University, East Bay. Before teaching, he worked for twenty-five years as a senior executive in the information security industry and ten years in law enforcement administration in New York City. He received his PhD in media studies at the University of Leeds. His most recent publications are Noise and Surveillance: Object, Event, Limit (forthcoming), The Technics of a Gnostic World: An Ontogeny of Big Data (2020). Triton Mobley is Professor of Practice at the School of the Museum of Fine Arts at Tufts University, USA. He is a new media practitioner and researcher. His guerrilla praxis and computational interventions have been presented at CURRENTS Virtual Festival, Tokyo Geidai’s Games Online, Harvard’s (IM)POSSIBILITY conference, University of Maryland’s Intentionally Digital, Intentionally Black conference, and Art Machines: Computational Media Art Symposium at City University of Hong Kong. Triton’s research is framed within architectures of digital perceptions and cultural optics – problematizing notions of being – situated between the discontinuities of emergent technologies and communities of marginality. Renata Lemos Morais is Associate Professor of Digital Design at the Dubai Institute of Design and Innovation (DIDI). Her approach to research is transdisciplinary and its current focus explores the intersections between design, psychology, computation and data systems. She has published three books and various journal articles on technological aesthetics and mediation. Luke Munn is an emerging media studies scholar based in Tāmaki Makaurau Auckland, Aotearoa New Zealand. His research investigates the sociocultural impacts of digital cultures and their broader intersections

CONTRIBUTORS

xv

with race, politics and the environment. His previous studies have ranged from Hong Kong protest to Uber labour and far-right radicalization, and have been featured in highly regarded journals such as Cultural Politics, Big Data & Society and Information Communication & Society. He has two monographs Logic of Feeling: Technology’s Quest to Capitalize Emotion (2020) and Ferocious Logics: Unmaking the Algorithm (2018). His work’s impact has extended beyond academia in the form of numerous interviews and references in outlets including the Guardian and the Washington Post. Angela Ndalianis is Professor and Director of the Centre for Transformative Media Technologies at Swinburne University of Technology, Australia. Her research focuses on entertainment technologies and their histories; visual effects; and the superhero, horror and science fiction genres. She has published numerous articles in journals and anthologies, and some of her book publications include Neo-Baroque Aesthetics and Contemporary Entertainment (2004), Science Fiction Experiences (2010), The Horror Sensorium: Media and the Senses (2012) and the edited books The Contemporary Comic Book Superhero (2009), Baroque to NeoBaroque: Emotion and the Seduction of the Senses (co-editor, 2018) and Fans and Videogames: Histories, Fandom, Archives (co-editor, 2017). Michela Ornati is Lecturer at University of Applied Sciences and Arts of Southern Switzerland (SUPSI). Her research interests focus on fashion and luxury marketing, communication, digitalization and innovative technologies. Prior to entering academia, she was an executive in leading industry firms such as DMC, Mantero, Bulgari and Giorgio Armani. She is currently pursuing her PhD at Università della Svizzera italiana, Institute of Digital Technologies for Communication (ITDxC) in Lugano, Switzerland. Book publications include Beyond CRM: The Customer Experience in the Digital Era, Strategies, Best Practices, Scenarios in the Fashion and Luxury Sector (original in Italian; 2011). Merlin Seller completed their MA and MSt at the University of St Andrews and the University of Oxford respectively, obtaining their doctoral thesis at the University of East Anglia, concerning intermedium works between film, photography and painting. With a background in Art History and Visual Studies, and teaching experience in game design, they are currently Lecturer in Design and Screen Cultures at the University of Edinburgh, working across Film, Media and Game Studies. Their present research interests concern (Post)Phenomenology, Horror and the Non-Human. Darrin Verhagen is Associate Professor in Digital Media at RMIT University, Melbourne, Australia. He is an award-winning composer and sound designer for theatre, dance, screen and installation. He is also the Director

xvi

CONTRIBUTORS

of the Audiokientic Experiments (AkE) Lab, with his most recent research focusing on multisensory music experiences for dementia. Jordan Beth Vincent is Executive Manager/Head of Sales and Strategy for Fika Entertainment (www.fikaent.com). Jordan’s interdisciplinary research profile bridges technology (including HCI/human–computer interfaces/ motion capture), creative arts (film/television, animation) and performing arts (dance, dance history). She has led industry research projects with partners including, the City of Melbourne, Peter MacCallum Cancer Centre, Dementia Australia, the Australian National Academy of Music and Ozpix Entertainment, among others. Her research collaborations in the creative industries (cultural labour, gender research) have included co-authoring industry reports for the Ontario Arts Council, the Department of Foreign Affairs and Trade, and VicHealth. Jordan is Chief Investigator on the ARC Industrial Transformation Hub for Digital Enhanced Living. Kim Vincs is a professor and leading researcher in the creative arts, with six Australian Research Council grants, over fifty industry partnerships, and over thirty arts/science collaborations across motion capture, game development, robotics, haptics, app design, 3D stereoscopy, artificial intelligence, virtual reality, augmented reality, cognitive psychology, biomechanics, mathematics, architecture and exercise science. Vincs’s industry partnerships include national and international companies such as Autodesk, Motion Analysis, Act3animation, Iloura, Alt.vfx, Arts Access Victoria, Victorian Opera and Australian Dance Theatre. She has commercial motion capture credits including the Cannes Silver Lion winning Nocturnal Migration. Brianna I. Wiens (she/her) is a postdoctoral fellow at the University of Waterloo, Canada, and Director of the qcollaborative, a critical feminist design lab. Her SSHRC-funded dissertation research draws on her experience as a mixed-race queer activist-scholar to analyse and develop methods and practices for digital activism, focusing on intersectional feminist affective and material small data approaches to hashtag activism. This research was recently awarded a Provost Dissertation Scholarship. Wiens’s collaborative work has recently appeared in Feminist Media Studies, Digital Studies/ Le Champ Numériqe and Leisure Sciences, and they are co-editor of the forthcoming collection Networked Feminist Activisms (2021).

ACKNOWLEDGEMENTS

We wish to thank the authors for their valuable contributions, the industry participants for their expertise and our many thanks also to those that have read and reviewed versions of this book for their considered and detailed feedback. Your time is greatly appreciated and the chapters stronger for them. A final heartfelt word of thanks to Katie Gallof at Bloomsbury for their ongoing support.

xviii

Introduction Screen media, artefacts and intra-communication Toija Cinque and Jordan Beth Vincent

Visual media with growing affordances using augmented reality (AR) and virtual reality (VR) are found in most aspects of everyday life in workplaces and on household devices – computer and digital television screens, appliances such as refrigerators and home assistants, applications for social media and gaming. These affordances are facilitated by touch surfaces such as iPads and iPhones, which are increasingly ubiquitous in an evolving global technological climate. Personal and industrial data collection, data sharing and increased self-tracking practices using social media applications on mobile screen devices that are linked to wearable devices or recorded data from ingestible sensors are key parts of this (Klauser and Albrechtslund, 2014). Today, small mobile screens, enabled by computer networks and various networked digital technologies, make it possible for individuals, corporations and governments to accumulate, curate and distribute personal data on an unprecedented scale (see Pantzar and Ruckenstein, 2015; Reigeluth, 2014; Sonricker Hansen et al., 2012). The advent of ‘big data’ (and small data) technologies and the reach of social media have inexorably altered the boundaries between private and public life, and profoundly altered our sense of self (Whooley et al., 2014;

2

MATERIALIZING DIGITAL FUTURES

Williamson, 2015). This book considers how the former techniques of connection to community in cultural and leisure activities are reconfigured (Acquisti and Gross, 2006) through this changing landscape of digital media visibility, data agglomerations and personal engagement with an empirical digital self. In this evolving context, we find new virtual worlds forming. Each technologically enabled opportunity brings an increasingly sophisticated visual language with the act of pursuing the intra-sensorial ways of perceiving the world around us – through touch, movement, sound and vision – that is the heart of screen media use and audience engagement with unfolding, refolding, ever-changing digital artefacts. In this context, haptic sense (touch) and sound are accorded equal importance together with the visual. Emphasis is placed on all the senses that play important roles because they work together and afford thinking beyond a single image, a single sound, a single touch, a single movement, to imagine or do something else that positions us in the middle of a screen media revolution. It is also true that our digital artifacts are embedded in wider and constantly shifting ecosystems such that they become increasingly editable, interactive, reprogrammable, and distributable. This state of flux and constant transfiguration renders the value and utility of these artifacts contingent on shifting webs of functional relations with other artifacts across specific contexts and organizations. (Kallinikos et al., 2013: 357) Found in the work of Kallinikos et al. (2013) is the convincing position that sociocultural technological developments have evolved through iterative processes affording numerous opportunities for construal and interpretation – a ‘media manifold’ expanding media environment (Couldry, 2011: 220). The premise connects to this book’s overarching contention for a new perception of intra-communication in the everyday whereby digital media use emerges and meaning materializes from within various and complex sociocultural relationships and not outside of it (see Barad, 2003). We seek in this anthology to explore the forms, infrastructures and various relationships through situated critical analyses of the production, circulation and use of digital devices, screen and sound media, and systems, and suggest a teleological orientation in our contemporary sociocultural lives. This book uses the term ‘materialize’ with reference to unfolding, connecting and immersive digital media. The word has the same provenance as ‘mediatization’ and is situated in the context of ongoing vibrant mediation (Hepp, 2013). For its part, an understanding of ‘visual networking’ is useful in this context for underscoring the expressed pleasurable and functional activities across social, cultural and technical environments – including our social networks and other interpersonal communication at the level of

INTRODUCTION

3

the individual, as well as more broadly in ‘smart homes’, the management of transport, ‘smartgrids’ for power, e-government and the like (Cinque, 2015). This conceptual background is important for unpacking the future of emerging digital technologies in a book of this nature where visual and visually spectacular technology is being investigated. What the authors of this anthology do not do, however, is score the various digital media as the focal point in a way that supplants the individual. We are looking for something more meaningful with the intent to go further than to simply describe a technological system in favour of encapsulating the range of personal (human) elements that largely drives us – the perceptual senses. The authors in this anthology critically consider our sensory motivations and consequent interaction with various technologies and the choices that lie behind them. This is done with regard to how our haptic senses move us (hearing, seeing, touching), driven by active individual agency, when engaging with interconnected digital technologies across a range of everchanging platforms. The notion of individual agency in this context refers particularly to the ability of people, individually and collectively, to influence their own lives as well as the society and environment in which they live. This book’s motivation is not to simply classify any group of technologies but rather to engage meaningfully with what we are doing with them in contemporary settings. The capacity to build upon and systematically pursue the momentum of digital media studies to further address the important questions of the significance and value of sensory-based human–computer interactions (HCI) in contemporary society will allow for the development of robust conversations on ways forward. Our approach is to first introduce interested readers to the underpinnings of a swiftly materializing digital future, which is now so ubiquitous that we take it for granted. By way of concentrating research debate and critical analysis around the concept of digital media artefacts and human identity formation, this book will circumnavigate the significant implications of living in a contemporary information-based society. Towards this critical exploration of ‘the human’ in and outside the digital environment, the intention of this research is to delve into questions of whether immersive technologies have been overestimated as consumer gadgets or entertainment media. We consider the future of exhibition practices pertaining to the promises attached to ‘full immersion’ via mixed AR and VR and the extent to which they have created tensions between the technologies and physical spaces of exhibitions, museums, education institutions and the like. In this context is the further question of how the spaces between all-digital artworks and all-physical exhibition and learning spaces might be negotiated now and in the future. Will the design, marketing and use of digital applications and platforms determine the ways in which the offline and online [digital] self is formed? A key point of difference in this book, as compared to other texts in the field, is that it also looks at the practical application of digital

4

MATERIALIZING DIGITAL FUTURES

futures within an industry context. We capture how key industry players are rapidly adjusting in important ways as they address change, and hear their voices and opinions as theory moves to application in commercial and industry contexts. This book discusses the actualities and imaginaries of emerging digital technologies and aims its searchlight at their impact on society, finding important connections between the digital and the material. The book is driven to unearth and present theoretically innovative understandings of the various sociocultural implications of emerging digital technologies for individuals in their social lives. The authors in this volume present new concepts that reorient thinking towards an ontology of digital identities, not least where media and technologies are progressively demonstrating ‘signs of embodiment’ and ‘emotional understanding’ (McStay, 2018). Our goal in this book is to afford readers a strong foundation from which to facilitate critical engagement across non-binary perspectives with reference to utopian, functionalist and dystopic visions of digital futures. This book is an edited collection of sixteen original chapters that consider multiple aspects of digital media, data cultures, art platforms, and exhibition and performance spaces. The authors assembled here are positioned at the intersection of Digital Media Studies, Science and Technology Studies, Film and Television Studies, Creative Industries and Data Cultures, but uniquely and importantly drawing as well from the fields of Dance Research and Performance Studies and Arts and Cultural Management with relevance to Mediated Intelligence in Design and Architecture. Their academic background allows each of the authors to approach the question of how the future is materializing in digital forms from a range of critical entry points. The anthology is organized into four sections with four chapters per section. Each section has a detailed introduction to set the intellectual parameters for the selected chapters therein. Each chapter will assess an aspect of the section’s core theme to present the reader with the key terms and concepts, dilemmas and issues that are central to a critical understanding of digital media now. Section One, ‘Socio-aesthetics of sound and sight’, explores digital media through sound composition, sound dissemination, electronic media art, the representation of the visual image and creative process broadly conceived. This section acts to position digital media in a global context of interconnected and interrelated media ecologies. Moving to a fine-grained examination of creative work against the background of rapid data collection and sharing is Section Two, ‘Meaning-making in the data-driven era’. The chapters in this section consider in detail the collection and use of big and small data. Here, digital media is explored through nuanced accounts of the value of personal data, individualization of movement, personalization of information/data and the visualizing of data for the sociocultural affordances therein. Section Three, ‘Touch, body, metal, screen’, draws together the two

INTRODUCTION

5

threads of immersive sound and visual media from Section One and the evolving data practices highlighted in Section Two to explore performance and motion capture as a way to disrupt normal animation practices by using data in a different way. Digital media are consequently investigated through the emerging technological transformations of robotics, HCI and VR. Section Three questions the transference of ideas between machines and humans to critically appraise digitization systems and their various purposes. Section Four, ‘Digital futures’, offers a study of possible futures in the digital age through the ubiquity of computing hardware and software, satellite usage, and an advanced telecommunications infrastructure. It considers notions of being a post-industrial society among other elements. An outline of each chapter can be found in the respective section introductions. In preparing this volume we acknowledge that the relationships between research and industry are crucial for the application of knowledge in the rapidly shifting fields of emerging technology. A series of industry case studies are presented, in which industry-based organizations are invited to reflect on their integration of emerging technology and the impact it has had on their business in relation to the themes of Sections One to Four. Four different industry representatives were interviewed on each of the main themes: Ern Rose in ‘Socio-aesthetics of sound and sight’; Shaz Mohapatra in ‘Meaning-making in the data-driven era’; Amy Nelson and Astrid Scott in ‘Touch, body, metal, screen; and Joe Millward in ‘Digital futures’ (ethics approval: HAE-21-009). In this way, Materializing Digital Futures: Touch, Movement, Sound and Vision is not just a scholarly publication but one that captures key moments in this field as they relate to industry. The case studies take the form of interviews with a key industry professional and provide real-world examples of the disruption and potential of new technology. By embedding the themes of this volume within contemporary, industry-based examples, this volume enriches the field of study. With its interdisciplinary approach and its emphasis on the nexus between theory and industry application, this book builds a story of digital media’s evolving transformation and transforming capacity. It takes into consideration multiple processes in robotics and artificial intelligence (AI), VR, creative image and sound production, the representation of data and creative practice when discussing important issues around commodification, identity politics, identification and political economy for the emerging and affecting encounters and perceptions that are brought to bear. The affective phenomena of touch, movement, sound and vision are variously threaded throughout each of the four sections in the book for a deeper continuation about the affordances and implications of digital media use and what we might see, hear, feel as we move within our changing lived experiences. This rising strand of HCI studies offers a way to re-evaluate deeply evocative futures in which humans and intelligent systems increasingly engage in symbiotically connected experiences via continuous flows of data and

6

MATERIALIZING DIGITAL FUTURES

information exchanges. By its nature digital media is an interdisciplinary field and we want to draw on a number of lenses for a thorough examination that will lead to new understandings of digital media cultures.

References Acquisti, A. and R. Gross (2006), ‘Imagined Communities: Awareness, Information Sharing, and Privacy on the Facebook’, in G. Danezis and P. Golle (eds), Privacy Enhancing Technologies, PET 2006, Lecture Notes in Computer Science, Vol. 4258, 36–58, Berlin/Heidelberg: Springer. Barad, K. (2003), ‘Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter’, Signs: Journal of Women in Culture and Society, 28 (3): 801–31. Cinque, T. (2015), Changing Media Landscapes: Visual Networking, South Melbourne: Oxford University Press. Couldry, N. (2011), ‘The Necessary Future of the Audience … and How to Research It’, in V. Nightingale (ed.), The Handbook of Media Audiences, 213– 29, Oxford: Wiley-Blackwell. Hepp, A. (2013), Cultures of Mediatization, trans. K. Tribe, Cambridge: Polity Press. Kallinikos, J., A. Aaltonen and A. Marton (2013), ‘The Ambivalent Ontology of Digital Artifacts’, MIS Quarterly, 37 (2): 357–70. Klauser, F. R. and A. Albrechtslund (2014), ‘From Self-Tracking to Smart Urban Infrastructures: Towards an Interdisciplinary Research Agenda on Big Data’, Surveillance & Society, 12: 273–86. McStay, A. (2018), Emotional AI: The Rise of Empathic Media, London: Sage. Pantzar, M. and M. Ruckenstein (2015), ‘The Heart of Everyday Analytics: Emotional, Material and Practical Extensions in Self-Tracking Market’, Consumption Markets & Culture, 18: 92–109. Reigeluth, T. (2014), ‘Why Data Is Not Enough: Digital Traces as Control of Self and Self-Control’, Surveillance & Society, 12: 243–354. Sonricker, Hansen A. L., A. Li, D. Joly, S. Mekaru, J. S. Brownstein (2012), ‘Digital Surveillance: A Novel Approach to Monitoring the Illegal Wildlife Trade’, PLoS ONE, 7 (12). Whooley, M., B. Ploderer and K. Gray (2014), ‘On the Integration of Self-Tracking Data amongst Quantified Self Members’, Proceedings of the 28th International BCS Human Computer Interaction Conference on HCI 2014—Sand, Sea and Sky-Holiday HCI, BCS, 151–60. Williamson, B. (2015), ‘Algorithmic Skin: Health-Tracking Technologies, Personal Analytics and the Biopedagogies of Digitized Health and Physical Education’, Sport, Education and Society, 20: 133–51.

SECTION ONE

Socio-aesthetics of sound and sight Introduction This section establishes digital media in a global context of interconnected and interrelated media ecologies by exploring digital media through sound composition, sound dissemination, electronic media art, the representation of data and creative process broadly conceived. It positions music as affectively vital. For their insight into the affective impact of music we spoke to master audio engineer Ern Rose. Rose is predominantly a producer/engineer with a career in sound recording in the music industry stretching back to the late 1960s when he worked at Armstrong Studios – a commercial recording studio located in Melbourne, Australia. These studios grew out of Telefil Sound Recording and Film Studios, which was at the time housed in a converted cinema, subsequently becoming the largest studio complex in the southern hemisphere with five studios by 1972. After this time the company was renamed AAV (Armstrong Audio Visual). By the late 1980s and early 1990s, a staff consortium of AAV, headed by Rose, acquired the audio arm of operations of AAV, which recorded U2, Paul McCartney, Madonna, Bob Dylan and Crowded House. Rose worked directly with numerous

8

MATERIALIZING DIGITAL FUTURES

artists including Renee Geyer Band (RCA Victor), Rick Springfield (Wizard Records), Little River Band (Capital Records), Split Enz (Mushroom), Deep Purple (Thames), Kiss (Sanctuary Visual Entertainment) among multiple others. Ern Rose: You know I’m passionate about the role that music plays. Music is something that audiences hold in their memories. The recall associated with music I think is a massive aspect in remembering during our life or in our society. For example, when you relate music to film, the right piece of music can change how the audience feels or their reactions to what they see. It has the same impact in advertising or entertainment. It stirs emotions, our feelings and is a unique and powerful way to make a connection. Often you hear a song, and you can remember where you first heard it, who you were with. When an artist adds to that the value of the lyric, words about life experiences, the narrative coalesces the form. Certain music preferences and style are often ingrained generationally for what has storied people’s lives; for what one is attached to – it accompanies nostalgia. The great songwriters write our lifetimes in their lyrics. I think that we are seeing this personal connection in older people, especially through those with dementia that listen to music from their earlier years. They will hear a tune and suddenly many that never rise out of a wheelchair will get up and start dancing. Looking ahead, a form of artificial intelligence is entering the music market at the level of computer driven creation where it’s given a set of parameters – for emotions, style, timing – guidelines and data about music made over decades can be assessed for something ‘new’ that fits the exact formula. As much as a computer and artificial intelligence can look for patterns, however, it’s the human element of knowing the subtleties of culture, of society and human emotions, that is what inflects the music that cannot be fully re-produced by a machine. And there is something about experiencing music together. We are not going to move away from that, and I think seeing what Covid did to people when they were restricted during lockdown from going to live music underscores this – it was almost a form of starvation. We came up with ways to try to overcome this with home online performances and the like. Some were mediocre if not shocking, but audiences soaked them up. It is only coming back now to see and hear and just sense some live music that we appreciate this as what people have been craving. The collective experience of music is something special. Chapter 1 is situated in the context of people connecting virtually, so prevalent during Covid-19 lockdowns. Angela Ndalianis undertakes a critical discussion of ‘being in the [virtual] world’ through exploration

SOCIO-AESTHETICS OF SOUND AND SIGHT

9

of the nature of exchange that occurs between the person in real life and their virtual self. This navigation is in the context of virtual reality (VR) platforms where Ndalianis argues there is a perceptual shift occurring in the VR experience one enters that offers an alternative experience of presence. They draw on the notion of a symbolic virtual body which consequently becomes a literal virtual body in this realm complicating the nature of embodiment as the body is materially present in real life while simultaneously being present in a virtual space. The case studies for close analysis are a VR experience of Hieronymus Bosch’s painting The Garden of Earthly Delights and the VR painting application Tilt Brush (created by Drew Skillman and Patrick Hackett). The author contends that emersion in fictional spaces affect our bodies haptically through their imaginative geographies, surfaces and textures, and the materiality they convey. As with all modes of communication, a perception of a doubled body also develops in the virtual space. Here, a body in motion becomes the means by which to understand and interpret the virtual world being inhabited. In Chapter 2, Merlin Seller considers that nature of ludic gameplay that requires dedicated effort, skill and ingenuity and moves us to consider seriously how videogame aesthetics can articulate novel kinds of sense perception by taking a post-phenomenological understanding of atmospheres as ‘quasi-objective’ aesthetic objects. There is a continuum in gameplay along which all games can be placed, with ‘paidia play’ at one end and ‘ludus play’ at the other. According to this definition, paidia covers the spontaneous manifestations of the play instinct, or spontaneous or uncontrolled fantasy play – think of a child playing with an empty cardboard box and its seemingly endless possibilities for play options. This is not to say, however, that there are no rules. For example, a child pretending their cardboard box is a car will demonstrate actions of holding and turning a steering wheel and making the requisite brrrm brrrm noises. By contrast, ludus encompasses the various games which require concentrated industry and ability. Through two case studies of twenty-first-century videogames, Unfinished Swan (2012) and Scanner Sombre (2017), Seller argues that videogames offer users cyborg access to alien forms of experience, as well as new defamiliarized perspectives on sense perception, wherein each game challenges their respective players to sense hidden environments. Turning to the examination of sound in detail for Chapter 3, co-authors Darrin Verhagan and Ben Byrne explore contemporary screen and digital media for the role that sound specifically plays in embodied audiovisual experiences. Drawing on sound design, sound studies and interaction design literature, Verhagen and Byrne argue that methods of sound production for VR, augmented reality (AR), mixed reality (MR) to 4D cinema go far beyond traditional understandings of sound design and score to instead employ sound as a crucial aspect of multisensory, interactive experiences that extend beyond the screen. The authors take examples of VR experiences via

10

MATERIALIZING DIGITAL FUTURES

Collisions (2016) set in central Australia, 4D cinema and the superhero film Venom (2018) in 4DX, to the 360° documentary HOBO UFO (v. Chernobyl) (2019), an audiovisual release that employs Google Street View footage of Chernobyl. Each of these, the authors assert, employs digital technologies in aid of affective multisensory experiences in which ‘the viewer’ is explicitly spatially positioned, and the relationship of sound to movement is key such that the listener can see as much as hear sound. Triton Mobley explicitly underscores the limitations of producing Blackness in image-making technologies in Chapter 4. As Mobley states, Blackness in America (as elsewhere) has always been rendered with a veil of opaqueness over it and this results in a kind of unseeing visibility. Mobley proposes a series of radical possibilities for visible Blackness – situating their central argument between film, formal architectural infrastructure(s) and speculative design. Here they position filmic space as ‘architectural space’ that has conserved the function of reconfiguring the imaginative desires of the audience. Mobley writes that Volumetric Black creatively reimagines possibilities for digital mirroring of Black bodies as a viewing experience without computational inadequacies. The author’s aim is achieved through a jugaad combination of image-processing computations, microcontroller programming and the inclusion of RGB light emitting diodes so that images of Black bodies are produced with an illumination that appears less visually compromised on visual displays. The aesthetic reconfiguration of visual Blackness is to be found when the filmic-architectural space and cultural collapse (i.e. between audience, (image) frame and infrastructure) so that the Volumetric Black might be revealed.

1 Virtual reality, the chiasm and the doubled body Angela Ndalianis

Introduction The year 2016 was hailed as the year of virtual reality (VR). The commercial release of the Oculus Rift, HTC Vive, Google Cardboard, PlaystationVR and other VR platforms signalled a perceptual shift in our mediated experiences of alternative realities by finally bringing VR into the private sphere. I am interested in the nature of embodiment in these new forms of VR. In this chapter, I focus on representational spaces and how they express their unique articulations of what phenomenologist Maurice Merleau-Ponty called ‘the flesh of the world’ (1968: 267). I begin with Merleau-Ponty’s definition of the virtual body, which he argues comes to the fore in all modes of communication, but, specifically, I examine how he understands his relationship to the paintings of Cézanne. While I begin with painting, the main focus of this chapter is to explore and extend Merleau-Ponty’s phenomenological reflections and consider their usefulness in understanding the user’s experience with VR. I contend that, when applied to VR, MerleauPonty’s symbolic virtual body becomes a literal virtual body that complicates the nature of embodiment. In particular, I am interested in Merleau-Ponty’s discussion of the virtual body as one that occupies a space ‘in between’ – a space he calls the intertwining or chiasm. Extending this idea, through an analysis of a VR experience of Hieronymus Bosch’s painting The Garden of Earthly Delights and the VR painting application Tilt Brush, I explore

12

MATERIALIZING DIGITAL FUTURES

how the VR user occupies a space in between where an exchange occurs between the user and their virtual self; a new state of embodiment occurs, one that generates the perception of a doubled body. In examining this, I also turn to the writings of Maxine Sheets-Johnstone who elaborates on Merleau-Ponty’s ideas regarding thought and sensation being grounded in a body that moves. I argue that kinaesthetic perception becomes central to our experience of virtual space and that the body in motion becomes the means to making sense of the virtual world we inhabit.

The virtual and the virtual Bosch In August of 2016, I visited the Prado Museum in Madrid. There were so many works by Ribera, Goya, Velasquez and others that I wanted to see, but it was in the Hieronymus Bosch rooms that I experienced something I had never experienced before. To this day I am not quite sure what happened to me. I found myself standing in front of the amazing triptych The Garden of Earthly Delights (ca.1490–1505) and becoming overwhelmed with intense emotion. I have become teary before when confronted with works of art, but never like this. All I can recall of my initial response was registering the colour pink, and the floodgate of tears pouring down my face. As my eyes began to move around the painting, taking in more colours, the multiple actions that animated the landscape, and the bizarre otherworldly creatures, a fresh flood of uncontrollable tears gushed forward. What was especially telling was that I felt my body heaving and shaking, totally overwhelmed by intense emotion, and as my gaze shifted to new parts of the painting, the onslaught of raw, gut-level sensations would start all over again. My reaction was visceral and had little to do with the fact that I was aware that I was standing in front of a great masterpiece. Architectural theorist Juhani Pallasmaa notes, ‘Michelangelo himself argued that everything in art and architecture arises from the human body’ (2014: 82–3), and this observation is true not only of the artist who creates the work but of the person who experiences it. Perhaps, as I stood in front of The Garden of Earthly Delights, and as my entire body shook with emotion, I was granted access to Bosch’s marvellous imagination – and his capacity to conjure and give me access to the fantastic worlds of heaven and hell. The atmosphere and textures of the space created – the lush greens of the landscape, the soft pinks of the imaginary architecture and objects, the soft blue hues of the sky and water, and the dark spaces ignited by hellish fires – all vibrated with intensity across my body, sending messages that transcended cognitive understandings about scripture and the fates that befall humans in Heaven and Hell. In the same year that I saw the Bosch painting ‘in the flesh’, I also experienced a virtual version of The Garden of Earthly Delights, designed

VIRTUAL REALITY, THE CHIASM, THE DOUBLED BODY

13

by BDH Immersive to mark the 500th year of Bosch’s death and to celebrate the painting The Garden of Earthly Delights. The experience was – and still is – available for download on the iPhone and Android and can be viewed through Google Cardboard, or other VR systems (Bosch VR, 2016). In the VR experience, we appear to occupy the same space as the reimagined Garden of Earthly Delights, and its narrative vignettes become animated in space, allowing us to travel around and through them. The Bosch VR was created by using stereoscopy to transform the 2D image of the painting to create the illusion of 3D depth in VR. Donning a VR viewer, and through the movement of our head/vision, not only allows the wearer access to the painting in depth but also activates motion that send the viewer on a journey through the landscapes of Heaven, Hell and Paradise. Our ability to interact with the space is minimal and limited to our vision, which flies through the spaces and past objects. The two different experiences of the same work of art propelled me to ponder on how my body and mind responded to both, leading me to the question that concerns me in this chapter: despite their ephemerality, do examples of VR have their own material presence – one that can affect participants somatically in similar or, indeed, different ways? In my efforts to make sense of these two unique experiences – painting and VR – I begin with the writings of Maurice Merleau-Ponty who, in Phenomenology of Perception, deliberates on his embodied response to the art of Paul Cézanne (2002: 174). In attempting to explain his experience, he refers to an experiment performed by Gestalt psychologist Max Wertheimer, in which a subject looks at himself and the space around him in a mirror tilted 45° vertically. Merleau-Ponty states: After a few minutes, provided that he [sic] does not strengthen his initial anchorage by glancing away from the mirror, the reflected room miraculously calls up a subject capable of living in it. This virtual body ousts the real one to such an extent that the subject no longer has the feeling of being in the world where he actually is, and that instead of his real legs and arms, he feels that he has the legs and arms he would need to walk and act in the reflected room: he inhabits the spectacle. The spatial level tilts and takes up its new position. (2002: 291) Martin Pulido explains that while the ‘actual body plays a necessary part in establishing the spatial level’ of the room, the virtual body remains ‘an open field of possible corporeal actions … The virtual body accepts these coordinates, and in so doing adopts the virtual space of the tilted room and its possible situations’ (2010: 33–4). There are two key points here. The first is Merleau-Ponty’s observation that the deceptive reflection in front of the mirror can result in a ‘virtual body that ousts the real one ... and inhabits the spectacle’. The second, extends this virtual body to consider how the ‘flesh of the body’ responds to art – specifically, the paintings of Cézanne.

14

MATERIALIZING DIGITAL FUTURES

Merleau-Ponty’s virtual body is found in the state of flux that exists in any communication, the point between meaning being conveyed but not yet fully being fixed, and the point where it is finally fixed and finds a place in the world, in our bodies and in the domain he calls the ‘actual’. This second explanation of the virtual extends to esoteric issues and deep thinking. When confronted by painting, or ‘literature, music, the passions’, the visible world opens up to another form of reflection that is ‘the exploration of an invisible and the disclosure of a universe of ideas’ (1968: 149). Like the reflection in the mirror, a dialogue opens up, but in this case it is a philosophical one that is filled with the possibility of ideas that fold back and forth between spectator and painting. Regarding both points – the reflection in the mirror and the reflection on Cézanne’s painting – it is Merleau-Ponty’s discussion of the virtual body in occupying the space in between that interests me: what Merleau-Ponty refers to in The Visible and the Invisible as the intertwining or chiasm (which I will return to later). It is where the visible (the body grounded in the world) interacts with the ‘invisible’ (through the virtual body) and a ‘crisscrossing’, or chiasm, occurs (1968: 150). What I find powerful about Merleau-Ponty’s ruminations on the virtual body is how they translate more literally to our relationship to virtual technology. My experience of Bosch’s painting took my virtual body to a realm that transcended language; an intertwining occurred between me and the work that offered multiple sensations that were filled with possibilities of meaning that refused, to this day, to be fixed. The same could be said of my response to the Bosch VR: this time, my senses reacted not to the material presence of the 500-year-old painting, but to the technology and its illusionistic capacity to place me within a wondrous digital world that produced a different kind of material presence – one where it felt as if I had entered the space occupied by Bosch’s fantastic characters and their surreal world. My delight came from seeing some of the objects and characters – the giant ears that hold a plunging knife between them, the birds floating on the water, the birdman wearing a pot for a hat swallowing a naked body – animate as my virtual self approached them. The Bosch VR presented its own unique reality, one that became entwined with my body, inviting me to consider the ‘universe of ideas’ that erupted forth through the transformation of the painted work into a virtual experience. Yet our experience of the Bosch VR also evokes the example of Wertheimer’s mirror illusion, but now it is the VR apparatus that replaces the mirror. In the Bosch VR we appear to enter the painting where our virtual body is represented by our vision, which floats and moves in a 3D space that appears like a series of stage flats that produce a sense of being in front of and behind objects. Despite the evocation of light, shade and colour that normally create volume and spatial recession, planes and surfaces are perceived as flat, something that is very typical of stereoscopic vision, which Bosch VR relies on. An exchange occurs between me and

VIRTUAL REALITY, THE CHIASM, THE DOUBLED BODY

15

what I perceive. The ‘crisscrossing’ that Merleau-Ponty refers to does not now relate to conceptual or philosophical thoughts but rather to a more grounded dialogue between my actual, physical body and the virtual body I navigate through the VR space. I return to Merleau-Ponty’s words: while immersed in the virtual Garden of Earthly Delights, ‘this virtual body ousts the real one to such an extent that the subject no longer has the feeling of being in the world where he actually is’ (2002: 291). As I explain in the following section, however, this escape to the other world is dependent on the viewer not strengthening ‘his initial anchorage by glancing away from the mirror’. In the case of Bosch VR – and VR in general – this occurs when the user is made aware of their physical location in actual space and the technology that is mediating the perception of immersion in a virtual world.

Tilt Brush, haptics and kinaesthetics To further explore this idea of ‘being in the [virtual] world’, I turn now to a VR example whose first incarnation was specifically for VR, and which therefore offers more intense immersive and interactive capabilities than Bosch VR. The VR example is Tilt Brush (available on Oculus Rift and HTC Vive), which is an application by Google released in April 2016 (created by Drew Skillman and Patrick Hackett) that allows you to paint in VR. I began my adventure in Tilt Brush by taking baby steps. My mind and body communicated with the program in an exchange that went back and forth: What will I draw? How do I select a line for my brush? How do I select a colour? Can I create textures and depth? Furthermore, as I inserted what would become the stem of a flower into this dark space, I became aware that the movement of my body in real space – leaning low to begin a stem that was almost my height – translated into the shape of the virtual stem, and then it’s leaves, coming to life in the virtual world. I was surprised at how my body in VR could navigate the space, moving back and forward, up and down, left and right and rotating 360° around my emerging flower. Using the controllers, I accessed a virtual palette with my left hand, which offered a range of brush strokes, textures, brush sizes, visual effects and animations. And as my body moved in real space, translating its motions into the virtual space, I gave birth to my creation: a single red flower and its vibrant green stem and leaves. What also became clear as I painted was that this flower was both a virtual painting and a virtual sculpture that I could move around. This was spatial art that required my touch and movement across and within two spaces so that it could come into being. My flower was given shape because of the dialogue that occurred in the intertwining of the actual and virtual body. It was a dialogue that required my actions, my gestures and my movements.

16

MATERIALIZING DIGITAL FUTURES

Giuliana Bruno’s phenomenologically informed examination of the film experience in her book Atlas of Emotion is useful in thinking through the relationship between vision, movement and touch in Tilt Brush. Bruno describes the film experience as ‘geopsychic architexture’ (2002: 4) – the textures of architectures and surfaces that impact the mind and body; not only are films considered to be geopsychic journeys that trigger affective responses in their audiences, but they are also modes of travel that take the spectator to fictional places. The spaces affect our bodies haptically through their imaginative geographies, surfaces and textures, and the materiality they convey. When we watch a film, the spaces ‘in there’ impact on our bodies and the space ‘out here’. This experience is even more intense and literal when using Tilt Brush. Here, not only do users create the geopsychic architexture from a blank space (we enter a blank, dark space when we start the program and only the virtual controllers are visible), they also take an active/creator role in creating and simultaneously inhabiting a space that seems to occlude the ‘out here’. Drawing on the art historical writings of Alois Reigl, Bruno understands the haptic as not being limited to the sense of touch alone but also connected to vision or, what she calls, the ‘haptic eye’. What we see through our eyes, synaesthetically triggers the sense of touch so that the film experience becomes a ‘haptic way of site-seeing’ that creates ‘the reciprocal contact between us and the environment’ (Bruno, 2002: 6). Sight is intertwined with touch and to see is to also become bodily immersed in the site of the film space: its textures, the atmosphere it creates and the spatial relationships it establishes – what Bruno calls ‘a touching experience of feeling through the eye’ (2002: 296). Furthermore, haptics works hand in hand with kinaesthetics, whereby our perception of onscreen kinetics has the potential to give way to a ‘type of kinaesthetics’ that activates the senses and creates for the ‘viewer’ an offscreen response to the onscreen space. Applying this idea of a kinaesthetics aligned with haptics/vision to VR, Bruno’s description – like Merleau-Ponty’s – takes on a more literal meaning. Unlike film, in Tilt Brush there is no visible on and off screen – the perception is that we become part of the onscreen space. Our interaction with the technology, in fact, creates the contents of the onscreen space which we can than move around and interact with. As a result, there is little ‘site-seeing’ and more ‘site-action’. Tilt Brush demands that we move our hands, arms and bodies and while we may not actually be able to touch the surfaces we create (yet), we do touch the controllers that mediate our connection to the virtual space. Kasper Levin reminds us that ‘immanent to Merleau-Ponty’s embodied phenomenology in dialogue with art there is a radical consequence in thinking about movement’ (2016: 184). Maxine Sheets-Johnstone elaborates on Merleau-Ponty’s ideas regarding thought and sensation being grounded in a body that moves. In her book, The Primacy of Movement, she argues that rather than exploring the body as an object in motion ‘we should investigate

VIRTUAL REALITY, THE CHIASM, THE DOUBLED BODY

17

self-movement through our direct access to it in the introspective experiences of kinesthesia ... we discover that movement is the originating ground of our sense-making’ (Levin, 2016: 188; Sheets-Johnstone, 2011: 139). Returning to Merleau-Ponty’s thoughts on Cézanne, she observes, ‘Merleau-Ponty’s remark that Cezanne’s description of himself as “thinking in painting” is a description of a process in which “vision becomes gesture” ... perception is interlaced with movement’ (Sheets-Johnstone, 2011: 428). I felt this explicitly in Tilt Brush, where the vision I had in my mind of my flower was finally envisioned through my gestures. Sheets-Johnstone argues that it becomes ‘impossible to separate out where perception begins and movement ends or where movement begins and perception ends’ (2011: 429). This exchange became more palpable in Tilt Brush, where I was caught in a loop between movement, gesture and sensemaking as I struggled to give expression to my simple flower through the dialogue that occurred between the virtual machine I handled and the world I created with it. Sheets-Johnstone continues her discussion by approximating the act of painting to that of dancing and the movements of a choreographer’s body, explaining that the ‘choreographed form evolves spontaneously from the ongoing process of thinking in movement’ (2011: 429). It is to this ‘ongoing process of thinking in movement’ I now turn, to examine it further through Merleau-Ponty’s concept of intertwining and the ‘chiasm’.

Intertwining and the chiasm In a video recorded for a ‘Future of Storytelling’ summit in 2015, Disney artist Glen Keane – the animator of The Little Mermaid, Tarzan, Beauty and the Beast and many other Disney classics – was invited to be filmed using Tilt Brush (Keane 2015) where he drew some of his famous characters life size in VR. In the narration he explains how surprised he was at the intense physicality involved in animating in VR. He states that when he draws on paper, he wishes the flatness of the paper ‘could go away’, but when in VR ‘I can put goggles on and I just step into the paper and now I’m drawing in it’ and ‘the edges of the paper are no longer there’. In the video he goes on to say, ‘When you draw, you’re expressing something that’s real and visceral. By making a line you’re making a seismograph of your soul’ (Keane 2015) and this experience becomes all the more intense in his experience with his characters in VR. For Keane, the VR experience is all the more real and visceral because he can move freely in the space and between the characters he is creating. Keane’s thrill at being in the presence of his life-size Beast was tangible. Drawing, he explains, becomes like a sculptural drawing that is also like dancing (see also Ulrich-Verderber, 2020). As Keane choreographs his movements in actual space – through sweeping, bodily motions and wide, open gestures – to the spectator he does indeed look like a dancer who thinks in movement. It is when we gain access to his virtual space, where

18

MATERIALIZING DIGITAL FUTURES

his virtual body gives shape to his creations – Ariel and the Beast – that we become more fully exposed to the creation in process. Something powerful has occurred in the space ‘in between’. Quoting John Dewey’s Art as Experience (2005 [1934]), Pallasmaa explains, ‘All works of art, in fact, exist simultaneously in two realms, those of physical matter and of mental imagery. This dual existence and double focus is fundamental to the mental essence of art’ (2014: 83). But in the case of VR, a different duality emerges: a perception of dual embodiment. My actual body is in real space, while my virtual body is in virtual space. VR introduces new sensory, embodied and material experiences that humans have never experienced before. This audiovisual medium opens up perceptual systems that differ from the medium of painting – and even from cinema and videogames, which VR has more in common with given the emphasis on the moving image. Returning to Keane, we see his moving body occupy two spaces – one actual and one virtual – and his narration reveals his fascination with the process of sense-making that is taking place as it tries to make sense of this duality. This mental movement is what MerleauPonty defines as the chiasm – from the Greek χίασμα, meaning ‘crossing’. This chiasm, which he also calls an intertwining, is in a state of motion, back and forth, as an exchange of communication occurs between the actual body, in this case that of Keane, and the virtual body, in this case, Keane’s virtual body as he thinks through and gives shape to his creations. Where Sheets-Johnstone’s choreographer’s body creates kinetic meanings through their movements, Keane’s kinetic movements are transferred into virtual space and are given concrete visual expression. In Chapter 4 of his unfinished book The Visible and the Invisible, in the chapter appropriately titled ‘The Intertwining – The Chiasm’, MerleauPonty outlines the complex interweaving of communication that occurs in any relationship between self and other, including things that exist in the world we inhabit. He states: The look, we said, envelops, palpates, espouses the visible things ... Between the exploration and what it will teach me, between my movements and what I touch, there must exist some relationship by principle, some kinship, according to which they are not only ... vague and ephemeral deformations of the corporeal space, but the initiation to and the opening upon a tactile world. This can happen only if my hand, while it is felt from within, is also accessible from without, itself tangible, for my other hand, for example, if it takes its place among the things it touches, is in a sense one of them, opens finally upon a tangible being of which it is also a part. (Merleau-Ponty, 1968: 133) While Merleau-Ponty may not have written this in response to virtual technology and perceptual responses to it, the passage nevertheless lends

VIRTUAL REALITY, THE CHIASM, THE DOUBLED BODY

19

itself to an analysis of the communication that occurs in this ‘in between’ where a ‘crisscrossing’ occurs between actual and virtual bodies. For example, on entering Tilt Brush, Keane’s body transferred its internal thoughts about how he creates drawings on paper, to the virtual apparatus where he was required – as was I – to think about how to use controllers and the virtual palette to create differently. Merleau-Ponty stresses the ‘reversibility that defines the flesh’ in any process of communication, where we weave ‘relations between bodies’ (1968: 144) Here, the relation formed is in an ‘invisible’ space, the space of the chiasm, where meaning and comprehension are in flux because they are in a state of process. By moving, gesturing and touching in the actual world, Keane initiates this process, thereby requiring his body to also move and touch in new ways in the virtual world – while always trying to make sense of the relationship between the two worlds. As MerleauPonty explains, ‘Through this crisscrossing within it of the touching and the tangible, its own movements incorporate themselves into the universe they interrogate, are recorded on the same map as it; the two systems are applied upon one another, as the two halves of an orange’ (1968: 134). Again, this plays out literally in VR where what we touch (controllers) and how we use them forces the user to interrogate the relationship between the two worlds and how one gives meaning and shape to the other. Katherine N. Hayles arrives at similar conclusions in her examination of the relationship between the ‘mindbody’ and virtual environments. While not following the path of Merleau-Ponty’s chiasm, she nevertheless adopts a similar idea when stressing the central role performed by relationality in this relationship. Hayles begins by questioning the dualistic thinking she presented in her book How We Became Posthuman (1999: xi), where she argued cybernetics ushered in ‘the erasure of embodiment’. Instead, she proposes ‘the idea of relation’ that involves a ‘dynamic flux’ and process of interaction between entities such as the human and the virtual (2002: 298). Discussing videogames, for example, she notes that players project ‘their proprioceptive sense into the simulated space of the game world; in fact, they eloquently insist that being a good player absolutely requires this kind of projection. Their body boundaries have fluidly intermingled with the technological affordances so that they feel the joystick as an unconscious extension of the hand’ (2002: 300). Adopting the term ‘mindbody’ (from Mark Hansen) in order to avoid the trap of earlier thinking about virtual technologies that emphasised a mind–body split, Hayles asserts that ‘the mindbody is always understood as a two-way relation, a feedback loop between biologically evolved capabilities and a richly engineered technological environment’ (2002: 303). Keane and I were in this relational feedback loop between our mindbodies and virtual space that was mediated by VR technology and the Tilt Brush program. And in the process, communication registered both ways. I left my imprint on the VR world by (quasi)mastering its program and tools to produce my flower, but the actions of my VR self also left an

20

MATERIALIZING DIGITAL FUTURES

imprint on actual me, teaching me how to navigate the world of Tilt Brush more effectively. As Merleau-Ponty elaborates: The thickness of flesh between the seer and the thing is constitutive for the thing of its visibility as for the seer of his corporeity; it is not an obstacle between them, it is their means of communication ... There is double and crossed situating of the visible in the tangible and of the tangible in the visible; the two maps are complete, and yet they do not merge into one. (1968: 135; my emphasis) What Merleau-Ponty suggests here is a doubling that occurs between the seer and thing that is seen, which is made possible through chiasmic intertwining. This facilitates an exchange of ideas or ‘deeper thinking’, but this doubling is never actualized in the real world (even though the seer and thing have been transformed in the process). Creating objects and spaces in Tilt Brush made this intertwining process come alive: I was aware of the communication happening between my dual bodies and this became less burdensome and more fluid as I became more adept at using the technology. As Bruno explains, ‘Bodies in space design spatial fields, which, in turn, design corporealities’ (2002: 64). The distinction here, however, is that in VR I feel as if I have two corporealities.

Tilt Brush and the doubled body in virtual reality From a neurological perspective, the dual body phenomenon has been confirmed through a number of experiments – initially the rubber hand illusion, which was later applied to the full body – which confirmed that individuals could be convinced that their bodies existed at two separate locations (see Heydrich et al., 2013). In two experiments performed in 2013, Lukas Heydrich et al. investigated how the sense of self-location and self-identification were affected when participants witnessed a virtual body in two scenarios, one that was virtual reality (Study 1) and the other in 3D (Study 2). Their conclusions were that participants ‘can experience the sensation of having more than one body (… Study 1) and identify with the two bodies at the same time (… Study 2)’ (2013: 13). The VR experiment also produced a greater sense of double body ownership when the computergenerated imagery (CGI) offered a high-quality realism, which created a greater sense of ‘presence in the virtual space’ (13). Perhaps the one medium most useful for understanding embodiment in VR is the videogame. In videogames, entire environments are built and we, as players, traverse them, interact with them, and emotionally and haptically

VIRTUAL REALITY, THE CHIASM, THE DOUBLED BODY

21

engage with their spaces. At the core of any understanding of game haptics is the process of interactivity (problematic as the term may be). The word ‘interactivity’ is not simply about technological mediation, it carries with it a kinaesthetic and haptic power that becomes even more intense when considered in relation to VR. As Diane Carr explains in discussing the role of the avatar in gaming, Given the role performed by avatars it is arguable that any uncanny resonance potentially generated while we watch a body move on film is amplified when, as players, we operate and navigate avatars. We use avatars as embodiments or vehicles, as our agents in the gameworld ... Avatars are our emissaries and, at least to a degree, our doubles. (2010: n.p.) Tilt Brush (and VR) produces an amplification of this doubling through its immersive illusionism. Our body is not visible as an avatar in third person, but rather in first-person mode and, in this sense, is represented by pure motion. The perception of duality is more potent because of this – we move in one space and our actions are manifested in another. When I paint in Tilt Brush, I find myself awestruck at the technological virtuosity that allows me to construct these illusions – no matter how primitive – and to move within and through them, responding affectively to their presence as they come into being. Even more so, and despite the medium’s desire to efface the process of technological mediation, I remain aware of the perception of my doubled body. In Tilt Brush and other virtual experiences, I’m overcome by a weird sensation of dual embodiment. One of the questions posed by cognitive science is how we experience ourselves inside a body that interacts with a virtual environment. Neuroscientists Konstantina Kilteni and Raphaela Groten make a distinction between self-location – ‘the sense that one feels self-located inside a physical body’ – that is, our sense of embodiment and the experience of presence, which is the sense of ‘being there’ inside the world or an environment (which they call ‘place illusion’) (2012: 375). Usually, the two coincide; for example, I have a sense of self-location, sensing my body present in my lounge room getting ready to put on my Oculus Rift goggles and create in Tilt Brush. However, the VR experience in Tilt Brush complicates this relationship between self-location and presence by doubling bodies. I have one body that is self-located in my living room, and another that feels like it is self-located in a virtual space. Likewise, I can still feel and sense my body’s presence in my lounge room, but the VR world I have entered and created also offers me an alternative experience of presence. All these states communicate with each other (and my brain/body) trying to make sense of the relationship between the virtual and the real (the chiasm). Kilteni and Groten also argue:

22

MATERIALIZING DIGITAL FUTURES

The sense of body ownership has been proposed to emerge from a combination of bottom-up and top-down influences ... Here, bottom-up information refers to the afferent sensory information that arrives to our brain from our sensory organs; for example, visual, tactile, and proprioceptive input, whereas top-down information consists of the cognitive processes that may modulate the processing of sensory stimuli; for example, the existence of sufficient human likeness to presume that an artificial body can be one’s body. (2012: 377) There is a fluid connection that exists between mind and body and intellect and senses, and the flow of information between brain and body is instantaneous. While surrounded by my creations, in Tilt Brush, I’m regularly reminded of my body presence in a real space (e.g. if I bump on the furniture, my eyes feel tired, or the technology isn’t working) while my brain is also telling my body (and vice versa) it is located in another world. The power of VR is that it aims to wipe the presence of the real from our perceptual field. So far, most VR has not achieved this seamlessly, but it has managed to do enough so that it plays games with the relay of ‘bottom-up and top-down’ information. My cognitive processes respond to two things – my body being materially present in my home, and my body simultaneously being present in a virtual space. I conclude by returning to Merleau-Ponty’s words (2002: 291) relating to the mirror experiment cited at the beginning of this chapter: while the ‘virtual body ousts the real one to such an extent that ... he [sic] inhabits the spectacle’, both the mirror experiment and the VR experience return to the ‘anchorage’ that places the viewer solidly back in the realm of the flesh and the world.

References Bosch VR (2016), ‘BDH Immersive: Bosch VR’. Available online: https://www.bdh. net/immersive/bosch-vr. Bruno, G. (2002), Atlas of Emotion: Journeys in Art, Architecture and Film, London: Verso. Carr, D. (2010), ‘Textual Analysis, Digital Games, Zombies’, Proceedings of DiGRA 2009 Conference. Breaking New Ground: Innovation in Games, Play, Practice and Theory, 26 September 2010. Available online: http://www.digra. org/dl/db/09287.24171.pdf. Dewey, J. (2005 [1934]), Art as Experience, London: Penguin Putnam. Hayles, N. K. (1999), How We Became Posthuman: Virtual Bodies in Cybernetics, Literature, and Informatics, Chicago, IL: Chicago University Press. Hayles, N. K. (2002), ‘Flesh and Metal: Reconfiguring the Mindbody in Virtual Environments’, Configurations, 10 (2): 297–320. Heydrich, L., T. J. Dodds, J. E. Aspell, B. Herbelin, H. H. Bülthoff, B. J. Mohler and O. Blanke (2013), ‘Visual Capture and the Experience of Having Two

VIRTUAL REALITY, THE CHIASM, THE DOUBLED BODY

23

Bodies – Evidence from Two Different Virtual Reality Techniques’, Frontiers in Psychology, 4: 1–15. Available online: https://www.ncbi.nlm.nih.gov/pmc/ articles/PMC3866547/ (accessed 18 December 2013). Keane, G. (2015), ‘Step into the Page’, Future of Storytelling. Available online: https://vimeo.com/138790270. Kilteni, K. and R. Groten, (2012), ‘The Sense of Embodiment in Virtual Reality’, Presence, 21 (4): 373–87. Levin, K. (2016), ‘Aesthetic Movements of Embodied Minds: Between MerleauPonty and Deleuze’, Continental Philosophy Review, 49: 181–202. Merleau-Ponty, M. J. J. (1968), The Visible and Invisible, trans. Alphonso Lingus. Evanston, IL: Northwestern University Press. Merleau-Ponty, M. J. J. (2002), Phenomenology of Perception, trans. Colin Smith, London: Routledge and Kegan Paul (originally published in French in 1945 and in English in 1962). Pallasmaa, J. (2014), ‘Empathic Imagination: Formal and Experiential Projection’, Architectural Design, 84 (5): 80–5. Pulido, M. (2010), ‘“An Entirely Different Kind of Synthesis”: Reflections on Merleau-Ponty’s Analysis of Space in the Phenomenology of Perception’, Aporia, 20 (1): 27–44. Sheets-Johnstone, M. (2011), The Primacy of Movement, Amsterdam: John Benjamins. Ulrich-Verderber, L. (2020), ‘This Disney Animator Brings Characters to Life in Virtual Reality!’ Ever Widening Circles. Available online: https:// everwideningcircles.com/2020/01/09/tilt-brush-glen-keane-virtual-realitypainting/ (accessed 1 September 2020).

24

2 Sensing Sims: Atmospheres, aesthetics and the cyborg player Merlin Seller

Introduction: More-than-human phenomenology at play I want to know what it is like for a bat to be a bat. Yet if I try to imagine this, I am restricted to the resources of my own mind, and those resources are inadequate to the ask NAGEL, 1974: 439

In ‘What Is It Like to Be a Bat?’ Thomas Nagel uses his titular thought experiment to reflect on ‘the subjective character of experience’ (1974: 436) and raises the question of whether we can truly know what it is like to ‘be’ another being. In playing with such questions as ‘what might a new sense feel like?’ this chapter suggests that videogames present us with interesting potential for materializing digital futures. Using the term Sensing Sims for videogames which focus on the challenge of sensing the hidden environments, I argue that before narrative and mechanics comes the obstacle of perceiving space by indirect means. Taking this further, I consider how sensing videogame atmospheres aesthetically allows players to reflect on how they sense the world around them, as exemplified in the Sensing Sim.1 I do so through a visual cultural studies approach to theory-driven textual

26

MATERIALIZING DIGITAL FUTURES

close-reading and visual analysis. Here I explore this problematic from the fields of Game Studies and atmospherics, before ‘embodying’ theory in the two case studies of Unfinished Swan (2012) and Scanner Sombre (2017). While previous scholarship has focused on the experiential implications of virtual reality (VR) (Helsel and Roth, 1991) or the physical interface itself (Keogh, 2018), significant work remains to be done in engaging with the screened aesthetic content of videogames as thought experiments with, and articulations of, sense perception itself. Conferences (Bogost, 2008) and monographs (Kirkpatrick, 2011) have considered videogames as articulations of phenomenological perspectives, but these have largely made general claims of the medium, risking the relegation of videogames to mere illustrations of existing discourse rather than novel, visual and tactile arguments in their own right. Yet, as Darshana Jayemanne (2017: 5) argues, following Gordon Calleja (2010), videogames present an ‘extreme heterogeneity’, problematizing synoptic accounts of the medium. As Jesper Juul notes, Game Studies has historically either favoured synoptic accounts of the medium and its audiences or focused monographs on individual videogames rather than on middle-level problems, resulting in missed opportunities to explore close comparisons or subgenres (2010: 21). In this chapter I explore two such middle-level problems – the meanings and affects of a small constellation of videogames which eschew traditional mechanics in favour of unique traversal and navigational verbs. I propose that there exists a subgenre of Sensing Sims as first-person videogames. This subgenre falls between walking-sims (defined as first-person narrative games without combat, e.g. Dear Esther, 2012) and puzzle-platformers (defined as games based on the challenge of traversal, e.g. INSIDE, 2016) and is essentially videogames of exploration wherein challenge stems from fundamental attempts to observe one’s environment rather than master or dominate it.2 In Alien Phenomenology, Ian Bogost argues for attempts to reckon with non-human ‘experiences’ – to appreciate things for things’ sake rather than their instrumental functions or relationships (2012). Yet Bogost, in his selfconfessedly inescapable anthropocentrism, is faced with Nagel’s obstacle: ‘I want to know what it is like for a bat to be a bat. Yet if I try to imagine this, I am restricted to the resources of my own mind, and those resources are inadequate to the task’ (Nagel, 1974: 439). The assumption they share is that we all live lives that are restricted because we are prisoners of our finite minds. But Nagel falls victim to his own critique in presuming to know the limits of our own consciousness. That is, is the mind really finite if we are, as cyborg theory and radical ecology tell us (Haraway, 2008; Morton, 2013), both less and more than we think we are? As the following sections will unpack, what if we could marshal additional ‘resources’ from our technologies, texts and surroundings? I argue that through the Sensing Sim we can feel something of the more-than-human world.

SENSING SIMS

27

In videogames, as Brendan Keogh argues, we do become more than ourselves as we are hybridized into an ‘assemblage that is the player-andvideogame’ (2018: 44). Videogames extend us and provide conceptual spaces which allow for new configurations and appraisals of subject and object, player and world. Moreover, through points of friction and opacity in their aesthetics videogames reveal something of the ‘alienness’ of nonhuman phenomena. These spaces, and their partly objective/partly subjective relations of player-and-videogame, are the product not only of mechanics and narratives but also of visual and tactile outputs and inputs and the speculative synthesizing work of player-and-videogame. Whereas phenomenological accounts of videogames ‘ask how videogames engage the body as a form of audiovisual-haptic media’ (Keogh, 2018: 17), this present conceptual shift deeply questions how specific videogames, understood as audiovisual-haptic media, can themselves reflect on the methods by which bodies can perceive. Jon Dovey and Helen Kennedy have called for ‘a phenomenology of [video] games that takes account of both their textual and experimental properties’ (2006: 93). That is, through experimental videogame spaces, addressed in subsequent sections here as ‘post-phenomenological atmospheres’, the player-and-videogame cyborg is given unique perspectives from which to reflect on the capacities of sight, sound and touch. Marshal McLuhan cleverly described media technologies as extensions of the central nervous system in that they are deeply entwined in everyday human living. (2001: 58). Drawing on this notion, we might be considered as currently living under biotech conditions in which interactive media can help us reflect on these extended capacities. The videogames discussed here wrestle with both Keogh (2018) and Haraway’s (1985) ‘cyborg’ and Bogost’s ‘alien phenomenology’ (2012), materially and conceptually. Whereas popular perception holds virtual environments as places which facilitate free agency (as critiqued by Bogost, 2016, and Bowman, 2019), Unfinished Swan (2012) and Scanner Sombre (2017) provide atmospherics and ‘staged materialities’ (Boehme, 2013) for reflecting upon the limits and alterities of sight and touch, that is, the friction we experience both behind and in front of the screen. In the case studies discussed in the following sections, the firstperson protagonist reflects on epistemic issues concerning spaces they can only sense indirectly – by throwing paint or LIDAR point-data onto their environment, and probing surfaces they cast into ambiguous relief. Before drawing on these case studies, we need to test and probe the scholarship concerning phenomenology in videogames and digital interfaces and establish the theoretical underpinnings of Sensing Sims. To do so, I follow Steve Swink’s design work on ‘game feel’ (2009), and the recent scholarship of Jayemanne (2017) and Keogh (2018) in focusing on videogames as embodied experiences as a corrective to scholarship’s neglect of what (2007: 336) calls ‘powerfully carnal modes of player-game engagement’. Rather than a psychological, historical or sociological account

28

MATERIALIZING DIGITAL FUTURES

of videogame experience, phenomenology provides us with these ‘carnal modes,’ ‘an account of space, time and the “world” as we live them. It tries to give a direct description of our experience as it is’ (Merleau-Ponty, 2012: vii). Opposing Game Studies’ traditional focus on either the formalist analysis of mechanics and narrative or its ethnographic interest in player communities, Keogh asks us to instead consider videogames as primarily ‘audiovisual-haptic media’ (2018: 12) open to the tools of media studies, visual studies and phenomenological analysis.3 In developing the notion of Sensing Sims I follow in the footsteps of Melissa Kagen’s model of ‘antigames’ (2017) which challenges and deconstructs conventional narratives and systems of competition and domination. I take a path that complements Dean Bowman’s (2019) recent work on the semiotics of space in walkingsims. Like Bowman, I aim to challenge the more orthodox Game Studies’ emphasis on ludology and rule systems, but from an aesthetic perspective in keeping with his narrative frame. While responses to the ludology versus narratology ‘debate’ (see Frasca, 2003, for an overview) have tended to redress the dominance of ludological (rules-based) approaches by emphasizing narrative instead (Jenkins, 2006), the visuality of videogames remains neglected by this dyad. Whereas Mark Wolf has outlined the core tension and distinction of the medium as between ‘video’ (the audiovisual and technological components of a work) and ‘game’ (its supposedly transferable rule set and mechanics) (2001: 14), Keogh asks us to consider videogames as indivisibly both. Of significance is the author’s call to treat videogames as ‘intermediative’ media rather than simply interactive rule sets (Keogh, 2018: 43). To begin analysing videogames from such a critical vantage, Keogh argues: ‘We must look at what the player looks through’ (2018: 37). Contemporary videogames make use of a diverse array of physical and graphical user interfaces (GUIs) with differing mechanics and logics (Jorgensenn, 2013). Advances in GUI and ‘neural networks’ have improved data processing and machine learning with impacts for the gaming industry. My focus, however, is on sensory thought-experiments on first-person videogames in which the player identifies with an imagined player character (PC) body in the position of what Darshana Jayemanne terms the ‘subjective camera’ (2017). This ‘first-person’ interface between player and PC, however, is multiple and ambiguous – composed of digital (on/off buttons) and analogue (tilted stick) inputs, force-feedback, non-diegetic User Interface (UI), virtual camera rigs, sights, sounds, attachments, and affects. Ash construes this intricate reciprocal relationship of technologies and bodies as a complex ‘envelope’ rather than a single surface or point of contact through which we effect agency – digital interfaces are a ‘folding of space time’ which creates new ‘forms of embodied and habitual value’ (2016: 7). Discourse within both the videogame design community and Game Studies

SENSING SIMS

29

scholarship, however, has frequently positioned this as a unidirectional, instrumental relationship where the player is imagined to be a pilot that ‘steers’ their character like a ‘vehicle’ (Barlte, 2003; Swink, 2009: xiii). Yet this metaphor obscures the dense complexities and entanglements that accompany embodiment (virtual, physical and composite) with the divine and empowering associations of the player ‘avatar’. As Melanie Swalwell documents in her visceral auto-ethnographic account, this relationship is deeply reciprocal, a messy dialogue (Swalwell and Wilson, 2008: 73). Indeed, Jayemanne questions the traditional ‘flattering’ metaphor of the ‘avatar’, and suggests adapting from neuroscience the idea of a ‘performative homunculus’ in which a flexible sensory body image is mapped imperfectly between player and videogame (2017: 224). From phenomenology, Maurice Merleau-Ponty uses a vehicular metaphor for the relationship of consciousness to the body implying their separation and hierarchy, but this is somewhat limited (2012). What remains, however, is the significance of his emphasis on bodily entanglements: ‘having a body is, for a living creature, to be intervolved in a definite environment” (2012: 94). Building on the work of both Merleau-Ponty (2012) and Jayemanne (2017), Keogh argues that orthodox Game Studies’ ‘focus on player “actions” [...] is ultimately disembodying, a historically constructed mythos coming at the expense of the fully embodied, sensorial experience of engaging with a videogame as sights, sounds, and haptics’ (2018: 195). In contrast, he describes the kind of distributed and emergent subjectivity that videogames give rise to as ‘cybernetic’. This is a play of bodies across screens wherein we become with the computer as a ‘cyborg-player’ (Keogh, 2018: 182). Indeed, David Sudnow argued in his seminal Pilgrim in a Microworld (1983) that we gain an irreducible bodily knowledge through interaction with a game world, a different, corporeal understanding than that afforded by an abstract awareness of a videogame’s mechanics. Significantly for our reconsideration of Nagel, this is a mutual incorporation in that we are extended beyond ourselves by the videogame: ‘It’s as if instead of truly incorporating the events of the screen within the framework of the body’s natural way of moving and caring, the action on the screen must incorporate me, reducing or elevating me to some ideal plane of synaptic being through which the programmed coincidences will take place’ (Sudnow, 1983: 138–9). As Ash summarizes, ‘the human and the technical have co-evolved alongside one another and so cannot be thought apart’ (2016: 9). Going beyond Keogh now, we might think of the player-and-videogame as a mutually constructive cybernetic thing in which human capacities are shaped and, crucially, expanded through being enveloped by and with videogames. But what affectively ‘fills’ this envelope? How do its concrete parts – sights, sounds, fingers, buttons – become a phenomenal whole? To better demonstrate this ‘elevation’ through an examination of case studies, we can consolidate this Game Studies discourse and consider the materiality of

30

MATERIALIZING DIGITAL FUTURES

Sensing Sims with fields related to atmospherics. In discussing materiality, we can build on post-phenomenology’s expanded sense of sensation and the cyborg body (Ash and Simpson, 2016; Lapworth, 2013) with the related ontological and phenomenological aesthetics of atmospheres (Böhme, 1993; Griffero, 2014), moving beyond a simple human-subject understanding of play. As Ash and Simpson define it, post-phenomenology responds to phenomenology by exploring ‘a move away from a subject-centred approach to experience’ (2016) in order to rethink affective relationships in what Levi Bryant (2011) would call a ‘flat ontology’ uniting human ‘subject’ and nonhuman objects without hierarchical distinction. In questioning the centrality of a pre-given human subject of affective experiences, this destabilizes simplistic pilot/vehicle metaphors of subject and body, implicitly rejecting the avatar model of the PC, and supporting the figure of the player-andvideogame cyborg. Digital materiality, understood through the lens of atmospherics as an aesthetic ‘staging’ of affects (Böhme, 1993) with the player-and-videogame, unites Bogost’s interest in alien objectivity with Keogh’s interest in intimate subjectivity in the Sensing Sim. What Böhme has described as the ‘quasiobjective’ aesthetics of atmospheres in which subjects and objects meet (2013: 69) productively bridges ontological and post-phenomenological concerns and allows us to appreciate the experience of encounters in the Sensing Sim cyborg. This can be thought of as a form of harmony integrating the partly objective, partly subjective envelope of play. Tonino Griffero frames this moment of encounter as ‘pathic aesthetics’ in which presences and appearances ‘resound in the lived body’ (2014: 7); Böhme terms it ‘attunement’, wherein ‘the [homuncular] body is a sounding board’ for affects, moods and presences such as pain or silence (2013: 44). Here we have the harmonizing of player-and-videogame. If Keogh is predominantly interested in literal and physical audiovisualhaptic materiality, by applying Böhme’s post-phenomenological approach we can further appreciate digital atmospherics, or what Böhme (2013) calls the ‘staged presence’ of materiality as opposed to objective materials, such as the felt solidity of space connoted by lighting or veneer. For Böhme, materiality is the experience of entanglement between bodies, both physical and virtual, what he calls the experience of presence and friction in felt space in which: ‘felt space is the modulation or articulation of bodily sensing itself’ (2013: 92). In drawing together theories of atmosphere with digital space, Ash (2012) argues that GUIs can indeed create spatio-temporal atmospheres. We can, then, with this grounding, extend it to consider the ‘atmospheres’ designed and orchestrated by Sensing Sims. Our sense perceptions here, and the ontology Böhme (1993) builds upon them, are mediated and constructive – essentially synaesthetic and indirect – rather than the direct awareness of the qualities of matter. Videogame worlds are atmospheric fields, a stage, of presentations, ‘semi-things’ and affective

SENSING SIMS

31

apprehensions (Böhme, 1993), where we can emerge ‘outside’ ourselves (Böhme, 2013) in shared affective understanding through designed scenography, as in Sensing Sims. Like the illumination of a stage, where rather than affording the direct perception of the qualities of objects we sense the affective space in-between – the ‘ecstasies’ of objects; quasiobjective properties extend beyond the object and result in the diffuse mood or felt space of an ‘atmosphere’. The post-phenomenological atmospherics of the player-and-videogame thus impart, through aesthetics, affective understandings of human–non-human phenomena. Play in these Sensing Sims is the sensing of atmospheres and feeling out felt spaces, and through them arriving at new awareness of human and non-human phenomena.

Sensing Sim atmospherics in Unfinished Swan and Scanner Sombre Following Jayemanne’s comparative approach (2017: 2), I now turn to analyse two case studies in order to ground and explore how these quasiobjective spaces can afford new kinds of cyborg play. Both Unfinished Swan (2012) and Scanner Sombre (2017) initially present the player with an opaque and undifferentiated visual field which, following their narrative premises of exploring and recovering lost artefacts with personal affective resonances, the player ‘maps’ through the act of painting over their hidden topography. In the opening of Unfinished Swan, the player sees nothing. More accurately, one sees a white reticle/cursor on what appears to be a blank, white page. With the press of a button on the player’s controller the user throws a blob of black ink into the void (see Figure 2.1). But this is not a void: with a splat we see the ink has impacted a surface before us. When we move the left analogue stick (conventionally tied to moving forward/ back, left/right on the x and y axes) or the right analogue stick (reciprocally dedicated to pitch and yaw), the size of this splat changes and its dimensions appear distorted on the screen. Through this we immediately sense a relative object in 3D space which is affectively similar to Juhani Pallasmaa’s claim of how we sense the atmosphere of a space before we become consciously aware of its detail (2012: 15). With the smallest of changes in the visual field we readily apply and recognize what Merleau-Ponty terms ‘an annexe of our “bodily schema”, the immanent meaning of a shift of “gaze” ’ (2012: 55). We feel a space relative to our body. But stripped to this minimal toolset, the positions we move through – both before the screen in real-life and behind the screen virtually – differ from traditional models of empowered player agency comprising both able-bodied play (sight and body) and orthodox Game Studies. There is

32

MATERIALIZING DIGITAL FUTURES

FIGURE 2.1  Unfinished Swan from Giant Sparrow (2012). Sony Interactive Entertainment. Used with permission. Screenshot taken by the author.

no easy traversal here. The one-point perspective of the first-person resists the comfortable centring of the subject through the ambiguity of space. Instead we experience the alien thingness (Brown, 2001) of slow first-person navigation around obstacles through what Bogost (2016) would describe as ‘the pleasure of limits’. The semblance of flatness here dissolves into a dizzying space of unknown dimensions, and to apprehend each object, the player-and-videogame cyborg operationalizes a kinaesthetic feeling-out of space. The act of ‘looking’ must take place in motion in order to reveal the third dimension – painting, then moving, then painting in order to cast objects from different vantage points in the round. In step with Keogh (2018) and Merleau-Ponty, our body – across the screen divide – experiences space carnally (2012: vii), but as a Sensing Sim it is incarnated in a more-thanhuman body. Going forwards, the player must compare the surface emanations of objects relative to themselves and each other in a manner analogous to echolocation (see Figure 2.1); yet, interacting too much with the environment causes it to lose form like overworked clay, converting it fully from white to black and making three dimensions read as two once again, a return to the beginning. Each painterly act of making the environment legible is also, in the same manoeuvre, an act of occlusion. Unfinished Swan here tells us that touching is an intersubjective condition – feeling something leaves neither the subject nor the object unchanged from the encounter. Knowing this world, always threatened with loss, then hangs precariously between reading its surfaces and rewriting them.

SENSING SIMS

33

Like Brown’s broken ‘things’ (2001), tools no longer ‘ready-to-hand’ in the Heideggerian sense, the conditions of the player-cyborg’s perception are exposed by these collapses as always a precarious and dynamic balancing act. A sense of pattern, of a gestalt drawn together from ink splatters, is thus key to navigating around the invisible forms of objects in this world. This process is closer to Böhme’s model of atmospheric ‘attunement,’ or more precisely ‘mood’ (2013: 91), than the popular game design terminology of ‘flow’ (Csikszentmihalyi, 1990): the player-cyborg haltingly tests and contemplates their environment, feeling its objects halfway extended into the space, compromising rather than optimizing. We stop and start precariously, rather than fluently dominating what we survey. The thematic premise of this videogame has the player inhabit the role of a child stepping into one of the eponymous unfinished paintings of their late mother, and so stages both a playful childhood adventure and a melancholy dialogue with mimesis, indexicality and incompletion. Mark Wolf, in his analysis of videogame space configurations, notes that videogames can pose interesting questions pertaining to visibility and space – where the ‘off-screen’ is both a core component of the medium, but also porous and dynamic (2001: 51). In Unfinished Swan and Scanner Sombre, we see this in a combination of what Wolf typifies as the discoverable space of a 3D maze videogame such as Doom (1993), and the imaginative non-visual space of text adventures such as Zork (1979). Indeed, what at first seems to be an enclosed interior maze of corridors opens into an exterior, with a bench, and then foliage which we gradually apprehend as a bamboo grove; beside it is a giant frog and a serpent in the lake, and beyond them a neoclassical ruin. Each encounter is a playful unfolding of space, the sequence of objects a humorous chain of things and atmospheres more than a coherent environmental narrative, and with each surprise we experience something of the strange surreality of those early forays into videogaming’s third dimension. In doing so, the player-and-videogame intermediate the perception of three dimensions both intertextually, what Böhme terms a material’s ‘social characteristics’, and sensually, through what Böhme terms ‘synesthetic characteristics’ (2013: 60). Even within the childhood narrative, what we feel, as we project ourselves into the unknown of this porous off-screen space, is a ‘tension’, Swalwell’s ‘kind of holding-on feeling’ (Swalwell and Wilson, 2008: 73) of investment, anticipation and surprise in the revelation of space. In Keogh’s terminology, the black ink of Unfinished Swan lets us ‘[look] at what the player looks through’ (2018: 37), and in this echoing space the player-and-videogame begins to grapple with ‘post’-phenomenology. This stands in contrast to the geometric sense of space in traditional first-person 3D play spaces, which Michael Nitsche articulates from a design perspective: ‘Game spaces are approached not as foregrounded spectacles based on visual cues such as perspective and parallax but as presented spaces that are assigned an

34

MATERIALIZING DIGITAL FUTURES

architectural quality’ (Nitsche, 2008: 3). Yet Unfinished Swan necessitates seeing through those very 2D videogame conventions of suggesting depth through the differing rates of change in background and foreground during movement (parallax) which 3D ‘presented spaces’ usually subsume or obscure through illusionistic rendering. Here the slightest awareness of parallax between paint splatters introduces depth and spatial awareness as a product of movement. Yet this is not a simple or straightforward key to our environment, instead, the process of changing between black and white, and triangulating between positions A and B and splatter C, is a dizzying process of identifying situated and partial contrasts between the echoes of objects and never the objects themselves. To re-adapt Merleau-Ponty, we are ‘intervolved in an [in]definite environment’, where what we actually perceive is the negative space of objects, their atmospheric materiality, rather than the modelling of their forms (Merleau-Ponty, 2012: 94). Every perception is cast as fleeting, situated and contingent, while the fictive material of this world is revealed to be ‘staged materiality’ – the phenomenon of contrast: ‘the character of materials becomes autonomous: materiality becomes pure outward form … the staging of materiality, or material as theatre’ (Böhme, 2013: 95). Moving through this space, we experience a speculative spectacle of objects split and cut into discrete silhouettes and moments of encounter. Böhme’s ‘ecstasies’ (2013) are the aesthetic space in which things affect one another. Rather than touching each other directly, as our videogame’s metaphor of painting and parallax so eloquently conveys, things leave kinds of perceptible traces of their mysterious properties. The eponymous ‘Unfinished Swan’ we seek is repeatedly deferred, leaving traces in the form of the occasional footprint – puzzle prompts to visually filling in the world. Like the dead mother figure that motivates our journey, the object is only knowable through its echoes. It stands as a metonym for our playful relationship to the environment, a game of call and response as we project into the space and bump up against it. Slowly points of encounter turn into constellations, and – moving, throwing and moving again – we co-construct the space as an amalgam of facets. This is an appreciation of space twice removed, first by ink and second by parallax. We see objects by the fictive pigment that covers them, not the object covered, and even then this world is flat, cast shadow until we move, feeling difference over time, triangulating a volume. We shout and stumble through this videogame, spreading our reach to cover a space we cannot ‘see’. This atmosphere of sharp contrast and ambiguous voids, constituted through the perception of negative space and parallax through movement, presents what Nagel would call an ‘alien’ form of experience, but for Nagel this poses a problem: ‘To the extent that I could look and behave like a wasp or a bat without changing my fundamental structure, my experiences would not be anything like the experiences of those animals’ (1974: 439). Yet, as Keogh (2018) has argued, our ‘fundamental structure’, by virtue of

SENSING SIMS

35

cybernetic extension, is fundamentally changed in the moment of play. While the player-and-videogame is no bat, we experience an atmosphere of echolocation in which we project noise outwards and then piece together three dimensions from the traces of signal that return to us: walking, splashing, walking, splashing. As Jayemanne emphasizes, through the profusion of framing devices in videogames and the performative multiplicities they afford, videogames are able to produce new and unexpected experiences: ‘a balance between potential and act that offers the possibility of thinking beyond the limits of any particular definition or unit of performativity’ (2017: 134). What we then experience is this beyond – a felt atmosphere. Though it may not be identical to the experience of a bat, it is a quasiobjective space built with our involvement as well as taking us beyond ourselves to a strange experience of the more-than-human and sensation defamiliarized. We now turn from this unfamiliar but welcoming atmosphere in Unfinished Swan’s fairy tale colouring book to a spelunker’s dark resting place in Scanner Sombre (see Figure 2.2). In the game space of Scanner Sombre, our titular LIDAR scanner probes invisible cave surfaces in three dimensions, with lasers fixing solid points in space as we navigate our way back above ground. We are given the robotic sight of a self-driving car but from the point of view of the vehicle itself, or, closer to home, we find ourselves inhabiting the subject position of a Kinect sensor, each point

FIGURE 2.2  Scanner Sombre (2017). Introversion Software. © Mark Morris from Introversion Software. Courtesy of the developer.

36

MATERIALIZING DIGITAL FUTURES

indicating the memory of where a laser beam was interrupted by a solid object. The player-and-videogame can ‘paint’ the environment with various ‘brushes’, from spraying localized areas with different densities of dots, to quickly feeling out a panorama with a spherical scan sweeping in every direction. Interestingly, and even more explicitly than in Unfinished Swan, Scanner Sombre’s different spray patterns convey the importance and necessity of reading stimuli as embodied sensations: signals bouncing from handheld tool to cave to our fictive headset display which maps on to the screen before the player. There is no seeing without a contingent, specific and dynamic cyborg perspective. While stationary, the uniform, regimented dots of a radial scan superimpose the fleeting impression of a perfect sphere in any space, until we move and the optical illusion collapses in parallax, revealing a flat pointillist image be a sculpture in relief. However, unlike Unfinished Swan, which imagines the player to be an able body stepping into a fantastical world, Scanner Sombre frames a player with cyborg senses (a fictive headset allowing us to see sensor data) returning to the surface from a labyrinthine cave. This presents its own unique challenges to the senses, as if the player were a slave in Plato’s famous Allegory of the Cave and here too we find gamers repeating history to wrestle with different models of the world in an effort to perceive their own truth/s. Unlike Plato’s cave, however, we never ‘see the sun’ – we cannot strip the technology from our body because the two are inseparable, truth lies not in a discrete form but in our bodily entanglement. Even if one was to reach the surface, players would find everything in the Scanner Sombre world composed of dots. Objects, then, are instead brought together in an atmospheric unity, an undifferentiated ‘point cloud’ where material wraps objects together in new thing-like configurations, and envelopes them.4 Sight is slowed down to an incremental exposure of space, distance mapped by means of a colour gradient. By applying point data, we touch objects at a remove, and so experience an integration of sight and haptics. Thus, we feel the extension of the tactile sense that Pallasmaa argues subtends our other senses: ‘All the senses, including vision, are extensions of the tactile sense; the senses are specialisations of skin tissue, and all sensory experiences are modes of touching, and thus related to tactility’ (2012: 12). In atmospherics, this is what Böhme (2013) terms a ‘synesthetic atmosphere’ – the conjunction and projection of sensation – reflexively staging a medium Keogh describes as ‘audiovisual-haptic’ (2018: 17). By wrapping the texture of damp stone in the neon texture of dotted point-data, Scanner Sombre makes sight tangible and allows us another atmospheric vantage upon the act of looking – feeling it from a new cyborg entanglement. Moreover, by elaborating sight in time – the accretion of dots and our triangulation of them – our senses are spatially and temporally extended. As the player-and-videogame maps point data which is encoded into the

SENSING SIMS

37

fictive memory of the headset, line of sight does not effect visibility – we see all points from every other point; the felt ghost of a level. What we have felt we will always feel, memory made atmospheric. By extending sight around corners, and backwards in time, our senses are stretched and recomposed. Rather than echo-location, a better metaphor here might be ‘vegetal’ sensation that is almost plant-like. We flow around objects rather than seeing them directly, and only our latest scans, the tips of our ‘tendrils’ give us accurate information about the present moment. We grow through the cave, and we see each point as a record of our first encounter with it during our extension, our journey. As Andrea Stevenson Won suggests, we can incorporate new affordances into the body by inhabiting digital avatars, terming this capacity for integration ‘Homuncular Flexibility’ (Won et al., 2015). Here, in Scanner Sombre, cyborg player-and-videogame assimilates to a new schema of haptic touch, a new spatio-temporal envelope of interaction. But this transition is one of phenomenological difference, not instrumental enhancement because it traces limits and ambiguous atmospherics. We see this where moving objects appear either ossified on the spot or melt into invisibility – ghostly figures frozen in time or restless waters slick like oil. Ruptures in the player interface are exacerbated in moments of aesthetic ‘glitches’ when the player encounters something horrifying. These visible tear-like post-processing effects are ruptures exposing the thingness of 3D rendering, its machinic failures and elisions. As Böhme has it, tears reveal the materiality of the atmosphere: ‘When the atmosphere is torn open, it is not destroyed but becomes in a sense visible as such and transparent’ (2013: 107). More broadly these tears in the atmosphere allow us to reflect on the complexities of sight through our aesthetic engagement with the videogame as affective thought experiment. Scanner Sombre, then, to a greater degree than Unfinished Swan, presents a self-reflexive look at the act of looking. This also allows Scanner Sombre to press harder on the ambiguities of perception. Powerfully, the atmospheric materiality of Scanner Sombre LIDAR disorients us with ambivalences as we spelunk through the darkness. Amid point data, reverse and obverse become indistinguishable. Are we inside a given space, or outside it; are we seeing the back of a rock from in front of it or the opening of a cavern at its entrance? This is reminiscent of Merleau-Ponty’s model of motor aphasia or literal paraphasia (2012: 226– 7) where the distinction of figure and ground is destabilized: are we looking into a pit, or have we yet to scan the front of a mound before us? Parallax can confer a sense of convex and concave, but cannot necessarily determine, of an area far away, whether we are before or behind it. Thus leaving us ‘powerless to structurize the world and grasp its articulatory physiognomy’ (Merleau-Ponty, 2012: 227). But from this vertigo comes a new appreciation of what Böhme describes as the collaborative or quasi-objective character of atmospheres. The author clarifies that ‘rather than qualities, I speak of

38

MATERIALIZING DIGITAL FUTURES

ecstasies, of ways of stepping outside of oneself’ (2013: 163). Using the thought experiment of an enclosed space to illustrate this, he argues that objects manifest themselves through atmospheric emanations rather than fully objective properties, and so what is concave to the body it encloses is convex to the space surrounding it. By being able to perceive this contrast in space, through a kind of kinaesthetic ambiguity, the player-cyborg can feel the multiple ecstasies of things and see the process of atmospheric manifestation itself. Through this form of engagement, we come to an appreciation of what Morton argues is a struggle to comprehend our reality: that there is no foreground or background because everything is interrelated in every direction and every order, without hierarchy (2013: 110). The ambivalently concave/convex experience, like a 3D version of the kippbild illusion (an optical illusion that shows two figures in the same image, such as the famous rabbit-duck), arguably gives the player an ecstatic, synaesthetic awareness of the multiplicity of things’ materiality. Böhme has defined the phenomenal quality of ‘materiality’ more directly as a sensual perception irreducible to a specific object, a paradoxically familiar kind of roughness, a ‘typical form of irregularity’ (2013: 97). In Scanner Sombre, this materiality is the ‘typical irregularity’ of mottled point data dusted on the surface of the world, a staged materiality we try to attune ourselves to. We feel it and we transmit it. We sense its textural quality as a kind of friction, an entanglement of bodily presence where the highfidelity audio of our footsteps on wet stone contrast dissonantly with the uniform texture of point data, which appears to float freely in a vacuum. These points of light, glowing without illuminating space around them, act like stars which give volume to space by establishing direction, but they fail to illuminate the rock they enshroud. As Böhme argues: ‘The space that is structured by stars, however, remains itself dark’ (Böhme, 2013: 152). By this atmospheric materiality, we experience the echoes of objects withdrawn from direct access. Ecstatic light emanates into our perceptual field in a way which breaks and recombines us with the surrounding world just as atmospheres both reveal and obscure their objects. Things extend beyond themselves as they come to meet us, but we are forced to confront the fact that we cannot grasp them fully. Instead, we are forced to meet them halfway to take in their dizzying affects. In Scanner Sombre is a starkly post-phenomenological model of our cyborg body, a spatio-temporal envelope which transgresses its frontiers. This is in contrast to Merleau-Ponty’s phenomenology: ‘The outline of my body is a frontier which ordinary spatial relations do not cross … For example, my hand is not a collection of points’ (2012: 112). In Scanner Sombre, our cyborg body can, with that alien night sky, start to become a starry ‘collection of points’. In Scanner Sombre, the body is opened up and both distended and reincorporated. Sight is split into multiple spatial points and fissured in incremental temporal mappings, but also fused with

SENSING SIMS

39

touch as a lasting record of topology. Our senses are changed – they grow into our environment. Like the ghostly humanoid homunculi we encounter, our body is both an accreted, messy mass of points and an atmospheric gestalt. To conclude, we find ourselves dwelling in inky blackness. In our painterly journeys through these echoing spaces, we experience a blossoming genre which encourages sensitive attunement rather than competitive flow. In these two case studies we see the inextricably aesthetic nature of videogames, and through their atmospherics we gain an appreciation of the affective and generative power of visuality in digital materiality, just as important as the physical, narrative and rule set dimensions of videogame play. Crucially, atmospheres, when discussed in tandem with case studies allow us to appreciate how space is a synaesthetic and synergistic product of the player-and-videogame cyborg. Videogames are the product of what Böhme calls ‘aesthetic workers’, and by paying careful bodily attention to aesthetics these videogames allow us to extend ourselves and begin the work of understanding experiences that are more than human. As my examples attest, Sensing Sims offer dizzying and diverse affective understandings of the ecstatic interplay of things, built on a shared engagement with the complex intersubjective and indirect relationship we have with our environments. Unfinished Swan emanates a playful atmosphere of moment-to-moment ‘echo-location’, whereas Scanner Sombre manifests a melancholy reflection on echoes in time – a mood space of haptic, vegetal ‘sight’. While offering seemingly similar spatio-temporal envelopes – with the player pushing buttons and sticks to move, paint, pause and reflect on initially blank firstperson environments – Unfinished Swan offers a focus on the momentary and provisional, whereas Scanner Sombre’s cloud of point-data memory pushes us to consider the longer-term relationship of time to sensation. The former poses a careful balancing act between extremes of opacity that forces the player to carefully distinguish between signal and noise, while the latter more self-reflectively probes the aesthetic apparatus of atmospheres and the ambivalent externality and interiority of space. What they share is an interest in breaking open atmospheres, offering insights into how objects withdraw like echoes, and in how atmospheres resonate between time and space, sight and touch. In both Unfinished Swan and Scanner Sombre, there is an interesting flattening of ontology. Our understanding of objects modulates into an appreciation of the ecstasies of things – subjects and objects meeting and generating something both alien and familiar: atmospherics. Combinations and elisions between objects in proximity prompt new relationships, new atmospheres and new affordances. In doing so, these examples propose interesting relationships between time and perception. In Unfinished Swan we inhabit the metaphor of a blank canvas. By projecting ink around the environment, we participate in its becoming – but, as the title reminds us,

40

MATERIALIZING DIGITAL FUTURES

this space is unfinished: we are unable to tell whether the environment exists (hidden) before our attempt to sense it, or whether the act of sensing, of ‘inking’, is what gives it form. Even more radically, Scanner Sombre reifies fleeting traces at explicitly fixed points in the past – blurring the lines between memory, fantasy and actuality, revealing all to be ‘things’ and none to be any less a being than any other. These videogames, I argue, can be fruitfully approached as atmospheric metaphors for echo-location and vegetal sensation, and while speculative, they provide meaningful access to the more-than-human through the homunculus of the player-and-videogame cyborg. There is much to be gained from exploring phenomenological and epistemological questions articulated by videogames, which lie on the edges of the ludic and the narratological. By using the aesthetics of atmospherics as an analytic frame, we can examine more than textual and mechanical analyses afford. Scanner Sombre and Unfinished Swan give us useful examples of atmospheres of cyborg entanglement with which to explore and reflect on normative human phenomenology, and to intermediate ‘alien’ sensations. In these Sensing Sims, the players watch themselves in the act of watching, defamiliarizing the human and familiarizing the non-human through the materiality of digital atmospheres.

Notes 1 It is important to acknowledge the heterogeneity of bodily sense perception here, as Keogh does in citing feminist post-phenomenology (2018: 33), and James Ash (2016) does in locating digital phenomenology within economic and class-based bodily rhetorics. Though a Disability Studies approach is out of the scope and expertise of this chapter and its author, I would like to acknowledge the excellent work of Diane Carr (2014) on representation and the disabled body, and Sarah Gibbons (2015) for her wider and more anthropological approach to synthesizing Disability Studies and Game Studies. 2 Other examples of Sensing Sims, which also blur the line between aesthetics and mechanics but are not in the scope of this chapter, include Ian MacLarty’s Catacombs of Solaris (2016), Pillow Castle’s Superliminal (2019), Demruth’s Antichamber (2013) and Christoph Frey’s Horror Vacui (2012). 3 In exploring the cyborg possibilities of defamiliarizing the human and embodying the non-human, I also wish to acknowledge my debt to a growing scholarship concerning the diversification of the medium from feminist, queer studies and critical theory perspectives (Anable, 2018; Chess, 2017; Malkowski and Russworm, 2017; Ruberg and Shaw, 2017). 4 The player eventually unlocks abilities to make rough colour-coded distinctions between kinds of material such as organic and inorganic, but as in Plato’s allegory we still have no direct access, and instead of the flicker of the fire, everything we perceive is still mediated by the pinpricks of laser data.

SENSING SIMS

41

References Anable, A. (2018), Playing with Feelings: Video Games and Affect, Minneapolis: University of Minnesota Press. Ash, J. (2012), ‘Rethinking Affective Atmospheres: Technology, Perturbation and Space Times of the Non-Human’, Geoforum, 49: 20–8. Ash, J. (2016), The Interface Envelope: Gaming, Technology, Power, New York: Bloomsbury. Ash, J. and P. Simpson (2016), ‘Geography and Post-Phenomenology’, Progress in Human Geography, 40 (1): 48–66. Bartle, R. (2003), Designing Virtual Worlds, Indianapolis, IN: New Riders. Behrenshausen, B. G. (2007), ‘Towards a (Kin)Aesthetic of Video Gaming: The Case of Dance Dance Revolution’, Games and Culture, 2: 335–59. Bogost, I. (2008), ‘The Phenomenology of Videogames’, in S. Günzel, M. Leibe and D. Mersch (eds), Proceedings of the Philosophy of Computer Games Conference, 22–43, Potsdam: Universität Verlag Potsdam. Bogost, I. (2012), Alien Phenomenology, or, What It’s Like to Be a Thing, Minneapolis: University of Minnesota Press. Bogost, I. (2016), Play Anything: The Pleasure of Limits, the Uses of Boredom, and the Secret of Games, New York: Basic Books. Böhme, G. (1993), ‘Atmosphere as the Fundamental Concept of a New Aesthetics’, Thesis Eleven, 36 (1): 113–26. Böhme, G. (2013), ‘Staged Materiality’, Interstices, 14: 94–9. Bowman, D. (2019), ‘Domesticating the First-Person Shooter: The Emergent Challenge of Gone Home’s Homely Chronotope’, Press Start, 5: 2. Available online: https://core.ac.uk/download/pdf/227108711.pdf. Brown, B. (2001), ‘Thing Theory’, Critical Inquiry, 28 (1): 1–22. Bryant, L. R. (2011), The Democracy of Objects, Ann Arbor, MI: Open Humanities Press. Calleja, G. (2010), ‘Digital Games and Escapism’, Games and Culture, 5: 335–53. Carr, D. (2014), ‘Ability, Disability and Dead Space’, Game Studies, 14 (2). Available online: Game Studies – Issue 1402, 2014. Chess, S. (2017), Ready Player Two: Women Gamers and Designed Identity, Minneapolis: University of Minnesota Press. Csikszentmihalyi, M. (1990), Flow: The Psychology of Optimal Experience (1st edn), New York: Harper & Row. Dovey, J. and H. Kennedy (2006), Game Cultures: Computer Games as New Media, Berkshire, UK: Open University Press. Frasca, G. (2003), ‘Ludologists Love Stories Too: Notes from a Debate That Never Took Place’, paper presented at Level Up, 4–6 November, Utrecht, Netherlands. Available online: http://ludology.org/articles/Frasca_LevelUp2003.pdf (accessed 1 March 2020). Gibbons, S. (2015), ‘Disability, Neurological Diversity, and Inclusive Play: An Examination of the Social and Political Aspects of the Relationship between Disability and Games’, Loading..., 9 (14): 25–39. Griffero, T. (2014), Atmospheres: Aesthetics of Emotional Spaces, Burlington, VT: Ashgate.

42

MATERIALIZING DIGITAL FUTURES

Haraway, D. (1985), ‘Manifesto for Cyborgs: Science, Technology, and Socialist Feminism in the 1980s’, Socialist Review, 80: 65–108. Haraway, D. J. (2008), When Species Meet, Minneapolis: University of Minnesota Press. Helsel, S. K. and J. P. Roth (1991), Virtual Reality: Theory, Practice, and Promise, Westport, CT: Meckler. Jayemanne, D. (2017), Performativity in Art, Literature, and Videogames, Cham: Palgrave Macmillan. Jenkins, H. (2006), ‘Game Design as Narrative Architecture’, in N. Wardrip-Fruin and P. Harrigan (eds), First Person: New Media as Story, Performance and Game, Cambridge, MA: MIT Press. Jorgensen, K. (2013), Gameworld Interfaces, Cambridge, MA: MIT Press. Juul, J. (2010), A Casual Revolution: Reinventing Video Games and Their Players, London: MIT Press. Kagen, M. (2017), ‘Walking Simulators, #GamerGate, and the Gender of Wandering’, in J. P. Eburne and B. Schreier (eds), The Year’s Work in Nerds, Wonks, and Neocons, 275–300, Bloomington: Indiana University Press. Keogh, B. (2018), A Play of Bodies: How We Perceive Videogames (Kindle edn), Cambridge, MA: MIT Press. Kirkpatrick, G. (2011), Aesthetic Theory and the Video Game, Manchester: Manchester University Press. Lapworth, A. (2013), ‘Habit, Art, and the Plasticity of the Subject: The Ontogenic Shock of the Bioart Encounter’, Cultural Geographies, doi:10.1177/1474474013491926. Malkowski, J. and T. A. M. Russworm (2017), Gaming Representation: Race, Gender, and Sexuality in Video Games, Bloomington: Indiana University Press. McLuhan, M. (2001), Understanding Media: The Extensions of Man, London: Routledge. Merleau-Ponty, M. (2012), Phenomenology of Perception, trans. D. A. Landes, London: Routledge. Morton, T. (2013), Hyperobjects: Philosophy and Ecology after the End of the World, Minneapolis: University of Minnesota Press. Nagel, T. (1974), ‘What Is It Like to Be a Bat?’ Philosophical Review, 83 (4): 435–50. Nitsche, M. (2008), Video Game Spaces: Image, Play, and Structure in 3D Game Worlds, Cambridge, MA; London: MIT Press. Pallasmaa, J. (2012), The Eyes of the Skin: Architecture and the Senses, Chichester: Wiley. Ruberg, B. and A. Shaw (eds) (2017), Queer Game Studies, Minneapolis: University of Minnesota Press. Scanner Sombre (2017), Videogame, Introversion Software, Walton-onThames, UK. Sudnow, D. (1983), Pilgrim in the Microworld, New York, NY: Warner Books. Swink, S. (2009), Game Feel: A Game Designer’s Guide to Virtual Sensation, Amsterdam: Morgan Kaufmann/Elsevier. Unfinished Swan (2012), Videogame, Giant Sparrow, Santa Monica, CA.

SENSING SIMS

43

Wolf, M. (ed.) (2001), The Medium of the Video Game, Austin: University of Texas Press. Won, A. S., J. Bailenson and J. Lanier (2015), ‘Homuncular Flexibility: The Human Ability to Inhabit Nonhuman Avatars’, in R. Scott and S. Kosslyn (eds), Emerging Trends in the Social and Behavioral Sciences, Hoboken, NJ: Wiley.

44

3 Embodied audiovisual experience: The role of sound in contemporary screen and digital media Darrin Verhagen and Ben Byrne

Introduction Emergent currently is a creative field around the production of multisensory experiences using digital technologies. This is driven by, but by no means exclusive to, audiovisual work. The field brings together existing forms that have been remediated and reshaped by digital practices, such as film and TV production, playable and interactive media, and kinetics. These come together in various combinations as well as to produce the ‘new realities’, virtual reality (VR) (with its close relative 360° video), augmented reality (AR) and mixed reality (XR). Sound, we find, plays a key role in these multisensory experiences. Generally, perspectives on the role of sound in audiovisual digital media are based on theories of cinematic sound design and score, relying on the principle that sound ‘adds value’ to image (Chion, 1994: xxv). Further, discourse on sound in audiovisual digital media is often structured by a series of seemingly simple binaries such as diegetic and non-diegetic, or on and off screen. The usefulness of such an approach

46

MATERIALIZING DIGITAL FUTURES

in cinema notwithstanding, these ideas fail to account adequately for the role of sound in many other digital multisensory experiences, as we show here. Using contemporary examples, and drawing on existing sound design theory, extended with reference to key texts from sound studies and interaction design, here we demonstrate how multisensory digital experiences are often particularly sonic, whereby audiences have the experience as listening subjects positioned within acoustic space rather than as viewers peering into screen space. While the experiential nature of this shift from ‘spectator’ to ‘immersant’ (Davies, 1998) or ‘visitor’ (Benayoun, 2011) can share audiovisual characteristics of computer games, our interest is not in goal-based worlds. Rather, we focus on cinematic audiovisual works that leverage spatial media, including 360° video, kinetics, haptics and sound, and interactivity of various forms to create new multisensory experiences. These works are often navigated through a more active level of embodied (or more specifically perhaps, ‘transplanted’) listening than traditional cinema. Many of the examples we explore involve an explicit alignment of Point of View with Point of Audition, and for some, ‘Point of Kinetic Experience’, centrally involved (as well as centrally positioned) as the audience members often are in these new encounters. As individuals fall into an enveloping world, many of the score’s established roles, whether synchronizing group emotions or anchoring images into meaning through leitmotifs and carefully charted narrative arcs, can be complicated to varying degrees by these new technologies. We present examples of 360° documentary, 4D cinema and web-based audiovisual material – from Collisions (2016), a VR experience set in central Australia, the superhero film Venom (2018) in 4DX and HOBO UFO (v. Chernobyl) (2019), an audiovisual release that employs Google Street View footage of Chernobyl. Although each of these is very different and their success and affordances of audience agency differ, all employ digital technologies in aid of affective multisensory experiences in which ‘the viewer’ is explicitly spatially positioned, and the relationship of sound to movement is key. The space we shall now inhabit is one between, around and beyond audiences and the screens that are so often assumed as the locus of digital experiences. Writing about cinematic audiovisual digital media, unsurprisingly, has looked at the digital through the screens of our devices, understanding it as a part of screen cultures and practices. We, instead, listen to digital media through embodied multisensory experiences produced by cross-modal practices that are necessarily interactive, sometimes in the sense that audiences are explicitly given agency as users, but just as often in the ways in which digital systems function and act on the bodies subject to them. These systems are no longer particularly new and have been previously identified by others. Media theorist Marshall McLuhan (2005: 72), for instance, wrote in the 1980s of the impact of

EMBODIED AUDIOVISUAL EXPERIENCE

47

‘emerging mediums such as the satellite, the computer, the data base, teletext-videotext, and the international multi-carrier corporations’ as continuations of what he phrases the ‘aural-tactility’ of media. The focus on the use of digital systems to create spatial affects is key to this chapter. However here we address this practice with a focus on how it is discernible in contemporary extensions of screen-based and interactive consumer media, from documentary to big budget film productions, as well as art works. We relate the click of a mouse to push a button on a display to the lurch of a motion simulator, and the turn of a head wearing a VR headset to the modulation of a video output based on accompanying audio data. Interactivity, approached in this way, includes both interactions between audiences (often now understood as users) and systems as well as between various modules and components in software and hardware, such as camera position being manoeuvred via algorithmic interpretation of an audio signal. Collisions’s use of the sensors, screens and stereo audio outputs of mobile devices to produce an affective 360° audiovisual experience for users to explore is linked to the leveraging of motion simulators in concert with sound design to position audiences diegetically in Venom; further, an approach to composition in HOBO UFO (v. Chernobyl) that employs bespoke software allows the sonic navigation of Google Street View data. These works all employ interactive digital systems to produce multisensory experiences, driven significantly by sound.

Sound design and score, interaction design and digital media Although much sound design literature to date has focused on cinema, there exists a distinct thread of more recent work that emphasizes different forms of multisensory experience. This connects with work that has pointed towards the increasing need to focus on the intersection of interaction and sound design, and new research exploring approaches to emergent digital media technologies. To consider the challenges and opportunities now open to sonic practitioners exploring such media, however (particularly with VR, AR and 360° film), it is important to first consider some of the audiovisual contracts and their underlying assumptions which have been established through cinematic convention, and against which new languages and experiences are currently being developed. Michel Chion’s Audio-vision (1994) has been a critical text in countering the dominance of visual accounts of cinematic experience. Importantly, it does more than simply suggest that cinema is image plus sound. It instead argues that there is a sophisticated interrelationship between the two senses

48

MATERIALIZING DIGITAL FUTURES

where audio-vision is more than just the sum of its parts. The subtlety, complexity and power of that relationship – whether it be used to open up imagination or lock down meaning – has been well understood for close to a century. In a period that could be viewed as a historically equivalent turning point to the present, prominent Russian directors voiced concerns in 1928 about how this relationship in sound film might develop, worried about the suffocating potential for creative experience if audio were to be used too literally when applied to image (Eisenstein et al., 1985: 83–5). There was a concern that poetic potential might be lost if simple functional confirmation of one sense by another prevailed. Notwithstanding such apprehension, the drift towards sound design’s authenticating role in popular film (rather than any relationship with image more elegiac or experimental) in hindsight, seems almost inevitable. If sound is perceived as an ‘attribute’ of an ‘object’ (Metz, 1985: 157) and, in cinema, ultimately serves to implant ‘the existence of objects depicted on the screen more firmly in the physical world’ (Camper, 1985: 379), then sound design was always going to be a highly prized tool when confirming a viewed yet constructed reality. The visual language of cinema and how it assembles time, space, perspective and narrative logic had already matured well before audio was added. So, at the point where technology could facilitate the shift from silent film to talkies, there was a clear choice on offer. Sound could be introduced to complicate these existing conventions by starting again with the time-consuming and experimental work necessary to develop a new audiovisual language (with all its concomitant birthing and growing pains) or it could simply be applied to support and validate the grammar which had already been established. Both score and sound design have grappled with these options throughout the twentieth century, resulting in different approaches and, at times, oscillating between both sharpening and weakening demarcations. Considering a traditional cinemagoer as ‘a pure onlooker whose participation is inconceivable’ (Metz, 1975: 264) has an effect beyond a simple denial of active agency – an agency (or suggestion of agency) that many of the new technologies now foreground. This ‘onlooking’ also established a specific relationship with, and reliance on, the traditional film score. For if it can be argued that ‘the cinematic screen serves as barrier between us and the world on screen’ (Wheatley, 2019: 71) then it was the psychological mechanics inherent in the experience of music which would increasingly be used to weaken that alleged obstruction. Arguably, score continues to play the same role even when audiences are surrounded by the visual experience of VR, but if that is the case it does so in concert with interactivity. Two decades ago, when concluding his 2001 book Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema, David Sonnenschein foresaw and emphasized the importance of the rise of interactivity in addressing digital media, claiming

EMBODIED AUDIOVISUAL EXPERIENCE

49

‘interactivity will take on unforeseen permutations and possibilities, a truly golden opportunity for sound designers to investigate’ (Sonnenschein, 2001: 218). As McArthur et al. now suggest, ‘VR affords a development of the dialectic between sound and image which distinctively involves our spatial attention. The lines between referent and signified blur; the mediation between representations invoked by practitioners, and those experienced by audiences, suggest new opportunities for co-authorship’ (2017: 26). Stephan Schütze and Anna Irwin-Schütze’s New Realities in Audio: A Practical Guide for VR, AR, MR and 360 Video (2018) is a rare example of a text focused on sound design for contemporary digital media, what the authors call the ‘new realities’. It offers much in the way of practical advice for sound designers. Although much of its focus is on re-creating reality, as is often asked of sound designers, it does emphasize the need to balance narrative concerns, and audience expectations, with rendering ‘reality’ (2018: 6, 71–2). Schütze and Irwin-Schütze identify that new realities are different because ‘the entire point of the experience is the sense of immersion and agency’ (2018: 21). Further, they find, as we do, that sound in new realities is ‘a vital part of immersion, communication, navigation and emotional narrative’ (Schütze and Irwin-Schütze, 2018: 268). Schütze comments: Over the years working on game audio, I found that in most cases an event, or object, or model, or creature would be placed into the game world and I would respond by creating a sound for that thing. Film and TV almost always work in the same way. The visuals are defined, captured, and edited, and then the audio is dropped in at the end. The primary exception to this is music videos where the video content is designed and created to suit the music. But is this the best way to develop content for the new realities? (Schütze and Irwin-Schütze, 2018: 273) To address this challenge, Schütze prompts that ‘with a brand new form of art, we can explore brand new methods of production’ (Schütze and Irwin-Schütze, 2018: 273). Such options have the potential to move beyond the traditional application of Chion’s (1994: xxv) ‘added value’ where sound adds ‘expressive and informative’ detail to images to confer a sense of authenticity. In much the same way that editors might struggle with the logic of edits in VR (where visual cinematic conventions might be problematic in 360°) so too are sound designers and composers exploring the new possibilities as to what kinds of expressive and informative detail their craft may provide in these new environments. That this is still a form of ‘added value’ is incontestable. But as new visual languages and logic need to be explored to accommodate the experience, there is potential for sound to do more than just lock an image into perceptible reality.

50

MATERIALIZING DIGITAL FUTURES

Media scholars Edmund Carpenter and Marshall McLuhan discuss what they term alternately acoustic and auditory space, in the essay ‘Acoustic Space’ (1970). Auditory space ‘has no favored point of focus’ and ‘no boundaries in the visual sense’, they argue (Carpenter and McLuhan, 1970: 67–8). Putting aside their presumptuous and totalizing references to ‘preliterate cultures’ and ‘oral traditions’ in staging a historic tension between visual and auditory culture, as well as their limited account of the reality of listening, their ‘auditory space’ does articulate the sonic dimensions of the ‘new realities’, long before their advent (1970: 65). Writing as if yesterday but now some fifty years ago, they claim ‘today we are experiencing the emotional and intellectual jag resulting from the rapid translation of varied visual and auditory media into one another’s modalities’ (1970: 70). In her book Sounding New Media: Immersion and Embodiment in the Arts and Culture (2009), Frances Dyson discusses at length the role of sound in VR, what she calls ‘virtual audio’ (Dyson, 2009: 138). She identifies that, in line with VR’s ‘pursuit of immersion and embodiment’: Virtual audio acknowledges and technologizes the presence of the body and the environment in a manner that the visual aspect of VR is only just beginning to encompass. Like the visual component of VR, virtual audio technology creates an interactive aural space or ‘sound field’, in which sounds are heard as if they occurred in the lived environment. If digital audio transforms the ear of the listener into a hearing eye, virtual audio threatens to shift the visual orientation. (Dyson, 2009: 138) The hearing eye she refers to is the listener being able to see as much as hear sound, displayed as waveforms and in other ways through digital technologies and their powers of modulation (2009: 138). The shift she speculates on, meanwhile, she hears as arising from the 360° aural space offered by virtual audio, and the way such a 3D listening space allows the simulation of movement of sound in physical environments, the acoustic or auditory space conceptualized by Carpenter and McLuhan (2009: 138). These possibilities are extended further when interactivity is considered. Janet Murray’s Inventing the Medium: Principles of Interaction Design as a Cultural Practice (2012: 1–2) seeks to aid in the design of new ‘digital artifacts’, based on an understanding that ‘all things made with electronic bits and computer code belong to a single new medium, the digital medium, with its own unique affordances’. With this in mind, Murray (2012: 3) offers alternatives to key digital media terms that facilitate ‘the collective cultural task of inventing the underlying medium’. These include, established term followed by Murray’s alternative, new media/digital medium, intuitive/transparent, interface design/interaction design, user/interactor, content/information and

EMBODIED AUDIOVISUAL EXPERIENCE

51

interactivity/agency (2012: 8–13). Murray’s framing of the digital medium, transparency and agency are most useful for us here. The hybrid forms and systems of which we have been writing are all part of the digital medium of which she writes (2012: 8). Transparency, specifically the goal of transparent interfaces that provide clear feedback in response to interactions, helps us to extend our account of the role of sound in contemporary audiovisual digital media. Agency, meanwhile, on which we have already touched, is afforded according to Murray ‘when the interactor’s expectations are aroused by the design of the environment, causing them to act in a way that results in an appropriate response by the well-designed computational system’ (2012: 12). This accounts for the navigation, human or otherwise, of 360° virtual audio spaces as much as websites or other digital media. These are principles well understood in sound design for games, it must be said. However, their use for functional agendas in goal-based gameplay of AAA games and the cinematic audiovisuals about which we write here could not be more divergent. It is the creative potential of working at the meeting point of sound design, spatial audio and interaction in audiovisual digital media which is intriguing to us. Inevitably, there will be creative practitioners attempting to reconcile the possibilities of these new forms with previously established traditions, tethering to technical conventions and entertainment agendas of the past. More interesting perhaps, and as we explore, are the artists with no interest in the new technologies being at the service of such traditions. And as both digital acuities rise and price points fall, it is at this intersection that we can expect to see experimentation that is a true challenge to both entertainment ideas and production codes of the past. Each of the three examples we focus on occupies a different place on this continuum.

Collisions Collisions (2016) is a VR experience that tells the story of Nyarri Nyarri Morgan, an Aboriginal Australian from the Martu tribe. Morgan witnessed the Maralinga nuclear tests in the central Australian desert in the 1950s, before he had encountered Australian settlers, and the work expresses the profound, and at times horrifying, impact the tests have had on him, his people and his country. An eighteen-minute film, it is presented using mobile based VR headsets and headphones, with audiences seated in swivel chairs to enable 360° movement (ACMI, 2016; Mann, 2016). As director Lynette Wallworth identifies, this leads to a significant shift when compared to traditional film: In virtual reality, everything becomes personal, you’re present. You’re not standing outside or sitting outside the film, you’re actually inside the

52

MATERIALIZING DIGITAL FUTURES

world that is being shown to you. So that meant that I would be able to place you exactly where Nyarri stood, and that you could see what he saw. (Mann, 2016) The work is linear, but the headset and chairs allow viewers freedom to look around in all directions. This of course means that the audience frames the screen with their gaze, rather than this being done by the director or cinematographer. The agency and potential depth of immersion that this affords those experiencing the work is significant, but it raises difficulties too, especially for a linear work. How, for example, do you ensure audiences do not miss key characters, sequences or events? When experiencing the work, we noted the creative and deliberate ways in which sound is employed to address this issue, maximizing audience agency. The centrality of sound to the experience of the film, and the ability to navigate it, became clear in a scene in the yard outside a house, red earth around and below. Looking out in the distance we heard Morgan’s voice behind us and turned to focus on what he was saying. The sound of Morgan’s voice indicated where we needed to look to see him. Simple as this may seem, it is a key technique that is employed throughout the work, and that is strikingly different to the standard cinematic approach to the audiovisual, in which sound design and soundtrack are understood to support the focus of the framed image before the viewer. Even if an offscreen voice or sound effect is used to draw the viewer’s attention, that attention is then often shifted by a pan or cut in the image on screen. This approach is still employed in Collisions, and other 360° and VR works, but the affordances of VR headset technologies mean filmmakers must always attempt to draw the audience’s gaze, instead of being able to direct it entirely. This draws on the intuitive perspective that no interesting sound stays behind one’s head for very long. While this was used as a cautionary warning for consideration in 5.1 cinema mixes to avoid the distraction of Ben Burtt’s ‘exit sign effect’ (Holman, 2008: 116) where ‘sound objects emitting from the back and from the sides of the movie theatre are experienced both as distracting and disturbing’ (Elvemo, 2013: 33), it has now become a functional opportunity as hearing is called upon to direct one’s vision. And while cinema’s ‘multichannel soundtrack … is able to enlarge the diegetic space and immerse the audience in it’ (Manolas and Pauletto, 2009: 39), this immersion has previously always been an augmentation to give life to ‘the rectangle’ at the front of the room. The use of sound in Collisions is explicitly intended and driven by Wallworth from the outset, contrasting the cliché of film sound being addressed last in the production workflow and with little consideration from writers or directors. Wallworth demonstrates her focus on this aspect of the work in a promotional trailer for Dolby Atmos:

EMBODIED AUDIOVISUAL EXPERIENCE

53

Sound is so important in VR. It’s important because it becomes a tool that guides and directs the viewer. You’re calling attention continually and because you have the capacity then to see 360 then you use sound to frame the viewer’s experience. (Dolby Laboratories, 2016) Wallworth recognizes that the reality of 360° images and audience control of cinematic gaze changes the place of sound in her work, and that sound can be used as a tool to direct attention in a way that is neither possible nor necessary in screen-based cinema. The 360° images and movement of VR technologies change the significance of on- and off-screen sound. The distinction can still be made, but diegetic on-screen sound can always become off-screen sound and vice versa. Technically, sound can still be used to extend the screen space (Chion, 1994: 86–9) but it’s an extension which can be realigned with image at any time. The shift to viewer-controlled screen space affords new opportunities for the use of sound as well as necessitates new approaches to directing audience attention, realities that Wallworth handles deftly. Collisions is clearly a work produced using new workflows, as called for by Schütze and Irwin-Schütze (2018). Wallworth explains in more detail: There are trigger points or key moments that then make you look up if the action is above you, or if you need someone to look behind them, if you … there’s a tiny thing happening over here that someone might miss … because they can look anywhere in that space. So, that sound is incredibly effective, because you get that sense of where the sound was located. (Dolby Laboratories, 2016) Her point about knowing where sound is located requires further explanation. Sound mixing involves locating sounds in space, whether that is a stereo field, a 5.1 or other multichannel speaker system, or even a mono, or single channel, mix for AM radio, where depth becomes particularly important. The location of sounds in Collisions is, however, especially perceptible and this is due to the use of specialist techniques and software so that sound can be placed behind audiences in the mix with relative accuracy. Dolby Atmos is a standard developed to give spatial dimensions to audio mixes across mobile, home and theatre speaker systems. It allows sounds to be positioned in 3D space in a mix, independent of the output system, which will decode the mix to the correct output channels. This allows a 360° sound mix to be decoded into a binaural stereo headphone mix on a mobile device. Collisions goes one step further still, using Dolby Atmos but in an implementation in which the mix responds to the movement of the audience so that it is possible to turn towards Morgan’s voice, as previously described, and hear a corresponding shift in the location of his voice. Wallworth is enthusiastic about the possibilities this provides and describes ‘the joy of

54

MATERIALIZING DIGITAL FUTURES

seeing what would previously be framing being directed via the atmos. The sound is drawing my attention here, the sound is calling me over here … it’s giving me this incredible sense of being there’ (Dolby Laboratories, 2016). Collisions demonstrates the potential for sound to provide practical solutions for visual challenges caused when the reactive focus of the directed cinematic gaze is relinquished to the volitional vicissitudes of audience attention. But the use of audiovisual relationships involves more than exploiting sound’s exit sign effect to the director’s advantage. Beyond the potential for sound as a simple technical fix, Wallworth’s understanding of the immersive possibilities of audiovision services a far greater agenda. Whether in her VR projects or her other installation works, Wallworth uses digital media to ‘call us all inside these stories … to make spaces’ to ‘invite you in’ (2015). And in Collisions, the technology is crucial in offering a sense of place: ‘not just what it looks like in the desert but what it feels like’ (2016). Sound, as much as vision, is at the centre of this experience. If VR in Collisions uses digital technology to ‘amplify the possibility of connection’ (Wallworth, 2015) at an affective level, the agenda of 4D’s ‘connection amplification’, by comparison, is much more physical. And if VR is opening up the potential for new forms of cinema, then 4D (like 3D before it) can be argued as remaining firmly sutured as an authenticating digital ‘value add’ to a more established cinematic approach. Whereas a sense of immersion from a ‘light agency’, the ability to look around, is technologically facilitated in VR, 4D cinema can be seen to offer a ‘simulated agency’ through digitally choreographed haptics, attempting to somatically bind audiences to the audiovisual action on the screen ahead.

Venom The film Venom (2018) is based on the Marvel Comics character of the same name, telling a superhero (or villain) origin story. In addition to more traditional formats, it was released in 4DX, a 4D cinema format in which audiences sit in specially designed chairs that, in concert with additional technologies fitted to the theatre, support the supplementation of various practical effects to the cinematic experience, such as motion, wind, fog, mist, scent and lighting. These 4D technologies have been described by cinema academics such as Christine Petersen as a ‘Bazinian development towards greater narrative realism through embodied spectator participation’ (2019: 7). The experience is described on the Event Cinemas website as ‘providing a revolutionary cinematic experience that stimulates all five senses’ with effects that ‘synchronise perfectly with the action on screen to fully immerse you in the movie’ (Event Cinemas, 2020). However, watching Venom in 4DX, we mostly found that the experience distracted from immersion in the film itself, sometimes with the sound of the motors of the

EMBODIED AUDIOVISUAL EXPERIENCE

55

fans used to produce wind effects, or with a spray of water from the seat in front drawing vision away from the screen, but most of all with the motion produced by the chair. One scene in particular demonstrated the issue. The film is set in San Francisco and includes a dramatic motorcycle and car chase that exploits the city’s steep streets. During the scene, the 4DX motion chair lurches back and forth violently, simulating the crunching suspension of speeding cars crashing down these streets. Frequently while this occurs, however, the camera is positioned outside the various vehicles, tracking the action. This results in a dissonance between the audience’s positioning in the narrative space of the film, through the camera angle, and the visceral, embodied experience being provided by the motion. The sound design is also dominated by relatively generic crashing vehicle and screeching tire sounds and so the motion the audience experiences is often not supported by specific sonic information. It seems that the design of this sequence is supposed to immerse the viewer in the action but instead we found it highlighted the constructed nature of the multisensory experience and thus interfered with any heightened sense of immersion in the action. This is perhaps best explained by way of contrast with another scene in the film, where the use of the motion chairs is far more successful. In the scene some ‘bad guys’ go to the apartment of the protagonist Eddie Brock, who becomes Venom after being exposed to an ‘alien symbiote’. As Brock answers the door, the antagonists burst in, and an action sequence ensues. At a key point in the sequence, the smash of a bad guy into the neighbour’s door, seen from the hall, is accompanied by a sharp jolt backwards from the motors in the motion chair, a powerful effect, both physically and affectively, that is heightened by the shots and fast editing employed. The backward motion the audience experiences is consistent with the spatial positioning of the camera. Further, there is a tight alignment between the sound design we hear, of the bad guy hitting the neighbour’s door, and the motion we experience as we are knocked backwards. Contrasted with the chase scene, the apartment fight is viscerally immersive because the spatial positioning of the audience in the narrative space of the film is consistent between the cinematography, the sound design and the motion simulation – the points of view, points of audition and points of kinetic experience each align. This is not to say that the fight scene is more realistic. It is still obviously a highly constructed digital experience. The sound design employed aligns with what an audience member expects from an action film, rather than what someone being hit ‘really’ sounds like, and the motion design is consistent with this rather than what it would necessarily be like to be smashed into a door such as the one in the film. Although Venom seems to stick to a reasonably traditional approach to sound design, it does demonstrate the way Chion’s principle of ‘synchresis’ (1994: xviii), the perceptual linking of synchronous sound and visual events

56

MATERIALIZING DIGITAL FUTURES

on screen, can be extended from audiovisual to audio-kinetic relationships, and the potential impact of employing techniques based on this logic, especially when the spatial positioning of the audience is considered. Central to the inconsistency of the 4DX experience is the fact that the encoding of kinetic gestures is usually an add-on to an already existent piece of film, rather than an organic consideration of multisensory integration during its construction. As much as certain film scenes will provide affordances for logical and engaging amalgams of different modalities, there will be potentially more distracting problems of misaligned points of audition, view and kinetic experience until such considerations are built into the overall design from the outset. As with many of the new media technologies, however, there is a tension between the opportunities for creative exploration and the economics, which may impede such an objective. This 4D technology is designed to service the most popular end of entertainment, coming at a price point which is well and truly out of reach for independent practitioners, not to mention proprietary restrictions on how it can be deployed. As such, it is unsurprising that its full creative potential, arguably, is yet to be reached. Reflecting on the examples thus far, the role of sound in relation to an immersive field – whether in VR cinema or to support a perceptual meld between vision and motion in 4DX – can be seen as serving a clear objective function. It builds on precedent, both everyday experiences of sound and established narrative, as well as production precedents in cinema. Collisions uses audiovisual devices to assist audiences in navigating the narrative content of the film and to deepen empathetic response. Venom uses multisensory immersion to amplify arousal and, to hark back to CinemaScope’s claim for widescreen in 1953, to put ‘you in the picture’ (Belton, 1996: 265). So, what happens when an artist eschews established conventions and does something with digital technology that more playfully engages with the new affordances on offer?

Hobo UFO (v. Chernobyl) James Hoff’s HOBO UFO (v. Chernobyl) was released on the popular experimental music label PAN in 2019. It followed a series of performances titled HOBO UFO, in which he performed live, beats-driven sets of music, with visuals produced via the manipulation of Google Street View’s representation of the area surrounding each venue. Hoff (2019), with the assistance of coder and artist Reuben Son, developed a setup to get Street View to respond to audio, producing a first-person perspective experience that is driven by his music, engaging the digital systems of which McLuhan, Murray and we write, on a planetary scale. During the performances, presented completely from the standard Google Street View first-person

EMBODIED AUDIOVISUAL EXPERIENCE

57

viewpoint, the visuals would zoom forward and back with accompanying blurring and blackouts, often stopping abruptly and switching to dizzying horizontal spins. Here the multisensory experience is driven by its sonic component, interacting with global systems of surveillance and computation, to affect the embodied, and located, experience of audiences. Whilst Collisions offers the audience light agency in aid of navigation and empathy, and Venom simulated agency for deepened embodiment, HOBO UFO is quite different. It exploits the contrast between denied audience agency and the visceral visual representation of a violently moving point of view. The result is intriguing. When composing what would become HOBO UFO (v. Chernobyl), initially commissioned by the Polish festival Unsound in 2017, Hoff worked extensively on the piece before visiting the Chernobyl exclusion zone in 2018 and changing direction (Hoff, 2019). Eschewing the conceptual and software-driven approach he had taken with the work to date, he instead composed a delicate, and quite ambient, piece including everything from woodwinds to GPS signals recorded on site, and used these to produce a visual experience that takes audiences from the empty streets of the formerly grand Soviet city of Pripyat into the Chernobyl nuclear facility, the Duga Radar System and the surrounding landscape, before taking off into the black void of digital space, in keeping with the UFO reference in the work’s title (Hoff, 2019). The tone of the release is very different from that of the performance, but it retains a close relationship between the movement of the camera’s perspective and the audio. The first-person perspective remains throughout, as do the stylized arrows of Google Street View’s user interface. While the work is still screen-based audiovision, its internal logic is free of any narrative obligations, instead driven by its musical aesthetic, intersecting with the affordances of digital code. As such, it is an example of a practice unconcerned with either authenticating an environment through sound design or synchronizing audience emotions musically. At its core is a literally dizzying disjunct between rapidly spinning Google-provided images and a dispassionate ‘soundtrack’ unfettered by service to story or any clearly discernible heightened emotional agenda. It is a testament to how digital tools themselves can play a crucial part in exploring new audiovisual possibilities – where the idiomaticism of the digital system being developed opens up options unrestricted by previous histories and expectations of ‘appropriate’ multimodal relationships. The mediation of code in the relationship between sound and image should not be underestimated. Unlike VR which employs digital media’s manipulation of sound to solve a visual and interaction design problem, or 4D, which adds an additional sense to corroborate pre-existing others, works like Hoff’s demonstrate how it is the underexplored possibilities of digital media and its capacity to bend, break and blend established conventions – to use Brandt and Eagleman’s definition of creativity (2017: 47) – which can result in the generation of innovative artwork. To flip Schütze’s (Schütze and

58

MATERIALIZING DIGITAL FUTURES

Irwin-Schütze, 2018: 273) assertion, with ‘brand new methods of production’ we can start exploring brand new forms of art.

Conclusion Digital media are usually multisensory, already plural and without the mechanical stipulations of the appropriate modalities for data common to many analogue media, a reality that affords significant opportunities for working creatively across the senses. Collisions, Venom and HOBO UFO (v. Chernobyl) all use digital technologies to create multisensory experiences that rely on a connection between sonic, visual and kinetic experience in manners that extend well beyond sound as just ‘added-value’. Synchresis in these digital media takes place with the pairing of sound and movement at different levels, movement on-screen still but also bodily movement, whether imposed by cinematography or kinetic effects or structured by interaction design. In different ways and to varying degrees of success, they feature movement-based interactivity that is guided by sound. Audio is used to guide the audience’s gaze in Collisions, Venom in 4DX uses sound to connect kinetic motion chair-based effects with action on screen, and in HOBO UFO (v. Chernobyl) sound directly controls camera direction, producing kinaesthetic affects through audiovisual movement. Each explicitly positions audiences spatially. This is most affective when supported by tight synchretic relationships between audio, vision and interactivity, whether connected to user agency or the interaction of modalities in the work. Interactivity in each contributes to creating and making convincing a first-person, and so explicitly embodied, point of audition and thus experience. The way this is achieved ranges from audiences being able to control the direction of listening in Collisions, to haptic confirmation of location in the diegetic space of the film – through which audiences are positioned in the space physically as listeners – in Venom, to sound being used to navigate 3D space on audiences’ behalf and soundtrack that movement in HOBO UFO (v. Chernobyl). Together these creations serve as examples of a developing field of practice around embodied multisensory digital experiences, in which sound performs a key role.

References ACMI (2016), ‘ACMI Collisions’. Available online: https://guides.acmi.net.au/ collisions/ (accessed 2 April 2020). Belton, J. (1996), ‘Technology and Innovation’, in G. Nowell-Smith (ed.), The Oxford History of World Cinema, 259–67, New York: Oxford University Press. Benayoun, M. (2011), ‘Art after Technology’, in The Dump, 207 Hypotheses for Committing Art’, FYP Editions.

EMBODIED AUDIOVISUAL EXPERIENCE

59

Brandt, A. and D. Eagleman (2017), The Runaway Species, Edinburgh: Cannongate Books. Camper, F. (1985), ‘Sound and Silence in Narrative and Nonnarrative Cinema’, in E. Weis and J. Belton (eds), Theory and Practice of Film Sound, 369–81, New York: Columbia University. Carpenter, E. and M. McLuhan (1970), ‘Acoustic Space’, in E. Carpenter and M. McLuhan (eds), Explorations in Communication: An Anthology, 65–70, London: Jonathan Cape. Chion, M. (1994), Audio-Vision: Sound on Screen, trans. Claudia Gorbman, New York: Columbia University Press. Collisions (2016), Documentary, dir. Lynette Wallworth, 2016. Davies, C. (1998), ‘OSMOSE: Notes on Being in Immersive Virtual Space’, Digital Creativity, 9 (2): 65–74. Dolby Laboratories (2016), Video, ‘Dolby Atmos Takes You to the Outback in Jaunt’s “Collisions” ’. Available online: https://www.linkedin.com/pulse/ virtual-reality-hot-its-success-depends-great-content-john-couling (accessed 2 April 2020). Dyson, F. (2009), Sounding New Media: Immersion and Embodiment in the Art and Culture, Berkeley: University of California Press. Eisenstein, S., V. Pudovkin and G. Alexandrov (1985), ‘A Statement’, in E. Weiss and J. Belton (eds), Film Sound: Theory and Practice, 83–5, New York: Columbia University Press. Elvemo, J. M. (2013),‘Spatial Perception and Diegesis in Multi-Channel Surround Cinema’, New Soundtrack, 3 (1): 31–44. Event Cinemas, (2020), ‘4DX’. Available online: https://www.eventcinemas.com.au/ Promotions/4DX (accessed 25 March 2020). HOBO UFO (v. Chernobyl) (2019), Music Album, Comp. James Hoff, Pan Studio, London. Hoff, J. (2019), Video, ‘HOBO UFO (v. Chernobyl)’. Available online: https://www. youtube.com/watch?v=ERbfczLUr-A&feature=youtu.be (accessed 2 April 2020). Holman, T. (2008), Surround Sound – Up and Running (2nd edn), Amsterdam: Focal Press. Mann, A. (2016), Video, ‘Aboriginal Man’s Story of Maralinga Nuclear Bomb Survival Told with Virtual Reality’. Available online: https://www.abc.net. au/news/2016-10-07/aboriginal-mans-story-of-nuclear-bomb-survival-toldin-vr/7913874 (accessed 2 April 2020). Manolas, C. and S. Pauletto (2009), ‘Enlarging the Diegetic Space: Uses of the Multi-Channel Soundtrack in Cinematic Narrative’, The Soundtrack, 2 (1): 39–55. McArthur, A., R. Stewart and M. Sandler (2017), ‘Sounds too True to Be Good: Diegetic Infidelity – The Case for Sound in Virtual Reality’, Journal of Media Practice, 18 (1): 26–40. McLuhan, M. (2005), ‘Visual and Acoustic Space’, in C. Cox and D. Warner (eds), Audio Culture: Readings in Modern Music, 67–72, New York: Continuum. Metz, C. (1975), The Imaginary Signifier: Psychoanalysis and the Cinema, Bloomington: Indiana University Press. Metz, C. (1985), ‘Aural Objects’, in E. Weis and J. Belton (eds), Theory and Practice of Film Sound, 369–81, New York: Columbia University.

60

MATERIALIZING DIGITAL FUTURES

Murray, J. (2012), Inventing the Medium: Principles of Interaction Design as a Cultural Practice, Cambridge, MA: MIT Press. Petersen, C. (2019), ‘The Address of the Ass: D-BOX Motion Code, Personalized Surround Sound, and Focalized Immersive Spectatorship’, Journal of Film and Video, 71 (1): 3–19. Schütze, S. and A. Irwin-Schütze (2018), New Realities in Audio: A Practical Guide for VR, AR, MR and 360 Video, Cleveland, OH: CRC Press. Sonnenschein, D. (2001), Sound Design: The Expressive Power of Music, Voice and Sound Effects in Cinema, Century City, CA: Michael Wiese Productions. Venom (2018), Film, dir. R. Fleischer, Columbia Pictures. Wallworth, L. (2015), Video, ‘Visions of Storytelling: Immersive Storytelling’, https://www.youtube.com/watch?v=1M22WaL6nB8 (accessed 4 May 2020). Wheatley, C. (2019), Stanley Cavell and Film: Scepticism and Self-Reliance at the Cinema, London: Bloomsbury Academic.

4 Volumetric Black: Post-cinematic Blackness Triton Mobley

Introduction Within the margins of accessible histories of visual documentation of Blackness what are the metrics for a qualitative rendering of the sociocultural and historic value of a Black aesthetic?1 With the many existing adaptive processes making only partial renderings of Black imagery, is there an associated cost to the mass consumption of these inconclusive composites? How might the chemically fixed realities that are (con)c/sealed from a once expansive cultural memory attached to a more expansive architectural environment – cropped within the photographic frame of an image be recovered? What are the retrieval processes – exclusive in their registers unable to convey with intention the scars on Black bodies and the lives of those on the margins? How might the photographic renderings of Blackness – free of technological distortions – possibly serve as a mode of active resistance within the frame? Is it too much of a speculative imagining to conceive of a mediated visual experience that could document, project, broadcast and stream a more complete composite of Black bodies subjects with enveloping granularity? Is it a futile exercise to consider what a dominant visual culture and its corresponding industries might have been if racial equity, through the inclusion of Black bodies, had been a consideration in the formation

62

MATERIALIZING DIGITAL FUTURES

of image-making technologies? Perhaps more importantly, could such a speculative reimagining provide fertile ground for creative possibilities to intervene through art hacktivist outputs to reclaim the lost technological potential? It is the speculative reimagining that I intend to place at the fore throughout this chapter on the issue of the technological and computational absence of Blackness throughout the history of image-making. I also put forward an outlined protocol for a kind of bug clustering2 the existing image-making technologies designed to both break the technologies, and expose its failures, radically transforming the Black image viewing experience. I conclude by offering a series of speculative design and interventionist art hacktivist possibilities for remaking these technologies. This will not be from an existing vantage point of pre-production coded3 adaptations, nor will it form the post-production milieu of corrective processes, but rather be an exacting critical proposal at the inherent limitations of image-making technologies. The racially codified techniques for supplemental image corrections before (and after) capture that is a specific predeterminant process for the imagemaking of non-white bodies. These are variances in the configurations, and reconfigurations of the filmic space (Jaikumar, 2019); the conceptual arrangement of both material and immaterial – culturally racial experiences, recording technologies and architectural manifestations bound together by registers of economies. The specific mode of configuration for the filmic space that I attempt to assemble in this chapter, through the combination of historical and contemporary aesthetic markers and speculative design, is one that is dependent on modularity in its functionality. I further layout within this document descriptions of interventionist ideations that operate as alternative reflections on the history of Black image-making – an existing practice that I argue serves primarily as an adaptive process. I will define a dynamic reconfiguration of digital perceptions designed to be constitutive of a new visual Blackness. The cross sections of mediated Black experiences within digital and cultural optics I refer to as Volumetric Black. When one considers the history of image-making systems of technologies and chemical emulsifiers, the technical limitations to produce any viable discernment within variations of Black bodies are inescapable. Whether it was the failure of photosensitivity to construct an image that did not have whiteness centred as a default calibration, or the absence of non-white bodies in the creation and distribution of the ‘reference’ images in the form of Shirley cards (Roth, 2009: 111–16), image-making technologies appear to have never felt the need to explicitly correct this oversight, but instead forged ahead through image-making innovations devoid of Blackness. If there had been non-white collaborators involved in the research and development procedures of chemical processing for the multinational film manufacturers that brought us the image rendering substrate that we know today, perhaps its photosensitivity reactions would not respond well only to light(er) skin

VOLUMETRIC BL ACK

63

(Roth, 2009: 13). To have had access to an image rendering substrate, and its corresponding devices, that placed the embellishment and aesthetic representation of Black bodies prioritized above the indiscernible variations of wood furniture or chocolate bars (Roth, 2009: 119) in low light prior to the mid-1960s would arguably have brought forth advancements for the field. What cultural effect might this have had in producing an alternate history of Black visual aesthetics and the fundamentally invalid technological mediations through which we have only ever seen images of ourselves reproduced? For as long as these technologies have existed, we, purveyors of Black bodies and culture, have only ever seen ourselves as the mirrored image bounced off the technically engineered reflected surfaces calibrated for whiteness. And due to this fundamental error in chemistry, engineering and computation we may have never truly seen ourselves. This image-making history that brought about photochemical processes and filmic production standards centred on whiteness has also replicated a similar centricity within the engineering of the digital photo and videography technologies4 that consumes much of our mediated realities for the foreseeable future. This is by no means a suggestion to demote or reduce the vibrant visual culture that has been established by the Black American diasporic pre-production and post-production innovations. Not at all. However, it does speak to the shared approach in computational adaptations and workarounds utilized by many of the industry technologies. The seemingly ubiquitous ‘black box’ of computational recipes for constructing multi-image composites for producing ‘likeness’. This is an attempt to reconcile historical failings of the (image-making) technologies and the forced development of adaptive measures to produce Black aesthetics in image-making. This is part critical imagining and part speculative art praxis to see what the Black American aesthetic might have been if it was not largely dependent upon the graphing of (pre/post)production improvisations onto Euro-centric imagemaking, cultural and technological, standards that have been maintained in the ‘New World’. As a primary mode for manifesting actionable tools within a speculative design space, I use curated historical and theoretical references as departure for the distillation of improvisational achievements within aesthetic Blackness. This reclamation effort is attempted through a combination of tactical media protocols of interventionist art hacktivist tools and creative praxis bent on remaking and reengineering the existing technological apparatuses based in functional fabrications of speculative design.

A cultural optic To adequately qualify the metrics for what I define as the Volumetric Black aesthetic, it is necessary to outline this framing with a key historical cinematic

64

MATERIALIZING DIGITAL FUTURES

figure in the production(s) of early Black cinema and image-making. In the early 1900s Oscar Micheaux, a filmmaker and producer of ‘Race Films’, created and operated the Micheaux Film Corporation. With his eyes set on creating Black films for Black audiences, starring all Black casts, Micheaux made a name for himself throughout segregated Black communities and urban centres, though not always under the best circumstances within the Black community. He even made a name for himself with many of the states’ film advisory boards for the overtly racial themes depicted in his films (Regester, 1995a,b). Although Micheaux was successful in constructing his own narrative aesthetic for Black productions, when all was said and done the technological resources that he employed for his productions were the same image-making devices and technologies that were never intended to document and correctly expose Black bodies. Blackness in the West has always been rendered not only as ‘other’, but to a more devastating effect, even today, as a dangerous other. This imagined fear was exemplified in the ‘Black faces’ on the white actors in The Birth of a Nation (Griffith, 1915). D. W. Griffith’s vision of the American threat can perhaps best be described by the film’s direction of an all-fronts assault from the haunted(ing) Black (Cripps, 1977) bodies delirious and mad, lusting for the divine ‘purity’ (Siomoupolis, 2006) of white(ness) women – who can only be saved by the Klan. This visual history of Westernized representations of Black bodies on screen has had a profound influence on all American culture – its past, present and future. Perhaps more importantly, it also further biased both white and Black audiences against Black communities, about what they should expect both on and off screen in their permeable real-world encounters. Although not widely released, Micheaux filmed what was said to be a direct response to Griffith’s 1915 film The Birth of a Nation. The later 1920 film, Within Our Gates, was seen as an attempt to counterprogram what Micheaux saw as the intentional misrepresentation of the Black race (Grupenhoff, 1988). Micheaux’s race films provided a platform for both the bodies and voices of Black performers to be represented as themselves by themselves. The advent of these productions meant that Black audiences would no longer need to be compelled or limited to the white-owned filmic space(s) that merely presented them and their culture(s) solely as Hollywood stereotypes. Despite having a significantly smaller budget than the rivalled Hollywood productions and concerns over lacking other production resources, Micheaux’s films still managed to be screened in concentrated Black communities across the Midwest and the South. Although these films provided an alternative viewing experience for many Black audiences, some local papers reported about the disapproving reception for many race films within Black communities. Media Studies scholar Mary Carbine offers Thomas Cripps’ reading of the accounts ‘that black audiences preferred [the] “slick Hollywood product” over “race” movies ineptly

VOLUMETRIC BL ACK

65

produced by Blacks. In large part, this pattern was the result of economic factors related to discrimination in the industry, which tipped the balance in favor of Hollywood product even in black venues’ (Grupenhoff, 1988). In addition to not having the available resources of the Hollywood productions, and all of the economic constraints, Micheaux further compromised some of his films by occasionally emulating common racist tropes and stereotypes of the era. His trafficking in covert respectability politics, as reported in newspapers, caused outrage and the boycotting of some of his screenings by Black communities and Black organizations in a few cities.5 Because neither the image-making substrate nor any of the entities of mass produced filmed entertainment were ever concerned with the spectral specificity of Black bodies, this filmic space with its ability to shape culture gave the dominant white filmmakers a kind of authorship over the cultural, social and visual remaking of Blackness for the American moviegoing audience. Micheaux’s contributions to this visual space only further delayed efforts in the making of an uncompromising Black aesthetics on screen. Many read Micheaux’s choice to assign specific character roles based on the skin colour of his cast – that is, the use of lighter skin actors playing thinking characters and darker skinned actors in the role of criminals – as a perpetuation of the existing stereotypes of the dominant white society (Green and Neal, 1998: 67). This is not to suggest that all of Oscar Micheaux’s intentions were poorly considered, he remains both a controversial and contradictory historical character among scholars for this very reason. However, I am making a case for the structural failures in the history of Black image-making and the continued efforts of the time by a white majority class in Hollywood to actively erode efforts in establishing a Black aesthetic. The resulting composite of Micheaux’s works and the role they played in this process, factoring the targeted distribution, should also be included. How might the projections of a conventional otherness, with the marginal semblance, alter a community’s disembodied self-image? Over a sustained period of time could the filmic space, a configuration of creative expression, novel spectacle and passive racism, compel Black audiences to adjust not only the way they see themselves but how they encounter one another outside of the films’ frame? Could the flickering images of the cinema have been a precursor to the ‘researches that might make it possible for the miserable Negro to whiten himself’, as Fanon suggested (Fanon, 1967: 111)? There is a documented history making the case for how the economics of Hollywood’s productions due to their production value and film distribution created unfavourable economic challenges for both Black filmmakers and venues. What remains are questions of the cultural implications of the filmic space and whether its contents function as signifiers of visually mediated denegrification (Fanon, 1967: 111).

66

MATERIALIZING DIGITAL FUTURES

Otherness outside the frame From the arrival of Portuguese, Dutch, British, French and other European vessels and merchants washing ashore on the western and southern African coasts from the sixteenth century, and enduring over the three centuries that followed, the Black body was assigned the ability to generate capital and was in that moment rendered the object of mediated experiences to be commandeered. The Black body was appropriated and refashioned in the New World as both an object of fear as well as showcased and marketed to non-Black bodies that this new object of fear was also the source of their superiority in the world. Throughout the history of the Black body as object of fear, Black diasporic peoples, through active defiance, have resisted these connotations of their undesired existence in the New World. These resistive acts through protest were always constructions of violence (the attempts at freedom were met with violence, the system of enforced containment was kept through violence and in the white imaginary the Black body was rendered as a kind of visual violence), which is all they could ever be given the circumstances that brought about the colliding forces of the transatlantic slave trade. In considering one of the first acts of redefining the terms of their capture, carried out by Black bodies of resistance, one must follow the triangular path that constructed the Middle Passage, connecting not only continents but more importantly the trails of lost bodies desperately fleeing for an alternative survival. The Middle Passage conjures the ‘always present throwing and jumping overboard, and the fish fed and feed on those bodies in the wake of the ships’ (Sharpe, 2016). The second act of resistance enacted by the Black body in defiance of their captivity was through the active devaluing of the object of capital through self-imposed starvation.6 For those that had to witness the torture of fellow slaves being force-fed through broken teeth, making it to the end of the voyage alive must have been an inconceivable calculus into the unknown. Upon arrival to the New World the auction blocks at the ports of arrival in the Americas served as the first elevated frame in which Black bodies were displayed, humiliated and brutalized for the satisfaction of white audiences. It was also where the observing crowds present were both enriched and frightened yet reassured about the security of their place in life. Another cruel and devastating exercise in the further dehumanization of the Black body was done through the branding of slaves with letters/initials of the awaiting slave owners. Across multiple ports of the transatlantic movement of slaves between the African continent, Europe and the Americas this process was used to identify and maintain surveillance of Black bodies. This process only led to additional modes of resistance by the appropriated Black bodies and their commodified abilities for manual labour, breeding and caretaking. Slaves would attempt night-time escapes on foot, despite knowing that the consequences could

VOLUMETRIC BL ACK

67

very well lead to death if found or captured (Browne, 2015: 101). Elsewhere in the Americas the same tactic of branding Black bodies by white owners was subverted in other ways as described in Simone Browne’s research on ‘Branding Blackness’: In Slavery and Social Death: A Comparative Study, Orlando Patterson explains that slave branding ‘backfired’ in Brazil, where the letter F that branded a recaptured runaway was ‘proudly displayed’ to the ‘more cautious but admiring fellow sufferers’, marking its resignification as a mark of honor, not of capture. (2015: 102) When Black (American) bodies, culture(s) and identity were rendered merely as object-of-capital, this act of (economic) violence served only to exacerbate the circumstances of their captivity. This form of appropriated Blackness, operating both as object of capital and the creation of mythologies for safeguarding white supremacy, should serve to acknowledge the limited means to resistance left available (through the acts of jumping overboard along the transatlantic voyage, refusing food and medicine to the point of starvation and fleeing on foot through a foreign land) knowing that death was the likely consequence. These once lived Black experiences constitute an exemplar of resistance to discrimination against Black bodies in the Americas as a form of media (as tools of production) in the sixteenth century. The entire process of removal of Black bodies from the African continent, to be relocated to ‘civilized’ societies in Western Europe and the Americas presents a curious rearrangement in the reading of cultural appropriation. With the extraction and replacement of language from enslaved peoples with that of a white majority class in the New World,7 the resulting output is a sensorial immersion through a forced cultural assimilation that can only bear trauma. This reality reconfigures the dynamics by which Black bodies are literally communicating culturally in the New World, priming new possibilities for cultural resistance. Prior to reconstruction the Black body was only permitted to exist as object-of-value(s) one derives from property ownership. In the era of Jim Crow, the Black body as object-of-value(s) slowly transitioned to become identifiably Black body as object-of-other. This process is one that develops through a century of conflict, mass death and what Fanon would later describe as the denegrification of the Black body (Fanon, 1967: 111). This attempt at denegrification played itself out through the productions of media entertainment at the turn of the twentieth century. It was clear that the goal of ‘throw[ing] off the burden of that corporeal malediction’ (Fanon, 1967: 111) was not aimed at making the Black body equitable to whites in filmic representation, but rather the goal was to make the Black body less dislikeable to white sensibilities, as the reinvented object of economic value, on the one hand (e.g. the buffoonish slouch, the docile uncle and Stepin

68

MATERIALIZING DIGITAL FUTURES

Fetchit characters), and to maintain a safe distance from the present danger on the other (e.g. Griffith’s crazed savages). This was achieved with great success through the image-making possibilities of minstrel productions and Blackface characters on screen in movie houses. Notably D. W. Griffith’s The Birth of a Nation that was released in 1915 and appropriates what he believed to be the Black body as ‘other’ with desires for whiteness (Griffith, 1915). As it was seen clearly through the provocations of productions like Birth of a Nation, the simulacra of Blackness was (re)weaponized through this new image-making medium (Baudrillard, 1994). In the wake of Griffith’s production, elements of the great migration in the north and upper Midwest saw vastly different possibilities for the new medium outside of merely consuming its contents. Due to the newfound opportunities of employment and living standards in the region (Carbine, 1990: 12), the circumstances also brought about access to disposable incomes for many, allowing members of the Black community to consume this new media on their own (economic) terms. Mass culture and commercial entertainment are seen as operating with inexorable force to integrate workers and immigrants into mainstream society. (Carbine, 1990: 10) Although the move from the south to the Midwest alleviated many of the daily traumas Black bodies experienced as other, during this period of Black migration there were still many race-related challenges that did not go away. The communities in Chicago and Detroit that were being built up by the influx of Black southerners were still very much segregated in many ways. There were still white parts and Black parts of town, and the enduring indignities of contending with the fact that whites received preferential treatment over their Black neighbours in all the accommodations they sought. The pervading attitude in the separation of the races was also reflected in the Hollywood entertainment that many in these new Black communities of the Midwest would avail themselves of in their leisure time. However, the indignities that many in the community felt they had to endure in order to attend white theatres, screening films that would not be representative of their lived experiences, was a trade-off that some were no longer willing to make (Carbine, 1990: 18). The vacuum created by the racialized practices of white theatres, and the Hollywood productions they screened, made way for Black owned production houses, filmmakers and their actors (though predominantly screened in white owned theatres in Black communities) to produce stories for Black people, about Black experiences. However, to the disappointment of some Black journalists and film critics of the time, the distribution and screening for many of the ‘race movie’ (Carbine, 1990: 20, 27) titles did not have a broad appeal among segments of the Black community. As other members of the community still

VOLUMETRIC BL ACK

69

had much interest in consuming brand name products sold in the shops on the stroll (Carbine, 1990: 19), and demonstrated a preference for screening Hollywood films due to their high production value, the white theatre operators in Black neighbourhoods found a way to culturally bifurcate their audiences, while maximizing profits. For some Black audience members, it was the show before the show that brought them to the theatre. There they were able to revel in the sights and sounds of Black performers from the vaudeville circuit (Carbine, 1990: 21). These live performances constituted an expansive, multidimensional Blackness that spilled outside the frame of the filmed image: As long as the black man is among his own, he will have no occasion, except in minor internal conflicts, to experience his being through others. (Fanon, 1967: 109) The history of media representation of Black bodies in America, whether broadcast over radio or projected from celluloid, alienated many Black audiences with Blackface caricatures and other forms of minstrel performances – programming that was wedded to productions of racial stereotypes having a broader appeal mostly with white audiences.8 However, the transformation of the entertainment infrastructure – the film theatre – became something entirely different for members of Chicago’s Black metropolis, repurposing the racially patronizing context with a layering of aesthetic Blackness that would provide the kind of insular,9 culturally specific entertainment appealing to Black audiences in predominantly white owned theatres (Carbine, 1990: 25).

Filmic infrastructures The previous sections of this chapter discussed in detailed the distinct, yet connected, heterogeneous markers of image-making, binding associations of racialized motivations, cultural and economic activities, and cultivated forms of spectatorship producing foundational criteria for the constructed cultural infrastructures. To conclude this section, I expand on the component of the theatre(s) that showcased such events, and the forced adaptability these filmic infrastructure(s) had to preserve for variety in the form of exhibited content. It is hard to conceive of the filmic space (as cultural, economic and social constructions) and not consider the formal architectural infrastructure(s) that contain it, with its ability to transpose the context of image-making, content-altering dynamics of audience reception. The clustering components of filmic space operate, within its infrastructural environment, simultaneously on multiple different fronts. These elements operate on distinctive registers of social entertainment and function,

70

MATERIALIZING DIGITAL FUTURES

domestic economies, and perhaps most significant, cultural self-image. Not only are they responsive to redirecting interactions between individual audience members with one another, but perhaps most notably they limit opportunity for individuals to see and experience reflections of themselves. This was particularly true of race films screened in Black churches in the south or the jazz band accompaniment to Hollywood films in the theatres of ‘Black Metropolis’. Filmic space – as architectural space – conserved the function of reconfiguring the imaginative desires of the audience, either through the way they saw themselves within the frame or in relation to the frame through shared collective spectatorship. This cultural burden underpinned the limited permeability of social experience(s) of public life. As the audience we ultimately must physically contend with the filmicarchitectural space. The affected manner by which we contort our bodies and compose ourselves socially is made permissible by the filmic infrastructure – filmic-architectural space. I would argue that these many modes and rules for spectatorship were acutely felt by Black bodies as members of the nondominant racial and economic class in America. With consideration for the abilities of filmic-architectural spaces to create cultural misrepresentations and a partial reality that can penetrate receptive viewers, as well as socially mismanage the cultural behaviours of Blackness, publicly, one need to look no further than the visceral harm created in the wake of Griffith’s magnum opus (Benjamin, 1968). I am suggesting there is a correlation between the filmed events of imagined Black bodies printed on Griffith’s celluloid and the public spectators’ response to the film’s screening. Perhaps no single screening was more significant and alienating to an entire racial population than President Woodrow Wilson’s special screening of The Birth of a Nation at the White House (Benbow, 2010). The ‘People’s House’ hosted a known bigot and elevated the platform for his divisive wares. An anecdotal reference that best summates an understanding of what I am describing would be the experience I had during a screening of Steve McQueen’s 12 Years a Slave in New York (McQueen, 2013). The screening was at the BAM theatre (this detail is significant because of the sociocultural demographics of the Fort Greene neighbourhood). As all those who have screened and experienced the film will note, it is an unflinching depiction of the brutal violence that was slavery in America. As I sat in the theatre mostly occupied by white spectators and the film approached its final 30 minutes, the audible sobs from the audience began all around. In this moment I experienced a visceral mixture of heightened self–awareness and hypersurveillance brought about by the honest reflected image of my likeness from the contents’ frame and what I can only describe as the weighted performative guilt from those seated around me. This is not commentary on the sincerity or psychology of the other spectators in that moment, but rather a lived illustration of a culturally malleable and socially permeable environment made possible through the filmic-architectural space. In that

VOLUMETRIC BL ACK

71

moment a psychic reverberation seemed to correspond with the digital projections’ imperceptible flicker – an expression of Fanonian bodily schema that was all-encompassing. A weightiness that clung to me long after leaving the filmic infrastructures of the screening. The filmic-architectural space is a triptych of sorts. A phenomenon that combines the socioeconomic and physical impositions of filmic infrastructures. The spectral partialities of Blackness captured within the white light of filmic frames, and the cultural reverberations of performative spectatorship. Digital technology has given us all abilities to transport image-making possibilities, remaking filmic-architectural space(s) on the go. Whether it is huddled towards the end of a crowded subway car, mounted to a car dashboard in highway traffic, or 40,000 feet above the ocean on a transatlantic flight, we can engage with the cultural, economic, social constructions of the image – allowing a kind of spatial-sensibilities spill beyond the frame of a mobile device on the commute to work, carpooling children or passing through airport security checkpoints. A more tragic and widely known framing of this prescribed curation of a singular Western gaze for the Black body is exemplified in the transcript of the 911 call of George Zimmerman, on 26 February 2012 in Sanford, Florida (Rose, 2012). On the 911 recording, Zimmerman is heard actively negating Trayvon Martin’s Black body in his attempt to describe his observations to the emergency dispatcher. What was the distracted excitement of a teenage Black boy, meandering through the rain to steal extra time on a phone call with a friend, and would have registered as such if Trayvon occupied a different racial body schema, instead became the literal manifestation of ‘Maman, look a Negro; I’m scared! Scared! Scared!’ (Fanon, 1967: 91). Given the overestimations (Goff et al., 2014) made about Trayvon’s age, size and general description of his disposition – a constructed narrative imagined entirely outside of the image-making frame – yet its racialized context massaged from the medium of the filmic-architectural space. This is not an attempt to pathologize George Zimmerman. What is undeniable based on his own admission is that he received the same messaging – a conditioning of racial animus towards Black (brutes) bodies caught within the filmic, social, cultural, economic, racial frame(s), historically. This is a historic rendering set out before him just as it was peddled ninety-seven years earlier at the White House when United States President Woodrow Wilson invited D. W. Griffith to screen his film in 1915. The image-making process of distilling human experiences in the guise of empathy creation is at a very basic level a configuration towards othering. The process by which we consume these truncated realities is bridled by the infrastructural environment of the formal screening. As stated previously, the advent of digital tools has allowed for a mobilization for consuming these truncated realities, the (framed) focusing which excludes the possibility for rendering a complete composite – regardless of the content’s entertainment

72

MATERIALIZING DIGITAL FUTURES

value. Inaccurate estimations on Blackness (cultural and aesthetic) weather the same computationally. The exception, of course, is when you are a member of the dominant racial group. The history of image-making has informed our consumption for and recognition of a more limited rendering of a humanity within the frame (e.g. news coverage of mass shootings, ethnographic video essays on homelessness, VR Syrian migrant camp visits). When you inhabit a Black body, even this already heavily mediated construction of your humanity is not only cropped but is also not fully registered on the techno-cultural substrate. It is easy for dominant races anywhere to affirm the truncated(ness) of their realities beyond the frame simply by virtue of being the majority. The fullness of white visuality in America is always rendered to completion: within the boundaries of racially segregated suburban neighbourhoods, in the reflections of white performativity from non-white bodies, and it in the pervasive distribution of the white imagination across media culture(s). The white American cultural aesthetic governs standards on beauty, proper use of language, perpetrated violence, the passing normativity of heterosexuality, Christian religiosity and variations of racial appropriation that permeate much of the granular aspects of daily life for anyone outside of the majority class. ‘To photograph is to appropriate the [thing] that has been photographed’, Susan Sontag once wrote on the subject of photography (1977). In our collective screen induced existence both the framed image(s) and the technology itself renders a forced appropriation. What I see as the potential remaking of the filmic-architectural space is the confluence of analog and digital creative potential for expansive technological development of image-making and viewing systems. A provocation of penetrating, low-frequency audience discomfort and spectral interactivities between Black spectatorship and projected (streamed) image, converging within the filmic- architectural space. Where architectural infrastructures and filmic images (of Blackness) collapse, merging frame and audience in a functional immersion that is volumetric, the speculative design and interventionist art hacktivist tools reimagine the approach as a multifaceted jugaad (Rai, 2019). This is use of low-tech hacking innovation systems for problem solving originated in Southeast Asia, where engineered solutions employ both analog and digital interventions – combining mixtures of creative technical tinkering in mechanical and industrial design. It is worth stating that I have a deep kinship of sorts with this way of creative problem-solving as a Black American that grew up in an urban centre, largely cut-off from many of the city’s resources. Although I will not digress much further on what I see as distinct characteristics shared between the jugaad practices of Southeast Asia and Black American communities in this chapter, suffice to say that we too used Ghetto solutions. Although the concept of ghetto solutions is less formally defined and is commonly used as a marker of derision to both reduce distinct cultural features of

VOLUMETRIC BL ACK

73

Black communities and as a dog whistle for a kind supremacy that even well-meaning liberals find acceptable, there are identifiable traits shared between both conceptual systems. This Black American function of ghetto operated to address neighbourhood repair, innovation and invention of play when the state would not provide it. As I begin to describe new ways of seeing Black bodies, it is appropriate to briefly address and draw a clear distinction between being completely seen (Blackness is humanity) and being surveilled (apparatus of state violence) as a Black body in America. Both the history and record of American surveillance of Black people/ Blackness has been thoroughly researched by many scholars underscoring the overreach of state actors violating their own laws to observe, to harass, and at times resulting in the deadly consequences of killing Black people/ Blackness (Browne, 2015: 92). What I propose as speculative design and interventionist art hacktivist possibilities regarding Black body imagemaking should maintain a position of non-compliance, with no attempts at co-opting partnerships or exchanges in resources with any local, state or national government actors. The realities of the state surveillance of Black bodies must always be considered when reimagining new systems for viewing Blackness. It is no secret, as I have stated previously on the history of hyper-surveilling Blackness – that in our present climate Black bodies in America are only partially seen and seldom humanized within the framing of the gaze.

Optimizing cultural optics Volumetric Black is the filmic-architectural space, the speculative imagining, where Black bodies on screen and Black spectatorship extend themselves beyond the borders and boundaries of the image-making frames and the audience environment. It also references the activity and speculative design tools behind the collapse where the praxis of interventionist art hacktivism, the jugaad, occurs. The Volumetric Black series of objects focuses on two main sources of technologies for image-making of Black bodies. The first in the series of speculative design systems is a re-engineering kit for commercially available television equipment. The second is a fully fabricated digital light filtration lens for DSLR (digital single-lens reflex) cameras. This speculative strategy for ushering in a new visuality for Black bodies employs a triadic approach that merges both cultural and social-spatial dynamics specific to Black American culture jamming – with the informal tactics of hacker augmentations of existing image-making technologies. The filmic-architectural space requires as much from the disruptions of cultural sensibilities as it does from the contents framed before the audience. It calls for a kind of crafting of blues aesthetic (Carbine, 1990: 23) reminiscent of the period from the talkie era when white Hollywood productions,

74

MATERIALIZING DIGITAL FUTURES

screened in Black communities did not speak to the lives of the audience, and so the pre-screening of Vaudeville or Chitlin’ Circuit entertainment culture jammed the films’ content. Or the use of jazz bands to provide the accompaniment for silent films over the traditional orchestral arrangements of the period. This intersection of culturally specific intermediaries, between the framed white images and Black spectator, provides a processing of both entertainment and spatial infrastructures for audience immersion. This kind of spectral Black immersion, of both architectural and cultural spaces, was a useful mode of disruption, or culture jam, that could reverse the cold alienation effectively communicated regarding white sensibilities through American cinema. It is the mashing up of culturally specific sites and sounds, recontextualizing the tools of production in the image-making of Blackness that visually mends together audience imaginary and the darkened eaves of the infrastructure(s) that bear markings of the spatial collapse constitutes a discernible space that presents sociocultural volume. Infrastructural objects add to the disorienting complexity of blurred elements, contributing the shadows that push functional structures deeper into recesses. The interior surface, syncopated in its breathing, is set to flickering projections. There are several examples of contemporary film productions released in America within the last ten years that have achieved the visual collapse (absent is the cultural collapse) between framed content and infrastructures using the existing trade tools within production of image-making. Perhaps the most recent example of this othering in visuality is Pedro Costa’s newest exploration in slow cinema, Vitalina Varela (Costa, 2020). In the productions’ wanderings through many night scenes and window-lit interior spaces, many of the images are compositional voids of Black that only partially reveal the Black bodies (Costa’s collaborators) that live within the narrative of the film. The richness of the filmed Black spaces and Black bodies, seemingly devoid of light, though clearly produced by it, tugs at the infrastructure around the projection surface area to the point of barely visible spatial contours. Vitalina Varela begins with a spectral procession of men filing out at night from a cemetery. The image is nearly drained of color, so dark that you might wonder for a moment whether that really is a metal cane swinging out of the blackness, or just a phantom movement swimming in the dim theater. (Noah, 2020) Another recent film that constructs a similar achievement is Andrew Dosunmu’s Mother of George (Dosunmu, 2013). Once again, the film’s contents allow for a kind of Blackness to exist in an expressly textured assemblage where lowlights, interior spaces and organic forms converge in an intertwined collage of Black bodies, architecture and muted visual space to construct a hybrid composite of Black body image-making. This

VOLUMETRIC BL ACK

75

purplish-blue interior of Blackness periodically gives way to reveal the enveloped facial expressions and darkened limbs to reappear with subtlety. This filmic production, though still operating within allowable guardrails of industries, establishes a transgressive stance that orients it towards a visual honesty. Although the previous films’ descriptions are examples of a filmicarchitectural space with compositions of visual Blackness, they do not, however, conclusively extend themselves to the cultural collapse between the spectator and the contents of the frame. This effect is partially due to cultural specificities within Black America and the relational narrative content of the productions. A brief example of a film with significant resonance within cultural collapse yet falters in its aesthetic abilities to render the visual Blackness of the filmic-architectural space is the 1967 Norman Jewison film, In the Heat of the Night, featuring the actor, Sydney Portier. The significant context for the film’s cultural collapse centres around a generational Black actor, as the central character in a major Hollywood production whose narrative setting was a depiction of the deep South – all filmed amid the American Civil Rights movements. Finally, what I propose next is what I have come to identify as Volumetric Black. A system of reconfigurable interventions employing the use of digital tools and computational hackings (or jugaad) aimed at remaking the image of Blackness across screen-based technologies. The speculative design objects for achieving Volumetric Black requires two methods for implementation of jugaad interventions in the production of Black body image-making. As previously discussed in this chapter, we use material(s), chemicals and technologies today for producing images that are partially arbitrary with partial negligence advanced by economics; however, both approaches are devoid of Blackness as a consideration. Research conducted by the University of Leiden and the Getty Conservation Institute on historical painting techniques and materials used by painters as early as the fifteenth century, utilized in frescoes and on altarpieces, indicate, through a series of x-rays and cross sections, the development and use of underpainting processes to construct blocking compositions, but also used to address tonal shifts of light creating volume (Barrett and Stulik, 1995). These master works’ underpaintings, or dead-colouring techniques, commonly consisted of selected variations of greys, blue-greys, vermillion, greens, pinks, yellows, ochres and ultramarines (Barrett and Stulik, 1995: 7). This was all before the actual construction of the image. This technique for image-making that implemented the use of a colour treatment to the background surface prior to the rendering’s application suggests that the modern shift to use white material substrates, in the case of photography and white LED (light-emitting diode) matrices for the backlighting of displays, should also consider a similar taxonomy and undergo further scrutiny of its technological methodologies. This is the critical space where the Volumetric Black jugaad takes place. The Volumetric Black display is a

76

MATERIALIZING DIGITAL FUTURES

repurposing of the existing commercially available televisions with default white LED panels for illuminating all images produced on the TFT (thinfilm transistor) liquid crystal display. Through a combination of imageprocessing computations, microcontroller programming and finally the replacement from white to RGB light-emitting diodes, images of Black bodies produce an illumination that appear less visually compromised. The Volumetric Black lens filter shares a similar computational application to the jugaad display fabrications. The lens fabrication is a four-part, 3D printed module that incorporates an array of surface mounted RGB LEDs and a Pico micro-controller in its assembly. The lens mechanism employs the same image-processing data programmed into the controls, allowing the user to toggle through a selection of alternate (non-white) light settings for the image-making of Black bodies. The combined output of the speculative designs used to document and visualize Black bodies with the Volumetric Black jugaad does not serve, within a set of parameters, as a new technical standard, rather the objects situate themselves well within a frame of culture jamming criticality – with an experiential praxis response to the history of image-making and the associated technologies. The filmic-architectural space and cultural collapse – between audience, (image) frame and infrastructure – is the aesthetic reconfiguration of visual Blackness that is Volumetric Black (Maurice, 2013). The system of enveloping sensorial optics and spatiality of Black bodies pushing boundaries of hyper-visual stimuli, creating architectural displacement and specificity of cultural discomfort for non-participants is the intended output of Volumetric Black. This research and praxis in speculative design utilizes a jugaad application in response to the history of American spatial sensibilities in relation to, and negation of, Black bodies during co-encounters of shared public-domestic (filmic) space. This includes the hybrid spaces of the public-domestic co-encounters made possible through the proliferation and mobilization of screens. This research works to operate multiple registers of visual Blackness, simultaneously oscillating between deliberate opacity and techno-visibility (Glissant, 1997). Outlined ecologies for cultivating imagemaking strategies for Black bodies operate at every stage of production from the replacement of production tools at an industry level, to performative spectatorship, and end consumer tactics as expressions for modulating aesthetics of Blackness. Ultimately the aim of this research and praxis is to provide optional possibilities and varied possessions of Blackness capable of toggling between modes of visibility and opacity at the affordance of Black bodies. The registers of Blackness that we operate within are seldom on our own terms or with our own technological agency. It is a kind of sociocultural discomfort permeating through shared public spaces, brought about solely through the existence of spatially heterogeneous functions of Blackness operating wholly. The image-making frame is only one of the many mediated systems of representation that require both critical re-examination

VOLUMETRIC BL ACK

77

and praxis interventionist approaches. This permanent technicity demands reconfigurations for racially heterogeneous aesthetics of Blackness that exist volumetrically. The reimagining of possibilities for viewing Black bodies – literally in a new light.

Notes 1 With a distribution of potential limited to southern segregated theatres and northern Black neighbourhood theatres, even low-budget ‘Race movies’ were more expensive to book than those offered by national chains, and theatres passed this cost on to patrons (Carbine, 1990: 20). 2 ‘D. Clustering Process: A cluster of defect reports wherein each individual report is ‘similar’ (with respect to our model) to all others is amenable to aggregate triage. Having defined similarity between defects reports, we now require a lightweight and accurate method for clustering related defects … We thus adopt a well-known algorithm for measuring interconnectedness of components for the purpose of clustering’ (Fry and Weimer, 2013: 285). 3 Photographers of African American and Asian subjects, who had developed methods for dealing with these ‘problems’ independently, began to share their knowledge with each other and with the public sometime around the late 1950s, at around the same time Kodak was experiencing some criticism of its photo emulsions regarding this very issue (Roth, 2009: 117). 4 ‘When I meet a German or a Russian speaking bad French I try to indicate through gestures the information he is asking for, but in doing so I am careful not to forget that he has a language of his own, a country, and that perhaps he is a lawyer or an engineer back home. Whatever the case, he is a foreigner with different standards. There is nothing comparable when it comes to the black man. He has no culture, no civilization, and no “long historical past” … Whether he likes it or not, the black man has to wear the livery the white man has fabricated for him’ (Fanon, 1967: 17). 5 Among those groups protesting the picture were the Young Communist League and the National Negro Congress. A clipping from a New York City newspaper reads, ‘the picture creates a false splitting of Negroes into light and dark groups. It slandered Negroes, holding them up to ridicule’ (Green and Neal, 1998: 67). 6 In an earlier passage, Barbot writes that although he was ‘naturally compassionate’, he sometimes caused ‘the teeth of those wretches to be broken, because they would not open their mouths’ in their refusal to eat. The false pretence of naming resistance to force-feeding as unruliness is an attempt to mask the violence of the slave trader by displacing the violence of slavery onto the African (Browne, 2015: 96). 7 ‘Every colonized people – in other words, every people in whose soul an inferiority complex has been created by the death and burial of its local cultural originality – finds itself face to face with the language of the civilizing nation’ (Fanon, 1967: 18).

78

MATERIALIZING DIGITAL FUTURES

8 And network radio had certainly never been a force for racial amelioration and uplift. Network radio’s first phenomenal hit show beginning in the 1920s had been Amos ’n’ Andy, voiced by two white men engaged in minstrel ventriloquism (Bodroghkozy, 2012: 3). 9 Aimed largely at Black audiences, these works freed Black actors from the burden of tailoring their performances to white preconceptions (Flory, 2005: 334).

References Barrett, S. and D. C. Stulik (1995), ‘An Integrated Approach for the Study of Painting Techniques’, in C. Lightweaver and J. Hill (eds), Historical Painting Techniques, Materials, and Studio, 6–11, Lawrence: Allen Press. Baudrillard, J. (1994), Simulacra and Simulation, trans. S. F. Glaser, Ann Arbor: University of Michigan Press. Benbow, M. E. (2010), ‘Birth of a Quotation: Woodrow Wilson and “Like Writing History with Lighting”’, Journal of the Gilded Age and Progressive Era, 9 (4): 509–33. Benjamin, W. (1968), Illumination: Essays and Reflections, trans. H. Zohn, New York: Schocken Books. Bodroghkozy, A. (2012), Equal Time: Television and the Civil Rights Movement, Urbana: University of Illinois Press. Browne, S. (2015), Dark Matters: On Surveillance of Blackness, Durham, NC: Duke University Press. Carbine, M. (1990), ‘ “The Finest Outside the Loop”: Motion Picture Exhibition in Chicago’s Black Metropolis, 1905–1928’, Camera Obscura, 8 (23): 9–42. Costa, P. (2020), Film, Vitalina Varela, Grasshopper Film, USA. Cripps, T. (1977), Slow Fade to Black, London: Oxford University Press. Dosunmu, A. (2013), Mother of George, Oscilloscope Laboratories, USA. Fanon, F. (1967), Black Skin, White Masks, trans. C. L. Markmann, New York: Grove Press. Flory, D. (2005), ‘Race, Rationality, and Melodrama: Aesthetic Response and the Case of Oscar Micheaux’, Journal of Aesthetics and Art Criticism, 63 (4): 327–38. Available online: https://www.jstor.org/stable/3700509?seq=1&cid=pdfreference#references_tab_contents (accessed 29 April 2020). Flory, D. (2016), ‘Racialized Disgust and Embodied Cognition in Film’, Projections, 10 (2). Fry, Z. and W. Weimer (2013), ‘Clustering Static Analysis Defect Reports to Reduce Maintenance Costs’, Working Conference on Reverse Engineering (WCRE), Koblenz: IEEE. Glissant, E. (1997), Poetics of Relation: On Opacity, trans. B. Wing, Ann Arbor: University of Michigan Press. Goff, P. A., M. C. Jackson, B. A. L. Di Leone, C. M. Culotta and N. A. DiTomasso (2014), ‘Essence of Innocence: Consequences of Dehumanizing Black Children’, Journal of Personality and Social Psychology, 106 (4): 226–45. Available online: http://citeseerx.ist.psu.edu/ (accessed 6 January 2020).

VOLUMETRIC BL ACK

79

Green, J. R. and H. Neal (1998), ‘Oscar Micheaux and Racial Slur: A Response to “The Rediscovery of Oscar Micheaux” ’, Journal of Film and Video, 40 (4). Available online: https://www.jstor.org/stable/20687841 (accessed 11 October 2019). Griffith, D. W. (1915), Film, The Birth of a Nation, Epoch Producing Co., USA. Grupenhoff, R. (1988), ‘The Rediscovery of Oscar Micheaux, Black Film Pioneer’, Journal of Film and Video, 40 (1). Available online: https://www.jstor.org/ stable/20687803 (accessed 17 May 2019). Jaikumar, P. (2019), Where Histories Reside: India as Filmed Space, Durham, NC: Duke University Press. Maurice, A. (2013), Cinema and Its Shadow: Race and Technology in Early Cinema, Minneapolis: University of Minnesota Press. McQueen, S. (2013), Film, 12 Years A Slave, Fox Searchlight Pictures, USA. Noah, W. (2020), ‘Pedro Costa, Filming the Saga of Lisbon’s Cabo Verdeans’, The New York Times Review of Books. Available online: https://www.nybooks. com/daily/2020/03/03/pedro-costa-filming-the-saga-of-lisbons-cabo-verdeans/ (accessed 8 November 2020). Rai, A. (2019), Jugaad Time: Ecologies of Everyday Hacking in India, Durham, NC: Duke University Press. Regester, C. (1995a), ‘The Misreading and Rereading of African American Filmmaker Oscar Micheaux: A Critical Review of Micheaux Scholarship’, Bloomington: Indiana University Press. Available online: https://www.jstor.org/ stable/3815383 (accessed 9 August 2019). Regester, C. (1995b), ‘Oscar Micheaux on the Cutting Edge: Films Rejected by the New York State Motion Picture Commission’, Studies in Popular Culture, 17 (2). Available online: https://www.jstor.org/stable/23413703 (accessed 28 August 2019). Rose, J. (2012), ‘911 Tapes Raise Questions in Fla. Teen’s Shooting’, NPR News, 12 March. Available online: https://www.npr.org/2012/03/19/148902744/911tapes-raise-questions-in-fla-teens-shooting-death (accessed 20 March 2020). Roth, L. (2009), ‘Looking at Shirley, the Ultimate Norm: Colour Balance, Image Technologies, and Cognitive Equity’, Canadian Journal of Communication, 34 (1). Sharpe, C. (2016), In the Wake: On Blackness and Being, Durham, NC: Duke University Press. Siomopoulos, A. (2006), ‘The Birth of a Black Cinema: Race, Reception and Oscar Micheaux’s Within Our Gates’, Moving Image, 6 (2): 111–18. Available online: https://muse.jhu.edu/article/209581 (accessed 9 April 2020). Sontag, S. (1977), On Photography, New York: Picador.

80

SECTION TWO

Meaning-making in the data-driven era Introduction In this section, digital media is explored through the overarching notion of the Quantified Self formed through multifaceted data-gathering processes with implications for data security, digital privacy, the value of personal data, individualization of movement, personalization of information/data and the visualizing of data. This section serves as a vital critical study that reflects and comments on current issues and sociocultural effects through key digital media for communication and the individual registers of the digitally assembled ‘self’. The chapters in this section present the current dilemmas and issues that are central to the critical understanding of quantified media and digital analytics. At present, key players in telecommunications, entertainment and internet service companies as well as public policymakers are all responding to the evolving capabilities that are afforded by new, often disruptive, digital

82

MATERIALIZING DIGITAL FUTURES

technologies, for which appropriate usage norms have not yet definitively emerged. Big data is related to the contemporary rise of multimedia devices and applications such as social media use and cloud computing in everyday life. Cloud computing for its part provides access websites and channels for the storage of extremely large amounts of personal and business-related data. In the paradigm of the ‘internet of things’ (IoT), sensors all over the world are collecting and transmitting data that is stored and processed in the cloud. Here cloud computing, which is essentially a shared and rented data centre, can be seen to hold an advantage, because instead of single businesses each running processing and storage for themselves, they have a company such as Microsoft do it for them. Users, consumers, companies and governments are all producing online content at unprecedented rates. Everyday consumer engagement online and in social media is normalized. Companies for their parts have embraced the internet as a strategic arena to promote their brand, products and jobs, and have increased their online investment and content creation accordingly. With the rise of new, richer content openly available on the internet, ever more sophisticated social and corporate insights can be created. Just as electricity transformed the way industries functioned in the past century, artificial intelligence – the science of programming cognitive abilities into machines – now has the power to substantially change society where there is growing focus on insights from external data sources for application in commercial, creative and research settings. If data can be likened to the new oil, then Meltwater is Saudi Aramco. Meltwater is a software as a service (SaaS) company that started as the world’s first online media monitoring company. The now global company was founded in 2001 in Oslo, Norway, by Jørn Lyseggen. It has headquarters in San Francisco, California, and has now established a network of offices stretching across Europe, North America, Asia/Pacific, Australia and Africa. Shaz Mohapatra is the Area Director (CS, CX, Insights) at Meltwater: Shaz Mohapatra: Meltwater basically looks into all the news that is published, all around the world 24-7 be it news from print, magazine, newspapers, online media, radio, TV and even social media. Initially what we would do is we would collect it and put it into an easily searchable ‘data lake’ and companies paid for access to the content to find out what has been written about them. Companies would take out a subscription and they would be able to find out what’s being said about them publicly in the media. How it evolved, though, is the most interesting part because it has led to the formation of an industry called ‘media intelligence’ as a result of the amount of news that is going in [to the data lake] every day, and then you compound it with the amount of social media content going in and

MEANING-MAKING IN THE DATA-DRIVEN ERA

83

it is astronomical. For example, we now ingest about 3 million news articles every single day. These are unique news articles; and then we also ingest 300 million posts just from Twitter – so it’s a huge amount of information being compiled every day, but nobody has the time to read it all. If you look at China, it has some 21 different social media channels alone that exist that have different users. This is by way of comparison to the liberal West that has the usual Facebook, Twitter, Instagram, YouTube, to some extent LinkedIn and Reddit, then some blogs, and used somewhat are dark social like Telegram, WhatsApp or Clubhouse [an invitation only audio-based social media app]. In China, however, there are many more channels, and they are used prevalently as business communication tools – as well as for social purposes of course. What that has allowed companies such as Meltwater to do is to give insight; so rather than giving your communications team all the collected news articles or say 50 news articles for teams to read or 500 social media posts for analysis Meltwater can collate and summarize the key information thematically for a subscription-holder and summarize the main issues that are being spoken about. Especially, what is ‘catching fire’ [positive comments and sentiments] or for whether there is negative commentary that needs to be attended to. For example, a government department might want to know if there are protests coming because voters have an issue to bring to public attention. In this way, they know instantly what to prioritize and what to escalate, rather than having to read all that media content and then deciding. The data lake is used to track a topic, for example, say it was around the PR efficiency of Donald Trump during his time in office. We can now go back and scrape the Meltwater platform for everything that Donald Trump said, what had the most impact on social media and you can get a really detailed, dataheavy insight into key actions and events. For example, I spoke with a client who makes gimbals [the pivoting support that allows you to mount along a single steady axis an object such as a camera and smoothly rotate it for image capture]. These are great products that are used creatively for making films to videos for TikTok and the like. This manufacturer sells the gimbals on Alibaba [an e-commerce company] with a turnover and revenue in the many millions of dollars. While they know there are many people buying their products globally, they wanted to know more about the public perception of their brand name in the market. Some countries block external communication, so it is not possible for the manufacturer to know what is happening in particular markets concerning their product. Consequently, they are now using social media rather than traditional media as a way of gaining insight into the attitudes and behaviours of their customer-base. This is because once the product leaves the shelves, they literally have no

84

MATERIALIZING DIGITAL FUTURES

idea what’s happening in regard to customer satisfaction and so forth. This capacity for data analysis and to study large amounts of crucial information has become their new customer relationship management strategy for assessing interactions with their buyers. On a personal level, personal data collection can have a real impact on one’s life. For example, I pay a monthly subscription for this APP called Life Cycle which gives me an insight day-to-day, how much time I spend on traveling; how much time I spend at home with my kids; driving to and from particular locations; calory intake and sleep patterns. I have restless sleep and I received good pointers from the data about what I can do to my habits for when I should eat, when I should drink water and so on. That has made a very significant impact on my quality of life where I am now sleeping a lot better, I regulate my travel, I now go to the gym and it has had a general cascading effect so that I am waking up feeling a lot better. I know Google is tracking my movements in real time, as is Apple, and you know I’m happy for them to track me as long as I can tell what’s happening and there are positive results for me. Taking up the threads of data gathering, Toija Cinque writes in Chapter 5 that the advent of ‘big data’ (and small data) technologies, artificial intelligence (AI) and social media underpin an obscure datafied existence of life in which we have inexorably altered the boundaries between private and public communication. Cinque uses a digital (auto)ethnographic method applied to a case study analysis of Microsoft productivity apps Cortana and MyAnalytics. She notes that those who cognitively labour are sent regular personal workday analytics by email that provide users with personal ‘deep dives’ into their e-work patterns in connection to their Focus, Wellbeing, Networks and Collaboration. The actuality of daily work patterns now comprizing such ‘people analytics’ is, however, frequently different by way of comparison to the recorded data. The author asserts that overarchingly one’s professional and social profiles, personalized pages and customized accounts are not merely reflections of the self, but to all intents, constructions and purposes are constitutive of the self they apparently represent. The spectre of self emerges as a result of digital ‘identity construction’ positioned by moments/actions in/across time revealing aspects of users’ significant lived experiences beyond single pieces of personal data. Cinque contends that the outcome is a version of self that is rendered pixelated and ghostlike. Cinque concludes by drawing attention to the existing standards that provide coverage for many aspects of AI infused communication systems to consider some socially just, ethical options. In Chapter 6, Kim Vincs underscores the tensions involved in bridging virtual and physical spaces that they proffer are thrown into sharp relief as virtual spaces have become increasingly entwined with everyday work and life. Vincs adroitly argues that we now look to conduct everyday business and

MEANING-MAKING IN THE DATA-DRIVEN ERA

85

social functions virtually, not through choices made for specific situations or contexts but utilizing technologies that were never designed for these contexts. This chapter addresses one of the key affordances of virtual reality – the sense of being physically present within a virtually created ‘world’ for its potential to transform simulation-based interventions in fields as broad as medical rehabilitation, workplace training and managing chronic pain. Vincs emphasizes, however, that VR is ‘persuasive’ at the level of the human sensorium, and this opens the possibility of VR as a manipulative as well as an emancipatory medium. Vincs takes up this uneasy tension between VR as immersive and/or coercive by understanding embodied action and cognition as reflexive rather than automatic, and hence able to deal productively and creatively with the illusion of ‘being there’. Vincs employs thinking drawn from dance technology which affords a world of physics (distances, depths, volumes, masses and forces), imbued with cultural, social and artistic meaning caught through actions. Chapter 7 takes a different approach to sociocultural meaning-making via big and small data by way of study into feminist activist memes. These are raised up as being complex sociocultural performatives that can speak to a range of structural inequalities. In the context of social media activism, Shana MacDonald and Brianna I. Wiens take feminist witch memes as a means of protest as their tool of discovery. The authors use their chapter to expand understandings of memes as having the capacity to iteratively circulate and (re)organize forms of discursive resistance and refusal into dynamic flows that consequently evoke an urgency through their resonances among audiences. MacDonald and Wiens consider memes as contributing to the formation of an explicitly feminist digital self. They do so by offering a critical analysis of select feminist digital witch memes, using a small data approach to meme collection and analysis that responds to a need to engage with methods that do not rely solely on the randomization, scaling and coding of algorithmic data, which can overlook power imbalances and communicative acts of solidarity that are integral to digital activism. The authors argue that digital meme activism and their resistance afford embodied experience and personal memory as distinctly feminist tools of resistance for Black, Indigenous and racialized communities, as well as queer, non-binary and women-identified social media participants. The authors conclude that it is through the creative (re)employment of tactics that name embodied experiences and their materialities that digital feminist activists publicize and remix personal memories and experiences for social and political ends. Focus on meaning and the image is considered in Chapter 8 through recent advances in creative machine learning which have led to an increased proficiency in artworks being digitally produced by AI. Andrew McIntyre argues that simply reducing AI to being a complex tool used to fulfil a predetermined intention misrepresents the reality of the image-making process and further undermines the AI’s role within this process. For

86

MATERIALIZING DIGITAL FUTURES

McIntyre a new conception of authorship theory is necessary to address these instances of AI art that are becoming increasingly prevalent in many cultures as opposed to upholding anthropocentric views. In Chapter 8, the case of AI art installation Memories of Passersby I (Mario Klingemann, 2018) is presented to demonstrate how such works might undermine the humanist assumptions that underpin authorship theory and further discussions around aesthetic qualities of the work as influenced by its posthuman production. To be found in McIntyre’s detailed unpacking of the vectors around Memories of Passersby I (along with other examples of AI art) is the claim for AI not as sole author. Rather he makes the suggestion for a nuanced relationship between human intent and the program.

5 Quantified me: Curatorial lives and the pixelated spectre of self Toija Cinque

Introduction: Algorithmic cultures The global technological climate is evolving and has changed irrevocably with the growth of personal and industrial data collection, data sharing as well as increased data tracing practices using personal mobile devices. For example, see the new Contact Tracing apps that are widely used/mandated since the global Covid-19 pandemic to track and trace the movement of individuals with the intention of limiting the viral transmission, and wearable computers such as Fitbits and smart watches. Introduced incrementally have been applications and instant messaging services such as Tik Tok (2016) and Signal (2021) that are linked to mobiles since the advent of Facebook (2004), YouTube (2005), Twitter (2006), Instagram (2010). Algorithms, the overarching artificial intelligence (AI) and big data are today impacting sociocultural and technological relations internationally and our everyday experiences. While algorithms have themselves been in use for centuries, and basic artificial intelligence has been in cars and electrical appliance since the 1950s, advances in mechanical and computing powers, transmission capacity and data storage have increased exponentially thus advancing their capabilities (Makridakis, 2017). The writing of code, once the modern domain of Information Technology and Computer Science, has now been taken up in the Digital Humanities by social scientists undertaking certain social cartographies, media researchers, journalists and interested others for

88

MATERIALIZING DIGITAL FUTURES

finding new ways to scrape (collect) data to make sense of current trends and issues. Because data is obscure and enigmatic, we are frequently caught in its heuristic gaze. Data might be quickly understood as some ‘hidden’, subtle function and means of supporting active agency and promoting positive benefits (of facilitating data/information processing, social connection, increased health and fitness as exemplars). A balance must be struck, however, between the opportunities and positive benefits with concerns for individuals with reference to data/digital tools being used for negative purposes such as for cybercrime, compromizing e-safety, supporting eating disorders or self-harm to digital privacy more broadly. Certain digital literacies are necessary, therefore, to navigate the many services and applications now increasingly interconnected across multiple technological devices in home and workspaces. This is in addition to developing an informed awareness of the online self that is created in/for/by our human actions and systems behaviour, but which also have consequent effects on the natural environment through mining for machine-required minerals, high electricity requirements, e-waste and the like. Considering that technological devices are embedded in (and from) childhood development and can be found in social, educational and familial settings, a greater understanding of the interaction between networked, connected technologies is needed. This understanding should encompass telecommunications infrastructures (including information technology and software) and social media (incorporating text and video from information media across networked news/entertainment websites and online communities) with the rise of wearable and traceable options. Such insight would allow for early informed and appropriate decision-making about their collective affordances. This chapter unpacks and uses the expression spectre of self, a neologism proposed for this study to re-conceptualize the nature of online engagement for where we are now. The framing goes beneath the important changes effecting and affecting our cultural, social and embodied emotional lives and the pressures and opportunities occasioned by reconfiguring social connection. In doing so, this chapter draws on the notion of a ‘quantified self’, a term first used by Gary Wolf and Kevin Kelly in 2007 for their lifelogging on thequantifiedself.com (see the later work by Lupton 2013; 2014; 2015; 2016) that encompasses the collection of personal data such as health information, fertility, sport performance, heart rate over time and the like and the blog discussions and empirical conclusions that ensue relating to this data. The spectre of self here describes a mythical self that is fashioned as part of our ‘curatorial lives’ going beyond single data events. It is an existence digitally assembled and formed, built for/by a non-unitary ‘us’ in and around our everyday actions. Further, it is through the enabling conjunctions between ubiquitous, networked computing technologies and communications systems that the spectre of self is truly experienced.

QUANTIFIED ME

89

This chapter’s primary objective is to intervene and delve into descriptive notions of ‘datafication’ towards unearthing the implications and value of materializing networked technologies. This will be navigated through critical consideration of how the former techniques of connection to community are reconfigured through this changing landscape of social media (in)visibility, big data agglomerations and personal engagement with an empirical spectre of self. In the following sections, I undertake a case study pertaining to the use and ownership of data alongside public understandings of, and social/relational implications for, ubiquitous data collection. Many discussions centred upon our algorithmic culture/s are around changed behaviour and also improving outcomes for people, but this emerging strand of Human–Computer Interaction (HCI) studies offers a way to re-evaluate deeply evocative futures in which humans and intelligent systems increasingly engage in symbiotically connected experiences via continuous flows of data and information exchanges.

The everydayness of big data and the internet of things The term ‘big data’ is used frequently to refer to the ‘free’ or at least widely circulating flow of information (paywalls and firewalls place some restrictions on data) and to describe and encapsulate the trend towards increasing amounts of information being created, stored and captured by individual users, as well as by corporate and not-for-profit enterprises. More specifically, Chen et al. argue that big data refers to the datasets that could not be perceived, acquired, managed, and processed by traditional Information Technology (IT) and software/hardware tools within a tolerable time. It can be characterized by four Vs, i.e. Volume (great volume), Variety (various modalities), Velocity (rapid generation), and Value (huge value but very low density). (2014: v) Adding to this description is that big data is related to the contemporary rise of multimedia devices and applications such as social media use and cloud computing in everyday life. The key sources for big data are social media platforms (including short-form blogging, link sharing, expert blog comments, user forums and ‘likes’, purchase histories), machine-generated content (such as device log files, search histories), software and applications (geolocation data signals taken from Wi-Fi routers and mobile phone towers), and ‘data languishing in legacy systems’ (such as medical records and customer correspondence) (Runciman, 2014: 1). Moreover, big data is linked to the functions from the late 1990s of the internet of things (IoT). A useful understanding of the IoT can be drawn from the Institute of Electrical and Electronics Engineers (IEEE):

90

MATERIALIZING DIGITAL FUTURES

The fundamental concept is to emphasize the ubiquitous computing among global networked physical objects, or things including RFID [radio-frequency identification] tags, sensors, actuators, and mobile phones. It covers a wide scope of technologies including wireless/wired sensing, networking, computing and control, which together build feasible complex cyber physical systems (CPS) to support diverse applications, including smart grid, healthcare, intelligent transportation, and logistics. (http://iot-nc2013.weebly.com; see also Liu and Gao, 2014) Expanding on these details, there is a fundamental dynamic at play around the difference between cyber physical systems (CPS), which are largely based on ‘closed’ systems (controllable and partly predictable), and IoT, which is characterized by an ‘open’ system, whereby it is more difficult to control or to predict system behaviour (Jeschke, 2014). What we see in contemporary CPS/IoT ‘embedded systems’ is an increasingly expanding and networked infrastructure connecting to computer-augmented physical systems worldwide that can be understood as ‘black boxes’ (Pazzi and Pellicciari, 2017: 989). By definition, data is in closed systems where what happens between input and output is opaque. That is, users might be aware of what a device/app/algorithm is meant to do but not specifically how it works; nor perhaps are they aware of the subsequent social/environmental implications. However, data does move in embedded systems across liminal spaces of indistinguishability whereby some hidden/closed data is in our most freely accessible spaces from Facebook to refrigerators, home ‘assistants’ such as the Google Nest, Amazon Alexa, Amazon’s Echo Dot to organizations’ doorbells, our cars and power meters. As a result, the spectre of self is being assembled by prosaic algorithmic traces from interconnected ‘things’ used every day. Some devices such as the Samsung Smart TV (Grimm, 2015) and apps such as Facebook (Booth, 2014) have included automatic sound/voice detection and ‘secretly’ listen in on users and/or save the data for opaque purposes or to specifically determine mood states (Kotenko, 2013). Such instances come with concern that the data can ‘implicitly contain a rich array of personal information, including cues to a speaker’s biometric identity, personality, physical traits, geographical origin, emotions, level of intoxication and sleepiness, age, gender, and health condition’ (Kröger et al., 2020: 242). Against the background of recent assessments about the obscure nature of data analytic processes (Andrejevic and Gates, 2014) and autarchic management of the conditions for the commodification of data (Crain, 2016), there is a well-rehearsed view of radical transparency that describes the idea of everyone having the capacity for knowing everything. Without reading the terms and conditions of service as well as by choice, many ‘over share’ the most personal of details and images. That a number give up care for protecting personal digital data is in part due to suffering ‘privacy fatigue’ and feeling powerless to control the outcomes for where

QUANTIFIED ME

91

data ends up and the purposes to which it is ultimately put (Choi et al., 2018). Further, the impacts are not always imminent and the services the data is being traded for are desirable.

Spectre of self: The shaping of data ghosts With reference to Jacques Derrida’s paronomastic compound term, ‘hauntology’, a pixelated, digital spectre of self is a Derridean ghost of one’s real identity, within an ontology of the being of non-being and the nonbeing of being, of something/someone ‘not there’. For ‘there is no Dasein of the specter, but there is no Dasein without the uncanniness, without the strange familiarity (Unheimlichkeit) of some specter’ (Derrida, 1994: 165). The everyday is punctuated by returns such that ‘the specter is thus both past and future; it is from the past but waiting to come back. The metaphor of a specter haunting the present establishes the idea/image of the existence of something ghostly which stands over and outside the present, something which does not belong to time, and is waiting to come’ (Hughes, 2012: 15). The digital, prosopopoeial spectre of self, acts here to position moments/actions in/across time revealing aspects of users’ significant lived experiences. The transmuted, pixilated spectre of self encourages thinking beyond a single piece of personal data (a post, a photo, a comment, like or share, geolocation data) to build a new perception and presentation of self (Goffman, 1959) that is constructed in parallel to that formed by traditional in-real-life agents of socialization such as friends, peers, religion, culture, media, legal and economic systems, and so forth (Giddens, 1993). Our profiles, personalized pages and customized accounts do not merely reflect the self, but to all intents, constructions, and purposes ‘are constitutive of the self they apparently represent’ (Kant, 2020: 57). Digital culture and communication are inevitably changing as technologies, media infrastructures, media practices and social environments become increasingly ‘datafied’ (Schäfer and van Es, 2017). In the nearness of now, the technical ability to leverage the power of machine (deep) learning and AI to recreate (fake) existing images, photographs, videos and/or sound files as replicated or synthetic versions of an original (or Deep Fakes) has the immense capacity to deceive. Examples include Mark Zuckerberg ‘admitting’ that Facebook’s true goal is to manipulate and exploit its users, to elected officials appearing drunk or making statements that never happened (see https://www.youtube.com/watch?v=NbedWhzx1rs). This has seen a rise in fake and misleading news and misrepresentation online, which impacts public opinion. Such troubling practices underscore significant implications in this neocommunicative process. Not only do the various media forms have the potential to bring heterogenous actors and events into alliance (Bratton, 2016), but they are technically ‘open’ to allow manipulation by end users. Once particular

92

MATERIALIZING DIGITAL FUTURES

connections and patterns of interaction are established, algorithms that personalize digital content can lead to filter bubbles (Marchi, 2012), which limits the platform’s potential as a democratic public sphere (Habermas, 1989), leading to ‘us and them’ narratives that potentially erode social cohesion. Thus, concerns about the role that digital and social media play in politics and society are increasing (Omidyar, 2018; Tucker et al., 2017). The ‘fears’ surrounding technology raised in society are often unpacked in popular media such that the spectre of self can be found and is depicted as being on the precipice of a societal fall. Notable examples are Zoe Greystone and the Cylons in Caprica (2011), Ash (amongst others) in Black Mirror (2011) and the hosts of Westworld (2016). A re-occurring theme within these screen examples is that of faithful reproduction or in using the key phrase of Westworld, ‘fidelity’ (2016). In Caprica (2011), Zoe discovers the algorithm to digitally recreate a person by searching and scraping any available data, including but not limited to social media and health records, and compiling this with biometric data collected from the original person. Social media is presented as a recurrent source of data in the examples here. The concerns raised in the narratives noted here are to be read as key issues in the social construction of contemporary ‘information societies’ (see Beniger, 1986; Castells, 2000). In addition to this, social media is intrinsically linked to society: Through an anthropological perspective, social media shows us that what individuals post (even posts that appear as forms of self-expression) are in fact a product of society itself–the norms, aspirations, tensions and contradictions that exist simultaneously. (Sinanan, 2017: 206) Epistemologically, Grace’s (2014) ‘particulate vision’ describes a different relation to reality, one that is vital/affirmative (Braidotti, 2013), and that is reflective of the atomization or fragmentation of contemporary experience especially apparent in everyday online activity and social media use. Beyond well-rehearsed dystopian visions of the internet (Stoll, 1995) and fearful visions for society (Turkle, 2017), Grace (2014) has argued that when spontaneous decisions are made to capture objects and events deemed worthy of attention, such as a visual image (via camera phone or the like), a certain everydayness is produced that is far from meaningless, shallow or characteristic of a rising ‘culture of narcissism’ (Lasch, 1979). The author argues against the negative proposition that random picturetaking and sharing – of healthy food or outdoor activities and gym-based accomplishments by ‘QSers’ (discussed and defined earlier as those that measure and record their physical achievements) – are no more than forms of narcissistic attention-seeking. Instead, Grace poetically presents the captured digital image as the user’s thought or impression of a feeling at a particular moment in time, from which emerges a more meaningful world – a multiplicity of ‘sphericules’ to use Gitlin’s (1998) term for how ‘we’ might

QUANTIFIED ME

93

be imagined if our thoughts are able to ‘wander’ (or circulate) far enough. At the level of the individual, the focus on the spectre of self can be used then with critical consideration given to our active/affirmative individual agency when engaging with interconnected digital technologies across a range of ever-changing platforms. Accentuating this claim is this chapter’s non-binary assertion that while for some, technology and what it can do allows many to embrace the possibilities and actively take it on, for others the extent to which we can truly apprehend the uses is problematic. Here we might look to Habegger and Mancila (2006: 6) for social cartography as a research methodology that allows us to know and build a comprehensive knowledge of the territory so as to collectively choose a better way of living, from a better understanding of the territorial reality, of how we live in our own territory and of how we build the future territory we want. The choices made for manufacturing hardware and software, generating algorithms, language choices, fabrication of code, interface designs and the licensing systems exercised all work in combination from, and to generate, certain values. Exactly ‘whose’ values is increasingly indistinct and called into question here as we find examples where the precipitous growth and use of AI and machine learning have enabled power-relations to enter a new era. Amazon, for example, needed to scrap its ‘secret’ AI recruiting tool that showed bias against women (Dastin, 2018: n.p.). Whether intentional or not, this ‘laundering’ of human prejudice through computer algorithms can, if allowed to be unfettered, continue making those biases along racial lines or those gender-neutral (see also ‘Traveling when Trans’ by Costello, 2016) and not appear justified or objective (y Arcas et al., 2017: n.p.). What is being underscored is that the process of digitalization, HCI and the subsequent ‘algorithmic turn’ facilitate versions of ‘the self’ from multiple sociocultural influences felt across time from the spectral data traces created then left online. That users emerge through and as part of their entangled, social intra-relating, and actions do not pre-exist their interactions, Karan Barad writes: Which is not to say that emergence happens once and for all, as an event or as a process that takes place according to some external measure of space and of time, but rather that time and space, like matter and meaning, come into existence, are iteratively reconfigured through each intra-action, thereby making it impossible to differentiate in any absolute sense between creation and renewal, beginning and returning, continuity and discontinuity, here and there, past and future. (2007: ix) We might then heed the call for an approach to design that includes and, better yet, is led by marginalized communities themselves that aims explicitly to challenge, rather than reproduce, structural inequalities (Costanza-Chock, 2020). Hence, the use of digital tools by particular groups

94

MATERIALIZING DIGITAL FUTURES

FIGURE 5.1  Mediation on the network of networks.

in combination with network, quantitative and qualitative analyses offers the capacity to unearth the platform specificity of technologies that allow us to better grapple with problems, issues and controversies. This might come from the perspective of user/producer practices with the capacity of hardware and IT that together provide mediation between them for what applications and software can facilitate (see Figure 5.1). The self and our intersecting sociocultural myths are evolving in iterative processes that offer mutually constitutive opportunities for construal and (re)interpretation (Soussloff, 1997). An outcome is, however, a version of self that is rendered pixelated and ghostlike.

Delineating the spectre of self: A methodology Without doubt there are a variety of health, sleep and calory-intake tracking apps, including, for example, Fitbit for exercise (https://www.fitbit.com/ global/au/home), Sleep Cycle for rest patterns (https://www.sleepcycle.com/) or the more wholistic Life Cycle (https://lifecycle.zendesk.com/hc/en-us), that ‘keep track of your time and present your life sorted into slices’ and are thus capable of assessing personal behaviour and individual performance data. For the purposes of this chapter, however, I will undertake a close analysis of the Microsoft email services in widespread use in business and education settings where the tool is deployed by way of an emerging exemplar. What follows is a critical review of my own recorded work habits as captured from Microsoft’s Outlook email through Cortana (for daily briefing emails) and information ‘Editions’ from MyAnalytics. I use a digital (auto)ethnographic

QUANTIFIED ME

95

approach (Finnegan, 1997; Pink et al., 2015) in this brief case study analysis of Microsoft’s current services as a way to assess an aspect of my spectre of self through workplace ‘identity construction’ for the wider points for consideration this might bring to bear. Taking the key methodological principle of ‘multiplicity’ that stresses the importance of customizing digital ethnography to the research (Pink et al., 2015: 8–9), I explore the recent trend in ‘bossware’ or people analytics (Tursunbayeva et al., 2021). For Pink, understanding the wider implications of digital research is important such that research which uses digital tools, or takes place in a digital environment, will likely have implications beyond the digital sphere alone. The combined approach is underpinned by critical questions of so-called wellness and productivity management. A non-intrusive methodology is used to specifically ask these questions: (1) how do Microsoft productivity apps Cortana and MyAnalytics engage with users? (2) how might users benefit from ‘wellness’ updates? and (3) What is the nature of the practices and implications that might emerge as a result. In a focused analysis of professional e-working spaces that brings to the surface perceptions of identity, tracking and performance monitoring in which many might feel themselves to have a stake, the focus here turns to explore ‘boss-ware’ communications, which are now made salient. This study involved reading relevant email updates and viewing visual data connected to my own Outlook work account. A one-year period was selected to begin gathering data as part of the chapter’s preparatory research. Using my institutionleased machine with access via two-factor authentication supported by Duo Mobile, I recorded data from June 2020 when I was alerted to my Office 365 account including a new MyAnalytics function to provide me with a personal ‘deep dive into [my] work patterns on Focus, Wellbeing, Network and Collaboration’. I continued through November 2020 when I received my first email from Cortana until May 2021. The focus was predominantly on the text and the number of messages sent from the company to the user (me). Of the hundreds of messages collected, checked and analysed manually, 100 were selected and used in detail. A limitation associated with the dataset is that all messages were in English. This is a cultural bias to be addressed in future studies, perhaps examining how the product is deployed in other languages and professional and educational settings for greater representation from international user experiences.

Welcome to your new Daily Briefing email from Cortana: Some results In my welcome email message from Cortana was the explanation, ‘You are receiving this email because Briefing emails are currently on within

96

MATERIALIZING DIGITAL FUTURES

your organization’ and ‘You’ll receive a Briefing email whenever you have upcoming meetings to prepare for and recent tasks you may have missed so that you can make the most of each day’. As foretold by Negroponte (1995), Cortana ‘interacted’ with me akin to a personal digital assistant. I was asked in the first email: What do you want to set aside time for? You have 29 hours available this week. I can help you reserve time for what’s important. (Cortana email, November 2020) Suggestions included: Focus time ‘Up to 2 hours daily to get your work done’; Catch up on messages ‘30 minutes daily to catch up on your email and chats’; Take a break ‘15 minutes twice a day to disconnect and recharge’; and Learning ‘Up to 2 hours to learn a new skill and grow your career’. Each briefing comes with a different work-related icon (a notepad or page of a report) which are differently coloured in soothing pastels (sometimes pink or blue or green). They are accompanied with inspirational messages such as: ‘Make today count!’ ‘Off to a good start!’ ‘Have a great Thursday!’ and ‘It’s a new day!’. At the end of the week, I receive prompts: ‘Commitments to follow-up’; to action tasks as ‘Done’; or ‘Add to To Do’ list. Further, a list is generated comprising all the meetings that I accepted – and those that I did not. In addition to Cortana’s Briefings, by way of further example from the Microsoft suite of products that collect user activity data, is the MyAnalytics service. MyAnalytics gives a user their purported ‘Month in Review’. By way of verifying what I already suspected, I am very connected through the day from approximately 6:30 am to 11 pm on average. Indicative data suggests I respond to some 621 emails in a month (or 31 per day) while reading some 2,620 (see Figure 5.2). In November/December my email hours increased from ‘7 to 12 hours over the past few weeks’. In undertaking a personal evaluation of the MyAnalytics tool by way of exemplar and drawing on my own data, apparently, in 2020, I was able to maintain an average of ‘73 percent Available to focus in a typical week’ (this figure remained unchanged over four consecutive months September– December 2020). In this same period, I had ‘27 percent Collaboration time’, a figure that remained unchanged over the same four consecutive months. The reported ‘Quiet days’ (based on meeting, email, chat and calls activity outside working hours as set in Outlook) for my personal well-being was 6 days in September 2020, 4 days in October, 8 days in November but only 3 quiet days in December. I have emailed, chatted or met with over 700 people (705) in the past 12 months. To date, in 2021, my average is down to 60 per cent available time to Focus, with 40 per cent of time spent collaborating and only 2 quiet days. The function to determine my number of collaborators (albeit GDPR compliant) gave me my usage statistics including: ‘In the past 4 weeks, you’ve

QUANTIFIED ME

97

FIGURE 5.2  Communication habits.

added [X number of] new people to your network’ (including my contacts’ photos if uploaded to Microsoft by way of ‘personalization’). This is in addition to: ‘Your top collaborators’, ‘the contacts with whom you’ve spent the most time in the last month’ (where-upon their photographs, names, email and purported total hours of engagement with me are subsequently detailed). For example, my Collaborator A had 4.1 hours of my time and Collaborator B only 3.9 hours in the last month. I also get a ‘read rate’ for each collaborator as a percentage of emails received from them that I have opened as well as the response time to reply. My figures are, however, underrepresenting actual engagement and thus inaccurate when compared to itemized actions written in the work diary by way of record. These sorts of digital performative impressions and erroneous assumptions – this spectre of my self – may, however, become a major driver of increased individual and/or organizational performance. The broader outcomes of such a trend are yet to be discerned for individuals and society. One can opt out of the Cortana or MyAnalytics services of course, and one may choose not to engage with them at all. This does not necessarily mean that big and small data are not collected and used. According to the MyAnalytics Privacy Guide: ‘The majority of the data that you see in MyAnalytics is simply an aggregation of information to which you already have access, but that you wouldn’t be able to quickly perform calculations on without some support’ (a paper diary and a pencil might, however, suffice). Further: ‘MyAnalytics has no mechanism or option that allows anyone but you to access the personalized information that is displayed through these surfaces, unless you purposefully and

98

MATERIALIZING DIGITAL FUTURES

independently share it’ (https://docs.microsoft.com/en-au/workplace-analytics/ myanalytics/overview/privacy-guide-users). The boundaries for access to, and retention of, personal data are opaque. For example, a statement in app permissions explains: ‘Some apps use user data to customize the experience. If you don’t want to allow an app to access your data, you can revoke permission’. Immediately under the statement, however, is this declaration: ‘You can’t revoke permission for these apps because this has been assigned to you as part of a subscription or an admin role’. It would appear that even if one makes the active choice to not provide access to personal data, large corporations have technical access to company emails regardless through IT departments; the issues are even more serious ‘where the employer provides an employee with a mobile phone, or with homeequipment and Internet connections, because company staff could end up monitoring entirely personal activities undertaken in personal time’ (Australian Privacy Foundation, n.d.). There is a potential trade (private data for convenience) where some autogenerated suggestions become useful and enhance performance throughout the working day. For example, I was recommended a PDF document that appeared to be associated with a meeting that I was to attend on the day that I was updated. This would have been useful had I not stored a copy previously in preparation. I sense that I could easily become reliant on the machine or feel guilty when chastised as in: ‘People have spent over 4.3 hours outside working hours on the emails you’ve sent to your company in the past 4 weeks. Consider reducing the audience or trying to delay delivery.’ Albeit technologies are frequently developed benevolently, they have often succumbed to ‘function creep’, to be adopted and used in ways that are counterintuitive and astray from the original intent and needs of both their creators and society. I can of course send the company my feedback by indicating if the Briefing was useful by choosing ‘thumbs up’ or ‘thumbs down’ clickable icons.

The pixelated spectre of self: A discussion Those that cognitively labour may be sent regular personal workday analytics by email. The actuality of daily work of such people analytics is frequently different when compared to the recorded data: collaboration is much higher, focus is low in the balancing of multiple tasks and quiet days are a luxury. Performance scores are generated that often have little to do with actuality. The provocation is that services that quantify ‘productivity’ are cloaking it as increased performance in the workplace while using a form of ‘software-surveillance’. My ‘feed’ includes the slogan ‘Discover your habits. Work smarter’, which might be to encourage users, or in fact that with key information one can be evaluated and rated as to how well one

QUANTIFIED ME

99

is doing, and if not ‘productive’ or meeting set key performance indicators (KPIs), certain consequences might follow. Here, Microsoft’s services for example are marketed simply as maintaining (monitoring) a subject’s wellbeing (Cortana) or making workers ‘smarter’ (MyAnalytics). This data could provide employers with a certain confidence that the worker is productive (or in other cases, not managing), which begs the question of whether we are now, or swiftly becoming, the sum of our autonomously curated bits of pixelated information. I now place this microanalysis in the broader context of contemporary algorithms and interconnected big data that are also shaping the daily experiences of our sociocultural and technological relations. Research by Gui, Chen, Caldeira, Xiao and Chen (2017) found that sharing fitness data with one’s pre-existing social networks motivates users to continue their self-tracking practices and enhances their existing social relationships. The authors also noted that users’ concerns about their online personal images led to challenges of digital privacy – this important point about digital privacy is discussed further in the following. Online applications embedded in wearable computer-devices are not always considered for their significance in terms of being ‘items’ worn on the body and also connected and networked across devices and services. Undoubtably, there is increasing commercial value in archived and real-time data, its collection and interpretation where data is a commodity (Kant, 2020). In a world where everything happens increasingly quickly, from shorter innovation pipelines to the speed with which digital advertising campaigns are launched, realtime data is proving increasingly useful. Social media sites are themselves monitored for vital clues and intelligence for marketing, advertising, research and analysis (Kennedy, 2016). Consequently, big and small data are valuable as ‘the new oil’ (Hirsch, 2014; Palmer, 2006). For its part, because some 90 per cent of Twitter accounts are publicly available (not protected), it is a rich source of data yielding insights into peoples’ lives, and also into the way they are using the platform to communicate. Amid a digital visual revolution, a critical lens has been applied to the hegemony of digitally produced, stored, shared data and curated content (Cinque, 2015). Contemporary use of digital screen and communication technologies, which are not value neutral in society, comes with both effects and affects for individuals and communities (Clough, 2016: 437). Clough writes with conviction on this datalogical turn: Biology (and now neuroscience) … is infused with technicity or technicality – the technicity of measuring for starters. In turning all contexts of populations into a biotechnicity of calculation, quantification, and measure, population racism makes way for the health or lack thereof of populations to be part of a global market, beyond national boundedness, beyond the boundaries of the body as organism. (2016: 438)

100

MATERIALIZING DIGITAL FUTURES

Clough’s (2016) datalogical turn is reflecting on practices of ‘mining of social media, tracking devices, and biometric and environmental passive microsensors’. Authors Román, Bujanda and Zerega (2017: 85) liken the digital age to the ‘eyes of God’ with the omnipotence and omnipresence given to everyone by recording our lives through digital means with inherent share-ability resulting in us being the ‘protagonist and observer’ at the same time. They discuss the next digital era where technology is automated and outside the realm of human control (Román et al., 2017: 83–4). These convincing arguments echo those of Clough (2016: 38) in discussing the cyclic nature of the digital revolution such that society brings forth new technologies that embed themselves so deeply so as to become integral. What is being emphasized is that what we do via our social networks and everyday activities using favourite applications for things such as online social connection, study, work, shopping, gaming, movies in our ‘smart’ homes with ‘smart’ devices are all done in ways that are networked, interactive and curatorial. Evermore, the spectre of one’s real self is haunting the digital realm. This is an apropos way of discussing how emerging technologies intertwine with and transform societies that are consequently imbued with a new existence; not destroying society, but forever changing it. It is the aftermath of this radical shift that is often cathartically explored in literature (Aldous Huxley’s 1932 book Brave New World; George Orwell’s 1949 work Nineteen Eighty-Four: A Novel; I, Robot from Isaac Asimov (1950); Alan E. Nourse’s 1974 book The Blade Runner) by depicting dystopian futures that have succumbed to the machine. Utopian counter definitions for their part consider the capacity of interconnected CPS/IoT as being bound to future imaginings wherein new ‘panoptical systems’ detail all subjects – the internet of all things – to imagine a universal system of seamless functioning. Intelligence passes to things communicating information about themselves. Taking the non-binary position, however, there are certainly varying affordances of ‘always-on’ connection from smart-grids to public safety being enhanced using data from, and to, social media for information about security alerts or local bushfires or air/water quality and the like, which can be gathered easily and shared quickly. We see that in developed countries (and more unevenly in developing economies) young people are growing up in what Ohler (2010: 170) adroitly describes as natural, human and digital ‘ecosystems’, with important complex implications for education, environment, health, communication, work, and social connection. A conceptual understanding of users might now be that we are increasingly one and the same as other non-human interconnected matter; just another variable in the network of networks (Latour, 2005). As underscored in the following section, algorithmic cultures are inexorably altering not only the overarching concept of self as we are [re]curated digitally, but in the process altering the boundaries between private and public life also.

QUANTIFIED ME

101

Ethics of AI: Explain-AI and socially just standards Many of the discussions happening at a global level frame issues of AI ethics around the ways to instil trust in the AI applications that are being developed and used across the sectors of economy and business, politics and government, and research for the benefit of society and the environment. In a discussion paper from the Australian Commonwealth Scientific and Industrial Research Organisation (CSIRO), Artificial Intelligence: Australia’s Ethics Framework, the authors suggest: Artificial Intelligence (AI) has the potential to increase our well-being; lift our economy; improve society by, for instance, making it more inclusive; and help the environment by using the planet’s resources more sustainably … to realise these benefits however, it will be important for citizens to have trust in the AI applications developed by businesses, governments and academia. One way to achieve this is to align the design and application of AI with ethical and inclusive values. (Dawson et al., 2019: 3) Arguably, this approach locates the problem of ethics with users themselves rather than with the organizations developing and deploying AI. It might be suggested instead that an ethical framework should be oriented beyond manufacturing trust toward a broader desire to do good in the community. Along with practical outcomes, it is reasonable that socially just standards should be motivated by aspirational goals and values. This may manifest in a shared responsibility model encompassing both AI exercisers and providers as well as users and an informed community (den Hollander, 2019: 1). Overarchingly, proponents argue for AI systems that at their core generate benefits for societies. In accord with the affirmative ideals of every generative innovation or technology at their introduction to society, AI should be imbued with social justice intentionality. This means acting for not only human rights but for all species with minimal impact on the environment as well as for advancing economies and increasing revenue. Underscored must be the safety of civilian users and their protection – in terms of human dignity, human rights and not intentionally concealing the ‘true’ purpose of an AI component – as well as the across-the-board principle of harm minimization through both the design process and implementation stages. However, the challenges of ensuring transparency and protecting people who are highly dependent on medical care, those who are unable to give consent, children, people with a cognitive impairment, intellectual disability, or mental illness are magnified in the case of AI. AI systems should be designed and operated so as to be compatible with the ideals of respect, human dignity, rights, freedoms and cultural diversity. Such

102

MATERIALIZING DIGITAL FUTURES

an inclusive approach acts to recognize the potential of harm to individuals, communities, environment, businesses and organisations. What would be useful are explanations about what the ‘black box’ does in a manner that users can understand. Coupled with this would be greater transparency of key legislation and policies, and industry practices governing the collection, use, storage and sharing of data, which ensures that when users sign up for a service, they can do so by providing informed consent. Raising awareness facilitates better opportunities to understand why we should actively demand better digital privacy protection that respects people’s boundaries pertaining to online actions, with consideration given to the different contexts in which people experience and/or use the technologies and services.

Conclusion: Quantified me as digital curation People’s lives are lived neither wholly online nor wholly offline, but at the liminal intersection of the two. The spectre of self goes beyond a simple understanding of offline versus online selfhood to think about how the nomenclature has much to do with subject quantification generated from data and the digital traces left behind during online actions and transactions. These traces are digital breadcrumbs scattered and found in dark (closed) and open places creating a ghost-like presence of ourselves, phantasmic and not quite real; yet, retaining so much of the essence of what is/what was/what might be in our curatorial lives. This is the direct result of the autonomous creating, disseminating, collecting, storing of everyday data in quantity and at scale that can reveal and/or disguise identity from a series of connected, but (in)distinct ‘pixels’. Within the context of contemporary networked communications, it must be emphasized that the online presentation of self by (social) digital (net)workers is not completely unhampered by the conditions of the in-real-life world. Rather, the pixelated spectre of self develops within a complex (eco)system of local and global cultural conventions, social expectations and other people’s identities, private industry and governmental interactions. The concepts of identity and ‘the self’ are being reshaped in this context of social networking, quantified media and the ‘quantifiable’ self. The creative technologies field is broadly interdisciplinary and transdisciplinary whereby the rise of AI and emerging technologies have forced scholars from a range of disciplines to reassess notions of contemporary computer communications. Interdisciplinary teams frequently draw now from design, health and social development, communications, arts and the humanities, thus fostering a closer relationship with mathematical sciences, computer sciences, digital studies, and ecology and conservation biology.

QUANTIFIED ME

103

As such, the different theoretical and applied approaches to the medium provide space/s for a multitude of research questions to emerge on the ethics and impact of AI and neo-communicative processes. Such questions are tackled by the combined creative intelligence of interdisciplinary knowledge. Many contemporary discussions about AI centre around its ability to change behaviour and promote the social good. Indeed, future research lies in understanding this intersection and how to advance skills and resources in proactive sociality that straddle digital, face to face and natural environments.

References Andrejevic, M. and K. Gates (2014), ‘Big Data Surveillance: Introduction’, Surveillance & Society, 12 (2): 185–96. y Arcas, B. A., M. Mitchell and A. Todorovl (2017), ‘Physiognomy’s New Clothes’, Medium. Available online: https://medium. com/@ blaisea/physiognomys-newclothesf2d4b59fdd6a (accessed 25 May 2020). Asimov, I. (1950), I, Robot, New York: Gnome Press. Australian Privacy Foundation (n.d.), ‘Workplace Email’, Policy Statements. Barad, K. (2007), Meeting the Universe Halfway: Quantum Physics and The Entanglement of Matter and Meaning, Durham, NC: Duke University Press. Beniger, J. (1986), The Control Revolution: Technological and Economic Origins of The Information Society, Cambridge, MA: Harvard University Press. Black Mirror (2011), Television Series, ‘Zeppotron’, Channel 4 Television Corporation and Gran Babieka, United Kingdom. Booth, R. (2014), ‘Facebook Reveals News Feed Experiment to Control Emotions’, The Guardian (UK). Available online: https://www.theguardian. com/technology/2014/jun/29/facebook-users-emotions-news-feeds (accessed 30 June 2014). Braidotti, R. (2013), Metamorphoses: Towards a Materialist Theory of Becoming, Hoboken, NJ: John Wiley & Sons. Bratton, B. H. (2016), The Stack: On Software and Sovereignty, Cambridge, MA: MIT Press. Caprica (2011), Television Series, David Eick Productions and Universal Media Studios, Canada and United States of America. Castells, M. (2000), The Rise of The Network Society: The Information Age – Economy, Society and Culture, Vol. 1 (2nd edn), Cambridge, MA: Blackwell. Chen, M., S. Mao, Y. Zhang and V. Leung (2014), Big Data: Related Technologies, Challenges and Future Prospects, New York: Springer. Choi. H., J. Park and Y. Jung (2018), ‘The Role of Privacy Fatigue in Online Privacy Behaviour’, Computers in Human Behavior, 81: 42–51. Cinque, T. (2015), Changing Media Landscapes: Visual Networking, South Melbourne: Oxford University Press. Clough, P. T. (2016), ‘Rethinking Race, Calculation, Quantification, and Measure’, Cultural Studies Critical Methodologies, 16 (5): 435–41.

104

MATERIALIZING DIGITAL FUTURES

Costanza-Chock, S. (2020), Design Justice: Community-Led Practices to Build the Worlds We Need, Cambridge, MA: MIT Press. Costello, C. G. (2016), ‘Traveling While Trans: The False Promise of Better Treatment’, in Trans Advocate. Available online: http://transadvocate.com/thetsa-a-binary-body-system-in-practice_n_15540.htm (accessed 22 March 2020). Crain, M. (2016), ‘The Limits of Transparency: Data Brokers and Commodification’, New Media & Society, 20 (1): 88–104. Dastin, J. (2018), ‘Amazon Scraps Secret AI Recruiting Tool that Showed Bias Against Women’, Reuters, San Fransico, CA. Available online: https://www. reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scrapssecret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G (accessed 28 May 2020). Dawson, D., E. Schleiger, J. Horton, J. McLaughlin, C. Robinson, G. Quezada, J. Scowcroft and S. Hajkowicz (2019), ‘Artificial Intelligence: Australia’s Ethics Framework’, Data61 CSIRO, Australia. den Hollander, J. (2019), ‘Response to the CSIRO Data61 Discussion Paper “Artificial Intelligence: Australia’s Ethics Framework”, Department of Industry, Innovation and Science, Australian Commonwealth Government, Response 1004995891’. Available online: https://consult.industry.gov. au/strategic-policy/artificial-intelligence-ethics-framework/consultation/ view_respondent?uuId=1004995891. Derrida, J. (1994), Specters of Marx, London: Routledge Classics. Finnegan, R. (1997), ‘Storying the Self: Personal Narratives and Identity’, in H. Mackay (ed), Consumption and Everyday Life, 65–112. London: Sage. Giddens, A. (1993), Sociology (2nd edn), Cambridge, UK: Polity. Gitlin, T. (1998), ‘Media Sphericules’, in T. Liebes and J. Curran (eds), Media, Ritual and Identity, 79–88, London: Routledge. Goffman, E. (1959), The Presentation of Self in Everyday Life, Scotland: Doubleday. Grace, H. (2014), ‘Culture, Aesthetics and Affect in Ubiquitous Media: The Prosaic Image’, Media, Culture and Social Change in Asia Series, London: Routledge. Grimm, N. (2015), ‘Samsung Warns Customers New Smart TVs “Listen In” on Users’ Personal Conversations’, ABC News. Available online: https://www.abc. net.au/news/2015-02-10/samsung-warns-customers-new-smart-tvs-listen-in-onusers/6082144?nw=0 (accessed 10 February 2015). Gui, X., Y. Chen, C. Caldeira, D. Xiao and Y. Chen (2017), ‘When Fitness Meets Social Networks: Investigating Fitness Tracking and Social Practices on Werun’, in Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO, 6–11 May, 1647–59. Habegger, S. and I. Mancila (2006), ‘El poder de la Cartografía Social en las prácticas contrahegemónicas o La Cartografía Social como estrategia para diagnosticar nuestro territorio’, Revista Araciega, 14: 12–22. Habermas, J. (1989), The Structural Transformation of the Public Sphere: An Inquiry into a Category of Bourgeois Society, trans. Thomas Burger with the assistance of Frederick Lawrence, Cambridge, MA: MIT Press. Hirsch, D. D. (2014), ‘The Glass House Effect: Big Data, the New Oil, and the Power of Analogy’, Maine Law Review, 66 (2): 373–95.

QUANTIFIED ME

105

Hughes, C. (2012), ‘Dialogue Between Fukuyama’s Account of the End of History and Derrida’s Hauntology’, Journal of Philosophy: A Cross-Disciplinary Inquiry, 7 (18): 13–26. Huxley, A. (1998), Brave New World (1932), London: Vintage. Jeschke, S. (2014), ‘“Society 4.0” – (R)Evolution of society?’, Concordia-Grundtvig Workshop IV, Karlsruhe, 5. Available online: www.imazlwifu.rwthaachen.de/ leadmin/user_upload/INSTITUTSCLUSTER/Publikation_Medien/Vortraege/ download/Society_4_0_ KIT_5March2014.pdf. Kant, T. (2020), Making It Personal: Algorithmic Personalization, Identity, and Everyday Life, New York: Oxford University Press. Kennedy, H. (2016), Post, Mine, Repeat: Social Media Data Mining Becomes Ordinary, London: Springer Nature. Klingemann, M. (2018), Artwork, Memories of Passersby I. Kotenko, J. (2013), ‘To Infinity and Beyond Verbal: The Web App Revolutionizing the Science of Voice Analysis’, Digital Trends. Available online: https://www. digitaltrends.com/social-media/exploring-beyond-verbal-the-technology-ofemotions-analytics/ (accessed 23 July 2021). Kröger, J. L., O. H. M. Lutz, P. Raschke (2020), ‘Privacy Implications of Voice and Speech Analysis–Information Disclosure by Inference’, in M .Friedewald, M. Önen, E. Lievens, S. Krenn, S. Fricker (eds), Privacy and Identity Management. Data for Better Living: AI and Privacy. Privacy and Identity 2019. IFIP Advances in Information and Communication Technology, 576, Cham: Springer. Available online: https://doi.org/10.1007/978-3-030-42504-3_16. Lasch, C. (1979), The Culture of Narcissism: American Life in an Age of Diminishing Expectations, New York: W.W. Norton. Latour, B. (2005), Reassembling the Social: An Introduction to Actor-NetworkTheory, Oxford, UK: Oxford University Press. Liu, W. and Z. Gao (2014), ‘Study on IoT Based Architecture of Logistics Service Supply Chain’, International Journal of Grid & Distributed Computing, 7 (1): 169–78. Lupton, D. (2013), ‘Quantifying the Body: Monitoring and Measuring Health in the Age of mHealth Technologies’, Critical Public Health, 23: 393–403. Lupton, D. (2014), ‘Critical Perspectives on Digital Health Technologies’, Sociology Compass, 8: 1344–59. Lupton, D. (2015), ‘Quantified Sex: A Critical Analysis of Sexual and Reproductive Self-Tracking Apps’, Culture, Health & Sexuality, 17 (4): 440–53. Lupton, D. (2016), The Quantified Self, Malden, MA: Polity Press. Makridakis, S. (2017), ‘The Forthcoming Artificial Intelligence (AI) Revolution: Its Impact on Society and Firms’, Futures, 90: 46–60. Marchi, R. (2012), ‘With Facebook, Blogs, and Fake News, Teens Reject Journalistic “objectivity” ’, Journal of Communication Inquiry, 36 (3): 246–62. Negroponte, N. (1995), Being Digital, New York: Alfred A. Knopf:. Nourse, A. E. (1974), The Blade Runner. David McKay Publications: Philadelphia. Ohler, J. (2010), Digital Community, Digital Citizen. Thousand Oaks: Corwin. Omidyar, P. (2018), ‘6 Ways Social Media Has Become a Direct Threat to Democracy’. New Perspectives Quarterly, 35(1): 42–5. Orwell, G. (1949), Nineteen Eighty-Four: A Novel, London: Secker & Warburg.

106

MATERIALIZING DIGITAL FUTURES

Palmer, M. (2006), ‘Data is the New Oil’, ANA Marketing Maestros, 3 November. Available online: http://ana.blogs.com/maestros/2006/11/data_is_the_new.html (accessed 22 May 2021). Pazzi, L. and M. Pellicciari (2017), ‘From the Internet of Things to Cyber-Physical Systems: The Holonic Perspective’, Procedia Manufacturing, 11: 989–95. Pink, S., H. Horst, J. Postill, L. Hjorth, T. Lewis and J. Tacchi (2015), Digital Ethnography: Principles and Practice, London: Sage. Román, C. T., H. Bujanda and M. M. Garaycoa (2017), ‘The Future Is Broken: Heterotopic Readings of Black Mirror’, Nómadas, 47: 81–95. Runciman, B. (2014), ‘Where Are We with Big Data’, in Big Data: Opportunities and Challenges, 1–3, BCS The Chartered Institute for IT, BCS Learning & Development Limited. Available online: http://deakin.eblib.com.au/patron/ FullRecord.aspx?p=1650370. Schäfer, M. T. and K. van Es (eds) (2017), The Datafied Society: Studying Culture Through Data, Amsterdam: Amsterdam University Press. Sinanan, J. (2017), ‘Conclusion: Social Media Through Ethnography’, in Social Media in Trinidad: Values and Visibility, 199–208, London: UCL Press. Soussloff, C. (1997), The Absolute Artist: The Historiography of a Concept, Minneapolis: University of Minnesota Press. Stoll, C. (1995), Silicon Snake Oil: Second Thoughts on the Information Highway, New York: Anchor Books. Tucker, J. A., Y. Theocharis, M. Roberts and P. Barbera (2017), ‘From Liberation to Turmoil: Social Media and Democracy’, Journal of Democracy, 28 (4): 46–59. Turkle, S. (2017), Alone Together: Why We Expect More from Technology and Less From Each Other, United Kingdom: Hachette. Tursunbayeva, A., C. Pagliari, S. Di Lauro and G. Antonelli (2021), ‘The Ethics of People Analytics: Risks, Opportunities and Recommendations’, Personnel Review. Available online: https://doi.org/10.1108/PR-12-2019-0680. Westworld (2016), Television Series, ‘Bad Robot’, Jerry Weintraub Productions, Kilter Films and Warner Bros. Television, United States of America.

6 Virtual reality and kinaesthetic connection: Qualities of ‘being there’ Kim Vincs

Introduction Bridging physical and virtual spaces has never been more critical. The impact of Covid-19 has been that many parts of the world have entered and re-entered lockdown conditions designed to prevent physical contact. As I write this chapter, early in 2021, it is not yet clear how long this situation will last – at best months, at worst, years. The ongoing reality of sudden and unexpected lockdowns that can take place at almost any time and in any part of the world has rapidly accelerated our reliance on digital forms of communication across many industrial, social and cultural domains. It has also revealed how lacking we are in 3D immersive platforms for digital communication. Information is increasingly visual and dimensional, yet our most common forms of digital communication operate via flat screens and zoom conferences. The sudden physical and geographical isolation our world is experiencing calls into question the nature and function of immersive technologies in new ways. In many ways, the relationship between physical and virtual spaces has never been more complex. On the one hand, virtual spaces have, of necessity, become a new normal for workplaces, meetings and social gatherings. On the other hand, our sudden confinement to connection via virtual spaces

108

MATERIALIZING DIGITAL FUTURES

emphasizes all that virtual spaces are not, and foregrounds the elements of human interaction digital screens cannot emulate. Touch is one example. Spatiality is another, along with proxemics – the amount of space needed between people – and body language. The tensions involved in bridging virtual and physical spaces, which this book examines, have been thrown into sharp relief as virtual spaces have become more entwined with everyday work and life. That this has not happened by choice but through necessity changes the dynamic of this book’s premise. We now look to conduct everyday business and social functions virtually, not through choices made for specific situations or contexts but utilizing technologies that were never designed for these contexts. Understanding the affordances of virtual reality (VR) – what it can, cannot and could, in the future, potentially do – is critical to developing the marriage of affordances and use cases needed to improve the function and value of virtual connection technologies. VR is effectively a new medium for experiencing physical presence. In VR, one literally moves and interacts within an immersive, 3D world. While film and video offer a ‘window’ into another world, VR offers a frameless experience in which one moves and interacts within a digitally created environment. Current VR headsets detect and track participant’s movements and, based on this information, calibrate and update the participant’s point of view in real time so that the visual environment appears to respond as if they were experiencing the scene. This creates the illusion that one is ‘really there’. When the participant looks and moves around the virtual ‘space’, the space responds as if they were physically present. The value proposition of VR lies in this ability to generate a compelling sense of physical presence – of ‘being there’ – and to deliver this experience remotely via headset or 3D display. For these reasons, VR is being widely explored across a range of fields such as medical rehabilitation, virtual simulation training and workplace behaviour and empathy training (e.g. Kim, 2018; Smith et al., 2019; Fusaro et al., 2018; Lugrin et al., 2018). Given this exploration and the potential benefits of VR in facilitating virtual presence across many areas of society and commerce, it is critical to understand how VR mediates physical and virtual interaction – its interactive affordances – and how these advance and challenge the various communication needs of our current and future society. In this chapter, I aim to undertake an analysis of some of the conceptual aspects of 3D immersive technologies that are foundational to developing new approaches and new use cases for immersive technologies. More specifically, I address what I perceive to be an unresolved tension in the use of VR. At the heart of VR’s perceived value is the creation of an illusion of presence, of ‘being there’, which provides the preconditions for ‘immersion’ – a state in which one is completely engaged within the interactive construct of a virtual environment. However, the concept of immersion in VR is complicated by its inescapable relationship with embodied experience. While almost all VR

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

109

literature understands immersion in VR as being embodied in the sense that one physically feels as though one ‘is really there’, it is important to unpack what is meant by ‘physically there’ in different disciplinary contexts. Some analyses consider embodiment a condition or property that is automatically driven at the sensory level, and hence non-reflexive (e.g. Hartmann and Fox, 2021). This approach makes it possible to claim that VR is, on the one hand, a wonder drug for education and change (e.g. the ‘empathy machine’; Milk, 2015), but on the other, a potentially dangerous form of manipulation that can be used for ill (Slater et al., 2020). My aim in this chapter is to rethink the concept of embodiment in VR in a way that resolves this uneasy tension between VR as immersive and/or coercive by understanding embodied action and cognition as reflexive rather than automatic, and hence able to deal productively and creatively with the illusion of ‘being there’. Dance provides an ideal disciplinary starting point for this process because dance is fundamentally concerned with embodied modes of ‘being in the world’. Contemporary dance practices stem from the assumption that physical movement is not simply instrumental but an enactment of embodied cognition, thus defining dance as an art form, as opposed to a sport. Theories of embodied cognition hold that human cognition is formed in and through interactions between the body’s sensations, perceptions and actions and the surrounding environment (Varela et al., 1993). Meaning is therefore inseparable from the environment within which it is created. In dance, this environment is the world of physics (distances, depths, volumes, masses and forces), imbued with cultural, social and artistic meaning through the dancer’s actions. Thinking drawn from dance technology is particularly valuable for understanding physical and virtual interaction because dance technology artworks integrate dance’s embodied knowledges and practices within interactive computational and data-driven systems. Both human computer interaction (HCI) and dance researchers have recognized the necessity of engaging embodied perspectives such as dance in evaluating existing HCI systems and developing new ones (Loke and Robertson, 2013). Several largescale international dance research projects have applied dance–technology interaction as a method for HCI development. Key projects include Moving Stories led by Simon Fraser University in Vancouver, introducing dance-based wearable technologies (Schiphorst and Pasquier, 2015) and WhoLoDance, a European Horizon 2020 project developing motion capture and holographic techniques for dance teaching (WhoLoDance, 2016). Such projects are at the forefront of what I term ‘practice-based research 2.0’ (Vincs, 2017). Practice-based research ‘1.0’ began some twenty-five years ago in the UK and Australia. It was based on the realization that knowledge production is inherent in the creative process. This gave rise to the attempt to extract and articulate, via textual and representative forms, the new knowledge created in the process of making an artwork. A key tenet

110

MATERIALIZING DIGITAL FUTURES

of this approach was that the tight coupling between artwork and form means that some aspects of this knowledge are impervious to translation and can only be fully articulated within the artwork itself. Practice-based research 2.0 takes the further step of applying knowledge generated through art form innovation to fields and applications beyond the arts. In this chapter, my approach draws on practice-based research ‘2.0’. I aim to draw on knowledge generated through my dance technology and interactive media practice to examine the complex relationships between ‘embodiment’ and ‘being there’ in VR systems.

The Walk experience I want to begin with a story about an experience I had with VR some years ago. I was in LA at IMAX’s VR lounge. It was October 2017, and I wanted to see how commercial operations were beginning to roll out VR experiences as entertainment. I wanted to understand how a VR lounge was set up and observe the conventions around it. Was it like a cinema? How would invigilation be managed? How much hands-on instruction did patrons need, and how would they manage (choreograph) the experience of entering the lounge, putting on VR headsets and learning how to operate the interactive systems? As I made my online bookings and then arrived at the lounge, I was impressed by the choreography of the VR lounge experience. I was ushered to a private booth, carefully shepherded through the process of donning and using my VR headset, and the invigilator helped me play the experiences, explaining how I could get more out of them and stepping in and playing with me to help me get the most out of the multiplayer experiences. So much for VR being socially isolating. The need for one-on-one supervision – no doubt a function of the caution with which VR has been introduced to the general public given the potential for motion sickness and accidents such as collisions and falls – made me feel the opposite of distanced. I was, on the contrary, the centre of attention. I was also surprised to see that the VR Lounge was almost empty. I seemed to be one of only a very few people accessing this carefully thought-out experience. However, the big surprise came not from these factors but from the effects of the VR experience itself. Research into VR experiences in contexts such as medical rehabilitation, pain relief, virtual simulation training and workplace and empathy training reveals that VR generates an extraordinary level of engagement. What has been more difficult to determine has been the extent to which the compelling and engaging nature of the experience translates into the desired outcomes, for example, faster or more effective learning, pain relief, and the like. My experience shed a new light for me on why this might be the case.

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

111

The first experience I did in the VR lounge was called ‘The Walk VR’ (Sony Pictures, 2015). The Walk presents a scenario in which the player re-enacts Philippe Petit’s famous highwire crossing between the Twin Towers of the World Trade Centre on a high in New York in 1974. In Sony Pictures’ VR recreation, a rope is laid on the floor to simulate the highwire. As one walks across the rope, the illusion is that you are actually many stories high, walking a tightrope between the buildings 94 floors up. Having no fear of heights, I began by walking back and forth across the wire a few times. I experimented with looking down at the streetscape, 400 virtual metres below me, up into the sky and at the buildings around me. I could feel that I was in between those buildings, even though I was fully aware that I was not. I could sense the narrowness of the rope, even though it was clear that the rope I was standing on in the physical world was a completely different size and shape than the one represented in the VR experience. As I became more acclimatized to the sensation of being so high in the sky, I began to experiment further. I stood on one leg, raised the other behind me for balance and began to lean forward from the hips to look down at the ground. As I bent forward, hovering over the immense drop below me, I could feel the impact of this revised spatiality in my body. I felt my chest curve slightly inwards and upwards, as if to start to form a concave opposition to the space below. I began to sense and simultaneously respond to the distance between me and the virtual ground in my torso. I began to play with the distance between myself and the virtual ground by subtly increasing and decreasing the concave curve. Doing this gave me a means of becoming aware of my torso’s muscular resistance as a factor in giving me a level of awareness and control of my situation. I moved subtly towards and away from the ground below me, as if to calibrate my ability to judge and control my relationship to it. My torso seemed to rearrange itself subtly to respond to the immense distance. I felt suspended, hovering above an extraordinary volume of air and streetscape. Was I frightened? I was not. I was inspired. If this virtual landscape could so profoundly affect my awareness of my body in space, what then were the implications for designing VR experiences specifically to experiment with body awareness? Could such experiences be designed to shift people’s awareness of their bodies through movement? Could VR experiences alter the foundational sense of a person’s relationship with the world around? Could such experiences continue to influence people’s behaviour when they are outside VR? Could specific effects and attitudes towards moving in space be cultivated in VR and revisited outside it? If so, could VR be used educationally and/ or therapeutically to help people to learn to respond differently in their bodies to specific circumstances and situations? And could VR be used to develop new approaches to embodiment in circumstances not accessible without the benefit of a carefully constructed virtual environment? Could

112

MATERIALIZING DIGITAL FUTURES

we learn to ‘float’, for example, or brace against forces or impacts, using graded experiences that cannot safely be created in the physical world? VR has been used for physical training in several contexts, for example, to allow NASA’s astronauts to practice space-walk procedures on earth (Carson, 2015). However, my thoughts turned to whether VR training environments could go beyond replicating specific real-world experiences to tap into the capabilities of a reflexive ‘thinking’ body. Could the physical parameters of VR environments be manipulated to develop new movement skills that can be applied across multiple contexts, rather than simply the specific scenario presented? I left the VR lounge that day with a profound new knowledge produced in my body – the capacity of VR to induce new experiences. Since then, I have often wondered why VR development often seems so fixated on replicating physical reality. The transformative potential of VR seems of much more value to me.

Embodiment, immersion and VR Embodiment, the personal sense, participation and control of our bodies, underpins and enables everything humans do. We cannot move or act in the world without understanding ourselves as embodied beings, and we do this in real-time through perceptual and sensorimotor experience and feedback. Immersion – in the sense of a deep mental involvement in something – is always embodied. VR, augmented reality (AR) and mixed reality (MR) extend the close coupling between human sensation, perception and action with the physical world, the basis of human action and cognition for millennia, to seamlessly incorporate digital computational processes within the feedback loop. While all action and experience are embodied, VR takes embodied processes out of their everyday context and activates them within artificially created virtual situations and scenarios. VR is, therefore, a medium in which embodied actions can be developed in response to environments that go beyond what is accessible or sometimes even possible in the physical world. However, beyond its sheer ability to place people in exotic environments, VR takes the further step of inducing what is often referred to as ‘immersion’. The sense of immersion changes the quality of embodied experiences in VR from quotidian to something more powerfully engaging. It is this sense of immersion that gives successful VR experiences the sensory and emotional power for which VR is known. Investigation into the affordances of VR as a unique new medium for experiencing physical presence has therefore focused on how the sense of immersion is constructed in VR, and the manner in which it depends on our understanding of how embodiment underpins the concept of ‘presence’.

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

113

Kilteni et al. (2012) developed a new term to define the concept of embodiment within VR. They describe a ‘sense of embodiment’ (SoE) that has three constituent elements. These are the senses of self-location, agency and body ownership. Self-location refers to the volume of space in which one senses one is located. This refers specifically to being inside a ‘body’ – virtual or real – rather than simply being located somewhere in a world. Agency refers to having motor control over one’s ‘body’ – again, real or virtual. One can move one’s ‘body’ and by doing so, has agency in the ‘world’ – real or virtual. Body ownership refers to the sensation that the body in question is ‘mine’. Kilteni et al. relate this to a combination of ‘top-down’ cognitive processing, for example, morphological similarity to one’s actual body, and ‘bottom-up’ visual, tactile and proprioceptive input. Kilteni et al. explore how the SoE can be ‘fooled’ by experiments such as the now famous rubber hand, in which participants experience a virtual hand as their own even when it is spatially displaced from their own hand. Their concern is to probe the extent to which VR users identify a SoE with a virtual representation of themselves. Thus, their schema provides a means of understanding SoE in VR scenarios that incorporate some form of visual representation of the user/player (commonly as an avatar or a pair of virtual hands). However, in scenarios that do not provide a representation of the user/player, SoE becomes more difficult to apply since self-location, agency and body awareness are implied within the scenario, but must be constructed by the user/player. Ratan and Dawson (2015) define embodiment as ‘the extent users experience the body of their virtual representation, or avatar, as their actual body or an extension thereof’ (2015: 3). Hartmann and Fox (2021) draw on this and similar definitions of embodiment to make the argument that VR feels ‘more real’ than other visual media. They argue that the process of embodiment allows the user to feel that they are the virtual character and that the sense of presence this generates is reinforced by spatially co-locating the avatar with the user and through illusions of co-presence with other objects in the VR scene. ‘VR may feel more real because it enables users to interact naturally with the environment. Unlike non-interactive media, objects have action potential and can respond in natural ways to the user’s actions’ (Hartmann and Fox, 2021: 10). Approaches such as these link the concept of immersion with the capacity to mentally replace the physical world with a virtual one. This displacement relies on identifying with a virtual body as if it is one’s own. T. Iachini et al. (2019 )and Burdea and Coiffet (2003) argue that the sense of presence in VR arises from ‘three Is: Immersion or capacity to isolate from the external world, Interaction or capacity to naturally exploring the virtual environment, and Imagination or individual aptitudes with mental imagery’ (Iachini et al., 2019: 1). In this definition, immersion in VR requires isolation from the external world. Without first tuning out the external world, it is

114

MATERIALIZING DIGITAL FUTURES

not possible to fully ‘immerse’ within a virtual one. At first glance, this seems quite reasonable since the point of VR is to inhabit and interact within nonreal (virtual) situations and environments. However, the deeper implication is that VR distances the user from their own body, which is replaced by the illusion of a body, sometimes visually represented and sometimes simply implied, which is created within the VR design.

Some perspectives from dance Dance theorist Sita Popat presents a different perspective. She argues that VR places us more in touch with our bodies rather than less. ‘The nature of action in virtual worlds is such that our bodies are both present and absent, experiencing agency and aspects of sensation even though there is no direct contact between flesh and world’. Popat asks: ‘How do we approach the nature of embodied experience in VR when anything can be done, but the body is apparently missing?’ (Popat, 2016: 359). Popat acknowledges the disorientation and sense of the body’s absence that arises in VR experiences that do not represent the body. She describes her own feelings of disorientation and sense of a kind of ‘missing, or phantom body’ while experiencing Ruth Gibson and Bruno Martelli’s VR installation, White Island. In this work, the user/player rides in a virtual hot air balloon over the Arctic Circle. Popat describes how her body was proprioceptively present but visually absent. However, she also explains, quoting Maxine Sheets-Johnstone, that ‘the tactile-kinaesthetic body cannot be fooled … Space at its source is a corporeal space defined by the intrinsic spatiality of animate form and the inherent spatial possibilities of the tactile-kinaesthetic body.’ This fundamental corporeal space arising from proprioceptive sensation is the basis by which I inhabit the virtual environment, whether an avatar is visible or not … My body always knows where I am in corporeal space, and I make cognitive sense of that corporeal space in relation to the world that I see around me, folding physical and virtual together rather than experiencing a binary division. My body cannot be missing because my corporeal space is ‘here’, engaging in action. (Popat, 2016: 367) This description resonates with my experience of The Walk. I do not experience my body as absent or missing. I experience a synthesis between my body and the virtual world. I can feel myself standing on ground – the actual ground – while I simultaneously sense standing on a ledge 400 m above the virtual ground. I had no physical sense that I could actually fall in The Walk, because I could sense my position in physical space as well as in virtual space. My body – and my SoE in Kilteni’s terms – is in two places

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

115

at once. And this feels very familiar. I am a dancer. Sensing, consciously exploring and manipulating how my body ‘is in the world’, and doing this in real-time – as I move – is a core element of my practice. Dance technology practice and research provides a basis for thinking through how this duality can play out in virtual spaces. Interactive dance artist Megan Beckwith describes her use of projected 3D stereographic environments in her work as a means of overlaying a physical volume with a virtual volume (Beckwith and Vincs, 2015) (Figure 6.1). This is the basic principle of AR and MR systems – a virtual world is superimposed and spatially matched to a physical one. Beckwith constructs stereoscopic environments that create a visual sense of ‘Gestalt’. The virtual images ‘make sense’ in relation to the physical world. When Beckwith dances in front of/ within her projected environments, she appears to be spatially integrated with them. She dances, for example, ‘within’ a virtual door frame, or ‘walks down’ a virtual corridor. This is the conceptual equivalent of the physical rope on which I stood in The Walk, which also represents the rope in the VR scenario. The spatial and conceptual congruence of the two ‘ropes’ assists the perception that real and virtual space form a single narrative and construct. The duality of real and virtual can also be negotiated via juxtaposition or disruption. In my own stereoscopic works, I have taken an opposite approach, creating virtual environments that clash with each other rather than reinforce the creation of a spatially cohesive scene. In my work The

FIGURE 6.1  The layered design of the space. Reproduced from Beckwith & Vincs 2015, © Megan Beckwith. Used with permission.

116

MATERIALIZING DIGITAL FUTURES

FIGURE 6.2  The Crack Up (2014). Performer: Brodie Chesher; Choreographer/ Director: Kim Vincs; 3D Image: Bobby Lin. Used with permission.

Crack Up, for example, a larger-than-life virtual performer ‘cracks’ and breaks into pieces above the head of a physical performer (Vincs et al., 2014) (Figure 6.2). In Choreotopography (Vincs et al., 2010), virtual tiles ‘float’ around the dancers creating a disjuncture between real and virtual space, which the audience must synthesize perceptually despite its improbability (Figure 6.3). Jordan Vincent, John McCormick and I have described this approach as ‘splitting centre’ (Vincent et al., 2015), describing work in which we do not try to hide the sensory disjuncture between live performers and virtual imagery, but instead highlight and creatively manipulate the resulting clashes and juxtapositions. In these works, as in VR, the fundamental question is not whether it is possible to experience, process and respond to more than one type of embodied experience at a time, but how the differently experienced elements

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

117

FIGURE 6.3  Choreotopography (2010). Performer: Sakura Shimizu; Choreographer/Director: Kim Vincs; 3D Image: Daniel Skovli. Used with permission.

of physical and virtual environments are ‘composed’ and presented to audiences. From my perspective as a dancer and dance technology artist, I would argue that embodied experiences – and the sense of presence that comes with them – do not have to be consonant. It is entirely possible to perceive and work creatively with disjunctures between different modes of experience. This argument turns on the extent to which embodied processes are reflexive as opposed to automatic. Hartmann and Fox argue, ‘Embodiment, spatial presence, and co-presence are mainly automatic, bottom-up, sensorydriven perceptual sensations that together define the typical VR experience’ (Hartmann and Fox, 2021: 20). They understand embodiment as the opposite of what they term cognitive distancing, which is the ability to be aware

118

MATERIALIZING DIGITAL FUTURES

that a VR experience is not real while experiencing it. Cognitive distancing relies on ‘top-down, higher-order cognitive processing’ (Hartmann and Fox, 2021: 6). Embodiment, however, as an automatic process driven entirely by the tight coupling between sensation and perception, remains outside the domain of higher-order cognitive function. Hartman and Fox argue that the automatic nature of embodied process results in VR being a ‘high-risk, highgain scenario’. Users might be so immersed via their embodied identification as an actor within the scenario that they may temporarily forget what they are experiencing is not real. Given VR’s affordances, the VR experience can result in powerful primary responses, including intense arousal levels. The risk is that users may find it difficult to regulate these primary responses, if they become too intense or are otherwise at odds with the experience users seek (Hartmann and Fox, 2021: 12). This is a logical outcome of theories that consider the human sensoryperception system as automatic, and hence opaque to higher-order cognitive processes. In contradiction to Popat’s experience of White Island and my experience of The Walk, the body can be fooled because its SoE is not reflexive. Dance offers an alternative scenario – the construction and cultivation of the ‘intelligent body’. When I go into a VR space, I make choices to explore different things. But perhaps when I say this, I am talking about something that is integral to my ‘dance knowledge’. As a dancer, when I say I make choices, I am normally moving when I make them, and I make them by moving. I do not need to remind myself that a mediated experience is not a real one because to me, all movement experiences are real, whether mediated or not. My movement choices mediate my interaction with the environment inside or outside VR. My issue here is not with the idea of cognitive distancing, but that it is placed in opposition to embodiment. Can I not be embodied and cognitively distanced? Can I not actually enact a form of cognitive distancing in and by moving? As a dancer I do this all the time. I notice while I dance and the noticing forms a delicate and complex feedback loop that informs me while I dance, and upon which I reflect when I am not dancing. My experience is of a reflexive embodiment. I understand this as a dialogue between how I am moving and how I could be moving, what I am experiencing in my body and what I could be experiencing. I am ‘taking notice’ in a very fine-tuned way to my movement as it happens. I am not necessarily confined to one modality or another, for example, vision versus proprioception, as if these were competing cognitive functions waging some sort of ongoing war for dominance. Dance offers a different perspective – that all forms of embodied experience might be integrated within a more holistic cognitive process, and that we can consciously and creatively shift between modalities, and simultaneously and consciously consider the outcomes and imagine what our next choices might be.

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

119

Conclusion The concept of ‘being there’ in VR is central to VR’s claims as a valuable new medium for entertainment, education and health. However, ‘being there’ in VR is a complex matter that exposes fault lines in our understanding of embodied processes and embodied experience. If embodied processes are automatic, driven by sensory-perceptual feedback alone, and if they are isolated from higher cognitive functions that drive thinking, decisionmaking and creativity, then the body in VR can indeed be fooled. If this is the case, we need to be very careful about what scenarios, environments and situations we create for VR users, since we risk harm as well as benefit with every experience we create. However, dance offers another perspective on embodiment. Dance experience and dance practice would say the body cannot be fooled because dance is precisely the artform of experiencing and consciously and creatively manipulating how the body ‘is in the world’. This perspective raises questions and possibilities for VR experiences beyond simply whether they are immersive or coercive. If the body can learn directly from and within VR experiences because the body can be an active rather than a passive participant in cognitive function, then it becomes possible to create VR experiences that enable users to learn, develop and change their responses to various scenarios and situations at the movement awareness level. This has profound implications for movement rehabilitation applications of VR since it raises the possibility of improving quality and ease of movement through imagery made virtual and interactive. It also has implications for VR as a medium for both entertainment and communication since the interaction experience engages and can potentially be a means of further integrating embodied experience.

References Beckwith, M. and K. Vincs (2015), ‘Parallax: Dancing the Digital Space’, ISEA2015, 21st International Symposium on Electronic Art, 14–18 August 2015, Vancouver, Canada. Burdea, G. C. and P. Coiffet (2003), Virtual Reality Technology, New York: Wiley. Carson, E. (2015), ‘NASA Shows the World Its 20-Year Virtual Reality Experiment to Train Astronauts: The Inside Story’. Available online: https://www. techrepublic.com/article/nasa-shows-the-world-its-20-year-vr-experiment-totrain-astronauts/ (accessed 18 May 2021). Fusaro, M., G. Tieri and S. M. Aglioti (2018), ‘Influence of Cognitive Stance and Physical Perspective on Subjective and Autonomic Reactivity to Observed Pain and Pleasure: An Immersive Virtual Reality Study’, Conscious Cognition, 67: 86–97. Hartmann, T. and J. Fox (2021), ‘Entertainment in Virtual Reality and Beyond: The Influence of Embodiment, Co-Location, and Cognitive Distancing on Users’ Entertainment Experience’, in P. Vorderer and C. Klimmt (eds), The Oxford

120

MATERIALIZING DIGITAL FUTURES

Handbook of Entertainment Theory, Oxford Handbooks online, doi: 10.1093/ oxfordhb/9780190072216.013.37. Iachini, T., L. Maffei, M. Masullo, V. P. Senese, M. Rapuano, A. Pascale, F. Sorrentino and G. Ruggiero (2019), ‘The Experience of Virtual Reality: Are Individual Differences in Mental Imagery Associated with Sense of Presence?’ Cognitive Processing, 20: 291–8. Kilteni, K., R. Groten and M. Slater (2012), ‘The Sense of Embodiment in Virtual Reality’, Presence, 21 (4): 373–87. Kim, J. H. (2018), ‘Effects of a Virtual Reality Video Game Exercise Program on Upper Extremity Function and Daily Living Activities in Stroke Patients’, Journal of Physical Therapy Science, 30 (12): 1408–11. Loke, L. and T. Robertson (2013), ‘Moving and Making Strange: An Embodied Approach to Movement-Based Interaction Design’, ACM Transactions on Computer–Human Interaction, 20:1, doi: 10.1145/2442106.2442113. Lugrin, J., S. Oberdorfer, M. Latoschik, A. Wittmann, C. Seufert and S. Graffe (2018), ‘VR-Assisted vs Video-Assisted Teacher Training’, 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Reutlingen, 625–6, doi: 10.1109/VR.2018.8446312. Milk, C. (2015), ‘How Virtual Reality Can Create the Ultimate Empathy Machine’, TED Talks. Available online: https://www.ted.com/talks/chris_milk_how_ virtual_reality_can_create_the_ultimate_empathy_machine (accessed 14 February 2021). Popat, S. (2016), ‘Missing in Action: Embodied Experience and Virtual Reality’, Theatre Journal, 68: 357–78. Ratan, R. and M. Dawson (2015), ‘When Mii Is a Me: A Psychophysiological Examination of Avatar Self-Relevance’, Communication Research, 43 (8): 1065–93. Schiphorst, T. and P. Pasquier (2015), ‘Moving Stories, Simon Fraser University’, ACM Interactions. Available online: https://interactions.acm.org/archive/view/ january-february-2015/movingstories-simon-fraser-university (accessed 14 February 2021). Slater, M., C. Gonzalez-Liencres, P. Haggard, C. Vinkers, R. Gregory-Clarke, S. Jelley, Z. Watson, G. Breen, R. Schwarz, W. Steptoe, D. Szostak, S. Halan, D. Fox and J. Silver (2020) , ‘The Ethics of Realism in Virtual and Augmented Reality’, Frontiers in Virtual Reality, 1 (1): 10.3389/frvir.2020.00001. Smith, M. J., J. D. Smith, M. F. Fleming, N. Jordan, E. A. Oulvey, M. D. Bell, K. T. Mueser, S. R. McGurk, E. S. Spencer, K. Mailey and L. Razzano (2019), ‘Enhancing Individual Placement and Support (IPS) – Supported Employment: A Type 1 Hybrid Design Randomized Controlled Trial to Evaluate Virtual Reality Job Interview Training Among Adults with Severe Mental Illness’, Contemporary Clinical Trials, 77: 86–97, doi: 10.1016/j.cct.2018.12.008. Sony Pictures Home Entertainment (2015), VR Installation, The Walk VR. Varela, F. J., E. Thompson and E. Rosch (1993), The Embodied Mind: Cognitive Science and Human Experience, Cambridge, MA: MIT Press. Vincent, J., K. Vincs and J. McCormick (2015), ‘Splitting Centre: Directing Attention in Transmedia Dance Performance’, ISEA2015, 21st International Symposium on Electronic Art, 14–18 August 2015, Vancouver, Canada.

VIRTUAL REALITY AND KINAESTHETIC CONNECTION

121

Vincs, K., R. Vincs, J. McCormick, D. Skovli and P. Divers (2010), 3D Stereo Dance Performance (Live Motion Capture), Choreotopography, The Arts Centre Melbourne, December 2010. Vincs, K., J. McCormick, R. Vincs, D. Skovli, S. Taylor, K. Wallace, B. Lin and P. Divers (2014), Transmedia Dance Performance, The Crack Up, Merlyn Theatre, The Coopers Malthouse, City of Melbourne Knowledge Week, 31 October–1 November 2014. Vincs, K. (2017), ‘Directions in Dance Technology Research’, NiTRO: NonTraditional Research Outcomes, Deans and Directors of Creative Arts, 6 July 2017. Available online: https://nitro.edu.au/articles?author=595a9a00f7e0ab046 550e3a3 (accessed 30 October 2017). WhoLoDance (2016), ‘WhoLoDance: Whole Body Interactional Learning for Dance Education’, http://www.wholodance.eu (accessed 23 February 2017).

122

7 Feminist memes: Digital communities, identity performance and resistance from the shadows Shana MacDonald and Brianna I. Wiens

Introduction In the current era of social media activism, feminist memes perform digital resistance against misogyny through an appeal to collective experiences of humour and rage. These memes often employ a visual signifier, or figure, to hold the complex entanglements of affect and critique they advance. Consider for instance the recent influx of memes centring the witch as a symbol of political feminist action. Since the outset of the viral MeToo hashtag in 2017, cries of ‘witch hunts’ against famous men rang out across physical and digital media spaces, with feminists quickly firing back (see, e.g. CBC News, 2019; McLaren, 2017; West, 2017). Here witch hunts morphed from something historically enacted against women to something done to male perpetrators of sexual harassment, and by proxy, to men more broadly. These public discussions centred the experiences of white celebrities and harassers, troublingly overlooking the important work of Black feminist activist Tarana Burke who started the MeToo Movement to

124

MATERIALIZING DIGITAL FUTURES

speak out against sexual harassment and assaults of young women of colour in 2006 (North, 2018; West, 2019). Thus, #MeToo was mired in erasures and the rewriting of historical contexts. While erasures and misuses of the term witch hunt are tangential to the larger questions of this chapter, they foreground the slippery nature of signifiers in the digital age and the political and material consequences such slippages cause for those most marginalized by digital cultures. The witch as a figure of contemporary feminism shows no signs of abating. As hashtags like #HexThePatriarchy and books like Witches, Sluts, Feminists: Conjuring the Sex Positive (Sollée, 2017), The Witches Are Coming (West, 2019) and Becoming Dangerous: Witchy Femmes, Queer Conjurers, and Magical Rebels (West and Elliott, 2019) attest, the resurrection of the witch by feminists (and their misogynistic detractors) is significant. Witches hold the dual signifier of women’s ‘ferocity’ and ‘transgression’ as well as her persecution (Sollée, 2017: 17–19). As such, the history of witches is a direct look ‘into the abyss of fear, sexist violence, and toxic masculinity that dominates feminist discourse today’ (Sollée, 2017: 22). The history of the witch is indeed steeped in resistance and revolt. Under patriarchy, witches were situated in opposition to men, or were charged with having an ‘inherent wickedness’ that translated into too much power by their detractors (Gasser, 2017; Rowlands, 2013; Sollée, 2017: 32). Those who challenged structures of patriarchy were persecuted, outlawed, put to death or punished through the strategic misrepresentation of their stories, sometimes finding themselves completely erased from history (Gasser, 2017). During the infamous Salem Witch Trials in New England, around 78 per cent of people who were accused of witchcraft and ultimately executed were women. Furthermore, simply through their associations with women found guilty of witchcraft, men and Black people who were enslaved faced their own trials and deaths (Demos, 2004; Karlsen, 1998). This indicates that people found guilty of witchcraft were those who signified opposition to patriarchal and white supremacist rule, pushing the boundaries of imposed gendered and racialized roles. With the contemporary resurgence of ‘witch culture’, what remains clear is that any resistance to hegemonic conditions is still feared. Feminism has taken up the mantle of the witch as a performance of resistance and the ‘prevalence of witch-infused messaging at this historic moment in support of gender equality reaffirmed the witch’s continued role in feminist activism’ (Sollée, 2017: 62). The figure of the witch currently offers a compelling manifestation of feminist digital performances of identity, providing clear insights into the desires and anxieties of social media users engaged in conversations around gender, equity and misogyny. As a feminist performance, the witch generates contingent, shifting and affectively constituted crowds with differing experiences and perspectives on ‘networked misogyny’ (Banet-Weiser and Miltner, 2016) that collectively form a digital community of resistance.

FEMINIST MEMES

125

Feminist witch memes are thus complex sociocultural performatives that iteratively circulate and (re)organize forms of cultural knowledge into dynamic flows of discursive resistance and feminist refusal. Memes tied to the occult position our affective experiences of laughter, anger and collective recognition as distinctly feminist tools of resistance. Such appeals to humour and the threat of a rage that can be wielded at one’s enemies are employed by digital feminists in order to connect and create resonance with others. The feminist witch memes we consider in this chapter respond to the urgency felt by media users who are surrounded by the tumultuous unfolding of social and political events. These memes offer a space for processing and meaning making. Through the use of affective tropes, the witchy memes articulate desires for resistance that publicize and remix personal feminist positionalities for social and political ends. The emotive force, the bawdy humour and the undercurrent of the violent reprisal of the witch in these memes provide a locus for feminist frustrations and offer a collective catharsis. Strong affects motivate social media users to make, circulate, amplify these memes and turn their sentiments into the grounds of political action.

Theoretical framing Our intervention into feminist media scholarship frames feminist memes as a repertoire that encourages forms of affective intimacy through their circulation, reception and meaning making. Situating ourselves in the sphere of interdisciplinary studies amid feminist performance studies, feminist media studies and the emerging field of feminist internet critique, we draw on Jac sm Kee’s contention that ‘online is always located in the materiality of the people who engage in a multiplicity of spaces’ and foreground the vital function of feminist ‘resilient networks’, which are ‘both embodied as flesh, and as discursive informational flows’ (2018: 85). Exploring ‘how users deal with the worlds they are thrown into by designers’ (Lialina, 2019: para. 1), we situate feminist witch memes as an example of counter-public worldmaking (Warner, 2002) and feminist performative assembly (Butler, 2015) that iteratively respond to contemporary political currents. Our understanding of world-making draws on Judith Butler’s premise that bodies assembled in public space function as a performative, collective gesture. In the context of protest culture, such assemblies assert ‘the right to appear’, eliciting ‘a bodily demand for a more livable set of lives’ (2015: 25). In their effort to counter dominant forms of institutional power, such assemblies also create the provisional powers to ‘bring about a new situation or to set into motion a set of effects’ (2015: 28). Butler situates assemblies as performative acts of collective expression. Feminist memes online offer a virtual form of performative assembly: they assert the right of feminist

126

MATERIALIZING DIGITAL FUTURES

voices to be heard and validated in their use of meme figures that can be shared widely and remixed into a chorus of collective voices. Further, as Aimée Morrison argues, digital practices ‘assemble publics’ that ‘are marked not just by their humour but by their bent towards the colloquial, the vernacular, and the anecdotal’ (2019: 24–5). In this way, we see the world-making of the digital witch within their production of vernacular content (de Seta, 2016) tied to ongoing feminist currents. Digital feminist practices are thus important for the ways in which they draw on seemingly ordinary practices – practices that take place under the watchful eye of the internet – with the intent of bringing people together into communities of support. However, community formation as world building has consequences. The ‘forms of knowledge’ that these memetic performances display are ‘produced through the affordances and limitations of their platform’ (Mauro-Flude, 2019: 207), while also transcending them as they gesture towards different discursive and material futures. Feminist memes thus stand as networked media events, particularly when their virality extends beyond their original contexts. This offers key insights into how the digital allows for a particular kind of feminist enactment predicated on emotion and connectivity (Rentschler, 2019). Memes can become events in their own right which echo through the ether and allow social media participants to reframe how they enact their identity and resistance in the virtual but also their lived offline spaces. Situating the performative digital assemblies of memes as media events allows further insight into their role as world-making. In this framework, memes do something, they enact discursive changes and create forms of digital kinship that expand the frames of reference and relationality. Feminist memes thus exhibit ways of ‘doing feminism’ (Rentschler and Thrift, 2015), because through their circulation, they can address diverse feminist audiences in the hopes of fostering (re)new(ed) connections that extend beyond the digital life of the memes themselves. Calling on affective and material histories of the witch within contemporary digital culture is one iteration of the current feminist movement. As such, it ‘demonstrates the shifting terrain of our movements, and the fluidity of spaces for our political acts of expression, occupation, re-territorialization, solidarity and resistance’ (Kee, 2018: 87). In particular, witchy feminist memes push against the hegemonic status quo, allowing social media participants to engage in a mimetic collapsing of the ‘personal is political’ in order to articulate affectively charged ethos of collectivity and resistance. From this framework of world-making and performative assembly, in this chapter we suggest that the affects operating at the core of meme culture can open the forms of resistance required to counter the neoliberal misogyny that currently runs rampant in many social media spaces. The paradoxical nature of simultaneous (feminist) creation and (misogynist,

FEMINIST MEMES

127

white supremacist) constraint in social media spaces (Wiens and MacDonald, 2020) speaks to Lee Humphreys’ work on the everyday life of social media, where ‘broader social values and systems … shape the human condition’ (2018: 6). Humphreys reads digital media events as both connective and contextual (2018), which MacDonald (2020) takes up to situate memes as feminist digital media practices that offer a snapshot of ordinary or everyday technologies. Memes are relatable acts of communication that often focus on the emotions that orient our everyday worlds. As such, this chapter focuses directly on the particular emotions that feminist witch memes are producing, namely humour and rage. The range of affective and political responses that the memes evoke are crucial for the further mobilization of the politics such spaces express and encourage. Our understanding of affect is grounded in Sara Ahmed’s writing on affective economies, wherein feelings circulate as a means of creating the surfaces and boundaries that both define and connect us to other people and things (2004: 8–10). To analyse the material effects of circulating affects, we take Ahmed’s affective economy alongside Diana Taylor’s notion of the repertoire, wherein performances, such as those circulating within and throughout digital media spaces, ‘function as vital acts of transfer, transmitting social knowledge, memory, and a sense of identity through reiterated’ gestures (2003: 2–3). The affect(s) tied to our social knowledge, memories and identities orient us towards other people, temporalities, spaces and places and in this way set up a performative scene that is ‘rehearsed and performed daily in the public sphere’ (2003: 3). By reading a media event as a performance scene, we may ask how the performances within this scene offer ‘conscious strategies of display’ while also indexing the occlusions found within more official discourses (2003: 28–9). One way to expand the conversation around feminist digital practices is to consider the valuable forms of embodied, affective knowledge they transfer and enable (MacDonald, forthcoming). We argue that that one way of approaching the affective and performative assemblies of feminist digital practices is to strive for a more holistic analysis of their sociocultural, political, communicative and material consequences in the world.

Methodology As memes are multi-platform, visual and textual artifacts that perform intimate forms of world-making, they offer much more than big data inferences on overarching trends and patterns can suggest (Brooker et al., 2018; Housley et al., 2018; Kim et al., 2018; Langlois et al., 2015). Memes, as repositories of visual-cultural articulations, need to be read for the meanings embedded in both their form and their circulation (de Seta, 2016). To do so, we read our focused collection of witch memes in this chapter as

128

MATERIALIZING DIGITAL FUTURES

a ‘small genre’ repertoire that feminists are using to ‘continuously interpret, negotiate, and improvise’ (de Seta, 2016: 480–1) white supremacist patriarchal culture. The memes we have chosen to analyse here are artifacts circulated within the social media spaces that we engage in as participantobserver researchers as we ‘virtually dwell’ (Wiens, forthcoming) within digital spaces in order to discover how memes as visual data are formed and gain affection and material traction through their circulation. Drawing on Haraway (2016), Wiens argues that, as a method, ‘dwelling … is like “staying with the trouble”, it’s about assuming a responsibility and relationship to the present moment to be open to new ideas and knowledges – especially if they contradict or seem antithetical to current power structures’ (forthcoming: 3). In this way, ‘practices of dwelling, although seemingly simple, matter for the ways in which they create opportunities to settle into the data and follow them through to new ends via our affective responses to them’ (forthcoming: 3). This affective data gathering informs the subsequent analyses of the memes, wherein we interrogated the affective economies that propel feminist activism and online solidarity, specifically looking to the ways that memes signify the multiple political, social and technological networks that they belong to. In this way, we use ‘tentacular’ thinking (Haraway 2016) to notice how one artifact can tendril out in many different ways towards different ends. The memes were collected from the curated news feed of our research account Feminists Do Media (@aesthetic.resistance) on Instagram. We employed a keyword search for hashtags, searching #HexThePatriarchy, #FeministWitch and #WitchesUnite and explored witch-based accounts within our network between 12 September and 12 October 2020, although it should be noted that some of the memes that we encountered during this time were produced before this period. We then shared a set of twenty collected memes in a research folder, settling on memes that clearly called forth for us the affective elements of humour or rage and gave formal attention to how they visually performed the witch repertoire. In order to narrow down the selection of memes for analysis for this chapter, we drew on data ‘glows’ (Maclure, 2013) to focus on the memes that ‘instill[ed] in us a curiosity, a fascination that impels us as researchers to follow where the data may take us’ (MacDonald and Wiens, 2019: 368). Here, we stayed with the glows, following them to see what kinds of networks they came from and which networks they might send us towards. Glowing memes were the digital feminist artifacts that signalled to us that trouble was stirring against patriarchal structures, and that impelled us to see how these memes were taken up among the platforms that they were circulating within. It is important to underscore that the memes that we have collected through our social media account could not have been found so easily in a larger dataset. This research experience can be lost or glossed over in big data’s aims to determine generalized trends from randomized data. We treat

FEMINIST MEMES

129

the memes we have encountered in our traversing and dwelling in feminist spaces of the internet as iterative performances of the ‘glitch witch’ – neither compliant nor silent, but rather bawdy, raucous and operating under the spectre and promise of powerful, collective rage-fuelled responses. As glitch witch artifacts, these memes demand that we dwell in space and pay attention to how the sociocultural, political, communicative and material factors work together to nurture feminist digital identity and community.

Hell hath no fury like a woman’s rage: Feminist remixes of patriarchal bullshit It is notable, and yet entirely understandable, that the two affective states that are largely embedded within memes that feature witches and witchrelated content are humour and rage. Much has already been said about how women’s anger and rage have entered everyday conversation since the 2016 US presidential election (Keller and Ryan, 2018), with the New Yorker going so far as to situate women’s anger-informed activism as a ‘recurring figure of American history’ (Cep, 2018: para. 3). Sara Ahmed (2017) has expertly situated this anger via a correlate figure to the witch: the ‘feminist killjoy’ who ‘kills’ patriarchal white supremacist capitalist joy through her cultural critiques. The killjoy, like women’s rage, and the figure of the witch that shadows it, amplifies critiques of sociopolitical crises from a feminist standpoint. The collective rage named by the figures of the killjoy and the witch encourages feminists to carve out digital identities from their different experiences and come together for the greater goal of feminist resistance. In this way, these connected figures offer an example of internet folklore wherein feminist critiques are ‘iteratively modified by each user … transforming the archive with unparalleled immediacy’; as such this ‘offers us a glimpse at the power of the Internet as a repository and its transcendental pull’ (MauroFlude, 2019: 207). Internet folklore helps us understand ‘how computational technologies do not only reveal new insights about post-digital culture but also transform propensities for embodied contemplation’ (Mauro-Flude, 2019: 207). Key here is how ‘the ubiquity of computational media has influenced our desires and fears, concerns and prejudices’ (Mauro-Flude, 2019: 219). Current witch memes are not the first to imbue the witch in politics. The history of witches is sometimes framed as a history of women’s resistance as outlined here. Most notable in more recent history is W.I.T.C.H., Women’s International Terrorist Conspiracy from Hell, also called Women Inspired to Tell Their Collective History (and a number of different names, changing their name to suit the issue) who in 1968 stormed the streets of New York

130

MATERIALIZING DIGITAL FUTURES

and later Chicago to ‘hex the patriarchy’, catcalling men who had made unwanted sexual moves on them, critiquing capitalism and speaking out against marital rape (McGill 2016). W.I.T.C.H employed guerrilla theatre tactics that focused on ‘capitalism and corporations as the engines driving sexism of the day’ (Sollée, 2017: 53). Their message cemented within visual culture the figure of the witch as a form of political dissidence (Sollée, 2017: 52). For instance, following Trump’s numerous claims that he was the victim of a witch hunt, a collective of ‘resistance witches’ known as The Magic Resistance perform a binding spell on Trump each month so that ‘he may fail utterly/that he may do no harm’ (Burton, 2017: para. 3). Identifying as neo-pagan, committed activists, the resistance witches explore forms of inclusive community-building and activist practices, while performatively channelling a sense of loss and grief under the Trump administration. Because the Magic Resistance has ‘its roots in internet culture’, this mode of protest has encouraged witch-activists to ‘reimagine the binding spell to suit their own needs’ through the online promotion of their practices (Burton, 2017: para. 7). Interestingly, then, both the memes and the monthly binding spell rely on the internet’s remixing abilities (Shifman 2014): the spell remixes ‘elements of different faith traditions and pantheons … [and] emphasized a pragmatic, personal approach to the occult’ (Burton, 2017: para. 6) that rely on digitally networked collaboration. The collective’s digital performances situate these histories of witch culture within the current political landscape as a form of feminist identity and activist participation (see Figure 7.1). In this contemporary meme, which offers an iteration of witches as political dissidents we see the theatricality of W.I.T.C.H.’s public interventions reasserted in the digital age. Drawing on an urgency to resist the Trump administration and expressing a collective rage built over centuries of violence against women, the meme exemplifies a form of collective mobilization by calling other witches into the digital coven at a distinct moment in time. The meme’s vernacular invites others to join in the action of binding while visually evoking distinct witch tropes. The meme from 24 February 2017 uses a vintage poster aesthetic with ‘Witches’ and ‘Mass Ritual’ written in a Victorian-era script. The witch in the image is cloaked with a crown and is hunched over the glowing orb she raises up. She casts her eyes askew off to the right of the page as if looking out for detractors or persecutors. The spectre of violence and prohibition haunt her actions. A wise but sceptical owl is peering on, while a series of fantastical ghouls emerge from the shadows behind her: both offering a rich intertextuality of wisdom and danger in equal measure. In less ornate text directly above the witch are the words, ‘We Need You’. In their positioning, they are reminiscent of the iconic poster (and its endless iterations) of Uncle Sam calling American men to join the military and fight for US freedom. All the elements together evoke both the history of witchy iconography and its shadowy existential threat

FEMINIST MEMES

131

FIGURE 7.1  Black and white image of a witch and the witch’s ghouls in the background. Image reading: ‘Witches, we need you. A spell to bind Donald Trump and all those who abet him. Mass ritual. February 24th, 2017. Midnight, Friday evening’ Designed by Kitty Lemiew, official graphic designer for The Magic Resistance.

132

MATERIALIZING DIGITAL FUTURES

to men abusing their positions of power. The image equally remixes earlier abuses of power tied to the nationalist military ethos of the US, shifting the call to arms to the realm of rebellious witches. The image is theatrical in its evocation of an old playbill and in its staging of a mass ritual, a form of performance itself. In doing so, it indexes the digital binding ritual in 2017 as a performance to be witnessed and a performative act to be engaged in. Even for those not skilled in or convinced of the results of binding spells, this action to ‘bind Donald Trump & all those who abet him’ expresses a widely shared desire to constraint the harm and destruction of his actions. It is an admission and assertion that, as writer Lindy West says in her defiance of Donald Trump, ‘fine, if you insist. This is a witch hunt. We’re witches, and we’re hunting you’ (2019). In this call lies a performance of self-identification and an invitation to community with like-minded feminists. Here then, the rage evoked by Trump’s rise to power is transfigured into an ethos of connected, concentrated rebellion. This is one key possibility offered by the digital performance of the witch at this moment in time.

Hex them with humour: Or, another way into the feminist fray In tandem with discussions of feminist rage in public discourse, recent feminist media scholarship notes the important phenomena of networked feminist humour (Rentschler and Thrift, 2015), which, unlike the alt-right’s use of lulz, ‘endeavours to shed light on sexism … exposing and criticising [it] via satire’ (Ringrose and Lawrence, 2018: 686). This humour offers space for recognition and connection via shared laughter. As Willet and Willet note, this form of ‘humour from below’ is a ‘source of empowerment, a strategy for outrage and truth telling … a means of empathetic and alliance’ (2019: 2). Take, for example, the Soulshine Tarot account on Twitter, Facebook and Instagram, run by a self-described ‘tarot-reading astrologycharting whiskey-drinking banjo-picking art-making foul-mouthed witch aunt’. On 19 November 2019, she tweeted: ‘them: every girl is into astrology and witchy shit now. basic. trendy. *eyeroll*. me: maybe it’s basic and trendy. or maybe it’s a global feminist awakening wherein we collectively return to our true nature before we overthrow the patriarchy and devour your very soul, KEVIN’ (see Figure 7.2). This tweet, which circulated across various social media platforms, addresses several strands of witchy digital performances. It begins by reciting a common critique that current evocations of the witch are a laughable trend in popular culture. The ‘me’ of the tweet, evoking a strong feminist narrator, disputes the critique and asserts the figure of the witch currently

FEMINIST MEMES

133

FIGURE 7.2  Tweet by soulshine_tarot – Twitter account. Image reading: ‘them: every girl is into astrology and witchy shit now. basic. trendy. *eyeroll*. me: maybe it’s basic and trendy. or maybe it’s a global feminist awakening wherein we collectively return to our true nature before we overthrow the patriarchy and devour your very soul, KEVIN’.

trending is in fact a response to the patriarchal norms of culture. Within this vignette-as-tweet, the humour of the dialogue indexes an undercurrent of not-so-subtle rage. What the ‘me’ articulates is a global feminist awakening that seeks to overthrow the dominant order and threatens to eat the soul of the generic every-male speaker, here named Kevin. The mix of awakened resistance with the possibility of being a soul eater is where the tweet’s performance pokes fun at the cultural fear of angry and/or powerful women, most likely directly addressing a feminist audience that is already well aware of these tropes. Through the generic language of them and me, the meme invites a form of identification with the theatrical scene wherein the intended feminist audience gestures towards the threat of a collective of killjoys overthrowing the patriarchy by consuming unrepenting misogynist souls with no chance for redemption. The tweet’s choice to humorously ‘call out’ Kevin incorporates an antiwhite supremacist position into its critique. Here, Kevin stands in as a male version of ‘Karen’, a Gen Z and millennial shorthand for someone who symbolizes the dangerous practices of white women exerting their privilege over Black people and other marginalized people by appealing to authorities (store managers, police, government) over perceived threats to their security and comfort. Including Kevin at the end of the tweet, Soulshine Tarot provides a comic beat for the joke and also indicates that feminist resistance necessarily must resist white supremacist practices in everyday life. While the tone is comedic, the tweet’s position of prominence (it is currently pinned at the top of the account’s Twitter profile page) connotes it functions as a digital identity performance. Here this figure of the rebellious

134

MATERIALIZING DIGITAL FUTURES

witchy aunt who swears and drinks whiskey and is layered with a political position tied to anti-racist and anti-sexist intentions. The coding of feminist rage via the humorous imagined dialogue mixes intersectional feminist politics with a playful performance of the self, calling others to dispel the haters delegitimizing witchiness as a trend, and join the feminist campaign it contains.

#HexthePatrirachy: A twenty-first-century digital witches brew What memes offer within the context of the occult are ‘new adhoc feminist publics and ways of knowing’ as well as ‘new forms of communication, community, and consciousness raising’ (Ringrose and Lawrence, 2018: 687–90). This is certainly the case with the feminist move to transform the notion of a witch hunt from pejorative threat to empowered resistance. Humour and rage allow the witch to iteratively move across platforms, memes and hashtags, with each repost asserting and performing the emotional labour of feminist identities within a moment of great change. Affective states are ‘often contagious’ and importantly, ‘carry culturally imbued meanings across porous borders’ (Willet and Willet, 2019: 6). As the examples in this chapter suggest, the witch meme is a way of doing feminisms that is catchy and catching, imbuing many sites within the current cultural moment with a meaningful assertion of frustration, rebelliousness and collective cackling. Feminist memes offer scholars a historical repertoire of collaborative resistance. As Kee’s (2018) formative work ‘Imagining a Feminist Internet’ suggests, these kinds of community-building practices contribute to the formation of our feminist identities – an identity that spans our on- and offline worlds. As part of the feminist internet, witch memes perform a ‘double ontology’ wherein they are ‘simultaneously operating as artworks … and, crucially, as concrete interventions that start, below the radar of official culture’ (cited in Mauro-Flude, 2019: 207). Significantly, the material outcomes of memes’ digital communicative, political and cultural interventions offer action-oriented forms of counter-public world-making (Warner, 2002) that are responsive to the current moment. For instance, in October 2020, in a digital performance of personal identity and community building, the illustrator @crucifix.vi posted a drawing of a coven of witches sending up a hex against all fascists in a simple visual black and white aesthetic (see ­Figure 7.3). As part of the #inktober challenge on social media that asked social media participants to simply draw a picture in ink for every day of October, the image became widely circulated in feminist and occult circles, prompting @crucifix.vi to sell the print, with all proceeds

FEMINIST MEMES

135

FIGURE 7.3  Image posted on Instagram by illustrator @crucifix.vi. Three witches with their arms above their heads casting a spell to ‘hex all fascists’.

going to National Bailout, a Black-led, Black-centred abolitionist collective (nationalbailout.org). Similarly, in the weeks after the brutal murder of George Floyd became a catalyst for another wave of anti-Black racism protests, @thehoodwitch posted this illustration (see ­Figure 7.4) by @bluecollards who sold prints of it to raise funds for Black-trans youth organizations in her area as an act of allyship. Alongside the illustration, @thehoodwitch wrote: ‘Today I felt called to meditate on the justice card. Justice’s energy is about sticking to your guns, not being influenced by persuasive talk. Whenever we begin to feel emotional tensions building up inside, I’ve found that meditating on justice cards scales helps to remind us to regain equilibrium. Know Justice, Know peace’ (see ­Figure 7.5). The illustration on its own is deeply

136

MATERIALIZING DIGITAL FUTURES

FIGURE 7.4  Image posted on Instagram on the @thehoodwitch and @bluecollards accounts. A woman of colour’s manicured hand with broken chains at the wrist holding two tarot cards: One card on the left says ‘No Justice’ with an image of scales on a decorative floral and yellow background; the other card on the right says ‘No Peace’ with a tower broken in two parts, one on top of the other, with flames escaping from the top and the windows on a blue background https:// www.instagram.com/p/CBOqG73JgSp/.

evocative as it brings together the practice of tarot, with a femme presenting hand that has broken out of chains. In relationship to the accompanying text, a message of intersectional feminism is made clear. In it, the oftentimes white-dominant aspects of witch culture are decentred while Black women’s bodies and reflections on justice and peace are recentred at a key moment within the Black Lives Matter movement. That the cards read ‘No Justice’, ‘No Peace’ offer a direct promise that protest and rebellious refusal will continue as it always has until there is justice for those most

FEMINIST MEMES

137

FIGURE 7.5  A post from @thehoodwitch account that accompanies the image in Figure 7.4.

marginalized by a white supremacist, capitalist, heteropatriarchal order. We close with these two examples because it brings home a key point of this chapter: while feminist witchy memes may catalyse the creation of digital communities, prompting journeys of feminist self-discovery and coalition building, they can also be used to actively promote anti-racist, ableist, trans and homophobic activism. These are not just illustrations but calls to action that bring witchy discourses into dialogue with activist movements, where perhaps they have actually been all along. In our work as media scholars, we must recognize and credit online actions that continue to extend particularly intersectional feminist frameworks within digital activism. If performative assemblies truly are oriented towards the demands for more liveable lives, our work needs to amplify when, why and how this occurs within the context of digital performances of self and the

138

MATERIALIZING DIGITAL FUTURES

vernacular’s that make it possible. Digital feminist memes are an important site for further consideration, both for what feminists can offer as well as the exclusions that can occur under their fourth-wave feminist banner. Our focus on humour and rage highlights how emotions are not just psychological states, but ‘social and cultural practices’ that determine ‘how we come to be invested in social norms (Ahmed, 2004: 9–12). Within feminist hashtag movements, humour and rage are only two of the dozens of emotions and affects that instigate the performative assembly of digital and physical crowds. The fascination we see right now with all things witchy is not new – it is premised on centuries of revolution and resistance from people who have been marginalized by society. Grappling with the ‘why’ and ‘how’ of such large-scale assembly can help to create the blueprints for continued resistance against the white supremacist capitalist heteropatriarchy and for the courage to bind it, curse it and banish it.

References Ahmed, S. (2004), The Cultural Politics of Emotion, UK: Routledge. Banet-Weiser, S. and K. M. Miltner (2016), ‘MasculinitySoFragile: Culture, Structure, and Networked Misogyny’, Feminist Media Studies, 16 (1): 171–4. Brooker, P., J. Barnett, J. Vines, S. Lawson, T. Feltwell, K. Long and G. Wood (2018), ‘Researching with Twitter Timeline Data: A Demonstration via ‘Everyday’ Socio-Political Talk around Welfare Provision’, Big Data & Society, 5 (1): 1–13. Burton, T. I. (2017), ‘Each Month, Thousands of Witches Cast a Spell Against Donald Trump’, Vox, 30 October. Butler, J. (2015), Notes Toward a Performative Theory of Assembly, Cambridge, MA: Harvard University Press. CBC News (2019), ‘Trump Complains about ‘Witch Hunts’ to Oil and Gas Conference Crowd’, CBC News, 23 October. Cep, C. (2018), ‘The Perils and Possibilities of Anger’, New Yorker, 8 October. Demos, J. P. (2004), Entertaining Satan: Witchcraft and the Culture of Early New England, Toronto: Oxford University Press. De Seta, G. (2016), ‘Neither Meme Nor Viral: The Circulationist Semiotics of Vernacular Content’, Lexia, doi: 10.4399/978882550315926. Gasser, E. (2017), Vexed with Devils: Manhood and Witchcraft in Old and New England, New York: New York University Press. Haraway, D. J. (2016), Staying with the Trouble: Making Kin in the Chthulucene, Durham, NC: Duke University Press. Housley, W. et al. (2018), ‘Interaction and Transformation on Social Media: The Case of Twitter Campaigns’, Social Media & Society, 4 (1): 1–12. Humphreys, L. (2018), The Qualified Self: Social Media and the Accounting of Everyday Life, Cambridge, MA: MIT Press. Karlsen, C. F. (1998), The Devil in the Shape of a Woman: Witchcraft in Colonial New England, New York: W.W. Norton.

FEMINIST MEMES

139

Kee, Jac sm (2018), ‘Imagine a Feminist Internet’, Development, 60: 83–9. Keller, J. and M. E. Ryan (2018), Emergent Feminisms: Complicating a Postfeminist Media Culture, New York: Routledge. Kim, H., S. Mo Jang, Sei-Hill Kim and A. Wan (2018) ‘Evaluating Sampling Methods for Content Analysis of Twitter Data’, Social Media & Society, 4 (2): 1–12. Langlois, G., G. Elmer and J. Redden (eds) (2015), Compromised Data: From Social Media to Big Data, London: Bloomsbury. Lialina, O. (2019), ‘Once Again, the Doorknob’, Media Theory, 3 (1). Available online: http://mediatheoryjournal.org/olia-lialina-once-again-the-doorknob/. MacDonald, S. (2020), ‘What Do You (Really) Meme? Pandemic Memes as Social Political Repositories’, Leisure Sciences, doi: 10.1080/01490400.2020.1773995. MacDonald, S. (Forthcoming), ‘ “Vital Acts of Transfer”: Affective Economies and Embodied Knowledge in #MeToo’, in J. Rudakoff (ed.), Performing #MeToo: How Not to Look Away, UK: Intellect Books. MacDonald, S. and B. I. Wiens (2019). ‘Mobilizing the “Multimangle”: Why New Materialist Research Methods in Public Participatory Art Matter’, Leisure Sciences, 41 (5): 366–84. MacLure, M. (2013), ‘Researching without Representation? Language and Materiality in Post-Qualitative Methodology’, International Journal of Qualitative Studies in Education, 26 (6): 658–67. McGill, M. (2016), ‘Wicked W.I.T.C.H: The 60s Feminist Protestors Who Hexed Patriarchy’, Vice Media, 28 October. Available at: https://www.vice.com/en_us/ article/43gd8p/wicked-witch-60s-feminist-protestors-hexed-patriarchy (accessed 30 May 2020). Mauro-Flude, N. (2019), ‘Performing the Internet: Post Internet Folklore’, in S. R. Wong, H. Li and M. Chou (eds), Digital Humanities and Scholarly Research Trends in the Asia-Pacific, 200–27, Hershey, PA: IGI Global. McLaren, L. (2017), ‘Me Too: It’s Not just Hollywood, It’s Canada’, MacLean’s, 18 October. Morrison, A. (2019), ‘Laughing at Injustice: #DistractinglySexy and #StayMadAbby as Counternarratives’, in D. C. Parry et al., Digital Dilemmas: Transforming Gender Identities and Power Relations in Everyday Life, 23–52, New York: Palgrave Macmillan. North, A. (2018), ‘The #MeToo Movement and Its Evolution, Explained’, Vox, 11 October. Rentschler, C. A. (2019), ‘Making Culture and Doing Feminism’, in T. Oren and A. Press (eds), Routledge International Handbook of Contemporary Feminism, 127–47, UK: Routledge. Rentschler, C. A. and S. C. Thrift (2015), ‘Doing Feminism: Event, Archive, Techne’, Feminist Theory, 16 (3): 239–49. Ringrose, J. and E. Lawrence (2018), ‘Remixing Misandry, Manspreading, and Dick Pics: Networked Feminist Humour on Tumblr’, Feminist Media Studies, 18 (4): 686–704. Shifman, L. (2014). Memes in Digital Culture, Cambridge, MA: MIT Press. Sollée, K. J. (2017), Witches, Sluts, Feminists: Conjuring the Sex Positive, Los Angeles, CA: Three L Media.

140

MATERIALIZING DIGITAL FUTURES

Taylor, D. (2003), The Archive and the Repertoire: Performing Cultural Memory in the Americas, Durham, NC: Duke University Press. Warner, M. (2002), ‘Publics and Counterpublics’, Public Culture, 14 (1): 49–90. West, L. (2017), ‘Yes, This Is a Witch Hunt. I’m a Witch and I’m Hunting You’, The New York Times, 17 October. West, L. (2019), The Witches Are Coming, New York: Hachette Books. Wiens, B. I. and S. MacDonald (2020), ‘Feminist Futures: #MeToo’s Possibilities as Poesies, Techné, and Pharmakon’, Feminist Media Studies, doi:10.1080/146807 77.2020.1770312. Wiens, B. I. (Forthcoming), ‘Virtual Dwelling: Orientations to Feminist Digital Communities’, in S. MacDonald, M. Radzikowska, M. MacArthur and B. I. Wiens (eds), Networked Feminisms: Activist Assemblies and Digital Practices, Lexington, KY: Lexington Press. Willet, C. and J. Willet (2019), Uproarious: How Feminists and Other Subversive Comics Speak Truth, Minneapolis: University of Minnesota Press.

8 The infinite portrait: A case of post-human authorship Andrew McIntyre

Introduction In October 2018, Christie’s gallery in London made headlines as the first auction house in history to exhibit and sell a piece of art attributed to an artificial intelligence (AI): the Portrait d’Edmond de Belamy produced by Paris-based art collective Obvious, shown in Figure 8.1 (Cohn, 2018). While the piece gained widespread notoriety for demonstrating the potential of such programs to generate works that are near-indistinguishable from human art, many practitioners and theorists working in the field of so-called creative AI have dismissed the work as a mere experiment or novelty rather than ‘true’ AI art.1 As the program was selectively trained by human programmers some would argue that it is little more than a precursor to the messianic ‘machine artist’ yet to come while others have argued against the very notion of a ‘machine artist’. Notably, computer scientist and AI artist Ahmed Elgammal contends that art-making is an inherently social activity and though a machine may be capable of creating masterpieces without human input it exists in an ‘isolated creative space that lacks social context’ unlike human artists who ‘create to tell stories and make sense of the world’ (Elgammal, 2019). For Elgammal and others, then, today’s creative AI constitutes a nascent medium and the AI program will eventually join the camera, pen and paintbrush as just another creative tool for the human artist to wield.2

142

MATERIALIZING DIGITAL FUTURES

FIGURE 8.1  Portrait d’Edmond de Belamy by Obvious (2018). Used with permission of Obvious.

Despite the media attention afforded to it, the Portrait d’Edmond de Belamy is by no means the first of its kind nor are the questions it raises about authorship entirely novel. Indeed, the prospect of artificially constructed artworks may be traced back to early computer and algorithmic art, such as the generative music of Brian Eno and Harold Cohen’s painting bot AARON, and could be further linked to the expansive tradition of artistic experimentation with agency, rules and randomness.3 These concepts and their relationship to authorial intention and originality have been explored, for example, in the expressionist action painting of Jackson Pollock, the sculptures and cut-up techniques of Dadaism and the mathematically constrained writing of members of the Oulipo (Ouvroir de littérature potentielle) including Raymond Queneau and Italo Calvino.

THE INFINITE PORTRAIT

143

In light of this century-long history of deconstructing the concept of the artist or author, I contend that the arguments against the Portrait d’Edmond de Belamy present an oversimplified view of artistic production. Though recontextualized for the twenty-first century, these positions maintain a more conservative, sentimental notion of authorship that echoes the humanist ideology of the Romantic period wherein authors such as Lord Byron and Percy Bysshe Shelley were idolized as ‘solitary geniuses’. Such a theory of authorship does not simply attribute works of art solely to human artists but envisions the human as a transcendental individual that is fully autonomous in their expression. For a machine to be counted an author it must reach the mythical status of the genius where the human being already awaits or be relegated to the lowly status of a tool used to fulfil a predetermined idea spawned from the human imagination.4 To continue promoting such a theory in the case of AI-generated images misrepresents the complexity of the human–machine relationship and misunderstands the reality of the imagemaking process as something entirely under human dominion. Further, it wrongfully presents works such as the Portrait d’Edmond de Belamy as either human-made artefacts to be consumed and critiqued like any other or as anomalous curiosities to be ignored and, in doing so, obscures their significance as post-human creations. In the following, I use the example of a lesser-known work, the digital installation Memories of Passersby I created by visual artist Mario Klingemann, to discuss the notion of post-human authorship. Drawing on the work of philosopher and media theorist Vilém Flusser, particularly his theories on technical images and photography, I show how AI opposes this humanist perspective on authorship by influencing the aesthetic experience of such works. In doing so, I hope to show that by ignoring the post-human authorship of AI art we may be marching blindly towards a meaningless and deceptive ‘utopia’.

Memories of Passersby I The installation itself, shown in Figure 8.2, consists of a wooden cabinet housing the AI hardware that is connected to two wall-mounted screens displaying digital portraits (one male and one female) generated through the use of a machine learning algorithm trained to emulate the styles of seventeenth- to nineteenth-century portraiture. However, the faces of these non-existent subjects are constantly changing with a continuous and almost fluidic motion as the AI, using the previous image as a reference, repeatedly creates new images to replace the old every instant. Over time, the figure on-screen might morph into a grotesque caricature, or sprout a second head or third eye, or their facial features might very well deteriorate entirely into a formless blur. As no one image or sequence is stored or repeated, there is

144

MATERIALIZING DIGITAL FUTURES

FIGURE 8.2  Memories of Passersby I by Klingemann (2018). Used with permission of Onkaos.

no fixed or finalized piece; there is only an ever-changing stream of surreal, dreamlike faces that the AI will endlessly generate from the moment it is switched on in a process that cannot be strictly defined as entirely random or entirely pre-programmed. The AI system used in the case of Memories of Passersby I is a type of artificial neural network known as a generative adversarial network (GAN). Such neural networks are characterized by their ability to ‘learn’ certain tasks that they have not been directly programmed to do with minimal assistance from a human programmer. As such, these systems have found use in a variety of different contexts, particularly in image recognition and reproduction software. In what is perhaps the system’s most famous and most controversial application, GANs have been used to produce artificial footage of celebrities and public figures that is near indistinguishable from

THE INFINITE PORTRAIT

145

real footage. These so-called deepfakes are artificially constructed by GANs ‘learning’ to identify and reproduce certain identifiable characteristics (e.g. facial features and muscle movements) from a database of footage and photographs. In the artworld, however, numerous artists and programmers have been experimenting with the same image-making GANs as a means of artificially creating ‘original’ works from a database of human-created works.5 While the Portrait d’Edmond de Belamy stands as a clear example of artificial portraiture, artists such as Robbie Barrat have extended this technique to other artistic styles including landscapes and nudes. Whether or not these images are considered accurate imitations, what is clear is that the GAN is attempting to construct an image exhibiting the recurrent visual elements of the database images, such as the figures of human bodies or the features of a countryside. It could be argued that these GAN-produced images are not original at all but are simply amalgamations or the mathematical ‘average’ of the database images. However, in examining the program of the GAN more thoroughly it becomes clear that this is not precisely the case. In simple terms, a GAN consists of two artificial neural networks: a ‘generator’ and a ‘discriminator’ (see diagram in Figure 8.3). The generator produces a synthetic dataset (e.g. an artificial image) to submit to the discriminator which in turn compares the synthetic data to the authentic data (e.g. a database of ‘real’ images). The process has been likened to an art critic (the discriminator) judging the work of an art forger (the generator); however, as the two compete against one another over time they improve in their function. The generator ‘learns’ to produce synthetic images that more closely resemble the authentic ones, while the discriminator ‘learns’ to more accurately differentiate synthetic images from authentic images. Crucially, the generator has no direct access to the authentic database. Instead, it creates synthetic images from an unspecified data input (shown

FIGURE 8.3  An approximation of a GAN program.

146

MATERIALIZING DIGITAL FUTURES

in Figure 8.3 as the random data input) and relies solely on the reactions of the discriminator to determine whether these synthetic images are accurate or inaccurate imitations (Creswell, 2018: 54). One might then think of this process as the GAN ‘interpreting’ the data input in terms it understands, and so it could be argued that the images are visualizations of this input data rather than the cannibalized bits of old art reassembled. With the writhing figures of Memories of Passersby I, the data input is ‘interpreted’ by the generator as a synthetic image that is then visualized on one of the two screens. If the synthetic image adheres more closely to the features of the authentic database of images, the distinct form of a person may appear on screen that echoes the style of seventeenth- to nineteenthcentury portraiture. However, should this synthetic image deviate, all that would be observed on screen is a digital sludge with no discernible forms to be found. As the GAN continues to run, forms may then appear to emerge from this sludge, move, change shape and colour, or dissolve entirely. This is, however, only a superficial observation. If Memories of Passersby I was paused at different intervals and the static images scrutinized, it would become clear that these digital portraits are not surfaces of changing forms but a series of individual visualizations of data continually being created and destroyed with every iteration of the GAN code. The differences between images may seem minute because each iteration is used to inform the next, but every image is truly a unique visualization of new data. The forms that appear to the human eye are simply data fitted to a mould that is shaped by the governing mathematical equations of the program.

Chaotic images For a rule-based system governed by a set of deterministic mathematical equations, that are themselves abstractions of concrete reality, it would seem logical to conclude that the behaviour of such a system is entirely predictable to the writer of those equations: ‘understand the laws and you understand the universe’ (Gleick, 1988: 12). Viewed through the lens of authorship theory, this Newtonian mantra implies that the author of a computer program, such as a GAN, determines the behaviour of said program and is, therefore, author of its outputs (e.g. the images it produces). From such a position that establishes an authorial hierarchy at the top of which stands the human, one might very well attribute any and all paintings produced by the painting bot AARON to programmer Harold Cohen, even those produced long after the programmer’s death. In effect, creating a kind of creative immortality. This hierarchical position certainly appears to be the case with simple machines, such as cameras, that adhere to a program with a strictly linear relationship between input and output (i.e. were I to direct the camera’s

THE INFINITE PORTRAIT

147

lens towards a particular object and press the shutter button, I can predict that the resulting image will be of said object). However, the same cannot be said of the highly complex, non-linear GAN program, the behaviour of which is analogous to a ‘chaotic’ mathematical system where ‘the future of a chaotic system is indeterminable even though the system is deterministic’ (Hilburn, 2000: 37). This unpredictability in chaotic systems is the result of a sensitive dependence on initial conditions, the so-called butterfly effect described by meteorologist Edward Lorenz (Lorenz, 1963). While simulating the behaviour of weather systems over a long period of time using standard non-linear equations, Lorenz noted: ‘a small change in initial conditions leads to grossly different long-term behaviour of the system, and we cannot in practice predict that long-term behaviour in detail’ (Hilburn, 2000: 37). This kind of chaotic behaviour, I argue, can also be observed in the infinite portraiture of Memories of Passersby I as, just as the flapping of a butterfly’s wings in Brazil may set off a tornado in Texas, minute differences in the system’s data input will produce vastly different images ensuring that every time the installation is switched on it is truly unique and entirely unpredictable even to the programmer. Furthermore, as each image is fed back into the GAN to inform the production of the next, these minute changes escalate into larger changes over time. For example, the random appearance of a small blemish on the GANgenerated face in the first iteration may then influence the next causing the facial features to distort, if only slightly, which in turn leads to more pronounced distortions in the following images. Over time, a singular blemish may very well result in a cascading effect that eventually causes the face to take on a completely different form or dissolve entirely. Of course, this is not truly the result of a ‘blemish’ but of a numerical value that is fed back into the system’s governing equations and thus changes its behaviour over time. This is not to say that the GAN goes ‘off script’ or defies its programming by any act of agency but that the non-linearity inherent in the system’s equations means that ‘the act of playing the game has a way of changing the rules’ (Gleick, 1988: 24). The GAN program itself changes ever so slightly with each iteration such that, after a period of time, it may produce visual forms that are both accounted for within the initial program written by Klingemann but that are also, crucially, entirely unpredictable or indeterminable at the moment the program was written. To say that the GAN’s behaviour is unpredictable does not, however, imply that it will eventually degrade into an entirely randomized or unstable system. Chaos is not the same as disorder, it lies at the transition point between order and disorder. To visualize this, consider Lorenz’s ‘strange attractor’ shown in Figure 8.4. It is a dimensionless plot of a simple chaotic system’s behaviour as it spirals further and further from its origin and has come to represent the order and disorder of chaos itself, as historian of science James Gleick comments:

148

MATERIALIZING DIGITAL FUTURES

FIGURE 8.4  Three-dimensional plot of Edward Lorenz’s ‘strange attractor’.

It always stayed within certain bounds, never running off the page but never repeating itself either. It traced a strange, distinctive shape, a kind of double spiral in three dimensions, like a butterfly with its two wings. The shapes signalled pure disorder, since no point or pattern of points ever recurred. Yet it also signalled a new kind of order. (Gleick, 1988: 30) With this image in mind, compare it to the changing portraiture of Memories of Passersby I, shown in Figure 8.5 (Sotheby’s, 2019). Every image shown is entirely unique, in terms of both visual form and numerical data, but each still adheres to certain conditions or constraints. For example, facial features may bulge, or shrink, or become distorted but are always accounted for and arranged in a relatively consistent pattern or position to one another (e.g. a mouth or eye does not spawn at the perimeter of the image while the nose remains in the centre). The behaviour of the system is, therefore, neither pre-programmed (ordered) nor is it entirely random (disordered) but instead exists at the boundary between the two. Therefore, the hierarchical or Newtonian position on authorship that would declare Klingemann as the sole author of Memories of Passersby I because he wrote the program and thus determines its behaviour does not hold up under closer inspection. However, to disregard Klingemann entirely and attribute sole authorship to the GAN itself is equally misguided as such a position fundamentally misunderstands that which makes works of art meaningful: the gesture itself.

The gesture of artificial intelligence As with any and all instances of art-making, Memories of Passersby I did not emerge naturally from the pure randomness of the universe but was

THE INFINITE PORTRAIT

149

FIGURE 8.5  Successive screenshots displaying the changing imagery produced by the GAN used for Memories of Passersby I. Used with permission of Onkaos.

150

MATERIALIZING DIGITAL FUTURES

produced through a human action or gesture; what Flusser defines as ‘a movement of the body or of a tool attached with the body, for which there is no satisfactory causal explanation’ for its meaning (Flusser, 2014: 3). In other words, a gesture is characterized by its symbolic meaning rather than its instrumental purpose. Such gestures are encountered in every aspect of daily life from immediate bodily movements, such as hand signs, to more prolonged ‘motions’ such as the meticulous crafting of a work of art. While bodily gestures are fleeting and their meaning is communicated instantly, a work of art might be considered a ‘frozen gesture’, the meaning of which is preserved in the medium. Gestures stand apart from other incidental motions that are purely reactionary (a knee jerk, e.g.) because they are motions aimed towards the realization of a future state (e.g. a work of art) that articulates something and are, therefore, symbolic of a ‘state of mind’. To say that a gesture is symbolic of a ‘state of mind’ is not, however, to say that anything of the artist’s biography or personality may be deciphered from the work itself, nor is it to imply that the work articulates a particular intended message, as other intentionalist theories of authorship claim.6 While intentions may be broadly defined as ‘attitudes we take toward ends and means to those ends’ and artists certainly commit gestures with intent, it is misguided to say that the meaning of a gesture derives from this intent as to do so ignores the gesture itself in favour of a causal explanation (Livingston, 2005: 208). To analyse Memories of Passersby I, for example, causally is to isolate a ‘catalog of moving bodies involved’ and establish a chain of events (e.g. writing the GAN program, selecting the training set, designing and constructing the installation) that eventually points back to Klingemann’s initial intention (Flusser, 2014: 62). Every motion thereafter, the concrete gesture itself, is then seen as subservient to this invisible intention and works solely towards its fulfilment. Though this may seem a logical suggestion from a humanist perspective, it is reminiscent of Romantic authorship wherein the human is viewed as a transcendent individual whose thoughts and ideas, especially artistic ones, are seen as ‘strokes of genius’ that occur in isolation beyond the influence of external forces. In opposition to this humanist notion of intention driving gestures, consider the gesture of writing that Pierre Macherey equates to an inexact labour in which the writer neither commands nor wields the words but, instead, plays with a ‘set of finite combinations’ available (Macherey, 2006: 45). As the writer conducts the gesture, new combinations reveal themselves and others disappear such that the final work is determined at every stage of its development and, as Macherey argues, ‘the author cannot deduce, from an initial intention, the means by which that intention shall be realised’ (Macherey, 2006: 27). It is only through the completion of the concrete gesture itself that the final form may be realized and there exists no alternative or idealized version that might adhere to the initial intention. For Roland Barthes and other anti-intentionalist critics, such an admission

THE INFINITE PORTRAIT

151

would seem to support the notion that a work is a ‘multi-dimensional space’ to be endlessly interpreted by the reader thus spelling the death of the author (Barthes, 1987: 146).7 However, to replace the author with the reader as the origin of meaning is to simply suggest yet another causal explanation, this time rooted in the experiences of the reader and the context in which they live. Rather than look to abstract and unknowable intentions or interpretations orbiting the gesture in order to explain its meaning, the gesture itself should be analysed in terms of its ‘gestalt’ (Flusser, 2014: 63). To explain this concept, Flusser uses the example of a painter and asserts that the human and the non-human (the GAN in the case of Memories of Passersby I) are parts of the overall motion of the gesture, stating: We are not seeing some sort of mysterious merging of painter and material in a process from which the painting emerges as a synthesis but rather the gesture of painting. ‘Painter’ and ‘his material’ are words we use to explain the gesture and not the other way around … there is a concrete gesture of painting, and in it, painter and brush are realized. (Flusser, 2014: 67) In terms of its gestalt, Memories of Passersby I can be understood as a gesture that does not end with Klingemann’s construction of the GAN but one that continues on forever as the GAN produces images ad infinitum. Unlike a traditional painting, Memories of Passersby I is not so much a ‘frozen’ gesture but a gesture of endless, chaotic motion and within this chaos ‘Klingemann’ and ‘the GAN’ are realized as different phases of motion. In allowing the machine to create of its own accord, human motion is being ‘replaced’ or diminished within the gestalt and so the human is further and further removed from the production process, from painter to producer of painters to producer of producers and so on until the human’s only meaningful movement is the touch of a button marked ‘on’. Where previously works of art might come to symbolize the ‘state of mind’ of the human artist behind their production, the case is not so clear with Memories of Passersby I and other AI art. In order to understand precisely what these gestures symbolize, the analysis must move towards the portraits themselves for the ‘meaning of the gesture, the painting to be painted, is the future of the gesture’, that is, the technical images produced (Flusser, 2014: 64).

The game at play While the features of the shifting human forms observed (if momentarily) in Memories of Passersby I certainly resemble the portraiture from which they derive, to propose that it be critiqued in the same manner as a product of

152

MATERIALIZING DIGITAL FUTURES

human imagination and skill is to misunderstand how such technical images are created and what they mean. Those portraits that Klingemann has plucked from across the centuries were images directly crafted by humans who observed phenomena in the concrete world (e.g. the portrait subject) and sought to communicate their significance by producing representative imagery. These are, therefore, not simply near-accurate translations of ‘something “out there” in space and time’ – be it an object or landscape view or person – but are 2D compositions of semiotic signs arranged upon surfaces so that others may receive information through decoding (Flusser, 2012: 8). In order to decode such works, and therefore ascertain their significance, Flusser asserts that it is necessary to speculate on the ‘encoding that took place ‘in the head’ of the painter themselves’ (i.e. to determine the painter’s intention or what might be romantically referred to as their ‘worldview’) (Flusser, 2012: 16). This is not the case, however, with technical images such as photographs, film and the GAN-generated portraits of Memories of Passersby I. With the integration of automatic machines or apparatuses (e.g. cameras or computers or GANs) into image-making practices, the human being has become further distanced from this process of encoding. No longer is it confined to the ‘head’ of the human artist as they witness and interpret reality but rather it occurs within the apparatus itself as it conducts the procedure it was programmed to complete, be it mechanical or digital in nature. As apparatuses are themselves products of scientific principles and mathematical laws that are already interpretations of reality, the technical images they produce signify these abstract concepts rather than phenomena observed in the concrete world (Flusser, 2012: 14). Therefore, technical images such as the Portrait d’Edmond de Belamy and Memories of Passersby I cannot be decoded through a superficial analysis of the content (i.e. the phenomena that appear to be represented). Instead, we must look beyond the shifting human forms that materialize on screen and towards their production according to the program of the apparatus. In the case of Memories of Passersby I, this is the GAN itself. In all instances, the fundamental nature of an apparatus program is the same: it is an autonomous ‘combinatory game’ that will produce different outputs for different inputs according to the rules that govern the game (i.e. the mathematical process) (Flusser, 2012: 32). It is, therefore, only the complexity of the game’s rules that separates a GAN program from the program of a camera. With simple apparatuses that adhere to a linear input– output relationship (e.g. cameras), the human operator (e.g. photographer) is able to govern what combinations or possibilities are made concrete. These operators are seen to ‘play’ with the apparatus, cycling through different combinations to arrive at their desired outcome and no other. Imagine the case of a photographer moving around their subject, clicking away and selecting their desired image from the multitude of possibilities

THE INFINITE PORTRAIT

153

before discarding the rest. While the camera presents numerous possible outcomes, the operator exerts limitation on what outcomes may be realized; this limitation or dominance over the gesture of photography is what establishes authorship. However, were the camera absentmindedly left to record any and all things that passed it by, making concrete its multitude of possibilities, the matter may be quite different and comparable to the case of Memories of Passersby I.8 The human eye is limited by its speed of response and cannot properly view the individual images produced by the GAN, which occur at lightspeed, and instead can only see visual forms coming into being, morphing and fading away. However, looking past the visual features, the digital portraits displayed on-screen in Memories of Passersby I are revealed to be nothing more than different outputs of the program. They are different possibilities or combinations that are all accounted for within the rules of the apparatus program. Previously, I have discussed how the GAN of Memories of Passersby I produces an endless stream of images and that within this gesture the influence of the human operator shrinks to the point of insignificance, therefore, ‘the possibility of stopping the apparatus at the desired consequence is lost. The program becomes independent of human intention. It becomes autonomous and rolls on until every coincidence has been realized, even those human beings originally wanted explicitly to avoid’ (Flusser, 2011: 73–4). With the operator so diminished, they cannot exert dominance or limit the possible outcomes to be realized, so once the GAN is switched on it will run forever trying to exhaust its inexhaustible program by producing an infinite number of images. Both the camera and the GAN are ‘black boxes’ from which different possibilities are seen to emerge. With the camera (and other linear programs) the viewer may only witness a finite number of possibilities as limited or curated by the operator who might be seen as a kind of ‘middleman’ or filter between the apparatus and viewer, stemming the flow of images. With the GAN, however, there is no such filter and so, rather than simply see a selection of possibilities, the viewer is forced to look directly into the ‘black box’ and observe the infinite number of possibilities unbounded by the limitations of the human. To witness these moving images, then, is to witness the ‘game at play’ and one might infer its rules; what is accounted for within the program and what is not. In short, what is observed and understood is the very function of the GAN program itself which is to create from nothing: ‘particles into two-dimensional images, to rise from no dimensions to two dimensions, from the abyss of intervals to the surface, from the most abstract into the apparently concrete’ in a specific form: a portrait (Flusser, 2011: 21). To view the endless imagery of Memories of Passersby I is to be confronted with what Andreas Broeckmann describes as the ‘machinic aesthetic’ which is ‘an experience in the face of art that hinges on machine-based process that are beyond human control’ (Broeckmann, 2007: 205). While Broeckmann

154

MATERIALIZING DIGITAL FUTURES

likens this to the notion of the sublime, I would argue that a machinic aesthetic is incompatible with traditional notions of aesthetic theory.

The collapse of the aesthetic universe In an attempt to quantify the aesthetic experience of art, Flusser proposes a model for approximating the subjective experience of works of art in terms of information theory and thermodynamics deriving from the work of Claude Shannon, shown in Figure 8.6. Within this model, artworks exhibit a kind of looping lifecycle, a ‘circularity of the aesthetic universe’, that reflects how the subjective experience of such objects changes over time and in relation to entropy, flitting between two extreme boundaries: the totally improbable (information in abundance) and the totally redundant (information lack) (Flusser, 2002: 55).

FIGURE 8.6  Approximation of the aesthetic universe.

THE INFINITE PORTRAIT

155

According to the laws of entropy, it is the natural tendency of the universe to progress towards the most probable and least informative situation (i.e. heat death). However, along this inevitable march towards heat death improbable situations may still arise that defy this tendency, if only temporarily, by creating and preserving information. Works of art, among other forms of cultural communication, are considered among these ‘negentropic’ occurrences that increase the sum-total of information available because they have come into being through an intentional gesture rather than being the likely or expected outcome of the pure randomness of the universe (Flusser, 2002: 9). Any new work of art is initially seen as improbable and overly informative, such that, from the perspective of a human existing within this universe, these works are considered ‘ugly’ or ‘terrifying’. To describe an artwork as ugly, however, is not to make a value statement of the piece but is, instead, to say that such works are affective because they differ so drastically from the accepted norm. Subjectively, any ‘new’ work of art may be defined as something that ‘makes us tremble because it is unexpected’ (Flusser, 2002: 52). In the aesthetic universe, new works are initially located at the highest point of the loop, nearest the boundary of total improbability. However, with the passage of time these ugly works become normalized within their environments and are seen as more and more probable but less informative as a result. As probability increases and information decreases, the work passes through other stages wherein it is experienced as ‘beautiful’ and then ‘pretty’ until reaching towards the other boundary of total redundancy where it is seen to be so commonplace as to be uninformative (i.e. ‘kitsch’). Again, these rather vague terms are not to be used as judgements but are simply labels or descriptors that convey how changes in entropy might be perceived subjectively; indeed, the entire model is metaphorical. A concrete image, be it produced by a human directly (e.g. a portrait) or an apparatus (e.g. a polaroid photograph), emerges into the aesthetic universe in a particular and unchanging form. The image is a ‘reservoir’ of information preserved visually and, as time passes, it traverses the loop as expected (Flusser, 2011: 18): from improbable and informative (ugly) upon first observation, to probable and uninformative (kitsch) upon the hundredth. In the case of Memories of Passersby I, the ephemeral images that pass in and out of existence in an instant present an entirely different situation. As previously discussed, the GAN-produced images of Memories of Passersby I are visualizations of data and, as Alexander Galloway argues, data requires such a visualization because it has no inherent visual form. Data exists as a discreet, dimensionless numerical unit that is imperceptible to the human eye because it lacks form and so, for data to be rendered perceptible, there must be a ‘contingent leap from the mode of the mathematical to the mode of the visual’ (Galloway, 2011: 88). Each iteration of the GAN

156

MATERIALIZING DIGITAL FUTURES

code is a ‘leap’ in Galloway’s terms, from nothingness to the realm of perception and, therefore, constitutes the production of new information. As the information spewing forth in every fleeting image is never the same, it seems inappropriate to describe Memories of Passersby I as a ‘reservoir’ and, instead, it is tempting to describe it as more akin to a ‘spring’ of information. Every momentary image of Memories of Passersby I is improbable and informative or aesthetically ugly but as each is destroyed and replaced instantaneously by the next ugly image, there is no traversal of the loop. The images are instantaneous and have no history. In relinquishing human control and allowing the apparatus to cycle through its infinite combinations, the viewer of Memories of Passersby I is presented with a constant overabundance of information, a continuous escalation of negentropy, and so is permanently affixed to the zenith of the aesthetic universe. Once circular and governed by the progression of historical time, the aesthetic universe collapses to a singular point, a position of infinite ‘ugliness’. In confronting the machinic aesthetic of Memories of Passersby I, the viewer does not so much ‘tremble’ at the unexpected and then relax as time passes but they appear to ‘convulse’ in endless terror of the infinite new; a horrific posthistorical experience.

The post-historical condition The notion that the technical conditions of media technologies (i.e. their form rather than content) shapes perceptions of the world is not exactly new and has been articulated in the well-known mantras of Marshall McLuhan (‘the medium is the message’) and Friedrich Kittler (‘media determine our situation’), as well as in more recent works of media theory and media archaeology.9 For Flusser, however, the privileging of the eye, above all other senses, in meaning-making processes suggests that images play a significant role in shaping experiences of time.10 As such, he suggests that different kinds of images presuppose different forms of temporal consciousness: mythical, historical and post-historical (Flusser, 2002: 117). For the prehistoric humans of Lascaux, cave paintings of animals were representative of phenomena in the world (i.e. traditional images) and in their static form allowed the eye to wander and scan the surface, encouraging a ‘mythical’ consciousness that experienced time as cyclical and never-ending: ‘while wandering over the surface of the image, one’s gaze takes in one element after another and produces temporal relationships between them. It can return to an element of the image it has already seen, and ‘before’ can become ‘after’: The time constructed by scanning is an eternal recurrence of the same process’ (Flusser, 2012: 8–9). It was not until the invention of linear writing that ‘historical’ consciousness and the very concept of ‘history’ as a sequence of events began. Rather than present the

THE INFINITE PORTRAIT

157

eye with static representations of the world, linear writing arranged images as discretized symbols that must be read chronologically to be understood as meaningful: ‘the eye travels along the page. Time and space, the event translated onto the page, is lineally set out before the eye. History comes into being’ (Barker, 2018: 29). In the linear motion of the eye, time was no longer experienced as static and instead seemed to flow along a chain of causality in which individual events, with definitive beginnings and endings and gaps to separate one from the other, are seen to be strung along a straight line where ‘everything is the consequence of causes and the cause of consequences’ (Flusser, 2002: 118). While the printing press and the widespread distribution of linear texts established historical consciousness as the dominant mode of experiencing time, Flusser asserts that the invention of photography and the rise of technical images in the past century is ushering in a ‘post-historical’ consciousness. Technical images, as previously discussed, are different to traditional images in that they do not signify phenomena in the world but, instead, signify the program of the apparatus; a combinatorial game that produces a concrete outcome from a range of abstract possibilities. As such, technical images are not understood as static representations of the world (traditional images) or flowing chronologies (linear texts) but are viewed as simply one possible outcome of many: ‘Prehistoric images are worldviews (copies of the environment). Photographs are computed possibilities (models, projections onto the environment)’ (Flusser, 2002: 129). Time is not experienced as cyclical or flowing but is ‘diced up’ into numerous disconnected temporal situations that emerge, not from ‘causality’ as consequences of the previous situation but, from the pure ‘chance’ of the combinatorial game. Therefore, while historical consciousness treats the present as ‘nothing more than a zero-dimensional point’ in the constant flow from past to future, in a post-historical consciousness ‘only the moment is real; it is a node in the surrounding field of possibilities’ that changes with every ‘click’ of the apparatus (Flusser, 2002: 118). The past is not a trailing line of previous presents but is an archive of ‘already realized possibilities’ while the future remains as those ‘yet unrealized possibilities’ that are splayed out flat, equally likely to occur, and seem to be ‘coming from all sides into the present’ (Flusser, 2002: 118). The photograph produced by a camera and a GAN-produced digital portrait are both just realized possibilities accounted for by the apparatus. Writing in the 1980s, Flusser theorized that the widespread emergence of technical images in television, advertisement and film would induce a grand, sweeping change in temporal consciousness across society. While I do not directly dispute this, I would limit my discussion to the work at hand and instead re-conceptualize this post-historical consciousness as a particular aesthetic condition or experience that is intensified in the case of Memories of Passersby I because of the GAN’s autonomy, as previously discussed. In

158

MATERIALIZING DIGITAL FUTURES

the historical condition, the chronological eye identifies individual symbols, words, gaps and punctuation that segment time and distinguish events that are meaningful from times of meaninglessness; words contrast with blank space on a page, new forms contrast with the habitual, periods of ‘beauty’ contrast with periods of ‘ugliness’. It is only through these binary relationships, in which something is because it is not something else, that images may become meaningful in a historical sense. As Flusser states: ‘consciousness, to be consciousness at all, is an unhappy consciousness. If all pain were relieved, all suffering numbed, the economy would be superseded. We could turn our backs to it and practice philosophy. But then there would be nothing left over which to philosophize’ (Flusser, 2011: 145). This existential suffering is what is truly at risk in the post-historical condition presented in the infinite imagery of Memories of Passersby I. With continual newness, or ‘ugliness’, and no perceived progression of time, there is no suffering from lack and so though the viewer may be eternally ‘terrified’ they will forget what it is to experience terror at all. Such a state is inhuman and, therefore, meaningless: ‘there can be no happiness without suffering. Utopia is impossible’ (Flusser, 2011: 145).

An inhuman utopia In confronting the machinic aesthetic, an eternal state on the cusp of pure improbability, the GAN shapes temporal experience according to a new scale of measurement, one that is inhuman. With traditional images, linear texts and other media that are limited or governed by the actions of human operators, the eye considers these images from a human perspective. A painting, for example, is understood as a single gesture that is symbolic of the ‘state of mind’ of the human artist and is, therefore, meaningful or fathomable. However, there are other scales within and without that of the human; other time scales that measure phenomena in terms of millennia or in fractions of seconds. As Flusser describes it, we exist ‘somewhere in the interior of a matrjoschka (Russian doll), a hierarchy of orders of magnitude in which each contains all smaller ones’ and the world is measured on a different scale depending on the order of magnitude (Flusser, 2002: 161). Viewing the same painting on the scale of microorganisms it is seen to be too grand or too slow to be of immediate concern, while on the cosmic scale of galaxies it is seen as too minute or fleeting to be noticed. Only on the human order of magnitude do artistic gestures, such as paintings and linear texts, carry any meaning; either viewed as cyclical/time-recurrent (i.e. mythical) or chronological/time-bound (i.e. historical). The infinite/unbounded and non-recurrent (i.e. post-historical) images of Memories of Passersby I lie within the ‘gray zone’ between the human and the inhuman order. They present continual information but are

THE INFINITE PORTRAIT

159

near-incomprehensible in their current context because they are so improbable. While such images are affective, they must be measured as fleeting possibilities or minute motions in an endless gesture, mere fractions of infinity (i.e. zero), and so are not seen to be symbolic in themselves. In viewing from the scale of the infinite, the eye becomes inhuman or ‘nihilistic’ in its meaning-making processes. This is a notion that Juhani Pallasmaa has discussed with reference to modern architecture and how the ‘deluge’ of new images in culture has led to the development of a nihilistic eye that is concerned solely with visual pleasure rather than meaning or being: ‘the art of the eye has certainly produced imposing and thought-provoking structures, but it has not facilitated human rootedness in the world’ (Pallasmaa, 2005: 19). The post-historical condition induced by Memories of Passersby I presupposes such a nihilistic eye but one that is also lidless, consuming all possibilities presented to it such that ‘the world becomes a hedonistic but meaningless visual journey’ (Pallasmaa, 2005: 22). The explosion of possibilities that the GAN, along with other AI, presents is not then an act of empowering the consumer but is a means of fashioning the perfect functionary by condensing the human’s being-in-the-world to a quantifiable input for the game at play. In this post-human ‘utopia’, in which machines carry out every function, the human’s role is minimized to the point that the subject no longer gestures at all and, instead, becomes static while the programmed ‘world’ moves about it. It is a world in which all meaningful things are created by that which is fundamentally inhuman and so, paradoxically, symbolize nothing of meaning at all. This is the world glimpsed in Memories of Passersby I.

Notes 1 For further material on the sale of the Portrait d’Edmond de Belamy and the criticisms regarding its authorship see references relating to articles from the Verge, the Mancunion, the Guardian, Mind Matters, Artnet News and Artsy. 2 Elgammal specifically compares the emergence of contemporary AI art to early photography which was disregarded as an artistic endeavour because ‘a machine was doing much of the work’. It was only after Edward Steichen’s The Family of Man exhibition at the New York Museum of Modern Art in 1955 that photography became widely regarded as fine art and photographers were considered artists and authors of the images. 3 The Portrait d’Edmond de Belamy is also just one instance of AI programs being used in contemporary art. Numerous artists such as Memo Akten, Trevor Paglen, Ian Cheng, Taryn Southern, Mike Tyka and Ross Goodwin are integrating AI programs into their varied practices. 4 For a general overview of authorship theory see the work of Seán Burke, Andrew Bennett and John Caughie. For more in-depth discussions on

160

MATERIALIZING DIGITAL FUTURES

Romantic theory see the work of M. H. Abrams, A. J. Minnis, Ernst Kris and Otto Kurz, and Jack Stillinger’s Multiple Authorship and the Myth of Solitary Genius (OUP, 1991) in particular. 5 See the GAN-generated artwork of artists Mario Klingemann, Robbie Barrat, Anna Ridler, Tom White and Libby Heaney, among others. 6 For further material on intentionalist theories of authorship see the work of E. D. Hirsch, Paisley Livingston, Berys Gaut, and Steven Knapp and Walter Benn Michaels. 7 For further material on anti-intentionalist theories of authorship see the work of W. K. Wimsatt and Monroe Beardsley, Cleanth Brooks, Hélène Cixous, Mary Jacobus and Peggy Kamuf. 8 For a more in-depth discussion of non-human photography see Joanna Zylinska’s Nonhuman Photography (MIT Press, 2017). 9 For further material on media theory and media archaeology see the work of Marshall McLuhan, Friedrich Kittler, Eric A. Havelock, Walter J. Ong, Jussi Parikka, Wolfgang Ernst, Siegfried Zielinski, Jonathan Crary, David Harvey and Timothy Barker. 10 For more on Flusser’s conception of temporal consciousness see Post-History (University of Minnesota, 2013).

References Barker, T. (2018), Against Transmission: Media Philosophy and the Engineering of Time, London, UK: Bloomsbury Academic. Barthes, R. (1987), Image Music Text, trans. S. Heath, London, UK: Fontana Press. Broeckmann, A. (2007), ‘Image, Process, Performance, Machine: Aspects of an Aesthetics of the Machinic’, in O. Grau (ed.), MediaArtHistories, 193–205, London: MIT Press. Cohn, G. (2018), ‘AI Art at Christie’s Sells for $432,500’, New York Times, 25 October. Available online: https://www.nytimes.com/2018/10/25/arts/design/ ai-art-sold-christies.html. Creswell, A. et al. (2018), ‘Generative Adversarial Networks: An Overview’, IEEE Signal Processing Magazine, 35 (1): 53–65. Elgammal, A. (2019), ‘AI Is Blurring the Definition of the Artist: Advanced Algorithms Are Using Machine Learning to Create Art Autonomously’, American Scientist, 107 (1) (January): 18. Flusser, V. (2002), Writings, ed. A. Ströhl, trans. E. Eisel, Minneapolis: University of Minnesota Press. Flusser, V. (2011), Into the Universe of Technical Images, trans. N. A. Roth, Minneapolis: University of Minnesota Press. Flusser, V. (2012), Towards a Philosophy of Photography, London, UK: Reaktion Books. Flusser, V. (2014), Gestures, trans. N. A. Roth, Minneapolis: University of Minnesota Press.

THE INFINITE PORTRAIT

161

Galloway, A. (2011), ‘Are Some Things Unrepresentable?’ Theory, Culture & Society, 28 (7–8): 85–102. Gleick, J. (1988), Chaos: Making a New Science, London, UK: William Heinemann. Hilburn, R. (2000), Chaos and Nonlinear Dynamics: An Introduction for Scientists and Engineers, Oxford, UK: Oxford University Press. Klingemann, M. (2018), Memories of Passersby I. Composition of multiple GANs. Mario Klingemann’s website: https://underdestruction.com/2018/12/29/ memories-of-passersby-I. Livingston, P. (2005), Art and Intention: A Philosophical Study, Oxford, UK: Oxford University Press. Lorenz, E. (1963), ‘Deterministic Nonperiodic Flow’, Journal of the Atmospheric Sciences, 20: 130–41. Macherey, P. (2006), A Theory of Literary Production, Abingdon, UK: Routledge. Pallasmaa, J. (2005), The Eyes of the Skin: Architecture and the Senses, Chichester, UK: Wiley-Academy. Sotheby’s (2019), Video, ‘The Hypnotic Allure of the AI Art Generator’, Youtube, 8 February. https://www.youtube.com/watch?v=Jjv3m5oWICA&t=1s. Vincent, J. (2018), ‘How Three French Students Used Borrowed Code to Put the First AI Portrait in Christie’s’, The Verge, 23 October. https://www.theverge.com/2018/10/23/18013190/ ai-art-portrait-auction-christies-belamy-obvious-robbie-barrat-gans.

162

SECTION THREE

Touch, body, metal, screen Introduction Blood, bone and flesh knitted together in the human body propel its movement. In and of the body, touch is multifaceted and a complex sense. This is made more so in the digital domain. Digital media are explored in this section through the emerging technological transformations of robotics and human–computer interaction or HCI (human–robot interaction or HRI) virtual and augmented realities, motion capture and animation. The section questions the transference of ideas between machines, media and humans to critically appraise digitization systems and their various purposes in individual, social and culturally important contexts. For their part, new communication technologies have presented fresh opportunities for legacy media public broadcasters to look forward – which offer important entertainment and informational services to the public – and the ways in which their public interest obligations are met. Established in the United Kingdom, Canada, Japan, Australia with variations in Europe they essentially have the general obligations to ensure a variety of cultural, educational and public affairs programming is put to air. The programming should be diverse so that there is ‘something for everyone’, regardless of

164

MATERIALIZING DIGITAL FUTURES

their socioeconomic status or whether they live in the city or the country. The public interest obligations of delivering universal service and access that were met through legacy analogue radio and television broadcasts are additionally realized via ‘new media’ such as digital television and the internet. Streamed content online and via digital television broadcasting currently presents additional means to deliver a diversity of educational, informational and entertaining content to people in urban and rural regions, providing universal service and access throughout a nation. We spoke to Amy Nelson and Astrid Scott from the public service broadcaster, the Australian Broadcasting Corporation (ABC). Amy Nelson and Astrid Scott are Senior Producers and Content Strategists for ABC Education. They specialize in the development of interactive content and new digital formats. ABC Education produces, acquires and distributes free curriculum-matched teaching resources to Australian primary and secondary school teachers: Amy Nelson and Astrid Scott: We did a project in ABC research and development (R&D) about the future of family entertainment. The context is that there is a multiscreen focus in media companies like ours such that we make content not just for broadcast on TV any longer, but also delivery on a laptop, on a phone or a tablet or on a watch – each of which is a different user experience. The focus was all about how virtual and augmented reality might play out in the home for entertainment. This was undertaken back in 2017 or 2018 and at the time we were looking five years ahead and speculating about content should VR and AR truly start to take off where the actual content might be experienced – not through goggles – but all around you in a virtual space and you all have a family entertainment experience together. Of course, it hasn’t happened quite yet. What we did was to bring together a number of TV executive producers who were at the time either making content for families or were about to commission some and we asked them to imagine the sorts of experiences they would produce for families. An experience that we are now envisaging for a museum could happen in the home instead or in a classroom. In R&D we were giving deep consideration to how certain public spaces might take up partnerships and act as gateways, for the time being at least, for showcasing immersive content and experiences because it is an easy way to deliver content in education. That is, schools are always doing tours of museums, zoos or galleries and albeit that it’s very challenging probably for a museum or the like to have multiple VR headsets to maintain for wear and cleanliness, it is optimal over any attempt to get the necessary technology into all schools or into every home which won’t happen for some time. The technology is expensive still. When we get to

TOUCH, BODY, METAL, SCREEN

165

the point of consumer-level products that are lightweight easy to take on and off the industry will move to another level. For content producers and especially directors, understanding of the technology as well as a spatial awareness is needed because images can be rotating 3600 which brings a completely different creative perspective to the process. In many respects we understand that at present a VR experience is not easily repeatable on a range of screens such as mobiles or tablets. There does not seem to be enough good experience to be gained yet to drive the audience to the technology. I think VR is slowly creeping along with improvements in real-time rendering, so you are getting better VR experiences. As a result, gamers want to use it increasingly and that’s happening a bit more, but I feel like we need another technology leap before you get that widespread take-up. Users need to get to a point in time, where engagement with the technology is organic and easy – akin to using a mobile – where you would pick it up or turn it on and straight-away you’re getting what you need, it’s intuitive and there is no lag [delayed content delivery]. In terms of AR, from the research we did in R&D a few years back, we knew then it was going to be a long time before there would be something like the Magic Leap [an augmented reality platform] or HoloLens [mixed reality smart glasses developed and manufactured by Microsoft] because what is being demonstrated is never quite there in actuality from the point of being able to truly deliver on what they’re showing you in the videos. This is really upsetting because it’s so tantalizingly close. Future facing and set in the context of theatrical productions that introduced public audiences to robots, Amy LaViers shares the results of several artistic projects designed as empowering introductions to robots in the form of public-facing outreach. These exposures were the first time many participants had seen a robot in real life – and for many more their first creative experience with the devices. The experiences shared the usage of embodied observation, whereby participants moved their own bodies to understand the movement of robots; imagined narrative, whereby participants used their own creativity to ascribe or describe the potential behaviour and applications of robots; and choreography, whereby participants made choices about movement during the engagement. The wider impact of such connection can be found in installations in art museums and on the web that situate robots inside performative contexts where their action is viewed through social, expressive and animated ends as opposed to their being functional tools for manufacturing. LaViers contends that creative experiences with robots might be substantive in engaging and retraining a generation of humans who will increasingly interact with automation and mechanization in factories and beyond.

166

MATERIALIZING DIGITAL FUTURES

Consideration of the physicalities of digital life – and the digitality of physical life – is amplified further in Chapter 10, which examines how humans and non-humans alike create a reality that is sensed and act upon. Luke Heemsbergen, Greg Bowtell and Jordan Beth Vincent have developed a conceptual framework for understanding the ways that physical digitality might be registered and made knowable. The authors first comment on the myriad ways that 3D printing creates a physical artifact from the digital in tangible data-rich ways. By extension, Heemsbergen, Bowtell and Vincent elaborate that whereas 3D printing (3DP) starts from virtual data designs to realize the physical, the inverse is found in Augmented Reality (AR) which perceives virtual forms of physical data. Their critique focuses specifically on the digitalities of sight and touch while acknowledging that multimodal perceptions of self and environment extend between the two senses essentialized in AR and 3DP. They explore how physical digitality relates new forms of power by making new types of things such that the relations they facilitate become perceptible and manipulable as a result. Heemsbergen, Bowtell and Vincent suggest the notion of visibilities of recognition which they adeptly argue make tangible the digitality of objects and persons that we relate to. Michela Ornati underscores that touch is a highly complex sense and extends this claim by studying the important, central role it plays in the daily activity of dressing. Using an enclothed cognition framework, Ornati carefully moves us through a fashion journey impacted by the process of digitalization and what this means for the embodied practice of dress where the touch sense, normally crucial to fashion consumption, is removed. In the digital domain, Ornati notes, clothes are visually accessible, but can neither be actively touched nor tried on prior to purchase. This leads to issues of mismatched size, fit and feel on the body, and contributes to unsustainable consumption practices with considerable environmental impact due to high product returns. Consequently, the author observes that industry is responding by giving due consideration to sensory inputs in the e-commerce experience and is actively seeking innovative solutions for enriching the way dress is embodied in digital fashion futures. This imperative for having an e-commerce presence is made even more real as a practical necessity during the ongoing Covid-19 pandemic which has limited access to physical retail experiences. Reaching outward, the final chapter in this section by Steph Hutchison and John McCormick explores what robots themselves learn from HCI and performative relationships. They present their work via case studies that used bodily processes drawn from dance and other movement practices in order to develop strategies for a robot’s movement. They investigate choreographic and embodied practice approaches involving motion capture, animation, robots, artificial intelligence (AI) and virtual reality (VR). The authors’ research approach, simultaneously highlighting the methods used,

TOUCH, BODY, METAL, SCREEN

167

allow them to critically demonstrate how choreographic and improvisational practices provide fertile ground for opening the collaborative potential of humans and robots. As a result, Hutchison and McCormick assert that a greater understanding of HRI, shared movement invention and kinaesthetic empathy is the result. In accord with others in this collection, the authors discuss issues that undergird emerging and connected media systems to impart that the shared movement between humans, robots and technologies has impacted areas in and also beyond the arts. If robots for their part are to truly share the spaces that humans work, play and socialize in, learning to move with, around and between each other is imperative.

168

9 First encounters with robots through embodied observation, imagined narrative and choreography Amy LaViers

Introduction First encounters with robots are notably charged with novelty and wonder – such as with any new device, product or experience but heightened by the fictional tales that are associated with the developing devices. Most people first encounter robots as literary devices in fictional books and fantastical characters in movies. This is even the source of the introduction of the term ‘robot’ in a 1920 science fiction play (Capek, 2004). But, 100 years after the introduction of the idea, real devices that claim the moniker exist, creating a confusing divide between science and fiction. As robots move out of cloistered cages on factory floors and into human-occupied spaces, this divide may create challenges around acclimatization for humans in spaces shared with robots. Already, there are trends that indicate this acclimatization to new technology is progressing with a negative bent: anecdotes of factory workers describe people going out of their way to avoid crossing paths with autonomous forklifts that are increasingly present in human-filled areas of

170

MATERIALIZING DIGITAL FUTURES

a factory; campaigns of political candidates are centred around the idea that job replacement will create apocalyptic unemployment; and sensational headlines claim that robots, like Boston Dynamics Atlas humanoid, are outperforming humans in activities like gymnastics that require agility and grace. On the other hand, a narrower group of people is exposed to the failures and limits of these devices. Working with robots, either in hardware or simulation, is limited to a subset of people studying engineering and design, and these individuals not only have a better understanding of the limits of the devices, they also experience making choices about the end system behaviour, appearance and aesthetic. It is in this context that this chapter asks: What makes an empowering first experience with a robot? What if we could design an ideal first encounter? This prompts a design question: What features should an ideal first encounter with robots have? Such a design would be dependent on the goals for human–robot relationships. On the one hand, the aim may be that a majority of humans are empowered to create and manipulate the behaviour of the robots around them, for example, a consumer buys a robotic vacuum with customizable modes and active infrastructure, guards, beacons and the like that outfit unique home layouts to control where the device operates; on the other, the endeavour may be to influence humans to successfully accommodate the behaviours of extant robots, designed by a small minority, for example, a consumer buys a robotic vacuum cleaner with pre-prescribed functionality and no required setup. In the former model, humans need an understanding of machines as manipulable, editable and fallible. In the latter model, humans need to expect certain levels of performance, accommodate possibly foreign conventions and learn about a novel set of affordances. Both types of understanding are probably needed to create harmony between man and machine; however, it was not until interfaces developed allowing ‘common’ users to interact with computers creatively that the computer transformed so much of human life (Isaacson, 2014). Interaction modalities between humans and robots form a new kind of ‘interface’ that can be designed to support creative expression from users. Likewise, activities that introduce participants to robots – be it the unboxing of a new purchase, an educational course, an artistic event or a depiction in other forms of media – may support one model or the other. This chapter describes a variety of examples of work that aims to provide accessible creative experiences with robots and provides a description of an activity that integrates broad themes from across multiple events. This work builds on a range of prior work that is reviewed in the second section. The educational pedagogy used internally within a research group that emphasizes somatic and choreographic practice is discussed. A series of artistic events with different time, space and resource constraints are described in the third section. Common themes from these artistic exercises, embodied observation, imagined narrative and choreography, are analysed

FIRST ENCOUNTERS WITH ROBOTS

171

in the fourth section. The fifth section describes an educational activity template that incorporates features of these events into a holistic robotics education design that emphasizes empowering students with creative agency over the robot design and behaviour, with little prior training required. The final section summarizes these themes and points to future directions.

Background The work presented here is inspired by decades of using robots in creative pursuits and scholarship for understanding how humans interact with robots. This work spans scholarly research in education, kinesiology, dance and robotics as well as artistic practice and the interdisciplinary application of these areas – and, as such, is too expansive to exhaustively cover in this chapter. Instead, this section focuses on a few key areas that directly impact and contextualize the presented approach to introducing people to robots through creative experiences, considering especially how early exposure impacts longer-term perceptions and how kinaesthetic and creative experiences have been leveraged.

Impact and content of first experiences In HCI and interface design, first impressions have been shown to be crucial in eventual adoption of new technology. Researchers describe disorientation and cognitive loading as key factors to avoid for successful adoption in selfguided interactions with new interface screens (Saadé and Otrakji, 2007). In accessing a new website, first impressions often determine whether users stay on the site and navigate beyond the landing page (Kim and Fesenmaier, 2007). Others have studied robots as interactants (agents rather than tools), finding that initial impressions influence the mental model formed over repeated interactions (Paetzel et al., 2020), including perceived gender (Bernotat et al., 2019). HRI studies involve recruiting human participants to conduct experiments. Most people, however, are exposed to robots through media, including news, books and film. An analysis of the sentiment of human–robot relationships in film showed strongly negative tones, including violence between humans and robots, in the majority of popular films that include robots (Cuan et al., 2020). When combined with research on the importance of first impressions with technology, such fictional portrayals may be influencing downstream interactions, colouring future experiences with negativity and antagonism. Further, news outlets often leverage anthropomorphism and hyperbole to describe new advances in robotics, which may contribute to the deployment of biased systems (Howard and Borenstein, 2018). Thus, popular media,

172

MATERIALIZING DIGITAL FUTURES

one of the most accessible inroads to passively learning about robots, can be hyperbolic and negative.

Modulating the parameters of first experiences Researchers have firmly established that perceptions of robots can be modulated through how the devices are introduced. Giving simple buglike robots a touching backstory made it less likely for participants to be willing to smash the devices with a hammer later in the experiment (Darling et al., 2015). Introducing a robot as a ‘toy’ versus a ‘robot’ changed how participants evaluated the anthropomorphism and ‘personality’ of the device (De Graaf and Allouch, 2014), and previously held high expectations of lifelikeness resulted in a more positive evaluation of a robot overall with additional evidence supporting human-like treatment of a zoomorphic device (De Graaf et al., 2017). It is not yet well-understood how familiarization towards robots evolves with time, but evidence suggests that some elements of first impressions, such as competency, persist over repeated interactions (Paetzel et al., 2020).

Creative experiences The notion of creativity in education – and specifically in STEM education – is a core component of experience design. Just as computers have been tools that facilitate creativity (Lubart, 2005), the embodied nature of robots makes them apt for similar processes. Moreover, it has been discussed that creativity as a social construct within educational robotics is uniquely important because ‘in order to become humanlike, the robot needs to be built and imagined as such. Therefore, by encouraging students to make meanings of what makes the robot humanlike, and hence, what it means to be human, anthropomorphic social robots foster student creativity’ (Zawieska and Duffy, 2015: 330). However, the outcomes of such creatively framed interactions can be hard to measure (Alves-Oliveira et al., 2019).

A bodily inroad to understanding artificial movement It has been shown that unstructured creative interaction can improve outcomes in educational settings with robots (McCoy-Parker et al., 2017), but providing these experiences can be hard with technical content that requires heavy explanation. One modality for coming to an understanding of robot features, such as competency, is through observation of the device’s movement. The notion of ‘kinaesthetic empathy’ has been observed and discussed for

FIRST ENCOUNTERS WITH ROBOTS

173

a wide variety of non-anthropomorphic objects (Miyoshi, 2019). Likewise, the act of moving itself has been noted as essential to creating a sense of animacy in robots (Gemeinboeck and Saunders, 2017), and this kinaesthetic channel has been used to generate behaviours for robots (Gemeinboeck and Saunders, 2018).

Using the arts to lend a creative, expressive lens to experience A natural extension of this line of work that uses kinaesthetic modality of sensing to interact with and design robots is the inclusion of expertise of dancers (Gemeinboeck Saunders, 2015). Situating robots inside a performance setting has been used both for public outreach (Schwartz, 2014) and artistic ends (Apostolos, 1992) as well as research – as described in the next section. In these settings, the treatment of robots by onstage performers influences how audience members treat the devices (Nabais, 2014). Installations in art museums (Gannon, 2016) and on the web (Goldberg, 2001) further situate robots inside performative contexts where their action is viewed through social, expressive and animated ends – rather than as functional tools for manufacturing. Such settings provide viewers the opportunity to imagine new functions, consider new applications and decide on their own personal relationship to such devices. Thus, these applications inspire a style of outreach described in the next section.

Outreach events inspired by somatic and choreographic practice The Robotics, Automation, and Dance (RAD) Lab served primarily as a space for academic scholarship and research (due to its setting at researchactive universities between 2013 and 2020), studying how somatic and choreographic practice could advance foundational knowledge in the field of robotics; however, occasional outreach events open to the public were also held. More critically, the lab needed to educate graduate and undergraduate researchers to engage in the research discourse at hand. Thus, inside a confluence of training advanced engineering students, researching questions about meaning-making in movement, and sharing this work with the public, a particular style of outreach event was developed. The model has been used with children barely in school, their mothers standing by to help facilitate, through to adult learners, curious about how their own expertise in somatics and choreography can impact the design of technology. This chapter focuses on the influence of concurrent artistic projects in the lab of these outreach activities, suggesting a particular activity design in the fifth section.

174

MATERIALIZING DIGITAL FUTURES

In our work, we have come to use ‘somatics’ and ‘choreography’ to describe two halves of the same process: making meaning through motion (LaViers et al., 2018). From an internal, first-person point of view, we have somatics, which deals with the lived experience at the nexus of ‘mind’ and ‘body’, typically eschewing dualistic models that try to divide the two. From an external, third person point of view, we have choreography, which deals with the process of creating a specific, intentional meaning for an observer of motion. To engage in both simultaneously is to bring one’s own embodiment, or bodily intelligence, to the process of creating externalized models and instances of movement, for example, an algorithm for activity detection or a dance about the creative process. This process has been written about inside the practice of research but not how this approach to work translates to public outreach, which is the subject of this essay.

Core pedagogy developed through education for engineers In any program design, decisions must be made about what to include and what to prioritize in a temporally and spatially limited setting. Complicating this choice is that due to the interdisciplinary nature of the work, educational objectives range from topics in coding and mechanics to movement theory and performance. Striking a balance between these foundational topics requires some relaxation of details in each. Rather than teaching a particular style of terminal-based coding, equation of motion, principle of movement or choreographic tool, we designed for each setting a through-line that captures the qualities of the brittleness of code, the malleability of human movement and the role of context in assembling the success of a particular instance of technology. Thus, participants may leave without knowing some technical details, for example, the syntax of a ROS-based script, but they also leave without the associated frustration of adhering to tedium. Our goal is that these initial introductions leave participants feeling empowered to design artificial systems – rather than intimidated by a lexicon of foreign terms and conventions.

Public interactive art installations A key inspiration for the design of such educational experiences is through the public interactive art installations that have gone up in parallel. Rather than being inset in an educational context, these events come with a performative implication; that is, there is an automatic invitation for audiences to engage their imagination and dreams. Typically, these events have their own ends – artistic and/or scholarly – but they also serve as public points of interaction for our work, serving as a key component of the proposed plan for broader

FIRST ENCOUNTERS WITH ROBOTS

175

impacts and dissemination of publicly funded work, and this tonality inherent in the performances described here will carry over into the more targeted educational experiences described in the next section, sharing key themes common to both distinct modes of public engagement. Time to Compile. In collaboration with artist-in-residence Catie Cuan, who has discussed the residency inside her creative process (Cuan, 2021), the RAD Lab developed a piece exploring the creative barriers in working with robots and contrasting the experience within a robotics lab with the experience outside of it. The work, entitled ‘Time to Compile’, involved stage-based choreography and performance as well as an interactive installation shown at several venues between 2017 and 2019, beginning with an informal showing in the lab itself, an in-progress showing at two academic conferences and culminating in polished, free presentations at public venues: MindSet BrainGym in Toronto, CAN (alongside a showing at an industry-based summit sponsored by ThoughtWorks) and the Mountain Arts Center in Prestonsburg, KY. In this work participants were invited to mirror the motion of a series of moving bodies: a large (~8 ft tall) robot shadow, a small (~2 ft tall) humanoid robot, a digital avatar of a human actor’s movement and a live human performer. Each participant was told that perfectly imitating the motion they would see would be impossible – that they would have to make creative choices with their bodies about how to mirror the movement. This facilitated an intimate observation of a series of natural and artificial bodies mediated by technology and heightened by the dark, meditative spaces that housed a series of four isolated rooms, which participants inhabited one by one (except in the cases of small children who went through the installation with guardians). After the experience, participants were asked to draw pictures of the overall installation and/or to imagine the moving elements inside different imagined narratives. This work is described for its use in academic research (Cuan et al., 2019), but it was also a public event held outside of university ecosystems that involved a first exposure and interaction with a robot in real life for many participants. A Machine. Another theme in the work of the lab is to broaden the notion of ‘robots’ beyond the human-like characters shown in science fiction to include the myriad automated and partially automated machines we interact with every day. In an interactive piece ‘A Machine’, shown in a free public installation in the Krannert Center in December 2018 (as well as later at an academic conference), the lab engaged audience cell phones to create a digital, chaotic sound score for a piece that explored the Turing Machine metaphor and academic writings on the ends and goals from the field of dance. This work explored the deep distinction between human experience and machine inner workings. Audience members, sitting on cushions adjacent to their surrendered cell phones, as shown in Figure 9.1, watched choreographed movement that played the curved, organic body of the performer antagonistically inside the lines of a large, linear grid marked

176

MATERIALIZING DIGITAL FUTURES

FIGURE 9.1  Performance of ‘A Machine’, 13 December 2018, Krannert Center, IL, USA, engaging participants (and their cell phones) in an interactive performance. Photographer, Natalie Fiol. Used with permission.

on the performance space, set to the soundscape of audience member cell phones, which were being called through an off-stage computer script. In addition, audience members were invited to answer true/false questions that demonstrated the texture of logic gates found inside computers; each member was asked a true/false question, for example, ‘we will all have a happy reunion with our cell phones at the end of the show’ or ‘I, the performer, believe that I have three arms’. This interaction provided at times humorous and at times confrontational interactions between the performer and the audience and highlighted how many questions do not have a clear binary answer. This work is described in an academic context as part of a practice-based research project in prior publications (LaViers, 2019), but also served as an access point for members of the Urbana, IL, community (most of them related to the university in some way) to easily engage in academic research topics. Babyface. Working with artist-in-residence Kate Ladenheim, the lab developed the work ‘Babyface’, which comprised staged choreography, augmented by a wearable robotic device, and interactive sculptural installations of articulated machines, whereby audience members could activate the motion of a wall-mounted robot simply by breathing in and out. This piece explored the representation of femme bodies in technology and the resulting ways that the pressures applied by society on such bodies were reinforced and heightened by the designs of that technology. It also invited audience

FIRST ENCOUNTERS WITH ROBOTS

177

members to explore their relationship to machines, contrasting an internal, meditative experience – keying into their experience of their own breath during the calibration procedure – with an external, performative experience facilitated by performers once the participant was successfully engaging the machine. This work was presented to the public on the outdoor waterfront of Wellington, NZ, during a five-day container-based free arts festival, The Performance Arcade. and is detailed in a prior publication (Ladenheim and LaViers, 2021). As a form of public outreach, this work allowed participants to physically control the motion of a non-anthropomorphic, large (~8 ft by 18 ft) robot with the subtle action of their own body during breathing. This was designed to be an empowering feeling that shows the ability of technology to magnify small actions taken by humans.

Key themes used to provide creative experiences with robots to untrained participants Installations and presentations like the three discussed in the prior section have common themes we want to use in the design of an outreach activity, one that shares a similar level of accessibility and emphasizes robots as machines for creative expression. These features are embodied observation, imagined narrative and choices in choreography. After first thoroughly identifying each feature in turn in this section, the next section will explain their implementation in feasible activities for outreach. Note, it is not the goal of this chapter to argue that every educational experience with robots needs to have these features. Instead, the chapter describes another insight developed through specific events in the performing arts, ‘Time to Compile’, ‘A Machine’, and ‘Babyface’, and informed by broader literature discussed in the ‘Background’ section. Embodied Observation. First, these events ask participants to move their bodies in order to engage in the activity. In ‘Time to Compile’ participants mirrored the action of other moving agents. This task required first observing the action of a foreign and unfamiliar body and then moving one’s own body in response. In ‘A Machine’ participants seated themselves inside the performance space and engaged in structured conversation with the performer who asked them ‘true/false’ questions. Each participant could follow this structure or break it and, in either case, was forced to reckon with the physical closeness of the performer and fellow participants. In ‘Babyface’ participants had to attune to their breath and follow choreographic prompts that played aloud, inviting large bodily actions. Each of these modalities served to bring the lived, felt and breathed somatic perspective to the experience.

178

MATERIALIZING DIGITAL FUTURES

Imagined Narrative. Another feature of these events is the opportunity for participants and audience members to bring their own imagination to the work – a feature of any work of art. In ‘Time to Compile’ participants were explicitly asked, via research surveys before and after the experience, to select narratives and describe the experience. In ‘A Machine’ suggestive and outlandish true/ false questions like ‘I believe I have three arms’ required imagination to make sense. In ‘Babyface’ participants stepped into a separated physical space with a fantastically dressed performer and then were asked questions like ‘don’t you feel good now?’ To engage in such work, audience members must construct a narrative (literal or abstract) for why the performance is proceeding as it is. Such imagination allows each audience member to develop a personal relationship to the work, elucidating their own meaning from the experience. Choices in Choreography. Finally, each event invited the audience member to make choices – to help choreograph the event. In ‘Time to Compile’, as the movements of each body were impossible to mirror perfectly, alongside the invitation to make choices, participants were put inside a creative space. Further, some of the installation elements were influenced by their movement, changing the experience for themselves and others. In ‘A Machine’ audience members could choose to follow the structure of the ‘true/false’ prompt – or push against it. Moreover, within the structure of answering ‘true’ or ‘false’ they exercised tone, emphasis and their own bodily movement to communicate a dubious ‘true’ or a cautious ‘false’ (and so on). In ‘Babyface’, participants were left to ‘play’ in the space once the sensor had successfully calibrated to their breath. They could choose to engage with the choreography, ignore it or find more comfortable movements similar to the prompts and actions of the performer. Each of these formats required little learning on the part of the audience, offering an accessible space in which to make choices about machines.

An integrated activity for creative first encounters with robots Reflective of the practice of artmaking described in the prior section, the design of outreach events can offer embodied observation, imagined narrative and choice in choreography. These themes are slightly resituated inside an educational experience with robots in the outreach activity outlined next. Elements of this outline were used in several outreach activities held over the course of three years, each of which provided new context for how young people and adult learners alike respond to the material (which indeed also influenced the design of the previously described interactive installations). These venues included the MechSE Girls’ Adventures in Mathematics, Engineering, and Science (GAMES) camp, the Urbana Neighborhood Connections Center (UNCC), the East Kentucky Science Center (EKSC), the Laban/Bartenieff Institute of

FIRST ENCOUNTERS WITH ROBOTS

179

Movement Studies and Curious Jane. Images of these events, which were designed and implemented with the aid of lab members, artists-in-residence and community outreach organizations, are provided in Figures 9.3–9.5. Context Design. Before introducing any new concepts, which could be intimidating or imply (possibly imagined) constraints to students, we take a moment to allow students to situate the idea of new technology in their own lives. Providing a worksheet (or working in small discussion groups) with an image of a robot, we ask students to engage in a free writing exercise with the prompt: ‘Tell a story about this robot. What will it do in the future?’ This initiates the process of feeling empowered over the action of the device and opens minds to possible application areas relevant to their own lives (often a least favourite chore or task for that individual is listed). This exercise builds on existing skills and puts each student in control of a robotic device without ever lifting a finger to code. Movement Analysis and Design. Next, students learn about a symbolic taxonomy for describing different actions, their quality, locations in space and body parts used (see Figure 9.2). This is facilitated by the Laban/Bartenieff Movement System (LBMS), which offers an analytic insight into the conscious experience of human movement – and a symbolic taxonomy for movement description (Studd and Cox, 2013). This set of symbols is used to help students generate, observe and describe movements with greater detail. Each symbol is introduced with students embodying examples of the action, creating a moment that feels like an exercise or dance class. Then, students create their

FIGURE 9.2  Symbols from the Laban/Bartenieff Movement System commonly used in Motif.

180

MATERIALIZING DIGITAL FUTURES

own movement phrases, notating them with the symbol set, with assistance from the teaching team, creating concrete movement scores and practicing their movements to refine the specificity of each action and the phrase overall. Hardware Design. Then, the students design or learn about the morphology of the robot. In some cases, where time permits, students create their own robots out of pipe cleaners. Here, students are introduced to distinct modes of actuation and sensing for their devices. For actuation, they are shown examples of rotary (to facilitate rotation) and linear (to facilitate translation) actuators mocked up in pipe cleaners, wire or other craft elements. For sensors, we discuss types of quantities we can measure about the environment: light, including images from cameras, chemicals, distances, and so on. These sensors are represented abstractly with elements like pom-poms and google eyes. In other cases, where resources permit, students learn about an existing platform, like the NAO humanoid robot, reviewing its list of onboard actuators, sensors and computational elements (see Figure 9.3). In both cases, students analyse

FIGURE 9.3  NAO hardware with movement design. Photographed by the author © Amy LaViers.

FIRST ENCOUNTERS WITH ROBOTS

181

the distinct morphology of each body, comparing it to their own body’s architecture, noting areas where degrees of freedom and ranges of motion are distinct. Software Design. Now, the students are tasked with taking their embodied movement phrases, notated with symbols from LBMS, and translate them to the artificial body they have just analysed. Students are given a worksheet with open, empty circles and arrows in between – a finite state machine. This mathematical abstraction is used to explain how a computer (or robot) needs explicit instructions to operate. In other words, the devices cannot interpret the created movement phrases automatically. Noting the ways their own movement cannot be exactly replicated on the robot, they make choices in the configuration space of the device, using the finite state machine to organize their drawings. Then, if we are using a real, working robot like the NAO, students get to program their movements into the device and watch their phrases come to life on a new body. Environment Design and Contextual Analysis. In some cases, each student may have their own robotic platform (Figure 9.4). For example, working with the Champaign-Urbana Fab Lab, we created music-box like robotic platforms that students could program and decorate themselves. This opportunity for personalization harks back to the opening activity, allowing the students to create a personalized – and personally meaningful – application for the device. In this case, we ask students to create the appropriate context for the movement they have created. This exercise again engages

FIGURE 9.4  Robotic platform. Pipe cleaners, wire and other craft materials attach to two reprogrammable servo motors; the low-cost form factor allowed each student to have their own platform, personalizing the context to their movement design. Photographed by the author © Amy LaViers.

182

MATERIALIZING DIGITAL FUTURES

FIGURE 9.5  Adult learners presenting a robot design and purpose to a larger group. Photographed by the author © Amy LaViers.

imaginative design that builds on their own existing skills. If they struggled with the programming, this gives them an opportunity to engage in a more comfortable, familiar activity to end the workshop and experience. Evaluate and Present Design. Finally, presenting their design to others – and evaluating ways to improve it – provides students a moment of celebration, sharing and reflection (see Figure 9.5). Thinking about how they would remake the design or challenges with extant technology, for example, if they required an ‘emotion’ sensor, adds realism to the imaginative sequence of design actions. Further, seeing examples where movement design successfully blends with the environment, emphasizes a key principle in LBMS as well as in robotics: movement must be created to cope with the environment.

Conclusion This chapter presents an integrated activity in contextual, embodied robotic design appropriate for a short introduction to participants from multiple age groups. The activity is inspired by prior literature in creativity, education and

FIRST ENCOUNTERS WITH ROBOTS

183

robotics as well as by interactive robotic art installations, which explicitly invite viewers to use their imagination to understand the work. To a similar end, the chapter suggests specific modalities to be used in outreach events that might be especially effective advances for facilitating creative experience for students and the lay public. Specifically, this activity leverages embodied observation, imagined narrative and choices in choreography. Creative experiences with robots may prove to be critical to engaging and retraining a generation of humans who will increasingly interact with automation and mechanization.

Acknowledgements The events described in this chapter were made possible by many internal and external collaborators: artists-in-residence Catie Cuan and Kate Ladenheim; lab members Hang Cui, Ishaan Pakrasi, Erin Berl, Reika McNish and Wali Rizvi; MechSE organizer Joseph Muskin and student Ashley Armstrong, who wrote up the first handbook of this activity design; Fab Lab director Jeff Ginger and members Colton Jackson, Dot Silverman and Heather Arnett; UNCC director Janice Mitchell; EKSC director Steve Russo and staff member Susan Scott; MAC organizer Joe Campbell; LIMS director Curtis Stedge and staff member Claire Baum; and Curious Jane director Samantha Razook. Funding for these events came from these collaborating organizations as well as ThoughtWorks Arts, National Science Foundation (NSF) grants #1528036, #1701295 and #1834893, and Defense Advanced Research Projects Agency (DARPA) grant #D16AP00001.

References Alves-Oliveira, P., S. Tulli, P. Wilken, R. Merhej, J. Gandum and A. Paiva (2019), ‘Sparking Creativity with Robots: A Design Perspective’, in 14th Annual ACM/ IEEE International Conference on Human Robot Interaction (HRI). Apostolos, M. K. (1992), ‘Robot Choreography: Kinesthetic Creativity at the Mind–Metal Interface’, Presence: Teleoperators & Virtual Environments, 1 (1): 149–50. Bernotat, J., F. Eyssel and J. Sachse (2019), ‘The (Fe) Male Robot: How Robot Body Shape Impacts First Impressions and Trust Towards Robots’, International Journal of Social Robotics, 1–13. Capek, K. (2004), RUR (Rossum’s Universal Robots), London: Penguin. Cuan, C., E. Berl and A. LaViers (2019), ‘Time to Compile: A Performance Installation as Human–Robot Interaction Study Examining Self-Evaluation and Perceived Control’, Paladyn, Journal of Behavioral Robotics, 10 (1): 267–85. Cuan, C., J. Hoffswell and A. LaViers (2020), ‘Stories About the Future: Initial Results Exploring How Co-Movement with Robots Affects Perceptions About

184

MATERIALIZING DIGITAL FUTURES

Robot Capability’, in Proceedings of the 7th International Conference on Movement and Computing, 1–8 July. Cuan, C. (2021), ‘Dances with Robots: Choreographing, Correcting, and Performing with Moving Machines’, TDR: The Drama Review, 65 (1): 124–43. Darling, K., P. Nandy and C. Breazeal (2015), ‘Empathic Concern and the Effect of Stories in Human–Robot Interaction’, in 2015 24th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), August, 770–5, IEEE. de Graaf, M. M. A and S. B. Allouch (2014), ‘Expectation Setting and Personality Attribution in HRI’, in 9th ACM/IEEE International Conference on HumanRobot Interaction (HRI), March, 144–5, IEEE. de Graaf, M. M. A. and S. B. Allouch (2017), ‘The Influence of Prior Expectations of a Robot’s Lifelikeness on Users’ Intentions to Treat a Zoomorphic Robot as a Companion’, International Journal of Social Robotics, 9 (1): 17–32. Gannon, M. (2016), ‘Mimus’. Available online: https://atonaton.com/mimus. Gemeinboeck, P. and R. Saunders (2015), ‘Towards Socializing NonAnthropomorphic Robots by Harnessing Dancers’ Kinesthetic Awareness’, in International Workshop on Cultural Robotics, August, 85–97, Cham: Springer. Gemeinboeck, P. and R. Saunders (2017), ‘Movement Matters: How a Robot Becomes Body’, in Proceedings of the 4th International Conference on Movement Computing, 1–8 June. Gemeinboeck, P. and R. Saunders (2018), ‘Human–Robot Kinesthetics: Mediating Kinesthetic Experience for Designing Affective Non-Humanlike Social Robots’, in 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), August, 571–6, IEEE. Goldberg, K. (ed.) (2001), The Robot in the Garden: Telerobotics and Telepistemology in the Age of the Internet, Cambridge, MA: MIT Press. Howard, A. and J. Borenstein (2018), ‘Hacking the Human Bias in Robotics’, ACM Transactions on Human-Robot Interaction, 7 (1): 1–3. Isaacson, W. (2014), The Innovators: How a Group of Inventors, Hackers, Geniuses and Geeks Created the Digital Revolution, New York, NY: Simon and Schuster. Kim, H. and D. R. Fesenmaier (2007), ‘The Persuasive Architecture of Destination Websites: The Effect on First Impressions’, in Information and Communication Technologies in Tourism 2007, 255–66, Vienna: Springer. Ladenheim, K. and A. LaViers (2021), ‘Babyface: Performance and Installation Art Exploring the Feminine Ideal in Gendered Machines’, in D. C. Herath, E. A. Jochum, C. Kroos and D. St-Onge (eds), The Art of Human–Robot Interaction: Creative Perspectives from Design and the Arts Topic in Frontiers in Robotics and AI, 8, 1–21. LaViers, A., C. Cuan, C. Maguire, K. Bradley, K. B. Mata, A. Nilles, I. Vidrin, N. Chakraborty, M. Heimerdinger, U. Huzaifa and R. McNish (2018), ‘Choreographic and Somatic Approaches for the Development of Expressive Robotic Systems’ Arts, 7 (2): 11. LaViers, A. (2019), ‘Ideal Mechanization: Exploring the Machine Metaphor through Theory and Performance’, Arts, 8 (2): 67. Lubart, T. (2005), ‘How Can Computers Be Partners in the Creative Process: Classification and Commentary on the Special Issue’, International Journal of Human-Computer Studies, 63 (4–5): 365–9.

FIRST ENCOUNTERS WITH ROBOTS

185

McCoy-Parker, K. S., L. N. Paull, A. C. Rule and S. E. Montgomery (2017), ‘Challenging Elementary Learners with Programmable Robots During Free Play and Direct Instruction’, Journal of STEM Arts, Crafts, and Constructions, 2 (2): 6. Miyoshi, K. (2019), ‘What Allows Us to Kinesthetically Empathize with Motions of Non-Anthropomorphic Objects?’ Journal of Somaesthetics, 4 (2): 52–67. Nabais, F. (2014), ‘I-CARE-US: Flying Robots and Human-Robot Interaction in Digital Performance’, in N. S. Sutil (ed.), Digital Movement: Essays in Motion Technology and Performance, New York: Palgrave Macmillan. Paetzel, M, G. Perugia and G. Castellano (2020), ‘The Persistence of First Impressions: The Effect of Repeated Interactions on the Perception of a Social Robot’, Proceedings of the 2020 ACM/IEEE International Conference on Human-Robot Interaction, March, 73–82. Saadé, R. G. and C. A. Otrakji (2007), ‘First Impressions Last a Lifetime: Effect of Interface Type on Disorientation and Cognitive Load’, Computers in Human Behavior, 23 (1): 525–35. Schwartz, B. (2014), ‘Communicating Science through the Performing Arts’, Interdisciplinary Science Reviews, 39 (3): 275–89. Studd, K. and L. Cox (2013), Everybody Is a Body, Indianapolis, IN: Dog Ear. Zawieska, K. and B. Duffy (2015), ‘The Social Construction of Creativity in Educational Robotics’, Progress in Automation, Robotics and Measuring Techniques, 329–38, Cham: Springer.

186

10 Physical digitality: Making reality visible through multimodal digital affordances for human perception Luke Heemsbergen, Greg Bowtell and Jordan Beth Vincent

Introduction This chapter considers the physicalities of digital life – and the digitality of physical life – as humans and non-humans alike create a reality that we perceive and act upon. We chose 3D printing (3DP) and Augmented Reality (AR) to highlight with ‘extreme cases’ the reality we have been living for some time: that there is no such magical place as ‘online’ and digital dualism is a fallacy (Jurgensen, 2011); that our data do not exist discretely from their related organics; and that as Haraway (1985: 39) contends, we must acknowledge a need to take responsibility for the social relations of science and technology so as to reconstruct the boundaries of daily life as we connect with others and all of our parts. Yet, as Hayles (2006) critiques, what ‘we’ are in these configurations of computation is not clear nor stable. She suggests that what we make and what (we think) we are, co-evolve. We are then less concerned about the ‘immaterial physicality’ of how digital capitalism and the ideology of the digital reinforce each other through technology (cf.

188

MATERIALIZING DIGITAL FUTURES

Betancourt, 2013), and more concerned with the co-evolution of experience of having physical digitality construct environments and humans. We suggest a critical stance less to do with Marxist critique of man and machine through an approach with its attendant considerations of labour and instead offer a reinterpreted critique through ‘physical digitalities’ with more radically pluralistic phenomenological considerations of the (post)human. This chapter narrows its critique of these issues to digitalities of sight and touch through AR and 3DP as extreme cases of the digital made physical and the physical made digital. We do, however, acknowledge multimodal perceptions of self and environment extend between the two senses essentialized in AR and 3DP. Our argument explores how physical digitality relates to new forms of power by making new types of things and the relations they afford – perceptible and manipulable. We understand affordances from literature that consider how social and mediated relations allow complex sets of perceived and imagined capacity and constraint (Nagy and Neff, 2015) for communication (Schrock, 2015; Treem and Leonardi, 2012) and, by extension, social life (Costa, 2018). At the same time, we suggest that social perception of what is – and what can be – is underpinned by the concept of visibility, as described by sociologist Andrea Brighenti (2007; 2010). To understand visibility past the optical and as a social category, Brighenti contemplates how managing visibility, ‘lies at the core of all forms of social control, whether formal or informal’ (2010: 148). To explain further, consider that managing visibility delineates between visibilities of recognition and visibilities of control. Recognition of another self-consciousness bestows emancipatory effect. That is, when humans are ratified by others as human, they gain equal dignity (Brighenti, 2010: 40). We extend this recognition to post-human considerations and new-materialist concerns that reflect on the physical digitalities that are emerging across media and humans. We must recognize how the non-human renders itself visible and in turn contributes to the rendering of our own digital selves and lives in new ways. Our use of physical digitality mirrors, but is distinct from, the concerns that build on what Betancourt (2013) calls immaterial physicality. Like us, Betancourt is interested in ‘objects [that] reflect digitally-derived features displaying the existing capacities (both current and historical) of digital technology’ (Betancourt, 2013: n.p). Yet we do not see this as an illusion that suggests Marxist ‘reification becomes realization’ via ‘what was immaterial [crystallizing] into solid, tangible form’ (Betancourt, 2013: n.p.). Instead, we embrace the critique that if all mediation is material – from cuneiform to code – the more interesting question then is considering how experiences change when we accept that post-human objects mediate their existence as part of complex relations. Recognition does not give the immaterial physicality. Rather, we suggest visibilities of recognition make tangible the digitality of objects and persons that we relate to.

PHYSICAL DIGITALITY

189

Where Betancourt and ‘immaterial physicality’ might have stronger alignment is in visibilities of control. This differs from visibilities of recognition because conferred is a ‘purposeful and contextual asymmetrisation [SIC] and hierarchisation of visibilities’ (Brighenti, 2010: 148) that both suppress and clarify what is possible. Reification indeed. For example, certain language/identity/technology is used to describe a problem and its solutions over possible others (in a simple overview of one example is the idea of privacy being variously defined as a right, a feature or a flaw). More than a question of discourse, visibilities of control in the digital age are definitive of what can be perceived and produce physical digitality as a result. Visibilities of control through digitality is affirmed through Franklin’s (2015) take on control, which he suggests renders the world legible, recordable and knowable, insofar as it presents the fundamental conditions for computing and programming life. According to Franklin, the logic of control describes a wholesale reconceptualization of the human and of social interaction under the assumption … that information storage, processing, and transmission … not only constitute the fundamental processes of biological and social life but can be instrumentalized to both model and direct the functional entirety of such forms of life. (Franklin, 2015: xviii) While this accurately describes one logic and effect of digital physicality, it remains closed to the creative and emancipatory role that Brighenti suggests is possible through visibilities of recognition. In fact, Brighenti’s (2010: 70) study of visibility as a social spectrum explicitly tries to avoid such pitfalls of ‘essentialism-as-reification’. Or on another track, if Franklin hopes for some sort of analogue escape from the digital, dependent on obscurity, opacity (Blas, 2016) or obfuscation (Brunton and Nissenbaum, 2015), then users must still at some point acknowledge each other to enable social relations – and these will be material. In this way the dualisms apparent in the disjuncture between ‘real’ and ‘virtual’ are an artefact of earlier technological/cultural moments in which the digital did not embed within and provide a conduit for everyday life, up to and including the embodied experience of the human body (Vincs et al., 2018). Thus, this chapter offers consideration of how visibility is managed in acts of recognition and control observed in digital physicalities through perceptions of touch and sight to show life and living are rendered perceptible and manipulable in new ways. The chapter presents two distinct categories of perception as we consider the argument of making reality figuratively tangible (tangible: it can be laid hold of or grasped by the mind or dealt with as a fact that can be realized or shown to have substance; palpable; Oxford English Dictionary, 2020) through multimodal digital affordances for human perception across touch and sight. For touch we consider 3DP as a communication platform rather than a manufacturing process. With 3DP media we make

190

MATERIALIZING DIGITAL FUTURES

the digital physical and re-embody not only bytes with bits but, in some instances, our own bodies in tangible, data-rich ways. These physicalized data can for instance make various maladies and pathologies both literally and conceptually graspable for patients and practitioners, offering new categories of recognition and control (Aimar et al., 2019). Beyond 3DP, we can also reconsider what and how touch makes things (objects, UI elements) perceptible; we tap, swipe and pinch screens, press keys and gesture with spatially located controllers in mediated facets of communication through various devices, even if these interactions are often constrained to 2D contexts. Yet, new forms of spatiality – whether delivered by sound, light or other environmental stimulus – offer distinct if related modalities with which we construct digital physicality. If 3DP manifests digital data as physical and tangible, AR presents the inverse by allowing perception of relatable virtual forms in the physical world. For sight, we comment on how AR’s construction of environment informs (social) identity. That which users perceive in AR and that which is made manipulable are not just an overlay but an augmentation of the relations (Schraffenberger and van der Heide, 2014) between the things that comprise reality. Further modes of perception, including spatially computed sound, are beyond the scope of this chapter, but warrants further research in extended forms. Here we consider how digital physicalities can be explained through sight and touch; how our digital present is materially perceived; and how these considerations might allow a reconfiguration of social visibility potentials of the humans operating as part of the system. This chapter seeks to address the entanglements of humans and nonhumans within the discourse of new materialist media studies, and situates our cases mediated through 3DP and AR. The entanglements present are often defined through digital networks designed to spread data. Identifying and observing what data is spread to whom (e.g. users and communities) or what (e.g. corporations, institutions and government entities) enables the creation of the perceivable world for humans by making certain things visible, and others less so. As Latour (2007: 139) configures it, materiality can be both ‘the way we move knowledge forward in order to access things that are faraway or otherwise inaccessible; and, second, the way things move to keep themselves in existence’. Humans and non-humans alike contribute to the building of reality from both of those threads. Specific to our focus, digital networked media and their users are creating a world of things that jump the air gap to become embedded in perceptions of that which comprises physical life. These perceptions are emergent insofar as they create meaning through the richness of the physical and human intellect (Hayles et al., 2002: 32). Thus, materiality here is considered through human strategies of action concerning a spectrum of perception, from what is manipulable to conceptual frameworks that interact with the potential resources that artefacts afford. Our argument does not, however, interpret

PHYSICAL DIGITALITY

191

the world of physical (and digital) things as texts (Hayles, 2002), or consider the normalization of digital methods subsuming equivalent physical tasks (Bridle, 2018: 43) in the context of immersive reality technologies. Instead, we see the interactive and relational set of potentials of things in a manner similar to how media scholars have described the social-contextual use of digital systems as affordances in context (Costa, 2018). These affordances are both restrictive and creative, offering a type of governmentality that both provokes and guides the potentiality of what can be mediated into existence (Heemsbergen et al., 2016). They are also relational insofar as they acknowledge the active capacities of things – human and otherwise – in relation to each other. They represent the digital physicality of life, lived in new forms of recognition and control.

Touch Physical digitality is developed through touch by first considering how 3DP makes the digital physical and can re-embody our own physical presence in tangible, data-rich ways. We also reconsider what touch makes perceptible via touch-interactive media and other digital media that consider humans’ own digital dexterity as part of how their interaction can fabricate reality. It is commonplace to categorize 3DP as a manufacturing technology, where computer-controlled machines construct shapes additively, layer by layer, from a digital design file. Digital bits represented by algorithmic curves and pixels on a screen are transfigured to have volume (voxels) and then are remediated in physical matter, commonly as nylon, PVA or other polymers through to metals, wood derivatives and organic matter (Yoo, 2015). However, framing 3DP solely as a manufacturing technology diminishes its capacity as a networked communication technology that remediates networked programming into code and then into tangible physical outputs. We note that the productive and political economies of digital (abundance) and physical realms (scarcity) make for complex legal, social and environmental practice that remediate extant power structures to configure what has so far been a 3DP revolution underwhelmed (Daly, 2016). Nevertheless, 3DP showcases the evolving potentials of physical digitality across various industries and concerns, ranging from the mundane to the surprising. Remixed designs of phone cases (Heemsbergen et al., 2016) seem to be the most shared if mundane items on Thingiverse.com, the largest repository of 3DP object designs. Simultaneously, sites like dildo-generator. com offer custom manipulation of physical forms that craft the most surprising personal experiences. Between these extremes, the kinetic end of 3DP is made visible across computer aided design (CAD) files for firearms, the regulation of which remains politically fraught (Daly, 2018). The physical touch afforded by 3DP goods can offer an extension of self from sexuality

192

MATERIALIZING DIGITAL FUTURES

to customization and personal aesthetic, yet like any other media these artefacts are also discrete things that spread out – in the Latourian sense – to digital networks, intellectual property regimes and silicone substrates. At a structural level, 3DP critiques the logics of scarcity with digital abundance to entangle concerns and economic logics that threaten to upend extant models of intellectual property (IP) and redefine what it means to create, own, author or share (Daly, 2016; Weinberg and Knowledge, 2010). Further, the decentralized nature of 3DP production, from design to manufacture, suggests post-industrial relations of production and consumption that make visible paths and partnerships typically utilized and found in the domain of open source software design (Bauwens, 2013) yet is hampered by current IP models (Heemsbergen et al., 2016; Heemsbergen and Fordyce, 2019). This means that in the context of touch the physical digitality of 3DP objects affords alternative fundamental processes of recognition and control for the creation, sharing and use of physical things in our everyday. Here we see the logics of the digital entangling with the physical world to make visible a new environment to be accessed and perceived. While potential futures envisioned here are diverse (Birtchnell et al., 2018), we focus on the compelling, already tangible case of 3DP in the medical industry. The medical industry provides an insightful set of 3DP practices that help explain what digital physicality makes visible and its implications. Take, for instance, the radical transformation of relations between the physical and digital readily apparent in medical devices, operating guides and teaching aids that present our own bodies in new material forms to new audiences. As one specific example, 3DP is used by medical practitioners for surgical simulations of high-risk and low-frequency procedures, forming objects that often mimic patient-specific biology (Chen et al., 2020). This requires scanning, computing and printing matter to afford new types of bio-digital precision. This physical digitality literally embodies ‘bodies’, but also embodies networks of pathology, privacy and industry that come together in new realms of what Cohen (2018) terms a bio-political public domain. 3DP mediations of self rely on new communicative relations between doctors and patients, and the subsequent configurations of medical practitioners, industry and regulation. Physical digitality for 3DP goods in the medical space deals not only with privacy and communication issues that digital sharing affords, but also the highly regulated medical governance that balances health and safety with extant political economies of device manufacturers and regulation jurisdiction (Heemsbergen and Fordyce, 2019). In this sense, the 3DP medical object that mediates a patient organ in a new physical form makes visible a whole different set of concerns and connections. Each printed object offers us (as users, designers and developers) an artefact of programmable media in direct, tangible and manipulable forms. Each reveals the digitality of life – both new capacities and reforming

PHYSICAL DIGITALITY

193

constraints – that ‘explodes-out’ in Latourian terms from what seems to be an object in hand. Consider the surgeon who prints a model of a patient’s lung to show the extent of tumour growth and uniquely planned surgical approach. In the patient–doctor relation, the object offers new material forms of recognition and empathy (we see here your tumour, and can explain our shared relations to it), as well as control (this is an operation where instruments will cut you and remove the tumour as shown). Moving beyond the doctor–patient relation, the object also makes visible new forms of data-tracking (with resulting issues for privacy) though the decentralized and data-rich manufacturing process. For instance, the ‘immaterial’ supply chain of 3DP objects makes it easy to ask (and answer) questions that were previously elusive. For example: How many tumours look like this? Are there algorithms that can guide the surgical approach and offer successful operable outcomes to patients? How does the new information influence insurance premiums? Will printing patients’ own biological materials be considered an act of/issue for manufacture, communication or embodiment? What governance is required? This example object affords not just its base use-case of operating guide or doctor–patient communication aid, but a network of affordances in context that concern the thing as it is linked to a digitally enabled network of things: this is digital physicality.

Other physicalities of touch Apart from graspable 3DP products, the physical digitality of touch is ‘felt’ in other ways that amend how we can think about touch in digital media. Swiping across screens has, since the introduction of the iPhone in June of 2007, irrevocably changed the perceived capacities of manipulating pixels displaying programmable data. Billions of humans (3.8 billion smartphones in use as of 2020) are now accustomed to the fluid manipulation by humans of computed shapes across an array of pixels using nothing more than their fingertips. Prior to 2007, touch interfaces were used in niche and commercial/ industrial applications – such as Hewlett Packard’s HP150 in 1982, which used infrared sensors to measure fingers across the screen but had a user interface that failed to make data figuratively tangible; it achieved limited commercial success as a result. The more recent tangibility of the responsive touchscreens and graphical user interfaces (GUI) allow users to interact in a more direct sense with their digital spaces. Mediating touch by a flat interface, and/or motionbased interaction at a commonly accessible level, affords specific Human– Computer Interaction (HCI) methods that are replete with new capacities and constraints. This touch-based mode of interaction made computing and programmed environments more immediate, visible and accessible in a way previously unavailable in digital media. Touch-centric user interfaces

194

MATERIALIZING DIGITAL FUTURES

abstract the underlying code, data and technological complexity with which applications and experiences are built to a greater extent than keyboard, command line or mouse and windows. The rapidity with which young children can adapt to a touchscreen device use demonstrates the increased accessibility of the digital afforded by touch (Lovato and Waxman, 2016). Finger movement is more accessible to computers and this ‘more natural’ interface gives greater access for the majority of humans, providing visibilities of recognition and control. We point out that the digital physicality of touch is present in manifestations outside touchscreens and 3DPs. A simplistic environmental digital physicality is introduced to preschool-age children with toys that use physical environmental cues to spatially manipulate robotic computer objects as a precursor for learning object-oriented programming (Shumway et al., 2019). In the arts, the touch of digital physicality can, as Michailidis et al. (2018) capture, involve transposing dancers’ movements into tactile feedback to inform pianists in real time, who are themselves part of a joint performance. In this sense touch can surpass the 2D touchscreen, or even the three dimensions of 3DP objects. Touch extends to multimodal inputs and feedback systems for (post)human understanding. Most feedback for touch, however, is tied to human digital dexterity. For instance, HaptX mechanical glove technology (Perret and Vander Poorten, 2018) allows users to sense texture through touch, making perceptible the physical characteristics of a virtual object that would otherwise be manipulated through a less feedback-dense means of interaction like a touchscreen or game controller. Further, techniques involving ultrasound frequency manipulation can create standing waves of sound that allow touch sensations to feed back from the digital to physical sensations discernible by human skin (Carter et al., 2013). Ultrasound-based haptics, by the company Ultraleap, combine dextral surveillance with targeted resonance of ultrasound to create discrete feelings of touch for each finger. Yet even traditional game controllers acknowledge the opportunity of touch in mediating the digital. The announcement of the PlayStation 5 controller featured an increased focus (compared to its predecessor) on touch feedback (PlayStation 5, 2020), providing a higher resolution of haptic resistance and feedback to actions that might warrant it, such as the draw of a bow or the pull of a lever. According to Sony, the goals centred on ‘sense of touch within gameplay’ so that the perception for users is that ‘the controller is an extension of themselves when they’re playing – so much so that they forget that it’s even in their hands!’ (PlayStation 5, 2020). But the marketing goals of haptic touch speak not just to video games as digital objects attempt to be recognized by and offer relations of control with humans. The sophistication of the human hand in terms of dexterity, sensation and grip has long influenced how mechanical interaction takes place where digital things are concerned. A genealogical example are British military

PHYSICAL DIGITALITY

195

aircraft that pioneered digital systems of targeting and flight control in the 1960s. They introduced digital touch interfaces of control mapped to keep ‘hands on throttle-and-stick’ (HOTAS) at all times. On these control surfaces, fingers interact by touch without the need for a visual check in a way that affords pilots increased situational awareness. While these digital buttons might seem ubiquitous now (see discussion on PlayStation or car steering wheels), they presented a novel way to program the digital by touch compared to punch cards, flip switches or keyboards. The button as an interaction device evoked binary control of the digital with carry-on affordances outside of the discrete system at hand (Plotnick, 2018). More cumbersome than the button are whole-body haptic training systems, such as the immersive fire-fighting training system Flaim Trainer that can provide temperature-based feedback by increasing the internal temperature of the suit as a user moves closer to a virtual fire (Nahavandi, 2019). As new consumer and industrial applications (across the growing immersive reality sector, the games industry and through advancement of smart device technology) are developed and reach their respective audiences, feedback and control mechanisms that more directly engage a digital physicality of touch will again redefine how the digital is both controllable and recognizable. The capacity to feel the ‘shape’ of certain characteristics of data – potentially enhancing the way in which users can perceive it – presents different ways to understand, interpret and interrogate vast and complex data sets. Such modes of engagement might enable a more meaningful human presence within the programmed networks of media as they expand to describe more richly (and ascribe meaning to) the lived environment. At the same time, the capacity for such touch to affect the human compounds notions introduced by Hayles (2002). Until these strange new co-evolutions become, for better or worse, common, many ‘HP-150s’ will offer strange glimpses – and touches – of our malleable future. Interestingly, the nascent user interface (UI) of spatial computing is commonly mediated by existing grammars of touch in 2D spaces. That is, the lexicon of touch, tap, swipe and pinch transmute into push, pull and rotate to encompass a range of basic interactions that AR applications afford a user such that 2D gestures (constrained by the flat interface) are effectively re-contextualized for 3D (Ze Dong et al., 2020).

A summary of touch The physical digitality of 3DP objects encompasses more than just technological transmutation from bytes of data into bits of matter. This transmutation makes visible the extended relations of potential these digital– physical things exhibit through physical digitality. They make cultures and systems of production knowable and manipulable in ways that mirror the fundamental conditions of networked computing. Yet, at the same time, as

196

MATERIALIZING DIGITAL FUTURES

media, these objects offer extensions of self from the banal and generalizable to the customized and uniquely personal. As Latour (2007) might interpret it, the graspable touch afforded by 3DP sees knowledge being accessed in ways that were previously considered remote. Further, creating and sharing 3DP objects change what can be perceived both of self and of wider contexts via the decentralized geographic and political-economic reach of 3DP practices. At the same time, these newly printed things require a cacophony of digital movement to exist as they are conceptually, physically and structurally with specific relations of power and control. Perceptions of touch through 3DP make visible many new digital ways of being. Examples might include industrial safety and training applications, or free and web-based software–hardware experience that offer user-friendly and user-designed 3DP sex toys. In the medical space, cranial inserts that perfectly match patient anatomy or printed models that inform preoperative communications and planning between patients and doctors open new ways of perceiving self and life – and extending both. Socially and economically speaking, this changes things. It changes expectations about scarcity and abundance (potentially problematic if we expect to just print another). It changes capacities for bespoke personal extensions of self. It changes practices of computer aided and iterative design across socio-technical complexities of law and regulation, and systems of production. It offers a new chapter in the physical digitality that we live in. In short, what 3DP makes visible in fundamental physical forms are fantastical objects created by digitality that otherwise could not exist. These things offer a material life made possible only through tendrils of networked data siphoned through programmed pathways to reshape perceptions of what is and what can be. Further, their touch brings knowledge within our grasp from what was once a remote digital realm. Graspable, physical and manipulable objects tell digital stories; their digitality make visible networks of meaning, capacity and constraint around what is and what can be. They organize and create the digital physicality of life.

Sight If 3DP offers new ways to perceive life by creating material objects imbued with digitality, AR allows us to perceive life through virtual forms of physical environmental data. For sight we comment on how AR’s construction of potentials of relation within an environment (Schraffenberger, 2018), creates physical digitality. First, we must consider what AR makes visible, and the boundary work that goes into what AR is and is not (Liao, 2016). The answer shared by pioneers of AR technical processes suggests that AR ‘allows the user to see the real world, with virtual objects superimposed upon or composited with the real world’ (Azuma, 1997: 356). However,

PHYSICAL DIGITALITY

197

more than an overlay, we can consider it to be a spatial environment wherein information is dynamically changing as per user (Manovich, 2006: 219). What users perceive in AR and what is made manipulable are not just an overlay outside and discrete from the real, but an augmentation of the relations between the things that make up phenomenological experience – the physical and digital are both manipulable and interpretable in AR media. This digital manipulation of the physical environment might tend towards the informational or perceptual. In the informational sense, numbers, signs or symbols can augment informational meaning to enhance what is seen. While the display technologies that enable AR ‘overlay’ this information to corresponding physical, environmental features, what is perceived in situ is the digitality of the spatial environment through dynamically changing information. For instance, in dystopian fictions of AR informational overlay (see the works of Matsuda, 2016 and May-raz and Lazo, 2012), everyday users are inundated with perceptions of in real life (IRL) pop-up advertisements that afford visibilities of control over other humans as well as against the users themselves. Non-human systems add information into these worlds, but are not seen themselves (Katell, 2019). Currently, as the world grapples with waves of Covid-19, we can imagine futures where individuals’ pathogenic/ immunization status is broadcast and visible to users with requisite hardware and social capacities. The implications of such technology being ubiquitous are worth considering; visibilities of recognition between citizens that are ‘safe’ to interact with could allow more efficient social distancing or improved application of other mandated non-pharmaceutical interventions that reduce spread of the virus. At the same time, these technologies cast a hierarchical visibility of control. Social rejection of or anger towards citizens who are involuntarily identified as infected creates an underclass of pariahs or targets for retribution for the social and economic impact that current suppression and recovery strategies have on a populace (O’Sullivan et al., 2020). However, dynamic information, which users must address through additional cognitive capacity, is just one option of what AR can make visible. We must also consider subtle perceived changes to the physical environment that do not increase cognitive load but do augment what is perceived and can be acted upon and interacted with. Raja and Calvo (2017) consider this option to explain AR through an ecological lens, suggesting users do not experience AR objects themselves in the real world, rather they experience the relational properties derived from augmentation of the world. A simple example of the informational capacity of an AR are the ‘stickers’ appended to the heads of users in hand-held video calls, allowing users to add to or modify their features with an aesthetic informational signal not-oftheir physical traits. These stickers augment reality insofar as they add information to experience from collected facial geometry data, computed display characteristics and displayed real-time information additions to the user’s face that otherwise would not be there. They also mediate emotion/

198

MATERIALIZING DIGITAL FUTURES

informational need/expression through the addition of signs/symbols. Snapchat, TikTok and Facebook (to name three current platforms among a rapidly increasing number) all provide these AR sticker overlays. The environmental augmentation that does not add information but does augment relational perceptions works differently. For example, Apple has implemented a subtle form of environmental augmentation in its FaceTime software as of late 2019. The video chat software shifts how it displays users’ gazes so that communication partners perceive each other to be looking at the camera/each other, when in fact they are looking below the camera at their screen-mediated image. Apple augments the environment of the face to facilitate communication potentials of eye contact, where there was previously none, signalling a new affordance of AR. Here, a mediated layer of reciprocal recognition is afforded through real time environmental manipulation that does not ‘add’ data or ‘overlay’ ‘virtual’ objects. The experience is user recognition of each other made visible in a new and, potentially, socially desirable way. This is a very simple example of how shifts in perception and expectation, rather than information, can manage visibility in AR. The digital shifts of user physicality in the Apple example shows that rather than merely an overlay, AR provides an augmentation of potential relations between the things that make up reality (Schraffenberger and van der Heide, 2014). To explain the difference in capacity of informational and environmental augmentations in the pandemic example, imagine navigating a city on foot where clusters of disease have sprung up. In our example, enhancing the relations of walkability of the city to public health could be defined through not straying through clusters of disease. In one instance, AR could make relevant informational warning signs, alerts or arrows for cognitive interpretation to avoid certain paths. Yet, at a perceptual level, other augmentations of the environment can make a safer path visible without the informational clutter. Rather than signs and symbols, darkening or defocusing some routes in comparison to others (Raja and Calvo, 2017) makes relations between walkability and public health, in our example, ‘perceptually salient’. Occluding or defocusing paths to avoid manages visibility in a specific way that has political and social consequences by literally and figuratively making some things known as bright and vibrant, whereas others fade from the light. Or as Bridle (2018: 36) puts it, ‘for everything that is shown, something is hidden’. On the technical level, the occlusion or defocusing textures do not hold meaning in themselves, even when attached to this wall or that alley. Their physical digitality – their meaning making in and of the social world – is experienced only in their relational properties to other datasets (geographic information system or GIS aware bio-pathogen risk data), physical structures of the built environment, human wayfinding paths through it and sociopolitical constructions of risk. Whether informational or perceptual, the sight that AR offers allows new relations to be made visible, actionable, and relatable between humans via their devices.

PHYSICAL DIGITALITY

199

Beyond the subtle example of augmented eye contact in video chat and fanciful notions of augmented quarantine, the gradual normalization of AR has already exposed various spatial ramifications of digital physicality. For some time, digital artists have been subversively augmenting public and private space with virtual art installations that are accessible to users of AR hardware, although these activations are effectively ungovernable by the institutional or public spaces they ‘inhabit’. Skywarek (2018) links the hackability aesthetic from disclosing digital secrets to producing ‘secret’ manifestations of activist art that transfigure the visibility of corporate logos (British Petroleum spewing pollution) and political spaces (Wall Street occupied with anti-capitalist provocations), or to ‘deleting’ structures of separation (in Israel and North Korea) and environmental artefacts (e.g. visualizing receding glaciers, light pollution and so forth). AR changes how these struggles are visible not only in the literal sense through an augmented relationship to an environment, but in terms of the categories of recognition and control that make up visibility as a social category. Control of space through Newtonian hierarchies and the political hierarchies of who governs the use of spaces are both upended by AR activists ‘hacking’ political sites like Wall Street. At the same time, new forms of recognition are available by perceiving things in and of the physical environment that were formerly not possible, by watching glaciers recede or walls dissolve to explicitly perceive life without these distinctive signifiers. In this way, AR affords visibility towards how social life has developed one specific set of relations to define a problem, instead of another. These art and political AR projects make governmentality visible by helping us perceive the rationalities and technologies of government (Rose and Miller, 1992) by showing their challenges in interactive, real-time spatial computing. As a final example, which shows how the visibilities of control and recognition are not necessarily in opposition to one another, consider the NAMAland project. NAMAland was an AR experience for mobile phones built on top of the Layar platform that placed the ‘monopoly man’ symbol on certain buildings. Each of these properties was under the control of Ireland’s National Assets Management Agency as a result of the 2009 financial collapse. As McGarrigle (2018: 121) summarizes: Despite (or perhaps because of) its pivotal role in the financial collapse, NAMA was very secretive in its workings … It became obvious that mapping NAMA’s property holdings was essential to gain an understanding of the organisation, and the events which led to its creation, in order to open it to scrutiny and critique. At the same time that mutual recognition was being made visible between citizens and what their government owns as part of a mutual political public, these recognitions then exposed and translated NAMA’s otherwise

200

MATERIALIZING DIGITAL FUTURES

obscured activities, rendering their world legible, recordable, and knowable. What these examples share is a desire (on the part of the authors) to imbue the physical world with digital manipulability. The ease of cut, copy, paste, manipulate and share has been extrapolated to inform and augment human perception of the physical environment. Exploring what complex or native affordances of AR media (see McGarrigle, 2018) are and offer remains under-theorized and presents opportunity for future research trajectories.

A summary of sight For sight we comment on how AR’s construction of environment informs social space and identity through physical digitality. The social recognition and control of elements that make up space – those Latourian things that define a space and our place in it – are what AR can make visible in ways that were not available in previous media. We should also note that augmenting the physical digitality of environments and objects should not be understood as limited to sight. Sound has been used to offer new relations to place and spatiality specifically (implicitly or explicitly) in ways meant to reconfigure subjectivities to institutional narrations of a location (Soria-Martínez, 2017). As the ubiquity of the relevant technologies that shift visibility towards physical digitality increases, new questions will arise. For instance, Chang (2019) considers how the digital gaze of algorithms augments traditional hierarchies of power while continuing to threaten autonomy with automation. More pessimistically, Bridle (2018: 43) describes an increasing infringement on the physical world by seductive digital manifestations of formerly physical interactions. The computerized reality we inhabit and the increasing capacity to perceive the digital in the physical is made explicit through the physical digitality of AR. While technologists envision a Mirror World (Kelly, 2019) of virtual twins for the real world as key to achieving ubiquity of spatial computing in reality, we disagree. Will we reach a point where people understand and perceive their place to be in a virtually real reality more than a physical one? Future research might consider not only resistance to these trends, but activist activation of the new affordances of sight offered through a lens of physical digitality.

Conclusion Physical digitality offers a way to think about the capacity and unique constraints to our perceptions that come with digital media being integrated into everyday life in multimodal forms. The feel of 3DP goods, the tactility of manipulating software on a screen and the barely perceived environmental cues that guide AR experience, each present physical manifestations of digital logics, economies and programmable media. While this chapter’s

PHYSICAL DIGITALITY

201

focus on touch and AR has largely defined scope in terms of the body and individuals’ perception of the post-human environment, digital physicality renders at multiple scales. The limits of such physicality scale from the active nano matter (Tibbits, 2017) to Geomedia (McQuire, 2017) as new forms of programmability are rendered networkable at scale. In between are examples like Zurich and Hong Kong linking via the Z/Z Twin Lab as ‘entangled co-actors’ (Mansbridge, 2020: 88) by exchanging performance and data in real time to expand and animate ‘interactions among cultures, bodies, technologies, and spaces’. These digital physicalities bring into existence what we can make ourselves to be. They are coded and programmable yet at the same time are manipulated in perceptual human space and cadence; physical digitality requires affordances that humans can grasp tangibly. As such, it also offers, as a hope, ways to be more aware of our post-human environment. Lupton (2019) suggests that the quantification of self, despite political-economic critiques, does lead people to be more aware of their selves. Critiquing digital physicality within the media which exhibits it, allows an extension of Lupton’s point from the self to the environment. This includes the political affordances that flow-on from making the digital aspects of things visible. Specifically, the instances of physical digitality explored in this chapter have opened for consideration how visibility can be treated as a social category that entices both recognition and control through objects’ and environments’ digital networks. Whether media that is printed in plastic, or augments the physical, the perceptions of digitality is seeping past a firewall of ephemeral ones and zeros to embed in perceived reality in novel ways. To conclude then, we might return to Hayles (2006: 32) suggestion that materiality as an emergent property is the dynamic interplay of a rich physicality and human intelligence as a way to create meaning. When we apply this to digital physicality, we see that the matter we make visible matters in terms of the digital networks that connect the datafied reality we now inhabit.

References Aimar, A., A. Palermo and B. Innocenti (2019), ‘The Role of 3D Printing in Medical Applications: A State of the Art’, Journal of Healthcare Engineering. Available online: https://doi.org/10.1155/2019/5340616. Azuma, R. T. (1997), ‘A Survey of Augmented Reality’, Presence: Teleoperators & Virtual Environments, 6 (4): 355–85. Bauwens, M. (2013), ‘Thesis on Digital Labor in an Emerging P2P Economy’, in T. Scholz (ed.), Digital Labor: The Internet as Playground and Factory, 207–10, Oxfordshire, UK: Routledge. Betancourt, M. (2013), ‘Automated Labor: The “New Aesthetic” and Immaterial Physicality’, CTheory. Available online: http://www.ctheory.net/articles.aspx? id=717.

202

MATERIALIZING DIGITAL FUTURES

Blas, Z. (2016), ‘Opacities: An Introduction’, Camera Obscura: Feminism, Culture, and Media Studies, 31 (2) (92): 149–53. Birtchnell, T., A. Daly, T. Rayna and L. Striukova. (2018), ‘3D Printing and Intellectual Property Futures’, Final report for project commissioned by the UK Intellectual Property Office (2018/03). Bridle, J. (2018), New Dark Age: Technology and the End of the Future, London: Verso Books. Brighenti, A. M. (2007), ‘Visibility: A Category for the Social Sciences’, Current Sociology, 55 (3): 323–42. Brighenti, A. M. (2010), Visibility in Social Theory and Social Research, New York: Palgrave Macmillan. Brunton, F. and H. Nissenbaum (2015), Obfuscation: A User’s Guide for Privacy and Protest, Cambridge, MA: MIT Press. Carter, T., S. A. Seah, B. Long, B. Drinkwater and S. Subramanian (2013), ‘UltraHaptics: Multi-Point Mid-Air Haptic Feedback for Touch Surfaces’, Proceedings of the 26th Annual ACM symposium on User Interface Software and Technology, October, 505–14. Chang, V. (2019), ‘Catching the Ghost: The Digital Gaze of Motion Capture’, Journal of Visual Culture, 18 (3): 305–26. Chen, G., M. Jiang, J. Coles-Black, K. Mansour, J. Chuen and D. Amott (2020), Three-Dimensional Printing as a Tool in Otolaryngology Training: A Systematic Review’, Journal of Laryngology & Otology, 134 (1): 14–19. Cohen, J. E. (2018), ‘The Biopolitical Public Domain: The Legal Construction of the Surveillance Economy’, Philosophy & Technology, 31 (2): 213–33. Costa, E. (2018), ‘Affordances-in-Practice: An Ethnographic Critique of Social Media Logic and Context Collapse’, New Media & Society, 20 (10), doi: 1461444818756290. Daly, A. (2016), Socio-Legal Aspects of the 3D Printing Revolution, London, UK: Palgrave Macmillan. Daly, A and M. Mann (2018), 3D Printing, Policing and Crime. Available online: https://eprints.qut.edu.au/119689/. Franklin, S. (2015), Control: Digitality as Cultural Logic, Cambridge, MA: MIT Press. Haraway, D. (1985), ‘A Manifesto for Cyborgs: Science, Technology, and Socialist Feminism in the 1980s’, Socialist Review, 15 (2): 65–107. Hayles, N. K. (2002), Writing Machines, Cambridge, MA: MIT Press. Hayles, N. K. (2006), ‘Unfinished Work: From Cyborg to Cognisphere’, Theory, Culture & Society, 23 (7–8): 159–66. Heemsbergen, L. J., R. Fordyce, B. Nansen, T. Apperley, M. Arnold and T. Birtchnell (2016), ‘Social Practices of 3D Printing: Decentralising Control, and Reconfiguring Regulation’, Australian Journal of Telecommunications and the Digital Economy, 4 (3): 110–25. Hyper Reality (2016), Film, dir. Keiichi Matsuda. Available online: http://km.cx/ projects/hyper-reality (accessed 17 September 2021). Jurgenson, N. (2011), ‘Digital Dualism versus Augmented Reality’, Cyborgology. Available online: https://thesocietypages.org/cyborgology/2011/02/24/digitaldualism-versus-augmented-reality/ (accessed 23 July 2020).

PHYSICAL DIGITALITY

203

Katell, M., F. Dechesne, B. J. Koops and P. Meessen (2019), ‘Seeing the Whole Picture: Visualising Socio-spatial Power through Augmented Reality’, Law, Innovation and Technology, 11 (2): 279–310. Kelly, K. (2019), ‘AR Will Spark the Next Big Tech Platform – Call It Mirrorworld’, Wired, 12 February. Latour, B. (2007), ‘Can We Get Our Materialism Back, Please?’ Isis, 98 (1): 138–42. Liao, T. (2016), ‘Is It “Augmented Reality”? Contesting Boundary Work over the Definitions and Organizing Visions for an Emerging Technology across FieldConfiguring Events’, Information and Organization, 26 (3): 45–62. Lovato, S. B. and S. R. Waxman (2016), ‘Young Children Learning from Touch Screens: Taking a Wider View’, Frontiers in Psychology, 7. Available online: https://doi.org/10.3389/fpsyg.2016.01078 (accessed 15 January 2021). Lupton, D. (2019), ‘“It’s Made Me a Lot More Aware”: A New Materialist Analysis of Health Self-Tracking’, Media International Australia, 171 (1): 66– 79. Available online: https://doi.org/10.1177/1329878X19844042. Manovich, L. (2006), ‘The Poetics of Augmented Space’, Visual Communication, 5 (2): 219–40. Mansbridge, J. G. (2020), ‘Choreographing the Nonhuman: Cross-Cultural Entanglements and Technologies of Capture in Zuni Icosahedron’s and Zurich University of the Arts’ Z/Z Twin Lab’, International Journal of Performance Arts and Digital Media, 16 (1): 88–103. McGarrigle, C. (2018), ‘Augmented Interventions: Redefining Urban Interventions with AR and Open Data’, Augmented Reality Art, 115–30, Cham: Springer. McQuire, S. (2017), Geomedia: Networked Cities and the Future of Public Space, Hoboken, NJ: John Wiley. Michailidis, T., J. Dooley, N. Granieri, and B. Di Donato (2018), ‘Improvising through the Senses: A Performance Approach with the Indirect Use of Technology’, Digital Creativity, 29 (2–3): 149–64. Nagy, P. and G. Neff (2015), ‘Imagined Affordance: Reconstructing a Keyword for Communication Theory’, Social Media+ Society, 1 (2), doi: 2056305115603385. Nahavandi, S., L. Wei, J. Mullins, M. Fielding, S. Deshpande, M. Watson, S. Korany, D. Nahavandi, I. Hettiarachchi, Z. Najdovski, R. Jones, A. Mullins and A. Carter (2019), ‘Haptically-Enabled VR-Based Immersive Fire-Fighting Training Simulator’, CompCom 2019: Intelligent Computing – Proceedings of the Computing Conference, 11–21, Cham: Springer, doi: 10.1007/978-3-030-22871-2_2. O’Sullivan, D., M. Rahamathulla and M. Pawar (2020), ‘The Impact and Implications of COVID-19: An Australian Perspective’, International Journal of Community and Social Development, 2: 134–51. Available online: https://doi. org/10.1177/2516602620937922 (accessed 15 January 2021). Perret, J. and V. P. Emmanuel (2018), ‘Touching Virtual Reality: A Review of Haptic Gloves’, ACTUATOR 2018; 16th International Conference on New Actuators, Bremen, Germany, 1–5. PlayStation 5 (2020), Blog, ‘Introducing DualSense, the New Wireless Game Controller for PlayStation 5’, Sony Corporation. Available online: https://blog. us.playstation.com/2020/04/07/introducing-dualsense-the-new-wireless-gamecontroller-for-playstation-5/ (accessed 9 April 2020).

204

MATERIALIZING DIGITAL FUTURES

Plotnick, R. (2018), Power Button: A History of Pleasure, Panic, and the Politics of Pushing, Cambridge, MA: MIT Press. Raja, V. and P. Calvo (2017), ‘Augmented Reality: An Ecological Blend’, Cognitive Systems Research, 42: 58–72. Rose, N. and P. Miller (1992), ‘Political Power Beyond the State; Problematics of Government’, British Journal of Sociology, 43 (2): 173–205. Schraffenberger, H. K. (2018), ‘Arguably Augmented Reality: Relationships between the Virtual and the Real’, doctoral dissertation, Leiden University, the Netherlands. Schraffenberger, H. K. and E. van der Heide (2014), ‘Everything Augmented: On the Real in Augmented Reality’, Journal of Science and Technology of the Arts, 6 (1): 17–29. Schrock, A. R. (2015), ‘Communicative Affordances of Mobile Media: Portability, Availability, Locatability, and Multimediality’, International Journal of Communication, 9: 18. Shumway, J. F., J. Clarke-Midura, , V. R. Lee, M. M. Hamilton and C. Baczuk (2019), ‘Coding Toys in Kindergarten’, Teaching Children Mathematics, 25 (5): 314–17. Sight (2012), Film, dir. E. May-Raz and D. Lazo. Available online: http://vimeo. com/46304267 (accessed 17 September 2021). Skwarek, M. (2018), ‘Augmented Reality Activism’, Augmented Reality Art, 3–40, New York: Springer. Tibbits, S. (2017), Active Matter, Cambridge, MA: MIT Press. Treem, J. W. and P. M. Leonardi (2012), ‘Social Media Use in Organizations: Exploring the Affordances of Visibility, Editability, Persistence, and Association’, Communication Yearbook, 36: 143–89. Vincs, K., A. Bennett, J. McCormick, J. B. Vincent and S. Hutchison (2018), ‘Skin to Skin: Performing Augmented Reality’, Augmented Reality Art, 195–209, New York: Springer. Weinberg, M. and P. Knowledge (2010), It Will be Awesome If They Don’t Screw It Up: 3D Printing, Intellectual Property, and the Fight Over the Next Great Disruptive Technology, Public Knowledge. Yoo, S.-S. (2015), ‘3D-Printed Biological Organs: Medical Potential and Patenting Opportunity’, Expert Opinion on Therapeutic Patents, 25 (5): 507–11, doi: 10.1517/13543776.2015.1019466. Ze D., T. Piumsomboon, J. Zhang, A. Clark, H. Bai and R. Lindeman (2020), ‘A Comparison of Surface and Motion User-Defined Gestures for Mobile Augmented Reality’, Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA ’20). Association for Computing Machinery, 1–8, New York, doi: https://doi. org/10.1145/3334480.3382883.

11 A true feel: Re-embodying the touch sense in the digital fashion experience Michela Ornati

Introduction Clothing ourselves is a communicative experience. Through dress, we express our identity, situate ourselves in the context of our historical time and social space, and contribute to the phenomena of evolving modes and styles called fashion (Kalbaska et al., 2018). Donning garments is also a bodily act – a multisensory experience triggering sensations and perceptions (Bruno and Pavani, 2018) in which touch, in all its complexity (Field, 2014), plays a central role. Thus, dress is an embodied practice (Entwistle, 2000; 2015) which affects emotional and psychological processes in distinct ways. Over the last twenty years, the rise of electronic, mobile and social media platforms have created new venues for clothing distribution, promotion, consumption and communication (Kalbaska et al., 2019; Rocamora, 2019). This process has digitized but also de-materialized dress practices (Shinkle, 2013). In fact, a fundamental difference between a brick and mortar apparel store and an e-commerce website is the physical accessibility of clothing. In the digital domain, clothes are visually accessible but cannot be touched or tried on; this leads to issues of size, fit and feel. This disembodied nature of the e-commerce fashion experience contributes to the costly phenomena

206

MATERIALIZING DIGITAL FUTURES

of product returns (Lieber, 2019). Thus, reintroducing touch is critical for the sustainable evolution of digital fashion. Indeed, touch sensations and its potential surrogates have fascinated scientists for over a century (Parisi, 2018). Today, haptic (touch) technology is a lively field of research. Haptic devices such as interactive touchscreens and wearables are increasingly present in consumer markets. Also, virtual and mixed reality developers are integrating haptics into their solutions to better approximate touch effects in virtual settings. This accelerating technological convergence points to a possible future of immersive virtual shopping in the home, on the go and in fashion outlets; a multisensory, re-embodied experience of dress which will affect the way clothing is chosen, purchased and consumed (Bonetti et al., 2018; Flavián et al., 2019). In this chapter I explore the constitutive elements and theoretical underpinnings of this scenario. First, I introduce the sense of touch, and how wearing clothes activates the somatosensory system. Then, I review two distinct, but related, theoretical approaches to embodiment focused on the effects of dress: the enclothed cognitive sciences approach and the embodied dress fashion perspective. Via these interpretive lenses, I discuss the physical experience of fashion retail practices and its disembodied counterpart in the online domain. Finally, I discuss haptic technology systems and devices, suggesting that technological convergence with virtual reality might change the way dress is embodied and experienced in digital fashion futures.

Clothes on the body: The sense of touch The sense of touch plays a fundamental role in development (Field, 2014), social communication (Jewitt et al., 2019) and in the exploration and understanding of the world which surrounds us (Fulkerson, 2014); its centrality in experience exerts a powerful influence on our lives (Linden, 2015: 18). Together with other sensory stimuli arising from vision, hearing, olfaction and the body itself, touch sensation contributes to the body’s cyclical, multisensory perceptual processes. The ‘optimal integration’ of these sensory stimuli in the brain (Bruno and Pavani, 2018: 103) gives rise to the ‘rich, nuanced perception’ we have of the world (Linden, 2015: 77). Consequently, its nature being both passive and active influences emotions, thoughts and actions, including decisions regarding our clothing choices (Peck and Childers, 2003a). In the following section I briefly illustrate, without exhaustive detail, the manner in which this highly complex sense functions, consequently suggesting connections to the everyday practice of dress. The sense of touch is a complex system1 of information gathering, also called somatosensory, from the Greek word soma for body. This system is distributed in almost the entire body: skin (mucosal, hairy and glabrous),

A TRUE FEEL

207

connective tissues (tendons and ligaments), and muscles. The type of information the system relays to the brain (Bossomaier, 2012; Linden, 2015) can be described in terms of the effects of wearing a particular garment, including proprioceptive signals (the pressure of tight jeans on the lower body), exteroceptive signals (external touch stimuli affecting the skin, such as the smooth or rough texture of cloth), thermal sensations (the warmth of cashmere or the coolness of linen on the skin), itch (the prickly stimuli of a rough wool sweater), pain (the effects of toes squeezed inside tight shoes), and inflammation (which might be induced by a reaction to textile fibres or to the chemical substances present in a garment (Ngan, n.d.). Despite the complexity of somatic input arising from the somatosensory system, the sense of touch is usually described primarily in terms of two modalities or states: kinaesthetic (proprioceptive) and tactile. Kinaesthetic refers to the sensation of forces and torques relayed by connective tissues and muscles. Tactile indicates sensations of pressure, vibration and shear picked up by highly specialized touch receptor systems. These systems are also called mechanoreceptors, because they transform mechanical energy into electrical signals which are then relayed to the brain via the corresponding nerve fibres embedded in the skin. Mechanoreceptors are particularly dense in the glabrous skin of the hand and fingertips, which explains the organ’s sensitivity to touch. Specifically (Linden, 2015: 92–107), Merkel disks ‘allow us to distinguish individual surface features’ like the fine ridges on a coin (or of a fabric weave); Meissner’s corpuscles enable us to ‘manipulate objects with delicacy, using the minimal force for the job at hand’ as in handling a coin (or holding a button); Pacinian corpuscles ‘provide a high-fidelity neural image of transient and vibratory stimuli transmitted to the hand by an object held in the hand’ such as a chisel, so that ‘we can perceive tactile events at the working end of the tool almost as if our fingers were present there’ (as when holding the sliding piece of a clothes zipper to open and close its two metal rows); Ruffini endings sense horizontal skin stretching. All four touch sensors are activated when grasping and applying force to an object, as would be the case when we grasp and manipulate a garment to put on, for example, a t-shirt or a shoe. Other non-mechanical touch receptors, such as free nerve endings, directly convey sensations of temperature, pain, itch and chemical inflammation. Finally, specialized nerve fibres, which innervate the hairy, non-glabrous skin, react to hair deflection (such as can be caused by a caress, or by fabric sliding on the body) giving rise to ‘multiple sensations’ which may or may not be emotionally neutral (Linden, 2015: 170). Touch is passive and active: we identify and perceive the properties of objects through explorative touch (Jones, 2018). Active touch is also referred to as haptics. Haptics derives from the Greek word haptikós meaning the ability to grasp or to perceive (Jones, 2018). Haptic sensing enables a bi-directional exchange of information with the environment, a unique property of the sense of touch as opposed to vision or hearing (Bayousuf

208

MATERIALIZING DIGITAL FUTURES

et al., 2019). Explorative touch is performed primarily through the organ of the hand, which unifies both motor and sensory functions (Hayward, 2018). Not surprisingly, it is the hand which is generally associated with sensing and manipulating textiles and garments, literally and metaphorically (as in expressing appreciation for fabric with the comment, ‘a nice hand’). Touch sensations unleash complex multisensory processes. Put simply, stimuli are transmitted via the nervous system (or other pathways) to different areas of the brain responsible for emotion, action and thought. ‘Ultimately’, states Linden (2015: 147), ‘representing the tactile world in the brain is in service of achieving some particular outcome: making a decision, forming a memory, or initiating an action’. A tactile representation of dress in the brain serves the same aims. To explore this concept further, I turn to research on the effects of clothing on the body, the mind and the wearer’s experience of the world within two very diverse fields of enquiry: the cognitive sciences, specifically research in embodied cognition, and fashion studies.

Embodying dress: Theoretical perspectives Embodied cognition theories are a perspective on the connection between bodily and mental processes which consider the physical body of an agent as relevant to cognition (Miłkowski et al., 2018). Theoretical approaches focus on the role bodily states (modalities) and sensorimotor abilities play in affecting cognitive processes. Grounded cognition is specifically concerned with the role of simulations, which are ‘the reenactment of perceptual, motor, and introspective states acquired during experience with the world, body, and mind’ (Barsalou, 2008: 618). Embodiment theories view cognition as being ‘constrained and enabled by the specific characteristics of our own brain-body system’ (Miłkowski et al., 2018: 2). They also acknowledge external factors, situating cognition within social contexts and stressing the role of the natural and cultural environments in which cognitive processes take place. Donning clothes is a tactile, bodily experience. Embodied cognition theories are, therefore, an appropriate perspective from which to study the effects that wearing clothes – and the context in which this process occurs – may have on cognitive processes. In fact, embodied (and implicitly grounded) cognition frames the theory of enclothed cognition proposed by social behaviourist Hajo Adam and social psychologist Adam D. Galinsky (Adam and Galinsky, 2012). In a series of experiments set up by the researchers, participants were given a plain lab coat to wear that could be interpreted as belonging either to a doctor or to a painter; then they were entrusted with a specific cognitive task. Based on their results, Adam and Galinsky argue that ‘actually wearing a piece of clothing and having the accompanying physical experiences (e.g. seeing it

A TRUE FEEL

209

on one’s body, feeling it on one’s skin, etc.)’ will make it ‘significantly more likely’ for the piece of clothing and its attendant symbolism to influence the wearer’s psychological processes and behavioural tendencies ‘above and beyond basic material priming effects’ (2012: 919). The embodiment of the clothing’s symbolic meaning, that is, ‘the co-occurrence of two independent factors: the symbolic meaning of the clothes and the physical experience of wearing the clothes’ (2012: 922) is therefore central to the enclothed cognition framework. To further the discussion, I wish to anchor the concept of embodied dress arising from the perspective of the cognitive sciences within the social sciences and more specifically, fashion theory. The work of fashion sociologist Joanne Entwistle serves that purpose. In her writings (Entwistle, 2000; 2015), the author constructs a theoretical framework centred upon dress as a socially situated, embodied practice. Fashion, she writes, ‘is about bodies: it is produced, promoted and worn by bodies’ (2015: 1) located within social, historical and cultural contexts which define ‘expectations about the body and about what constitutes a “dressed” body’ (2015: 11). But in the everyday practice of dress, of interacting with the body by clothing it, social norms are constantly renegotiated and redefined. For example, wearing a corset or bustier to achieve a fashionable hourglass-shaped figure was the norm in Western countries up to the early 1900s (Steele, 2001). But at the turn of the twentieth century, women’s emancipation movements and the introduction of the brassiere (the precursor of today’s undergarment) made softer silhouettes both desirable and socially acceptable (Steele, 2005: 188–92). Entwistle’s embodied and socially constituted dress framework draws from sociology, anthropology and philosophy (2000: 323–48). Her discussion of Merleau-Ponty’s phenomenology is especially relevant in the context of this chapter because phenomenology is, in some respects, the philosophical counterpart of embodied cognition: it emphasizes the embodied nature of subjectivity, meaning the body is an agent – rather than a passive object – in the active perception and experience of the self and its temporal and spatial location in the world. In Phénoménologie de la Perception (2009), Merleau-Ponty extensively investigates the subjective effects of passive and active touch, and in the context of fashion theory, his writings have ‘drawn attention to the ways in which body adornment is experienced by its wearers not just as a visual phenomenon but also as a haptic experience’ (Negrin, 2015: 122). For Entwistle, approaching dress from a phenomenological viewpoint means acknowledging dress as part of the experience of the body ‘which in turn works on and mediates the experience of self’ (2000: 334) in a specific time and place. In fact, identity is routinely ‘managed through dress’ (2000: 338) in a process of ongoing negotiation within socially constituted norms and codes. Connecting the enclothed cognition perspective to Entwistle’s theory of embodied dress

210

MATERIALIZING DIGITAL FUTURES

helps define and enrich the concept further: clothing the body is a (socially) contextualized experience which affects the wearer’s physical states and, consequently, psychological processes (the sense of self) and behaviour (the adherence or rejection of norms and codes of dress). In the Western world, the mundane act of embodying dress theorized by Entwistle, and the cognitive processes it triggers, usually begin in front of a closet or a chest of drawers. Garments and accessories stored therein, if not otherwise acquired, have been at some point purchased somewhere; that is, engagement with dress and its effects begins in a physical or in a digital store. However, a fundamental difference between these two retail touchpoints – as they are aptly called in the field of marketing – is the possibility they afford to actually enjoy a multisensory, on-body experience of dress before a decision is made regarding them. The online experience of fashion is digital and, lacking touch, disembodied.

The dis-embodiment of dress: Digital fashion actualities From eye-catching window displays to in-store atmospherics, physical retail offers a multisensory shopping experience (Arnold, 2009: 67–84; Classen, 2012: 191–6) and, most importantly, a place where customers can inspect products. Garments accessibly displayed on racks or shelves are a fixture of the retail experience, because the need to touch clothes (Peck and Childers, 2003b) is ‘a critical and symbolic aspect of the apparel purchase decisionmaking process’ (Lund, 2015: 19). Even in the most technologically advanced, prototypical store such as the one temporarily showcased by online multibrand seller Farfetch in London, the clothes are tangibly present (‘Inside Farfetch’s Store of the Future’, 2017). Inspecting garments to evaluate their feel and trying them on to check how they fit are core elements of the physical retail experience which are still lacking in digital retail. This experiential barrier has not held back fashion e-commerce. Over the last twenty years, fashion and luxury websites have evolved from being little more than digital storefronts to full-fledged sales channels. Today, most apparel and fashion brands feature dedicated online business divisions which directly manage digital marketing, sales and promotion strategies. The ongoing Covid-19 pandemic has underscored the importance of having a digital presence and accelerated the growth of fashion e-commerce. About 71 per cent of the executives interviewed for the State of Fashion annual report expect their online business to grow 20 per cent in 2021 (Imran et al., 2020), while a survey conducted by online marketplace Zalando on a panel of 118 fashion brands selling through the platform indicated that by 2023, online sales could account for 57 per cent of their total sales (‘After Covid: The Two-Year Plan for Omnichannel Relevancy’, 2020).

A TRUE FEEL

211

One of the key priorities for apparel brands is online personalization (Nobile and Kalbaska, 2020) particularly product size and fit. Websites usually include country-specific sizing charts or sizing guides (e.g. bust, hip, waist measures in inches or centimetres and corresponding sizes) by product type. Some websites also integrate cloud-based, third-party online virtual fit engines (e.g. FitFinder or TrueFit) offering size recommendation, fit recommendation and fit visualization options (Gill, 2015). However, the need to innovate garment sizing and fit functionalities to facilitate purchasing, to increase customer satisfaction and therefore to decrease product returns (Miell et al., 2018) has overshadowed the closely connected element of garment feel – the sensorial and perceptual affordances of dress. All fashion e-commerce websites need to showcase products, which are depicted worn on a model or displayed flat, sometimes both. Some brands include videos of a model wearing a garment and moving. These video clips provide some visual cues about a garment’s characteristics when worn: for example, how it drapes over the body or flows with the wearer’s movement. As mentioned, fashion websites also include textual descriptions, which tend to focus on technical or functional elements: for example, style, cut, length, material composition and so on. Feel is conveyed, if at all, using a limited range of technical terms (e.g. ‘velour’, ‘twill weave’, ‘bi-stretch’) and adjectives (e.g. ‘soft’, ‘snug’, ‘smooth’). Thus, to understand the chosen garment’s properties, performance and effects when worn, customers must be able to interpret the textual information in relation to the moving, still, or 3D visual information provided, and project the outcome onto their own mental body image. This simulation is a re-enactment of dress which – as explained in the discussion on grounded theory – builds upon prior perceptual experience, stored in memory (Papies et al., 2017). This condition may not be fulfilled for all consumers and all garments in question. Thus, customers make online buying decisions based on assumptions about fit and on partial information about garment characteristics, particularly feel. When the chosen items are delivered and tried on, these decisions might prove to have been misguided. The garment might fit but not drape well, because bodies are rarely perfectly symmetrical; or it might feel unpleasant to the touch or on the skin. Consumers manage the perceived risk engendered by the uncertainties connected to buying online in a variety of ways (Ornati and Cantoni, 2020). They practice webrooming, which entails searching for brands online but trying them on and purchasing them in a store; and showrooming, which involves evaluating items in a store and then shopping for them online. They order multiple sizes of the same item, or material variants, and take advantage of lenient shipping and return policies (Wang et al., 2019) to send back the garments which do not correspond to the fit or the look they had anticipated. In fact, according to a survey conducted on end customers of e-commerce platform provider Shopify.com, ‘consumer preference-based return reasons (e.g. size, fit, style, etc.) tend to drive around

212

MATERIALIZING DIGITAL FUTURES

72 per cent of all returns in fashion product categories’ (Orendorff, 2019). In the US alone, clothing and shoes bought online have return rates of 30 to 40 per cent (versus 5 to 10 per cent if purchased in-store) (Reagan, 2019) and fashion product returns generate 5 billion pounds of landfill waste a year (Schiffer, 2019) – thus contributing to the increasingly unsustainable environmental impact of the fashion and textile industry (Dean, 2020). When considering these developments, the implications of the sensory limits of the digital fashion experience, particularly relating to the sense of touch, become readily apparent. However, the fashion industry has taken notice. In a recent live discussion organized online by Vogue.com, Stephanie Phair, Chief Customer Officer of Farfetch, mentioned ‘huge developments’ in size and fit technologies as well as in augmented and virtual reality applied towards fashion; technologies which, she believes, will eventually provide a digital sensory experience approximating ‘feel’ (Yotka, 2020). Technological innovation might indeed help the fashion industry overcome the sensory limitations of e-commerce. Ongoing research in haptics (touch technologies) and virtual reality might create the necessary conditions for the re-embodiment of the digital fashion experience sometime in the (near) future. In the next section I will provide a brief overview of these technologies and their application, introduce some devices currently entering or already on the market and discuss their potential relevance for fashion business-to-business and end-customer contexts.

Technological embodiments: Haptics Haptics is touch, engineered. In fact, haptics refers to integrated mechanical, electronic and computational systems and non-invasive applications enabling human–computer interaction (HCI) by artificially reproducing the sense of touch (Culbertson et al., 2018; Kuchenbecker, 2018). By definition, these systems involve ‘sensing, processing, and actuating’ (Kuchenbecker, 2018: 3). Haptic systems can be deployed for interaction in different kinds of environments, from the user’s physical setting, to a tangible but remote environment (e.g. when performing surgery on a patient at a distance), to an immersive, virtual one (Kuchenbecker, 2018). Following is a short introduction, by no means exhaustive, to those haptic systems and applications which are more widely discussed in the literature and are particularly relevant to digital fashion. Haptic systems can be categorized by type of interaction: graspable, wearable, touchable (Culbertson et al., 2018: 387–8) and contactless or mid-air (Rakkolainen et al., 2020). Graspable systems are kinaesthetic devices, typically manoeuvred via a hand-held tool, which enable the user to exert force and feel force or other sensory feedback. Wearable systems are cutaneous devices which are worn on the body – typically the hands – and

A TRUE FEEL

213

which convey sensations directly to the skin. Touchable systems are also cutaneous devices, but usually in the form of a surface which changes tactile properties based on the location of the user’s fingers during the interaction. Contactless or mid-air haptic systems function by exerting pressure on the skin (via ultrasound, air jet or other technologies); unlike the systems just described, they do not require physical contact with the user. Haptics systems can also be distinguished in terms of where they are mounted: grounded devices are objects attached to a surface; ungrounded, or – confusingly – body-grounded devices are held in the hand or worn on the body (e.g. thimble-like devices or exoskeletons). Surface haptic interfaces are surfaces which may be stationary, such as a computer screen or mobile, as in the case of a smartphone screen; their defining characteristic – with reference to the other types of systems described here – is the fact that tactile feedback is generated via direct interaction with the fingers or a tool, such as a stylus. Haptic devices usually incorporate actuators, sensors or both. Actuators are machine components which control the haptic system or parts of it. Tactile sensors are data acquisition devices designed to detect diverse properties of objects upon direct physical contact; these devices approximate human touch sensing and perception capabilities, converting the information acquired so that it can be processed and analysed within an intelligent system (Martinez-Hernandez, 2015). Using actuators and/or sensors, most haptic systems deliver tactile information to the user in one of the following ways: via kinaesthetic forces or pressure felt on joints and body parts; through vibrations or pulses, which exploit the human ability to sense and distinguish between frequencies; and by skin stretching, which triggers the skin’s sensitivity to tangential forces. (Basdogan et al., 2020; Bayousuf et al. 2019; Culbertson et al., 2018). Three examples of haptic systems and devices which could be particularly relevant in the context of touch sensing for the digital fashion domain are TanvasTouch® (www.tanvas.co), WeArt (www.weart.it) and TeslaSuit (www.teslasuit.io). The Tanvas haptic system includes an interactive screen which enables garment texture surface effects simulation (Ornati and Cantoni, 2020)2. WeArt is a double ring, wearable haptic sensor system which reproduces tactile cues (forces, textures and temperature changes) on the wearer’s skin, and has been tested in virtual reality (VR) environments (Spagnoletti et al., 2018). Tesla’s full-body suit is currently used to simulate real-life bodily interactions (such as impact) and sensations (such as heat) for personnel training purposes in public safety scenarios (e.g. firefighters and construction workers). Were it to evolve into a lighter – and more affordable – version, it could very well be used in consumer settings, possibly even the home, to deploy haptically onto the wearer the fit and feel properties of clothing. The haptic interfaces described are a limited sample and a modest indication of the innumerable interfaces and devices engineered, prototyped

214

MATERIALIZING DIGITAL FUTURES

and developed in interdisciplinary research centres around the world, as exemplified by publications in IEEE Transactions on Haptics and dedicated conferences such as EuroHaptics. Haptic interfaces and the ‘fundamental understanding of touch’ have made ‘remarkable progress in recent years’ (‘Eurohaptics 2020 Conference’, 2020). In the last decade, haptic systems ‘have started to be designed with wearability and portability in mind’ thereby increasing adoption within ‘teleoperation, gaming, rehabilitation’, augmented reality (AR), VR, mixed reality (MR) ‘and many other application domains’ (Prattichizzo et al., 2019: 228). In addition to being lighter and smaller, haptic systems are becoming less expensive, a fact which ‘opens great opportunities in the consumer market for applications such as gaming and interaction in VR and AR’ even though there are still ‘significant drawbacks’ to their diffusion such as ‘limited actuation and sensing capabilities’ (Prattichizzo et al., 2019: 228). Haptic systems, however sophisticated, are still quite far from being able to comprehensively reproduce the exquisite sensitivity of the somatosensory system. Yet, TanvasTouch, WeArt and TeslaSuit afford two constitutive dimensions of the embodiment of dress: the tactile perception of material feel (roughness, temperature, shear and so on) under the hand and on the skin; and the on-body kinaesthetic sensation of being clothed (pressure, pull, and so on) both in the physical and in the virtual domain. In fact, the vision of researchers in the haptic field is one of ‘rich touch interactions’ with 3D objects in AR, MR and VR applications – ‘embodiments of an immersive haptic reality’ which ‘have the potential to change the way we live, work, learn and leisure’ (‘Eurohaptics 2020 Conference’, 2020); and thus, I add, potentially modify the practice of dress in the digital fashion domain. Innovation at the crossroads of haptic and virtual technologies, coupled with the fashion industry’s pressing need to deliver engaging, multisensory digital customer experiences, is already impacting the way clothing is promoted, sold and experienced. In the following section I discuss recent AR and VR technology applications in the fashion industry and speculate on the digital future of virtual dress embodiment made possible by converging technologies.

The re-embodiment of dress: Technological convergence Luxury brands – particularly watches and jewellery – have been experimenting with AR as a promotional tool for almost two decades (Javornik, 2016); more recently, fashion brands have introduced AR effects and virtual try-on mirrors in physical stores (Bonetti et al., 2019). Spurred by the need to digitize store experiences made inaccessible during the Covid-19 induced confinement, fashion brands are adopting AR technology

A TRUE FEEL

215

mobile applications such as smart mirrors and add-on filters (for makeup and clothing). Cases include AR applications developed by Holition for l’Oréal and Uniqlo (www.holition.com/work), Louis Vuitton promotions (Papagiannis, 2019) and many others (Boardman et al., 2020: 156). While AR technologies superimpose virtual content onto real-world environment, VR technologies afford the user multisensory experiences of immersion and the psychological phenomena of presence (Herrera and Bailenson, 2020). Immersion and presence are exactly what the fashion industry was unable to offer its business and end customers during pandemic-induced shutdowns. Fashion shows scheduled during the Copenhagen, London and Paris summer of 2020 fashion weeks – when buyers and journalists the world over still could not travel – were video streamed synchronously or asynchronously online. Retrospectively, industry experts judged 2D screenings unengaging, because the sensorial dimension was missing (Farra, 2020). Thus, fashion industry experts have been increasingly discussing VR technologies and haptics (Turra et al., 2020) as an alternative approach to brand promotion and sales. When asked how live fashion shows could be recreated effectively in a virtual setting, designers and consultants interviewed by Vogue Business (‘Forced Cancellations Jumpstart Virtual Fashion Technology’ n.d.) emphasized the importance of creating virtual ‘presence’ via multisensory stimulation – being able to smell something or to touch fabric swatches. They also mentioned the possibility of using haptic gloves to replicate the sensory feedback one might get from touching a garment. The need for presence was exacerbated during the winter of 2021 fashion shows, which again were presented exclusively online. The technological convergence of haptic and virtual technologies, coupled with the trend towards smaller, more accessible haptics and VR devices (‘2020 Augmented and Virtual Reality Survey Report’, 2020; Robertson, 2020) is bound to attract the most innovative players in the fashion industry. In addition to existing physical and digital channels, brands will be able to leverage V-commerce, or ‘electronically mediated commercial transactions that originate from an alternate reality technology platform and involve either digitally-generated or real-world products and services’ (de Regt and Barnes, 2019: 19). Fashion brands working with 3D computer-aided design and garment prototyping software systems will be well positioned to do so, as they will be able to deploy the technology downstream – for example, by integrating digital production with existing e-commerce platforms’ fit and sizing solutions – thereby offering consumers a ‘made-to-measure’ virtual fashion experience (Tabuchi, 2015) fuelling more sustainable, on-demand supply mechanisms of production and distribution. The customer experience will be virtual, but multisensory and embodied, thus impacting the perceptual, emotional and cognitive processes of situated dress practices in novel ways.

216

MATERIALIZING DIGITAL FUTURES

I envision two embodied customer experience future scenarios: out-ofhome, in physical retail, and at home or on the go, in digital retail. The first scenario has already been tested by Nike. In the NikeiD VR STUDIO clients were able to customize, order (and later purchase) sneakers while immersed in a 3D environment (Adlatina, 2016). The immersive in-store experience pioneered by Nike could also be offered in pop-up or itinerant brand store formats. Nomadic (www.blurtheline.com), a VR production firm which designs MR environments incorporating real objects and touch feedback into the virtual experience, has developed a modular VR system which can be adapted to a brand’s need and integrated into existing spaces such as shopping malls (‘Nomadic Nabs $6M for Its Modular VR System for Retail Spaces’, n.d.). The second scenario envisions digital retail as an immersive, MR experience leveraging body-scanning mobile phone applications, AR applications, VR consumer systems (including smaller headsets and haptic wearables simulating not only fit but feel, such as gloves, vests or very lightweight haptic body suits, like undergarments) and necessarily, brand websites designed for V-commerce usability. Materializing this future vision of touch in the digital fashion embodied dress experience will be a challenge for the fashion industry. It will require a digital transformation of the supply chain, the technological convergence and market readiness of innovations still in the prototypal stage, consumer acceptance and the implementation of sustainable and ethical promotional and sales practices. But the innovation imperative (Imran et al., 2020) forced upon the industry by the Covid-19 pandemic has accelerated these processes of radical change, propelling digital adoption (Imran et al., 2020).

Conclusion Clothing ourselves is an embodied practice which involves physical and cognitive processes. Donning garments activates our diffused sense of touch, unleashing sensations, perceptions and emotions. These multisensory processes affect the way we experience clothes, ourselves and the world around us, impacting our behaviours. The acts afforded by dress practices are palpable and have real-world outcomes. In the last two decades, the rise of electronic commerce has created a digital channel for fashion experience and consumption which limits the embodied, multisensory quality of dress experience. Size, fit and feel are unresolved issues in online shopping. For most consumers, nothing can quite replace the act of physically handling and trying on clothes. Haptic technologies aim to replicate this active touch experience, via interactive, wearable and other devices increasingly accessible to the ordinary consumer. Advances in haptics and convergence with virtual

A TRUE FEEL

217

technology systems suggest electronic commerce might one day become virtual commerce, an immersive meta-experience involving body and mind. ‘Faster and more realistic computer graphics, goggles that allow threedimensional immersive contexts, and sensors capable of detecting body movements in real-time are making the experience of visual duplicates (avatars) of ourselves increasingly common’ (Bruno and Pavani, 2018: 38). In the future, trying clothes on our projected selves, directly experiencing their true feel, might become part of a normal digital fashion experience, extending the embodied, situated practice of dress to another dimension of time and place.

Notes 1 Perception scientists question the traditional separation and classification of the senses, but concede it is convenient to study them as systems, defined as ‘a set of interconnected elements forming a complex whole’ (Bruno and Pavani, 2018: 2). 2 A video recording of the interaction can be viewed here: https://youtu.be/ NEtf1d53eZ8.

References ‘2020 Augmented and Virtual Reality Survey Report’ (2020), Vol. 4, Perkins Coie LLP. Available online: https://www.perkinscoie.com/en/ar-vr-surveyresults/2020-augmented-and-virtual-reality-survey-results.html. Adam, H. and A. D. Galinsky (2012), ‘Enclothed Cognition’, Journal of Experimental Social Psychology, 48 (4): 918–25. Available online: https://doi. org/10.1016/j.jesp.2012.02.008. Adlatina (2016), ‘R/GA Buenos Aires Launches NIKEiD VR STUDIO | Articles’, R/GA, December 2016. Available online: https://www.rga.com/news/ articles/r-ga-buenos-aires-presents-nikeid-vr-studio. ‘After Covid: The Two-Year Plan for Omnichannel Relevancy’ (2020), Vogue Business, 16 December 2020. Available online: https://www.voguebusiness.com/ consumers/after-covid-the-two-year-plan-for-omnichannel-relevancy. Arnold, R. (2009), Fashion: A Very Short Introduction, Very Short Introductions 210, New York: Oxford University Press. Barsalou, L. W. (2008), ‘Grounded Cognition’, Annual Review of Psychology, 59 (1): 617–45. Available online: https://doi.org/10.1146/annurev. psych.59.103006.093639. Basdogan, C., F. Giraud, V. Levesque and S. Choi (2020), ‘A Review of Surface Haptics: Enabling Tactile Effects on Touch Surfaces’, IEEE Transactions on Haptics, 1–1. Available online: https://doi.org/10.1109/TOH.2020.2990712. Bayousuf, A., H. S. Al-Khalifa and A. Al-Salman (2019), ‘Haptics-Based Systems Characteristics, Classification, and Applications’, in D. B. A.

218

MATERIALIZING DIGITAL FUTURES

Mehdi Khosrow-Pour (eds), Advanced Methodologies and Technologies in Artificial Intelligence, Computer Simulation, and Human-Computer Interaction, 778–94, Hershey, PA: IGI Global. Available online: https://doi. org/10.4018/978-1-5225-7368-5. Boardman, R., C. E. Henninger and A. Zhu (2020), ‘Augmented Reality and Virtual Reality: New Drivers for Fashion Retail?’, in G. Vignali, L. F. Reid, D. Ryding and C. E. Henninger (eds), Technology-Driven Sustainability: Innovation in the Fashion Supply Chain, 155–72, Cham: Springer. Available online: https://doi. org/10.1007/978-3-030-15483-7_9. Bonetti, F., E. Pantano, G. Warnaby, L. Quinn and P. Perry (2019), ‘Augmented Reality in Real Stores: Empirical Evidence from Consumers’ Interaction with AR in a Retail Format’, in M. Claudia tom Dieck and Timothy Jung (eds), Augmented Reality and Virtual Reality, 3–16, Cham: Springer. Available online: https://doi.org/10.1007/978-3-030-06246-0_1. Bonetti, F., G. Warnaby and L. Quinn (2018), ‘Augmented Reality and Virtual Reality in Physical and Online Retailing: A Review, Synthesis and Research Agenda’, in Timothy Jung and M. Claudia tom Dieck (eds), Augmented Reality and Virtual Reality: Empowering Human, Place and Business, 119–32, Cham: Springer. Available online: https://doi. org/10.1007/978-3-319-64027-3_9. Bossomaier, T. R. J. (2012), Introduction to the Senses: From Biology to Computer Science, Cambridge: Cambridge University Press. Bruno, N. and F. Pavani (2018), Perception: A Multisensory Perspective, 1st edn, Oxford, UK: Oxford University Press. Classen, C. (2012), The Deepest Sense: A Cultural History of Touch, Studies in Sensory History, Urbana: University of Illinois Press. Available online: https:// www.press.uillinois.edu/books/catalog/97kfh2np9780252034930.html. Culbertson, H., S. B. Schorr and A. M. Okamura (2018), ‘Haptics: The Present and Future of Artificial Touch Sensation’, Annual Review of Control, Robotics, and Autonomous Systems, 1 (1): 385–409. Available online: https://doi.org/10.1146/ annurev-control-060117-105043. Dean, C. (2020), ‘Waste – Is It “Really” in Fashion?’ Available online: https://www. fashionrevolution.org/waste-is-it-really-in-fashion/. Entwistle, J. (2000), ‘Fashion and the Fleshy Body: Dress as Embodied Practice’, Fashion Theory, 4 (3): 323–47. Available online: https://doi. org/10.2752/136270400778995471. Entwistle, J. (2015), The Fashioned Body: Fashion, Dress and Modern Social Theory, 2nd edn, Cambridge, MA: Polity Press. ‘Eurohaptics 2020 Conference’ (2020), 6-9 September, Leiden. Available online: http://eurohaptics2020.org/ (accessed 10 September 2020). Farra, E. (2020), ‘Paris Fashion Week Will Return with IRL Shows This September’, Vogue, 24 June 2020. Available online: https://www.vogue.com/article/ paris-fashion-week-shows-returning-september-spring-2021. Field, T. (2014), Touch, 2nd edn, A Bradford Book, Cambridge, MA: MIT Press. Flavián, C., S. Ibáñez-Sánchez and C. Orús (2019), ‘The Impact of Virtual, Augmented and Mixed Reality Technologies on the Customer Experience’, Journal of Business Research, 100 (July): 547–60. Available online: https://doi. org/10.1016/j.jbusres.2018.10.050.

A TRUE FEEL

219

‘Forced Cancellations Jumpstart Virtual Fashion Technology’ (n.d.), Vogue Business. Available online: https://www.voguebusiness.com/technology/forcedcancellations-remote-working-jumpstart-virtual-vr-ar-fashion-technology (accessed 16 April 2020). Fulkerson, M. (2014), The First Sense: A Philosophical Study of Human Touch, Cambridge, MA: MIT Press. Gill, S. (2015), ‘A Review of Research and Innovation in Garment Sizing, Prototyping and Fitting’, Textile Progress, 47 (1): 1–85. Available online: https:// doi.org/10.1080/00405167.2015.1023512. Hayward, V. (2018), ‘A Brief Overview of the Human Somatosensory System’, in S. Papetti and C. Saitis (eds), Musical Haptics, 29–48, Cham: Springer. Available online: https://doi.org/10.1007/978-3-319-58316-7_3. Herrera, F. and J. Bailenson (2020), ‘Virtual and Augmented Reality Interfaces’, in M. H. Ang, O. Khatib and B. Siciliano (eds), Encyclopedia of Robotics, 1–6, Berlin, Heidelberg: Springer. Available online: https://doi. org/10.1007/978-3-642-41610-1_32-2. Imran, A., A. Balchandani, A. Berg, S. Hedrich, F. Rölkens, R. Young, J. E. Jensen and A. Peng (2020), ‘The State of Fashion 2021’. Available online: https://www.mckinsey.com/industries/retail/our-insights/ the-state-of-fashion-2019-a-year-of-awakening. Imran, A., A. Berg, A. Balchandani, H. Saskia, R. Felix, R. Young and J. E. Jensen (2020), ‘State of Fashion Coronavirus Update: Rewire the System’, McKinsey. Available online: https://www.mckinsey.com/industries/ retail/our-insights/its-time-to-rewire-the-fashion-system-state-of-fashioncoronavirus-update. ‘Inside Farfetch’s Store of the Future’ (2017), The Business of Fashion, 12 April 2017. Available online: https://www.businessoffashion.com/articles/ bof-exclusive/inside-farfetchs-store-of-the-future. Javornik, A. (2016), ‘The Mainstreaming of Augmented Reality: A Brief History’, Harvard Business Review, 4 October 2016. Available online: https://hbr. org/2016/10/the-mainstreaming-of-augmented-reality-a-brief-history. Jewitt, C., S. Price, K. Mackley, N. Giannoutsou and D. Atkinson (2019), Interdisciplinary Insights for Digital Touch Communication, 1st edn, New York: Springer. Jones, L. A. (2018), Haptics, The MIT Press Essential Knowledge Series, Cambridge, MA: The MIT Press. Kalbaska, N., T. Sádaba and L. Cantoni (2018), ‘Editorial: Fashion Communication: Between Tradition and Digital Transformation’, Studies in Communication Sciences, no. 2. Available online: https://doi.org/10.24434/j. scoms.2018.02.005. Kalbaska, N., T. Sádaba, F. Cominelli and L. Cantoni (eds) (2019), Fashion Communication in the Digital Age: FACTUM 19 Fashion Communication Conference, Ascona, Switzerland, July 21–26, 2019. Available online: https:// public.ebookcentral.proquest.com/choice/publicfullrecord.aspx?p=5785106. Kuchenbecker, K. J. (2018), ‘Haptics and Haptic Interfaces’, in M. H. Ang, O. Khatib and B. Siciliano (eds), Encyclopedia of Robotics, 1–9, Berlin, Heidelberg: Springer. Available online: https://doi. org/10.1007/978-3-642-41610-1_19-1.

220

MATERIALIZING DIGITAL FUTURES

Lieber, C. (2019), ‘Customers Love Free Returns. Here’s How Brands Are Navigating the Costs’, BOF Professional, 31 May 2019. Available online: https://www.businessoffashion.com/articles/professional/ customers-love-free-returns-heres-how-brands-are-navigating-the-costs. Linden, D. J. (2015), Touch: The Science of Hand, Heart, and Mind. Available online: http://banq.lib.overdrive.com/ContentDetails. htm?id=C7A5B78A-3011-42BD-B010-CF2DC890F562. Lund, C. (2015), ‘Selling through the Senses: Sensory Appeals in the Fashion Retail Environment’, Fashion Practice, 7 (1): 9–30. Available online: https://doi.org/10. 2752/175693815X14182200335619. Martinez-Hernandez, U. (2015), ‘Tactile Sensors’, Scholarpedia, 10 (4): 32398. Available online: https://doi.org/10.4249/scholarpedia.32398. Merleau-Ponty, M. (2009), Phénoménologie de la perception, Tel 4, Paris: Gallimard. Miell, S., S. Gill and D. Vazquez (2018), ‘Enabling the Digital Fashion Consumer through Fit and Sizing Technology’, Journal of Global Fashion Marketing, 9 (1): 9–23. Available online: https://doi.org/10.1080/20932685.2017.1399083. Miłkowski, M., R. Clowes, Z. Rucińska, A. Przegalińska, T. Zawidzki, J. Krueger, A. Gies, et al. (2018), ‘From Wide Cognition to Mechanisms: A Silent Revolution’, Frontiers in Psychology, 9. Available online: https://doi. org/10.3389/fpsyg.2018.02393. Negrin, L. (2015), ‘Maurice Merleau-Ponty: The Corporeal Experience of Fashion’, in Thinking Through Fashion: A Guide to Key Theorists, Dress Cultures, London: I.B. Tauris. Ngan, V. (n.d.), ‘Textile Contact Dermatitis’, DermNet NZ. Available online: https://dermnetnz.org/topics/textile-contact-dermatitis/ (Accessed 28 August 2020). Nobile, T. H. and N. Kalbaska (2020), ‘An Exploration of Personalization in Digital Communication. Insights in Fashion’, in F. Fui-Hoon Nah and K. Siau (eds), HCI in Business, Government and Organizations, 12204: 456–73, Lecture Notes in Computer Science, Cham: Springer. Available online: https:// doi.org/10.1007/978-3-030-50341-3_35. ‘Nomadic Nabs $6M for Its Modular VR System for Retail Spaces’ (n.d.), Blog, TechCrunch. https://social.techcrunch.com/2017/06/12/nomadic-nabs-6m-inseed-funding-for-its-modular-tactile-vr-system/ (accessed 17 April 2020). Orendorff, A. (2019), ‘The Plague of Ecommerce Return Rates and How to Maintain Profitability’, 27 February 2019. Available online: https://www. shopify.com/enterprise/ecommerce-returns. Ornati, M. and L. Cantoni (2020), ‘FashionTouch in E-Commerce: An Exploratory Study of Surface Haptic Interaction Experiences’, in F. Fui-Hoon Nah and K. Siau (eds), HCI in Business, Government and Organizations, 12204: 493–503, Lecture Notes in Computer Science, Cham: Springer. Available online: https:// doi.org/10.1007/978-3-030-50341-3_37. Papagiannis, H. (2019), ‘LV x Snapchat’, XR Goes Pop, 16 May 2019. Available online: https://xrgoespop.com/home/lvsnapchat. Papies, E. K., M. Best, E. Gelibter and L. W. Barsalou (2017), ‘The Role of Simulations in Consumer Experiences and Behavior: Insights from the Grounded Cognition Theory of Desire’, Journal of the Association

A TRUE FEEL

221

for Consumer Research, 2 (4): 402–18. Available online: https://doi. org/10.1086/693110. Parisi, D. (2018), Archaeologies of Touch: Interfacing with Haptics from Electricity to Computing, Minneapolis: University of Minnesota Press. Peck, J. and T. L. Childers (2003a), ‘To Have and to Hold: The Influence of Haptic Information on Product Judgments’, Journal of Marketing, 67 (2): 35–48. Peck, J. and T. L. Childers (2003b), ‘Individual Differences in Haptic Information Processing: The “Need for Touch” Scale’, Journal of Consumer Research, 30 (3): 430–42. Prattichizzo, D., M. Otaduy, H. Kajimoto and C. Pacchierotti (2019), ‘Wearable and Hand-Held Haptics’, IEEE Transactions on Haptics, 12 (3): 227–31. Available online: https://doi.org/10.1109/TOH.2019.2936736. Rakkolainen, I., E. Freeman, A. Sand, R. Raisamo and S. Brewster (2020), ‘A Survey of Mid-Air Ultrasound Haptics and Its Applications’, IEEE Transactions on Haptics, 14 (1): 2–19. Available online: https://doi.org/10.1109/ TOH.2020.3018754. Reagan, C. (2019), ‘That Sweater You Don’t like Is a Trillion-Dollar Problem for Retailers. These Companies Want to Fix It’, CNBC, 12 January 2019. Available online: https://www.cnbc.com/2019/01/10/growing-online-sales-means-morereturns-and-trash-for-landfills.html. Regt, A. de and S. J. Barnes (2019), ‘V-Commerce in Retail: Nature and Potential Impact’, in M. Claudia tom Dieck and T. Jung (eds), Augmented Reality and Virtual Reality: The Power of AR and VR for Business, 17–25, Progress in IS, Cham: Springer. Available online: https://doi. org/10.1007/978-3-030-06246-0_2. Robertson, A. (2020), ‘HTC Is Prototyping an AR Headset That Looks like Sunglasses’, The Verge, 20 February 2020. Available online: https://www.theverge.com/2020/2/20/21145260/ htc-project-proton-vr-ar-xr-headset-prototype-cosmos-vive-5g. Rocamora, A. (2019), ‘Mediatization and Digital Retail’, in The End of Fashion: Clothing and Dress in the Age of Globalization, 99–111, London: Bloomsbury Visual Arts. Schiffer, J. (2019), ‘The Unsustainable Cost of Free Returns’, Vogue Business, 30 July 2019. Available online: https://www.voguebusiness.com/consumers/ returns-rising-costs-retail-environmental. Shinkle, E. (2013), ‘Fashion’s Digital Body: Seeing and Feeling in Fashion Interactives’, in Fashion Media: Past and Present, London: Bloomsbury Academic. Spagnoletti, G., L. Meli, T. L. Baldi, G. Gioioso, C. Pacchierotti and D. Prattichizzo (2018), ‘Rendering of Pressure and Textures Using Wearable Haptics in Immersive VR Environments’, in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), 691–2, Reutlingen: IEEE. Available online: https://doi. org/10.1109/VR.2018.8446128. Steele, V. (2001), The Corset: A Cultural History, New Haven, CT: Yale University Press. Steele, V. (ed.) (2005), Encyclopedia of Clothing and Fashion, Scribner Library of Daily Life, Farmington Hills, MI: Charles Scribner’s Sons. Tabuchi, H. (2015), ‘Tommy Hilfiger Introduces Virtual Reality Headsets for Shoppers’, New York Times, 20 October 2015. Available online: https://www.

222

MATERIALIZING DIGITAL FUTURES

nytimes.com/2015/10/21/business/tommy-hilfiger-introduces-virtual-realityheadsets-for-shoppers.html. Turra, J., L. Diderich, A. Templeton, J. Diderich, L. Templeton and A. Turra (2020), Blog, ‘Coronavirus Crisis Could Spur Digital Strategies’, WWD, 11 March 2020. https://wwd.com/business-news/technology/fashion-brands-step-updigital-strategies-to-make-up-for-coronavirus-travel-restrictions-1203534738/. Wang, Y., J. Anderson, S.-J. Joo and J. R. Huscroft (2019), ‘The Leniency of Return Policy and Consumers’ Repurchase Intention in Online Retailing’, Industrial Management & Data Systems, 120 (1): 21–39. Available online: https://doi. org/10.1108/IMDS-01-2019-0016. Yotka, S. (2020), ‘Virgil Abloh, Stephanie Phair, and Remo Ruffini Imagine a New Future of E-Commerce’, Vogue, 16 April 2020. Available online: https:// www.vogue.com/article/vogue-global-conversation-e-commerce-virgil-ablohstephanie-phair-remo-ruffini-angelica-cheung.

12 What robots learn from performative relationships and interactive performance Steph Hutchison and John McCormick

Introduction There are many questions and possibilities that arise from the interrogation of encounters between human and robot movement. Artform specific considerations include the creative, choreographic and performative possibilities offered by creative robotics in the area of dance and movement. What new choreographic forms are made possible? How can both human and robot movement vocabularies be extended? Can existing choreographic structures be used to enable shared movement invention and synergy? Are there new choreographic structures that will need to be developed to better manage the collaborative relationships between humans and robots? Investigation of shared movement between humans and robots has the potential to impact areas well beyond the arts. If robots are to truly share in the spaces in which humans work, play and socialize, we must learn to move with, around and between each other. As with humans, robots will not be engaged in work-task driven behaviour all the time. Much of the time will be spent moving from one task to another, navigating space and others, observing, planning, responding. Small, seemingly inconsequential movements, devoid of productivity goals, will come to have enormous consequences for the way we work and socialize with robots. The movement

224

MATERIALIZING DIGITAL FUTURES

structures that enable our relationships with robots to remain physically and emotionally continuous are dependent on the movement exchanges that carry continual meaning between robots and humans. This research also looks beyond the arts for further inspiration towards methods for enabling collaborative dance between humans and robots. Artificial Intelligence (AI), Virtual Reality (VR) environments and the related Digital Twins from Industry 4.0 are being used extensively in contemporary robotics to manage a robot’s capabilities. What can these offer the creative process and in particular choreographic process as structures for enabling robot movement capabilities and human robot collaboration? Conversely, can choreographic process contribute to the development of approaches to AI and VR that extend their capabilities within and beyond artistic practice? At the heart of this problem is movement itself. How do we learn to navigate shared spaces? What strategies will help humans and robots learn from each other in order to facilitate better cooperation? Dancers are not only experts at constructing complex movements for themselves they are also experts at engaging with another physical body through a trained understanding of kinaesthetics, spatial awareness and physical contact. Moreover, contemporary dancers are experts at engaging with complex systems and virtual bodies (Kirsch, 2011). In our research we leverage the expert knowledge of a trained contemporary dancer as well as choreographic structures and apply these to the problem of humans moving with collaborative robots.

Emergence Prior to collaborating with robots, we investigated software agents as performing partners. The performance project Emergence (2014) used machine learning to enable a software agent to learn how to perform live with a dancer. The agent was trained using motion capture examples from the dancer. A combination of neural networks (NNs; discussed further later) and Hidden Markov Model (McCormick et al., 2014) was used to enable the agent to engage in simple choreographic structures with the dancer; follow the dancer’s movements, create new movement sequences based on the dancer’s current movement, recognize short dance sequences performed by the dancer and respond with appropriate movement and sound. The performance was presented at the International Symposium on Electronic Art (ISEA, 2015). The work was successful because of the interdependent nature of the dancer–agent relationship (McCormick and Hutchison, 2015). Socially distributed cognition provided a framework whereby the onus of developing capability was not entirely on the self-contained agent, it could draw on the spatio-temporal intelligence of the dancer throughout the

WHAT ROBOTS LEARN

225

developmental and performance process (Barnier et al., 2008; McCormick et al., 2014; Sutton et al., 2010).

Dancer as support structure for a performing agent Contemporary dancers have a highly developed understanding of the moving body, both their own and the bodies they collaborate with. Dancers support each other through subtle cues and suggestions that keep their movements attuned to each other. It is the nature of this relationship that allowed the use of relatively simple NNs to enable a performing agent to dance with a human dancer. The agent was continually supported by the dancer’s movement as reference. If the agent was looking to synthesize a movement sequence it could look at what the dancer was doing and use those current movements to seed its own movement. Having been trained using the dancer’s captured movement data, the agent had an internal representation of the dancer’s movement that allowed it to match what the dancer was performing and follow her with similar movements. It could put together movements it had learned, to create sequences in response to what the dancer was performing creating an improvised duet. As well as recognizing individual postures, the agent could also recognize short sequences of movement as discrete choreographed movement motifs that were distinct from one another. It responded to sequences it recognized with verbal and animated commentary on the dancer’s actions. Emergence used a software agent in a sense as a simulator for potential relationships between a dancer and a software agent or physical agent such as a robot. It is standard practice to use a simulator to trial movement possibilities for a robot before applying them to the real robot. In Emergence the dancer used her motion captured movement to animate the agent directly as well as to train the agent’s NN enabling it to follow her by using its learnt movement to best approximate what the dancer was doing. This reproduced one of the most prevalent choreographic structures of one dancer following another dancer.

Artificial neural networks for agent learning The agent and NN were developed in the Unity game engine which enabled a range of visualizations to be generated that described the relationship between the dancer and performing agent. This created a form of live annotation over the duet performance. The audience viewed the projected virtual environment using stereo glasses in order to get a sense of depth in the visual relationship between dancer and agent. The agent used both a humanoid body and more abstract botanically inspired representations (Figure 12.1). The relationship of dancer and agent’s movement was

226

MATERIALIZING DIGITAL FUTURES

FIGURE 12.1  Emergence.

visualized through the 3D annotated imagery, for example, the closer in shape the agent’s movement was to the dancer’s movement, the shorter the limbs of the resultant avatar became. In the visualized environment the bodies could represent dancer, agent or a combination of both. Using the 3D environment to create an annotated visualization of the combined movement of both the dancer and agent is a technique used in all our subsequent work developed with humans and robots. Another choreographic structure used to develop a relationship between the agent and dancer used a trained NN to enable the agent to develop its own movement derived from the dancer’s current movement. This enabled a semi-improvised duet where the agent responded to the dancer’s movement with similar movement of its own and the dancer could similarly cue her movement from the agent’s. Geometry was created along the agent’s movement pathway that created a real-time 3D sculpture derived from the movement improvisation and could be played as a musical instrument. This creation of choreographic objects that embody aspects of the choreographic relationship is a development seen in some of the choreographic objects showcased in the MotionBank project (deLahunta and Jenet, 2017; Leach, 2014). Emergence pointed to directions for research into using dance with physical agents as well as software agents. Relatively simple choreographic structures could be used to develop movement for the agent, structure

WHAT ROBOTS LEARN

227

choreographic relationships and frameworks for a human dancer and enable real-time performance interactions between human and agent dancer. In subsequent work we looked at applying similar choreographic structures to the development of a physical robot. After the success of using a virtual environment to encapsulate and visualize the motion capture data from the dancer and the movement generated by the agent, we progressed to employing similar methods and structures for working with robots in performance.

From software agents to physical agents After working with performing software agents and dancers in performance we had the desire to give the agent physical agency and liberate it from the screen. Hence the current preoccupation with performing collaborative robots. We hypothesized that this strategy of harnessing the movement intelligence of the dancer is transferable to the development of a collaborative robot’s movement capabilities. Collaborating with robots can be seen as a partnership rather than two completely independent entities continually attempting to find common ground. The robot can be supported by the dancer at each moment and this intrinsically changes the design approach. Our practice-led research uses a framework of embodied and distributed cognition to develop supportive, collaborative human–robot relationships in order to develop synergistic movement between the two (Clark, 1997; Clark and Chalmers, 1998: 7–19; Noё, 2012: 192). New knowledge is discovered from actually moving, from the experiences of the physical bodies and the exploration of collaborative movement and collective cognition.

Pinoke In the 2015 artwork Pinoke (Figure 12.2), we began to extend the work with performing agents to robots. We used a number of choreographic structures similar to those used with the software agent to develop the robot’s movement. The fundamental structure of one dancer following another was achieved using motion capture to transfer the movements of a human to the robot. Differences in morphology were mediated by the human observing the robot and adjusting their movement to compensate, providing an unsubtle but quick solution to the ‘correspondence problem’ whereby human movement cannot be directly mapped onto a robot with different morphology. In the performance Pinoke, the dancer also used some guided examples to teach the robot movement, physically moving the robot and recording the movements. Moving the robot by hand and contact was a quick method for

228

MATERIALIZING DIGITAL FUTURES

FIGURE 12.2  Pinoke Kneeling.

generating robot movement. Along with dancer mediated motion capture data, where the motion data is filtered by the dancer to suit the robot’s morphology more closely, physically moving the robot is also a promising means of generating appropriate movement data as both recorded movement to animate the robot directly and as data with which to train a digital NN.

Digital puppeteering Animating the robot with live motion capture data and through physically manipulating the robot are both forms of puppeteering, the former moving it from inside’ and the latter from ‘outside’. Both methods offer many possibilities for generating direct movement for the robot as well as data for machine learning. However, the transfer of ‘movement data’ is not just one way, from human to robot. The human dancer is also working within the movement constraints generated by the robot’s morphology when using their motion capture to animate the robot in real time. When physically manipulating the robot, the human dancer must adjust their own movement to serve the purposes of working with the weight and physicality of the robot to achieve a desired aesthetic. The dancer has the opportunity to understand the characteristics of the robot and how their own movement can fit in the same movement volume as the robot. This shared inhabitation of the same volume of space can be encoded into data for future machine learning.

WHAT ROBOTS LEARN

229

We used an NN to learn movement from motion capture data of the dancer to give it some independence in creating movement. This was limited to upper body movement as the NN could not compensate for morphology correspondence in the way the dancer could, and bipedal movement from the NN was consequently unstable. These early experiments in using NN for movement learning were quite modest yet showed that an NN could successfully learn from human movement for the purpose of enabling robot movement. Advances in guided reinforcement learning could be applied to the problem of a robot learning from a human dancer’s motion capture data and adjusting for the morphological differences. Simulated robots have been successfully trained using motion capture examples to guide reinforcement learning of a deep NN (Peng et al., 2018). They work using physics-based simulation environments enabling the robots and humanoids to learn extremely complex movements adapted to their own morphologies. Our ongoing research is looking at applying such techniques to our physical robots to improve their movement learning from human experts.

Spatial movement structures The kinesphere is distinguished by the personal space around the body that is defined by extension of limbs around the body without stepping away from the centre point (Laban and Ullmann, 2011: 11). Adapting the spherical form of the kinesphere to the form of the cube, as fitting inside the sphere, the dancer and robot work within the cube as a frame for directing movement spatially (Laban and Ullmann, 2011: 18). The application of the cube as an external framework for teaching robots to dance enables the robot to develop a 360° perception of its body in space and time. Replicating conditions familiar to the dancer/choreographer, this perception of the kinesphere and of activating points in space just beyond the reach of the robot/dancer enables both to construct their movement in relation to the external framework. Seemingly, with reference to the stability of the humanoid robot in Pinoke the application for the cubic framework was improved by the structure to support movement design. Additionally, the sense of presence and three-dimensionality of both performers was enhanced. In Pinoke the dancer/choreographer used the cube and alphabet as a means to develop a bespoke movement vocabulary. With one movement assigned to each letter of the alphabet the dancer/choreographer had developed a series of movement gestures that could be shared with the robot. The movements could be performed in any order. The use of pangrams provided a choreographic score to order the movement vocabulary in an arbitrary way. For both the robot and human dancer alike the process of performing pangrams involves a process of physically puzzle-solving the

230

MATERIALIZING DIGITAL FUTURES

transitions between the movements. The transitions may or may not have an already established kinaesthetic pathway and therefore may or may not have an embodied logic. Here, there is an opportunity for both dancer and robot to extend their embodied experience and movement capacity. Like improvisation this form of choreographic score creates an opportunity to construct novel phrasing and combination of movement sequences that deliberately destabilized familiar/habitual movement patterns. How both dancers navigate the unfamiliar sequencing provides an opportunity to further understand the process of embodied cognition and movement potential (Noland, 2009a,b), Moreover, each movement as assigned to a letter of the alphabet contains its own distinct pathway through space. Using motion capture the movement pathway could be rendered as an animated representation thus creating a unique movement signature. The animation in this way created a form of annotation and a choreographic object.

Dance Haptics For the Dance Haptics project, we developed methods for sharing dance with visually impaired artists and audience members. Movement of the dancer was interpreted through a haptic cushion attached to the user’s chair and through a small humanoid robot (Figure 12.3). The haptic cushion had an array of actuators that provided tactile stimulation to the back of the user (Hossny et al., 2015). Patterns of touch representing spatial movement

FIGURE 12.3  Robot Margaret.

WHAT ROBOTS LEARN

231

of the dancer were traced across the users back representing movement of the dancer through the physical space (McCormick et al, 2020). The system drew inspiration from the practice of social haptics, a method for portraying information to a blind person through touch, that does not rely on an alphabet or traditional sign language (Lahtinen, 2003; Lahtinen and Palmer, 2000). Social haptics practitioners are extremely adept at describing spatial and interpersonal relationships and this system was successfully used to describe the dynamics and spatial orientation of a live dancer to blind participants. The haptic cushion mimicked the human social haptics practitioner albeit with less fidelity by imparting spatial information through the participant’s back.

Tactile robot The second method used saw a small humanoid robot programmed to mimic the dancer’s movements. The user could hold the robot in their hands to get a sense of the postures of the dancer. The robot’s movement was largely pre-programmed based on the dancer’s choreography. The robot was able to give a sense of the dynamics and postures of the dancer to the visually impaired audience member. While the representation of the dance is physically limited by the robot’s movement capabilities, its smaller range and degrees of freedom compared to the dancer, nevertheless the experience was reportedly profound in the sense of clarity of movement portrayed by the robot. The portrayal of movement through the user’s hands was inspired by the practice of an interpreter guiding the hands of the visually impaired person to emulate the dancers upper body movement, giving some sense of the dance. The small humanoid robot enabled the user to feel most of the robot’s body and so give a good sense of the overall movement of the dance. The choreographic process for Dance Haptics required the dancer to rely more heavily on the feedback being provided in real time by the participants. In order to transmit the dance, the dancer needed to understand the constraints of the various systems (technological-humanoid robot and haptic cushion, and Auslan and social haptics) and also consider what was fundamental to dance and how this could be mapped across to the various systems working to perform the dance for the visually impaired person. Not just any movement or gesture could be communicated. The haptic cushion had 64 actuators in an 8 × 8 grid providing less resolution than a human social haptics practitioner’s ability to transmit continuous information through the back of the recipient. Nevertheless, the haptic cushion provided a means of successfully transmitting real-time spatial information derived from the motion capture data of the dancer’s movements and speed across the performance space. The Dance Haptics project also raised questions regarding what might be perceived as a robot. The humanoid form was recognized by all

232

MATERIALIZING DIGITAL FUTURES

participants as a robot, while the haptic cushion, being camouflaged within the back of a seat, was perhaps less recognizable as a robot, even though both were multiple degree of freedom, mechanically articulated devices. We considered both as within the field of robotics with their morphologies being appropriate to the form of expression they were emulating and the type of information they were trying to convey. In the larger context of a performance, the two systems might even be considered part of the same robotic performance system that incorporates the motion capture system, human dancer, humanoid robot, haptic cushion and the virtual software environment that is enabling all the parts to communicate and share data. Robots do not always need to exist in a single body and can present as complex distributed systems, using motion capture cameras as sensory input and projection, audio and haptic systems as part of their expression.

Eve of Dust For the performance Eve of Dust, presented at Tokyo International Forum for SIGGRAPH Asia 2018, we used a Sawyer collaborative robot to explore human robot choreography (McCormick et al., 2018). We created a virtual environment containing both a digital twin of the robot, animated in real time by its movement and the tracked data of the dancer (Figure 12.4). While

FIGURE 12.4  Eve of Dust.

WHAT ROBOTS LEARN

233

the audience did not see the VR world, it played an integral part in mediating the collaborative relationship. The movement of both dancer and robot created a soundscape within the VR world providing a sonic description of their combined movement which the audience could hear. The choreography of the human and robot consisted of a number of approaches: guided recorded movement and contact improvisation using variable compliance of the robot’s motors and movement generated from a motion planner, which responded to the dancer moving with a Vive VR controller. In Eve of Dust we developed very fluid movement between the dancer and robot, particularly in the contact improvisation mode which was also the most challenging choreographic structure as weight changes could happen quickly. For the next developments we take these structures and expand on their possibilities, using machine learning to enable the robot to learn further from these choreographic explorations. While these past human–robot explorations have resulted in significant examples of human– robot collaboration in a creative context, the addition of more sophisticated machine learning techniques will enable more complex choreographic relationships to be developed as they will offer the robot greater movement generation capabilities and better recognition of the movement the dancer is performing. Contact improvisation, as exemplified by dancers such as Steve Paxton, is a particular duet dance structure where both dancers are equally active in giving and taking weight and instigating movement (Stark Smith, 2008). The dancers attempt to maintain some form of contact at all times. It is extremely nuanced in the subtle reading of shifts in weight and other shared proprioceptive cues. A form of contact improvisation was employed in Eve of Dust as a means of exploring fine transfers of action, weight and intent. One method on the robot side was to work with variable compliance of the robot’s motors. Compliance in robots is the amount of force the motors transmit and how it responds to external forces. While most collaborative robots are capable of variable compliance, there are subtle cues to be recognized as to who is becoming more active or more passive. Methods for the human and robot to be sensitive to the changes in weight and force of the other need to be investigated further and will benefit activities requiring intimate collaboration between robot and human. This is an example of using dance structure to investigate real-time dynamic interactions between robot and dancer. Through the process of choreographing the Sawyer robot, the robot’s movement developed a sense of breath—breathing with movement which was amplified as its movement also generated the sound. As the dancer worked with the robot, imparting her understanding of breath, the movement developed a spaciousness for the relationship between to deepen. The breath provided timing and it influenced behaviour as it was expressed through the robot’s movement.

234

MATERIALIZING DIGITAL FUTURES

Contemporary dancers as agents for training robots Trained contemporary dancers possess the ability to engage in complex systems both physical and metaphysical. Using advanced kinaesthetic intelligence, contemporary dancers are able to engage with imagined external frameworks such as the architecture of a cube. Holding the framework in their mind they are able to consistently undertake a physical enquiry within the external framework. We refer to external frameworks as Physical Thinking Prototypes (PTP) (Hutchison, 2017; Hutchison and Vincs, 2013; Hutchison et al., 2015). The novel ability to devise and physically solve complex movement generation systems makes dancers the perfect vehicle for problematizing how to generate movement for and move with robots.

Physical Thinking Prototypes Physical Thinking Prototypes such as McCormick’s artificially intelligent performance agent in Emergence, or Merce Cunningham’s use of Lifeforms/ Danceforms provide dancers with new systems to interact within. Within these PTPs (that which is extrinsic to the dancer), the dancer works collaboratively using their own embodied system to continually mine the potential for movement within the broader system in which they are embedded. Emergence (2013–15) demonstrated the value of established dance practices such as learning from visual examples, mirroring or following processes already found in dance that could be transferred to the teaching of an AI to dance. Perhaps what was unanticipated in the beginning was the transference of the experience of working in a motion capture environment and dancing alongside the agent from the dancers’ perspective as equally rewarding and informing of her practice externally. The virtual tactility is also formative in the collaborative relationship between the dancer and the robots. In subsequent projects the dancer’s construction of the imagined body of the other when developing choreographic frameworks and systems for duet movement exploration enabled the reading of the body of the robot as though another (post)human body. Understanding the embodied system of the robot enables the dancer to engage and interact intimately. This empathy for another agent’s physicality is a powerful tool for projecting appropriate movement onto another body.

Intrinsic movement This research investigates intrinsic movement of humans and robots when moving together rather than other communication methods such as speech,

WHAT ROBOTS LEARN

235

facial expression or symbolic gesture. Dancers are intrinsically embodied and contemporary dance in particular methodically creates movement for its own sake (Stevens and McKechnie, 2005: 243–52). Dancers as movement experts use their proprioception and kinaesthetic awareness to understand how their bodies are transformed by movement (Gemeinboeck and Saunders 2016; Noland 2009a,b). Investigating movement from an embodied, enacted perspective provides an alternative to representational approaches to the functioning of humans and robots in a creative context (Clark, 1997; Manning, 2009; Noё, 2012: 192; Varela et al., 1991: 308). There are many possibilities for using dance to generate movement for robots. Gemeinboeck and Saunders use dancers to prototype movement to test the expressivity of different non-humanoid robot morphologies. Their performative body mapping (PBM) provides a novel methodology for the exploration of robot movement through dance (Gemeinboeck and Saunders, 2016; 2017; 2018; Saunders and Gemeinboeck, 2018). PBM uses dancers’ movement expertise to help design non-humanoid robot morphologies and as a means of generating affective movement behaviours for the robot through the use of dance exploration. Our performance work differs in that it uses two-way movement interaction rather than primarily transfer from human to robot and both non-humanoid and humanoid robots. This research also uses dance exploration with the robots directly and not through costume proxies as in PBM, as the focus is with collaborative movement and not also with design of new morphologies for social robots. Our performance work investigates not only how robots can learn from human example but also how our relationship with robots can be developed through movement.

Learning and training approaches in dance applied to robots and AI Transmission of dance knowledge between human dancers often involves processes that are visual, tactile and verbal. Rapport between two human dancers develops through the perception of these shared experiences and the development of a mutual exchange between them. These same processes have been applied to the collaborative process between a human dancer and robots. In these collaborative relationships the mutual exchange can be sensed literally and metaphorically/virtually in visual and tactile forms. In Dancing in Suits, Hutchison and Vincs (2013) draw attention to the dancer’s perspective of working in motion capture environments and on dancing duets with an artificially intelligent performance agent. Based on this early work by Hutchison and Vincs (2013) for understanding of the environment and of the different technologies as collaborators, our current work positions the systems intelligence as an ecosystem within which the agent’s actions and those of other collaborators interact and affect each other.

236

MATERIALIZING DIGITAL FUTURES

Learning by visual example One of the most used and direct methods for sharing movement is for one dancer to teach another movement. This may be seen as a type of transfer learning with the result that the dancers have a common movement vocabulary to draw on. There are a number of ways we have employed this simple model in the context of human and robot dancers. Motion capture technology was used to enable the dancer to animate the robot. This type of ‘puppeteering’ or remote operation of the robot is usually done in a single direction, human controlling robot. However, the dancer can also allow the connection to affect her body, exploring the differences in morphology and what potential mappings can be experienced through the process of connecting with the robot. A dancer can be sensitive to the nuanced shifts in body movement required to translate into sensible movement for the robot. Where PBM uses physical costumes as limiting structures, here the robot’s body becomes a limiting structure on the dancer’s movement which can lead to unique movement invention. This concept of the robot affecting the human’s body when they are linked is grossly illustrated in our research work investigating the ergonomics of robot-assisted laparoscopic surgery compared to traditional laparoscopic surgery. The operation of a surgery robot places constrictions on the physicality of the surgeon that can lead to musculoskeletal disorders over prolonged use (Hislop, 2020: 31–8). While the specific symptoms are different to those of traditional surgical approaches, many surgeons still report adverse physical symptoms. Even without these ‘hard’ links between human and robot, the use of motion capture to sensibly move a robot requires the human to adapt in some way to the robot’s morphology, changing the human’s own posture. This can lead to interesting movement possibilities for both human and robot performers. The sense of empathy for the other body can contribute to both new movement for the robot and the dancer as the dancer is challenged to work outside of the normal body image and extend their proprioception outwards to include the robot’s body (Kirsh, 2012).

Learning by tactile demonstration The above form of transfer learning is largely visually cued – that is, dancer viewing robot, robot viewing dancer through camera-based motion capture system. Another method for transfer of movement is through touch, tactile moving of one person by another. The dancer moves the robot physically in order to choreograph its movements. This type of learning by demonstration was used as a quick method for recording a robot’s movement through

WHAT ROBOTS LEARN

237

manipulation. It can also be used to inform the robot of the morphology of the dancer and the space surrounding her and give the dancer an understanding of the morphology, volume, weight and physics of the robot. In Eve of Dust (McCormick et al., 2018), the dancer used not just her hands but any part of her body in order to move the robot. This gives both an indication of appropriate movements that could be used by the robot in close proximity to the dancer, as well as the physical extent of the dancer’s body in the negative space between robot and dancer.

Learning by improvisation Improvisation forms a large part of our practice in both performance and research/rehearsal phases of all works. Improvisational structures and taskbased processes, as dynamical systems, used in both performance and in the generation of choreographic material for dance works, can be considered as self-organizing systems – leading the dancer and robot to discover emergent movement languages through an ongoing exchange. The application of dynamical systems in our human–robot collaborations enables us to observe how interactive behaviours are generated (Hansen and House, 2015; Hansen et al., 2014). The dancer works as an agent to facilitate the improvisation. In her role the dancer is searching for kinaesthetic empathy, connection, between the robot and herself. She is paying attention to the relationship that is forming and the behaviour of the robot in different relationship contexts. The process is complex and non-linear as in any given improvisation multiple interactions between subsystems of the body, the task and the environment all play out (Edelman and Giullo, 2000; Fuchs and Viktor, 2008; Haken, 1987; Kelso, 1995; Payne, 2006; Smith and Thelen, 2003; Thelen et al., 2001; van Gelder, 1995).

Motion capture, machine learning and visualization strategies Capturing and re-synthesizing movement and relationships The dance investigations discussed here used the recordings of the movement outcomes to develop their vocabulary. This includes recording the movement of the motors and joint angles of the robot, external volume of the robot and dancer and motion capture of the dancer. This made it possible to replicate developed movements and relationships by playing back the recorded movements.

238

MATERIALIZING DIGITAL FUTURES

Machine learning–reinforcement learning–neural network learning Developing on from the work with NN learning, an advanced method will use machine learning to learn to synthesize aspects of the movement relationship. Deep NNs have been used to synthesize movement based on motion data (Saunders and Gemeinboeck, 2018; Taylor and Hinton, 2009). Reinforcement learning guided by motion capture reference data has been used successfully to simulate robots doing very complex movements while being able to respond to external forces and adapt the movement to their own morphology (Peng et al., 2018). In the early stages of the research, recordings of interesting relationships can be used for reproduction. Later these recordings can become the data to train machine learning models of new collaborative choreographic structures.

Virtual learning, visualization and annotation environment In most of our robotic works a virtual environment developed in Unity was used to capture the motion capture data from the dancer and the movement of the robots, visualizing their relationships and movement patterns. The environments annotate the movement in real time indicating dynamics and relationships. Data captured through the environment was used to train machine learning models to give the performing agents and now robots autonomous movement capability. The environment is able to replay captured movement relationships for fine-grained analysis as well as output the data for external analysis.

Conclusion: Dancing bodies learning together The projects we have undertaken with robots have a common approach, the use of bodily processes drawn from dance and other movement practices in order to develop strategies for a robot’s movement. Choreographic structures have provided a rich source for creating movement from human example and interaction. One of the long-term goals is to have robots and humans able to move sensitively together. Another is to use forms of learning and demonstration rather than traditional programming to enable a more natural interaction in how we work with the robot and how the robot comes to understand how to move with humans. Simple procedures such as physically moving the robot can quickly generate movement for a robot as well as developing human sensitivity to

WHAT ROBOTS LEARN

239

moving with a particular robot and describing how the human might fit spatially in relation to the robot. These same learnings can form the basis for machine learning methods as the movement is already filtered by the trained understanding of the human dancer. Other movement-based practices such as the use of social haptics to interpret and relay information to the visually impaired provide rich structures that can be drawn on in order to generate communicative movement for a robot. As we develop our collaborations with robots combining the knowledge gained from software agents in Emergence with new learnings from working with industrial and humanoid robots, we enter an emerging frontier for artistic performance. The understandings we have developed about the application of choreographic and improvisational dance practice to collaborations with AI and robots lends itself to further exploration. The experiential knowledge we have developed through the projects described in this chapter provide a foundation from which to develop a series of embodied practice methods derived from dance for the development and collaboration with robots. Moreover, the virtual environments developed for these projects lend themselves to be tools for annotation of the relationships between human and robot in performance. The application of this form of annotation may support the development of collaborative and/or symbiotic robots in different contexts where the virtual environment as an annotation tool can assist in analysing the relationship between human and robot and see emergent patterns and/or opportunities for further iteration.

References Barnier, A. J., J. Sutton, C. B. Harris and R. A. Wilson (2008), ‘A Conceptual and Empirical Framework for the Social Distribution of Cognition: The Case of Memory’, Cognitive Systems Research, 9 (1–2): 33–51. Clark, A. (1997), Being There: Putting Brain, Body, and World Together Again, Cambridge, MA: MIT Press. Clark, A. and D. Chalmers (1998), ‘The Extended Mind’, Analysis, 58 (1): 7–19. deLahunta, S. and F. Jenet (2017), ‘Making Digital Choreographic Objects Interrelate: A Focus on Coding Practices’, in M. Leeker, I. Schipper and T. Beyes (eds), Performing the Digital: Performance Studies and Performances in Digital Cultures, 63–79, Bielefeld, Germany: Transcript-Verlag. Edelman, M. G. and G. Tononi (2000), A Universe of Consciousness: How Matter Becomes Imagination, New York: Basic Books. Fuchs, A. and V. K. Jirsa (eds) (2008), Coordination: Neural, Behavioral and Social Dynamics, Heidelberg: Springer. Gemeinboeck, P. and R. Saunders (2016), ‘Towards Socializing NonAnthropomorphic Robots by Harnessing Dancers’ Kinesthetic Awareness’, in Cultural Robotics, 85–97, New York: Springer.

240

MATERIALIZING DIGITAL FUTURES

Gemeinboeck, P. and R. Saunders (2017), ‘Movement Matters: How a Robot Becomes Body’, Proceedings of the 4th International Conference on Movement Computing, Article 8, London: Association for Computing Machinery. Gemeinboeck, P. and R. Saunders (2018), ‘Human–Robot Kinesthetics: Mediating Kinesthetic Experience for Designing Affective Non-Humanlike Social Robots’, 2018 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 27–31 August 2018, Nanjing, China. Haken, H. (1987), ‘Synergistics: An Approach to Self-Organization’, in F. E. Yated (ed.), Self-Organizing Systems: The Emergence of Order, 417–35, New York: Plenum Press. Hansen, P. and C. House (2015), ‘Scoring the Generating Principles of Performance Systems’, Performance Research: On An/Notations, 20 (6): 65–73. Hansen, P., K. Kaeja and A. Henderson (2014), ‘Transference and Transition in Systems of Dance Generation’, Performance Research: On Turbulence, 19 (5): 23–33. Hislop, J., O. Tirosh, J. McCormick, R. Nagarajah, C. Hensman and M. Isaksson (2020), ‘Muscle Activation During Traditional Laparoscopic Surgery Compared with Robot-Assisted Laparoscopic Surgery: A Meta-Analysis’, Surgical Endoscopy, 34 (1): 31–8. Hossny, M., S. Nahavandi, M. Fielding, J. Mullins, S. Mohamed, D. Creighton, J. McCormick, K. Vincs, J. Vincent and S. Hutchison (2015), ‘HapticallyEnabled Dance Visualisation Framework for Deafblind-Folded Audience and Artists’, IEEE International Conference on Systems, Man, and Cybernetics, 9–12 October 2015, Kowloon Tong, Hong Kong, 446. Hutchison, S. (2017), Meta: Discourses from Dancers Inside Action Machines, PhD thesis, Deakin University. Hutchison, S. and K. Vincs (2013), ‘Dancing in Suits: A Performer’s Perspective on the Collaborative Exchange between Self, Body, Motion Capture, Animation and Audience’, ISEA 2013: Resistance Is Futile: Proceedings of the 19th International Symposium on Electronic Art, 16 June 2013, Sydney. Hutchison, S., J. McCormick and K. Vincs (2015), ‘Meta: Notes from a Dancer from Inside a Duet with an AI Agent’, in P. Pasquier and T. Schiphorst (eds), Proceedings of the 21st International Symposium on Electronic Art (ISEA 2015), ISEA International, Canada, 1–6. Kelso, J. A. S. (1995), Dynamic Patterns: The Self-Organization of Brain and Behaviour, Cambridge, MA: MIT Press. Kirsh, D. (2011), ‘Creative Cognition in Choreography’, Proceedings of 2nd International Conference on Computational Creativity, 27–29 April 2011, Mexico City. Kirsh, D. (2012), ‘Running It Through the Body’, Proceedings of the 34th Annual Cognitive Science Society, Mahwah, NJ: Lawrence Erlbaum. Laban, R. and L. Ullmann (2011), Choreutics, Alton, England: Dance Books. Lahtinen, R. (2003), Development of the Holistic Social-Haptic Confirmation System. A Case Study of the ‘Yes’ & ‘No’ – Feedback Signals and How They Become More Commonly and Frequently Used in a Family with an Acquired Deafblind Person, Department of Teacher Education, University of Helsinki. Lahtinen, R. and R. Palmer (2000), Holistic & Interactive Communication Methods with Acquired Deafblind People & Families – A Practical Approach.

WHAT ROBOTS LEARN

241

Joint Training Initiative, Distance Learning Course (incl. Video), Manchester University, UK. Leach, J. (2014), ‘Choreographic Objects: Contemporary Dance, Digital Creations and Prototyping Social Visibility’, Journal of Cultural Economy: Prototyping Cultures, 7 (4): 458–75. Manning, E. (2009), Relationscapes: Movement, Art, Philosophy, Cambridge, MA: MIT Press. McCormick, J. and S. Hutchison (2015), ‘Emergence’, International Symposium on Electronic Art (ISEA) 2015, Vancouver, Canada. McCormick, J., A. Nash and S. Hutchison (2018), ‘Eve of Dust’, Siggraph Asia Art Gallery, Tokyo International Forum, Tokyo, Japan. McCormick, J., K. Vincs, S. Nahavandi, D. Creighton and S. Hutchison (2014), ‘Teaching a Digital Performing Agent: Artificial Neural Network and Hidden Markov Model for Recognising and Performing Dance Movement’, Proceedings of the 2014 International Workshop on Movement and Computing, ACM, Paris, France, 70–5. McCormick, J., M. Hossny, M. Fielding, J. Mullins, J. Vincent, M. Hossny, K. Vincs, S. Mohamed, S. Nahavandi, D. Creighton and S. Hutchison (2020), ‘Feels Like Dancing: Motion Capture Driven Haptic Interface as an Added Sensory Experience for Dance Viewing’, Leonardo, 53 (1): 145–9. Noё, A. (2012), Varieties of Presence, 192, Cambridge, MA: Harvard University Press. Noland, C. (2009a), Agency and Embodiment, Cambridge, MA: Harvard University Press. Noland, C. (2009b), ‘Coping and Choreography’, UC Irvine: Digital Arts and Culture 2009. Available online: https://escholarship.org/uc/item/0gq729xq. Payne, H. (2006), ‘Tracking the Web of Interconnectivity’, Body, Movement and Dance in Psychotherapy, 1 (1): 7–15. Peng, X. B., P. Abbeel, S. Levine and M. van de Panne (2018), ‘DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills’, ACM Transactions on Graphics, 37: 4. Saunders, R. and P. Gemeinboeck (2018), ‘Performative Body Mapping for Designing Expressive Robots’, in Proceedings of the 9th International Conference on Computational Creativity, Salamanca, Spain. Smith, L. and E. Thelen (2003), ‘Development as a Dynamic System’, Trends in Cognitive Sciences, 7 (8): 343–8. Available online: https://doi.org/10.1016/ S1364-6613(03)00156-6. Stark Smith, N. (2008), ‘Nancy’s Introduction – What Is Contact Improvisation?’ in D. Koteen and N. Stark Smith (eds), Caught Falling: The Confluence of Contact Improvisation, Nancy Stark Smith, and Other Moving Ideas, Toronto, ON: Contact Editions. Stevens, C. and S. McKechnie (2005), ‘Thinking in Action: Thought Made Visible in Contemporary Dance’, Cognitive Processing: International Quarterly of Cognitive Science, 6: 243–52. Sutton, J., C. B. Harris, P. G .Keil and A. J. Barnier (2010), ‘The Psychology of Memory, Extended Cognition, and Socially Distributed Remembering’,Phenomenology and the Cognitive Sciences, 9 (4): 521–60.

242

MATERIALIZING DIGITAL FUTURES

Taylor, G. and G. Hinton (2009), ‘Factored Conditional Restricted Boltzmann Machines for Modeling Motion Style’, Proceedings of the 26th International Conference on Machine Learning, Montreal, Canada. Thelen, E., G. Schner, C. Scheier and L. Smith (2001), ‘The Dynamics of Embodiment: A Field Theory of Infant Perseverative Reaching’, Behavioral and Brain Sciences, 24 (1): 1–34. Available online: https://doi.org/10.1017/ S0140525X01003910. van Gelder, T. (1995), ‘What Might Cognition Be, If Not Computation?’ Journal of Philosophy, 92 (7) (July): 345–81. Varela, F. J., E. Thompson and E. Rosch (1991), The Embodied Mind: Cognitive Science and Human Experience, 308, Cambridge, MA: MIT Press.

SECTION FOUR

Digital futures Introduction Media landscapes have gone through major changes throughout history, have been undergoing fundamental changes in recent years and will continue to do so in the near future. The changes have intensified in the last two decades starting with media digitalization, advances in telecommunications and cloud platforms that will eventually result in the end of dedicated infrastructures for content distributions. Time shifting, place shifting, video and audio on demand, and global networks for distribution of audiovisual content are just a few examples of these changes. The implications are, for example, the disruption of traditional media distribution and delivery forms, distribution of content markets, disruption of home video markets, disruption of the record and music industry. Further, there are major changes in the usage and consumption behaviour connected with terminal devices and the combination of audiovisual applications and services with social networking applications. These changes are driven by the interplay between technological developments, market developments, new business models and the policy and regulatory environment. The aim of this section is to critically engage with important driving forces for such changes and to examine what each means for the market, industry, users and technological development. The combination of Smart Home technologies with Smart Appliances is transforming consumer lifestyles. Tasks that were purely a drudgery in the

244

MATERIALIZING DIGITAL FUTURES

past are now easy, and in some cases actually engaging and fun as appliances also act as information and entertainment conduits by way of connected home solutions. In addition, major home appliances are increasingly becoming much easier to maintain and replace as the cloud-based service model transforms products into easy-to-use and informative services. This rapid growth in technology and digital communication infrastructure— coupled with increasing bandwidth and its pervasiveness—alters the interactions between people and their environment. Innovation with broad application is presently found in Mixed Reality (MR) which blends real and virtual worlds along the reality–virtuality continuum; one in which MR and Virtual Reality (VR) can provide an immersive experience in either real or virtual environments. Being in a 3D environment means that a person can make use of their natural ability for spatial interaction. Well-designed interfaces make interaction and collaboration inside such systems more natural, effective and engaging beyond entertainment purposes alone. This final section is a study of possible futures for citizens communicating in the digital age through the ubiquity of computing hardware and software, satellite usage, and an advanced telecommunications infrastructure. It considers notions of being a post-industrial society among other elements. For their vanguard understanding of materializing digital futures, we interviewed Joe Millward who is an Innovation Manager with over eighteen years’ experience in Fortune 500 companies across diverse sectors including education, manufacturing, fast-moving consumer goods, computer hardware and software development. Recognized for his expertise in the immersive technology industry, Joe Millward is currently leading a team to research the latest emerging technologies for TAFE NSW, an Australian vocational education and training provider in New South Wales (NSW) established as an independent statutory body under Tertiary and Further Education TAFE Commission Act 1990. Joe Millward critically explores the potential of augmented reality (AR) and VR technology for how it will enhance education, training and industry more broadly: Joe Millward: There is an erroneous expectation that people think mainstream use of immersive virtual reality is coming tomorrow, then they are disappointed in three years’ time when it hasn’t arrived. Important to consider then is what the building blocks are that we need to have in place when the technology arrives. So, what I’m doing now is I’m taking steps to ensure that our content is not invisible when we move away from screen to spacial media. Everything needs to be in 3D, assets need to have a digital double for a recreation or simulation opportunity, or spacial mapping is required so we can apply a scenario back to the real world from the 3D one. That’s the preparatory work that is happening at the moment and much is happening in the background.

DIGITAL FUTURES

245

For example, Facebook is very clever with what they are doing with their Project Aria [a research project being undertaken by Facebook Reality Labs to build the first generation of wearable augmented reality devices] because they have been spending a couple of years now understanding how humans interact with the world. This is the perfect way you do it for a human-centred design approach. If you understand how things work before you build something as opposed to all of these clunky attempts where (a) the hardware is not quite mature enough, and (b) the infrastructure is not there yet. In terms of the privacy of personal digital data that is collected there is a lot of ‘privacy theatre’ as opposed to actuality and I am not quite convinced that all the giants’ privacy protections they portray are accurate. But Facebook, on the other hand, may give you the glasses for free and you simply give away your entire databank. That is going to be a really interesting dynamic because of the types of data and range collected and ultimately what is done with it. I am looking to the blockchain platforms such as Hedera [a purportedly more secure alternative to blockchain consensus mechanisms that use Hashgraph to securely verify transactions between users]. Arguably, Blockchain and AI convergence is inevitable as each concerns data and value. Blockchain for example enables secure storage and sharing of data or anything of value whereas AI can analyse the data and generate insights from the data to generate value. Hedera assess issues of privacy, identity and security to work out the possibilities for data. We need a mechanism to expose what the value of data actually is. Once you understand the value of your data then it changes the dynamic as soon as it has a dollar value. Unfortunately, I think it will be something that the wealthy will have access to first. Of course, there are definitely long-term negative aspects, but there are also positive aspects as well. I do think data security and digital privacy require people to have influence over decision mechanisms. Developers also need to ensure their products are authentic. I remember walking down a platform at Central Station and I had these two ladies stop me, both of whom would have been in their late-seventies earlyeighties, to ask what I was wearing on my Face. I had on a pair of Google glasses which they asked to try on. They asked if they could display natural language translation because they would be traveling overseas and wanted to have conversations with people in their language as they only spoke English. Their request made it clear that having a distinct purpose is where the adoption of this sort of technology can be of value to consumers. Apart from bridging communication divides, another problem the technology can be applied to is in the construction industry. The construction industry is a dangerous field, and we want to deliver

246

MATERIALIZING DIGITAL FUTURES

simulations that would be hazardous to do in the real world or something that you would have to practice, such as using a welding device or a handsaw, in a safe environment. Similarly, we are developing tools for the medical field where incredibly expensive equipment is used, but students might not be able to get access to learn on an apparatus so we can simulate an action prior to actual exposure and application. Another very bright area is for virtual presentation where you might have an amazing speaker who is incredibly engaging, but they are only one person and audiences don’t have direct access to them. A developer can take the essence of their presentation – content including the way they speak, their enthusiasm and capture their mannerisms in a generated hologram or avatar with AI driving the performance which is nonlinear and one-on-one with hundreds of people. In this example, the AI demonstrates step-by-step procedural aspects, but then the speaker or teacher watching their hologram performing can dive into its system to assist with more complex elements on topic. Everyone might have their own personal tutor and it will be Einstein or they may take a Masterclass with film director Federico Fellini. We know some are more comfortable engaging with a non-human avatar than with people in real-life situations for reasons of race, gender or disability. There may be an advantage for their communication, learning and work prospects where they have the ability to digitally converse using a personalized avatar. A number of years ago at Disney World Resort was an experience between a young boy and a character from the film Finding Nemo called Crush that is real-time animated and speaks to the children. Crush swam up to this boy and asked his name and the child responded and then a two-minute conversation followed between the pair. After the show, the parents were highly emotional because their child was diagnosed as autistic and non-verbal, never having spoken to his parents, yet was able to with this avatar. Another positive affordance is giving a person with paraplegia the ability to embody an avatar with full mobility. These are really significant steps into experiential space such that if one cannot be somewhere physically then at least these digital means can be used to have intimate experiences of place or interaction and I think the evolution of this will come with digital fine-tuning. In this final section, Chapter 13 discusses smart home technologies and how the introduction of Smart Appliances might transform consumer lifestyles. Xi Cui explores the uses of smart home devices and how they shape—and are shaped by—the sense of everyday normalcy soon after their introduction into the home environment. The author emphasizes that the smart home devices are designed to be invisible and ‘unremarkable’ and gradually do fade into the background of everyday activities. For their

DIGITAL FUTURES

247

critical investigation, the author draws qualitative and quantitative data from home visits, through interviews and use-logs respectively to discover three major ways in which the smart home devices are related to the participants’ phenomenological experiences of everyday home life. They assert that the selected smart home devices arrange the action in the home whereby they align the atmosphere through, for example, body ballets, time–space routines, and, when multiple persons are present in the same space, place ballets, which adds a sense of sociality through interpersonal, spatio-temporal sharing to the experience of the environment. Xi Cui concludes that the uses of the smart home devices sit between structure and agency in ordinary domestic life. From a close examination of the future home to place more expansively, John MacWillie considers in Chapter 14 the impact on our lived experience ‘at home’ through a critical perspective of contemporary warfare. The author outlines that for millennia, military strategy has focused on dominating physical domains such as land and sea. In the past century, the domains of air and space have been added to this strategic focus. Over the past few decades, the significance of these domains has been increasingly displaced by constructs of data and information interconnected by computers, applications, databases and networks. Political and economic decision-makers, as well as consumers, have come to rely on these digital assets to cognitively understand the organization of the world, as well as make decisions about how to best achieve one’s interests. While much has recently been written about cyberwar, information war and cognitive war, this chapter addresses the theoretical and practical implications of the next frontier in this competitive race where operatives, using digital platforms, seek to dominate the affective layer of human experience. This frontier is one where the target is not cognition or perception but feeling and emotions or what MacWillie lucidly refers to as affective warfare. The ensuing Chapter 15 is a considered account of a fully self-directed automated future being a myth. Luke Munn takes a step back from a focus on potential universal automation – where associated technologies are framed as a de-situated force that will sweep across societies and ripple across the globe – to argue instead that this expectation needs to be reconsidered. That is, while the myth of automation holds out the promise of economic prosperity it simultaneously warns of a set of potentially disastrous effects. Munn states that far from being self acting, instead we encounter, again and again, the fleshiness of human labour. On a warehouse floor a fleet of automated guided vehicles might dart back and forth, each shuttling a package to a chute, but follow the packages and you will find laborers collecting them on the floor below, placing them in bags and trucking them to ports for shipping. Moreover, for the author, labour is socially stratified and so automation’s fallout will also be highly uneven, falling heavier on some populations (immigrants, people of colour, women) and lighter on others.

248

MATERIALIZING DIGITAL FUTURES

Munn’s chapter offers a nuanced, localized and racialized understanding of automated technologies and their intersections with labour regimes. In Chapter 16 by Renata Morais the landscape of media artefacts is analysed for how technological networks give rise to new modalities of material mediation. By way of context, Morais foregrounds emerging technologies from the micro dimension (nanotechnologies, quantum computing) to the macro dimension (space exploration, pervasive intelligence, planetary computation) as bringing new sightlines for material mediation in the context of entangled human–computer interaction (HCI). In order to establish a normative framework for unpacking processes of technological convergence, Morais devises three transdisciplinary categories of material mediation – ontological, intentional and vital. These categories are aligned to three main typologies of material mediation. The first of which is subject-oriented mediation that focuses on semiotic perception and translation according to subjective interpretation. The second is objectoriented mediation that focuses on artifacts and technology networks, infrastructures and assemblages. The third is subject–object interaction that focuses on the relational dynamics and properties of mediation between the two. This concluding chapter underscores the importance of understanding the socio-material world that moves beyond the socio-natural and cultural– material dualisms that have characterized modernity. Readers are called to consider that ecological categories such as planet, environment, habitat and species have also become categories of material mediation.

13 Smart home: Smart devices and the everyday experiences of the home Xi Cui

Introduction: Smart home devices and notions of home Internet-enabled smart home devices have seen an exponential increase recently (Statista, n.d.). Although the fast adoption of these new technologies in homes shows a striking resemblance to the historical popularization of radio and television in the home space (Moores, 1988; Morley, 1988; Scannell, 1996; Silverstone, 1994; Spigel, 1992), empirical studies on the cultural implications of these smart technologies are scant (e.g. Hine, 2020; Spigel, 2010) outside those concerned with computer designs (Harper, 2003; 2011). The imperative to study this new set of communication technologies in the home, this chapter contends, rests on the following. First, the smart devices are designed to be invisible and ‘unremarkable’ (Spigel, 2010). As time goes on, these devices will gradually fade into the background of everyday activities, despite their significant potential in reconfiguring life in the home like television did (Morley, 1988; Silverstone, 1994; Spigel, 1992). Second, not only will our uses of them become habituated and embodied (Moores, 2017), our understandings of the ‘idealized home’ (Silverstone, 1994) and the everyday normalcy may also be reconfigured, disguising the

250

MATERIALIZING DIGITAL FUTURES

power relationships behind the norms of gender, class, parenting, ageing and so forth (Hine, 2020; Moores, 2017; Spigel, 2001; 2010 ). Finally, these connected devices reconfigure our homes in ways beyond the ability of previous generations of mass media. The smart devices are connected to the internet and permeate the boundaries around the home with constant data flows (Lupton, 2020). This will inevitably reconfigure the relationships between the inside and the outside, the private and the public, and autonomy and control. The following sections first provide a review of the socially constructed nature of the home. The main focus is on: (a) how the phenomenological experiences of the home environment contribute to the idea of home, and (b) how modern technologies interact with the sense of normalcy in the home. Next, the research participants’ uses of the smart devices are thematized in terms of their environmental experiences of the home. These themes are then articulated to the sense of everyday normalcy and the potential for change in the home.

The socially constructed idea of home We tend to conflate the idea of ‘home’ with the place in which we live. Home, Silverstone (1994: 45) argues, is ‘the product of our practical and emotional commitment to a given space’ and it is a ‘reality in which our identities are forged and our security maintained’. It is the mutual constitution of the physical environment and, more importantly, our practices, meanings and affects towards it. This study focuses on how people’s uses of smart home devices interact with their experiences and understandings of the home. Home is first and foremost a place that is meaningful to us. Tuan (1977) in his analysis of our geographic experiences distinguishes a space from a place. Space is associated with openness, infinity and movement, whereas place reminds us of security, certainty and dwelling. A place is a space that is given ‘geometric personalities’ through our activities in it (Tuan, 1977: 17). When we are new to a location, the space around it is blurred, unsorted and ‘out there’. It only becomes a concrete reality and feels like home with our full engagement ‘through all the senses’ (Tuan, 1977: 18). The mundane activities in everyday life, many of which certainly involve communication technologies, give a space meanings and constitute the experience of at-homeness, that is, the ‘unnoticed, taken-for-granted situation of being comfortable in and familiar with the everyday world in which one lives’ (Seamon, 1979: 70). Meanwhile, the individual practices constituting the home are also part of the larger social, economic and technological contexts. Putnam (2006) and Spigel (1992) traced the modern history of the home in the United Kingdom and the United States, respectively. They showed that our domestic space

SMART HOME

251

as well as the activities in them had been transformed by the infrastructure such as sewers, electricity, telephone and television in the twentieth century. In these social processes, the functions and meanings of different home spaces have been transformed along with the socialization among family members. Moreover, home is also built upon ideologies. We tend to use words like haven, warmth, security, and the like to describe the ideal home even though conflicts and suffering are often part of the home experience, especially for certain socioeconomic groups (Hine, 2020; Pink 2004). Indeed, the home as an isolated and apolitical space is closely linked to (mostly Western white male subject’s) individual freedom from the period of Enlightenment onwards (Kaika, 2004). More recently, the idealized home is also predicated against the contrasting concepts of outside/urban factory/impoverishing/ dangerous environments on the one hand and the inside/suburban residence/ nurturing/safe aspects on the other. Media technologies in the home play a critical role in the ideological construction of home (Morley, 1988; Spiegel, 1992; 2001). As we will see, a constant negotiation of ideology and power is disguised in everyday mundane practices, from cooking to dressing to reading in the home (Pink 2004; 2012).

Home as environmental experiences Most of the social, cultural and technological influences shaping the home are only felt, if at all, through the everyday dwelling in the home. To most people, the home is not an abstract reflection but direct phenomenological experiences in the home environment. Cultural geographer David Seamon (1979: 15–16) defines one’s everyday environmental experience as ‘the sum total of a person’s first-hand involvement with the geographical world in which he or she typically lives’. His phenomenological perspective is also echoed by many media and cultural scholars who try to understand how communication technologies intersect with the subjective experiences in the home (Morley, 1988; 2005; Pink, 2004; Spigel, 1992). Seamon (1979), through his longitudinal study with the members in the Environmental Experience Group, extracted three essential themes of our experiences with the living environment: movement, rest and encounter. Movement refers to ‘any spatial displacement of the body and bodily parts initiated by the subject’ (Seamon, 1979: 33). In the experiences of the home environment, movements consist of body ballets, time–space routines, and, when multiple persons are present in the same space, place ballets. Body ballets refer to ‘a set of integrated gestures and movements which sustain a particular task or aim’, while time–space routine is ‘a set of habitual bodily behaviors which extend through a considerable portion of time’ (Seamon, 1979: 54–5). Much of the activities carried out in the home

252

MATERIALIZING DIGITAL FUTURES

are subconscious fusions of body ballets and time–space routines. Place ballets, in the meantime, add a sense of sociality to the experience of the environment. In place ballets, the environment becomes a meaningful place through interpersonal, spatio-temporal sharing. The smooth carrying out of everyday movements in the home contributes to our sense of thorough familiarity and social affinity, or what Edward Relph (1976: 55) calls the ‘existential insideness’ in the home. The choreographed ballets in the home, individual or social, spontaneous or routine, are frequently evidenced in media scholarship. The traditional place ballets of the husband reading in the library, the wife knitting in the bedroom, the children playing in the back parlour or the guests meeting in the front parlour were gradually reconfigured by the introduction of television for family viewing (Spigel, 1992). Personal computers also transformed the space–time routines of many homes as family members mutually accommodate each other’s needs and computer skills (Frohlich and Kraut, 2003). Part of our experience of the home arises from the smooth and choreographed ballets in the home’s physical spaces. Ballets in the home are complemented by ‘rest’, which refers to situations where one’s contact with an object is relatively fixed in a place for a period of time (Seamon, 1979). In other words, rest establishes a sense of ‘centre’ in our everyday experience that we might orient ourselves towards. If ballets establish the body-subject, that is, the inherent capacity of the body to function as a kind of subject which acts intelligently and expresses itself in a pre-conscious way (Merleau-Ponty, 2002), rest gives rise to the ‘feelingsubject’ (Seamon, 1979: 75), a matrix of emotional intentionalities within the person which sustain positive feelings for well-used centres and places, and negative feelings for centres that are changed in some way. The experience of ‘rest’ in the home cultivates the feeling of at-homeness (Seamon, 1979), a direct result of subjects’ orientations towards the established centre without conscious reflection on their experiences. Affectivity and identity arise with the sense of at-homeness. This is quite often achieved through the location of furniture and appliances, and the decoration and styles chosen for the home (Spiegel, 1992). When radio, television and even futuristic smart home technologies are introduced into the homes, both their design, placement and their contents intertwine with the subject-feeling of at-homeness (Moores, 1988; Scannell, 1996; Spigel, 2010), illustrative of Seamon’s rest dimension of the home experience. The last aspect of Seamon’s (1979) conceit with regard to the environmental experience of the home is ‘encounter’. This refers to any contact of various degrees of attentiveness between one and the world at hand. Although encounter can vary between oblivion and heightened attention, home normalcy is characterized by what Seamon (1979: 11) calls ‘basic contact’, the ‘preconscious perceptual facility of body-subject’. It provides a ‘perceptual matter-of-factness, which comes to light only when

SMART HOME

253

the world is changed in some way’ (Seamon, 1979: 117). In the mediasaturated home environment, basic contact is provided by what Bausinger (1984: 346) calls ‘inconspicuous omnipresence of the technical’. Radio, for one example, when relocated to the more private backroom from the formal front parlour became increasingly used for companionship and background ‘noise’ against other activities (Moores, 1988). Even smart home designers strive to achieve the unmarvellous status of marvellous technologies (Tolmie et al., 2003). These smooth experiences of basic contact in the home environment maintain the sense of order and norm in everyday home life. Therefore, home, as reviewed here, comprises both experience and cultural constructs that are shaped in numerous social processes. The technical infrastructure, spatial arrangement, ideologies as well as the body-subject and feeling-subject should all manifest in the ways smart home devices are placed, used or rejected in the home. These conceptualizations provide us with lenses through which to view the role of smart devices in the home as both shaping and being shaped by our experiences and understandings of the home.

Everyday experiences of communication technologies in the home The home is important in the constitution and articulation of the sense of normalcy. Our ballets and rest in the home as well as the precognitive basic contact with the home environment all signal a sense of business-as-usual for us (Seamon, 1979; 2018). When we are ‘at-home’, a sustained attitude of familiarity and at-easeness permeates our daily existence. However, media scholar Shaun Moores (2006) raised a few caveats against directly employing Seamon’s theory to understand the sense of normal life in the media-saturated home. First, he argues, Seamon seems to have brushed aside the importance of the media because, following cultural geographers like Edward Relph, mass media are considered to promote placelessness, which undermines the idea of home. Second, Seamon’s phenomenological approach prioritizes subjective experiences and ignores the structural contexts of the experiences. Last, Seamon’s categories of movement in ballet, rest, and encounter tend to idealize the home experience and neglect exclusion and conflicts in the everyday life in the home. Therefore, to understand the everyday experience of the smart home, Seamon’s theory must be articulated based on the way the sense of normalcy is constructed, especially through communication technologies in the home. Two lines of research studying the experience of communication technologies as part of the everyday life in the home complement Seamon’s theory by addressing some of Moores’s concerns. First, cultural studies on the domestication of technologies saw technologies as part of a complex web of values, relationships, symbols and routines that make up domestic

254

MATERIALIZING DIGITAL FUTURES

life (Habib and Cornford, 2002). Silverstone (1999: 251) defines the process of domestication as ‘fitting and fixing the new into the familiar and the secure, moulding its novelty to the needs, desires, and culture of the family or household’. Second, the computer-supported cooperative work (CSCW) research comes from computer science scholars focusing on designing computer systems that support collaborative activities and their coordination in daily work and home environments. They have a practical concern of helping the new technologies to integrate smoothly into everyday activities in the home (Harper, 2003; 2011). Despite their divergent conceptual goals, both approaches conduct ethnographic studies and articulate the everydayness of media uses in the home to three experiential dimensions: routines and repetition, embodiment, and ready-to-hand.

Routine and repetition The sense of everyday normalcy partially comes from the spatio-temporal repetition of mundane activities. Reiss (1981: 238) argues that life in the home largely follows what he calls ‘pattern regulators’. He distinguished these taken-for-granted daily routines from the high-intensity ceremonials that are episodic, consciously mandatory and emotionally charged. It is precisely these pattern regulators, Reiss argues, that shape and stabilize the minute and subliminal sense of normalcy day in and day out. In media studies, Scannell (1996) interprets the institutionalization of broadcasting from programming schedule to predictable genres as efforts of routinization through repetition. Our routinized engagement with the mass media, he argues, maintains our sense of the world’s everydayness. In CSCW research, Crabtree and Rodden (2004) and Crabtree and Tolmie (2016), through ethnographic accounts, found that information uses in the home comprise sequences of activities that are spatio-temporally repeated on a regular basis. These routines and repetitions are not simply what home dwellers do repeatedly, but what acculturate them in a specific environment. By studying the extraordinary spatial ability of the Arctic indigenous population in North America living on the snow-covered tundra, Tuan (1977) finds that spatial skills and awareness lie in the repeated performance of ordinary daily tasks in a specific physical space. What is more, when people live under the same roof, their routines interact regularly to form, what Seamon (1979) calls, place ballets. Frohlich and Kraut (2003) found that in the early days of home computer use, family members coordinated their computer time and developed highly predictable sequences of activities on the computer (Frohlich and Kraut, 2003). Further, within the first few months online, their monthly log-on time first exponentially increased and quickly stabilized as routines developed. Depending on the locations of computers, family members’ socialization was either increased or decreased. Social meanings

SMART HOME

255

of relationships, gender roles, hierarchies also arise, sustain and change in these routines as part of the normalcy in the home. However, normalcy in the home is not just repeating exactly the same activities. When objects are moved around, spaces are occupied and people’s paths intersect, we do not necessarily feel that normalcy is disrupted. Rather, we try to adjust accordingly to sustain the at-homeness. This adjustment rests on another aspect of the sense of normalcy, being ready-to-hand.

Ready-to-hand ‘Ready-to-hand’ refers to our practical relationship with the world that is handy or ‘graspable’ (Heidegger, 1962: 99). If the environment we live in is ready-to-hand, we use it without theorizing it. In this way, the ordinariness of life not only appears as fixation of particular activities but a general attitude regarding the essence of the situation (Merleau-Ponty, 2002). Only when the world becomes unready-to-hand, by not working the way it is supposed to, will it rise to our consciousness and become abnormal. Our engagement with the home through ballets, rest and basic contact in normal life as just examined depends on the home environments being ready-to-hand. Scannell (1996) argues that media, as an integral part of everyday normalcy, always works in accordance with our practical knowhow and never breaks down, and further the contents also make sense to us, appear intuitive and graspable. Tolmie et al. (2003) argue that for smart home devices to become ‘unremarkable’, they should focus on the users’ instead of the designers’ ‘semantics’ of the devices. Only when the devices make sense to the users in their own home environment, in other words when they are ready-to-hand, can the devices become part of normalcy. To the contrary, however, Jakobi et al. (2017) found that the user interfaces of early smart home devices fragmented the flow of the users’ experience. As such, they constantly rise to our consciousness for reflection and compromises the sense of everyday normalcy. The concept of ready-to-hand complements the functionalist tendency of the routine and repetition aspect of everyday life and offers explanatory power for our apt adjustments to everyday variations, including those about the communication technologies in the home.

Embodiment The reason that both repetition and ready-to-hand appear normal in our experiences of everyday life is that they do not rise to our conscious reflection. They are embodied (Merleau-Ponty, 2002; Moores, 2017). The body knows, intelligently, how to do and how to feel. In Merleau-Ponty’s

256

MATERIALIZING DIGITAL FUTURES

phenomenology (2002), the body-subject is the inherent capacity of the body to intelligently direct one’s behaviour and express a special kind of subject that is precognitive. This perspective provides an alternative to the body–mind dualism in modern thinking which trivializes the repetitive and routinized behaviours in everyday life. In Seamon’s (1979) environmental experience, ballets, rest, and basic contact all constitute the body-subject and its relationship with the home. Sarah Pink (2004; 2012) through her sensory anthropology explicates how sensory experiences in the home directly guide our feelings and behaviours in the home to constitute the sense of normalcy. Pink, however, takes a more critical stance than Seamon to show how expressivity can also be embodied in the most mundane and routine activities to negotiate power and induce change. Cleverly playing on the dual use of ‘digital’ to simultaneously refer to media and a finger (now used frequently in tandem), Moores (2017) calls our embodied experience in today’s highly mediated world a ‘digital’ orientation. He argues that our relationship with the sense of place and the sense of normalcy is largely embodied in media uses. Tolmie et al. (2003) in CSCW research argue that smart devices should augment the users’ embodied actions in the home instead of dictating how users use them. Only in this way can smart home dwellers develop an orientation towards normal life in a home with smart devices. Taking routine and repetition, ready-to-hand, and embodiment together, we can articulate Seamon’s ballets, rest and basic contact dimensions of environmental experiences to the sense of everyday normalcy in the home. Keeping these diverse conceptual approaches in mind, we now move to the ethnographic interviews with smart home dwellers to answer the following research questions: (1) Where are smart home devices located? (2) How are they used? (3) How do the uses shape – and how are they shaped by – the everyday experience of the home.

Sample and data collection For this present study, 12 residents were recruited in Charleston, South Carolina, in the United States through local community Facebook groups, the researcher’s personal connections as well as some participants through a snowballing recruitment technique. Volunteers were asked to fill in a form indicating their neighbourhoods in the greater Charleston area, age, range of household income, number of people living in the home, ownership of any smart home device and contact information. Out of 39 volunteers, 16 volunteers were contacted and 12 consented to participate. The participants varied in their age, income and the number of residents in the home. Ages were in the ‘20–29 years’ to ‘70 years and above’. Their incomes were in the range ‘under $30,000’ to ‘above $90,000’ per year. The

SMART HOME

257

number of residents in the home ranged from 1 to 7 with 6 families having either a young child or children. To understand how home experiences may change or not with the introduction of smart devices, none of the participants had already been using smart devices prior to this study. The small sample is not meant to be representative of overall smart home device users. The demographic variations in terms of age, income, occupation, and family composition might, however, help us link idiosyncratic smart home users to potential patterns in environmental experiences of the home. The study was conducted in late 2019 and early 2020 and spanned 4–5 weeks for most participants with use-log collections and home interviews. At the beginning of the study, each participant was given one Google Home Mini smart speaker, one General Electric smart lightbulb and one General Electric smart plug to use as they saw fit in their home. The researcher helped one elderly couple set up the devices in their home and other participants set up the devices on their own based on the product manuals. During the second week, the researcher checked in with the participants to see how the devices were working but they were neither pushed to use the devices more frequently nor directed to specific uses. During the home visits and use-log collections, the researcher drew a simple layout of each participant’s home and the location of the smart speaker, smart lightbulb and smart plug. The participants went to Google’s personal data management site to retrieve their use data of Google Assistant which is linked to the smart speaker. A detailed instruction was given to the participants for them to review their activities and delete any record they did not wish to disclose. After any deletions, the participants’ use logs were converted to PDF and emailed to the researcher. Each interview lasted between 40 and 75 minutes. The data analysis takes a typical grounded theory approach (Glaser and Strauss, 2017). Each activity in the use log is considered in terms of the participant’s background, the content of the commands, the time of activity, the frequency of it in the overall use log as well as the location of the devices in the house. The interviews were analysed in conjunction with the participants’ use logs. Attention was given to the rationales for specific uses, the descriptions of the experiences of the home as well as their home routines and everyday rhythms. The research questions regarding the location, uses and their relationships to the experience of normal life in the home are answered in the following thematic analysis.

Locations of the devices: The centre– margin continuum Locations of the smart devices not only reveal how participants understand these new additions to their homes, but also influence the temporal and

258

MATERIALIZING DIGITAL FUTURES

social dimensions of the uses. It seems the smart speakers are either positioned in a convenient location of a specific room such as on a nightstand in the bedroom or at the intersection of foot traffic in the house such as the hallway. Tuan (1977) argues that our environmental experiences are always organized by certain social or cultural centres. They not only guide our movements in the space, but also provide a centre for our worldview. For the smart speakers that were at a convenient location in a specific room, their uses are also understood to be directly tied to the activities in that space such as the kitchen or the bedroom. For example, Participant 2, a single young professional living in an apartment, installed her smart speaker in the bedroom. Her uses of the device, as revealed in her use log, were mostly related to sleep timer, sleeping music and some random interpersonal dialogues with the Google Assistant. On the other hand, some participants chose to install the speakers at the intersection of multiple spaces such as the hallway connecting the house door and the living room or that space between the living room and the bedroom. These locations are almost ‘liminal’ in the sense that it is hard to define where intended sound was directed, or the rooms being covered. Participant 8, a single young professional living in an apartment, bought a wall mount for the speaker which directly plugged into a wall outlet in the hallway outside her bedroom/bathroom leading to the living room. Her device uses were mixed including playing news in the background for breakfast, controlling the hallway light, playing soothing music for sleeping and setting alarms to wake her up in her bedroom. The uses of the smart speakers at different home locations seem to be geared towards either the rest or the ballets dimension in the environmental experiences of the homes (Seamon, 1979). Speakers located in the centre of the living room or the bedroom, as will be detailed in later themes, seem to be mostly used to construct the environmental quality through music, lighting and socialization activities such as trivia games and, for one family, parents-and-kids calisthenics exercises. They accentuate the existing meanings of the space mostly through cultivating the at-easeness and the emotional experiences associated with the space. For speakers located at the intersections of spaces within the home, the uses did not take on the meaning of a particular space. Rather, they were more dependent on the geography of the homes, their family members as well as their family culture. It is in the ways that different users see them as being useful, or ready-to-hand, that define the uses. For example, having their smart speakers installed in the hallway between the home’s entrance and the living room, Participant 8, a single professional in a small apartment, used the speaker extensively for news, music and lighting control whereas Participant 3, the mother of a family of five only used it to control the welcoming porch light outside the home.

SMART HOME

259

It should be noted that sometimes the participants did try to achieve the convenience of ‘fluid’ purposes by slightly tweaking the placement of the devices. Therefore, the dynamic between the central and the marginal use-locations should be seen as a continuum. For example, Participant 2, although installing her smart speaker in the bedroom, places it on the chest opposite to the bed near the bedroom door which leads to the living room instead of on the nightstand at the far end of the bedroom. Therefore, although the smart speaker was mostly related to her activities in the bedroom, as she describes in the interview, she did sometimes ask recipe questions of the device when she was in the open kitchen on one side of the living room. In addition to the locations of the devices, to understand the mutual constitution between the participants’ uses of them and the everyday experience in the home environment, in the following section, the relationship is thematized into three dimensions: (1) arrange the action; (2) align the atmosphere and (3) arrest the attention. It is important to note that the themes are conceptualized in terms of the relationship between the uses and the environmental experience in the home. They do not correspond to particular uses or smart speaker commands. The same use, as will be shown in the following, may manifest different dimensions of the environmental experience in different homes.

Arrange the action In many circumstances, the smart speakers, lightbulbs and plugs are used to organize one’s own bodily movement as well as coordinate the actions of other family members. The uses that are related to the environmental experiences by arranging the actions in the home fall under two large categories, random burst of needs and routines of repetition. The burst of needs use seem to be satisfying unpredictable but practical information needs such as ‘what’s the equivalent of one cup of flour and grams?’ while cooking, or ‘when will the rain stop?’ These uses, as reviewed earlier, contribute to, or undermine, the sense of everyday normalcy in the home through the devices being ready-tohand or not. They respond to the users’ requests, either by answering questions or turning on and off the lights or appliances, whenever the users need them to. The frictionless satisfaction of the burst of needs eliminates the necessary movement of walking to the switch or searching for the information on the phone or computer. For the burst of needs use, there are certainly times when the devices are not ready-to-hand and compromise the sense of normalcy in the home. Participant 11, after failing to control her Roku smart television, had to resort to the remote control and reported disappointment about the device and a sense of

260

MATERIALIZING DIGITAL FUTURES

frustration. However, like she said, ‘you gradually learn what it is capable of … I simply don’t bother asking for the things too complicated for “her” [the smart speaker]’. The ready-to-hand aspect of the smart devices is indeed an understanding built through the numerous interactions between the user/s and the devices over time, in other words, through living in a smart home. The second way to arrange the actions is through routines and repetitions. These actions (ballets) are highly repeated spatio-temporally. Some participants in the study gave the command to turn the lights on and off at a certain time on weekdays or weekends, and other participants simply set up a routine through the app to automate these commands. Participant 4, a work-from-home dad of a family of five people, used the smart light installed in his 9-year-old daughter’s room to wake her up in the morning. He said that it ‘saves me a trip every morning’ when he uses his phone to turn on the light. Participant 2, a single young professional, used their smart device to check the weather and traffic to determine if she had more time for her morning routines or needed instead to rush out of her home. Participant 8 lined up a series of news shows from major news media to play on the command of ‘play news’ while preparing and eating breakfast. Their routines as well as the sequence of media content resemble how newspaper, radio and television themselves construct the sense of everyday normalcy in the home (Bausinger, 1984; Moores, 1988; Scannell, 1996). These uses are directly related to the everyday rhythm of Seamon’s (1979: 54) time–space routines, ‘the habitual bodily behaviors which extend through a considerable portion of time’. The repetition of these behaviours over time becomes ritualistic and gains norma(l)tive significance. By reducing the constant vigilance and dulling the sense of striving, the bodysubject joins the feeling-subject as manifested in the embodied routine behaviours and the sense of at-homeness. However, the breaking of it, like when Participant 4’s daughter fell ill and did not go to school, becomes ‘chaotic’ as he puts it and invoked ‘a lot of anxiety’. By arranging the ballets in the home, these uses are not only about eliminating movements, but sometimes also enable participants to move more freely in the home environment. Participant 11 said in the interview: I like using the music function coz before I’d have to play it through my phone, which automatically drains the battery ... So now that I’m playing it through there and it’s just streaming through the Wi-Fi, it’s not using any power from my phone, which is nice. By not keeping the phone charged, this Participant was free to move around the apartment and used her phone anywhere.

SMART HOME

261

Align the atmosphere Another important aspect of our environmental experience of the home is the ambience. Not only is what we do or not do important to define the home as suggested in the previous section, but the quality of the environment where we do or do not do things also provides or compromises a sense of home. By ‘align the atmosphere’, I refer to the smart home uses that are directly related to the manipulation of the environmental quality through sound, sight and sociality. As conceptualized by Seamon (1979), rest refers to situations where one, or one’s contact with an object, is relatively fixed in a place for a period of time. This is when people appropriate the home environment to further their particular understanding of the ideal home. Participant 2 likes to read in bed before going to sleep and she manipulates the brightness of the smart lightbulb in her nightstand lamp to create a ‘comfy environment’. Participant 6 asks for specific songs when she is in the kitchen or the bedroom. She explains in the interview that the frequent adjustment of the volume as shown in the use logs is due to her either working in the kitchen or in the bedroom. At the end of the day, she always puts on ‘country sound for 30 minutes’ to help her ease into sleep. The sound and sight are important experiential aspects of the home. As Pink (2004) argues, people sense everyday normalcy through their particular sensory experiences in the home. These mundane sensory experiences constitute larger structural power relations such as gender and class. Other than the sound and sight quality of the home environment, smart devices are also used to align the atmosphere in social situations such as a trivia game or a dance party in the home of Participant 4, the work-fromhome dad: The kids like what they call their own dance party. So, they’ll go out and pick a song and dance along and they’ll … get entertained for themselves for an hour … each of them would go back and forth picking songs and playing on it and there is dancing in there and they’re going nuts. The sounds, lights and trivia games played through the smart devices contributed to the sense of at-homeness and attachment through the embodied feeling-subject. Immersing oneself in a certain atmosphere can ‘negate a person’s awareness of directional time and space’ (Tuan, 1977: 128). Especially the rhythms of music, dance, turns of talk and laughter can ‘free people from the demands of purposeful goal-directed life, allowing them to live briefly in what Erwin Straus calls ‘presentic unoriented space’ (Tuan, 1977: 129) which is differentiated from the rational, goal-directed mode of existence. The subject’s feelings and the uses invoking the feelings are

262

MATERIALIZING DIGITAL FUTURES

embodied through the bodily and sensory, but not cognitive, engagement with the home environment. The home’s atmosphere must be aligned to our cultural understanding, or the semantics, of particular situations and they must be working frictionless on demand to provide the sense of normalcy and expectedness.

Arrest the attention An important dimension of the environmental experience of the home is the ability to conduct many day-to-day activities subconsciously. One simply cannot live in constant vigilance in one’s own home. Seamon (1979) terms this precognitive and embodied aspect of everyday experience in the home ‘basic contact’. It is the basic contact, the embodied and automatic carrying out of mundane activities in the home, that gives the opportunity for the cognitive power to attend to other activities. In other words, many activities in the home are delegated to the body as a subject to intelligently deal with the environment without the cognition’s intervention. It is found that smart devices are used to arrest the users’ attention either to carry out non-routine tasks that demand heightened attention or to activate a set of routines that need to be carried out at a specific temporal rhythm. Participant 11 cooks at home frequently. She used to use the microwave timer and often forgot what the timer was for when cooking multiple things. ‘I feel like the Google home one [timer function] is a lot easier to use because I can just adjust it as I’m going, like, I don’t have to reset it or remember what this one means.’ For participant 5, the alarm in the afternoon was to remind her to go pick up her daughter from school. Regardless of what she was doing at home, when the alarm went off, another set of routines kicked in. She checked the traffic, the weather, grabbed keys and left the home. The peace of mind they gained after setting the timer maintained their sense of comfort in a stress-free home with normalcy. Another way in which the devices arrest the users’ attention is through the gadget themselves catching residents’ attention. As a domestic communication technology, smart home devices not only mediate messages but are visible objects in the home environment themselves, what Silverstone (1994) calls double articulation. Especially for internet-enabled smart devices, it is important to pay attention to the connection between the symbolic content and material artefact (Livingstone, 2007). Across the diverse families of our participants, the playful uses of these devices during the period immediately after the installation seems to be a pattern. Young children are especially aware of the environmental changes and are playful about these new additions to their homes. In the first two weeks, Participant 4’s children routinely asked the speakers silly questions. Participant 3’s daughter teased her older brother by asking ‘Hey Google,

SMART HOME

263

is [brother’s name] stupid?’ Participant 1, a Chinese immigrant, used the speakers to play the Radio Calisthenics to exercise with his two young children. However, such uses became less frequent within a few weeks of installation. As time goes on, the physical devices appear to get domesticated into the home environment and the marvellous and attention-getting nature fades into basic contact. The three ways in which the smart devices interact with the environmental experience of the home – arrange the action, align the atmosphere, and arrest the attention – are indeed intertwined. The reason for which we do not thematize the uses per se is that many uses simultaneously contribute to different aspects of our experiences of the home. For example, setting the baking timer on the smart speaker instead of the oven serves both to arrest attention and to remove the action of walking up to the oven to stop the timer when it goes off. Dimming the smart lightbulb via the smart speaker before going to sleep serves both to save the action of opening the app to control the light and to align the atmosphere to the night-time environment. The smart device uses often manifest the regularity of a routine, the readyto-hand of the devices and the embodied actions/feeling in the home, all of which contributes to our sense of everyday normalcy of the home. Moving on from the explications of the relationships between the smart devices uses and our experiences of the home, it is now necessary to discuss the implications of the smart home devices’ roles in constructing the sense of normalcy in the home. While many participants enjoy the at-homeness with these new technologies, the question arises as to the extent to which these technologies are channelling new forms of control into our everyday lives in the home. Are these technologies fulfilling the promise of the home to let us be our authentic selves in such an environment?

Discussion and conclusion: The ‘smart life’ and everyday normalcy The phenomenological approach taken in this study is largely descriptive. It tries to capture the everyday normalcy of the home as what directly appears to the users of the smart devices through their experiences in the home. Seamon (1979; 2018) in his theorization of the everyday environmental experience argues that at-homeness in a familiar environment like the home supports both openness and habituality. Openness manifests a mode of being which Heidegger (1962) calls authenticity where one ‘accepts responsibility for his existence and seek to be consistent and honest in his dealing with the world’ (Seamon, 1979: 118). Feeling at-home, one maintains the basic contact in the familiar environment with peace of mind and conserves the physical and psychic energies for heightened attention and discovery, should

264

MATERIALIZING DIGITAL FUTURES

one choose to pay attention and try to discover something. Meanwhile, at-homeness also fosters habituality. The comfortable monotony of the everyday normalcy promotes order and continuity and prevents chaos and anxiety, but also shuts off many possibilities. Generations of scholars have argued about both the oppressing and the subverting powers of the everyday life. On the one hand, from the Frankfurt school to Lefebvre and Bourdieu, scholars seem to view the everyday life as politics mistaking play for power. External structures, in their views, exerts insurmountable power over helpless individuals. On the other hand, de Certeau, and Fiske by way of example, see a politics in which play is power. They see that everyday life bears the possibility of resistance and transcendence. Silverstone (1994: 161), in his analysis of the role of television in everyday life asks us to give up on the dichotomous views and examine both television and the television audience as ‘multiply structured and structure, making, remaking and unmaking meanings’. As Pink (2004; 2012) and Spigel (2001) have argued, everyday life in the home, with its dual powers to foster both openness and habituality, is the backdrop from which authenticity is possible, but not guaranteed. Gender, class and other power structures can either be perpetuated or challenged through the most mundane life in the home. Indeed, both the authenticity and inauthenticity of one’s being in everyday life can be witness in the day-to-day uses of the smart home devices. On the one hand, most of the uses of the smart devices to arrange the actions, align the atmosphere and arrest the attention are directly derived from existing patterns of everyday life without reflection upon the normative implication of these practices. In a sense, the smart devices are simply replicating or even reinforcing the existing structure of everyday life such as gender roles and parental hierarchy. In the present study, participants had trouble understanding the range of possible uses of these devices until the researcher made suggestions for certain possibilities during the interviews. Here the researcher offered insights with their fresh eyes on the participants’ seemingly taken-for-granted everyday normalcy. For example, Participant 6 did not realize that, as shown in her use log, she only listened to four different music artists. When asked about her music preferences, she said she had always wanted to listen to new ones but never actually explored far beyond the familiar genres known to them. When it was suggested that she could use the smart speaker to explore more possibilities, she started to venture into a diverse array of artists as reflected in a follow-up use-log collection. To what extent, however, the new music genres contributed to, or compromised, her embodied at-homeness remains to be seen over time. On the other hand, there are also moments of authenticity manifested in the uses of these devices. For some participants, the introduction of these devices affords an opportunity to reflect upon the taken-for-granted aspects of everyday life. Participant 2 explained her thought process of installing

SMART HOME

265

the smart lightbulb on her nightstand lamp. She deliberately did this so she could spend less time in the living room watching television at night and could have a comfortable environment to read more in bed before going to sleep. Moreover, the family game and music functions of the smart speaker gave multiple participants ‘family fun time’ and bonding opportunities. For other participants, the ready-to-hand and embodiment affordances of the smart devices facilitated opportunities to delegate the repetitive routines to basic contact so that they might pay more attention to the things that emotionally open and intellectually ‘grow’ themselves. Just like any other part of everyday life, the uses of smart devices sit between structure and agency in ordinary domestic life. The everyday normalcy in the home as experienced through the smart device uses is indeed paradoxical. It is the source of both security and anxiety. It encompasses both passivity and activity. It crosses between the private and the public. It fosters both consumption and production. It manifests both independence and dependence. The increasingly mediatized home is so marvellous, but felt completely ordinary. Although the current study is largely descriptive, future scholarship may want to focus on the sociocultural implications of these domestic technologies. As Spigel (2010: 84) argues, the smart home presents us with ‘futures worth imagining. After all, if we don’t imagine the future then Disney [or Google] most certainly will’.

References Bausinger, H. (1984), ‘Media, Technology and Daily Life’, Media Culture & Society, 6 (4): 343–51. Crabtree, A. and T. Rodden (2004), ‘Domestic Routines and Design for the Home’, Computer Supported Cooperative Work, 13 (2): 191–220. Crabtree, A. and P. Tolmie (2016), ‘A Day in the Life of Things in the Home’, Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, San Francisco, CA. Frohlich, D. and R. Kraut (2003), ‘The Social Context of Home Computing’, in Inside the Smart Home, 127–62, London: Springer. Glaser, B. G. and A. L. Strauss (2017), Discovery of Grounded Theory: Strategies for Qualitative Research, New York: Routledge. Habib, L. and T. Cornford (2002), ‘Computers in the Home: Domestication and Gender’, Information Technology & People, 15 (2): 159–74, doi: 10.1108/09593840210430589. Harper, R. (ed.) (2003), Inside the Smart Home, London: Springer. Harper, R. (ed.) (2011), The Connected Home: The Future of Domestic Life, London: Springer. Heidegger, M. (1962), Being and Time. New York: Harper & Row. Hine, C. (2020), ‘Strategies for Reflexive Ethnography in the Smart Home: Autoethnography of Silence and Emotion’, Sociology, 54 (1): 22–36.

266

MATERIALIZING DIGITAL FUTURES

Jakobi, T., C. Ogonowski, N. Castelli, G. Stevens and V. Wulf (2017), ‘The Catch(es) with Smart Home: Experiences of a Living Lab Field Study’, Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, Denver, CO. Kaika, M. (2004), ‘Interrogating the Geographies of the Familiar: Domesticating Nature and Constructing the Autonomy of the Modern Home’, International Journal of Urban and Regional Research, 28 (2): 265–86. Livingstone, S (2007), ‘On the Material and the Symbolic: Silverstone’s Double Articulation of Research Traditions in New Media Studies’, New Media & Society, 9 (1): 16–24. Lupton, D. (2020), ‘The Internet of Things: Social Dimensions’, Sociology Compass, 14 (4): e12770. Merleau-Ponty, M. (2002), Phenomenology of Perception, New York: Routledge. Moores, S. (1988), ‘The Box on the Dresser’: Memories of Early Radio and Everyday Life’, Media, Culture & Society, 10 (1): 23–40. Moores, S. (2006), ‘Media Uses and Everyday Environmental Experiences: A Positive Critique of Phenomenological Geography’, Particip@tions, 3 (2): 233–56. Moores, S. (2017), Digital Orientations: Non-Media-Centric Media Studies and Non-Representational Theories of Practice, New York: Peter Lang. Morley, D. (1988), Family Television: Cultural Power and Domestic Leisure, London: Routledge. Morley, D. (2005), ‘What’s “Home” Got to Do with It? Contradictory Dynamics in the Domestication of Technology and the Dislocation of Domesticity’, European Journal of Cultural Studies, 6 (4): 435–58. Pink, S. (2004), Home Truths: Gender, Domestic Objects and Everyday Life, New York: Berg. Pink, S. (2012), Situating Everyday Life: Practices and Places, Los Angeles, CA: Sage. Putnam, T. (2006), ‘Postmodern Home Life’, in I. Cieraad (ed.), At Home: An Anthropology of Domestic Space, 144–52, Syracuse, NY: Syracuse University Press. Reiss, D. (1981), The Family’s Construction of Reality, Cambridge, MA: Harvard University Press. Relph, E. (1976), Place and Placelessness, London: Pion. Scannell, P. (1996), Radio, Television, and Modern Life: A Phenomenological Approach, Cambridge, MA: Blackwell. Seamon, D. (1979), A Geography of the Lifeworld: Movement, Rest and Encounter, New York: St Martin’s Press. Seamon, D. (2018), Life Takes Place: Phenomenology, Lifeworlds, and Place Making, New York: Routledge. Silverstone, R. (1994), Television and Everyday Life, New York: Routledge. Silverstone, R. (1999), ‘Domesticating ICTs’, in W. H. Dutton (ed.), Society on the Line: Information Politics in the Digital Age, 251–2, Oxford, UK: Oxford University Press. Spigel, L. (1992), Make Room for TV: Television and the Family Ideal in Postwar America, Chicago, IL: University of Chicago Press.

SMART HOME

267

Spigel, L. (2001), ‘Media Homes: Then and Now’, International Journal of Cultural Studies, 4 (4): 385–411. Spigel, L. (2010), ‘Designing the Smart House, Posthuman Domesticity and Conspicuous Production’, in C. Berry, S. Kim and L. Spigel (eds), Electronic Elsewhere: Media, Technology and the Experience Social Space, 55–94, Minneapolis: University of Minnesota Press. Statista (n.d.), ‘Smart Home’. Available online: https://www.statista.com/ outlook/279/109/smart-home/united-states (accessed 31 March 2021). Tolmie, P., J. Pycock, T. Diggins, A. MacLean and A. Karsenty (2003), ‘Towards the Unremarkable Computer: Making Technology at Home in Domestic Routine’, in R. Harper (ed.), Inside the Smart Home, 183–206, London: Springer. Tuan, Yi-Fu (1977), Space and Place: The Perspective of Experience, Minneapolis: University of Minnesota Press.

268

14 Affect and the digitalization of war John MacWillie

Introduction: Affective warfare Affective warfare is just one tool among an arsenal of tactical weapons used to conduct political warfare where political communications may be ‘combined with violence, economic pressure, subversion, and diplomacy, but [where] its chief aspect is the use of words, images, and ideas, commonly known … as propaganda and psychological warfare’ (Smith, 1989: 3). This spectrum of weaponry includes: (a) public diplomacy and propaganda; (b) psychological warfare, in which propaganda is accompanied by the threat of violence, where, in the words of the CIA, ‘the most critical point of the human being is the mind [and] once the mind has been reached, the “political animal” has been vanquished, without necessarily having received any shots’ (CIA, 1984: n.p.); (c) information warfare designed to distort, disrupt or undermine the trust governments and their citizens have in the integrity, reliability, resilience and legitimacy of their own social and political institutions and processes; and finally (d) affective warfare, which targets the emotional integrity of a population in furtherance of undermining the confidence in, and legitimacy of, the consensus underlying governance. Each of these stratagems are often operationalized with different technics and objectives to be projected by state as well as non-state actors. However, there is an important difference between most forms of political warfare and what I am referring to as affective warfare. Conventionally,

270

MATERIALIZING DIGITAL FUTURES

political warfare targets the cognitive domain – focusing on propositions, beliefs, values, opinions, perceptions and ideologies, namely those forms of conceptual knowledge which are consciously accessible. Moreover, what characterizes the production of most of these types of political warfare tactics is their reliance on broadcast forms of communication, including print, radio, television and cinema, to reach large audiences. This use of linguistic mediums – text, narratives, discourses – are comparatively slow to adoption, readily denied and easily attributable to their origin. Understanding how this domain works is an epistemological and linguistic question and interpreting it is a problem for semantics, logical analysis and verification. By contrast, affective warfare targets the sub- or preconscious experiential domain, primarily using aural and visual communication to target subconscious reactions or sympathies and does so in a way that masks forensic attribution. This entails a form of preconceptual experience that is, at least in part, neurophysiological and, in part, psychological. To be clear, affective warfare is not new and many forms of political warfare have sought to manipulate affect. The act of war and the forcing of an opponent to submit to an other’s will is inherently affective and the exploitation of affect can be a useful tactical weapon in achieving that submission. In World War 1, for example, British propagandists assembled atrocity propaganda integrating images and text to invoke fear and hatred of Germans and promote solidarity for war aims among the British citizenry. In the Korean War, the Chinese Red Army ‘swarmed over the hills, blowing bugles and horns, shaking rattles and other noisemakers, and shooting flares in the sky. They came on foot, firing rifles and burp guns, hurling grenades, and shouting and chanting shrilly. The total surprise of this awesome ground attack shocked and paralyzed most Americans and panicked not a few’ (Becton, 2017: 44). However, the emergence of computers, networks, social and interactive media, and mobile computing have exposed new possibilities for accessing and manipulating affect in the twenty-first century. These new possibilities leverage the following characteristics of affect: l

l

l

affect is ab initio an inherent property of individuals; the effect of affect can be transferred to others; and affect can be mobilized as a contagion.

At the same time: l

l

l

affect is material; affect is externally susceptible to stimulation and provocation; affect can be micro-targeted at individuals to manipulate individual sentiment; and

AFFECT AND THE DIGITALIZATION OF WAR

l

271

affect has the capacity to be mobilized as shared sentiment into action in furtherance of disrupting or destabilizing social and political bodies.

Here, affective messages which appeal to stimulating subconscious prejudices and predispositions can be targeted to select individuals. Cognitive researchers have demonstrated the effectiveness of this strategy for generating consumer desires and marketing products (Ariely, 2009; Bayle-Tourtoulou and Badoc, 2020). When embedded into the flows and connections of social networks, these prejudices and the likelihood of their accessibility can be algorithmically identified and effectively narrowcast to maximize reaction (Benkler et. al., 2018; Singer, 2018). As discussed in detail below, this content can be further propagated to ‘friends’ and family who hold similar views and whose sympathy is reflected in ‘likes’ and ‘shares’. This recirculation and repetition further amplifies the significance and confirms the validity (truth-value) of the affective content (Benkler et al., 2018), enabling what Ignas Kolpakas refers to as ‘mimetic warfare’ (Kalpokas, 2016). The ultimate strategic objective is achieved when individuals begin moving from the digital into the external physical domain – be it purchasing products, socializing or political advocacy. But, when the ultimate goal of a digital operation is to instigate political discord, distract political processes and undermine confidence in political institutions, affect is weaponized. Affective warfare can be extremely valuable when there is an asymmetric balance of forces, when political domination must be achieved without incurring costs associated with a kinetic war or when weakening of an opponent through subversion of social cohesion is the objective (Erbschloe, [2017] 2020). To launch an affective warfare campaign, what is required is a theoretical model which can be operationalized to generate provocative messages, a technical method for distributing this content, a form of media that enables and encourages the amplification of the emotional effect of this content through redistribution and rapid recirculation, and an affectively overly sensitized population to exploit. But before we examine how such affective warfare works, we need to understand what affect is and the way it becomes socialized and made available to the provocateurs that want to employ it.

Theory of affect What then is affect? Though it has gained currency in contemporary theory (e.g. Clough and Halley, 2007; Gregg and Seigworth, 2010), the concept of affect has its roots in the ancient Greeks. In the Anima, Aristotle writes of something that is a ‘sensation that [depends] … on a process of movement or affection from without, for it is held to be some sort of change of quality’

272

MATERIALIZING DIGITAL FUTURES

(1941: ii.5.416b33–34). He identifies affection as a movement of appetite of the same genus as desire and passion (1941: ii.5.414b3). Similarly, Aristotle observes in the Rhetoric that affect can be exposed in rhetorical acts of persuasion to stimulate pleasure or pain. In this way, affects can also be employed as a means to an end. In the Nichomachean Ethics (Book VII), Aristotle observes that there are people who know that their actions are not right but do them anyway driven by feelings rather than reason – a condition often translated as ‘incontinence’. Nearly two thousand years later, Hobbes, Descartes and Spinoza each confront just how affect plays a role in human psychology. For example, Benedict Spinoza asserts that affect is an important philosophic question. In Part III of the Ethics, he writes: ‘By emotion (affectus) I understand the affections of the body by which the body’s power of activity is increased or diminished, assisted or checked, together with the idea of those affections’ (Spinoza, 1992: 103). He declares that affections are (accidentally) free to cause pleasure, pain, or desire (Prop. XV), that our subsequent encounter to an object (even by resemblance) will be first influenced by our first affective attachment (Prop’s XVI, XVII, XVIII). By the time contemporary theorists begin to consider the importance of affect, there is already, then, an inherited body of speculation and research on what affects might be and their place in human physiological, psychological, social and cultural formation. The specific motivation for the present ‘rediscovery’ of affect and affectivity among cultural theorists introduced here can be traced to the mid-1990s, when many researchers with differing approaches (e.g. the later Foucault, Deleuze, Judith Butler, among others) sought to surmount the inadequacies of post-structuralist theory and its reliance on language as the principal source for the construction of personal and social reality. To formulate a theory of affective warfare, my approach follows an architecture of affect that connects autonomy with sensation from the work of Brian Massumi (2002); sensation with the body based on the writing of Eve Sedgwick (2003); and the materiality of the body with the political as exemplified in the work of Lauren Berlant (2011). My argument is that this triadic approach helps frame our understanding of how affect and digital media parallel and amplify the effects of each other, particularly amid a contagion of affect.

Affect as intensity Affect is not a singular thing. Rather it is a state that modulates between neurophysiological sensation and psychological experience. It is, as Brian Massumi (2002) defines it, an intensity. He gives an example that begins with a ‘story about a snowman’, and recounts a silent film shown to

AFFECT AND THE DIGITALIZATION OF WAR

273

children, many of whom are disturbed by the image effects. Researchers are bewildered by the children’s response and proceed to compare children’s reactions to different versions of the same film. The first is a highly textualized ‘factual’ version of the film, the second incorporates minimal text necessary to highlight emotionally significant moments in the narrative and the third version is the original unedited version. When shown the three versions, children rate the first as the least pleasant and least remembered. But, surprisingly, it is the third (originally silent) version that stirs the greatest emotive reaction, as it is rated the saddest, but simultaneously, the most pleasant. By way of assessment, Massumi observed: [The] only positive conclusion emphasized the primacy of the affective in image reception. The affective is marked by a gap between content and effect … the strength or duration of an image’s effect is not logically connected to the content in any straightforward way … [T]‌he content of the image is its indexing to conventional meanings in an intersubjective context, its sociolinguistic qualification. This indexing fixes the determinate qualities of the image; the strength or duration of the image’s effect could be called its intensity. What comes out here is that there is no correspondence or conformity between qualities and intensity. (Massumi, 2002: 23) There are three aspects of affect that Massumi draws out from his analysis. First is the distinction between affect and emotion. Affect is intensity, while emotion is ‘the sociolinguistic fixing of an experience which is … personal. Emotion is qualified intensity’, where qualification is imposed by semiotic and narrativized progressions (Massumi, 2002: 28). Second, affect is the most fundamental point of contact between the body and the world in which it is surrounded. In our media and information intensive environment, Massumi writes, ‘affect is central’ (Massumi, 2002: 27). Third, affect, is not only an intensity but also an excess, more than what we can process in the time available to us. As a result, what we think of as will or intentionality is really a second order reduction of that excess, of what has already been experienced unconsciously. Volition is already a subset of affect processed by an autonomic reaction ‘outside of consciousness [and] prior to action and expression’ (Massumi, 2002: 29). It is an emergent connection between affect as something open and autonomous (virtual) and the way in which it is captured and qualified as an emotion (actual).

The embodied affect Affect is also material and embodied. Eve Kossofsky Sedgwick (2003) along with her colleague Adam Frank point to the earlier work of psychologist

274

MATERIALIZING DIGITAL FUTURES

Silvan Tomkins (1962–92) who first proposed that the primitive origins of shame in an infant could be identified long before the child could have understood any form of prohibitive expression. As Sedgwick (2003) observes, the affective development of shame is now considered by most development psychologists as the foundation for the psychic formation of the self. For Tomkins, affect can attach to any object and, unlike the assumption underlying behaviorist models, can produce multiple outputs. Affect is not driven by a goal. Citing Tomkins, “It is enjoyable to enjoy. It is exciting to be excited. It is terrorizing to be terrorized and angering to be angered. Affect is self-validating with or without any further referent” (Sedgwick, 2003: 99–100). What Tomkins recognized was that affect constituted a relationship among parts (brain, nervous system, emotions, and so on) and that the whole of the parts and the relationships constituted a ‘system’. For Tomkins, the work of cyberneticists like Norbert Weiner (1948) and Gregory Bateson (1977) contributed to his understanding that affect was both a form of vitality and a means of self-organization. As Tomkins recounts: ‘One could not engage in such [an understanding] without the concept of multiple assemblies of varying degrees of independence, dependence, interdependence, and control and transformation of one by another’ (Tomkins cited by Sedgwick (2003): 100). This emphasis upon a  multiplicity of structures suggests that Tompkins’s model might also apply to the present layering of digital and analog communication channels emphasizing not the components but the relationships among entities of body and world, dermis, nerve and response, body and body, emotion, and cognition. These channels are material, all formed around a single principle, ‘the density of neural firing’, which, occurring over a short span of time, may increase, decrease or remain constant and correspond to the changes in the density of neural firing over time and specific types of affect: fear, interest, anger, joy and so forth. Affect theory emphasizes the potential complexity of relations and outcomes which, in Tompkins interpretation, ‘offers a wealth of sites of productive opacity’ (Sedgwick, 2003: 106–7). The relationships among these sites offer enormous opportunities for the distribution, recirculation and amplification of affects and constitutes the circulation of affect in and through the body. Affect is systemic.

The politics of affect A body exists in a material world and it is Lauren Berlant’s objective to trace ‘the force that moves you out of yourself and into the world in order to bring closer the satisfying something that you cannot generate on your own’ (Berlant, 2011: 1–2). This force she asserts is optimism, but it is an optimism with a double-bind which then establishes relations with objects

AFFECT AND THE DIGITALIZATION OF WAR

275

that are, in fact, an obstacle to one’s own flourishing (Berlant, 2011: 1). This knot is what Berlant refers to as ‘cruel optimism’ (Berlant, 2011: 1). It arises from encounters – casual, inadvertent and promiscuous, as well as those that are intentional, that might invoke anything quite different from enthusiasm: ‘dread, anxiety, hunger, curiosity, the whole gamut from the sly neutrality of browsing the aisles to excitement at the prospect of “the change that is going to come” ’ (Berlant, 2011: 2). In other words, a reticent anticipation is seductively drawn into an innocent engagement that, in turn, becomes a subliminal expectation. For Berlant, ‘one makes affective bargains about the costliness of one’s attachments, usually unconscious ones, most of which keep one in proximity to the scene of desire/attrition’, which means that ‘to understand ‘cruel optimism’, therefore, one must embark on an analysis of indirection, which provides a way to think about the strange temporalities of projection into an enabling object that is also disabling’ (Berlant, 2011: 25). If interactive media has been proven to accomplish anything, it is the fetishization of anxiety, or what Landau, Rothschild and Sullivan (2011: 132) refer to as ‘existential uncertainty’, in which ‘people may compensate for high levels of existential uncertainty by means of fetishism – investing in narrow dimensions that afford certain (albeit circumscribed and rigid) bases for affirming their life’s significance’ (Landau et al., 2011: 132). This fetishism is manifested by compulsion, addiction and the artifice of virtual identity. The digital ‘app’ thus becomes the material object through which affect flows revealing itself as the repetitive obsession of returning again and again to verify that ‘my friends are still my friends’ and that what I feel is still important. Multiple studies point to a relationship between Facebook addiction and low self-esteem. For example, one literature review concluded ‘in regard to Facebook addiction, the findings … paint the following picture: individuals with low psychosocial well-being, such as loneliness, anxiety or depression, are motivated to use Facebook to find social support or to pass time’ (Ryan et al., 2014: 145; see also Błachnio et al., 2016). Neither the friend nor the feeling is as important, in and of themselves, as is the acknowledgement and validation by others that is critical to one’s own self-esteem and worth. As another group of researchers observed: ‘Individuals with lower levels of self-esteem and higher levels of self-monitoring are more likely to think that Likes are important and to feel bad if they do not receive ‘enough’ Likes’ (Scissors et al., 2016: 1501). Though these researchers also found ‘they desire Likes more from certain types of relational ties than others’ (Scissors et al., 2016: 1507–8). Anxiety subsides as affect flows across the neurophysiological to the digital network and back. Interrupt this flow and the anxiety, fear and dread return to the fore. The cruelty, to which Berlant’s critique is directed, is derived from an ‘impasse of desire’ where consequent hoarding takes on the character of a

276

MATERIALIZING DIGITAL FUTURES

solution (Berlant, 2011: 42). The parallel cathexis between an overcharged affectivity and a deficit fetish occurs because ‘hoarding controls the promise of value against expenditure, as it performs the enjoyment of an infinite present of holding pure potential’ (Berlant, 2011: 42). This is a political act that seeks to negotiate a balance of power where no such negotiation can occur. Berlant concludes: This means that the object of cruel optimism here appears as the thing within any object to which one passes one’s fantasy of sovereignty for safekeeping … (where) the subject or community turns its treasured attachments into safety-deposit objects that make it possible to bear sovereignty through its distribution, the energy of feeling relational, general, reciprocal, and accumulative. (Berlant, 2011: 43) In short, affects are heterogeneous assemblages of power and constitute a psychic transference that is primarily political. As Berlant puts it, ‘the process of dynamic sensual data-gathering through which affect takes shape in forms whose job it is to make reliable sense of life’ (Berlant, 2011: 52).

The fields and contagion of affect If affects are free and autonomous, then they must be able to move about. Brian Massumi (2002: 43) claims that that is precisely what occurs as: ‘[affective atoms] are autonomous not through closure but through a singular openness. As unbounded regions in an equally unbounded affective field, they are in contact with the whole universe of affective potential, as by action at a distance’ (Massumi, 2002: 43). The affective fabric is constituted from the actual and virtual relations of affects alone – it can expand or contract, become more intense or dimmer, engage or withdraw. Its capacity to modulate these differences lies in its potential. Massumi theorizes: Concepts of the virtual in itself are important only to the extent to which they contribute to a pragmatic understanding of emergence, to the extent to which they enable triggerings of change (induce the new). It is the edge of virtual, where it leaks into actual, that counts. For that seeping edge is where potential, actually, is found. (Massumi, 2002: 43) This potentiality constitutes the sociality of affect that Sedgwick, drawing on Tomkins’s research, focuses on in terms of the role of imitation and mirroring in constructing socially affective relations. Tomkins’s ideas as framed by Sedgwick (2003) draw the conclusion that: ‘If you liked

AFFECT AND THE DIGITALIZATION OF WAR

277

to be looked at and I like to look at you, we may achieve an enjoyable interpersonal relationship. If you like to talk and I like to listen, this can be mutually rewarding’ (Sedgwick, citing Tomkins, 2003: 96). The essence of successful imitation is repetition and synchrony, where incitement is met by receptivity, suggesting that imitation and desire are simply different aspects of the other. For the nineteenth-century sociologist Gabriel Tarde, the social is not meaningful at either the macro (sociological) or micro (psychological) scales of the individual, but rather, it exists within the relationships themselves (Tarde, 1969: 177–94). These relationships are initiated, sustained and transformed as the result of repetitious imitations across the affective field. According to Tarde, imitation always proceeds from the inner to the outer – one seeks to imitate what is experienced: [The] imitation of ideas precedes imitation of their expression [and ...] imitation of ends precedes imitation of means … [W]‌e are led to copy from others everything which seems to us a new means for attaining old ends, or satisfying our old wants, or a new expression for our old ideas; and we do this at the same time that we begin to adopt innovations which awaken new ideas and new ends in us. (Tarde, 1969: 186) In this way, imitation is the production of desire, vis-à-vis relations and connections, or what Berlant refers to as ‘the force that moves you out of yourself and into the world’ (Berlant, 2011: 1). This is a force of production engendering reality. Affects constitute not only the mirroring of the world but enable adaptation and invention through consonance and dissonance, synchronicity and syncopation. Preconscious sensation, movement and desire propagate their effects across entire fields of affective relations – while their effects are simultaneously absorbed and reconstituted to produce further affective responses in imitative, oppositional, and innovative forms. By social contact and proximity with sufficiently large number of others, affect become a nidus of contagion. But if affects are embodied, then not all connections in practice are equal (or even actual), as they are often hierarchically stratified (e.g. among social classes), more differentially efficient (e.g. frequently traversed), or preferentially connected (e.g. familiarity) in ways that make some connections more favourably equal than others. The lumpiness of this connective tissue directly affects the influence of individual points on their attendant connections. Similarly, the greater the number of connections with a single point, the greater the potential intensity and influence that point has with all other points. Nevertheless, despite the constraints of inequivalence, it is the communicability of affect that affective warfare seeks to exploit.

278

MATERIALIZING DIGITAL FUTURES

Mapping affect to war Existentially, affect is universal, but its presence is always particular. How then can it be assembled and mobilized on a digital platform for the purposes of warfare? To illustrate how this works, I extrapolate from a real-world example and map this data to a theoretical model of how affect can be digitally weaponized. The following empirical example is based on affective warfare campaigns conducted by Russia and my model draws upon the application of epidemiology to social networks. What has transformed the potentiality of affect into a political weapon is the emergence of the internet and social networking applications as platforms. Internet-based social media is nearly perfect for affective warfare. It is ubiquitous (in Western democracies), has a near zero incremental marginal cost for each new attack, provides low latency between production and distribution, where its content is independent of device, geography, or time of day, and where one can easily obfuscate the origin of attacks. The decentralized design characteristics and peer-to-peer affordances for sharing content via social media and the like make it easy to personally target content to individuals with specific characteristics; each of whom can then readily redistribute this content to likeminded cohorts creating an ever-expanding audience. In short, because internet-based social media is far more frictionless than other media platforms, is integrated into everyday communicative experiences and accelerates message amplification, it is ideally suited for experimenting with new forms of communicative belligerence. Nearly every nation with the resources and technical capabilities to execute a political warfare campaign employing propaganda and information operations is already doing so. But so far, only a few nations including Russia and Israel routinely initiate campaigns of affective warfare, while China, Iran and North Korea are beginning to exhibit some of their own capabilities. Nor are state actors the only practitioners. Al-Qaeda and Da’esh have launched campaigns primarily for recruitment and fund-raising, while private, wealthy activists, like US citizen Robert Mercer, have invested in key affective warfare capabilities, including Cambridge Analytica in the United Kingdom.1 For these practitioners planning an affective warfare campaign requires asking four primary questions: Why conduct a campaign (goals, objectives, benefits, risks)? Who is the target (characteristics, size, distribution, susceptibility)? What comprises the campaign that achieves the objectives (affective message, lure, expected effect)? How is a campaign to be conducted given the deliverables, the population and the objectives (delivery platform, rate, effectiveness)? The first two questions are problems of strategy (what is the problem and what is the solution?) and the second two are tactical issues (process and platforms). To examine how these questions may be addressed in actual acts of belligerence, I turn to Russia

AFFECT AND THE DIGITALIZATION OF WAR

279

and its foreign policy in the post–Cold War and map this evidence to an epidemiological framework.

Aetiology By way of background, the dissolution of the Soviet Union and the implosion of its economy, military capabilities and global leadership role constituted a traumatic experience for most Russians. The details are beyond the scope of this chapter, but the reaction to the political disintegration is crucial to understanding contemporary Russian polices and their objectives in the use of affective warfare. In the mid-1990s, Russian Foreign Minister Yevgeny Primakov (appointed by President Boris Yeltsin), established the crucial strategic doctrine to resist US unilateralism by assuring that Russia was in a position to serve ‘as an indispensable actor with a vote and a veto, whose consent is necessary to settle any key issue facing the international community’ (Rumer, 2019: n.p.). To accomplish this without necessarily engaging in a kinetic war, Russian Chief of Staff General Valerii Gerasimov drew on a ‘new form of warfare’, he claimed was inspired by his observations of the Color Revolution and the Arab Spring (Gerasimov, 2013: 1). Further, that ‘the role of non-military methods in achieving political and strategic goals has increased, which in a number of cases have significantly surpassed the power of weapons in their effectiveness’ (Gerasimov, 2013: 1). He also emphasized ‘the use of the protest potential of the population’ as a way of leveraging force in hybrid warfare (Gerasimov, 2013: 1). Russia began experimenting with information warfare on scale with initiatives against former Soviet territories in Estonia (2007), Georgia (2008) and the seizure of Crimea (2014) and regions of Ukraine (2014). But it was with operations in 2016 in Britain (related to the Brexit referendum), France (the gilets jaunes protests), Germany (AfD protests) and the United States (presidential elections, immigration policy, Covid mitigation) that Russia began more targeted affective warfare campaigns, indicating that some of these programmes really do work. If we turn now to consider the goal of such actions our knowledge of specific Russian motives for interference in Western democracies is not definitive, but it is not difficult to surmise. Russian President Vladimir Putin has made clear his disdain for Western democracies when he asserted that ‘liberal ideology has “outlived its purpose”, “liberalism is obsolete”, and points out ‘[Liberals] cannot simply dictate anything to anyone just like they have been attempting to do over the recent decades’ (Barber and Foy, 2019: n.p.). The ‘liberal ideologies’ to which President Putin refers are forms of governance, wherein its legitimacy and authority rely on a democratic consensus among the governed, and whereby those that govern do so on the

280

MATERIALIZING DIGITAL FUTURES

citizens’ behalf. Consensus requires some degree of equanimity and social harmony among shared public interests. Yet, as Putin senses, this legitimacy and concomitant consensus is breaking down as many Western liberal democracies undergo increasing disparities in the distribution of wealth and a greater concentration of power by elite segments of society. Looking now at the objectives for such motivations, sensing an opportunity with this emerging fault line, Russian dezinformatsiya focused on the heightening resentment between and among ethnic, economic and social groups. Each was acting to create seemingly irreversible fractures, exacerbating these divisions and increasing political polarization, undermining consensus, demoralizing a will to act and, in general, promoting social and political disruption and distraction. As a US investigation by the Office of the US Special Counsel (Robert Mueller) undertaken in 2019 puts it, the Russian government organized ‘a social media campaign designed to provoke and amplify political and social discord’ (Special Counsel, 2019: 4).

Susceptibility If we turn now to consider the target of such actions just outlined, of note is that survey polls have suggested the popular mood in the United States is ripe for exploiting shifts in social status and satisfaction.2 In a culture that has been suspended for twenty years in a state of exception for a war on terror (Agamben, 2005; Dillon and Reid, 2009), it is not surprising that one poll revealed that 70 per cent of adult US subjects were ‘very or somewhat’ angry ‘because our political system seems to only be working for the insiders with the money and power’ (NBC, 2019). Another study, comparing ideological views over a twenty-year period, showed a marked degree of sharply increasing political polarization among those considering themselves liberal or conservative, while, at the same time, there was a sharp reduction in those reporting a mix of views (Pew, 2016). The great middle class, which has long served as ballast for American consensus is disappearing, as an older middle class moves upward, while a younger middle class slips into a lower economic status. There is increasing alienation between the rust belt and coasts, rural and urban, young and old, and religious and secular, collectively providing a rich environment for fostering further division, suspicion and resentment. The lumpiness in fields of affect described earlier are becoming high mountains separated by deep valleys of sentiment.

Pathogens To exploit the vulnerabilities of this increasingly polarized ecosystem, the Russians formulated a strategy, created provocative affective content, utilized an efficient delivery platform in internet-based social media and

AFFECT AND THE DIGITALIZATION OF WAR

281

identified initial targets. The content, or what might be thought of as digital pathogens, and the effects they were intended to invoke, turned out to be more problematic than anticipated. As the review of affect theory here suggests, affect is a free agent within and betwixt a field of affects. The best proximation of effective content is that which is resonant with a field of affect already sufficiently predisposed towards one of the negative emotions noted here (anxiety, fear, resentment, anger, and so on). Given the conditional probability of whether any specific content will cause a specific effect, only large-scale iteration and repetition will increase the possibility of achieving the desired impact. Just as viruses have different species, digital pathogens have different morphologies. The most common found in affective warfare today are of a type considered ‘punctuated discourse’, comprising short, pithy declarative or imperative statements, whereby even the text is abbreviated into an idiosyncratic slang, injecting terse but highly emotive provocations underscoring a primary visual or aural message.

Exposure Further, in terms of target, unlike broadcast platforms, such as television or radio, where content is produced and controlled by large studios and network oligopolies, access to create content on social media is open and given that 80 per cent of Americans use social media, it is the largest unrestricted source of information content in the United States (Gramlich, 2019). Of that 80 per cent, nearly 90 per cent of Americans have a Facebook account, and of these, ‘three-quarters (74%) visit the site at least once a day’ (Gramlich, 2019: n.p.). For the Russians, Facebook (as well as other social media applications like Instagram, Twitter, YouTube and Pinterest) was a primary platform for exploitation. What is clear is that ‘venues’ are crucial. Social media sites primarily make money from selling digital real estate to advertisers who seek prime locations that offer a high volume of highly engaged users. In the case of Facebook, engagement is measured by the volume of comments posted, pictures and video uploaded, subscriptions to news feeds, affiliations with interest groups and events, and active messaging with friends. This is not an incidental feature. On average, every user on Facebook has hundreds of ‘friends’ whose posts about their lives and opinions are automatically shared among each other. Every minute, Facebook processes four million ‘likes’ indicating an affirmative response by one user of another user’s post. (Domo, 2015). There is an active vitality in these affirmations in which embodied affect is extended through a multiplicity of digital relationships. Moreover, Facebook enables users to cross-link Facebook features to amplify the engagement experience. The multiplicity of entry points gives

282

MATERIALIZING DIGITAL FUTURES

affect-warriors multiple vectors to launch an attack and to simultaneously cross-influence these attacks by adding their own ‘likes’, affirmative or derogatory comments, and ‘shares’. Indeed, reports from industry sources and government investigations indicate that Russian operations exploited nearly all these vectors to reach their target populations (cf: DiResta et al., 2018: 3–101). Selection is also important. Targeting focused on identifying clusters of potentially disaffected populations (not all retaining the same disaffection) with a view towards intensifying these disaffections. In a study by DiResta and her colleagues (2018), these clusters were classified into themes associated with accounts including: ‘Black Lives Matter’, ‘Blue Lives Matter’, ‘Being Patriotic’, ‘Heart of Texas’ and other Southern topics such as ‘LGBT’ and ‘Second Amendment’, among others (DiResta et al., 2018: 3–101). The technics of targeting were based on the tools available in each social media application. In Facebook, selection was based on a combination of inclusive and exclusive factors, for example, ‘interest in Black culture’ and excluding those interested in Black Lives Matter, to presumably focus on older, more traditional African American netizens (DiResta et al., 2018: 8). The actual content of messaging was tailored to these themes. For example, in defense of the Second Amendment, an advertisement portrayed a mother holding a large pistol entitled ‘Why do I have a Gun? Because it is easier for my family to get me out of jail than out of a cemetery’. Another, under the theme of ‘Being Patriotic’, displayed an unflattering photograph of US Democratic Party politician Hillary Clinton, with the words ‘Benghazi victim’s mother wishes Hillary to burn in hell’. This was neither followed by counter nor supportive arguments; just provocative statements underscoring an emotive image.

From Infection to Contagion The primary Russian entity carrying out most of the attacks for the 2016 US election was the Internet Research Agency (IRA), a private enterprise believed to have been directly authorized and funded by the Kremlin (Senate, 2020: 5). Authorship for this material was completely hidden by false names and accounts employing the Russian strategy of ‘maskirovka’ – translated literally as disguise. The IRA appears to have begun operations at least by January 2012 on Twitter, and perhaps earlier (DiResta, et al., 2018: 15). Facebook-related operations began in January 2015. In 2017, Facebook executives acknowledged that the IRA published 80,000 pieces of content on the Facebook platform, which was initially seen by 29 million people. More significantly, this initial reach was multiplied by user engagements (likes, comments, shares, and so on) to reach 126 million Americans or nearly one Facebook engagement for every one of the 129 million voters in the 2016 election (Isaac and Wakabayashi, 2017: B1).3

AFFECT AND THE DIGITALIZATION OF WAR

283

An aftershock is evident in the United States where many incredulously question how this strategy might have worked? There are a few answers. First, research shows that affect moves across social media without explicit content. A research team at Facebook altered the news feeds of nearly 690,000 Facebook users by randomly giving some users ‘happy’ news while others consistently received ‘sad news’ (Kramer et. al., 2014). Those that received sad news often went on to make sad posts and vice versa (Kramer et al., 2014). The researchers found ‘that emotional states can be transferred to others via emotional contagion, leading people to experience the same emotions without their awareness … emotional contagion occurs without direct interaction between people … and in the complete absence of nonverbal cues’ (Kramer et al., 2014: n.p.).4 Second, the relationship among these negative affects is complex. It is fear that appears to establish a predisposition for suspicion and resentment, laying the groundwork for the amplification of negative affect and anger (Glassner, 2009). More significantly, several studies reveal that there is an inverse relationship between fear and anger and the level of risk individuals perceive. The greater the fear, the greater the perceived risk, and vice versa, suggesting that when acted upon anger mitigates fear (and anxiety) (Habib et al., 2015; Lerner and Keltner, 2001; Lerner et al., 2003). This raises important questions about the role of social media in intergroup conflict. Given the putative anonymity of most social media, where personal risk of disclosure is low, individuals may be more risk-prone in their behaviour. One would expect that given the right conflictual circumstances social media would engender angry communications. Indeed, research suggests that social media may be an even more potent multiplier of affect intensity, specifically demonstrating that anger replicates faster and more extensively over social media than any other affect (Fan et al., 2014). This research study also found that ‘there is a stronger sentiment correlation between a pair of users if they share more interactions. And users with larger number of friends possess more significant sentiment correlation with their neighborhoods’ (Fan et al., 2014: n.p.). Moreover, individuals who are already angry before going online are more likely to amplify this anger when these feelings are transferred to the digital realm and to engage others of a similar disposition creating an echo chamber of anger and resentment (Wollebaek et al., 2019). Third, even when individuals are cognitively aware of a contradiction, they often rely on or are more comfortable with what is repetitiously familiar (Fazio et Al. 2019). This is an example, at an affect level, of what cognitive economists refer to as ‘arbitrary coherence’, in which initial exposure (to prices, emotional interactions, and the like) anchor subsequent affective events (Ariely, 2009). In other words, the combination of homophily, selfreferential confirmation bias and social networks amplifies the affective intensity of anger and consequential polarization antagonistic to others.

284

MATERIALIZING DIGITAL FUTURES

The weaponization of affect, then, only requires identifying the heuristics of a target audience resonating with their fears, and which amplifies their resentment to exploit their anger. This is most effective when these heuristics are consistent with an existing frame in which new events are already explained and understood (Lucas, 2012).

Conclusion Affective warfare is in its infancy and some theorists of war might plausibly dispute whether it is a distinct form of weaponry. But there is enough evidence to suggest that given the rapid development of information and communications technology enabling new media, and the ways in which these technics are radically reshaping social relations, as well as our sense of self, we should be mindful of its future potentiality for new forms of warfare. What makes affective warfare distinctive is its objective of manipulating the subconscious feelings, predispositions and prejudices towards action. This is not a ‘Manchurian Candidate’ form of brainwashing in which volition is supplanted by the will of another, but rather an attack on one’s way of being-in-the-world. It is a substitution of the real with the illusion of another world, where objects of feeling replace real objects. It is a fetishization of emotion itself, where the goal is to collect as much of these substitute feelings together in one place so that they can affirm what should be felt without, at the same time, acknowledging what is actually felt – anger substituting for shame, resentment fetishizing hurt, rage masking loss. This is the sense in which I read Lauren Berlant when she writes: If consumption promises satisfaction in substitution and then denies it because all objects are rest stops amid the process of remaining unsatisfied that counts for being alive under capitalism, in the impasse of desire, then hoarding seems like a solution to something … [B]‌eing in circulation denotes being in life, while an inexhaustible hoard denotes being in fantasy, which is itself a hoarding station against a threatening real, and therefore seems like a better aspirational realism. (Berlant, 2011: 42) Affective warfare is made possible, in large part, because new media interactively discloses the relationship between the virtual and the actual, between what is potential and what is real. This revelation overflows our conscious state with subliminal affect, setting aside cool reason for hot affect. It is only in retrospect that we may acknowledge its effect. If one is sceptical about the significance of this strategy, I would suggest imagining what affective warfare looks like when it is integrated with other emergent technologies such as surveillance, biometrics, reality augmentation, big data

AFFECT AND THE DIGITALIZATION OF WAR

285

and artificial intelligence. The more tightly integrated the interface between human and machine, the more abbreviated and efficient the interaction and the more easily organic affect is displaced by artificial affect.

Notes 1 Robert Mercer, a US hedge fund manager has underwritten right-wing political organizations including Cambridge Analytica which aggregated data and developed algorithms to target likely sympathizers for the UK Brexit campaign and for Donald Trump’s run for president in 2016 (on Mercer and Cambridge Analytica see Kaiser, 2019; Wylie, 2019). 2 Just how vulnerable US society has become to divisive polemics became manifestly clear with the storming of the Capitol on 6 January 2021 by rightwing protestors seeking to block the certification of the election of Joe Biden to be the next president. And recent polls suggest that the divisions are hardening rather than receding (Agiesta, 2021; Wike et al., 2021). 3 Nor was this a one-time operation as US intelligence officials warned that Russian operatives were active in the 2018 and 2020 election cycles (Goldman et al., 2020; Rosenberg et al., 2018). 4 This chapter was going to press just as the ‘Facebook papers’ were leaked to the media revealing the algorithmic mechanics Facebook uses to stimulate and accelerate negative affect to promote stickiness and intensity.

References Agamben, G. (2005), State of Exception, Chicago, IL: University of Chicago, Agiesta, J. (2021), ‘CNN Poll: Most Americans Think Elections Results Could Lead to Political Violence in the Coming Years’, CNN, 12 March 2021. Available online: https://www.cnn.com/2021/03/12/politics/cnn-poll-political-divisions/ index.html (accessed 26 May 2021). Ariely, D. (2009), Predictably Irrational: The Hidden Forces That Shape Our Decisions, New York: HarperCollins. Aristotle (1941), The Basic Works of Aristotle, ed. Richard McKeon, New York: Random House. Barber, L. and H. Foy (2019), ‘Vladimir Putin: Liberalism Has “Outlived Its Purpose”’, Financial Times, 27 June. Available online: https://www. ft.com/content/670039ec-98f3-11e9-9573-ee5cbb98ed36 (accessed 11 September 2020). Bateson, G. (1977), Steps to an Ecology of Mind, New York: Ballantine. Bayle-Tourtoulou, A.-S. and M. Badoc (2020), The Neuro-Consumer: Adapting Marketing and Communication Strategies for the Subconscious, Instinctive and Irrational Consumer’s Brain, New York: Routledge.

286

MATERIALIZING DIGITAL FUTURES

Becton, J. (2017), Becton: Autobiography of a Soldier and Public Servant, Annapolis, MD: Naval Institute Press. Benkler, Y., R. Faris and H. Roberts (2018), Network Propaganda: Manipulation, Disinformation, and Radicalization in American Politics, New York: Oxford University Press. Berlant, L. (2011), Cruel Optimism, Durham, NC: Duke University Press. Błachnio, A., A. Przepiorka and I. Pantic (2016), ‘Association between Facebook Addiction, Self-Esteem and Life Satisfaction: A Cross-Sectional Study’, Computers in Human Behavior, 55 (B): 701–5. CIA (1984), ‘Psychological Operations in Guerilla Warfare’, 18 October. Available online: https://www.cia.gov/library/readingroom/document/ciardp86m00886r001300010029-9 (accessed 7 September 2019). Clausewitz, C. (1976), On War, Princeton, NJ: Princeton University Press. Clough, P. and J. Halley (eds) (2007), The Affective Turn, Durham, NC: Duke University Press. Dillon, M. and J. Reid (2009), The Liberal War of War: Killing to Make Life Live, New York: Routledge. DiResta, R., K. Shaffer, B. Ruppel, D. Sullivan, R. Matney, R. Fox, J. Albright, B. Johnson (2018), The Tactics and Tropes of the Internet Research Agency, Austin, TX: New Knowledge. Domo (2015), ‘Data Never Sleeps: Infographic’, American Fork, UT: Domo. Available online: https://web-assets.domo.com/blog/wp-content/ uploads/2015/08/15_domo_data-never-sleeps-3_final.png (accessed 11 September 2020). Erbschloe, M. ([2017] 2020), Social Media Warfare: Equal Weapons for All, Boca Raton, FL: Auerbach. Fan, R., J. Zhao, Y. Chen and K. Xu (2014), ‘Anger Is More Influential Than Joy: Sentiment Correlation in Weibo’, PLoS ONE, 10 (9). Available online: https://journals.plos.org/plosone/article?id=10.1371/journal. pone.0110184 (accessed 27 January 2020). Fazio, L., D. Rand and G. Pennycock (2019), ‘Repetition Increases Perceived Truth Equally for Plausible and Implausible Statements’, Psychonomic Bulletin and Review, 26: 1705–10. Gerasimov, V. (2013), ‘Ценность науки в предвидении: Новые вызовы требуют переосмыслить формы и способы ведения боевых действий’, ВОЕННОПРОМЫШЛЕННЫЙ КУРЬЕ, 26 February: 1. Available online: https://vpk-news.ru/ articles/14632 (accessed 1 March 2019). Glassner, B. (2009), The Culture of Fear (Updated Edition), New York: Basic Books. Goldman, A., J. Barnes, M. Haberman and N. Fandos (2020), ‘Lawmakers Are Warned That Russia Is Meddling to Re-Elect Trump’, New York Times, 20 February 2020. Gramlich, J. (2019), ‘10 Facts About Americans and Facebook’, Washington, DC: Pew Research Center, 16 May. Available online: https://www.pewresearch.org/facttank/2019/05/16/facts-about-americans-and-facebook/ (accessed 12 July 2020). Gregg, M. and G. Seigworth (eds) (2010), The Affect Theory Reader, Durham, NC: Duke University Press.

AFFECT AND THE DIGITALIZATION OF WAR

287

Habib, M., M. Cassotti, S. Moutier, O. Houde and G. Borst (2015), ‘Fear and Anger Have Opposite Effects on Risk Seeking in the Game Frame’, Frontiers in Psychology, 10 March. Available online: https://www.frontiersin.org/ articles/10.3389/fpsyg.2015.00253/full (accessed 12 December 2020). Isaac, M. and D. Wakabayashi (2017), ‘Russian Influence Reached 126 Million through Facebook Alone’, New York Times, 30 October 2017. Kaiser, B. (2019), Targeted: The Cambridge Analytica Whistleblower’s Inside Story of How Big Data, Trump, and Facebook Broke Democracy and How It Can Happen Again, New York: Harper. Kolpakas, I. (2016), ‘Social Media: Mimesis and Warfare’, Lithuanian Foreign Policy Review, 35: 119–36. Kramer, A., J. Guillory and J. Hancock (2014), ‘Experimental Evidence of Massive-Scale Emotional Contagion through Social Networks’, Proceedings of the National Academy of Sciences USA, 111 (24): 8788–90. Available online: https://www.pnas.org/content/111/24/8788 (accessed 23 February 2020). Landau, M., Z. Rothschild and D. Sullivan (2011), ‘The Extremism of Everyday Life: Fetishism as a Defense against Existential Uncertainty’, in M. Hogg and D. Blaylock (eds), Extremism and the Psychology of Uncertainty, Chichester: Wiley-Blackwell. Lerner, J. and D. Keltner (2001), ‘Fear, Anger, and Risk’, Journal of Personality and Social Psychology, 81 (1): 146–59. Lerner, J., R. Gonzalez, D. Small and B. Fischoff (2003), Psychological Science, 14 (2): 144–50. Lucas, S. (2012), ‘Mobilizing Fear: U.S. Politics before and after 9/11’, in S. Thompson and P. Hoggert (eds), Politics and the Emotions: The Affective Turn in Contemporary Political Studies, 79–92, New York: Continuum. Massumi, B. (2002), Parable for the Virtual: Movement, Affect, Sensation, Durham, NC: Duke University Press. NBC (2019), Study 19305 NBC News/Wall Street Journal Survey August 10–14, 2019. Available online: https://www.documentcloud.org/documents/633678719305-NBCWSJ-August-Social-Trends-Poll.html (accessed 12 September 2020). Pew (2016), ‘A Wider Ideological Gap between More and Less Educated Adults’, U.S. Politics and Policy, 16 April, Washington, DC: Pew Research Center. Available online: https://www.pewresearch.org/politics/2016/04/26/awider-ideological-gap-between-more-and-less-educated-adults/ (accessed 13 September 2020). Rosenberg, M., C. Savage and M. Wines (2018), ‘Russia Sees Midterm Elections as Chance to Sow Fresh Discord, Intelligence Chiefs Warn’, New York Times, 13 February 2018. Rumer, E. (2019), ‘The Primakov (Not Gerasimov) Doctrine in Action’, 5 June, Washington, DC: Carnegie Endowment for Internal Peace Available online: https://carnegieendowment.org/2019/06/05/primakov-not-gerasimovdoctrine-in-action-pub-79254 (accessed 11 September 2020). Ryan, T., A. Chester, J. Reece and S. Xenos (2014), ‘The Uses and Abuses of Facebook: A Review of Facebook Addiction’, Journal of Behavioral Addictions, 3 (3): 133–48.

288

MATERIALIZING DIGITAL FUTURES

Scissors, L., M. Burke and S. Wengrovitz (2016), ‘What’s in a Like? Attitudes and Behaviors around Receiving Likes on Facebook’, Proceedings of the 19th ACM Conference on Computer Supported Cooperative Work and Social Computing, 1501–10. Sedgwick, E. (2003), Touching, Feeling: Affect, Pedagogy, Performativity, Durham, NC: Duke University Press. Singer, P. (2018), Like War: The Weaponization of Social Media, Boston, MA: Mariner Books. Smith, P. (1989), On Political War, Washington, DC: National Defense University Press. Special Counsel (2019), Report on the Investigation into Russian Interference in the 2016 Presidential Election: Vol I, Washington, DC: U.S. Department of Justice. Available online: https://www.justice.gov/storage/report.pdf (accessed 15 June 2019). Spinoza, B. (1992), Ethics: Treatise of the Emendation of the Intellect and Selected Letters, Indianapolis, IN: Hackett. Tarde, G. (1969), On Communication and Social Influence, Chicago, IL: University of Chicago Press. Tomkins, S. (1962–92), Affect Imagery Consciousness, 4 vols, New York: Springer. Wiener, N. (1948), Cybernetics, or Control and Communication in the Animal and the Machine, New York: John Wiley & Son. Wike, R., L. Silver, S. Schumacher and A. Connaughton (2021), ‘Many in U.S., Western Europe Say the Political System Needs Reform: Americans Are Especially Likely to Say Politicians Are Corrupt’, Pew Research Center. Available online: https://www.pewresearch.org/global/2021/03/31/manyin-us-western-europe-say-their-political-system-needs-major-reform/?utm_ content=buffer07010&utm_medium=social&utm_source=twitter. com&utm_campaign=buffer (accessed 26 May 2021). Wollebaek, D., R. Karlsen and K.Steen-Johnsen (2019),‘ Anger, Fear, and Echo Chambers: The Emotional Basis for Online Behavior’, Social Media and Society, 9 April 1–14. Available online: https://journals.sagepub.com/doi/metr ics/10.1177/2056305119829859 (accessed 8 February 2020). Wylie, C. (2019), Mindf*ck: Cambridge Analytica and the Plot to Break America, New York: Random House.

15 Automation is a myth Luke Munn

Introduction Automation is a myth, a long-running fable about the future of work that needs to be reconsidered. From industry forums to technology journalism and the frontpages of the popular press, this fiction abounds, and for good reason. This is a tale of drama, where technologies suddenly sweep in and society undergoes a massive upheaval. This is a tale of crisis, where work as our longstanding means of identity and economic support is now placed in jeopardy. And this is a tale that is universal, obliterating geographical and cultural distinctions to insist that this will affect everyone, everywhere. The myth of automation, then, is a powerful narrative, one that simultaneously holds out the promise of economic prosperity while warning about a set of potentially disastrous effects. Yet whether embraced as dream or cautioned as nightmare, automation is ultimately a fiction, a fantasy. Myth does not imply that automated technologies do not exist, nor that there have not been technically driven transformations in the nature of work over the last century. But these transformations have been piecemeal rather than total. They have taken place differently within different cultures and locations. And they have impacted particular races and genders rather than a generic humanity. In speaking of ‘myth’, then, we are taking aim at the universal understanding of automation constructed by population automation discourse, a conception that abounds in press headlines and popular bestsellers. Indeed, this version of automation must be a fable, because it rests on a set of triple fictions: the

290

MATERIALIZING DIGITAL FUTURES

myth of autonomy, the myth of automation everywhere and the myth of automating everyone. These claims are disconnected from real-world conditions, from the socio-material conditions experienced by particular people in particular places. As a result, this version of automation becomes a free-floating concept, untethered from the hard ground of reality. In grabbing headlines and framing the debate, this myth is not just misguided but dangerous. It obscures the material transformations taking place in factories, warehouses and hospitals, changes that are reconfiguring labour conditions for particular people in particular places. And it prevents us from tackling a series of more nuanced questions that result from the complex new entanglements of machinic and human labour. Before this work can begin, the myth of automation must be problematized. But more than merely debunking, this chapter aims to use the myth as a springboard, pushing back against its claims in order to gesture to the sharper conceptualization that needs to take place.

The myth of automated autonomy Automation is a myth because its central conceit of human-less work is unattainable. At the root of the word automation is a claim to become self-acting, to move and act on its own (Harper, 2021), and automation continues to insist that the human is on the way out, that machines – better, faster, more productive – will soon take over all the essential operations. This assumption drove the concept of the ‘lights out’ factory, a vision initially championed by General Motors (GM) in the 1980s as a means of clawing back dominance from Japanese automakers (Holusha, 1982). In this dream, every aspect of production would be mechanized; the product would be assembled from start to finish by machines. Once the factory had attained this lofty peak of efficiency, the last symbolic concession to the human – light to work by – could finally be removed. While business analysts generally consider GM’s foray a failed endeavour, this powerful dream has persisted and even been reenergized based on claims around newer technologies (Linder, 2020). The earlier technologies were too crude – but this time will be different. Listen to highly influential tech leaders like Elon Musk, and you’ll hear about the ‘alien dreadnought’ factory, a facility asserting that the future of work will be ‘completely inhuman’ by design (Debord, 2017: n.p.). Watch the promotional videos of technology titans like Alibaba and Amazon, and you’ll be presented with a vision of smooth logistical flow devoid of human touch. In these visions, machines ceaselessly create products and ship packages in a choreography governed by vague algorithms or ethereal artificial intelligence (AI). But when we search for the locus of automation, for that imagined point where autonomous machines extinguish the human, we come up empty

AUTOMATION IS A MYTH

291

handed. The fabled site does not exist. Instead we encounter, again and again, the fleshiness of human labour. On a warehouse floor in Hangzhou, a fleet of automated guided vehicles dart back and forth, each shuttling a package to a chute and dropping it in before returning to grab another. The vast facility, run by the Alibaba logistics arm Cainiao, seems to be empty save for these robots. ‘Packages are moving around freely in the warehouse with no people present’, states the video’s description (AlibabaTech, 2018: n.p.). Clearly an example of automation, the promo proclaims. And yet once we start unpacking this scenario, the human re-emerges and grand claims of autonomy begin to crumble. Follow the packages down the chute and you will find laborers collecting them on the floor below. These are bundled into paper bags and hauled into the back of a truck for shipping. Automation imaginaries may nod briefly to this kind of labour while insisting that it slots into a broader production process governed with machine-like precision. The human is a cog in the machine, necessary for now. Yet this labour can be disobedient rather than tightly controlled. Last year, just before the biggest shopping day of the year, logistics companies in China faced a formidable challenge: its couriers were not showing up to work. In a country where strikes are officially illegal under the law, companies like YTO Express and Yunda Express began reporting ‘abnormal operations’ (Feng, 2020: n.p.). The mostly young male drivers were walking off the job, protesting unfair wages and harsh work conditions. Express deliveries in Hunan, Jiangsu and Shanghai were affected (Feng, 2020), delaying packages and forcing sellers to scramble to find alternative couriers. This type of labour intervention would hardly have been effective if humans are as superfluous as automation rhetoric makes them out to be. But this is the ‘last mile’, some might argue, the difficult-to-automate segment of a production chain that is rapidly becoming routinized. And yet human labour persists in the main when we start to peel back claims around ‘smart’ or ‘realtime’ warehouses. At Cainiao, the applications that support this array of functionality run in server containers, requiring intensive computing power. In a three part series, engineers outline the work they have done to optimize these processes, employing an ‘elastic scheduling system’, gathering metrics on existing use, establishing policies for allocating tasks and building in tolerance for glitches (Yongzhou, 2020: n.p.). After deploying their changes, they monitor the impacts, correct for any mistakes, optimize if possible and deploy again. While these details are highly technical, the point here is that the engineering team needed to carry out an enormous amount of meticulous work. The performativity of the smart warehouse does not simply happen, but rather depends on a performance that was developed by human labour and is dependent on it for ongoing operations. This is maintenance not magic. In his book New York 2140, sci-fi author Kim Stanley Robinson suggests that this labour of repairing and tending will not go away. In

292

MATERIALIZING DIGITAL FUTURES

describing the kinds of energy and transportation systems needed in a postAnthropocene world, he foregrounds the persistent role that humans will play. These systems ‘provided a lot of employment needed to install and maintain such big and various infrastructures. The idea that human labor was going to be rendered redundant began to be questioned: whose idea had that been anyway?’ (Robinson, 2017: 406). Another post from Cainiao’s Chief Technology Officer applauds how technical teams willingly gave up their holidays, ‘pulled all nighters’ and ‘buried themselves in a lot more workload than usual’ in response to pressures on technical systems during the pandemic (Gui, 2020: n.p.). Gui’s statements point to the immense amount of labour poured into ‘automated’ systems. This labour, highly skilled and highly paid, depends in turn upon a vast edifice of human labour from chipmakers and server technicians to the lecturers and assistants who helped train these software engineers in the first place. Of course, not all of this labour is desirable. We might think here of the ‘digital piecework’ (Dubal, 2020) introduced by crowdsourcing marketplaces like Amazon Mechanical Turk. On these platforms, workers take on hundreds of microtasks per hour, each paying pennies. This labour might involve classifying a product, labelling an image or moderating a piece of content. Technical systems rely heavily on this work to produce highquality datasets and high-quality user experiences, and yet these workers are typically anonymous and have little agency to address work conditions. This is the hidden labour that is crucial for ‘autonomous’ systems, the ‘ghost work’ (Gray and Suri, 2019) behind the technological fantasy. In Our Robots, Ourselves: Robotics and the Myths of Autonomy, technologist David Mindell echoes many of these points, pushing back against the inevitability of ‘full automation’. The concept of linear progress presented by some (Singer, 2009) assumed that technology is constantly improving, that human presence will move from direct involvement to a remote or mediated intervention before autonomous systems supplant them altogether (Mindell, 2015: 14). According to this fable, humans will soon be ‘out of the loop’ altogether. But there is no evidence to suggest that this is the case. In fact, counters Mindell, the opposite is true – the future suggests a closer coupling between technical systems and human workers, a deeper intimacy between people and their machines. ‘For any apparently autonomous system’, notes Mindell (2015: 15), ‘we can always find the wrapper of human control that makes it useful and returns meaningful data’. Machines require regularity and uniformity. But labour, it turns out, is highly complex and highly differentiated. Tasks might require a huge variety of hand gestures, or a deep knowledge of the domain to make decisions, or the ability to improvise and deal with crises – areas where machines struggle and humans excel. Transforming these diverse tasks into a process that is repeatable, predictable and therefore machine-operable is easier said than

AUTOMATION IS A MYTH

293

done. The mantra that new technology will resolve these issues has been repeated until it is no longer believed. Trust has been eroded by too many empty promises. ‘Full automation is not looking promising at all’, stated one engineer (Russell, 2020: n.p.): ‘The idea of fully eliminating workers hasn’t been practical in the past 70 years, and there have been hundreds of prototypes of full automation that have never been commercialised.’ These views are all the more powerful in that they come from technologists rather than sociologists, from industry insiders rather than critical outsiders. Such statements puncture the overinflated myth of full automation and reconfirm the centrality of human labour for the future. Certainly, technology brings with it change. There is no question that work conditions in the twenty-first century will differ from the nineteenth or twentieth century. And these shifts bring with them fundamental questions about the future of work. But this future is increasingly spoken of in terms of a collaboration between human labour and mechanic processes – integrating rather than replacing (Munn, 2021). We might think here of an Uber driver and her deep entanglements with a range of digital sensors, mobile apps and systems processes. For those who are managed by these automated systems, it becomes crucial to understand the logic underpinning them: How are tasks broken down? What variables are tracked Where are the gaps or slippages in understanding? This adjustment is key for ensuring that the worker makes her labour algorithmically legible (Munn, 2017; 2018). The worker alters her practices and gestures so that they are recognized as the performance that is desired. This is a kind of mutual subjectivation, where the worker accommodates her labour to technology, and the technology, in turn, ‘learns’ from the worker. This partial and collaborative automation is a more modest proposal, but also a more realistic one that prompts a sharper set of inquiries. How might technical systems assist humans with key tasks? What kinds of values and assumptions are embedded within these systems? Where exactly are the hand-off points in this mixed machinic–human labour? And what happens to oversight and ethics when automated decision-making is employed? Here retaining the fantasy of human-less labour in the face of reality becomes dangerous, obscuring the real issues occurring ‘on the ground’ and the critical research needed to work through them.

The myth of automation everywhere Automation is a myth because automation is universal. Whether described in boosterish business terms or predictions of doom, automation is framed as a ubiquitous force rippling across the globe. Bestsellers warn that an ‘automation wave’ (Ford, 2016) is about to hit the world. Business reports echo this understanding, describing three ‘waves’ that will sweep over

294

MATERIALIZING DIGITAL FUTURES

society (Hawksworth et al., 2018). For other pundits, automation is better described as an ‘age’ (Oppenheimer, 2019), a new era that will usher in global change. These conceptions can often be witnessed in discussions of automation, which quickly leap into a highly abstracted debate employing grand terms like ‘humanity’, ‘technology’ and ‘society’. Together, these terms work to toss out the specificities of city and country, culture and region. Automation, it seems, is a universal condition, a de-situated force that will soon envelop the planet. Where exactly is automation to be found? Automation stands in for a vast and confusing assemblage of historical events, productive processes, mechanical tools and computational systems. In refusing to be pinned down, it loses most of its analytical purchase. Automation is everywhere and nowhere. But dig into these systems and you quickly discover their sited-ness, their grounding in a particular time and place. Open up the automated guided vehicle in the Hangzhou warehouse discussed earlier, and you will find a chip made by HiSilicon, a Chinese manufacturer. Several years ago, the inroads that Chinese firms have made in the fields of machine learning, wireless protocols and particularly in 5G – a key critical infrastructure technology for the foreseeable future – caused unease in Washington (Munn, 2020). The Trump administration responded by blacklisting information technology providers like Huawei and Xiaomei along with dozens of other Chinese tech firms. US companies were purchasing or supplying electronic components from any corporation on the list. These moves escalated into a ‘technology war’ (Dieter, 2020) which meant that cooperation or even communication between Chinese and American tech firms was prohibited. Chinese companies responded by domesticating their supply chain: American components were stripped out and alternative Sino suppliers were found. Of course, supply chains are complex and these substitutions will not take place overnight. Yet these technology wars and regulatory red tape will no doubt persist for many years, forcing global alliances to be replaced with technology which is designed and developed locally. Indeed, commentators have begun speaking of a ‘red stack’ (de Seta, 2018) – layers of Chinese-made technology that start at the bare metal of the chip and move all the way up through operating systems and wireless protocols to the software applications at the top. These dynamics point to a basic but fundamental point: automation is not a universal state, but rather a fragmented condition, one shaped as much by geopolitical forces as by the march of technological progress. Indeed, technology itself is deeply situated, grounded in a particular place and time. Context shapes how these techniques, infrastructures and processes will be taken up and implemented. Just as there is no singular television but rather ‘this television, our television’, there is no singular technology; technologies must be understood as a ‘cultural form’ (Yang, 2012: 49). Yuk Hui (2019) makes this same point by excavating a distinctly Chinese philosophy of

AUTOMATION IS A MYTH

295

technology from traditional Daoism and Confucianism, arguing that there is not a single technic but multiple technics. In the same way, we can assert that there is not a single automation, but rather many automations. These versions are distinct, varying from place to place. Automation in China is different from its deployment in the United States, which is different again from its use in Brazil or Australia. The nationalization of technology happens in several ways. First, each country has its own unique blend of software developers, hardware suppliers, research and development laboratories, and academic institutions. This domestic ecosystem has certain strengths and weaknesses. A confluence in factors in China, for example, has meant that it excels at facial recognition intelligence and applications, while lagging behind in general AI research and chip production (Ernst, 2020). These characteristics, in turn, privilege certain aspects while ignoring others. Chinese companies, according to one influential expert (Lee, 2018), place a premium on applying technology in practical ways to generate profit, while accepting imitation as part of the landscape. In fact, companies in Shenzhen have turned imitation into a new art, drawing on the supplier ecosystem around them to produce wildly inventive digital devices for niche markets, a phenomenon now known as shanzhai (Lindtner et al., 2015). This novel form of cultural production leverages the unique environment around it, reminding us of the importance of place in shaping technological development and production. Second, the economic and policy environment differs enormously from place to place. For some governments, automation is a ‘dirty word’; for others, it is a buzzword (Khalid, 2017). This view determines the particular mix of political and financial incentives that are offered around technology development. Enticements might include a laissez faire approach to regulation alongside long-term loans and tax breaks. One Shenzhen firm focused on automation disclosed that it was receiving large pay outs from the city of Shenzhen as well as a state grant to produce a new factory, where ‘the land, the factory and even the office furniture are all offered by the government for free’ (Khalid, 2017). Together, these measures aim to lower the financial outlay of purchasing machines, tooling and training, reducing the risk of being an early adopter. For better or worse, they alter the ‘landscape’ around companies and institutions, shaping their stance towards labour and technology. And finally, even ostensibly international technologies like software standards or wireless protocols get modified as soon as they ‘touch down’ at the level of the nation, the state or the region. The global gets adapted for the local. For nation-states, this adaptation is a way to ensure that technologies match the values they ascribe to the nation. For companies, it is savvy localization, ensuring an optimum fit between a technology and a particular market. And for individuals, it is often simply a way to make things work, to overcome hurdles and get some practical benefit. ‘If it is to be of any

296

MATERIALIZING DIGITAL FUTURES

use’, asserted Arnold Pacey in The Culture of Technology (1983: 3), any technology ‘must fit into a pattern of activity which belongs to a particular lifestyle and set of values’. During this acquisition process, the technology itself becomes changed: some functions are prioritized, some are tweaked and others are scrapped entirely. Global technology is reworked, taking on the hues of its cultural surroundings. Going further, we might note the diverse ways that different industries within a country have adopted automated techniques. Human resources has seemed to embrace robotic process automation as a faster (if problematic) way to screen potential applicants. Healthcare, by contrast, is still among the least digitized sectors in certain areas (Spatharou et al., 2020). Part of this is simply a lack of spending. Underfunded health district boards have struggled to pool together the funds needed to invest in key areas. But part of this is due to the ethical and moral issues surrounding new technologies. AI and automation spin up a whole range of controversial issues from data collection and privacy to the use of automated decision making in the highstakes area of medicine. The same kinds of deep problems can be seen when introducing automation into the social safety nets meant to alleviate poverty. As Virginia Eubanks (2018) has extensively documented, high-tech tools have produced devastating outcomes for many, forcing some systems to be rolled back and replaced with hybrid approaches that blend computerized processes with face-to-face interactions. Digging deeper, we could even start to examine how these technologies differ from city to city and region to region. China’s economic planning, for example, has long used a ‘specialized towns’ model (Bellandi and Di Tommaso, 2005), employing financial and policy incentives to privilege certain kinds of technologies in certain areas (Zheng and Kuroda, 2013). These specificities mean that the application and location of automation is key – the ‘where’ of automation cannot simply be ‘wherever’. As a brief example, we might contrast the automation of logistics in Hangzhou mentioned earlier with the automation of surveillance in Xinjiang. Situated in northwest China, Xinjiang is an autonomous region home to millions of Uighur, an ethnic minority that is mostly Muslim. In response to ‘unrest’ in the region, the Chinese state has waged an increasingly brutal campaign against this people group in the form of re-education camps, population control and one of the most sophisticated surveillance regimes in the world. Through digital checkpoints, phone tracking and a network of facial detection cameras, technologies collect data, monitor movement and behaviour, and use this as evidence to punish Uighur citizens on a daily basis. This apparatus has been described as ‘automated oppression’ (Human Rights Watch, 2020) or ‘automated racism’ (Mozur, 2019). Rather than simply parroting labour-saving as a benefit, then, we need to ask what labour is being saved in what context, who it benefits, and who it harms. ‘Automation’ is not the same thing in Xinjiang as it is

AUTOMATION IS A MYTH

297

in Hangzhou. Instead we see how they operate in distinct ways, enfold particular actors and prompt different sets of questions. All of this means that we need to localize our focus and acknowledge the social, cultural and geographical context when considering the impacts of automation. To speak of automation occurring everywhere is to speak of nothing in particular.

The myth of automating everyone Automation is a myth because the catchall figure of ‘the human’ at the centre of its claims does not exist. Over the last century, discussion on automation has consistently evoked a generic humanity as the target of its impacts. In the 1920s, the New York Times asked: ‘Will Machines Devour Man?’ (‘Man Devoured by His Machines’, 1921). In the 1940s, one author warned that the machine was ‘gaining on mankind’ (Boyle, 1949). In the 1960s, academic anthologies featured articles such as ‘Man, Automation, and Dignity’ (Brickman and Lehrer, 1966). And in the 1980s, Leontief and Duchin (1986) investigated The Future Impact of Automation on Workers over a sprawling 170-page report, but never specified this worker in detail and scarcely acknowledged disparities of race and gender across the workforce. An earlier and more naive time, some might argue. But even Frey and Osborne’s (2013) far more recent study sketches out a fairly generic worker. Their task-based breakdown of work aims to see how susceptible certain tasks are to automation, or what they term computerization. But this seemingly egalitarian approach smuggles in a view of man-as-motorfunction through the back door, blurring out the race, culture and class of the individual. This abstraction ignores the sociocultural factors that have historically steered certain groups into certain positions. ‘Anyone’ can code and ‘anyone’ can drive a forklift, but it is clear that not just anyone does. Stereotypes like the white male brogrammer or immigrant care worker point to the skewed demographics in certain professions, demonstrating that other forces are at play. In evoking ‘man’ or ‘humankind’, automation posits a universal subject. Racial, cultural and socioeconomic differences are erased, evoking the human as a beige body and humanity as a shapeless mass of labour. Automation discourse adopts what Atanasoski and Vora (2019: 28) term a technoliberalist stance, asserting that ‘racial difference, along with all human social difference, is transcended’. Indeed, claims about automation derive their power in large part from this universalizing tendency. If automation is proclaimed as a gateway to a labour-free utopia, it will float all boats, lifting humanity as a whole into a new era. If automation is derided as a slippery slope to an economic nightmare, there will be no escape. Either way, it will

298

MATERIALIZING DIGITAL FUTURES

happen to us all. In these framings, ‘the labourer’ becomes a fungible object, interchangeable with any other. A worker is a worker is a worker. Yet we know that this generic worker does not exist, only individuals and communities do. Work is marked at all times by social difference – indeed, the history of labour is a history of racial, cultural and sexual stratification. Some workers are privileged; others are oppressed. Some workers earn more; others earn less. Jason Moore (2015) has shown how colonial capitalism valued humans differently, exploiting cheap labour as a value-generating mechanism to reap enormous profit. Cedric Robinson (2000) has tracked how the development and expansion of capital pursued racial directions, racially stratifying labour from very early on. These scholars and many others have demonstrated that this unequal treatment was a feature not a glitch. Accumulation requires ‘loss, disposability, and the unequal differentiation of human value’, asserts Jodi Melamed (2015: 77), ‘and racism enshrines the inequalities that capitalism requires’. Based on these uneven conditions, automation’s fallout will also be uneven. The mythic ‘human’ at the heart of automation’s claims masks the fact that some humans will benefit enormously from these transformations while other humans will greatly suffer. For one, the ability of a worker to be ‘resilient’ to these changes and adapt to them is a function of financial security and stability. Here we are talking about wealth rather than simply income, the command of financial resources that provides access to life chances (Oliver and Shapiro, 2006). Wealth provides a kind of buffer against technological upheaval. Access to wealth might allow an unemployed worker to go without a pay check for months while looking for work, or to pay tuition fees for retraining, pivoting to a new career. A lack of wealth, by contrast, leaves workers without any fallback, exacerbating an already precarious situation. And the distribution of wealth is not just uneven, but racially inflected. In the United States, for instance, there is already a historic imbalance when it comes to race and wealth, and because of this, one report suggested that automation may be ‘widening the racial wealth gap between African American families and white families in America’ (Cook, 2019: 1). For another, automation is taking place in industries that are already racially stratified. Automation has long promised to replace jobs that tick any of the ‘4 Ds’: dull, dark, dirty or dangerous (Ahlen, 1987). But it is not just anyone who performs this kind of labour. Historically, as sociologist Mignon Duffy (2007) notes, ‘dirty people’ do the dirty work. In the notoriously dangerous warehouse sector, for instance, women, immigrants and people of colour make up significant portions of the workforce and will be disproportionately impacted. Amazon’s (2019) own statistics reveal that women comprise 42 per cent of employees globally, yet only 27 per cent are in management. Similarly, 26.5 per cent of the workforce identity as Black or African American, but only 8 per cent hold managerial positions (Amazon, 2019). Indeed, one review of the company’s workforce found that

AUTOMATION IS A MYTH

299

85 per cent of Amazon’s black employees work in ‘unskilled’ warehouse jobs (Demmitt, 2015). On a broader industry level, workers of colour constitute 66 per cent of warehouse workers, even though they account for just 37 per cent of the total US labour force (Gutelius and Theodore, 2019). These numbers mean that the drastic reorganization of warehouse labour will impact these demographics most directly. The black sorter on the factory floor is exposed to the full physical and psychological brunt of automation, while the white manager in the office above largely witnesses these effects from afar. It is not ‘humanity’ whose jobs will be replaced, but particular humans.

Conclusion Automation is a myth, predicated on a set of triple fictions. There is the myth of full autonomy, the fable that machines will soon operate completely independently, taking over the production cycle and supplanting the human. But this dream is being rapidly scaled back and converted into a more modest set of interventions. Far from being self-acting, these technical systems require development and maintenance, reminding us of the human labour behind ‘autonomous’ processes. Second, there is the myth of universal automation, where these technologies are framed as a desituated force that will sweep across society and ripple across the globe. But this fiction ignores the social, cultural and geographical forces that shape technologies at a local level. Automation is both technical and geopolitical, and any discussion must situate the impacts of these technologies within a specific context. Third, there is the myth of automating everyone, the generic figure of ‘the human’ at the heart of automation claims. Labour is socially stratified and so automation’s fallout will also be highly uneven, falling heavier on some populations (immigrants, people of colour, women) and lighter on others. Built atop these three fictions, automation itself is a fiction. This fable is powerful but unhelpful, derailing a more articulated set of problematics. In one sense, this means breaking down automation to be more specific. What, where, and who are we talking about when we talk about automation? Zooming into a particular company or region – automated systems across Alibaba’s warehouses, for instance – is one way to counteract the grand tendencies of automation discourse. Sharpening our understanding of labour – moving from ‘humanity’ to East African workers at Amazon – would be another means of pushing back against these sweeping generalizations. Narrowing down these frames of reference helps us to better grasp the particular people and things that are being transformed and the socio-technical forces at work. In another sense, this means acknowledging that automation is fundamentally cross-cutting, a social, cultural and political project as much

300

MATERIALIZING DIGITAL FUTURES

as a technological one. A siloed approach where sociologists work in one corner, computer scientists in another, and race and cultural scholars in a third will not suffice. Any critical engagement with automation will need to also cut across disciplinary boundaries, understanding technical logics but also folding in powerful insights from history and the humanities. This is a both/and approach: enfolding both operations and representations, both processes and people. By sharpening the focus and thickening the description, the grand sweep of ‘Automation’ in all caps is reduced to a real research object. Better questions can be asked; better insights can be generated. Only then can we make progress on this timely and urgent issue.

References Ahlen, J. E. (1987), ‘Impact of Robots on the Workforce’, Los Angeles, CA: California State University. Available online: https://www.elibrary.ru/item. asp?id=6758431. AlibabaTech (2018), Cainiao Smart Warehouse. Available online: https://www. youtube.com/watch?v=jRYaAGH_Z-0. Amazon (2019), ‘Our Workforce Data’, About Amazon, 10 April. Available online: https://www.aboutamazon.com/news/workplace/our-workforce-data. Atanasoski, N. and K. Vora (2019), Surrogate Humanity: Race, Robots, and the Politics of Technological Futures, Perverse Modernities, Durham, NC: Duke University Press. Bellandi, M. and M. R. Di Tommaso (2005), ‘The Case of Specialized Towns in Guangdong, China’, European Planning Studies, 13 (5): 707–29. Available online: https://doi.org/10.1080/09654310500139244. Boyle, H. (1949), ‘Machines Are Laughing at Men’, Evening Independent, 14 April. Available online: https://news.google.com/newspapers?nid=950&dat=19490414 &id=LPhPAAAAIBAJ&sjid=glUDAAAAIBAJ&pg=5992,5416204&hl=en. Brickman, W. W. and S. Lehrer (1966), Automation, Education, and Human Values, New York: School & Society Books. Cook, K. (2019), ‘The Future of Work in Black America’, Washington, DC: McKinsey. Available online: https://www.mckinsey.com/featured-insights/ future-of-work/the-future-of-work-in-black-america#. Debord, M. (2017), ‘Tesla’s Future Is Completely Inhuman – and We Shouldn’t Be Surprised’, Business Insider, 21 May. Available online: https://www. businessinsider.com.au/tesla-completely-inhuman-automated-factory2017-5?r=UK. Demmitt, J. (2015), ‘85 Percent of Amazon’s Black U.S. Workers Hold Unskilled Jobs’, Puget Sound Business Journal, 11 June. Available online: https://www. bizjournals.com/seattle/blog/techflash/2015/06/85-percent-of-amazon-s-black-us-workers-hold.html. Dubal, V. (2020), ‘Digital Piecework’, Dissent Magazine, Fall. Available online: https://www.dissentmagazine.org/article/digital-piecework. Duffy, M. (2007), ‘Doing the Dirty Work: Gender, Race, and Reproductive Labor in Historical Perspective’, Gender & Society, 21 (3): 313–36.

AUTOMATION IS A MYTH

301

Ernst, D. (2020), ‘Competing in Artificial Intelligence Chips: China’s Challenge Amid Technology War’, Waterloo, Canada: Centre for International Governance Innovation. Eubanks, V. (2018), Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor, New York: St Martin’s Press. Feng, J. (2020), ‘Couriers Quit before China’s Ecommerce Shopping Festival, Demanding Better Pay and Work Conditions’, SupChina, 30 October. Available online: https://supchina.com/2020/10/30/couriers-quit-before-chinasecommerce-shopping-festival-demanding-better-pay-and-work-conditions/. Ford, M. (2016), Rise of the Robots: Technology and the Threat of a Jobless Future, Illustrated edn, New York: Basic Books. Fredrickson, G. M. (1981), White Supremacy: A Comparative Study of American and South African History, Cambridge: Oxford University Press. Frey, C. B. and M. A. Osborne (2013), ‘The Future of Employment: How Susceptible Are Jobs to Computerisation?’ Technological Forecasting and Social Change, 114 (September): 254–80. Available online: https://doi.org/10.1016/j. techfore.2016.08.019. Gray, M. L. and S. Suri (2019), Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass, Illustrated edn, Boston, MA: Houghton Mifflin Harcourt. Gui, X. (2020), ‘Crisis Is a Test of Our Spiritual Strength’, Alibaba Tech, 13 August . Available online: https://alibabatech.medium.com/ crisis-is-a-test-of-our-spiritual-strength-1270d8c6d85c. Gutelius, B. and N. Theodore (2019), ‘The Future of Warehouse Work: Technological Change in the U.S. Logistics Industry’, Berkeley, CA: UC Berkeley Center for Labor Research and Education. Harper, D. (2021), ‘Automatic’, Online Etymology Dictionary.https://www. etymonline.com/word/automatic. Hawksworth, J., R. Berriman and S. Goel (2018), ‘Will Robots Really Steal Our Jobs?’ London: Price Waterhouse Coopers. Holusha, J. (1982), ‘General Motors: A Giant in Transition’, New York Times, 14 November. Available online: https://www.nytimes.com/1982/11/14/magazine/ general-motors-a-giant-in-transition.html. Hui, Y. (2019), The Question Concerning Technology in China: An Essay in Cosmotechnics, Vol. 3, Cambridge, MA: MIT Press. Human Rights Watch (2020), ‘China: Big Data Program Targets Xinjiang’s Muslims’, Human Rights Watch, 9 December. Available online: https://www. hrw.org/news/2020/12/09/china-big-data-program-targets-xinjiangs-muslims. Khalid, A. (2017), ‘A Dirty Word in the U.S., “Automation” Is a Buzzword in China’, 20 November. Available online: https://www.wbur.org/bostonomix/2017/11/20/ china-automation. Lee, Kai-Fu (2018), AI Superpowers: China, Silicon Valley, and the New World Order, Boston, MA: Houghton Mifflin Harcourt. Leontief, W. and F. Duchin (1986), The Future Impact of Automation on Workers, Illustrated edn, New York: Oxford University Press. Linder, N. (2020), ‘Are We Inching Toward the Lights Out Factory?’ Forbes, 17 June. Available online: https://www.forbes.com/sites/natanlinder/2020/06/17/ are-we-inching-toward-the-lights-out-factory/.

302

MATERIALIZING DIGITAL FUTURES

Lindtner, S., A. Greenspan and D. Li (2015), ‘Designed in Shenzhen: Shanzhai Manufacturing and Maker Entrepreneurs’, Aarhus Series on Human Centered Computing, 1 (1): 1–12. Available online: https://doi.org/10.7146/aahcc. v1i1.21265. ‘Man Devoured by His Machines’ (1921), New York Times, 2 October. Available online: https://timesmachine.nytimes.com/timesmach ine/1921/10/02/107026085.pdf. Melamed, J. (2015), ‘Racial Capitalism’, Critical Ethnic Studies, 1 (1): 76–85. Mindell, D. A. (2015), Our Robots, Ourselves: Robotics and the Myths of Autonomy, New York: Viking Adult. Moore, J. (2015), Capitalism in the Web of Life: Ecology and the Accumulation of Capital, London: Verso. Mozur, P. (2019), ‘One Month, 500,000 Face Scans: How China Is Using A.I. to Profile a Minority’, New York Times, 14 April. Available online: https://www. nytimes.com/2019/04/14/technology/china-surveillance-artificial-intelligenceracial-profiling.html. Munn, L. (2017), ‘I Am a Driver-Partner’, Work Organisation, Labour & Globalisation, 11 (2): 7–20. Available online: https://doi.org/10.13169/ workorgalaboglob.11.2.0007. Munn, L. (2018), Ferocious Logics: Unmaking the Algorithm, Lüneburg: Meson Press. Munn, L. (2020), ‘Red Territory: Forging Infrastructural Power’, Territory, Politics, Governance, 1–20 October. Available online: https://doi.org/10.1080/21622671 .2020.1805353. Munn, L. (2021), ‘Imperfect Orchestration: Inside the Data Center’s Struggle for Efficiency’, Computational Culture, 8 July. Available online: http:// computationalculture.net/imperfect-orchestration-insidethe-datacenters-struggle-for-efficiency/. Oliver, M. and T. Shapiro (2006), Black Wealth, White Wealth: A New Perspective on Racial Inequality, New York: Routledge. Oppenheimer, A. (2019), The Robots Are Coming!: The Future of Jobs in the Age of Automation, trans. Ezra E. Fitz, New York: Vintage. Pacey, A. (1983), The Culture of Technology, Cambridge, MA: MIT Press. Robinson, C. J. (2000), Black Marxism: The Making of the Black Radical Tradition, Durham: University of North Carolina Press. Robinson, K. S. (2017), New York 2140, Rome, Italy: Fanucci Editore. Russell, M. (2020), ‘Automated Picking Bins Set to Help New Zealand Growers Save Labour Costs’, Fresh Plaza, 10 June. Available online: https://www.freshplaza .com/article/9224099/automated-picking-bins-set-to-help-new-zealand-growerssave-labour-costs/. Seta, G. de. (2018), ‘Into the Red Stack’, Hong Kong Review of Books 香港書評, 17 April. Available online: https://hkrbooks.com/2018/04/17/into-the-red-stack/. Singer, P. W. (2009), Wired for War: The Robotics Revolution and Conflict in the 21st Century, London: Penguin. Spatharou, A., S. Hieronimus and J. Jenkins (2020), ‘Transforming Healthcare with AI: The Impact on the Workforce and Organizations’, London: McKinsey. Available online: https://www.mckinsey.com/industries/ healthcare-systems-and-services/our-insights/transforming-healthcare-with-ai.

AUTOMATION IS A MYTH

303

Yang, G. (2012), ‘A Chinese Internet? History, Practice, and Globalization’, Chinese Journal of Communication, 5 (1) (March): 49-54. Yongzhou, Y. (2020), ‘Why Cainiao’s Special and How Its Elastic Scheduling System Works’, Alibaba Tech, 2 January. Available online: https://www. alibabacloud.com/blog/why-cainiaos-special-and-how-its-elasticscheduling-system-works_595702. Zheng, D. and T. Kuroda (2013), ‘The Impact of Economic Policy on Industrial Specialization and Regional Concentration of China’s High-Tech Industries’, Annals of Regional Science, 50 (3): 771–90. Available online: https://doi. org/10.1007/s00168-012-0522-4.

304

16 A triadic typology of material mediation: Ontology, intentionality and vitalism Renata Lemos Morais

Introduction In the context of macrosystems of planetary computation (Bratton, 2015), emerging technologies such as nanocommunication (Chandra and Debashis, 2018), quantum computation and pervasive intelligence (Lee et al., 2020) are pushing the boundaries of material mediation. Advances in computational data processing and visualization technologies have enabled humans to have access to progressively larger areas of the universe, be it in outer space (Parikka, 2015) or in quantum space (Lee et al., 2020). Ever more powerful telescopes and microscopes expand our awareness about what lies beyond the reach of our human senses and perception, at the same time that artificial intelligent agents permeate the vast majority of daily use devices and gadgets, tracking and mapping out global patterns of consumption, mobility and engagement (Zuboff, 2019). When considered as a whole, this emerging technological apparatus that integrates micro and macro levels of reality instigates new theoretical reflections. The broad panorama that is delineated by the combination of these kinds of emerging technologies is one of enormous and serious consequences, as even stable world democracies begin to be influenced and manipulated via the global apparatus of technological mediation (Zuboff, 2019).

306

MATERIALIZING DIGITAL FUTURES

Even though a huge transformative potential is inherent to this global, multi-layered technology apparatus that Benjamin Bratton (2015) calls the Stack, there seems to be little or no comprehensive effort toward the development of a systemic and integrated application of emerging technologies to the current challenges being faced by humanity in the twenty-first century (such as global warming and migration, Covid-19, water insecurity, and the like). On the one hand, the capacity of emerging technologies to bring relief and create solutions to many of these complex problems has never been so promising due to the speed of technological innovation. On the other hand, these technological innovations have worsened the conditions of capitalist exploitation promoting truth obfuscation, distorting and confusing our shared notion of reality (Morais, 2014). Material mediation, in particular, seems to be a concept that needs better defining and articulation in the context of emerging technologies. Material mediation can be defined according to a variety of viewpoints, usually belonging to specific disciplines. In telecommunication science and cybernetics, material mediation is associated with signal transduction and transmission (Widrom and Stearns, 1985). In psychological and cognition science, material mediation is akin to neurotransmission and encompasses emotional responses to sound frequencies (Leman, 2007), for instance. In the field of interaction design, material mediation takes place in the context of responsive digital materials (Wiltse, 2014). These examples showcase a small sample of the fascinating variety of angles through which material mediation can be seen. The disconnect between these diverse disciplinary understandings of material mediation is clear in our literature review. The need for a transdisciplinary system that is able to articulate a common theoretical axis around the concept of material mediation emerged in the course of this research. In order to make proper sense of the complexity of these processes of technological convergence, I have devised three transdisciplinary categories of material mediation: (1) ontological, (2) intentional and (3) vital. These categories are aligned to three main typologies of material mediation: (1) subject-oriented mediation that focuses on semiotic perception and translation according to subjective interpretation; (2) object-oriented mediation that focuses on artifacts and technology networks, infrastructures and assemblages; and (3) subject–object interaction that focuses on the relational dynamics and properties of mediation between the two. Before contextualizing our understanding of material mediation in terms of subjectoriented, object-oriented and subject–object interaction semiotic processes, I will begin by laying down the methodological principles that have inspired and enabled this inquiry. The methodology of transdisciplinarity (Nicolescu, 2014) brings a unique frame of reference to material mediation. As an epistemology rooted in the idea of multidimensional levels of reality, transdisciplinarity allows us to

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

307

reinterpret the coexistence between the different layers present in matter (D’ariano et al., 2017), each operating according to either classic physical laws or quantum physical laws. Material mediation in the quantum level operates in a radically different way than in the ordinary level of human perception. Material properties at the quantum level bring to fore the direct relationship between the subjective dimension of semiotic processes and material reality itself. The way we perceive objects in our daily lives relies on stable material properties that are manifested according to the classical laws of physics. However, beneath the apparent stable surface of objects lies a vibrant modulating field of quantum waves and particles that behave differently depending on the presence or absence of observers. Relational quantum mechanics is the foundation of subject–object interaction approaches to material mediation since it states that ultimately, all objective material mediation is subject-dependent since at the quantum level all matter is observer-dependent (Rovelli, 1996). Relational systems illustrate the diversity and complexity of levels of reality that can be encountered in the analysis of transdisciplinary processes of material mediation. Levels of reality are a ‘multidimensional and multi-referential structure of Reality’ that is ‘a key concept to transdisciplinarity’ and is defined as ‘a set of systems which are invariant under certain general laws: for example, quantum entities are subordinate to quantum laws, which depart radically from the laws of the macrophysical world’ (Nicolescu, 2012: 20). The diversity of principles regulating different levels of material mediation (such as physical, symbolic and embodied) is multidimensional in a way consistent to Nicolescu’s (2012) approach. Further, the transdisciplinary idea of levels of reality enables a unified definition of material mediation that is not confined to any particular level, having the potential to cut across level boundaries and to go beyond particular differences into the realm of universality. This latter possibility (a conception of material mediation that is universal) resonates with Nicolescu’s proposition of zones of non-resistance, defined as the zones that exist ‘between two different levels and beyond all levels’ of reality (Nicolescu, 2012: 20). Nicolescu’s zones of non-resistance imply that the total sum of all levels of reality is not equal to a finite integral reality, because non-resistance transcends all boundaries. By applying this principle to material mediation, one could, in theory, envision infinite levels of mediation in the thresholds connecting classical and quantum physics, mind and matter, sign and meaning. This possibility has inspired the application of Nicolescu’s methodology of transdisciplinarity to the analysis of the three aforementioned categories of material mediation. Transdisciplinary methodology is based on three fundamental axioms: 1.The ontological axiom: There are, in Nature and in our knowledge of Nature, different levels of Reality of the Object and, correspondingly,

308

MATERIALIZING DIGITAL FUTURES

different levels of Reality of the Subject. 2. The logical axiom: The passage from one level of Reality to another is insured by the logic of the included middle. 3. The epistemological axiom: The structure of the totality of levels of Reality is a complex structure – every level is what it is because all the levels exist at the same time. (Nicolescu 2014: 187) These axioms are aligned with the three transdisciplinary categories of material mediation proposed in this chapter. Recognizing the transdisciplinary nature of the semiotic continuum required by processes of material mediation, I have also categorized it as a field of transdisciplinary knowledge. According to Sue McGregor, transdisciplinary knowledge is characterized by three main elements: First, transdisciplinary knowledge is complex and emergent, meaning the knowledge is continually changing as it is created. Second, the creation of transdisciplinary knowledge entails the Logic of the Included Middle wherein as many perspectives as possible are integrated. Third, transdisciplinarity assumes that many levels of reality are central to knowledge creation, including the internal mind of humans (their consciousness) and their external world (information flows). (McGregor, 2013: 63) The relation between the triadic typology of material mediation and transdisciplinary knowledge is expressed by the following arguments: (1) The complexity of the relations between subject, object and subject–object interaction present in material mediation goes beyond all disciplines. Since this field is continuously changing in tandem with the evolution of emerging technologies, it is aligned with the first tenet of transdisciplinary knowledge. (2) The Logic of the Included Middle (Nicolescu, 2014: 2012) inherent in the connection between one level of reality to another enables a pluriversal approach to the analysis of material mediation, making it consistent with the second tenet of transdisciplinary knowledge. (3) Finally, subjective perception and cognition processes are indissociable from material mediation, aligning it with the third tenet of transdisciplinary knowledge. Even though there is no direct correspondence between single categories of material mediation and each of the axioms of transdisciplinary knowledge and methodology, the latter enable consideration of the effects of the interaction between the three categories (ontological, intentional and vital) from an integrative perspective. Each category belongs to a particular level of reality; however, through the zones of no-resistance (Nicolescu, 2012; 2014) that exist across and beyond them, a glimpse of transversal continuity emerges in which mediation presents itself as the bridge between them. It is here that the transdisciplinary potential of material mediation can be seen in its full potency: not only is it able to show the interdependence between different categories, it expands its horizon beyond any categories.

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

309

Expanding the boundaries of mediation Defining material mediation in a comprehensive manner, inclusive of multiple levels of reality such as social technologies, material surfaces, cognitive and psychological dimensions, embodied affect, and so on, requires redefining the nature of the relationship of subject and object. When we consider material mediation as having multidimensional properties depending on which level of reality is being analysed, its scope goes beyond the limits of linguistics, culture and sociology and into the transdisciplinary space of ontology and epistemology. The concept of mediation, material or otherwise, acquires different undertones depending on the context in which it is being used. The word mediation is usually found in two main spheres: subject-oriented and therefore pertaining to psychological and social circumstances,1 or object-oriented and related to the transfer of information and meaning across different channels (e.g. in fields such as telecommunication or nanocommunication). Subject-oriented mediation is therefore context-dependent and relative to cultural and social constraints such as language, semantics, and the like, while object-oriented mediation is determined by material infrastructures of communication such as signals, transmission systems, and so on. All categories of material mediation are context-dependent and will vary in accordance to their level of reality. What is or is not considered as a medium will also determine the way material mediation is understood. Its definition will change according to the standpoint applied to its interpretation. If a medium is understood solely in terms of its material structures of message transmission, then material mediation is something that is not inherent to matter. Mediation has often been understood exclusively as the continuum integrating objective transmission and subjective translation of signs, information and meaning in the realm of human communication and linguistics. This sociological approach to mediation has prevailed throughout modernity and is clearly manifested in the socio-centric interpretation of mediation that is the rule in classical theories of mass communication such as authoritarian/free press, media audiences/content and structural schools (Perry, 2009). Its reign has been long in the media theory field, and it has gone unchallenged up until Marshall McLuhan’s game-changing statement: ‘the medium is the message’ (1994). This Zen-like koan of classic media theory has given rise to multifarious interpretations that have then originated new variations on the idea of mediation such as remediation (Bolter and Grusin, 2000). Digital technologies have enabled social processes of mediation to shift from the material into the virtual realm. Remediation is mediation that jumps from material space into cyberspace, within networks of transparent media (Bolter and Grusin, 2000). Digital media networks expand our understanding of the medium beyond a mere physical channel of message

310

MATERIALIZING DIGITAL FUTURES

transmission, and into a virtual simulacrum of physical reality itself. The interaction between mind and matter, subject and object is transformed by digital technologies engendering new layers of complexity to the study of material mediation. Material mediation can be understood according to three very different, however complementary, approaches. The first is ontological and draws upon the philosophy of materiality and considers mediation as a phenomenon that transcends and is not dependent upon human culture and communication. Material mediation, when seen from this perspective, is a compositional layer of physical reality. Natural processes such as the biological rhythms of living systems, the geological material formation of various kinds of minerals and the particle dynamics of quantum physics can all be understood as processes of material mediation. The second is intentional, and addresses material mediation as either individual or collective agency in technological networks that are intrinsically connected to culture, design and its social implications. This approach is not concerned with the ontological status of matter, but instead focuses on the complex relationship between subject and object and the role of intentionality in socio-technic assemblages. Emphasis is here given not to the substance and nature of matter, but to the pragmatic socio-material applications of technological mediation. The third approach to material mediation is vital and brings together the previous two under the broad theoretical umbrella of new materialism (Barad, 2007; Bennett, 2010; Braidotti, 2013). New materialism is a transdisciplinary field of inquiry that includes radical propositions such as Barad’s material-discursive conditions of reality (2007), Braidotti’s neo-vitalist theory of matter (2013) and Bennett’s thing-power (2010) as non-intentional distributed agency.2

The ontological view: The semiotics of matter The entire universe is perfused with signs, if it is not entirely composed of signs. CHARLES SANDERS PEIRCE (CP, 1958: 5.448, NOTE 1)3

Charles Sanders Peirce has left us an extensive legacy of transdisciplinary concepts that refer to material mediation. Peircean semiotics is intricate and complex, and there are innumerable different interpretations of his work that vary in relation to how they portray the materiality of semiotic processes. His triadic theory of signs has often been applied to the social realm of cultural and internet studies, advertising and mass communications. The interest that the fields of digital design and media have had on Peircean

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

311

semiotics has been mostly human-centred. Before new materialism, most applications of Peirce’s theory of signs in the field of media and design were not related to its material ontology. This changed after the technological processes of convergence seen in quantum, pervasive and unconventional computation brought new attention to the semiotic properties of matter. Against the background of emerging technologies and computation, Peircean semiotics stands out due to its prescience. Peirce’s triadic categories can help us understand technological processes of mediation in a new light. Peirce’s general theory of signs provides us with elements of analysis that are anchored both in objective material reality (substance, form and structure) and in subjective reality (perception and meaning). Peirce’s theory of signs is a general theory of knowledge, addressing simultaneously material processes of signification (sign-based) and virtual processes of meaning interpretation (interpretant-based). The fundamental structures of this material semiotic continuum are as follows: (1) the sign as a vehicle of meaning; (2) the object as meaning itself; and (3) the interpretant and the idea generated by the interpretation of the sign/object. In sum, a sign is ‘a cognizable that is determined by something other than itself, called its object’ (EP, 1897–1909: 2:492). An object is, therefore, what is contained and expressed through the sign, and that is then reckoned with by the subject/interpretant. The action uniting the three is semiosis that happens according to the universal principle of continuity of synechism (CP, 1958: 1.172), irrespective of the nature of its substrata. Peirce’s proposition of semiosis is an absolute form of mediation that is both a material and a virtual constituent of reality. Within the framework proposed by Peirce, the fundamental elements of any kind of mediation are always triadic and composed of sign, object and interpretant. Each of these categories cannot be understood independently from each other, being a part of an ontological semiotic continuum that consists of infinite and universal semiosis – synechism (CP, 1958: 1.172). Peircean semiotics considers signs and their relationships to exist at all levels of reality, subjective or objective. A sign exists only through a relationship between its substance or form of expression, its representative structure and its meaning. This sign relation, in turn, presupposes a subject or a level of consciousness. The principle of synechism is of particular relevance in the context of this research, considering that, according to Peirce, all mediation is continuous: ‘the principle of continuity as the supreme principle of mediation and synthesis (...) continuity is, therefore, mediation’ (CP, 1958: 5.104; 5.436). Synechism presents sign relations as a continuum between matter, systemic structures of representation and thought. The promise of nanotechnological convergence brought about by NBIC (nano-info-bio-cogno) technologies (Dotsenko, 2017) is a concrete example of synechism. Nano manipulation of matter in liminal spaces such as neurons and atoms has materialized Peirce’s principle of semiotic

312

MATERIALIZING DIGITAL FUTURES

continuity. Although the full potential of this semiotic unity of matter is still unfulfilled, it continues to inspire new imaginings about the future possibilities of material mediation. The grandiosity of the claim that cognitive nanotechnology makes in relation to a radical unity between mind and matter could be considered as equivalent to a technological singularity. However, in the name of simplicity and to avoid unnecessary extrapolations that would do little to help us clarify its significance, I will set aside farfetched ideas such as NBIC and technological singularity. The chapter focuses instead on a single, yet multidimensional, example of mediation at the nexus of virtual and material thresholds: slime mould. Slime is both matter and metaphor. It has been used as a metaphor in the book Slime Dynamics (Woodard, 2012), a seminal work of dark vitalism and speculative realism, to represent the alien dynamics of slime as an organism without individuality. In a very real sense, the biological nature of slime mould is itself a form of material mediation. Its very existence is a paradox: it refuses to be categorized as either one or many. It manifests itself at the boundary between virtuality and materiality, in a fluid liminal existence that oscillates between visible and invisible states. Its existence also oscillates between individual and collective manifestation depending on its reading of external environmental signs, in a binary dynamic that is a living example of material mediation. The connection of slime mould to material semiotics is not solely symbolic. As recently as 2016, slime mould has been categorized as ‘a living embodied computational material’ or ‘virtual computing material’ (Jones, 2016: 1). Slime mould is both virtual and material because it mediates the change in external environmental stimuli by self-programming new material pattern formations. The emergent properties of slime material are congruent with the semiotic patterns of computational agents (Jones, 2016). Slime mould makes visible the invisible semiotic nature of matter. Slime mould dynamics exemplify natural processes of computation that mediate the relationship between agency and matter via the semiotic continuity of matter, perception and intentionality. Peircean semiotics is a theory compatible with the phenomenon of material–semiotic assemblages because signs are entities that can be found at all levels of reality. Computation operates through information encoded in signs; biological systems can be analysed as semiotic processes; digital networks create, interpret and reorganize codes and signs; and finally, all manifestations of intelligence, whether human or artificial, require the movement of signs. The flow of information and meaning is the thread that interconnects all these levels into a single material ontology in which all levels of existence are semiotic: material, cognitive, abstract (virtual) or concrete, human or nonhuman, living or non-living. From an ontological perspective, the chapter acknowledges, in line with Peircean semiotic argument, that the fundamental semiotic categories of material mediation are sign, object and interpretant.

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

313

However, once one changes their horizon from the transversal ground of semiosis into the human landscape of media artifacts and networks, material mediation acquires different tones. It is here that new sightlines emerge for material mediation as I propose: (1) ontological, (2) intentional and (3) vital.

Intentionality and mediation: Responsive digital materials Being Digital, the 1995 seminal book by Nicholas Negroponte, was one of the first to point to a proto instance of intentional material mediation when describing the collapse of the separation between categories of natural and technological processes of mediation. This folding of material-based media (which he called atoms) into virtual-based media (referred to as bits) is representative of the new kind of relationship between material and virtual realities that digital technologies had brought to the fore. More than twenty years later, emerging media technologies continue to blur the difference between media atoms and media bits. The internet of things (IoT) can be understood as an example of literal material mediation that enables the functioning of a smart house’s appliances, electrical systems, alarms, air conditioning, windows and doors, and the like. Autonomous driving vehicles are another example. The continuous mediation of data flows by intelligent sensors enables decisions based on navigation systems determined by live information processing in real time, much in the same way that smart wearables can adapt to the weather thanks to sensors. Artificial intelligence has the capacity to be embedded into any material surface. Any kind of material surface can acquire responsiveness when data sensing, processing and flow are combined with artificial intelligence (Tsiatsis et al., 2019). The IoT was originally a field of discrete spatial applications based on seamless interfaces, responsive objects and telepresence robotics. It has evolved into a material–semiotic continuum that goes beyond systems of artificial intelligence and is not confined to the scale of things or humans anymore. There is a change of scale, but also in directionality, currently taking place in digital processes of material mediation that involve autonomous agents. We are moving from the social dimension of networks of people and objects and into the networks of geoengineering artefacts (Parikka, 2015) that configure a material infrastructure of planetary computation (Bratton, 2015). Ecological categories such as planet, environment, habitat and species have also become categories of material mediation. Environments pervaded by wireless technologies and mobile communication devices – increasingly varied, progressively more interactive, powerful and integrated – have become our next nature: ambient computational networks that are so pervasive and seamless that they disappear into one’s surroundings as

314

MATERIALIZING DIGITAL FUTURES

completely as to become a new version of nature (van Meensvort, 2005). Koert van Meensvort, a transdisciplinary thinker and designer, has coined next nature as a term that represents the absolute merging of nature and culture via material mediation. By ‘covering our natural environment with a layer of language, technology and economy, we have modified our living environment and transformed it into a technological interface’ (van Meensvort, 2005). Meensvort has successfully applied the idea of nature as technological interface to pivot the design of a futures-oriented design movement called NNN (Next Nature Network). NNN is more than a social construct based on the idea of material mediation. It exists as a mutant material–semiotic entity that operates as company, community, platform, brand, trope, product, vision, satire, and the rest. Meensvort’s NNN illustrates how material mediation as design is an action of intentional self-expression, a ‘materializing’ of agency that flows in the direction of subject to object, and also how material mediation as ecosystem, on the other hand, is a form of distributed agency that is autonomous and flows in the direction of object to subject. Therefore, intentionality can be either human or programmed and operates in two main directions in situated processes of material mediation: (1) subjective and (2) objective. At a planetary scale, geoengineering as material mediation is objective since it actualizes intentionality in the design of whole new material ecologies. On an individual level, mobile screens act as our primary interfaces of technological mediation to materialize agency that is inherently subjective and that is becoming more and more intertwined with artificial intelligence. In the objective reality of planetary infrastructures, the design category of user shifts from the individual level into the level of multispecies. In parallel to it, artificial intelligence as programmed intentionality ends the monopoly of human agency in material mediation by challenging the notion of the user as an entirely subjective category. Artificial intelligence can be considered either as subjective or as objective, depending on one’s specific understanding of the reach of computational models of mind. Although this discussion goes beyond the scope of this chapter, it must be taken into consideration as we explore the role of autonomous agents in material mediation. Verbeek’s (2012) technological mediation argument infers that the relationship between design and intentionality in material mediation goes beyond functionality by placing the user in an active and not simply passive position in relation to technology. It is the user’s intention that triggers the interaction with the machine in traditional computing systems. Human–Computer Interaction (HCI) requires conscious and intentional individual action on the part of the user. However, in ambient intelligent networks, human intentionality is not always necessary to trigger interactions between humans and materials. The mere perception of the user by the machine’s sensors can trigger sequences of interaction that can be visible

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

315

or not. Another significant difference is that, in the context of pervasive intelligence, any object can mimic human intentions. The coexistence of human and artificial intentionality enables a hybrid form of technological agency that is both subjective and objective. Human intentionality is repositioned in relation to non-human intelligence, since ‘it is through nonhuman agency that we can create nuanced links between design intent, context of use and people intra-acting with technology’ (Frauenberger, 2019: 2:22). These ‘nuanced links’ are explored in Fraunberger’s theory of Entanglement HCI. Entanglement HCI is a new theoretical perspective that ‘seeks to develop an understanding of a socio-material world that moves beyond the socio-natural and cultural-material dualisms that have characterized modernity’ (Frauenberger, 2019: 4). Intentionality in HCI design has traditionally been considered as exclusive to humans, particularly through the notion of user-centric design. Human-centric approaches to HCI are being challenged in the context of emerging technologies, when new levels of entanglement between human actors, multi-species and artificial intelligences go beyond the boundaries of user-centric interaction design. These ‘fuzzy borders between humans and machines and the distribution of agency’ (Frauenberger, 2019: 22) make the distinction between them ever more difficult to establish. Material mediation becomes a double-sided mirror through which both the human and the non-human are objectified through datafication and psycho-politics (Han, 2017; Zuboff, 2019). We live within Byung-Chul Han’s transparency society (2015): the levels of exposure and accessibility of personal data is unprecedented. Each individual’s digital traces can be reconfigured into a data double, an artificial construct that represents a person based on their digital interactions and virtual traces. Data doubles abound: each transaction, click and swipe trigger new data flows that are stored, processed and categorized and is then reflected back at us in both our virtual and material environments. In many ways, due to technologies of geolocation, facial recognition, RFID tagging, mobile device tracking and so forth, our urban environments have become responsive assemblages that operate according to a hybrid intentionality comprising both human and technological agency. This hybrid intentionality emerges from the interaction between the self and its reflection via technological mediation and can be observed in responsive digital materials that exist as the superposition of code and material surfaces in interactive devices (Wiltse, 2014). In the case of mobile applications and wearables, for example, material mediation is what enables data harvesting from a user’s data double for the creation of self-portraits built out of this collection of digital traces. These traces bridge the physical realm of objects and interfaces into the subjective realm of user experience (UX). Responsive digital materials are the material–semiotic assemblage that integrates the informational layer of the technological capture and processing of user activities, and the physicality

316

MATERIALIZING DIGITAL FUTURES

of the activities themselves. This subjective–objective intentionality is found in ‘traces of activities [that] are registered … as a change in their system status that is often (but not always) made visible in the digital material (interface) itself’ (Wiltse, 2014: 165). From the perspective of design, intentional material mediation is understood as a creative language that can either consist of the objective expression of subjective intentionality, or as a dynamic field of interaction between human and non-human agents. Artificial intelligence and pervasive and planetary computation are changing the rules of this game, as they challenge the common category of user and the formal separation between subjective and objective layers of material mediation as we move into a next nature. Even though the idea of next nature is far removed from any kind of philosophical or computational ontology and only applies to the cultural surface of social and technological interfaces, it could possibly be classified as a synechist approach to material mediation according to Peircean semiotics. The idea of next nature presupposes the radical continuity of mediation as a sine qua non condition for a homogeneous habitat that is a whole made of interchangeable natural materials and artificial signs, or vice-versa.

Vital mediations: Embodied affect Even though Meensvort’s NNN are interesting interpolations of the technological dynamics of nature and culture from the standpoint of design, its frame of reference is unable to engage with the biological vitality of life. The idea of a next nature is grounded in a superficial, human-centred, often humorous approach to material mediation that does not engage with the ontological depth of vitalism. Donna Haraway’s naturecultures (Haraway, 2003) is a concept that overcomes dualities such as human and nonhuman under the umbrella of new materialism. Branching into feminism, philosophy, media and science, new materialism argues that non-human (and particularly non-biological) matter is imbued with a liveliness that can exhibit distributed agency in material–semiotic assemblages connecting human and non-human actors. New materialism understands material mediation as a post-human manifestation of embodied affect (Braidotti, 2013). Nonetheless, there is no individual agency in these assemblages. New materialism considers agency as intrinsically distributed and never as the effect of situated intentionality (Bennett, 2010). Bennett’s thing-power exemplifies the ability of objects to manifest a vital kind of agency: ‘thingpower gestures toward the strange ability of ordinary, man-made items to exceed their status as objects and to manifest traces of independence of aliveness, constituting the outside of our own experience’ (Bennett, 2010: 2). The thing-power of objects is an artificial kind of vitalism, one that replicates the biological logic of slime mould. Slime mould is a literal

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

317

manifestation of embodied affect—its jump from the invisible dimension of a virtual individuality into the visible dimension of material pattern as a collective only becomes possible due to a vital urge for survival as an organism. The dark vitalism (Woodard, 2012) of slime mould is twofold: it is vital due to its capacity for reproduction, mutation and adaptation; and dark due to its liminal existence in an invisible threshold between materiality and virtuality. Vital mediation as a concept emulates the entanglement between materiality and virtuality in a similar way. Vital mediation is directly connected to embodied–subjective experience, however even though it mediates subjective perception, life itself is not limited by human individuality or agency. The threshold between the conscious self and external reality (between subject and object) takes place through the complex dynamics of embodied affect that sustains vital mediation.

Conclusion The typology of material mediation proposed in this chapter is triadic, in alignment with Peirce’s Triadic Model (CP, 1958: 1–8). Peircean semiotics has long considered material mediation to be a part of its ontology of matter that considers physicality to be, in itself, semiotic. I have categorized this understanding of material mediation as ontological. My second category of material mediation is intentional and concerns the cultural and social praxis of technological interfaces, user experience and design. This approach relies on entanglement HCI (Frauenberger, 2019), Verbeek’s technological mediation argument (2012) and next natures and design (van Meensvort, 2005) to outline material–semiotic assemblages that are responsive digital materials (Wiltse, 2014). The third category, with reference to the work of Braidotti (2013), is vital and considers material mediation as the thingpower of embodied affect that exists in the liminal space between virtuality and materiality (Bennett, 2010; Woodard, 2012). All of these three approaches to material mediation presuppose the existence of a semiotic continuum that aligns with Peircean synechism. Semiotic continuity implies distributed agency across subject and object, matter and mind, human and non-human. Seeing technological mediation from a transdisciplinary point of view allows us to access deeper grounds of reflection in relation to its possible psychological, social and political effects. Perhaps, a better understanding of what is at stake in the many levels of these massive global technological infrastructures will inspire a clearer awareness in relation to their social implications (Zuboff, 2019) and the toll these new planetary architectures (Bratton, 2015) are exerting across different environmental layers (natural, cultural, social).

318

MATERIALIZING DIGITAL FUTURES

Notes 1 Vygotsky’s emphasis on subject-oriented mediation exemplifies the importance of language and signs as psychological tools that can mediate the relationship between the inner experience of self and external reality (Wertsch, 2007). 2 Also belonging to this approach to material mediation is Donna Haraway’s material–semiotic knot (Haraway, 1987) and Bruno Latour’s actor-network theory (Latour, 2005). 3 Collected Papers of Charles Sanders Peirce (1958), cited in the text as CP, followed by volume and paragraph numbers. C. S. Peirce (1897–1909), Letters to William James, cited in the text as EP.

References Ascott, R. (1995), ‘The Architecture of Cyberception’, in M. Toy (ed.), Architects in Cyberspace, 38–41, London: Academy Editions. Barad, K. (2007), Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning, London: Duke University Press. Bennett, J. (2010), Vibrant Matter: A Political Ecology of Things, Durham, NC: Duke University Press. Bolter, J. D. and R. Grusin (2000), Remediation: Understanding New Media, Cambridge, MA: MIT Press. Braidotti, R. (2013), The Posthuman, Cambridge, MA: Polity Press. Bratton, B. (2015), In the Stack: On Software and Sovereignty, Cambridge, MA: MIT Press. Chandra, J. D. and D. Debashis (2018), ‘QCA Based Design of Polar Encoder Circuit for Nano Communication Network’, Nano Communication Networks, 18: 82–92. D’Ariano, G. M., Chiribella, G. and Perinotti, P. (2017), Quantum Theory from First Principles: An Informational Approach, Cambridge, MA: Cambridge University Press. Dotsenko, E. (2017), ‘NBIC-Convergence as a Paradigm Platform of Sustainable Development’, E3S Web of Conferences, 21: 1. Eco, U. (1976), ‘Peirce’s Notion of Interpretant’, MLN, 91(6): 1457–72. Frauenberger, C. (2019), ‘Entanglement HCI The Next Wave?’, ACM Transactions on Computer–Human Interaction, 27: 1–27. Han, B.-C. (2015), The Transparency Society, Stanford, CA: Stanford University Press. Han, B.-C. (2017), Psychopolitics: Neoliberalism and New Technologies of Power, London: Verso Books. Haraway, D. (1987), ‘A Manifesto for Cyborgs: Science, Technology, and Socialist Feminism in the 1980s’, Australian Feminist Studies, 2:1–42. Haraway, D. (2003), The Companion Species Manifesto: Dogs, People, and Significant Otherness, Chicago, IL: Prickly Paradigm Press.

A TRIADIC TYPOLOGY OF MATERIAL MEDIATION

319

Jones, J. (2016), ‘Applications of Multi-Agent Slime Mould Computing’, International Journal of Parallel, Emergent & Distributed Systems, 31 (5): 420–49. Latour, B. (2005), Reassembling the Social – An Introduction to Actor-Network Theory, Oxford: Oxford University Press. Lee, D., J. Bonitati, G. Given, C. Hicks, N. Li, B. Lu, A. Rai, A. Sarkar, J. Watkins (2020), ‘Projected Cooling Algorithm for Quantum Computation’, Physics Letters B, 807: 1–4. Leman, M. (2007), Embodied Music Cognition and Mediation Technology, Cambridge, MA: MIT Press. McGregor, S. (2013), ‘Transdisciplinarity and Biomimicry’, Transdisciplinary Journal of Engineering & Science – TheATLAS, 40: 57–65. McLuhan, M. (1994), Understanding Media: The Extensions of Man, Cambridge, MA: MIT Press. Morais, R. L. (2014), ‘Sky High, Skin Deep: Dark Technologies of Mediation’, CTheory, 15 September:1–8. Negroponte, N. (1995), Being Digital, New York: Alfred A. Knopf. Nicolescu, B. (2012), ‘Transdisciplinarity: The Hidden Third, between the Subject and the Object’, Human and Social Studies, 1:13–28. Nicolescu, B. (2014), ‘Methodology of Transdisciplinarity’, World Futures, 70: 186–99. Parikka, J. (2015), A Geology of Media, Minneapolis: University of Minnesota Press. Peirce, C. S. (1897–1909), Letters to William James, L [R]‌224, 2:492. Peirce, C. S. (1958), Collected Papers of Charles Sanders Peirce, Vols 1–6 ed. C. Hartshorne and P. Weiss, Vols 7–8 ed. Arthur W. Burks, Cambridge, MA: Harvard University Press. Perry, D. K. (2009), Theory and Research in Mass Communication, New York: Routledge. Roese, H. (1982), ‘Some Aspects of Topographical Locations of Neolithic and Bronze Age Monuments in Wales’, Bulletin of the Board of Celtic Studies, 29 (1): 763–5. Rovelli, C. (1996), ‘Relational Quantum Mechanics’, International Journal of Theoretical Physics, 35: 16–37. Tsiatsis, V., S. Karnouskos, J. Holler, D. Boyle and C. Mulligan (2019), Internet of Things: Technologies and Applications for a New Age of Intelligence (2nd edn), Cambridge, MA: Academic Press. van Meensvort, K. (2005), ‘Exploring Next Nature’, NNN. Available online: http:// www.nextnature.net (accessed 17 May 2020). Verbeek, P. P. (2012), ‘Expanding Mediation Theory’, Foundations of Science, 17 (4): 391–5. Wertsch, J. V. (2007), ‘Mediation’, in H. Daniels, M. Cole and J. Wertsch (eds), The Cambridge Companion to Vygotsky, Cambridge, MA: Cambridge University Press. Widrow, B. and S. D. Stearns (1985), Adaptive Signal Processing, Englewood, NJ: Prentice-Hall.

320

MATERIALIZING DIGITAL FUTURES

Wiltse, H. (2014), ‘Unpacking Digital Material Mediation’, Techné: Research in Philosophy and Technology, 18 (3): 154–82. Woodard, B. (2012), Slime Dynamics, New York: Zero Books. Zuboff, S. (2019), The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power, New York: Public Affairs.

INDEX

Note: page locators in italics indicate images. AAA games 51, see also video game AARON, painting bot 142, 146 Acoustic Space 46, 50 activism: digital 85, 137; women’s anger-informed 129, see also under feminism; social media advertising campaigns, digital 99 aesthetic: machinic 153–4, 156, 158; narrative 64; universe 154–6; white American cultural 72 Aetiology 279–80 affect 269, 272–4 affective warfare 247, 269–72, 277–9, 281, 284 agency 48–9, 51–2, 113, 142, 147, 247, 265, 292, 312, 314–17 Ahmed, Sara 127, 129, 138 algorithms 77, 87, 92, 174, 193, 200, 290; cultures of 87–9, 100 Alien Phenomenology 26–7 Alibaba 83, 290; automation in warehouses of 299; Cainiao of 291 align the atmosphere 247, 259, 261, 263–4 Amazon 93, 290, 298–9; Alexa 90; Echo Dot 90 American Civil Rights 75 AM radio 53 Android 13 animation 5, 163, 166, 230 anthropomorphism 171–2 apparatuses 152, see also devices Apple 84, 198

appliances 1, 243, 252, 259; home 244, 313; smart 246, see also devices; gadgets ‘architectural space’ 10, 70, 72 Ariel and the Beast 18 Armstrong Audio Visual (AAV) 7 arrange the actions 260, 264 arrest the attention 259, 262–4 Art as Experience 18 artifacts 2, 128, 248, 306; digital media 3; environmental 199 artificial intelligence (AI) 82, 84, 87, 148, 166, 224, 285, 290, 313–16; art 86, 141, 143, 151, 159; creative 141; ethics of 101–2; hardware 143–6; human– machine relationship 143; in music market 8; systems 85, 101, 103, 296 artificial: interactive installations 174– 7; movement 172–3; neural networks 144–5, 225–7 Ash, James. 92 Atanasoski, N. 297 Atlas of Emotion 16 atmosphere 12, 16, 247, 259, 261–4 audiences 8, 10, 16, 46–9, 51–8, 69–70, 72–4, 76, 85, 116–17, 174, 176, 178, 225, 233; environment 73 audio-kinetic relationships 56 Audio-vision, Chion 47 audiovisual digital media 45–6, 51; interaction in 51

322

INDEX

augmented reality (AR) 1, 3, 9, 45, 47, 49, 112, 115, 164–6, 187–8, 190, 195–201, 214–16, 244–5; normalization of 199 authorship 65, 86, 142–3, 146, 148, 150, 153, 282 ‘automated oppression’ 296 automation 165, 173, 183, 200, 247, 289–91, 293–300; machines 152; racial stratification and 298 avatar 21, 113–14, 217, 226, 246 Babyface 176–8 BAM theatre 70 Barad, Karan 93, 310 Barbot 77n6 Barthes, Roland 150 Bateson, Gregory 274 Bausinger, H. 253 BDH Immersive 13 Beauty and the Beast 17 Beckwith, Megan 115 Being Digital, Negroponte 313 Bennett; J., thing-power of 310, 316–17 Berlant, Lauren 272, 274–7, 284; cruel optimism 275–6; Betancourt, Michael. 188–9 big data 1, 82, 84, 87, 284; agglomerations 89 biology 99, 192 The Birth of a Nation 64, 68, 70 Black: aesthetics 61, 63, 65; audience 64–5, 69, 72–4; churches 70; communities 61, 64–5, 68, 73–4, 124, 133; diasporic 66; experiences 62, 67– 8; filmmakers 65; films 64; image-making 61–2, 65; neighbourhoods 69; organizations 65; performers 64, 69; race 64; women’s bodies 136 ‘black boxes’ 63, 90, 102, 153 Black American: culture jamming 73; ghetto 73 Black bodies 10, 61–8, 70–7; aesthetic representation of 63; in America 63, 69, 72–3, 75; and Black

spectatorship 73; branding 66–7; dehumanization of 66; media representation of 69; of resistance 66; triadic approach 73, 272 Blackface caricatures 69 Black Lives Matter movement 136 Black Metropolis 70 Black Mirror 92 Blackness 62–5, 67–9, 71–5; cultural behaviours of 70; heterogeneous aesthetics of 77; Mobley on 10 Blockchain and AI convergence 245 body: movement 58, 150, 178, 259; ownership 22, 113 Bosch, Hieronymus 9, 11–13; painting ‘in the flesh’ 12 Bosch VR 13–15 boss-ware communications 95 Boston Dynamics Atlas 170 bottom-up information 22 Braidotti, Rosi, neo-vitalist theory of matter 310 Bratton, Benjamin 306 Brighenti, Andrea 188–9 Broeckmann, Andreas 153 Browne, Simone 67, 73 Bruno, Giuliana 16, 20, 217 Bujanda, H. 100 Burke, Tarana 123 Butler, Judith 125, 272 Byron, Lord 143 Caldeira, C. 99 Calvino, Italo 142 Calvo, P. 197–8 camera-based motion capture system 236 capitalism 130, 284; colonial 298; digital 187 Caprica 92 Carbine, Mary 64 Carpenter, Edmund 50 Carr, Diane 21 Cézanne, Paul 11, 13, 17; paintings 14 Chernobyl 10, 46–7, 56–8 Chesher, Brodie 116

INDEX

‘chiasm’ 11, 14–15; intertwining and 17–20; meaning of 18 Chion, Michel 45, 47, 49, 53, 55 choreographer 229, see also dancers choreography 110, 165–7, 169–70, 173–4, 177–8, 183, 223, 233, 290; practice 170, 173; process 224, 231; structures 224, 226–7, 233, 238–9 Choreotopography 116–17 cinema 7, 9, 10, 18, 46, 47, 48, 52, 53, 54, 56, 64, 65, 74, 110, 270, see also film cinematic, audiovisuals 46, 51; Blackness 61, see also Blackness cinematography 55, 58 Clinton, Hillary 282 clothes 166, 205–6, 209, 217; disembodiment of dress 210–12; re-embodiment of dress 214–16, see also fashion cloud-based service model 244 Clough, Patricia T. 99–100 clustering process 77n2 cognition: grounded 208; socially distributed 224 Cohen, Harold 142, 146 collaborative resistance 134 collective: cackling 134; rage 129–30 Collisions 10, 46, 51–4, 56–8, 110 Commonwealth Scientific and Industrial Research Organisation (CSIRO) 101 communication 9, 11, 14, 20, 91, 100, 102, 107–8, 119, 127, 134, 188, 190, 193, 309–10; between bodies 18; MerleauPonty on interweaving of 18–19; technologies 99, 249–50, 253, 255 communities 2, 65, 68, 89, 99, 101–2, 126, 129, 132, 134, 176, 190, 276, 298, 314 computational 10, 51, 63, 72, 76, 112, 180, 212, 312, 314, 316; Blackness 62; data processing 305; hackings 75; interactive 109;

323

media 129; networks 313; systems 212, 294 computer aided design (CAD) 191, 215 computer-controlled machines 191 computer-generated imagery (CGI) 20 computerization 297 computer-supported cooperative work (CSCW) 254 consciousness 134, 255, 273, 308, 311; Flusser on 158; historical 156–7; mythical 156; post-historical 157 consumers 82, 159, 170, 211, 215–16, 245, 247 consumption 72, 192, 205, 216, 265, 284, 305 Contact Tracing apps 87 Cortana 84, 94–9 Costa, Pedro 74 Costello, C. G. 93 Covid-19 107, 166, 197, 214, 306 The Crack Up 116 creativity 57, 119, 165, 181; in education 172; process 4, 7, 109, 174–5, 224; technical tinkering 72 Cripps, Thomas 64 Crow, Jim 67 Crowded House 7 cultural optics 62, 63–5, 73 culture 8, 63–5, 67, 77, 86, 91, 195, 201, 280, 289, 294, 297, 309–10, 314, 316; jamming 73, 76; of narcissism 92 The Culture of Technology, Pacey 296 ‘curatorial lives’ 87–8, 102 customers 210–11, 215 cybernetics 19, 306 cyber physical systems (CPS) 90 cyborg 25–7, 29–33, 36–40; and alien experience 9 Dadaism 142 dance 119, 166, 171, 173–5, 180, 223–6, 229, 233–5, 237–9, 261; haptics 230–1; perspectives from 114–18; re-synthesizing movement 237; technology 85, 109–10, 115, 117; with visually impaired 230

324

INDEX

dancers: contemporary 224–5, 234; human 225, 227–9, 232, 235, 239; movements 224–6, 231, 236; and performing agent 225 Dancing in Suits 235 dark vitalism 312, 317 data: biometric 92; collection 1–2, 4, 84, 87, 89, 95, 128, 256, 296; doubles 315; driven systems 109; ghosts 91–4; recorded 1, 84, 95, 98; security 81, 245; sharing 1, 87; tracking 193; visualizing of 4, 81 data-rich manufacturing process 193 Dawson, M. 101, 113 Deep Fake 91 Deep Purple (Thames) 8 Derrida, Jacques 91 Derridean ghost 91 design, speculative 10, 62–3, 72–3, 76 devices 46–7, 63–4, 87–89, 99, 169– 70, 172–3, 180, 212, 215–16, 245, see also apparatuses; data acquisition 213; digital 2, 295; haptic 206, 213; internet-enabled smart 262; locations of 257–9; Samsung Smart TV 90; smart 100, 249–50, 253, 256–7, 260–5; smart home 246–7, 249, 253, 255–6, 262–4; workplace and household 1 Dewey, John 18 digital: artifacts 2–3, 50, 199; communication 107, 244; cultures 91, 124, 129; dual use of 256; fashion 166, 205–6, 210–12, 213–14, 216–17; feminist memes 138; feminist practices 126; humanities 87; identities 4, 129, 133, see also collective rage; media 2–5, 7, 9, 45–51, 54, 57–8, 81, 124, 127, 163, 191, 193, 200, 309; meme activism 85; networks 190, 192, 201, 275, 312; performances 124, 132, 134, 137; physical 190–1; privacy 81, 88, 99, 245;

puppeteering 228–9; retail 210, 216; systems 47, 56–7, 191; technologies 1, 3–4, 10, 45–6, 50, 54, 56, 58, 71, 93, 188, 309–10, 313; traces 102, 315; visual revolution 99; witches 134–8 digital single-lens reflex cameras (DSLR) 73 digital television 1, 164 Digital Twins 224, 232 Disney classics 17 Dolby Atmos 52–3 Dosunmu, Andrew 74 doubled body 11–12, 19–20; perception of 9, 12, 15, 20–1; in virtual reality, Tilt Brush and the 20–2 dualisms 189, 315; body–mind 19, 256; cultural–material 248; embodiment 18, 21 duality 18, 115, 316; perception of 21 Duffy, Mignon 298 Duga Radar System 57 Duo Mobile 95 Dylan, Bob 7 Dyson, Frances 50 e-commerce 166, 212, 216–17; experience 166; platforms 215 economic violence 67 ‘elastic scheduling system’ 291 Elgammal, Ahmed 141, 159n2 embodiment 9, 11–12, 21, 50, 109–13, 117–19, 174, 193, 206, 254–6; affect 273–4, 281, 309, 316–17; cognition 109, 208–9, 230; dress 206, 208–10, 216; observation 165, 169–70, 177–8, 183; system 234; technological 212–14 Emergence 224–6, 234, 239 emotions 8, 12, 16, 90, 126–7, 138, 197, 206, 208, 216, 247, 272–4, 283–4 enclothed cognitive sciences approach 206 Eno, Brian 142 Entanglement HCI 315, 317

INDEX

entertainment: commercial 68; media 3 Entwistle, Joanne 209–10 environmental augmentations 198 Environmental Experience 251 Eubanks, Virginia 296 Eve of Dust 232–3, 237 ‘existential uncertainty’ 275 exposures 165, 281–2, 315 eye 12, 16, 22, 50, 126, 143, 148, 156– 9; human 146, 153, 155 Facebook 83, 87, 90–1, 132, 198, 245, 275, 281–3 Facebook Reality Labs 245 Fanon, F. 65, 67, 69, 71 fantasy play 9 fashion 205, 209; brands 210, 214–15; e-commerce 210–11; garments prototyping software systems 215; industry 212, 214–16; weeks 215 feedback 19, 51, 98, 112, 118–19, 194–5, 212–13, 215–16, 231 Fellini, Federico 246 feminism 124, 134, 136, 316; activism 124, 128; global awakening 132– 3; hashtag movements 138; humour 132; identities 130, 134; internet 134; media 125, 132; memes 123, 126, 134; rage 132, 134; remixes of patriarchal bullshit 129–32; resistance 129, 133; witch memes 85, 125, 127, 137 #FeministWitch 128 fetishism 275 film 8, 10, 16, 21, 45, 47, 49, 51–2, 54–6, 58, 64–5, 70, 74–5, 152, 171; architectural space 10, 70–3, 75–6; ‘factual’ version of 273; space 62, 65, 69–73 Finding Nemo 246 Fitbits 87, 94 Flaim Trainer 195 ‘the flesh of the world’ 11 Floyd, George 135 Flusser, Vilém 143, 150–1, 153–8 4D 9–10, 46, 54, 56–7 4DX 10, 46, 54–6, 58

325

Fox, J. 109, 113, 117–18 Frank, Adam 273 Franklin, S. 189 Fraunberger, C. 315 Frohlich, D. 254 ‘Future of Storytelling’ 17 gadgets 3, 305, see also devices Galinsky, Adam D. 208 Galloway, Alexander 155 The Garden of Earthly Delights 9, 11–13, 15 gaze 12, 52–3, 58, 71, 73, 88, 156, 200; cinematic 54 Gemeinboeck, P. 173, 235 General Electric 257 generative adversarial network (GAN) 144–8, 150–3, 155, 157–9; deepfakes by 145; imagery by 149 ‘geopsychic architexture’ 16 Gerasimov, Valerii 279 Gestalt 33, 39, 115, 151 gestures 15, 17, 77, 126–7, 148, 150–1, 153, 159, 190, 195, 231, 290, 293 Gibson, Ruth 114 Gitlin, Todd 92 Gleick, James 147 Google: Cardboard 11, 13; Home Mini 257; Nest 90; Street View 10, 46, 56–7; Tilt Brush 9, 11, 15–17, 19–22 Goya 12 Grace, Helen 92 graphical user interfaces (GUIs) 28, 30, 193 Greystone, Zoe 92 Griffith, D.W. 64, 68, 70–1 Groten, Raphaela 21 Gui, X. 292 hacker augmentations 73 Hackett, Patrick 9, 15 hacktivism 62–3, 72–3 ‘hands on throttle-and-stick’ (HOTAS) 195 haptic (touch) 2, 15–17, 46, 207, 215; cushion 230–2; ‘eye’ 16; interface 213–14; social 231, 239;

326

INDEX

systems 212–14, 232; technology 206, 216; ultrasound-based 194 Haraway, Donna 128, 187, 316 Hartmann, T. 109, 113, 117–18 hashtags 124, 128, 134, see also under separate entries hauntology 91 Heidegger, Martin 255, 263 Hewlett Packard 193 hex all fascists 135 #HexThe Patriarchy 128, 134–8 HOBO UFO (v. Chernobyl) 46–7, 56–8 Hoff, James 56–7 Hollywood 65, 68–70, 75 HoloLens 165 home: at-homeness 250, 252, 255, 260–1, 263–4; ballets 247, 251–6, 260; basic contact 252–3, 255–6, 262–3, 265; body-subject 252–3, 256, 260; environmental experiences 250–2, 256–9, 261–3; sense of everyday normalcy 246, 250, 254–6, 259–60, 263 @thehoodwitch 135–6 How We Became Posthuman 19 HP-150s 193, 195 HTC Vive 11, 15 human 89, 109, 193, 296; artists 141, 143, 151–2, 158; body 12, 145, 163, 189, 234; intentionality 314– 15; and robot relationships 170–1, 223–4, 227 Human Robot Interaction (HRI) 163, 167, 171 Human–Computer Interaction (HCI) 3, 5, 89, 93, 109, 163, 166, 171, 193, 212, 248, 314–15 humanoid robot 175, 180, 229–32, 235, 239 Humphreys, Lee 127 identity 61, 67, 84, 90–1, 95, 102, 124, 126–7, 129–30, 134, 189–90, 200, 205, 209, 245; formation, human 3 image-making 68–9, 72–5; of Black bodies 73, 76; devices 64;

Euro-centric 63; frames 71, 73, 76; substrate 65; technologies 10, 62, 73 images: as reservoir of information 155; technical 143, 151–2, 157; traditional 156–8 Imagined Narrative 165, 170, 177–8, 183 ‘Imagining a Feminist Internet’ 134 IMAX’s VR lounge 110 immaterial physicality 187–9 Immersion 15, 49–50, 52, 54–5, 108–9, 112–13, 215 immigrants 68, 298–9 Included Middle, logic of 308 Industry 4.0 224 ‘information societies’ 92 Information Technology (IT) 87, 89 information warfare 269, 279 infrastructures 2, 10, 74, 76, 245, 248, 251, 292, 294, 306 Instagram 83, 87, 132, 135–6, 281; Feminists Do Media (@aesthetic. resistance) on 128 Institute of Electrical and Electronics Engineers (IEEE) 89; Transactions on Haptics 214 intellectual property (IP) 192 interaction 3, 37, 92–3, 174–5, 190–1, 193–4, 212–14, 244, 246, 283, 285, 308, 310, 314–16; in audiovisual digital media 51; design 9, 46–51, 57–8, 306, 315; with game world 29; human and virtual 19; between networked technologies 88; with technology 16 interactive: computational and datadriven systems 109 interactivity 19, 21, 46–50, 58, 92–3, 174–5, 190–1, 193–4, 212–14, 244, 246, 283, 285, 308, 314–16 interface 50, 57, 93, 170–1, 193–5, 285, 314, 316 internet of things (IoT) 82, 89–90, 313 Internet Research Agency (IRA) 282 intertextuality 130 intra-sensorial 2

INDEX

iPhones 1, 13, 193 Irwin-Schütze, Anna 49, 53, 58 Jewison, Norman 75 jugaad 10, 72–3, 75–6 Kallinikos, J. 2 ‘Karen’ 133 Keane, Glen 17–19 Kee, Jac sm 125 Kelly, Kevin 88, 132–3 Kilteni, Konstantina 21, 113 kinaesthetics 15–17, 21, 171, 207, 224; awareness 235; empathy 167, 172, 237; experience 55, 58; intelligence 234; modality 173; sensation on-body 214 kinesphere 229 kinetics 45–6, 191 Kiss (Sanctuary Visual Entertainment) 8 Klingemann, Mario 144, 147–8, 150–2 Kolpakas, Ignas, ‘mimetic warfare’ 271 Kraut, R. 254 Laban/ Bartenieff Movement System (LBMS) 179–81; symbols from 179 Lascaux 156 Latour, Bruno 190 Levin, Kasper 16 Life Cycle 84, 94 light-emitting diode (LED) 10, 75–6 The Little Mermaid 17 Little River Band (Capital Records) 8 Lorenz, Edward 147; ‘strange attractor’ 148 ‘ludus play’ 9 Lupton, Deborah 88, 201 Lyseggen, Jørn 82 Macherey, P. 150 machine 8, 17, 28, 88–9, 91, 95, 98, 100, 143, 151, 153, 170, 180, 213, 238, 285, 291–2, 294, 297, 314; artist 141; learning 28, 85, 93, 143, 224, 228, 233, 237–9, 294; motion capture in 237–8

327

A Machine 175–8 Madonna 7 Magic Leap 165 The Magic Resistance 130–1 Mancila, I. 93 Martelli, Bruno 114 Martin, Trayvon 71 Marvel Comics 54 mass communications 309–10 Massumi, Brian 272–3, 276 materialism 310–11, 316 materiality 9, 16, 85, 125, 190, 201, 272, 310, 312, 317 material mediation 305–6, 309–10, 312–17; transdisciplinary categories of 248, 307–8 McCartney, Paul 7 McGregor, Sue 308 McLuhan, Marshall 46, 50, 56, 156, 309 McQueen, Steve 70 mechanism 76, 97, 245; valuegenerating 298 mechanoreceptors 207 media: intelligence 82; interactive 45, 176, 270, 275; studies 64, 254; technologies 156, 251 media-saturated home environment 253 media spaces: digital 123, 127; social 126–8 mediation 248, 310, 311, 312, 313, 314—7 mediates 192, 209, 312, 317 Mediatization 2 Meensvort, Koert van 314 Meissner’s corpuscles 207 Melamed, Jodi 298 Meltwater 82–3; Trump on 83 memes 85, 123, 125–30, 133–4 Memories of Passersby I by Klingemann 86, 143–4, 146–8, 150–3, 155–9 Mercer, Robert 278, 285n1 Merkel disks 207 Merleau-Ponty, Maurice 11–18, 209, 252, 255; on Cézanne 17; chiasm of 19; Levin on 16; on

328

mirror experiment 22; on seer-seen doubling 20; SheetsJohnstone on 12, 16; virtual body of 14 MeToo movement 123–4 Micheaux, Oscar 65; Race Films 64 Micheaux Film Corporation 64 Microsoft 82, 95–7, 165; Outlook 94 Middle Passage 66 Millward, Joe 244 ‘mindbody,’ Hayles on 19 Mindell, David 292 Mirror World 200 misogyny 124; digital resistance against 123 mixed reality (MR) 9, 49, 112, 115, 165, 206, 214, 216, 244 mixed reality (XR) 45 Modular VR System 216 Mohapatra, Shaz 82 Moore, Jason 298 Moores, Shaun 249–50, 253, 256 Morgan, Nyarri Nyarri 51–3 Morrison, Aimée 126 Mother of George, Dosunmu 74 MotionBank project 226 Motion capture technology 236 motion data 228, 238 movement: analysis 179–81; human 174, 179, 227, 229; individualization of 4, 81 multiple sensations 14, 207 multisensory experiences 10, 45–7, 58, 215; affective 10, 46; digital 46, 50, 58, 214 Murray, Janet 50–1, 56 Musk, Elon 290 MyAnalytics 84, 94–9 nanocommunication 305, 309 nano-info-bio-cogno (NBIC) 311–12 NAO hardware with movement design 180 National Assets Management Agency (NAMA) 199 National Bailout (nationalbailout. org) 135 navigation 9, 49, 51, 57

INDEX

Negroes 65, 71, 77n5 Negroponte, Nicholas 313 Nelson, Amy 164 neo-communicative processes 91, 103 network: media 126, 190; of networks 94, 100 neural network (NN) 224–5, 229, 238 New Realities in Audio, IrwinSchütze 49 New World 63, 66–7 Next Nature Network (NNN) 314, 316 Nichomachean Ethics, Aristotle 272 Nicolescu, B. 306–8 NikeiD VR STUDIO 216 ‘No Justice’ No Peace’ 136 Nomadic (www.blurtheline.com) 216 non-intrusive methodology 95 Nourse, Alan E. 100 object-oriented: mediation 248, 306, 309; programming 194 Oculus Rift 11, 15, 21 Office 365 95 online personalization 211 onscreen space, offscreen response to 16 otherness 65, 66–9 Oulipo (Ouvroir de littérature potentielle) 142 Our Robots, Ourselves 292 Pacey, Arnold 296 Pacinian corpuscles 207 ‘paidia play’ 9 painting 11–14, 17–18, 146, 151, 158 Pallasmaa, Juhani 12, 18, 159 PAN experimental music 56 pathogens 280–1 patriarchy 128 Paxton, Steve 233 Peirce, Charles Sanders 310–13 Peircean: semiotics 310–12, 316–17; synechism 317; Triadic Model 317 people analytics 84, 95, 98 Performance of ‘A Machine’ 176 performative assemblies 125–7, 137–8 performative body mapping (PBM) 235–6

INDEX

personal data 1, 4, 81, 84, 88, 91, 98, 315 Petersen, Christine 54 Petit, Philippe 111 Phénoménologie de la Perception, Merleau-Ponty 209 phenomenology 13, 16, 25–31, 33, 40, 209, 256 Phenomenology of Perception, Merleau-Ponty 13 photographers 77n3, 152, 159, 176 physical digitality 166, 187–9, 191–3, 195–201 physicalities 166, 201, 228, 236, 315, 317; of digital life 187 Physical Thinking Prototypes (PTPs) 234 Pico micro-controller 76 Pinoke 227, 229 planetary computation 248, 305, 313, 316 PlayStation 5 194 PlaystationVR 11 political warfare 269–70, 278 Pollock, Jackson 142 Popat, Sita 114, 118 Portier, Sydney 75 Portrait d’Edmond de Belamy 142, 152, 159n3; in Christie’s gallery 141–3, 145 post 9, 61–3, 83, 86, 91–2, 129–30, 134–7, 141, 143, 156–9, 188, 192, 194, 201, 292 (post)human 143, 159, 188, 194, 201, 234, 316 post-structuralist theory 272 practice-based research 2.0 109–10 The Primacy of Movement 16 Primakov, Yevgeny 279 propaganda 269, 278 Pulido, Martin 13 Putin, Vladimir 279 Putnam, T. 250 PVA 191 QSers 92 quantified: media 4, 81, 102; self 81, 88

329

Queneau, Raymond 142 race films/movies 64, 68, 70, 77n1 racial: challenges 68; experiences 62; population 70 radio-frequency identification (RFID) 90; tagging 315 Raja, V. 197–8 Ratan, R. 113 ready-to-hand 254–6, 258–60, 263, 265 realism 20, 54, 181, 284, 312 reality 307 Reigl, Alois 16 reinforcement learning 238 Reiss, D. 254 relational dynamics 248, 306 Relph, Edward 253; ‘existential insideness’ 252 Renee Geyer Band (RCA Victor) 8 resistance 61, 66, 85, 124–6, 138, 200, 264; against discrimination 67 re-territorialization 126 RGB LEDs 10, 76, see also lightemitting diode (LED) Ribera, José de 12 Robinson, Kim Stanley 291 robotics 5, 163, 171, 173, 181, 183, 232; computer objects 194; creative 223; platform, pipe cleaner 182; vacuum 170 Robotics, Automation and Dance (RAD) 173, 175 robot learning 229; by tactile demonstration 236–7; by visual 236 Robot Margaret 230 robots 100, 167, 170, 173, 178–81, 226, 230, 232, 291–2; dance, training and AI 235; design and adult learners 182; environment design 180; hardware design 180; humanlike 172; as literary devices 169, see also robot learning; morphology 228, 236; movement 165–6, 223–4, 227, 229, 231, 233–6, 238; social 172, 235;

330

INDEX

software design 180; training, dancers as agent for 234–7 ROS-based script 174 Rose, Ern 8, 199 Ruffini endings 207 Salem Witch Trials 124 Saunders, R. 173, 235 Sawyer robot 232–3 Scannell, Paddy 249, 255 Scanner Sombre 9, 26–7, 31–40 Schütze, Stephan 49, 53, 57–8 science fiction play 169 Scott, Astrid 164 screen: contemporary 9; interactive 213; media revolution 2 Seamon, David 250–1, 254, 256, 260– 2, rest dimension of 252–3; Sedgwick, Eve Kossofsky 273–4, 276–7 self-identification 20, 132 self-location 20–1, 113 self-tracking 1, 99 sensation 12, 16, 20, 111, 113–14, 118, 194, 207, 213, 271–2 sense-making 17–18 sense of embodiment (SoE) 21, 113– 14, 118 senses 2–3, 9, 14, 22, 47, 54, 58, 109, 113, 156, 188, 280, 305; of normalcy 250, 253, 255–6, 259, 262–3; of touch 16, 166, 206–7, 212 Sensing Sims 25, 26, 27, 28, 29, 30, 31, 33, 35, 37, 39, 40, 41 sensors, ingestible 1 sensory stimuli 22, 206 sexual harassment 123–4; see also MeToo Movement shared movement 167, 223 Sheets-Johnstone, Maxine 12, 16– 18, 114 Shelley, Percy Bysshe 143 Shimizu, Sakura 117 Shirley cards 62 Shopify.com 211 sight 4, 7, 16, 69, 166, 188–90, 196– 200, 261

signal 35, 39, 47, 306 Signal 87 Silverstone, R. 249, 254, 262, 264 simulation 50, 55, 85, 108, 110, 170, 211, 213, 229, 244 Skillman, Drew 9, 15 Skovli, Daniel 117 Slavery and Social Death 67 Slime Dynamics 312 small data 1, 4, 84–5, 97, 99 smart: home 3, 100, 243, 246–7, 249, 253, 255–6, 260, 262–4; lightbulb 257, 261, 263, 265; speakers 257–60, 263–5 smartgrids 3 social: cartographies 87, 93; communication 206, see also mass communications; social media sociality 247, 252, 261, 276 socialization 251, 254; agents of 91 social media 82–4, 88–9, 92, 100, 127, 283; #inktober challenge on 134; activism 85, 123; applications 1, 281–2; internetbased 278, 280; participants 126, 134; sites 99, 281 social networks 2, 99–100, 271, 278, 283 sociocultural 4, 61, 70, 74, 76, 81, 85, 87, 93–4, 99, 125, 127, 129, 265, 297; relationships 2; technological developments 2 socio-material world 248, 315 software as service 82 software-surveillance 98 somatics 173–4 somatosensory system 206–7, 214 Son, Reuben 56 sonic 46–7, 50, 57–8, 233; information 55 Sonnenschein, David 48–9 Sontag, Susan 72 Sony Pictures’ VR recreation 111 sound 2, 10, 54, 56–8, 69, 74, 190, 200, 224, 233, 261; in audiovisual digital media 45; in Collisions 52; design 9, 47–52,

INDEX

55, 57; dissemination 4, 7; mixing 53; studies 9, 46 Sound Design, Sonnenschein 48 Sounding New Media, Dyson 50 space 9–22, 46, 50, 52–3, 57–8, 62, 64–5, 70–6, 111–16, 176–80, 199–201, 228–30, 246–51, 258, 260–1; dark 12, 15–16; design 63; e-working 95; isolated creative 141; multidimensional 151; physical 3, 84, 108, 114, 231, 252, 254; transdisciplinary 309 spatiality 76, 108, 190, 200; movement structures 229–30 spatio-temporal: intelligence 224; repetition 254, 260; sharing 247, 252 spectator participation 54 spectre of self 84, 91, 94–5, 102 speculative 61, 75–6; art 63; design 10, 62–3, 72–3, 75–6; imagining 61–2, 73; realism 312 Spigel, Lynn 249–52, 264–5 Split Enz (Mushroom) 8 Springfield, Rick (Wizard Records) 8 Steichen, Edward 159n2 Straus, Erwin, ‘presentic unoriented space’ 261 structural inequalities 85, 93 subject–object interaction 248, 306–8 subject-oriented mediation 248, 306, 309 symbiotic robots 239, see also robots synchresis 55, 58 synechism 311 synthetic images 145–6; see also Memories of Passerby I tactile robot 231–2, see also robots TanvasTouch® (www.tanvas.co) 213 Tarde, Gabriel 277 Tarzan 17 Taylor, Diana 127 technology/technological 87, 89, 215, 292, 296; climate, global 1, 87; convergence 206, 214–16, 248, 306; mediation 21,

331

305, 310, 314–15, 317; networks 248, 306; relations 87, 99; transformations 5, 163 telecommunications 81, 243, 306, 309 TeslaSuit (www.teslasuit.io) 213–14 thin film transistor (TFT) 76 3D 14, 20, 53–4, 58, 76, 107–8, 115–17, 187, 195, 211, 214–16, 226, 244 3D printing (3DP) 166, 187, 189–92, 194, 196; ‘immaterial’ supply chain of 193 360°: audiovisual 47; aural space 50; images 53; movement 51; perception 229; sound mix 53; video 46; virtual audio 51 Tik Tok 83, 87, 198 time–space routines 247, 251–2, 260 ‘Time To Compile’ 175, 177–8 Tokyo International Forum 232 Tomkins, Silvan 274, 276–7 touch 4, 108, 191–2, see also haptic; as complex sense 163; explorative 207–8; perceptions of 189, 196; physical digitality of 193–5; sensations 206, 208; sensors 207 touchable systems 213 touch-centric user interfaces 193 touch-interactive media 191, 194 touchscreens: interactive 206; responsive 193; 2D 194 tracking 95 transdisciplinary 102, 248, 309–10, 314, 317; knowledge 308; methodology 306–7 transparency 51, 101–2 Trump, Donald 83, 130–2, 294 Tuan, Yi-Fu 250, 254, 258 twitter 83, 87, 132, 281–2 2D: compositions 152; image 13, 190, 194 Unfinished Swan 9, 26, 31, 32, 33, 34, 35, 36, 37, 39, 40 Unity game engine 225 Unsound, Polish festival 57 Urbana 176, 178

332

INDEX

user experience 164, 315, 317 User Interface (UI) 57, 193, 195, 255 utopia, inhuman 158–9 U2 7 Vaudeville or Chitlin’ Circuit entertainment 74 V-commerce 215–16 Velasquez, Diego 12 Venom 47, 55–6, 57; in 4DX 10, 46, 54, 58 Verbeek, Peter-Paul 314, 317 vibrations 207, 213 videogame 9, 18–20, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 39, 40 videography 63 virtual: audio 50; body 9, 11, 13–15, 18–20, 22, 113, 224; commerce 217; environments 19, 21, 108, 113–14, 117, 227, 232, 238–9, 244; learning 238; spaces 9, 12–13, 15–19, 21–2, 84, 107–8, 114–16, 164; technology systems 217 ‘virtual computing material’ 312 virtual reality (VR) 1, 16, 18, 45, 48–50, 53, 85, 212– 15, 224, 244; Bosch 13; cinema 56; in Collisions 54; consumer systems 216; example 15; experience 9, 11, 13, 17, 21–2, 46, 51, 110–12, 114, 117–19, 165; experiment 20; headsets and headphones 47, 51, 110; Lounge 110–12; for physical presence 108; real-time rendering 165; space 15, 118; Syrian migrant camp 72; technologies 19, 53, 215, 244; 3D depth in 13; users 113, 119; videogame 20; world 19, 21, 233 visibilities 10, 20, 76, 89, 166, 188, 198–9, 201; hierarchisation of 189 The Visible and the Invisible 14, 18 visual: Blackness 10, 75–6; image 4, 92; media 1, 5, 113; networking 2

visualization 146, 155, 211, 225–6, 237–8, 305 Vitalina Varela 74 Vive VR controller 233 Volumetric Black 61–3, 65, 67, 69, 71, 73, 75–7 The Walk by Sony 110–11, 114– 15, 118 Wallworth, Lynette 51–4 War 270, 278–80 wearables 206, 315 WeArt (www.weart.it) 213–14 Weiner, Norbert 274 Wertheimer, Max 13 Westworld 92 white: audiences 66; sensibilities 67, 74; visuality 72 White Island 114, 118 Wilson, Woodrow 70–1 witch culture 123–6, 129–30, 132, 134–6 Witches, we need you 131 witch hunt 123–4, 130, 132, 134 witch memes 127, 129, 134, see also memes #WitchesUnite 128 Within Our Gates 64 Wolf, Gary 88 Women Inspired to Tell Their Collective History, see Women’s International Terrorist Conspiracy from Hell (W.I.T.C.H.) Women’s International Terrorist Conspiracy from Hell (W.I.T.C.H.) 129–30 women: rage 129–32; resistance 129 Work 98, 298 Xiao, D. 99 Youtube 83, 87, 281 Zimmerman, George 71 Zuckerberg, Mark 91

333

334