Bloomsbury Semiotics Volume 4: Semiotic Movements 9781350139404, 9781350139435, 9781350139428

Table of contents :
Cover
Contents
List of Figures
List of Tables
List of Contributors
Acknowledgements
List of Abbreviations
Introduction Paul Cobley
1 Communication Theory and Semiotics Richard L. Lanigan
2 Media/Culture Studies and Semiotics Sophia Melanson Ricciardone and Marcel Danesi
3 Digital Humanities and Semiotics Alin Olteanu and Arianna Ciula
4 Systems Theory and Semiotics Ricardo Gudwin and João Queiroz
5 Phenomenology and Semiotics Peer F. Bundgaard
6 Hermeneutics and Semiotics Ronald C. Arnett and Susan Mancino
7 Translation Studies and Semiotics Evangelos Kourdis and Ritva Hartama-Heinonen
8 Pragmatics and Semiotics Per Aage Brandt
9 Gesture Studies and Semiotics Irene Mittelberg and Jennifer Hinnell
10 Multimodality and Semiotics David Machin and Ariel Chen
11 Discourse Analysis and Semiotics Kay L. O’Halloran and Sabine Tan
12 Integrational Linguistics and Semiotics Adrian Pablé
13 Cognitive Linguistics and Semiotics Jordan Zlatev and Möttönen Tapani
14 Cognitive Science and Semiotics Göran Sonesson
Index

Citation preview

BLOOMSBURY SEMIOTICS VOLUME 4

Bloomsbury Semiotics General Editor: Jamin Pelkey Volume 1: History and Semiosis Edited by Jamin Pelkey Volume 2: Semiotics in the Natural and Technical Sciences Edited by Jamin Pelkey and Stéphanie Walsh Matthews Volume 3: Semiotics in the Arts and Social Sciences Edited by Jamin Pelkey, Susan Petrilli and Sophia Melanson Ricciardone Volume 4: Semiotic Movements Edited by Jamin Pelkey and Paul Cobley

BLOOMSBURY SEMIOTICS

SEMIOTIC MOVEMENTS VOLUME 4

Edited by Jamin Pelkey and Paul Cobley

BLOOMSBURY ACADEMIC Bloomsbury Publishing Plc 50 Bedford Square, London, WC1B 3DP, UK 1385 Broadway, New York, NY 10018, USA 29 Earlsfort Terrace, Dublin 2, Ireland BLOOMSBURY, BLOOMSBURY ACADEMIC and the Diana logo are trademarks of Bloomsbury Publishing Plc First published in Great Britain 2022 Copyright © Bloomsbury Publishing Plc, 2022 Jamin Pelkey and Paul Cobley have asserted their right under the Copyright, Designs and Patents Act, 1988, to be identified as Editors of this work. For legal purposes the Acknowledgements on p. xvii constitute an extension of this copyright page. Cover design: Tjaša Krivec Cover illustration by Rebecca Heselton All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage or retrieval system, without prior permission in writing from the publishers. Bloomsbury Publishing Plc does not have any control over, or responsibility for, any third-party websites referred to or in this book. All internet addresses given in this book were correct at the time of going to press. The author and publisher regret any inconvenience caused if addresses have changed or sites have ceased to exist, but can accept no responsibility for any such changes. A catalogue record for this book is available from the British Library. A catalog record for this book is available from the Library of Congress. ISBN: HB: 978-1-3501-3940-4 ePDF: 978-1-3501-3942-8 eBook: 978-1-3501-3941-1 Set: 978-1-3501-3944-2 Series: Bloomsbury Semiotics Typeset by Integra Software Services Pvt. Ltd. To find out more about our authors and books visit www.bloomsbury.com and sign up for our newsletters.­

CONTENTS

L ist

of

F igures

vii

L ist

of

T ables

x

L ist

of

C ontributors

A cknowledgements L ist

of

A bbreviations

Introduction

xi xvii xviii 1

Paul Cobley

1 Communication Theory and Semiotics

15

Richard L. Lanigan 2 Media/Culture Studies and Semiotics Sophia Melanson Ricciardone and Marcel Danesi

47

3 Digital Humanities and Semiotics Alin Olteanu and Arianna Ciula

67

4 Systems Theory and Semiotics Ricardo Gudwin and João Queiroz

87

5 Phenomenology and Semiotics Peer F. Bundgaard

103

6 Hermeneutics and Semiotics Ronald C. Arnett and Susan Mancino

123

7 Translation Studies and Semiotics Evangelos Kourdis and Ritva Hartama-Heinonen

143

8 Pragmatics and Semiotics Per Aage Brandt

163

9 Gesture Studies and Semiotics Irene Mittelberg and Jennifer Hinnell

183

10 Multimodality and Semiotics David Machin and Ariel Chen

215

11 Discourse Analysis and Semiotics Kay L. O’Halloran and Sabine Tan

231

vi

CONTENTS

12 Integrational Linguistics and Semiotics Adrian Pablé

253

13 Cognitive Linguistics and Semiotics Jordan Zlatev and Möttönen Tapani

269

14 Cognitive Science and Semiotics Göran Sonesson

293

I ndex

313

LIST OF FIGURES

1.1

Ruesch and Bateson discourse model of communicology

18

1.2

Category description of Ruesch and Bateson model

19

1.3

Discourse communicology model

20

1.4

Complexity description of semiotic systems

21

1.5

Etymology of C. S. Peirce logic typology

22

1.6

Scholastic trivium discourse model

24

1.7

Roman Jakobson communicology theory and applied model

26

1.8

Jakobson communicology process model

28

1.9

Jakobson chiasm matrix model

29

1.10 Semiotic phenomenology matrix model

30

1.11 Communicology matrix of elements and functions

31

1.12 Matrix of the logic and semiotic squares

34

1.13 Chiasm process matrix for apperception and apposition

35

1.14 Trope logic in rhetoric and semiotic

36

1.15 Communicology matrix of intelligibility and relevance

38

1.16 Discourse trope model of communicology

39

2.1

Two-step flow model

54

8.1

The complex functions of language

165

8.2

The rhetoric of image and text

168

­8.3

The semiotic constitution of power through enunciation

170

8.4

A sociogram example

171

8.5

The complex semiosis of music

172

8.6a

The blending model

174

viii

LIST OF FIGURES

8.6b Metaphor

175

8.6c

Metonymy

175

8.7a

The schema: ‘the road not taken’

176

8.7b Irony

177

9.1

A seventeenth-century illustration of a gesture signalling antithesis

184

9.2

Thumbs-up emblem signalling approval

189

9.3

Pointing gesture (‘there is’) and metaphoric container gesture (‘the main verb’)

191

9.4

Painting: P. Klee, Dance of a Mourning Child 1922

197

9.5

Motion-capture gesture diagram of travel itinerary: Stockholm, Prague, Kiev, etc.

198

Metonymy: contiguity relations between hands and the virtual object they seem to be holding. Metaphor: Imaginary object represents the abstract entity (‘subcategory’)

200

Embodied contrast: On the one hand/on the other hand, bilateral point sequence

203

9.6

9.7

10.1 UFIT protein shake and Slimfast protein shake

216

10.2 SCI-MX protein

225

10.3 Bounce protein energy ball

226

11.1 Multimodal analysis of Just Terror movie poster

236

11.2 Resemiotizations and intertextuality of Just Terror tactics and knife attacks in Dabiq and Rumiyah magazines

240

11.3a Just Terror article: Image and text

243

11.3b Language analysis for Just Terror article

244

11.3c Image analysis for Just Terror article

245

11.4 K-Modes cluster visualization

245

11.5 Clusters with image categories, Watson categories and website taxonomy and tonality

246

13.1 Meaning as construal, with some of the relevant structures

272

13.2 Diagrammatic iconicity between embodied and linguistic distance in Paamese possessive

277

LIST OF FIGURES

ix

13.3 Greece Flag ≡ Toilet paper, leading to the metaphorical interpretation GREECE IS SULLIED, given appropriate knowledge on the Sedimented and Situated levels

281

13.4 The Motivation & Sedimentation Model (MSM) with the upward arrows signifying motivation, and downward arrow sedimentation

282

LIST OF TABLES

11.1 Semiotic concepts for MDA

234

11.2 Interpretations of Just Terror

237

11.3 Bonding icons in violent extremist terrorist communication

239

11.4 MDA framework and IBM Watson and Clarifai algorithms

247

13.1 The sign systems of Language, Gesture and Depiction, with some of their properties

285

14.1 The relations between principles, grounds and signs in the present interpretation of Peirce

302

14.2 ‘The semiotic hierarchy’: Five types of selves, worlds and significations

304

14.3 The distinction of ipseity, dialogicity and neutrality in the phenomena accessed and the modes of access

307

LIST OF CONTRIBUTORS

Ronald C. Arnett is Chair and Professor in the Department of Communication and Rhetorical Studies, the Patricia Doherty Yoder and Ronald Wolfe Endowed Chair in Communication Ethics, and the Henry Koren, C.S.Sp., Endowed Chair for Scholarly Excellence (2010–15) at Duquesne University, Pittsburgh, Pennsylvania, USA. He has received eight book awards, co-edited seven books and authored/co-authored twelve books, most recently Communication Ethics and Tenacious Hope: Contemporary Implications of the Scottish Enlightenment (2022, Southern Illinois University Press). He was named a Distinguished Scholar by the National Communication Association and is currently President of the Semiotic Society of America. Per Aage Brandt is Full Professor in the Department of Cognitive Science at Case Western Reserve University, Cleveland, Ohio, USA. He is the author of a dozen books and more than 150 published papers on cognitive and semiotic theory of language, grammar, aesthetics, art and music. As a scholar trained in romance philology (French and Spanish), he has worked his way through structural linguistics and structural semantics, and elaborated a series of models – in particular those related to the technical and formal representations of textual phenomena such as enunciation, diegesis and modal schematisms – for describing patterns of meaning in the framework of a discourse-oriented (Greimas) and later a formalized phenomenological (Thom, Petitot) and cognitively (Talmy) oriented semiotics. In 2002, he was awarded the Grand Prix de Philosophie by l’Académie française and was made Officier de l’Ordre des Arts et des Lettres by the French Ministry of Culture. Peer F. Bundgaard is Associate Professor in the Center for Semiotics, School of Communication and Culture at Aarhus University, Denmark. He holds a PhD in general semiotics. He is editor-in-chief of Cognitive Semiotics. His research interests are cognitive linguistics and phenomenology as well as the cognitive semiotics of literary and visual art and aesthetic experience. He has published articles in Synthese, Phenomenology and the Cognitive Sciences and Semiotica. He has edited Investigations into the Phenomenology and the Ontology of the Work of Art, Springer, open access (with Frederik Stjernfelt), and Semiotics (4 vols), Routledge (with Frederik Stjernfelt). Ariel Chen (PhD Cardiff University) is a researcher of media and communication studies at Örebro University. Her research interests include social semiotics, visual communication, food and health culture, and global media. She is currently working on a research project entitled “Communication of ‘Good’ Foods and Healthy Lifestyles”. Using Multimodal Critical Discourse Analysis, she examines the way the discourse of ‘good’ foods and healthy lifestyles becomes colonized and shaped by commercial interests. Her recent articles can be seen in Critical Discourse Studies; Discourse, Context and Media; and Food, Culture and Society.

xii

LIST OF CONTRIBUTORS

Arianna Ciula is Deputy Director and Senior Analyst in the King’s Digital Lab at King’s College London, UK. She has broad experience in digital humanities research and teaching, research management and digital research infrastructures. She holds a PhD in manuscript and book studies. Her personal research interests focus on the modelling of scholarly digital resources related to primary sources. She lectured and published on humanities computing, in particular on digital manuscript studies and editing; she has organized conferences and workshops in digital humanities and is an active member of its international community. ­ aul Cobley is Full Professor and Deputy Dean (Research and Knowledge Exchange) P in the Faculty of Arts and Creative Industries, Middlesex University, London, UK. He is the author of a number of books, including Cultural Implications of Semiotics (2016) and Narrative, 2nd edn (2014). He is co-series editor (with Kalevi Kull) of Biosemiotics, Communication and Cognition (de Gruyter Mouton) and co-editor (with Peter J. Schulz) of the multi-volume Handbooks of Communication Sciences (de Gruyter) and (with David Machin) the journal Social Semiotics. Among his edited volumes are The Routledge Companion to Semiotics (2009), Theories and Models of Communication (2013, with Peter J. Schulz), Semiotics and Its Masters vol. 1 (2017, with Kristian Bankov), Realism for the 21st Century: A John Deely Reader (2009) and The Communication Theory Reader (1996). He is the 9th Thomas A. Sebeok Fellow of the Semiotic Society of America, President of the International Association for Semiotic Studies (elected 2014) and secretary (since 2012) of the International Society for Biosemiotic Studies. Marcel Danesi is Professor Emeritus in the Department of Anthropology and ex-director of the Program in Semiotics at Victoria College, University of Toronto, Canada. He was editor of Semiotica from 2004 to 2019. Among his recent works are Semiotics of Emoji (2016), Memes and the Future of Popular Culture (2019), Semiotics of Love (2019) and Linguistic Relativity Today (2021). He is also a co-founder and currently co-director of the Cognitive Science Network of the Fields Institute for Research in Mathematical Sciences. Ricardo Gudwin is Associate Professor at DCA-FEEC-UNICAMP, the School of Electrical and Computer Engineering, University of Campinas, Brazil. He received a BS degree in electrical engineering in 1989, an MS degree in electrical engineering in 1992 and a PhD in electrical engineering in 1996, all of them from UNICAMP. His current research interests include the study of cognitive architectures, cognitive systems, intelligence and intelligent systems, intelligent agents, (Peircean) semiotics and computational semiotics. He co-edited the books Artificial Cognition Systems (2006), Semiotics and Intelligent Systems Development (2006), Computação, Cognição, Semiose (2007) and Systems, Self-Organisation and Information – An Interdisciplinary Perspective (2018). He was the editor-in-chief of the journal Controle & Automação, published by the SBA, the Brazilian Society for Automation, from September 2004 to December 2008, and the vice-president of SBCC, the Brazilian Society for Cognitive Science, from 2013 to 2017. Ritva Hartama-Heinonen is Senior Lecturer in Swedish Translation and Interpreting Studies in the Department of Finnish, Finno-Ugrian and Scandinavian Studies at the

LIST OF CONTRIBUTORS

xiii

University of Helsinki, Finland. She also holds the title of Docent in Translation Studies at the same university. After finishing her doctoral dissertation Abductive Translation Studies: The Art of Marshalling Signs (2008), a combination of translation studies and Peirce’s semeiotic, she has continued to contribute to the semiotics of translation. In addition, she has published widely on translation-theoretical issues, for instance, by developing the notion of intracultural translation. Her recent research interests include intralingual and interlingual translation into Easy Language. Jennifer Hinnell is a Postdoctoral Research Fellow at the University of British Columbia, Vancouver, Canada. As a cognitive linguist, her research applies corpus, motion capture and experimental methods to the investigation of co-speech gestures. She has (co-)published journal articles and book chapters in Language and Cognition, Cognitive Linguistics, The American Journal of Semiotics and Frontiers in Psychology  –  Cognition, among other venues, and co-edited Language and the Creative Mind (2013). Jennifer has been an invited speaker in the United States, Asia and Europe. Awarded the Governor General’s Gold Medal for her research in 2020, Jennifer holds an MA in linguistics from Simon Fraser University and a PhD in linguistics from the University of Alberta. She is a member of the international collaborative multimodality lab ‘Red Hen’ and Emerging Scholars Representative of the International Cognitive Linguistics Association. Evangelos Kourdis is Professor of Translation Semiotics in the Department of French Language and Literature, Faculty of Philosophy, Aristotle University of Thessaloniki. His main research interests include translation and semiotic theory, intersemiosis, semiotic landscape, language ideology, social dialectology, cultural adaptation and cultural communication. He is Director of the Joint Master Semiotics, Culture and Communication, Aristotle University of Thessaloniki, and Director of the Laboratory of Semiotics (AUTH SemioLab) of the Faculty of Philosophy, Aristotle University of Thessaloniki. He is also the National Representative of Greece at the International Association for Semiotic Studies and Vice-President of the Hellenic Semiotic Society. Richard L. Lanigan is Executive Director and Laureate Fellow, International Communicology Institute, Washington, DC, USA, and University Distinguished Scholar and Professor of Communicology (Emeritus), School of Communication, Southern Illinois University. He is Past President, Semiotic Society of America; Editor, The American Journal of Semiotics (ten years); Past Vice-President, International Association for Semiotic Studies; Senior Fulbright Fellow (China 1996, Canada 2007); Fellow of the International Academy for Intercultural Research; Fellow, Polish Academy of Science (Philology); ‘International Scholar Award in Philosophy of Communication’, Duquesne University, USA (2011); elected member, American Philosophical Association; founding member, Philosophy of Communication Division, National Communication Association (USA) and International Communication Association. Festschrift: Communicology for the Human Sciences: Lanigan and the Philosophy of Communication, ed. A. R. Smith, I. E. Catt and I. E. Klyukanov (New York: Peter Lang, 2018). PDF publications: Academia. edu/Richard L. Lanigan. David Machin is Full Professor in the School of International Studies at Zhejiang University, China. He is author of a number of books in the field of critical discourse

xiv

LIST OF CONTRIBUTORS

studies, such as Doing Visual Analysis (Sage, 2018), Introduction to Multimodal Analysis (Bloomsbury, 2020) and Visual Journalism (Palgrave, 2015). He has published many papers in leading journals in the field. He is co-editor of the Taylor & Francis journal Social Semiotics and the Bloomsbury book series Advanced in Critical Discourse Studies. Susan Mancino is Assistant Professor in the Department of Communication Studies, Dance, and Theatre at Saint Mary’s College, Notre Dame, IN, USA. She is the co-editor of Communication and Learning in an Age of Digital Transformation and An Encyclopedia of Communication Ethics: Goods in Contention. Additionally, she is the author/co-author of ten articles and nine book chapters. Her works have appeared in Versus, The Atlantic Journal of Communication, Review of Communication and Empedocles: European Journal for the Philosophy of Communication. Her research interests include semiotics, public memory, religious communication, philosophy of communication and communication ethics. S­ ophia Melanson Ricciardone is a PhD candidate (ABD) with the joint programme of Communication and Culture with York and Toronto Metropolitan Universities in Toronto, Ontario, Canada. She is a PhD research assistant with Toronto Metropolitan University’s Meaning Lab, which is dedicated to research into various facets of language and culture and cognition. Broadly, her work examines the ways in which we offload facets of cognition to digital technology, which invariably affects how we collectively use language, how we reason through complex ideas about the world and how patterns of thought (mental schemas) are altered in the process. She is currently working on a research project empirically evaluating the impact of Twitter-bots on intersubjective discourse. More specifically, she is exploring whether and to what extent bot-generated contributions to political Twitter discourse stimulate the circulation of affectively charged content and what consequences such arrangements impart on minds and bodies. Irene Mittelberg is Full Professor of Linguistics and Cognitive Semiotics at the Institute of English Studies of RWTH Aachen University (Germany), where she directs the Natural Media Lab and the Centre for Sign Language and Gesture. She holds a PhD in linguistics and cognitive studies from Cornell University. Her work combines semiotic theory (e.g. Peirce, Jakobson) with embodied approaches to language, cognition, multimodal interaction and the visual arts. Her recent interdisciplinary work includes the adoption of Peirce’s universal categories for neuroscientific research into gesture. She wrote a monograph on Metaphor and Metonymy in Language and Gesture (2006), co-edited Methods in Cognitive Linguistics (2007) and has (co-)published over sixty journal articles/ book chapters. She serves on the board of the International Association for Cognitive Semiotics (IACS) and the German Association for Semiotics (DGS). Tapani Möttönen is a postdoctoral fellow of Finnish language in the Department of Finnish, Finno-Ugrian and Scandinavian Studies at the University of Helsinki, Finland. His general research interests include metatheory and methodological development of linguistics and semantics, and his PhD research focused on the ontology of meaning in cognitive grammar. He currently works on adapting methodology from cognitive linguistics to the analysis of spoken interaction as a part of industrial design processes.

LIST OF CONTRIBUTORS

xv

Kay L. O’Halloran is Chair Professor and Head of the Department of Communication and Media in the School of the Arts at the University of Liverpool. Her research area is multimodal analysis, involving the study of the interaction of language with other resources in texts, interactions and events. A key focus of her work is the development of digital tools and techniques for multimodal analysis, as well as mixed methods approaches that combine multimodal analysis, data mining and visualization for big data analytics within and across different media platforms. Alin Olteanu is Postdoctoral Researcher and Publications Coordinator at the Käte Hamburger Kolleg Cultures of Research of RWTH Aachen University, Germany. Having worked on the dialogue between semiotic theories and educational philosophy, Alin is currently interested in the potential of biosemiotics to contribute to society and technology studies, with a focus on digitalization and sustainability. Adrian Pablé is Associate Professor in the School of English at the University of Hong Kong and Professor Extraordinarius at the University of South Africa (Pretoria). He is a Roy Harris scholar and an integrational linguist, with an interest in the history of linguistics and in ideologies of language. He is the secretary of the International Association of the Integrational Study of Language and Communication (IAISLC) and the series editor of the Routledge Advances in Communication and Linguistic Theory. His recent publications include the monograph Signs, Meaning and Experience (de Gruyter, 2015, with Chris Hutton) and the edited volume Critical Humanist Perspectives (Routledge, 2017). He has published several articles on the relationship between semiotics and semiology, and he is currently applying an integrationist critique to the posthumanist turn in linguistics. João Queiroz is Professor at the Institute of Arts, Federal University of Juiz de Fora, Brazil, and in the Postgraduate Program in Communication at the same institution, where he coordinates the Iconicity Research Group (IRG). He has been teaching courses on cognitive semiotics and intermediality studies. He has supervised PhD and master’s students in the fields of semiotics, Latin-American art and literature and cognitive aesthetics. He has several publications in international journals, books and conferences, including the Commens Digital Companion to Charles S. Peirce (co-organized with Mats Bergman and Sami Paavola). Queiroz is a member of the International Association for Cognitive Semiotics (IACS), a member of Group for Research in Artificial Cognition (UEFS, Brazil) and an associate researcher of the Linguistics and Language Practice Department, University of the Free State (South Africa). Website: https://joaoqueirozsemiotics. wordpress.com Göran Sonesson is Professor Emeritus at the Division of Cognitive Semiotics, Lund University, Sweden. He holds doctorates in general linguistics from Lund and in semiotics from Paris. He has published numerous papers, both theoretical and experimental, on pictorial, cultural and cognitive semiotics, as well as on the semiotics of communication and translation and the evolutionary foundations of semiosis. Apart from anthologies, his papers have appeared in journals such as Semiotica, Cognitive Semiotics, Cognitive Development, Sign System Studies, Degrés, Signa, Signata, Sign and Society and Frontier of Psychology. His main book-length works are Pictorial Concepts (1989), which is a critique of the critique of iconicity, and Human Lifeworlds (2016), which is a study of

xvi

LIST OF CONTRIBUTORS

cultural evolution. His new book, The Pictorial Extensions of Mind, will be published next year by de Gruyter. Sabine Tan is Senior Research Fellow at Curtin University, Western Australia. She has a background in critical multimodal discourse analysis, social semiotics and visual communication. She has applied multidisciplinary perspectives to the analysis of institutional discourses involving traditional and new media. She has worked on interdisciplinary projects involving the development of interactive software for the multimodal analysis of images, videos and 360-degree videos for research and educational purposes. She has also worked on projects to develop mixed methods approaches which combine multimodal analysis, data mining and visualization for big data analytics in areas such as violent extremism, news discourse and 360-degree video. Jordan Zlatev is Full Professor of General Linguistics and Director of Research for the Division of Cognitive Semiotics at Lund University. He served as the first president for the Swedish Association of Language and Cognition (SALC), 2006–9, and for the International Association for Cognitive Semiotics (IACS), 2013–14. His current research focuses on polysemiotic communication and, more generally, on the nature of language in relation to other semiotic systems like gesture and depiction, as well as to consciousness. He is editor-in-chief of Public Journal of Semiotics.

ACKNOWLEDGEMENTS

Out of all the hundreds of individuals involved in this project, we owe a big thank you up front to Andrew Wardell, Senior Commissioning Editor of Linguistics at Bloomsbury from 2017–20. Without Andrew’s invitation and his ensuing vision, enthusiasm and patient support for this project, it would never have gotten off the ground; and it certainly wouldn’t have grown from a single volume ‘companion’ into a four-volume ‘major reference work’. Andrew issued the invitation for this project while still serving as Editorial Assistant to Gurdeep Mattu. Many thanks are also due to Becky Holland, Editorial Assistant to Andrew from 2018–21; to Morwenna Scott, Senior Commissioning Editor of Linguistics since 2020; and to Laura Gallon, Editorial Assistant to Morwenna since the end of 2021 – each of whom played key roles in shepherding the project along to publication against the bottomless backdrop of tragedy, uncertainty and delay that marked the Covid-19 pandemic. Additional vital support from Bloomsbury during the project’s final stages came from Production Editor, Elizabeth Holmes, and from Dharanivel Baskar, Team Lead for project management at Integra Software Services, both of whom engaged untold support from their own dedicated teams. It has been a pleasure to work with all of you: thank you once again. During the final two years of intensive work, this project benefitted financially from a number of research grants, including a 2020–21 collaborative publishing grant from the Faculty of Arts and Creative Industries, Middlesex University, London (via Paul Cobley) and three grants from Toronto Metropolitan University – including a 2020–21 Faculty Research Grant from the Faculty of Arts (via Jamin Pelkey), a 2021 graduate research assistant grant from the TMU-York Graduate Program in Communication and Culture (via Stéphanie Walsh Matthews) and a 2021–22 Work Study Research Assistant Grant from the Office of the Vice President of Research and Innovation (via Jamin Pelkey). These funds paid for logistical and copy editing support involving seven student researchers: three PhD researchers (Sophia Melanson Ricciardone, Richard Rosenbaum and Jan Vykydal) and four undergraduate researchers (Sari Park, Irene Storozhuk, Leonard Pamulaklakin and Kai Maurin-Jones). Many thanks to all. Appreciation is also due to nine anonymous peer reviewers for their invaluable guidance and feedback, including a converging series of recommendations that ultimately led to the expansion of the project from one volume to four. The project also benefited from countless consultations, conversations and inspirations afforded by connections with semioticians around the world. Let us say thank you in this regard to colleagues in the Semiotic Society of America, the International Association for Cognitive Semiotics, the International Cognitive Linguistics Association, the International Society for Biosemiotic Studies, and the International Association for Semiotic Studies, many of whom have played a formative role, suggesting in the process the need for additional complementary volumes of this nature to offer expanded scope, depth and range, in terms of topics, angles of coverage, and qualified participants. Semiotics, after all, is still only just getting started; and if these volumes serve the enterprise en route to its next milestone, it is thanks to everyone involved.

­L IST OF ABBREVIATIONS GUIDE TO CRITICAL PEIRCE EDITIONS

Charles S. Peirce: Primary Sources

CD

CN

CP

EP 1

EP 2

LoF

LI

MS

Peirce, C. S. (1889–91), entries in The Century Dictionary: An Encyclopedic Lexicon of the English Language, ed. W. D. Whitney, New York: Century Co. Cited as CD, followed by page number. Peirce, C. S. ([1901–8] 1975–9), Charles Sanders Peirce: Contributions to ‘The Nation’, 3 vols, ed. K. L. Ketner and J. E. Cook, Lubbock: Texas Tech University Press, 1979. Cited as CN, followed by volume number and page number. Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP, followed by volume number and section number. Peirce, C. S. ([1867–93] 1992), The Essential Peirce: Selected Philosophical Writings, vol. 1, in N. Houser and C. Kloesel (eds), Bloomington: Indiana University Press. Cited as EP 1, followed by page number. Peirce, C. S. ([1893–1913] 1998), The Essential Peirce: Selected Philosophical Writings, vol. 2, ed. Peirce Edition Project, Bloomington: Indiana University Press. Cited as EP 2, followed by page number. Peirce, C. S. ([1895–1910] 2019–21), Logic of the Future: Writings on Existential Graphs, ed. A.-V.Pietarinen, vol. 1: History and Applications; vol. 2/1: The Logical Tracts; vol. 2/2: The 1903 Lowell Lectures; vol. 3/1: Pragmaticism; vol. 3/2: Correspondence, Berlin: De Gruyter. Cited as LoF, followed by volume number and page number. Peirce, C. S. ([1891–1910] 2009), The Logic of Interdisciplinarity: The Monist Series, ed. E. Bisanz, Berlin: Mouton de Gruyter. Cited as LI, followed by page number. Peirce, C. S. ([1857–1914] 1787–1951), The Charles S. Peirce Papers Harvard University, Houghton Library, MS Am 1632. Individual papers are referenced by manuscript number in R. Robin (ed), Annotated Catalogue of the Papers of Charles S. Peirce, Amherst: University of Massachusetts Press, 1967, and in Robin (1971), ‘The Peirce Papers: A Supplementary Catalogue’, Transactions of the Charles S. Peirce Society 7: 37–57. Cited as MS, followed by manuscript number and page number.

­LIST OF ABBREVIATION

xix

Note: Houghton Library catalog available online: https://hollisarchives.lib. harvard.edu/repositories/24/resources/6437. Many of Peirce’s papers and letters are available in the Microfilm Edition, Harvard University Library, 38 Reels (1966-70). Digital images of the microfilm, in the Robin catalog number sequence, are also available online: NEM

PoM

PPM

SS

SWS

W

https://rs.cms.hu-berlin.de/peircearchive/pages/home.php. Peirce, C. S. ([1866–1913] 1976), The New Elements of Mathematics, 4 vols, ed. C. Eisele, The Hague: Mouton Press. Cited as NEM, followed by volume number and page number. Peirce, C. S. ([1888–1908] 2010), Philosophy of Mathematics: Selected Writings, ed. M. E. Moore, Bloomington: Indiana University Press. Cited as PoM. Peirce, C. S. ([1903] 1997), Pragmatism as a Principle and Method of Right Thinking: The 1903 Harvard Lectures, ed. P. Turrisi, New York: SUNY Press. Cited as PPM, followed by page number. Peirce, C. S. and V. L. Welby ([1903–11] 1977), Semiotic and Significs: The Correspondence between Charles S. Peirce and Victoria Lady Welby, ed. C. Hardwick and J. Cook, Bloomington: Indiana University Press. Cited as SS, followed by page number. Peirce, Charles S. ([1894–1912] 2020), Selected Writings on Semiotics, 1894–1912, ed. F. Bellucci, Berlin: Mouton de Gruyter. Cited as SWS, followed by page number. Peirce, C. S. ([1857–92] 1982–2010), Writings of Charles S. Peirce, 7 vols (1–6, 8), ed. Peirce Edition Project, Bloomington: Indiana University Press. Cited as W, followed by volume number and page number.

xx

Introduction Semiotic Movements PAUL COBLEY

This fourth volume of Bloomsbury Semiotics might be best introduced – perhaps surprisingly – with reference to a play first performed in 1670. Molière’s Le Bourgeois gentilhomme (The Bourgeois Gentleman) features the attempts of a middle-class man to realize, through instruction, his aristocratic aspirations. Monsieur Jourdain, wishing to ascend to the noble class, enlists the services of a number of teachers who will educate him in the ways of the aristocracy. In line with his ambition, he desires union with a Marchioness and asks his philosophy master to help him compose a love note to her. The philosophy master asks him whether he wants the note composed only in prose. Then: Monsieur Jourdain: No, I don’t want either prose or verse. Philosophy Master: It must be one or the other. Monsieur Jourdain: Why? Philosophy Master: Because, sir, there is no other way to express oneself than with prose or verse. Monsieur Jourdain: There is nothing but prose or verse? Philosophy Master: No, sir, everything that is not prose is verse, and everything that is not verse is prose. Monsieur Jourdain: And when one speaks, what is that then? Philosophy Master: Prose. Monsieur Jourdain: What! When I say, ‘Nicole, bring me my slippers, and give me my nightcap,’ that’s prose? Philosophy Master: Yes, Sir. Monsieur Jourdain: By my faith! For more than forty years I have been speaking prose without knowing anything about it, and I am much obliged to you for having taught me that [. . .] (Molière, Le Bourgeois Gentilhomme, Act II, Sc. IV) This brief interaction dramatizes what has become known as the ‘Jourdain effect’. An amusing episode, it demonstrates the naivete of Jourdain in failing to recognize the dominance of prose in discourse but also, more importantly, the fact that he is not conscious of the very architecture of his own speech. This lack of awareness of the mechanics of signification processes even while one is using them is common over a

2

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

number of domains. Many people around the globe talk to each other every day without ever registering that they are employing verbs, nouns, tenses and cases. Yet, the effect operates also at the level of academic fields and disciplines: no matter how self-conscious and methodologically precise any discipline might be, it is still susceptible of utilizing procedures and assumptions which it barely acknowledges but which are central to its epistemology and its operations. Among the many things that semiotics does and the many areas of scholarly endeavour that it pervades, there are still fields and domains where a semiotic perspective is only found to be implicitly at work. Sometimes that perspective is acknowledged without necessarily being named (e.g. cognitive theory); sometimes it occupies a major part of the domain (e.g. cultural studies); sometimes it is unacknowledged but not necessarily named (e.g. linguistics); sometimes it is deliberately repressed for methodological purposes (e.g. information theory and, again, linguistics); sometimes it is repressed with bad faith (e.g. biology). The founding father of contemporary semiotics, Thomas A. Sebeok, frequently referred to the Jourdain effect or ‘factor’ (1998: 35) in his interdisciplinary studies (e.g. 1991: 45) and, indeed, as Danesi (2011: 116) points out, made a vast number of converts to semiotics in this way (cf. Houser 2016: 321; Favareau 2010: 44; Aroni, Chapter 12, Vol. 2). It should be pointed out, too, that Roman Jakobson (1960: 356) made exactly the same point about the Jourdain effect in relation to metalanguage in one of the most famous contributions to semiotics and communication theory, which appeared in a book edited by Sebeok over sixty years ago. The general contention, then, is that in many disciplines where signification is a central – or even marginal – process, any consideration of it, however unsystematized it may be, amounts to a work of semiotics. This may seem a straightforward issue at first glance. Since semiotics is concerned with signification of all kinds, it seems to follow that any field which features sign trafficking and refers to this is, effectively, doing semiotics. Indeed, that field will be in want of semiotics, particularly if it takes signification for granted, treats it as an unconsidered trifle or, worse, represses the fact that its processes involve signs in action. Something of each of these scenarios is often present in the approaches of a number of disciplines. Biology, for example, features sign action at higher, organismic levels (animal and plant signals) as well as at lower thresholds (the neurons, mRNA). Yet, when biology is taught, a semiosic entity such as mRNA is usually pronounced to be a ‘messenger’ effective in the production of proteins. Vanishingly rare is the case that there will be, as part of biology as a disciplinary field, explication of the messenger mechanism and its relation to other biological functions with palpably semiotic constitutions. Indeed, addressing this lacuna in biology is one of the aims of biosemiotics within general semiotics. However, confronting the matter of semiosis in terms of the Jourdain effect may not always be well advised. Norris (1986: 229) suggests that figuring semiotics in this fashion may seem like a threat to instal it as a ‘master-science’ while also depriving it of the possibility of enjoying any clear disciplinary status. Certainly, this would not fit well with the orthodoxy of, say, postmodernism and, in particular, its customary claim that there is incredulity towards meta (or master) narratives – such as semiotics in this figuring – and that universals need to be superseded by the specifics of the local political conjuncture. Norris suggests that the semiotician says, with ‘boundless confidence’, humani nil a me alienum putum. Well, since Norris was writing, the situation has got worse than that because, as the aforementioned biosemiotics has attested over the last thirty years, the domain of semiotics has expanded beyond the human to all known life forms.

INTRODUCTION

3

In the face of postmodernism and its decline as an academic trend, many disciplines have, while remaining alive to the importance of local configurations, gone back, or continued, to explore the possibility that there may exist universals. In the current volume, such universals include interpretative and communicative processes (chapters on hermeneutics, phenomenology, communication theory), the operation of systems (chapter on systems theory), human processes of knowing (chapters on digital humanities, media and cultural studies and on translation studies), human modes of signification (chapters on pragmatics, multimodality, discourse analysis and integrationism) and cognition (with chapters on cognitive linguistics and cognitive science generally). Potentially, there could be other fields, disciplines and domains where semiosis and semiotic understanding are involved in the investigation of what some consider to be universal processes and attributes. These include medicine (see Tredinnick-Rowe and Stanley, Chapter  6, Vol. 2), finance and economics (see Oakley, Chapter 10, Vol. 2), religion and the idea of a supreme being (see Leone, Chapter 4, Vol. 3), mainstream linguistics, cultural anthropology (see Ness and Coleman, Chapter 2, Vol. 3), palaeontology, the possibility of artificial intelligence, law (see Chapdelaine-Feliciati, Chapter 8, Vol. 2) and performance studies. So, in pursuit of some of these processes, the current volume is concerned with semiotic movements. Whereas Volume 1 featured contributions on movements and areas of prominence in the history of semiotics, with Volume 2 featuring contributions specifically concerned with areas of the natural and technical sciences which have been infused with semiotics, and Volume 3 containing the same for the arts and social sciences, the current volume slightly re-figures the term ‘semiotic movements’. Rather than being movements that have characterized the development of semiotics as a disciplinary field, the contributions here are largely concerned, in line with the Jourdain effect, with aspects of signification – but have not necessarily declared, fully realized or completely embraced that fact. They are movements that are semiotic, but often proceed with areas of functioning that do not necessarily advertise their semiotic bearing. Communication theory (Lanigan, Chapter 1, this volume) has had a relationship to semiotics which has often been so close that it has been impossible to determine the boundary line (Sebeok 1977: 185). Indeed, it has been closer, perhaps, than the field of communication studies in general, which has been overwhelmingly concerned with human communication, particularly as it might be approached from a social sciences perspective (Cobley and Schulz 2013). Qualifying this last observation, it should be noted that there are indications that the two fields, with communication theory at the forefront, are inexorably moving closer (Self 2013). Although the convergence of semiotics, particularly in its more contemporary concerns beyond human communication, and media/culture studies (Melanson and Danesi, Chapter 2, this volume) cannot be said to be proceeding as rapidly as the convergence of semiotics and communication theory, there has been progress. The second part of the media and cultural studies couplet, particularly in some aspects of ‘posthumanism’, has started to acknowledge common grounds and approaches, albeit often with its own preliminary agenda. Indeed, digital humanities (Olteanu and Ciula, Chapter  2, this volume) as a whole has, as its raison d’etre, a concern with signification – both in the differences in signification between digital and non-digital methods for research and between the same for dissemination of knowledge. As with communication theory, systems theory (Gudwin and Queiroz, Chapter 4, this volume) has also had an immeasurably close relationship with semiotics, arising, initially, from the post–Second World War ‘great expectations’ (Sperber 2008) of a unified science featuring cybernetics, information theory, linguistics, semiotics and

4

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

game theory. Cannizzaro (2010, 2013, 2014) argues, persuasively, that systems theory is effectively a continuation of semiotics, particularly in the centrality of the concept of modelling after the work of Lotman from the 1950s onwards. Where it differs from media/culture studies, communication theory and digital humanities is that systems theory has not harboured a semiotic wing itself but, rather, studied systems in a manner that is very much analogous and allied to that of semiotics’ approach. Semiotics has also been a concern for particular developments in philosophy in the last century, in terms of its implications both for ontology and for epistemology. In this volume, phenomenology and hermeneutics have been chosen as emblematic of this concern as a ‘semiotic movement’. Although semiotics has been resolutely committed to an ‘anti-psychologistic’ (Stjernfelt 2014) perspective on the existence of signs – sometimes in contrast, sometimes overlapping with contemporary speculations from cognitive perspectives – it has acknowledged, in various ways, the crucial role of first-person experience in the fortunes of semiosis. One reason why phenomenology and semiotics can be so closely related is that the latter, while seeming to propose an entity called ‘the sign’ and a larger one called ‘semiosis’, both of which are putatively irreducible and stable, has never, in practice, presented such a proposition. In the oft-quoted definition from Peirce, for example, a sign ‘stands to somebody for something in some respect or capacity. It addresses somebody that is, creates in the mind of that person an equivalent sign or perhaps a more developed sign’ (c.1899: CP 1.564). The sign, in this sense, is very much bound up with phenomenology (Bundgaard, Chapter 5, this volume), notwithstanding the fact that ‘somebody’ in this formulation need not even be human. This phenomenological orientation of the sign, of course, entails that signs are variably interpreted. Originally applied to sacred texts, the close study of written sources that provided the crucible for hermeneutics (Arnett and Mancino, Chapter 6, this volume) is to be found in the work of ‘Plato, Augustine, Humanism, the early Heidegger and Husserl’, according to Grondin (1995: xi). He adds (1995: 36) that phenomenology gave birth to hermeneutics, almost unwittingly or against the wishes of Husserl, one of modern phenomenology’s founders. What hermeneutics does carry out, however, is a kind of intense phenomenological semiotics of texts which is allied to other kinds of textual semiotics such as narratology but, because of its institutional history, has managed to remain distinct. Effectively, a form of textual semiotics is quite explicitly carried out by four more of the ‘semiotic movements’ covered in this volume. In the burgeoning field of translation studies (Kourdis and Hartama-Heinonen, Chapter 7, this volume), there is specific focus on how one text, with all the determinants specific to it, might undergo a transmutation into another text, often involving shared determinants but also involving quite different ones. The circumspection of the formulation in the last sentence will be noted because, in demotic conceptions of translation, what is taken to be involved is merely a conversion from one national language to another. What semiotics has demonstrated for translation studies is that national language correspondences and divergences are merely the tip of the iceberg. Translation takes place across a number of semiotic modes, both verbal and non-verbal, with ‘interlinguistic’ translation only constituting a small part of a general process featuring in myriad transmissions at lowly organismic levels upwards (Marais 2020). Similarly, the studies of multimodality (Machin and Chen, Chapter 9, this volume) that have come to dominate the analysis of discourse in the last twenty-five years have arisen from fairly modest beginnings. Mainstream linguistics, constantly reinventing itself and attempting to protect against extinction, started to realize that contemporary culture has involved other modes than the word. Most commonly identified has been pictorial

INTRODUCTION

5

representation, especially in digital interfaces, with the move from page to screen (Kress 2003). Often, ‘semiotic’ has been taken by linguists as a synonym for that which is not solely linguistic; however, the field of multimodality studies has developed in a limited way, focusing on the pictures and the visual, with research into other non-verbal modes (e.g. Jaworski and Thurlow 2011) still a rarity. Nevertheless, both translation studies and multimodality studies retain and share a fairly generalist remit with semiotics. Discourse analysis and pragmatics, by contrast, are more avowedly specific in their concerns. One part of linguistics’ reinvention in the face of semiotics over the last fifty or sixty years has involved fragmentation. In addition to multimodality studies, linguistics has also developed sociolinguistics, discourse analysis, Critical Discourse Analysis (CDA), systemic functional linguistics (SFL), ordinary language philosophy, pragmatics, relevance theory, (its own version of) ethnomethodology, conversation analysis (CA), psycholinguistics, generative linguistics, generative semantics, neurolinguistics, cognitive linguistics and numerous others, many of them continuing to focus on linguistics’ traditional moieties of morphology, phonology, phonetics, grammar, syntax, lexicon and semantics. Yet, it is worth remembering that a more general division which has applied in linguistics was actually proposed from within semiotics as early as 1938. In Foundation of the Theory of Signs, Charles Morris identified the semantical dimension of semiosis (the relation of signs to objects), the pragmatical dimension of semiosis (the relation of signs to interpreters) and the syntactical dimension of semiosis (the formal relation of signs to one another) (1938: 6–7). Taken up also by Carnap (1948: 8–11) in the service of logic and deduction, the division effectively launched the linguistic field of pragmatics which has enjoyed immense success. Enjoying the benefit of multiple journals, a plethora of textbooks, specifically titled professors and a well-subscribed international association, pragmatics has expanded the study of the relation of signs to interpreters with an extensive armoury of technical applications. Arguably launched as a discipline in ordinary language philosophy and, especially, the work of Austin ([1955] 1962), pragmatics has been, from its wider semiotic beginnings, principally a pursuit confined to linguistics (Moeschler 2021). As a semiotic movement (Brandt, Chapter 8, this volume), its attempts to move beyond a curricular prison-house of language are still in gestation (Wharton 2009; Scarantino 2017; Xie, Yus and Haberland 2021), although the benefits of pragmatics in understanding the movement from non-verbality to acquisition of language by children have been made palpable (Clark 2017; Bohn and Frank 2019). In its relation to some multimodality studies, discourse analysis, by contrast with pragmatics, has made significant inroads in the move beyond language (Machin 2013). Nevertheless, it is once more a semiotic movement whose main profile has primarily been in the field of language. Developing in tandem with other linguistic text-focused semiotic movements such as narratology and myth criticism, discourse analysis (O’Halloran and Tan, Chapter 10, this volume) has both a general existence and a more specified one. The more general one arises out of the impetus to analyse features of language beyond or bigger than the traditional linguistic domain of the sentence. Arguably, this impetus has a lineage which runs in parallel with the relatively recent (Harris 2001) establishment of linguistics in the academy. It has often been manifest in fairly consistent concerns with figures in language such as metaphor, metonymy, simile, irony and so forth. Yet, along with the development of Saussurean semiotics (or semiology) in Europe during the 1960s, and especially the ‘invention’ of the ‘text’ (Marrone 2014), such concerns turned increasingly towards a more critical analysis which exposed the broadly political implications of the options taken within discourse. Roland Barthes, whose work differed

6

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

from discourse analysis in that he took linguistic models and expanded them to analyses of non-verbal phenomena from the outset, nevertheless provided inspiration for CDA (Waugh, Masaeed, Hong Do and Renigar 2016; Catalano and Waugh 2020). As with Barthes’s early essays (1973, 1977), discourse analysis increasingly attempted to reveal not just the mechanics of texts, but the ‘agenda’ of any instance of discourse: how it worked – largely unconsciously – to instil particular messages which were often far from being disinterested ideologically, politically or both. As discourse analysis developed in the 1980s and 1990s, the question of power became absolutely central, with many analyses learning from Foucault – not an avowedly semiotic thinker – that institutions, currents of thought and even whole societies might act in a programmed fashion under the yoke of one or more discourses. Critical discourse analysis (in upper case – CDA), developed by Norman Fairclough (1989), is a particular case in point, ossifying into a method which involves a kind of close reading or semiotics of texts in order to demonstrate a larger point about how a given text instantiates the embedding of power relations in sign systems. One benefit of early discourse analysis, perhaps, was its relative lack of methodological precision. It could combine a number of approaches and adjust its own overall approach in line with the situations that gave rise to the texts upon which it focused. Later, CDA and critical discourse studies enjoyed a great deal of success and are sometimes considered as rigorous methods, supported by a number of journals invoking their name. Yet, as Peter Jones (2007) argues, CDA has been successful because it flatters the culture which sponsors it. As the holders of power in contemporary Western social formations have become more aware of the importance of sign systems, so critical discourse analysts – with good intentions to criticize unequal power relations and even promote social change – have responded in a way which not only affirms the current state of affairs but reifies the phenomenon under scrutiny: that is, language. Jones’s argument is mounted from the position of the thoroughgoing critique of the ‘language myth’ (Harris 1981) which characterizes integrationism (Pablé, Chapter 11, this volume). Rather than ‘segregating’ communication processes (the actual activities in which sign users engage) from language (routinely conceived as an entity, or an abstract system), an integrationist approach, as the name implies, brings both together as one process. As a result, integrationism is an anomaly in a long history, as well as amidst contemporary practices, of conceiving signification in terms of the myth of language as something that can be located, quantified and labelled. For Jones (2007: 367), ‘thinking about language in terms of “codes”, “models”, “frameworks” of rules, rule systems, or choices within inventories or networks is not part of the solution but part of the problem’. As a semiotic movement, then, integrationism or ‘integrational semiology’ and contemporary semiotics have been steadily converging, as the latter has moved away from its decades-long (1960s to 1980s) fixation on codes and invariants in signification (Pablé and Hutton 2015). Similar issues are evinced in the development of what might be the quintessence of a certain kind of semiotics, recapitulating linguists’ synonym for that which is not solely linguistic: gesture studies. Again, gesture studies, as a particularly strong branch of nonverbal communication studies in general, has sometimes been riven by the relationship of semiosis and invariance. Sebeok (2001a: xiii) refers to the work of his ex-University of Chicago office mate Ray Birdwhistell, whose kinesics introduced a programme devoted to gesture but which became ‘effectively moribund’ owing to its attempt to align nonverbal communication with grammar. Foremost in semiotics, with programmes devoted to a more semiotic rather than codified perspective on gesture, are the works of Adam

INTRODUCTION

7

Kendon (e.g. 2004) and David McNeill (e.g. 2007). Yet, as a field of study, gesture itself has tended to be emblematic of the division of non-verbal communication in the academy. In communication study, for example, non-verbal communication has become a massive endeavour (Hall and Knapp  2013), but this is within a context in which ‘communication’ implies only human communication and largely focuses on bodily communication other than endosemiosis. Despite the inaugural work of Ruesch and Kees (1956), there have been very few attempts to situate gesture and human bodily communication within the wider context of the non-verbal communication which includes the whole complex of visual communication in general, endosemiosis and nonhuman communication (Sebeok 2001b). Nevertheless, the proximity of gesture as a field to that of human communication, along with its address of some fundamental issues, has maintained its central place in semiotics. Equally enmeshed in a complex relationship with semiosis and invariance have been the semiotic movements associated with the so-called ‘cognitive revolution’. Brandt (2011: 49) has suggested: ‘If semiotics studies meaning, and cognitive science studies the mind, then cognitive semiotics is the study of mind and meaning – the way meaning exists and works in human minds (and ideally, in animal minds in general).’ This is not at all unreasonable. Furthermore, the fact that his statement bears upon ‘cognitive semiotics’, a major strand of general semiotics, with its own successful society, regular conferences and an esteemed journal makes it seemingly uncontentious as a statement. Notwithstanding the Jourdain effect, however, a strong case could be made for cognitive linguistics and cognitive science as semiotic movements – that is, as studying ‘meaning’, in Brandt’s formulation, from the very start. Both of them trade, primarily, in signs or the concept of signs. Indeed, despite the fact that he did not commit it to print, more than once Thomas A. Sebeok referred to research in cognitive science as ‘semiotics plus money’. Sebeok was referring not so much to the coded semiotics of cultural theorists, but to the dynamic sign as conceptualized by Peirce and a semiotics derived from the new conceptualization of language in which he had participated in the immediate post–Second World War period. For Sebeok, whose own work contributed to resolving the impasse of behaviourist research created by the likes of his mentor Charles Morris, the perspective of Chomsky as well as the cognitive linguistics and the cognitive revolution that followed were pivotal (see Cobley 2016: 34–8). Cognitive linguistics (Zlatev and Möttönen, Chapter  13, this volume), stemming from Chomsky’s work in the first instance, forced semiotics (as well as other fields) to consider the prominent role of syntax. Syntax, of course, had already been introduced into the semiotic programme by Morris, but was afforded comparatively lowly status. Cognitive linguistics raised the stakes by insisting that the very definition of language lay in the innateness, as well as the formality, of syntactic rules (Chomsky 1957). Yet, what was to characterize cognitive science was the ‘plus money’ aspect of Sebeok’s formulation – that is, cognitive science (as the name may even suggest) conceived itself in scientistic fashion, seeking to carry out experiments and empirical investigations rather than laying out formal rules (Sonesson, Chapter 14, this volume). This would not necessarily guarantee that it became a science; however, empirical work can provide considerable cachet in the world of grant-driven research, particularly when it purports to deal with something as fundamental as consciousness. Even so, a glimpse of the paraphernalia of much thinking in cognitive science – signalling, schema, rules, models, as well as semantics, pragmatics, syntactics and aspects of existing linguistics – makes it hard to avoid concluding that cognitive science is a semiotic movement.

8

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

As has been argued, the semiotic movements featured in this volume are, in their own way, attempts to come to terms with universals, although not necessarily by utilizing universalizing principles. Thus, they have tended to correspond to historical landmarks in the development of semiotics in the last two-and-a-half thousand years. As possibly the founder of semiotics, Hippocrates (c.460–370 BCE) certainly seems to have inaugurated the systematic investigation of signs in the construction of the corpus of symptoms and treatments, observing that a symptom exists as such when it is found to be identical in Delos, Scythia and Libya (Hippocrates [430-330 BCE] 1983: 185). At the very root of semiotics, then, there is an attempt, for good reasons, to identify a potentially universal phenomenon through signs. Yet, of course, diagnosis or prognosis is not always watertight; as Peirce acknowledged (e.g. 1893: CP 5.265), it should be understood as subject to fallibilism. Does this entail that there are some semiotic rules in some domains and different ones in others? Is it the case that some disciplines are more suited to rigid definitions in signification or actively demand them? Volume 2 of this Bloomsbury Companion, featuring contributions to the natural and technical sciences, deals with fields that have tended to rely on invariance in signification. Yet, the contributions to that volume reveal, in consonance with contemporary semiotics, the extent to which variance in sign use very much dictates the results and orientations of fields and disciplines. This is probably to be expected in, say, jurisprudence (Chapdelaine-Feliciati, Chapter 11, Vol. 2); yet it is likely that most lay people would be unaware of the extent to which this is true of biology, ecology, zoology, medicine and even mathematics (Kull and Favareau, Chapter  2; Maran, Chapter  3, Tønnessen, Chapter  4; Tredinnick-Rowe and Stanley, Chapter  4; Stjernfelt and Pietarinen, Chapter 1; respectively, all in Vol. 2). The semiotics which has arisen since the twentieth century has had a natural home in the arts and humanities; so, it is perhaps unsurprising that its interpretative orientation has infused fields from musicology, education and archaeology (Dougherty, Chapter  10; Olteanu and Stables, Chapter 7; Tamm and Preucel, Chapter 3; respectively, all in Vol. 3). Nevertheless, what is demonstrated overall is that some intellectual movements, as well as those inhabiting the Jourdain effect, have become semiotic or realized some of the implications of their significatory proclivities. Put yet another way, intellectual movements through a semiotic lens traverse grounds where it is now possible to contemplate the project of those movements in their oscillation between certainty and doubt. Even Hippocrates, whose project was committed to the goal of certainty lest a shortfall of certainty should precipitate harm, would have been compelled to operate with some residual doubt. As the groundbreaker for both a semiotics of interpretation and a semiotics of invariance, effectively his work is the ultimate reference point certainly for a semiotic history of the universe as adumbrated in Volume 1, but also for all the movements covered in the current volume. Interpretative and communicative processes (hermeneutics, phenomenology, communication theory) seek ways to establish both principles of invariance and invariant properties while stressing the mutability, locality and indeterminacy of interpretation. The same can be said, from the opposite direction, in respect of the operation of systems (systems theory), where the fallibility of systems corresponds to the interplay of agency in semiosis in relation to the constraints of semiosis. That same phenomenon of agency, associated most readily with the interpretative behaviour of humans’ specific forms of oscillation between certainty and doubt in acts of knowing (digital humanities, media and cultural studies, translation studies), also characterizes the very modes of human signification (chapters on

INTRODUCTION

9

pragmatics, multimodality, discourse analysis, gesture and integrationism). It may even be a manifestation of cognitive processes generally (anthropological linguistics, cognitive linguistics, cognitive science). Much more than doubt and fallibilism, however, semiotics – particularly in relation to semiotic movements – finds itself confronting a particular challenge which was mentioned at the start of this Introduction to Volume 4. That is, there is fear that semiotics might be posed as a ‘master-science’ while at the same time being a discipline with no purchase at all. Certainly, its lack of a firm institutional basis – strong as a subject area, with many international journals and handbooks in semiotics such as this one, but less strong as an embedded subject area outside Latin America, with relatively few courses and professorial appointments – suggests some truth in the latter. Indeed, it is possible that semiotics’ alleged positing of itself as a master-science and its weak institutional basis are intimately connected. Yet, however marginalized semiotics has been in the academy since Hippocrates, it is a survivor, adapting to the demands of rapidly changing circumstances in the realm of knowledge and, as this volume attests, being ready to support, facilitate and stabilize some intellectual movements whether they explicitly acknowledge semiotics or not. The contributions to the current volume present a range of semiotic movements and, in addition to locating them historically in a manner corresponding to this Introduction, they present specific perspectives on the controversies, debates and challenges that make up the ‘state of the art’. The volume opens with Richard L. Lanigan’s overview of communication theory. Proceeding from the distinction between competence and performance, Chapter 1 goes on to outline a ‘communicology’, drawing from phenomenology but also from some generalized models which have haunted communication theory: the human observer model; the human discourse model; the trivium model; and the chiasm model of discourse. The chapter sees the interface of semiotics and communication theory in, particularly, the sign’s ability to mediate a consciousness of time and in the mind which is simultaneously possible in the mind of the other, the interlocutor, a human that is not me. Melanson and Danesi’s Chapter  2 shows how media/culture studies have employed terms and strategies that draw on semiotics, such as opposition theory, code, text, iconicity and deconstruction. Especially important to modern media theory has been the work of McLuhan. Not usually associated with semiotics’ history, this chapter shows how McLuhan’s work reveals media to be an extension of the human’s sensorium, also creating a global village which is akin to a semiotic life of the mind. This is not only changing media but also changing what it is to be human. Where the old models of orality and literacy, with their semiotic overtones, served for previous media formations, both now have to be extended for the world of memes and perpetual electronic connection into digital literacy, computer literacy, mobile literacy and virtual worlds literacy. In their consideration of education, Olteanu and Ciula propose new avenues for digital humanities. Taking the concept of modelling, which has been influential in Lotman’s work and in biosemiotics, Chapter  3 suggests that it is especially apposite in understanding mediality. Following on from Melanson and Danesi’s discussion of mediality in terms of extensions of sensoria, this chapter argues that the digital humanities inculcate a special kind of literacy, one that is akin to the process of modelling in humans, an awareness of which might be taught in order to foster digital skills in general rather than simply teaching each localized digital skill in turn. In addition, Olteanu and Ciula see possibilities for semiotic and ecological approaches in digital humanities to encourage a ‘biosphere consciousness’.

10

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Gudwin and Queiroz’s Chapter  4 suggests that systems theory and semiotics – in particular, the semiotics of Charles Sanders Peirce – could mutually assist one another. What they share is the ability to see and even assess entities as processes rather than just as agents and objects. Semiosis, then, is not to be conceived simply as subject to the vagaries of all agents; nor does the chapter argue that semiosis is an unchanging system. Instead, it explores the possibility of a semiotic system – a stable ‘cluster of semiotic processes’. This, it is argued, has significant implications for the understanding of computer simulations. Despite the fact that Peirce proposed a phenomenology (his ‘phaneroscopy’), Bundgaard’s Chapter 5 suggests that this is not necessarily evident in the development of semiotics. The chapter cites Greimas and Peirce specifically. Yet, Bundgaard argues, a widened perspective reveals that Husserl’s phenomenology has decisively influenced the development of semiotics since the early twentieth century. Closely reading Husserl, Chapter 5 makes similar conclusions to Chapter 1, observing that a phenomenological semiotics invokes not a self-contained language system but, rather, interaction between acting intentional subjects giving rise to intersubjective communicating consciousnesses. As Chapter  6 by Arnett and Mancino avers, semiotics and hermeneutics have an intellectual objective in common. Comparing them with the work of Gadamer, and considering the themes of dialogue and semiosis, the chapter analyses two works by Eco: his rich treatise, A Theory of Semiotics, and his last, plot-driven novel, Numero Zero. It argues that hermeneutics explicates texts in a manner that exemplifies dialogic understanding; semiotics, as a complement, is discussed in terms of signs’ relation to other signs and their translatability. Ultimately, Arnett and Mancino’s conclusion suggests that hermeneutics and semiotics carry strong traces of their mutual progenitor, Augustine. Although Augustine stressed signification’s ability to beget further, future-orientated, signification, the idea of semiotics as translation, particularly intersemiotic translation, was properly set in motion by Roman Jakobson. Chapter 7, by Kourdis and HartamaHeinonen, identifies three main currents of thinking translation semiotically after Jakobson: from the perspective of Peircean semiotics; from the perspective of the Paris School, paying particular attention to ‘intertextuality’; and from the perspective of the Tartu-Moscow School. These have led to broadly different approaches depending on whether the objects of translation are linguistic or non-linguistic, whether they are cultural, cognitive, sociological or technological in nature, or whether they are more broadly concerned with objects of different kinds whose common denominator is that they are made up of signs. A movement as influential, particularly in communication studies, as pragmatics clearly deserves its own chapter in a volume on semiotic movements. Brandt’s Chapter 8 considers the interface of pragmatics and semiotics, finding it sufficiently significant to warrant the proposal of a semio-pragmatics. The chapter proceeds from the conception of meaning as both ‘immanent’ and ‘transcendent’, conceptual and contextual. Brandt sees semiotics as principally concerned with the former in these couplets and pragmatics as principally concerned with the latter. The chapter suggests that, in fact, pragmatics is constantly prevented from drifting off into sociology, linguistics or anthropology by semiotics’ insistence that pragmatics is called back to the ‘fundamental life of signs in human life’. Chapter  9, by Mittelberg and Hinnell, focuses on gesture, or ‘how interlocutors gesturally indicate objects, ideas, locations, or events and enact physical habits such as interpersonal interaction, movement patterns, or object manipulation, as well as more abstract, yet deeply embodied, schemata of experience’. Seemingly not a semiotic

INTRODUCTION

11

movement as such, the field of gesture studies invokes other ‘movements’ – cognitive science and multimodality, especially – and its concern with both human communication and the interpretation/invariance couplet, as noted above, offers it the flavour of a movement. Unlike many other discussions of gesture in semiotics which either adapt linguistic models or develop their own for discussing non-verbal communication, this chapter draws on Peirce, particularly his understanding of how signs relate to their objects and the embedding of these relations in habits. Mittelberg and Hinnell survey theories and empirical work, ultimately concluding that gesture studies not only throw a great deal of light on human gestures in a developmental ecology of human non-verbal communication, but also provide indications for understanding iconicity, indexicality and habit in semiosis more widely. Signs in human life partake of many modes. Machin and Chen’s Chapter 10 makes the point at the outset that ‘Communication was never “mono-modal” but always involved things like images, fonts, types of paper, gestures, sound, music’. Multimodality, as an approach that exemplifies the recognition that there are many modes of communication, has also customarily indicated that ‘choices’ of mode are very significant and that, for the addressee of communications, any choices offered are usually partial and limited. Taking a case study of food packaging which presents comestibles purporting to be healthy, and does so in a gendered way, the chapter shows how, say, energy or protein food bars aimed at men feature packaging that connotes science and technology, whereas those for women connote ‘nature’. Multimodality is used here, then, to expose how food marketing companies use communication modes strategically in order ‘to colonise and shape discourse and practices of taking care of our bodies’. In Chapter  11, O’Halloran and Tan trace the origins of multimodality and social semiotics to the seminal work of Michael A. K. Halliday, finding that some key concepts which his work allowed to flourish – system, metafunction, intersemiosis and resemiotization – provided the underpinnings for contemporary multimodal discourse analysis. Carrying out a number of approaches from the latter, including cluster visualization, along with the application of IBM Watson and Clarifai models, the chapter finds that the implementation of such mixed methods with current computer technology offers considerable possibilities for textual and contextual understandings through collection of metadata and existent semantic classifications. Any combination of semiotics with other movements necessarily entails an embrace of what semiotics understands to be the ontology of the sign. Yet, is it possible that the ontology of the sign for many in semiotics has become a little too settled? This is the key question asked by Pablé in Chapter 12. As opposed to the version offered by Saussure and Peirce, Pablé presents a Third Ontology of the sign, based on human creativity rather than a ‘standing for’ function. Derived from the integrationist principles of Roy Harris, the chapter outlines an ontology in which signs are what sign-makers make them. It suggests that integrationism is antithetical to a global semiotics because human activities and non-human activities ‘must ultimately be treated as “activities” tout court which trigger signs’ although it must be conceded that ‘integrationism is deeply humanist and anthropocentric’. Chapter 13 by Zlatev and Möttönen, whilst noting the Peircean tradition’s attempt to ‘dementalize’ semiosis but also overlooking Sebeok’s incorporation of post-Chomskyan linguistics into semiotics, observes that semiotics and cognitive linguistics seem to be miles apart. Yet, drawing attention to construal, iconicity, metaphor and multimodality, the chapter suggests that the subjectivist conception that dominates cognitive linguistics

12

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

and the emphasis on intersubjective structures in semiotics is resolved to some extent by the relatively recent development of cognitive semiotics, encompassing phenomenology and cognitive science generally, as well as semiotics and cognitive linguistics. Finally, Chapter  14, pace the Sebeok formulation on cognitive science cited earlier in this chapter, treats semiotics and cognitive science as two distinct research traditions. Semiotics, according to Sonesson, is concerned with ‘meaning’; cognitive science is concerned with consciousness. His chapter compares and contrasts semiotics and cognitive science, drawing out the stages of the latter, commencing with a focus on artificial intelligence and culminating, in recent years, with a movement from its fringes towards a more phenomenologically based concern with the nature of consciousness. As in the previous chapter, this one draws attention to the emergence of cognitive semiotics. Observing that linguistics is not to be considered as a third ingredient of cognitive semiotics, because it is already embedded in semiotics and cognitive science, Sonesson nevertheless notes that linguistics’ penchant for setting up rules and regularities is carried over in aspects of cognitive semiotics.

REFERENCES Austin, J. L. ([1955] 1962), How to Do Things with Words: The William James Lectures Delivered at Harvard University in 1955, ed. J. O. Urmson and M. Sbisà, Oxford: Clarendon Press. Barthes, R. (1973), Mythologies, trans. A. Lavers, London: Paladin. Barthes, R. (1977), Image – Music – Text, ed. and trans. S. Heath, Glasgow: Fontana. Bohn, M. and M. Frank (2019), ‘The Pervasive Role of Pragmatics in Early Language’, Annual Review of Developmental Psychology, 1: 223–49. Brandt, P. A. (2011), ‘What Is Cognitive Semiotics? A New Paradigm in the Study of Meaning’, Signata, 2: 49–60, https://journals.openedition.org/signata/526. Cannizzaro, S. (2010), ‘On Form, Function and Meaning: Working Out the Foundations of Biosemiotics’, Hortus Semioticus, 6: 40–52. Cannizzaro, S. (2013), ‘Where Did Information Go? Reflections on the Logical Status of Information in a Cybernetic and Semiotic Perspective’, Biosemiotics, 6 (1): 105–23. Cannizzaro, S. (2014), ‘Transdisciplinarity for the 21st Century, or “Biosemiotics as Systems Theory”’, Cybernetics and Human Knowing, 21 (3): 45–59. Carnap, R. (1948), Introduction to Semantics, Cambridge, MA.: Harvard University Press. Catalano, T. and L. Waugh (2020), Critical Discourse Analysis, Critical Discourse Studies and Beyond, New York: Springer. Chomsky, N. (1957), Syntactic Structures, The Hague: Mouton. Clark, E. V. (2017), Language in Children, London: Routledge. Cobley, P. (2016), Cultural Implications of Biosemiotics, Dordrecht: Springer. Cobley, P. and P. J. Schulz (2013), ‘Introduction’, in Cobley and Schulz (eds), Theories and Models of Communication, 1–16, Berlin: de Gruyter. Danesi, M. (2011), ‘The Semiotic Foundations of Knowledge: Remembering Thomas A. Sebeok’, in P. Cobley, J. Deely, K. Kull and S. Petrillil (eds), ‘Semiotics Continues to Astonish’: Thomas A. Sebeok and the Doctrine of Signs, 115–22, Berlin: de Gruyter. Fairclough, N. (1989), Language and Power, London: Longman. Favareau, D. (2010), ‘An Evolutionary History of Biosemiotics’, in Favareau (ed.), Essential Readings in Biosemiotics: Anthology and Commentary, 1–80, Dordrecht: Springer.

INTRODUCTION

13

Grondin, J. (1995), Sources of Hermeneutics, Albany: SUNY Press. Hall, J. A. and M. L. Knapp, eds (2013), Nonverbal Communication, Berlin: de Gruyter. Harris, R. (1981), The Language Myth, London: Duckworth. Harris, R. (2001), ‘Linguistics after Saussure’, in P. Cobley (ed.), The Routledge Companion to Semiotics, 118–33, London: Routledge. Hippocrates (1983), Hippocratic Writings, ed. A. Lloyd, Harmondsworth: Penguin. Houser, N. (2016), ‘Semiotics and Philosophy’, Cognitio: Revista de Filosofia, 17 (2): 313–36. Jakobson, R. (1960), ‘Closing Statement: Linguistics and Poetics’, in T. A. Sebeok (ed.), Style in Language, 350–77, Cambridge, MA: MIT Press. Jaworski, A. and C. Thurlow (2011), ‘Gesture and Movement in Tourist Spaces’, in C. Jewitt (ed.), The Routledge Handbook of Multimodal Analysis, 363–72, London: Routledge. ­Jones, P. E. (2007), ‘Why There Is No Such Thing as “Critical Discourse Analysis”’, Language & Communication, 27: 337–68. Kendon, A. (2004), Gesture: Visible Action as Utterance, Cambridge: Cambridge University Press. Kress, G. R. (2003), Literacy in the New Media Age, London: Routledge. Machin, D. (2013), ‘What Is Multimodal Critical Discourse Studies?’ Critical Discourse Studies, 10 (4): 347–55. Marais, K. (2020), A (Bio)Semiotic Theory of Translation: The Emergence of Socio-cultural Reality, London: Routledge. Marrone, G. (2014), The Invention of the Text, Milan: Mimesis. McNeill, D. (2007), Gesture and Thought, Chicago: University of Chicago Press. Moeschler, J. (2021), Why Language? What Pragmatics Tells Us about Language and Communication, Berlin: de Gruyter. Morris, C. (1938), Foundations of the Theory of Signs, (vol. I, no. 2, of International Encyclopedia of Unified Science), Chicago: University of Chicago Press. Norris, C. (1986), ‘Semiotics in Great Britain’, in T. A. Sebeok and J. Umiker-Sebeok (eds), The Semiotic Sphere, 229–51, New York: Plenum. Pablé, A. and C. Hutton (2015), Signs, Meaning and Experience: Integrational Approaches to Linguistics and Semiotics, Berlin: de Gruyter. Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Ruesch, J. and W. Kees (1956), Nonverbal Communication: Notes on the Visual Perception of Human Relations, Berkeley: University of California Press. Scarantino, A. (2017), ‘How to Do Things with Emotional Expressions: The Theory of Affective Pragmatics’, Psychological Inquiry, 28 (2–3): 165–85. Sebeok, T. A. (1977), ‘Ecumenicalism in Semiotics’, in Sebeok (ed.), A Perfusion of Signs, 180–206, Bloomington and London: Indiana University Press. Sebeok, T. A. (1991), Semiotics in the United States, Bloomington and London: Indiana University Press. Sebeok, T. A. (1998), ‘The Estonian Connection’, Sign Systems Studies, 26: 20–39. Sebeok, T. A. (2001a), Global Semiotics, Bloomington and London: Indiana University Press. Sebeok, T. A. (2001b), ‘Nonverbal Communication’, in P. Cobley (ed.), The Routledge Companion to Semiotics and Linguistics, 14–27, London: Routledge. Self, C. (2013), ‘Who’, in P. Cobley and P. J. Schulz (eds), Theories and Models of Communication, 351–68, Berlin: de Gruyter.

14

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Sperber, D. (2008), ‘Claude Levi-Strauss, a Precursor?’ European Journal of Sociology/Archives Européennes de Sociologie/Europäisches Archiv für Soziologie, 49 (2): 309–14. Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns, Boston, MA: Docent Press. Waugh, L., Catalano T., K. Masaeed, Do T. Hong and P. Renigar (2016), ‘Critical Discourse Analysis: Definition, Approaches, Relation to Pragmatics, Critique, and Trends’, in A. Capone and J. Mey (eds), Interdisciplinary Studies in Pragmatics, Culture and Society, 71–136, New York: Springer. Wharton, T. (2009), Pragmatics and Nonverbal Communication, Cambridge: Cambridge University Press. Xie, C., F. Yus and H. Haberland, eds (2021), Approaches to Internet Pragmatics, Amsterdam: John Benjamins.

­C HAPTER ONE

Communication Theory and Semiotics RICHARD L. LANIGAN

‘For all human beings, and only for human beings, language is the vehicle of mental life and communication’, says Roman Jakobson as he begins the task of suggesting just how communication theory and semiotics relate to one another as verbal comportment (1972; Holenstein 1974). While it helps enormously to view what is semiotic about communication and how semiotics assists our understanding of communication (Nöth 2014), the issue of the current moment is why they must be explicated together as we encounter our futures in the symbolic systems of our cultures (Whorf 1952; Ruesch 1967; Wilden 1972, 1987; Lotman 1994). The emergent answer is the human trait of wonder  – the experienced uniqueness of speech expression wherein the perception of language founds an ability to express the mind’s consciousness (Fuchs 2018). The moment and event of wonder (ékstasis) are our own human embodied mental life (consciousness) and the awareness (experience), as Charles Peirce (1902: CP 3.621) would say, that others also have this communication capacity to intertwine expression as perception (mental causation) in the modality of discourse tropes of meaning (Shapiro 1988; Lotman 1990; Rota 1997; Ruthrof 1997; Ehring 2011; Bondi 2012). Human beings have the capacity to first learn a language from others, yet use that very tool to identify both the similarity and the difference between one’s self and the other – as object or not-me person (Bühler 1934; Ruesch and Bateson 1951). In a process of auto-communication (Lotman 1991; Jakobson 1960b), the act of speaking is simultaneous with that of hearing and language is developed as performance (verbal and gestural). Yet this experience occurs after hearing others speak and seeing them gesture to us, when language is learned as embodied competence (expression and perception) (Britton 1939: 2; Ruthrof 1997, 2000, 2020). Competence we know as culture (our entry into all semiotic systems) just as we understand performance to be the awareness of self-identity or mind (our entry into all communication systems: linguistics, logics, mathematics). Because these two basic sign-systems of mind and culture are simultaneous in human beings, i.e. in logic and language (Sapir 1931; Ruesch 1953; Alexander 1988), human symbolic capacity stands apart from machines (they must be built and maintained by humans) and apart from animals (who lack the ability to move referents in space-time [mediate by symbolic forms]) (Wiener 1948a, b, 1950; Fuchs 2018: 22, 167, 170; Lanigan 2013). It is Ernst Cassirer (Urban 1949b; 1957; DePalo 2010; Chang 2013; Lanigan

16

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

2017a, b) who proposes that the phenomenology (conscious experience) of being human consists in three capacities: 1. The ability to perceive objects (the world of icons investigated by natural science); 2. The ability to perceive expressions (the world of indices explored by human science); 3. ­The ability to communicate by speaking (the world of symbols created and learned by human beings). Human beings have the unique capacity to combine these abilities as the ‘art and science of . . . ’ (Ruesch 1953; Eco 1976). As Fuchs (2018: 22) summarizes, ‘we do not perceive stimuli or images, rather gestalt units, meanings and affordances. Perception avails itself of the mediating process, in order to establish a direct relation to things – in other words, a mediated immediacy.’ The paradigm tool for exploring this art and science combination is the Discourse Model of Communication (symbolic meaning as the medium for both Addresser and Addressee) that is prior to, and contextualizes, the Exchange Model of Information (signal signification as the channel between either Sender or Receiver). The discourse model is generally known by the disciplinary name Communicology (Hunt 1970; DeVito 1978; Lanigan 1992, 2018a), while the exchange model is called Informatics, often more a ‘computer metaphor or metonymy of the mind’ vaguely labelled as inorganic ‘computer science’ or organic ‘cognitive science’ (Jakobson [1962] 1981; Marcus 1974; Gleick 2011).

COMMUNICOLOGY MODELS While it is a commonplace assumption that ‘communication’ and ‘information’ are synonymous, we must realize that this is a scientific vulgarity comparable to saying today that ‘chemistry’ is synonymous with ‘alchemy’ (Gleick 2011). Our analysis will not explore the mathematics of ‘information theory’ (or its statistical cousins ‘computer science’ or ‘cognitive science’), inasmuch as ‘information’ is simply a numerical-indexical referent for an object. And, only human beings create such symbolic reference systems (linguistic communication) for objects in the natural world within which they live (Lanigan 2013). Making this judgement is an example of what we shall explore: how human beings observe (mediate) their own behaviour (immediate) in the moment (time binding) as a meaningful event (space binding). Moments account for our consciousness and events record our experiences as signs (sēmeion). Moments and events are a continuing process of observation known simply as intentionality (awareness of consciousness as the experience of living). Human development founds a series of learning stages wherein the perception of others constitutes a sense of similarity and difference from which we abstract an idea of who (nóēsis) and what (nóēma) we are (Dreyfus 2009). Importantly, the combinatory idea (symbolon) guides our understanding of moments (index) as past, present, future and of events (icons) that name [ónoma] what is taken as mediated self–other (measure) and immediate same–different (magnitude). The best example in the human Umwelt is (1) the transmutation from real sound (speech) to symbolic sound (language), which we know as speaking and writing, and (2) the translation of one symbol system into another, which we call linguistics and mathematics (Holenstein 1974: 159; Powell 2017). As Ernst Cassirer (1942: 3) summarizes, ‘Speech and writing are considered as the origin of measure; for they are characterized above all by the ability to fix the fleeting and variable and to remove it from the accidental and arbitrary’.

COMMUNICATION THEORY AND SEMIOTICS

17

The human observer model As in any scientific publication, I shall use figures to condense and present large amounts of factual material, the evidence from which we make judgements. In particular, figures are intended to visually simplify complex relationships of process and provide tabular data categories for further exploration in the references given. In addition, I should note that there is only one comprehensive theory of human communication (Roman Jakobson); yet, there are a great many models of aspects of the theory. A theory is a comprehensive narrative account that purports to explicate a complete depiction of a process used for future prediction. A model is a selective account (often just pictorial) of a few variables abstracted from the theory. The only complete model for a theory is the textual narrative explicating it (what you are reading right now; or for example, see Alexander 1988). Some parts of a narrative can be condensed into content tables; other parts are connected by forms of dynamic relationship requiring visual figures. Please be aware that just as a text is a continuous narration, sequential figures cross-refer to one another and are a progression of thought complexity grounded in logic. The Observer Model of Human Communication was proposed by Jürgen Ruesch and Gregory Bateson (1951). Their book articulates a human-centred view of science discussed at the 1950s New York Macy Conferences by a group that included Bateson, Ruesch, Margaret Mead, Norbert Weiner, Ernst Cassirer, Chiam Perelman, Alfred Schütz, Charles Morris, Claude Lévi-Strauss and Roman Jakobson (Lanigan 2018b). All were concerned to account for the full range of observation from the mind inside the human to the body outside, from psychiatry (Ruesch) to cultural anthropology (Bateson), from logio-mathematics (Weiner) to linguistics (Jakobson), from rhetoric (Perelman) to poetic (Jakobson) in discourse – all combinatory symbolic forms (Lanigan 2017a, 2018b; Rota 1997). Ruesch and Bateson (Figures 1.1 and 1.2) suggest that human observation begins ‘within one’ as the embodiment of consciousness designated as Intra-Personal communication (Level I). This level of language (langage) defines the symbolic systems (codes) learned by a human being as a result of being born into a given culture. Observation is a search for similarity that is constricted by tone of voice heard (gērys). The child learns by measuring the future channel utility of particular sounds (verbal functions) and their contextual facial expressions (nonverbal functions). The child begins to experiment with the magnitude of expressive production (rhetoric) of these sounds and the response they get (network levels as a medium of communication). These initial events lead to Inter-Personal behaviour (Level II) where ‘one to one’ dialogue emerges. The priority for the child is the discovery of the Self, finding one’s own voice as speaking (parole). The task is sometimes difficult in a bilingual family, but it is universally difficult with parents whose private language ‘baby talk’ is useless and confusing compared to the soon-to-be-needed ‘public talk’. Interpersonal communication is distinguished by the ability to recognize the ‘one-to-one’ difference (perspectives) between the Self and the Other person as a voice of embodiment in conversation. Public talk (langue) engaging other people in a social group constitutes Social communication (Intra-Group Level III) as an intra-group function. At this stage, culture makes a noticeable entrance into conversational events in the form of social preference for human actions and their names. Cultures have a preference for how groups are organized and graded with respect to activity. Some cultures prefer an egocentric focus (individual/ identity by rule, e.g. gender designation) where people are an aggregate collection (e.g. United States; ‘one to many’, ‘a bag of marbles’). Other cultures prefer a sociocentric focus

18

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.1  Ruesch and Bateson discourse model of communicology.

(group/identity by role, e.g. family or professional) where there is an organic collection (e.g. PR China; ‘many to one’, ‘a pile of salt’). When and where groups begin to encounter one another, we have the discovery of human difference manifest as wholistic, unified communicative preference (discours) when people engage in group-to-group interaction. Cultural communication (InterGroup Level IV) is the highest level of complexity inasmuch as social preference expands to cultural choice adherence. Differences between persons become differences among groups by preferred affordance (measured magnitude). Groups organize by choice of rule (egocentric; space binding) or choice of role (sociocentric; time binding) for making analogue choices about ‘normal, acceptable, good, bad, evil’ behaviour (cultural mores). Culture becomes a reality guidepost for ‘many to many’ future interactions. Examples are commonplace on both a positive (‘peace’) and negative (‘war’) note. A word of caution here, the comparisons and contrasts that I depict are preferences meaning that all cultures have affordances for individuals and groups. Their practices are simultaneously social and cultural, so that choice is a matter of habitual practice carried

COMMUNICATION THEORY AND SEMIOTICS

19

FIGURE 1.2  Category description of Ruesch and Bateson model.

forward as anticipated future actions. Echoing Charles Peirce, Norbert Wiener (1915: 570, 574) long ago argued about meaning functions that ‘the life of every branch of mathematics lies in a habit’ and those habits are perceived first as ‘words not symbols’. Communication as discourse is an attempt to perceive those actions (words – embodied speaking) as expressions (symbolic formations; Gestaltung). For Cassirer, the habits of culture are the source of all symbolic forms: language, art, myth and religion (Lanigan 2017a, b). Peirce (1906: 196–7) calls such forms the commens of mind constituted by communication or simply commind as ‘double consciousness’ (1897: CP 1.153).

The human discourse model Linguistic anticipation is our orientation to the unknown (ágnōstos), to the future. Expression is an anticipation that someone will engage in perception. Our tool use of language assumes a hierarchy of utility for such expressive and perceptive behaviour (Figure 1.3). These expressions are typically spoken as oral messages where we get our sense of connotative or informal meaning, or they are written texts with a code or style that suggests a more formal denotation. The result is our sense of personal narrative expression (my story, spoken by me) compared to a public narrative perception (my history, spoken by others). As communication forms, these functions can be combined in autobiography (story of my history = same-self = discourse trope) or biography (history of my story = other-different = discourse figure). These expressive levels or registers (Figure  1.3) have a definite social and cultural value implication about us as a person.

20

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.3  Discourse communicology model.

We use the register to orient our communicator perspective as either the person speaking (Addresser) or as the one listening in a communicatee perspective (Addressee). How the speaking is encoded or nominated by the Addresser (metaphor = A—as—B) is then decoded or predicated in reverse order by the Addressee (metonymy = B—is—A). This binary analogue interaction we tag as metaphor in the use of analogy (Ruthrof 1997: 172–4). A non-technical account of this concept of a logic constituted by founding tropes in speech is the semiotician Yuri Lotman (1990: 36–53) and the mathematician Rota (1997: 188–9). The more detailed explication is Ehring (2011) and the relation to Hegel’s foundational logic as the semiotic dash [—] is Comay and Ruda (2018). Nonetheless, keep in mind that Aristotelian logic per se derives from the consideration of oral rhetoric; hence, tropes become informal (rhetoric) and formal (logic) laws of analogy to which we adhere as beliefs (Lanigan 2014, 2019b, c; Houser 2020).

COMMUNICATION THEORY AND SEMIOTICS

21

However, before we can explore this founding trope logic of chiasm in its dynamic process (A : B :: b : A = A as B, if b is a), it is helpful to preview the complexity system of communication upon which we are embarking. Figure 1.4 summarizes where we have been (choice : preference :: habit : affordance) and where we are going with logic and semiotic models. We begin with Husserl’s (1948: 115, 116, 154) concept of the Founding Logic [Fundierung] of Chiasm, more commonly called tropic or rhetorical logic (Lotman 1990). Figure 1.5 gives the working definitions that correlate to the semiotics of Charles S. Peirce ([1866–1913] 1931–58) that also will guide us. For context, Figure 1.6 illustrates the Scholastic trivium definitions of logic, rhetoric and grammar that are the historical concept base for all contemporary theoretical work

FIGURE 1.4  Complexity description of semiotic systems.

22

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.5  Etymology of C. S. Peirce logic typology.

in communicology, but especially for Peirce, Husserl, Heidegger and Merleau-Ponty. Other popular authors who worked in this tradition are Umberto Eco (1976) and Michel Foucault (Lanigan 1992). Noting the ‘rhetorical branch of linguistics’, Roman Jakobson also uses this foundation logic to specify the discourse model of communication as the symbolic form (‘text’) of tropes (speaking) and figures (writing). As noted in Figures 1.3 and 1.6, for Jakobson the master tropes are metaphor (qualitative meaning) and metonymy (quantitative signification) as interactive combinatory elements (poetic function) in every register of communication (metalinguistic function); this is also Heidegger’s analysis of poiēsis (1929: 63; cf. view of Heidegger in Britton 1939: 170).

The Basic Model of Georges Gusdorf Gusdorf was a philosophy professor in charge of exam tutoring at the famous École Normale Supérieure (ENS) in Paris. He began his career by succeeding Maurice MerleauPonty in the job. He is famous for writing a small tutoring book called Speaking [La Parole]

COMMUNICATION THEORY AND SEMIOTICS

23

(1957) used by generations of French scholars, most notably Louis Althusser and Michel Foucault (Bondi 2012, 2014). The book outlines the philosophy of speech foundational for the analysis of all human sciences (Buyssens 1943, 1968; Swiggers 1987; Mulligan 1990; Rollinger 2009, 2010). Gusdorf (1966–93) also authored a twelve-volume history of the human sciences during his lifetime for which this book is the key. The basics of this wonderful little book about speaking are summarized in Figure 1.3. The figure displays a hierarchy of perception, who a listener encounters in speech. Recall this is a decoding process in which the trope of metonymy is operating with a whole analysed by its parts. The cultural criterion of language [langage] is used to identify the speaker’s code and the judgement value for this evidence is difference. The conversational listener is always thinking by analysis, i.e. divide into parts, assemble into wholes. For example, does this person speak my language? If I speak English, I recognize English, otherwise the speaking is noise. But then this discovered similarity of code continues to suggest difference, e.g. the tone or accent is new to me, some of the words are not known to me, etc. This is group identity; the speaker is using American English; I know because I learned a different version, British English. So, we, speaker and listener, are now the same, but different! We share a cultural logic in language (code), but our grammar is showing signs of difference (context). This American is using lots of personal (informal) references (‘I, me, you’) that make me slightly uncomfortable (I prefer more formal reference [contact], that is what ‘one’ does). Social usage [langue] is suggesting this American probably does not have much in common with me. I start to perceive that a great deal of the speaker’s vocabulary is adjectives, making most nouns incoherent by exaggeration! I have the British preference for specific nouns and no adjectives. My exception for conversation [discours] is not being met. I am becoming very aware of how I speak [parole] and it is not like this! Now, my oral experience is helped or hindered by the matching nonverbal codes that contextualize the situation: how close is the American standing (Proxemics), how long will this take (Chronemics), can I tolerate the way his clothing looks (Ocularics), will this person ever leave (Kinesics), do I have to shake hands on departure (Haptics), can I tolerate the accent (Vocalics), can I take the smell any longer (Olfactorics)? Every culture has exacting affordance norms for the non-verbal codes that are not as flexible as the verbal codes, especially when social preference operates inside the cultural context (the source of class, caste, etc.) (Ruthrof 2000). Figure 1.4 helps correlate the issues we are analysing. Our example took us through the Gusdorf discourse model and introduced us to a working example of communication network levels. For analytic purses, Husserl and Jakobson formalize our levels of conscious experience as founding relations of implication. Humans organize these experiences as levels of meaning or what Peirce calls typologies (universal categories) consisting of types (particular wholes), their tokens (particular parts) and their tones (singular attributes). In order to interpret the relationships of all the connected levels (gestalt), Peirce uses the traditional division of logic into its types (according to its semiotic function in communication): (1) ADduction (Interpretant), (2) ABduction (Symbol), (3) DEduction (Index) and (4) INduction (Icon). We need to recall that ever since the Greeks, logic and language have used triadic (three parts; think grammatical subject, verb, object) relations of process that are called appositions (each part related to the other two). If one of the parts is the context, then the

24

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.6  Scholastic trivium discourse model.

other two become an opposition. However, it is also critical to remember that appositions also have a context which is the definition of a logic (qualitative) or a mathematic (quantitative) called a quadratic (four attributes; think a ‘window frame’ 2 × 2 matrix). The ‘Peirce Semiotic’ in Figure 1.4 is an example. While Figure 1.4 gives us a general summary of our analysis so far, Figure  1.5 provides an accurate definition of Peirce’s usage names and definitions over time so that his terminology is not confusing.

The trivium model of the Scholastics Scholasticism was the dominant philosophical orientation in Europe from about 1100 to 1700 CE and was centred mainly at Paris, France, although there is a heavy German influence including Cassirer, Husserl and Heidegger. The most notable persons were Albertus Magnus and his assistant, Thomas Aquinas. These scholars were focused on two goals: 1. How do you come to understand things (philological analysis)? 2. How do you communicate this knowledge to others (logical analysis)? ­ hey decided the key was listening to orators and reading books for evidence, then T writing out summaries that could be orally presented (lectures). Of course, these written synthetic summaries and critical commentaries became the new (text) books to be read and then discussed. How to do this efficiently? They devised a four-part

COMMUNICATION THEORY AND SEMIOTICS

25

method that applied to speaking (lecture format) and listening (note taking) called explication de texte: 1. Text – Literal Meaning: Factual Summary of the content of a text (oral [lógos] or written [lexis] evidence); Peirce’s term (see Figures 1.4 and 1.5) for this category in Logic as expression is Case, and in Semiotic as perception is Symbol (Immediate/ Intentional Interpretant; Data = what is given). 2. Grammar – Moral Meaning: Description of what is perceived by typology (argument type, token, tone); Peirce’s term for this category in Logic as expression is Result, and in Semiotic as perception is Index (Dynamical/Effectual Interpretant; Acta = what is done). 3. Rhetoric – Allegorical Meaning: Reduction of the description to definition as semantic tropes or syntactic figures; Peirce’s term for this category in Logic as expression is Rule, and in Semiotic as perception is Icon (Final/Communicational Interpretant; Verba = what is said) (Charaudeau 1983: 77–80; Lotman 1990: 36–44). 4. Logic – Anagogical Meaning: Interpretation by evaluation of the definition for the future (hermeneutic judgment); Peirce’s term for this category in Logic as expression is Representamen, and in Semiotic as perception is Interpretant (Commens/ Commind; Capta = what is taken). A contemporary example of the method occurs significantly in the work of Maurice Merleau-Ponty and Michel Foucault (Lanigan 1992: 89, 119, 127ff.). Of course, anyone who has been to law or medical school encounters this method as learning by case studies, the art foundation of doing science (Hunt 1970; Houser 2020). I should also note that the case method is foundational to the Ruesch (psychiatrist) and Bateson (anthropologist) model (Figure 1.1) where r/receiving (Self) is literal meaning, c/channel (Other) is moral meaning, s/sending (Same) is allegorical meaning and e/evaluation (Different) is anagogical meaning, and each semantic function is dominant in one of the communication Levels (I to IV) of signification (Hunt 1970). The tabular information in Figure 1.6 illustrates the basic organization of the Trivium as both: 1. A general application of philological interest in the code used by the Addresser and the message understood by the Addressee in communication, whereas 2. The special application of logical (dialectic) interest is the medium of contact created by the conjunction with the context for the message. The right-hand column in Figure 1.6 notes the Jakobson communication elements and functions derived from the trivium, and Figure 1.8 illustrates those categories as a process.

THE DISCOURSE COMMUNICOLOGY OF ROMAN JAKOBSON Roman Jakobson was a polyglot whose main work takes up nine large volumes in English, French and several Slavic languages. Like others discussed thus far, he is at the centre of work on language, semiotics and their application to the human sciences based in linguistic theory. He brings the Ruesch and Bateson model (Figure  1.1) into the larger action perspective (praxeology) of the human sciences on the one hand; yet, he also focuses on the place of language within the hierarchy of those sciences of life: Linguistics, Semiotics, Cultural Anthropology, Economics and Biology (Figure 1.7; tabular version is Figure 1.11).

26

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.7  Roman Jakobson communicology theory and applied model.

COMMUNICATION THEORY AND SEMIOTICS

27

Basic models of semiotics and communication All this is another way of saying that the rhetoric of philology founds the logic of philosophy. Jakobson’s Human Science Model provides a context or fame for his Communicology Model. The elements or content categories of his theory constitute the basic, unique parts of the human communication phenomenon. The elements and functions are found at each level: 1. Intra-personal – the thought process in our own mind where the active voice of consciousness is a simultaneous dialogue of Addresser (‘I’) and Addressee (‘Me’); 2. Inter-personal – the dyadic conversation between the passive voice of one Self and an Other person (‘You [thee] / You [thou]’); 3. Intra-group – the social dialogue of middle voice among three or more Similar people as the collective voice of a family or a work team (‘We / Us’); and 4. Inter-group – the interaction of mainly ethnic groups where cultures form and encounter one another as Different (‘They/Them’) ways of giving particular voice (a reality) to the experienced natural world of actuality (Lanigan 2015c).

Tropes: Complexity of elements and functions What makes the communication elements discrete is their function, i.e. how each element is perceived by the Addressee as an orientation, an indication of the future consciousness (intentionality) of the Addresser. All human encounters begin with such nominated expectations – future predictions that are meant to be descriptions of behaviour with which we are familiar already in the present situation. The basic comparison (what Husserl calls founding implication) that we engage with human communication is the result of a shared code, both as expressed intentionality (Addresser representamen, to use Peirce’s term) and as perceived interpretation (Addressee interpretant). Figure 1.8 gives a static picture of this process event. The encoding process is the speaker’s attempt at intelligibility for the present, while the listener’s decoding process is the counter-attempt at relevance for the future. The whole domain of misunderstanding (ambiguity of past referents) comes about because the speaker’s code-to-message construction sequence must be reversed by the listener as a message-to-code deconstruction. Unfortunately, the reversal brings along with it a tendency to now view the message/code (listener view) as reflexive (message different from listener’s view) and reflective (code same as listener’s view). Most adults discover this referent confusion phenomenon in conversation when using pronouns, only to later (future) discover the assumed (past) noun referents of speaker and listener are different. The semiotic point here is that the code gives us an expression type, but the message gives us a perception token, and only the comparison of both the type and the token allows a symbolic judgement of tone. Indeed, the semiotic function of logic is the combinatory foundation of unity in difference (see the Fundierung column in Figure 1.4). While we have explored the complexity of message and code as elements and functions, we must now consider several more levels of complexity. This amounts to saying that we must take the basic pairs of Code /Message and combine them with Contact/Context [note the way elements and functions are illustrated in Figure  1.10]. Then, we must consider that all of them can interrelate as pairs (oppositions), triads (appositions) and

28

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.8  Jakobson communicology process model.

as quads (reverse oppositions and appositions) with equally interrelated functions. The static list of elements and function in Figure 1.7 becomes a process matrix as illustrated in Figure 1.9. For convenience, the matrix is called a chiasm and is represented by the logic symbol . It is helpful to gather the various parts of our analysis thus far into a summary that is visually static in Figure 1.10. The manner in which the communication elements and functions work from the practical perspective of thinking (logic), writing (grammar) and speaking (rhetoric) constitutes Figure  1.11 (see Figure  1.7). Note here that Figure  11

COMMUNICATION THEORY AND SEMIOTICS

29

FIGURE 1.9  Jakobson chiasm matrix model.

depicts the integration of subjectivity (elements) and intersubjectivity (functions) so that we become aware of what is called double articulation or ‘double judgement’ (Peirce 1893: CP 4.52–3). This is a way of saying that while in the actual world of experience, there are not just two voices talking (speaker and listener) but four reality referent voices that dialogue as Addresser (one embodied speaker [voice-1] and listener [voice-2]) and Addressee (an other embodied speaker [voice-3] and listener [voice-4]). We know this process as two people talking to one another while each person is thinking about what the other person is saying and doing. Karl Jaspers calls this human conjunction of minds the encompassing. The four-part interaction is called the Perspectives Model of Communication; a detailed account is Lanigan 2015, and the logic is detailed in Peirce (1903: CP 1.347 and 1.370). The chiasm square illustration in Figure 10 is an adequate visual representation of the four perspectives for most purposes. For our immediate need, the following

30

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.10  Semiotic phenomenology matrix model.

quadratic schema of perspectives is a sufficient explication of the talk/thought semantic field as in (1): (1) Open : Blind :: Hidden : Unknown

Knowing (public) : Imagining (overt) :: Imagining (covert) : Thinking (private)

  ⬅ Known to Self and Others ➡ OPEN    : ⬅ Not Known to Self yet Known to Others ➡ BLIND     :: ⬅ Known to Self yet Not Known to Others ➡ HIDDEN     : ⬅ Not Known to Self and Others ➡ UNKNOWN

COMMUNICATION THEORY AND SEMIOTICS

31

FIGURE 1.11  Communicology matrix of elements and functions.

The chiasm model of discourse For most readers, it is enough to know that the founding trope logic of chiasm is a simple ratio expressed as: 1. A : B :: b : a Our ‘common sense’ version of the ratio is implication thinking by analogy (Rota 1997), whereby: 2. A as B if b is a This analogical implication thinking (‘if’) is the Husserl and Jakobson source for metaphor (‘as’) and metonymy (‘is’). Or, we should note its traditional French trope version of le même et l’autre (Descombes 1979): 3. Self : Other :: Same : Different

32

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

The linear illustration is deceptive; keep in mind that (1) the four terms make up a ‘square’ (as in Figure 1.10, but see the Note in Figure 1.9 for visual help) and (2) the four are a ‘mirror’ reversal (vertically and horizontally; Jakobson’s poetic function; see [1962] 1981) such that ‘Self: Other’ outside the mirror is reflected backwards and upside down inside the mirror as ‘Same: Different’ [the symbol :: would be the edge the mirror glass] (Sonesson 2015). You experience this visual chiasm when you touch your face while looking in a mirror and have trouble distinguishing ‘right side’ from ‘left side’, up from down, since you forget to first pick a perspective of ‘inside’ or ‘outside’! There is an extended cultural explication in Lanigan (2015c). The concise point is this: a quadratic (four parts) is a combination that allows any one term (one part) to be the context for the other three terms (three parts). The context has a valence (positive/negative) so that the remaining triad (three parts) is in an apposition. Any one of the three parts can take the opposing valence (negative/ positive) as a criterion to select compatibility (similarity) with one of the two terms remaining as a dyad (making two parts). The chosen part now has the same valence as, at least, one other part, leaving the last one part to have the reverse valence – by force of division this part stands in opposition and is a monad. Thus, we have philosophical terms of convenience, i.e. monad (singularity, ‘one’; Position; Peirce’s Firstness as spontaneity [1892: CP 3.422]), dyad (particular, ‘two’; ‘Some’; Opposition; Secondness as dependence) and triad (universal, ‘three’; ‘All’; Apposition; Thirdness as mediation). Do not forget their context which provides the quad (universal particular, ‘fourth’; Peirce call this an instance [1905: CP 4.537]). An analysis parallel to that of Peirce, but in comparison to Husserl, is Ruthrof (2020). Cultural preferences for the positive/negative valences of conduct create semantic meaning (metaphor), while the quantitative syntactic signification (metonymy) creates our pragmatic choices for future behaviour (affordance, comportment). We just used the cultural tropic logic at work here, i.e. deictic expressions/perceptions such as inside/ outside, right/left, up/down. Each deictic expression (one part) displays (is the context for) an apposition selection (one part) of the remaining opposition choice (two parts). Precisely this semiotic process allows a human being to make a choice, yet ‘change their mind’ by choosing an alternative context (a new quad, i.e. changing both logical levels and semantic registers). Recall we started with a quad and choose one part; this means we have three choices left, and we assign them a place in our memory called present, past and future. You learned this process as grammar (rules for listening) then learned to control it as rhetoric (rules for talking), all the time assuming it was the correct logic to use (rules for thinking). The semiotic creativity of human beings is this capacity to be wrong (Peirce’s Fallibilism), change our mind and create a different future, because there is an absolute continuity (Synechism) and probability (Tychism) for our conscious experience (Lanigan 2014, 2018b, 2019a; [see the levels of apperception in Figure 1.13]). Jakobson’s use of the chiasm discourse model is heavily contextualized by its use among French scholars, most notably Merleau-Ponty and Kristeva on issues of perception, and Foucault and Lacan on issues of expression (Kristeva 1974, 1981; Lanigan 2018b, c; Bondi 2012, 2014; Lacan 1956). Following Edmund Husserl’s work on logic (Holenstein 1974: 145) and Claude Lévi-Strauss’s work on chiasm in semantic exchange (Lévi-Strauss 1958: 228; Lanigan 2017b: 210), Jakobson formulates the chiasm in linguistic terms as Stawarska (2015: 185) summarizes: ‘Instead of a distinction between deep structural level and the surface phenomenon of speech, there is a crisscrossing or a chiasm between expression and language systems.’ The crossover

COMMUNICATION THEORY AND SEMIOTICS

33

poetic function occurs as the encoding/decoding reverse conjunction (see Figure  1.8) between metaphor (the relation between substance and attribute, often called ‘content’ nomination) and metonymy (the relation between whole and part, often referred to as ‘form’ predication). The result is a chiasm implication model defined as:

Substance : whole :: part : attribute Herein metaphor is a semantic (trope) relation of substance and attribute that is mediated by metonymy as a syntactic (figure) relation of whole and part. Metaphor is the paradigmatic/synchronic (‘vertical’ now/here) process of selection, substitution and similarity. Metonymy is the syntagmatic/diachronic (‘horizontal’ there/then) process of combination, contexture and contiguity (Holenstein 1974: 139, 159). Jakobson ([1962] 1981) shorthands the reversible process explication by calling it simply poetic function (transmutation; see Figure  1.3) and metalinguistic function (translation) – the combination is synonymous with the Greek aitia process concept. Note the hypercode here, the phonology of the word aitia itself is an iconic palindrome chiasm: a - i — t — i - a. Sebeok (1972: 17) argues in agreement, affirmed by Deely (1994: 71), that Jakobson’s poetic function (semantics/temporality) and metalinguistic function (syntactics/spatiality) are a coding capacity for judging logical ‘sameness or difference’ (chiasmatic exchange) that is ‘exclusively human’ as a biological boundary between the human and animal Umwelt (see Figure 1.7). The boundary is manifest as a habit in human language by Sapir’s (1944: 23; Sebeok 1972: 23) concept of semantic grading wherein any process category (capacity) is marked by analogue degree differences of both ‘mores’ and ‘lesses’ replacing a ‘dichotomy with a continuum’, e.g. the English static case set: ‘good, better, bad, worse, of average quality, interesting’ or causative case set: ‘to fall, to fell’ (1944: 125, 155). Another example is the contemporary German da, meaning by analogue abessive case set: ‘not-here, there, here, because, in-order-to’. Such a chiasm model illustrates the complete integration of the trivium categories of logic, rhetoric and grammar as the ‘immediate inference’ of communication per se. Lanigan (2017a, 2017b) is a summary of this long tradition, but for the technically minded Figure 1.12 gives a summary of its use in semiotics, especially the popular Greimas Semiotic Square that is built upon the Scholastic version of Aristotle’s aitia [αἰτία] metaphysics, i.e. (1) explication by ‘mental causation’ implication (material, formal, efficient, final answers [thematics] to questions [problematics]) and (2) explication by enthymeme as a chiasmatic ‘syllogism’ (Peirce 1893: CP 2.450; Lanigan 1974, 1995). For an applied example of the Greimas theory and model, see Lanigan (2019d). These explications are used by Peirce to account for his ‘argument cycle’ (Figure 1.5; Lanigan 1974, 1995). Aristotle’s aitia is also tagged with the Husserl/Jakobson ‘laws of foundedness’ (Figure 1.12). Figure 1.13 summarizes the over-reaching analysis made by Franz Brentano, Edmund Husserl, Anton Marty and others with regard to the function of apperception in the chiasm structure (Rollinger 2009, 2010). I should note that Jakobson’s theory of marked and unmarked terms functions semantically as this chiasm relation, thereby providing a systematic account of Karl Bühler’s (1934) analysis of anaphoric and cataphoric deixis (DePalo 2010; Chang 2013; Lanigan 2018b). Last, Figure  1.14 summarizes the basic convergence of the classical and medieval view of the trivium as it bears on Jakobson’s linguistics reflecting the influence of Husserl and converging with Merleau-Ponty’s enduring work on chiasm when he prematurely died at age fifty-four in 1961 (Lanigan 1992: 119).

34

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.12  Matrix of the logic and semiotic squares.

MODELLING THE FUTURE: INTELLIGIBILITY AND RELEVANCE A discussion of the relation between chiasm and the combinatory future of communication theory and semiotics necessarily begins with Maurice Merleau-Ponty. He takes up Martin Heidegger’s discourse definition of human Being as a projection (Geworfenheit; ‘thrownness’ = Greek symbolon; Da—Sein, ‘there—Being’) into the future (the ‘law of projection’ in Peirce (1892: CP 3.419; Ijsseling 1970; Rota 1997). Merleau-Ponty (1964/1968: 170; Wayne J. Froman 2005) offers us these working notes: ‘a new effort of expression’, ‘Being in Heidegger’s sense’ [Dasein], ‘which, apprehended by philosophy in its universality, appears as containing everything that will ever

COMMUNICATION THEORY AND SEMIOTICS

35

FIGURE 1.13  Chiasm process matrix for apperception and apposition.

be said, and yet leaving us to create it’ (Proust): it is the λόϒοϛ ένδίάϴϵτος [lógos endiathetos; the speech that remains within; private narrative, interpretive] which calls for the λόϒοϛ προϕορικόϛ [lógos prophorikos; the speech manifests without; public narrative, revelatory] – –. (my bracket inserts and Greek transliteration to Latin and English) We already know this invoked semiotic methodology of existential discourse as phenomenology and revelatory discourse as hermeneutics, respectively, the Scholastic lógos and lexis. The simultaneous semiotic combination (metaphor) and division (metonymy)

36

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 1.14  Trope logic in rhetoric and semiotic.

COMMUNICATION THEORY AND SEMIOTICS

37

is precisely Jakobson’s use of chiasm, captured so well by Merleau-Ponty’s reference to Proust. Figure 1.15 provides a contextual view of the binary analogue (unity of division in combination) operation of everyday human communication, whether as encoding, decoding or both together as the hypercode allowing simultaneous transmutation and translation: Symbolicity.

­Binary analogue model of communicology Husserl’s model of perception (core—field—horizon) allows us to see the existential core (Self) as opposed to the revelatory field (Other), and the analogue boundary of the horizon (Same/Different; lesser vs greater scale) that moves from low to high intelligibility (left-to-right), yet simultaneously is balanced by an equal degree of relevance (low to high) moving right-to-left. Compare Figures 1.8 and 1.9 to see the process dynamic at work. Figure 1.15 also incorporates the basic logic of founding implication by specifying the basic elements of chiasm perception (a : b :: B : A) and chiasm expression (A : B :: b : a) as they relate to the discourse hierarchy in Figure 1.3. The main communication components of Figure 1.15 are intelligibility and relevance as ‘implicational argument’: ‘The possibility of intelligible communication is the ultimate postulate of all thought, all knowledge. It itself cannot be explained but is the presupposition of all knowledge and science’ (Urban 1929: 46). ‘Mutual acknowledgment on the part of transcendent minds of common meanings and values is as much a condition of communication and genuine knowledge as the independent being of the meanings and values thus acknowledged. Transcendence of both Object and Subject is presupposed’ (Urban1949b: 147). Yet again, Peirce’s commind is affirmed by Urban’s description of ‘double transcendence’ that echoes Husserl’s theorem that ‘Subjectivity is Intersubjectivity’. Intelligibility is a human foundational belief (performance; depiction) in discourse meaning (lógos endiathetos): ‘Now, there are three “well-nigh invariable beliefs” certainly necessary propositions of intelligible communication, namely, that I exist and others like me, inhabiting a world.’ ‘They are not the outcome of reflection and communication, but their co-implication’ (Urban 1929: 47–8; 1939: 634; 1949a: 144). Encoding is our shorthand for naming discourse (vocabulary selection that nominates) which creates reason [lógos in Figure  1.15], the capacity to be understood, i.e. allows others to comprehend our meaning [páthos]. Decoding is the reverse process leading to a determination, a judgement of reasonableness [eulógos]. ‘The condition of intelligible communication is, indeed, logical form – an intelligible logic is one that must be interwoven with our natural language’ (Urban 1929: 122–3). As Heidegger (1929: 136) affirms, ‘lógos mean thinking. Lógos means the word, discourse, and legein means to discourse, to talk’. Relevance is a human foundational belief (competence; prediction) in discourse signification (lógos prophorikos): As William James suggests, ‘we can choose which way of thinking to adhere to and which to disregard’ (Schütz 1970: xiv). Here, decoding is the parallel shorthand for implication (articulation selection that predicates). In this regard, Schütz (1970: 70) suggests a model for relevance which moves perceptually from: 1. Topical connection to the point at issue [páthos], then to combinatory comparison of 2. Interpretation of meaning [eulógos], and then 3. Determination of value [ḗthos].

38

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

This is, of course, Peirce’s (1905: CP 4.538) sequence of type, token and tone, but more specifically, his use of the Scholastic term, proposition and argument (he specifies the ‘argument’ as the Greek dèlôma (‘revealed by speech’) which is contextualized by communication (1902: CP 3.621; 1906). Heidegger (1929: 63–8) and Lanigan (2015b) make this same analysis about dèlôma and its ‘gathering’ [naming] function in discourse by saying: ‘Ónoma in the wider sense is dèlôma tèi phönéi peri tén ousian: a revelation by means of [hearing tonal] sound in relation to, and in the sphere of, the Being of being’ (my bracket trans. insertion; Peirce defines tone exactly this way as hearing one’s unique embodied voice [gērys] where ónoma literally means ‘made symbolic by speech’).

FIGURE 1.15  Communicology matrix of intelligibility and relevance.

COMMUNICATION THEORY AND SEMIOTICS

39

FIGURE 1.16  Discourse trope model of communicology.

Trope matrix model of intentionality Recall the tropic structure of thinking, as Jakobson conceives it (Figure  1.9). With Figure 1.16, we flesh out the various concepts of tropic structure developed thus far in our analysis. The overall matrix arrangement displays the chiasm by grouping related explications of causal mentality (tropes). Thus, the levels of communication (Figure 1.1) are placed with those of trope description (substance, attribute; whole, part) as chiasm ratios (A—B; b—a) and the logic correlation to Aristotle’s syllogism is noted. The schema allows the easy recognition of types of tropic thinking: Metaphor/Paradox; Metonymy/ Dilemma; Simile/Irony; Asyndeton/Zeugma; Prosopopoeia/Synecdoche (see Figures 1.13 and 1.14 for definitions).

40

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

As such, each trope structure suggests a mediating analogue (more/less) combination (both/and) context that, in turn, frames an immediate binary (either/or) division by choice. While Figure 1.16 is a static illustration, it allows us to visualize why the horizon boundary in Figure  1.15 is dynamic as a moving analogue (lesser/greater) boundary between intelligibility and relevance meaning. For example, an Addresser may overcode the message (‘talk too much, too long’, etc.) causing the Addressee to undercode the message (increasingly ‘stop listening, loose focus’, etc.). Here, speaker presence turns into listener absence leading to misunderstanding, misinterpretation and the like. The speaker’s intentionality (aseity: conscious experience as self-sufficiency) fails as a symbolic form for the listener’s intentionality (ipseity: conscious experience as self-identity). While the symbolic form is shared as signification, the symbolic content constitutes a meaning ambiguity (good) or paradox (bad) as lived (Peirce’s commind). ‘The content of consciousness is a sign resulting from an inference’ (Peirce 1893: CP 5.313). Our analysis began in the wonder of human language, the symbolic capacity to be conscious of time and space, embodied and mediated, within our own mind and, yet, immediately present within other people’s mind. As such, in the claim to immediacy, we humans come to discover the chiasm transcendent moments and events wherein we may communicate: 1. [a] Depict (encode) and 2. [b] Predict (decode) the future of 3. [B] Communicology (consciousness), and 4. [A] Semiotics (experience) as simply a continuing discourse expression: A : B :: b : a. The subjective and intersubjective matrix founds discourse as culture, if nature is explicated in language by human beings. Communicology is human intentionality as an embodied three-part discourse relation: ‘something means something for somebody’ (Fuchs 2018: 37). Perhaps Charles S. Peirce (1905: CP 5.506) says it best as an abduction: ‘No communication [Rule] of one person to another [Result] can be entirely definite, i.e., non-vague [Case].’ Or even better, Martin Heidegger’s famous Dasein/Da—Sein chiasm:  ‘Discourse which Expresses Itself is Communication’ (Heidegger 1926: 211; 1939: 25, para. 27; Lanigan 2015b). Expressed as a Greimas Square (see Figure 1.12), we have: Dasein [not-here/Being] ↑ Mitsein [being with] Da— [there] ↑ —Sein [Being]; note that the chiasm crossover means both past ‘because-motive’ [Folglewelt] and future ‘in-order-to motive’ [Vorwelt] (Schütz 1970; Lanigan 1974), i.e. the human ability to ‘change your mind’. By way of conclusion, let me provide a summary definition of the Discourse Model of Communication: Communicology, as a founding logic, (1) names (ónoma) the wonder (ékstasis) of the human mind (logismós) that is the transmutation (sēmeion) and translation (symbolon) of the past (experience) into the future (consciousness) by meaning (discours, the voice of the Other) and (2) expresses (parole, the voice of the Self) itself (langue, the voice of Society) as communication (langage, the voice of Culture) in the continuing discovery (Mitsein) that both not-here as time (Dasein) and there as place (Da—Sein) signify understanding (dianoia) as being humane (logós/ḗthos/páthos). In short, we come to the semiotic Theorem of Human Communication: Speech as Language, if a Sign is a Symbol, more than a Signal.

COMMUNICATION THEORY AND SEMIOTICS

41

REFERENCES Text page references are to the English translation where one exists. Alexander, H. G. (1988), The Language and Logic of Philosophy, Lanham: University Press of America. Reprinting of revised and enlarged edn by University of New Mexico Press, 1972, of Language and Thinking: A Philosophical Introduction, Princeton: D. Van Nostrand Co. Inc., 1967. Bondì, A. (2012), ‘Le sujet parlant comme être humain et social’, Cahiers Ferdinand De Saussure, 65: 25–38. Bondì, A. (2014), ‘L’expérience de la parole: le thème du sujet parlante’, Texto ! Textes & Cultures, 19: 1–19. Britton, K. (1939), Communication: A Philosophical Study of Language, London, UK: Kegan Paul, Trench, Turner & Co. Ltd. Bühler, K. (1934), Sprachtheorie: Die Darstellungsfunktion der Sprache, Jena/Stuttgart: Gustav Fischer Verlag. Buyssens, E. (1943), Les Langages et le Discours: Essai de linguistique fonctionnelle dans le cadre de la sémiologie, Brussels: Office de Publicité. Buyssens, E. (1968), ‘Le Langage et la Logique. Le Langue et la Pensée’, in A. Martinet (ed.), Le Langage, 76–90, Paris: Éditions Gallimard. Cassirer, E. (1942), ‘Zur Logik der Kulturwissenschaften: Fünf Studien’, Götesborgs Högskolas Arsskrift, 48: 1–139. Cassirer, E. ([1957] 1995), The Philosophy of Symbolic Forms: Vol. 1: Language, Vol. 2: Mythical Thought; Vol. 3: Phenomenology of Knowledge; Vol. 4: The Metaphysics of Symbolic Forms, trans. R. Manheim, New Haven, CT: Yale University Press. Chang, H. (2013), ‘Cassirer, Benvenisté, and Peirce on Deictics’ and “Pronominal” Communication’, Sign Systems Studies, 41 (1): 7–20. Charaudeau, P. (1983), Langage et Discours: Éléments de sémiolinguistique (Thêorie et pratique), Paris: Hachette. Comay, R. and F. Ruda (2018), The Dash – The Other Side of Absolute Knowing, New York: MIT Press. ­Deely, J. (1994), The Human Use of Signs or: Elements of Anthroprosemiosis, Lanham: Rowan & Littlefield Publishers, Inc. DePalo, M. (2010), ‘Le “je”, la phénoménologie et le discours: Bühler, Benvenisté et Husserl’, Beiträge zur Geschichte der Sprachwissenschaft, 20: 155–65. Descombes, V. (1979), Le Même et L’Autre: Quarante-Cinq Ans de Philosophie Française (1933–1978), Paris: Éditions de Minuit. DeVito, J. A. (1978), Communicology: An Introduction to the Study of Communication, New York: Harper and Row, Publishers, Inc. Dreyfus, H. L. ([2001] 2009), On the Internet, 2nd edn, New York: Routledge; Taylor & Francis Group. Eco, U. (1976), A Theory of Semiotics, Bloomington: Indiana University Press. Ehring, D. (2011), Tropes: Properties, Objects, and Mental Causation, New York: Oxford University Press. Froman, W. J. (2005), ‘Merleau-Ponty and the Relation between the Lógos Prophorikos and the Lógos Endiathetos’, in A.-T. Tymieniecka (ed.), Logos of Phenomenology and Phenomenology of Logos, 409–16, [Book One], New York: Springer. Fuchs, T. (2018), Ecology of the Brain: The Phenomenology and Biology of the Embodied Mind, New York: Oxford University Press.

42

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Gleick, J. (2011), The Information: A History, A Theory, A Flood, New York: Pantheon. Gusdorf, G. (1953), La Parole, Paris: Presses Universitaires de France. Gusdorf, G. (1966–93), Les Sciences Humaines et la Pensée Occidentale, Tomes 1–12, Paris: Payot. Hardwick, C. E., ed. (1977), Semiotics and Significs: The Correspondence between Charles S. Peirce and Victoria Lady Welby, Bloomington, IN: Indiana University Press. Heidegger, M. ([1926] 1962), Sein und Zeit, 7th edn, Tübingen: Neomarius Verlag, trans. J. Macquarrie and E. Robinson (eds), New York: Harper & Row, Publishers. Heidegger, M. (1929), Einführung in die Metaphysik, Tübingen: Max Niemeyer Verlag. Heidegger, M. (1939), Die Geschichte des Seyns, Frankfurt Main: Vittorio Klostermann GmbH. Holenstein, E. (1974), Roman Jakobson’s Approach to Language: Phenomenological Structuralism [from German Habilitationschrift, Zurich 1974; French edn, Jakobson, ou sur le structuralisme phénoménologique, Paris, FR: Editions Seghers, 1974]. Houser, R. E. (2020), Logic as a Liberal Art: An Introduction to Rhetoric and Reasoning, Washington, DC: Catholic University of America Press. Hunt, E. (1970), ‘Classical Rhetoric and Modern Communicology’, Western Speech, 34 (1): 2–7. Husserl, E. (1948), Erfahrung und Urteil: Untersuchungen zur Genealogie der Logik, Hamburg: Klassen and Govert. Ijsseling, S. (1976), ‘Who Is Actually Speaking Whenever Something Is Said?’ in Rhetoric and Philosophy in Conflict: An Historical Survey, 127–36, Boston: Martinus Nijhoff. Jakobson, R. O. ([1962] 1981), ‘Linguistics and Poetics’, in Stephen Rudy (ed.), Roman Jakobson Selected Writings III: Poetry of Grammar and Grammar of Poetry, The Hague: Mouton Publishers. Jakobson, R. O. (1960b, 1961), ‘Linguistics and Communication Theory’, in R. Jakobson (ed.), Structure of Language and Its Mathematica Aspects, vol. 12; vol. 20, 570–9, Providence: American Mathematical Society. Jakobson, R. O. (1962–2002), Selected Writings, 9 vols: vol. 1, Phonological Studies, 1962, 2nd edn 1971, 3rd edn 2002; vol. 2, Word and Language, 1971; vol. 3, Poetry of Grammar and Grammar of Poetry, S. Rudy, ed. (1981); vol. 4, Slavic Epic Studies, 1966; vol. 5, On Verse, Its Masters and Explorers, S. Rudy and M. Taylor, eds (1979); vol. 6, Early Slavic Paths and Crossroads: Part 1 and Part 2, S. Rudy, ed. (1985); vol. 7, Contributions to Comparative Mythology; Studies in Linguistics and Philology, 1972–82, S. Rudy, ed. (1985); vol. 8, Completion Volume One: Major Works, 1976–1980, S. Rudy, ed. (1988); vol. 9, A Complete Bibliography of His Writings, compiled and edited by Rudy (1990). Berlin: Mouton de Gruyter. [Unless noted, all vols. edited by Jakobson.] Jakobson, R. O. (1972, September), ‘Verbal Communication’, Scientific American, 227 (3): 37–44. Kristeva, J. (1974), La révolution du langage poétique, Paris: Éditions du Seuil. Kristeva, J. (1981), Le Langage, cet inconnu, Paris: Éditions du Seuil. Lacan, J. ([1953] 1966), Fonction et champ de la parole et du langage en psychanalyse, vol. 1, Paris: Éditions du Seuil. Lanigan, R. L. (1974), ‘Enthymeme: The Rhetorical Species of Aristotle’s Syllogism’, Southern Speech Communication Journal, 39 (3): 207–22. Lanigan, R. L. (1988), ‘A Treasure House of Preconstituted Types: Alfred Schütz on Communicology’, in R. L. Lanigan (ed.), Phenomenology of Communication: MerleauPonty’s Thematics in Semiology and Communicology, Pittsburg: Duquesne University Press. Lanigan, R. L. (1988), Phenomenology of Communication: Merleau-Ponty’s Thematics in Semiology and Communicology, Pittsburgh, PA: Duquesne University Press.

COMMUNICATION THEORY AND SEMIOTICS

43

Lanigan, R. L. (1992), The Human Science of Communicology: A Phenomenology of Discourse in Foucault and Merleau-Ponty, Pittsburgh, PA: Duquesne University Press. Lanigan, R. L. (1994), ‘Capta Versus Data: Method and Evidence in Communicology’, Human Studies, 17 (1): 109–30. Lanigan, R. L. (1995), ‘From Enthymeme to Abduction: The Classical Law of Logic and the Postmodern Rule of Rhetoric’, in L. Langsdorf and A. R. Smith (eds), Recovering Pragmaticism’s Voice: The Classical Tradition, Rorty, and the Philosophy of Communication, 49–70; notes 278–83, Albany: State University of New York Press. Lanigan, R. L. (2012), ‘The Two Senses of a Phenomenology of the Weltanschauung: An Essay in Honor of Émile Benvenisté’, The American Journal of Semiotics, 28 (1–2): 63–72. Lanigan, R. L. (2013), ‘Information Theories’, in P. Cobley and P. Schulz (eds), Theories and Models of Communication, vol. 1, 58–83, Berlin, DE; Boston, MA: De Gruyter Mouton. Lanigan, R. L. (2014), ‘Charles S. Peirce on Phenomenology: Communicology, Codes, and Messages, or, Phenomenology, Synechism, and Fallibilism’, The American Journal of Semiotics, 30 (1–2): 139–58. Lanigan, R. L. (2015a), ‘Semiotic Paradigms of Self and Person: The Perspectives Model of Communicology as the Logic Foundation of Human Science’, Language and Semiotic Studies, 1 (1): 106–29. Lanigan, R. L. (2015b), ‘Philosophy of Communicology: “Discourse Which Expresses Itself Is Communication” [Heidegger]’, Review of Communication, 15 (4): 332–58. Lanigan, R. L. (2015c), ‘Semiotic Confusion in the Phenomenology of Perception West Meets East, One Actuality Becomes Two Realities’, Chinese Semiotic Studies, 11 (2): 227–62. Lanigan, R. L. (2017a), ‘Cassirer on Communicology: The Symbolic Forms of Language, Art, Myth, and Religion in Cultural Semiotics’, The American Journal of Semiotics, 33 (3–4): 135–40. Lanigan, R. L. (2017b), ‘Ernst Cassirer’s Theory and Application of Communicology: From Husserl via Bühler to Jakobson’, The American Journal of Semiotics, 33 (3–4): 181–231. Lanigan, R. L. (2018a), ‘Franklin H. Knower (1901–93): A Model for Communicology’, Review of Communication, 18 (4): 391–413. ­Lanigan, R. L. (2018b), ‘Communicology Chiasm: The Play of Tropic Logic in Bateson and Jakobson’ (Special Issue: The Play of Signs/The Signs of Play), Language and Semiotic Studies, 4 (2): 67–92. Lanigan, R. L. (2018c), ‘The Rhetoric of Discourse: Chiasm and Dialogue in Communicology’, in R. C. Arnett and F. Cooren (eds), Dialogic Ethics, 215–43, Philadelphia, PA: John Benjamins Publishing Co. Lanigan, R. L. (2019a), ‘Applied Communicology in Organizational PR and R&D: Peirce on Synechism, Fuller on Synergetics, Gordon on Synectics, and Alinsky on Socialism’, Language and Semiotic Studies, 5 (4): 1–27. Lanigan, R. L. (2019b), ‘Immanuel Kant on the Philosophy of Communicology: The Tropic Logic of Rhetoric and Semiotics’, Semiotica, 227: 273–315. Lanigan, R. L. (2019c), ‘Perelman’s Phenomenology of Rhetoric: Foucault Contests Chomsky’s Complaint about Media Communicology in the Age of Trump Polemic’, Semiotica, 229: 273–328. Lanigan, R. L. (2019d), ‘Crossing Out Normative Boundaries in Psychosis: The Logic of a Social Semiotic Passage in Dickens’, Bleak House’, The American Journal of Semiotics, 35 (3–4): 335–64. Lévi-Strauss, C. (1958), Anthropologie Structurale, Paris: Pion.

44

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Lotman, Y. M. (1994), The Universe of Mind: A Semiotic Approach to Culture, Bloomington: Indiana University Press. Marcus, S. (1974), ‘Fifty-two Oppositions between Scientific and Poetic Communication’, in C. Cherry (ed.), Pragmatic Aspects of Human Communication, 83–96, Boston: D. Reidel Publishing Co. Merleau-Ponty, M. (1964), Le Visible et l’invisible, Paris: Librairie Gallimard. Mulligan, K. ed. (1990), Mind, Meaning, and Metaphysics: The Philosophy and Theory of Language of Anton Marty, Boston: Kluwer Academic Publishers. Nöth, W. (2014), ‘Human Communication from the Semiotic Perspective’, in F. I.-San Juan and T. M. Dousa (eds), Theories of Information, Communication and Knowledge: A Multidisciplinary Approach, 97–119, New York: Springer Science & Business Media. Peirce, C. S. (1906), March 9, 52-page draft letter to Lady Welby (under Robin L493, p. 200; EP 2: 478), ms. pp. 24–30 excerpted in Hardwick 1977: 195–201. Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Powell, E. A. (2017 May/June), ‘When the Ancient Greeks Began to Write: Newly Discovered Inscriptions Help Explain How Literacy Spread’, Archaeology, 3: 43–9. Ruesch, J. ([1953] 1972), ‘Synopsis of the Theory of Human Communication’, in T. Sebeok and D. J. Umiker (eds), Semiotic Approaches to Human Relations, 47–94, Paris: Mouton. Ruesch, J. ([1967] 1972), ‘Social Control of Symbolic Systems’, in Ruesch, Semiotic Approaches to Human Relations, 277–300, Paris, FR: Mouton. Ruesch, J. and G. Bateson (1951), Communication: The Social Matrix of Psychiatry, New York: Norton. Rollinger, R. D. (2009), Austrian Phenomenology: Brentano, Husserl, Meinong, and Others on Mind and Object, New Brunswick, NJ: Ontos Verla. Rollinger, R. D. (2010), Philosophy of Language and Other Matters in the Work of Anton Marty: Analysis and Translations, New York: Rodopi. Rota, G.-C. (1997), ‘Three Senses of “A is B” in Heidegger’ [Husserl’s Fundierung]’, in F. Palombi (ed.), Indiscrete Thoughts, 188–91, Boston, MA: Birkhäuser. ­Ruthrof, H. (1997), Semantics of the Body: Meaning from Frege to the Postmodern, Toronto: University of Toronto Press. Ruthrof, H. (2000), The Body in Language, New York: Cassell. Ruthrof, H. (2020), ‘Phenomenology’, in Oxford Research Encyclopedia of Literature, New York: Oxford University Press. Sapir, E. (1931), ‘Communication’, Encyclopedia of the Social Sciences, 78–81, New York: Macmillan. Sapir, E. ([1944] 1949), ‘Grading: A Study in Semantics’, in D. G. Mandelbaum (ed.), Selected Writings of Edward Sapir in Language, Culture, and Personality, 122–49, Berkeley: University of California Press. Sebeok, T. A. (1972), ‘Coding in the Evolution of Signaling Behavior’, in Perspectives in Zoosemiotics, 7–33, Paris: Mouton. Schütz, A. (1970), Reflections on the Problem of Relevance, New Haven: Yale University Press. Shapiro, M. (1988). Figuration in Verbal Art, Princeton: Princeton University Press. Sonesson, G. (2015), ‘The Mirror in-between Picture and Mind: A Phenomenologically Inspired Approach to Cognitive Semiotics’, Chinese Semiotic Studies, 11 (2): 159–80. Stawarska, B. (2015), Saussure’s Philosophy of Language as Phenomenology: Undoing the Doctrine of the Course in General Linguistics, New York: Oxford University Press.

COMMUNICATION THEORY AND SEMIOTICS

45

Swiggers, P. (1987), ‘Eric Buyssens’s Les Langages et le Discours: A Functional Analysis of Man’s Use of Signs’, in T. A. Sebeok and J.-U. Sebeok (eds), The Semiotic Web 1986, New York: Mouton de Gruyter. Urban, W. M. (1929), The Intelligible World: Metaphysics and Value, London: George Allen & Unwin. Urban, W. M. (1939), Language and Reality: The Philosophy of Language and the Principles of Symbolism, London: George Allen & Unwin. Urban, W. M. (1949a), Beyond Realism and Idealism, London: George Allen & Unwin. Urban, W. M. (1949b), ‘Cassirer’s Philosophy of Language’, in P. A. Schilpp (ed.), The Philosophy of Ernst Cassirer (LLP vol. VI), 401–44, Evanston: The Library of Living Philosophers. Whorf, B. L. (1952), ‘Language, Mind, and Reality’, Et cetera, 9 (3): 203–26. Wilden, A. ([1972] 1980), System and Structure: Essays in Communication and Exchange, 2nd edn, London: Tavistock Publications Ltd. Wilden, A. (1987), The Rules Are No Game: The Strategy of Communication, New York: Routledge and Kegan Paul. Wiener, N. (1915, 14 October), ‘Is Mathematical Certainty Absolute?’ The Journal of Philosophy, Psychology and Scientific Methods, 12 (21): 568–74. Wiener, N. (1948a), ‘Time, Communication, and the Nervous System (Teleological Mechanisms)’, Annals of the New York Academy of Sciences, 50 (4): 197–220. Wiener, N. ([1948b] 1961), Cybernetics: Or Control and Communication in the Animal and the Machine, 2nd edn, Cambridge: MIT Press. Wiener, N. (1950), The Human Use of Human Beings: Cybernetics and Society, Boston: Houghton Mifflin Co.

46

­C HAPTER TWO

Media/Culture Studies and Semiotics SOPHIA MELANSON RICCIARDONE AND MARCEL DANESI

INTRODUCTION The emergence of media/culture studies as an autonomous discipline is traced to the 1940 work The Invasion from Mars: A Study in the Psychology of Panic by Princeton psychologist Hadley Cantril. In it, Cantril investigated the likely source of the public panic created by the 1938 CBS radio docudrama based on H. G. Wells’s 1898 novel about interplanetary invasion, War of the Worlds, directed by Orson Welles as part of his Mercury Theater of the Air Presents. The broadcast interspersed fake news reports of Martian landings in New Jersey, which were so realistic that many listeners believed that they were genuine, despite periodic announcements that the programme was a fictional dramatization. Some citizens in the New Jersey area even fled their homes and phoned the local authorities to report the purported landing. After interviewing 135 subjects, Cantril concluded that better-educated listeners were more inclined to recognize the broadcast as fiction than less-educated ones. The study was criticized on several counts, since the statistical correlation between the radio broadcast, educational background and the degree of reported panic was weak. Moreover, the panic may have been exaggerated by subsequent media stories. Despite the critiques, the implications of Cantril’s research could not be ignored by a world saturated with mass media communications and culture. So, by the early 1950s, the academic study of the relationship between mass media and mass culture (henceforward abbreviated as media studies) started taking shape (Williams 1950; McLuhan 1951; Gerbner 1956), becoming an autonomous discipline by the 1960s. In 1957, French semiotician Roland Barthes, along with his colleagues at the Centre d’Études des Communications de Masse (the Centre for Mass Communication Studies, or CECMAS), brought out the benefits of analysing popular mass media texts and performances (spectacles, movies, consumer products, etc.) in semiotic terms, with his book Mythologies. In 1962, CECMAS began publishing the journal Communications, which traced the semiotic dimensions of communication and culture (Simonson and Durham Peters 2008). Although Barthes did not name his approach in any specific way, eventually it came to be called media semiotics, becoming a branch of both media culture studies and cultural semiotics. This chapter will deal with the interface between general media studies and media semiotics. It will cover several thematic areas including the disciplinary origins of both fields of study; the main critical and socio-psychological approaches to the media; the

48

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

media semiotic approach that traces its roots to Barthes (classic media semiotics); and semiotic views of digital media (new media semiotics).

ORIGINS The Cantril study made it obvious that the mass media had psycho-behavioural effects on individual consumers of media content. At first, the study of these effects was incorporated as a target of investigation by psychology, sociology and anthropology. But soon after, the importance of the mass media culture that was spreading broadly caught the attention of scholars from across the academic landscape. This led eventually to the establishment of media studies as an autonomous discipline, characterized by the integration of findings and ideas emanating from the different fields (above). By the mid-1960s, departments of media studies started cropping up on campuses across America, Europe and other countries where the mass media played a critical role in daily life. One of the first to examine the mass media in terms of its relation to technology was the Canadian communications theorist Marshall McLuhan. In several key works (e.g. 1951, 1962, 1964), McLuhan developed the theory that media are extensions or amplifications of human faculties – biological, cognitive and emotional. So, for example, tools such as an axe, a wheel and a telescope are media, in this sense, because they extend (amplify) specific biological capacities – an axe extends the ability of the human hand to break wood, the wheel of the foot to cover greater distances and the telescope of the eye to see further. In this framework, communications media extend the ability of the individual to go beyond the immediate environment, increasing the frequency, efficacy, range and rapidity of communication across time and space. The advent of the electronic mass media in the twentieth century further extended this ability and, in the process, became a factor in changing the world. McLuhan coined the catchphrase the medium is the message to convey that communication and meaning-making are shaped by the nature of the communications media we devise and use, because they condition the codes for expression and the modes of experience that emerge within specific social environments for messages to be conceived and delivered. He used the example of a light bulb to illustrate this. A light bulb has no meaning or content in itself. However, it enables people to see in the dark and thus creates ‘an environment by its mere presence’ (1964: 8) – a physical environment in which people can carry out activities involving sight. Clearly, as environments become altered by a medium, the modalities of meaning, expression and perception are adjusted accordingly, thus influencing the ‘message’. In his 1967 book The Medium Is the Massage: An Inventory of Effects, co-authored with Quentin Fiore, McLuhan used the word ‘massage’, rather than ‘message’, to reinforce the notion of media creating environments that directly affect human beings, as if they were being given a massage. He elaborates as follows: All media work us over completely. They are so persuasive in their personal, political, economic, aesthetic, psychological, moral, ethical, and social consequences that they leave no part of us untouched, unaffected, unaltered. The medium is the massage. Any understanding of social and cultural change is impossible without a knowledge of the way media work as environments. All media are extensions of some human faculty – psychic or physical. (McLuhan and Fiore 1967: 26)

MEDIA/CULTURE STUDIES AND SEMIOTICS

49

McLuhan’s work on media, which goes back initially to his 1950 book The Mechanical Bride, showed how a culture cannot be studied apart from the dominant communications media that it uses. As a result of McLuhan’s ground-breaking work, departments, journals, book series and associations of media studies started burgeoning in the 1960s, 1970s and 1980s. By the 1990s, the field had carved a significant niche for itself in academia. Today, media studies is a flourishing discipline across the world, encompassing a broad range of theories and interests. Within the discipline, branches have emerged focusing on ‘area-specific’ study – for example, television, radio, internet and video games. Of particular relevance is the study of digital media and their effects on culture – a line of investigation that clearly continues McLuhan’s work, albeit often in an unwitting and thus unacknowledged way. As mentioned above, the foundational work for media semiotics, as separate from, but integrated with, general media studies, was established by Barthes’s Mythologies. Guided by Barthes’s contributions, CECMAS became a leading institution where the mass media became the target of serious semiotic analysis – what was then termed semiology. Barthes saw the mass media culture as a ‘bastard form’ plagued by ‘humiliated repetition’ and generating a constant need for ‘new books, new programs, new films, news items, but always the same meaning’, as he emphatically put it later (Barthes 1975: 24). For Barthes, the content of mass media texts and performances (blockbuster movies, sports spectacles, etc.) was a mere recycling of the classic myths (as will be discussed below). Despite Barthes’s ideological take on mass media culture, it is his semiotic approach to mass culture that has remained a significant point of reference not only in semiotics proper but in media culture study generally. The ideas of philosopher and literary theorist Mikhail Bakhtin, which predate those of Barthes, going back to the late 1920s (see Bakhtin 1981), are now included under the rubric of media semiotics retroactively. Bakhtin claimed that the more profane aspects of modern mass culture are similar to the medieval carnivals, since they allow common people to laugh and vent their frustrations as a unified community – in contrast to the idealistic and individualist culture produced by the nobility and by artists of the Romantic period. The language used at carnivals is polyphonic, exchanged between performers and audiences with no regard to any pre-existent hierarchical social relations among the speakers; it is a form of dialogue where everyone’s voice has value since there is no one voice of authority imposing itself on the dialogical system. Bakhtin saw polyphony as even a feature of great literature, from Dante and Rabelais to Goethe and Dostoyevsky, because readers feel that they are in an internal dialogue with the author – a communicative relation that became a key one within semiotics and literary criticism in the analysis of literary texts a few decades ago. Media semiotics today has spread its tentacles broadly, including within its purview everything from film, television, video games, the algorithmic generation of signifying content, the structure of websites to the evolution of new languages in cyberspace. Since the late 1990s and into the millennium, it has produced several textbook-like surveys and introductions (e.g. Jensen 1995; Nöth 1997; Bignell 2002; Danesi 2018). Overall, media semiotics aims to explore the implicit or unconscious signifying structures and practices present in mass media content. As an analytical and conceptual framework, it has proven itself to be particularly well-suited to deciphering media effects on cultural trends and on how meaning systems inherent in human codes, such as language, are transformed for various purposes by the media. Like the other branches of the discipline, media semiotics interweaves insights and findings from cognate fields in order to unravel the modalities implicit in ‘mediated semiosis’ (Danesi 2018).

50

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

CRITICAL APPROACHES TO MEDIA STUDIES Long before the rise of the academic study of mass media culture in the 1950s, social critics of the Industrial Age were already assailing mass culture as being vulgar and profane. The nineteenth-century British social critic Matthew Arnold (1869), for example, attacked the mass culture of his era as insipid and homogenized, warning his contemporaries that it was a threat to civilized society, turning the clock back to an undomesticated stage of humanity. Arnold’s book, titled Culture and Anarchy, can be seen retrospectively to constitute the initial work in what can be called a critical approach to mass culture. Known today as the ‘mass society thesis’, Arnold’s main contention was that a mass culture based on materialism and affluence had a deleterious effect on human growth and potential. Also falling under the rubric of a critical approach is the 1922 book Public Opinion by American journalist and essayist Walter Lippmann. Lippmann argued in it that mass culture had a powerful negative impact on people’s minds and behaviours, producing ‘pictures in our heads’, and thus adversely affecting public opinion and politics, a view espoused as well by Harold Lasswell a little later in Propaganda Technique in World War I (1927). In the 1950s, this view was adopted by English literary critic Frank R. Leavis (1952), who saw the spread of an ever-expanding mass media culture as a product of banal bourgeois tastes and thus evidence of the decline of civilization. But perhaps the best-known critical approach emerged from the Frankfurt Institute for Social Research, founded in 1922, which condemned the mass-mediated cultures of modern capitalist societies from a specific philosophical angle – Marxism. Members of the School, which included Theodor W. Adorno (1941), Max Horkheimer (1947), Leo Lowenthal (1949) and Herbert Marcuse (1972), characterized capitalist culture as an anti-aesthetic one, merely reproducing art forms, such as music, as if they were ‘commodities’ to be sold and quickly discarded in the marketplace just like manufactured commodities. As Adorno saw it, works of art in a mass capitalist culture, literature, music and so on followed the same pattern of reproducibility. In contrast, true art ennobles the human spirit and cannot be reproduced so easily. Despite the many counter-arguments to Adorno’s views, they continue to have a broad intellectual appeal today, because of their implications for studying new media, where cultural products can be produced, consumed and reproduced instantly and collectively ad infinitum, and often take the form of viral content on the internet. Adorno and Max Horkheimer coined the term culture industry to characterize culture in a mass capitalist society – in analogy with the industrial manufacturing of commercial products. Marcuse argued further that the mass media indoctrinate and manipulate common people, promoting a false consciousness of the self that plays into the hands of the system of power that oversees the social order. Following on the coattails of the Frankfurt School was the Centre for Contemporary Cultural Studies (CCCS), founded at the University of Birmingham in 1964 to study mass culture from the standpoint of hidden discursive practices within it. The CCCS was one of the first to study systematically the negative representation of underprivileged classes and people. Leading members of the Centre included Richard Hoggart (1957), E. P. Thompson (1963), Raymond Williams (1977), Stuart Hall (1978), Dick Hebdige (1979) and Angela McRobbie (1991). Their focus was on the struggle that powerless people faced in achieving a cultural voice. Under the leadership of Stuart Hall, the CCCS conducted an important research project that led in 1978 to the publication of Policing the Crisis, which showed how African Americans were misrepresented in the media and the negative social consequences this entailed (for an overview of the CCCS, see Turner 2002).

MEDIA/CULTURE STUDIES AND SEMIOTICS

51

The CCCS was also instrumental in promoting feminist theories of media culture, focusing on how representations, or misrepresentations, of women in the mass media kept women subjugated and repressed as artists and free thinkers. This line of inquiry no doubt had a profound impact on changing attitudes in how media representations of gender are realized and in bringing about a greater gender-equality. In the 1970s and 1980s, feminist critics argued that images of women were highly biased towards patriarchal views of gender. But as society changed, by the 1990s the feminist approach had also changed, leading to the so-called post-feminist movement, which did not see female sexuality in media as exploitation, but rather as a transgressive form of representation against previous and largely religious restrictions on the public display of women’s bodies (on feminist theories of the era, see Van Zoonen 1994; Phoca and Wright 1999). However, social commentator Ariel Levy (2005) has asserted that this particular brand of post-feminism only implied an emancipation from the shackles of patriarchy through sexuality and was, thus, a superficial enticement merely distorting reality. According to Levy, the mass media has simply appropriated feminism and commodified the male perspective through the production of a ‘raunch’ culture that seduces women into asserting their so-called liberation by accommodating a highly sexualized model that is heavily contoured by a nuanced projection of male desire. Recent events around the #metoo movement during the second decade of the twenty-first century have begun to challenge these masculinized framings of female sexual liberation that accompanied the mediation of ‘raunch’ culture during the 1990s. The semiotic salience of the hashtag used to circulate awareness of movements such as #metoo will be discussed later in this chapter in relation to memetic (meme-based) internet culture. The critical approaches of both the Frankfurt School and the CCCS are sometimes classified under Hegemony Theory – a concept developed initially by Italian Marxist Antonio Gramsci (1937) – which states that genuine culture under modern capitalism is improbable, since media representations are typically nothing more than forms of propaganda designed to pacify the masses and disguise social inequalities. In this system, true artists and opposing voices are punished either directly (by incarceration) or indirectly and thus more effectively (by critique and marginalization). The concept of hegemony is inherent in another major critical approach, called Propaganda Theory, a framework associated with Edward Herman and Noam Chomsky (1988), which maintains that those in power, such as the government and media owners, influence how the media present news coverage for the simple reason that they control the funding. In consequence, the media tend to be little more than a propaganda arm of those in power or of particular media owners. The media are thus set up to ‘manufacture consent’. They do this by selecting the topics to be showcased, establishing the tone of the issues that are discussed, and filtering out any contradictory information.

­CRITICAL APPROACHES TO NEW MEDIA More recent critical scholarship in new media studies has revitalized former approaches and philosophical framings of how power is distributed and negotiated across media networks. Philosopher and sociologist Bruno Latour, for example, emphasizes the importance of analysing non-human activities, such as computational operations, mediating networks and the machinations of the political economy, in order to trace how they shape the contours of social and cultural life. He refers to this sociological approach as the Sociology

52

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

of Associations and has put forth an associated theory, called Actor Network Theory (Latour 2007: 7–9). Latour argues that to fully ascertain the nature of ‘power relations’ within society, one must trace the processes by which various associations are constituted (86). Recall that McLuhan had made similar observations in his assertion that any medium is an extension of human faculties, and will therefore transform the environments that are inhabited, altering the cultural conditions and the social ordering of everyday life in the process. This perspective aligns well with the general semiotic law of media which stipulates that ‘ . . . as the medium changes, so too do the patterns of semiosis’ (Danesi, in Pericles Trifonas 2015: 485). One may conclude, therefore, that as the various machinations of new media influence the signifying order of power relations within society, the rhythms of social and cultural life become altered. Thus, understanding how new media influence the patterns of signification in everyday life is integral to ascertaining the conditions of social and cultural existence at any given time. Along a similar though perhaps more deterministic vein, sociologist Maurizio Lazzarato (2014) borrows from Deleuze and Guattari’s conception of machinic enslavement to describe the new media conditions outlined above. He remarks that a-signifying processes of new media – the signalling qualities of algorithms and digital processes that have no semantic dimension – capture human subjectivities within a technosocial machine, and he explains that human subjects are rendered conceptually contiguous with machines within the logic of the new media conceptual system (Lazzarato 2014: 27). Lazzarato outlines the process through which new media operationalize traces of human behaviour online as terminals for a-signifying, symbolic and signifying processes (96), and that within such arrangements, ‘ . . . signs function in place of the object to which they refer . . . [and through this] they influence subjectivity’ (100). Also borrowing from Guattarian perspectives, communications scholar, Ganaele Langlois (2014), put forward a mixed semiotic approach to ascertain how meaning emerges from the confluence of material, social, technical, economic, psychological and political processes, and proposes a theoretical framework to map the ways in which various interactions coalesce into meaning-making processes. She argues that interactions between disparate units within the circuitry of new media create meaning machines, which render mediated forms of semiosis susceptible to the manipulative interference of economic and political powerbrokers (19). Guattari has suggested that the recent miniaturization of technology has enabled powerbrokers to equip individuals with devices that permit increased management of their perceptions and interpretations of meaning by plugging them into what he refers to as the machinic phylum – an ongoing process through which singular units belonging to disparate wholes are dismembered and assembled into something else (Guattri 1995: 103). As new media hardware continues to contract in size, the algorithmic machinations of new media that define the neuro-technological ordering of signifying content online have become progressively more portable and increasingly more accessible. Congruent with Guattari’s observations, cognitive semiotician Merlin Donald (2010) notes that as intimacy between humans and their mediating devices intensifies, society becomes increasingly dependent upon technology and its mediating logic – what he terms exographic media – in making decisions and planning various courses of action (2010: 76) on the micro- and macro-levels, some of which undoubtedly impact the meaning, distribution and relations of power. Interactions between technological circuitry, artificial intelligence (AI), artificial neural networks, human digital and physical activity, human cognition and the political economy

MEDIA/CULTURE STUDIES AND SEMIOTICS

53

within cyberspace generate a myriad of simulacra – representations or abstractions of the real – that through the affordances of algorithms signal the production, assemblage and reproduction of signifying matter ad infinitum. Within these conditions, digital impressions of our identities are abstractly contoured as simulacra (Baudrillard 1981) of the embodied self, which are translated into bits and bytes, rendering abstractions of the ‘self’ legible to the algorithmic (or, statistically salient) computations of new media. The resulting digital transmutations of the self are what communications scholar, John Cheney-Lippold, refers to as our measurable type (Cheney-Lippold 2017: 66) – a term inspired by Max Weber and Erving Goffman’s conception of the ‘ideal type’. CheneyLippold argues that these datafied, abstracted identities are constructed by algorithms in the background of cyberspace according to the programmed priorities of commercial and political powerbrokers without our knowledge, thus reinforcing existing monopolies of power.

PSYCHOLOGICAL-SOCIOLOGICAL APPROACHES The psychological analysis of the mass media, as mentioned, started with the Cantril study, which led shortly thereafter to the first cognitive theory of media, called the Hypodermic Needle Theory (HNT), which claims that the mass media can directly influence minds and behaviours with an analogous impact to the one that a hypodermic needle has on the body. In its initial form, HNT was critiqued substantively, although recent versions that apply to digital media have been put forth, such as the idea that we have become overly dependent on digital media or psychologically primed by media effects. People habituated to large doses of information and visual stimulation, constantly navigating the Web, or being constantly involved in social media chatter, tend to become psychologically dependent on these in ways that parallel substance dependency. Indeed, research has suggested that lower working memory and reduced attention control result from prolonged use of mobile devices (Unsworth, McMillan, Brewer and Spillers 2012). Furthermore, Hadlington established a significant probability that excessive use of the internet and mobile phones is linked with common cognitive failures, such as physically walking into other people and objects (Hadlington 2015). Another symptom is an abiding preoccupation with all aspects of the internet, including compulsive anticipations of digital contact from online interlocutors. Research scattered across medical and psychology journals is suggesting, in fact, that this kind of addictive behaviour might potentially be having several negative effects (e.g. Carr 2011, 2015; Greenfield 2015). So, while a revised version of HNT is gathering momentum today, the original one, which was associated with electronic media before the advent of the internet, was critiqued at first by American sociologist Paul Lazarsfeld (Lazarsfeld, Berelson and Gaudet 1948), who found that the media had very little (if any) ability to change or influence people’s minds directly. A little later, Lazarsfeld and Elihu Katz (1955) showed that audiences react to media content as members of interpretive communities – families, unions, neighbourhoods, churches, peer groups and so on. In such communities, there are ‘opinion leaders’ (e.g. church ministers) who influence how the members will interpret a media event. So, in contrast to HNT, which portrays media impact as a onestep flow reaching a homogeneous and passive audience directly, the work by Lazarsfeld and his co-researchers indicate that it is more accurately a two-step flow, as illustrated in Figure 2.1.

54

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 2.1  Two-step flow model.

Katz and Lazarsfeld also argued that the mainstream media have a conservative social function, rather than a disruptive one. For example, the representation of violence and deviancy on TV crime programmes is hardly a causative factor of real violence or crime in society, because such representations have a moral subtext built into them, warning people about the dangers of violence and crime. This came to be known as Cultivation Theory, which claims that the mainstream media ‘cultivate’, not threaten, the status quo. The over-representation of violence actually reinforces an abiding respect for law and order in most people. And even if there are copycat criminals, it was suggested that they would be inclined to perpetrate their misdeeds anyway. In a related 1976 study, George Gerbner and Larry Gross looked at the beliefs of habitual television viewers and those of non-viewers, finding that awareness of violence was higher in the former than it was in the latter. Those watching television seemed to believe that there is more violence in society than there actually is. In a related previous study, Katz (1959) argued that audiences use the media for their own purposes and gratifications, known as Uses and Gratifications Theory. In 1984, Elisabeth Noelle-Neumann demonstrated that exposure to the media impels many simply to go along with the opinions that the media propagate. Known as Spiral of Silence Theory, she claimed that those who perceive their opinion as being a minority one tend to remain silent fearing ridicule or marginalization (Scheufele and Moy 2000). Thus, the media reinforce the majority opinion by ‘silencing’ dissenting voices. One final psychological approach that implies an incorporation of some of the implications of the HNT is Agenda-setting Theory, which claims that the mainstream mass media influence opinions because they select which ideas and events are worthy of public exposure. The choices then get transferred to public and political agendas – an unconscious process called salience transfer. The theory was introduced by Maxwell McCombs and Donald Shaw in a 1972 study of the impact of media coverage on the 1968 American presidential election. They found a correlation between the degree of coverage and the rankings of the interviewees. In effect, voter opinions were conditioned

MEDIA/CULTURE STUDIES AND SEMIOTICS

55

by the quantity and nature of media information. Media emphasis thus translates into public emphasis. New media, which includes the internet and its related mediating technologies, tends to be cognitive by nature. This cognitive dimension of new media, what Latour refers to as mental software (2007: 3), introduces a new dynamic to mediation, in that it has become highly participatory – we participate within the technological assemblages of mediated content, and new media participates with us through the semi-autonomous automations of algorithms. Canadian neuroanthropologist and cognitive semiotician, Merlin Donald, asserts that technologically meaningful objects ‘ . . . can be said to have cognitive “lives” inasmuch as they enter into the mind, solidify and channel shared experience and define the social order . . . ’ (Donald 2010: 71). Donald notes that the proliferation of new media produces significant revolutionary implications for human cognition since it supplies a means of performing cognitive work collaboratively across hybrid networks that link many minds with new technologies. He claims that as new media have produced hybrid human-machine networks, the broader cognitive-cultures that society has constructed have changed, producing a detailed division of cognitive labour (Donald 2010: 72). Borrowing from Deleuze and Guattari, and in line with Donald’s insights, Katherine Hayles argues that cognitive assemblages are the result of these hybrid human-machine networks, and states that the interconnection between human minds and technical systems affects the cognitive processing of mediated information on the conscious, unconscious and embodied levels, as sensory systems carry mediated signals across the nervous system (Hayles 2017: 118). Given the analytical rigour of seasoned semioticians, media semiotics is likely the most appropriate intellectual means of assessing these perspectives and making sense of how meaning is shaped within the confluence of cognitive assemblages rendered within hybrid human-machine networks.

CLASSIC MEDIA SEMIOTICS The type of semiotic approach to traditional media, from print to television and cinema, can be called classic media semiotics, while the one that applies to digital and computer media of various kinds can be called, instead, new media semiotics (see below). Even though they share much theoretical ground, there are some techniques in the latter that apply specifically to the emergence of the new media. Classic media semiotics, as discussed above, began with Barthes’s Mythologies, in which he argued that mass-mediated texts and spectacles do nothing more than recycle ancient mythic themes, characters and settings, adapting them to the modern world. So, for instance, the action hero Superman possesses the same kinds of superhuman powers that an Atlas or a Hercules had; he also has a tragic flaw, like Achilles. Because of the unconscious power of myth, such fictional media heroes become cultural icons, symbolizing virtue, heroism and righteousness above and beyond the comic and movie scripts in which they appear. They are recycled mythic personae. To distinguish between the original myths and their contemporary media versions, Barthes designated the latter mythologies. Barthes claims that the code (system of meanings) is the same, but its textualization and representation are adapted and updated to fit the present day. Classic media semiotics involves penetrating the mythological (coded) level of all kinds of media texts and performances. As a structuralist, Barthes adopted various techniques, such as the notion of opposition, initially called différence by Ferdinand de Saussure (1916), to aid in the decoding of

56

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

media texts and spectacles. Essentially, this meant that if there was a hero in a story, then there would also have been a villain in it, and the différence between the two comes out in signifying elements (signifiers) such as their diverse appearances, behaviours and so on. Mythic oppositions are found throughout media texts. Now, despite its obvious utility, the structuralist approach was challenged as early as the 1960s, by the movement called post-structuralism, spearheaded by the late French philosophers Michel Foucault (1972) and Jacques Derrida (1976, 1978). According to Derrida, all sign systems are self-referential – signs refer to other signs, which refer to still other signs, and so on ad infinitum. Thus, what appears stable and natural turns out to be illogical, not oppositional in the structuralist sense. In contrast to Saussure’s idea of différence, Derrida (1978) coined the word différance (spelled with an ‘a’, but pronounced in the same way) to overtly critique Saussurean theory. As Nesselroth (2007: 442) has remarked, Derrida’s approach was one that ‘constantly decentres fixed meanings and puts into question the ontological status of language (both written and spoken) and of communication in general’. Now that the movement has subsided somewhat, it is becoming evermore evident that structuralist notions, such as those used by Barthes, still have value in understanding all kinds of mediated artefacts, spectacles, genres, etc. since they are based on oppositional codes that recycle ancient mythic notions and language. Another critique of opposition theory that emerged in semiotics at around the same time as poststructuralism was that it ignored the role of metaphorical meaning in representational activities – a critique dovetailing with the crystallization of cognitive linguistics (Pollio, Barlow, Fine and Pollio 1977; Lakoff and Johnson 1980, 1999; Fauconnier and Turner 2002). But as it turns out, the concept of conceptual metaphor within cognitive linguistics is hardly incompatible with that of opposition. A simple linguistic metaphor such as ‘My sister is a tiger’ is a token of an associative structure that portrays personality in connotative (coded) animal terms (‘Sam is a gorilla’, ‘Sarah is a puppy’, etc.). That structure is the conceptual metaphor, people are animals. Now, conceptual metaphors are formed through image schemata, as Lakoff and Johnson have cogently argued (Johnson 1987; Lakoff 1987; Lakoff and Johnson 1999). The image schematic source for the people are animals conceptual metaphor seems to be an unconscious perception that human personalities and animal behaviours are linked in some way. In other words, it is the output of an ontological opposition: humans-asanimals, constituting an example of how opposition manifests itself as an associative phenomenon. In this case, the two parts of the opposition are not contrasted (as in night vs day) but equated: humans-as-animals. This suggests that oppositional structure operates in a non-contrastive way at the level of figurative meaning. As mentioned, poststructuralism has receded, but it has left several interesting ideational residues that are now incorporated into media semiotics. One of these is the claim that media texts have rhetorical structure, which, as just mentioned, is covered by conceptual metaphor theory. Actually, in a 1964 article, ‘The Rhetoric of the Image’, Barthes had established a method for fleshing out rhetorical meaning in visual texts, leading shortly thereafter to the emergence of Visual Rhetoric (VR), which has since become a branch of media semiotics and other disciplines, from psychology to graphic design (Barthes 1964/2004). The Belgian Group µ, founded in 1967, further developed Barthes’s original ideas. Their 1970 publication A General Rhetoric reformulated classical rhetoric in Barthesian ways, and in their Traité du signe visuel (1992) they elaborated a rhetorical grammar of the image. More recently, Hawreliak (2018) has applied the

MEDIA/CULTURE STUDIES AND SEMIOTICS

57

notion of rhetorical structure to the multimodality of video games, showing that classic media semiotics still has a significant role to play in the context of current digital media. It should be mentioned that aspects of VR analysis, as envisioned by Barthes and illustrated in his 1964 article, are consistent with Charles Peirce’s notion of iconicity (Peirce 1866–1913) and how it shapes the understanding of many media texts, such as common ads. Reductively, iconicity is defined as the process of making signs and texts resemble their referents. Iconic brand names and logos, for example, dominate the marketing world – Splash (detergent) evokes what is done with the product through sound imitation (‘splashing’); the Polo logo represents the sport of polo visually with a horse and a rider dressed in polo garb; etc. (Danesi 2018). The relation between Peircean and Barthesian analysis of media is beyond the present scope. Without going into detail here, suffice it to say that a combination of both approaches has characterized classic media semiotics since its inception.

NEW MEDIA SEMIOTICS Though the term new media emerged in common parlance alongside digital media in the 1990s (Manovich 2001: 31), the cyberculture within which we operate today has long been percolating beneath society’s surface during its shift from a post-industrial state in the 1970s (Bell 1976) to the network society (Castells 2011) of the late twentieth century, which materialized fully equipped with a service and information economy. AI, as the pulse of digital new media, was first conceived in 1958 by Frank Rosenblatt and evolved throughout the 1960s to automate the statistical induction of probability in order to detect patterns and make predictions about the world based on data fed into a machine. However, as Minsky and Papert (1969) noted at the time, the AI theories and technologies of the period were inadequate to fully accommodate the vision of what AI might accomplish in simulating the neural networks of the human brain. Thus, AI pursuits were temporarily suspended and the dominant trend in computer science until the early 1990s was based on the expert systems model, which necessitated the tedious task of manually programming specific knowledge into software. Dissemination of this knowledge involved the integration of networks and the maintenance of cumbersome databases. These dynamics of computer programming and software performance defined the hypertexted, multi-mediated culture of the late 1990s and early 2000s. The advent of the internet – what has been coined the Big Data revolution of the mid-2000s – coupled with the development of powerful computer hardware for gaming consoles in the 1990s provided the extensive data and processing speed needed to reinvigorate new developments in AI (Hosanagar 2019), which today fuels what is called new media. Contemporary forms of AI are equipped with deep learning algorithms and artificial neural networks that have transformed the more static and anchored quality of internet content at its nascence to the more kinetically adaptive and distributed content that is curated and personalized for each individual user today. Herein lies the distinction between mass media and new media: the homogeneous quality of mass-mediated popular culture contrasts with the adaptive and customized quality of participatory digital culture. This distinction will be expanded upon further below – it is a direct result of significant technological transformations. Media theorist Matteo Pasquinelli (2017) offers Peirce’s differentiation between abduction and induction to supply insights into the potential limitations of new media

58

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

AI (2017: 12). Peirce discerns that inductive inference only determines ‘ . . . the value of a quantity. It sets out with a theory and it measures the degree of concordance of that theory with fact’ (1903: CP 5.145). Pasquinelli echoes Peirce’s proclamation that abduction is much more dynamic and serves as the engine of creative insight and invention. Though artificial neural networks perform a myriad of complex statistical inductions and may even come close to accomplishing an anaemic form of abduction in producing statistically constituted predictions, the human brain’s capacity for invention and creativity remains out of reach for the artificial neural networks of AI. That said, within cyberspace, the human brain responds to the signifying content and assemblages curated and assembled by deep learning algorithms, and so the computational operations of AI remain a target of semiotic analysis among contemporary theoreticians. In other words, the signifying output of AI computations becomes real in its consequences as human minds interpret the resulting assemblage of signifying matter projected through the user interface. These interpretations, guided by mediated processes, are schematically mapped within individual and collective memory. However, as Pasquinelli so aptly summarizes, ‘if pattern recognition via statistical induction is the most accurate descriptor of what is popularly termed Artificial Intelligence, the distorting effects of statistical induction on collective perception, intelligence and governance . . . are yet to be fully understood’ (Pasquinelli 2017: 2). New media semiotics is tasked with the charge of understanding these effects. In addition to the description of AI logistics offered above, the participatory and convergent dimensions of new media also warrant some consideration. New media, according to Henry Jenkins, is ‘convergent’ media, through which ‘old and new media collide’ (Jenkins 2006: 3) and where content flows across a multitude of media platforms that reproduce and remediate traditional mass media forms. Lev Manovich refers to this process as meta-mediation – media that mediates other media forms (Manovich 2002: 32). Such convergence certainly impacts the signifying order of content and the negotiation of meaning within current new media environments. Convergent media also represents the ways in which digital technologies enable the intersecting of media industries and media consumers, who collaboratively generate the ephemera of internet content. This dynamic has resulted in an effusion of participatory culture, which contrasts with previous notions about the mass media as passive media, cultivating a docile spectatorship (Manovich 2002: 3). In consequence of this participatory dynamic of new media, the circulation of signifying matter within cyberculture operates according to a logic of individual customization rather than that of mass standardization (Manovich 2002: 51). Within the participatory culture of convergent new media, humans collaborate in acts of semiosis alongside computational codes and a-signifying orders, which generate shifting modes of interpretation and tenuously distractible patterns for interaction. The vacillating and variable nature of participatory new media cultivates conditions within which the meaning of culturally constituted signifiers becomes destabilized where every meaning-making instance is subjected to the influence of what Lévi-Strauss refers to as the ‘floating signifier’, which animates ‘the disability of all finite thought’ (Lévi-Strauss 1987: 63). This particular mode of suspending finite thought is perpetuated by what Heidegger (2010) describes as a ‘ . . . restlessness and excitement from continual novelty and changing encounters’ (166) brought into existence by transient experiences with signifying matter online – a semio-technical simulation of ‘place’ that may be conceptually encompassed by the term ‘cyberspace’. Cyberspace, as experienced through the facility of user interfaces, is an interactive and immersive virtual electronic space. The internet is a primary environment within it,

MEDIA/CULTURE STUDIES AND SEMIOTICS

59

in the McLuhanesque sense (above). It now has its own communities and its own set of conventions for communicating and interacting. Movement and interaction in cyberspace are ‘hyperreal’, to use Jean Baudrillard’s (1983) apt term. The modern human being lives in two universes, that of physical reality and that of hyperreality, and the distinction between the two is becoming more and more indistinguishable. Baudrillard introduced the term simulacrum to describe this form of ‘amalgamated consciousness’, whereby reality and fantasy mesh in an entrenched way. Baudrillard’s concept of the simulacrum can be considered the starting point for new media semiotics to emerge a decade or so later, since it prefigured how meaning unfolds in cyberspace – as an unconscious blend of reality and hyperreality. While still based on fundamental semiotic notions, new media semiotics has expanded its range of application to encompass such phenomena as multimodality, algorithmic design, psychometrics, micro-targeting, digital profiling, memes and memetic codes, and other specific aspects of cyberspace (e.g. Mazzali-Lurati 2007; Warschauer and Grimes 2007; Albertazzi and Cobley 2010; Canizzaro 2016; Poulsen, Kvåle and van Leeuwen 2018; Danesi 2019). In addition to simulacrum theory, new media semiotics often incorporates McLuhan’s Four Laws of Media into its modus operandi, since they seem to be particularly relevant to the study of how media, technology and meaning-making codes intersect in the digital universe (Danesi 2018). The laws are called amplification, obsolescence, reversal and retrieval (McLuhan and McLuhan 1988). As discussed above, a new technological mass communications medium will at first amplify some sensory, intellectual or other human faculty. While one faculty is amplified, another is lessened or rendered obsolete – it is amputated – until it is used to maximum capacity whence it reverses itself and is retrieved in another medium. An important target of new media semiotics is how AI is shaping meaning systems and how we now read or experience texts (Walsh-Matthews and Danesi 2019). For instance, augmented reality (AR), which is the technology that superimposes a digital image on someone’s view of the real world, providing a composite augmented view of reality, manifests how Baudrillard’s simulacrum concept is an applicable one to the world of AI. AR projects elements of the consciousness that accrues in the virtual world onto the perceptual mechanisms involved in the real world, by amplifying hearing and seeing. In consequence of such technology, texts such as novels are being created in ‘augmented’ ways so that the author is no longer the key element in the text’s interpretation or resolution – it is the reader, who becomes enmeshed in the simulacrum. This has led to reconsiderations of semiotic theories about the ways in which authors and readers are intertwined in a text, such as those by Bakhtin (above) and the late Umberto Eco (1979, 1992). We have clearly moved away from the Gutenberg Galaxy, based on print media, into the ‘Matrix Galaxy’, as it can be called, in homage to the movie. Today, the ability to decipher, create, understand and interpret digital texts, platforms and social media venues meaningfully involves understanding the relation between reality and hyperreality more than ever before. Digital literacy, moreover, is an extension of print literacy – again, to use McLuhan’s key notion. Without such literacy today, people’s ability to function in modern society diminishes considerably. It now encompasses a host of subliteracies, including computer literacy, or how to use the computer to read and write, mobile literacy, which involves the ability to create and understand texts on mobile devices, virtual worlds literacy, such as Twitter, Instagram and Facebook communities, which have developed new forms of language and textuality within them, and others (for an overview, see Wong 2019).

60

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A few thematic areas of media semiotic research include local and global discourse structures, which reveal shifts in linguistic behaviours, including sentence typologies, speech acts, turn-taking, text messaging, the use of hashtags, the virality of memes and the contagious spread of content online. Studying local discourse structures involves examining how the internal structures of linguistic and communicative competence are responsive to traditional, real-world, social contexts, reflecting and encoding them simultaneously. Studying global discourse involves examining the themes and topics of conversations as they occur in digital contexts and how they are organized structurally and deliver meaning through the new technological channels. Global discourse also involves the kinds of speech genres that have developed online. For example, the Facebook homepage, which allows people to update their activities and leave relevant messages for visitors, follows specific stylistic conventions across the globe that identify it as such. Instant messaging or text messaging reveals a compressed and multimodal style of writing, including visual supports (emoji, GIFs) and abbreviated writing styles (lol, brb). Social media discourse patterns also fall into the global domain, but they vary according to social media platform – Facebook, Instagram, Snapchat, Twitter, etc. – each of which has developed their own textual genres with a specific style (see Poulsen, Vigild, Kvåle and Leeuwen 2018). Hashtag (i.e. #thisisahashtag) literacy is also becoming a primary area of interest of new media semiotics within the global discourse frame. The hashtag emerged in 2007 as a system for categorizing and tagging tweets. It slowly gained traction, migrating to various social media platforms. Adding hashtags to a post enables multiple posts to become threaded into a common network of related subject matter. Hashtags can be used as well for finding users who are looking for topics on social media. Hashtags also have a ‘commentary’ function. For example, a linguistic marker referencing the target of evaluation in a tweet not only renders the content searchable, but also constitutes a means to affiliate with values expressed in the tweet. This has led some linguists to refer to hashtags as part of ‘searchable talk’, whose subtext is ‘Search for me and affiliate with my values’ (Zappavigna 2019). It also implies that hashtags can become ideological and politically subtle codes. In effect, hashtags are now intrinsic to a subtle discourse strategy, constituting a cue for expressing meaning and tone (irony, sarcasm, surprise, etc.). Memes are signs or texts of short duration that are transmitted throughout the internet virally (Canizzaro 2016; Danesi 2019). Meme theory provides a useful lexicon for describing the ‘contagious’ spread of something through the internet, even though it fails to explain why memes have a short life, whereas certain important ideas and movements do not. As is well known, the term was coined by sociobiologist Richard Dawkins in The Selfish Gene (1976) to explain how information gets passed on from generation to generation in a specific culture in order to enhance survivability and promote progress and further evolution by replicating the functions of genes. Culturally speaking, memes are structures of information (ideas, laws, clothing fashions, artworks, tunes and so on) and of behaviour (marriage rites, courtship rituals, religious ceremonies and so on) that people inherit directly from their cultural environments. An internet meme may be an image or a video, a hyperlink, a hashtag, a website or even just a word or phrase that spread ‘virally’ via social networks, blogs and email. This implies that internet culture is based on mimicry (memetic culture). Some see memes as a new form of gossip or a bizarre word-of-mouth speech in a global digital context, known as secondary orality. Primary orality, as Walter Ong (1982) explained at the threshold of the digital age, refers to a culture in which information

MEDIA/CULTURE STUDIES AND SEMIOTICS

61

and traditions are passed on from one generation to the next orally. Some forms of oral culture include poetry, folktales, and proverbs as well as magical spells, religious incantations, and stories of the past. Orality involves a face-to-face form of contact. Cyberspace has led to the emergence of a secondary form of orality, since social media interactions occur as if they were oral and their content is similar, with memes constituting a form of gossip or, often, a way of joking. As McLuhan (1964) argued, the primary form of orality is the fundamental one and still exists in many societies or groups, but in the age of electronic communications, a secondary form has emerged which has blended, strangely, with writing, transforming writing into a simulative-oral medium – it is elliptical and compressed (lacking many punctuation marks and correct grammar), in ways that parallel vocal speech; it is synchronous (occurs in real time) like vocal speech, while it is also editable, like writing; it is informal and highly conversational in style and mode of delivery; it encourages immediate replies from one’s interlocutor, as in oral communication.

CONCLUDING REMARKS Media semiotics, in both its classic and current versions, is about how meaning structures are embedded in different media and how they evolve over time through technological and social changes. It seeks to flesh out meanings from media texts, venues and performances through a series of specific notions, such as opposition theory, code, text, iconicity, simulacra, a-signification, deconstruction and a few others. It has provided a lexicon that allows scholars from various disciplines to extract from media semiotic analysis relevant findings and theories for their own utilization. The semiotic study of media culture is fundamentally an exercise in unravelling the psychological reasons why we use media to extend ourselves, thus aligning itself with McLuhan’s framework of the effects of our media on our evolution. Media semiotics started with Barthes, but it could be claimed, in fact, that the true founder was Marshall McLuhan, as intimated throughout this chapter. His views of how media extend the human body into the world, allowing it to do more than would be possible by biology, are actually a fundamental principle of semiotic science as a whole. McLuhan never used the word semiotics in his writings. But there is little doubt that many, perhaps all, of his ideas are fundamentally semiotic. A McLuhanesque media semiotics would put into its core approach a study of the ways in which new media change people and societies. It would constitute a form of semiotic investigation that goes beyond explorations of ways in which our individual intelligence is affected by new media. In addition to this, it would also explore ways in which the global village we inhabit, based on the internet, has led to a connected form of intelligence, as Toronto communications scholar Derrick De Kerckhove (1997) has argued, which means that the sum total of the ideas of people will be vastly more important than those of any one individual. He speculates that we are undergoing one of the greatest evolutionary leaps in the history of our species. The architecture of connected intelligence, as represented by the World Wide Web, resembles that of a vast brain whose cells and synapses are encoded in software and hardware that facilitate the assemblage of minds. Because of this, individual brains are able to see more, hear more and feel more. In this environment, experts are just one part of the collective mind, whose ideas are carried by software and hardware systems that overlap with them and with relevant data and information.

62

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

As the media for knowledge-making and communicating change, we develop a different sense of ourselves. Semiotics has always studied this sense in terms of the body-mindenvironment connection. It is thus well-adapted to study how this connection is unfolding in cyberspace. As McLuhan put it (1970: 180), ‘When the evolutionary process shifts from biology to software technology the body becomes the old hardware environment. The human body is now a probe, a laboratory for experiments.’

REFERENCES Adorno, T. W. (1941), The Culture Industry, London: Routledge. ­Albertazzi, D. and P. Cobley (2010), The Media: An Introduction, New York: Routledge. Arnold, M. (1869), Culture and Anarchy, London: Smith, Elder, & Co. Bakhtin, M. M. (1981), The Dialogic Imagination: Four Essays by M. M. Bakhtin, trans. C. Emerson and M. Holquist, Austin: Texas University Press. Barthes, R. (1957), Mythologies, Paris: Éditions du Seuil. Barthes, R. ([1964] 2004), ‘Rhetoric of the Image’, in C. Handa (ed.), Visual Rhetoric in a Visual World: A Critical Sourcebook, New York: Bedford/St. Martin’s. Barthes, R. (1975), The Pleasure of the Text, New York: Hill and Wang. Baudrillard, J. ([1981] 1994), Simulacra and Simulation, Ann Arbor: University of Michigan Press. Baudrillard, J. (1983), Simulations, New York: Semiotext(e). Bell, D. (1976), The Coming of Post-industrial Society, New York: Harper Colophon Books. Bignell, J. (2002), Media Semiotics: An Introduction, 2nd edn, Manchester: Manchester University Press. Canizzaro, S. (2016), ‘Internet Memes as Internet Signs: A Semiotic View of Digital Culture’, Sign Systems Studies, 44: 562–86. Cantril, H. (1940), The Invasion from Mars: A Study in the Psychology of Panic, Edison, NJ: Transaction Publishers. Carr, N. (2011), The Shallows: What the Internet Is Doing To Our Brains, New York: Norton. Carr, N. (2015), The Glass Cage: How Our Computers Are Changing Us, New York: Norton. Castells, M. (2011), The Rise of the Network Society, Hoboken: Wiley-Blackwell. Cheney-Lippold, J. (2017), We Are Data: Algorithms and the Making of our Digital Selves, New York: New York University Press. Danesi, M. (2015), ‘The Semiotics of the Mass Media’, in Peter Pericles Trifonas (ed.), International Handbook of Semiotics, New York: Springer. Danesi, M. (2016), The Semiotics of Emoji, London: Bloomsbury. Danesi, M. (2018), Understanding Media Semiotics, 2nd edn, London: Bloomsbury. Danesi, M. (2019), Memes and the Future of Popular Culture, Leiden: Brill. Dawkins, R. (1976), The Selfish Gene, Oxford: Oxford University Press. De Kerckhove, D. (1997), Connected Intelligence: The Arrival of the Web Society, Toronto: Somerville. Deleuze, G. and F. Guattari (1987), A Thousand Plateaus, Minneapolis: University of Minnesota Press. Derrida, J. (1976), Of Grammatology, trans. G. C. Spivak, Baltimore: Johns Hopkins Press. Derrida, J. (1978), Writing and Difference, London: Routledge & Kegan Paul. Donald, M. (2010), ‘The Exographic Revolution: Neuropsychological Sequelae’, in L. Malafouris and C. Renfrew (eds), The Cognitive Life of Things: Recasting the Boundaries of the Mind, 71–9, Cambridge: McDonald Institute Monographs.

MEDIA/CULTURE STUDIES AND SEMIOTICS

63

Eco, U. (1979), The Role of the Reader. Explorations in the Semiotics of Texts, Bloomington: Indiana University Press. Eco, U. (1992), Interpretation and Overinterpretation, Cambridge: Cambridge University Press. Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities, New York: Basic. Foucault, M. (1972), The Archeology of Knowledge, trans. A. M. Sheridan Smith, New York: Pantheon. Genosko, G. (2008), ‘A-signifying Semiotics’, The Public Journal of Semiotics, 11 (1): 11–21. Gerbner, G. (1956), ‘Toward a General Model of Communication’, AV Communication Review, 4: 171–99. ­Gerbner, G. and L. Gross (1976), ‘Living with Television: The Violence Profile’, Journal of Communication, 26 (2): 173–99. Gramsci, A. (1937), Letters from Prison, New York: Columbia University Press. Greenfield, S. (2015), Mind Change, New York: Random House. Group µ (1970), A General Rhetoric, Paris: Larousse. Group µ (1992), Traité du signe visuel: Pour une rhétorique de l’image, Paris: Seuil. Guattri, F. (1995), ‘Machinic Junkies’, in Soft Subversions, New York: Semiotext(e). Hadlington, L. (2017), Cybercognition, Los Angeles: Sage. Hall, S. (1978), Policing the Crisis, New York: Palgrave. Hawreliak, J. (2018), Multimodal Semiotics and Rhetoric in Videogames, New York: Routledge. Hayles, K. (2017), Unthought: The Power of the Cognitive Nonconscious, Chicago: The University of Chicago Press. Hebdige, D. (1979), Subculture: The Meaning of Style, London: Methuen. Heidegger, M. ([1953] 2010), Being and Time, trans. Joan Stambaugh, New York: State University of New York Press. Herman, E. and N. Chomsky (1988), Manufacturing Consent: The Political Economy of the Mass Media, New York: Pantheon. Hoggart, R. (1957), The Uses of Literacy, London: Chatto & Windus. Horkheimer, M. (1947), Eclipse of Reason, New York: Oxford University Press. Hosanagar, K. (2019), A Human’s Guide to Machine Intelligence: How Algorithms Are Shaping Our Lives and How We Can Stay in Control, New York: Viking. Jenkins, H. (2006), Convergence Culture: Where Old and New Media Collide, New York: New York University Press. Jensen, K. B. (1995), The Social Semiotics of Mass Communication, London: Sage. Johnson-Laird, P. N. (1983), Mental Models, Cambridge: Harvard University Press. Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason, Chicago: University of Chicago Press. Katz, E. (1959), ‘Mass Communication Research and the Study of Culture’, Studies in Public Communication, 2: 1–6. Katz, E. and P. F. Lazarsfeld (1955), Personal Influence: The Part Played by People in the Flow of Mass Communications, Glencoe: The Free Press. Lakoff, G. (1987), Women, Fire, and Dangerous Things: What Categories Reveal about the Mind, Chicago: University of Chicago Press. Lakoff, G. and M. Johnson (1980), Metaphors We Live By, Chicago: Chicago University Press. Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought, New York: Basic. Langlois, G. (2014), Meaning in the Age of Social Media, New York: Palgrave. Lasswell, H. D. (1927), Propaganda Techniques in World War I, Cambridge: MIT Press.

64

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Lazarsfeld, P. F., B. Berelson and H. Gaudet (1948), The People’s Choice, New York: Columbia University Press. Lazzarato, M. (2014), Signs and Machines: Capitalism and the Production of Subjectivity, South Pasadena: Semiotext(e). Latour, B. (2007), Reassembling the Social: An Introduction to Actor-Network-Theory, Oxford: Oxford University Press. Law, J. (2008), ‘Actor Network Theory and Material Semiotics’, in B. S. Turner (ed.), The New Blackwell Companion to Social Theory, 141–58, Oxford: John Wiley & Sons. Leavis, F. R. (1952), The Common Pursuit, London: Chatto & Windus. Lévi-Strauss, C. (1987), Introduction to Marcel Mauss, London: Routledge. ­Levy, A. (2005), Female Chauvinist Pigs: Women and the Rise of Raunch Culture, New York: Free Press. Lippmann, W. (1922), Public Opinion, New York: Macmillan. Lowenthal, L. (1949), Triumph of Mass Idols, New Brunswick: Transaction Books. Manovich, L. (2002), The Language of New Media, Cambridge: MIT Press. Marcuse, H. (1972), An Essay on Liberation, Harmondsworth: Penguin. Mazzali-Lurati, S. (2007), ‘Here Is the Author! Hyperlinks as Constitutive Rules of Hypertextual Communication’, Semiotica, 167 (2): 119–34. McCombs, M. and D. Shaw (1972), ‘The Agenda-Setting Function of Mass Media’, Public Opinion Quarterly, 36: 176–87. McLuhan, M. (1951), The Mechanical Bride: Folklore of Industrial Man, New York: Vanguard. McLuhan, M. (1962), The Gutenberg Galaxy, Toronto: University of Toronto Press. McLuhan, M. (1964), Understanding Media: The Extensions of Man, New York: McGraw Hill. McLuhan, M. (1970), Culture Is Our Business, New York: Wipf & Stock. McLuhan, M. and E. McLuhan (1988), Laws of Media: The New Science, Toronto: University of Toronto Press. McLuhan, M. and Q. Fiore (1967), The Medium is the Massage: An Inventory of Effects, New York: Random House. McRobbie, A. (1991), Feminism and Youth Culture: From ‘Jackie’ to ‘Just Seventeen’, Boston: Unwin Hyman. Minsky, M. and S. Papert (1969), Perceptrons: An Introduction to Computational Geometry, Cambridge, MA: MIT Press. Negroponte, N. (1995), Being Digital, New York: Knopf. Nesselroth, P. W. (2007), ‘Playing Doubles: Derrida’s Writing’, Semiotica, 166 (3): 427–32. Noelle-Neumann, E. (1984), The Spiral of Silence: Public Opinion – Our Social Skin, Chicago: University of Chicago Press. Nöth, W. (1997), Semiotics of the Media: State of the Art, Projects, and Perspectives, Berlin: Mouton de Gruyter. Ong, W. (1982), Orality and Literacy, New York: Methuen. Pasquinelli, M. (2017), ‘Machines that Morph Logic: Neural Networks and the Distorted Automation of Intelligence as Statistical Inference’, Glass Bead: Logic Gate, The Politics of the Artifactual Mind. Site 1, www.glass-bead.org/wp-content/uploads/GB_Site-1_MatteoPasquinelli_Eng.pdf Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Peirce, C. S. (1868 [1992]), ‘Some Consequences of Four Incapacities’, in Nathan Houser and Christian Kloesel (eds), The Essential Peirce 1 (1867–93), Bloomington: Indiana University Press.

MEDIA/CULTURE STUDIES AND SEMIOTICS

65

Phoca, S. and R. Wright (1999), Introducing Postfeminism, Cambridge: Icon Books. Pollio, H. R., J. M. Barlow, H. J. Fine and M. R. Pollio (1977), The Poetics of Growth: Figurative Language in Psychology, Psychotherapy, and Education, Hillsdale, NJ: Lawrence Erlbaum Associates. Poulsen, V., G. Kvåle and T. van Leeuwen, eds (2018), Social Media as Semiotic Technology, Special Issue of Social Semiotics, London: Routledge. de Saussure, F. (1916), Cours de linguistique générale, ed. C. Bally and A. Sechehaye, Paris: Payot. ­Scheufele, D. A. and P. Moy (2000), ‘Twenty-five Years of the Spiral of Silence: A Conceptual Review and Empirical Outlook’, International Journal of Public Opinion Research, 12: 3–28. Simonson, P. and J. Durham Peters (2008), ‘Communication and Media Studies, History to 1968’, in W. Donsbach (ed.), International Encyclopedia of Communication, 764–71, Oxford: Blackwell. Thompson, E. P. (1963), The Making of the Working Class, London: Vintage. Turner, Graeme (2002), British Cultural Studies: An Introduction, 3rd edn, London: Routledge. Unsworth, N., B. D. McMillan, G. A. Brewer and G. J. Spillers (2012), ‘Everyday Attention Failures: An Individual Differences Investigation’, Journal of Experimental Psychology Learning, Memory, and Cognition, 38 (6): 1765–72. Van Zoonen, L. (1994), Feminist Media Studies, London: Sage. Walsh-Matthews, S. and M. Danesi (2019), ‘AI: A Semiotic Perspective’, Chinese Semiotic Studies, 15 (2): 199–216. Warschauer, M. and D. Grimes (2007), ‘Audience, Authorship, and Artifact: The Emergent Semiotics of Web 2.0.’, Annual Review of Applied Linguistics, 1: 1–23. Wiener, N. (1948), Cybernetics, or Control and Communication in the Animal and the Machine, Cambridge: MIT Press. Williams, R. (1950), Reading and Criticism, London: Frederick Muller. Williams, R. (1977), Marxism and Literature, London: New Left Books. Wong, M. (2019), Multimodal Communication: A Social Semiotic Approach to Text and Image in Print and Digital Media, New York: Palgrave. Zappavigna, M. (2019), Searchable Talk: Hashtags and Social Media Metadiscourse, London: Bloomsbury.

66

­C HAPTER THREE

Digital Humanities and Semiotics ALIN OLTEANU AND ARIANNA CIULA

EXISTING APPROACHES: A MODELLING FOCUS The prominent role of modelling activities in digital humanities (McCarty 2005), hereafter DH, was recently addressed from a semiotic perspective (Ciula and Eide 2017; Ciula and Marras 2019), as a way to foster a modelling theory fitted for this new and quickly developing area of research. This approach is not unexpected, as semiotic theories can justify in many regards the use of digital methods in humanities research, which they can help develop. These two areas of scholarship share modelling, understood as schematic (diagrammatic), as a key interest. By schematic we refer to structures of meaning that are more accurately represented through diagrams than in verbal language. Given the swift uptake and advancement of digital technology in recent decades, converting (non-digital) objects into digital forms, processable by computers, a process termed digitization, and the analysis of those digitized objects via computational methods have acquired momentum in art and humanities research as well as in the cultural heritage sector. This has been occurring in parallel with the restructuring and expansion of human societies onto digital media, in general, a process termed digitalization (Brennen and Kreiss 2016). Digitalization produces a social networking effect: it merges human communication channels into vast networks, on a global scale, where nodes can be regarded as signal emitters. Thus, in social sciences and humanities research on digitalization, a schematic (network-like) notion of model can be observed. In this direction, we remark, among others, boyd’s (2014) approach to (cultural) identity vis-à-vis social networks and Hartley’s (2015) notion of urban semiosis. boyd explains that the structures of communication of social media networks have consequences for the mental modelling of audiences, which further has implications for the identities of individuals and, hence, social organization (see also Marwick and boyd 2010). The engineering of digital communication networks is not distinct or inconsequential for cultural modelling: ‘increasingly, software engineers are designing and building algorithms to observe people’s practices and interests in order to model who they are within a broader system. [. . .] What becomes visible – either through people or through algorithms – can affect how people understand social media and the world around them’ (boyd 2014: 160–1). Relying on Lotman’s semiotic theory of modelling, Hartley (2015: 80) explains that cities, as sites of creativity where systems ‘clash’, are not so much geographical locations as they are networks, irrespective of geographic proximity. Also, inspired by Lotman, as well as other major semiotic scholars, such as Umberto Eco, Bankov’s recent book (2022) addresses digital culture from a semiotic  perspective.

68

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

While  Bankov  does not focus on modeling and digital humanities methods, the book demonstrates the intricacies of semiotic theories and digital transformations. Construing models as networks is commonplace in semiotics. The diagram-centred semiotics of Peirce (see Pietarinen 2006; Stjernfelt 2007) has been given renewed interest recently because of its compatibility with the revolution in linguistics and philosophy of mind produced by the emergence of the notion of schema in cognitive linguistics (Johnson 1987; Lakoff 1987). In general, this schematic trend in linguistics and semiotics can be seen as aligned to, if not a major part of the iconic turn in the humanities (Moxey 2008; Boehm and Mitchell 2009). Here, we explain why it is relevant for DH that the primary consequence of the iconic or schematic turn has been a surge in the popularity of certain schools of semiotics. These are, to begin with, 1. Peirce’s semiotics, given the centrality of iconicity in this theory 2. Cognitive semiotics, where the cognitive linguistic notion of schema has been adopted with a semiotic scope 3. The multimodality framework, which considers meaning (expressed in text) as irreducible to a double articulation 4. Biosemiotics, given its focus on modelling and embodiment, and 5. Ecosemiotics, the sub-branch of biosemiotics that focuses on networks This chapter explains the synergy between semiotics and DH, particularly in relation to modelling, and suggests potential directions for further theoretical development, which could contribute to overcoming what has been called the crisis of the humanities (see Nussbaum 2010; Jay 2014), in itself of particular interest to semiotics (Martinelli 2016; Cobley 2017) as well as DH.1 The remediation processes connected to digitization of the objects of study of the humanities and their modelling are a critical matter in overcoming this impasse. Digitization – when dealing with conversion across media – and digitilization are forms of intermedial translation, hence of relevance to translation studies that found its theoretical grounding in semiotics (e.g. Elleström 2014; Pérez-González 2014; Kourdis 2015; O’Halloran et al. 2016). Observing that semiotic theories fit this purpose, O’Halloran (2015) established multimodal digital semiotics as a lens for multimodal digital humanities. As this approach is focused on the comprehensive analysis of multimodal texts, primarily relying on the traditional theoretical tools of systemic functional linguistics (Halliday 1978) and social semiotics (Kress and van Leeuwen [1996] 2006, 2010; O’Toole 2011), it does not present modelling as a theory-framing notion. It does, however, rely on modelling as an analytical method (O’Halloran 2015: 390), as common for DH, albeit within the limited context of cultural analytics. The adoption of semiotic modelling in the DH scholarship (Ciula and Eide 2017) followed one of the extrapolations of Charles Peirce’s semiotics as modelling theory, particularly as developed in a philosophy of science view (Kralemann and Lattmann 2011). Models and modelling are a common, central concern in both DH, in its research products and process, and in semiotics, as a theoretical cornerstone. Computers are modelling machines (see McCarty 2005: 22; Mahoney 2000, 2005) and, more often than not, semiotic theories are modelling theories. The work of signs, according to many accounts, is modelling (Lotman 1977, 1991; Myrdene and Merrell 1991; Sebeok 1991; Sebeok and Danesi 2000; see also Nöth 2018). To make sense of things, or, putting it differently, to discover meaning in experiencing the factual world is to make useful models of those things or facts. A commonly accepted hypothesis of semiotics is that meaning is a relation

DIGITAL HUMANITIES AND SEMIOTICS

69

(and models are relational), and thus to discover meaning is to establish (or observe) a relation. As such, the models by which semiotics describes the meaningful environments of living organisms have the structure of networks – interconnected sums of relations. For this reason, semiotic modelling theory particularly fits the work of and with computers. This semiotic perspective is of immediate value for DH (Ciula and Marras 2016, 2019). Connections between DH in its earliest manifestations (humanities computing) and semiotic theories have been more or less explicit. Pending a systematic review of these intersections, they can tentatively be clustered in three non-exclusive categories: 1. More or less explicit overarching references to historical analysis and philosophical underpinnings around the meaning of information, data, systems and infrastructures converging cyclically towards reflections and critiques around the agency of the machine and platforms in processing symbols in relation to human intelligence, interpretation and bias (in this respect Smithies (2017) offers an insightful analysis of DH seen from the socio-technical lens of the digital modern) 2. Practical approaches inspired by semiotic concepts to inform the creation of computable models and standards as well as interactive interfaces 3. Attempts to connect the two levels (theory and practice) in particular via a semiotic theorization of modelling Approaches that we subsume under 2 vary extensively. For example, they include mainly text-focused approaches grounded in philosophy of language and attempting to provide formal models of the linguistic sign in areas akin to Natural Language Processing methods and computational linguistics. This category can also account for user designoriented methods partially anchored to media studies (see, e.g. Drucker 2013), as well as to the field of human-computer interaction. Further, under the second category we can also group more inclusive analysis of cultural forms and objects leading for example towards the multimodal turn discussed further below. For example, the project of a semiotic branch of DH was addressed explicitly in multimodality studies, particularly within a project that Kay L. O’Halloran started at the University of Singapore (O’Halloran 2015: 389–90; see also O’Halloran et al. 2013, 2014). The third category is an area with potential that calls for further research. Indeed, while semiotics concepts have informed and continue to inspire DH research, only rarely a semiotic approach has been explicitly adopted to underpin data models (see for example Boot 2006). Particularly, the semiotic proposal in DH found inspiration in Peirce’s pragmatic and context-aware notion of meaning-making (semiosis) and in his notion of iconicity, as exploited in recent scholarship on modelling (Kralemann and Lattmann 2013) and medial iconicity (Elleström 2013). It is noteworthy that the proposal for a semiotic approach to DH modelling came from DH scholarship. So far, there seem to be only suggested implications coming from semiotic scholarship, given its ramifications into areas of research such as information theory, design and, in a very general sense, textual studies. Ciula and Eide’s (2017) initial proposal reveals numerous applications and can find support in semiotic theories broadly, beyond the sources explicitly explored so far. Semiotics can offer a robust philosophical and theoretical underpinning of DH for various reasons that have not been explicated yet in semiotic scholarship but are implicit in semiotic research on modelling, multimodality, translation, iconicity, embodiment, literacy and industry. Recent semiotic scholarship richly explores these topics of relevance for the DH, but there seems to be no remarkable attempt from this end at developing a semiotic theoretical underpinning for the DH.

70

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Also, while non-discursive languages are part of the epistemological foundations of DH (e.g. for visual languages see Drucker 2011), its most recognized landmarks are founded on (McCarty 2005; Berry 2012), or tend to be pulled towards, discursive theory,2 arguably a language-centred semiotic theory (Cobley 2016: 18, 95). Here we draw attention to a mismatch: as digit(al)ization boosts the modal plurality of social representations, thus bringing to the fore the (cross-)iconicity of semiotic systems, its language-centred construal is prone to miss many of its important characteristics. Model-focused semiotics, as mostly explored in light of the Peircean school, tends to distance from language-, discourse- and text-based conceptions in favour of more comprehensive schematism (e.g. Pietarinen 2006; Stjernfelt 2007, 2014; Cobley 2016). A semiotic exploration of possible contributions to DH should account for non-language-centred modelling tools, that is, develop a theory that does not conceive of meaning as necessarily dependent upon a linguistic representation. Language-centrism, as Kress and van Leeuwen point out (2001: 4, see also Olteanu 2019: 124, 2020), relies on a rigid notion of meaning as fixed form-content articulation. The limitations of this traditional linguistic axiom have been observed in DH, for instance in discussions on layered models for digitizing and interpreting handwritten objects, in digital palaeography (Ciula 2017). Arguably, recent trends in semiotics suggest that semiotic theory is particularly fit to apprehend the way in which digit(al)ization models human cultures and societies of the past, present and futures, in ‘data worlds’ (Gray 2018), structures of networks and infrastructures.3 Studying and informing the construction of these data worlds, networks and infrastructures falls within the scope of DH and digital social sciences more widely. As such, accounting for the entanglement of the technical and ethical dimensions of the modelling process in digital media and its development in light of semiotic theory could offer an opportunity to address the ongoing crisis of the humanities. The decline of humanities studies in universities and the relatively low financing that research funders direct to the humanities are due, among other things, to the uninvolvement of the humanities in engineering digital societies (see Smithies, ffrench and Ciula, forthcoming). A proper inter- (or trans-) disciplinary dialogue between research on meaning (semiotics) and infrastructure studies could open up opportunities in this respect.

MULTIMODALITY In the past two decades, discussions in social semiotics have been dominated by the concept of multimodality (Kress and van Leeuwen 2001), which has become a framework for communication studies altogether (Kress 2010). This term refers to the articulation of meaning through the use of the plurality of sense perception channels and representation modalities available. It is a development of and a critique of the traditional humanistic, literary notion of text (see, e.g., Lotman 1991), particularly as developed in systemic functional linguistics (see Halliday 1978; Halliday and Webster 2009). The need for such an approach to meaning and communication was inspired by the diversification of media technologies in recent decades. Particularly, this awareness was accelerated by digit(al)ization, as digital media afford and make use of designs which, to be produced and interpreted, require the bringing together of various modalities. As traditional theories that regard meaning as a double articulation of form and content miss essential aspects of multimodally constructed discourses, social semiotics drew its attention to the multimodal articulation of meaning. While the multimodality framework has become

DIGITAL HUMANITIES AND SEMIOTICS

71

highly popular, arguably becoming the default perspective of social semiotics, currently used in the study of many social phenomena, it has been seldom invoked in a full-fledged critique of digitalization (suggestions in this direction might be found, arguably, in some works such as Kress Selander 2012) or in regard to its potential relevance for DH (as gestured by Clivaz 2017).

TRANSLATION Translation appears an insightful area of interest at the intersection of semiotics and DH because it is a form of modelling. Any model-driven engineering activity requires an operation of translation across multiple dimensions (Génova et  al. 2009). As a poietic creation, conceptualizing and writing software in code is a matter of translation, from human models of reality and systems into computable or computational models. As noted above, digiti(li)zation is an intermedial and, also, intersemiotic translation which, following Jakobson (1959), stands for a translation from a linguistic to a non-verbal system. Digit(al)ization brings to the fore a panoply of examples of translation processes that do not involve language at all. As O’Halloran et al. (2016: 199) observe, For Jakobson, intersemiotic translation involved language. He does not discuss translation from one non-verbal semiotic system to another non-verbal semiotic system or the translation of multisemiotic texts. Because of its language-centrism tainting, Jakobson’s classic notion of intersemiotic translation is not a suitable theoretical lens for understanding digitization and modelling in DH. Moreover, modelling in DH reveals not only that some translation processes, such as in human-machine interaction, do not necessarily involve language but that monolingualism is a flawed doctrine altogether because translation and modelling themselves are always inter- or even poly-semiotic. As such, digitalization produced critical changes in translation theory and practice (see Olteanu 2020). A revealing example that theoretical work on digitization offers, which we discuss below, is the translation of existential graphs, as naturalistic representations of logical operations, into conceptual graphs, embodied in software design and development (Sowa 2000, 2014). Jakobson’s semiotic theory, given his view on language, could nevertheless provide another solid pillar for the semiotic take on DH. He was an avant-garde promoter of Peirce’s semiotics, observing that, as Peirce’s semiotics implies, iconicity is essential to language, much more so than pure convention, which allows for the articulation of opposites (Jakobson [1965] 1987). This was at least controversial, in the structuralist ambiance of the (semiotics of the) 1960s. His advocacy for iconicity in language implies what in contemporary terminology can be defined as the multimodal quality of language. Thus, we note the relevance of pursuing Jakobson’s iconicity-based semiotics, with critical revisions (e.g. with respect to translation) in the scope of DH. For the same reasons that led to the switch from double articulation to multimodality in social semiotics, in translation studies a semiotic turn can be noticed of recently. In a society that increasingly relies on intermedial and multimodal designs, traditional linguistic theories fall short in grasping the meaning effects of translating multimodal texts, particularly when such translations require cross-modal operations. Texts in contemporary advertising, marketing, education and art, to name just a few social practices, explicitly and purposefully require an awareness of their inherent multimodality in order

72

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

to be created and interpreted. Thus, the relevance of non-verbality and of schematicism in meaning-making has drawn the attention of translation studies (Bezemer and Kress 2008; Torresi [1998] 2009; Pérez-González 2014). For this reason, contemporary construals of translation tend to coincide with medial adaptation (Elleström 2014, 2017), or what in classic Greek terminology was expressed as ekphrasis. This indicates the necessity of an overarching theory, simultaneously comprehensive of both semiotics and mediality. The media semiotic theories of Danesi (2002) and Elleström (2014, 2018) present such perspectives. However, despite its critical innovations, Danesi’s theory holds an anchor in language-centred theories of (post)structural inheritance. A semiotic approach to DH should seek to clear away language-centred concepts of meaning. For example, the study of translation and multilingual corpora, which are traditionally approached from language-centred perspective, has provided opportunities in DH research for nonlanguage-centred analysis via data curation, processing and visualizations.4 Elleström’s theory (2013, 2014), as already illustrated by its adoption by Ciula and Eide (2016), constitutes a good starting point for a semiotic theory of digit(al)ization and of digital modelling. As its purpose is the thorough analysis of meaning effects of media, it can be (and has been at least in Eide 2015) employed as one core aspect of DH modelling, from a semiotic perspective. Following Peirce, translation ‘is semiotic mediation’ (Nöth 2012: 279), which leads to construing translation as a type of intermedial modelling. This opens up potentially new integrated approaches informed by semiotics to interactions between humans and computers and among humans within and across digital media. From a pragmatic perspective, models in DH are artefacts produced during modelling activities in iconic relation with other objects or phenomena they aim to represent, based on specific interests and biases, contexts and purpose. They are experienced, manipulated, reasoned upon artefacts of different nature. They are not only computational models instantiated into formal code but also sketches and diagrams, for example, often documented in stages of design and analysis as part of model-driven engineering processes of non-linear translations (for a recent contribution to the discussion on translation across models in DH from a research software engineering perspective, see Ciula et al. 2020 and forthcoming). As any translation is an interpretation and vice versa (e.g. Pajević 2014) and models are nonlanguage centred interpretations with specific interests, the theoretical construal of digital modelling can rely on (some accounts in) the vast semiotic scholarship on translation (e.g. Gorlée 1994; Stecconi 1994, 2004; Nöth 2012; Kourdis 2015). In turn, by addressing modelling as a translation process, translation studies would develop towards a more finely tuned understanding of the specific meaning effects of computational methods, digital media and platforms.

ICONICITY Elleström’s medium-centred theory of meaning and communication relies on the concept of iconicity, particularly as developed in light of Peirce’s semiotics. Peirce termed icon the sign type that signifies due to similarity. Iconicity refers to how, or on what basis does one phenomenon signify another by means of similarity (e.g. Stjernfelt 2007: 49). In a Peircean view, iconicity is present in any instance of meaning-making and, also, can constitute meaning on its own, without reliance on other types of representation. Given this aspect of Peirce’s semiotics, his work is one of the sources that inspired

DIGITAL HUMANITIES AND SEMIOTICS

73

the recent iconic turn (Moxey 2008; Boehm and Mitchell 2009), which started in the visual arts. While at first a matter of (visual) art research, the iconic turn has been gaining ground in the humanities and social sciences. It consists in the switch from verbal language to schematic structures as the basic and general vehicle of knowledge, understood as both a further development of the linguistic turn and a rejection of some of the latter’s central claims. In brief, the iconic turn regards operations with image schemata as basic for modelling. In philosophy of science, the widespread use of models and the iconic turn, away from analytical methods in scientific research, have been attributed to the prominence of computational models and the widespread adoption of visualization tools (Godfrey-Smith 2009: 108). As Boehm and Mitchell remark in their landmark conversation (2009), digitalization played an important role in stimulating the iconic turn. After this realization in visual art research, more fields of the humanities have followed suit, such as cognitive linguistics, phenomenology, theory of embodiment and semiotics. Also, the iconic turn is intimately related to the multimodality turn, as image schemata afford various simultaneous modalities, unlike the modelling at which linear, verbal language, particularly in the common perception of print mediality,5 aims.

EXISTENTIAL GRAPHS A central thesis in Peirce’s semiotics is that icons, that is, signifying relations of similarity, are the only sign types that can be used as predicates, thus having a critical role in logic. This position has been instrumental and of much interest particularly in the iconic turn in semiotics (Pietarinen 2006; Stjernfelt 2007), which is focused on Peirce’s notions of hypoicons (image, diagram, metaphor), as semiotic counterparts to the cognitive linguistic notion of schema. Peirce introduced and explained the notion of iconicity to justify the rationale of his system of graphic representations of logic (Stjernfelt 2015), termed existential graphs. Existential graphs have already been adopted in information systems development, as translated by Sowa (see 2000, 2014) into linearly represented conceptual graphs. Conceptual graphs have been included as a formal knowledge representation system in the ISO standard for common logic (see Sowa 2008). While Peirce found an advantage in representing logical operations graphically, his system of existential graphs proves useful to represent knowledge for computational processing because, while its translation into a linear syntax and semantics hides the schematic into a symbolic representation, it retains the iconic ground of logic. The purpose of existential graphs is ‘to afford a method (1) as simple as possible (that is to say, with as small a number of arbitrary conventions as possible), for representing propositions (2) as iconically, or diagrammatically and (3) as analytically as possible’ (Peirce 1905: CP 4.561). This is also the rationale within which Elleström develops his media communication theory, by considering that ‘[a] model should be understood as a clearly outlined cognitive scheme that is both described with the aid of language, and depicted as a diagram’ (2018: 270). Thus, as stated previously, modelling must be approached as a cross-modal process of translation. Following this logic, a conceptual graph is not a different model than its existential graph equivalent, but simply an analytic translation of its diagrammatic counterpart. The work of modelling consists in conceptualizing the graph and perceiving the correlation between its schematic and analytic representations. It deals with a process of cross-modal translation.

74

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Even if multiple approaches are possible, DH modelling has to take into consideration the underpinning justification of existential graphs, namely the iconicity of logic, and practised in the emerging adoption of graph databases and conceptual models in particular in relation to the Linked Open Data paradigm.6

EMBODIMENT The iconic and multimodal turns are related to yet another turn, namely an embodiment turn. Or, rather, these could be considered three aspects of one and the same turn currently occurring in the humanities. The idea that the vehicle of knowledge consists in schemata, some of which may be non-verbalizable, is connected with the idea that the mind is embodied (e.g. Lakoff and Johnson 1999; Merleau-Ponty 1995; Stjernfelt 2007, 2014; Mittelberg 2013, 2019). The multimodality framework brought awareness to the fact that humans have always made sense of their environment multimodally. Some media technologies, among which especially the conventions of print (with which we are so familiar in our daily encounters to the point of printing cultures codes being overlooked by most but bibliographic experts), have inculcated an illusion of modelling, and particularly of scholarly modelling disseminated in linear publications, as monomodal. The discovery that social semiotics arrived at, thus, is not multimodality itself but that monomodality is impossible, yet taken for granted in modern philosophy and science.7 Monomodality is an abstraction. Conceiving a monomodal model is not phenomenally possible for human beings or any animals that have semiotic competencies comprising a plurality of representation modalities. This is of some importance for thinking through digitalization and modelling in the digital humanities for at least two reasons. First, as observed by Danesi (2002), unlike previous technological media, digital media structure social awareness in ways that reintegrate body and mind, undoing the dichotomizing effect of print, which shaped modern dualism (analogically, for instance, consider the emergence of nationalism in modernity, as a social consequence of print-capitalism, according to Anderson [1983] 2006). Second, as observed by Elleström (2018), mediality must be understood as initially evoked by corporeality. Mediality consists in a relation between the body and the environment, which implies an affordance (Gibson 1979) perspective for media and representation (see Majchrzak et  al. 2013; Treem and Leonardi 2013). As we discuss below, the implications for ecology of digitalization should also be considered in light of modelling. Hence, as digital modelling unfolds – for example, in the translations across verbal and non-verbal languages, natural and programming languages, human and machine interfaces – it has to be construed in light of the human body’s basic semiotic competencies, as discussed in biosemiotics (Stjernfelt 2006; Campbell et al. 2020).

LITERACY A main concern of the humanities tradition, literacy is also a central concern for research on digitalization. Moreover, digitalization produced a turn in (new) literacy studies (Mills 2010), as well. A critical aspect of this turn, yet again, consists in a reorientation towards semiotic theories (Mills 2016: 65–6). The new literacies and the lifestyles that they (will) support (see boyd 2014: 176–97) are essential to modelling society

DIGITAL HUMANITIES AND SEMIOTICS

75

and culture. Since each medium presents different affordances to a given species or individual for representation – and, hence, for modelling – digital media have their own specific effects on humans’ modelling of their world. For instance, Anderson (1983) claimed that the spreading of literacy that was caused by print led to specific models of organizing society. Without wanting to embrace over-deterministic views nor to elect one culture and context as representative of global phenomena, the spreading of any medium-specific literacy skills is deemed to affect social restructuring (McLuhan 1995: 177–8; Olteanu 2020). As such, since digital models depend on models of the (socalled) empirical and social world of humans, there is no reason not to believe that the future development of DH modelling will be shaped by the prospects brought about by what is now termed digital literacy (Lankshear and Knobel 2008). Hartley (2011: 20–1) explains this notion as an extension ‘beyond the defensive notion of “critical reading” and “media literacy” as taught in schools, towards [. . .] a form of hands-on productive expression, taught by and within a milieu in which it is deployed, using multiplatform devices to “write” as well as “read” electronic media’. Because the theoretical notion of digital literacy has had to keep up with the fast development of digital technology lately, it is unlikely a long-term comprehensive definition will consolidate. In addition, the plural digital literacies (Lankshear and Knobel 2008: 2) better reflects the plurality of forms of literacy emerging in a dynamic digital media context. Literacy is essentially understood as consisting in those competences required for surviving and thriving in a certain human society.8 Human competences for modelling are intimately related to literacy. For reasons similar to the iconic turn and the semiotic turn in translation studies, the study of the specific literacies of digital media has found inspiration in semiotic theories (e.g. Scolari et  al. 2018). Also, the multimodality framework found literacy as a particularly rich area of application (e.g. Jewitt 2006; Bezemer and Kress 2008; Kress and Selander 2012). In light of the same limitation that literary theory encounters in tackling multimodal representations, Stables (2012: 76–7) proposes replacing the term literacy, which suggests competences for operating with fixed symbolic signs, namely letters, with a more comprehensive semiosy. This more holistic concept is also better suited to the DH context, where objects of study are surely verbal texts but also span visual objects, sculptures, maps, cultural artefacts of different kind, and derivative and intermediate products of modelling from diagrams, formulas and snippet of code to three-dimensional models. Natasa Lacković’s (2018, 2020) recent, impactful and distinctively Peircean perspective on critical thinking and multimodal literacy can provide insights on the mutuality between digital literacies and DH modelling; for, even the more software-intensive or mathematically informed branches of DH, it has been argued, critical and historical understanding of cultural objects cannot rely on statistical analysis alone, but need ‘critically analysing and historicizing of data set’ (Bode 2020). Typical of modernity, the traditional concept of literacy centres on competences of coding and decoding texts. Arguably, this construal of literacy is still applicable, but digitalization critically changed the concept of text. Thus, understanding the new literacies of the digital age depends on construals of digital textuality. Since digital modelling, in a very wide sense, is the result of the interaction between humans and computers, digital literacies are instrumental as to how digital modelling techniques and methodologies will develop. In light of the many social and cultural changes it entails, digitalization supposes different educational programmes and curricula

76

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

than developed, so far, in view of modern ideas of science and social organization. A main difference comes from the reality that digital technologies afford so many possibilities for industry, many of which unpredictable, that digital literacies cannot be exhausted by fixed lists (Lankshear and Knobel 2008). While this is particularly so in digitalized societies, it is a valuable insight that digitalization provides for educational philosophy and theory. Digitalization could well inspire an anti-deterministic philosophy, in contrast to mechanistic mindsets which see modernity unidirectionally shaped by technology. Nevertheless, the modelling work of digitalization can offer some intuitions as to what could be useful skills, as deliverable through education, be it formal or informal (see Scolari et al. 2018) and how professional carriers of the future could be affected by alternative paths suited to a maker-culture, beyond or to complement traditional academic research for the humanities.9 What is at stake in conceiving digital literacies and their apprehension by educational systems is, of course, the (future) competences of current students on job markets, as well as their social integration and mobility. Without ignoring the unpredictability of future digital technologies and markets, there must also be a large number of learnable and transferable predetermined skills pertaining to digital modelling, from high-quality coding in various programming languages to data analysis and visualization techniques. However, arguably, it is the process-oriented character of modelling that represents the most interesting pedagogical challenge. Processes of making and negotiating meaning collaboratively and translating across verbal and non-verbal languages are educationally more valuable than supposed educational outputs or goals, as recent semiotic approaches in philosophy of education also argue (e.g. Pesce 2010, 2020; Stables  2012, 2016; Stables et  al. 2018). Being literate, or semiate (following Stables  2012: 80), in the digital age can equate to any version of combinations of practical skills and knowledge that allow an individual (or community) to thrive, which means to be able to learn more. These skills and knowledge belong to a broad spectrum going from the basics of computational skills and modelling competences to advanced methods as required for a research software engineering career (see recent recommendations on training and education at the intersection between DH and data science suggested by McGillivray et al. 2020). In any instance, it is important not to lose out of sight that knowledge of how and when to use non-digital media is an essential part of digital literacies (Bawden 2008: 28). Thus, digital literacies require further investigation in light of the modelling processes, both by which they were established and by which they will likely foster further. Most important, semiotic investigation should aim at revealing the underpinning meaning patterns in digital modelling, which could be used to conceive general principles of digital literacies. Arguably, modelling, as a logical operation, should be taught as a curricular subject. Existential graphs could be one school subject, for instance. Existential graphs or other illustration of schematic logic, logical principles of DH modelling could be introduced in pre-university curricula. This argument posits that such a curricular use of semiotics would serve as a scaffolding for students’ acquisition of specific digital skills, such as translating across styles of reasoning and expressions including coding (see Scolari et al. 2018), and it is, therefore, pedagogically more valuable, from the perspective of liberal education, than the teaching of predetermined skills. This would mean the cultivation in schooling of a semiotic (networked, relational) manner of thinking.

DIGITAL HUMANITIES AND SEMIOTICS

77

INDUSTRY According to Floridi (2014), a current philosophy of information ought to account for the way Information and Communication Technologies have transformed ourselves and our environment (the infosphere) where, rephrasing Hegel, ‘what is real is informational and what is informational is real’ (Floridi 2014: 41): The most obvious way in which ICTs are transforming the world into an infosphere concerns the transition from analogue to digital and then the ever-increasing growth of the informational spaces within which we spend more and more of our time. [. . .] This radical transformation is also due to the fundamental convergence between digital tools and digital resources. The intrinsic nature of the tools (software, algorithms, databases, communication channels and protocols, etc.) is now the same as, and therefore fully compatible with, the intrinsic nature of their resources, the raw data being manipulated. [. . .]. In the infosphere, populated by entities and agents all equally informational, where there is no physical difference between processors and processed, interactions become equally informational. (Floridi 2014: 41) In the infosphere, industry is of course digital, which means digital modelled and engaged in digitally modelling. Hartley (2015) considers the Lotman’s celebrated concept of semiosphere is explicative for the cultural dynamics of creative industries. Similarly, O’Halloran (2013: 412) observes the need for a concept of ‘digital semiosphere for multimodal digital humanities research’ that ‘involves the development of computational tools and visualization techniques for analysing semiotic data’. As mentioned above, many areas of industry (cultural, advertising, marketing) are working with designs that require multimodal semiotic operations. As suggested in the previous section, for the newly emerging creative industries (e.g. Hartley 2005), learning a fixed list of skills and following of a predetermined school curriculum is deemed to be inadequate for a market geared towards digital creativity which favours, for example, collaborative thinking and making via rapid prototyping and participatory design. Traditional notions of literacy and text are unsuited and foreign to the digital job market. In addition to the acquisition of skills, students must cultivate creativity and imagination as semiotic resources that allow a high adaptability in a society where change is the status quo. Moreover, digitalization offers an opportunity, which humanity does not afford to miss, for an equitable and sustainable development. Arguably, the ongoing third industrial revolution (Rifkin 2011) consists in the merging of digital media networks with renewable energy grids. The resulting global network, which should distribute resources as necessary, exhibits a diagrammatic structure, akin to semiotic modelling theories. Recent scholarship alludes to the inherently semiotic structure of digitally modelled societies, which made the creative industries possible (Hartley 2015). Thus, a main concern of the semiotic approach to DH, not immune from the ethics in the infosphere, consists in modelling the social networks resulting from the digitalization of industry. Indeed, the integration of digital technology in the labour market come with benefits in the form of new behavioural affordances (Treem and Leonardi 2013: 178) as well as risks (which the DH community is increasingly aware of),10 if roles are not adequately professionalised.

78

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­A BIOSEMIOTIC APPROACH: FULL-BLOWN MODELLING THEORY Unlike previous technological media, the modelling work channeled via digital media is not uni-directionally linear or nucleic but spreads the flow of meaningful content in network structures. Semantic network analysis is one of several mainstream modelling techniques in DH. It is akin to the main motivation of the biosemiotic approach to culture and ecology, given the focus on schematic representation here (Kull 2003). The turn from ladder-like or tree-like to web-like models that ecology produced in the natural sciences is similar to the effect of (Peirce’s) semiotics in the humanities: it highlights the network structure of systems. Further, this reveals the insightfulness of a modelling approach, in contrast to the textual one, to culture and society. The semiotic modelling theory per excellence, which has not been considered yet in regard to the digital, is biosemiotics. From its conception, Thomas Sebeok (e.g. 1991) defined biosemiotics as a modelling theory. Sebeok was inspired by Lotman’s cultural semiotic theory, but also criticized it. He adopted Lotman’s idea that the work of signs is modelling, that language is a modelling system and that human culture and society are pregnant with supralinguistic, secondary modelling systems, such as art, science, technology and so on, that culture itself is such a top-layer modelling system. In accordance with contemporary evolution  theories (e.g. Gould and Vrba 1982), Sebeok argued that if meaning is to be conceived in terms of modelling systems, two levelled-modelling does not suffice. Since most of communication in the biological world is non-verbal, then the primary modelling system, it has to be assumed, is non-verbal. Language, then, is a secondary modelling system, dependent upon the primary system, and supralinguistic modelling is yet a third system. This has major consequences for conceptualizing meaning, language, culture and modelling itself. From Sebeok’s perspective, the primary, non-verbal modelling system is schematic, roughly corresponding to Peirce’s iconic representations (see also Sebeok and Danesi 2000). It first and foremost shapes an organism’s intimate, inner world. This undermines the rigid and traditional view in the humanities that modelling is textual (challenged by Bod 2018), an idea that otherwise runs through Lotman’s theory (1991). Notions such as culture text are deemed unnecessary, thus. Biosemiotics implies, as more recently explained by Cobley (2010, 2016), that cultural phenomena should be understood from the perspective of modelling. In the scope of the present paper, this implies that DH modelling processes must rely on the embodied, schematic human competences for modelling, as well as on language and, possibly, other medial modelling systems including those embodied by information agents and platforms (indeed both human and machine materiality affect our reasoning apparatus and its potential – e.g. see Dourish 2014). While Sebeok’s version of biosemiotics is foundational and still innovative, more recent developments on the modelling work of organisms must be considered. Particularly, holding on to a Peirce-faithful notion of iconicity, as the justification of existential graphs, reveals more finely grained modelling activities. From this perspective, perceptual judgements in the form of propositions have a pivotal role in modelling (Stjernfelt 2014). Perceptual judgements are the models, not necessarily cognitive, that an organism primarily experiences, as a reaction to its environmental situated presence. Their propositional structure affords, in view of the organism’s semiotic competences, both their decomposition into more basic sign types (predicates, icons, indexes) and the further using of these discovered signs as models or tools for modelling and their copulation with other signs, towards the construction of more

DIGITAL HUMANITIES AND SEMIOTICS

79

complex models, such as arguments (Stjernfelt 2014; Olteanu et al. 2020). In the scope of DH, such modelling operations can therefore also be multi-layered and distributed across human and technical objects (see Smithies, ffrench and Ciula, forthcoming), and can be communicated in the form of existential (or conceptual) graphs, or any other system of logical representation based on the principle of iconicity. Another more recent concept that biosemiotics can provide for such an analysis is that of semiotic scaffolding (e.g. Hoffmeyer 2007). From this perspective, modelling in a digital and computational environment must be understood as supported (scaffolded) by pre-existing human environmental and technological affordances. The futureoriented question, in turn, is what can and does modelling for-and-with digital tools and computational methods support further on. Such an approach is inherent of the positions of the iconic, multimodal and embodiment turns and can be further employed in the conceptualization of modelling in the digital humanities. Furthermore, for his semiotic modelling theory, Sebeok had another foundational source besides Peirce and Lotman, namely Jakob von Uexküll’s (1926) theoretical biology. One of the great merits of Uexküll’s biology, which Sebeok eagerly adopted, is the merging of the notions of modelling and environment in the celebrated Umwelt (arguably, a more obvious connection for a German-speaking scholar, such as Uexküll, than for a non-German one). Sebeok’s biosemiotics is rooted in the analysis of how cycles of perceptions and actions form an organism’s model(s) of the world (i.e. Umwelt) in terms of Peircean categories and sign types. Discussing both human apprehension and the work with and of computers in terms of modelling allows for a comparison of the two, which brings a compelling answer to McCarty’s (2005: 21) foundational idea for the DH that computer systems must have non-virtual corresponding models. Contradictions (aniconicity) between human semiotic competences and given digital modelling methods are thus cause for rethinking digital methods or questioning human understanding. The friction, when it occurs and is noticed, has epistemological potential.

DIGITAL MODELLING AND THE NATURAL ENVIRONMENT Further, this leads to interrogating the environmental aspects and implication of DH modelling. Biosemiotics produced an ecological semiotics theory, named ecosemiotics, on the ground that organisms have inner cognitive models of the environment (Nöth 1998), coinciding with the Uexküllean Innenwelt. Since these inner cognitive models are based on primary modelling, they are schematic and, thus, the scope of ecosemiotics is that of investigating ‘the impact of maps on the mapped’ (Maran and Kull 2014: 41). The assumption here is that organisms’ mapping of the environment impacts on the environment – a rather recent realization, missed by modern philosophy and its corresponding medialities. Thus, the question that all of this points to is the following: What do digital models instantiated in artefacts imply for the environment? In this regard, there is an unmissable hope, which scholarship has only scratched the surface of, and which semiotics could equip DH to exploit. According to Rifkin (2011), since digital media newtoks and renewable energy grids can be merged, then digital media can bring about a ‘biosphere consciousness’ (2011). Kull’s argument that semiotics and ecology are easily compatible because of the similarity of their underpinning models, as networks, endorses this (2003, see above). Also, inspired by Lotman’s cultural

80

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

theory, Hartley (2015) argues that the digitally modelled creative industries require a theoretical approach that takes the biosphere as a starting point. These are research avenues that semiotics, as a modelling theory, can help DH and data science more generally explore, providing both the theoretical justification and methodological means for doing so.

CONCLUSIONS This overview points to the vast possibilities of semiotic research methods in the scope of DH. While these two disciplines have many common interests and (at times, apparently accidental) intersections, we consider that a solid theoretical framing for their interdisciplinary merging revolves around the notions of model and modelling. These reveal common research avenues around key concepts such as mediality, multimodality, literacy, embodiment and language, to name some. Literature in this regard started to develop recently, but this new common philosophical direction of research remains largely unexplored. The digital turn in the humanities gives way to a semiotic approach by its consideration of the meaning effects that modelling unavoidably has. The reason for which DH finds modelling a central and framing concept has to do with modelling processes in general, often considered independently of mediality (see Bod 2018). By foregrounding the modelling work of humans in general, the digital turn in the humanities gives new scope to the schools of semiotics which have been traditionally model-centred. Processes of digit(al)ization reveal that meaning and, therefore, knowledge are always situated and embodied. This also highlights that media are semiotically non-neutral. In one of his last contributions to biosemiotics, Jesper Hoffmeyer (2018: 5) confirmed that digitization illustrates that knowledge is by default embodied: Knowledge is a bodily phenomenon, which is why we can’t obtain it – as people often seem to believe – simply by summing up information. The only reason why we can so easily retrieve any thinkable piece of information, is that Google already has digitised it. But in the very process of digitizing it, the information necessarily loses its natural connection to the rest of the world. Information is converted to fact. But this implies that the information must be given a new context – or otherwise remain incomprehensible to the human mind. Most of the time we manage this without giving it another thought, and experience the information as meaningful. This means that the information has now – unknowingly perhaps – become re-contextualised. Without this recontextualisation, we would have only the ‘naked information’, and no matter how many pieces of such ‘naked information’ we add, we will never reach any kind of understandable knowledge.

NOTES 1 As initiatives such as https://4humanities.org/ demonstrate. 2 See for example van Zundert et al. (2017) for code criticism in relation to the performative dimension of programming or Galey and Ruecker (2010) for prototypying as a form of argument in design practices. 3 See https://cistudies.org/ 4 See the experiments in https://dral.kdl.kcl.ac.uk/

DIGITAL HUMANITIES AND SEMIOTICS

81

­5 Note that philological, editorial and bibliographic studies informed by material culture approaches have studied the complex structure and layout of visual signs in print material and informed document and interface theories in humanities computing and DH (see, e.g., McGann 2014). 6 See, e.g. http://dataforhistory.org/ and https://lincsproject.ca/ 7 With important exceptions, unsurprisingly, in the study of the role of modelling and models in the natural, physical and social sciences (see Nersessian 2008 and Morgan 2012). 8 See the multi-dimensional definition of cultural literacy proposed at http://cleurope.eu/about/ key-concepts/ 9 See, for example, the Research Software Engineering career pathways emerging in laboratories such as King’s Digital Lab: https://www.kdl.kcl.ac.uk/blog/rse-career-development/; Smithies and Ciula (2020). 10 See, for example, https://dh2018.adho.org/en/precarious-labor-in-the-digital-humanities/

REFERENCES Anderson, B. ([1983] 2006), Imagined Communities: Reflections on the Origin and Spread of Nationalism, revised edn, London: Verso. Bankov, K. (2022). ‘The Digital Mind: Semiotic Explorations in Digital Culture’, Cham: Springer. Bawden, D. (2008), ‘Origins and Concepts of Digital Literacy’, in C. Knobel and M. Lankshear (eds), Digital Literacies, 17–32, New York: Peter Lang. Berry, D. M., ed. (2012), Understanding Digital Humanities, New York: Palgrave Macmillan. Bezemer, J. and G. Kress (2008), ‘Writing in Multimodal Texts: A Social Semiotic Account of Designs for Learning’, Written Communication, 25 (2): 166–95. Bod, R. (2018), ‘Modelling in the Humanities: Linking Patterns to Principles’, Historical Social Research, Supplement, 31: 87–95. Bode, K. (2020), ‘Why You Can’t Model Away Bias’, Modern Language Quarterly, 81 (1): 95–124. Boehm, G. and W. J. T. Mitchell (2009), ‘Pictorial Versus Iconic Turn: Two Letters’, Culture, Theory and Critique, 50 (2–3): 103–21. boyd, d. (2014), It’s Complicated: the Social Lives of Networked Teens, New Haven: Yale University Press. Boot, P. (2006), ‘Decoding Emblem Semantics’, Literary and Linguistic Computing, 21 (suppl. 1): 15–27. Brennen, J. S. and D. Kreiss (2016), ‘Digitalization’, in K. Jensen, C. T. Robert, J. D. Pooley and E. W. Rothenbuhler (eds), The International Encyclopedia of Communication Theory and Philosophy, Oxford: Wiley Blackwell. Campbell, C., A. Olteanu and K. Kull (2020), ‘Learning and Knowing as Semiosis: Extending the Conceptual Apparatus of Semiotics’, Sign Systems Studies, 47 (3–4): 352–81. Ciula, A. (2017), ‘Digital Palaeography: What Is Digital about It?’ Digital Scholarship in the Humanities, 32 (2): ii89–ii105. Ciula, A. and C. Marras (2016), ‘Circling around Text and Language: Towards “Pragmatic Modelling” in Digital Humanities’, Digital Humanities Quarterly, 10 (3): 17–29. Ciula, A. and C. Marras (2019), ‘Exploring a Semiotic Conceptualisation of Modelling in Digital Humanities Practices’, in A. Olteanu, A. Stables and B. Dumitru (eds), Meanings & Co.: The Interdisciplinarity of Communication, Semiotics and Multimodality, 33–52, Cham: Springer. ­Ciula, A. and Ø. Eide (2017), ‘Modelling in Digital Humanities: Signs in Context’, Digital Scholarship in the Humanities, 32 (1): i33–i46.

82

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Ciula, A., G. Noël, P. Caton, G. Ferraro, B. Maher, and M. Vieira. ‘The Place of Models and Modelling in Digital Humanities: Intersections with a Research Software Engineering Perspective’. In Book of Abstracts of Digital Humanities 2020. The Alliance of Digital Humanities Organizations. http://dx.doi.org/10.17613/789z-9p59. Ciula, A., G. Noël, P. Caton, G. Ferraro, T. Ong, J. Smithies, and M. Vieira. ‘The Place of Models and Modelling in Digital Humanities: Some Reflections from a Research Software Engineering Perspective’, in J. Burkhardt, E. Arestau, N. Herold, R. Sierig and M. Annisius (eds). Elisabeth Burr Festschrift, Leipzig: Universitätsbibliothek, Forthcoming. Clivaz, C. (2017), ‘Multimodal Literacies and Continuous Data Publishing: Une Question de Rythme’, in P. Boot, A. Cappellotto, D. Woot, F. Fischer, A. Kelly, A. Mertgens, A. M. Sichani, E. Spadini and H. van Hulle (eds), Advances in Digital Scholarly Editing, Papers presented at the DiXiT conferences in The Hague, 99–104, Leiden: Sidestone Press. Cobley, P. (2010), ‘The Cultural Implications of Biosemiotics’, Biosemiotics, 3: 225–44. Cobley, P. (2016), Cultural Implications of Biosemiotics, Dordrecht: Springer. Cobley, P. (2017), ‘What the Humanities Are for: A Semiotic Perspective’, in K. Bankov and P. Cobley (eds), Semiotics and Its Masters, 3–23, Berlin: de Gruyter Mouton. Danesi, M. (2002), Understanding Media Semiotics, London: Arnold. Dourish, P. (2014), ‘No SQL: The Shifting Materialities of Database Technology’, Computational Culture, 4, http://computationalculture.net/no-sql-the-shifting-materialitiesofdatabase-technology/ Drucker, J. (2011), ‘Humanities Approaches to Graphical Display’, Digital Humanities Quarterly, 5 (1), http://digitalhumanities.org/dhq/vol/5/1/000091/000091.html Drucker, J. (2013), ‘Performative Materiality and Theoretical Approaches to Interface’, Digital Humanities Quarterly, 7 (1), http://www.digitalhumanities.org/dhq/vol/7/1/000143/000143.html Eide, Ø. (2015), Media Boundaries and Conceptual Modelling: Between Texts and Maps, London: Palgrave Macmillan. Elleström, L. (2013), ‘Spatiotemporal Aspects of Iconicity’, in L. Elleström, O. Fischer and C. Ljungberg (eds), Iconic Investigations, 95–117, Amsterdam: John Benjamins. Elleström, L. (2014), Media Transformation: The Transfer of Media Characteristics among Media, London: Palgrave Macmillan. Elleström, L. (2017), ‘Transfer of Media Characteristics among Dissimilar Media’, Palabra Clave, 20 (3): 663–85. Elleström, L. (2018), ‘A Medium-centered Model of Communication’, Semiotica, 224: 269–93. Floridi, L. (2014), The Fourth Revolution: How the Infosphere Is Reshaping Human Reality, Oxford: Oxford University Press. Galey, A. and S. Ruecker (2010), ‘How a Prototype Argues’, Literary and Linguistics Computing, 25 (4): 405–24. Gibson, J. J. (1979), The Ecological Approach to Visual Perception, Boston: Houghton Mifflin. Génova, G., M. C. Valiente and M. Marrero (2009), ‘On the Difference between Analysis and Design, and Why It Is Relevant for the Interpretation of Models in Model Driven Engineering’, The Journal of Object Technology, 8 (1): 107–27. Godfrey-Smith, P. (2009), ‘Models and Fictions in Science’, Philosophical Studies, 143 (1): 101–16. Gorlée, D. L. (1994), Semiotics and the Problem of Translation: With Special Reference to the Semiotics of Charles S. Peirce, Amsterdam: Rodopi. Gould, S. J. and E. S. Vrba (1982), ‘Exaptation – A Missing Term in the Science of Form’, Paleobiology, 8 (1): 4–15. ­Gray, J. (2018), ‘Three Aspects of Data Worlds’, Krisis: Journal for Contemporary Philosophy, 37 (1): 4–17, https://archive.krisis.eu/three-aspects-of-data-worlds/

DIGITAL HUMANITIES AND SEMIOTICS

83

Halliday, M. A. K. (1978), Language as Social Semiotic: The Social Interpretation of Language and Meaning, London: Edward Arnold. Halliday, M.A.K. and J. J. Webster, eds (2009), Continuum Companion to Systemic Functional Linguistics, New York: Continuum. Hartley, J., ed. (2005), Creative Industries, Oxford: Blackwell Publishing. Hartley, J. (2011), The Uses of Digital Literacy, London: Transaction Publishers. Hartley, J. (2015), ‘Urban Semiosis: Creative Industries and the Clash of Systems’, International Journal of Cultural Studies, 18 (1): 79–101. Hoffmeyer, J. (2007), ‘Semiotic Scaffolding of Living Systems’, in M. Barbieri (ed.), Introduction to Biosemiotics: The New Biological Synthesis, 49–166, Dordrecht: Springer. Hoffmeyer, J. (2018), ‘Knowledge Is Never Just There’, Biosemiotics, 11 (1): 1–5. Jakobson, R. (1959), ‘On Linguistic Aspects of Translation’, in R. A. Brower (ed.), On Translation, 232–9, Cambridge: Harvard University Press. Jakobson, R. ([1965] 1987), ‘Quest for the Essence of Language’, in K. Pomorska and S. Rudy (eds), Language in Literature, 413–27, Cambridge: Belknap Harvard. Jay, P. (2014), The Humanities ‘Crisis’ and the Future of Literary Studies, New York: Palgrave Macmillan. Jewitt, C. (2006), Technology, Literacy and Learning: A Multimodal Approach, Abingdon: Routledge. Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination and Reason, Chicago and London: The University of Chicago Press. Kourdis, E. (2015), ‘Semiotics of Translation: An Interdisciplinary Approach to Translation’, in P. P. Trifonas (ed.), International Handbook of Semiotics, 303–20, Dordrecht: Springer. Kralemann, B. and C. Lattmann (2011), ‘The Semantics of Models: A Semiotic Philosophy of Science Approach’, in K. D. Schewe and B. Thalheim (eds), Semantics in Data and Knowledge Bases, 50–69, Dordrecht: Springer. Kralemann, B. and C. Lattmann (2013), ‘Models as Icons: Modelling Models in the Semiotic Framework of Peirce’s Theory of Signs’, Synthese, 190: 3397–420. Kress, G. (2010), Multimodality: A Social Semiotic Approach to Contemporary Communication, London: Routledge. Kress, G. and T. van Leeuwen (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, London: Arnold. Kress, G. and T. van Leeuwen ([1996] 2006), Reading Images: The Grammar of Visual Design, 2nd edn, London: Routledge. Kress, G. and S. Selander (2012), ‘Multimodal Design, Learning and Cultures of Recognition’, Internet and Higher Education, 15: 265–8. Kull, K. (2003), ‘Ladder, Tree, Web: The Ages of Biological Understanding’, Sign Systems Studies, 31 (2): 589–603. Lacković, N. (2018), ‘Analysing Videos in Educational Research: An “Inquiry Graphics” Approach for Multimodal, Peircean Semiotic Coding of Video Data’, Video Journal of Education and Pedagogy, 3 (6): 1–23. Lacković, N. (2020), ‘Thinking with Digital Images in the Post-truth Era: A Method in Critical Media Literacy’, Postdigital Science and Education, 2: 442–62. Lacković, N. and A. Olteanu (2020), ‘Rethinking Educational Theory and Practice in Times of Visual Media: Learning as Image-concept Integration’, Educational Philosophy and Theory, 53 (6): 597–612. DOI: 10.1080/00131857.2020.1799783. ­Lakoff, G. (1987), Women, Fire, and Dangerous Things: What Categories Reveal about the Mind. Chicago, London: University of Chicago Press.

84

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought, New York: Basic Books. Lankshear, C. and M. Knobel, eds (2008), Digital Literacies: Concepts, Policies and Practices, New York: Peter Lang. Lotman, J. (1977), ‘Primary and Secondary Communication-modelling Systems’, in D. P. Lucid (ed.), Soviet Semiotics: An Anthology, Baltimore: Johns Hopkins University. Lotman, J. (1991), Universe of the Mind: A Semiotic Theory of Culture, trans. A. Shukman, London: I.B. Tauris. Mahoney, M. (2000), ‘Historical Perspectives on Models and Modelling’, Zürich: DHS– DLMPS Joint Conference 13, on ‘Scientific Models: Their Historical and Philosophical Relevance’, www.princeton.edu/~hos/Mahoney/articles/models/models.html Mahoney, M. (2005), ‘The Histories of Computing(s)’, Interdisciplinary Science Reviews, 30 (2): 119–35. Majchrzak, A., S. Faraj, G. C. Kane and B. Azad (2013), ‘The Contradictory Influence of Social Media Affordances on Online Communal Knowledge Sharing’, Journal of ComputerMediated Communication, 19 (1): 38–55. Martinelli, D. (2016), Arts and Humanities in Progress: A Manifesto of Numanities, Cham: Springer. Marwick, A. E. and d. boyd (2010), ‘I Tweet Honestly, I Tweet Passionately: Twitter Users, Context Collapse, and the Imagined Audience’, New Media & Society, 13 (1): 114–33. Maran, T. and K. Kull (2014), ‘Ecosemiotics: Main Principles and Current Developments’, Geografiska Annaler: Series B, Human Geography, 96 (1): 41–50. McCarty, W. (2005), Humanities Computing, New York: Palgrave Macmillan. McGann, J. (2014), A New Republic of Letters: Memory and Scholarship in the Age of Digital Reproduction, Cambridge: Harvard University Press. McGillivray, B., B. Alex, S. Ames, G. Armstrong, D. Beavan, A. Ciula, G. Colavizza, J. Cummings, D. De Roure, A. Farquhar, S. Hengchen, A. Lang, J. Loxley, E. Goudarouli, F. Nanni, A. Nini, J. Nyhan, N. Osborne, T. Poibeau, M. Ridge, S. Ranade, J. Smithies, M. Terras, A. Vlachidis and P. Willcox (2020), The Challenges and Prospects of the Intersection of Humanities and Data Science: A White Paper from The Alan Turing Institute, Figshare, dx.doi.org/10.6084/m9.figshare.12732164 McLuhan, M. (1995), Essential McLuhan, ed. E. McLuhan and F. Zingrone, London: Routledge. Merleau-Ponty, M. (1995), La Nature. Notes. Cours de Collège de France, Paris: Seuil. Mills, K. A. (2010), ‘A Review of the “Digital Turn” in the New Literacy Studies’, Review of Educational Research, 80 (2): 246–71. Mills, K. A. (2016), Digital Literacy Theories for the Digital Age: Social, Critical, Multimodal, Spatial, Material and Sensory Lenses, Bristol, Buffalo, Toronto: Multilingual Matters. Mittelberg, I. (2013), ‘The Exbodied Mind: Cognitive-semiotic Principles as Motivating Forces in Gesture’, in C. Müller, A. Cienki, E. Fricke, S. Ladewig, D. McNeil and S. Teßendorf (eds), Body – Language – Communication: An International Handbook in Human Interaction, vol. 1, 755–84, Berlin: De Gruyter Mouton. ­Mittelberg, I. (2019), ‘Visuo-kinetic Signs Are Inherently Metonymic: How Embodied Metonymy Motivates Forms, Functions, and Schematic Patterns in Gesture’, Frontiers in Psychology, 10: Article 254. Morgan, M. (2012), The World in the Model, Cambridge: Cambridge University Press. Moxey, K. (2008), ‘Visual Studies and the Iconic Turn’, Journal of Visual Culture, 7 (2): 131–46.

DIGITAL HUMANITIES AND SEMIOTICS

85

Myrdene, A. and F. Merrell, eds (1991), On Semiotic Modelling, Berlin: Mouton de Gruyter. Nersessian, N. J. (2008), Creating Scientific Concepts, Cambridge: MIT Press. Nöth, W. (1998), ‘Ecosemiotics’, Sign Systems Studies, 26: 332–43. Nöth, W. (2012), ‘Translation and Semiotic Mediation’, Sign Systems Studies, 40 (3–4): 279–98. Nöth, W. (2018), ‘The Semiotics of Models’, Sign Systems Studies, 46 (1): 7–43. Nussbaum, M. C. (2010), Not for Profit: Why Democracy Needs the Humanities, Princeton: Princeton University Press. O’Halloran, K. (2015), ‘Multimodal Digital Humanities’, in P. P. Trifonas (ed.), International Handbook of Semiotics, 389–415, Dordrecht: Springer. O’Halloran, K. L., M. K. L. E., A. Podlasov and S. Tan (2013), ‘Multimodal Digital Semiotics: The Interaction of Language with Other Resources’, Text and Talk: Special Edition for Michael Halliday, 33 (4–5): 665–90. O’Halloran, K. L., A. Chua and A. Podlasov (2014), ‘The Role of Images in Social Media Analytics: A Multimodal Digital Humanities Approach’, in D. Machin (ed.), Visual Communication, 565–88, Berlin: de Gruyter. O’Halloran, K. L., S. Tan and P. Wignell (2016), ‘Intersemiotic Translation as Resemiotization: A Multimodal Perspective’, Signata: Annals des sémiotiques/Annals of Semiotics, 7: 199–229. O’Toole, M. ([2010] 2011), The Language of Displayed Art, 2nd edn, London: Routledge. Olteanu, A. (2019), Multiculturalism as Multimodal Communication: A Semiotic Perspective, Cham: Springer. Olteanu, A. (2020), ‘Translation from a Contemporary Media Perspective: Avoiding Culturalism and Monolingualism’, Social Semiotics, aop: doi.org/10.1080/10350330.2020. 1714204. Olteanu, A., C. Campbell and F. Sebastian (2020), ‘Naturalizing Models: New Perspectives in a Peircean key’, Biosemiotics, 13: 179–97. Pajević, M. (2014), ‘Translation and Poetic Thinking’, German Life and Letters, 67 (1): 6–21. Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Pesce, S. (2010), ‘From Semiotics of Teaching to Educational Semiotics’, in I. Semetsky (ed.), Semiotics, Education, Experience, 99–113, Rotterdam: Sense Publishers. Pesce, S. (2020), ‘From Analogical to Analytical Thinking and Back: The Adaptation of Teacher’s Reasoning to Complex Situations’, in H. Yama and V. Salvano-Pardieu (eds), Adapting Human Thinking and Moral Reasoning in Contemporary Society, 223–44. IGI Global. Pérez-González, L. (2014), ‘Multimodality and Translation and Interpreting Studies: Theoretical and Methodological Perspectives’, in S. Bermann and C. Porter (eds), A Companion to Translation Studies, 119–31, Oxford: Wiley Blackwell. Petrilli, S. (2003), ‘Translation and Semiosis: Introduction’, in S. Petrilli (ed.), Translation, Translation, 17–37, Amsterdam: Rodopi. Pietarinen, A. V. (2006), Signs of Logic: Peircean Themes on the Philosophy of Language, Games, and Communication, Dordrecht: Springer. Rifkin, J. (2011), The Third Industrial Revolution: How Lateral Power Is Transforming Energy, the Economy, and the World, London: PalgraveMacmillan. ­Scolari, C. A., M. J. Masanet, M. Guerrero-Pico and M. J. Establés (2018), ‘Transmedia Literacy in the New Media Ecology: Teens’ Transmedia Skills and Informal Learning Strategies’, El Profesional de la Informacion, 27 (4): 801–12.

86

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Sebeok, T. (1991), A Sign Is Just a Sign: Advances in Semiotics, Bloomington: Indiana University Press. Sebeok, T. A. and M. Danesi (2000), The Forms of Meaning: Modelling Systems Theory and Semiotic Analysis, Berlin: Mouton de Gruyter. Smithies, J. (2017), The Digital Humanities and the Digital Modern, London: Palgrave Macmillan. Smithies, J. and A. Ciula (2020), ‘Humans in the Loop: Epistemology & Method in King’s Digital Lab’, in K. Schuster and S. Dunn (eds), Routledge International Handbook of Research Methods in Digital Humanities, 155–72, London: Routledge. Smithies, J., P. ffrench, and A. Ciula. ‘Droit de Cité: Towards a Socialized Model for Technology Development in the Arts & Humanities’, in U. Pawlicka-Deger and C. Thomson (eds.) Digital Humanities and Laboratories: Perspectives on Knowledge, Infrastructure and Culture, edited. Routledge, Forthcoming. Stables, A. (2012), Be(com)ing Human: Semiosis and the Myth of Reason, Rotterdam: Sense Publishers. Stables, A. (2016), ‘Education as Process Semiotics: Towards a New Model of Semiosis for Teaching and Learning’, Semiotica, 212: 45–58. Stables, A., W. Nöth, A. Olteanu, S. Pesce and E. Pikkarainen (2018). Semiotic Theory of Learning: New Perspectives in the Philosophy of Education, London: Routledge. Stecconi, U. (1994), ‘Peirce’s Semiotics for Translation’, Koiné, 4: 161–80. Stecconi, U. (2004), ‘Interpretive Semiotics and Translation Theory: The Semiotic Conditions to Translation’, Semiotica, 150: 471–89. Stjernfelt, F. (2006), ‘The Semiotic Body. A Semiotic Concept of Embodiment?’ in W. Nöth (ed.), Semiotic Bodies, Aesthetic Embodiments and Cyberbodies, 13–48, Kassel: Kassel University Press. Stjernfelt, F. (2007), Diagrammatology: And Investigation on the Borderlines of Phenomenology, Ontology and Semiotics, Dordrecht: Springer. Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns, Boston: Docent Press. Stjernfelt, F. (2015), ‘Iconicity of Logic and the Roots of the “Iconicity” Concept’, in M. K. Hiraga, W. J. Herlofsky, K. Shinohara and K. Akita (eds), Iconicity: East meets West, 35–53, Amsterdam: John Benjamins. Sowa, J. F. ([1999] 2000), Knowledge Representation: Logical, Philosophical, and Computational Foundations, Pacific Grove: Brooke Cole Publishing Co. Sowa, J. F. (2008), ‘Conceptual Graphs’, Foundations of Artificial Intelligence, 3: 213–37. Sowa, J. F. (2014), ‘From Existential Graphs to Conceptual Graphs’, in L. Johnston, K. Wolfe, J. Yoder and A. DeMarco (eds), Computational Linguistics: Concepts, Methodologies, Tools, and Applications, 439–72, Hershey: IGI Global. Torresi, I. ([1998] 2009), ‘Advertising’, in M. Baker and G. Saldanha (eds), Routledge Encyclopedia of Translation Studies, 2nd edn, 6–10, Oxford: Routledge. Treem, J. W. and P. M. Leonardi (2013), ‘Social Media Use in Organizations: Exploring Affordances of Visibility, Editability, Persistence, and Association’, Annals of the International Communication Association, 36 (1): 143–89. Uexküll, J. von. (1926), Theoretical Biology, London: Kegan Paul, Trench, Trubner & Co. van Zundert, J. J. and R. Haentjens Dekker (2017), ‘Code, Scholarship, and Criticism: When Is Coding Scholarship and When Is It Not?’ Digital Scholarship in the Humanities, 32 (suppl. 1): i121–i123.

­C HAPTER FOUR

Systems Theory and Semiotics RICARDO GUDWIN AND JOÃO QUEIROZ

INTRODUCTION Here, we explore the idea that a systems theory-based approach can help us to define more accurately many of the properties of semiosis for C. S. Peirce. For the sake of our arguments, an approach to semiosis (from Peirce’s mature semiotics) requires the notion of an open self-organized emergent and anticipatory system evolving in time. This theoretical position has many implications. A number of investigators maintain that semiosis (meaning process), and its metaphysical counterpart, must be considered in terms of complex adaptive systems (see Merrell and Queiroz 2009; Loula and Queiroz 2011; Bickhard 2007; Steels 1999; Steels and Kaplan 1999; Wagner et al. 2003; Christiansen and Kirby 2003; Cangelosi 2002; Cangelosi and Parisi 1998; Cangelosi and Turner 2002; Vogt 2007; Vogt and Coumans 2003; Jung and Zelinsky 2000; Briscoe 1998). We can say that, from a Peircean perspective, there are reactive and semiotic systems. For the cognitive scientists Nolfi and Floreano (2002: 93), reactive systems are systems in which sensors and motors are directly linked and which always react to the same sensory state with the same motor action. In these systems, internal states, if any, play a comparatively limited role in determining the motor output of the system. By contrast, in semiotic systems, some (internal or external) states are crucial for the interpretation of other states perceived through sensory apparatuses, and, as sign interpretation plays a fundamental role in the processes through which the system answers to environmental cues, different actions can result from the same sensory state, depending on the relationships established within the system and between the system and its semiotic niche. A ‘semiotic system’ is a system that produces, transmits, receives, processes and interprets signs of different types. As Fetzer (1997: 358) writes: ‘What makes a system “semiotic” thus becomes that the behaviour of the system is causally affected by the presence of a sign because that sign stands for something else iconically, indexically, or symbolically, for that system.’ Signs can be found in many domains and levels of descriptions – physico-chemical (physiosemiosis), biological (biosemiosis), animal (zoosemiosis), ecological (ecosemiosis), computational (artificial semiosis), cognitive, social, etc. By interpreting signs, semiotic systems show self-corrective behaviour (Ransdell 1977: 162). Such self-corrective behaviour depends on the capability of semiotic systems of using signs as media for

88

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

the communication of habits (regular patterns of semiotic activity) from objects to interpretants so as to constrain their own behaviour (Queiroz and El-Hani 2006).

PEIRCE’S PROCESS SEMIOTICS: SOME BASIC NOTIONS Peirce’s process semiotics is grounded on a list of logical-phenomenological categories  – Firstness, Secondness, Thirdness – which corresponds to an exhaustive system of hierarchically organized classes of relations (Houser 1997). This system makes up the formal foundation of his model of semiosis as a process and of his classifications of signs (Murphey 1993: 303–6). Firstness as a mode of being is related to the modality of possibility. It is the category of vagueness and novelty – ‘the mode of being which consists in its subject’s being positively such as it is regardless of anything else. That can only be a possibility’ (Peirce 1903: CP 1.25). Secondness is the mode of being ‘which is as it is relatively to a Second but regardless of any Third’ (1898: CP 6.200). It is a kind of reaction. Like Firstness, Secondness can be related to a modality, namely, the modality of actuality (Peirce 1908: CP 6.455; Parker 1998). The actuality of a thing is simply its occurrence. Rephrased, actuality is the realization of a possibility, without thereby making reference to something larger, be that a general law or an interpretation. Peirce considered ‘the idea of any dyadic relation not involving any third as an idea of secondness’ (1904: CP 8.330). Thirdness is the category of mediation, habit, generality and conceptualization (Peirce 1895: CP 1.340). The example par excellence is Peirce’s semiotic process (semiosis) in which a sign is related to an object by mediation through an interpretant. Semiosis implies process (Queiroz and Merrell 2008). Indeed, one of the most remarkable characteristics of Peirce’s theory of signs is its processual nature. As a genuine process thinker, it was quite natural that Peirce conceived semiosis as a process in which triads (sign-object-interpretant) are systematically linked to one another so as to form a web (Gomes et al. 2007). In this regard, we follow Rescher in his definition of a process as ‘a coordinated group of changes in the complexion of reality, an organized family of occurrences that are systematically linked to one another either causally or functionally’ (1996: 38). Peirce developed, between 1867 and 1908–11, a model of sign as process, and designed elaborated divisions of signs in order to describe these processes. According to Peirce, semiosis (sign in action) entails an irreducible triadic relation between a sign, its object and its interpretant (1902: CP 2.171; 1903: 2.274). That is, according to Peirce, any description of semiosis involves a relation constituted by three irreducibly connected terms, which are its minimal constitutive elements (1907: MS 318.81; 1903: CP 2.242). In Peirce’s words: My definition of a sign is: A Sign is a Cognizable that, on the one hand, is so determined (i.e., specialized, bestimmt) by something other than itself, called its Object, while, on the other hand, it so determines some actual or potential Mind, the determination whereof I term the Interpretant created by the Sign, that that Interpreting Mind is therein determined mediately by the Object. (n.d.: CP 8.177; emphasis in the original) ­In his ‘most fundamental division of signs’ (1903: CP 2.275), Peirce characterized icons, indexes and symbols as matching, respectively, relations of similarity, contiguity and

SYSTEMS THEORY AND SEMIOTICS

89

law between S and O (sign-object relation) in the triad S-O-I. Icons are signs which stand for their objects through similarity or resemblance (Peirce 1903: CP 2.276), irrespective of any spatio-temporal physical correlation that S may have with an existent O (Peirce 1893: CP 2.299). If a determinative relation of the sign (S) by the object (O) is a relation of analogy, that is, if S is a sign of O in virtue of a certain quality that S and O share, then S is an icon of O. If S is an icon, then S communicates to I a quality of O: ‘An Icon is a sign which refers to the Object that it denotes merely by virtue of characters of its own, and which it possesses, just the same, whether any such Object actually exists or not’ (1903: CP 2.247). In contrast, if S is a sign of O by reason of ‘a direct physical connection’ between them (Peirce 1885: CP 1.372), then S is said to be an index of O. In that case, S is really determined by O, and both must exist as events: ‘An Index is a sign which refers to the Object that it denotes by virtue of being really affected by that Object’ (Peirce 1903: CP 2.248). The notion of spatio-temporal co-variation is the most characteristic property of indexical processes. The examples range from a pronoun, demonstrative or relative, which ‘forces the attention to the particular object intended without describing it’ (Peirce 1885: CP 1.369), to physical symptoms of diseases, photographs, weathercocks, thermometers. In a Peircean analysis, small red spots in a child’s skin, for instance, can be treated as a sign (S) which stands for a disease, say, measles, its object (O), so as to constrain its interpretant, the effect the red spots have on an interpreter, say, a doctor performing a diagnosis. The sign processes involved are indexical. Finally, in a symbolic relation, the interpretant stands for ‘the object through the sign’ by a determinative relation of law, rule or convention (Peirce 1903: CP 2.276). According to Peirce (1901: CP 2.307), a symbol is ‘a Sign (q.v.) which is constituted a sign merely or mainly by the fact that it is used and understood as such, whether the habit is natural or conventional, and without regard to the motives which originally governed its selection’. Our effort here is, by approaching semiosis from a perspective of a process and a dynamical system, to make the understanding of Peirce’s notion more precise. But before that, we need to introduce some theoretical background on systems theory.

SYSTEMS THEORY According to von Bertalanffy (1972), the notion of system is as old as European philosophy. The term system was not born yesterday, out of current questions of mathematics, science, philosophy and technology. Rather, it is a fruit of a methodological investigation of perennial problems which have been recognized for centuries and discussed in the language available at the time. Two principal ideas were advanced in order to deal with the problem of order and organization, which is subjacent to the idea of a system (von Bertalanffy 1972). One was the comparison with man-made machines. The other was to conceive of order as a product of chance (von Bertalanffy 1972). The theory of the living organism as a machine was constructed by the combination of these two, and was a natural derivation, which apparently could be explained by the evolution of science (von Bertalanffy 1972). Notwithstanding the singular success achieved in the explanation of ever more and finer life processes, basic questions remained unanswered, at least for a while. The evolution of machines by events at random appears to be, at the beginning, self-contradictory (von Bertalanffy 1972). Mitochondrial ‘machines’ of enzymatic organization in even the simplest cell or nucleoprotein molecules are incomparably more complex than a watch or other man-made artefacts.

90

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

The evolutionary concept of ‘survival of the fittest’ seems to lead to a circuitous argument. Self-maintaining systems must first exist before they can enter into competition, which leaves systems with higher selective value predominant. Self-maintenance, however, is not provided by the ordinary laws of physics. Rather, the second law of thermodynamics prescribes that ordered systems in which irreversible processes take place tend towards most probable states and, hence, towards destruction of existing order and ultimate decay. Thus, how to explain biological systems as being instances of the same kind of general principles ruling man-made machines? How to explain the finalistic behaviour exhibited by living organisms? Since the fundamental character of the living things is its organization, the customary investigation of the single parts and processes cannot provide a complete explanation of the vital phenomena. This investigation gives us no information about the coordination of parts and processes. These questions led to the creation of a whole research programme in biology which was later called ‘the organismic programme’ and was the germ of what later became known as general systems theory (von Bertalanffy 1972). The old Aristotelian dictum of the whole being more than its parts, which was originally neglected by the mechanistic conception, and which later led to a vitalistic demonology, was evolved to a more satisfactory explanation: ‘the properties and modes of action of higher levels are not explicable by the summation of the properties and modes of action of their components taken in isolation. If, however, we know the ensemble of the components and relations existing between them, then the higher levels are derivable from the components’ (von Bertalanffy 1972: 411). In order to understand an organized whole, we must know both the parts and relations between them. Later, the notion of a general systems theory was formulated by von Bertalanffy (1972: 411), orally in the 1930s: There exist models, principles and laws that apply to generalized systems or their subclasses irrespective of their particular kind, the nature of the component elements and the relations or forces between them. We postulate a new discipline called General System Theory. General System Theory is a logico-mathematical field whose task is the formulation and derivation of those general principles that are applicable to systems in general. In this way, exact formulations of terms such as wholeness and sum, differentiation, progressive mechanization, centralization, hierarchical order, finality and equifinality, etc, become possible, terms which occur in all sciences dealing with systems and imply their logical homology. General systems theory, then, consists of the scientific exploration of ‘wholes’ and ‘wholeness’ which, not so long ago, were considered to be metaphysical notions transcending the boundaries of science. Novel concepts, methods and mathematical fields have developed to deal with them. At the same time, the interdisciplinary nature of concepts, models and principles applying to systems provides a possible approach towards the unification of science. Now, with a general unified theory, physical, biological, social, psychological and higher-order systems might share the same background and conceptual framework. Even though it was an outstanding progress, things were not that easy. There were different situations in which General Systems Theory might be employed by the different sciences. Its formulation might be acceptable and satisfactory for analytical sciences, like biology (excluded synthetic biology), psychology and social sciences, but it might become a problem for synthetic sciences, like engineering and computer science. The

SYSTEMS THEORY AND SEMIOTICS

91

finalistic behaviour exhibited by living beings and investigated since Aristotle’s first proposition of final cause, as the tendency to a final state, was still a challenge for synthetic computational approaches. As already mentioned, the idea of a final cause, a purposive, intentional or goal-based behaviour, was criticized for a long time as being a non-scientific kind of mysticism introduced by Aristotle’s ideas and further explored by the catholic church (Mayr 1992). General systems theory apparently solved the mystery by proposing that goal-based behaviour might be just a matter of coordinating both random events (chance) and mechanical deterministic processes in specific ways. But how? How to create a mechanistic explanation aligned with general systems theory for purposive or intentional actions? A system is usually defined as a set of elements that maintain relations with one another. ‘Elements’ are conceived as primitive entities which are found at each instant in one among several possible states. Elements establish ‘relations’ when the state of an element depends on the state of another one. Bunge (1979) defines a system as possessing composition (the set of its components), structure (the set of relations between its components) and environment (the set of other systems with which a system establishes relations). According to Bresciani and D’Ottaviano (2018), and aligned with General Systems Theory (von Bertalanffy 1950, 1972), a system is a unity, constituted by a non-empty set of active elements, which maintain to each other, in an organized nature, a given set of relations with time-invariant characteristics, giving them, as a whole, an identity. In other words, a system is a whole, composed by a set of interacting parts. Each part (or element) might have a set of properties which might be fixed or vary through time. These properties might be observable (which means they are measurable) or non-observable (which means that despite having some value, this value cannot be measured by a physical device). Properties might assume different values through time. This set of elements forms a structure with a given functionality, which gains an identity due to providing this functionality. Systems can be natural, as in the case of living systems, or artificial, as in the case of technological artefacts built by men. The organization1 of a system pertains to the structure connecting the many different system parts, meaning that each part is in some kind of relation to one or more other parts of the system. This connection might be physical, or simply logic, implying that there is some relation between two specific parts within the system. In a man-made technical system, the idea of organization might not be really related to true organs, as in a living body, but have to do with the existence of subsystems of the overall system, carrying on some sort of functionality, and having on themselves some sort of identity as a composite of parts. These definitions are definitely quite abstract, but can be applied to, since collections of physical objects (particles, atoms, molecules, physical bodies, etc), up to living beings, technical artefacts or social entities like business organizations/groups of people, and either human minds or consciousnesses. But it is important to point out that the term organization holds a double meaning. It either might designate a fixed structure defining the system (an atemporal structure or, in a temporal system, its particular structure in a given instance of time), which we will be calling here an s-organization (meaning an organization as a structure) or can be related to a dynamical process, where this structure is modified along time, which we will be calling here a p-organization (meaning an organization as a process). In other words, the term organization might be related either to the structure configuring a system in a given instant of time (as e.g. when we talk about the current organization of a system, an s-organization), or to the process where this structure might be changing over time (as e.g. when we talk about the process of

92

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

organization of a business company department along history, a p-organization). This double meaning can be quite misleading, especially if we are talking about a specific phenomenon: self-organization. Before touching the issue of self-organization we need first to address a more elementary phenomenon: dynamical systems.

DYNAMICAL SYSTEMS A system might be static or dynamical, depending on its relation to time. A static system does not change through time. A dynamical system, on the contrary, has its structure or their component properties changing through time. Dynamical systems theory conceives systems as sets of interdependent variables. By ‘variable’ we mean some entity that can change, i.e. that can be in different states at different times – similarly to the concept of element. The state of a system is simply the state or value of all its variables at a given time t. The behaviour of a system, in turn, consists of transitions between states (Van Gelder and Port 1995: 5). General Systems Theory describes (represents) a dynamical system in terms of two distinct features: 1. Structure 2. Behaviour Static systems don’t have in fact a behaviour, which is a characteristic only of dynamical systems. A behaviour describes how the system structure or its properties change across time. Both structure and behaviour can be described (and defined) in different ways. One popular and useful way is through mathematics. This means that systems components are referred to as variables, and their relations are described as formulas and equations. This form of description is very popular in engineering and physical sciences. In computer sciences, other means for describing systems might be used, such as data structures and computer programs. Other sciences might prefer other ways of describing systems, as e.g. a collection of statements (observations, facts) and rules connecting them. These are more popular in sciences like economy, social sciences and psychology. Additionally, other means might be used, such as diagrams and figures. In some situations, a mixture of mathematical/ linguistic/diagrammatic descriptions might be used. One important concept within general systems theory is the notion of state. The state of dynamic system is the smallest set of variables (called state variables) such that knowledge of these variables at t = t0, together with knowledge of the input for t ≥ t0, completely determines the behaviour of the system for any time t ≥ t0. Dynamical systems allow the description of a system by means of a process. This system-as-a-process aspect is crucial for our understanding of semiosis (action of sign), from a system perspective.

­CLOSED VERSUS OPEN SYSTEMS A system can be either a closed or an open system (Veeke et al. 2008). A closed system is a system that does not interact with neither another system nor the environment. It means that all possible behaviour is determined by only the system itself. An open system, on the contrary, has an interface, where it might receive inputs from either the environment or other systems, and also determine outputs, which might affect other systems or the environment.

SYSTEMS THEORY AND SEMIOTICS

93

DETERMINISTIC VERSUS NON-DETERMINISTIC SYSTEMS A system might be deterministic, if its behaviour through time can be precisely defined by its state, and a given set of rules governing its state transitions. It might be non-deterministic (Bartlett 1978) if there is any sort of randomness or chance affecting its behaviour, and as such preventing a precise determination of its current state. Nevertheless, nondeterministic systems might be described and represented under some sort of imprecision, as e.g. through states defined under intervals and/or fuzzy variables. Also, many nondeterministic systems can be represented by means of statistical laws, governing its behaviour, which is described by means of it following some sort of probabilities.

COMPLEX SYSTEMS Systems theory can be used in many different scopes involving sciences and humanities. A system can be since a physical system, which can be investigated and studied using standard methods from physics, passing through biological organisms in biology, up to human societies or social groups, which might be investigated and studied by methods from psychology, anthropology and social sciences. Amongst this multitude of phenomena which might be addressed by the name of system, an important concept arises – complexity. The concept of what is complexity is though, at a minimum, quite complicated. For many researchers, a system can be said to exhibit complexity if its behaviour cannot be easily inferred from its component properties. From one side, part of the scientific community believes that complexity is a matter of compositionality and hierarchy (Salthe 1985). In this sense, a complex system might be simply a system which can be decomposed into subsystems. This belief comes mainly from fields of research like physics, chemistry or engineering, where systems of an arguable complexity (or complicatedness) can be derived by means of compositionality and hierarchy. But another part of the scientific community (mainly coming from biology) alerts us that just compositionality and hierarchy might not be enough (Mikulecky 2001). Particularly, what has been the source of uncountable disputes in the history of science is what since Aristotle is known as final cause, or teleological behaviour, the tendency of a system to a final state. Despite its common use in biology to refer to many kinds of an organism’s behaviour, this notion was criticized for a long time as being a non-scientific mysticism (Mayr 1992). In biology, a completely new word, teleonomy, was created to designate an apparent purposefulness or goal-directedness of structures and functions in living organisms. It was just after Cybernetics (Rosenblueth et  al. 1943; Christensen 1996) that the real nature of teleological/teleonomic behaviour became clear in science. Nowadays, we know that this tendency to a final state is just a collateral effect caused by feedback. Feedback provides an ingredient which is essential to complexity: closure. Connecting some of the system’s outputs to some of its inputs, these feedbacks create closures, which provide the missing piece, together with compositionality and hierarchy, in order to create complexity. In this phenomenon, some system property acts as a reference set point, becoming a parameter which, entering the feedback loop, drives an update into another system property value, forcing successive updates in these values until a final value will lead this property to assume the same value prescribed by the original reference property. Nowadays, this is a common knowledge widely used by control engineers to

94

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

set up control systems. To summarize, a property which a true complex system has, and which a simple system (despite its complicatedness, in the sense of composition and hierarchy) does not exhibit, is the property to anticipate its future states. Among others, one who studied the phenomena of such anticipatory systems was Robert Rosen (1985).

SELF-ORGANIZED SYSTEMS According to Debrun (2018a, 2018b), a system can be hetero-organized, if the system structure is imposed by an external source, or self-organized, if it produces itself, i.e. if it evolves by itself, as an offspring of an interaction of a set of parts, which were independent before they gain together its entity status (its identity as a system) due to providing a functionality. Debrun also differentiates between a primary selforganization, if the system gains its identity during the initial interaction of its many parts, before they can be seen as a system, and secondary self-organization, if the system, after its identity is already determined, is able to preserve its structure, if this structure is disturbed in any form. Using our terminology, self-organization is achieved if a system might reach a stable s-organization due to its p-organization, or similarly, if it reaches its s-organization due to its p-organization. If this stable s-organization is achieved during the p-organization process, primary self-organization is achieved. If the system is originally hetero-organized, but is able to maintain its stable s-organization due to its p-organization, if its structure is disturbed in some way, we might say that the system reaches just secondary selforganization. Usually, machines and other man-made systems are hetero-organized, as their structure don’t change along time. This is the most common case for man-made systems. Conversely, natural systems like living beings can be seen as (primary) self-organized systems. It is important to point out, though, that a system not necessarily must have material parts. They might be just logical.

EVOLUTIONARY SYSTEMS Another kind of system which is worth mentioning here is the class of evolutionary systems (Csányi 1989; Fogel 2006). Evolutionary systems are a kind of intelligent systems where many aspects of biological evolution are simulated in a computational environment. Despite its many variations, evolutionary systems usually comprise a population of elements, which are processed in an iterative way, using combinatorial operators like crossover, mutation and others, generating new populations along time. Each element in the population represents the solution of a mathematical problem, in the form of a structured collection of parameters, which are important for characterizing the mathematical problem. This element is usually called a chromosome, in a straight analogy to biological evolution, which can be evaluated by a fitness function, providing a measuring for the quality of the solution brought by a particular element. With this, in a given population many different possible solutions can be evaluated, and usually the best solution is considered as an output from the evolutionary system. The initial population is usually randomly generated, and as soon as new populations are generated through this evolutionary process, solutions of a better quality are derived (or ‘evolved’), making the evolutionary system a kind of optimization process. Each evolutionary step implies

SYSTEMS THEORY AND SEMIOTICS

95

into a first step, where the population first grows, using the many combinatorial process (like crossover and mutation), and later contracts, when a selection process maintains in the final population, only the elements considered more apt, i.e. those with a greater fitness value. With this expansion/contraction movement, the size of a population can be maintained constant along the generation of successive populations. Evolutionary systems, as in the case of neural networks, can be classified as self-organized systems. We might identify either primary or secondary aspect of self-organization in their functioning. As populations are initially randomly generated (using, of course, pseudorandom number generators in their implementation in computers), they might require many steps of evolution before reaching their final configuration, where a good solution for a mathematical problem is generated. This characterizes its primary self-organization properties. But then, after good solutions are available in the population, the evolutionary system preserves those of a better quality in the next populations, meeting the criteria for being classified also as a secondary self-organized process.

ANTICIPATORY SYSTEMS An anticipatory system is a system whose current state is determined by a future state, or, according to Robert Rosen (1985), ‘A system containing a predictive model of itself and/or its environment, which allows it to change state at an instant in accordance with the model’s predictions pertaining to a latter instant’. These predictions can be goals, plans or simply estimations of future states. According to Mihai Nadin (1999; 2003), ‘Anticipation is a recursive process described through the functioning of a mechanism whose past, present, and future states allow it to evolve from an initial to a final state that is implicitly embedded in the mechanism’. Anticipatory systems are somewhat different from the usual kind of systems we generally find in engineering and systems sciences and have many interesting properties that make them more than pure mechanical deterministic systems. Rosen (1991) argues that their behaviour is what makes living systems different from non-living systems. Living systems would be anticipatory systems. Anticipatory systems may also provide the kind of finalistic behaviour that is particularly related to the property we use to call ‘intelligence’ in human beings. This finalistic behaviour situates anticipatory systems as a natural candidate to instantiate the Peircean notion of ‘thirdness’. Peirce argues that all that can be known must fit into three different categories: Firstness, Secondness and Thirdness. This is in the kernel of Peircean philosophy. But how to understand Peirce’s claims if we assume very simple systems as our reality? A simple system like 1. y(t+1) = Random() will be a system of pure Firstness. Supposing that t+1 is equivalent to the present time, we have a system where the present is completely random. A system like 2. y(t+1) = f (y(t)) will be a purely deterministic, reactive or mechanical system. This is a system where there is only Secondness. It is a system where the present is completely determined by the past.

96

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A system like 3. y(t+1) = f (Random() + y(t)) will be a system where there is Firstness and Secondness. A candidate for a system with Thirdness will be something like: 4. y(t+1) = f (y(t+τ)) or, in other words, a system where the present depends on the future. But, if time passes from the past to present, this seems to be impossible. How can this be possible? To understand that, we need to make a change in the equation: 5. y(t+1) = f (E(y(t+τ),t)) In this case, E(.) is the estimation in time t of a future state y(t+τ). This system is perfectly feasible. In fact, this estimation is possible, because an anticipatory system carries within itself a model of its environment and uses this model to anticipate whatever is going to happen, before it really happens. This predictive capability enables an anticipatory system to generate future estimations, simulating possible futures and using these estimations to base its current behaviour. These estimations are as good as the system’s model is tuned to its environment, but are not perfect. It is important to notice that, despite the quality of a model, there is always a level of uncorrelation between a model and what it is supposed to model. Nevertheless, anticipatory systems have an advantage over purely reactive systems, because they are able to include possible future states in their decisionmaking, and with that analyse multiple possible courses of action before deciding for a specific one, which will force the future to be as previously specified. This ability to forge the future onto specific goals allows anticipatory systems to exhibit a final-like behaviour, which is indistinguishable from a finalistic behaviour. This is generally called a goal-based behaviour. Several implications of anticipatory systems approach are explored in the context of AL and AI (artificial life and artificial intelligence) experimental protocols.

ARTIFICIAL SEMIOTIC SYSTEMS Computational-based methodologies have been used to design virtual/embodied experimental protocols, where it is possible to simulate the predictions derived from theoretical models (Parisi 2001), in particular those describing semiotic processes in artificial semiotic systems. Computer simulations can be used to study different levels of the organization of semiotic processes (Cangelosi and Turner 2002; Parisi and Cangelosi 2002; Perfors 2002). These levels include the simulation of syntactic structures (Batali 1994, 1998; Kirby 1999), morpho-syntactic compositionality (Christiansen and Ellefson 2002), lexicalization phenomena (Steels 1999; Cangelosi and Parisi 1998; Steels et  al. 2002), symbolic competence (Cangelosi 2001), communication (Steels 1997; Steels and Kaplan 1999) and meaning creation in communication. Computational approaches have been used to model and simulate semiosis (meaning processes) from many different perspectives, including Cognitive and Evolutionary

SYSTEMS THEORY AND SEMIOTICS

97

Robotics, Artificial Life, and Synthetic Ethology. Among the advantages of the ‘buildingup synthetic’ approaches is that one can create and modify the parameters defining the patterns of events, the cognitive architectures of creatures, the environment and the laws controlling the behaviour of creatures; one can alter each parameter separately, couple selected parameters, temporally combine their variations in cascade, and observe the consequences resulting from one or from several of these proceedings; one can replicate all these proceedings, introduce new parameters and remove old ones; one can see again the interactive history of particular organisms with conspecifics, with their environments, and from this information calculate ‘probabilities of meetings of organisms’ or predict the emergence of certain activities, when some conditions are fixed. What is of particular importance regarding the premises underlying this chapter, when semiotic process is the target of simulation, the approaches have worked on different levels including syntactic properties (Batali 1994; Kirby 1999), morpho-syntactic compositionality (Ellefson and Christiansen 2000), emergence of grammaticality (Batali 1998), lexicalization phenomena (Steels et  al. 2002; Vogt and Coumans 2003) and communication (Steels and Kaplan 1999; MacLennan 2002). In spite of these new developments, and in light of recent, undeniably important technological discoveries and innovations among members of diverse communities of investigators involved in modelling and simulating projects, there is little regard for the nature of the genuine, ongoing process of semiosis making emerging from the senses. This is based on considerable evidence pointing towards the notion that models modelling the meaning process are lacking a sufficiently strong sense of flexibility of the sort found in complex semiotic systems.

CONCLUSION A semiotic system is causally affected by signs in action (semiosis). As we argued, semiotic systems can be regarded as the embodiment of semiotic processes (Peirce 1868: CP 5.314). Surely, this blurs a distinction between agents and processes. Peirce was a process thinker, i.e. a representative of a philosophical tendency of treating processes as being more fundamental than entities as metaphysical and epistemic categories. A process philosophy can address things and entities as stable bunches of processes. A semiotic system can be understood in these terms as a stable (both spatially and temporally) cluster of semiotic processes, or semiosis. An approach based on the theory of dynamical systems provides a new perspective on the main properties that characterize semiosis associated with a theoretical framework not yet explored by Peirce’s scholarship. What are the advantages? Advantages for whom? One of the advantages related to the use of systems theory is that it is based on a very formal structure, which can lead to a necessary terminological disambiguation of Peirce’s semiotics, a mind-tool in understanding such a complex and abstract theory. At the same time, Peirce’s philosophy of sign provides a powerful collection of theoretical constraints to model and simulation of semiosis. Such constraints are crucial, for example, when we design computational simulation protocols for evolutionary semiotic systems. Once computer simulation becomes tantamount to theoretical simulation, involving epistemological metaphors of ontological and semiotics versions (see Parisi 2001), the selection and choice of models can dramatically affect the nature of all work involving simulation. This is to say, the goal of computer simulation is to pattern theoretical

98

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

simulation through modelling – simulation that is created out of, and supports, the theory, insofar as that theory is constructed in an effort to pattern some aspect of the physical world (Hartmann 1996; Humphreys 2004).

NOTE 1 The term organization can be etymologically traced back to the process leading to the formation of organs in a living body, i.e. a collection of parts of a physical body exhibiting collectively some sort of functionality, and gaining together the status of an organ due to performing this functionality (the Greek word organon means ‘tool’).

REFERENCES Bartlett, M. S. (1978), An Introduction to Stochastic Processes: With Special Reference to Methods and Applications, Cambridge: Cambridge University Press. Batali, J. (1994), ‘Innate Biases and Critical Periods: Combining Evolution and Learning in the Acquisition of Syntax’, in R. Brooks and P. Maes (eds), Proceedings of the Fourth Artificial Life Workshop, Cambridge, 160–71, MA: MIT Press. Batali, J. (1998), ‘24 Computational Simulations of the Emergence of Grammar’, Approaches to the Evolution of Language: Social and Cognitive Bases, 405–27, Cambridge: Cambridge University Press. Bickhard, M. (2007), ‘Modern Approaches to Language’, New Ideas in Psychology, 25: 67–9. Bickhard, M. (2011), ‘Some Consequences (and Enablings) of Process Metaphysics’, Axiomathes, 21 (1): 3–32. Bresciani, F. E. and I. M. L. D’Ottaviano (2018), ‘Basic Concepts of Systemics’, in A. Pereira Jr., R. Gudwin and W. A Pickering (eds), Systems, Self-Organisation and Information: An Interdisciplinary Perspective, 47–63, London: Routledge. Briscoe, T. (1998), ‘Language as a Complex Adaptive System: Coevolution of Language and of the Language Acquisition Device’, in S. Verberne, H. Van Halteren and P. A. J. M. Coppen (eds), Proceedings of the 8th Meeting of Computational Linguistics in the Netherlands Conference, 3–40, Nijmegan: Rodopi Amsterdam. Bunge, M. (1979), Treatise on Basic Philosophy: Ontology II: A World of Systems, vol. 4, Berlin: Springer Science & Business Media. Cangelosi, A. (2001), ‘Evolution of Communication and Language Using Signals, Symbols, and Words’, IEEE Transactions on Evolutionary Computation, 5 (2): 93–101. Cangelosi, A. and D. Parisi (1998), ‘The Emergence of a Language in an Evolving Population of Neural Networks’, Connection Science, 10 (2): 83–97. Cangelosi, A. and D. Parisi (2012), Simulating the Evolution of Language, London: Springer. Cangelosi, A. and H. Turner (2002), ‘L’emergere del Linguaggio’, in A. M. Borghi and T. Iachini (eds), Scienze della Mente, 227–44, Bologna: Il Mulino. Christensen, W. (1996), ‘A Complex Systems Theory of Teleology’, Biology and Philosophy, 11 (3): 301–20. ­Christiansen, M. H. and M. R. Ellefson (2002), ‘Linguistic Adaptation without Linguistic Constraints: The Role of Sequential Learning in Language Evolution’, in A. Wray (ed.), The Transition to Language, 335–58, Oxford: Oxford University Press. Christiansen, M. H. and S. Kirby (2003), ‘Language Evolution: Consensus and Controversies’, Trends in Cognitive Sciences, 7 (7): 300–7.

SYSTEMS THEORY AND SEMIOTICS

99

Csányi, V. (1989), Evolutionary Systems and Society: A General Theory of Life, Mind, and Culture, Durham: Duke University Press. Debrun, M. (2018a), ‘The Idea of Self-Organization’, in A. Pereira Junior, R. Ribeiro Gudwin and W. A Pickering (eds), Systems, Self-Organisation and Information: An Interdisciplinary Perspective, 7–22, London: Routledge. Debrun, M. (2018b), ‘The Idea of Self-Organization’, in A. Pereira Junior, R. Ribeiro Gudwin and W. A Pickering (eds), Systems, Self-Organisation and Information: An Interdisciplinary Perspective, 23–46, London: Routledge. Fogel, D. B. (2006), Evolutionary Computation: Toward a New Philosophy of Machine Intelligence, vol. 1, Hoboken: John Wiley & Sons. Fetzer, J. H. (1988), ‘Signs and Minds: An Introduction to the Theory of Semiotic Systems’, in J. Fetzer (ed.), Aspects of Artificial Intelligence, 133–61, Dordrecht: Kluwer Academic Publishers. Fetzer, J. H. (1997), ‘Thinking and Computing: Computers as Special Kinds of Signs’, Minds and Machines, 7: 345–64. Gomes, A., C. El-Hani, Gudwin R. and J. Queiroz (2007), ‘Towards the Emergence of Meaning Processes in Computers from Peircean Semiotics’, Mind & Society-Cognitive Studies in Economics and Social Sciences, 6: 173–87. Hartmann, S. (1996), ‘The World as a Process: Simulations in the Natural and Social Sciences’, in K. G. Troitzsch, R. Hegselmann and U. Mueller (eds), Modelling and Simulation in the Social Sciences from the Philosophy of Science Point of View, 77–100, Dordrecht: Kluwer. Humphreys, P. (2004), Extending Ourselves: Computational Science, Empiricism, and Scientific Method, Oxford: Oxford University Press. Kirby, S. (1999), ‘Learning, Bottlenecks and Infinity: a Working Model of the Evolution of Syntactic Communication’, in K. Dautenhahn and C. Nehaniv (eds), Proceedings of the AISB’99 Symposium on Imitation in Animals and Artifacts, 55–63, Cambridge: Cambridge University Press. Jung, D. and A. Zelinsky (2000), ‘Grounded Symbolic Communication between Heterogeneous Cooperating Robots’, Autonomous Robots Journal, 8 (3): 269–92. Loula, Â. and J. Queiroz (2011), ‘Modeling the Emergence and Evolutionary History of Semiotic Systems and Processes’, International Journal of Signs and Semiotic Systems, 1: i–iv. MacLennan, B. J. (2002), ‘Synthetic Ethology: A New Tool for Investigating Animal Cognition’, in Magnani and N. J. Nersessian (eds), Model-based Reasoning. Science, Technology, Values, New York: Kluwer Academic and Plenum Publishers. Mayr, E. (1992), ‘The Idea of Teleology’, Journal of the History of Ideas, 53 (1): 117–35. Mikulecky, D. C. (2001), ‘The Emergence of Complexity: Science Coming of Age or Science Growing Old?’ Computers & Chemistry, 25 (4): 341–8. Murphey, M. G. (1993), The Development of Peirce’s Philosophy, Indianapolis: Hackett. Nadin, M. (1999), ‘Anticipation – A Spooky Computation’, Third International Conference on Computing Anticipatory Systems (CASYS 99), Haute Etudes Commerciales, Liege, Belgium, 9–14 August. Nadin, M. (2003), Anticipation – The End Is Where We Start from (English-German-French text), Baden: Lars Müller Publishers. ­Nolfi, S. (2002), ‘Power and Limits of Reactive Agents’, Neurocomputing, 49: 119–45. Parisi, D. (2001), Simulazioni – Larealta`rifatta nel Computer, Bologna: Il Mulino. Parker, K. (1998), The Continuity of Peirce’s Thought, Nashville: Vanderbilt University Press.

100

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Peirce, C. S. ([1857–1914] 1787–1951), The Charles S. Peirce Papers Harvard University, Houghton Library, MS Am 1632. Individual papers are referenced by manuscript number, in R. Robin (ed.), Annotated Catalogue of the Papers of Charles S. Peirce, Amherst: University of Massachusetts Press, 1967, and in Robin, ‘The Peirce Papers: A Supplementary Catalogue’, Transactions of the Charles S. Peirce Society 7, 1971: 37–57. Cited as MS. Peirce, C. S. ([1866–1913] 1931–58) The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Perfors, A. (2002), ‘Simulated Evolution of Language: A Review of the Field’, Journal of Artificial Societies and Social Simulation, 5 (2): 1–46. Queiroz, J. and S. Ribeiro (2002), ‘The Biological Substrate of Icons, Indexes, and Symbols in Animal Communication’, in M. Shapiro (ed.), The Peirce Seminar Papers – The State of the Art, vol. 5, 69–78, Oxford: Berghan Books. Queiroz, J. and C. El-Hani (2006), ‘Semiosis as an Emergent Process’, Transaction of C. S. Peirce Society, 42 (1): 78–116. Queiroz, J. and F. Merrell (2006), ‘Semiosis and Pragmatism: Toward a Dynamic Concept of Meaning’, Sign System Studies, 34 (1): 37–66. Queiroz, J. and F. Merrell (2009), ‘On Peirce’s Pragmatic Notion of Semiosis – A Contribution for the Design of Meaning Machines’, Minds & Machines, 19: 129–43. Queiroz, J. and F. Merrell (2008), ‘Some Remarks on Bickard and Process Metaphysics’, Cybernetics and Human Knowing: A Journal of Second Order Cybernetics, Autopoiesis & Semiotics, 15: 64–70. Ransdell, J. (1977), ‘Some Leading Ideas of Peirce’s Semiotic’, Semiotica, 19 (3–4): 157–78. Rescher, N. (1996), Process Metaphysics: An Introduction to Process Philosophy, New York: State University of New York Press. Rosen, R. (1985), Anticipatory Systems: Philosophical, Mathematical, and Methodological Foundations, (vol. 1 of IFSR International Series on Systems Science and Engineering), Amsterdam: Elsevier Science & Technology Books. Rosen, R. (1991), Life Itself: A Comprehensive Inquiry into the Nature, Origin and Fabrication of Life, New York: Columbia University Press. Rosenblueth, A., N. Wiener and J. Bigelow (1943), ‘Behaviour, Purpose and Teleology’, Philosophy of Science, 10 (1): 18–24. Salthe, S. N. (1985), Evolving Hierarchical Systems: Their Structure and Representation, New York: Columbia University Press. Steels, L. (1997), ‘The Synthetic Modeling of Language Origins’, Evolution of Communication, 1 (1): 1–34. Steels, L. (1999), ‘The Talking Heads Experiment: Vol I. Words and Meanings, Pre-Edition’, VUB Artificial Intelligence Laboratory, LABORATORIUM, Antwerpen. Steels, L. and F. Kaplan (1999), ‘Situated Grounded Word Semantics’, in T. Dean (ed.), IJCAI’99 Proceedings of the 16th International Joint Conference on Artificial Intelligence, 862–7, San Francisco: Morgan Kaufmann Publishers. Steels, L., F. Kaplan, A. Mcintyre and J. Van Looveren (2002), ‘Crucial Factors in the Origins of Word-Meaning’, in A. Wray (ed.), The Transition to Language, 252–71, Oxford: Oxford Press. ­Van Gelder, T. J. and R. Port (1995), ‘It’s about Time: An Overview of the Dynamical Approach to Cognition’, in R. Port and T. van Gelder (eds), Mind as Motion: Explorations in the Dynamics of Cognition, 1–43, Cambridge: MIT Press.

SYSTEMS THEORY AND SEMIOTICS

101

Von Bertalanffy, L. (1950), ‘An Outline of General System Theory’, British Journal for the Philosophy of Science, 1 (2): 134–65. Von Bertalanffy, L. (1972), ‘The History and Status of General Systems Theory’, Academy of Management Journal, 15 (4): 407–26. Veeke, H. P., J. A. Ottjes and G. Lodewijks (2008), The Delft Systems Approach: Analysis and Design of Industrial Systems, Berlin: Springer Science & Business Media. Vogt, P. (2007), ‘Language Evolution and Robotics: Issues on Symbol Grounding and Language Acquisition’, in A. Loula, R. Gudwin and J. Queiroz (eds), Artificial Cognition Systems, 176–209, Hershey: IGI Publishing. Vogt, P. and H. Coumans (2003), ‘Investigating Social Interaction Strategies for Bootstrapping Lexicon Development’, Journal of Artificial Societies and Social Simulation, 6 (1): 1–25. Wagner, K., J. A. Reggia, J. Uriagereka and G. S. Wilkinson (2003), ‘Progress in the Simulation of Emergent Communication and Language’, Adaptive Behaviour, 11 (1): 37–69.

102

­C HAPTER FIVE

Phenomenology and Semiotics PEER F. BUNDGAARD

PREAMBLE If one considers two of the historical beacons of semiotics on either side of the Atlantic – Algirdas J. Greimas and his Paris School of Semiotics in Europe and Husserl’s almost contemporary Charles Sanders Peirce’s work in America – it would be tempting to say that phenomenology has only occasionally, perhaps even incidentally, influenced the development of a theory of signs and a theory of meaning. However, if one broadens the perspective, it is clear that aspects of Husserl’s phenomenology in particular and the phenomenological tradition in general have decisively, albeit indirectly, both shaped the epistemological foundation of what could be called classical continental (Greimasian) semiotics (Greimas 1966, 1970, 1982) and prepared the ground for a non-axiomatic, non-formalist semiotic theory of human making-making – all this while at the same time addressing issues and proposing hypotheses that are cognate with some of those Peirce elaborated in parallel with and independently of Husserl. The crucial import of Husserl’s third and fourth Logical Investigations ([1900–1] 1984)1 on the development of structural linguistics has long since been laid down by Roman Jakobson (1990), and subsequently by Elmar Holenstein (1976, 1992; see also Bundgaard 2003). Its constitution of the notion of ‘structure’ in terms of dependency relations is indeed at the core of this important branch of linguistics developing from the 1920s and on, but it is also found verbatim in Louis Hjelmslev’s Prolegomena (1943) – yet without any mention of Husserl2 – which itself constitutes the epistemological and methodological framework that Greimas intended to transpose from the domain of language proper to the domain of meaning. Similarly, but on the other side of the Atlantic, Husserl’s idea that there exists a universal, formal and ideal scaffolding of language as such, underpinning and constraining any empirical language formation, and thus from which any empirical language (use) can be generated is obviously a predecessor (most likely mediated by Jakobson) of Chomsky’s generative grammar (the generative hypothesis is by the way also at the core of Greimas’s semiotics). Now, if we look at more recent developments of semiotics, Husserl has played a pivotal role, along with Kant’s transcendentalism, in Petitot’s morphodynamic refoundation of and subsequent breakaway from Greimas’s axiomatic semiotics (Petitot 1982, 1992); Petitot was himself, along with Per Aage Brandt (1992), the first to introduce Cognitive Linguistics (Lakoff and Johnson 1980, 1999; Langacker 1987–91;

104

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Talmy 2000) in continental semiotics (in the mid-80s), many of whose key theoretical assumptions have roots directly back to Husserl3: e.g. the embodiment of cognition, the import of intersubjectivity on cognition and human meaning-making, the idea that language expresses a ‘construal’ of a referent scene or a situation, i.e. the situation such as it has been experienced or intended, and that language possesses a design structure capable of specifying such construals.4 Husserlian phenomenology and what we could call Merleau-Ponty’s post-phenomenology are also recurrent references in the work of cognitive semioticians such as Jordan Zlatev (2018) and Göran Sonesson (2015). Finally, Frederik Stjernfelt has in a series of pathbreaking works, notably Diagrammatology (2007), established the profound, and before his book rather neglected, affinities between Husserl and Charles Sanders Peirce. In this chapter, I shall not unfold the history of how (Husserlian) phenomenology has influenced semiotics writ large, since his import can be grasped directly at the source, that is in the works mentioned above (see also Bundgaard 2010). Rather, I would introduce to and elaborate on an aspect of Husserl’s work which has been less discussed by semiotic scholars, at least recently, even though it is of core relevance for this field; I am thinking of his theory of signs: a work he initiated in 1890 in a manuscript entitled ‘On the Logic of Signs (Semiotics)’ (Husserl [1890] 1970), that is to say ten years before the publication of the LI in 1900–1: it is indeed an attempt to determine the concept of sign as well as to minutely classify different sign types by means of carefully drawn distinctions, and it constitutes the foundation of the 1st Logical Investigation, ‘Expression and meaning’, itself devoted to: 1. The definition of the concept of sign 2. The essential distinction of different sign types 3. The determination of the essential properties of genuine (symbolic) signs 4. The constitution of the relation between mind and signs and 5. The relation between sign, meaning and the intended objective states of affairs. As Thomas Byrne (2018) has observed, Husserl returned to his theory of signs and meaning in the manuscripts preceding the planned revisions of the sixth LI in 1913–14 (Husserl [1893–1921] 2005). In the following, I shall address Husserl’s theory of signs and meaning, according to the abovementioned dimensions of it, both as they are developed in the first LI and according to the amendments he makes in the manuscripts from 1913 to 1914. I shall do so, however, with a specific emphasis on the cognitive acts he integrates into the essence of genuine signs in their communicative use. This implies some walking on the knife’s edge, as it were, between the (formal) logic of signs and the (pragmatic) psychology of sign use. Husserl was a methodological anti-psychologist (just like Peirce), and the LI were intended both to lay the ground for such a philosophical antipsychologism and to determine essential properties of the human mind: consciousness, intentionality, meaning (as experienced directly in perception or imagination, or indirectly via linguistic and other sign types) within such an anti-psychological framework. This implies that the object of study – e.g. ‘sign’ or ‘meaning’ – will be defined in its objective essence independently of the vagaries of the individual consciousness, its psychology, its particular history, its concrete situation of use and so on. So, for example, the content of a given sign, the meaning it signifies and is intended to signify, is to Husserl an ideal, objective essence which by no means can be reduced to its situation of use. Likewise, a sign – as we shall see – has formal properties that are independent of any individuals’

PHENOMENOLOGY AND SEMIOTICS

105

use of them. Now, on the other side, Husserl is, of course, perfectly aware of the fact that signs are indeed used in given contexts, involving sign users and specific mental acts engaged in meaning production and meaning processing. This, then, implies that his descriptions also reach outside the realm of the purely objective properties and essences and into the psychology of sign use with a view to characterizing important aspects of this psychology. My approach in this chapter is double. I shall first and foremost try to lay down the components of Husserl’s semiotics, both the essential, logical ones and the pragmatic, psychological ones. But I shall also, while doing this, try to illustrate how Husserl’s theory of signs (its questions and his answers) still informs linguistic and semiotic research in signs, sign-processing and meaning; as well as how recent research in turn can be used to illustrate and in a sense further strengthen some of Husserl’s claims; even if these cognitive-pragmatic extensions of Husserl’s theory come with a rather considerable prize: a much more psychological, semiotic theory than Husserl himself would have endorsed, but perhaps also a descriptively more adequate or powerful theory.

WHAT IS A SIGN? INDICATIONS AND EXPRESSIONS Husserl operates with a somewhat broad and flatfooted general notion of signs, which simply reads that a sign (outside its use in solitary life or private thinking) has a physical appearance given to the senses that is associated with something else, which is not necessarily sensed. However, ‘not every sign has meaning’ (Husserl 1900–1: 183); that is to say, not every sign expresses a meaning. ‘Indications’, which are to be understood as Peircean indices, are such signs that elicit the representation of something else, without expressing any meaning. So, smoke indicates the simultaneous reality of fire, but it is not a sign that expresses any meaning beside that indication; likewise, a footstep indicates the past presence of a human being, and fossil vertebrae are signs of prediluvian animals (Husserl 1900–1: 184). The relation between the indication and what it is a sign for is considered by Husserl to be a motivational unity in that the existence of ‘certain objects or states of affairs of whose reality someone has actual knowledge indicate to him the reality of certain other objects or states of affairs, in the sense that his belief in the reality of them is experienced [. . .] as motivating a belief or surmise in the reality of the other’ (Husserl 1900–1: 184; his italics). In short, the experience of indications is essentially one of a ‘descriptive unity’ of the sign proper (the phenomenon given to the senses) and the object that it is a sign for, and which must exist or have existed since the former exists.

Expressions and meaning-giving acts Now, how about signs used in communication, that is to say here in the LI, linguistic signs, which Husserl calls ‘expressions’? Just as was the case for indications, expressions are also given to the senses, but contrary to indications, they carry meaning, and they carry meaning not in and of themselves, or simply by virtue of associations established in the past, but because the addresser in his use of them bestows an intended meaning upon them. Expressions are signs to the extent that someone produces them with the intention of expressing himself about something to someone. In other words, expressions are sound formations (Wortlaute) that have been animated by certain sense-giving acts, without which they would remain simple sound gestalts.5 We shall return to the sense-giving

106

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

acts in due course, but let us first observe that Husserl integrates the cognitive meaninggiving acts into the essence of the word sign, which has a couple of consequences that are worthwhile stressing. One is that word signs do not simply mean what their definition in the dictionary reads, but what I, within certain limits, make them mean in a given context, relative to a given state of affairs. Another consequence is that if, for example, we take a walk at the beach, and the vagaries of waves, stones and sand produce marks on the ground which perfectly resemble words aligned in a well-formed sentence, these marks would be just that: marks that resemble words, but which are not expressions, since they have not been endowed with meaning by someone who had the intention of expressing himself.

Intimation From this follows another consequence or another essential property of word signs, which logically precedes the meaning-giving acts. This property is that expressions manifest a meaning intention, i.e. an intention to communicate certain mental states or thoughts, Thus, expressions are also indications, namely of the fact that speaker has thoughts, means to say something to someone: they show forth the existence of such mental states or meaning intentions (just like smoke shows forth the existence of fire). This indicating function of the expression is dubbed ‘intimation’ (Kundgabe) by Husserl, and is the function that sets off the whole sign-processing dynamics, that is to say, that which makes physical marks or sound formations appear as genuine signs. It is of course because the otherwise well-formed sentences produced by a parrot do not display any intention to intimate mental states that they do not pass the semiotic threshold. As regards the logic of signs, expressions are (conventionally constituted) sound patterns (1) which intimate the hearer to believe that the speaker has an intention to express something (they are indications of a communicative intention); (2) which have been endowed with meaning by the speakers meaning-conferring acts; it is by virtue of such acts that they mean something, and it is by virtue of what they mean that they relate ‘to what is objective’ (Husserl:1900–1: 192). From this strictly logical perspective there is nothing more to it.

Meaning-fulfilling acts Now, expressions appear in speech or discourse, and discourse and speech are by and large communicative: we use signs to express something about something to someone. Thus, even though, logically speaking, the essence of the sign is exhaustively established by the intimating function and the meaning-giving acts, we still need to characterize signs as they are used in social life, we need to shed light on their communicative use. To this end, Husserl introduces another essential distinction which leads him to the essential notion of the meaning-fulfilling acts. The distinction is the one between full and empty meaning intentions. So, a meaning intention confers meaning to a word which, in accordance with that meaning, relates to something objective, the object, state of affairs or event that is being spoken about. In some cases, the object, which the sign is related to, is given in front of us, directly in perception. You can be speaking about something we both have perceptual access to. This is, for example, the case in all cooking shows, YouTube recipes and live comments on sports events, or when we exchange judgements about the beauty or ugliness of a landscape, a painting or a piece of music (cf. Zwaan 2014). In such cases,

PHENOMENOLOGY AND SEMIOTICS

107

we are talking about what we are witnessing, or what we are actually doing here and now, and Husserl would say that the relation to the object established by the sign (and the meaning-conferring acts) is given in ‘intuitive fulness’ (Husserl 1900–1: 192). The meaning intention is said to be full or fulfilled.6 Without being exotic, the above cases are nevertheless rather rare or not standard cases of communication. Typically, we use word signs, and speech, to talk about what is not there, which is of course one of the major advantages of language: it enables us to represent the absent. In such situations, the meaning intentions are said to be ‘empty’, and the relation to the object is not immediately realized. Meaning is all there is, as Husserl perhaps a bit enigmatically puts it: A name, e.g., names its object whatever the circumstances, in so far as it means that object. But if the object is not intuitively before one, and so not before one as a named or meant object, mere meaning is all there is to it. If the originally empty meaningintention is now fulfilled, the relation to an object is realized, the naming becomes an actual conscious relation between name and object named. (Husserl 1900–1: 192) What happens, then, in a communicative situation where there isn’t any perception to fulfil the meaning and realize the relation to the object? Here the addressee must produce acts that fulfil the meaning of the sign. Language-processing consists in performing meaning-fulfilling acts. These acts, Husserl stresses this time and again, are not essential to the expression proper (only intimation and sense-conferring acts or meaning intentions are), but they are essential to the communicative situation as such, because they are the ones by virtue of which the hearer actualizes the expression’s relation to the object. But how are these acts to be described? One should tread carefully here because, even though I will indeed assign particular importance to the considerable subset of them that could be dubbed quasi-perceptual or quasi-experiential acts, they are by necessity of different sorts. If I say ‘the three perpendiculars of a triangle intersect in one point’ or ‘the square root of sixteen is four’, the meaning fulfilling acts that relate the propositions to their object cannot be characterized as being quasi-perceptual, i.e. acts that establish this relation by means of imagery or intuitive contents. The same may hold true for propositions involving abstract concepts, such as ‘Freedom of speech is one of the pillars of the democratic society’. Yet, I will nevertheless propose that an important subset of meaning-fulfilling acts could be aptly considered as quasi-perceptual acts or acts involving intuitive or experiential contents, which would entail that to process signs is to ‘simulate’ or cognitively re-enact the experience of the situation the expression is about.

Meaning-fulfilling acts as quasi-perceptual simulations In the cognitive psychology of text-processing, Rolf Zwaan has for years developed a theory that considers what he calls ‘simulation’ (Zwaan 2004, 2016) as the pivotal element of language comprehension. I here simply propose that an important subset of meaning-fulfilling acts could be adequately described as ‘simulations’. What is, then, a simulation? It is a representation that is a quasi-experience of the situation (writ large) described or expressed by the linguistic input. Meaning-fulfilling acts thus realize the relation of the sign(s) to the object by simulating the experience of the object (or the way it has been intended) such as it is encoded in the sign(s). Here is how Zwaan succinctly

108

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

puts it: ‘The basic premise is that language is a set of cues to the comprehender to construct an experiential (perception plus action) simulation of the situation’ (2004: 36). In other words, the comprehension of an expression (which in Peircean terms could be the interpretant of the sign) is the ‘reconstitution’ of the experience expressed by the sign (Zwaan 2004: 38). Zwaan, and others (notably Barsalou [1999]), have provided abundant evidence for the quasi-perceptual or quasi-experiential nature of language comprehension, that is of the meaning that is the endpoint of meaning-fulfilling acts and the communicative finality of the meaning-giving acts. A classical experiment consists in having participants read either of the following sentences (Zwaan 2004):

(1a) The ranger saw the eagle in the sky. (1b) The ranger saw the eagle in the nest. They are subsequently shown a picture of an eagle and asked if it was mentioned in the sentence. Interestingly, response time is faster when there is a match between the shape of the eagle in the picture and the one implied by the sentence; response time is thus faster when people after having read (1a) see a picture of an eagle with its wings outstretched. In other words, the phenomenal shape of items referred to by a complex of signs is routinely integrated in the comprehension of those signs. This is part of what I mean when I say that an important subset of meaning-fulfilling acts can aptly be described as quasi-experiential simulations.7 Other kinds of examples, here adapted from Zwaan (2004), show the quasi-perceptual nature of linguistic contents: (2a) The red squirrel approaches the fence. (2b) The red tractor approaches the fence. Here, obviously the understanding of the sentence, which relates word expressions and their meanings to a given objective scene, implies rather different construals of both word meanings and the referred relations between objects. As Zwaan remarks, ‘red’ will not be represented in the same way in both sentences, nor will ‘approach’, since in the squirrel scene ‘approach’ requires much more proximity than in the ‘tractor’ scene; and finally, the implied perspectives in each scene are essentially different (proximal vs. distal). These interpretations, construals or simulated quasi-perceptual experiences are elicited by the meaning-giving acts. Now, consider a somewhat different case since it does not imply a straightforward perceptual simulation of a scene (for the simple reason that the information does not concern the perceptual properties of the scene), which, of course, implies that meaningfulfilling acts and simulations can be more than proxies for genuine perceptual meaningfulfilling acts. Consider someone informing someone else of the state of affairs in example (3): (3) This computer is not child safe. For reasons pertaining to the fact that language essentially underdetermines its referent, both syntactically and semantically, the meaning of the sentence cannot be computed following compositional principles. The sentence can mean one thing and its opposite, namely that it is not safe for children to use the computer, since I haven’t

PHENOMENOLOGY AND SEMIOTICS

109

installed any parental control software on it; or that it may not resist children’s playing with it. The experienced meaning of the sentence, which is of course dependent on contextual information, will be the outcome of the meaning-fulfilling acts. Now, these acts are perhaps not as perceptual as the ones triggered off by examples [1a through 2b], but they can still be said to be experiential, and not simply abstract representations, since they elicit different simulations of the things you can and cannot do with the computer, for example its fragility vis-à-vis children’s playing with it. Here’s then the complete model for the word sign and its use in communication: 1. An expression is shaped so as to intimate to the hearer that speaker has a communicative intention; an expression indicates a meaning intention. 2. Meaning is conferred to the sign in meaning-giving acts. 3. In cases where the meaning intention is not immediately fulfilled by perception, meaning-fulfilling acts realize the sign’s relation to the object, according to its meaning. As we can see, this semiotic setup by far exceeds a simple signifier/signified relation of a Saussurean sort. Moreover, even though the logical basis of the word sign is limited to its intimating function and the meaning-giving acts, it is impossible in practise to avoid the essential interference from the situation of sign use and the crucial role played by the meaning-fulfilling acts elicited by the word sign such as it has been used. This has a consequence which Husserl acutely observes and then attempts to restrict the validity of. The consequence is that the meaning of a sign is not ‘simply’ the meaning bestowed upon it by a sign user, but rather the meaning-fulfilling acts that realize the meaning intention’s relation to the object; that is, those acts that represent (or simulate) the way the sign intends a given object. In other words, when meaning is conferred to signs, then the ultimate intention of this act is to elicit meaning-fulfilling acts that correspond to the original meaning intention. The meaning of a sign is, in a sense, the meaning-fulfilling act and its content. Here’s how Husserl puts it: The word ‘expression’ is normally understood [. . .] as the sense-animated expression. One should not, therefore, properly say (as one often does) that an expression expresses its meaning (its intention). One might more properly adopt the alternative way of speaking according to which the fulfilling act appears as the act expressed by the complete expression: we may, e.g., say that a statement ‘gives expression’ to an act of perceiving or imagining. (Husserl 1900–1: 192) As I have already mentioned, there is a sort of double approach to word signs in the LI. On the one hand, Husserl attempts to lay down the essential properties of such signs, their logical foundation; on the other hand, he intends to describe signs in communicative use, which makes him recruit acts, namely the meaning-fulling acts, that are essential to the communicative situation, but not essential properties of the sign as such. In short, we seem to have both a logic of signs and a pragmatics of sign use. This distinction between elements that are essential to the sign and elements that are essential to its use, but not to what it really is, may seem artificial, and as we shall see Husserl himself may have come to that conclusion at a later point in time. Suffice here to say that he maintains the restricted definition and the difference in the LI because he wants a definition of the sign that

110

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

seamlessly covers both its dialogical (communicative) use and its monological use (what he calls ‘expressions in solitary life’), and here, obviously, the meaning-fulfilling acts are not an essential element of the way the sign intends an object through its meaning. The question is now whether this distinction can be upheld, and whether Husserl himself by preparing the ground for a pragmatic semiotics has not, in a sense, undermined the restricted definition of the sign, not in the sense that it cannot be logically maintained, but in the sense that this restricted definition of the sign is so compressed that it cannot be used to adequately describe the communicative use of signs. Let’s consider the case of the communicative use of signs again, which complexifies things considerably: so we have one speaker who intends to use signs to express something (a meaning) about an object (writ large) to a receiver, we therefore have the speaker’s meaning-giving acts, we have the situation of communication, we have an object related to and a manner of intending the object expressed by the signs, and we have the receivers meaning-fulfilling acts that are representations of the way in which speaker intends the object in case. Now, and this is the crux of the question, if the speaker’s meaning intention is to elicit in the receiver, by means of a combination of signs, a representation of the manner in which he relates to or intends a given object, then, of course, it seems strange, if not infelicitous, to relegate the meaning-fulfilling acts – which, again, realize that relation – to something extraneous to the sign proper, something beyond or somehow secondary to the essence of the sign.

THE REDEFINITION OF SIGNS IN THE MANUSCRIPTS FROM 1913 TO 1914 Husserl himself may have reached that conclusion because in his, posthumously published, revisions to the second edition of the Logical Investigations, he proposed a series of interesting modifications of his original theory of signs, which assigns particular importance to the communicative use of signs proper. In these manuscripts, Husserl proceeds by making a more fine-grained division of signs into inauthentic signs (indications) and authentic signs, which again are divided into authentic categorial signs and authentic signal signs; only the latter two are expressive in the sense that they carry or express an intention to communicate something to someone, but only the authentic categorial signs are linguistic. We shall now see how Husserl, by virtue of this crucial distinction between inauthentic and authentic signs, so to speak puts the communicative function of the sign back into its essence. As we have already seen, indications elicit the representation of, and the belief in, the existence of another object; they do so compellingly by virtue of a past association that motivates the link between the two objects. So, dark clouds elicit the representation of (possible) rain, smoke elicits the representation of (and thus signifies) fire and so on. However, the motivated association between sign (indication) and signified object may wane, as it were; that is to say, the compelling link between the two objects may come in degrees of compulsion; and the essential reason for this can be expressed in terms of Husserl’s semiotics. Indications or indicative signs do not have any intimating function; therefore they do not express any meaning intention or communicative intention; and therefore the degree to which indications (smoke, footsteps) compellingly elicit a representation of their signified object (fire, human beings) varies in function of their relevance in the situation where they are perceived. Footsteps may strongly indicate human presence for Robinson Crusoe for reasons pertaining to his situation; not so or

PHENOMENOLOGY AND SEMIOTICS

111

hardly so for me if I am taking a stroll on the beach; smoke compellingly signifies fire or combustion of some sort if it is in my kitchen; less so or hardly so if I see it above a cityscape, perhaps not at all if I am more interested in knowing the force of the wind or its direction.8 The situation of use is inarguably essential for the comprehension of any sign, including, of course, linguistic signs (Bundgaard 2019) the meaning of which is tributary to the context. However, the relevance of the situation where an indication appears is even more crucial in that it depends on its situational relevance whether or not the indication will even start signifying, that is, whether it will, and to what degree, elicit the representation of its associated object. This is not the case for linguistic signs which compulsively will elicit a representation of their default meaning and its relation to an object, even in the absence of a situation of use: in such – probably purely fictive – situations, the meaning-fulfilling acts will construct a default situation where the word signs will take on their default meaning.

Genuine signs come with the demand that they should be acted upon Here is then the new distinction Husserl proposes in his revisions to the sixth (and also the first) LI: the one between authentic non-categorial signs and authentic categorial signs (the latter being linguistic signs). A non-categorial sign is any non-linguistic sign which has a communicative purpose and thus expresses the intention to communicate something to someone. It could be a flag raised by pirates communicating that no prisoners will be taken, or a flag communicating that there are contaminated people on this ship which should therefore not be approached; or it could be Husserl’s own main example, namely a siren signalling an incoming storm from north-east. We shall shortly see what, according to Husserl, makes such signals different from categorial signs, but before that, we shall consider what makes signal sign authentic signs on a par with linguistic signs. That is of course the fact that they are communicative, express a state of affair and are used for this very purpose (for example by the harbour authorities). Yet, there is more to it since Husserl now introduces a property of genuine signs which he didn’t mention in the first LI and which appears as an essential refinement of his notion of ‘intimation’ (see also Byrne 2018 for this). Any genuine signs, whether storm signals, road signs or linguistic signs, are accompanied with or manifest the addresser’s demand (Zumutung) that the addressee should (sollen) pay attention to what is being communicated, that is to say, he ‘should know this’. So, any genuine sign is bestowed not only with a meaning, but also with what scholars in a continental tradition of semiotics9 have called an ‘enunciative’ demand and a should: ‘Alle echten Zeichen haben den Ursprung aus solchem zumutenden, von Subjekten ausgehenden Sollen’ (Husserl [1893–1921] 2005: 97); ‘All genuine signs originate in this demand, in this Should that emanates from the subject.’ Husserl himself doesn’t develop this ‘Sollen’ in much detail, but we can try here to derive some consequences for a theory of signs that may exceed the limits of what Husserl actually proposes here, but which is nevertheless still in accordance with the main tenets in it. The first consequence is that genuine signs are genuinely communicative, that is to say, if the communicative sign comes with such a demand, the addressee and, along with him, the meaning-fulfilling acts are essential moments of the sign in use, they are the instance towards which the sign tends. This is an important extension, and also an improvement, of the restricted definition from the first LI. The next consequence is that this extension implies a pragmatic definition of the sign that, of course, goes beyond the relation between sign and signified. The sign user not

112

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

only expresses himself in such and such a way, for such and such reason, but also demands of the receiver that he act upon what is being communicated to him; in Husserl’s more restricted understanding of this ‘Sollen’, it could simply mean ‘you should pay attention to this’, ‘it is important that you know this’ and so on. Now, the pragmatic character of this new definition of the sign becomes evident if we consider its dual form, as it were, that is, if we inverse the perspective. If it is the case that what the receiver is being told is something that by definition comes with the demand ‘you should know this’, then the receiver must consequently also expect that what he is being told is relevant in some respect. The pragmatic notion of relevance is therefore built into the key notions of ‘demand’ and ‘Should’ (Zumutung and Sollen). I suggested above that the notion of ‘should’ implies that the sign comes with the demand that the receiver act upon what is being said (and that indexical signs do not come with that demand). It should be mentioned that Husserl may stop short from fully developing the sense ‘should’ and the full, pragmatic, meaning of ‘acting upon’ a communicative sign. Let’s return to Husserl’s own example: a storm siren is signalling an incoming storm from north-east; by hearing it, I know I should pay attention to it, and paying attention to it, I understand what it intends to express. An analogous situation would be that I am in a theatre and a man yells ‘Fire!’, which elicits the representation that the theatre is on fire. Could we in either of these situations exhaustively and adequately describe the semiotic situation, and thus the sign, in terms of it eliciting a representation of an incoming storm from north-east or that the theatre is on fire? Hardly so, since the meaning of the storm siren is crucially not simply that there is a storm coming in, but to make you act accordingly, that is to say: you should stay in the harbour. In Peircean terms, we could say that the final interpretant of the storm signal is the adequate way of acting on it; the conclusion ‘so I should stay in the harbour’ is not an inference that is drawn beyond the realm of the sign, something extraneous to its formal essence, it is indeed the genuine meaning of that sign. This, trivially, explains why it is prohibited to yell ‘Fire!’ in a theatre if there isn’t any fire in it. In short, genuine signs come with an extended demand, which is part of their intended meaning, namely that a corresponding act should be executed (physically, intellectually or perceptually).

The difference between categorial (linguistic signs) and signal signs The above explains why signals, contrary to indicative signs, are genuine signs: they express a meaning relative to a state of affair; they come with an intersubjective demand and they call for an appropriate action. Now, what makes them different from categorial signs, that is to say, linguistic signs? The difference rests on the fact that ‘in the realm of signals, there doesn’t exist any grammar’ (Husserl [1893–1921] 2005: 53). This may at first glance seem a bit astonishing since they do seem to display some sort of minimal compositionality in that we can have articulated signals like [A storm is coming] + [from north-east], and we can easily transpose that meaning into words. Now, according to Husserl, what signals do is that they simply indicate the state of affairs consisting of a storm and where it comes from; contrary to linguistic signs they do not articulate meanings and their categorial forms so as to constitute them as specifically intended states of affairs. They point at a fact [Storm coming] + [from north-east] and leave it to us to combine these elements and shape them into given categorial expressions: as in, e.g., ‘There is a storm; the storm is from north-east’; or ‘There is a north-east storm’; or ‘A north-east storm is expected’, and so on (Husserl [1893–1921] 2005: 53). While the

PHENOMENOLOGY AND SEMIOTICS

113

signal is indifferent to the various ways these significations can be combined, Husserl insists that these sentences are different as regards their significations: i.e. the object is intended through differently composed meaning intentions; or to put it in cognitive linguistic terms: such expressions articulate different construals of the state of affairs. Let’s take a closer look at the differences between signals and signs in order, once again, to develop an understanding of the functioning of linguistic signs that, despite other kinds of essential differences, is close to the one developed within cognitive linguistics. Signals are univocal, clear and distinct. However, if you consider a sentence like ‘The lion is over there’, its meaning is by no means unambiguously anchored in the sign, but rather10 in the addresser’s meaning intentions, it is relative to what he intends to express when he uses the sign in such and such situation. So, trivially, the above sentence could mean that the lion cage in the zoo is over there, that the toy lion is under the sofa over there, that the portrait of the lion is over there, that the person dubbed Lion in my family is sitting over there, and so on and so forth. Another difference, which is already implied in what Husserl said about the indication (the signalling) of a state of affairs and its linguistic expression, is that signals do not have any or imply any viewpoint (i.e. they don’t express any construals). When you become conscious about a signal, the latter leads you into consciousness about the object it points at. So, you hear SOS, and you then, subsequently, pass to the consciousness of an object, namely a ship in distress. There is nothing between the sign and the object it univocally makes you conscious about. This is not the case for linguistic signs which express the way in which an objective state of affairs has been intended and therefore bring the referent object to a certain manner of categorial manifestation, what Husserl calls the  categorial ‘Erscheinungsweise’ of the referent scene (Husserl [1893–1921] 2005: 125). Simple examples are sentences such as: (1) A is parallel to B; (2) B is parallel to A; (3) A and B are parallel. The object referred to by the sentences is of course the same, but the way it is categorially shaped, in terms of perspective and fore- and backgrounding, is not. Husserl puts it like this: Linguistic signs contrary to signals are ‘articulated expressions of a cognizing consciousness [Erkentnissbewusstsein]’, i.e. they express the way in which such a cognizing consciousness has comprehended the state of affairs it is or was intending. In this sense, and even though Husserl’s technical language is far from psychological, his conception of what linguistic signs express, or what is expressed through language, seems to be a forerunner of what has been marshalled in Cognitive Linguistics in the past forty years: language expresses construals of referent scenes and through the expression of such construals the speaker intends to elicit representations in the addressee that match such construals, that is to say, as we saw it above: speaker wants to elicit meaning-fulfilling simulations.

Linguistic signs do not designate objects, they construct meaning formations Here’s then a final difference between signals and linguistic signs, but also a difference with respect to Husserl’s definition of sign-processing in the first LI (Thomas Byrne [2018] has insisted on this new definition of the sign). In the case of the signal, we saw that first there is a phenomenal consciousness of the signal as, say, an acoustic phenomenon Z, and then we have a passage to another type of consciousness, namely of the acoustic phenomenon as a sign, and finally we have a consciousness of the object signified by Z: GZ. Similarly for linguistic signs in the Logical Investigations, you would first have a

114

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

phenomenal consciousness of, say, certain marks on paper, and then you would have a passage into another kind of consciousness when these marks are apprehended as signs signifying something, and finally a consciousness of what is meant by the signs. Now, in the manuscripts from 1913 to 1914, Husserl proposes a strong amendment of this structure in that for him there is no such passage from a phenomenal consciousness to another type of consciousness (of the signified object). Marks taken as linguistic signs are intimately united with what they name and mean; in this respect, Husserl claims, they behave like pictures ([1893–1921] 2005: 126). In a picture you can, indeed, analytically distinguish between the depicting moments and the depicted moments, but only analytically: there isn’t any first intake of the picture as an aggregate of physical marks on a canvas, and then, subsequently, another kind of intake, i.e. another kind of consciousness, namely of the depicted objects. Both these sets of moments coincide or, as Husserl has it, cover (decken sich, Deckung) each other or overlap. The same is then supposed to be the case for the representing moments of the sign and its represented moments. There is not first one type of apprehension of the former, which then leads to another type of apprehension, namely of the latter. How is this coextension or coverage between representing and represented moments in the sign to be understood? The point is the following: once linguistic signs are taken as signs, that is to say, as expressing a meaning intention and a given way of attending to a state of affairs, we are immediately in a realm of significations in which an object is being constituted: the object as it is meant; that is, an object that is not identical with the object referred to, but that object such as it is intended. In this specific sense, linguistic signs do not ‘designate’ (bezeichnen] anything; they do not have any direct referential (hinweisende) function proper (Husserl [1893–1921] 2005: 128), as signals have, because they don’t mean the object as such, but the way in which it comes to appearance for a cognizing consciousness, as we saw before. The distinction between meaning and object of reference of course still obtains, but what the signs do is not simply to point to some state of affairs, but to shape it, gestalt it, as it were, to a meant meaning; and what the receiver does is similarly to continuously process the signs’ manner of articulating a meaning formation, relative to speaker’s intention, and relative to the situation or objects referred to, and crucially this manner of processing does not require a specific kind of consciousness – one uniquely sign-oriented – which then leads us into an entirely different kind of consciousness, one which is object-oriented. We acknowledge the meant object as it is being articulately expressed by the signs.11 It is worthwhile remarking that essential elements of the analogy with the picture and the corresponding picture consciousness still obtain here. As already remarked, in general, there is no such thing as a specific consciousness of physical marks on a surface that constitutes the gateway into another, different consciousness of the objects depicted in the picture. Now, just as was the case for the linguistic, this does not imply that the relation to a reference object (whether fictional or not) has been cut off. The fact is simply that I don’t directly experience the very object represented in the picture, or the picture is not a gateway to that object: I see that it is Napoleon and the Alps, but what I experience is him triumphantly crossing the Alps such as Jacques-Louis David meant him to do it.12 It is in this sense that, in Husserl’s later developments of his theory, signs can be said to signify rather than designate (bezeichnen), and also in this sense that sign consciousness and meaning consciousness constitute from the outset a unitary semiotic consciousness in which sign processing is coextensive with the grasping of an articulated, meant state of affairs.

PHENOMENOLOGY AND SEMIOTICS

115

Having reached the, at least provisory, endpoint of this presentation of Husserl’s theory of signs, let me sum up some of its main points, along with the specific pragmatic and cognitive semiotics extensions I have proposed of it.

­CONCLUSION: THE COMPONENTS OF A PHENOMENOLOGICAL SEMIOTICS In the guise of a conclusion, let me further discuss the four main tenets of what Holenstein (1992) already called Husserl’s phenomenological semiotics: 4. The meaning-giving acts 5. The intimating and demanding sign function 6. The meaning-fulfilling acts 7. Sign use as meaning construction.

Meaning-giving acts The integration of meaning-conferring cognitive acts into the very definition of the sign is a core element of Husserl’s semiotics and theory of meaning. It paves the way for a dynamic understanding of human sign use in that meaning is not seen as simply carried by or attached to a sign vehicle; rather signs are defined as expressing intentional relation to a given object (broadly taken), and the meaning they express is conferred to them by virtue of this intentional relation to the object. In this respect, Husserl is clearly a – relatively ignored – forerunner for one of the fundamental theses in modern Cognitive Linguistics: namely that language does not simply signify objects as such, but expresses construals of objects, specifically structured experiences of objects. Construals of referent scenes or states of affairs come with a perspective, which is a speaker’s way of attending to the given object; and ‘perspective’ here simply means the way in which one intends the relation between the elements in the referent scene (writ large) to be; or how speaker has distributed his attention over the referent scene, foregrounding this, backgrounding that. Now, perspective is not necessarily limited to experiential or intuitive cases, but could be claimed to play a shaping role also in formal uses of expressions, e.g. in the articulation of ideal objects so that the difference between [2+2=4] and [√16=4] pertains to the manner in which these objects have been categorially intended, that is, it pertains to their categorial manner of manifestation. It is also worthwhile noting that the essential role assigned by Husserl to the meaning-giving acts also makes him capable of accommodating a key property of language: namely that language by essence underdetermines its referent, but still is an efficient tool for communication in that signs in use are sets of prompts for the addressee to represent the meant situation and thus the meaning-giving acts that articulate speaker’s intentional relation to that situation.

The intimating function; the demand and the Should We have seen that another essential property of communicative signs is their intimating function – the fact that for something to be a sign it must announce an intention to express something about something to someone. Husserl’s own extension of this function in his 1913–14 manuscripts stipulates that the sign does not simply announce a meaning intention, but doing so also comes with the demand that the addressee attend to it, that

116

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

he should know this or pay attention to it. In that respect the addresser’s demand is to be understood as a claim of relevance, which again is correlated with the addressee’s expectation of relevance. Here again, we have a determination of a logical property of signs, the intimating, demanding function and the ensuing claim of relevance, which forces the semiotician to address the nature of signs outside signs themselves as it were: namely, relative to the intersubjective dynamics in which they are used. As explained above, Husserl may nevertheless operate with a too restricted scope of demand, or a too restricted semiosis, which is then in need of further extension. The point is here that signs used in communication do not simply come with the demand that the addressee should listen to them because the information they convey is relevant; they also, and crucially, come with the demand that the addressee act adequately on them. This implies that in the absence of such adequate action, the sign signifies or expresses in the void, a bit like a road signal does when it is not in use. This may seem straightforward for cases like ‘Fire!’ or storm signals and imperatives, but what does ‘act upon’ mean for everyday communicative exchanges, where physical action is not required? In such cases Husserl’s minimal definition of the ‘should’ may suffice: it may be enough to take into account what is being communicated; yet, ‘to take into account’ probably implies a reaction of some sort, which again is to be considered the final interpretant of the sign as it is used by the addresser. If someone tells me ‘It’s raining outside’, and I answer ‘Hmm’ without lifting my eyes from the crossword section of my paper, the other person may consider his attempt to communicate as failed, and me as a rude person. Similarly, if I am told that Trump won Florida, but lost Pennsylvania, the finality of that piece of communication is not simply that information, but rather to elicit some sort of emotional response. In short, signs used in communication call for (re)action; they are by essence perlocutionary, as Austin (1962) had it.

The meaning-fulfilling acts Expressions in their communicative use are prompts for the addressee to simulate the referent scene as intended by the addresser. So, the sign-processing acts that expressions elicit are simulations of the meaning-intending acts and their relation to the object. Such simulations are in some situations necessarily quasi-perceptual, but come of course in degrees of abstraction according to the nature of the meaning-giving acts. In short, ‘the mouse approaches the fence’ necessarily triggers of a quasi-perceptual simulation (which need not be detailed), whereas ‘Freedom of speech is one of the pillars of a free democratic society’, or ‘√16=4’ does not require such acts. It is, however, still much debated whether even very abstract propositions or concepts (such as ‘equity’ or ‘justice’ or ‘moral’) require recruitment of intuitive contents, i.e. quasi-experiential situations. It may very well be difficult, perhaps impossible, to explain the concept of ‘justice’ purely in abstracto, i.e. without recurring to situations likely to illustrate (aspects of) the concept (e.g. ‘See, this could be called a just action’, or ‘This is something unjust’). In that sense understanding a concept amounts to simulate situations where the concept has been put into play. A last word on the cognitive-pragmatic extension of Husserl’s notion of meaninggiving and meaning-fulfilling acts. Cognitive Linguistics provides abundant well-founded illustrations of what it means for an act to bestow meaning on signs in accordance with the way an object has been intended, and of the design features of language that specify this meaning. Similarly, the cognitive psychology of language-processing has elaborated

PHENOMENOLOGY AND SEMIOTICS

117

concise theories about and given enlightening examples and evidence of the nature of meaning-fulfilling acts as quasi-perceptual simulations: i.e. simulations of the referent scene such as the addresser has intended it. The ideal meaning of signs – their finality – in a pragmatic, communicative situation of use is to elicit simulations that match sufficiently well their constitutive meaning intentions. It should be noticed that the fact specific meaning-fulfilling acts can be mistaken implies that the meaning bestowed upon the sign cannot be reduced to the way in which the meaning-fulfilling acts realize the relation of the sign to the object. It is still the case, however, that the finality of the sign in use is to have these meanings cover each other adequately.13

Sign use as construction of meaning formations, and sign processing as the apprehension of meaning formations The revisions from 1913 to 1914 unfold a seemingly radical claim to the effect that word signs do not designate (bezeichnen) an object, they do not simply refer to it (but are of course related to it). They are rather said to unfold, by virtue of their internal articulation, a certain predicative apprehension of a state of affairs; they are, as we saw, the articulated expressions of a cognizing consciousness (‘Erkenntissbewusstsein’). This means that instead of referring directly to such and such objects or situations, they construct a meaning formation which is the object such as it has been acknowledged, apprehended or experienced. Correlatively, sign-processing is not a two-step endeavour where signs qua signs with given contents are first computed, after which their relation to an object is represented; sign-processing consists in apperceiving, as it were, the meaning formation as it is being shaped, the relation to the object as it is being articulated within a dynamic, intersubjective communicative system. Here’s how Husserl puts it: [The communicative function] is activated by the fact that the addresser and the addressee, the sign giver and the sign receiver constitute themselves in a intersubjective consciousness, and by the fact that they face each other in mutually understanding acts of consciousness, where one transmits the sign with its signification, and where the other understands it exactly with this signification. (Husserl [1893–1921] 2005: 54) The reason why we can talk about a phenomenological semiotics in this context is not simply because its author is a phenomenologist called Husserl, but because it is a theory of sign which from the outset defines the communicative sign system as something essentially different from a self-contained language system. Instead, a phenomenological semiotics founds the communicative sign system on the interaction between experiencing, perceiving and acting intentional subjects and their objects intended and meant in such and such a way, as well as on the intersubjective interaction of communicating consciousnesses in given situations, relative to given objects. These elements – cognitive acts, intentional relations to construed objects, signs and interactions between intersubjectively related and situated consciousnesses – constitute the semiotic ecology of communicative sign use. The semiotic legacy of Husserl’s phenomenology in general, his theory of signs in particular, as well as of the work of disciples such as Roman Ingarden (1973) and Merleau-Ponty (1945, 1964) extend, of course, far beyond the aspects of his theory introduced and discussed here. Husserl has for example devoted a considerable amount of work on and proposed acute descriptions of the nature of images or pictorial signs,

118

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

both plain and aesthetic, as well as of the type of consciousness they elicit – a work, which, unfortunately, does not seem to have had the same kind of (indirect) impact on the semiotic community as his analyses of the linguistic sign and the language system: a remarkable exception is again Stjernfelt (2007). Similarly, and as already suggested, Husserl’s decisive attempt to constitute the essence of the sign as something irreducible to an encapsulated semiological formal structure, that is, by integrating interacting communicative minds and the situations of sign use, may be considered an anticipation and a still not sufficiently well-explored foundation for recent and not so recent branches of cognitive science and semiotic and linguistic theory, stressing the enactive and the embodied nature of cognition and semiosis (Varela et al. 1991; Lakoff and Johnson 1999; Pelkey 2017, 2018, Mittelberg 2018), its social, interactive nature (Tylén et al. 2013). This history of influence, as it were, is still by and large to be written (see Gallagher 1997, though). It sets off early with the texts I have briefly examined here, and it goes on with the abovementioned analyses of the image signs (present also in the Ideen, Husserl 1952), the anti-representationalism developed in The Idea of Phenomenology (Husserl 1950b), the thorough examinations of the relations between experience, judgement and language in Experience and Judgement (Husserl [1939] 1999) as well as in the meditations about the constitutive function of intersubjectivity as exposed in the dense Cartesian Meditations (Husserl 1950a). The manner in which these hypotheses, claims and insights have sedimentated in contemporary theories of human meaning-making constitutes a rich and promising horizon for future research.

NOTES 1 Logical Investigations will henceforth be abbreviated ‘LI’ and referred to as ‘Husserl (1900–1)’. References are to the English translation. 2 Hjelmslev does, however, mention Husserl elsewhere (1928: 40), in rather negative terms, when commenting on ‘the philosopher Husserl’s strange hypothesis’ (‘la thèse étrange du philosophe Husserl’). 3 Two caveats: (1) they have roots directly back to Husserl, but have been recruited via Maurice Merleau-Ponty; (2) their being generated from Husserlian phenomenology does of course not mean that they are structurally identical; Husserl would most likely have considered Cognitive Linguistics much too psychologistic. 4 There is abundant literature within Cognitive Linguistics on the relation between language, semantic structure and ‘construals’; see, e.g. Langacker (1987/91; 2015) and Talmy (2000). 5 In the following I shall interchangeably be using ‘meaning-conferring’, ‘meaning-giving’ and ‘sense-giving’ acts for those cognitive acts where meaning is assigned to a sign in accordance with a meaning intention. 6 The fullness does not need to be perceptual; I can, for example, in a conversation with my brother evoke common memories of past experiences we had; in that case, the meaning intention would also be full. 7 In the concluding section of this chapter, I shall again address the issue about expressions which hardly can be said to elicit quasi-experiential simulations, e.g. abstract expressions or propositions. 8 These observations are mine. Husserl doesn’t consider the existence of a graduated link between the signifying indication and its signified. 9 Cf. Greimas (1982), Greimas and Courtés (1986): art. ‘Énonciation’.

PHENOMENOLOGY AND SEMIOTICS

119

10 Here is not the place to answer the question whether linguistic signs or concepts have core meanings and then different sorts of extensions from that core, or whether they do not have any such core meanings, but are vague predicates that are specified in the situation of use. Casasanto and Lupyan (2015) have made a strong case for the latter option; I myself have marshalled cognate ideas in Bundgaard (2019). Be it as it may, the non-univocity of linguistic signs is hardly disputed. 11 Husserl’s argument is here of a Chinese Room sort (cf. Searle 1980). There is not first some syntactic computation in one compartment which is then handed over to another compartment in which the semantic outcome of the first computation is extracted. The articulation is semantic from the outset, namely of the significant moments of the object as they are meant and expressed linguistically. 12 Again: the analogy with picture consciousness is Husserl’s own, so it does not at all follow that images are like linguistic signs; it’s the unitary structure of linguistic sign-processing that is like the unitary structure of picture consciousness. I have dealt with Husserl’s theory of depiction in Bundgaard (2022); see also Stjernfelt 2007: chapter 14). 13 For a non-psychological, realist phenomenological interpretation of meaning-fulfilling acts as quasi-perceptual acts, cf. Roman Ingarden (1973). Here, he qualifies readers’ ‘filling-out’ of the famous ‘spots of indeterminacy’ in literary artworks, but also in texts in general, as ‘a function analogous to perception’. See Bundgaard (2013) for a presentation of Ingarden’s theory.

REFERENCES Austin, J. L. (1962), How to Do Things with Words, Oxford: Oxford University Press. Barsalou, L. (1966), ‘Perceptual Symbol Systems’, Behavioral and Brain Sciences, 22: 577–660. Brandt, P. Aa. (1992), La charpente modale du sens, Amsterdam: John Benjamins. Bundgaard, P. F. (2003), ‘The Ideal Scaffolding of Language: Husserl’s Fourth Logical Investigation in the Light of Cognitive Linguistics’, Phenomenology and the Cognitive Sciences, 3 (1): 49–80. Bundgaard, P. F. (2010), ‘Husserl and Language’, in D. Schmicking and S. Gallagher (eds), Handbook of Phenomenology and Cognitive Science, 368–99, Dordrecht: Springer. Bundgaard, P. F. (2013), ‘Roman Ingarden’s Theory of Reader Experience: A Critical Assessment’, Semiotica, 194: 171–88. ­Bundgaard, P. F. (2019), ‘The Structure of Our Concepts: A Critical Assessment of Conceptual Metaphor Theory as a Theory of Concepts’, Cognitive Semiotics, 12 (1): 1–11. Bundgaard, P. F. (2022), ‘Aesthetic Perception as Vision for Appearance – On Husserl’s Theory of Depiction’, in F. Vassiliou (ed.), Phenomenology, Art, and the Aesthetic Mind. The New Yearbook for Phenomenology and Phenomenological Philosophy, vol. XIX, London: Routledge. Byrne, T. (2018), ‘The Evolution of Husserl’s Semiotics: The Logical Investigations and Its Revisions (1901–1914)’, Bulletin d’analyse phénoménologique, XIV (5): 1–23. Casasanto, D. and G. Lupyan (2015), ‘All Concepts Are Ad Hoc Concepts’, in E. Margolis and S. Laurence (eds), The Conceptual Mind: New Directions in the Study of Concepts, 543–66, Cambridge: MIT Press. Gallagher, S. (1997), ‘Mutual Enlightenment: Recent Phenomenology in Cognitive Science’, Journal of Consciousness Studies, 4 (3): 195–214. Greimas, A. J. (1966), Sémantique structurale, Paris: PUF.

120

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Greimas, A. J. (1970), Du sens, Paris: Seuil. Greimas, A. J. (1982), Du sens, II, Paris: Seuil. Greimas, A. J. and J. Courtés (1979), Semiotique, Paris: Hachette. Greimas, A. J. and J. Courtés (1986), ‘Énonciation’, in J. Courtés and A. J. Greimas (eds), Sémiotique: Dictionnaire raisonné de la théorie du langage, vol. 2, 75–77, Paris: Hachette. Hjelmslev, L. (1928), Principes de grammaire générale, Copenhagen: Det Kgl. Danske Videnskabernes Selskab, Historisk-Filologiske Meddelelser, vol. XVI (1). Holenstein, E. (1976), Roman Jakobson’s Approach to Language – Phenomenological Structuralism, Bloomington and London: Indiana University Press. Holenstein, E. (1992), ‘Phenomenological Structuralism and Cognitive Semiotics’, Scripta Semiotica, 1 (interview by Roberto Benatti), 133–50. Husserl, E. ([1900–1] 1970), Logical Investigations, trans. J. N. Findlay, London: Routledge. Husserl, E. ([1890] 1970), Philosophie der Arithmetik: Logische und psychologische Untersuchungen, ed. L. Eley, Husserliana XII, Den Haag: Nijhoff. Husserl, E. ([1893–94, 1921] 2005), ‘Logische Untersuchungen Ergänzungsband. Zweiter Teil. Texte für die Neufassung der VI. Untersuchung. Zur Phänomenologie des Ausdrucks und der Erkenntnis’, in U. Melle (ed.), Husserliana, XX–2, Dordrecht: Springer. Husserl, E. (1950a), ‘Cartesianische Mediationen und Pariser Vorträge’, in S. Strasser (ed.), Husserliana I, Haag: Nijhoff. English translation: Cartesian Meditations, trans. D. Cairns, The Hague: Nijhoff. 1960. Husserl, E. ([1907] 1950b), ‘Die Idee der Phänomenologie’, in W. Biemel (ed.), Husserliana II, Haag: Nijhoff. English translation: The Idea of Phenomenology, trans. L. Hardy, Dordrecht: Kluwer Academic, 1999. Husserl, E. ([1913] 1952), ‘Ideen zu einer reinen Phänomenologie und phänomenologischen Philosophie II: Phänomenologische Untersuchungen zur Konstitution’, in W. Biemel (ed.), Husserliana IV, The Hague: Nijhoff. English translation: Ideas Pertaining to a Pure Phenomenology and to a Phenomenological Philosophy. First Book: General Introduction to a Pure Phenomenology, trans. F. Kersten, Dordrecht: Kluwer Academic, 1982. Husserl, E. ([1898–1925] 1980), ‘Phantasie, Bildbewusstsein, Erinnerung. Zur Phänomenologie der anschaulichen Vergegenwärtigungen 1898–1925’, in E. Marbach (ed.), Husserliana, XXIII, The Hague: Nijhoff. English translation: Phantasy, Image Consciousness, and Memory (1898–1925), trans. J. B. Brough, Dordrecht: Springer, 2005. ­Husserl, E. ([1939] 1999), Erfahrung und Urteil, Hamburg: Meiner. English translation: Experience and Judgment, trans. S. Churchill, Evanston: Northwestern University Press, 1973. Ingarden, R. (1973), The Literary Work of Art, Evanston: Northwestern University Press. Jakobson, R. (1990), On Language, Cambridge: Harvard University Press. Lakoff, G. and M. Johnson (1980), Metaphors We Live by, Chicago: Chicago University Press. Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh, New York: Basic Books. Langacker, R. (1987–91), Foundations of Cognitive Grammar, vols 1–2, Stanford: Stanford University Press. Langacker, R. (1991), Concept, Image, and Symbol. The Cognitive Basis of Grammar, Berlin: Mouton de Gruyter. Langacker, R. W. (2015), ‘Construal’, in E. Dąbrowska and D. Divjak (eds), Handbook of Cognitive Linguistics (Cognitive Linguistics 39), 120–142, Berlin: Walter de Gruyter Merleau-Ponty, M. (1945), Phénomenologie de la perception, Paris: Gallimard. English translation: Phenomenology of Percepion, trans. C. Smith, London: Routledge and Kegan Paul, 1962.

PHENOMENOLOGY AND SEMIOTICS

121

Merleau-Ponty, M. (1960), Signes, Paris: Gallimard. English translation: Signs, trans. R. McCleary, Evanston: Northwestern University Press, 1964. Mittelberg, I. (2018), ‘Gestures as Image Schemas and Force Gestalts: A Dynamic Systems Approach Augmented with Motion-capture Data Analyses’, Cognitive Semiotics, 11 (1): 1–21. Pelkey, J. (2017), The Semiotics of X. Chiasmus, Cognition and Extreme Body Memory, London: Bloomsbury Academic. Pelkey, J. (2018), ‘Upright Posture and the Meaning of Meronymy: A Synthesis of Metaphoric and Analytic Accounts’, Cognitive Semiotics, 11 (1): 1–22. Petitot, J. (1982), Pour un schématisme de la structure, Doctoral Thesis, Paris: École des Hautes Études en Sciences Sociales. Petitot, J. (1992), Physique du Sens, Paris: éditions du CNRS. Searle, J. (1980), ‘Minds, Brains and Programs’, Behavioral and Brain Sciences, 3: 417–57. Sonesson, G. (2015), ‘The Mirror in-between Picture and Mind. A Phenonomenologically Inspired Approach to Cognitive Semiotics’, Chinese Semiotics, 11 (2): 159–80. Stjernfelt, F. (2007), Diagrammatology: An Investigation on the Borderlines of Phenomenology, Ontology, and Semiotics, Dordrecht: Springer. Talmy, L. (2000), Toward a Cognitive Semantics, Cambridge: MIT Press. Tylén, K., R. Fusaroli, P. F. Bundgaard and S. Østergaard (2013), ‘Making Sense Together: A Dynamical Account of Linguistic Meaning-Making’, Semiotica, 194: 39–62. Varela, F., E. Thompson and E. Rosch (1991), The Embodied Mind, Cambridge: MIT Press. Zlatev, J. (2018), ‘The Semiotic Hierarchy and Phenomenology’, Cognitive Semiotics, 11 (1): 1–18. Zwaan, R. (2004), ‘The Immersed Experiencer – Toward an Embodied Theory of Language Comprehension’, The Psychology of Learning and Motivation, 44: 35–62. Zwaan, R. (2014), ‘Embodiment and Language Comprehension: Reframing the Discussion’, Trends in Cognitive Sciences, 18 (5): 229–34. Zwaan, R. (2016), ‘Situation Models, Mental Simulations, and Abstract Concepts in Discourse Comprehension’, Psychonomic Bulletin and Review, 23: 1028–34.

122

­C HAPTER SIX

Hermeneutics and Semiotics RONALD C. ARNETT AND SUSAN MANCINO

Hermeneutics and semiotics share an intellectual objective: to discern meaning and to provide conversational dwellings for and to another sign that can direct interpretation differently than expected. The connecting link between semiotics and hermeneutics is twofold: attentiveness to what is before us and interest in the ‘not yet’ envisioned. Both fields of study move from what is to the possible. Text and sign welcome respectful learners who seek insight about current conditions that make possible a future that one cannot predict but can potentially surmise. Martin Buber queried: What happens when life seems to turn in a problematic direction? And you ask: How did I get here? He stated that whenever one ignores existential signs, one invites unforeseen challenges. Buber wrote, ‘Each of us is encased in an armour which we soon, out of familiarity, no longer notice. There are only moments which penetrate it and stir the soul to sensibility’ (Buber [1947] 2002: 12–13). To fail to understand the text, to disregard the signs, invites a practical and existential ignorance that makes one vulnerable to a world that goes unacknowledged. Without alertness to text and sign, one falls prey to taken-for-granted assumptions that ignore both hermeneutics and semiotics, attempting to live in unreflective reaction alone (Schutz 1967: 74). Attentiveness to text and sign, hermeneutics and semiotics, invites dialogic participation in and with existence. Interpreters are not in control of the signs they encounter but must be responsive to them. Responsiveness engages semiotics and hermeneutics in performative action; this chapter explicates that interaction in the following three sections. The first, ‘Textual Traces: Historical Examinations’, provides an overview of hermeneutics followed by an examination of semiotics. These brief historical accounts yield a dialogic perspective that intersects with the mutual insight in each area of study. The second section, ‘Dialogic Voices: Gadamer and Eco’, centres on our reading of hermeneutics and semiotics, stressing Hans-Georg Gadamer’s Truth and Method and Umberto Eco’s A Theory of Semiotics as representative texts that demonstrate the interconnections between each interpretive tradition. The third section, ‘Dialogue and Semiosis in Action: Numero Zero’, explores Eco’s final novel, Numero Zero, as an interpretive artefact elucidating the interplay of hermeneutics and semiotics in action, emphasizing the interconnecting themes of dialogue and semiosis. This chapter frames hermeneutics and semiotics as essential co-players in the pursuit of dialogic knowledge and understanding. This chapter provides a descriptive history of hermeneutics and semiotics with a stress on two related performative metaphors: dialogue and semiosis. Hermeneutics

124

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

and semiotics explicate signification of human meaning. Hermeneutics depends upon interpretive difference and uniqueness of questions that open a text within dialogic understanding. Semiotics reveals an existential fact: one sign yields interpretive insight for and about another. Multiplicity of possibilities, interpretations and discernment originate with signs, questions, and texts. For example, Eco’s first novel and international bestseller, The Name of the Rose,1 ‘offers a dialogue of many texts’ that moves the reader to hypertextual connections embedded within history and ‘textual traces’ (Arnett 2017: 145–6). These textual traces announce dialogic semiosis in the interplay between hermeneutics and semiotics.

TEXTUAL TRACES: HISTORICAL EXAMINATIONS This section outlines descriptive histories of hermeneutics and semiotics as distinct areas of inquiry. For both interpretive traditions, key texts recount the emergence and evolution of each field of study. An emphasis on dialogue and semiosis informs our brief historical examinations, beginning with hermeneutics and then moving to semiotics, as they manifest reciprocal insights.

Hermeneutics Three texts guide a short history on hermeneutics: Richard Palmer’s Hermeneutics: Interpretation Theory in Schleiermacher, Dilthey, Heidegger, and Gadamer; Jean Grondin’s Sources of Hermeneutics; and Maurizio Ferraris’s History of Hermeneutics. Each text surveys the history of hermeneutics distinctively, offering a context for understanding Gadamer’s philosophical hermeneutics. Palmer’s2 work surveys the history of hermeneutics and provides an overview of defining themes and theses. He delivers a thoughtful examination of the evolution of interpretive inquiry from its ancient roots to the mid-twentieth century. Through Friedrich Schleiermacher (1786–1834), Wilhelm Dilthey (1833–1911), Martin Heidegger (1889–1976) and Gadamer, hermeneutics moves from precision of answer and reified signs to a horizon of multiple possibilities. Palmer expands biblical interpretation to an understanding of human experience. His final chapter lists thirty theses on interpretation grouped into five central themes. 1. Palmer conceptualizes the hermeneutical experience as situated historically, linguistically and dialectically, with the whole offering insight into the part and the part into the larger understanding of the text (1969: 242). The hermeneutical experience is ontological as it discloses being in the interplay of text and interpreter. The hermeneutical experience is a ‘language event’ that both illuminates and conceals (Palmer 1969: 243). Furthermore, it is objective; Palmer underscores that one cannot impose meaning upon a text. One must disclose the ground from which one stands, looks, observes and interprets (Palmer 1969: 243). The hermeneutical experience is text driven (Palmer 1969: 244). It illuminates meaning within the present and is a temporal ‘disclosure of truth’, attending to that which is before us (Palmer 1969: 244–5). Finally, the hermeneutical experience recognizes aesthetics as dependent upon the encounter of interpreter and text (Palmer 1969: 245). 2. Palmer conceptualizes transcendence as the hermeneutical experience inclusive of interpreter and text, forgoing a subject-object framework (1969: 246).

HERMENEUTICS AND SEMIOTICS

125

3. Palmer conceptualizes autonomy and objectivity as an interpretive fact: the meaning of a text goes beyond interpreter sensibilities (1969: 246–7). 4. Palmer conceptualizes methods as inclusive of the situatedness of the interpreter, contrary to any effort to master a text – attentive to implications and not just form (1969: 247–8). Palmer understands the text as speaking, requiring an interpretive engagement that does not seek to dominate or smother (1969: 248). The interpreter is a participant within the experience of the parole of the text (Palmer 1969: 248–9). Hermeneutics is an encounter as one brings attentiveness beyond innocence to the experience (Palmer 1969: 249). Hermeneutical experience seeks to understand the interaction of interpreter and text (Palmer 1969: 249). A hermeneutical method discloses temporal, not final, answers (Palmer 1969: 250). Finally, questions that open the horizon of the text emerge within the experience of a given interpreter (Palmer 1969: 250). 5. Palmer conceptualizes interpretive historical consciousness as one’s own experience interrogating the historical consciousness of a given text (1969: 250–1). All literature is historical and shaped within experience; such is the reason that hermeneutics requires historical distance, respect for the text and recognition that understanding is dependent upon the standpoint of a given interpreter (Palmer 1969: 251–2). Understanding a text is never abstract or non-temporal; one must attend to the dwelling of the work (Palmer 1969: 252). History and experience yield interpretive insight. The task of hermeneutics is not scientific objectivity but historical encounter between a given text and the particular lived experience of a given interpreter. Palmer’s introductory text provides the coordinates from which we understand the contributions of Grondin and Ferraris as they announce the evolving influences of metaphysics, phenomenology and authorial intent. Grondin’s3 (1995) Sources of Hermeneutics examines metaphysics and hermeneutics, with an explication of phenomenology’s influence. From the work of Heidegger forward, metaphysics increasingly takes on questionable status. Grondin contrarily relies upon the insights of Gadamer to renew and broaden hermeneutic insights exemplified with the importance of metaphysics. Metaphysics carries with it a given method of thought that uplifts meeting and learning from recognition of bias in interpretive engagement with a text from a particular standpoint. Grondin attends to phenomenology’s contribution to hermeneutics, as seen in his chapter emphasizing ‘Husserl’s Silent Contribution to Hermeneutics’. Husserl’s notion of intentionality explicates an existential fact: ‘the thing itself’ changes in its engagement with a given event, object or person. The lifeworld calls forth particularity of the things themselves that morph and shift with responsive perspective to a changing historical moment. Grondin examines the early work of Husserl’s student Heidegger and his young Hegelian insights, which emphasized the notion of the ethical (1995: 47). Emmanuel Levinas, however, provided a public counter to Heidegger’s position, with a focus on ethics as first philosophy. Grondin contends that there was a serious and significant implication in Heidegger’s stress on the awaking of an ethical consciousness. This orientation reflects a Hegelian perspective in which an original ethical consciousness can seduce one into the search for the correct moral conscience. Grondin continues this caution with Jürgen Habermas’s argument that Heidegger’s philosophical turn lacked argumentative rigour and actually justified Nazi involvement. Heidegger radicalizes a path towards thinking, free from metaphysics and attentive to the immediacy of thinking itself. Grondin points

126

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

to a rigidity that rests within a critical disposition towards metaphysics. The existential reality of particularity and ‘the things themselves’ provides, for Grondin, a call for multiplicity that undergirds Gadamer’s project. Grondin announces phenomenological influences on hermeneutics with particular insight into how they contributed to Gadamer’s philosophical hermeneutics. Grondin indicates that Gadamer’s most well-known tome, Truth and Method, originally published in 1960, introduces philosophical hermeneutics as the primary focus of Gadamer’s corpus with an emphasis on a triad of philosophy, art and history. Grondin adds a perspective that language is the channel, the driver of ontological content and a guide for philosophy, art and history. Such insight philosophically does not come from a method or a final story. Understanding emerges from an interpreter tied to a given event or object situated within a particular historical moment. Philosophical hermeneutics is always reflexive, working within limits that are organically before us and require acknowledgement. Grondin stresses that both Gadamer and Augustine point to hermeneutics as universal. Hermeneutics, for Augustine, houses the revelatory, and Gadamer stresses hermeneutics as the home of the unexpected and ‘not yet’. Gadamer presupposes humanism in hermeneutics, but not an anthropocentric humanism. The human is not at the centre, but is the responsive recipient within experience and the historical moment. The philosophical humanism of Gadamer offers an embedded and situated sense of interpretation, forever responsive to limits. Heidegger, of course, abandons all humanism and develops an anti-humanism. Gadamer, however, emphasizes a humanism deeper than the communicative agent alone, recognizing that we live within a moment historically defined. The interpretive task is one of dialogic responsiveness, not imposition of one’s own demands and expectations of a given text. Gadamer’s hermeneutics ultimately situates the human being within a dialogic moment, one that is ever responsive to the past, the present and the ‘not yet’. Understanding is an engagement of difference responsive to a bias that is not psychological in nature but historically given and dialogically understood. In History of Hermeneutics, Ferraris4 ([1988] 1996) acknowledges the shift from authorial intent into the realm of an interpretive exchange between and among texts, readers, authors and the historical moment, with each temporally situated. He begins in the classical period with Hermes, the messenger god, who witnesses to the practical and communicative nature of interpretation. Hermeneutics, in a classical sense, announces expression; early work on hermeneutics privileges authorial intent. This authorial orientation continues through the Enlightenment with increasing emphasis on the communicative agent. This line of inquiry undergirds Friedrich Nietzsche’s (1844–1900) and Sigmund Freud’s (1856–1939) discussions of a deconstructive hermeneutic of suspicion. The ontological project of Heidegger brings forth an existential reality: the human’s thrownness into existence. Heidegger undercuts authorial intent and stresses existential givenness. He shifts hermeneutics into ontological existence, the field of interpretive understanding and play, which opens the text for the reader. Gadamer, a student and research assistant for Heidegger, adds bias to the notion of existential thrownness. Gadamer’s philosophical hermeneutics situates bias as the central disruption of the assumption of authorial intent. Philosophical hermeneutics presupposes that meaning resists ownership by authorial mandate. This position finds its greatest emphasis in Jacques Derrida’s (1930–2004) deconstruction; Gadamer acknowledges multiple biases that open texts differently and, at times, contentiously.

HERMENEUTICS AND SEMIOTICS

127

Ferraris concludes with Gadamer and Ricoeur, perhaps the most definitive scholars on philosophical hermeneutics. Gadamer provides a dialogic entrance into the humanities and the human sciences, recognizing the importance of a community of scholars in the discernment of temporal validity of a given interpretation. Similarly, Ricoeur immersed hermeneutics in multiple conversations that seek to emancipate interpretation from the text alone. Ricoeur, however, distinctively embraced the narrative nature of interpretation, as well as the existential reality of a hermeneutics of suspicion played out thoroughly by Freud, Nietzsche and postmodern discourse that called a hermeneutics of the universal into question. Ferraris describes the historical shifts in the hermeneutic project from a classical era of expression, to one of authorial intent, and finally to the onset of philosophical hermeneutics and its interpretive engagement between and among author, text, history and interpreter. Hermeneutics assumes the interpreter dwells within existence, bringing particularity of perspective and experience in the meeting of the lifeworld. Bias is a pragmatic and existential reality that opens up and limits interpretive insights. This perspective guides the insights of semioticians such as Susan Petrilli, Augusto Ponzio and John Deely.

Semiotics We turn to two texts on the history of semiotics: Petrilli and Ponzio’s Semiotics Unbounded: Interpretive Routes through the Open Network of Signs and John Deely’s Semiotic Animal: A Postmodern Definition of ‘Human Being’ Transcending Patriarchy and Feminism. Each text situates semiotics historically. We begin with Petrilli and Ponzio’s text, which provides the historical scope of semiotics and semioethics, and then transition to Deely’s work, which situates semiotics in a postmodern recognition of multiple interpretive possibilities co-present in the signification of signs. Petrilli, Ponzio and Deely collectively assert that, in an era of difference and hypertextual meaning, the interpretation of signs carries semioethical implications. Signs and interpretation are value laden, not neutral or objective. Petrilli and Ponzio5 (2005), in Semiotics Unbounded, provide a thorough and accessible history of semiotics. Their approach is attentive to dialogue, otherness, communication, ethics and responsibility; their project is a value-laden interpretive undertaking. We know of no other scholars who provide such a consistent and extensive commitment to the intersections between and among communication, semiotics and ethical implications. Semiotics Unbounded covers more than 600 pages. It begins with an introduction to their orientation towards semiotics followed by an overview of semioticians central to their work: Charles Sanders Peirce (1839–1914), Lady Victoria Welby6 (1837–1912), Mikhail Bakhtin (1895–1975), Charles Morris (1901–79), Thomas Sebeok (1920–2001), Ferruccio Rossi-Landi (1921–85) and, ultimately, Eco. Petrilli and Ponzio include forms of application commencing with a section on ‘Modelling, Writing, and Otherness’ (2005: 341) that stresses dialogue. The next section, ‘Predicative Judgment, Argumentation, and Communication’ (Petrilli and Ponzio 2005: 429), examines (mis)understanding, closed and open communities, global communication, biosemiotics and semioethics. Petrilli and Ponzio engage semioethics in a number of projects. They explicate semiotics and semiosis within an interpretive context of responsibility, attending to issues of dispossession, extralocalization (Petrilli and Ponzio 2005: 558) and our biosemiotic condition. They engage in critical, philosophical, theoretical and interpretive judgement that moves

128

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

otherwise than conventional efforts of dominated technical control. Their intellectual contribution situates an interpretive position within semiotic responsibility ever attentive to semiosis. Petrilli and Ponzio explicate the semiotic significance of otherness and ethics in semiosis as they review Peirce’s emphasis on Firstness, Secondness and Thirdness, which they connect, respectively, ‘to abduction, deduction, and induction in logic;7 to icon, index, and symbol in semiotics;8 and to agapism, anancism, and tychism in ontology9’ (2005: 28). Petrilli and Ponzio emphasize Firstness as ‘originality’, in phenomenological terms, a ground, undifferentiated material that requires eventual differentiation. Secondness is an ‘over-againstness’ that offers a first step of differentiation in relation to an object (Petrilli and Ponzio 2005: 28). Thirdness is that which emerges in between signs in triadic relations, enhancing differentiation dialogically through mediation (Petrilli and Ponzio 2005: 28–31). With each move among Firstness, Secondness and Thirdness, greater clarity of signification emerges. Petrilli and Ponzio provide an examination of Eco’s reliance on Peirce and the triadic relationship of sign, object and interpretant. This triadic notion forms semiosis, which makes otherness consistently elusive. Petrilli and Ponzio emphasize Eco’s ‘limits of interpretation’ as a reminder of the otherness of signs (2005: 298);10 signs do not adhere to our demands but yield meaning in their otherness. Understanding otherness as difference makes interpretive insight of semiosis possible. Understanding interpretation as potentially solidified and unduly clear moves contrary to Eco’s understanding of semiotics in everyday engagement. His contribution influenced Italy and made a global contribution to semiotics. Petrilli and Ponzio credit Eco with the introduction of ‘interpretation semiotics’ as a passage beyond decodification semiotics (2005: 299). Decodification presupposes an ability to stand above the human context and the historical moment; interpretation relies upon such existential realities. For Eco, however, codes provide interpretive paths connecting source and destination without guaranteed agreement of signification. Semiotics, particularly as understood through Peirce’s ontology, walks between the objective and the subjective, and is often described as a form of ‘objective idealism’ (Petrilli and Ponzio 2005: 306). Eco understood the importance of unlimited semiosis without falling prey to deconstructionism. Following the work of Peirce, Eco unites interpretation with unlimited semiosis and an emphasis on openness (Petrilli and Ponzio 2005: 303). Eco distinguishes between open texts and closed texts as a means for understanding the interpretive engagement among authors, texts and readers. While a closed-text implements constraints to guide readers in the interpretation of signs, the open text contains an excess of information that requires readers to venture deep into the path of unlimited semiosis, navigating multiple possibilities for meaning and uncovering interpretive plurality. Eco’s project moves contra to the extremes of ideology as witnessed by his notion of communication as a theory of sign production. Petrilli and Ponzio refer to Eco’s description of ideology as ‘a false consciousness’, placing it within sign production inattentive to the existential normality of semiosis (2005: 321–2).11 Interpretation and semiotics, for Eco, are kindred activities announcing ongoing dialogic relationships between and among multiple texts and signs. With attentiveness to Eco and Bakhtin, Petrilli and Ponzio presuppose that dialogue constitutes how human beings make sense of the world. Interpreting meaning within the dialogue between and among signs engages abduction as a response to unlimited semiosis. Abduction presupposes the necessity of background in order to discern foreground issues in a hypertextual historical moment.

HERMENEUTICS AND SEMIOTICS

129

Eco defined hypertextuality as the co-presence of multiple texts, codes and signs in a given moment. Deely12 (2010) situates semiotics in a hypertextual and postmodern historical moment with his book Semiotic Animal. Deely and Eco deal with postmodernity as an interpretive revolution, reclaiming attentiveness to a hypertextual recognition that does not fall prey to modernity’s linear perspective and hypermodernity’s emphasis on the individual. Postmodernity is often confused with hypermodernity. Postmodernity, however, resists the paradigmatic progression of modernity, which blinds one to the responsibility of being ever-vigilant semiotic animals. Deely contends that human interpreters act as semiotic animals in an age thematized by an existential fact: we disagree on the value implications and signification of signs, making the notion of semioethics vital. Semioethics attends to how signs matter within a human community and how their meaning shifts, ever attentive to semiosis. Deely reminds us that semiotics does not live within an ideological system in a dwelling of semiotic animals that attends to semiosis in individual and collective lives (2010: 32–3). Historical epochs shape signification of signs and codes. Postmodernity presupposes hypertextual meaning, suggesting multiple interpretive possibilities are co-present in a given sign. Awareness of semiosis is the semiotic heart of a postmodern/hypertextual age (Bain 2006). As a single sign morphs in semiosic engagement, it carries forth hypertextual meanings. Hypertextuality suggests temporal clarity without universal final answers by presupposing meaning situated within signs, relations and intersubjectivity. Deely attends to an intersubjective world that lends itself to Peirce’s triadic relations of representamen (form of/potential for meaning), object (sign vehicle represented by sign) and interpretant (action of sign).13 From this pragmatist perspective, the structure of a sign is a triadic relationship, which forms a trinity that makes semiosis and the advancement of insight organically inevitable. Deely discusses the semiotic animal as beyond mere separation, ever responsive to intersubjective understanding. The semiotic animal dwells distinctly situated in place, space and nature. He emphasizes that semioethics does not view the human as merely rational, but called forth as a semiotic animal that is responsible, upright and willing to attend to existential dilemmas. The semiotic animal responds to a life of responsibility inclusive of the biosemiotic sphere. Semioethics functions as an interpretive code for semiotic maturity defined by responsibility for others. Semiotics, like hermeneutics, opens possibilities for interpretive insight and recognizes the historical, cultural and temporal situatedness of signs, texts and interpreters. The history of semiotics witnesses movement from universal reification and certitude to an increasing awareness of uncertainty and ongoing semiosis. The history of hermeneutics follows a similar course, moving from certitude and psychological imposition to the admission of bias and prejudice as fundamental to the act of interpretive understanding. These historical examinations illuminate two key thinkers within hermeneutics and semiotics: Gadamer and Eco, respectively. Gadamer and Eco offer dialogic voices that build upon historical attentiveness. From a continental understanding of dialogue, this interpretive engagement commences long before one meets another, a text, or an event (Buber [1947] 2002). Dialogue lives within history, meeting the present with responsiveness to the ‘not yet’. Dialogue functions as a communicative phenomenon between and among historical moments, with semiosis engaging a similar creative interplay of meaning. Gadamer and Eco recognize dialogue and semiosis as guiding metaphors that elucidate the interconnections between hermeneutics and semiotics.

130

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­DIALOGIC VOICES: GADAMER AND ECO The dialogic voices of Gadamer and Eco represent philosophical hermeneutics and interpretation semiotics, respectively. The following discussion situates each thinker within his historical moment before outlining a work reflective of his thinking – Truth and Method for Gadamer and A Theory of Semiotics for Eco. These works characterize their authors’ expansive contributions to hermeneutics and semiotics. Their insights illuminate dialogue within philosophical hermeneutics and semiosis within the pragmatic perspective of interpretation semiotics. The temporal pursuit of truth and understanding coordinates the contributions of hermeneutics and semiotics.

Gadamer’s Truth and Method Gadamer (1900–2002) studied at the University of Marburg and the University of Freiburg, where he was a student of and research assistant for Heidegger. He began his teaching career at Marburg in 1929 before teaching at Leipzig University from 1938 to 1948 and then at Heidelberg University, where he succeeded Karl Jaspers as chair and where he would later retire as professor emeritus in 1968 (Mueller-Vollmer [1985] 2006: 256). At the age of sixty, Gadamer penned Truth and Method, what Georgia Warnke (1987) named his magnum opus, what Palmer referred to as a ‘masterwork’ (1987: 135) and what Klaus Dockhorn described as ‘a liberating breakthrough to new horizons’ (1980: 160). According to Palmer, Truth and Method introduced philosophical hermeneutics and revolutionized existing perspectives on aesthetics, history and language with practical insight and a counter to modern thought (1987: 135). Truth and Method has three parts: the first is on truth and art, the second on truth and the human sciences, and the third on the ontological shift to language in hermeneutics. Ultimately, Gadamer illustrates philosophical hermeneutics with truth as a temporally situated principal objective and method as a desire for application. Gadamer’s scholarship rests within a humanities-based commitment to temporal understanding, in which one attempts to discern appropriate questions and responds to the conditions of a given historical moment. He frames understanding as temporal, finite and dialogically responsive. The aesthetic dimension is important to Gadamer as he emphasizes a humanistic tradition of the human sciences. Gadamer begins with an emphasis on four concepts central to the aesthetic dimension: Bildung, sensus communis, judgement and taste. Gadamer invokes the notion of Bildung as more than culture alone; he emphasizes its ancient mystical tradition, in which the image of God, carried in one’s soul, requires the cultivation of self ([1975] 1993: 11). This cultivation is a practical spirit of working with historical consciousness in order to attend to the questions of a given moment and to render proper temporal responses. Likewise, Gadamer contends that sensus communis, Vico’s notion of common sense, is attentive to community. Sensus communis initiated Vico’s use of a humanistic rhetoric against a modern understanding of science and makes the sympathy that the Earl of Shaftesbury discussed in the early Scottish Enlightenment possible (Gadamer [1975] 1993: 24). Sensus communis manifests the practical knowledge of phronesis (Gadamer [1975] 1993: 21). Gadamer links sensus communis to judgement where one learns that common sense emerges in judgements about ‘right and wrong, proper and improper’ ([1975] 1993: 32). The break from sensus communis to the notion of judgement comes with Immanuel

HERMENEUTICS AND SEMIOTICS

131

Kant (1724–1804). His doctrine of schematism understands the human being as engaged in self-legislation. However, Kant places judgement of taste within communities, the social life of persons. Judgements attending to sensus communis require taste educated within, or at least about, a given community. Kant actually equates taste with what could be termed a ‘true sense of community’, in which he brings the transcendental to his understanding of sensus communis (Gadamer [1975] 1993: 34). Kant enters into pure judgement, which endangers observations of and about a culture or phenomenon contrary to one’s own, freeing the notion of taste from sensual and rational prejudices. This transcendental engagement brings Kant to the connection between taste and genius, with genius beginning to remove the notion of taste from consideration. As taste loses its power, it becomes something more akin to testimony (Gadamer [1975] 1993: 56). Gadamer enters into a critique of pure rationality with the concept of Erlebnis, or experience, which he connects to historical consciousness via tradition. Consciousness, for Gadamer, is not a given event, but a time in which there is a form of consciousness where one experiences a sense of knowing in the lifeworld. His understanding of experience and a historically developed sense of consciousness contrasts with the transcendental position. Gadamer critiques the abstraction of the aesthetic. Gadamer argues that speculative idealism challenged the aesthetic subjectivism and agnosticism of Kant with the adoption of a standpoint of ‘infinite knowledge’ ([1975] 1993: 99). Contrarily, Gadamer emphasizes finiteness; he underscores the rightness of Heidegger’s modern subjectivism that opens understanding temporally without reliance upon the transcendent and the infinite. Being dwells, for Heidegger, within experience and existential life, a point that Gadamer carries forth in his emphasis on experience and tradition. Interestingly, phenomenology returns to the aesthetic experience tied to the transcendental, with existential phenomenology attentive to experience and traditions. However, when Gadamer refers to tradition, he emphasizes the plural: traditions. He jettisons a modern universal for acknowledgement of multiplicity of traditions. The notion of play is Gadamer’s signature display of the union of temporal experience and embodiment. One cannot transcend play; one is within it – a characteristic of experience is one’s transformation into and within the structure of play. It is not a second version, copy, imitation or representation, but rather engagement in the activity itself. Play requires structure and engaging the confines of coordinates. The art of play attending to historical consciousness and particularity results in an interpretive self-forgetfulness within the engagement of play. There are limits on one’s ability to play; undue reflection interrupts play as an embodied doing (Arnett 2018). The embodiment of experience permits a dramatic piece or musical work to announce itself differently, no matter how many performances. Play has a communion component of dialogic engagement. Gadamer understood the ‘subjectivist attitude’ of play in modern aesthetics as essential to ‘the event of art proper’ ([1975] 1993: 145). Play underscores his discussion of pictures, decorative arts and architecture in relation to signs and symbols. Gadamer situates play in relation to a picture, which rests somewhere halfway between a sign (what he understands as ‘pure substitution’) and a representation (what he refers to as ‘pure indication’), offering an experiential space that renders a temporal glimpse of play in action ([1975] 1993: 152). Gadamer identifies an ontological difference between picture and sign: ‘The picture does not disappear in pointing to something else but, in its own being, shares in what it represents’ ([1975] 1993: 153). Likewise, the picture dwells halfway between a sign (‘pure taking-the-place-of-something’) and a symbol (‘pure pointing-to-something’) (Gadamer [1975] 1993: 154). The task of play is not one of restoration, but temporal

132

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

understanding driven by knowing imagination. Hermeneutics is akin to play in that experience, the historical moment and the activity invite dialogic meeting. Gadamer’s dialogic engagement with philosophical hermeneutics relies upon Schleiermacher’s effort to make texts understandable. Schleiermacher employs a psychological move of one seeking to place oneself in the space of a writer or reader (Gadamer [1975] 1993: 187). Schleiermacher, however, points to a ‘paradoxical formula’: the original writer is generally not the best interpreter of a given text (Gadamer [1975] 1993: 193). Schleiermacher understood the expressive nature of interpretation. Gadamer contends that there is a beginning shift from individual expression in interpretation to historical thinking, which offers a space for interpretive insight. Productivity, meaning and understanding dwell within history. Gadamer cites Leopold von Ranke (1795–1886), tying hermeneutics to historical thinking and the power of necessity (Gadamer [1975] 1993: 204–12). This understanding of history embraces interpretive engagement within tradition. There is a movement from Schleiermacher’s romantic theory of understanding, where interpretation belongs to a single interpreter, to attentiveness to shifts in the historical moment. For Gadamer, dialogue between and among historical moments situates interpretive understanding. As Gadamer shifts to Dilthey’s influence, he writes, ‘Dilthey’s hermeneutics of life fundamentally seeks to retain the historical worldview’ ([1975] 1993: 226). History moves interpretation from a psychological framework into the temptation of a historical consciousness that flirts with objectivity, universality and scientific certainty. Dilthey sought a human science that challenged the dogmatic, while attempting to forge an insight closer to objectivity than raw subjectivity. Dilthey understood the dangers of objectivism and the reification of insight; he sought a horizon of interpretation without a ‘rigid boundary’ (Gadamer [1975] 1993: 245). Dilthey began with experience; yet, Heidegger attacked his desire for a transcendental reduction for an emphasis on the existential thrownness of human life. For Heidegger, history does not lend itself to a transcendental lens. There is no such thing as ‘a particular ideal of existence’ (Gadamer [1975] 1993: 263). Thrownness indicates not a transcendental gaze, but a situated and embodied effort to understand temporally. All insight, whether historical or situated, begins with an existential position resulting from thrownness. Gadamer outlines the Enlightenment prejudice against bias itself ([1975] 1993: 271). It is impossible to understand a historical moment or tradition from a transcendental posture. This perspective evades a recognition that the interpreter begins with assumptions, not transcendental clarity. These suppositions are sometimes known, assumed and unearthed by others. Understanding is finite; one cannot gain an absolute, perfect or pure historical knowledge of existence. Western history and the Enlightenment share a love for progress that permits us to distance ourselves from the preconditions of tradition, history and environment. This interpretive stance is existentially wrongheaded and opposed to a hermeneutical attitude respectful of the text and the limits of one’s own gaze. Gadamer emphasizes historicity for the text, the interpreter and the finiteness of possibilities for interpretation. Historicity offers a unity of contraries: finiteness linked directly with multiplicity of possibilities. Likewise, interpretive horizons embody both multiplicity and limitations (Gadamer [1975] 1993: 302). Historical knowing makes practical knowledge possible and absolute knowledge unattainable. Gadamer also stresses the relevance of Aristotle’s emphasis on action and its implications for interpretation engaged in practical knowledge, contrary to technical language ([1975] 1993: 315). Practical knowledge represents a deep interpretive

HERMENEUTICS AND SEMIOTICS

133

humanities engagement, with application void of absolute certainty associated with technical language. Interestingly, Gadamer cites the Pietist theologian tradition as a basis for the issue of application ([1975] 1993: 307). Gadamer’s humanities-centric form of hermeneutics emphasizes reading as a form of application that dwells in tradition responsive to change and not infinitude ([1975] 1993: 340). Gadamer’s emphasis on experience tied to finitude invites dialogic engagement with the ‘thou’ of historical consciousness and assumes interpretive bias, perpetually announcing limits. Gadamer places interpretation within the engagement of questions and answers; he writes, ‘Thus interpretation always involves a relation to the question that is asked of the interpreter. To understand a text means to understand this question’ ([1975] 1993: 370). Gadamer understands opinion as an effort to answer a question within a historical horizon. He writes, ‘To think historically always involves mediating between those ideas and one’s own thinking. To try to escape from one’s own concepts in interpretation is not only impossible but manifestly absurd’ (Gadamer [1975] 1993: 397). Gadamer articulates the fundamental alignment of dialogue, interpretation and understanding. As dialogical creatures, humans understand within limited historical perspectives, permitting questioning and interpreting within particular experience. Interpretation requires greater knowledge and reflection beyond the immediacy of contexts between persons whose practices unite them. The further one is from a given text historically, the greater the need for reflection upon the biases of text and interpreters. One of Gadamer’s most fundamental contributions is moving hermeneutics from the psychological and the individual to the historical, emphasizing temporal limits, dialogue, multiple horizons of interpretive possibilities and the reality of incorrect interpretations. For Gadamer, limits surround texts, warning against the extremes of subjective and relativistic interpretation without assuming an objective, universal hermeneutic. Gadamer’s philosophical hermeneutics privileges dialogue in the emergence of meaning, as interpreters and texts meet in the discernment of multiple horizons of insight. Multiplicity of possibilities unites Gadamer’s philosophical hermeneutics and Eco’s interpretation semiotics. For Eco ([1990] 1994), a commitment to multiple paths of meaning, particularly through semiosis, limits the possibilities of overinterpretation, unresponsive to the constraints of a text. Eco’s semiotic engagement, like Gadamer’s hermeneutics, positions interpretation as a meeting of texts between the extremes of relativism and objectivity, which is central to Eco’s highlighting of semiosis.

Umberto Eco’s A Theory of Semiotics Eco (1932–2016) studied medieval aesthetics under Luigi Pareyson at the University of Turin,14 where he penned what would become his first book, The Aesthetics of Thomas Aquinas. His early work focused on cultural aesthetics, which framed his hermeneutic entrance into semiotics. Eco approached semiotics as ‘a logic of culture’, which he describes in the first chapter of A Theory of Semiotics, the inaugural text in Indiana University Press’s Advances in Semiotics series edited by Sebeok. Deely portrays the work as ‘one small step for philosophy, one giant leap for the doctrine of signs’ (1997: 82). Reviews from Richard L. Lanigan (1978), Frank E. X. Dance (1977) and Thomas W. Benson (1977) announce its creative relevance for communication studies. Lanigan emphasizes the work’s distinctive and novel contribution in its theoretical union of communication theory and rhetorical theory (1978: 345). Although Eco’s semiotic project continued to

134

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

evolve throughout his corpus, A Theory of Semiotics outlines the heart of his interpretation semiotics (Deely 1997: 82; Petrilli 1997). In A Theory of Semiotics, published in 1976, Eco translates his existing research on semiotics into English and introduces interpretation semiotics (1976: 314). The work contains five parts: 1. A framing of semiotics as a logic of culture 2. An examination of the issues of signification and communication 3. An explanation of signification as a theory of codes 4. An exploration of communication as a theory of sign production 5. An acknowledgement of the human interpreter as ‘the subject of semiotics’ In the foreword, Eco outlines four major objectives as he advances his semiotic project (1976: viii): 1. Seeking a semiotic theory of referents 2. Bringing semantics and pragmatics together 3. Questioning the common classical topologies that underscore and define the notion of signs 4. Advancing his critique of iconism beyond the excesses of trivialization and undue clarity Additionally, Eco’s theory of sign production moves beyond a typology of signs to a typology of sign production Eco begins by framing semiotics as ‘a logic of culture’ (1976: 3). As one sign yields insight into another, it is simultaneously a trace of yet another sign – each contextually and experientially understood within culture(s). Much like Gadamer’s use of tradition, Eco’s discussion of culture lies within a multiplicity of cultures. To apprehend a sign, one must understand the cultures situating it. Sign functions lend insight into a given time, place and mode of interaction. Eco’s logic of culture differentiates information, signification and communication. Information measures probability void of cultural meaning. Signification considers the codes, patterns and possibilities of meaning within cultural exchange between people. Communication requires embodied labour to produce and interpret meaning within the experience of cultural signs. Eco understands communication and signification as deeply intertwined in the recognition that the meaning of signs emerges in the meeting between and among persons and codes. With reliance upon Peirce, Eco attends to the cultural import of semiosis between and among persons. As one navigates a culture composed of sign functions, one participates in signification and communication emergent in temporal groupings. Eco refers to signification as a theory of codes and communication as a theory of sign production; codes offer semiotic spaces for the labour of sign production. Eco’s theory of sign production revolves around the notion of labour beginning with the mere ‘act of uttering’ (1976: 151). In the labour of sign production, semiotics and interpretation of signs guide the interplay of experience and culture. Eco overviews multiple types of labour, including abduction, overcoding, undercoding, rhetorical code switching and code making (1976: 153–5). Semiosis itself is an act of labour as one sign lends itself to producing another (Eco 1976: 71). Labour is essential to Eco’s understanding of semiotic literacy. Semiotics as a logic of culture augments judgement that recognizes semiosis in action. Semiosis navigates a multiplicity of codes and engages texts according to the perceptual possibilities of interpretation (Eco 1976: 159).

HERMENEUTICS AND SEMIOTICS

135

For Eco, ‘perceptual signification’ is one of ‘semiotics’ limits’ (1976: 167). He traces conceptualizations of perceptual meaning as it applies to semiotics through Locke, Berkeley, Peirce and Husserl. From this position, Eco emphasizes how perceptions form ideas; their meaning evolves in the infinite regression of unlimited semiosis, necessitating interpretation of signs. Eco cites Peirce’s understanding of ideas as signs that emerge from experience and cultural engagement (1976: 165). Eco describes this perspective as Peirce’s ‘philosophicosemiotical position’ (1976: 166); from this insight, Eco situates linguistic and perceptual meaning in the philosophical tradition of phenomenology of perception as being equivalent to the semiotic cultural unit. Eco references Husserl, writing, ‘semiotic meaning is simply the socialized codification of a perceptual experience in which the phenomenological epoché should restore to us in its original form’ (1976: 167). Signs, for Eco, have phenomenological points of origin. No matter the source, experience and cultural framings within a social world influence the interpretation of signs. Eco then shifts to a critique of Peirce’s trichotomy of symbols, icons and indices, which he describes as ‘untenable’, particularly when placed within the context of signification (1976: 178). This chapter primarily focuses on Eco’s critique of iconism. Eco jettisons the notion that iconism relies on similarity. He provides a phenomenological reading that attends to uniqueness. For Eco, the key is discerning and engaging difference, not embracing a modern understanding of commonality. Within semiotic study, ‘isomorphism’ depicts a modern cul-de-sac. Isomorphism is simply unresponsive to difference. In phenomenological terms, Eco is receptive to Husserl’s later work The Crisis of the European Sciences and Transcendental Phenomenology, in which a given event, or in this case sign production, is like an onion yielding multiple layers and possibilities (Husserl [1954] 1970: 11–12). Gadamer, of course, would refer to this reality as a horizon in which there are multiple points of connection, with none providing isomorphic agreement. To resolve what Eco finds as unsatisfactory and naive in the trichotomy, he replaces Peirce’s typology of signs with a classification of modes of sign production (1976: 217). His discussion of sign production emphasizes two essential coordinates: grammatical units in ‘isolated sign functions’ and global ‘textual units which assume the role of largescale (undercoded) sign functions’ (Eco 1976: 217). These macro-units are impossible to separate completely from grammatical units in the communicative labour of sign production. Eco’s modes of sign production emphasize labour in four distinct and related dimensions: 1. Labour physically engaged prior to discerning a given sign 2. Labour that determines how closely signs adhere to established codes 3. Labour that moves beyond assumptions of similarity to attend to signs materially and/or arbitrarily representing referents 4. Labour that reflects cultural knowledge of pre-existing codes or invents new possibilities for interpretive meaning. Modes of sign production are a priori, including the complexity of that which one discerns and the necessity of context. Eco offers four forms of physical labour that produce signs: recognition, ostension, replica and invention. Recognition produces signs via empirical traces motivated by experience and pre-existing codes. Ostention accounts for representations that aim to produce meaning through signs created via perceived similarities between expression and content (Eco 1976: 227). Replica creatively employs arbitrarily assigned signs that

136

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

move between and among pre-established codes and proposed texts. Invention produces signs while simultaneously constructing codes. Invention permits the possibility of code making based on the recognition, ostension and replica of signs situated in codes; invention produces what is ‘not yet’ culturally known, grounded in what is (Eco 1976: 249–50). Eco relies upon Vico to explain that ‘[n]o new culture can ever come into being except against the background of an old one’ (1976: 256). Moving from existing signs, invention produces new sign functions in uncoded contexts based upon interpretive inferences of general rules and specific cases (Eco 1976: 131–2). Invention, ‘the not yet’, requires attentiveness to what is before us and what has preceded us – the invention of new signs emerges only from semiosis of existing signs. For Eco, invention embodies the aesthetic experience and, specifically, the aesthetic text as an act of productive labour. Eco suggests that philosophical aesthetics and the layperson embrace invention. What differentiates the two are rhetorical devices that invite ‘imaginative interplay of metaphors’ (Eco 1976: 262). The aesthetic is a poetic function with possibilities for both ambiguity and directive focus. Eco suggests that an aesthetic text has ambiguity in its expression-plane (transmission of signs and denotative markers) and content-plane (channel of sign transmission with connotative markers), which interrupt conventional pairings of sign functions and require reflection of new possible meaning (1976: 264). A self-focusing aesthetic experience is a moment in which the ambiguity ratio is minimal in relation to temporal clarity. When the aesthetic text is overcoded, in either expression or content, its organization increases and becomes more ambiguous with increasing information. The ambiguity of aesthetic texts leads Eco to ‘an idiolect’ or ‘new coding possibility [. . .] spoken by only one speaker and understood by a very restricted audience’ (1976: 272). For Eco, the idiolect is a space of intuition in which expression and content have not yet fully correlated in a sign function. In the invention of aesthetic texts, one seeks the expression and content of rhetoric, elocution and overcoding. Particularity of cultural knowledge grants overcoding clarity of expression and content, which permits the rhetorical labour of code switching (Eco 1976: 276). When the capacity of code changing is absent, one falls within the semantic category of ideology, refusing to attend to semiosis and transformation. Social and cultural transformation occurs through the rhetorical labour of inventing aesthetic texts that uphold or divert from existing cultural codes. All transformations rely upon existing cultural sign systems generated through the labour of sign production that frames semiotics as a path for both ‘social criticism’ and ‘social practice’ (Eco 1976: 298). In Eco’s concluding chapter, he turns to the semiotic subject engaged in possibilities for semiotic criticism and practices (1976: 314). Eco articulates the embodied and performative nature of semiosis in his ongoing emphasis on interpretation semiotics. The ‘paramount’ term for Eco is semiosis; labour of sign production generates new insights of signification (1976: 316). He does not rely upon a methodological subject but rather a subject embodied in the labour of producing signs within codes that lend themselves to overcoding, undercoding and code switching. This semiotic experience brings forth new insights. Eco understands labour tied to sign production through social practice, social criticism and responsiveness to traces. He addresses the sender-addressee relationship in the semiosic act. The speaking subject is a situated ‘concrete historical, biological, psychic subject’ (Eco 1976: 314). For Eco, the subject engaged in the act of utterance becomes a sign that carries forth his or her situatedness and brings forth ‘conveyed content’ (1976: 315). Eco works carefully to respect the cultural boundaries

HERMENEUTICS AND SEMIOTICS

137

of semiotics. Eco focuses semiotics on signification and the labour of sign production, bypassing psychological ‘“deep” individual origins’ (1976: 315). Eco stresses codes and sign functions tied to situated understanding, a particular ‘way of looking at the world’, that brings new or different cultural insights made available through semiosis (1976: 315). His understanding of semiotics veers from idealism into the practical, the pragmatic, the social and the cultural. Eco shares with Gadamer the practical, pragmatic, social and cultural importance of interpretation. Their work unites hermeneutics and semiotics with differing descriptions of how one idea, one event, one activity shifts and morphs into another. Their work reminds us that interpretation is a revelatory activity, not an action of transcendental objectification. Gadamer’s hermeneutics and Eco’s semiotics articulate the ‘how’ and the ‘why’ of the ‘not yet’. The central metaphors for this ongoing communicative transformation are dialogue and semiosis. The following section engages Gadamer’s dialogue and Eco’s semiosis in relationship to Eco’s final novel Numero Zero, which embodies the possibilities of multiple of interpretive directions, each with a bias and a shifting sense of signification.

DIALOGUE AND SEMIOSIS IN ACTION: NUMERO ZERO This section turns to Eco’s seventh and final novel, Numero Zero, as a literary case study demonstrating the intersections between hermeneutics and semiotics. Eco compared Numero Zero, published in 2015, to his second novel, Foucault’s Pendulum, published in 1988. In an interview with National Public Radio’s (NPR’s) Scott Simon, Eco described both novels as stories about ‘losers’, because they are ‘more interesting than the winners’ and are more likely to resonate with readers since losers outnumber winners (Simon and Eco 2016: 1:30–1:50). Eco’s interest in losers announces the reality of interruption as the heart of dialogue. Differing perspectives interrupt and educate us routinely. Losers unfortunately ignore semiosis and fail to learn from interruptions. Eco’s work dwells within dialogue attentive to interruption and semiosis that transforms one’s perception of existential reality. Rocco Capozzi’s (2015) review of Numero Zero identifies Eco’s semiotic stress on sign, perception, the power of media and interpretation. Capozzi captures Eco’s work within the realm of questioning and uncertainty, the home of the revelatory power of dialogue and semiosis. Numero Zero provides a satirical examination of a tabloid press set in Italy in 1992 and is told from the perspective of Dottor Colonna, a middle-aged failed journalist described as ‘a compulsive loser’ (Eco 2015: 4). Colonna finds himself recruited to work for a newspaper called Domani (which translates to ‘tomorrow’ in Italian); it is an outlet for blackmail, conspiracy theories and speculation meant to threaten the realms of institutional power, such as the government, the church, the business sector and the military. Eco describes these dwellings as the ‘inner sanctum of finance and politics’ capable of disrupting human lives (2015: 17). Commendator Vimercate, owner of a television station, several magazines and a hotel chain, finances Domani. Vimercate functions as a parvenu;15 he wants acceptance and access to the ruling power and institutions. Continually, he finds this social status denied. The parvenu lives in a space void of dialogue and semiosis that has the power of labour to transform an individual life and a sense of place. The parvenu rigidly meets monologic engagement with those in power, denying the possibility of transformation. Expectedly, Vimercate’s audience for Domani is the very group he seeks to join without success – the powerful and socially elite. He approves stories capable of drawing their eye and appreciation.

138

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Vimercate secretly informs the paper’s editor, Simei, about the ruse. Vimercate and Simei understand that the paper will go undistributed, used merely as a tool for power and influence. As Simei leads an editorial team of ‘losers’ commissioned to complete Domani’s issue number zero, he reminds his team that the paper’s audience must ‘draw the conclusions’ about the questions raised in its articles (Eco 2015: 19). This enthymematic characteristic is consistent with Eco’s theory of interpretation semiotics, requiring the communicative labour of sign production. Readers encounter texts without certainty of there being a single correct interpretation. In accordance with the theme of Numero Zero, Eco acknowledged in the NPR interview his hope that readers ‘become more suspicious and attentive’ of news media (Simon and Eco 2016: 3.06). He laments public indifference to tragic news stories. This indifference is the ‘real tragedy’ and the ‘final moral’ of Numero Zero (Simon and Eco 2016: 4:00–4:15). Both dialogue and semiosis work within a realm that necessitates active participation and awareness of the fallacy of absolute truth. Eco calls for reflective and thoughtful interpretive engagement as a primary tool, which protects against ideological false consciousness, the anathema of dialogue and semiosis. Both dialogue and semiosis illuminate revelatory insight, contrary to a subjective monologic imposition upon existence. One of Eco’s characters, Romano Braggadocio, substitutes dialogue and semiosis for imposition of conspiracies. Braggadocio is an anxious and paranoid man, who questions historic events from the Gulf War and the moon landing to Mussolini’s death. Braggadocio suspects that Mussolini escaped to Argentina at the end of the Second World War, perhaps with the protection of the papacy. As he joins one conspiracy with another, Braggadocio contrives stories about bombings in Italy in the 1970s and 1980s, the kidnapping and murder of Aldo Moro, and the death of Pope John Paul I. He suspects that Gladio, the Italian branch of NATO’s stay-behind operations after the Second World War, attempted to destabilize the political environment for a potential resurgence of Mussolini’s power. Through Braggadocio, Eco emphasizes the role of questioning run amok, arrogantly unresponsive to the world around us. As Braggadocio finalizes the series of conspiracies for publication in Domani, Colonna learns that ‘they’ve killed Braggadocio’ (Eco 2015: 169). Although Eco never identifies the ‘they’, Colonna and Simei are confident that Braggadocio was too close to the truth in at least one of his own conspiracies (2015: 171). As they navigate official investigations surrounding Braggadocio’s death, Colonna and Simei work hard to conceal the theories Braggadocio shared with them. Ultimately, Vimercate views Domani as ‘too dangerous’ and shuts the paper down (Eco 2015: 171). For Eco, the possibility to lie characterizes semiotics (1976: 7). Semiosis, like dialogue, does not guarantee truth, just different insights. Both semiosis and dialogue take us to the ‘not yet’, with neither offering assurance about a pure truth. Interpretation is the house of semiotic and dialogic insight. For both Eco and Gadamer, the home of truth dwells forever within interpretive questioning. Eco and Gadamer are attentive to the revelatory: semiosis and dialogue, respectively. Concrete meetings and understanding lead to interpretive insight and richness. Eco and Gadamer do not suggest a universal truth. Their work pursues understanding within temporality. Semiosis and dialogue are essential continuations of the story of the human condition; they illuminate the ‘not yet’ and the revelatory. Eco and Gadamer are respectful of the past, the shifting present and the anticipatory next, yielding a pragmatic display of the significance of semiosis and dialogue in action.

HERMENEUTICS AND SEMIOTICS

139

­NOTES 1 The novel tells the story of William of Baskerville, a detective monk who investigates a series of deaths in a medieval abbey. As he enters the infinite regression of semiosis, one sign leads towards signification of another in the dialogic engagement between and among texts. 2 Palmer (1933–2015) was a renowned American scholar in hermeneutics who dedicated his life to translating and interpreting Gadamer’s corpus. He received two Fulbright Fellowships to study with Gadamer. The first of his six books was Hermeneutics: Interpretation Theory in Schleiermacher, Dilthey, Heidegger, and Gadamer, published in 1969. 3 Grondin (1955–) was Gadamer’s pupil, specializing in the thought of Kant, Gadamer and Heidegger. Translations of his work on hermeneutics, phenomenology and metaphysics appear in twelve languages. His well-known books include Introduction to Philosophical Hermeneutics (1991), Sources of Hermeneutics (1995) and Hans-Georg Gadamer: A Biography (2003). 4 Ferraris (1956–) was a student at the University of Turin, taught in the tradition of Luigi Pareyson and Gianni Vattimo (under whom Ferraris studied directly). Eco also attended the University of Turin, completing his graduate work under the direction of Pareyson, who had significant influence on Eco’s cultural aesthetics as a precursor to his semiotic project. From this tradition, Ferraris examines hermeneutics and is particularly attentive to the interplay of Nietzsche and Heidegger, as well as to the later work of Derrida, Foucault and Deleuze. Additionally, Ferraris studied under Gadamer at Heidelberg University during the 1980s. Ferraris also contributed to developing ‘New Realism’ as an orientation shared by continental and analytic philosophers alike. 5 Susan Petrilli (1954–) is the seventh Thomas A. Sebeok Fellow of the Semiotic Society of America. Augusto Ponzio (1942–) has been instrumental in introducing the works of Bakhtin, Levinas, Marx, Rossi-Landi and Sebeok to an Italian audience. Their work includes over three dozen articles and four books exploring the intersection of ethics and semiotics. They rely upon dialogic scholars including Bahktin and Levinas as they frame semioethics as a pervasive and unceasing call of responsibility to and for the other, grounded in the sign. 6 Petrilli’s well-known contributions on Lady Victoria Welby include Signifying and Understanding: Reading the Works of Victoria Welby and the Signific Movement and book Victoria Welby and the Science of Sciences: Significs, Semiotics, Philosophy of Language. ­7 Deduction relies upon generalizable principles and theories to understand specific cases, whereas induction moves from observation of special cases to draw generalizable principles. Peirce introduced abduction as a form of logic that relies upon deductive and inductive reasoning to creatively respond to and anticipate insights. 8 The icon, index and symbol are three types of signs introduced by Peirce. The icon emphasizes resemblance and likeness to the object. The index demonstrates a relation, a trace of the object. The symbol is a conventional agreement of meaning oftentimes based upon culture. 9 Peirce identifies agapism, anancism and tychism as three modes of evolution in the cosmos. In ‘Evolutionary Love’, he describes agapism as ‘evolution by creative love’, anancism as ‘evolution by mechanical necessity’ and tychism as ‘evolution by fortuitous variation’ (Peirce 1893: 188). 10 Eco’s notion of the limits of interpretation is central to his semiotic perspective and is the title of a collection of essays published in 1990. 11 Petrilli and Ponzio are critical of the ‘weakness’ of Eco’s placement of ideology within his theory of sign production and emphasize his failure to define ideology (2005: 321–2).

140

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

12 John Deely (1942–2017) was an American philosopher and semiotician. He was a research fellow under Mortimer J. Adler at the Institute for Philosophical Research during the 1960s. While studying the work of Jacques Maritain and John Poinsot, he met Sebeok. Deely helped Sebeok found the Semiotic Society of America in 1975 and 1976. Important works of Deely’s include Introducing Semiotic: Its History and Doctrine (1982), Basics of Semiotics (1990), Four Ages of Understanding (2001) and Semiotic Animal (2010). 13 For Peirce, the representamen is the form of the sign and its potential to carry meaning as it stands for something else. The object is the socially constructed and agreed-upon entity of what a sign represents. Peirce’s notion of the interpretant is the action and sense produced by the sign. 14 Luigi Pareyson (1918–91) completed his graduate work at the University of Turin in 1939, where he later taught Eco and Vattimo. Pareyson’s work aims to understand interpretation of aesthetic texts. 15 A parvenu participates in a wicked social game that includes (1) an invitation, (2) a rejection, (3) another invitation, (4) another rejection and (5) a continuation of this pattern. The parvenu witnesses impossibility in attempts to change class status (Arendt [1957] 2000).

­REFERENCES Arendt, H. ([1957] 2000), Rahel Varnhagen: The Life of a Jewess, Baltimore: Johns Hopkins University Press. Arnett, R. C. (2017), Levinas’s Rhetorical Demand: The Unending Obligation of Communication Ethics, Carbondale: Southern Illinois University Press. Arnett, R. C. (2018), ‘Gadamer: Ethics and the Dialogic Character of Play’, Language and Semiotic Studies, 4 (2): 19–35. Bain, P. (2006), The Primacy of Semiosis: An Ontology of Relations, Toronto: University of Toronto Press. Benson, T. W. (1977), ‘A Theory of Semiotics’, Philosophy & Rhetoric, 10 (3): 214–16. Buber, M. ([1947] 2002), Between Man and Man, trans. R. Gregor-Smith, New York: Routledge. Capozzi, R. (2015), ‘Umberto Eco’s Numero Zero: A J’Accuse of Forgeries, Lies, Conspiracies and Bitter Truths’, Italica, 92 (1): 222–40. Dance, F. E. X. (1977), ‘A Theory of Semiotics’, Quarterly Journal of Speech, 63 (3): 333–5. Deely, J. (1982), Introducing Semiotic: Its History and Doctrine, 1st edn, Bloomington: Indiana University Press. Deely, J. (1990), Basics of Semiotics, 1st edn, Bloomington: Indiana University Press. Deely, J. (1997), ‘Looking Back on A Theory of Semiotics: One Small Step for Philosophy, One Giant Leap for the Doctrine of Signs’, in R. Capozzi (ed.), Reading Eco: An Anthology, 82–110, Bloomington: Indiana University Press. Deely, J. (2001), Four Ages of Understanding: The First Postmodern Survey of Philosophy from Ancient Times to the Turn of the Twenty-first Century, Toronto: University of Toronto Press. Deely, J. (2010), Semiotic Animal: A Postmodern Definition of ‘Human Being’ Transcending Patriarchy and Feminism, South Bend: St Augustine’s Press. Dockhorn, K. (1980), ‘Hans-Georg Gadamer’s “Truth and Method”’, Philosophy & Rhetoric, 13 (3): 160–80. Eco, U. (1976), A Theory of Semiotics, Bloomington: Indiana University Press.

HERMENEUTICS AND SEMIOTICS

141

Eco, U. (1979), The Role of the Reader: Explorations in the Semiotics of Texts, Bloomington: Indiana University Press. Eco, U. (1984), Semiotics and the Philosophy of Language, Bloomington: Indiana University Press. Eco, U. ([1990] 1994), The Limits of Interpretation, Bloomington: Indiana University Press. Eco, U. (2015), Numero Zero: A Novel, trans. R. Dixon, New York: Houghton Mifflin Harcourt. Ferraris, M. ([1988] 1996), History of Hermeneutics, trans. L. Somigli, Atlantic Highlands: Humanities Press International. Gadamer, H. ([1975] 2013), Truth and Method, trans. J. Weinsheimer and D. G. Marshall, New York: Bloomsbury. Gadamer, H. (1976), Philosophical Hermeneutics, trans. and ed. D. E. Lange, Berkeley: University of California Press. Grondin, J. ([1991] 1994), Introduction to Philosophical Hermeneutics, trans. J. Weinsheimer, New Haven: Yale University Press. Grondin, J. (1995), Sources of Hermeneutics, Albany: SUNY Press. Grondin, J. (2003), Hans-Georg Gadamer: A Biography, Princeton: Yale University Press. ­Husserl, E. ([1954] 1970), The Crisis of the European Sciences and Transcendental Phenomenology: An Introduction to Phenomenological Philosophy, trans. D. Carr, Evanston: Northwestern University Press. Lanigan, R. (1978), ‘Contemporary Philosophy of Communication’, The Quarterly Journal of Speech, 64: 335–60. Mueller-Vollmer, K. ([1985] 2006), ‘The Historicity of Understanding: Hans-Georg Gadamer’, in K. Mueller-Vollmer (ed.), The Hermeneutics Reader: Texts of the German Tradition from the Enlightenment to the Present, 256–92, New York: Continuum. Palmer, R. E. (1969), Hermeneutics: Interpretation Theory in Schleiermacher, Dilthey, Heidegger, and Gadamer, Evanston: Northwestern University Press. Palmer, R. E. (1987), ‘Gadamer’s Hermeneutics: A Reading of “Truth and Method” by Joel Weinsheimer’, Philosophy & Rhetoric, 20 (2): 135–8. Peirce, C. S. (1893), ‘Evolutionary Love’, in P. Carus (ed.), The Monist: A Quarterly Collection of Essays, 3, 176–200, Chicago: Open Court Publishing Co. Petrilli, S. (1997), ‘Toward Interpretation Semiotics’, in R. Capozzi (ed.), Reading Eco: An Anthology, 111–36, Bloomington: Indiana University Press. Petrilli, S. (2009), Signifying and Understanding: Reading the Works of Victoria Welby and the Signific Movement, Berlin: De Gruyter Mouton. Petrilli, S. (2015), Victoria Welby and the Science of Sciences: Significs, Semiotics, Philosophy of Language, New York: Transaction Publishers. Petrilli, S. and A. Ponzio (2005), Semiotics Unbounded: Interpretive Routes through the Open Network of Signs, Toronto: University of Toronto Press. Schutz, A. (1967), The Phenomenology of the Social World, Evanston: Northwestern University Press. Simon, S. and U. Eco (2016), ‘“Numero Zero” Reprises Umberto Eco’s Fascination with “Losers”’, NPR, 20 February, https://www.npr.org/2016/02/20/467468433/numero-zeroreprises-umberto-ecos-fascination-with-losers Warnke, G. (1987), Gadamer: Hermeneutics, Tradition and Reason, Stanford: Stanford University Press.

142

­C HAPTER SEVEN

Translation Studies and Semiotics EVANGELOS KOURDIS AND RITVA HARTAMA-HEINONEN

INTRODUCTION Despite the fact that Roman Jakobson linked translation practice to semiotics as early as in 1959, and Gideon Toury introduced a novel approach to translation – the semiotics of translation – in 1980, and regardless of the interdisciplinarity that characterizes these research fields interested in cultural communication, the breadth and the depth of the semiotic nature of translating have not thus far been extensively studied. Furthermore, although distinguished semioticians like Charles Sanders Peirce ([1866–1913] 1931–58), Victoria Lady Welby (1903), Juri Lotman (1990, 2005, 2009), Umberto Eco (1976, 2001, 2004) and Paolo Fabbri (2008) have carried out research on the concepts of translation and interpretation, thereby emphasizing the translational and interpretative nature of the semiosic processes involved in all understanding, their contributions have not yet significantly influenced the field of translation studies. Semiotics became misinterpreted and was pushed aside, seen as yet another complex philosophical theory – which semiotics might be for many, though it definitely reaches far beyond. Translation studies has recently adopted a more positive attitude towards semiotics, especially when it was widely realized that multimodal cultural communication with its ever-expanding digital spaces creates new data on the translatability of texts, which are not necessarily verbal, but polysemiotic, multimodal or intermodal as well. During this process, it has become more and more apparent that there is neither one monolithic semiotics nor one conception of translation; instead, there are many complementary approaches and schools that can be gathered under the respective conceptual umbrellas of both semiotics and translation. This makes any effort to combine translation practice, sign theory and translation theory an intriguing vantage point and a task worth attempting to complete, and yet, finally, a complicated pursuit. In this chapter, we will present mainly theoretical approaches to the research area and field of knowledge called translation semiotics, and examine two key perspectives: first, translation scholars’ positions on how semiotics can contribute to translation studies and, second, semioticians’ positions on translation and translatability, which were, and are still in part, influenced by three dominant semiotic schools: 1. Translation from the perspective of Peircean semiotics, which is the approach that, with its fundamental concepts such as ‘semiosis’ and ‘interpretant’, created the basis for a general semiotic theory of translation and interpretation. This theory has

144

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

gradually been constructed further, by testing and proving the applicability of the cornerstones of Peirce’s semeiotic. 2. Translation from the perspective of the Paris School of Semiotics which provided semiotic theory with important concepts, such as intertextuality (the original idea though from Mikhail Bakhtin) and transposition (a designation also employed in Jakobson 1959), which were passed on to translation, mainly through literary theory and literary translation. Although this school has not focused on translation, it endorsed the concept of the translatability of all semiotic systems. 3. Translation from the perspective of the Moscow-Tartu School of Semiotics (nowadays also known as the Tartu School of Semiotics) which connected translation to culture via the conception of semiotization, and adopted the stance that culture is an inherent mechanism of translation, and vice versa. Finally, we will examine some subfields where a semiotic theory can make a significant contribution. These include phenomena, such as intericonicity, adaptation and transduction, that emerge within the discipline of translation studies, and are considered to have a semiotic core. The purpose of this chapter is to show that these two interdisciplinary areas, translation studies and semiotics, share to some extent a common ground, the application of which, together with the recognition that there may also be shared interests, can primarily deepen our idea of translating, and highlight the multi-dimensionality of translational practices in cultural communication and beyond – in all communication.

SEMIOTICS FROM THE TRANSLATION SCHOLARS’ PERSPECTIVE We begin our overview of how translation researchers have approached semiotic vantage points by placing the above-mentioned proposal by Jakobson at the centre of attention. His proposal is not at the centre within either semiotics or translation studies today, but it will function here as a link between the predecessors and the interpreters with their echoing or non-echoing stances. In this hub-like function, Jakobson’s proposal may explain the complex yet evolving relation between those who approach translations and translating from a semiotic angle and those whose approach is translation-theoretical (see, e.g., Hartama-Heinonen 2015, 2008: 29–80). The following may also explain the growing awareness of a shared ground and shared interests, or interdisciplinary unification that Jakobson appears to have pushed forward within translation studies and semiotics. Jakobson, the linguist and semiotician, introduced in 1959 his pioneering division of translation into the intralingual, interlingual and intersemiotic types, describing how verbal signs can be interpreted within verbal or non-verbal sign systems: within the same verbal sign system; between two different verbal sign systems; between a verbal and a non-verbal one. The three-part typology is language-centred (with the choice of the verbal as the source system), yet sign-theoretical (signs, sign systems), featuring simultaneously a linguistic and a semiotic line of approach and respecting both of them. This proposal constitutes a landmark the importance of which has been evaluated and re-evaluated more prominently since the 1980s. When not directly ignored, his proposal has been continuously referred to and revisited: sometimes accepted as such, sometimes criticized and rejected, and sometimes reread, updated and modified to suit the changing

TRANSLATION STUDIES AND SEMIOTICS

145

spatiotemporal constraints, contexts and purposes. This applies even today, with some exceptions: see, for example, Sütiste (2008) on Jakobson in linguistic, semiotic and translation-theoretical reference works; note also Toury’s (1986) rethink of the typology, resulting in the dichotomy of intrasemiotic and intersemiotic types, where the intrasemiotic forms consist of intrasystemic (such as translating within a language) or intersystemic (between two languages) translation; or Gottlieb who advocates a polysemiotic approach, attesting that Jakobson’s contribution can, today, be passed without comment (2017, cf. also [2000] 2004). Jakobson’s article is both innovative and seminal, and heralds a semiotic turn in translation research, a circumstance that has not been fully comprehended despite the decades of reassessment. The reason is obvious: interlingual translation, Jakobson’s translation proper, has been and is still for many translation scholars the prototype of translation and the only reasonable focus of interest; their translation-theoretical studies continue to strengthen the interlingual translation type. To suggest a semiotic turn might be ambitious, the acceptance being far from self-evident. In The Turns of Translation Studies: New Paradigms or Shifting Viewpoints? Mary Snell-Hornby mentions Jakobson for introducing the Peirce-inspired typology, and she also observes the novelty and applicability of Jakobson’s proposal when studying modern translation genres such as intralingual subtitling or intersemiotic multimodal translation (2006: 21). For SnellHornby, this is apparently all that can be said about Jakobsonian or, for that matter, other semiotic influences; they have hardly launched a semiotic turn or paradigm. Translation studies appears to be somewhat selective in its approach to Jakobson. While interlingual translation between two natural languages is acknowledged, and intersemiotic translation from verbal signs to non-verbal ones (and vice versa), such as when a novel is filmed, has found more adherence, intralingual translation has not until recently received due attention or use as a term. For Jakobson, the intralingual type comprises choosing synonyms and paraphrases within the same language (1959: 233). But in a modernized version, intralingual translation can manifest itself together with the intermodal, such as in speech-to-text interpreting or subtitling for the deaf and hard of hearing, when the signs in one mode are rendered with signs in another; spoken and aural with written and visual. In this intralingual-intermodal-intersemiosic type, the transfer mechanism inherent in translation is harnessed to serve accessibility and barrier-free communication – observe also adaptations and translations into easy language or texts in such varieties as inclusion-enhancing clear administrative language. According to Karen Zethsen (2009), we ought to study more extensively the similarities and differences between interlingual and intralingual translation to refine our idea of translation, even towards a more open-ended definition. There are also critical voices. Brian Mossop (2016) maintains that despite the shared features, variance-oriented intralingual activities – rewording such as plain writing, paraphrasing or editing – should not be included in the scope of translation studies: they are so different from interlingual translating conceived as equivalencing; invariance is a criterion that neither intersemiotic translation might meet (Mossop 2019). Jakobson’s translation typology, however, requires us to broaden our notion of translation beyond the threshold of invariance, and do it in accordance with Peirce’s fundamental idea which originally inspired Jakobson: that signs become signs through translating (1903: CP 5.594). We may gain some insight into the direct and indirect role semiotics has recently played when we examine what semiotic(s) appears to be for translation scholars, or what sort of context may prompt them to apply semiotics, whatever it may mean in individual studies.

146

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A search with the entry semiotic in the Translation Studies Bibliography (Gambier and van Doorslaer 2004–) results in 100 abstracts from 2010 to 2019. We base the following outline on some fifty explicit hits in these abstracts, but our account is definitely not exhaustive, since the bibliography material presents no balanced cross-section. Furthermore, contributions included in the bibliography, qualifying as translation studies, can represent other disciplines. In any case, this account presents some signposts to areas in which semiotic ideas gradually establish foothold. First, semiotic(s) most often occurs in references to semiotic resources, that is, together with mention of Halliday, multimodality, sociosemiotics, meaning-making, and semiotic modes and codes. Second, there are some genres of specialized translation that apparently have a close connection to what can be called the semiotic nature of the phenomenon in question. These include audiovisual translation (subtitling, surtitling, audiodescription or adaptation). Translation of comics, picture books and advertisements also belongs to this group as well as song translation and theatre translation, genres of intersemiotic nature. Third, some abstracts explicitly refer to a certain semiotic branch, a particular semiotic school or certain semioticians. Fourth, we encounter some mentions of linguistic as opposed to semiotic (culture, transfer) – so language is not always considered as a semiotic system. In an increasingly cross-semiotic world, intersemiotic translation has gained wide currency. For many, this type constitutes the noteworthy novelty in Jakobson’s contribution and is actually its eye-opening core finding: translation occurs between verbal systems, but also between verbal and non-verbal sign systems and, in addition, can proceed from a non-verbal to a non-verbal one as well. The step from the dominantly interlingual to the intersemiotic and multimodal facet in translational communication means a definite step towards semiotic thinking. The world of professional translating – language industry, language service providers and freelance or in-house translators – is facing new challenges, and so is Translation Studies. We see changes in its research objects: products, processes, actors and contexts involved. These include a gradual shift from studying translation products towards studying translation processes reaching from the cognitive aspects of human translation to the development and the growing use of translation technologies. The changes also concern new emphases and focuses (Translator Studies instead of Translation Studies; user-centredness) and non-profit endeavours (non-professional translation and interpreting; crowdsourcing). Despite the developments in translation practice, semiotics will assumedly preserve its place in the research contexts of today, and even Jakobson’s typology will exist as a basic yet implicit framework for what translation scholars focus on.

THE TRANSLATION SEMIOTICS RESEARCH FIELD To explore the division of labour between fields and disciplines is not easy. This applies to Translation Studies in relation to Semiotics (of Translation) as well. Semiotics, the general sign theory, also characterized as the reuniting ‘master science’ (Ketner 1998: 306) and the language connecting scholars across disciplines constitutes the foundation and umbrella for all disciplines (in practice often integrated into disciplines). Translation Studies in turn positions itself as an interdiscipline, explicitly since 1994 (see Snell-Hornby [1988] 1995: 133, 2006: 70–2). Its current level of interdisciplinarity is reflected in its methodology which as expected is not its own, but borrowed from nearby disciplines, such as semiotics, modified and refined, thus both adopted and adapted.

TRANSLATION STUDIES AND SEMIOTICS

147

To illustrate this point, in the article ‘Semiotic Modalities in Translation Causality’ (2002), Andrew Chesterman, one of the leading Translation Studies scholars, draws on A. J. Greimas’s idea of modalities and analyses the work of a translator as reflections of these modalities, relating them to causality and translation effects. This Greimasinspired approach to translatorial activities contributes, no doubt, to Translation Studies, but does it simultaneously, de facto, contribute to Semiotics? Or should it be classified more cautiously among semiotic translation theory, semiotic translation studies, semiotic approaches to translation (studies), semiotic translation research, semiotics of translation, semiotics and translation, semiotranslation, or translation(al) semiotics – designations denoting research that combines semiotic ideas and translation? Instead of discussing issues like what can count as semiotics of translation (Semiotics of Translation), or whether this discussion is relevant at all, we will concentrate on how translational matters are approached from two disciplinary angles and emphases. In the 1980s–90s, when Translation Studies took its first steps towards a success, there were, according to Snell-Hornby, several factors that increased its viability: enthusiastic scholars with merging interests, disciples (scholars-to-come), conferences, the European translation scholars’ society, new journals, expanding translator education, a series of ‘turns’ and research innovations that later on proved to be landmarks, such as skopos theory ([1988] 1995: 133–5, 2006: 69–70). Translation semiotics cannot present similar achievements, if this branch is supposed to be to some extent comparable to an already-established discipline. Ritva HartamaHeinonen (2008: 70) describes the status of this subfield as vague, when compared to such acknowledged semiotic branches as semiotics of culture, musical semiotics or semiotics of theatre. There are scattered scholars and translation-semiotic agendas and pursuits that apparently do not aim at unified views; only one international conference series dedicated expressly to semiotics and translation, SemTra, with thirteen conferences thus far (Imatra, Finland, and Kaunas, Lithuania, 2000–17); no own society nor a journal, instead special issues of journals (e.g. Athanor, Parallèles, Punctum, Semiotica, Sign Systems Studies, Signata, Versus), and what is particularly noteworthy, visibility-promoting state-of-the-art overviews, as our chapter, in semiotic and translation-theoretical encyclopaedias, reference works and handbooks (from Toury 1986 to Kourdis 2015 and Gottlieb 2017); mostly separate university-level courses; and finally, one particular innovation, semiotranslation. To sum up, the infrastructure of this semiotic branch is despite the efforts still under construction. Eero Tarasti (2000: vii, 3–4) characterizes the development of general semiotics with three generations: the first-generation semioticians represent classical semiotics from Peirce, Ferdinand de Saussure, Claude Lévi-Strauss and A. J. Greimas to Thomas A. Sebeok; the second generation consists of Michel Foucault, Roland Barthes, Julia Kristeva, Eco and Jacques Derrida, among others; the third is neo-semiotics, actually Tarasti’s existential semiotics which aims at renewing semiotics to serve our millennium and its new challenges. When outlining the following short description of the advent and evolution of translation semiotics, we have combined Tarasti’s proposal of generations with the ideas and designations employed for instance, by Snell-Hornby (2006: 5, 20, 30, 47; based on Radnitzky [1968] 1970). The first generation, Peirce and Welby – translation semioticians avant la lettre – are the precursors who in their writings (classics) laid the foundations for the generations to come. The second generation are the pioneers: Jakobson, Lotman, Anton Popovič1 and Toury, semioticians who reflected the relation between translation and semiotics, and prepared the way for emerging translation-semiotic thinking. The third generation consists of Eco, Dinda L. Gorlée, Susan Petrilli and Peeter Torop, the acknowledged

148

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

first masters who have during years and decades devotedly constructed their semiotic approaches to translation and become forerunners in and through their programmatic, leading-edge research. We might also categorize Pirjo Kukkonen and Douglas Robinson among the third-generation scholars, as widely published and consequent in their individual contributions and applications of semiotic insights. The fourth generation are disciples and potential masters-to-be: Hartama-Heinonen, Evangelos Kourdis, Bruno Osimo, Ubaldo Stecconi and Elin Sütiste might qualify as candidates. Their duty is to put the masters’ work to the test, and challenge the tradition. In this partly translation-studies-inspired reading of translation semiotics, we have observed slow semiotic progress and movement. This picture must be completed and anchored further. Translation semiotics emerges from the mainstream schools (Prague, Paris, Moscow-Tartu) as well as from the theoretical and empirical (applied) methods and approaches of its mother discipline. Translation semioticians have drawn on philosophical (Peirce, Morris, Bakhtin), linguistic (Hjelmslev, Jakobson), cultural (Lotman), structuralist (Saussure, Greimas) and poststructuralist (Barthes, Kristeva) insights. Impacts from several main branches and attitudes are also manifest: semiotics of culture, of language, of literature and of theatre, legal, musical and visual semiotics, bio- and zoosemiotics, and semioethics, to mention a few. Synergy can be created when studying translation from several semiotic perspectives (for instance, from a biosemiotic angle and complement this approach with a Peircean one, as in Marais and Kull 2016). To understand the mentioned still vague situation and the scholars’ preferences, some observations, definitely not generalizations, are appropriate. Very few translation scholars appear to have a permanent interest in exploring translation with the explicit help of semiotics proper. In addition, very few semioticians have a permanent interest in translations and translating, while keen translation semioticians are not necessarily interested in what Translation Studies has achieved, nor always committed to a certain semiotics but may act in their pursuit of knowledge eclectically, also using mixed methods. Furthermore, even though English is an academic lingua franca, you ought to know major languages like Chinese, or minor languages like Finnish and Swedish (to read, e.g., Kukkonen 2014), if you aim at getting an up-to-date, overall view of the translation-semiotic field of study. To succeed in its mission, translation semiotics has to be competitive and bring added value by addressing questions that no other field has on its agenda, and to justify its further existence, create new theoretical and empirical knowledge and, thereby, enhance our understanding of translational phenomena. The strength of semiotics does not lie so much in the versatility of its objects of study than in its flexibility: like a chameleon, semiotics adjusts itself to its surroundings and, while recontextualizing itself, does not lose its sign-theoretical characteristics. What this means in a translation-semiotic context and inquiry (and probably for semiotics applied) becomes evident below through the presented achievements.

­TRANSLATION FROM THE PERSPECTIVE OF PEIRCEAN SEMIOTICS Semeiotic, Peirce’s uncompleted semiotic project, has been characterized as interpretive/ interpretative, processual or pragmatic/pragmatistic semiotics. These are designations that are employed to highlight possible approaches to and facets of Peirce’s non-structuralist

TRANSLATION STUDIES AND SEMIOTICS

149

semiotic thought, according to which the meaning and the nature of a sign are revealed through successive interpretative (translative) efforts, in a practically endless sign process called semiosis, resulting in the growth of knowledge through pragmatic consequences and effects, interpretants. Peirce’s general theory of signs is not so much about signs (icons, indices and symbols) or sign systems as it is about the ubiquitous two-sided semiosis that embraces both meaningful dynamic sign action and sign interpretation, that is, a sign functioning as a sign and determining an interpretant of itself, or more precisely, creating a triadic interrelation between the sign, its object (which empowers the sign) and interpretant. This interpretant, a translation of its preceding sign, is a sign as well, and can, through its own action, determine a new interpretant, or even a chain of interpretants. This is how the meaning of the sign (its object) is gradually unfolded in a semiotic and semiosic action: the sign is translated into, renders itself as, another sign (1903: CP 5.594), an interpretant which is an equivalent or perhaps a more enhanced version (c.1897: CP 2.228), and a future-oriented interpretation, translation, inference, effect. This is why any semiosis is inherently translational semiosis, with interpretants being translations of antecedent signs. Since translating goes from sign to sign and occurs between signs, the process is intrinsically intersemiotic. From the viewpoint of semiosis, the world manifests itself as a universe of signs and sign translation. When we add to this the notion that the involved signs can be verbal or non-verbal (verbal language being only one possible sign system), we have produced a simplified Peircean account of translation(al signification and communication). Peircean semiotics can namely be approached as an unwritten general theory of translation as sign translation (including sign interpretation and sign production); often cryptic and fragmentary yet constituting the foundation and framework for Peircean translation semioticians to build on, theorize about, apply, modernize and elaborate. Translation appears to amount to semiosis and semiosis to translation (Gorlée 1994: 226–7), with translation being realized in and through the universal mechanism of semiosis and vice versa. Furthermore, translation emerges wherever we encounter signs and semiosic processes, that is, in the semiobiosphere (Ponzio 1999–2000: 5–6). For many, these starting-points are too vague, obscure and open-ended. But it is also evident why Peirce’s semiotic thought has also inspired scholars to explore translation and interpreting semiotically. Dinda L. Gorlée, Douglas Robinson and Janice Deledalle-Rhodes were among the first in the 1980s who saw how Peirce’s sign theory can enhance translation research. Even though Gorlée, the semiotician, translation scholar and Peirce scholar, has since 1986 published exceptionally widely on translation semiotics, her Semiotics and the Problem of Translation (1994) is even today the most influential single introduction to Peircean translation studies, not least because she was the first to apply Peirce’s thought to translation extensively. Gorlée’s proposal for a Peirce-based approach to translation as practice and theory is realized in her interpretation of translational semiosis, semio(-) translation. Gorlée has gradually integrated translation-related interpretations of Peirce’s central concepts, ideas and often triadic distinctions into semiotranslation theory. These include semiosis, types of signs and interpretants, the role of the object, signhood, the ontological categories, the modes of being, truth in semiosis, abduction, induction, deduction in translatorial reasoning, etc. Semiotranslational research is highly theoretical, but Gorlée has since the 2000s also developed applied semiotranslation, such as Peircebased analyses of vocal translation and intersemioses and partly Peirce-inspired readings

150

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

of Wittgenstein (see Gorlée 2012 and the references therein). Semiotranslation is no longer exclusively Peirce-centred, but draws on other semioticians as well. Semiotranslation is Gorlée’s coinage, but the term has been used without commitment to Gorléean semiotranslation, for instance, by Goethals et  al. (1999–2000) when they demonstrated the practical applicability of Peircean semiotics to multimedia, biblical texts and comics, the black box, and reasoning. Of these researchers, Robinson has published widely on translation and semiotics, also on Peirce, but selectively. His Becoming a Translator ([1997] 2003) is a book with applications of Peirce’s three types of reasoning. Robinson has also elaborated other Peircean concepts, and re-interpreted semiotranslational views. Stecconi (e.g. 2004, 2007 and the references therein) advocates semiotics and Peirce’s sign theory, since the latter in its generality reassesses many modern ideas of translation. Stecconi’s (2004, 2007) contribution to Peircean translation semiotics is what he calls the Foundation of translation, characterizing the conditions to translation semiosis with the triplet of similarity (First), difference (Second) and mediation (Third) (note also the triplet of conditions, events, norms/habits). For him, translation represents a specific type of semiosis. Neither does Eco accept the above proposal that interpretation is translation (and semiosis), since they are not coextensive (2001: 67–71; cf. also Vassallo 2015): Jakobson drew on Peirce when associating interpretation with translation, but did not mean it literally, only that it is worth approaching interpretation and meaning as translations. As we can see, Peirce can be read in several ways, and his followers do not necessarily agree on the interpretations. Hartama-Heinonen (e.g. 2008, 2015) has explored the status of translation semiotics within both semiotics and Translation Studies, and particularly Peircean semiotranslation. Her main focus is though on the art of inferring from signs in translation and inquiry, or Peirce-based abductive translation (studies), probing such claims as ‘reasoning is (sign) interpretation is translation is semiosis’ and ‘translation is translation and beyond’ (2008: 61, 179, 299). We will also mention the Peircean contributions of Susan Petrilli and Cécile Cosculluela, and add that there are only few, if any, ‘full-time’ Peircean translation semioticians, and many of those who are active may draw on Peirce eclectically and sporadically. To provide a wider picture of Peircean translation semiotics, we end this section with three recent studies, and ask, perhaps provocatively, whether they point to future tendencies. João Queiroz and Daniella Aquiar (2015) discuss semiosis and iconic processes, when developing their model of intersemiotic translation, which they illustrate with an example of a transcreation from text to dance. Would it be time to give up the ambitious and extensive theoretical pursuits and begin to scrutinize more restricted topics empirically? Adopting Gorlée’s term semiotranslation and also applying Peirce, Kobus Marais and Kalevi Kull (2016) call for an interdisciplinary dialogue between biosemiotics and translation studies, since that can lead to a conceptual renewal and promote mutual understanding of what constitutes translation. Could biosemioticians succeed in what Peircean translation semioticians have not, in convincing that translating definitely reaches beyond language, to the whole semiobiosphere? Sergio Torres-Martínez (e.g. 2018, 2020) has developed his idea of semiosic translation, a multi-disciplinary approach that fuses translation, Peircean semiotics, abduction, sign and signification, and certain Wittgensteinian ideas. He appears to resume topics already studied, but could it be so that the future of the field emerges from the abductive power and synergy of similar symbioses, parallels and combinations?

TRANSLATION STUDIES AND SEMIOTICS

151

TRANSLATION FROM THE PERSPECTIVE OF THE PARIS SCHOOL OF SEMIOTICS This section presents the positions of semioticians of the Paris School of Semiotics (also known as Semio-linguistic Research Group) concerning semiotics and translation in an effort to reveal their approach on translation in a period when translation was not an autonomous field of research and it was studied under a linguistic prism. This broadening was done thanks to the elaboration of concepts like signification, meaning, intertextuality, transposition, synergy, fusion, syncretism, redundancy, anchoring function, significant concepts provided by scholars of the Paris School of Semiotics. Even though this semiotic school had no direct links to translation studies, we believe that it provided concepts which were adopted by translation studies scholars and contributed to the autonomy of translation studies, thus escaping the predominance of the semiotic system of language, although it continues to be considered by semioticians as the primary semiotic system for the study of culture. From the 1980s, some semioticians began to study translation more systematically through semiotic theories. This current, known today as Semiotics of translation, contributed, in our opinion, significantly to the revival of translation studies, and in this effort the contribution of the French school is evident. As Joseph Courtés mentions,2 what interests the Paris School of Semiotics is ‘rather knowing what is happening “under the signs” or “between the signs”, what which is the basis of their mutual relationships’ (2001: 123). The translatability of signs also belongs to this context. The subject’s activities and attitudes towards the object, as well as towards its modalities (see Tarasti 2017: 44), gave to the notion of translatability of semiotic systems a central position in the discourse of the Paris school. It should be noted that, according to the Paris school of Semiotics, ‘translatability stands as one of the fundamental properties of semiotic systems’ (Greimas and Courtés 1982: 351), and, they continue, ‘translation, comes in between the existential judgment “there is meaning” and the possibility to say something about it. “To speak of meaning” is both to translate and to produce signification’ (1982: 351). We adopt this position considering that the first attempts at a certain widening of the notion of translation were made by semioticians who are classified within the Paris School of Semiotics or are considered close to it. Jakobson (1959), a leading figure of the Paris School, proposed a typology of translation and with the introduction of the concept of intersemiotic translation or transmutation, he clearly linked, for the first time, translation to semiotics. The concept of intersemiosis as the act of intersemiotic translation was further promoted by Julia Kristeva (1969), who analysed not only the notion of intertextuality, an important concept in the study of intersemiosis, but also the notion of transposition (Kristeva 1974), a term also proposed by Jakobson (1959: 238). In our view, Jakobson is ‘scientifically cautious’. He takes a careful step but avoids insisting on his analysis on intersemiotic translation. His scholarly background seems to confirm this observation. Jakobson is no longer concerned with the phenomenon of intersemiosis although he has included polysemiotic/syncretic systems, as well as non-linguistic systems, in the translation process. This appreciation of non-linguistic signs is in keeping with the need to include them in the study of daily communication. For Greimas, the founder of the Paris School of Semiotics, a translation that is not properly linguistic is also possible. As Greimas remarks, ‘any signifying ensemble of a different nature from that of the natural language can be translated, with more or

152

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

less exactitude, into any natural language: for instance, painting and its translation by pictorial criticism’ (1983 [1966]: 11). However, the fact that for Greimas language always remains the primary semiotic system the participation of which in the translation process is considered to be primary prevents him from deepening his above-mentioned position. It should be noted that all these proposals come from linguists who very early noticed the importance of non-verbal communication, at a time when mass communication was not as widespread as today. It is also interesting that the legitimacy of the exclusive use of linguistics for the study of translation is refuted by another Paris school, that of the Paris School of Translation. As Marianne Lederer mentions ‘whatever the merits of linguistics [. . .] it cannot claim to explain the complexity of translation’ (1994: 92). We will see later that this complexity included a series of semiotic facts which had already influenced linguistic and literary studies before studying them for translation. The concept of intertextuality has been known in European thought since the 1920s thanks to the translinguistic analysis of Mikhail Bakhtin and it was disseminated in France thanks to the studies of Julia Kristeva (1969). As Greimas and Courtés note, ‘covering and expanding, without contradicting it, the concept of intertextuality, that of intersemiosis is imposed, in semiotic theory, in the name of respect for immanence. Its construction is made possible by the distinction between the two types of macrosemiotics which are semiotics of the natural world on the one hand, and natural languages on the other’ (1986: 119). Just like Jakobson (1959), Greimas and Courtés (1986) could not imagine a phenomenon of intersemiosis which excluded the language system. But how did this transition from one key semiotic concept to another come about so early if one takes into account that even today intersemiotic translation is a concept carefully studied and not so widely used in semiotic and translational studies? In our view, it was Roland Barthes’s essay ‘Rhétorique de l’image’ (1964), based on the notion of synergy of semiotic systems, which laid the foundations for the study of intersemiosis. He showed that it is a selective and intentional coexistence where different signs can coexist and say the same thing, even at the connotative level. But how is this concept defined? Parret defines synergy as ‘[. . .] the coordinated action of several systems [. . .] from which results the accomplishment of a function, the execution of a movement’ (2006: 121). Parret adds that ‘synergy [. . .] as a harmony of trends oriented by the same purpose, must be thought of in the world of fusion’ (2006: 130). The notion of fusion is not a new term, since there is also the related notion of syncretism. Jakobson argues that ‘the study of communication must distinguish between homogeneous messages which use a single semiotic system and syncretic messages based on a combination or merger of different sign patterns’ (1971: 705). It is this synergy/ fusion in visual communication that Barthes (1964: 43) became aware of when he carried out the first semiotic study of advertising, and he wondered if the image duplicated certain information from the text, through a phenomenon of redundancy, or the text adds new information to the image. It does not escape our attention that, when Barthes (1964) deals with the anchoring function in the case where a ‘legend’ accompanies an iconic message, the linguistic message (the legend) can share the same load of information with certain iconic elements. Thus, two different signs can say the same thing which classifies this semiotic fact in the field of translation. In other words, the fact of intersemiosis is produced in order to combat the ambiguity of the image. In our view, the notions of synergy, fusion, syncretism, redundancy have paved the way for the concept of intersemiosis. The definition of intersemiosis by François Rastier and Carine Duteil-Mougel, ‘intersemiosis defines an interaction between system of signs’,

TRANSLATION STUDIES AND SEMIOTICS

153

relates to the contribution of the Paris School (2009: 215). As early as 1973, Jakobson emphasized that ‘there is a variety of semiotic systems that include various substitutes for spoken language’ (1973: 28). Jakobson clearly links polysemiosis to translation, citing that all forms of communication ‘are accompanied by certain verbal utterances or other semiotic manifestations or both [. . .] they are not verbalized, they are all verbalizable, that is to say into verbal messages, or, at least, lower’ (1973: 34). In our view, this position, which harmonizes with Barthes’s positions, has advanced visual semiotics which, in turn, has developed and disseminated the concept of intersemiosis. The Paris School of Semiotics greatly influenced the Italian School of Semiotics. Eco (2001), an iconic figure in Italian semiotics, wrote an important book on this field. For Eco (2001, 2004), translation is based on negotiation, a process through which one renounces something to gain something else, and in the end, those involved should end up with a sense of reasonable and mutual satisfaction. Eco also made two further points that are of particular interest. First, Eco suggests a new definition for intersemiotic translation, where intersemiotic translation is to be understood as an adaptation (2001: 118, 125). Second, Eco argues that interpretation is not always synonymous with translation, since the universe of interpretations is much larger than that of translation (2001: 73). Paolo Fabbri, an equally important figure in Italian semiotics, diverges from Eco in both the above points. Fabbri posits that every semiotic system can be translated into another semiotic system (2008: 161). Thus, novelistic writing can be translated into film for television or cinema, and when instances of untranslatability occur, it is a question of changing strategy in order to allow for every fundamental element of the source text to come through. Fabbri seems to accept the view that interpretation is a form of translation (2008: 112). The positions and research proposals mentioned above only point to some of the topics of the Semiotics of translation, an innovative approach to translation, different from the classic approach that accompanied translation in the past. In our view, to the Semiotic School in Paris we owe the widening of the notion of translation. From this perspective, the influence of the Paris Semiotic School is important. Even if a large number of translators limit the contribution of semiotics to the presence of the iconic messages which accompany the written text, the semiotic notions presented in this study can help in the cultural understanding of the function of translation, a phenomenon which seems to be primarily verbal, but not always.

TRANSLATION FROM THE PERSPECTIVE OF THE MOSCOW-TARTU SCHOOL OF SEMIOTICS The Moscow-Tartu School established a theoretical framework for the semiotics of culture under the influence of the Russian semioticians, and in particular Mikhail Bakhtin. Eminent scholars belonging to the Moscow-Tartu School of Semiotics are Juri Lotman, Vjacheslav Ivanov, Boris Uspenskij, Alexander Pjatigorskij, Vladimir Toporov, Peeter Torop, Kalevi Kull, to mention but a few. Juri Lotman is considered the most eminent scholar of the Moscow-Tartu School. His work on translation is far less known than the rest of his work, despite his belief that ‘the instrument of semiotic research is translation’ (1990: 271). The School broadened the notion of translation based on a series of notions such as cultural text, semiosphere and cultural boundary, cultural equivalence,

154

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

(hetero)communication and autocommunication, and introduced the notion of cultural translation in semiotic studies. As concerns the concept of cultural text, for the Moscow-Tartu School, language is not used in the sense of a natural language but in the ‘specifically semiotic sense’ described by Uspenskij et al. ([1973] 2003: 297) which is applied ‘also to any carrier of integral (“textual”) meaning – to a ceremony, a work of the fine arts, or a piece of music’. According to Lotman (1990: 125), cultural texts ‘relate to each other along the spectrum which runs from complete translatability to just a complete mutual untranslatability’. The disengagement of the concept of the text from the linguistic text had already been proposed in the field of semiotics by Roland Barthes in the 1960s, when Barthes (1964) spoke not only of linguistic message in advertising, but also of uncoded iconic (image) and coded iconic or symbolic message. A new concept also introduced into semiotics by Lotman is the concept of semiosphere, which is used quite often in semiotic studies for translation. Lotman defines the semiosphere as ‘that same semiotic space, outside of which semiosis itself cannot exist’ (2005: 208–9). In other words, the semiosphere includes all these signs that belong to a cultural system and which seem to be not only a first, but also a second set of signs, that is, they are a record of cultural information. For Lotman, translating information across these semiospheric boundaries creates meaning, creating new information (2005: 215). This new information could be culturally equivalent to an information of another cultural system. The notion of equivalence was also proposed by Jakobson (1959). For Lotman, ‘at the basis of every act of exchange lies the contradictory formula, “equivalent but different”: the first part of the formula makes an exchange technically possible and the second part makes it meaningful in content’ (1977: 96). The translated semiosphere has to appear more or less intact in its translated representation and this semblance of formal equivalence, i.e. equivalence of signifiers of the two semiotic systems, is the job of this intermediate semiosphere, the sphere where informational loss is, as it were, manufactured. This notion of an intermediate sphere is useful, even though it creates an ad absurdum infinite creation of semiospheres for it accounts for the fact that semiosis occurs within systems of meanings with different, competing, cross-destructive or crosscomplementary (even self-destructive) semantic values. Scholars of the semiotics of culture have often underlined the contribution of the notion of equivalence in cultural translatability, as a function of controlling informational loss. According to Uspenskij et al., ‘translation from one system of text to another always includes a certain element of untranslatability’ ([1973] 2003: 311).3 Untranslatability is detected in the function of equivalence from one semiotic system to another. Translation will draw its informational material from those elements that the translated and the translatable semiospheres have in common and to this task the contribution of intersemiotic translation seems very useful. For Lotman translation is connected to human thought. Lotman (1990: 43) claims that ‘an elementary act of thinking is translation’, explaining that ‘the elementary mechanism of translating is dialogue’. Lotman (2009: 6) also adds that ‘[. . .] even the nature of the intellectual act could be described in terms of being a translation, a definition of meaning as a translation from one language to another, whereas extra-lingual reality may be regarded as yet another type of language’. If extra-lingual reality is another type of language, then it follows that intersemiosis is a wider phenomenon, as well as cultural translation in the cultural space, as already Eco (1976) aptly has claimed.

TRANSLATION STUDIES AND SEMIOTICS

155

This dialogue could be realized in the frame of heterocommunication (also known as communication) or autocommunication (Lotman 1990). In Lotman’s autocommunication the subject is transmitting a message to him- or herself. It is worth mentioning that translation as communication and autocommunication is also approached by Sütiste and Torop who mention that ‘as the concept of translation broadens, it approaches the concept of understanding – understanding through translation and understanding the translation itself. To understand different kinds of translation means to understand both communication and autocommunication processes’ (2007: 202). The most important contribution of the Moscow-Tartu School is the correlation of the concept of culture with the concept of translation. This school not only linked culture with translation, since ‘[culture] is not only the place where meanings are born, but the space in which they are being exchanged, “transmitted” and seek to be translated from one cultural language into another’ (Toporov 1992: 30) but went one step further than the French school by linking the concept of text/semiotic system with translation. They argued that a message can be considered a text when it is open to later translations and interpretations (Lotman and Pjatigorskij 1969). An important contemporary figure of this school is Peeter Torop (1998, 2002, 2003, 2007, 2008, 2012) who sees the concepts of translation and culture as overlapping. As Torop (2002: 603) states, ‘in the discipline of the semiotics of culture it comes naturally to say that culture is translation, and also that translation is culture’, in the sense that meaning is always something to be transferred from one locus to another, and this metaphoric act is at once an act of translation and an act of culture. Torop’s (2003) introduction of the concept of total translation is of particular importance. As Torop (2003: 271–2) explains, total translation describes different types of textual communication in culture, not only interlingual translation, but metatextual, intratextual, intertextual and extratextual translation as well. The change of the text’s ontology in contemporary culture due to the possible simultaneous existence of various forms of the same text in different media and discourses allows us to regard culture as a process of intersemiotic translation. An interesting approach and expansion of the Lotmanian theory constitutes the openness to translation by the Lund School of Semiotics and mainly by Göran Sonesson. Sonesson (2014a) approaches translation as a double act of communication and proposes the replacement of the notion of ‘translation’ with the notion of ‘interference’ in the three kinds of translation offered by Jakobson in 1959. Moreover, Sonesson (2014a, 2014b) argues that translation is a special case of transposition and that Jakobson’s intersemiotic translation, in contrast to intralingual translation, maintains the characteristics of translation proper, which is (usually) a double act of communication.

­SIMILAR CONCEPTS FOR INTERSEMIOSIS IN TRANSLATION STUDIES We argue that there are subfields where a semiotic theory can make a significant contribution, and that is, phenomena, such as adaptation, interpictoriality, intericonicity and transduction. It is worth mentioning that the terms intersemiotic translation and transmutation were proposed by Jakobson (1959) to describe the interpretation of verbal signs by non-verbal, but many semioticians accept that intersemiotic translation may occur among non-verbal sign systems as Lawendowski (1978), Klinkenberg (1996), Sonesson (1996), Torop (2000), Petrilli (2003), Fabbri (2008) argue, to mention but a few.

156

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

What is widely known in Semiotics and Translation Studies as intersemiotic translation is widely known in film and theatre studies as adaptation. According to Bastin, ‘adaptation may be understood as a set of translative interventions which result in a text that is not generally accepted as a translation but is nevertheless recognized as representing a source text’ (2008: 5). Eco argues that ‘many adaptations are translations in the sense that they isolate one of the levels of the source text, and it is this level that the adapter wishes to render in another continuum’ (2001: 125). It is worth noting that this kind of intersemiotic translation concerns Jakobson’s definition of translation of verbal signs to non-verbal signs systems which is the transfer of a literary work for the theatre, cinema or television. But what is happening when the verbal system is not involved? In this case, other terms are used instead, such as interpictoriality, intericonicity or transduction. According to Sonesson, ‘we will have to take into account the possibility of intrapictorial translation (e.g., exchanging one drawing for another) and interpictorial translation (e.g., substituting a photograph for a drawing)’ (1996: 10). Sonesson (1996) classifies as translation practices the case of the exchange and substitution of an iconic message by another. He goes on to describe how the production of a cultural text could be based on oppositions in absentia. The term of interpictorial translation might not be widely diffused in semiotic studies, but in Interart and Translation Studies the term intericonicity is frequently used. According to Gamer, [T]he concept of ‘intericonicity’ itself – other terms or descriptions for the phenomenon are ‘interpictoriality’, ‘intervisuality’ or simply ‘intertextuality of images’ – is located at the point of intersection of art history and literary criticism and is widely thought of as analogous to the concept of literary intertextuality. (2013: 115) Gamer defines intericonicity as the ‘relationships between images as well as the modi of their transformation from one into another’ (2013: 116). Gamer also highlights that ‘on closer inspection, it does not seem necessary to limit the usage of this term to art/art relationships’ (2013: 215). In this perspective, Kourdis and Yoka (2014) tied to relate this notion to translation studies based on Lotman’s and Fabbri’s theory. For Kourdis and Yoka, words and images are inseparable and all pictures are not just similar images presupposing one another but actual semiotic transpositions (2014: 174–5). Like with every intericonic incident, intersemiosis between pictures (visual signs) and images (on different media) points to the constructedness of the pictorial and the linguistic signification underlying and cross-supporting it. A special type of intersemiotic translation is transduction. Even though the term transduction derives from biology, it is used in both semiotics and translation studies, two disciplines that essentially study cultural communication. Kull mentions: [T]he process of translation is central to the genetic code, which is rather a complex of processes that manages the correspondence between nucleotide triplets and amino acids. While the concepts of ‘code’ and ‘translation’ were initially used metaphorically, one of the more important conclusions of biosemiotics is that these are in fact semiotic processes, that is, actual aspects of semiosis. (2015: 521)

TRANSLATION STUDIES AND SEMIOTICS

157

Influenced by Thomas Sebeok, Petrilli mentions that ‘transduction consists of a series of transformations or translations in the source and in the destination, operated on the basis of the interpretation of a probable homology between meaning and an externalized serial string (e.g., speaking, writing or gesturing)’ (2014: 256). Petrilli is not the only scholar who puts side by side the notions of transformation and transduction. Bezemer and Kress, in their framework, use translation as the general semiotic term, transformation for changes within a mode and transduction as the more specific term for the move of semiotic material from one mode to another (2008: 169, 175). The notion of transduction seems to be used in translation semiotics in a metaphorical sense. Thus, for Fabbri, transduction is ‘translation between different semiotic systems’, not excluding the language system (2008: 161). Thus, Fabbri supported the idea of a meta-painting, i.e. of a painting that can not only formulate through language general theoretical hypotheses, but which can also function through clearly iconic signifieds, which cannot be expressed in words (2008: 161). Though Fabbri relies more on a semiotic leap of faith to hold on to the idea of a non-linguistic sign that can be translated into another non-verbal sign, he is certain that in the case of iconic signs we can actually talk of transduction. Gorlée also highlights that transduction moves beyond language and ‘is the new term that relates to the parallel of “translation” not applied to language but within intermediality of speeches of non-linguistic texture between the doctrinal, the educational, and the emotional aspects of different arts’ (2015: 9). All these terms are based on the notion of similarity and of the transfer of the information load in cultural communications. In our view, they refer to intersemiotic phenomena and could be studied through the lens of translation semiotics.

CONCLUDING REMARKS As scholars, we choose our theoretical approach and, consequently, methods with respect to our preconceived idea of the research object. If this object is translation and we consider it to exemplify linguistic activities, we probably search for a suitable framework in linguistics. If we conceive translation rather as a phenomenon of cultural, cognitive, sociological or technological nature, we acquaint ourselves with research into such disciplines as cultural studies, cognitive science, sociology or digital humanities respectively. If translation, in our view, primarily represents the realm of signs, we presumably look for potential sign-theoretical insights. In this way, our approach finds its justification in providing an angle and answers, interpretants of the sign ‘translation’, which together can challenge, complement and widen our state-of-the-art notion of translations and translating. The present chapter has concentrated on discussing how translation semiotics can construct and enhance our big-picture-perspective on translation. We have presented how translation studies can benefit from semiotic views and how translation has recently been studied within semiotics. Focusing on semiotics and its mainstream schools and branches, we have endeavoured to emphasize that translation semiotics has early on, since Jakobson, questioned many of the received or prevailing views, putting forth and recommending an (often theoretical) alternative which originates from the persistent revisiting and rethinking activities of translation semioticians. If we translate our observations into semeiotic, the growth of translation semiotics attests that what emerged as a mere possibility has, step by step, developed into the

158

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

actuality of today, reaching slowly towards the tendenciality of tomorrow. Despite this encouraging development, the short history of the branch cannot thus far be described as a success story. As members of the research community, translation semioticians are the agents who with their diverse motifs, goals and emphases, and in their Greimasian actantial roles as subjects and objects, senders and receivers, helpers and opponents, are themselves the key to the future of their field.

NOTES 1 Popovič (1975: 16) is probably the first to refer to the semiotics of translation, denoting the semiotic nature and aspect that characterizes the translation process due to spatiotemporal differences. 2 All translations from French are by the authors. 3 For Lotman’s influence on translation studies see also Gorlée (1994: 19–21).

REFERENCES Barthes, R. (1964), ‘Rhetorique de l’image’, Communications, 4: 40–51. Bastin, G. (2008), ‘Adaptation’, in M. Baker and G. Saldanha (eds), Routledge Encyclopedia of Translation Studies, 3–6, London: Routledge. Bezemer, J. and G. Kress (2008), ‘Writing in Multimodal Texts: A Social Semiotic. Account of Designs for Learning’, Written Communication, 25 (2): 166–95. Chesterman, A. (2002), ‘Semiotic Modalities in Translation Causality’, Across Languages and Cultures: A Multidisciplinary Journal for Translation and Interpreting Studies, 3 (2): 145–58. Courtés, J. (2001). ‘La sémiotique, comprendre l’univers des signes’, in J.-F. Dortier (ed.), Le langage. Nature, histoire et usage, 121–126, Auxerre : Editions Sciences Humaines. Eco, U. (1976), A Theory of Semiotics, Bloomington: Indiana University Press. Eco, U. (2001), Experiences in Translation, trans. A. McEwen, Toronto Italian Studies, Toronto: University of Toronto Press. Eco, U. (2004), Mouse or Rat? Translation as Negotiation, London: Phoenix. Fabbri, P. (2008), Le tournant sémiotique, Paris: Lavoisier. ­Gambier, Y. and van Doorslaer, L. eds (2004–), Translation Studies Bibliography, Amsterdam: John Benjamins. Gamer, E.-C. (2013), ‘Configurations of Emptiness: Intericonic Blanks in Louise Lawler’s A Movie without the Picture and Hiroshi Sugimoto’s Theatres’, in C. Taban (ed.), Meta- and InterImages in Contemporary Visual Art and Culture, 115–31, Leuven: Leuven University Press. Goethals, G., R. Hodgson, G. Proni, D. Robinson and U. Stecconi (1999–2000), ‘Semiotranslation: Peircean Approaches to Translation’, Athanor: Semiotica, Filosofia, Arte, Letteratura, anno X, nuova serie (2): 139–50. Gorlée, D. L. (1994), Semiotics and the Problem of Translation: With Special Reference to the Semiotics of Charles S. Peirce, Amsterdam: Rodopi. Gorlée, D. L. (2012), Wittgenstein in Translation: Exploring Semiotic Signatures, Berlin: de Gruyter Mouton. Gorlée, D. L. (2015), From Translation to Transduction: The Glassy Essence of Intersemiosis, Tartu: Tartu University Press. Gottlieb, H. ([2000] 2004), Screen Translation: Seven Studies in Subtitling, Dubbing, and Voice-over, 4th edn, Center for Translation Studies, Department of English, University of Copenhagen.

TRANSLATION STUDIES AND SEMIOTICS

159

Gottlieb, H. (2017), ‘Semiotics and Translation’, in K. Malmkjær (ed.), The Routledge Handbook of Translation Studies and Linguistics, 45–63, London: Taylor & Francis. Greimas, A. J. (1983 [1966]), Sémantique structurale, Paris: Presses universitaires de France. Greimas, A. J. and J. Courtés (1986), Sémiotique. Dictionnaire raisonné de la théorie du langage, vol. 2, Paris: Hachette. Hartama-Heinonen, R. (2008), Abductive Translation Studies: The Art of Marshalling Signs, Imatra: International Semiotics Institute at Imatra & Helsinki: Semiotic Society of Finland. Hartama-Heinonen, R. (2015), ‘Herding Together: On Semiotic-translational Branches, Fields and Disciplines’, Punctum: International Journal of Semiotics, 1 (2): 39–52. Jakobson, R. (1959), ‘On Linguistic Aspects of Translation’, in R. A. Brower (ed.), On Translation, 232–39, Cambridge, MA: Harvard University Press. Jakobson, R. (1971), Selected Writings, vol. 2, The Hague: Mouton Press. Jakobson, R. (1973), Questions de poétique, Paris: Editions Du Seuil. Ketner, K. L. (1998), His Glassy Essence: An Autobiography of Charles Sanders Peirce, Nashville: Vanderbilt University Press. Klinkenberg, J.-M. (1996), Précis de sémiotique générale, Bruxelles: De Boeck Université. Kourdis, E. (2015), ‘Semiotics of Translation: An Interdisciplinary Approach to Translation’, in P. P. Trifonas (ed.), International Handbook of Semiotics, 303–20, Dordrecht: Springer. ­Kourdis, E. and C. Yoka (2014), ‘Intericonicity as Intersemiotic Translation in a Globalized Culture’, in W. Yongxiang and J. Haihong (eds), Our World: A Kaleidoscopic Semiotic Network. Proceedings of the 11th World Congress of the IASS/AIS, 5–9 October 2012, Nanjing Normal University, volume 3, 162–76, Nanjing, Hohai University Press. Kristeva, J. (1969), Σημειωτική. Recherches pour une sémanalyse, Paris: Seuil. Kristeva, J. (1974), La révolution du langage poétique. L’avant-garde à la fin du XIXème siècle: Lautréamont et Mallarmé, Paris: Seuil. Kukkonen, P. (2014), I språkets vida rum: Salens språk – språkets sal: Volter Kilpis modernistiska prosaepos Alastalon salissa i Thomas Warburtons svenska översättning I salen på Alastalo, Helsinki: The Semiotic Society of Finland. Kull, K. (2015), ‘Introduction to Biosemiotics’, in P. Trifonas (ed.), International Handbook of Semiotics, 521–34, Berlin: Springer. Lawendowski, B. (1978), ‘On Semiotic Aspects of Translation’, in T. Sebeok (ed.), Sight, Sound and Sense, 264–82, Bloomington: Indiana University Press. Lederer, M. (1994), La traduction aujourd'hui, Paris: Hachette. Lotman, J. (1977), ‘Primary and Secondary Communication-Modelling Systems’, in D. Lucid (ed.), Soviet Semiotics: An Anthology, 95–8, Baltimore and London: The Johns Hopkins University Press. Lotman, J. (1990), Universe of the Mind: A Semiotic Theory of Culture, Bloomington and Indianapolis: Indiana University Press. Lotman, J. (2005), ‘On Semiosphere’, Sign Systems Studies, 33 (1): 205–29. Lotman, J. (2009), Culture and Explosion, Berlin: Walter de Gruyter. Lotman, J. and A. Pjatigorskij (1969), ‘Le texte et la fonction’, Semiotica, 1: 205–17. Marais, K. and K. Kull (2016), ‘Biosemiotics and Translation Studies: Challenging “Translation”’, in L. van Doorslaer and Y. Gambier (eds), Border Crossings: Translation Studies and Other Disciplines, 169–88, Amsterdam: John Benjamins. Mossop, B. (2016), ‘“Intralingual Translation”: A Desirable Concept?’ Across Languages and Cultures, 17 (1): 1–24.

160

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Mossop, B. (2019), “‘Intersemiotic Translating”: Time for a Rethink?’ Translation and Interpreting Studies, 14 (1): 75–94. Parret, H. (2006), Sutures sémiotiques, Limoges: Lambert-Lucas. Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Petrilli, S. (2003), ‘Translation and Semiosis. Introduction’, in S. Petrilli (ed.), Translation Translation, 17–37, Amsterdam: Rodopi. Petrilli, S. (2014), Sign Studies and Semioethics: Communication, Translation and Values, Berlin: de Gruyter Mouton. Ponzio, Augusto (1999–2000), ‘Presentazione/Presentation’, Athanor: Semiotica, Filosofia, Arte, Letteratura, anno X, nuova serie (2): 5–7. Popovič, A. (1975), Dictionary for the Analysis of Literary Translation, Edmonton: Department of Comparative Literature, University of Alberta. Queiroz, J. and D. Aquiar (2015), ‘C. S. Peirce and Intersemiotic Translation’, in P. P. Trifonas (ed.), International Handbook of Semiotics, 201–15, Dordrecht: Springer. Radnitzky, G. ([1968] 1970), Contemporary Schools of Metascience, 2nd edn, Göteborg: Göteborg University Press. ­Rastier, F. and C. Duteil-Mougel (2009), ‘Intersémioticité’, in D. Ablali and D. Ducard (eds), Vocabulaire des études sémiotiques et sémiologiques, 215–16, Paris: Honoré Champion & Besançon, Presses Universitaires de France-Compté. Robinson, D. ([1997] 2003), Becoming a Translator: An Introduction to the Theory and Practice of Translation, 2nd edn, London and New York: Routledge. Snell-Hornby, M. ([1988] 1995), Translation Studies: An Integrated Approach, revised edn, Amsterdam: John Benjamins. Snell-Hornby, M. (2006), The Turns of Translation Studies: New Paradigms or Shifting Viewpoints?, Amsterdam: John Benjamins. Sonesson, G. (1996), ‘The Quadrature of the Hermeneutic Circle: The Picture as “Text”’, in D. Wilske (preface), LSP and Theory of Translation. Acts of the XVI Vakki symposion, Text and Image, Vöyri, February 10–12,1996, 9–33, Vaasa: University of Vaasa. Sonesson, G. (2014a), ‘Translation as a Double Act of Communication: A Perspective from the Semiotics of Culture’, in W. Yongxiang and J. Haihong (eds), Our World: A Kaleidoscopic Semiotic Network. Proceedings of the 11th World Congress of the IASS/AIS, 5–9 October 2012, Nanjing Normal University, vol. 3, 83–101, Nanjing: Hohai University Press. Sonesson, G. (2014b), ‘Translation and Other Acts of Meaning: In between Cognitive Semiotics and Semiotics of Culture’, Cognitive Semiotics, 7 (2): 249–80. Stecconi, U. (2004), ‘Interpretive Semiotics and Translation Theory: The Semiotic Conditions to Translation’, Semiotica, 150 (1–4): 471–89. Stecconi, U. (2007), ‘Five Reasons Why Semiotics Is Good for Translation Studies’, in Y. Gambier, M. Shlesinger and R. Stolze (eds), Doubts and Directions in Translation Studies: Selected Contributions from the EST Congress, Lisbon 2004, 15–26, BTL 72, Amsterdam: John Benjamins. Sütiste, E. (2008), ‘Roman Jakobson and the Topic of Translation: Reception in Academic Reference Works’, Sign Systems Studies, 36 (2): 271–314. Sütiste, E. and P. Torop (2007), ‘Processual Boundaries of Translation: Semiotics and Translation Studies’, Semiotica, 163 (1): 187–207. Tarasti, E. (2000), Existential Semiotics, Indianapolis: Indiana University Press.

TRANSLATION STUDIES AND SEMIOTICS

161

Tarasti, E. (2017), ‘The Semiotics of A.J. Greimas: A European Intellectual Heritage Seen from the Inside and the Outside’, Sign Systems Studies, 45 (1): 33–53. Toporov, V. (1992), ‘Translation: Sub Specie of Culture’, Meta, 37 (1): 29–49. Torop, P. (1998), ‘The Limits of Translation: The Socio-Semiotic Aspect of Translation Semiotics’, Sign Systems Studies, 26: 136–50. Torop, P. (2000), ‘Intersemiosis and Iintersemiotic Translation’, European Journal for Semiotic Studies, 12 (1): 71–100. Torop, P. (2002), ‘Translation as Translating as Culture’, Sign Systems Studies, 30 (2): 593–605. Torop, P. (2003), ‘Intersemiosis and Intersemiotic Translation’, in S. Petrilli (ed.), Translation Translation, 271–82, Amsterdam: Rodopi. Torop, P. (2007), ‘Methodological Remarks on the Study of Translation and Translating’, Semiotica, 163: 347–64. Torop, P. (2008), ‘Translation and Semiotics’, Sign Systems Studies, 36 (2): 253–57. Torop, P. (2012), ‘Semiotics of Mediation’, Sign Systems Studies, 40 (3): 547–57. Torres-Martínez, Sergio (2018), ‘Semiosic Translation’, Semiotica, 225: 353–82. ­Torres-Martínez, Sergio (2020), ‘Translating Wittgenstein: A Semiotic Translation of the Tractatus’, Semiotica, 233: 91–123. Toury, G. (1980), In Search of a Theory of Translation, Tel Aviv: The Porter Institute for Poetics and Semiotics, Tel Aviv University. Toury, G. (1986), ‘Translation: A Cultural-semiotic Perspective’, in T. A. Sebeok (gen. ed.), Encyclopedic Dictionary of Semiotics Tome 2, 1111–24, Berlin: Mouton de Gruyter. Uspenskij, B.-A., V.-V. Ivanov, V.-N. Toporov, A.-M. Pjatigorskij and J.-M. Lotman ([1973] 2003), ‘Theses on the Semiotic Study of Cultures (as Applied to Slavic Texts)’, in M. Gottdiener, K. Boklund-Lagopoulou and A.-Ph. Lagopoulos (eds), Semiotics, vol. 1, 293–316, London: Sage. Vassallo, C. (2015), ‘What’s So “Proper” about Translation? Or Interlingual Translation and Interpretative Semiotics’, Semiotica, 206: 161–79. Welby, V. Lady ([1903] 1983), What Is Meaning? Studies in the Development of Significance, Amsterdam: John Benjamins. Zethsen, K. K. (2009), ‘Intralingual Translation: An Attempt at Description’, Meta, 54 (4): 795–812.

162

­C HAPTER EIGHT

Pragmatics and Semiotics PER AAGE BRANDT

Pragmatics, as well as semiotics, may be described as a discipline studying meaning and language. A possible way of distinguishing these disciplines would be to let pragmatics be the study of expressed meaning in a context of pragma, while semiotics takes care of the internal syntax and semantics of expressed meaning, or in other words, to understand semiotics as the study of the immanent aspects of meaning, and pragmatics as the study of its transcendent aspects.1 If meaning only had transcendent aspects, in this sense, pragmatics would be semiotics, and inversely. This view, however, presupposes, as the French-Lithuanian semiotician A. J. Greimas says,2 a general theory of language, thus of the general architecture of existing aspects of expressed meaning.3 In this respect, our discussion necessarily involves linguistics. What would an immanent-transcendent model of the componential structure of language look like? To answer this question would be to describe the relations between semiotics and pragmatics, and it would yield a view of a more comprehensive discipline that so far has no name. A semio-pragmatics? We will venture to draft a version of such an answer by presenting a general model of language in its structure and in its use.

MODELLING THE RELATION BETWEEN PRAGMATICS AND SEMIOTICS The first such model is based on the notion of semiosis. A semiosis is a ‘little machine’, some say a ‘coded’ function, that makes an expression refer to something, its ‘content’.4 As Louis Hjelmslev ([1943] 1993) and, after him, Roland Barthes (1985) suggested, expressions and contents can also each be entire semioses; so, one semiosis can be the expression of, that is, connote, another semiosis.5 And it can be the content of another semiosis as a text in a painting.6 Language can be described as an inter-semiosic agglomeration of these little semiotic machines, an inter-semiosic architecture. This is what the model shows (Figure 8.1, below).7 Let us take a look at the composition of this inter-semiosic model of language. The most conspicuous of the involved semioses is the one that makes phonetic strings, as expressions, refer to grammatical meanings, as their contents; these strings can be morphemes, words, phrases, clauses, sentences, entire texts, insofar as they have phonetic properties. The lexical, grammatical and discursive meanings are the meanings thus conveyed. Furthermore, phonetics is itself a semiosis, since the involved bodily, substantial expressions (writing, gesture, phonation) mean the formal phonemes and syllables that are the linguistic expressions. And the syntactic meaning of the resulting linguistic

164

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

expressions, that is, of the words, phrases, clauses, sentences and trans-phrastic units, which all have conceptual meaning to contribute, is a linguistic semantics. Therefore, we have, at the core of linguistic immanence, a general grammatical semiosis of two semioses, the phonetic semiosis and the semantic semiosis. The bodily ‘input’ (phonetics) is grammatically connected to a mental ‘output’ (semantics) of this process, namely what we call conceptual meaning. Conceptual meaning exists in different formats, of course: from single concepts to entire stories or doctrines. However, this triple semiosis – phonetics-through-grammar-to-semantics – producing conceptual meaning has to be inserted in an equally complex transcendent superstructure in order to give rise to what we are especially interested in here, namely referential meaning, i.e. meaning as we experience and understand it in context. Conceptual meaning has to be converted to referential meaning, as semantics feeds into pragmatics. In order to understand this wider framework, we first need to introduce the semiotic component of enunciation (from Benveniste 1966). Enunciation is a semiosis, whose ‘little machine’ lets certain expressions indicate states and attitudes of speaker’s and hearer’s subjectivity, social identity and condition. The simplest manifestations of enunciation in language are the systems of personal pronouns and the deictic forms (demonstratives, shifter adverbs, etc.) that languages use. The dimension we call prosody superimposes enunciational meanings onto the immanent linguistic semiosis. These meanings include speaking from a higher or lower status than the hearer’s; speaker and hearer knowing or not knowing each other; gender and gender difference; generational difference; and (personal or social) emotional tonality. The enunciational semiosis overdetermines all utterances uttered, and the result is then transposed into situational meaning through a specific pragmatic component that takes the personalized utterances as appropriate signifiers and the specific situation as signified. This pragmatic component is a semiosis in the sense that it makes the personalized utterances mean their situational context, contribute actively and intentionally to making the context specific and in this respect, meaningful. Contexts become meaningless if we do not ‘intend’ them by letting our language signify them. Ceremonial rituals become invalid if the formulaic language they involve is not displayed appropriately; conversations are not taking place if our utterances do not ‘converse’ in the right way (lecturing, for instance, is not conversing properly). Even quarrelling does not happen if our language does not know how to ‘quarrel’. The pragmatic ‘little machine’ will therefore have to know the culturally defined situational categories that language is supposed to signify; otherwise we would speak out of order, use wrong turn-taking, etc. Pragmatics in this sense is really a particular semiosis, and an important one, since it is when language is used in culturally defined situations that its conceptual meaning is converted into referential meaning. Finally, the personalization that enunciation provides, and the ‘situationalization’ that pragmatics provides, integrate in discourse, that is, in a discursive, rhetorical semiosis, in which an enunciative personalization expresses pragmatic situationalization: the roles we play as speakers and hearers theatrically determine and mean the situations we are in; so celebrative rhetoric sounds unlike political rhetoric or judiciary rhetoric, and the rhetorical ‘sounds’ will again mean these different situations, where language therefore will carry distinct referential meanings. Enunciation, pragmatics and discursive, theatrical rhetoric constitute the overarching, transcendent semiosic complex by which language is ‘carried’ into the world of its ‘use’. We might want to follow Ferdinand de Saussure (1915) or reinterpret him and call the lower triad of semioses, in French, la langue, or the language system, as opposed

PRAGMATICS AND SEMIOTICS

165

FIGURE 8.1  The complex functions of language.

to the upper triad of semioses, la parole, or language use. The hinge between the lower (immanent) and the upper (transcendent) part of the complex is the core utterance semiosis we also called grammar. A theory only accepting to study the lower semiosic triad would just be a grammar theory, rather than a comprehensive, linguistic, general science of language embracing both the immanent and the transcendent parts of language as a constellation of semiosic ‘little machines’.

PRAGMATICS AS SEMIOSIS AND DIEGESIS What is a situational context of an utterance? Eric Landowski8 suggests that such a ‘semiotics of situations’ would coincide with a complete narrative grammar, including an account of inter-actantial relations of ‘manipulation’, aspectualizations, modalizations, etc. A main point of Greimasian semio-narratology is in fact to theorize the narrated content, the story, apart from narration itself, thus as a series of situations where things simply happen to people, and the narrator has not yet arrived, so to speak (the narrator supposedly arrives through subsequent ‘discursivization’). The semantic level of the narrated is separated from the semantic level of the narrating, in the same way as the signified is seen as distinct from the signifier. A situation is thus simply an episode in the signified diegesis of a (possible) story. The next question will then be what the little pragmatic machine, or the semiosis of situations, recognizes as diegetically relevant episodes, which may thus contextualize language use. Is there really a canonical set of situational parts in a canonical sequence forming the ‘whole’ of a story? The claim to have found it would be strong, and no doubt far too strong, unless we believed that Propp and Greimas really solved the universal mystery of defining narrative structure.9 Nevertheless, the problem of classifying situations and thereby, if possible, opening a path towards a structural pragmatics is relevant; it has to be addressed, since as Landowski intimates in his 1983 lecture; otherwise pragmatics would let psychology and sociology define not only

166

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

contexts but pragmatics altogether. Such a relinquishment would in fact be unfortunate, because the sociological categories would be too distant from the situational format, and the psychological categories would be too proximal, too intimate, to match the format. Let us then assume that language is used in situations, and that situations are diegetic. Speaker and hearer are ‘actants’ and inscribed in a presupposed narrative and ‘actantial’ semantics. Now let us consider an example. The expression to make sure, Cambridge Dictionary: ‘to act so that, or check that, something is certain or sure’. Whether by acting or by checking to produce a state of belief. The conceptual meaning of the expression is open, so the task of the pragmatic setting of its use is to deliver its referential meaning in context. Here is a context10: 1. Two Hunters are out in the woods when one of them collapses. He doesn’t seem to be breathing and his eyes are glazed over. 2. The other guy whips out his phone and calls the emergency services. He gasps, ‘My friend is dead! What can I do?’ 3. The operator says, ‘Calm down. I can help. First, let’s make sure he’s dead.’ There is a silence, then a shot is heard. Back on the phone, the guy says, ‘Ok, now what?’ (Italics mine). The diegetic unfolding inscribes the expression /make sure/ in a dialogue, again inserted in a story. We may see this narrative context as consisting of four phases: (1) the collapse; (2) the call; (3) the advice, containing the performative utterance, let’s make sure; (4) the killing. The ‘actants’ are staged as hunters, therefore armed: (1) presents a crisis; (2) makes it a catastrophe: the speaking hunter, in panic, makes a mistake by mentioning his partner as deceased; (3) displays the consequence: the operator’s epistemic advice reuses the fatal expression; (4) thereby triggering the conclusion, the killing of the collapsed hunter – told from the operator’s viewpoint, so the act is only heard, not seen. Crisis, catastrophe, consequence, and conclusion form a diegetic series of narrative situations offering variable dynamic conditions. In this story, these first include the participants’ fragile bodily conditions (collapsing); then a fragile psychological condition (panic); a fragile linguistic sensibility and intelligence; and the fatal mistake rendered probable by the condition of carrying arms11: (4) refers back to (1) in this respect. Maybe the experience of life is diegetic in this or a similar sense? A recurrent contextual rhythm of four canonical phases might then underlie our apparently fluid experiential time and connect perception and reflection by the same backwards reference we find in narrative literature and jokes.12 This contextual rhythm, implicit or explicit, might then form a discreet elementary micro-narrative phenomenological music of time presupposed by the larger historical contextual frames that interest hermeneutics and the disciplines of historiography, where the experiential dimension disappears, and where the linguistic problem of contextual determinations of meaning is therefore less relevant.

THREE TYPES OF MEANING Making sure is either assertive (stating) or performative (acting). In modern standard pragmatics, the distinction between performative and assertive meaning is well-known and accepted as relevant, across various terminologies. Only assertive meaning can

PRAGMATICS AND SEMIOTICS

167

carry truth value. Performative meaning instead can carry validity or ‘speech act force’. In modern standard semiotics, as in Landowski, supra, this pragmatic distinction is translated into semiotic terms of modality: epistemic utterance force (truth value) versus deontic utterance force (performative value), respectively. In French, this translation is particularly tempting, since the French language only has two pure modal verbs, pouvoir and devoir, the former basically epistemic, the latter basically deontic – though A. J. Greimas’s modal syntax of course complexifies the field in many ways. Standard semiology also essentially distinguishes two semantic sign types, icons (signs of possibility, according to C. S. Peirce) and symbols (signs of necessity, whether logical or other). Of course, such dichotomies do not in any way explain the variations in force and value. We would like to better understand how the dichotomies work, why they work, and whether they account exhaustively for the types of meaning that exist in the pragmatic world. Terminologies alone do not satisfy our curiosity. Symbols are signs of doing, and icons are signs of being: symbolic signs are always instructions calling for acts, physical or mental (in math, music, social codes, etc.), whereas iconic signs are manifestations of seeing something, physically or mentally, in certain ways. In language, therefore, words are symbolic, and sentences are iconic.13 So, lexemic signs are ‘arbitrary’, conventional and, in a sense, performative, comparable to imperatives, orders in context (like social graphic signs: flags, emblems, injunctions: No smoking! Wrong way; No u-turn). Words are instructions for the mind to follow. Sentences are motivated co-lexemic signs, whose connecting morphemes create a sort of assembled conceptual predicate to a scene, an image of it, a simulacrum of a representation; only in ritual uses do sentences become symbols. Icons are signs of being, that is, of something being seen as something, as a subject with a predicate that invites the addressee to share the predication and thereby establish an affective bond with the sender. When icons and symbols combine, the operation consists in establishing the affective iconic bond and then superimposing a symbolic instruction to act accordingly, namely as the predication suggests (in caricatures: dissociate yourself from the iconic subject, reject it; in advertisements: associate yourself with the subject, buy it). Communication in the social sphere mainly uses such icono-symbolic messages; the combination is, so to speak, the basic formula for manipulation. Roland Barthes discusses it in his famous article ‘Rhétorique de l’image’ on a Panzani advertisement (Barthes 1964). Semiotically speaking, there is in this advertisement a double superposition, namely a (symbolic) text IN the image and a text ON the image. Figure 8.2 shows an unfolding of the result, an image that connotes, has a meta-semiotic comment and contains a text (naming the brand announced); hence a complex of four semioses. In this graph, the connotative semiosis taking the iconic semiosis as its expression derives its abstract content from the metonymic meaning of the iconic content. Composition and chromatic properties of the image can trigger rich associations. Connotation and metonymy are closely related. The driving force in such semiosic constellations is always the affective effect of the iconic parts (the referential image and its metonymic connotations) and the deontic effect of the symbolic parts (the metatext and the embedded text). There is, however, a third type of signs, besides symbols and icons, namely diagrams.14 These are neither arbitrary-and-performative nor image-like and similar to their representational content; they are neither about doing nor about being-and-feeling. They are instead graphic or gestural signs of minding – diagrams spontaneously use labelled arrows, boundary lines, bubbles and boxes, imaginary topologies of many sorts, which

168

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 8.2  The rhetoric of image and text.

are schemas, models or ‘theories’ of how things may be connected. Diagrams are signs of minding, modelling, ideation and surprisingly enough, the most important sign type in the social world, while also being the most ignored. There are reasons to believe that thinking itself is diagrammatic, and that the diagrammatic semiosis constitutes the semiotics of the mind itself, whether the conscious or the unconscious part. Ideas we exchange are most often neither instructions nor factual or idealized pictures; they may instead be sketches of a programme, a half-baked plan, a hypothetic construction, building blocks of a problématique, as the French say, an imaginary architecture to be completed by self or other. A diagram shows the thinking of something, simply as being thinkable (German: als denkbar). Disciplines of knowledge are mostly based on signs of this type, so the language that supports them is also of the third kind: dialogue and discourse, not just words and sentences. Dialogue and discourse allow us to ‘think aloud’ and thereby to think together, letting others complete our thought, or trying to complete others’ thought. When we discuss, we negotiate a shared imaginary architecture, corresponding to a diagram, and sometimes explicitly a drawn diagram on whiteboard or paper that we can point to. Symbols, icons and diagrams are three fundamentally different sign types in two ways. Firstly, by the constitution of their semiosis: the expression-to-content relation is conventional and non-motivated in the first case, non-conventional and motivated in the second case, and neither conventional nor motivated in the third case.15 Secondly, the type of meaning they convey is different: the addressee of symbols is called upon to act in some manner (the address is deontic: ‘you are suggested to do X’); the addressee of icons is called upon to perceive something in a specific manner (the address is alethic, and therefore affective16: ‘you are invited to feel this about X’); and diagrams invite the addressee to connect ideas, i.e. think (so the address is epistemic: ‘we may understand the problem X like this’). The sign types are as mutually irreducible as the meaning types.17 Responses and pragmatic exchanges differ according to these variations. You may obey or contest in the first case; be captivated or repelled in the second; and be interested or sceptical in the third. Discourse is possible in all cases, but

PRAGMATICS AND SEMIOTICS

169

dialogue is optimal in the third. Dialogue becomes quarrel if the exchange is entirely symbolic; by contrast, it vanishes or becomes a mutual echoing, if the exchange is entirely iconic. We will of course want to know why this is so. It will therefore be useful to consider the interrelations of meanings and their signs in social life and in our subjectivity.

POWER AND CRITIQUE Symbolic meaning rests upon principles of authority, namely that of the instances that back up the speaker, and entitles the enunciator to use performatives in specified situations. Authority in this sense includes named institutional identities (ministries, political parties, corporative, industrial, religious entities, etc.) and traditional cultural, historical references (memorable events, dates, personal statuses, domains of knowledge, etc.), and allows speakers and writers to express authoritative ‘values’ instead of their own thinking. All commands, instructions and otherwise imperative utterances receive their ‘speech act force’ from a presupposed entitlement signed by an authority, at least in the imaginary mode. The ritualization of commands (uniforms, gestures, formulaic phrasing) manifest this depersonalization of the performer. I would like to add that the ritualization and the depersonalization of the symbolic performer also produces another relevant effect: iconization. The performer plays a theatrical role and becomes an image of the non-personhood of authority – of its rigidity, the scary inanimate statue we call power. Ordinary iconic meaning is not backed by any such authority. It relies only on the affective meaning of the depicted motif, which means that making an iconic representation of something is (a symptom of) affirming its affective meaning. Therefore, icons of the imaginary gestalt of authority, that is, icons of the source of symbolic force, may be the foundational semiotic function that constitutes power in general.18 The mechanism described can be analysed in terms of enunciation. There is an elementary structure, in which first person (enunciator, P1) gives third person (object of attention, P3) to second person (enunciatee, P2). The constitution of power uses three embedded instances of this structure: I.

A human collective (P1) offers a cult ICON (P3) to itself (P2)

II. The ICON (as P1) then confers authority (A, P3) to a person S1 (P2) III. This person S1 (as P1) finally gives another person S2 (P2) a symbolic message, an order (SY) IV. The order (P1) tells S2 (P2) to act accordingly (P3). The uncanny statue of power (the icon of authority) stands behind the instance authorizing S1’s symbolic, performative acts, and it emerges as a compulsive motif of depiction: portraits of rulers hanging in the offices of their servants, monumental statues of famous and supposedly great historical persons, placed in the urban landscape, pompous architecture of banks and businesses, even advertisements linking the power of brands to images of their ‘greatness’, election posters, etc. – all such semiotic displays, iconic and symbolic, and eventually the entire social iconicity may consist in offering competing cults of implied symbolic power holders of all kinds. Universally, the iconic semiosis may have a symbolic semiosis hidden in its content, as in Figure  8.2. Thereby, affect and duty – fear and pride – will be connected in the

170

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­FIGURE 8.3  The semiotic constitution of power through enunciation.19

complying social addressee’s mind. Hence the interest of Barthes’s analysis: if social power is a pragmatic effect of socio-semiotic structures, a possible counter-force may be sought in critical structural commentary. Diagrammatic meaning, as it happens, can be critical. It can oppose iconic cult and symbolic obedience. It always ‘occurs’ to the thinking mind, which ‘gets’ ideas instead of actively producing them. Thoughts are mental diagrams that form the elementary, ‘occurring’ contents of consciousness, for example as maps of one’s present location, intuitions about relations between present persons, and about the time frame of present doings (something is still the case; but something else is already the case). More abstract diagrammatic contents are supported by worded labels or other markers, imported from memory, and inserted in the nodes of the intuitive graphics. A literary drama is slightly more abstract than an episodic experience, essentially because we are not in it. We are abs-tracted from it, and its understanding becomes an intellectual challenge to meet by diagramming. Science is built upon elaborate models of abstract empirical domains of the world; it ‘formalizes’ its intuitive diagrams into models which then underlie scientific theories. Political life would, on the other hand, be impossible without diagrams analysing the respective force of competing powers. The contemporary version of global political life has conferred overwhelming iconicsymbolic capacities to the digital media and is responsible for creating a dangerous imbalance between power and critique. Diagramming on screens is mainly seen in critical programmes, which are becoming rare; one reason is that diagrammatic thinking is slow, and that digital time is an expensive commodity demanding fast action; another reason is that global culture may be decreasingly prepared for autonomous alphanumerical, diagrammatical, mental activity by general education – except for weather or stock exchange reports. Critical activity is then bound to unfold wildly in the shape of bodily conflicts and material clashes of collective ‘identities’, each with its protective iconography.

PRAGMATICS AND SEMIOTICS

171

FIGURE 8.4  A sociogram example.

MUSIC Rhythmic emission of tonal phrases is probably the earliest semiotic activity of our species. Before serving as a generalized anaesthetizing tapestry for all sorts of messages in the media, it was mainly used for celebration. Music for weddings, funerals, coronations, memorial events of all kinds, was fundamental. It had, and still under certain circumstances, has the phenomenological capacity to extend the moment of celebration: to let ‘time stand still’. Concert music is self-celebrating and can be considered as experimental or laboratory music, as opposed to what I will call natural music, the sort that has been with us in our different cultures from their beginnings. The pragmatics of music20 is thus a global, universal and evolutionarily primordial concern. Its generality is the same as that of language, and curiously, music unfolds a closely similar structural complexity. Musical semiotics has in fact, like language, an immanent and a transcendent aspect. In music, certain sounds are performed and perceived as discrete tones, which are parts of melodic lines (syntagmatic axis) and also parts of tonal scales (paradigmatic axis) based on octave recognition. This aspect is the ‘phonetics’ of music. The melodic wholes have necessary rhythmic properties that determine how the involved bodies will move to it and especially dance, thereby expressing particular emotional states. These states are ‘staged’, induced by the music or called into being by its experience, and constitute the immanent semantic meaning of the specific songs played. In modern, academic music, where corporeal movements are often suppressed, the dancing and its meaning are mentalized and ‘performed’ by the imaginary homunculi in your head, whether you are a musician or a listener. The transcendent aspect includes the situation connoted by specific musical phrases and by the tonality chosen and used; scales are often specific for specific contexts: morning

172

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 8.5  The complex semiosis of music.

music, evening music, night music, winter music, spring music, summer music, fall music, high status scales, low status scales, etc., as if the calendar and the entire social spatiotemporal world were represented in the inventory of a culture’s acknowledged tonal scales. Tonality and pragmatics are thus naturally connected. The analogy between the complex semioses of language and those of music may make the following diagram easier to interpret. It is likely that the neuro-biological disposition for language is very similar to the disposition to make and enjoy music. Music may even have been laying the neural ground for language.21 In language, the integration of immanent, conceptual meaning in transcendent, pragmatic, referential meaning is an essential aspect of semantics, as we have seen. In music, this integration comes to the fore as a duality of feelings of the ‘character’ of a given work (if written) or song (if not) and the ‘character’ of the situation of performance: music ‘for’ funerals (sad), weddings (happy), coronations (solemn), military parades (awe-inducing), etc. – often linked to power displays. If power is rooted in collective, intersubjective iconicity (subjects seeing and believing what other subjects are seeing and believing), as suggested above, then the mere sharing of music, both seen and heard, will reinforce this constant re-founding effect of iterated collective performances that pertain to the existence of social powers, whether affirmative or contestant.

THE PRAGMATICS OF TROPES Through it all, Welser-Möst and the [Cleveland] orchestra are dashing partners, surrounding their guest [Radu Lupu] in vibrant colors without washing out any of his. On Thursday, they also helped steady him during the Adagio [of Bela Bartok’s Piano

PRAGMATICS AND SEMIOTICS

173

Concerto No. 3], when a ringing cell phone twice poked holes in the silken musical fabric he was weaving. To listeners, the noise was simply rude. But to Lupu, it must have been painful, like lightning striking. (Zachary Lewis, ‘Ensemble offers up an appealing climate’ Cleveland Plain Dealer, 01/17/09) Clouds occasionally gather in the E-flat trio, though Schubert is resilient enough to emergefrom minor-key territory and let the sun shine in his inimitable way (Donald Rosenberg, ‘Trio Conveys Wonder of Schubert’s Work,’ Cleveland Plain Dealer, 27/03/10 [Italics mine in both quotations]).22 Speaking about music as heard is not an easy task. Strikingly often, critics seem to refer to meteorology rather than to tonality. Above: lightning, climate, clouds, sunshine – but also vibrant colors, silken . . . fabric, and in all cases, visual, auditive and often also tactile sensations (silk) that happen in an imaginary space far from the concert hall, and far from what is written in the composition being played, in the above cases. You may be thinking of metaphor to explain this phenomenon. Language is not prepared for describing sensory perceptions that activate deep registers of the soul, so we use expressions of weather observations and other highly emotional imaginary contexts to express what we feel. However, metaphor would supply imagery to a topic that could also be covered literally; here, there is no literal wording of what is experienced available. In some books, the case is then called catachresis, or necessary metaphor (table leg, headquarter, etc.). But in metaphor, the image covers, and often hides, the referent (target), and, so to speak, signifies it. In the concert hall, the sounding tonal events inversely signify the reported imagery! This imagery seems to be the target of the tones. I have proposed (Brandt 2013) to treat the phenomenon as ‘inverted metaphor’. Anyway, the meaning of music is often experienced as related to this imaginary content.23 There is a mental space of imagery, often cosmic or meteorological, or just narrative (as in note 16), and there is a different mental space containing the sensory perception; they connect in interesting ways, not only by the possible mappings between items of each space, but also by appearing as if they were one and the same blended space: Radu Lupu weaves silken fabric, Schubert lets the sun shine (in the quotations above). The semiotic and cognitive theory of blending, or conceptual integration, offers a pragmatically relevant theory of such experiences and textual occurrences of double-space semantic descriptions.24 In this theory, there is a semiotic Base space, a situation of experience or communication, where subjects exchange signifiers, and signifieds therefore emerge in their mind according to the semioses discussed above. These signifieds are now further observed as forming spaces around themselves, instead of just appearing as isolated nominal concepts of thought. It is suggested that we think and produce meaning through processes that establish, hold and combine mental, semantic spaces where our isolated concepts are contextualized, allowing them to change and merge, thereby giving rise to new concepts.25 The most interesting aspect may be the discovery (Brandt and Brandt 2005) that the blending process must be stabilized by a schematization stemming from Base space, that is, from the reality of the thinking or communicating subject(s). If ‘this surgeon is a butcher’,26 the evaluative and affective meaning produced stems from the ethical schema of helping and harming applied to the blend, not from the generic butcher. Schemas of different kinds are activated by the immediate clash of mapped terms

174

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

(surgeon ≠ butcher!) and are semantically foregrounded in the blend. Each schema may be either generally cultural or locally social, it may be rooted in history or in phenomenology, or emerge as part of the present situation, and even be imported from ongoing dialogue, since the base space it comes from has variable ontological depth; and this pragmatic anchoring is thus a necessary condition for the meaning production in the blend to happen. This metaphor model, which can be generalized to other tropes, first of all to metonymy, shows that in blending processes, pragmatics is inherent in semantics (see Figure 8.6a–c). Whereas metaphor has input spaces whose contents are taken from different experiential domains,27 metonymy uses input from same domain in both mental spaces. Example: ‘I saw a Banksy in the street today’, meaning a graphical work by Banksy. The artist referred to is in one space, therefore called Reference Space, in the model, whereas the drawing is in the space of present perception, the Presentation Space. In the blend, the artist seems to materialize in the scenario as reincarnated in his artwork. Both metaphor and metonymy are counterfactual in this sense. But in the latter case, the schematization is of a particular nature: it expresses a respectful attitude towards an authority of some kind, positive or negative. Using a part to ‘mean’ a whole, or inversely, or a concept for a related concept, is invariably an operation schematized as an enunciative indication of attitude, in this sense. Again, this schema emerges from the speaker’s Base space, not from the semantic contents involved. In Figure 8.6a–c, I will first summarize the general model of semio-pragmatic blending, and then show the blending process supposed to structure our two examples of metaphor and metonymy. The field of application of this semio-pragmatic model is much larger than what can be discussed in this presentation. (The sentence you just read is an example of theoretical metonymy: what you see represents something that is not presented, but which is thus represented as worthy of your attention.)

FIGURE 8.6a  The blending model.

PRAGMATICS AND SEMIOTICS

175

FIGURE 8.6b  Metaphor.

FIGURE 8.6c  Metonymy.

We may, however, add the tricky case of elementary irony. Wayne Booth (1974) offers this example: A man enters his office totally soaked after having biked from home under a torrential rain, and greets his colleague with few words, something like . . . ‘Nice weather today’. Deidre Wilson (2006) has the same sort of examples: Mary (after a difficult meeting): ‘That went well.’

176

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Classical accounts define irony by formulae like this one: 1. A says p to B 2. A knows that ¬p 3. A intends B to understand p in (1) as meaning ¬p The lemmas (1) and (2) alone would define p as a lie. But the semantic miracle happens in (3), which therefore calls for a pragmatic explanation. What needs explaining is also why someone would think and talk like this at all. Note that in Booth’s example, it first has to rain heavily, and that in Wilson’s, the meeting has to be specified as difficult. We therefore need to add two more lemmas to the formula: (4) B knows that p/¬p is A’s evaluation of the situation X that p/¬p refers to, and which directly involved A28; and (5) The use of p in (1) and (3) expresses A’s frustration, to be empathically shared by B, caused by a very high degree of ¬p in X. The instantiations of the last lemma (5) often manifest a semiotic dimension, a facial or gestural expression, or a phonetic aspect, a modification of intonation, as if the speaker were quoting someone. To clarify, let us now project these observations onto the blending model. In the base space, we have A, B, the situation X (heavy rain or difficult meeting) and A’s utterance ‘p’. In the Presentation Space of the network, there is the evaluation p of X, and in the Reference Space, the contrasting evaluation ¬p of X. In the blend, p and ¬p of X somehow merge, and after an implicit schematization yield the content of the Meaning Space, ¬p! There is in fact a schema characteristic of irony; we might call it ‘the road not taken’ (see Figure 8.7a): S, the subject, follows a path that suddenly bifurcates in such a way that there is a very attractive path and a repulsive path, and the former is barred; therefore, S must take the latter, while still thinking of the other one and regretting not to have been able to be on it. Note that irony can be ‘biting’; this aspect ‘bites’ at the barrier on the p branch: something went wrong, and the cause is evident, namely that ‘you blew it!’29 In this case, the addressee is foregrounded as responsible for the barrier. Enunciation is active in this way particularly in dialogues of quarrelling. As illustrated in Figure 8.7b, the network activates the schema in the Relevance input to the blend, where p and ¬p merge, paradoxically, absurdly, but again are disambiguated

FIGURE 8.7a  The schema: ‘the road not taken’.

PRAGMATICS AND SEMIOTICS

177

FIGURE 8.7b  Irony.

by the schema and ‘resolved’ as an emphatic ¬p! to be shared in the pragmatic Base space by A and B or else by A and bystanders (C), as in polemical irony, where A and B is are antagonists. The blending mechanism is ubiquitous in language and in graphic imagination, such as cartoons, caricature drawings, advertisements and poetry. In all contexts where the expression itself is foregrounded, which is the case whenever emotion prevails, our communicative thinking tends to split the semantic content into two mental spaces interrelated as a Presentation to a Reference, and, fundamentally, as a foregrounded predicate to a known subject, in the logical, propositional sense. In poetry, where the stipulated situational context (X) is aesthetic (that is, determined as both sacred and intimate30), irony is a possible ‘strategy’ besides all sorts of tropes and linguistic and graphic operations on the form of the utterance. The primary such operation is of course the introduction of the phenomenon called verse, whether metrical or ‘free’, as an expressive gesture of disruption that may be compared to the implicit negation (¬) in irony. In poetry, a referential content r is intended behind a presentation where r is disrupted by verse, tropes, figures, graphic gestures, syntactic cut-ups and a more or less chanting vocal performance: (¬)r. We often oppose the ironic coolness and the pathetic heat in lyric contexts (poetry and songs); the poetic heat may stem from the fact that the disruption (¬)r appears in the Presentation space, whereas it appears in the Reference space of irony.31 The fundamental poetic schema may however be almost the same. The barrier blocks the enunciator from going ‘directly’ to the core of a content, which is sacred and intimate; the blocking is a call for circumlocution in a very wide sense.

­CONCLUSION: SEMIOTICS AND PRAGMATICS As we have seen, meaning is determined by both immanent and transcendent semiotic structuring: it is both conceptual and contextual. The recursion of semiosis makes it possible to understand and theorize this open but non-chaotic relation between minimal,

178

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

medial and maximal sign structures and the experiential life world that infuses social systems with meaning and lets cultural, semiosic and mental content develop as a continuity. Semiotics and pragmatics are interconnected, and their bonds are indissoluble; if cut off, pragmatics would become a part of psychology and semiotics a specialty of linguistics. The cognitive theory of mental spaces and conceptual blending needed a semiotic and, as we suggested above, a semio-pragmatic grounding in order to grow out of its initial format as a philosophical daydream. The model explained here shows how situational and experiential contributions, namely the schemas that blend activate, intervene in the sense-making, also called the meaning production, of the processes of creative thinking and signifying that semiotics studies; the pragmatic interest in signifying behaviour, on the other hand, would disappear into sociology and anthropology, or collapse into linguistics, if the semiotic insistence on iconicity, symbolicity, diagrams, enunciation and deixis did not call it back to the fundamental life of signs in human life.

NOTES 1 An example: The concept of justice means different things in the (transcendent) context of revenge or retribution and in that of jurisdiction. In itself, immanently, it contains a schema of balance or equilibrium between ‘weights’ of penalty and wrong-doing. 2 ‘the only way out is through the formulation of a general theory of language which will postulate a necessary complementarity between syntax and semantics on the one hand, and pragmatics on the other’ (A. J. Greimas in Perron and Collins 1989: 93). The paper, entitled ‘Pragmatics and Semiotics. Epistemological Observations’, was read as an introductory lecture to the International Symposium on Pragmatics and Semiotics held at the University of Perpignan from 17 to 19 November 1983. 3 Meaning is expressed in many ways, and language is only one of them, although presumably predominant. The realm of available expressive activities in the human world is extensive. For example, there are certainly reasons for developing a pragmatics of music, as of the vast family of iconic media and symbolic gestures and objects (think of architecture). Nevertheless, it may be wise to start out from considering language in particular; very few of the other forms would have evolved without it. 4 The function that ties a content to an expression, or a signified to a signifier, is an automatic mental mechanism, not a process of reasoning like the interpretation of a treatise of philosophy; I use the diagrammatic format of a triangle for a semiosis, so that the ‘coding’ mechanism can occupy an angle. This ‘little machine’ is a cognitive routine intimately related to the neural processes that make us think. We do not know yet what they are and how they causally work, but we can approach them by observing as closely as possible what they are achieving: the different forms of meaning-making that pervade the human world. 5 The principle of semiosic recursion is presented in ‘The Sign Cascade – A Basic Format of Semiotic Apperception’, in Fraser and Turner (2009). ‘From Linguistics to Semiotics: Hjelmslev’s fortunate Error’, in Hénault (2019), has a broader unfolding of this glossematic discovery. 6 Hjelmslev called this constellation a relation between a metalanguage and its object-language. The ‘meta-semiosis’ that has an ‘object semiosis’ in its content is rather the context of the embedded semiosis. This constellation is therefore of great interest to pragmatics as a study of contexts. 7 The model is explained in the chapter ‘The Role of Semio-Syntax’, in Brandt (2019).

PRAGMATICS AND SEMIOTICS

179

8 The (1983) symposium lecture, ‘Pragmatics and Semiotics. Some Semiotic Conditions of Interaction’, in Perron and Collins (1989). ‘What is being dealt with here is the résemiotization of context, or, rather, the setting up of a semiotics of situations’ (p. 101). 9 Innumerable attempts have been made, including my own, ‘Forces and Spaces’, in Brandt (2020). 10 This joke was rated and declared to be preferred over a thousand other jokes by a large American test population, as reported in the science journal Nature: “ScienceUpdate”, 4 October 2002. 11 This is a particularly fine joke for Americans. 12 Brandt (2020) has a chapter explaining this diegetic model and illustrating it by the reading of short stories by Maupassant, Borges and Hemingway. 13 This statement may seem a little blunt. What is meant is the following: words are firstly instructions for the mind to actualize concepts, whereas sentence construes imaginable scenarios. 14 Peirce’s third semantic sign type is of course the index, unfortunately a poorly elaborated type. When we perceive one event (smoke) and infer another event (fire), which is the case in all forms of perceptions, except some aesthetical forms, then the world is not signifying to us what it intends us to think, as by sending us a signifier to trigger a signified in our mind. The index is not a sign, since it is not intentional. What is happening instead is that the perception contains an inference. We have to interpolate a schema between the first categorization event (smoke) and the second (fire), either a mereological or a causal schema. These schemas are mental diagrams, and they are of the exact same constitution as our expressive diagrams. Therefore, the third sign type is the diagram, not the index. Furthermore, the index is often taken to include the phenomenon of deixis, as if the world wished to ‘point to’ an object in order to call our attention to its meaning. Deixis is instead the enunciational aspect of all expressed signs. The frame of the picture; the sign post of the traffic sign. Only mental diagrams do not have deixis. 15 This is not the standard explanation of the difference. ‘Motivation’ here means orientation towards a goal known beforehand, in casu imitation of the figurative properties of the target, or imaging. 16 Our feelings, whether they are the emotions of the moment, the moods of the day or the passions of our life, react to what we see as true. You do not get angry, if you do not feel certain you are being insulted – ‘perhaps’ is not enough. Inversely, feelings can trigger beliefs, which is why manipulation, in the unethical sense, is possible. 17 Diagrams may be the evolutionary origin of symbols and icons; gesture is often neither conventional-instructive (as greetings are) nor motivated-illustrative (as theatrical showing is), because its basic function may be inscribing the persons communicating in an imaginary space of presently shared meaning. We may find the same basic function in art, which is neither commanding nor illustrating. Roman Jakobson’s poetic function corresponds to the meaning of this basic diagramming aspect of gesture. 18 However, the ‘source of symbolic force’ still has to qualify for the iconic cult that inscribes it in the logic of enunciation. I have argued elsewhere (Brandt 2019, 2020) that the origin authorities of all kinds may be the need for and the belief in protection. Money protects, and ethnic, religious and other strong collective identities, even mafias and tribes, protect. Money was named after Juno Moneta, the protective goddess of Rome. Temples became banks. 19 This enunciative cascade would also be illustrated by a Christian priest (S1) who blesses a believer (S2) in the name of God. 20 See Small (1998).

180

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

21 The strange human habit of singing utterances instead of just humming or musicking by instruments makes the connection between music and language even stronger. Syllables are then superimposed on tones, and melodic phrases on grammatical phrases. Semantically, linguistic meanings and musical, emotional meaning will then cover each other and create strong impacts. Social life may have started as some sort of ongoing, non-fictive opera. 22 Quoted and discussed in Brandt (2013). 23 A professional violinist recalls her memory of hearing Carl Nielsen’s violin concerto for the first time at age twelve: ‘The orchestra bangs out an enormous chord, and then the violin takes over, as if it says: “I will take care of this!”, and it’s like hearing a bird you let out of the cage and which is now just flapping around in the room’ (Grit D-H Westi, Information Kultur, 29 May 2020). The music is heard as describing the imaginary scenario, not the other way around. 24 The theory of conceptual blending was developed by Turner and Fauconnier, who summarized and illustrated their approach in a very entertaining 2002 book. It was further developed by Line Brandt (2005, 2013) and the author, who expanded the model semiotically and pragmatically. 25 See also Oakley (2011). 26 The example discussed in Brandt and Brandt (2005). 27 Experiential domains are phenomenological regions of experiences that human cognition distinguishes and conceptualizes distinctly. Brandt (2004) has a first account of the phenomenon, after Sweetser (1990), who introduced the idea of basic semantic domains in order to account for meaning differences in modality. Domain difference as such was already presupposed in conceptual metaphor theory (Lakoff and Johnson 1980) but was never theorized or even discussed as an interesting phenomenon: why do we naturally develop domains of experience so that moving a categorial concept between domains changes its meaning? 28 Line Brandt (2013) discusses these and other more complex examples and pragmatically concludes on Wilson’s analysis of irony, that is, on ironic enunciation, that irony is not a propositional effect, but that it takes specific situational contexts and specific intersubjective speaker-hearer relations to achieve ironic effects. 29 Example: A and B are quarrelling. B, A’s wife, underlines her point of view by throwing the kitchen’s crockery against the walls and on the floor, creating an impressive mess. A: ‘Nice work, my dear!’. 30 See the analysis of semantic domains in Brandt (2004, 2019, 2000). 31 Poetry as such may therefore be seen as a sort of inverse irony, in this sense.

REFERENCES Barthes, R. (1964), ‘Rhétorique de l’image’, Communication, 4: 40–51. Barthes, R. (1985), The Responsibility of Forms, trans. R. Howard, New York: Hill & Wang. Booth, W. (1974), A Rhetoric of Irony, Chicago: University of Chicago Press. Brandt, L. (2013), The Communicative Mind: A Linguistic Exploration of Conceptual Integration and Meaning Construction, Newcastle upon Tyne: Cambridge Scholars. Brandt, L. and PAa. Brandt (2005), ‘Making Sense of a Blend: A Cognitive Semiotic Approach to Metaphor’, Annual Review of Cognitive Linguistics, 3 (1): 216–49. Brandt, PAa. (2000), ‘The Architecture of Semantic Domains: A Grounding Hypothesis in Cognitive Semiotics’, Revista Portuguesa de Humanidades, 4 (1–2): 11–51.

PRAGMATICS AND SEMIOTICS

181

Brandt, PAa. (2004), Spaces, Domains, and Meaning, Berne: Peter Lang Verlag, Series: European Semiotics, Number 4. Brandt, PAa. (2013), ‘Weather Reports: Discourse and Musical Cognition’, in K. Chapin and A. H. Clark (eds), Speaking of Music: Addressing the Sonorous, New York: Fordham University Press. Brandt, PAa. (2019), The Music of Meaning. Essays in Cognitive Semiotics, Newcastle upon Tyne: Cambridge Scholars. Brandt, PAa. (2020), Cognitive Semiotics: Signs, Mind and Meaning, London: Bloomsbury. Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities, New York: Basic Books. Fraser, B. and K. Turner, eds (2009), Language in Life, and a Life in Language: Jacob Mey – a Festschrift, Bingley, UK: Emerald. Hénault, A., ed. (2019), Le sens, le sensible, le réel: Essais de sémiotique appliquée, Paris: Sorbonne University Press. Lakoff, G., and M. Johnson (1980), Metaphors We Live By, Chicago: University of Chicago Press. Oakley, T. (2011), ‘Conceptual Integration’, in J. O. Östman and J. Verschueren (eds), Handbook of Pragmatics, Amsterdam: John Benjamins. Perron, P. and F. Collins, eds (1989), Paris School Semiotics: I. Theory, Amsterdam: John Benjamins. Saussure, F. de ([1915] 1972), Cours de linguistique générale, Critical edn, ed. T. de Mauro, Paris: Payot. Small, C. (1998), Musicking: The Meanings of Performing and Listening, Middletown: University Press of New England. Sweetser, E. (1990), From Etymology to Pragmatics, Cambridge: Cambridge University Press.

182

­C HAPTER NINE

Gesture Studies and Semiotics IRENE MITTELBERG AND JENNIFER HINNELL

INTRODUCTION As semiotic beings, we build our understanding of the discourses we participate in, and more generally of the world around us, by integrating signs in different modalities into dynamic meaningful ensembles. From the first exchanges between infants and caregivers, dialogue typically involves articulations in multiple modalities such as vocalizations, spoken or signed language, manual gestures, eye gaze, body posture and facial expressions. While listening to our conversational partners, we observe their facial expressions; we sense emotional qualities and inner dispositions in the ways they articulate themselves – both verbally and gesturally, and we might respond to them – both verbally and gesturally – or hold ourselves back. Put differently, spontaneous interactive discourse typically consists of concerted multimodal semiotic acts of contextualized meaning-making. In this chapter, we place the human body and its communicative behaviour at the centre of studying multimodal processes of semiosis (e.g. Peirce 1875: CP 1.337; 1907: 5.472, 5.484). We characterize various ways in which the human body functions as sign and sign-creator, bringing together semiotic and related accounts that help us understand how interlocutors gesturally indicate objects, ideas, locations, or events, and enact physical habits such as interpersonal interaction, movement patterns or object manipulation, as well as more abstract, yet deeply embodied, schemata of experience. Gestures here are broadly understood as discourse-embedded, kinesic actions that are performed with the hands and arms, head, shoulders, torso, or entire body, and have semiotic function(s). Imagine a conversation in which a friend explains to you that her plan for the weekend is to work. While she speaks, she performs a typing action in mid-air, simulating typing on an imagined keyboard. From her gesture you will likely infer that she will spend her weekend typing on an actual keyboard at an actual desk: i.e. you will infer that, here, ‘working’ means on a computer. You may also imagine the written text that will result from the work, as well as other ideas and states (e.g. your friend’s future mental or emotional state when she submits her final manuscript) resulting from her weekend of writing. This decontextualized example nonetheless shows how a gesture can evoke not only an implied object or tool, but also the associated pragmatically structured context of experience. In this chapter, we focus on aspects of the study of gesture most highly relevant to semiotics. After an overview of pioneering research in modern gesture studies, we

184

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

provide a semiotic characterization of gesture, especially as compared to more highly codified linguistic signs. Drawing on Peirce, we then demonstrate how in gestural signs basic semiotic modes such as iconicity, indexicality and conventionality/habit interact in modality-specific ways. We further highlight the inherently metonymic nature of gestures and their tendency to schematize experience. We close the chapter with a forwardlooking perspective on the field and an overview of new technologies being put to use in semiotic gesture analyses.

BACKGROUND: THREE WAVES OF MODERN GESTURE STUDIES Over the past five decades, modern gesture studies has developed into a flourishing interdisciplinary field that intersects semiotics, linguistics, anthropology, philosophy, psychology and neurology, among other disciplines (e.g. Müller et  al. 2013, 2014). While gesture was acknowledged as part of the act of oratory by Greek and Roman rhetoricians (Hall 2004), it wasn’t until the seventeenth century that a truly scholarly interest in gesture emerged that motivated a descriptive, naturalistic account of gesture (though still tied to rhetorical purposes as well). For example, Figure  9.1 features an

FIGURE 9.1  A seventeenth-century illustration of a gesture signalling antithesis (Bulwer 1644).

GESTURE STUDIES AND SEMIOTICS

185

illustration by seventeenth-century physician and philosopher John Bulwer of a gesture signalling antithesis.1 In this section, we present the past century as three ‘waves’ that constitute modern gesture studies. These waves are intended to be understood as time periods with unifying developments, rather than as discrete periods. (For a more complete history of gesture studies, see Kendon 2008, 2017; Müller 1998a; Bressem 2013a.)

The first wave: Founding treatises Founding treatises that could be considered a ‘first wave’ of the modern field include those of Wilhelm Wundt (1921, 1973), George Herbert Mead (1934, 1938), David Efron ([1941] 1972), Charles Morris (1946), Leroi-Gourhan (1964), Desmond Morris (1979), and Paul Ekman and Wallace Friesen (1969) inter alia. Here we briefly introduce the foremost contributions of several of these scholars. German psychologist Wilhelm Wundt (1921, 1973) viewed language as an interactive practice in which non-verbal behaviour played a role. His semiotic classification system emerged from his studies of the hand movements of deaf people, and co-speech gestures of Indigenous people of North America, Cistercian monks and Neapolitan speakers and featured a tertiary division between demonstrative/pointing, descriptive and symbolic gestures. Wundt’s classifications were foundational to later typologies (e.g. Efron’s and Ekman and Friesen’s). Efron’s ethnographic research ([1941] 1972) was part of the small ripple in the Zeitgeist that ran counter to linguistic structuralism, which viewed language as an abstract system and had little interest in gesture as a linguistic phenomenon. Efron addressed the question of whether different cultures use gesture differently. In his study of gestures of traditional and assimilated Italian and Yiddish speakers in New York, Efron found distinctive gestural characteristics between the traditional groups, but also concluded that the traditional gestures disappeared upon the speakers’ assimilation into the wider community. He thus suggested that gestural actions are socially shared beyond the individual and that different cultures and social groups have different patternings. Efron also presented a classification system ([1941] 1972), later refined by Ekman and Friesen (1969). Ekman and Friesen (1969) developed their classification system by investigating the full repertoire of bodily movement beyond the hands (facial expressions, posture shifts, etc.). It was based on three fundamental considerations: ‘how a [non-verbal] behaviour became part of the person’s repertoire, the circumstances of its use and the rules which explain how the behaviour contains or conveys information’ (1969: 49). They further distinguished between affect – displays of emotions largely in facial expressions; adaptors, e.g. scratching an itch or adjusting one’s glasses; regulators – now known as interactive gestures (Bavelas et al. 1995); emblems, such as the ‘victory’-sign; and illustrators – which portray speech content. Later typologies (see below) relied heavily on these distinctions.

The second wave: 1970s–90s In what could be considered a second – and pivotal – wave, the pioneering research of linguistic anthropologist Adam Kendon (e.g. 1972, 1980, 1988, 2004), psycholinguist David McNeill (e.g. 1985, 1992, 2000, 2005) and others (e.g. Calbris 1990) has greatly shaped our understanding of manual gestures as an integral part of thought processes, utterance formation and communicative interaction.

186

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Kendon presents comparative semiotic studies of the role of the body in interaction, or, to use his term, visible action as utterance (2004). His research brings together a structuralist approach to the form and function of kinesic utterances; ethnographic methods inspired by Efron ([1941] 1972) and Wundt (1921); and a focus on social interaction (Goffman 1955; Schegloff 1984; Seyfeddinipur and Gullberg 2014: 2). His studies of Indigenous sign languages of Australia (1988), gestures of Neapolitans in Italy (1995, 2004) and gesture families (2004) are among his most significant contributions and led to many original observations regarding different kinds of multimodal language (i.e. co-speech gestures and signed languages) (see also subsection entitled ‘Multi-functionality and Polysemy of Co-speech Gestures’ below and contributions in Seyfeddinipur and Gullberg’s (2014) Festschrift in honour of Kendon’s work). McNeill’s research into the interrelation between speech and gesture and on the cognitive function of gesture has formed the basis of much of his work. McNeill suggests that speech and gesture are ‘part of the same psychological structure’ (1985: 350 see also 2005 on ‘growth points’, dynamic units of online verbal thinking that combine imagery and linguistic content). Furthermore, McNeill’s (1992) Peirce-inspired gesture typology now represents one of the most widely used categorization systems in gesture research. In it, he differentiates beats – rhythmic gestures aligned with speech prosody and used mainly as emphasis markers; deictics – pointing gestures; emblems – codified gestures with a stable form-meaning relation, e.g. the ‘victory’ or ‘okay’ signs; iconics – which depict a physical action or a concrete object; metaphorics, which portray abstract entities; and cohesive gestures – repetitions of a gesture form or location across a stretch of discourse. Moving away from assigning gestural signs to a single category, McNeill (2005) later prefers to speak of mixing dimensions, such as iconicity and indexicality (see also the fourth main section below and Duncan, Cassell and Levy’s 2007 Festschrift in honour of McNeill’s work). Geneviève Calbris (1990) offered a detailed semiotic analysis of French gestures based on her elaborate coding system. Her account of meaning is truly multimodal; she examines the temporal and semantic coordination of meaningful movements of the hands, head, torso and facial mimics. She also highlighted gestures’ capacity to abstract from the concrete and to concretize the abstract (e.g. spatial representations of time), and how the motivation for gestural forms – i.e. through iconicity and analogy – involves conventionalization, including both cultural practices and cognitive schemata (Calbris 2011).

The third wave: Gesture studies as a multi-disciplinary field In a third wave, since the late 1990s, modern gesture studies has exploded into a multiand interdisciplinary field addressing questions in semiotics, psychology, linguistics, anthropology, philosophy, neuroscience and many more disciplines using a wide variety of methodologies. Here we briefly introduce research strands that emphasize the semiotics of gesture. Linguistic anthropologists Charles Goodwin (1981, 2007, 2018), John Haviland (1993, 2000), and Nick Enfield (2009, 2011, 2013) as well as interactional linguist Jürgen Streeck (1993, 2009) have focused on the situated nature of meaning, i.e. that conversation is anchored in the material world and is co-constructed dynamically by participants. Importantly, their work is also grounded in the study of a multitude of non-European languages, such as Haviland’s (1993) work on pointing gestures of the

GESTURE STUDIES AND SEMIOTICS

187

Guugu Yimithirr Indigenous community in Australia and Enfield’s (2009) collection of studies on Lao speakers of Southeast Asia. Streeck’s ecological, conversation-analytic approach employs microscopic analyses of interactions – e.g. car mechanics talking while repairing a vehicle – to study how we make meaning in and gather meaning from our environments, how we share meaning with others and how we organize our interactions (2009: 3, 1993; LeBaron and Streeck 2000). Streeck also proposed six ‘gesture ecologies’ or ‘patterns of alignments between human actors, their gestures and the world’ (2009: 7) (see discussion on ‘Gestural Practices of Sign Formation’ below and Streeck, Goodwin and LeBaron 2011). Cornelia Müller’s research, rooted in linguistic discourse analysis, has influenced the field in many ways. Müller (1998a, 2017a) showed how Bühler’s ([1934] 1982) Organon model of communication with its three functions – expressive, referential and appellative – illuminates the multi-functionality of gestures. She also introduced a widely used set of gestural modes of representation (Müller 1998a, b, 2014; see below). Her work on multimodal metaphor has demonstrated how gestures may bring out the dynamic nature of metaphor (Müller 2017b; Cienki and Müller 2008) and how speech, gesture and expressive movement are temporally orchestrated in multimodal interaction and film (Müller and Kappelhoff 2018). Furthermore, Müller (2017a, 2018) has offered insights into motivated and conventionalized facets of both gesture and sign, e.g. by introducing the notion of recurrent gestures (see also Ladewig 2011, 2014; Bressem 2014; Bressem and Müller 2014). Besides the individual scholars we have mentioned so far, there are also groups of gesture researchers whose work is anchored in a shared theoretical perspective. Here we name two schools that are particularly akin to semiotics. Cognitive Linguistics (CL) was one of the first linguistic fields to embrace gesture studies, building on a shared interest in thought processes, embodiment and multimodality (e.g. Johnson 1987; Cienki 1998, 2013; Müller 1998a; 2017b; Sweetser 1998, 2007, 2012; Taub 2001; Parrill and Sweetser 2004; Wilcox 2004; Gibbs 2005; Núñez and Sweetser 2006; Cienki and Müller 2008; Feyaerts, Brône and Oben 2017; Wilcox and Occhino 2017; Talmy 2018). There is some theoretical overlap between CL and a cognitive semiotic perspective on gesture, which similarly centralizes embodied meaning-making in the broader context of cognitive, social and neurobiological processes, but has a stronger anchorage in semiotic theory (e.g. Sonesson 2014; Zlatev 2015, this volume; Mittelberg 2006, 2008, 2013a; Wolf et al. 2017, 2018). What unites these three waves is the acknowledgement that human language, whether spoken or signed, is embodied, dynamic and multimodal and needs to be analysed and theorized as such (see also Liebal et al. 2013 for a multimodal approach to human primate communication). New interdisciplinary research strands are evolving as quickly as new technologies: e.g. the use of 3D motion capture systems, large-scale multimedia corpora and brain imaging to study gesture. Could these methodologies be leading us to a fourth wave? We return to this question to conclude the chapter.

GESTURES ACTING AS SIGNS A question central to gesture research, and to this chapter, concerns the cultural, cognitive, affective, physical, material and interpersonal factors that motivate the formation of individual gestural signs and gesture patterns, thereby lending a certain regularity to forms and functions of human communicative behaviours. In comparison to the Saussurian

188

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

understanding of language and linguistic signs (Saussure [1916] 1986), most of these regularities do not rely on highly conventionalized and socially agreed-upon codes. Thus, one of the most basic questions from a semiotic perspective is how a gesture acts as a sign. Like any material sign carrier in any semiotic process, in order for a gesture to function as a sign, it needs to be perceived and interpreted by a human mind or some other cognitive system (e.g. a machine). According to prominent American philosopher Charles Sanders Peirce (1839–1914), a ‘sign [. . .] is something that stands to somebody for something in some respect or capacity’ (c.1897: CP 2.228; see fifth main section for semiotic theory). These basic principles underlying semiotic processes require us to consider: In what ways and capacities can coverbal gestures stand for something and be interpreted as such by an addressee? Do they always clearly stand for something? And what role do the concurrent linguistic signs play?

Some prerequisites for verbo-gestural sign processes We begin by considering some factors that condition how gestures enter multimodal sign processes for both an addresser and their addressee. When interlocutors speak in face-to-face interaction, they ensure that they can be heard and can hear their interlocutors. A not-too-noisy environment and intact audio channel are prerequisites for the successful exchange of (verbal) linguistic signs. By contrast, with the exception of signers, interlocutors usually do not check whether they can see each other’s gestures (e.g. when video-conferencing) and they gesture even when their interlocutors cannot see them (e.g. on the telephone; Bavelas et al. 2008). Regarding gesture interpretation, it is thus important to take into account that many gestures are not consciously or not fully perceived by the addressee(s) of a multimodal utterance. People’s attention, rather, seems to narrow in on certain articulations (e.g. speech, head movement, gaze, gesture) in certain moments and then to change focus again to seize on others. Despite this gradient nature of gesture perception, interlocutors are clearly attending to them to some degree, as gestures enhance the understanding of oral discourse and the learning of new concepts, among other functions (e.g. McNeill 2000; Goldin-Meadow 2003). Gesture production shows a great deal of variation depending on discourse genres, personal styles, cultural practices and other factors (e.g. Müller et  al. 2013, 2014). Speakers can be more or less aware of the gestures they make while talking. The fact that speakers are often not aware of their gestural behaviour can reveal less monitored aspects of multimodal semiosis during communication (Sweetser 2007). As meaningful bodily actions influenced by cognitive and affective states, gestures provide valuable insights into the physical grounding, emotional dimensions, and socio-cultural situatedness of the semiotic processes humans rely on for thinking, imagination and communication (e.g. Streeck, Goodwin and LeBaron 2011; Müller 2017b). There are situational contexts in which gestures may become the focus of joint attention and joint intentionality (e.g. Tomasello 1999) and thus the focus of the communicative action of all participants involved, which makes it very likely that they fulfil semiotic functions for both the gesturer and the addressee(s). Such contexts include expert explanations (e.g. Streeck 2009) and language teaching (e.g. McCafferty and Stam 2008). Among the different kinds of gestural signs, pointing gestures, or deictics (McNeill 1992), in particular, need to be fully perceived by the addressee(s). Speakers use such indexical gestures to purposely direct their addressee’s attention and to indicate a specific object, location or event, or less specific targets such as broad directions (e.g. north, south) or regions in space (e.g. a mountain range) (Kita 2003; Fricke 2007; Goodwin 2007; see also discussion on ‘Contiguity – Indexicality – Metonymy’ below).

GESTURE STUDIES AND SEMIOTICS

189

Emblematic gestures (emblems, McNeill 1992, or quotable gestures, Kendon 2004) are also gestural signs that interlocutors need to attend to for a communicative exchange to be successful. Within the spectrum of less versus more conventionalized, or ‘languagelike’, gestural behaviour, emblems exhibit a high degree of conventionalization on the basis of well-formedness conditions and culturally codified meanings (see McNeill (2005) for the so-called ‘Kendon-continuum’ encompassing gesticulation-pantomime-emblemssign language). An example of an emblem is the ‘thumbs-up’ gesture signifying (in many Western cultures) approval and positive appraisal, formed with an outstretched thumb extended about 90 degrees away from the fingers, which are curled into a fist (shown in Figure 9.2). While it can signal approval without relying on the semantics of the concurrent speech, the interpreter still needs to consider the immediate discourse and situational context to understand what the speaker-gesturer is approving of. In Figure 9.2, we see a thumbs-up emblem produced with both hands by US Vice President Kamala Harris to give positive feedback to her student audience.2 In light of the highly conventionalized and codified nature of emblems, they seem to be, strictly speaking, the only type of gesture that can be adequately described with

FIGURE 9.2  Thumbs-up emblem signalling approval. US Vice President Kamala Harris gives a thumbs up to students at Miller Elementary School, Dearborn, Michigan, Monday, May 6, 2019. Copyright: Associated Press 2019.

190

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Saussure’s model of the linguistic sign ([1916] 1986). That is, reference is afforded via coded form-meaning pairings, i.e. between a signifier that is unequivocally assigned to a signified as in the thumbs-up gesture signifying approval. For the same reason, emblems (but not all gestures) may count as symbols in Peirce’s understanding of the term (see below on ‘Gesture and semiotic theory’). While popular understandings of the term gesture are often associated with emblems, they make up only a sub-group of gestures and subsequently also of gesture research (for cross-cultural uses of emblems, see, e.g. Calbris 1990; Brookes 2004; Kendon 2004; Müller and Posner 2004; for an overview of research on emblems, see Teßendorf 2013). This chapter is mainly concerned with spontaneously produced co-speech gestures that show a larger and more varied range of forms, functions and degrees of conventionalization.

Multi-functionality and polysemy of co-speech gestures Gestures typically fulfil more than one communicative function and therefore often defy neat categorization. Hierarchical understandings of semiotic processes (Jakobson 1960; Peirce (1903: EP 2.289–99); Bühler [1934] 1982) have been adapted to account for the multi-functionality of gestures, which often combine in varying degrees iconicity, indexicality and conventionality (Peirce c.1895: CP 2.302; McNeill 1992, 2005), and pragmatic and interactive functions described beyond Peirce-based paradigms (Bavelas et  al. 1995; Müller 1998a; Kendon 2004). Depending on local pragmatic forces and contextual factors, one of the modes comes to the fore, thus determining the predominant function of a given gestural sign. The non-dominant functions still contribute to the overall meaning of the utterance, e.g. referential functions may layer with the stance of the speaker towards the content of the utterance (see Müller (1998a, 2014), drawing on Bühler; Mittelberg (2013a) and Mittelberg and Waugh (2009), drawing on Jakobson and Peirce; and Debras (2017) and Hinnell (2020) on the multimodal expression of stance). The following example from a teaching context illustrates how gestures act as signs in multimodal utterances. When saying ‘there is the main verb’, the speaker in Figure 9.3 (adapted from Mittelberg 2013a), a linguistics professor, is drawing the students’ attention to the verb form ‘taught’ written on the blackboard behind him by creating a gestural index with his right arm and hand. Here the linguistic index ‘there’ and the gestural index pointing at the written form ‘taught’ jointly establish reference; they are semantically integrated (e.g. Cooperrider et al. 2021). Meanwhile, the speaker’s left hand exhibits basic form features of a receptacle representing ‘the main verb’ at a higher level of abstraction. Without considering the speech content, we would not know that the cupped hand does not portray a physical object, but, rather, a grammatical category. By representing an abstract entity through a physical structure, this form qualifies as a metaphoric gesture (McNeill 1992; Sweetser 1998; Cienki and Müller 2008; see also section ‘Gesture and Semiotic Theory’ below). Notably, these gestures exhibit both similar and different semiotic characteristics and functions within the multimodal utterance. Both are polysemous sign carriers that depend on linguistic and other contextual information to assume a specific, contextualized meaning; yet, different degrees of context-dependence are at work. The meaning of indexical signs is categorically highly context-sensitive, a fact well established for pronouns, demonstratives and other function words in language (or shifters, Jakobson 1957). Pointing gestures similarly depend more fully on the linguistic

GESTURE STUDIES AND SEMIOTICS

191

FIGURE 9.3  Pointing gesture (‘there is’) and metaphoric container gesture (‘the main verb’).

and extralinguistic context and cause the attendee’s attention to shift from the hand itself to the entity pointed at. By contrast, iconic and metaphoric gestures typically exhibit form features which, taken by themselves, evoke content, including embodied schemata (see Johnson 1987; Cienki 1998, 2013; Mittelberg 2006, 2018). Focusing only on the gestural part of the utterance above, we see that the gestural index refers to something external to the speaker’s body, while the container gesture also is what it signifies, a physical structure in the form of a receptacle (e.g. Merleau-Ponty 1962: 216; Mittelberg 2013b: 340). In the speech content, we observe another difference between these two multimodal sign processes. Whereas the referent of the linguistic deictic expression ‘there is’ can only be identified with a simultaneous gestural index, the noun phrase ‘the main verb’ could be understood without the container gesture illustrating it. Additional semiotic support comes from the written form ‘taught’ that represents an exemplar of main verbs. All in all, the speech content, the visual information on the board and the two coinciding gestures contextualize each other, thus co-constituting a multimodally performed semiotic act. Thus far, we have introduced the semiotic complexity that characterizes dynamic multimodal sign processes. Coverbal gestures are typically polysemous and multifunctional signs that assume their local meaning based on their own semiotic characteristics and in conjunction with concurrently uttered speech and other contextual factors. While questions of referentiality (e.g. Deacon 1997) are crucial to understanding how gestures act as signs, this chapter cannot provide a full treatment of this complex issue (but see ‘Gesture and Semiotic Theory’ below).

192

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

BODILY SEMIOTIC ACTION AND MULTIMODAL LANGUAGE Co-speech gestures are often described as part of visual communication or visual language (e.g. Sandler, Gullberg and Padden 2019). While they clearly constitute the visible, observable part of language use in interaction, a semiotic characterization of gesture needs to account for the fact that they are, compared to static visual signs such as drawings, paintings or hieroglyphs, more than merely visual in nature. Here, we address the role kinesic action plays in the semiotics of gesture and discuss connections to signed languages, thus broadening the scope of multimodal language.

Kinesic, visual and imagined facets of gestural signs Kendon’s (2004: 7) term visual action as utterance highlights the fact that gestures are dynamic bodily actions. While utterance stresses gestures’ affinity with verbal expression, here the term suggests that gestures can also be seen as utterances in their own right (Seyfeddinipur and Gullberg 2014). Drawing on Jakobson’s notion of motor signs (1972: 474), Mittelberg (2019a) recently introduced the notion of visuo-kinetic signs to capture the fact that gestures – as well as the various articulations in signed languages – are conditioned by the kinesic and sensory affordances of the speaker’s body and its interactions with the environment. In this understanding of gestures, semiosis (Peirce 1907: CP 5.472, 5.484) and embodiment (Varela et  al. 1991) are intimately linked: experientially grounded embodied schemata and force dynamics (Johnson 1987) are assumed to underpin both the mindful production and interpretation of bodily signs, thus allowing for (inter-)subjective understanding (see Mittelberg 2013a on the exbodied mind). Honing in further on bodily articulations, Boutet, Morgenstern and Cienki (2016) proposed a fine-grained kinesiological account of gestures in their examination of gestures reflecting grammatical aspect. Gestures’ deep grounding in physical actions and social interactions explains why speakers readily and effortlessly use their hands to hold or move around real or virtual objects (see, e.g. Kita 2000; studies in Streeck, Goodwin and LeBaron 2011). Participants further co-construct contextures of action with their conversational partners (Goodwin 2011, 2018). In principle, however, a gesture is a gesture – i.e. a reduced and more schematic bodily action with semiotic function – precisely because gesturers typically only pretend to be manipulating objects, touching surfaces, tapping on someone’s shoulder, etc. (see Clark (2016) on depictions; Hostetter and Alibali (2008) on simulated actions in gesture and Müller (1998a/b, 2014) on acting). When speakers do ‘pull’ real persons, physical objects or tools into their gestural actions, these elements become part of the respective visuo-kinetic sign process and thus themselves fulfil semiotic functions: they stand for someone or something else. Although the verbo-gestural co-expression of propositional content has always been a focus of gesture research (e.g. McNeill 1992; Kendon 2000, 2004), the action affinity of gestures has also offered insights into how gestures may do something in their own right. Examples include brushing away an argument (e.g. Bressem and Müller 2014), fending off a rhetorical attack (e.g. Wehling 2017) or inviting an interlocutor to take the floor (e.g. Holler and Bavelas 2017). Despite gestures’ visual and bodily mediality, a challenging factor in gesture interpretation and analysis resides in the fact that the ‘semiotic material’ we are looking at often provides rather sparse and short-lived imagery (e.g. Arnheim 1969). Gestures

GESTURE STUDIES AND SEMIOTICS

193

often consist only of quickly performed motion-onsets or furtive schematic figurations abstracted from, for example, the full action routine, object or spatial constellation (e.g. Müller 1998a, 2014; Mittelberg and Waugh 2014). Moreover, while a gestural form description starts with the observable physical components such as body posture, hand shape, palm orientation, finger configuration, and the position and action of gesturing hands (e.g., Bressem 2013b), gestural sign carriers also tend to involve facets that are not directly observable, yet are also signifying. These may include the imaginary interlocutors, objects or surfaces mentioned above, but also invisible movement traces or points set in gesture space, all of which need to be metonymically inferred from the visible, physical elements and actions (Mittelberg 2019a; see also below). For instance, the typing gesture described earlier necessarily implies an imagined keyboard, which is not itself gestured. A thorough gesture analysis thus needs to include elements of multimodal sign processes that are not visual, or visible, but still contribute to a gesture’s form, meaning, pragmatic function and kinesthetic feel.

Gestural practices of sign formation We now turn our attention to how communicating hands create gestural signs. Rudolf Arnheim gives a succinct impression of the ‘intelligent’ abstractions speakers perform with their gestures: By the very nature of the medium of gesture, the representation is highly abstract. What matters for our purpose is how common, how satisfying and useful this sort of visual description is nevertheless. In fact, it is useful not in spite of its spareness but because of it. Often a gesture is so striking because it singles out one feature relevant to the discourse. [. . .] The gesture limits itself intelligently to emphasizing what matters. (Arnheim 1969: 117) Spontaneously produced gestures like the ones described by Arnheim, particularly those that portray an aspect of an object, action or idea, may take shape through employing different kinds of semiotic practices. Kendon (2004), for example, distinguishes between different techniques of representation, namely modelling a body part to stand for something else, enacting certain features of an action pattern, or depicting objects through hand movements recognized as sketching or sculpting the shape of something. Müller (1998a, b) introduced four modes of representation in gesture, drawing on tools, media and mimetic techniques stemming from the visual arts: drawing (e.g. tracing the outlines of a picture frame); molding (e.g. sculpting the form of a crown); acting (e.g. pretending to open a window); and representing (e.g. a flat open hand stands for a piece of paper; Müller 2014). Some gesture scholars specifically differentiate between gestures that carefully depict a particular space or referent object from those gestures that seem to reflect a concept or an idea, for example, via metaphor (e.g. Calbris 1990; McNeill 1992; Fricke 2012). Streeck (2009: 151–2) distinguishes between depicting (e.g. via a gesture portraying a physical object) and ceiving (e.g. via a gesture conceptualizing a thematic object, such as an experience or idea), among others, attributing the latter mode to a more selfabsorbed, conceptually driven way of finding a gestural image for an emerging idea. Clark (2016) presented a detailed multimodal account of three main modes of communication including depicting, describing (with arbitrary symbols such as words) and indicating

194

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

(using indices such as pointing and placing). See Ferrara and Hodge (2018) for a review of the techniques addressed in this section.

Comparative semiotics: Co-speech gesture and sign language Reflecting on semiotic action and the multimodal nature of language leads us to consider, though briefly, how co-speech gestures relate to signed languages. In a historical context in which sign languages were long not accorded the same status as spoken languages, original accounts of signed languages comprised phonological descriptions in the structuralist approach that characterized linguistics at the time (e.g. Stokoe [1960] 2005 on ASL). The theoretical ‘rapprochement’ (Wilcox and Occhino 2017: 111) between the study of signed languages and gesture studies has occurred over the last twenty-five years (Kendon 1988, 2008; Armstrong, Stokoe and Wilcox 1995), a period in which cognitive approaches have come to play a greater role in studies of signed languages (Wilcox and Occhino 2017). This, in turn, has brought into focus the semiotic mechanisms that play a role in both gesture and signed languages. Gestures and signs in signed languages share similarities, yet also differ, in how they act as signs (e.g. Goldin-Meadow 2003; Liddell 2003; McNeill 2005; Sweetser 2009; Perniss, Thompson and Vigliocco 2010; Goldin-Meadow and Brentari 2017; Müller 2018; Perniss 2018). Regarding their semiotic status, signed languages compare to spoken languages in that they are highly conventionalized sign systems, each with a coded phonology, lexicon and grammar (e.g. Wilcox and Occhino 2017). With respect to their semiotic resources, gesture and signs share largely the same set of kinesic articulators and the same primary lieu of meaning-making, namely, the body-centred, three-dimensional signing/gesture space. However, in contrast to spoken discourse and concurrent ‘cospeech’ gestures, sign languages are entirely visuo-kinetic multimodal sign systems. The hands do most of the semiotic work, but in a composite manner with other articulations such as facial expressions. In every instance of signed interaction, several symbolic signs thus need to be simultaneously visually perceived by the addressee and semantically and grammatically integrated. The study of emerging sign languages, or homesign (e.g. Morford and Kegl 2000) can help us better understand how physical semiotic resources are dynamically coded by their users and thus develop into a multimodal language. Sign languages also comprise gestural strata and components (Liddell 2003; Kendon 2008; Wilcox and Occhino 2017). For example, some signs enter a sign language as a gestural form, e.g. as the substrate for lexical signs (Janzen and Shaffer 2002), which can grammaticalize to grammatical signs, such as is the case with modals (Wilcox and Shaffer 2006). By the same token, co-speech gestures may exhibit various degrees of grammaticalization (e.g. Fricke 2012; Mittelberg 2017b; Zima and Bergs 2017; Müller 2017a, 2018; Harrison 2018; Hinnell 2019, 2020; Ladewig 2020). There are also less codified articulations in sign language that may be ascribed a gestural quality. Constructed action, for example, is a widely used strategy for reporting narratives and for event enactment in which signers ‘encode different perspectives, either simultaneously or sequentially relaying information about referents by pivoting their gaze, shoulders, or bodies or by changing facial markers’ (Kurz, Mullaney and Occhino 2019: 90). Some sign linguists have also argued that certain sign forms are blends of conventional signs and gestural locations, given that the locations are not codified and are therefore part of gesture (e.g. a point in space to index a discourse referent) (Liddell 2003).

GESTURE STUDIES AND SEMIOTICS

195

We now turn to the basic semiotic modes that are at work in both co-speech gestures and signed language.

GESTURE AND SEMIOTIC THEORY In this section, we hone in on one of the fundamental semiotic theories applied to gesture and multimodal sign processes, namely, Peirce’s semiotic. We also draw connections to Saussure and Jakobson, as well as to the fields of cognitive linguistics and cognitive semiotics, which share basic assumptions with Peirce regarding embodied patterns of experience and expression. A Peircean perspective on multimodal interaction can account for both highly symbolic sign systems, such as spoken and signed languages, and less codified, dynamic visuo-spatial modalities, such as coverbal gestures. In Peirce’s pragmatist doctrine of signs, semiosis and cognition are tightly intertwined: ‘we think only in signs’ (Peirce c.1895: CP 2.302). Given the central question in gesture research of how gestures partake in expressive and cognitive action during communication, Peirce’s semiotic has informed work done in several intersecting fields. These include cognitive semiotics (e.g. Mittelberg 2006, 2008, 2013a, 2019a, b; Sonesson 2007, 2014), linguistics (e.g. Fricke 2007, 2012; Andrén 2010; Bressem 2014), anthropology (e.g. Haviland 1993, 2000; Enfield 2009, 2011, 2013), philosophy (Maddalena 2015), psycholinguistics (e.g. McNeill 1992, 2005), psychology (Clark 1996, 2016), and social neuroscience (e.g. Wolf et al. 2017).

PEIRCE’S TRIADIC SIGN MODEL APPLIED TO GESTURE Peirce’s widely cited definition of the sign reads as follows: A sign, [in the form of a] representamen is something which stands to somebody for something in some respect or capacity. It addresses somebody, that is, creates in the mind of that person an equivalent sign, or perhaps a more developed sign. That sign which it creates I call the interpretant of the first sign. The sign stands for something, its object. It stands for that object, not in all respects, but in reference to some sort of idea, which I sometimes called the ground of the representamen. (Peirce c.1897: CP 2.228; italics in original) The term Representamen refers to the material form the sign takes, e.g. a spoken word or a gesture; Interpretant refers to the response evoked in the mind of the sign receiver. Sign interpreters are seen as agents actively involved in meaning-making. Whether one recognizes a gesture as a sign and what one sees in it depends on one’s semiotic history, notably, socio-cultural knowledge, code(s), and other patterns of perception and interpretation, including both coded and less symbolic gestural practices (e.g. Mittelberg 2006, 2019b; Fricke 2007; Enfield 2013; Wolf et al. 2017). Peirce’s wide understanding of a semiotic Object encompasses both physical objects/ actions and abstract entities/processes, including qualities, feelings, relations, concepts, mental states and possibilities. As is true for any semiotic material or system, in gesture, too, the medium-specific affordances of the hands, arms and other physical articulators determine what may be represented or indicated, and how (see discussion on ‘Gestural Practices of Sign Formation’ above). Imitating manual typing on a keyboard comes

196

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

naturally. Whereas a hand can readily become a small receptacle (e.g. Figure 9.3), huge Objects such as buildings need to be cognitively and physically brought down to human scale in order to describe them with gestures. Some Objects thus do not lend themselves well to depiction or enactment via gesture; colours, for example, may be gesturally represented in terms of their vividness rather than actual colour (for how Peirce’s notions of the immediate and dynamic Object apply to gesture, see, e.g. Mittelberg (2006, 2013a) and Enfield (2013)). Peirce’s concept of the Ground highlights the fact that sign vehicles represent Objects not in their entirety, but only with respect to some perceptually salient and/or pragmatically relevant aspect. By entering the semiotic process, these abstracted and foregrounded features of the Object function as the Ground of the Representamen (Peirce c.1897: CP 2.229; Sonesson 2007: 40). Accordingly, gestural sign processes have been devised based on different semiotic grounding mechanisms following Peirce (e.g. Mittelberg 2013a; Mittelberg and Waugh 2014; Sonesson 2014). In what follows, we will discuss how the basic Representamen-Object correlations icon, index and symbol (Peirce 1893: CP 2.275) have been described in relation to coverbal gestures.

Similarity – iconicity – metaphor Similarity generally builds the basis for iconic signs. According to Peirce (1903: CP 2.276), ‘icons have qualities which “resemble” those of the objects they represent, and they excite analogous sensations in the mind’. Although there is a visual bias in the term icon, it encompasses a multimodal understanding of iconicity and thus also includes those ‘sensations’ that cause something to feel, taste, look, smell, move, or sound like something else. A gesture with a highly iconic ground gives a partial image of its Object based on a perceived or construed similarity (e.g. Andrén (2010), Lücking (2013), Mittelberg (2014) and Sonesson (2014) for more theoretical insights into gestural iconicity). In gesture, similarity may reside in structural resemblance, which has also been described in terms of isomorphism (e.g. Kita 2000; Mittelberg 2006; Fricke 2007). Similarity may also motivate how speakers foreground an expressive quality of a movement or a phase of an action routine (e.g. Müller 1998b; Mittelberg 2019a; also below). For example, in McNeill’s (1992: 12) well-known example of a speaker retelling the action of a cartoon character in conjunction with the utterance He grabs a big oak tree and he bends it way back, the gesturer first performs a grabbing action and then a pulling backward action with his right hand and arm. As McNeill (2005: 6–7) points out, ‘the gesture has clear iconicity – the movement and the handgrip; also a locus (starting high and ending low) – all creating imagery that is analogous to the event being described in speech at the same time’. In the McNeillean tradition, one focus of analysis has been on how speech and gesture encode different aspects of motion events, e.g. path and manner of movement, and how these strategies reveal patterns that correlate with typological differences (e.g. studies in Duncan, Cassell and Levy 2007; for an overview of work on iconic or representational gestures, see Mittelberg and Evola 2014). Iconic gestures also reflect the viewpoint strategies speakers adopt when describing events or experiences (e.g. McNeill 1992). A speaker can describe a scene from the inside, imitating the actions of a character (character viewpoint) or adopt an external viewpoint on the event and describe, for example, the motion path of a character (observer viewpoint). Speakers may also express multiple viewpoints on the same experience by employing several bodily articulators (see Calbris (1990) on body segments; Dudis (2004) on body

GESTURE STUDIES AND SEMIOTICS

197

partitioning in ASL; Parrill (2009) on dual viewpoint; and Stec (2012) for an overview). From a semiotic perspective, viewpointed iconic gestures are indexically conditioned, as they reveal the conceptual anchorage and/or physical perspective from which the gestural sign is created (e.g. Sweetser 2012; Mittelberg 2017a). Peirce’s distinction between three subtypes of icons (i.e. hypoicons; Peirce 1903: CP 2.276) – images, diagrams (i.e. icons of relations) and metaphors (implying a parallelism; Peirce c.1897: CP 2.228; Mittelberg 2014) – captures the multi-dimensional nature of gestural imagery. Image iconicity resides, for instance, in iconic gestures depicting a character’s actions, as in McNeill’s tree example above. Full body enactments of an action or object – e.g. a child ‘becoming’ a helicopter, with his torso representing the body of the helicopter and his arms the rotating blades (Bouvet 1997: 17) – show more iconic form substance than a virtual outline of an object, such as a picture frame sketched in the air (Müller 1998a,b). To further illustrate these tendencies, consider the gestural portrayal of a painting by Paul Klee in Figure 9.4 (‘Dance of a Mourning Child’, adapted from Mittelberg 2013b). As the video still in the centre image reveals, the speaker adopts character viewpoint when imitating the figure in the painting by mirroring its posture, tilted head and downward gaze.3 Here, the speaker’s entire body functions as an image icon with a high degree of iconicity. By contrast, the motion-capture plot on the right (Natural Media Lab, RWTH University, Aachen) highlights the balanced image-iconic structures the speaker created with both hands. Visualizing and freezing the motion traces produces static iconic signs of the eye slits (at the very top of the mocap plot) and the heart-shaped mouth, which the speaker drew on her own face, as well as the stretched-out arms and the skirt flowing around her legs (for a detailed image schema analysis of the painting and the multimodal description, see Mittelberg 2013b). Diagrammatic iconicity underpins gestures that trace connections between two or more locations in gesture space to schematically represent, for instance, kinship relations (Enfield 2009), syntactic structure (Mittelberg 2008) or a travel itinerary, as in Figure 9.5. Consisting of visualized motion traces, this gestural diagram is a digitally engendered iconic sign of a diagrammatic gesture (Schüller and Mittelberg 2016). The speaker on the right summarizes the itinerary of a train trip through Europe she and her

FIGURE 9.4  Painting (P. Klee, Dance of a Mourning Child 1922 as stimulus; speaker’s body as icon of figure in painting [video still]; iconic figurations [MoCap plot]).

198

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 9.5  Motion-capture gesture diagram of travel itinerary: Stockholm, Prague, Kiev, etc.

conversation partner agreed upon: ‘I think then we’ll go like this, you know, from (-) there to there across down and then to there and then back home again, no?’4 Given the high number of indexical function words in the verbal utterance, this synopsis can only be fully understood in the context of the incrementally emerging diagram, the preceding multimodal discourse and aspects of the common ground shared by the interlocutors (e.g. Holler and Bavelas 2017; for a detailed analysis, see Mittelberg and Rekittke 2021). An example of metaphor iconicity motivating a gesture is the open cupped hand discussed earlier (Figure 9.3) that represents an abstract category in the form of a small container. In Peircean terms, this metaphoric gesture represents a parallelism between an abstract semiotic object and a physical Representamen (c.1897: CP 2.228). According to Müller and Cienki (2009), this gesture is an instance of a monomodal metaphor, for the concurrent linguistic expression ‘main verb’ is not metaphorical, but technical in nature (Mittelberg 2008). Put into cognitive linguistic terms, the speaker here enacts the primary metaphor categories are containers, which is deeply rooted in experience (Grady 1997). Here, the embodied image schema container (Johnson 1987) provides structure that motivates the gestural form. (For more research on metaphor in gesture, mainly from a cognitive linguistic perspective, see, e.g., Calbris 1990, 2011; Cienki 1998; Sweetser 1998, 2007; Parrill and Sweetser 2004; Núnez and Sweetser 2006; Cienki and Müller 2008; Mittelberg 2008; Müller and Cienki 2009; Müller 2017b; Pagán Cánovas et al. 2020; for metaphor in signed languages: see, e.g., Taub 2001; Wilcox and Occhino 2017.)

Contiguity – indexicality – metonymy Contiguity relations are born out of a factual (physical or causal) connection between the semiotic Object and the sign carrier, notably physical contact, but also temporal and spatial proximity or distance. Contiguity underpins signs with a predominant indexical

GESTURE STUDIES AND SEMIOTICS

199

Ground (Peirce 1901: CP 2.306; Sonesson 2014: 1992). According to Peirce (c.1897: CP 2.228), ‘an Index is a sign which refers to the Object that it denotes by virtue of being really affected by that object’. As demonstrated above (see discussion on ‘Gestures Acting as Signs’, also Figure 9.3), the spatial orientation of pointing gestures is indeed affected by the location of their target Object. It is through the very act of pointing that the Object is established via a visual vector. Whereas the highly indexical gesture shown in Figure  9.3 clearly points at the intended referent located nearby, thus instigating a contiguity relation to it, other instances of pointing may be more ambiguous, evoking several possible meanings (e.g. Goodwin 2014), including abstract pointing (McNeill, Cassell and Levy 1993) and pointing at previously produced signs (e.g. Fricke 2007). Form variants of the indicating hand and arm have been shown to fulfil various pragmatic functions (cf. Clark 2003; Enfield, Kita and De Ruiter 2007; Talmy 2018; Hassemer and McCleary 2020). While pointing gestures can be observed across languages and cultures, field work in various parts of the world has demonstrated that pointing practices and conventions are not universal, but rather vary considerably and also include non-manual pointing (cf. Haviland 2000; Kita 2003; Wilkins 2003; Kendon 2004; Enfield 2009; Cooperrider, Slotta & Nuñez 2018; Cooperrider et al. 2021). We emphasize here that besides pointing, all gestures are, in principle, indexical to a greater or lesser extent, in that their meaning and pragmatic functions always depend on the speaker’s body’s spatio-temporal anchorage in a real or imagined context (Sweetser 2012). Indexicality thus conditions any gestural act of meaning-making. It also resides in interactive gestures (Bavelas et  al. 1995) and environmentally coupled gestures (Goodwin 2007). Furthermore, indexicality tends to interact with iconicity, not only regarding viewpoint (as discussed above), but also in gestural diagrams (Mittelberg and Rekittke 2021), gestural enactments of contrast (Hinnell 2019, 2020; see also below) and metonymy in gesture (Mittelberg and Waugh 2014), for instance. Jakobson’s (1956) balanced theory of metaphor and metonymy directly builds on Peirce’s notions of similarity and contiguity and shows how these fundamental modes of association and signification interact in linguistic and visual signs. Jakobson’s work, particularly his distinction of inner and outer contiguity (Jakobson and Pomorska 1983), is well suited to illuminate the experientially motivated and inherently metonymic nature of gestures (addressed in the third section above; see also Mittelberg and Waugh (2009, 2014). Here we only briefly treat the main principles. Jakobson understood inner contiguity (giving rise to internal metonymy) to underlie part-whole relationships (e.g. of a physical structure such as the human body). The cupped hand shown in Figure 9.3 metonymically represents a solid container. Brief action onsets or schematic movements may further evoke the full action routine in question. For example, Mittelberg (2019a) describes a speaker performing a swimming gesture that is metonymic in more than one respect: a stylized and reduced iconic movement of her arms evokes a full-fledged swimming action, with the leg movements left out. Outer contiguity (giving rise to external metonymy) holds among elements that are spatially, temporally or pragmatically correlated within a situational or semiotic context (Jakobson 1956). Let us consider Figure 9.6, which shows the same linguist as in Figure 9.3 continuing to explain the relation between main verbs and auxiliaries. Here, he stresses that the auxiliaries ‘have, will, being and been [. . .] must all belong to some subcategory’. To interpret this gesture, we must first see the speaker-gesturer as a dynamic image icon of someone holding something (based on internal metonymy) and then draw on the outer contiguity relation (contact/adjacency) between the vertical palms of the

200

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 9.6  Metonymy: Contiguity relations between hands and the virtual object they seem to be holding. Metaphor: Imaginary object represents the abstract entity (‘subcategory’).

hands (i.e. the source of the metonymic mapping in cognitive linguistic terms) and the virtual object between them (i.e. the metonymic target). Here we witness an instance of cross-modal metonymic inferencing, for the verbal expression ‘subcategory’ causes our attention to shift from the manual action of holding something (not referred to verbally) to the imaginary object being held. This gesture thus enacts a frame-metonymy (Dancygier and Sweetser 2014) and, more specifically, the metonymic mapping actionfor-object involved in action or presentation-for-presented (e.g. Panther and Thornburg 2003). Further examples of external metonymy in visuo-kinetic signs include speakers presenting reified discourse contents to be imagined on a palm-up open hand or between a hand’s index and thumb (Mittelberg 2006). (Re-)creating contact with the environment, these kinds of gestures exhibit comparably low degrees of indexicality, yet ‘point’ to the virtual object, tool, surface or space ‘at hand’. Depending on their respective shape, size and affordances, the (imaginary) elements resonate in a gesture’s particular hand shape and action pattern. Hence, such signs hence are – to return to Peirce’s notion of an index – dynamically affected by the Object. Finally, to arrive at the abstract grammatical category of the multimodal explanation in Figure 9.6, it is crucial to realize that metonymy leads into metaphor (Mittelberg and Waugh 2009, 2014) in such gestural actions that intuitively draw on familiar scenes of

GESTURE STUDIES AND SEMIOTICS

201

experience (Fillmore 1982). In short, the imaginary object functions as both the target of the metonymic mapping as explained above and the source of the interlaced conceptual metaphor ideas are objects or categories are containers (Lakoff and Johnson 1980; Dancygier and Sweetser 2014). Hence, the metaphoric target, i.e. the subcategory, is construed as a tangible object that can be intersubjectively shared with the student audience (see Mittelberg 2019a; for comparable processes in signed languages see, e.g., Taub 2001; Wilcox and Occhino 2017).

Conventionality – symbolicity – conceptual schemata According to Peirce (1903: CP 2.249), ‘[a] symbol is a sign which refers to the Object that it denotes by virtue of a law’. Strictly speaking, only codified emblematic gestures are, as noted earlier, truly symbolic signs, which is also why Saussure’s ([1916] 1986) model of the linguistic sign seems far too rigid when it comes to gesture. Rather, different levels and sources of both conventionality and iconicity tend to interact in communicative actions of the human body, thus driving the emergence and routinization of motivated signs (e.g. Calbris 1990 on cultural clichés in French gestures; see also Posner 2004). These gesture-based insights are valuable in the context of the long-standing debate regarding the structural integration of iconicity in predominantly symbolic linguistic signs. As Jakobson ([1966] 1990) claimed in his work based on Peirce, and as subsequent work on spoken and signed languages has clearly shown, iconicity is a general feature of language at all levels of linguistic structure (Jakobson and Waugh [1979] 2002; Perniss, Thompson and Vigliocco 2010; also above). As far as symbolic indices in gesture are concerned, they occur mainly in the form of pointing, which, as Peirce (c.1903: CP 2.262) maintained, may build a constitutive part of a (symbolic) proposition (e.g. Stjernfelt 2014). Compared to law-like conventionality, Peirce’s notion of habit (e.g. Peirce 1902: CP 2.170; Nöth 2016) seems particularly suited to accounting for gradually routinized correlations between recurring gestural forms and their functions (Mittelberg 2006). Peirce’s universal categories of perception and experience have been shown to illuminate the multi-dimensionality of gesture (e.g. Fricke 2007, 2012) and, particularly, the habitbased emergence of different kinds of gestural patterns and schematicity: Mittelberg (2019b) distinguishes between gestural enactments of habits of feeling (Firstness), habits of acting (Secondness) and habits of thinking (Thirdness) (see also Sonesson 2014; Nöth 2016). Accordingly, Thirdness-laden properties of gestures should allow interpreters to discern some of their semantic/pragmatic functions without considering the concurrent speech (see also Wolf et al. (2017) on perceived conventionality). Embodied image schemas and conceptual metaphor (e.g. Johnson 1987) mediate between experience, thought and expression and have thus been shown to motivate the schematic form and meaning of many iconic and metaphoric gestures (e.g. Cienki 1998, 2013; Mittelberg 2008, 2013a/b, 2018, 2019b; Ladewig 2011; Wehling 2017). Besides the basic schemata container (Figure 9.3) and object (Figure 9.6) illustrated above, and balance, to be discussed below, further schemata prone to manifest – and interact – in gesture include path, boundary and cycle, the spatial relation schemata up-down, nearfar and left-right (Lakoff and Johnson 1999) and force schemata such as resistance and blockage (Johnson 1987; Talmy 1988). Compared to these abstract schemata structuring certain gestures in adult speakers, the mimetic schemas introduced by Zlatev (2005) – such as jump, kick, grasp, push and hit – are anchored in specific physical actions. They play a fundamental role in the cognitive and linguistic development of children and ‘help

202

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

explain most literally the grounding of both communication and thought through action and imitation’ (Zlatev 2014: 2; see also Andrén 2010; Cienki 2013). It is precisely because most gestural signs are not fully coded, but combine different semiotic modes to quite varying degrees, that one of the central goals of modern gesture research has been to identify gestural forms that exhibit relatively high degrees of patterning and conventionalization. Gestures that are frequently used across individual speakers, speech genres and situational contexts belong to gesture families (e.g. Kendon 2004), gesture ecologies (Streeck 2009) and recurrent gestures (e.g. Müller 2004, 2017a; Ladewig 2011, 2014; Bressem and Müller 2014). Besides their tendency to develop pragmatic functions, some of these routinized forms may become grammatical markers and/or feed into multimodal constructions (e.g. studies in Zima and Bergs 2017; Hinnell 2018, 2019, 2020). This section has focused on how semiotic theory and gesture analysis can illuminate one another. In what follows, we provide insights into how theoretical questions and empirical work can advance each other in additional ways and on a larger scale.

RECENT EMPIRICAL RESEARCH STRANDS As discussed above, we are now in what we have identified as a third wave of modern gesture studies. In this section, we introduce technologies that are being used to explore semiotically oriented research questions in multimodal communication, such as largescale multimedia archives, fMRI imaging, and 3D motion capture and a sampling of research projects using these technologies. Empirical gesture research depends on the time-intensive task of producing annotated speech and gesture transcripts that allow for the analysis of the synchrony of co-occurring linguistic and gestural forms. First, one transcribes the speech and segments the gestural utterance into phrases and phases (e.g. McNeill 1992; Kendon 2004). Then, gestures are normally annotated according to their form and functions as per the specific research questions driving the study (see, e.g., Kendon (2004) and Bressem (2013b) for gesture annotation; Hinnell (2018, 2020) for detailed annotations of speech, gesture and other bodily articulators such as head movement). Video-annotation tools such as ELAN allow users to create and search utterance transcriptions and multiple annotation layers.5 These annotation methods form the basis for a range of methodologies, from experimental to corpus-based work we have referred to throughout this chapter. Due to theoretical perspectives informed by cognitive linguistics that focus on actual language usage in interaction (Feyaerts, Brône, and Oben 2017), as well as the availability of larger, text-searchable, multimedia databases, gesture research has recently featured a growth in multimodal corpus studies. Platforms such as Red Hen (Steen and Turner 2013; www.redhenlab.org) and the TV News Archive (www.archive.org/tv) allow researchers to harness data from publicly broadcast television and feature searchable text from television closed-captioning. Once a targeted search – e.g. for a specific linguistic construction – is complete, data are manually annotated for linguistic and gestural form and function, and then analysed using quantitative and/or descriptive statistical methods (e.g. Joo, Steen and Turner 2017; Hinnell 2018, 2019, 2020; Pagán Cánovas et al. 2020; Woodin et  al. 2020). The availability of interactional, multimedia data has resulted in studies featuring much larger data sets. Such studies address issues that are inherently semiotic, e.g. patterned communicative behaviour and the conventionalization of signs,

GESTURE STUDIES AND SEMIOTICS

203

including the degree and the loci of conventionality that designate components of a gestural sign as such and of multimodal sign processes. As an example, in a recent corpus-based study aligned with scholarship on the central role of the body and our experience in the world in driving linguistic and conceptual structure (e.g. in cognitive linguistics (Johnson 1987) and semiotics (Pelkey 2017)), Hinnell (2019) investigates how speakers of North American English build semiotic environments around the construal of contrast (recall the gesture of antithesis in Figure 9.1). Using data from Red Hen, the study documents how speakers mark contrast in gesture and head movement alongside spoken utterances such as on the one hand/on the other hand, the conjunction versus, comparative expressions (better than/worse than), antonym pairs such as David/Goliath and rich/poor, opposition as negation (should I/ shouldn’t I?), and others. Hinnell argues that both iconicity and indexicality motivate the multimodal forms that characterize her data. Iconicity is seen in the symmetrical nature of the gestures (and head movements) and the use of lateral space, which embodies the balance image schema that is the basis of contrast (Johnson 1987). Indexicality emerges in the handshape – frequently, but not always, a point with an extended index finger (see Figure 9.7) – and in the practice of indicating towards opposite sides of the lateral gesture space to reference opposing discourse objects. In Figure 9.7, the first line (S) signifies the speech utterance and the second (G) gesture description. Target utterances in the text search are underlined. Brain imaging studies (see review in Özyürek 2014) and 3D motion capture (MoCap) also facilitate empirical investigations of semiotic and related theories. Functional magnetic resonance imaging (fMRI) has been used to investigate the neuroscientific basis for Peirce’s universal categories (Wolf et al. 2017, 2018) and metonymy in gesture (Joue et al. 2018). As a digital medium, MoCap is semiotic in its very nature, given its capacity to render gesture both visually and algorithmically. Video recordings are made of dialogues in which speakers wear markers, e.g. on their hands, arms, shoulders, hips, and face (for marker-less tracking, see Trujillo et al. 2019; also Pouw et al. 2019 on video-based motion tracking). After data processing, the motion trace of a gesture as well as the skeletal morphology and position of a person’s body can be viewed without seeing the person’s physical body (as one can in video recordings), as in Figure 9.4. MoCap can serve both computational and linguistic semiotic purposes. For example, Mittelberg and colleagues (2015; building on

FIGURE 9.7  Embodied contrast: On the one hand/on the other hand, bilateral point sequence.

204

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Hinnell 2018) recorded a series of conversations among speakers of American English in a motion capture lab. Gestures marking specific event construals were identified and manually annotated. The MoCap profiles of identified gestural forms were then used to derive kinetic patterns and improve spatiotemporal similarity searching in 3D data (Schüller et al. 2017). Semiotic understandings of gesture have an important role to play in the development of artificial intelligence (AI) and human-computer interaction (HCI). In order to render human-like dialogue in HCI applications, the quality of multimodal utterances would, presumably, need to approach human-like behaviour (see Hinnell 2020). Wicke and Veale (2021), for example, have developed an embodied framework for the multimodal performance and interpretation of narratives within robotic storytelling. (See Bressem (2013a), Joo, Steen and Turner (2017) and Kopp (2017) for applications of gesture studies to AI and vice versa). Returning to the wave metaphor, we suggest that such methodological and technological advances, which, importantly, are driving interdisciplinary collaborations, are also driving a fourth wave of gesture studies. To have substantial and sustainable impact, we propose that the new technologies be developed with an aim of making them available beyond individual research programmes and that results should ideally feed back into semiotic theory-building.

CONCLUSION In this chapter, we have aimed to show that investigating gestural phenomena through a semiotic lens allows for deep insights into the specific semiotic nature of gesture. What we can draw from the expansive body of work discussed in this chapter is that gesture – at first glance a basic and natural human communicative resource – can teach us a great deal about semiosis in general, as well as about embodiment and multimodality. Gesture indeed reveals essential aspects about iconicity, metaphor, indexicality, metonymy, conventionality/habit, etc., and about the formation and functioning of signs. It is likely precisely because gestures are so essential to us as embodied social beings that they can reveal primordial, including preverbal, facets of how we experience our inner and outer world, express ourselves and understand others. Looking ahead, semiotic perspectives in gesture studies continue to demonstrate the potential to advance our understanding of the intricate bodily-based sign processes that underlie human cognition and communicative interaction. In order to do justice to the noted semiotic complexity of co-speech gestures, and multimodal language more broadly, the evolution of both theoretical and empirical work must feature focused studies within the intersecting disciplines, as well as interdisciplinary research initiatives that see humanities scholars, data scientists and many others pursue truly collaborative innovation.

NOTES 1 This image is in the public domain and was accessed online: https://publicdomainreview.org/ collection/chirologia-or-the-natural-language-of-the-hand-1644 2 US Vice President Kamala Harris gives thumbs up to students at Miller Elementary School, Dearborn, Michigan, Monday, 6 May 2019. Retrieved 4 June 2021. Copyright: Associated Press 2019. Source: https://www.taiwannews.com.tw/en/news/3697138

GESTURE STUDIES AND SEMIOTICS

205

3 Transcript of verbal transcription (Figure  9.4): ‘I thought it was a girl, because, uhm she looked like she was wearing some sort of short flowing skirt uhm, and . . . uh her head was turned to this side if I were mirroring what she was doing and her arms were like this. Uhm, and . . . uh.. her mouth was almost in the shape of a heart and . . . uhm, I kept trying to see if her eyes were open or closed, and it looked like they were just slits.’ 4 German original: ‘Nee, ich denke wir fahren dann so weißte von da nach da rüber runter und dann nach da und dann wieder nach Hause, nein?’ 5 https://tla.mpi.nl/tools/tla-tools/elan/ (ELAN 2020).

REFERENCES Andrén, M. (2010), Children’s Gestures from 18 to 30 Months, PhD.diss., Lund University, Lund. Armstrong, D. F., W. C. Stokoe and S. Wilcox (1995), Gesture and the Nature of Language, Cambridge: Cambridge University Press. Arnheim, R. (1969), Visual Thinking, Berkeley: University of California. Bavelas, J., J. Gerwing, C. Sutton and D. Prevost (2008), ‘Gesturing on the Telephone: Independent Effects of Dialogue and Visibility’, Journal of Memory and Language, 58 (2): 495–520. Bavelas, J. B., N. Chovil, L. Coates and L. Roe (1995), ‘Gestures Specialized for Dialogue’, Personality and Social Psychology Bulletin, 21 (4): 394–405. Boutet, D., A. Morgenstern and A. Cienki (2016), ‘Grammatical Aspect and Gesture in French: A Kinesiological Approach’, Russian Journal of Linguistics, 20 (3): 132–51. Bouvet, D. (1997), Le Corps et la Métaphore dans les Langues Gestuelles: À la Recherche des Modes de Production des Signes, Paris: L’Harmattan. Bressem, J. (2013a), ‘20th Century: Empirical Research of Body, Language, and Communication’, in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 1, 393–415, Berlin: De Gruyter Mouton. Bressem, J. (2013b), ‘A Linguistic Perspective on the Notation of Form Features in Gestures’, in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 1, 1079–98, Berlin: de Gruyter Mouton. Bressem, J. (2014), ‘Repetitions in Gesture’, in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 2, 1037–958, Berlin: de Gruyter Mouton. Bressem, J. and C. Müller (2014), ‘A Repertoire of German Recurrent Gestures with Pragmatic Functions’, in C. Müller et al. (eds), vol. 2, 1575–91, Berlin: de Gruyter Mouton. Brookes, H. (2004), ‘A Repertoire of South African Quotable Gestures’, Journal of Linguistic Anthropology, 14 (2): 186–224. Bühler, K. ([1934] 1982), Sprachtheorie: Die Darstellungsfunktion der Sprache, Stuttgart/New York: Fischer. Calbris, G. (1990), The Semiotics of French Gestures, Bloomington: Indiana University Press. Calbris, G. (2011), Elements of Meaning in Gesture, trans. M. M. Copple, Amsterdam: John Benjamins. Cienki, A. (1998), ‘Metaphoric Gestures and Some of Their Relations to Verbal Metaphoric Expressions’, in J.-P. Koenig (ed.), Discourse and Cognition: Bridging the Gap, 189–204, Stanford, CA: CSLI Publications.

206

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Cienki, A. (2013), ‘Image Schemas and Mimetic Schemas in Cognitive Linguistics and Gesture Studies’, Review of Cognitive Linguistics, 11 (2): 417–32. https://doi.org/10.1075/ rcl.11.2.13cie Cienki, A. and C. Müller, eds (2008), Metaphor and Gesture, Amsterdam: John Benjamins. Clark, H. H. (1996), Using Language, Cambridge: Cambridge University Press. Clark, H. H. (2003), ‘Pointing and Placing’, in S. Kita (ed.), Pointing: Where Language, Culture, and Cognition Meet, 243–68, Mahwah, NJ: L. Erlbaum Associates. Clark, H. H. (2016), ‘Depicting as a Method of Communication’, Psychological Review, 123 (3): 324–47. Cooperrider, K., J. Slotta and R. Núñez (2018), ‘The Preference for Pointing with the Hand Is Not Universal’, Cognitive Science, 42 (4): 1375–90. Cooperrider, K., J. Fenion, J. Keane, D. Brentari and S. Goldin-Meadow (2021), ‘How Pointing Is Integrated into Language: Evidence from Speakers and Signers’, Frontiers in Communication, 6: 567774. Dancygier, B. and E. E. Sweetser (2014), Figurative Language, Cambridge: Cambridge University Press. Deacon, T. (1997), The Symbolic Species: The Co-evolution of Language and the Brain, New York and London: Norton. Debras, C. (2017), ‘The Shrug: Forms and Meanings of a Compound Enactment’, Gesture, 16 (1): 1–34. Dudis, P. (2004), ‘Body Partitioning and Real Space Blends’, Cognitive Linguistics, 15: 223–38. Duncan, S., J. Cassell and E. T. Levy (eds) (2007), Gesture and the Dynamic Dimension of Language, Amsterdam: John Benjamins. Efron, D. ([1941] 1972), Gesture, Race and Culture: A Tentative Study of the Spatio-temporal and ‘Linguistic’ Aspects of the Gestural Behavior of Eastern Jews and Southern Italians in New York City, Living under Similar as well as Different Environmental Conditions, The Hague: Mouton. Ekman, P. and W. Friesen (1969), ‘The Repertoire of Nonverbal Behavior: Categories, Origins, Usage, and Coding’, Semiotica, 1 (1): 49–98. ELAN (Version 6.0) [Computer software] (2020), Nijmegen: Max Planck Institute for Psycholinguistics, The Language Archive. Retrieved from https://archive.mpi.nl/tla/elan. Enfield, N. J. (2009), The Anatomy of Meaning: Speech, Gesture, and Composite Utterances, Cambridge: Cambridge University Press. Enfield, N. J. (2011), ‘Elements of Formulation’, in J. Streeck, C. Goodwin and C. LeBaron (eds), Embodied Interaction: Language and the Body in the Material World, 59–66, Cambridge: Cambridge University Press. Enfield, N. J. (2013), ‘A “Composite Utterances” Approach to Meaning’, in C. Müller et al. (eds), vol. 1, 689–707, Berlin: de Gruyter Mouton. Enfield, N. J., S. Kita and J. P. De Ruiter (2007), ‘Primary and Secondary Pragmatic Functions of Pointing Gestures’, Journal of Pragmatics, 39 (10): 1722–41. Ferrara, L. and G. Hodge (2018), ‘Language as Description, Indication, and Depiction’, Frontiers in Psychology, 9: 1–15. Feyaerts, K., G. Brône and B. Oben (2017), ‘Multimodality in Interaction’, in B. Dancygier (ed.), The Cambridge Handbook of Cognitive Linguistics, 135–56, Cambridge: Cambridge University Press. Fillmore, C. J. (1982), ‘Frame Semantics’, in The Linguistic Society of Korea (eds), Linguistics in the Morning Calm, 111–37, Seoul: Hanshin.

GESTURE STUDIES AND SEMIOTICS

207

­Fricke, E. (2007), Origo, Geste und Raum – Lokaldeixis im Deutschen, Berlin and New York: de Gruyter. Fricke, E. (2012), Grammatik multimodal: Wie Wörter und Gesten zusammenwirken, Berlin: de Gruyter. Gibbs, R. W., Jr. (2005), Embodiment and Cognitive Science, Cambridge: Cambridge University Press. Goffman, E (1955), ‘On Face-work: An Analysis of Ritual Elements in Social Interaction’, Psychiatry: Interpersonal and Biological Processes, 18: 213–31. Goldin-Meadow, S. (2003), Hearing Gesture: How Our Hands Help Us Think, Cambridge, MA: Harvard University Press. Goldin-Meadow, S. and D. Brentari (2017), ‘Gesture, Sign, and Language: The Coming of Age of Sign Language and Gesture Studies’, Behavioral and Brain Sciences, 40: e46. Goodwin, C. (1981), Conversational Organization: Interaction between Speakers and Hearers, New York: Academic Press. Goodwin, C. (2007), ‘Environmentally Coupled Gestures’, in S. D. Duncan, J. Cassell and E. T. Levy (eds), Gesture and the Dynamic Dimension of Language, vol. 1: Gesture Studies 1, 195–212, Amsterdam: John Benjamins. Goodwin, C. (2011), ‘Contextures of Action’, in J. Streeck, C. Goodwin and C. LeBaron (eds), Embodied Interaction: Language and the Body in the Material World, 182–93, Cambridge: Cambridge University Press. Goodwin, C. (2014), ‘The Intelligibility of Gesture within a Framework of Co-operative Action’, in M. Seyfeddinipur and M. Gullberg (eds), From Gesture in Conversation to Visible Action as Utterance, 199–218, Amsterdam/Philadelphia: John Benjamins. Goodwin, C. (2018), Co-operative Action, Cambridge: Cambridge University Press. Grady, J. (1997), Foundations of Meaning: Primary Metaphors and Primary Scenes, PhD diss., University of California, Berkeley, CA. Hall, J. (2004), ‘Cicero and Quintilian on the Oratorical Use of Hand Gestures’, The Classical Quarterly, 54 (1): 143–60. Harrison, S. (2018), The Impulse to Gesture: Where Language, Minds, and Bodies Intersect, Cambridge: Cambridge University Press. Hassemer, J. and L. McCleary (2020), ‘The Multidimensionality of Pointing’, Gesture, 17 (3): 417–63. Haviland, J. B. (1993), ‘Anchoring, Iconicity, and Orientation in Guugu Yimithirr Pointing Gestures’, Journal of Linguistic Anthropology, 3 (1): 3–45. Haviland, J. B. (2000), ‘Pointing, Gesture Spaces, and Mental Maps’, in D. McNeill (ed.), Language and Gesture, 13–46, Cambridge, MA: Cambridge University Press. Hinnell, J. (2018), ‘The Multimodal Marking of Aspect: The Case of Five Periphrastic Auxiliary Constructions in North American English’, Cognitive Linguistics, 29 (4): 773–806. Hinnell, J. (2019), ‘The Verbal-kinesic Enactment of Contrast in North American English’, The American Journal of Semiotics, 35 (1–2): 55–92. Hinnell, J. (2020), ‘Language in the Body: Multimodality in Grammar and Discourse’, Ph.D. diss., University of Alberta, Edmonton. Holler, J. and J. Bavelas (2017), ‘Multi-modal Communication of Common Ground: A Review of Social Functions’, in R. B. Church, M. W. Alibali and S. D. Kelly (eds), Why Gesture: How the Hands Function in Speaking, Thinking and Communicating, 213–40, Amsterdam: John Benjamins. Hostetter, A. B. and M. W. Alibali (2008), ‘Visible Embodiment: Gestures as Simulated Action’, Psychonomic Bulletin & Review, 15: 495–514.

208

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Jakobson, R. (1956), ‘Two Aspects of Language and Two Types of Aphasic Disturbances’, in L. R. Waugh and M. Monville-Burston (eds), Roman Jakobson – On Language, 115–33, Cambridge: Harvard University Press. Jakobson, R. ([1957] 1984), ‘Shifters, Verbal Categories, and the Russian Verb’, in L. Waugh and M. Halle (eds), Russian and Slavic Grammar: Studies, 1931–81, 41–58, Berlin: Mouton. Jakobson, R. (1960), ‘Closing Statement: Linguistics and Poetics’, in T. A. Sebeok (ed.), Style in Language, 350–77, Cambridge: MIT Press. Jakobson, R. ([1966] 1990), ‘Quest for the Essence of Language’, in L. Waugh and M. Monville-Burston (eds), Roman Jakobson: On Language, 407–21, Cambridge: Belknap Press. Jakobson, R. (1972), ‘Motor Signs for “Yes” and “No”’, Language in Society, 1 (1): 91–6. Jakobson, R. and K. Pomorska (1983), Dialogues, Cambridge: Harvard University Press. Jakobson, R. and L. R. Waugh ([1979] 2002), The Sound Shape of Language, Berlin: de Gruyter Mouton. Janzen, T. and B. Shaffer (2002), ‘Gesture as the Substrate in the Process of ASL Grammaticization’, in R. P. Meier, K. Cormier and D. Quinto-Pozos (eds), Modality and Structure in Signed and Spoken Languages, 199–223, Cambridge: Cambridge University Press. Joue, G., L. Boven, K. Willmes, V. Evola, L. R. Demenescuz, J. Hassemer, I. Mittelberg, K. Mathiak, F. Schneider and U. Habel (2018), ‘Handling or Being the Concept: an fMRIstudy on Metonymy Representations in Coverbal Gestures’, Neuropsychologia, 109: 232–44. Johnson, M. (1987), The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason, Chicago: University of Chicago Press. Joo, J., F. F. Steen and M. Turner (2017), ‘Red Hen Lab: Dataset and Tools for Multimodal Human Communication Research’, Künstliche Intelligenz, 31: 357–61. Kendon, A. (1972), ‘Some Relationships between Body Motion and Speech’, in A. W. Siegman and B. Pope (eds), Studies in Dyadic Communication, 177–216, New York/Toronto/Oxford/ Sydney/Braunschweig: Pergamon. Kendon, A. (1980), ‘Gesture and Speech: Two Aspects of the Process of Utterance’, in M. R. Key (ed.), Nonverbal Communication and Language, 207–88, The Hague: Mouton. Kendon, A. (1988), Sign Languages of Aboriginal Australia: Cultural, Semiotic and Communicative Perspectives, Cambridge: Cambridge University Press. Kendon, A. (1995), ‘Gestures as Illocutionary and Discourse Structure Markers in Southern Italian Conversation’, Journal of Pragmatics, 23 (3): 247–79. Kendon, A. (2000), ‘Language and Gesture: Unity or Duality’, in D. McNeill (ed.), Language and Gesture: Window into Thought and Action, 47–63, Cambridge: Cambridge University Press. Kendon, A. (2004), Gesture: Visible Action as Utterance, Cambridge: Cambridge University Press. Kendon, A. (2008), ‘Some Reflections on the Relationship between “Gesture” and “Sign”’, Gesture, 8 (3): 348–66. Kendon, A. (2017), ‘Pragmatic Functions of Gestures: Some Observations on the History of Their Study and Their Nature’, Gesture, 16 (2): 157–75. Kita, S. (2000), ‘How Representational Gestures Help Speaking’, in D. McNeill (ed.), Language and Gesture, vol. 2: Language, Culture and Cognition, 162–85, Cambridge: Cambridge University Press.

GESTURE STUDIES AND SEMIOTICS

209

­Kita, S., ed. (2003), Pointing: Where Language, Culture, and Cognition Meet, Mahwah, NJ: L. Erlbaum Associates. Kopp, S. (2017), ‘Computational Gesture Research: Studying the Functions of Gesture in Human-Agent Interaction’, in R. B. Church, M. W. Alibali and S. D. Kelly (eds), Why Gesture: How the Hands Function in Speaking, Thinking and Communicating, 267–84, Amsterdam: John Benjamins. Kurz K. B., K. Mullaney and C. Occhino (2019), ‘Constructed Action in American Sign Language: A Look at Second Language Learners in a Second Modality’, Languages, 4 (4): 90. Ladewig, S. H. (2011), ‘Putting the Cyclic Gesture on a Cognitive Basis’, CogniTextes, 6: 1–23. Ladewig, S. H. (2014), ‘Recurrent Gestures’, in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 2, 1558–74, Berlin: de Gruyter Mouton. Ladewig, S. H. (2020), Integrating Gestures: The Dimension of Multimodality in Cognitive Grammar, Berlin: de Gruyter. Lakoff, G. and M. Johnson (1980), Metaphors We Live by, Chicago: Chicago University Press. Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought, New York: Basic Books. Lausberg, H. (2019), The NEUROGES Analysis System for Nonverbal Behavior and Gesture: The Complete Research Coding Manual Including an Interactive Video Learning Tool and Coding Template, Berlin: Lang. LeBaron, C. and J. Streeck (2000), ‘Gestures, Knowledge, and the World’, in D. McNeill (ed.), Language and Gesture, 118–38, Cambridge: Cambridge University Press. Leroi-Gourhan, A. (1964), Le Geste et la Parole: Techniques et Langage, Paris: Éditions Albin Michel. Liddell, S. K. (2003), Grammar, Gesture, and Meaning in American Sign Language, Cambridge: Cambridge University Press. Liebal, K., B. Waller, A. Burrows and K. Slocomb (2013), Primate Communication: A Multimodal Approach, Cambridge: Cambridge University Press. Lücking, A. (2013), Ikonische Gesten: Grundzüge einer Linguistischen Theorie, Berlin: de Gruyter. Maddalena, G. (2015), The Philosophy of Gesture: Completing Pragmatists’ Incomplete Revolution, Montreal/Kingston: McGill-Queen’s University Press. Mandel, M. (1977), ‘Iconic Devices in American Sign Language’, in L. Friedman (ed.), On the Other Hand: New Perspectives on American Sign Language, 57–107, New York: Academic Press. McCafferty, S. and G. Stam, eds (2008), Gesture: Second Language Acquisition and Classroom Research, New York/London: Routledge. McNeill, D. (1985), ‘So You Think Gestures Are Nonverbal?’ Psychological Review, 92 (3): 350–71. McNeill, D. (1992), Hand and Mind: What Gestures Reveal about Thought, Chicago: Chicago University Press. McNeill, D., ed. (2000), Language and Gesture, 2nd edn, Cambridge: Cambridge University Press. McNeill, D. (2005), Gesture and Thought, Chicago: University of Chicago Press. McNeill, D., J. Cassell and E. T. Levy (1993), ‘Abstract Deixis’, Semiotica, 95 (1/2): 5–19. Mead, G. H. (1934), Mind, Self and Society from the Standpoint of a Social Behaviorist, Chicago: University of Chicago Press.

210

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Mead, G. H. (1938), The Philosophy of the Act, ed. C.W. Morris, Chicago: University of Chicago Press. Merleau-Ponty, M. (1962), Phénoménologie de la perception, Paris: Gallimard. Mittelberg, I. (2006), Metaphor and Metonymy in Language and Gesture: Discourse Evidence for Multimodal Models of Grammar, PhD diss., Ithaca: Cornell University. Mittelberg, I. (2008), ‘Peircean Semiotics Meets Conceptual Metaphor: Iconic Modes in Gestural Representations of Grammar’, in A. Cienki and C. Müller (eds), Metaphor and Gesture, 115–54, Amsterdam: John Benjamins. Mittelberg, I. (2013a), ‘The Exbodied Mind: Cognitive-semiotic Principles as Motivating Forces in Gesture’, in C. Müller et al. (eds), vol. 1, 755–84, Berlin: de Gruyter Mouton. Mittelberg, I. (2013b), ‘Balancing Acts: Image Schemas and Force Dynamics as Experiential Essence in Pictures by Paul Klee and Their Gestural Enactments’, in B. Dancygier, M. Borkent and J. Hinnell (eds), Language and the Creative Mind, 325–46, Stanford: CSLI. Mittelberg, I. (2014), ‘Gestures and Iconicity’, in C. Müller et al. (eds), vol. 2, 1712–32, Berlin: de Gruyter Mouton. Mittelberg, I. (2017a), ‘Experiencing and Construing Spatial Artifacts from within: Simulated Artifact Immersion as a Multimodal Viewpoint Strategy’, Cognitive Linguistics, 28 (3): 381–415. Mittelberg, I. (2017b), ‘Multimodal Existential Constructions in German: Manual Actions of Giving as Experiential Substrate for Grammatical and Gestural Patterns’, Linguistics Vanguard, 3 (1): 20160047. Mittelberg, I. (2018), ‘Gestures as Image Schemas and Force Gestalts: A Dynamic Systems Approach Augmented with Motion-capture Data Analysis’, Cognitive Semiotics, 11 (1): 1–2. Mittelberg, I. (2019a), ‘Visuo-kinetic Signs Are Inherently Metonymic: How Embodied Metonymy Motivates Forms, Functions, and Schematic Patterns in Gesture’, Frontiers in Psychology, 10: 1–18. Mittelberg, I. (2019b), ‘Peirce’s Universal Categories: On Their Potential for Gesture Theory and Multimodal Analysis’, Semiotica, 228: 193–222. Mittelberg, I. and L. R. Waugh (2009), ‘Metonymy First, Metaphor Second: A Cognitive-semiotic Approach to Multimodal Figures of Thought in Co-speech Gesture’, in C. J. Forceville and E. Urios-Aparisi (eds), Multimodal Metaphor, 329–58, Berlin: de Gruyter Mouton. Mittelberg, I. and L.-M. Rekittke (2021), ‘Frames und Diagramme im Dialog: Indexikalische und Ikonische Zeichenmodi im Verbal-Gestischen Ko-Konstruieren von Reiserouten’, Zeitschrift für Semiotik, 43(1–2): 81–116. Mittelberg, I. and L.R. Waugh (2014), ‘Gestures and Metonymy’, in C. Müller et al. (eds), vol. 2, 1747–66, Berlin: de Gruyter Mouton. Mittelberg, I. and V. Evola (2014), ‘Iconic and Representational Gestures’, in C. Müller, et al. (eds), vol. 2, 1732–46, Berlin: de Gruyter Mouton. Mittelberg, I., J. Hinnell, C. Beecks, M. Hassani and T. Seidl (2015), ‘Emergent Grammar in Gesture: A Motion-capture Analysis of Image-schematic Aspectual Contours in North American English Speaker-gesturers’, 13th International Cognitive Linguistics Conference, Newcastle, 20–25 July. Morford, J. and J. A. Kegl (2000), ‘Gestures as Precursors to Linguistic Constructs: How Input Shapes the Form of Language’, in D. McNeill (ed.), Gesture and Language, 158–387, Cambridge: Cambridge University Press. Morris, C. W. (1946), Newcastle Signs, Language, Behaviour, New York: Prentice Hall. Morris, D. (1979), Gestures: Their Origins and Distributions, London: Cape.

GESTURE STUDIES AND SEMIOTICS

211

­Müller, C. (1998a), Redebegleitende Gesten: Kulturgeschichte – Theorie – Sprachvergleich, Berlin: Spitz Verlag. Müller, C. (1998b), ‘Iconicity and Gesture’, in S. Santi et al. (eds), Oralité et Gestualité: Communication Multimodale et Interaction, 321–8, Montréal/Paris: L’Harmattan. Müller, C. (2004), ‘Forms and Uses of the Palm Up Open Hand: A Case of a Gesture Family’, in C. Müller and R. Posner (eds), The Semantics and Pragmatics of Everyday Gestures, 233–56, Berlin: Weidler. Müller, C. (2014), ‘Gestural Modes of Representation as Techniques of Depiction’, in C. Müller, et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 2, 1687–702, Berlin: de Gruyter Mouton. Müller, C. (2017a), How Recurrent Gestures Mean: Conventionalized Contexts-of-use and Embodied Motivation, Gesture, 16 (2): 277–304. Müller, C. (2017b), ‘Waking Metaphors: Embodied Cognition in Multimodal Discourse’, in B. Hampe (ed.), Metaphor: Embodied Cognition and Discourse, 297–316, Cambridge: Cambridge University Press. Müller, C. (2018), ‘Gestures and Sign: Cataclysmic Break or Dynamic Relations?’ Frontiers in Psychology, 9: 1651. Müller, C. and A. Cienki (2009), ‘Words, Gestures and Beyond: Forms of Multimodal Metaphor in the Use of Spoken Language’, in C. Forceville and E. Urios-Aparisi (eds), Multimodal Metaphor, 297–328, Berlin: de Gruyter Mouton. Müller, C. and H. Kappelhoff (2018), Cinematic Metaphor, Berlin: de Gruyter. Müller, C. and R. Posner, eds (2004), The Semantics and Pragmatics of Everyday Gesture: The Berlin Conference, Berlin: Weidler Verlag. Müller, C., A. Cienki, E. Fricke, S. H. Ladewig, D. McNeill and S. Teßendorf, eds (2013), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 38/2, Berlin: de Gruyter Mouton. Müller, C., A. Cienki, E. Fricke, S. Ladewig, D. McNeill and J. Bressem, eds (2014), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, 38/2, vol. 2, Berlin: de Gruyter Mouton. Nöth, W. (2016), ‘Habits, Habit of Change, and the Habit of Habit Change According to Peirce’, in D. West and M. Anderson (eds), Consensus on Peirce’s Concept of Habit, 35–63, Dordrecht: Springer. Núñez, R. E. and E. Sweetser (2006), ‘With the Future behind Them: Convergent Evidence from Aymara Language and Gesture in the Crosslinguistic Comparison of Spatial Construals of Time’, Cognitive Science, 30 (5): 401–50. Özyürek, A. (2014), ‘Hearing and Seeing Meaning in Speech and Gesture: Insights from Brain and Behaviour’, Philosophical Transactions of the Royal Society B: Biological Sciences, 369 (20130296): 1–10. Pagán Cánovas, C., J. Valenzuela, D. Alcaraz Carrión, I. Olza and M. Ramscar (2020), ‘Quantifying the Speech-Gesture Relation with Massive Multimodal Datasets: Informativity in Time Expressions’, PLoS ONE, 15 (6): e0233892. Panther, K.-U. and L. L. Thornburg, eds (2003), Metonymy and Pragmatic Inferencing, Amsterdam: John Benjamins. Parrill, F. (2009), ‘Dual Viewpoint Gestures’, Gesture, 9 (3): 271–89. Parrill, F. and E. Sweetser (2004), ‘What We Mean by Meaning: Conceptual Integration in Gesture Analysis and Transcription’, Gesture, 4 (2): 197–219.

212

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Peirce, C. S. ([1866–1913] 1931–58), The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed), Cambridge: Harvard University Press. Cited as CP. Peirce, C. S. ([1893–1913] 1998), The Essential Peirce: Selected Philosophical Writings, Peirce Edition Project (eds), vol. 2, Bloomington: Indiana University Press. Cited as EP 2. Pelkey, J. (2017), The Semiotics of X: Chiasmus, Cognition, and Extreme Body Memory, London: Bloomsbury. Perniss, P. (2018), ‘Why We Should Study Multimodal Language’, Frontiers in Psychology, 9: 1109. Perniss, P., R. Thompson and G. Vigliocco (2010), ‘Iconicity as a General Property of Language: Evidence from Spoken and Signed Languages’, Frontiers in Psychology, 1: 227. Posner, R. (2004), ‘Everyday Gestures as a Result of Ritualization’, in M. Rector, I. Poggi and N. Trigo (eds), Gestures: Meaning and Use, 217–30, Porto: Edições Universidade Fernando Pessoa. Pouw, W., J. P. Trujillo, and J. A. Dixon (2020), ‘The quantification of gesture–speech synchrony: A tutorial and validation of multimodal data acquisition using device-based and video-based motion tracking’, Behavior Research Methods, 52: 723–740. Sandler, W., M. Gullberg and C. Padden, eds (2019), ‘Visual Language’, Frontiers in Psychology and Frontiers in Communication, 10: 1765, Lausanne: Frontiers Media. Saussure, F.de ([1916] 1986), Course in General Linguistics, trans. R. Harris, Chicago: Open Court. Schegloff, E. A. (1984), ‘On Some Gestures’ Relation to Talk’, in J. M. Atkinson and J. Heritage (eds), Structures of Social Action: Studies in Conversation Analysis, 266–96, Cambridge: Cambridge University Press. Schüller, D. and I. Mittelberg (2016), ‘Diagramme von Gesten: Eine Zeichentheoretische Analyse Digitaler Bewegungsspuren’, Zeitschrift für Semiotik, 34 (3–4): 3–32. Schüller, D., C. Beecks, M. Hassani, J. Hinnell, B. Brenger, T. Seidl and I. Mittelberg (2017), ‘Automated Pattern Analysis in Gesture Research: Similarity Measuring in 3D Motion Capture Models of Communicative Action’, Digital Humanities Quarterly, 11 (2): 1–14. Seyfeddinipur, M. and M. Gullberg, eds (2014), From Gesture in Conversation to Visible Action as Utterance, Amsterdam and Philadelphia: John Benjamins. Sonesson, G. (2007), ‘The Extensions of Man Revisited: From Primary to Tertiary Embodiment’, in J. M. Krois, M. Rosengren, A. Steidele and D. Westerkamp (eds), Embodiment in Cognition and Culture, 27–53, Amsterdam: John Benjamins. Sonesson, G. (2014), ‘Some Issues in the Semiotics of Gesture: The Perspective of Comparative Semiotics’, in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 2, 1989–99, Berlin: de Gruyter Mouton. Stec, K. (2012), ‘Meaningful Shifts: A Review of Viewpoint Markers in Co-speech Gesture and Sign Language’, Gesture, 12 (3): 327–60. Steen, F. and M. Turner (2013), ‘Multimodal Construction Grammar’, in M. Borkent, B. Dancygier and J. Hinnell (eds), Language and the Creative Mind, 255–74, Stanford, CA: Centre for the Study of Language and Information. Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Dicisigns, Boston: Docent Press. Stokoe, W. C. ([1960] 2005), ‘Sign Language Structure: An Outline of the Visual Communication Systems of the American Deaf’, Journal of Deaf Studies and Deaf Education, 10 (1): 3–37. Streeck, J. (1993), ‘Gesture as Communication I: Its Coordination with Gaze and Speech’, Communication Monographs, 60 (4): 275–99.

GESTURE STUDIES AND SEMIOTICS

213

­Streeck, J. (2009), Gesturecraft: The Manu-facture of Meaning, Amsterdam and Philadelphia: John Benjamins. Streeck, J., C. Goodwin and C. LeBaron, eds (2011), Embodied Interaction: Language and Body in the Material World, Cambridge: Cambridge University Press. Sweetser, E. E. (1998), ‘Regular Metaphoricity in Gesture: Bodily-based Models of Speech Interaction’, in Actes du 16e Congrès International des Linguistes (CD-ROM), Oxford: Elsevier. Sweetser, E. E. (2007), ‘Looking at Space to Study Mental Spaces: Co-speech Gesture as a Crucial Data Source in Cognitive Linguistics’, in M. Gonzalez-Marquez, I. Mittelberg, S. Coulson and M. J. Spivey (eds), Methods in Cognitive Linguistics, 201–24, Amsterdam: John Benjamins. Sweetser, E. E. (2009), ‘What Does It Mean to Compare Language and Gesture? Modalities and Contrasts’, in J. Guo, E. Liven, N. Budwig, S. Ervin-Tripp, K. Nakamura and S. Özcaliskan (eds), Crosslinguistic Approaches to the Psychology of Language: Research in the Tradition of Dan Isaac Slobin, 357–66, New York: Psychology Press. Sweetser, E. (2012), ‘Introduction – Viewpoint and Perspective in Language and Gesture’, in B. Dancygier and E. E. Sweetser (eds), Viewpoint in Language: A Multimodal Perspective, 1–22, Cambridge: Cambridge University Press. Talmy, L. (1988), ‘Force Dynamics in Language and Cognition’, Cognitive Science, 12: 49–100. Talmy, L. (2018), The Targeting System of Language, Cambridge, MA: MIT Press. Taub, S. (2001), Language from the Body: Iconicity and Metaphor in American Sign Language, Cambridge: Cambridge University Press. Teßendorf, S. (2013), ‘Emblems, Quotable Gestures, or Conventionalized Body Movements’, in C. Müller et al. (eds), Body – Language – Communication: An International Handbook on Multimodality in Human Interaction, vol. 1, 82–100, Berlin: de Gruyter Mouton. Tomasello, M. (1999), The Cultural Origins of Human Cognition, Cambridge, MA and London: Harvard University Press. Trujillo, J. P., J. Vaitonyte, I. Simanova and A. Özyürek (2019), ‘Toward the Markerless and Automatic Analysis of Kinematic Features: A Toolkit for Gesture and Movement Research’, Behaviour Research Methods, 51: 769–77. Varela, F., E. Thompson and E. Rosch (1991), The Embodied Mind: Human Cognition and Experience, Cambridge: MIT Press. Wehling, E. (2017), ‘Discourse Management Gestures’, Gesture, 16 (2): 245–76. Wicke, P. and T. Veale (2021), ‘Creative Action at a Distance: A Conceptual Framework for Embodied Performance with Robotic Actors’, Frontiers of Robotics and AI, 8: 662182, online: https://doi.org/10.3389/frobt.2021.662182/ Wilcox, S. (2004), ‘Cognitive Iconicity: Conceptual Spaces, Meaning, and Gesture in Signed Languages’, Cognitive Linguistics, 15 (2): 119–48. Wilcox, S. and B. Shaffer (2006), ‘Modality in American Sign Language’, in W. Frawley (ed.), The Expression of Modality, 207–37, Berlin: de Gruyter Mouton. Wilcox, S. and C. Occhino (2017), ‘Signed Languages’, in B. Dancygier (ed.), Cambridge Handbook on Cognitive Linguistics, 99–17, Cambridge: Cambridge University Press. Wilkins, D. P. (2003), ‘Why Pointing with the Index Finger Is Not a Universal (in Sociocultural and Semiotic Terms)’, in S. Kita (ed.), Pointing: Where Language, Culture, and Cognition Meet, 171–216, Mahwah, NJ: L. Erlbaum Associates. Wolf, D., L.-M. Rekittke, I. Mittelberg, M. Klasen and K. Mathiak (2017), ‘Perceived Conventionality in Co-speech Gestures Involves the Fronto-temporal Language Network’, Frontiers in Human Neuroscience, 11: 1–19.

214

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

­Wolf, D., I. Mittelberg, L. M. Rekittke, S. Bhavsar, M. Zvyagintsev, A. Haeck, F. Cong, M. Klasen and K. Mathiak (2018), ‘Interpretation of Social Interactions: Functional Imaging of Cognitive-semiotic Categories during Naturalistic Viewing’, Frontiers in Human Neuroscience, 12: 296. Woodin G., B. Winter, M. Perlman, J. Littlemore and T. Matlock (2020), ‘“Tiny Numbers” Are Actually Tiny: Evidence from Gestures in the TV News Archive’, PLoS ONE, 15 (11): e0242142. Wundt, W. (1921), Völkerpsychologie: Eine Untersuchung der Entwicklungsgesetze von Sprache, Mythos und Sitte, vol. 1: Sprache, Mythus und Sitte, 4th edn, Stuttgart: Kröner. Wundt, W. (1973), A Psychology of Gesture, trans. J. S. Thayer, C. M. Greenleaf and M. D. Silberman, The Hague: Mouton. Zima, E. and A. Bergs, eds (2017), ‘Towards a Multimodal Construction Grammar’, Linguistics Vanguard [Special Issue], 3 (s1). Zlatev, J. (2005), ‘What’s in a Schema? Bodily Mimesis and the Grounding of Language’, in B. Hampe (ed.), From Perception to Meaning: Image Schemas in Cognitive Linguistics, 313–43. Berlin: Mouton de Gruyter. Zlatev, J. (2014), ‘Image Schemas, Mimetic Schemas, and Children’s Gestures’, Cognitive Semiotics, 7 (1): 3–29. Zlatev, J. (2015), ‘Cognitive Semiotics’, in P. P. Trifonas (ed.), International Handbook of Semiotics, 1043–67, Dordrecht: Springer.

CHAPTER TEN

Multimodality and Semiotics DAVID MACHIN AND ARIEL CHEN

INTRODUCTION Social semiotics and multimodality are terms that have come to be used interchangeably. Both refer to a form of analysis applied to all forms of communication which has been developed mostly based on principles from the Systemic Functional Linguistics of Michael Halliday. Halliday (1978) called his own approach a ‘social semiotic’ model of language. The basic idea was that rather than modelling grammar, as had been the trend in linguistics, language was seen rather as a set of available tools, seen as flexible choices, to be used and combined in different ways in contexts. The task was therefore to identify what systems of choices were available for accomplishing different things by speakers in contexts. A number of scholars, such as Bob Hodge and Gunther Kress (1988), began to argue that this same model had the potential to be applied to all kinds of communication, such as in the use of images and visual design. The term ‘multimodality’ came into use to capture this extra-linguistic application of social semiotics. The term multimodality must be understood in the sense that it was designed to signal to linguists that meaning-making was always accomplished through instances of communication that comprised more than language alone. Communication was never ‘mono-modal’ but always involved things like images, fonts, types of paper, gestures, sound and music. Meaning-making was always ‘multimodal’. Any analysis which was monomodal would therefore neglect much of what was being deployed to make meaning, for example, in the case of linguistic analyses of advertisement posters. Here it would miss the communicative role of images, icons and symbols. Even, it was argued by Kress and Van Leeuwen (2001), communication was becoming more multimodal. In other words, as design became more sophisticated, aided by advancing technologies, language was used more and more in combination with other modes and in turn the use of these other modes, such as typefaces, colours and images, was becoming more sophisticated and important. So, linguistics was in danger of being left behind. It had to become multimodal. Of course, to scholars with any knowledge of semiotics, or working in the field of media studies, photography studies or film studies, the idea that communication is multimodal would be rather obvious and there is at least a century worth of studies and theories of the visual and film. But multimodality comprised a significant breakthrough within linguistics, where social semiotics has been applied to all manner of communicative forms, such as images, film, monuments, photojournalism and art. One of its major shortcomings has

216

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 10.1  UFIT protein shake (left) and Slimfast protein shake (right).

been, perhaps, that the linguistics who have taken up this social semiotic approach to all forms of communication have engaged little with existing literature in semiotics, visual studies, media studies, film studies, photography or graphic design. And one problem is that this model has been treated as a singular grand theory of all things and there has been little real critical interrogation of its concepts and principles and what kinds of limitations they may have (Ledin and Machin 2018). At the time of writing it was the scholarship in multimodality which was looking outwards also to link with other theories of the visual, with semiotics and more sociological understandings of communication which were looking much more robust. Nevertheless, social semiotics, certainly, in itself has much to offer our understandings of communication, and multimodality has developed into a range of exciting subfields each with its own emphasis, procedures and concerns (Jewitt, Bezemer and O’Halloran 2016). Overall, social semiotics and multimodality encourage us to think about how we communicate by drawing on inventories of existing choices of signs, and that these are used by people in a motivated way in contexts. And this is not neutral but related to power, persuasion and the negotiation of social relations. Multimodality can be very successful in drawing out the ideas and values which may be more buried in instances of

MULTIMODALITY AND SEMIOTICS

217

communication. In this chapter we show how this does indeed have something highly useful to contribute to the wider field of semiotics. Two books, Reading Images (Kress and Van Leeuwen 1996) and The Language of Displayed Art (O’Toole 1994), have been most influential, laying the foundations of multimodality. In fact, these two books pick up in different aspects of Halliday’s social semiotics in his systemic functional grammar and this can be seen echoed in work in the area to this day. As critical linguists Kress and Van Leeuwen were more interested in the functional side of communication. They were interested in motivated sign-making and ideology and did indeed engage with other works in semiotics, particularly Roland Barthes (1977). So here communication was always to be placed in social and political context. O’Toole, on the other hand, was more interested in the systemic side of communication. Here the aim is more to model visual communication in the fashion that linguists model grammar in language. In this chapter we draw on the former, more functionally and socially oriented approach in multimodality. In this chapter we now go on to lay out some of these basic principles and tools of multimodality. To illustrate how these work we show how they can be used as a research tool using the example of food packaging. Here we look at the way that foods marketed for fitness and health are very much gendered, even though they may contain the same ingredients. Multimodality here can help us to show how gender is communicated as well as fitness and healthiness itself. It helps us to draw out what kinds of ideas and values are associated with these. For example, in the case of the two fitness drinks in Figure 10.1, these contain similar ingredients. We do not need to look very hard to see that they are gendered, i.e. marketed differently for men and women. What multimodality can help us do is to focus in on details that we may miss as we describe how exactly this has been accomplished. And this focus on details can in turn help us to draw out the less obvious ways that ideas, values and identities are communicated. And indeed, it is an interesting thing, ideologically, that healthiness and fitness are so gendered.

SEMIOTIC RESOURCES AND CHOICES One basic idea in multimodality is that all acts of communication are accomplished through drawing on a set of available choices which users can apply to, and assemble appropriately, for specific contexts. These choices can be thought of as ‘semiotic resources’ since they are available to us pre-loaded with meaning potential to be used for our purposes in different instances. These choices are loaded with meaning potentials in the sense that a word like ‘youth’ has the potential to describe ‘young people’, or in the case of ‘youth of today’ be combined to suggest something more negative. Here the word ‘youth’ as opposed to ‘young’ carries, therefore, slightly different meaning potentials. In this way, the making of communication should be seen not as necessarily rigid, but as a process where communicators use and combine choices to suit their needs in contexts to achieve certain goals. And the choices they make are not simply about describing or representing the world but about how we make the world as we create priorities, plans, definitions and build institutions. There is no correct or neutral way to represent a thing, process or person, only available semiotic resources that are preloaded with meanings and which carry traces of the socio-political moment they were created (Kress 2010). Multimodal analysis, therefore, is interested in the available choices of the resources, what meanings these resources can be used to produce and then how they also shape how

218

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

we make our societies. New and fresh semiotic production is going on all the time in the processes of creativity, as meaning potentials are deployed in different ways. Yet these potentials nevertheless are always ideologically loaded. The notion of choice and semiotic resources can be applied to all modes of communications, for example, the design of a webpage, an exhibition space, a sculpture or a food package. In the case of designing food packages, it will involve not only choosing the words that are placed on the package, but also the material, the texture of the packaging, the shape of the package, the image, the colour and fonts and so on. We will go into details with these inventories in the second part of the chapter. The idea here is that if we could break down instances of communication in ways which would create inventories of available semiotic resources, we can give a more detailed, systematic and predictive form of analysis. Doing a multimodal analysis, therefore, is concerned with documenting semiotic resources used in the object we analyse. After realizing what options communicators use, we then can ask how these options are used, why these options are used and what the consequences of using these options are. As an example, we can see in our two fitness drinks in Figure  10.1 that there are different semiotic choices between the two in terms of the thickness and curvature of fonts, in the colours, the shape of the bottles, the use of iconography. We return to this shortly to show how we can be more systematic about how we go about making such descriptions.

SEMIOTIC MATERIALS, AFFORDANCES AND CANONS OF USE Besides the choice of semiotic resources, multimodality is also interested in the forms of communication, what we call semiotic materials. Semiotic materials carry, or are formed by, the semiotic choices. A food package, a monument, a document, a menu, a Facebook page are all semiotic materials. A semiotic material can accomplish a particular kind of communication act but not others. So semiotic materials all have affordances. Monuments provide some affordances which a piece of paper does not. Crucially, a semiotic material itself also comes loaded with ideas and assumptions, and shapes communication, social organization and social interactions. A monument and a piece of A4 paper shape what communication gets done and also the kinds of social interactions that take place. If we want to commemorate a leader who fought for social justice, a piece of A4 paper may not work so well as a large bronze figure in a town square. In the same way if we wanted to send our grandmother a message that we missed her, the A4 sheet would work better than the monument. Or a semiotic material like Facebook might do it. In each case a particular kind set of ‘moves’ in communication are laid out by the semiotic material. These allow certain things but not others, therefore templatizing communication and social interaction. Semiotic materials relate the bonds within which communication occurs as well as shaping the kinds of interpersonal and social relationships which can take place. Semiotic materials, like monuments, a sheet of A4, a Facebook page, create a kind of roadmap which we can fill with our semiotic choices. This templatization through semiotic materials means that the semiotic choices we make are done and encountered through what we can call ‘canons of use’. This term is useful as it allows us to capture how semiotic materials are always part of social practices. When we encounter semiotic choices in a material object, such as food-packaging, as

MULTIMODALITY AND SEMIOTICS

219

in our two examples in Figure 10.1, we do not have to make sense of meaning-making through the details of the semiotic choices such as fonts, colours and iconography, but understand this already as an instance of food packaging which is used to shape and load products with ideas and values. Canons of use relate to the traditions of use of a semiotic material as well as the kind of semiotic choices that we find with them. Photographs, for example, have different canons of use in photojournalism, in advertising, personal social media accounts, in galleries. In each case they come with different established meanings, and behind each lies different processes of production and consumption. So, in each case the use of the same kinds of semiotic choices may even take on slightly different kinds of meaning. In this model then there are three levels. There is the micro-level. This is the semiotic material, or ‘text’, which we encounter in situations, like the monument, the A4 sheet, a Facebook page or a fitness drink bottle. This semiotic material, the text, is part of culture, of a tradition of shared ideas, beliefs and behaviours within social groupings. This would include all kinds of artefacts and technologies, such as computers, cameras, fitness equipment and other products. There is the meso-level, which are the canons of use. This accounts for the meaningmaking potential for each semiotic material in contexts. So a photographer, or food packaging designer, will use the cultural resources of their moment in time, such as a digital camera, design software, to make meaning aligned with the existing canons of use, or the typical and established ways of using photography or food packaging in a culture. This leads to the macro-level where the broader available semiotic choices carrying meaning potentials, like words, colours, textures, iconography, will be deployed. To a large extent this will be routine. For example, a photojournalist may use a certain kind of image and style to connote ‘war and suffering’, say an image of a dirty faced child, standing alone outside a bombed-out house. A food-packaging designer may communicate fitness and strength, as we see in the left-hand image in Figure 10.1, by using bold fonts, saturated colours and a bottle whose shape appears stable, rather than tall and elegant. Or they may do so by using a slimmer, more gently curved font, a curved bottle, images of fruits and a busy list of ‘healthy’, properties, such as ‘gluten free’ which, if you are not gluten intolerant, will have no health benefits and means in fact much processing (De Giorgio, Volta, and Gibson 2016). Put simply materiality and communication become interdependent (Bateman et  al. 2017). And communication is not simply here about representation, but rather comes to be infused into our material world such that it forms and gives meaning to the world in which we live (Hjelmslev 1961). And as we can see this materiality houses ideas and values. News photographs claim to represent wars, although they tell us nothing of the political and economic motivations which cause them. Notions of health and physical fitness become presented and shaped to serve food marketing. Multimodality will carry out detailed analyses of the semiotic choices found on instances of communication such as photographs or food packaging. But this analysis must be placed in the context of the forms of activities, ideas and values of which they are part. And we must analyse those semiotic choices within their typical pattern of use in a canon of use. A word, a colour, a form of typeface may have slightly different meaning potential depending on where it is deployed. And this is an important methodological point since one weakness with the more systemic based work in multimodality is that it seeks to create one-size-fits-all models of semiotic choices regardless of context. It also means that we must be aware of how a semiotic material, like a food package, tends to be

220

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

used typically as a canon of use, but also that designers are using the cultural resources of their time, which afford certain kinds of communication and uses of semiotic resources (cf. Bezemer and Kress 2016). In the case of food packaging technology has brought about huge changes, for example, in regard to how liquids can be stored and presented. Before such technology, package designers would not be able to choose to use paper base packages to communicate a more environmentally friendly image of a milk product, for example. As well as this we must also grasp how there are constraints on the uses of materials and semiotic choices. Food-packaging regulation itself shapes this. The EU, regulation No 1169/2011 maps out how some specific semiotic resources can be used and leaves other semiotic resources relatively free in their use. For example, a list of ingredients is mandatory information that is required to be placed on the packaging in a specific way. There is also regulation No 432/2012 which defines when and which specific health claims can be used on food packages. These regulations are established to ensure the communication on food packages, according to the regulation, protects and does not mislead consumers. However, the use of other semiotic resources like the shapes, textures and colours is left with a larger degree of flexibility in their usage. This flexibility gives food packages the affordance to suggest ideas and values of a product to consumers. Looking at the two fitness products in Figure  10.1, research shows there is no evidence that these wheybased drinks will actually contribute to muscle building nor to fitness per se (Hansen et al. 2016). But if we look closely, as we do shortly, no such claim is actually made. It is communicated symbolically through fonts, colours and through buzzwords.

DISCOURSES AND DISCURSIVE PRACTICE This particular social semiotic and multimodal approach also draws on Foucault’s (1972) notion of discourse. Here discourses are like models that consist of our understandings of how things work in society. One discourse that has been extensively studied is the discourse of gender. In this discourse, people believe that, for example, there are certain characteristics, kinds of appearance and behaviours that belong to or are more appropriate for one gender but not others. Such discourse can appear simply natural and inevitable. But discourse is also about power and ideology and the struggle for the definition of reality (Flowerdew and Richardson 2017). So, multimodality here would be interested in analysing instances of communication to reveal how semiotic choices carry certain discourses and ideologies, legitimize and naturalize certain power interests. The notion of discourse is highly useful for approaching the analysis of semiotic resources in the case of our two fitness drinks found in Figure 10.1. In the two decades of the twenty-first century these has been a rising popularity of high protein diets, driven by people’s desire to maximize their exercise outcomes (Hartmann and Siegrist 2016). This must in itself be placed in a surge in interest in gyms and body management (Chen and Eriksson 2019). In supermarkets, we have seen an increasing number of products being marketed highlighting their high protein content and suggesting that this relates to a healthy and active lifestyle. Research shows that consumers believe protein has the capacity to increase training outcomes, reduce risk of injuries and illness, improve health and help control body weight (Heikkinen et  al. 2011; Royne et  al. 2014). However, nutritional scientists have reported that there is no conclusive evidence of a physiological effect of a high protein diet (Pasiakos et al. 2015; Churchward-Venne et al. 2016; Hansen

MULTIMODALITY AND SEMIOTICS

221

et al. 2016). Some researchers even argue that protein supplementation is unnecessary for healthy adults participating in general fitness programmes (Kreider et al. 2010). The myth of protein in relation to fitness points to discourses that are carefully constructed for specific consumer target groups in the interests of increasing sales for food producers (Chen and Eriksson 2019). What these observations show is the need to place the analysis of a set of semiotic choices within a set of discourses prevalent at any given time. So the designers here, or the marketing team, are using the technological and cultural resources of their time, aligned with existing canons of use for food packaging in a particular culture to load their protein products with a set of ideas and values related to muscle building and fitness. Clearly here semiotic materials relate to activities within cultures and to ideas and behaviours within social groupings. The term discursive practices (see Fairclough 1992) is also useful here to think about how instances of communication, semiotic materials and the semiotic choices deployed to form them, such as in the case of the fitness drinks, belong always in a broader framework of behaviours, forms of social interactions, social processes which are realizations of discourses. Put simply, how the world represented through communication is related to and influences how we think, talk, act and live. In this viewpoint, the protein discourse discussed earlier, which is loaded with commercial interests, plays a role in how we shop, eat, see our and others’ bodies and value our and others’ way of living. The production part of making such drinks both draws on these discourses and helps to shape, drive and legitimize them. A multimodality project is, therefore, a project of revealing discourses, the kind of discursive practice that is involved and the ideology they serve. Semiotic materials should be thought about as how the use of them has become established in social and political contexts in order to serve particular interests. The choices in semiotic resources should be examined through how they are related to ideology.

­ NALYSING FOOD PACKAGING AS PART OF A A DISCOURSE OF GENDERED HEALTH AND FITNESS What should be clear at this point is that social semiotics and multimodality are only effective and useful when applied to specific instances of communication understood within context. And it must be paired with theories and practices from other disciplines (Van Leeuwen 2005). For example, we are applying multimodality to the specific issue of how food packaging communicates ideas about gender and fitness; this must be placed against existing scholarship in these areas. This is necessary to help us to formulate the focus of the research question (Bateman et al. 2017). In this case, we want to understand more about the two fitness drink packages in Figure 10.1, relating to how whey protein drink is marketed in two different ways targeting men and women. We want to know what discourses are used and how these are communicated. So, in this case we also need to draw knowledge from scholarship in packaging design, marketing, food studies and gender studies. Here we are able to create more context. Food studies tells us that food should be understood not merely through its function to sustain life, nor about culinary pleasure, but also through what it symbolizes socially and culturally (Falk 1994; Beardsworth and Keil 1997). What is considered healthy food can in itself differ across cultures and also across socioeconomic groups, for example,

222

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

where middle-class consumers favour natural, handmade and organic foods as these bring associations of simplicity and authenticity rather than anonymous industrial production (Ledin and Machin 2018). Studies also show that food preferences can be highly gendered (Madden and Chamberlain 2004) and we see how food advertising is involved in (re)producing these ideas (see, e.g., Bordo 1993; Labre 2005; Gough 2007). In regard to healthy eating, while it is known that this is a growing market for men, this tends to relate to fitness and specifically the idea of developing muscles, bulking up and physical power/performance (Gough 2007). Researchers have long shown that physical form can substitute actual bodily health, where musculature can symbolize strength and vitality, even where it comes from diet and training regimes which place stress on the body (Markula 1997). On the other hand, media advertising of healthy food to women carries a different emphasis on bodily appearance (Schneider and Davis 2010). Here good food choices tend to be related to low-fat food and/or food that allows women to stay slender and therefore sexy and desirable or for them to maintain energy levels to be capable caregivers (Madden and Chamberlain 2004). More recent healthy lifestyle trends, such as ‘fitspiration’, have shifted to also emphasize strength and empowerment over being skinny (Boepple et  al. 2016; Tiggemaann and Zaccardo 2018). It is found that images associated with ‘fitspiration’ contain mainly healthy food such as low-fat/calories salad and smoothies, as well as protein-type supplements and overwhelmingly represent one body shape, thin and toned (Boepple et al. 2016; Tiggemaann and Zaccardo 2018). Scholars have documented the way that muscle became gendered as part of the rise of ideals of femininity as soft, non-aggressive and passive (Ian 1991). But in the second decade of the twenty-first century we have seen increasing representations of women in advertising which carries a rhetoric of empowerment (Toffoletti 2014). Rather than being passive objects to be looked at they have become independent, dynamic, assertive and have their own sexual desires (Gonick 2006; Gill 2008). This has been called a kind of ‘power femininity’ (Lazar 2006: 505). Some have related this to the rise of neoliberalism which demands that the self must be go-getting, focused, successful, where any form of passivity is a sign of unproductivity (Walkerdine 2003). Fitness regimes and the drive for a stronger more muscular body in the case of women can be seen as part of this striving for selfimprovement (Scott 2011). For Dawson (2015) this is about self-management, improving outputs. Cohen and Colino (2014) and Elliot (2013) also help us to think about how the appearance of more young women with toned and muscular bodies in the media also relates to a culture of consumerism which demands the constant ‘new’ and reinvention, where the self becomes one site where this takes place. And this is an era where we are judged chiefly on our consumption practices (Bauman 2011) whether we eat well, if we shop ethically, etc. Scholars have also shown how food products marketed as healthy often simply offer a hodgepodge of health buzzwords which are often contradictory and confusing for consumers (see, e.g., Lewis 2008; O’Niell and Silver 2017) and may even be used to brand foods that are otherwise pretty unhealthy (Chen and Eriksson 2019). While claims are regulated, buzzwords and packaging designs can, nevertheless, be used to signify healthiness and simplicity (Eriksson and Machin 2019). Research has shown that packaging designs can be highly influential to consumers’ perceptions and buying behaviour (Underwood, Klein and Burke 2001; Clement, Kristensen and Gronhaug 2013; Clement et al. 2017) and it has been shown that gender coding is one way this is accomplished through design features, including colour, shape, texture, logos, packaging and brand names (Alreck 1994).

MULTIMODALITY AND SEMIOTICS

223

Through this brief literature review we have been able to provide some of the context against which we can now go on to carry out the textual part of multimodal analysis. We know something about the production and discourses loaded onto these products. And we know something about the social practice, ideas and values of which they are apart. These are gender-targeted notions of fitness, in a cultural context where fitness and body management are part of neoliberal ideas and values associated with self-improvement. This has been particularly transforming for women’s notions of the body with the rise of the valuing of musculature, strength and power. We now go on to look closely at some different health and fitness food packages to see what kinds of semiotic resources they carry, so that we can consider what kind of discourse they communicate.

ANALYSING PACKAGING: HOW GENDER HAS BEEN INFUSED IN THE DISCOURSE OF FITNESS FOOD ON FOOD PACKAGES In what follows we present an analysis of the packages under a number of headings each of which presents tools of observation typically used in multimodality. We look at shape, size, textures, colour, fonts, iconography. Each of these can be seen as a set of meaning potentials. So what kind of shape or texture is chosen and what does it communicate? In multimodality it is usual to present the available repertoire of semiotic choices as a list of alternatives, such as hard-soft, rough-smooth, angular-curved where each tends to come with meaning potentials. These are based on distinctions, identified by a commutation test (Ledin and Machin 2020). So, in any instance of a semiotic choices we can change them to see what happens. In practice this leads to inventories of opposites which sit on a continuum such as light-dark, saturated dilute, rough-smooth. Such inventories have their basis in what are called ‘system networks’, in Hallidayan linguistics. These are the systems and layers of choices available to speakers as they build up meaning (Halliday 1978). Here we do not have the space to present full inventories, nor to carry out a complete analysis of all the qualities and features of food packages such as shape, materials, composition, colour, iconography. But we are able to provide an example of how this works looking at typeface and textures. To some extent this involves an artificial way of going about analysis as the qualities and features these tools address are never encountered individually as we look at or interact with the packages. We meet them as semiotic materials in contexts where they are already meaningful. But this kind of step-by-step analysis allows us a more systematic level of observation.

WRITING AND TYPOGRAPHY An obvious difference on the two drinks bottles in Figure 10.1 is the fonts that are used. At first glance we might say that the fonts on the bottle marketed at women are simply more feminine. But in multimodal analysis we want to ask how they communicate this. And sometimes looking a little more systematically reveals not exactly what we expected. So, in multimodality we would have a kind of inventory of meaning potentials for typefaces. These are based on what we call ‘experiential associations’, in other words associations in our everyday world. So wider objects tend to be more stable than narrow ones. Angular objects are harsher than curved ones, etc. Here is such an inventory for

224

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

typefaces. See Ledin and Machin (2020) and Van Leeuwen (2006) for a full account of how such inventories are created. 1. Weight – bold can mean substantial, stable, strong or overbearing. Slight can mean timid, insubstantial or subtle 2. Expansion – typefaces can take up space confidently or can look cramped, or they can appear gentle and unassuming 3. Curvature – angular can mean technical harsh or masculine. Curved can mean organic, nurturing feminine 4. Orientation – tall can mean aspirational or arrogance. Squat can mean down to earth, stable or inertia 5. Regularity – irregular can mean fun, chaos, non-conformist; regular can mean order, serious, formal, conformist Applying these to the two fitness drinks we see that the UFIT package caries heavy and wide fonts for the brand name. This can suggest that the product is solid and substantial. The font is also rounded, with no sharp edges, which can point to a smooth and silky texture of the drink or also suggest the roundedness of chubby muscles. The word ‘fit’ is also quite tall suggestion aspirational and certainly not as heavy as the number ‘fifty’. So as well as bulk this suggests something about mobility. The Slimfast drink has slimmer fonts overall, pointing to lightness rather than bulk, especially for the qualities like ‘High in Fibre’. We find curvature on the word ‘Slimfast’, but also angularity on the world ‘Vitality’ and on some of the drink qualities. Here this gives a sense of something technical of ‘information’. What we see is that on packages different font qualities can be used for different sections. So, curvature may be used in a brand word to suggest something emotional, expressive, but angular fonts to symbolize facts. We also find more font regularity on the Ufit package, where we find different sizes of the same typeface. The Slimfast package uses a range of different fonts and also different weights of the same font in the same word, such as in ‘Vitality’, here suggesting something dynamic, fun even. In terms of fonts then we see some typical older notions of gender. The men’s product is about bulking up; the women’s is about slimming and energy. We also see that the women’s product presents a range of health-related buzzwords, whereas the men’s does not. And it also carries these buzzwords using more formal, technical typefaces. What we begin to see from this analysis is that, at the level of font at least, we find welldocumented gendered discourses of the meaning of fitness. For men it is about bulk, or at least power, for women, more about slimming and having energy, as well as actual health properties being important – here where fonts present them as information. Nothing on the men’s product carried this level of formality.

TEXTURES We now move onto a different semiotic resource used by packaging designers and that is texture. We also introduce two new fitness foods where the same basic content is marketed differently for men and women. There are snack bars seen in Figures 10.2 and 10.3. Both of these are made from plastic; yet, very different textures have been used on each. Textures too can be broken down into a number of meaning potentials. For a

MULTIMODALITY AND SEMIOTICS

225

more detailed account of textures see Abousnnouga and Machin (2013), Djonov and Van Leeuwen (2011) and Ledin and Machin (2018). 1. Rigidity – surfaces can be hard or give to the touch. Tough, uncompromising solid, versus gentle, yielding, accommodating 2. Relief – surfaces can be flat or have contours, ridges, or this can simply be rough or smooth. This can mean natural versus unnatural/processes versus handmade 3. Regularity – surfaces can be irregular which can mean interest, complexity, individually made, while regular surfaces can be controlled, regulated, clean and certain, or even dull and conformist 4. Natural-artificial – the organic, unprocessed, personal versus processed, or technical impersonal 5. Sticky – this can mean dirty, but also the ability to grip as in rubber floors, so comfort. Unsticky can be clean but also uncomforting as in a marble floor 6. Liquidity – wetness can mean life and vitality or putridness. Dryness can mean cleanness but also lack of vitality. The packages in Figures 10.2 and 10.3, the SCI-MX and Bounce, both use basically plastic. However, the SCI-MX has a metallic and glossy surface, accomplished with layering of polymer film and metal. It appears metallic, shiny, polished and technical and is smooth to the touch. In contrast The Bounce package has a slightly rough, or grainy,

FIGURE 10.2  SCI-MX protein.

226

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 10.3  Bounce protein energy ball.

texture suggesting something more natural, less processed and polished. The surface of the SCI-MX package is without relief and is regular also pointing to control, consistency, processing. The Bounce package, in contrast, fits loosely round the product and is slightly thicker, though also comprising polymer, metal layering, which creates relief and irregularity, suggesting less consistent, less polished processing. Both products are high in fat and sugars, the latter of which provides the energy content. Yet we can see that one foregrounds something technical and scientific, the other the natural and the handmade. Returning to the fonts we can also see that the SCI-MX bar carries more angular and squat fonts compared to the rounded, even hand-written, personalized and slightly taller fonts on the Bounce bar. Of course, the very names of the products themselves bring different associations, of the scientific, versus fun. We can also think about textures in relation to the drink bottles in Figure  10.1. Sometimes men’s fitness drinks have stickiness with polystyrene grip-type bands a round them to suggest a kind of functionality and activeness, suggestive of the kinds of grips found on power tools. Looking at the two bottles we can see the men’s product has a more

MULTIMODALITY AND SEMIOTICS

227

regular surface with some contouring around the middle suggesting a kind of grip zone. The women’s product has more variety in relief with its slimmer mid-section and curved top. Both bottles have a rigid texture, unlike some drink bottles for natural products which may use thinner plastics to suggest being closer to the product. Here rigidity can be important to suggest the strength and dynamism provided by the product. A fitness or bulking drink in a very thin plastic which gave to the touch may not be quite appropriate. Other drinks bottles also use textures which carry liquidity to suggest vitality, or may have textured paper round them to communicate naturalness as in organic yoghurt drinks. At the level of texture, we can also see then that health-related foods are gendered in quite traditional terms. But for the snack bars it is slightly different. Here the men’s product is linked with science and technology. The women’s product is more personalized and natural. There are differences between the drink bottles and the snacks, but in both cases we find that for the women’s products the list of buzzwords appears with more formal fonts. Of course, gender is no single consume category in marketing terms and these products will be aimed at different platforms. So the female target group for the Bounce bar may not be the same as that for the Slimfit drink. But nevertheless, here we find gendering in both places. And, ultimately, we might want to ask what this means when ‘health’ is so communicated to men and women, and to different market platforms in different ways.

CONCLUSION The analysis of these food-packaging examples only begins to scratch the surface; and, of course, there are still other aspects of even these food packages that we could have looked at, such as colour, iconography, shape or materials. But here the aim was to purely introduce how multimodality goes about analysis. Multimodality looks to carry out high detailed and systematic analyses of instances of communication. But this must always be done with context in mind, so, understanding the nature of the semiotic material we are analysing the canons of use to which they are associated, and also the discourse and discursive practices of which they are a part. All of these help us to make sense of how and why semiotic choices are made as they are designed. To do this we need to provide some sociological depth by considering relevant literature which has already built up established knowledge. Multimodality can be applied for the analysis of many instances of communication using different semiotic materials as data, such as film, documents, data presentation and monuments (Ledin and Machin 2018). But analysis should be driven by a research question, which is clear and also clarified by reference to wider literature on the topic. And multimodality is best used in relation to the wider project of drawing out what kinds of ideas and values are buried within the communication and the semiotic materials around us, like news websites, social media platforms, the design of buildings and food marketing. In this chapter this very superficial analysis has already pointed to the way that food-marketing companies seek to colonize and shape discourse and practices of taking care of our bodies.

REFERENCES Abousnnouga, G. and D. Machin (2013), The Language of War Monuments, London: Bloomsbury. Alreck, P. (1994), ‘Commentary: A new Formula for Gendering Products and Brands’, Journal of Product & Brand Management, 3: 6–18.

228

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Barthes, R. (1977), Image, Music, Text, London: Fontana. Bateman, J., J. Wildfeuer and T. Hiippala (2017), Multimodality: Foundations, Research and Analysis: A Problem-Oriented Introduction, Berlin: de Gruyter Mouton. Bauman, Z. (2011), Collateral Damage: Social Inequalities in a Global Age, Cambridge and Malden: Polity Press. Beardsworth, A. and T. Keil (1997), Sociology on the Menu, New York: Routledge. Bezemer, J. and G. Kress (2016), Multimodality, Learning and Communication: A Social Semiotic Framework, London: Routledge. Boepple, L., R. Ata, R. Rum and K. Thompson (2016), ‘Strong Is the New Skinny: A Content Analysis of Fitspiration Websites’, Body Image, 17: 132–5. Bordo, S. (1993), Unbearable Weight: Feminism, Culture, and the Body, Berkeley: University of California Press. Chen, A. and G. Eriksson (2019), ‘The Mythologization of Protein: A Multimodal Critical Discourse Analysis of Snacks Packaging’. Food, Culture & Society, 22 (4): 423–45. Churchward-Venne, T., A. Holwerda, S. Phillips and L. Van Loon (2016), ‘What Is the Optimal Amount of Protein to Support Post-exercise Skeletal Muscle Reconditioning in the Older Adult?’ Sports Medicine, 46 (9): 1205–12. Clement, J., T. Kristensen and K. Grønhaug (2013), ‘Understanding Consumers’ In-store Visual Perception: The Influence of Package Design Features on Visual Attention’, Journal of Retailing and Consumer Services, 20 (2): 234–9. Clement, J., V. Smith, J. Zlatev, K.Gidlöf and J. Van de Weijer (2017), ‘Assessing Information on Food Packages’, European Journal of Marketing, 51 (1): 219–37. Cohen J. and S. Colino (2014), How Strong Is the New Skinny: How to Eat, Live and Move to Maximize Your Power, New York: Harmony. Dawson, M. (2015), ‘CrossFit: Fitness Cult or Reinventive Institution’, International Review for Sociology of Sport, July: 1–19. De Giorgio, R., U. Volta and P. Gibson (2016), ‘Sensitivity to Wheat, Gluten and FODMAPs in IBS: Facts or Fiction?’, Gut, 65: 169–78. Djonov, E. and T. Van Leeuwen (2011), ‘Bullet Points, New Writing, and the Marketisation of Public Discourse: A Critical Multimodal Perspective’, in E. Djonov and S. Zhao (eds), Critical Multimodal Studies of Popular Discourse, 232–50, London: Routledge. Elliot, A. (2013), Reinvention, London: Routledge. Eriksson, G. and D. Machin (2019), ‘Discourses of “Good Food”: The Commercialisation of Healthy and Ethical Eating’, Discourse, Context & Media, 33: 1–7. ­Fairclough, N. (1992), Discourse and Social Change, London: Polity Press. Falk, P. (1994), The Consuming Body, London: Sage. Flowerdew, J. and J. Richardson (2017), The Routledge Handbook of Critical Discourse Studies, London: Routledge. Foucault, M. (1972), The Archaeology of Knowledge, New York: Pantheon Books. Gill, R. (2008), ‘Empowerment/Sexism: Figuring Female Sexual Agency in Contemporary Advertising’, Feminism & Psychology, 18 (1): 35–60. Gonick, M. (2006), ‘Between “Girl Power” and “Reviving Ophelia”: Constituting the Neoliberal Girl Subject’, NWSA Journal, 18 (2): 1–23. Gough, B. (2007), ‘“Real Men Don’t Diet”: An Analysis of Contemporary Newspaper Representations of Men, Food and Health’, Social Science & Medicine, 64 (2): 326–37. Halliday, M. (1978), Language as Social Semiotics: The Social Interpretation of Language and Meaning, London: Edward Arnold.

MULTIMODALITY AND SEMIOTICS

229

Hansen, M., J. Bangsbo, J. Jensen, M. Krause-Jensen, B. Bibby, O. Sollie, U. Hall and K. Madsne (2016), ‘Protein Intake during Training Sessions Has No Effect on Performance and Recovery during Strenuous Training Camp for Elite Cyclists’, Journal of the International Society of Sports Nutrition, 13: 9–19. Hartmann, C. and M. Siegrist (2016), ‘Benefit Beliefs about Protein Supplements: A Comparative Study of Users and Non-users’, Appetite, 103: 229–35. Heikkinen, A., A. Alaranta, I. Helenius and T. Vasankari (2011), ‘Dietary Supplementation Habits and Perceptions of Supplement Use among Elite Finnish Athletes’, International Journal of Sport Nutrition and Exercise Metabolism, 21 (4): 271–9. Hjelmslev, L. (1961), Prelegomena to a Theory of Language, Madison: University of Wisconsin Press. Hodge, B. and G. Kress (1988), Social Semiotics, Cambridge: Polity Press. Ian, M. (1991), ‘From Abject to Object: Women’s Bodybuilding’, Postmodern Culture, 1 (3): 1–17. Jewitt, C., J. Bezemer and K. O’Halloran (2016), Introducing Multimodality, London and New York: Routledge. Kreider, R., C. Wilborn, L. Taylor, B. Campbell, A. Almada, R. Collins, M. Cooke, C. P. Earnest, M. Greenwood, D. S. Kalman, C. M. Kerksick, S. M. Kleiner, B. Leutholtz, H. Lopez, L. M. Lowery, R. Mendel, A. Smith, M. Spano, R. Wildman, D. S. Willoughby, T. N. Ziegenfuss and J. Antonio (2010), ‘ISSN Exercise & Sport Nutrition Review: Research & Recommendations’, Journal of the International Society of Sports Nutrition, 7 (7): 2–43. Kress, G. (2010), Multimodality: A Social Semiotic Approach to Contemporary Communication, London: Routledge. Kress, G. and T. Van Leeuwen (1996), Reading Images: The Grammars of Visual Design, London: Routledge. Kress, G. and T. Van Leeuwen (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, London: Arnold Publishers. Labre, M. (2005), ‘Burn Fat, Build Muscle: A Content Analysis of Men’s Health and Men’s Fitness’, International Journal of Men’s Health, 4 (2): 187–200. Lazar, M. (2006), ‘“Discover the Power of Femininity!”: Analysing Global “Power Femininity” in Local Advertising’, Feminist Media Studies, 6 (4): 505–18. Ledin, P. and D. Machin (2018), Doing Visual Analysis: From Theory to Practice, London: Sage. Ledin, P. and D. Machin (2020), Introduction to Multimodal Analysis, London: Bloomsbury. Lewis, T. (2008), Smart Living: Lifestyle Media and Popular Expertise, New York: Peter Lang. ­Madden, H. and K. Chamberlain (2004), ‘Nutritional Health Messages in Women’s Magazine: A Conflicted Space for Women Readers’, Journal of Health Psychology, 9 (4): 583–97. Markula, P. (1997), ‘Are Fit People Healthy? Health, Exercise, Active Living and the Body in Fitness Discourse’, Waikato Journal of Education, 3: 21–39. O’Neill, K. and D. Silver (2017), ‘From Hungry to Healthy: Simmel, Self-cultivation and Transformative Experience of Earring for Beauty’, Food, Culture & Society, 20 (1): 101–32. O’Toole, M. (1994), The Language of Displayed Art, London: Leicester University Press. Pasiakos, S., T. McLellan and H. Lieberman (2015), ‘The Effects of Protein Supplements on Muscle Mass, Strength, and Aerobic and Anaerobic Power in Healthy Adults: A systematic Review’, Sports Medicine, 45 (1): 111–31. Royne, M., A. Fox, G. Deitz and T. Gibson (2014), ‘The Effects of Health Consciousness and Familiarity with DTCA on Perceptions of Dietary Supplements’, Journal of Consumer Affairs, 48 (3): 515–34.

230

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Schneider, T. and T. Davis (2010), ‘Fostering a Hunger for Health: Food and the Self in ‘The Australian Women’s Weekly’’, Health Sociology Review, 19 (3): 285–303. Scott, S. (2011), Total Institutions and Reinvented Identities, Basingstoke: Macmillan. Tiggemann, M. and M. Zaccardo (2018), ‘“Strong Is the New Skinny”: A Content Analysis of #fitspiration Image on Instagram’, Journal of Health Psychology, 23 (8): 1003–11. Toffoletti, K. (2014), ‘Iranian Women’s Sports Fandom: Gender, resistance, and Identity in the Football Movie Offside’, Journal of Sport & Social Issues, 38: 75–92. Underwood, R., N. Klein and R. Burke (2001), ‘Packaging Communication: Attentional Effects of Product Imagery’, Journal of Product & Brand Management, 10 (7): 403–22. Van Leeuwen, T. (2005), Introducing Social Semiotics, London: Routledge. Van Leeuwen, T. (2006), ‘Towards a Semiotics of Typography’, Information Design Journal + Document Design, 14 (2): 139–55. Walkerdine, V. (2003), ‘Reclassifying Upward Mobility: Femininity and the Neo-liberal Subject’, Gender and Education, 15 (3): 237–48.

CHAPTER ELEVEN

Discourse Analysis and Semiotics KAY L. O’HALLORAN AND SABINE TAN

INTRODUCTION Semiotics provides the foundations for discourse analysis, most notably multimodal discourse analysis which is concerned with the meaning arising from the integration of language, images and other resources in semiotic artefacts and processes (e.g. O’Halloran 2021). In particular, Michael Halliday’s social semiotic approach to the study of language (e.g. Halliday 2009) provided key theoretical concepts which Kress and van Leeuwen (2006, 2020) and O’Toole (2011) further developed for the study of images and displayed art. From these early beginnings, multimodal discourse analysis was developed and applied to other forms of semiosis such as music (van Leeuwen 1999) and mathematics (O’Halloran 2015), leading to the field which is generally known as ‘multimodality’ (Jewitt, Bezemer, and O’Halloran 2016; Bateman, Wildfeuer, and Hiippala 2017). Although there are other approaches to the study of multimodal phenomena (e.g. cognitive approaches, mediated discourse analysis and interactional analysis), semiotics provides the basis for the most widespread approaches. In this chapter, key concepts in semiotics which provide the underlying foundations for multimodal discourse analysis (e.g. system, metafunction, intersemiosis and resemiotization) are explored and illustrated. In addition, recent research efforts aimed at developing digital approaches to multimodal discourse analysis and multimodal mixed methods approaches to big data are explored. The discussion aims to show how semiotics has provided the basis for various cross-disciplinary movements with the potential to develop into the multimodal sciences in the future (O’Halloran et al. 2020).

SEMIOTIC APPROACHES TO DISCOURSE ANALYSIS The term ‘discourse analysis’ covers a broad terrain, from linguistically orientated approaches which account for how language is used to achieve certain objectives, to sociolinguistic approaches which investigate social, political and psychological features of texts (see Kress 2011). In these traditions, the focus is spoken and written language and non-verbal communication, driven by the need to understand how knowledge and power are constructed and enacted through language use (e.g. Tannen, Hamilton and Schiffrin 2015). However, since the mid-1990s when the term ‘multimodality’ first started to appear (e.g. Jewitt et  al. 2016; Machin 2013, 2016), multimodal discourse analysis,

232

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

which is concerned with the meanings arising from the integration of language and other semiotic resources (image, sound, gesture, etc.), has become a well-established research tradition (e.g. Kress 2011; Tan, O’Halloran and Wignell 2020; O’Halloran 2021). As van Leeuwen (2015: 447) explains, given the increasing multimodality of contemporary communication, ‘[c]learly discourse can no longer be adequately studied without paying attention to multimodality’. Following van Leeuwen (2015), this chapter is concerned with the ways in which multimodality is conceptualized and studied by linguists and semioticians. Social semiotics provides the theoretical foundations for the most widespread approaches to multimodal discourse analysis today, although other approaches to the study of multimodal phenomena exist, such as cognitive approaches (e.g. Forceville 1996; Forceville and Urios-Aparisi 2009), mediated discourse analysis (e.g. Scollon 2001; Scollon and Scollon 2004) and interactional analysis (e.g. Norris 2004; Norris and Jones 2005). The term ‘social semiotic’ is derived from Halliday (1978) but its origins can be traced back to two different, but complementary linguistic and philosophic branches of semiotics developed in the late nineteenth and early twentieth centuries by the American philosopher Charles S. Peirce and the Swiss-French linguist Ferdinand de Saussure. These works in turn inspired the ground-breaking work in semiotics, literary criticism and social theory in the mid-twentieth century by the Russian formalists and the Paris schools of semiotics, as exemplified, for example, by the works of Russian philosopher and literary critic Mikhail Bakhtin, and French philosopher Roland Barthes (e.g. see van Leeuwen 2015). Social semiotics is aligned with cultural semiotics developed by Lotman and the Tartu-Moscow School (e.g. Lotman and Uspensky 1978; Lotman [1984] 2005) where culture is understood as interacting systems of meaningful signs (e.g. Halliday and Hasan 1985). The concept of multimodality arose in the pioneering works of Gunther Kress and Theo van Leeuwen, who introduced the term in their books Multimodal Discourse: The Modes and Media of Contemporary Communication (2001) and Reading Images: The  Grammar of Visual Design (2006, 2020). Drawing on the combined principles of social theory and social semiotics, and supplemented with insights drawn from art history, film, iconography, structural semiotics and cognitive psychology, their work provided the foundations for much multimodal research in the 1980s and 1990s. O’Toole (2011) developed a complementary social semiotic approach, drawing on insights from film studies, iconography and art history. O’Toole applied Halliday’s social semiotic model to the analysis of a variety semiotic artefacts including displayed art, paintings, sculpture and works of architecture, demonstrating how the approach can be applied across the arts. These and other approaches to multimodality inspired by social semiotics and critical discourse analysis (e.g. van Leeuwen 2005, 2008; Machin and Mayr 2012; Machin 2013, 2016) provided the theoretical basis for what became known as multimodal discourse analysis (MDA) (e.g. Kress 2011; van Leeuwen 2015; Jewitt, Bezemer and O’Halloran 2016; Bateman, Wildfeuer and Hiippala 2017). From these early beginnings, the social semiotic approach to multimodal discourse analysis has evolved to become an evergrowing platform for modelling, analysing and interpreting multimodal phenomena (e.g. O’Halloran 2021), ultimately leading to approaches referred to as systemic functional multimodal discourse analysis (SF-MDA) (e.g. O’Halloran 2008, 2021; O’Halloran, Tan and Wignell 2019; Tan, O’Halloran and Wignell 2020) and multimodal critical discourse analysis (MCDA) (e.g. van Leeuwen 2012; Machin 2013). The principles behind these

DISCOURSE ANALYSIS AND SEMIOTICS

233

approaches are elaborated below, with a focus on the social semiotic approach which provides the foundations and inspiration for multimodal discourse analysis (MDA).

MULTIMODAL DISCOURSE ANALYSIS Multimodal discourse analysis (MDA) largely derives from Halliday’s social semiotics, where language, images and other semiotic resources are viewed as resources for meaning-making. Originally developed and applied to the study of language, it formed the basis for what is now known as System Functional Theory (SFT). Essentially, SFT is regarded as a ‘theory of meaning as choice’ where language and other semiotic resources are conceptualized as networks of interrelated systems of meanings which are organized according to the functions that these semiotic resources serve in social contexts (e.g. Halliday 1978; Halliday and Hasan 1985). The approach is closely aligned with critical discourse analysis approaches to multimodality (e.g. Machin 2013; van Leeuwen 2015). Among the key concepts from SFT is the notion that the communicative function of any text or artefact is the result of specific choices which realize three strands of meaning simultaneously. The three strands of meaning, conceptualized in SFT as ‘metafunctions’ (e.g. Halliday and Matthiessen 2014), are: 1. ideational meaning, which consists of experiential and logical meaning for construing our experience of the world; 2. interpersonal meaning for enacting social relations, expressing attitudes and creating a stance towards these expressions; 3. textual (or compositional) meaning for organizing these meanings into coherent messages relevant to their context. In language, the three strands of meaning are mapped on the structure of a clause by specifying the grammatical systems through which these metafunctions are realized. For example, experiential meaning is realized through the grammatical system of Transitivity which accounts for the different types of processes that are found in a language, and the structures through which they are expressed (Halliday and Matthiessen 2014: 211–335). Interpersonal meaning, in turn, is realized through (a) the system of Mood, which realizes ‘meaning as an exchange’ (Halliday and Matthiessen 2014: 135–9), with choices for giving and demanding information (statements, questions), and for giving and demanding goods and services (offers, commands) and (b) the system of Modality (that is, expressions of probability, usuality, obligation and inclination). Lastly, textual meaning is realized through the thematic structure of the clause which organizes the flow of information with clearly defined points of departure. Martin and colleagues (e.g. Martin 1992; Martin and White 2005; Martin and Rose 2007) extended the systemic functional approach to account for metafunctionally based systems of meaning at the discourse level (i.e. paragraph and text). A similar approach has been taken to develop metafunctionally based systems for images and other semiotic resources (e.g. Kress and van Leeuwen 2006, 2020; O’Toole 2011). Kress and van Leeuwen (2006, 2020) adopt a (top-down) contextual approach to the study of images and visual design with particular emphasis on ideological meaning, while O’Toole (2011) employs a (bottom-up) approach to derive metafunctionally based frameworks which can be applied for the analysis of displayed art (e.g. paintings, architecture and sculpture). Regardless of the orientation, the metafunctional principle plays a critical role

234

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

in determining the underlying organization of semiotic resources and for investigating the ways in which semiotic choices combine and interact in a multimodal text or artefact. Another key concept adopted from SFT is the notion of stratification, which posits that the three strands of meaning are also realized through choices made at different ranks or constituent levels according to the context. Language is seen as a stratified semiotic system ‘involving three cycles of coding at different levels of abstraction’ (Martin and White 2005: 8). In this model, the three metafunctions are realized at through choices in discourse semantics and lexicogrammar, and through choices in phonology (for spoken language) and graphology (for written text, e.g. as realized through font size, colour and style). Following O’Toole (2011) who adopted this approach, O’Halloran (2008) proposed a framework for the analysis of multimodal texts and images where language is organized according to the constituent ranks of word, word group/phrase, clause and clause complex, and images are organized according to the ranks of Member (Part), Figure, Episode, Scene and Work. SFT concepts have been adapted and extended for a wide range of multimodal texts and artefacts, including visual art, music, children’s toys and games (e.g. Kress and van Leeuwen 2001, 2006, 2020; O’Toole 2011; van Leeuwen 1999, 2005), popular discourse (e.g. Djonov and Zhao 2013) and media practices (e.g. Djonov 2007; Knox 2007; Bednarek and Caple 2012; Stöckl, Caple and Pflaeging 2020) to account for the ways in which multimodal resources combine and interact intersemiotically in the communication of meaning. In each case, the multimodal texts and artefacts are interpreted in relation to the situational and cultural context, following Halliday’s (1978) social semiotic approach. Expanding on this work, O’Halloran, Tan and Wignell (2019) propose an integrated framework and approach for the analysis and interpretation of multimodal texts and artefacts, including online media. An overview of these semiotic concepts, as interpreted from a systemic functional multimodal discourse analysis perspective (SF-MDA), is provided in Table 11.1. The analytical approach is demonstrated in the following section.

TABLE 11.1  Semiotic concepts for MDA (adapted from O’Halloran et al. 2016) Metafunction

Rank

System

Description

Discourse

Configurations of activities

Content in relation to knowledge construction

Clause

Processes; Participant Roles; Circumstance

Happenings, actions and relations

Discourse

Appraisal

Evaluation in terms of attitude, emotion and judgement

Clause

Speech Function

Exchange of information (e.g. statements and questions) and goods & services (e.g. commands and offers)

text Experiential

Interpersonal

DISCOURSE ANALYSIS AND SEMIOTICS

Metafunction

235

Rank

System

Description

Textual

Clause & Discourse

Information Focus

Organization of information, with points of departure for what follows

Logical

Clause Complex

Logico-Semantic Relations

Logical relations between happenings, actions and relations

Work

Narrative Theme; Representation; Setting

Nature of the scene

Episode

Processes; Participant Roles; and Circumstance

Visual happenings, actions and relations

Figure

Posture; Dress

Characteristics of the participants

Work

Angle; Camera Distance; Lighting

Visual effects

Episode

Proportion in Relation to the Whole Image: Focus; Perspective

Happenings, actions and relations with respect to the whole image

Figure

Gaze-Visual Address

Direction of participant’s gaze as internal to image or external to viewer

Work

Compositional Vectors; Framing

The organization of the parts as a whole, with the visual marking (e.g. framing) of certain parts

Episode

Relative Placement of Episode; Framing

Position of the happenings, actions and relations in relation to the whole image, and the visual marking of certain aspects

Figure

Relative Placement of the Figure within the Episode; Arrangement; Framing

Position of figures in relation to happenings, actions or relations and the visual marking of certain aspects of those figures

text

Images Experiential

Interpersonal

Textual

KEY CONCEPTS WITH ILLUSTRATIONS The multimodal social semiotic approach can be applied to the analysis of multimodal texts by drawing on semiotic concepts for text and image analysis displayed in Table 11.1. The approach is demonstrated through the analysis of terrorist propaganda material drawn from the English-language magazines Dabiq and Rumiyah, the official online magazines

236

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

produced by the violent extremist terrorist organization known as Islamic State, or ISIS. Figure 11.1 is an advertisement for the ISIS video Just Terror, which first appeared in the ISIS online magazine Dabiq, Issue 13, page 55, on 19 January 2016. The analysis of the Just Terror movie poster in Figure 11.1 was first undertaken with the interactive software Multimodal Analysis Image.1 The software application has facilities for annotating images on the basis of purpose-built catalogues of semiotic resources and systems, using overlays in the form of geometrical shapes, arrows and pins, which correspond to the various semiotic choices displayed in Table 11.1. The analysis of selected semiotic choices in the Just Terror movie poster is discussed below. In terms of its composition and arrangement in space, the movie poster consists of three distinct sections or panels (marked A, B and C in Figure 11.1), following the genre convention of regular movie posters. For example, in movie posters elements placed in

FIGURE 11.1  Multimodal analysis of Just Terror movie poster, Dabiq, 13: 55.

DISCOURSE ANALYSIS AND SEMIOTICS

237

the top half of the image are sometimes used to express the ‘promise of the product’ by depicting a scene from the movie (Tan, E and O’Halloran 2012). Elements placed at the centre of the image, in turn, are commonly used to attract the reader’s attention, while elements placed at the bottom half may provide additional information about the movie. Kress and van Leeuwen (2006) interpreted such compositional arrangements in terms of representational (i.e. experiential) and interactive (i.e. interpersonal) meanings created through the three interrelated systems of: 1. information value, realized through the placement of elements (participants and syntagms that relate them to each other and to the viewer); 2. salience, realized through elements (participants as well as representational and interactive syntagms) will attract the viewer’s attention to different degrees; and 3. framing, realized through the presence or absence of framing devices which connect or disconnect elements of the image, signifying that they belong or do not belong together in some sense (Kress and van Leeuwen 2006: 177). Read from top to bottom, the poster analysed in Figure 11.1 displays the title or name of the advertised movie (Just Terror) in large capital letters at the top of the page, superimposed over a stylized monochrome representation of the Paris landscape, then features the names of the principal actors underneath their images, and ends with informing the reader of the movie’s overall motto or leitmotif: ‘Let Paris be a lesson for those nations that wish to take heed . . . ’. Linguistically, Just Terror can be interpreted in two ways, as displayed in Table 11.2. That is, it can be interpreted as ‘justifable’ terror, based on an interpretation of the experiential modifier ‘just’ as what is morally right or fair. Alternatively, the movie title Just Terror can be read as ‘simply’ terror, based on an interpersonal interpretation of ‘just’ as related to intensity, arising (in part) intertextually from Nike’s trademark, ‘JUST DO IT’. In this way, the movie title is designed to attract the attention of readers in the West. Indeed, Dabiq and other propaganda materials produced by ISIS exploit Western media traditions to attract a Western audience (e.g. Speckhard 2015). The movie title Just Terror is followed by the marked linguistic command to pay attention and learn from these latest terrorist attacks in Paris which took place on 13  November 2015, killing 130 people and injuring 416 others.2 As we shall see, the linguistic text functions intersemiotically with the images to realize the main narrative themes of the movie poster. According to Kress and van Leeuwen (2020), the information value in visual compositions which are arranged along the vertical axis can be interpreted as representing

TABLE 11.2  Interpretations of Just Terror Experiential interpretation: Just

Terror

based on what is morally right and fair

Interpersonal interpretation: Just Intensity

Terror

238

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

the ‘Ideal and the Real’, whereby the ‘ideal’ is presented as ‘the idealised or generalised essence of the information’ (2020: 191). The ‘real’ in contrast then presents ‘more “down-to-earth” information (e.g., photographs as documentary evidence, [. . .]), or more practical information (e.g., practical consequences, directions for action)’ (2020: 191). In terms of the experiential meaning potential conveyed at the level of Work, the scene depicted in the top panel of the movie poster is a stylized monochrome representation of the Paris streetscape, and can thus be interpreted as expressing the ‘promise of the product’ (in this case, alluding to the Paris attacks on 13 November 2015), while the bottom half of the poster displays an actual, ‘down to earth’ photograph of a realistic scene from aftermath of the Paris attacks. Together, they present the poster’s overall narrative theme. The title and motto, placed in bright big uppercase letters, which contrast and stand out against the muted background, suggest importance, even grandeur. Experientially, they represent ISIS’s worldview and ‘jurisprudential rationale’ (Ingram 2018: 22) for carrying out terrorist attacks against nations ISIS perceive to be their enemies (Wignell, Tan and O’Halloran 2017; Wignell, O’Halloran, Tan, Lange and Chai 2018). The centre panel of the movie poster, which in Kress and van Leeuwen’s (2006, 2020) terms can be interpreted as functioning as a mediator between the Ideal and the Real, showcases the main participants in the advertised video and – in a wider sense, in the military theatre of jihad, a term used by ISIS to refer to the ‘just’ warfare between the Muslim and non-Muslim world (Wignell, Tan and O’Halloran 2017), drawing attention to participant roles and processes. In terms of visual prominence or salience, certain elements in images can attract more attention than others due to their size, or because they are placed in the foreground. A common persuasive technique of movie posters is to highlight the qualities or characteristics of the principal actors by means of visual prominence. In this case, it is not surprising to find that the participant depicted at the top centre of the whole Episode (marked 1 in Figure 11.1), who in reality was the mastermind of the Paris attacks, is shown to tower over the other participants. Visually, he is an imposing figure, due to his relative size and the AK47 rifle he holds in his hand. His posture is carefully staged so that the way he extends his rifle upwards creates a compositional vector that is intensified by the bright blue spotlight placed upon it. Together, these elements draw the eye upward to the headline or movie title Just Terror, highlighting one of the central themes in ISIS’s propaganda campaign (Ingram 2018), as elaborated below. Vectors not only realize textual or compositional meaning, but they also contribute to the text’s experiential meaning potential. According to Kress and van Leeuwen (2020: 55), the hallmark of a visual narrative proposition is the presence of a vector, which emanates from a participant and connects to another participant or element in the image. In the movie poster, these action vectors are formed mostly by the participants’ hands holding up rifles and knives or performing the ‘Tawheed’ gesture (right forefinger extended skywards). In MDA terms, these participants are portrayed as being engaged in material processes of action. Other participants in the image, ostensibly supporting actors in the advertised video, are arranged in space beneath the main actor, disproportionally smaller in size. The majority of these participants are portrayed as being engaged in processes of reaction, that is, they are just looking, but they are not looking at each other, nor are they looking at the viewer. In terms of the interpersonal relations created, apart from the main figure (the mastermind of the Paris attack) who stares menacingly at the viewer, other figures arranged in the foreground at the bottom tier of this Episode, and who are shown holding up knives or performing the ‘Tawheed’ gesture, also interact directly with the viewer via direct visual address, as they are shown gazing directly at the

DISCOURSE ANALYSIS AND SEMIOTICS

239

viewer. The lead actor and main perpetrator of the Paris attacks also dominates over the others (including the viewer) in view of the power relations that are established through camera angle. For instance, he is the only figure that is captured from a low angle, so that the viewer is placed in the position of looking up at him. As such, the central figure is visually prominent and dominating. However, from a social semiotic viewpoint, the portrayed participants, their actions and the objects they are interacting with fulfil yet another purpose. According to van Leeuwen (2001), the key to interpreting visual meaning is to develop an understanding of the different layers of meaning. Drawing on Barthes’s ([1957] 1972, 1977) adaptations of Hjelmslev’s notions of denotation and connotation, van Leeuwen (2005) perceives of denotation representing the first layer of meaning, expressing the literal message of ‘what, or who, is being depicted here’ (van Leeuwen 2005: 37). Borrowing from SFT, Kress and van Leeuwen (2006, 2020) elaborate that this type of meaning is commonly realized through conceptual representations which convey information about the represented participants’ qualities and characteristics through processes of categorization or typification (e.g. clothing). Connotation, in contrast, is a more abstract concept, which can convey much broader socio-cultural ideas and values. These connotative values and ideas can be realized through symbolic processes which have to do with ‘what a participant means or is’ (Kress and van Leeuwen 2020: 102), or through processes of iconographic symbolism, which convey ideas and values attached to particular ‘object-signs’ or shapes endowed with symbolic value (van Leeuwen 2001: 107). A related concept that has been adapted from SFT and incorporated in the MDA approach is the notion of ‘iconization’. Different from the term used in Peircian semiotics, where an icon does not refer to or ‘stand for’ anything other than itself’ (Nöth 2007: 104), from an MDA perspective, the concept of iconization is viewed as a process whereby the ideational meaning of an object or an activity is backgrounded and the interpersonal connotations or shared values around which fellowships are formed are foregrounded instead (Martin 2017: 129). In this way, multimodal texts and artefacts turn into social emblems of belonging or ‘bonding icons’ (Stenglin 2008). Building on this notion, Wignell, Tan and O’Halloran (2016, 2017; see also Wignell, Tan, O’Halloran, Lange, Chai and Wiebrands 2020) propose a set of options for identifying different types of bonding icons in the context of violent extremist terrorist communication (see Table 11.3).

TABLE 11.3  Bonding icons in violent extremist terrorist communication (adapted from Wignell et al., 2016, 2017, 2020) Types of bonding icons

Description

ICON Hero

People (or personifications), e.g., revered members of a community; role models

Relic

Physical objects and images of physical objects, e.g. flags; emblems; logos

CREED Scripture

Written or spoken representations of scripture, e.g. the Qur’an

Enacted creed

Participation in rituals, e.g. religious services or communal prayer

240

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Interpreted in this way, the represented participants in Figure 11.1, i.e. the so-called ‘Soldiers of the Caliphate’, are the ‘heroes’, role models worthy of emulation and designed to inspire likeminded followers. In the same manner, the ideational (or denotative) meaning of the objects these ‘heroes’ are shown to be interacting with (i.e. knives) and their actions (i.e. performing the Tawheed gesture) is eclipsed by the interpersonal connotations these objects and actions have acquired in the context of violent extremist terrorist communication. Consequently, they become bonding icons for followers with similar shared values and dispositions (Wignell et al. 2016, 2017, 2020). Bonding icons, however, do not acquire their connotative layers of meanings spontaneously or in isolation, but through processes of resemiotization and intertextual references to other multimodal text types and genres. As Ingram (2018: 37) comments, the emergence of ISIS’s Just Terror campaign was the ‘product of several interconnected dynamics’. For example, the various themes and motifs that are present in the movie poster in Figure 11.1 were announced on the cover of Dabiq, Issue 12 (Figure 11.2, left), which features the slogan Just Terror in big bold white and red capital letters and depicts a realistic photograph of a scene from the 2015 Paris attacks. Likewise, the ideological value of ISIS relics such as knives which are heralded as the weapon of choice for lone-wolve terrorists in relation to ISIS’s Just Terror tactics campaign was reinforced in the online magazine Rumiyah which superseded Dabiq. For example, as shown in Figure  11.2, the theme of knives as the weapon of choice featured prominently on the cover as well as in a procedural text on page 12 of Rumiyah, Issue 2 (Figure 11.2, centre) and was reiterated in a full-page infographic in Rumiyah, Issue 4 (Figure 11.2, right). As Wignell et al. (2016, 2017, 2020) point out, the connotative meanings of bonding icons arise out of a community’s view or vision of the world, by highlighting aspects of this vision of an ideal world which a community treasures, which in this case had ‘an almost guaranteed “force multiplying” impact’ on ISIS’s propaganda aims (Ingram 2018: 37). When these and similar messages are redistributed in Western online media, the values encoded in these texts and images are then further reinforced or negated through resemiotization and recontextualization processes (Wignell, O’Halloran, Tan,

FIGURE 11.2  Resemiotizations and intertextuality of Just Terror tactics and knife attacks in Dabiq and Rumiyah magazines.

DISCOURSE ANALYSIS AND SEMIOTICS

241

Lange and Chai 2018), which according to Iedema (2003) ‘provide the analytical means for: 1. tracing how semiotics are translated from one into the other as social processes unfold, as well as for 2. asking why these semiotics (rather than others) are mobilised to do certain things at certain times’ (Iedema 2003: 29). In order to map such patterns, it is necessary to develop new techniques for large-scale analyses of multimodal texts, as discussed below.

DIGITAL APPROACHES TO MULTIMODAL DISCOURSE ANALYSIS Research in multimodal analysis has traditionally involved the development of theoretical frameworks which are demonstrated with examples, as illustrated above. However, this approach has limitations arising from the fact that the insights are derived and generalized from a few sample texts, and the theoretical frameworks are not rigorously tested under different conditions as a result. For this reason, interdisciplinary mixed methods involving computational approaches for large-scale multimodal analysis are being explored to overcome these issues (e.g. O’Halloran et al. 2018; O’Halloran, Pal and Jin 2021). This is a challenging undertaking, however, given the large amounts of training data which are required to develop machine learning and data mining approaches based on multimodal semiotic theory. Moreover, existing computational tools are largely monomodal, having been developed for language analysis, image analysis and video analysis alone. With this in mind, approaches that integrate existing computational tools with multimodal frameworks for large-scale semiotic analysis are being investigated to test the effectiveness of this approach and to identify the gaps in existing computational approaches. Initial explorations in this direction, involving the use of natural language understanding algorithms for MDA, appear promising (Wignell, Chai, Tan, O’Halloran and Lange, 2018). In what follows, a mixed methods approach for large-scale multimodal analysis is illustrated using online terrorist materials discussed above. Many images which appear in violent extremist propaganda are re-used and recontextualized across different websites (e.g. mainstream news media, magazines, blogs, photo sharing sites, etc.) and social media. In order to investigate the recontextualization patterns, reverse image searches of images from Dabiq and Rumiyah were undertaken using TinEye3 which crawls the web to identify the websites where these images appear. A sample set of 20 images from Dabiq and Rumiyah and 254 English language webpages with these images were identified to explore how the multimodal analysis framework (Table 11.1) can be integrated with computational tools (i.e. natural language processing and image processing) in order to analyse the text and images in the articles from these websites. The sample data set of images from Dabiq includes the Just Terror movie poster (Figure  11.1) discussed above. An example of an article4 with the image from Dabiq is displayed in Figure  11.3a. The article appears in FDD’s Long War Journal (LWJ),5 an American news blog which reports on the US war on terror. The linguistic text was

242

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

analysed using natural language understanding algorithms provided by IBM Watson6 and the image was analysed using image recognition models provided by Clarifai.7 The linguistic analysis involves IBM Watson’s algorithms for key concepts, entities, keywords, categories, relations and semantic roles and sentiment and emotion and the visual analyses involve Clarifai’s algorithms for concepts, focus and demographics in the image. The linguistic and visual features derived from these computational algorithms are organized according to the metafunctional systems and ranks from the multimodal theoretical framework (Table  11.1), as displayed below. The results for the linguistic and visual analyses derived from IBM Watson and Clarifai models for the article in Figure 11.3a are displayed in Figures 11.3b and 11.3c respectively.

Text Analysis: IBM Watson 1. Discourse/Register – Experiential: Watson’s categorical values (twelve categories/ features) 2. Discourse Semantics – Interpersonal: Watson sentiment and emotion analysis for each paragraph (Sentiment: three features, Emotion: five features) 3. Lexicogrammar – Logical: conjunctive adverb analysis (two features)

Image Analysis: Clarifai 1. Work – Ideational/field: Clarifai Concepts (195 features) 2. Episode – Interpersonal/tenor: Clarifai Focus (two features) 3. Figure – Ideational/field: Clarifai Demographics (seven features) The results of the language and image analyses were clustered using K-modes algorithm (Huang 1997, 1998), which is a form of unsupervised learning for data which does not have predefined categories. The clusters (or sub-groupings) are derived based on the features provided, which in this case are categorical features derived from the linguistic and visual analysis and the multimodal framework (i.e. metafunctions and rank). The K-modes algorithm defines the clusters based on the number of matching categories between data points – in this case, the vectors arising from the text analysis, image analysis and the metafunctional framework. In other words, the algorithm produces clusters depending on the dis/similarities between the features of the articles, which in this case is derived from the linguistic and visual analyses. As the process involves unsupervised learning with no prior knowledge of the texts, the Elbow Method (He, Deng and Xu 2006) is used to calculate the optimal number of clusters, beyond which no significant gain (in terms of average dispersion) is obtained by increasing k, the number of clusters. The results of the cluster analysis (for k = 9) are displayed using an interactive webbased visualization application in Figure 11.4. Each cluster (represented as a spiral) has a centroid with a mean value for that cluster, and the clusters are positioned according to their relations to the cluster at the centre of the display (in this case Cluster 8). The articles in the cluster (i.e. represented as points on the spiral) are positioned according to their relations to the centroid. Individual results for each article are displayed as separate webpages which can be accessed by clicking on the point. The webpages contain the article image and text (Figure 11.3a), the URL of the

DISCOURSE ANALYSIS AND SEMIOTICS

243

FIGURE 11.3a  Just Terror article: Image and text.

source article and a set of tables with the results of the language and image analyses (Figures 11.3b and 11.3c). The image from the Just Terror movie poster (Figure  11.1) appears in Cluster 6, positioned on the right of Figure  11.4. This is the largest and most diverse cluster

244

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 11.3b  Language analysis for Just Terror article.

containing 61 webpages with 11 different images. To interpret and evaluate the cluster analysis results, a Sankey diagram consisting of the: 1. type of visual bonding icon; 2. Watson category; and 3. website classifications derived from UClassify8 Taxonomy classifier and Tonality classifier (corporate or personal) was constructed, displayed in Figure 11.5.

DISCOURSE ANALYSIS AND SEMIOTICS

245

FIGURE 11.3c  Image analysis for Just Terror article.

FIGURE 11.4  K-Modes cluster visualization.

As can be seen from the Sankey diagram, the images in Cluster 6 are predominantly ICONS (i.e., ISIS HEROES and ISIS RELICS), together with ISIS advertisements, and images of enemy soldiers and actions by ISIS against local enemies. The Watson categories are predominantly Society, with others being Law, Government and Politics and, to a lesser extent, Religion and Spirituality. These categories form part of the

246

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

FIGURE 11.5  Clusters with image categories, Watson categories and website taxonomy and tonality.

taxonomy of News and Politics, with a corporate tone, following uClassify classifications of the websites. Closer inspection of the articles and websites reveals that the majority of the articles with the images appear on Western news media sites, as suggested by Figure 11.5. The results are promising, showing how multimodal texts can be clustered and analysed in order to display similar features in terms of linguistic and visual choices. The results are quite different when the linguistic and visual analyses are conducted separately. That is, the clusters for the text analysis (k = 6) and image analysis (k = 2, 3 and 4) are tightly grouped, showing similarities between the language and images in the articles. This suggests that a multimodal analysis approach is required in order to explore the features of multimodal texts. Most importantly, however, when the computational tools are mapped to the multimodal framework (see Table  11.4), it becomes evident how few multimodal semiotic systems are actually accounted for in the computational algorithms provided by major technology companies. In particular, gaps exist at the level of the clause for language and interpersonal meaning for images, and the textual metafunction for text and images. These findings suggest that multimodal frameworks have the potential to improve computational approaches to big data analytics by: 1. providing an overall coherent approach to multimodal analysis and 2. identifying areas in which computational tools need to be developed.

DISCOURSE ANALYSIS AND SEMIOTICS

247

TABLE 11.4  MDA framework and IBM Watson and Clarifai algorithms Metafunction

Rank

System

Description

Discourse

Configurations of activities Content in relation to knowledge construction IBM Watson – categorical values

Clause

Processes; Participant Roles; Circumstance

Happenings, actions and relations

Discourse

Appraisal

Evaluation in terms of attitude, emotion and judgement IBM Watson – sentiment and emotion analysis for each paragraph

Clause

Speech function

Exchange of information (e.g. statements and questions) and goods & services (e.g. commands and offers)

Textual

Clause & Discourse

Information Focus

Organization of information, with points of departure for what follows

Logical

Clause Complex

Logico-Semantic Relations

Logical relations between happenings, actions and relations Conjunctive adverb analysis

Work

Narrative Theme; Representation; Setting

Nature of the scene Clarifai – concepts

Episode

Processes; Participant Roles; and Circumstance

Visual happenings, actions and relations

Figure

Posture; Dress

Characteristics of the participants Clarifai – demographics

Work

Angle; Camera Distance; Lighting

Visual effects

Episode

Proportion in Relation to the Whole Image: Focus; Perspective

Happenings, actions and relations with respect to the whole image Clarifai – focus

Figure

Gaze-Visual Address

Direction of participant’s gaze as internal to image or external to viewer

Text Experiential

Interpersonal

Images Experiential

Interpersonal

248

Metafunction

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Rank

System

Description

Work

Compositional Vectors; Framing

The organization of the parts as a whole, with the visual marking (e.g. framing) of certain parts

Episode

Relative Placement of Episode; Framing

Position of the happenings, actions and relations in relation to the whole image, and the visual marking of certain aspects

Figure

Relative Placement of the Figure within the Episode; Arrangement; Framing

Position of figures in relation to happenings, actions or relations, and the visual marking of certain aspects of those figures

Figure

Gaze-Visual Address

Direction of participant’s gaze as internal to image or external to viewer

Text Textual

FUTURE DIRECTIONS: MULTIMODAL SEMIOTIC SCIENCES Multimodal discourse analysis holds much promise for analysing and interpreting communication in the digital era. In particular, the social semiotic approach offers a way of modelling semiotic resources as interacting systems of meaning and analysing multimodal texts as clusters of multimodal choices, following Halliday’s (2008) foundational principle of ‘language as system’ and ‘language as text’. As Lemke (2000) explains, semiosis involves different scales from the materiality of signs to acts of interpretation which are enabled and at the same time constrained by semiotic resources and socio-cultural practices (e.g. see van Leeuwen 2008). If semiotic behaviour is to be captured, mapped and interpreted across space and time, then multimodal contextual-based approaches which are capable of managing the complexity of large scale semiotic analysis are required. Inevitably, this will involve the development of interactive digital approaches to MDA. For example, it is possible to set up cloud-based platforms with facilities for collecting, indexing, storing and analysing online communications using multimodal frameworks and computational models for big data, cloud computing, natural language processing, image processing, video processing and contextual metadata, together with interactive visualizations and dashboard for displaying the results. The mixed methods big data approach appears promising, based on preliminary studies of online news and social media which reveal differences about how official news reports differ from public reactions on social media. For example, George Floyd’s death on 25 May 2020 gave rise to a higher collective level of negative reactions in online news media and social media, compared to the polarizing and highly differentiated response to the Covid-19 pandemic during the mid-2020s (O’Halloran, Pal and Jin 2021). Furthermore, it was possible to demonstrate

DISCOURSE ANALYSIS AND SEMIOTICS

249

that concerns about the coronavirus pandemic at this time are related predominantly to politics in the online newspapers, compared to health and well-being in social media. Semiotics as the foundational science of signs holds much promise for understanding society and culture in the age of digital media which has revolutionized the world socially, culturally, politically and economically. In particular, mixed methods approaches to multimodal discourse analysis provide the means for mapping and interpreting semiosis in ways which have not been possible before, given advances in technology and the contextual information which is now available. This includes metadata about communications (e.g. time, place, source, author, etc.) and semantic classifications which span the entire breadth of human activity (e.g. Wikipedia). From this perspective, the development of the multimodal semiotic sciences must necessarily be a collective multi-disciplinary effort involving discourse analysts, cultural semioticians, computer scientists and scientists. The need to bring together social semiotic theory and technology to develop the multimodal sciences for mapping and understanding human activity is evident. Furthermore, it is clearly an urgent research goal if we are to address major issues in the world today.

NOTES 1 2 3 4 5 6 7 8

http://multimodal-analysis.com/products/multimodal-analysis-image/software/index.html https://en.wikipedia.org/wiki/November_2015_Paris_attacks https://tineye.com/ https://www.longwarjournal.org/archives/2016/01/before-paris-attack-islamic-stateterrorists-committed-grisly-executions.php https://www.longwarjournal.org/ https://www.ibm.com/uk-en/cloud/watson-natural-language-understanding https://www.clarifai.com/ https://www.uclassify.com/

REFERENCES Barthes, R. ([1957] 1972), Mythologies, trans. A. Lavers, London: J. Cape. Barthes, R. (1977), Image-Music-Text, London: Fontana. Bateman, J., J. Wildfeuer and T. Hiippala (2017), Multimodality Foundations, Research and Analysis – A Problem-oriented Introduction, Berlin: de Gruyter Mouton. Bednarek, M. and H. Caple (2012), News Discourse, London: Continuum. Djonov, E. (2007), ‘Website Hierarchy and the Interaction between Content Organization, Webpage and Navigation Design: A Systemic Functional Hypermedia Discourse Analysis Perspective’, Information Design Journal, 15 (2): 144–62. Djonov, E. and S. Zhao (2014), Critical Multimodal Studies of Popular Discourse, London and New York: Routledge. Forceville, C. (1996), Pictorial Metaphor in Advertising, London: Routledge. Forceville, C. and E. Urios-Aparisi (2009), Multimodal Metaphor, vol. 11, Berlin: Walter de Gruyter. ­Halliday, M. A. K. (1978), Language as Social Semiotic, London: Edward Arnold. Halliday, M. A. K. (2008), Complementarities in Language, Beijing: Commercial Press. Halliday, M. A. K. (2009), Collected Works of M. A. K. Halliday (10 Volume Set), London and New York: Continuum.

250

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Halliday, M. A. K. and R. Hasan (1985), Language, Context and Text: Aspects of Language in a Social-Semiotic Perspective, Waurn Ponds, VIC: Deakin University. Halliday, M. A. K. and C. M. I. M. Matthiessen (2014), Halliday’s Introduction to Functional Grammar, London: Routledge. He, Z., S. Deng and X. Xu (2006), ‘Approximation Algorithms for K-Modes Clustering’, in D. S. Huang, K. Li and G. W. Irwin (eds), Computational Intelligence, ICIC 2006, Lecture Notes in Computer Science, 4114, 296–302, Berlin: Springer-Verlag. Huang, Z. (1997), ‘Clustering Large Data sets with Mixed Numeric and Categorical Values’, in H. Motoda and H. Lu (eds), Proceedings of the First Pacific Asia Knowledge Discovery and Data Mining Conference, Singapore, 21–34, Singapore: World Scientific. Huang, Z. (1998), ‘Extensions to the K-Modes Algorithm for Clustering Large Data Sets with Categorical Values’, Data Mining and Knowledge Discovery, 2 (3): 283–304. Iedema, R. (2003), ‘Multimodality, Resemiotization: Extending the Analysis of Discourse as a Multisemiotic Practice’, Visual Communication, 2: 29–57. Ingram, H. J. (2018), ‘Islamic State’s English-Language Magazines, 2014–2017: Trends & Implications for CT-CVE Strategic Communications’, International Centre for Counter Terrorism, 1–46. Jewitt, C., J. Bezemer and K. L. O’Halloran (2016), Introducing Multimodality, London: Routledge. Knox, J. (2007), ‘Visual-Verbal Communication on Newspaper Home Pages’, Visual Communication, 6 (1): 19–55. Kress, G. (2011), ‘Multimodal Discourse Analysis’, in J. P. Gee and M. Handford (eds), The Routledge Handbook of Discourse Analysis, Abingdon: Routledge. Kress, G. and T. van Leeuwen (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, London: Arnold. Kress, G. and T. van Leeuwen (2006), Reading Images: The Grammar of Visual Design, 2nd edn, London: Routledge. Kress, G. and T. van Leeuwen (2020), Reading Images: The Grammar of Visual Design, 3rd edn, London: Routledge. https://doi.org/10.4324/9781003099857. Lemke, J. L. (2000), ‘Opening up Closure: Semiotics Across Scales’, in J. Chandler and G. van de Vijver (eds), Closure: Emergent Organizations and their Dynamics, vol. 901, Annals of the NYAS, 100–11, New York: New York Academy of Science Press. Lotman, Y. ([1984] 2005), ‘On the Semiosphere’, Sign System Studies, 33 (1): 201–29. Lotman, Y. and B. Uspensky (1978), ‘On the Semiotic Mechanism of Culture’, New Literary History, 9 (2): 211–32. Machin, D. (2013), ‘What Is Multimodal Critical Discourse Studies?’ Critical Discourse Studies, 10 (4): 347–55. Machin, D. (2016), ‘The Need for a Social and Affordance-driven Multimodal Critical Discourse Studies’, Discourse & Society, 27 (3)A: 322–34. Machin, D. and A. Mayr (2012), How to Do Critical Discourse Analysis: A Multimodal Introduction, London: Sage. Martin, J. R. (1992), English Text: System and Structure, Amsterdam: John Benjamins. Martin, J. R. (2017), ‘Revisiting Field: Specialized Knowledge in Secondary School Science and Humanities Discourse’. Martin, J. R. and D. Rose (2007), Working with Discourse: Meaning beyond the Clause, 2nd edn, London: Continuum. ­Martin, J. R. and P. R. R. White (2005), The Language of Evaluation: Appraisal in English, London: Palgrave Macmillan.

DISCOURSE ANALYSIS AND SEMIOTICS

251

Norris, S. (2004), Analyzing Multimodal Interaction: A Methodological Framework, London: Routledge. Norris, S. and R. H. Jones (2005), Discourse in Action: Introducing Mediated Discourse Analysis, London: Routledge. Nöth, W. (2007), ‘The Death of Photography in Self-reference’, in W. Nöth and N. Bishara (eds), Self-reference in the Media, 95–106, Berlin: Walter de Gruyter. O’Halloran, K. L. (2008), ‘Systemic Functional-Multimodal Discourse Analysis SFMDA: Constructing Ideational Meaning Using Language and Visual Imagery’, Visual Communication, 7: 443–75. O’Halloran, K. L. (2015), ‘The Language of Learning Mathematics: A Multimodal Perspective’, The Journal of Mathematical Behaviour, 40 (Part A): 63–74. O’Halloran, K. L. (2021), ‘Multimodal Discourse Analysis’, in K. Hyland, B. Paltridge and L. Wong (eds), Bloomsbury Companion to Discourse Analysis, 2nd edn, 249–266, London: Bloomsbury. O’Halloran, K. L., S. Tan, P. Wignell, J. Bateman, D. S. Pham, M. Grossman and A. Vande Moere (2016), ‘Interpreting Text and Image Relations in Violent Extremist Discourse: A Mixed Methods Approach for Big Data Analytics’, Terrorism and Political Violence, 31 (3): 454–74. O’Halloran, K. L., S. Tan, D. S. Pham, J. Bateman and A. Vande Moere (2018), ‘A Digital Mixed Methods Research Design: Integrating Multimodal Analysis with Data Mining and Information Visualization for Big Data Analytics’, Journal of Mixed Methods Research, 12 (1): 11–30. O’Halloran, K. L., S. Tan and P. Wignell (2019), ‘SFL and Multimodal Discourse Analysis’, in G. Thompson, W. L. Bowcher, L. Fontaine and D. Schönthal (eds), The Cambridge Handbook of Systemic Functional Linguistics, 433–61, Cambridge: Cambridge University Press. O’Halloran, K. L., S. Tan, P. Wignell, R. Wang, K. Chai and R. Lange (2020), ‘Towards a Discipline of Multimodality: Parallels to Mathematics and Linguistics and New Ways Forward’, in J. Wildfeuer, J. Pflaeging, C. I. Tseng, O. Seizov and J. Bateman (eds), Multimodality: Towards a New Discipline, 35–54, Berlin: de Gruyter. O’Halloran, K. L., G. Pal and M. Jin (2021), ‘Multimodal Approach to Analysing Big Social and News Media Data’, Discourse, Context and Media, 40. https://doi.org/10.1016/j. dcm.2021.100467 O’Toole, M. (2011), The Language of Displayed Art, 2nd edn, London and New York: Routledge. Scollon, R. (2001), Mediated Discourse: The Nexus of Practice, London: Routledge. Scollon, R. and S. W. Scollon (2004), Nexus Analysis, London: Routledge. Speckhard, A. (2015), ‘The Hypnotic Power of ISIS Imagery in recruiting Western Youth’, in ISCVE Report, International Center for the Study of Violent Extremism. Stenglin, M. (2008), ‘Interpersonal Meaning In 3D Space: How a Bonding Icon Gets Its “Charge”’, in L. Unsworth (ed.), Multimodal Semiotics: Functional Analysis in Contexts of Education, 50–66, New York: Bloomsbury Publishing. Stöckl, H., H. Caple and J. Pflaeging (2020), Shifts towards Image-centric Practices in the Contemporary Multimodal Practices, London: Routledge. Tan, S. and K. L. O’Halloran (2012), Multimodal Analysis Image (Teacher edn and Student edn), Singapore: Multimodal Analysis Company. ­Tan, S., K. L. O’Halloran and P. Wignell (2020), ‘Multimodality’, in A. De Fina and A. Georgakopoulou (eds), The Cambridge Handbook of Discourse Studies, 263–81, Cambridge: Cambridge University Press.

252

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Tannen, D., H. E. Hamilton and D. Schiffrin (2015), The Handbook of Discourse Analysis, 2nd edn, New Jersey: John Wiley & Sons. van Leeuwen, T. (1999), Speech, Music, Sound, London: Macmillan. van Leeuwen, T. (2001), ‘Semiotics and Iconography’, in T. van Leeuwen and C. Jewitt (eds), Handbook of Visual Analysis, 92–118, London: Sage. van Leeuwen, T. (2005), Introducing Social Semiotics, London: Routledge. van Leeuwen, T. (2008), Discourse and Practice: New Tools for Critical Discourse Analysis, Oxford: Oxford University Press. van Leeuwen, T. (2012), ‘Critical Analysis of Multimodal Discourse’, in C. A. Chapelle (ed.), The Encyclopedia of Applied Linguistics, 4002–6, London: Blackwell. van Leeuwen, T. (2015), ‘Multimodality’, in D. Tannen, in H. E. Hamilton and D. Schiffrin (eds), The Handbook of Discourse Analysis, 2nd edn, 447–65, New Jersey: John Wiley & Sons. Wignell, P., S. Tan and K. L. O’Halloran (2016), ‘Violent Extremism and Iconisation: Commanding Good and Forbidding Evil?’ Critical Discourse Studies, 14 (1): 1–22. Wignell, P., S. Tan and K. L. O’Halloran (2017), ‘Under the Shade of AK47s: A Multimodal Approach to Violent Extremist Recruitment Strategies for Foreign Fighters’, Critical Studies on Terrorism, 10 (3): 429–52. Wignell, P., K. L. O’Halloran, S. Tan, R. Lange and K. Chai (2018), ‘Image and Text Relations in ISIS Materials and the New Relations Established through Recontextualisation in Online Media’, Discourse & Communication, 12 (5): 535–59. Wignell, P., S. Tan, K. L. O’Halloran, R. Lange, K. Chai and M. Wiebrands (2020), ‘Images as Ideology in Terrorist-related Communications’, in H. Stöckl, H. Caple and J. Pflaeging (eds), Shifts towards Image-centricity in Contemporary Multimodal Practices, 253–74, New York: Routledge. Wignell, P., K. Chai, S. Tan, K. L. O’Halloran and R. Lange (2021), ‘Natural Language Understanding and Multimodal Discourse Analysis for Interpreting Extremist Communications and the Re-use of these Materials Online’, Terrorism and Political Violence, 33 (1): 71–95.

CHAPTER TWELVE

Integrational Linguistics and Semiotics ADRIAN PABLÉ

SEMIOTIC IDEOLOGY: THE THREE ONTOLOGIES OF THE SIGN What discipline do you turn to when you wish to know about the nature of the sign? Semiotics, as the general study of signs, would seem the most obvious place to look for enlightenment. The two related questions ‘What is a sign?’ and ‘How does a sign acquire its meaning?’ have been approached from all kinds of angles in semiotics, depending on the various schools and theorists. As such, one would expect that these two questions have been settled definitely or at least discussed exhaustively, with little more to say about the ontology of signs. This was also the opinion of German linguist Rudi Keller, author of a work on theories of the sign entitled Zeichentheorie. Zu einer Theorie semiotischen Wissens (Keller 1995), translated into English three years later as A Theory of Linguistic Signs (Keller 1998). Keller (1995: 9) draws the following conclusion about the history of semiotic theory: ‘Alles was über sprachliche Zeichen gesagt werden kann, ist vermutlch irgendwann zwischen Platon und heute gesagt worden. In einem sprachphilosophischen Gebiet mit mehr als zweitausendjähriger Tradition lässt sich wirklich Neues wohl kaum noch entdecken.’ [Everything that can be said about linguistic signs has probably been said sometime between Plato and today. In a language-philosophical field that is more than two thousand years old, it seems unlikely that anything new will ever be discovered.] The present contribution would beg to differ on this point. In fact, Keller’s verdict corresponds pretty much to the official position in semiotics, with linguistic signs being only one type of sign among many others. When it comes to questions about the ontology of the sign, one endorses either a semiological position (i.e. Saussurean and Post-Saussurean dyadic sign conceptions) or the much older ‘semiotic’ position, which construes signs in a broader sense (i.e. the various ‘surrogational’ traditions, most notably the Peircean triadic sign model). It would thus seem that everything has been said about what a sign is and how it acquires meaning. What else could it be than a substitute for – or surrogate of – something other than itself, and its meaning being what the sign replaces (or ‘stands for’)? This conception broadly aligns with lay thinking about signs. Even Saussure had to concede that at the level of parole words are taken by the language users to be substitutes for material things or for private ideas in the mind. Saussure’s definition of the linguistic sign as having a purely psychological ontology – a ‘psychological reality’, as he insisted – and leading a separate real existence from its material manifestation as a word was the

254

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

idealist answer to lay materialist semiotics. What in ordinary English, French, Italian, etc., is called a ‘sign’ is not a sign in Saussurean terms, but the word sign, which identifies material ‘signs’, is one. According to Saussure, the human capacity to mentally construct collective systems of signs was to be at the centre of semiology as a ‘science’, while material ‘signs’ were relegated to objects like any others. Other forms of idealism, e.g. Benedetto Croce’s aesthetic science, equally rejected the materiality of art (including language) and privileged the individual’s mental creative powers. Unlike Saussure, Croce had no interest in a theory of signs that dealt with either abstractions or physical stimuli (Dessì Schmid 2004: 699), neither of which concerned language as ‘perpetual creation’. The Italian idealist linguistic schools of a Crocean stamp, among them Giulio Bertoni’s idealist ‘science’, exerted very little influence on general linguistics as a contemporary academic field, remaining limited to Romance linguistics (in spite of their universalist outlook). One reason for this was undoubtedly the idealist rejection of positivism as embodied in the Neogrammarian school (Bonfante 1947). General linguistics took a Saussurean turn instead and made Saussure’s idealism compatible with an empirical-historical study of language inasmuch as synchrony and diachrony came to be regarded as different perspectives of the same reality. Saussure’s dictum that it is ‘the viewpoint that creates the object’ (Saussure 1983: 8) was thus ignored in structuralist sociolinguistics and replaced by a ‘multi-fixed-code’ theory able to account for the alleged continuity of language variation and change (Weinreich, Labov and Herzog 1968). In other words, the ‘history of English’ (or of any other language for that matter) was to be tackled from within parole, however without having to give up the notion of langue. This also meant that linguists no longer had to bother about a strict separation between the word and the linguistic sign, and as a consequence they did not hesitate to treat languages as fixed-codes with continuously changing forms but relatively stable meanings. From the twentieth century onwards, the questions ‘What is a sign?’ and ‘How does a sign acquire its meaning?’ have thus had two possible answers within mainstream semiotics/semiology, roughly aligning with a Peircean and a Saussurean stance. However, this view is not representative of the whole range of ontologies of the sign proffered by theorists. This chapter will introduce a third approach to sign ontologies. Like Croce’s view of language as pure creation, it emphasizes human creativity, however without subscribing to any form of idealism (Pablé 2018). To the best of the author’s knowledge, no fourth ontology has been proposed and it is not likely that one ever will be. This third ontology has largely gone unnoticed in semiotics (with few exceptions, see Cobley 2011, 2014) and although it originated as part of semiology it has not received much attention in linguistics, a discipline (as mentioned above) with an ever-declining interest in theoretical questions about the linguistic sign and a predilection for empirical questions about phonetic variations in words. This third approach has been described as too ‘radical’ by empirically minded linguists (Linell 2018) or has been criticized for making nonsense of the abstract notion of signhood as established by Saussure (Weigand 2018). Being founded on a semiological approach, this third ontology has much to say about human signs but very little about non-human signs. Being semiological, moreover, it also rejects the ‘standing for’ formula common to semiotics (Daylight 2012). The sign conception at issue was developed by Oxford linguist Roy Harris (1931–2015) in the late 1970s, as part of an approach to language and communication named ‘integrationism’ and ‘integrational linguistics’, respectively. The latter approach is said to apply the insights of integrationism to the particular case of language (Harris 1998: 4). Throughout his work, Harris referred

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

255

to integrational linguistics as ‘lay-oriented’ (e.g. Harris 1981). The integrational sign conception, however, is not a lay conception any more than Saussure’s was. Throughout the history of Western lay linguistics, signs were construed as having a stable relation to something (other than itself) in the real world: the dark clouds in the sky are a natural sign of bad weather, while the words used to make that statement are conventional signs mirroring a state of affairs in the non-linguistic world. Both Saussure and Harris regarded the lay view of signs as inadequate for semiology, though they proposed very different answers to the question what a sign really was. The difference between their two sign conceptions is best explained as the difference between the abstract collective sign versus the contextualized personal sign. The belief that signs have a certain ontology is always embedded in a certain theory of the sign and presupposes a certain worldview. Keane (2018) speaks of ‘semiotic ideologies’ and uses this concept to denote people’s underlying assumptions of what signs are and how they function. Keane (2018: 66) explains that it is a matter of semiotic ideology ‘whether signs are taken to be interpretable because their relation to the world is arbitrary, or logical, or natural, or divinely ordained’. He is thus interested in semiotic ideologies insofar as they tell us what counts as a sign (and a sign of what) in a certain community among different sign-users. For the integrationist, ‘semiotic ideology’ is equally of interest as a term if extended to cover semioticians’ and linguists’ assumptions about the ontology of the sign and what conditions must be given for signs to exist. Integrationists thus take an interest in philosophy of linguistics, as Harris (1993) made clear, but theirs is also a commitment to philosophy of semiotics. Keane talks about icons and indexes as if the sign classification itself was not ‘ideological’, i.e. the result of what he (2018: 67) calls ‘semiotic reflexivity’ – and what would be subsumed under the label ‘decontextualization’ in Harrisian terms. For Keane, semiotic ideology guides people ‘as they sort out which aspects of their experience are or are not candidates for even being signs at all’ (2018: 69). But Keane seems oblivious to the possibility that this very view of semiotic ideology already presupposes a certain view of the sign. In Keane’s case it is a view linked to a human world in which only certain things become sign-mediated, in turn becoming different ‘objects’ in different cultures, societies and times. Thus Keane (2018: 69) asks the ontological question: what are possible objects for a sign? Keane discusses semiotic ideology based on the Peircean triadic sign model, presenting it as promoting a ‘quasi-scientific’ understanding of what a sign is. The tripartition into iconic, indexical and symbolic signs is thus treated as a universal semiotic reality. However, Keane (2018: 80) admits that analysts and laypeople do not always agree on the reality of an object or what the object is (i.e. what the sign ‘stands for’). And therefore there is also disagreement about the sign modality. Keane (2018: 82) concludes that in ‘interpretive communities’ sign vehicles remain the same while objects change. Keane (2018: 69) tells the reader that his semiotic approach is done ‘in the spirit of pragmatism’: i.e. his aim is to find ‘usable concepts for empirical purposes’. Against this background the legitimate question then arises whether the integrational conception of the sign may itself be ‘ideological’, e.g. motivated by the purpose of not being empirical. In other words, can there be a theory of the sign free of any disciplinary impositions? Like Saussure’s semiology, integrational linguistics is founded on a self-reflexive theory of the sign, but presupposes an altogether different view concerning the centrality of signs in human life. Just as Saussure declared that all questions about the sign were inseparable from raising these questions by means of linguistic signs, Harris introduced the revolutionary idea that signs were what their sign-makers ‘made’ them: no sign without a

256

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

sign-maker, including the word sign. Neither Saussure nor Harris thus believes that there is a meaningful world detached from the collective linguistic system or the individual sign-maker, respectively. Semiologists maintain that signs are the human reality and that the separation between signs and objects is the result of the human imagination of a detached but still graspable reality. Harris devoted much of his career to the critique of an all-encompassing ‘semiotic ideology’ which he termed the ‘Language Myth’ (Harris 1981). One facet of this myth is precisely the belief that there are things in the real world identified by their true names: e.g. the word universe ‘stands for’ the universe, whereby the universe exists independently of humans attaching a linguistic label (‘universe’) to it. What makes the Harrisian approach to the sign different from both the Saussurean and the Peircean counterparts is the insight that nobody (including the analyst or theorist) can tell what a sign means for somebody else, let alone for a whole community (e.g. the word universe). The objection that certain communities agree on what a sign means, and therefore that it is possible to tell what the sign means for at least some people (e.g. the word universe for the scientific community) is not an argument that an integrationist would endorse. On the contrary, the integrationist discards abstract and reductionist terms like Keane’s ‘interpretive community’ because they do not explain individuals’ meaning-making in given circumstances. Ultimately, the integrationist argues, resorting to entirely macrosocial explanations of sign-making (sign-makers A, B, C, D belong to the same ‘interpretive community’) does not take into consideration the individual’s personal communicational biography, replacing it with an abstract communal biography on which individual communication allegedly rests. In the latter framework individual variations are still entirely possible but are based on a shared underlying core. In other words, there is a communal psychological ‘fixed-code’ already in place. The question that Harris (1996: 7) invites his readers to ponder is precisely whether ‘communication presupposes signs’ or whether ‘signs presuppose communication’.

SIGNS AND ACTIVITIES Harrisian integrationism, discussed most comprehensively in the books Signs, Language and Communication (Harris 1996) and Introduction to Integrational Linguistics (Harris 1998), takes an altogether humanist and anthropocentric view on signs and sign-making. Unlike more traditional forms of humanism, it does arguably not operate with ethnocentric linguistic categories, even though Harris’s intellectual formation occurred in a Eurocentric context. At the same time integrationism claims to be ‘lay-oriented’ in a way that renders its understanding of human communication immune to the charge of ethnocentrism (Pablé 2019). In fact, Harris rejected the traditional humanist – or what Sebeok (1975) called the ‘glottocentric’ – views about communication and the role that ‘language’ and ‘languages’ play within these paradigms. Harris (2010) argued that linguistic theory has to start with ‘understanding linguistic experience from the inside’ (and hence the ‘lay-orientation’), instead of promoting itself as an empirical science tasked with interpreting other people’s sign-making. In fact, mainstream linguistics, despite its claims to being ‘scientific’, could actually be accused of harbouring a cultural bias in the way it investigates and describes human languages: thus while defining ‘languages’ as abstract systems of signs shared by the members of a collectivity (Saussure’s langue), it also proclaims the professional linguist to be an expert, whose knowledge transcends the limitations of the lay linguistic user. The view of ‘languages’ as consisting each of identifiable – though only interdependently

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

257

existing – abstract signs amenable to systematic description was indeed the result of a Western idea, originally developed in nineteenth-century Europe, and modelled on the natural sciences, which, as Harris argued in his challenging book The Semantics of Science (Harris 2005), equally subscribe to traditional humanist assumptions about how speaker A can rationally and successfully communicate with hearer B under ideal circumstances (e.g. both need to be native speakers of the same language, they need to have been brought up in a Western cultural context, etc.). On this view, science is grounded in a Eurocentric conception of how language relates to the world (the ‘semiotic ideology’ of science), i.e. the belief in true names, which correctly identify, irrespective of contextual factors, the things that they are names of (i.e. what they ‘stand for’). For the integrationist, in turn, there are no linguistic experts because there are no scientifically describable objects called ‘languages’ nor independently existing words of ‘a language’. Integrationists insist, however, that a theory of communication has to be semiologically grounded (Harris 1996: 12). Unlike Saussure’s semiology, Harrisian semiology does not take the linguistic sign qua psychological unit as its point of departure. In the integrational conception, the sign serves the activities it integrates for someone under the present circumstances. Anything (articulated sounds, material objects, etc.) can be a sign if someone bestows on it a semiological function as part of a programme of activities he or she is engaged in, but more often than not it would not be possible to identify ‘the sign’ or ‘the signs’ involved in the communicative situations typically analysed in linguistics (and how many there are exactly). The reason why this is not possible is the temporality that characterizes the sign: in fact, in order to analyse communication in terms of signs you need to decontextualize them (i.e. treat them as atemporal). The integrationist denies precisely this, namely that there is an ‘objective’ way of representing the (linguistic) signs which, mirabile dictu, corresponds exactly to how the communicants themselves experienced the signs when they made them. Harris’s own illustrative examples of what a sign is focus mostly on everyday situations in which something familiar is imagined to happen (sometimes involving Harris himself): someone knocks on the door, a landmark acquires a signalling function on one’s ride home, someone asks a question and someone else replies or does something nonverbally, someone sees a cat at the end of one’s garden and tells somebody else about it. Harris does not seem interested in engaging in ‘conversation analysis’, which calls for a third-person perspective: the complexity of such exchanges involving an open range of cotemporal activities can only be analysed at the cost of ‘segregating’ the linguistic from the non-linguistic activities, and artificially putting them together again a posteriori (as, e.g., in multimodal approaches to language). Other integrationists have provided more detailed introspective analyses of sign-making activities (e.g. Jones 2011; Conrad 2020) or have analysed the unfolding of longer stretches of other people’s written discourse, though without any accompanying ‘mind-reading’ claims (e.g. Duncker 2018). At any rate, all of these analyses by integrationists serve first of all to explicate the ontological nature of the sign and its embeddedness in human activities. Integrationists focus on the notion of activity as a fundamental lay concept when it comes to understanding human communication, for which they need no specific ‘theory’ of activities. Instead they start from the insight that communication is a process which consists of activities that are integrated by means of contextualized signs of all kinds. At the centre of an integrational theory are thus the concrete first-order sign-making processes that individuals (not members of ‘interpretive communities’) engage in, for whom the ‘linguistic’ and the ‘non-linguistic’ are precisely not separate but always integrated domains of knowledge, as stated by the Harrisian ‘principle of cotemporality’ (Harris

258

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

1998). In other words, what matters in human life is that activities of whatever kind are integrated by whatever semiological means are available to individuals engaged in specific programmes in order to further their specific communicational objectives. There is no detachable ‘linguistic message’ that is experienced identically by speaker A and hearer B (or analyst C). Communication, as far as human beings are concerned, cannot be analysed ‘scientifically’ in terms of ‘verbal’ and ‘non-verbal’. Not because lay speakers would not analyse communication in this way, but precisely because communication is not a fact but a lay perspective (Harris 2006a: 13). In other words, communication is radically open-ended (both ontologically and in its interpretations). This, in turn, means that communication needs to be treated as a much broader concept than any ‘glottocentric’ (Saussurean) model allows for. Communication, for the integrationist, is bound to one’s sign-making. Realizing the one activity is linked to another activity (in whatever way this link is construed) presupposes the making of signs. This communication with oneself – self-communication, as Harris (1996) calls it – may in turn lead to further activities (thinking, speaking, gesturing, etc.). What is radically open-ended about communication is not only what is part of it and for whom, but also when communication begins and ends, between whom, and which personal past and future experiences combine in order to make the present communication episode meaningful. Regarding the ontology of the sign, the integrationist holds that the signs exist only insofar as they serve an integrational purpose: once they have fulfilled their integrational function, the signs no longer exist. As Harris puts it: ‘the sign exists as a construct by reference to which certain communicational activities [. . .] are integrated’ (2006b: 715). It is important to keep in mind that integrationists deny that signs ‘pre-exist’ in any way, which is tantamount to rejecting the abstract sign altogether. In other words, there is no type that is repeatable innumerable times as a token, i.e. as the ‘same’ sign in different situations. Rather it is the other way round: it is the recurring of tokens that allows for the creation of the type as a product of the here-and-now (Ellis 1994; Harris 2013). ‘Sameness’ is surely a semiological process, constrained by cultural factors, that people recognize as playing an important communicational role, and as such integrationists do not necessarily regard it as problematic. However, its rejection at the level of the sign ontology is of paramount importance: arguably it is that very rejection which makes the Harrisian sign incompatible with both the Saussurean and the Peircean sign. What lay people might (retrospectively) argue is the ‘same sign’ produced several times (within a particular reconstructed communication episode) must be, from an integrational point of view, unique signs made on the various occasions along the sign-maker’s biographical continuum. In this respect, the integrationist does not accept that there are ‘facts’ (linguistic and otherwise) that can be established independently of the signmakers themselves (Harris 1998). Crucially, however, even a sign repeated is, from an integrational semiological point of view, a creation ex nihilo: its creation occurs in a new situation, but its integrational function relies on its being construed by the signmaker(s) as having a predecessor sign. The first sign (e.g. ringing the doorbell for the first time) and the second sign (ringing the doorbell for the second time) cannot possibly mean ‘the same’ for either the person ringing or the person hearing it. However, the two signs might be construed as being ‘the same’ under specific circumstances and for certain communicational purposes. Sameness is never ‘already there’. On Harris’s view, signs are thus the products of contextualization: they both result from and facilitate the integration of particular activities in particular circumstances (Harris 2006a). The signs do not exist prior to or independently of communication because

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

259

the signs are personal creations here-and-now: that is, there are no signs belonging to a metaphysical realm, but only signs that are made (by particular sign-makers) in order for various activities to be integrated. If signs only served one single activity rather than the integration of different activities, human communication would be radically different from how we know it based on personal experience. Activities are recognizable to us as general activities (e.g. thinking, speaking, listening, writing, reading and gesturing), but an activity only exists in a particular moment as engaged in by a particular person. There is only my view (and yours, etc.) on whether what I am doing right now is engaging in the activity of ‘reading’. Nobody else can determine this in my stead. What counts as ‘reading’, in fact, may arguably be a matter on which people might disagree. Moreover, does reading silently and reading aloud (respectively to oneself, to somebody else present or recording it for somebody else not present) count as the same basic activity? For one, reading silently and reading aloud involve very different activities. The same goes for writing a note to oneself as opposed to writing a note for somebody else. Integrationists recognize that humans are always circumstantially determined, aware of themselves, which is why the here-and-now is communicationally constrained for any individual by factors of three kinds, namely (i) biomechanical factors, which relate to the physical and mental capacities of the human being, (ii) circumstantial factors, which relate to the specifics of particular situations, and (iii) macrosocial factors, which relate to practices established in the community (Harris 1998: 29). Integration in any semiologically relevant sense is the individual’s continuous integration of (remembered) past experience and (anticipated) future experience with the here-and-now, which, the integrationist argues, cannot rely on a set of shared abstract signs but necessitates the making of signs. To manage the daily integrational tasks is to know how activities are temporally integrated in the macrosocial patterns of the community and to act accordingly (not doing anything may also be a form of action). Harris (2009a) calls this knowledge ‘integrational proficiency’; however, it is not a ‘proficiency’ that any traditional intelligence tests would be able to disclose (Harris 2013). It is not knowledge atemporally stored in the brain (‘knowing that’) but rather knowledge in the sense of ‘knowing what/how to do’, i.e. knowledge embedded in activities (Harris 2009b). Thus, if I do not know a word I have encountered as part of a particular activity (e.g. reading a newspaper article) and I look it up in the dictionary, and if as a result I can make sense of that word as it occurs in that particular article I am reading, I know its meaning sufficiently well for the purposes at hand, i.e. I know ‘what to do with it’ (Harris 1998: 63). On this view, the idea of knowledge for the sake of knowledge is a product of the Language Myth, i.e. our ability to decontextualize communication and extract from it something called ‘knowledge’ (‘linguistic knowledge’ in the case of the unfamiliar word): it is the belief that knowledge exists independently of the knower and thus independently of individuals integrating their own activities (with other people’s activities) in specific circumstances in order to achieve specific purposes, constrained by the three aforementioned parameters (biomechanical, circumstantial, macrosocial). In other words, for the integrationist, there is no contextless knowledge (Harris 2009b). Harris (1981) specifically rejects two fallacies that characterize the Language Myth as a Western cultural enterprise as well as academic linguistics as a ‘segregationist’ project: they are the fallacies of ‘telementation’ and of the ‘fixed-code’ (or ‘determinacy’), which, taken together, explain the role of words as enabling speakers of the ‘same language’ to transfer mental concepts from one mind to another, resulting in mutual understanding, which Pennycook (2018) identifies as a Eurocentric humanist ideal. The integrationist, however, goes on step farther than Pennycook, who equally rejects the notion of

260

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

‘languages’ as scientifically describable objects, and adds ‘words’ to the list of secondorder abstractions, hence viewing them as products of human communication – not as its prerequisites (e.g. Love 1990, 2017). Since integrationists acknowledge communicational activities of the first order as the only human reality, they view signs as spontaneous creations of the individual’s mind, i.e. they are, as Harris and Hutton (2007) put it, radically indeterminate in both form and meaning. ‘Radical indeterminacy’ refers to the impossibility to understand signs outside of their contexts: any understanding is done in ‘its’ context. In other words, there is no third-person perspective on signs and hence there can be no ‘science’ of semiology (pace Saussure). The signs are what they are and mean what they do to the various sign-makers when they make them. The signs are not radically indeterminate to their makers, though, as the signs they make integrate activities. In order to have any integrational function, signs need to be ‘determinate’ for their makers: failing to understand what the other is saying does not render my signs ‘radically indeterminate’. The signs – i.e. my interpretations – may simply be inadequate, in my perception, to integrate my activities with another person’s activities here-and-now. In order for them to be adequate I do not need to know the signs ‘objectively’. In other words, semiological determinacy, for the integrationist, is not of the (fixed-code) lexicographical formula ‘bachelor:unmarried man’. Signs do not acquire an integrational function the way determinate words are bestowed determinate meaning through the authority of the dictionary. Using a dictionary has certainly become one viable ‘glossing practice’ (Harris 1998) in order to reduce semantic indeterminacy and facilitate certain kinds of communication, but crucially using a dictionary is culturally conditioned: it is a particular (historically grown) activity embedded in other activities. On that score the integrationist agrees with Saussure: the linguistic signs we make are not the same as the words we find in the dictionary, though there must be some perceived connection between them for the individual sign-maker. The contextualized signs I am making when interacting with others, for example, have no first-order connection with the (second-order) words in the dictionary that my signs allegedly correspond to. To ask ‘what are these signs “signs of”?’ is a culturally conditioned question, with many communicationally relevant ramifications, but arguably it hinders a non-ethnocentric understanding of the ontology of the sign. A sign is never a sign of something independently of the activities for which it was created in the first place. Harris also does not treat the sign as having a form and a meaning (as in Saussure’s signifier and signified). When Harris talks about radical indeterminacy in both form and meaning, he does so through the lens of mainstream linguistics. Strictly speaking, a sign ‘made’ (in the Harrisian sense) cannot have ‘a form’ and ‘a meaning’ (Davis 1997). Analysing signs in this way presupposes their decontextualization: i.e. treating signs as detachable from communication. From this it follows that the integrationist rejects the substitute (or ‘surrogational’) sign (Harris 1996), i.e. the very notion that a sign is one thing ‘standing for’ (or representing) another thing. The surrogational sign is not a sign embedded in activities, created for and as part of the activities that particular individuals engage in. In everyday lay discourse it is common to explain meaning surrogationally, but in order to see such a relation between a sign and its ‘object’ (in the Peircean sense), I have to assign some sort of semiological value to something, i.e. make it a sign. The signs never antecede the activities that make them signs in the first place, but it is tempting to think otherwise, especially when it comes to material objects whose signification we assume to be permanent, such as a wedding ring or a traffic sign (Harris 1996). The integrationist equally has to reject the Saussurean sign, i.e. the very notion that signs are defined purely internally by an abstract (linguistic) system they belong to, and this allegedly is how a

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

261

sign is known by its user. Given the temporality of the sign, however, neither form nor meaning can exist as abstractions. In turn, Harris proposes a semiology grounded in firstperson experience: there is no sign without its personal maker.

­INTEGRATIONISM AND BIOSEMIOTICS The only branch of semiotics that shares several points of convergence with Harrisian integrationism is Thomas Sebeok’s global semiotics (Sebeok 1979), which is founded on a Peircean understanding of the sign. Peirce is a ‘surrogationalist’ (Harris 1996), construing a sign, as philosopher and semiotician John Deely (2003: 185) put it, as ‘one thing representing another than itself to yet another’. There is no indication that Sebeok was aware of Harris’s work, while Harris mentioned Sebeok only once in a footnote (1996). In terms of their sign conceptions, global semiotics and integrationism differ from each other in relevant ways, at least from an integrational point of view. In fact, from a semiotic standpoint it could be argued that Deely’s explanation of how organisms with cognitive abilities make meaning comes close to an integrationist position of a more ‘global’ stamp: The organism, according to its own nature and past experiences, attaches a value to the stimulus and relates that stimulus to its own needs and desires. In other words, the mere stimulus of sensation becomes incorporated objectively [i.e., species-specifically] into a whole network of experience wherein it acquires a meaning. (2003: 178–9) However, the concept of ‘activity’ does not take centre stage in Deely’s analysis – in fact, it is not on stage at all. The reason for this is arguably that the Peircean sign only presupposes activities against which the sign-object relation is established by the interpretant. Thus while the activities are clearly there in Peirce, the focus remains on the signs, without which there would be no activities in the first place for any sign relations to be realized. Activities presuppose signs from a Peircean semiotic perspective – which is why the signs and the activities can be considered separately – while signs presuppose activities in Harrisian semiology: the signs, for Harris, are only acknowledged as existing insofar as they integrate (human) activities. Those signs are unique and therefore they only make sense the way they do (to the sign-makers) because of the very activities hereand-now. This is not to deny that sign-makers make connections between the activities here-and-now and the activities of the past (as integrated by them). However, there is no neutral perspective on why someone makes the signs he/she makes, i.e. how exactly they connect to his/her communicational biography. So to say that a sign ‘stands for’ something other than itself to someone cannot capture the dynamic nature of human activities, in which signs facilitate their integration and at the same time result from those integrations. Peirce’s notion of ‘semiosis’ (signs generate further signs) comes close to capturing this dynamism, but, as the integrationist would maintain, the signs by themselves do not exist. Our reluctance to let go altogether of the sign in and by itself (if only as a potential), the integrationist argues, stems precisely from our ability to analyse signs as detached from the activities in which they were created (hence we cling to the idea that a sign is something that ‘stands for’ something else), while we easily forget that this very analysis occurs itself as part of a macrosocially, circumstantially and biomechanically conditioned activity.

262

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

In spite of significant differences in their respective conceptions of the sign, Harrisian integrationism and global semiotics, and in particular biosemiotics (the semiotic branch developed from biologist Jakob von Uexküll’s work [1982]), are united in their philosophical views on the issues of reality and objectivity. Building on Uexküll, John Deely (e.g. 2003, 2009) has developed the most comprehensive biosemiotic account of what constitutes ‘objective reality’ (as opposed to an orthodox scientific account of ‘reality’). In his work, Deely resurrected the intriguing distinction between thing and object, together with the distinction between physical reality and objective reality. His is a rejection of both naïve realism and nominalism, but it is not per se a critique of academic linguistics or its underlying theory of the sign. While not using Deely’s terminology, Harris deals with the same issues in his work on science (Harris 2003a, 2005), however with particular reference to the Language Myth as it pertains to the so-called ‘supercategories’ (e.g. Harris 2003b, 2004, 2005). How are we to understand Deely’s critique of objectivity, and how does it differ from an integrational critique? Building on the notion that the sign is constituted as a triadic relation, Deely affirms that reality is always ‘objective’ because it is species-specific and thus constrained by the sign-making capabilities of a species and its sensoria. Deely’s account makes clear how orthodox science has ignored ‘objective reality’ in the biosemiotic sense, confusing objectivity with the physical environment because of its Baconian goal to investigate the ‘things as they are’, i.e. as God knows them. As Deely rightly argues, only human beings are capable of contemplating objects in relation to the objects themselves, while non-human animals make sense of objects exclusively in relation to their own needs and perception. Contemplating objects for their own sake (i.e. as ‘things’) and to treat them as ‘objectively’ given (not in the biosemiotic sense) is an exclusively human attribute (among the terrestrial species). The cardinal mistake committed by orthodox science, for Deely (2009: 167), thus lies in assuming that the physical environment is the way it is regardless of the creatures that inhabit it and make sense of it. Deely does not go as far as to deny the reality of the physical universe (as a thing): he states, in fact, that there is a physical universe which is the same for all types of organisms, but adds that this physical environment is different from the points of view of the various ‘Umwelten’ (Uexküll [1940] 1982). This comes as close to admitting that ‘reality’ is unknowable as it gets in biosemiotics. On the other hand, Deely (2009: 160) talks extensively about so-called ‘hardcore reality’ when it comes to the relations between subjectivities (substances): thus the relation between a father and his biological son is intersubjectively real: in other words, it is not a sign-mediated relationship. On the other hand, the relation between a woman and her being a witch is sign-mediated – if witches are not real – i.e. it involves a triadic, and not a dyadic, relationship. As Deely argues at length in his work, the distinction between reality and unreality is not a consequential one from the point of view of semiotics. As Deely (2009: 167) puts it, ‘signs are indifferent to reality in the physical sense’. Biosemiotics cannot afford to take a ‘scientific’ point of view, as the latter pretends to be a perspective unmediated by signs. For the semiotician, in fact, science could be seen as a branch of anthroposemiotics, i.e. the institutionalized human activity of investigating a sign-less reality through signs (science communication in its various forms), a reality made of ‘things’, not ‘objects’. Crucially this insight is shared by Harris and elaborated on in his book The Semantics of Science (Harris 2005), i.e. the insight that the study of ‘things as they are’ is intrinsically part of the human umwelt and therefore species-specific. Science cannot exist detached from the integrated human activities that actually make it, including the activities lay people commonly identify as involving ‘language’. In this sense we may conclude that Science, Art, Religion and

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

263

History are human enterprises (among other so-called ‘supercategories’): they make sense the way they do because they are the product of the human ‘communicational universe’ (Harris 1996) – they do not have to be graspable by any languageless creature. Hence it would be absurd to imagine highly intelligent alien races that have ‘science’ as we know it. There simply is no languageless science. On that point, Sebeok and Harris would agree, ‘language’ is an entirely human experience, and only humans can attribute ‘language’ to other creatures (higher-order animals, extraterrestrials, even plants), which, however, does not mean that those creatures possess it (whatever their cognitive abilities). Recent developments within posthumanist approaches to language, such as Leonie Cornips’s suggestion to ‘put the animal on the linguistic research agenda’ (Cornips 2019), would have been rejected out of hand by both Sebeok and Harris. Deely’s critique of science’s failure to distinguish between objects and things is not at the same time a critique of the language of science. The latter is part of the human objective world, while the ‘things’ themselves are not. Sebeok’s semiotics thus views language as the ‘primary modelling device’ that humans make use of in the way they constitute their objective world (Petrilli and Ponzio 2011: 138). It is something additional that non-human animals lack. Harris (2005), in turn, criticized the worldview sustaining the scientific notion of ‘objective reality’ (in the non-biosemiotic sense) as part of his critique of the Language Myth. The very notion of such a reality is only possible because we construe languages to be labelling devices, as Saussure (1983: 16, 65) famously noted in his critique of nomenclaturism. Biosemiotics, for all its material grounding, cannot dismiss altogether Saussure’s idealist explanation of what ‘a language’ is, i.e. a system of abstract signs. Within segregationism, Saussure’s model of communication is the most convincing one, precisely because it constitutes an idealization. Harris, on the other hand, rejects the idea of ‘a language’ altogether (other than as a lay concept), thus dismissing both nomenclaturist materialism and Saussurean idealism. From an integrational point of view, there can be no ‘language of science’ understood as a mini fixed-code shared by the members of the scientific community. The idea that there is such a language is the intellectual product of segregating language from ‘communication’ (in the broad integrationist sense) and treating ‘linguistic sign-making’ as something sui generis. To see this in all clarity, however, one needs to theorize the sign in relation to the activities that it uniquely serves to integrate. The Harrisian argument is thus that science (as the ‘supercategory’ that we know it to be) owes its existence to (what is perceived as) the availability of a specific kind of language and a macrosocially institutionalized discourse, upon which membership and excommunication ultimately rest. As Harris shows in The Semantics of Science, science tacitly relies on a ‘surrogationalist semantics’, itself divided into two distinct branches: reocentrism and psychocentrism. It is chiefly the reocentric conception of how words get their meaning, namely by ‘standing for’ the things they are names of, that prevails in orthodox science. It is a language-philosophy that ties in with the view that the world exists mind-independently and that knowledge of the things that populate the universe requires knowledge of the correct linguistic labels, by means of which things can be identified as the same over and over again. Both types of knowledge are assumed to be context-independent (i.e. the biomechanical and circumstantial parameters of human communication are not taken into account). It has been the scientist’s job to continuously map the universe, as scientific knowledge progresses, whereby one name gets replaced by a new name (or the name remains but is redefined). Name variation, within a ‘reocentric’ model of language, must be handled with suspicion and possibly dissolved, i.e. cases in which one and the same thing is known

264

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

by different (potentially contradictory) names, as well as cases in which the same name refers to different things. In either case a disruption of the reocentric ideal (one name-one referent) has occurred. Some names must therefore be rejected – scraped off the linguistic map – on the grounds that they are psychocentrically defined, i.e. they merely ‘stand for’ ideas in the mind. From a Harrisian point of view, either surrogationalist conception of meaning is fallacious for its failure to treat words in relation to the activities they integrate, which is why the alleged ‘determinacy’ of scientific terms is an illusion brought about by decontextualization. For the integrationist, ‘hardcore reality’ is not necessarily a notion worth preserving: especially not if that commits one to the idea that humans can at any moment step outside their own human umwelt constituted by human communication (which involves but is not determined by what we commonly refer to as ‘language’). We may go as far as to say that there is something ‘there’, independently of how human individuals integrate it, but even that insight is semiologically grounded. While Harris views science as one of many terrestrial activities with a high degree of institutionalized knowledge, he is not a denier of science nor of applied science and its achievements. However, unlike biosemiotics, integrationism provides a powerful explanation of ‘languages’ (including the ‘language of science’) as second-order abstractions, i.e. products of communication presupposing linguistic reflexivity (Love 1990). The distinction between thing and object, as Deely argues, is an important one to understand ourselves as a species. What is equally important, however, is to recognize that both things and objects are integrational products of human thinking and human discourse. They are signs that serve different kinds of human activity, and personal experience tells us that things and objects are intertwined as part of the human umwelt in such ways that a clear distinction between the two requires an artificial categorization which cannot capture first-order human sign-making.

CONCLUSION: THE HUMANITY OF SIGNS Global semiotics, as the name suggests, is interested in theorizing the sign as a universal phenomenon. Human signs and non-human signs are explained through the same sign model, which is triadic and which treats sign-making as an attribute of life itself (Sebeok 1979). The Peircean sign is free of the limitations of the Saussurean dyadic sign, and is able to incorporate the symbolic sign into its triadic model and account for the nonidealist (i.e. material) dimension of Saussurean linguistics: parole. Biosemiotics, being committed to such a global view of signs, cannot afford to adopt the Harrisian sign conception, which takes personal linguistic experience as its guide to understanding what a sign is and how it means. In this respect, integrationism is arguably antithetical to a global semiotics. Within the latter, in fact, human activities and non-human activities must ultimately be treated as ‘activities’ tout court which trigger signs. Integrationism, in turn, takes the position that we cannot know other people’s minds let alone non-human minds. Observable behaviour per se, in the eyes of the integrationist, does not provide any insights into sign-making. Integrationism has thus very little to say about how non-human creatures integrate activities by means of signs of their own making, while insisting that ‘integration’ is not unique to human beings. It also adopts a sceptical position when it comes to the question whether lower life forms (e.g. micro-organisms) can be signmakers: does a bacterium make ‘its’ own signs? In this respect, integrationism is deeply humanist and anthropocentric. It is not, however, ‘glottocentric’ in Sebeok’s sense: first

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

265

of all, it is reluctant to admit any clear-cut separation between linguistic activities and non-linguistic activities. Strictly speaking, there is no linguistic sign (as theorized in segregational linguistics) but only the integrated sign. Secondly, it takes a lay-oriented approach to the phenomenon of ‘language’, i.e. there is no reocentric surrogational definition of ‘language (mass)’. Sebeok’s critique of semiology as glottocentric thus needs to be revised in light of the present contribution: the main characteristic of semiology (from a Harrisian point of view) would thus not be glottocentrism, but rather the more positive insight that ‘language’ is an integrated mode of communication, which also means that human communication cannot be decomposed in any facile way: the formula ‘human communication equals primate communication plus language’ is itself glottocentric, as it fails to take into account the differences that exist between speciesspecific ‘communicational infrastructures’ (Harris 1996). Those (glottocentric) linguistic theorists who argue that semiology can only be Saussurean (e.g. Weigand 2018) obviously have to reject Harris’s view of the sign. One way they typically dismiss Harris is to say that integrational linguistics is not ‘linguistics’ after all, or that integrationism takes too broad a conception of ‘communication’. Both Harris and Sebeok underlined the importance of taking a species-specific view on communication in their respective work, while they would have disagreed on questions concerning the ontology of the sign. For Harris (2009b), the term species is itself radically indeterminate: pace Darwin, it is not a determinate sign. At the same time, Harris does not discard the notion of a ‘species’, finding its integrational function a helpful one for the (humanist) project of self-knowledge. However, it cannot be denied that even our human experiences differ vastly across time, and between cultures and individuals (Pennycook and Makoni 2019). That is precisely why a lay-oriented approach to the human sign is indispensable.

REFERENCES Bonfante, G. (1947), ‘The Neolinguistic Position (A Reply to Hall’s Criticism of Neolinguistics)’, Language, 23 (4): 344–75. Cobley, P. (2011), ‘Mythbusting’, Language Sciences, 33 (4): 511–16. Cobley, P. (2014), ‘Codes and Coding: Sebeok’s Zoosemiotics and the Dismantling of the Fixed-code Fallacy’, Semiotica, 1998: 33–45. Conrad, C. (2020), ‘Creating Reality as a Locally Tailored Interface – An Integrational, Pragmatic Account of Semiosis’, Sign System Studies, 48 (1): 12–31. Cornips, L. (2019), ‘The Final Frontier: Non-human Animals on the Linguistic Research Agenda’, in Linguistics in the Netherlands, vol. 36, 13–19, Amsterdam: John Benjamins. Davis, D. R. (1997), ‘The Three-dimensional Sign’, Language Sciences, 19 (1): 23–31. Daylight, R. (2012), ‘The Difference between Semiotics and Semiology’, Gramma: Journal of Theory and Criticism, 20: 37–50. Deely, J. (2003), The Impact on Philosophy of Semiotics, South Bend: St. Augustine’s Press. Deely, J. (2009), Purely Objective Reality, Berlin: Mouton de Gruyter. Dessì Schmid, S. (2004), ‘Das Zeichen in der Sprachtheorie Benedetto Croces’, in G. Hassler and G. Volkmann (eds), History of Linguistics in Texts and Concepts/Geschichte der Sprachwissenschaft in Texten und Konzepten, vol. 2, 699–711, Münster: Nodus Publikationen. Duncker, D. (2018), The Reflexivity of Language and Linguistic Inquiry. Integrational Linguistics in Practice, Oxon: Routledge.

266

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Ellis, J. M. (1994), Language, Thought and Logic, Noyes: Northwestern University Press. ­Harris, R. (1981), The Language Myth, London: Duckworth. Harris, R. (1993), ‘What Is Philosophy of Linguistics?’, in R. Harré and R. Harris (eds), Linguistics and Philosophy: The Controversial Interface, 3–19, Oxford: Pergamon. Harris, R. (1996), Signs, Language and Communication, London: Routledge. Harris, R. (1998), Introduction to Integrational Linguistics, Oxford: Pergamon. Harris, R. (2003a), History, Science and the Limits of Language, Shimla: Indian Institute for Advanced Study. Harris, R. (2003b), The Necessity of Artspeak, London: Continuum. Harris, R. (2004), The Linguistics of History, Edinburgh: Edinburgh University Press. Harris, R. (2005), The Semantics of Science, London: Continuum. Harris, R. (2006a), ‘Communication: or How Jill Got Her Apple’, in Integrationist Notes and Papers 2003–2005, 11–14, Crediton: Tree Tongue. Harris, R. (2006b), ‘Integrational Linguistics and Semiology’, in K. Brown (ed.), Encyclopedia of Language and Linguistics, 714–18, Oxford: Elsevier. Harris, R. (2009a), ‘The Integrational Conception of the Sign’, in Integrationist Notes and Papers 2006–2008, 61–81, Gamlingay: Bright Pen. Harris, R. (2009b), After Epistemology, Gamlingay: Bright Pen. Harris, R. (2010), ‘Linguistic Inquiry: An Integrational Approach’, Unpublished Manuscript. (revised version of Harris 1998). Harris, R. (2013), Language and Intelligence, Gamlingay: Bright Pen. Harris, R. and C. Hutton (2007), Definition in Theory and Practice. Language, Lexicography and the Law, London: Continuum. Jones, P. E. (2011), ‘Signs of Activity: Integrating Language and Practical Action’, Language Sciences, 33 (1): 11–19. Keane, W. (2018), ‘On Semiotic Ideology’, Signs and Society, 6 (1): 64–87. Keller, R. (1995), Zeichentheorie: Zu einer Theorie semiotischen Wissens, Tübingen/Basel: Francke. Keller, R. (1998), A Theory of Linguistic Signs, trans. K. Duenwald, Oxford: Oxford University Press. Linell, P. (2018), ‘Dialogism Is an Integrationism: Reply to Peter Jones’, Language and Dialogue, 8 (2): 306–23. Love, N. (1990), ‘The Locus of Languages in a Redefined Linguistics’, in H. Davis and T. Taylor (eds), Redefining Linguistics, 53–117, London: Routledge. Love, N. (2017), ‘On Languaging and Languages’, Language Sciences, 61: 113–47. Pablé, A. (2018), ‘Radical Indeterminacy, Idealism, Realism: Benedetto Croce vs. Roy Harris’, Language & Communication, 61 (4): 46–57. Pablé, A. (2019), ‘Is a General Non-ethnocentric Theory of Human Communication Possible? An Integrationist Approach’, Lingua, 230: Article 102735. Pennycook, A. (2018), Posthumanist Applied Linguistics, London & New York: Routledge. Pennycook, A. and Makoni S. (2019), Innovations and Challenges in Applied Linguistics from the Global South, London and New York: Routledge. Petrilli, S. and A. Ponzio (2011), ‘A Tribute to Thomas A. Sebeok’, in P. Cobley, J. Deely, K. Kull and S. Petrilli (eds), Semiotics Continues to Astonish, 307–30, Berlin: Mouton de Gruyter. Saussure, F. ([1916] 1983), Course in General Linguistics, trans. R. Harris, London: Duckworth.

INTEGRATIONAL LINGUISTICS AND SEMIOTICS

267

Sebeok, T. (1975), ‘Zoosemiotics at the Intersection of Nature and Culture’, in T. Sebeok (ed.), The Tell-tale Sign. A Survey of Semiotics, 85–95, Lisse: Peter de Ridder. ­Sebeok, T. (1979), The Sign & Its Masters, Texas: The University of Texas Press. Uexküll, J. ([1940] 1982), ‘The Theory of Meaning’, Semiotica, 42 (1): 25–82. Weigand, E. (2018), ‘The Theory Myth’, Language & Dialogue, 8 (2): 289–305. Weinreich, U., W. Labov and M. Herzog (1968), ‘Empirical Foundations for a Theory of Language Change’, in W. Lehman and Y. Malkil (eds), Directions for Historical Linguistics, 95–195, Austin: University of Texas Press.

268

CHAPTER THIRTEEN

Cognitive Linguistics and Semiotics JORDAN ZLATEV AND MÖTTÖNEN TAPANI

INTRODUCTION Generative linguistics in the Chomskyan tradition has on occasion been referred to as ‘cognitive’, in the sense that it aspires to deal with ‘what the grammar purports to describe: a system represented in the mind/brain’ (Chomsky 1990: 679): the latter a phrase that is as common as it is misleading. However, Cognitive Linguistics (henceforth, CL) currently refers to a very different approach to language, with its dedicated journals, conference series and handbooks (e.g. Dancygier 2017; Geeraerts and Cuyckens 2007). This tradition emerged in the late 1970s, and a well-known book on how we ‘live by’ metaphor is often heralded as its birthdate (Lakoff and Johnson 1980). CL posed itself in opposition to generative linguistics, on the one hand, and to logical semantics on the other (e.g. Croft and Cruse 2004; Fauconnier and Turner 2002; Johnson 1987; Lakoff 1987; Langacker 1987; Talmy 2000). While Chomskyan linguists emphasize (a) syntax, (b) innateness and (c) modularity as fundamental characteristics of linguistic knowledge, CL researchers do the reverse, privileging (a´) semantics, (b´) ‘usage-based’ linguistic knowledge, which is grounded in (c´) general cognition (e.g. Saeed 2009). And while the logical tradition of semantics is based on truth and denotation (Dowty, Wall and Peters 2012), cognitive semantics boldly adopts the motto of ‘meaning as conceptualization’ (Langacker 1987; Gärdenfors 1999). On the face of it, while CL and semiotics share a pre-occupation with meaning, they also seem miles apart. The predominantly subjectivist, individual-psychological take on meaning typical for CL is clearly at odds with the structuralist tradition of semiotics stemming from Saussure ([1916] 1960). Perhaps less obviously, it is also at odds with the Peircian tradition, which has endeavoured to ‘dementalize’ semiosis (sign use) throughout much the twentieth century (see Daddesio 1995), converging in its criticism of psychologism with logic and Husserlian phenomenology (e.g. Stjernfelt 2014). Furthermore, while semiotics, of any kind, has always aimed at characterizing meaning (making) in general, and at the same time seeking to explain what makes different semiotic systems such as language, gesture, depiction and music unique, CL is, by definition, focused on language. Yet, at least three separate developments have contributed to rapprochement between CL and semiotics over recent years. First, cognitive linguists have, to an increasing degree, been engaged in analysing meaning-making beyond language, involving semiotic systems like gestures, pictures and music, often under the banner of ‘multimodality’ (see

270

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

section Multimodality and Polysemiosis), and thus finding themselves in increasingly semiotic territory. Second, in relation to this and other factors, some cognitive linguists have turned to semiotics (and phenomenology) for help with concepts like iconicity and intersubjectivity (e.g. Devylder 2018; Oakley 2009). Third, a number of semioticians have found cognitive linguistic constructs like mental spaces and image schemas consonant with structures and processes in meaning-making at large (e.g. Brandt 2004; Daddesio 1995). This mutual interest has been so strong in some quarters that, with additional impetus from cognitive science (Deacon 1997; Donald 2001; Konderak, 2018), a new discipline has emerged over the past decade: cognitive semiotics (Zlatev 2012, 2015). Within this, concepts and methods from semiotics, cognitive linguistics and cognitive science have been successfully integrated in the service of the study of social interaction (Tylén, Fusaroli, Bundgaard, and Østergaard 2013), literature (L. Brandt 2013), music (Giraldo 2019), choice making (Mouratidou 2019), street art (Stampoulidis and Bolognesi 2019) and many other topics. What we aim to show in this chapter is, above all, how through a dialectical synthesis of sorts, cognitive semiotics has been able to address problems with CL by bringing in correctives from semiotics and phenomenology, and thus avoiding two recurrent, and interrelated, fallacies: (1) the subjectivism that comes from the excessive psychologizing of notions that are essentially intersubjective and (2) confusing perceptual, and other non-signitive processes like imagination, with sign use and linguistic conventions. In the following sections we focus on four topics where CL research has displayed these fallacies, and suggest how (cognitive) semiotics can redress them. These four topics are construal, iconicity, metaphor and ‘multimodality’. First we discuss the notion of construal, which implies that ‘[. . .] meaning can be defined as non-objective [. . .] in that it always manifests some restricted and perspectivized take on the matters at hand’ (Möttönen 2016a: 13). The problem here has been that scholars like Langacker (1987) and Croft and Cruse (2004) do not distinguish between construal as (a) a private (and pre-linguistic) phenomenon, as (b) an intersubjective, pragmatic phenomenon of language/sign use, and (c) as an aspect of the conventional, socially shared semantics of the expressions in question. We review a number of ‘dimensions of construal’ from the CL literature, both distinguishing (a–c) and showing their interconnectedness. Given how frequently the notion of ‘imagery’ appears in CL, it is surprising how seldom the notion of iconicity is explicitly employed in analysis, though it is often implicitly assumed (see Möttönen 2016a). As we show in the section Iconicity, one problem is that iconicity is rarely defined, and when it is, this is commonly done in terms of a ‘resemblance-based mapping of form and meaning’ (Dingemanse, Perlman and Perniss 2020: 2). Unless language-independent criteria for such resemblance can be given, such a definition is potentially vacuous, and this problem is compounded in CL, where both ‘form’ and ‘meaning’ are typically understood as internal, mental phenomena. We illustrate the problem with Wilcox’s notion of ‘cognitive iconicity’ defined as distance between structures in conceptual space (Wilcox 2004), which both lacks such intersubjective criteria and makes iconicity indistinguishable from a complementary interpretative principle (or semiotic ground): indexicality. In contrast, we show how the seminal work of Jakobson (1965) lends itself to a cognitive-semiotic perspective and can be used to both avoid the typical CL pitfalls and rehabilitate the fundamental role of iconicity in language (Itkonen 2005; Devylder 2018).

COGNITIVE LINGUISTICS AND SEMIOTICS

271

In the section Metaphor we turn to the third and most complex type of iconic signs, or (hypo)icons in the Peircean classification: The topic is huge, but we focus on how most cognitive linguists have not taken a semiotic approach, but rather attempted to explain metaphor in terms of (unconscious) mental processes such as ‘cross-domain mappings’ and ‘mental simulations’ (Lakoff and Johnson 1999; Gibbs 2006). We argue that this once again commits the two fallacies of subjectivism and motivation/signification conflation. Based on recent work from cognitive semiotics (Stampoulidis, Bolognesi and Zlatev 2019; Devylder and Zlatev 2020) we show how these can be avoided in a model that distinguishes three different levels of meaning-making, while also highlighting the dynamic relations between them. In the section Multimodality and Polysemiosis, we briefly consider the area where CL and semiotics most obviously overlap: in the analysis of meaning-making when – as in most forms of human communication – different kinds of signs and signals like words, gestures, facial expressions, music and drawings combine. As mentioned, CL has used the ‘buzzword’ (Devylder 2019: 148) multimodality, but this term is highly ambiguous (i.e. sensorial, semiotic, communicative, etc., ‘modes’ and ‘modalities’) and confusing. Cognitive semiotics has helped to distinguish the combination of sensory modalities from polysemiosis, where more than one semiotic system (e.g. language, gesture, depiction, music) is used, irrespective of whether one or more perceptual modalities (e.g. vision, hearing and touch) are involved (Zlatev 2019), and to begin analysing the interactions between systems and modalities. In the final section, we conclude that the cognitive semiotic syntheses outlined in this chapter suggest that connections between CL and semiotics are likely to be further strengthened.

CONSTRUAL In order to emphasize the claim that language does not denote the world (directly), but that, if at all, it does this through the prism of the mind, many CL researchers adopt the notion of construal (Croft and Cruse 2004; Verhagen 2007). Most famous in this respect is the work of the founder of Cognitive Grammar (Langacker 1987, 2006, 2017), for whom the following definition of construal is typical: I will say that the speaker (or hearer), by choosing appropriate focal ‘settings’ and structuring the scene in a specific manner, establishes a construal relationship between himself and the scene so structured. The construal relationship therefore holds between the conceptualizer of a linguistic predication and the conceptualization that constitutes this predication. (Langacker 1987: 128) This excludes any possibility of describing meaning exhaustively in terms of truthconditions that render an assertion true relative to the extra-linguistic scenario it aims to depict. It is not a (representation of a) ‘state-of-affairs’, a situation or a scene per se that constitutes linguistic meaning, but a structure such as that shown in Figure 3.1, which includes a focused onstage region with the profile as the most focal element. The ‘conceptualizer’ is included within the maximal scope of conceptualization, even when not focused on, but assumed implicitly as a ‘tacit conceptualizing presence’ (Langacker 2006:  19). For example, when a speaker utters (1), apart from the ‘profiled’ clouds, presented in a dynamic spatial relationship with the horizon, there is an implicit

272

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

conceptualizer (the speaker and/or the addressee, see below) who ‘sees’ this scene from a certain kind of vantage point.1 (1)

The clouds are floating gently/boringly over the horizon.

The implicit presence of the conceptualizer in (1) shows itself in at least two aspects. Most obviously, the choice of adverb (gently/boringly) specifies the relationship to the described scene with entirely different emotional valance. More subtly, the mentioned horizon necessarily presumes the presence of a viewer, with a specific spatially positioned viewpoint. Over the years, Langacker and colleagues have proposed many different kinds of construal ‘operations’, and have endeavoured to group them under a set of ‘dimensions’ (Croft and Cruse 2004; Möttönen 2016a). No agreement on a general taxonomy can be found, but the following dimensions of construal are common. Starting from the profile and moving outwards (see Figure 13.1), the dimension of specificity concerns how much information is presented about the profile. As illustrated in (2), there is increase in specificity moving from left to right, and if the expressions are co-referential (denoting the former president of the USA), they follow an implicational hierarchy of sorts. The examples under (3) display the dimensions prominence and focusing. Such reversals between the trajector (what is focal) and the landmark (what serves as reference point) describe the same scene, but focus on the tree or the bench, respectively. The examples in (4) involve such a reversal as well, but also the perspective of the conceptualizer, since unlike beside, the predicates left/right are inherently relative to a viewpoint (Zlatev 1997; Levinson 2003). (2) the man < the old man < the old man with the fake blond hair < the old man with the fake hair and the red baseball hat (3a) The tree is beside the bench. (3b) The bench is beside the tree. (4a) The tree is to the right of the bench. (4b) The bench is to the left of the tree. A more subtle case of involvement of the perspective of the conceptualizer is shown in (5). In (5a) the described movement of the balloon takes place ‘onstage’ and is hence

FIGURE 13.1  Meaning as construal, with some of the relevant structures (adapted from Langacker 2006).

COGNITIVE LINGUISTICS AND SEMIOTICS

273

a case of what Langacker calls objective construal. In contrast, it is not the profiled path that is moving in (5b) but the ‘scanning’ of it, and given that this takes place ‘offstage’, the meaning of the sentence involves a subjective construal. (5a) The balloon is slowly rising in the sky. (5b) The path slowly rises toward the summit. As here only suggested due to space limits, but as elaborated by Verhagen (2007) and Möttönen (2016a, 2016b, 2019) the concept of construal has considerable descriptive and explanatory potential, under the condition that the two typical fallacies of CL (subjectivism and conflating perception with signification) are redressed. Typically, Langacker associates construal with individual-psychological ‘imagery’ (e.g. Langacker 1990). Even more problematic, in line with (current) neuro-reductionist trends, it is assumed that such ‘structures described in qualitative terms ultimately consist in neural processing’ (Langacker 2016: 473), thus placing their essence ultimately beyond consciousness. In an extensive meta-analysis and elaboration, Möttönen (2016a) shows that such claims are fundamentally at odds with phenomenology, and consequently with cognitive semiotics, given that the latter is strongly influenced by this philosophical tradition (Sonesson 2007; Zlatev 2015). At the same time, Möttönen (2016a) argues that a subjectivist (and even more so physicalist) interpretation of construal is not required for Cognitive Grammar. Its emphasis on being a ‘usage-based’ theory in fact implies considering construal as an intersubjective phenomenon. To accommodate this, however, it is imperative to distinguish between (a) acts of perceptual intentionality and other kinds experiential motivations, that may underlie but not do not determine the use and interpretation of linguistic utterances such as those in (3) and (4), and (b) such utterances themselves (Blomberg and Zlatev 2014; Zlatev and Blomberg 2016). The most obvious reason why trajector/landmark reversals and other ‘operations’ on an individual level cannot be determinative of meaning is the fact that language always takes place in one social context or another. For example, one’s choice of which referential expression to use in (2) will not so much depend on how specifically one needs to characterize Donald Trump for one’s own sake, but for the sake of communication (Möttönen 2016b). Verhagen (2007) correctly emphasizes that the ‘conceptualizer’ should not be equated with a single mind, but minimally include both the speaker and the hearer. Hence, construal in language cannot be a private, purely mental process, but needs to be seen as a pragmatic, communicative process, supervening on the experiences of individual speakers and hearers. This is the transition of construal as non-signitive intentional acts to construal as signitive, ‘meaning-intending’ acts in the sense of Husserl ([1900] 1970). The notion of construal can be extended even further with the help of the phenomenological concept of sedimentation, which denotes the process where, over historical time, through numerous individual acts of meaning-making, relatively stable intersubjectively shared structures emerge. As argued by Zlatev (2016), linguistic meanings (senses) can thus be understood as ‘conventionalized, socially shared construals of their intended referential objects’ (2016: 563). To take a famous example from the history of semantics, the expressions Morning Star and Evening Star are not synonymous even though they are co-extensional, as the contextual information pertaining to when

274

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

the referent is observed has become sedimented into the conventional semantics of the respective expression. As described by (Möttönen 2016a: 161–2): In this process, the multiple, intersubjectively-shared experiential designata, and the particular ways in which they are experienced, come to define the content of the expression. The designata are not only given as such, but also in their perspectival appearance to subjects. We return to the notion of sedimentation in the section Metaphor, but may for now summarize that the CL concept of construal is indeed productive for (cognitive) semiotics, under the condition that three kinds are both distinguished and interrelated: (a) individual people (may) construe1 situations variously in conscious experience; (b) speakers and hearers construe2 situations and referents in social interaction, motivated but not determined by (a); (c) words and other linguistic expressions construe3 certain situations and referents, to the degree that they are sedimentations of (b). These three senses of the verb construe exist in everyday English, and this could be a reason, but hardly an excuse, for commonly confusing them within CL.

ICONICITY The concept of iconicity is possibly the most direct ‘borrowing’ in (cognitive) linguistics from semiotics, and yet this is often not even acknowledged, along with terminological slippage where iconicity becomes conflated with any form of ‘non-arbitrariness’, as noted among others by Devylder (2018). CL overview books (e.g. Croft and Cruse 2004; Dancygier 2017) tend to mention iconicity only in passing, though there is an informative chapter on the topic in the first comprehensive CL handbook (van Langendonck 2007). On a general level, theorists of Cognitive Grammar (e.g. Langacker 2008: 209) occasionally refer to iconicity as ‘correspondences’ between grammatical patterns and their conceptual meanings. But as we argue below, this take on iconicity easily leads to the two familiar problems of subjectivism and conflation of experiential motivations with linguistic motivations. It was Jakobson (1965) who introduced the Peircian concept of icon(icity) to linguistics, albeit in a simplified way, in his ‘prelude to a theory of iconicity’ (De Cuypere 2008: 55), where he questioned ‘the Saussurean dogma’ of the arbitrariness of the linguistic sign.2 In accordance with Peircian semiotics, Jakobson described iconicity as a resemblancebased interpretative principle, which can only be understood in contradistinction with two other complementary principles (or ‘semiotic grounds’): indexicality, based on spatio-temporal contiguity, and symbolicity, based on shared agreement, or convention (see Ahlner and Zlatev 2010). Further, Jakobson made it clear that these three semiotic grounds are not mutually exclusive but can coexist (in different proportions) in the same sign. For example, the English demonstratives this/that are (a) indexical as the requite contiguity with a referent in the context, (b) symbolic as they are conventions in English (as opposed to, e.g., the corresponding conventions tozi/onzi in Bulgarian), and (c) arguably also iconic, as the vowel contrast [i/a] corresponds to the opposition [short/long] distance from the speaker, and this correspondence has been shown to be non-random in a large sample of languages (Johansson and Zlatev 2013). But is such a correspondence between (elements of) the expression (‘form’) and (aspects of) the meaning of the expression indeed a matter of resemblance, and thus

COGNITIVE LINGUISTICS AND SEMIOTICS

275

iconicity? A positive answer can be given, but only under two conditions. First, it is not a matter of imagistic iconicity, where the expression is like a photograph or a direct imitation of the referent, but of diagrammatic iconicity, where the resemblance takes the form of an analogy: the relation A-B (in expression) corresponds to the relation A´-B´ in experience. In the example of English demonstratives, the contrast between a close-front vowel (with its higher acoustic frequency) and open-back vowel (with its lower acoustic frequency) corresponds to the contrast between short distance and long distance. Second, it is imperative that such ‘isomorphism’ does not only hold between ‘form’ and ‘content’ within language, but to extra-linguistic experience, if the similarity in question is to be intersubjectively verifiable. This is precisely the take on linguistic iconicity as ‘structural similarity between linguistic and non-linguistic entities’ adopted by Itkonen (2005: 101). Iconicity concerns all levels of language, from the sounds of individual words to grammar and discourse. On the lowest, there is phonic iconicity, commonly known as (part of) the phenomenon of ‘sound-symbolism’, for example with English expressions for actions and events such as (6). (6)

bang, bong, boom, buzz, crack, flip, flop, splash, squeak, zip . . .

Such onomatopoetic words are often mentioned as superficial counter-examples to arbitrariness, only to be dismissed due to their supposed marginal numbers and role in language; even van Langendonck (2007) writes that ‘compared to normal vocabulary, onomatopoeia remains a marginal phenomenon in natural language’ (2007: 402). The last decade has made it clear that such judgements are mistaken, in part due to the fact that they are based on a ‘skewed database’ (Ahlner and Zlatev 2010: 305) of languages. As pointed out by Perniss, Thompson and Vigliocco 2010: 3): When we move outside the Indo-European language family, however, we find that iconic mappings are prevalent and are used to express sensory experiences of all kinds. Languages for which a large iconic, or sound-symbolic, lexicon has been reported include virtually all sub-Saharan African languages . . .,some of the Australian Aboriginal languages . . ., Japanese, Korean, Southeast Asian languages . . ., indigenous languages of South America . . ., and Balto-Finnic languages. Another reason for traditionally downplaying iconicity on this level is that onomatopoeia is based on both (a) unimodal (sound-to-sound) resemblance and (b) relatively imagistic (imitative) iconicity, while the massive amounts of iconicity on the ‘phonic level’ that are recognized in recent studies are both mostly (a´) cross-modal (e.g. sound-to-shape) and (b´) diagrammatic as defined above (Dingemanse 2012; Blasi et  al. 2016; Jääskeläinen 2016). Even though such iconicity is subtle, and far from obvious, it has been shown to offer advantages in terms of language acquisition (Imai et  al. 2012) and social transmission (Erben Johansson, Carr, and Kirby 2020), implying that it is indeed perceived by speakers, even if tacitly. Jakobson (1965: 33) recognized the ‘latent and virtual character’ of iconicity on this level, but claimed that ‘in syntax and morphology (both inflection and derivation) the intrinsic, diagrammatic correspondence between the signans [i.e., expression] and signatum [i.e., content] is patent and obligatory’. This statement appears to echo the way iconicity is understood in Cognitive Grammar, predating it with some thirty years. However, as we illustrate below, Jakobson’s ‘signatum’ does not correspond to Langacker’s notion of meaning as mental imagery, or of construal as something purely subjective (section Construal) and thus does not encounter the same problems.

276

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

A well-known example of iconicity on the level of morphology concerns the linguistic category number as shown in (7); if singular and plural have different manifestations, it is the plural form that (nearly always) has an extra morpheme, analogously to the ‘more’ in experience. This type of linguistic iconicity is commonly known as ‘iconicity of quantity’. (7a) boy vs. boy-s (English) (7b) pojke vs. pojk-ar (Swedish) (7c) momce vs. momce-ta (Bulgarian) (7d) poika vs. poja-t (Finnish) Another kind of iconicity on this level is the ‘iconicity of distance’, where ‘the distance between expressions corresponds to the conceptual distance between the ideas they represent’ (Haiman 1983: 782). In an extensive discussion of how such diagrammatic iconicity can explain asymmetries in possessive constructions in Paamese (and other languages), Devylder (2018) presents oppositions such as in (8), analysed as in Figure 3.2. In a debate with Haspelmath (2008), Devylder points out that usage frequency (‘economy’) cannot explain the distribution of possessive constructions, and that it is experienced distance – in (8) pertaining to one’s body but in other cases pertaining to social and emotional relations – that maps onto ‘linguistic distance’, with the indirect possessive construction in (8b) also being used, for example, for mother-in-laws, while the direct one in (8a) is used for mothers, etc. (8a) vatu-k (Paamese)

head-1SG

(8b) maleles ona-k (Paamese)

lungs POSS-1SG

Moving on to the sentence level, iconicity of distance applies to many linguistic constructions such as (9), where ‘absolute objective qualities like wooden and white are closest to the noun. Relative objective qualities (size, e.g., little) are expressed further away from the noun, and subjective qualities like nice are repressed still further. Furthest away . . . in the determiner, since is serves only to pick out the referent’ (van Langendonck 2007: 409). This is a pattern that is potentially universal, with exceptions that can be accounted of. (9)

those three nice little white wooden dolls

Again, to appreciate such iconicity, distance on the semantic side should be understood as experiential rather than ‘physical’ or so. This is even clearer in (10), where the contrast is between direct perception vs indirect perception (or evidence). The examples in (10a) and (10b) could denote the same situation, but with different construal, the first using a closer, and the latter a more distanced viewpoint. But (10b) can also have the interpretation that the speaker did not at all hear the song him/herself, but was given an indirect report of the fact. On the expression side, this difference is mirrored by (10a) being a single clause construction, and (10b) consisting of two clauses, with a subordinate finite verb. (10a) I heard him sing/singing. (10b) I heard that he sang.

COGNITIVE LINGUISTICS AND SEMIOTICS

277

FIGURE 13.2  Diagrammatic iconicity between embodied and linguistic distance in Paamese possessive (from Devylder 2018, with permission from the author).

Moving on to the discourse level, and ‘iconicity of sequence’, the famous (11) reportedly uttered by Caesar, implies both temporal and causal order of the denoted events being mirrored by the order of the verb/clauses). In (12), taken from a well-known Bulgarian poem by Hristo Botev, it is even clearer that the order in events is not causal but experiential. (11) Veni, vidi, vici (Latin) ­

‘I came, I saw, I conquered’.

(12) Gora zašumi, vyatar povee . . . (Bulgarian)

‘Forest starts to sound, wind to blow’

We can generalize that in all these cases of linguistic iconicity, the resemblance (mapping, correspondence, analogy, etc.) in question is between expression and a more

278

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

or less schematic experiential structure, and in most cases, this mapping is relational, so that ‘more/less of’, ‘close to/away from’, ‘before/after’, etc., make sense only when two or more linguistic and experiential elements form an analogical diagram as in Figure 13.2. This serves to show that iconicity in language is ‘cognitive’ in the sense of requiring conscious speakers/signers who can appreciate the subtle and relational resemblances between linguistic and experiential structures. But it does not imply ‘cognitive’ in the sense of subjective (‘in the head’) meaning. Unfortunately, the latter is how iconicity is typically understood in CL, as illustrated by Wilcox (2004). The author first assumes that ‘the iconic relation is between construals of real-world scenes and construals of forms’ (2004: 123, our emphasis), with a subjectivist understanding of construal (see section Construal). Even more problematically Wilcox then defines cognitive iconicity as ‘a distance relation between the phonological and semantic poles of symbolic structures’ (2004: 122): the ‘closer’ this distance, the more ‘cognitively iconic’ the construction should be. Thus, Wilcox ends up defining iconicity in terms of invisible (internal, mental, conceptual) contiguity, i.e. indexicality, and even more, does not provide criteria of how the latter could be independently established, making the proposal quite circular. What leads Wilcox to this problematic idea is the valid need to account for substantial degrees of iconicity in signed languages like American Sign Language (ASL), despite the fact that single signs are very often less iconic than corresponding gestures. He illustrates this with the ASL sign SLOW, which is produced with a slow movement of one hand along the other top of the other hand. But VERY SLOW is produced with ‘an initial hold of the sign’s movement followed by sudden, rapid release’ (Wilcox 2004: 123), with the result that VERY SLOW is produced faster than SLOW. This has been used by some analysts to argue that signed languages are in fact arbitrary. Wilcox correctly questions this claim, since the sign VERY also modifies all signs denoting qualities (e.g. FAST, SMART) in essentially the same way, which makes the expressions of all such phrases diagrammatically iconic: qualities without VERY will be expressed ‘normally’, and those with VERY will have this extra very ‘sudden, rapid release’. But this is clearly an intersubjective, sedimented sense in ASL, and there is hardly any need to evoke any special kind of ‘cognitive iconicity’ (based on internal distance) to acknowledge its iconic character. The attitude displayed in the ‘cognitive iconicity’ suggested by Wilcox is symptomatic for CL at large. As pointed out earlier, Langacker and others regard iconicity as a generalized form of cognitive ‘isomorphism’ (van Langendonck 2007: 400), understood as a relationship between two mental entities: internal phonological (or signed) form and internal meaning. But if the claim of resemblance between these ‘poles’ is to be non-vacuous, there must be non-linguistic criteria for establishing the structure of the (experiential) meaning, which is why Itkonen (2005), among others, insists that what iconic linguistic expressions correspond to are non-linguistic entities. In other words, CL theorists once again commit the two fallacies of (a) meaning-subjectivism and (b) confusing experiential motivations with linguistic semantics. In contrast, by making the latter distinction clear, and by pointing out that iconicity holds not between ‘form’ and ‘meaning’, but between linguistic expressions and experiences of the kind that motivate (but do not constitute) the intersubjective conventional construals of the expressions in question, cognitive semiotic analyses are able to avoid both pitfalls (Blomberg and Zlatev 2014; Zlatev and Blomberg 2016; Devylder 2018). Further, such a perspective can easily be extended to metaphor, where it is even clearer that resemblance in question

COGNITIVE LINGUISTICS AND SEMIOTICS

279

is not between ‘form’ and ‘meaning’, but between different interpretations of the same (linguistic) expression, as we explain in the following section.

METAPHOR CL is famous for its research on metaphor, and in particular for the theory proposed in the well-known book Metaphors We Live by (Lakoff and Johnson 1980), which became eventually known as Conceptual Metaphor Theory (CMT, Johnson 1987, 2010; Grady 1997, 2007; Lakoff and Johnson 1999; Kövecses 2000; Gibbs 2017). This theory redefined metaphor as a scheme of mental correspondences (a ‘mapping’) from more concrete (e.g. SPACE, MOTION) to more abstract (e.g. TIME, EMOTION) conceptual domains. As stated clearly by Grady (2007) ‘within Cognitive Linguistics the term metaphor is understood to a pattern of conceptual association, rather than to an individual metaphorical usage, or a linguistic convention’ (2007: 188). Metaphorical expressions in language (or any other semiotic system) like (14) and (15) thus become surface realizations of underlying ‘conceptual associations’. (14) Let us forget the past, and look ahead! (15) He fell into a deep depression. Lakoff and Johnson (2003: 256) re-affirm the emphasis on ‘a stable, conventional system of primary metaphors that tend to remain in place indefinitely within the conceptual system and that are independent of language’. But one may reasonably wonder how a ‘conventional system’ could exist in the mind/brain of individual speakers, given that conventions are essentially social and normative phenomena (Itkonen 2008), and how it could remain there ‘indefinitely’, given that neural connectivity is under constant change (Edelman 1992). Furthermore, the definition of ‘a cross-domain mapping of structure from a source domain to a target domain, where the two domains are regarded as different in kind’ (Johnson 2010: 407) hinges on being able to individuate ‘conceptual domains’, which is anything but a trivial matter (Croft and Cruse 2004). For example, are the above-mentioned MOTION and EMOTION truly distinct domains (Zlatev and Blomberg 2016)? The spectre of subjectivism reappears once again. As can be expected, many from both within and outside CL have reacted to such problems. Discourse metaphor theory (Cameron and Deignan 2006; Zinken 2007), has brought back the attention to specific metaphorical expressions like drive X crazy, or bleed (X) to death and how they become conventionalized in a language community as a result of repeated use in social interactions. Even more radically, some theorists have focused on the role of situated context and proposed that ‘metaphors come to exist only in the moment’ (Müller 2016: 50). Further, given the difficulties of individuating ‘conceptual’ metaphor, and the need for intersubjectively valid criteria to decide what is metaphor has prompted many to propose ‘procedures’ for identifying metaphors in text (Pragglejaz 2007). The debates surrounding CMT continue to rage and have even been characterized as ‘metaphor wars’ (Gibbs 2017). We cannot hope to do them proper justice here, but only show how cognitive semiotics could possibly resolve some of the debated issues. From the perspective of semiotics, what is lacking in the CMT approach is first of all an acknowledgement that metaphors are a particular kind of sign (process), and secondly, that they are iconic signs. As seen above, proponents of CMT characterize metaphors and mental ‘associations’, but as pointed out in the section Iconicity, associations are

280

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

at best indexical, not iconic processes. On the other hand, and as stated clearly by van Langendonck (2007: 398): ‘A metaphor, in Peirce’s view, brings out the representative character of a sign by representing a parallelism with something else.’3 Unlike images and diagrams, in metaphors the resemblance in question is not between a linguistic (or other) expression and some extra-linguistic object, related to its meaning (though not the meaning itself, as argued in the section Iconicity) but between two interpretations of the expression in question. A recent cognitive semiotic theory that capitalizes upon this insight is the Motivation & Sedimentation Model (MSM; Stampoulidis, Bolognesi and Zlatev 2019; Devylder and Zlatev 2020), which offers the following definition of metaphor: A metaphor is a (simple or complex) sign in a given sign system (or a combination of systems) with (a) at least two different potential interpretations (giving rise to tension), (b) standing in an iconic relationship with each other, where (c) one interpretation is more relevant in the communicative context, and (d) can be understood in part by comparison with the less relevant interpretation. (Stampoulidis 2021: 54) To illustrate, we may consider a pictorial metaphor from the corpus of Greek street art of Stampoulidis et al. (2019), shown in Figure 13.3. This may be interpreted as GREECE IS SULLIED, provided the conventional metonymy (association) between national flag and the country it represents. This pictorial expression/sign is (a) ambiguous between at least two conflicting interpretations: the Greek flag and a roll of toilet paper, (b) standing in a relatively imagistic iconic relation to one another. The question of relevance depends on the context in which the metaphor is created and interpreted. Given that the street artwork is a commentary on the political situation of Greece, (c) it is reasonable that Greece is the topic (target) of the metaphor, which (d) is interpreted via the notion of toilet paper, and its pejorative associations. Concerning linguistic metaphors like that in (15) we can first establish that the expression (type) fall into X has at least two different interpretation by means of a ‘polysemy test’ showing that a sentence like ‘A fell into X but did not (really) fall into X’ is not self-contradictory (Torstensson 2019; Geeraerts 2010). It is the figurative kind of falling that is relevant in the context, and it is in part understood through comparison with actual falling, given that there is an experiential similarity in their event structure: initial stability, loss of stability, unpredictable but likely negative consequences, etc. A number of studies have shown that this theoretical definition of metaphor leads to intersubjectively verifiable operational definitions for different semiotic systems (Stampoulidis et al. 2019; Torstensson 2019; Devylder and Zlatev 2020; Moskaluk 2020). MSM is based on phenomenology (e.g. Merleau-Ponty 1962), and integral linguistics (Coseriu 1985, 2000), which itself has roots in both phenomenology and (structural) semiotics. In brief, MSM (see Figure 13.4) grounds meaning in the Embodied level, which consists of non-linguistic cognitive and experiential processes like cross-modal perceptual experience (Abram 1996), bodily mimesis (Donald 2001) and analogy making (Itkonen 2005). On the opposite side is the Situated level, which is that of actual social interaction, spontaneous language use and artistic improvisation. This is where all particular linguistic and extra-linguistic meaning-making take place, and where diversity and creativity abound. Linking the two is the Sedimented level of historically derived, relatively stable

COGNITIVE LINGUISTICS AND SEMIOTICS

281

FIGURE 13.3  Greece Flag ≡ Toilet paper, leading to the metaphorical interpretation GREECE IS SULLIED, given appropriate knowledge on the Sedimented and Situated levels (Stampoulidis et al. 2019). Creator: Unknown. Photography: Georgios Stampoulidis © July 2015. (Reproduced with permission from the author and copyright owner.)

linguistic and other social norms (Itkonen 2008; Zlatev and Blomberg 2019). It includes shared beliefs and communicative and other communal conventions. The most basic relation that links the levels is that of motivation, which grounds, but does not determine, the meaning of the expression that is motivated. The second relation, sedimentation (see section Construal), goes in the opposite direction, stabilizing linguistic and other semiotic expressions, as well as other cultural phenomena, as a repository of common knowledge for the community. These can now co-motivate expressions on the Situated level, which are thus doubly motivated – not only by the visceral experiences and non-linguistic thought processes on the Embodied level, but also by the norms of the Sedimented level. The degree of metaphoricity of an expression on the Situated level is, among other factors, proportional to the degree to which its use is motivated by the Embodied level, and inversely proportional to the degree to which it is

282

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

motivated by the Sedimented level (Moskaluk 2020). Examples (14) and (15) would thus qualify as relatively low on this scale, while the pictorial metaphor in Figure 3.4, as well as (authentic) examples like (16), taken from a go-along interview with a Greek street artist (Stampoulidis 2021), clearly display active analogy making, and hence the predominant role of the Embodied level. (16) You may think of people going to the gym, working out really hard to show off their abs in the summer. Similarly, here, the people of art do the same. We show our artistic abs. So, in essence, street art making is the same as going to the gym. And your artworks are your abs. (Stampoulidis 2021, Paper 3: 31) The cognitive semiotic MSM accommodates some of the insights from CL, including the role of ‘mappings’, especially once it is acknowledged that this is identical with the classical notion of analogy (Itkonen 2005), which as pointed out in the section Iconicity also overlaps with that of diagrammatic iconicity. Further, the emphasis on grounding language and other sign systems in bodily experience is shared by the two approaches. But while MSM regards the mappings/diagrams and other cognitive processes as motivations for sign use, CMT follows the usual pattern of CL of conflating the two. By regarding metaphorical expressions and conventions as more or less epiphenomenal reflections of the crucial cognitive infrastructure, CMT also cuts off the link to social interaction and becomes subjectivist. In line with phenomenology, MSM, on the other hand, emphasizes that even the Embodied level is not ‘in the head’, but rather structured by embodied intersubjectivity and thus not solipsist (Zlatev and Blomberg 2016; Möttönen 2016a, 2019). Further, MSM integrates elements of some of the metaphor theories that have been proposed as alternatives to CMT mentioned earlier. The Sedimented level accommodates shared socio-cultural knowledge and conventions as emphasized by proponents of discourse metaphors, and the Situated

FIGURE 13.4  The Motivation & Sedimentation Model (MSM) with the upward arrows signifying motivation, and downward arrow sedimentation (adapted from Stampoulidis et al. 2019; Devylder and Zlatev 2020).

COGNITIVE LINGUISTICS AND SEMIOTICS

283

level is indeed the level where metaphors ‘dynamically emerge, are elaborated, and are selectively activated over the course of a conversation’ (Kolter et al. 2012: 221), as argued by proponents of dynamic-enactive accounts (Müller and Kappelhoff 2018). However, these could neither emerge nor be elaborated without the other two levels, the Situated and the Embodied.4 In sum, metaphor continues to be a very hot topic, to use a highly sedimented but still ‘alive’ metaphor, and CL should be given credit for making it so. Its dominant theory CMT, however, has been found increasingly problematic theoretically, empirically and methodologically. And the main reason for this is once again the falacy of subjectivism: placing metaphors in the (individual) head, rather than in communication, culture and embodied intersubjectivity. We reviewed MSM as a contribution from cognitive semiotics that clearly distinguishes between cognitive motivations and metaphor use, and links them through a level of shared conventions, thus offering the blueprint for a peace treaty in the ‘metaphor wars’.

MULTIMODALITY AND POLYSEMIOSIS Somewhat later than other fields such as social semiotics (Kress 2009) and gesture and sign language studies (Vigliocco, Perniss and Vinson 2014), CL has recently embraced the notion of multimodality, with a recent handbook (Dancygier 2017) devoting a whole part to ‘Language, Body and Multimodal Communication’. Unfortunately, as pointed out by Devylder (2019: 149) in the review of this book, the term ‘is not defined and approached cohesively within CL’. Devylder illustrates this claim clearly, with references to particular chapters in the handbook: Defining something as multimodal naturally implies that it involves at least two modalities, or modes. That is about as far as the consensus goes in CL, and that is not very far because we already do not know if we are talking about modes or modalities in ‘multimodality’, and what both terms mean. For Vandelanotte (p. 158), a modality is a sensory channel (e.g., ‘the visual modality’), for Sullivan (p. 389–91) in line with the CL tradition that uses the term ‘visual metaphors’ in contrast to ‘linguistic metaphors’, modality has to mean that pictures and written text are two modalities (the combination of which is thus ‘multimodal’), for Feyaerts et al. (p. 135), the term extends to just about any aspect of a face-to face interaction. [. . . Thus,] when one reads a CL paper on multimodality, one can expect the term to either mean: the combination of text and image, the combination of gesture and speech, the combination of vision and hearing, or a combination of all of the above and more. (Devylder 2019: 149–50) The problem is in part, as suggested above, that the term ‘multimodality’ has sprung forth in different fields, from cognitive psychology to conversation analysis, and many of these differ in terms of ontological and methodological commitments. While ‘modality’ commonly refers to a ‘channel’ of perception (in psychology), ‘mode’ is a very general term, used for anything that could be meaningful in social interaction. This, however, is problematic, as ‘there is no theoretical limit to the number of modes that may be recognized in various socio-cultural contexts, and this leads to an abundance of modes that are difficult to compare’ (Green 2014: 9–10).

284

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

At least some CL theorists recognize the terminological challenges with the term. For example, Vandelanotte (2017: 161) distinguishes between ‘multimodality in the strict sense’ for different sensory (and productive) channels and ‘in the loose sense’ for different ‘semiotic resources’ like text and image. Similarly, a prominent analyst of ‘multimodal metaphor’ acknowledges that ‘what constitutes a mode is a much-debated and hitherto unresolved issue’ (Forceville 2017: 27). The problem, however, is more than terminological and reveals a deeper concern. For example, Forceville continues the citation from above by stating: ‘I distinguish the following modes: written language, spoken language, visuals, music, non-verbal sound, gestures, olfaction, taste, and touch.’ This is indeed far from ‘cohesive’, given that it includes (a) different semiotic systems like language, gesture and music, (b) different sub-systems of language: spoken and written, and (c) different sensory modalities. It is hard not to see the two CL fallacies of subjectivism and conflation of semiotic and perceptual systems implicated here as well. First, purely perceptual (and thus nonsignitive) experiences are conflated with sign and signal use. Second, this is facilitated by the general ‘cognitive’ bias to treat everything as essentially mental, or in other words, subjectivism. Recent work in cognitive semiotics (Louhema et  al. 2019; Zlatev 2019) has once again attempted to redress the problem by distinguishing types of signification from types of experience, as a preliminary to investigating their interrelations. Such authors re-introduce a familiar term from semiotics: semiotic system, but make it clear that this should not be understood as a generalization based on language, which was arguably the case in structuralism (Saussure [1916] 1960). First, semiotic systems can consist of either signs, where expressions clearly denote (intentional) objects, or signals, which are only associated with such objects (Zlatev, Zywiczynski and Wacewicz 2020). Human language is a paradigmatic example of a sign system, while various animal alarm calls, as well as spontaneous human facial expressions, are signal systems. Sign systems may be defined as combinations of particular kinds of signs, along with their properties and relations. Table  13.1 shows what are arguably the most fundamental sign systems in human communication, with some of their central properties. The semiotic system of language differs, depending on whether it is realized as speech, writing or signing. In the case of writing, it is characterized by a high degree of permanence of its expressions, at least in prototypical written media. In the case of the signed languages of the deaf, with respect to production and perception, language adopts the features of Gestures. Language has so-called ‘double articulation’: phonemes or graphemes combine systematically to form meaningful morphemes, and this is arguably also the case for some but not all signed languages, in particular those which have emerged more recently (Sandler 2012). Its semiotic ground is predominantly conventional, even if iconicity and indexicality are also present (see section Iconicity). Its ‘syntagmatic’ (sequential) relations are characterized by a high degree of compositionality, where the meaning of a composite sign is built up (at least in part), from the meanings of its constituent signs, and the rules for combining these. In the case of Gesture and Depiction, on the other hand, the predominant grounds are iconicity and indexicality, even if conventionality is also important. The signs of Gesture and Depiction can be analysed into phrases and units (Kendon 2004; Green 2014), but these are not made up of minimal distinctive elements, and hence lack double articulation. Further, these sign systems have much less systematic manners in arranging sequences of signs, making it more difficult (but not impossible) to express complex messages such as

COGNITIVE LINGUISTICS AND SEMIOTICS

285

TABLE 13.1  The sign systems of Language, Gesture and Depiction, with some of their properties (adapted and extended from Zlatev et al. 2020). Capital letters are used here and in the text for the systems to highlight that the terms are used in a specific (technical) sense ­ Properties

Sign systems Language

Gesture

Depiction

Speech

Writing

Signing

Production

Vocal

Material

Bodily

Bodily

Material

Perception (Modality)

Auditory (+Visual)

Visual

Visual (+Tactile)

Visual (+Auditory, + Tactile)

Visual (+Tactile)

Permanence Low

High

Intermediate

Intermediate

High

Double articulation

Yes

Yes

Yes/No

No

No

Semiotic grounds

Conventional > Iconic, Indexical

Conventional > Iconic, Indexical

Conventional > Iconic, Indexical

Iconic, Iconic > Indexical > Indexical, Conventional Conventional

Syntagmatic Compositional Compositional Compositional Linear relations

Possibly linear

narratives. In terms of permanence, Gesture (and Signing) is intermediate to Speech and Writing, given that one can ‘hold’ gestural expressions in mid-air, when the need for emphasis or audience attention requires it. The permanence of Depiction is similar to that of Writing, with differences having to do with the ‘Material’ in which signitive acts are produced (e.g. stone vs paper vs sand). Communication through a single semiotic system (either a sign or signal system) is by definition monosemiotic. But actual human communication, especially in faceto-face contexts, is as a rule polysemiotic, combining multiple sign systems (as those in Table 1) and signal systems, like facial expressions and postures. Much recent and ongoing work in cognitive semiotics is formulated under the banner of polysemiotic communication or polysemiosis for short. This includes work on intersemiotic translation, i.e. transferring a message from one semiotic system to another, with or without change of modality (Louhema et  al. 2019), analysing the practice of ‘audio description’ of films for blind viewers as intersemiotic translation (Diget 2019), polysemiotic and monosemiotic street art (Stampoulidis et  al. 2019), polysemiotic ‘path metaphors’ (Devylder 2020) and the analysis of the role of pantomime for the evolution of language (Zlatev et al. 2020). We lack the space to review this work here, but can point out its most relevant feature for the present context: clearly distinguishing perceptual modalities and their combinations (i.e. multimodality) and semiotic systems and their combinations (i.e. polysemiosis). The classification in Table 13.1 allows us to clearly acknowledge that there is no one-to-one

286

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

relation between semiotic system and perceptual modality. Even considering only Speech, it is typically perceived not only as sound, but visually as mouth movements, as shown by the famous McGurk effect (Munhall et al. 1996). Various manifestations of Depiction, such as sculptures and reliefs, afford perception through multiple modalities. From the other side, vision is typically, or can be, involved in the perception of all sign systems. Thus, we can see once more that cognitive semiotics helps de-conflate (perceptual) experience from signification, in a first step, so as to investigate their interactions more systematically in a second step.

CONCLUSIONS In this chapter, devoted to the relation between the forty-year-old tradition of Cognitive Linguistics (CL) and the much older traditions of semiotics, we chose to focus on four theoretical concepts, which show both similarities and differences between the two traditions: construal, iconicity, metaphor and multimodality. From the onset, we noted a tension between the dominant subjectivist conception of meaning in CL, and the quest for intersubjective structures in communicative and other kinds of acts of meaning making in semiotics. The relatively recent discipline of cognitive semiotics (Zlatev 2015; Stampoulidis 2021) combining concepts and methods from these traditions, as well as from cognitive science (Konderak 2018) and phenomenology (Sonesson 2009), was shown to be able to offer solutions to dilemmas for each of the four concepts. In the case of construal, we pointed out the need to distinguish mental processes from pragmatic interactions and conventional meanings. Concerning iconicity, we argued that it should not be understood as a mapping between ‘mental form’ and ‘mental meaning’, but as resemblance between intersubjectively observable expressions, and independently perceivable (or inferable) non-linguistic events and experiences. Thus, iconicity can indeed be shown to be prevalent on all levels of language, though not in the same way nor to the same degree as in Gesture and Depiction, which are distinct semiotic systems. Similarly, metaphor is from the cognitive semiotic perspective not a ‘mapping’ or ‘simulation’ in the subjective mind, but a complex iconic sign (process), where the resemblance in question is between different interpretations of the relevant signs, alongside more or less creative tensions between these, and the surrounding context. Finally, the fact that CL currently analyses non-linguistic phenomena like gestures, ‘memes’ and images in advertisements, cartoon, film, etc., is to be applauded, but we argued that it is problematic to do so in terms of ‘multimodality’. This is so both due to the ambiguity of the term and due to the latent subjectivism of CL, where ‘modes’ of expression become conflated with ‘modes’ or ‘modalities’ of thought and perception. In sum, we have offered a ‘diagnosis’ to CL in terms of the two fallacies of subjectivism and expression/motivation conflation, along with prescribed medications from the pharmacy of cognitive semiotics. For the sake of clarity, we have consistently contrasted CL and cognitive semiotics. But given that the latter constitutes a synthesis of semiotics, linguistics and cognitive science (as discussed in the introduction above), it is clear that no sharp distinctions between CL and cognitive semiotics can be drawn. Both of the present authors present their research also in CL conferences and publications, and we hope that this chapter will spark further dialogue and collaboration between CL and cognitive

COGNITIVE LINGUISTICS AND SEMIOTICS

287

semiotics, despite their – at first glance – distant perspectives. We may conclude with the final lines of a publication where the first author was asked to ‘predict’ the development of CL in the future: Having reviewed some of these differences, I have argued that methods and concepts from phenomenology can help re-unite various strands in an epistemologically pluralistic, truly experiential linguistics. My prediction would be that in the coming decades, this would merge with broader fields that likewise adopt a phenomenological perspective in the understanding of mind and meaning, such as enactive cognitive science (Thompson 2007) and cognitive semiotics. (Zlatev 2016: 570)

ACKNOWLEDGEMENTS We wish to thank Johan Blomberg, Simon Devylder and Georgios Stampoulidis, as well as the editor Jamin Pelkey, for many helpful comments on previous versions of this chapter.

NOTES 1 Reflected in Figure 13.1 are also the roles of trajector (tr) and landmark (lm), which the clouds and the horizon respectively assume in (1). These notions are meant to capture the semantic asymmetry internal to the profile, with the (grammatical) subject by default coinciding with the trajector. 2 Jakobson can be said to have ‘abridged’ Peirce for a linguistic audience, and may have involuntarily contributed to underestimating the concept of iconicity in linguistics. Nöth (1999, 2008) can be consulted for a more elaborate application of Peircean ideas to language. Nevertheless, we wish to acknowledge Jakobson (1965) both for its temporal primacy and for the inspirational character of this short text, which fifty-five years after publication still surpasses the way iconicity is dealt with in much current (cognitive) linguistic literature. 3 Peirce’s original definition (which is famously open to different interpretations) states that metaphors are types of hypoicon ‘which represent the representative character of a representamen by representing a parallelism in something else’ (Peirce c.1902: CP 2.277). 4 Somewhat related ideas within CL, with an explicitly diachronic (historical) focus, on how linguistic structures are motivated by but not determined by experience have been proposed by Heine (1997) and more recently by Schmid (2020). Some of these ideas have also been fruitfully integrated within cognitive semiotics (e.g. Pelkey 2017, 2018).

REFERENCES Abram, D. (1996), The Spell of the Sensuous: Perception and Language in a More-than-human World, New York: Vintage. Ahlner, F. and J. Zlatev (2010), ‘Cross-modal Iconicity: A Cognitive Semiotic Approach to Sound Symbolism’, Sign System Studies, 38: 298–348. Blasi, D. E., S. Wichmann, H. Hammarström, P. F. Stadler and M. H. Christiansen (2016), ‘Sound–meaning Association Biases Evidenced across Thousands of Languages’, Proceedings of the National Academy of Sciences, 113 (39): 10818–23.

288

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Blomberg, J. and J. Zlatev (2014), ‘Actual and Non-actual Motion: Why Experientialist Semantics Needs Phenomenology (and vice versa)’, Phenomenology and the Cognitive Sciences, 13 (3): 395–418. Brandt, L. (2013), The Communicative Mind: A Linguistic Exploration of Conceptual Integration and Meaning Construction, Newcastle: Cambridge Scholars. Brandt, P. A. (2004), Spaces, Domains, and Meaning: Essays in Cognitive Semiotics (European Semiotics Series 4), New York: Peter Lang. Cameron, L. and A. Deignan (2006), ‘The Emergence of Metaphor in Discourse’, Applied Linguistics, 27 (4): 671–90. Chomsky, N. (1990), ‘Language and the Problem of Knowledge’, in A. P. Martinich (ed.), The Philosophy of Language, 675–93, Oxford: Oxford University Press. Coseriu, E. (1985), ‘Linguistic Competence: What Is It Really?’ The Modern Language Review, 80 (4): xxv–xxxv. Coseriu, E. (2000), ‘The Principles of Linguistics as a Cultural Science’, Transylvanian Review (Cluj), 9 (1): 108–15. Croft, W. and D. A. Cruse (2004), Cognitive Linguistics, Cambridge: Cambridge University Press. Daddesio, T. C. (1995), On Minds and Symbols: The Relevance of Cognitive Science for Semiotics, Berlin: Walter de Gruyter. Dancygier, B. (2017), The Cambridge Handbook of Cognitive Linguistics, Cambridge: Cambridge University Press. De Cuypere, L. (2008), Limiting the Iconic: From the Metatheoretical Foundations to the Creative Possibilities of Iconicity in Language, Amsterdam: John Benjamins. Deacon, T. (1997), The Symbolic Species, New York: Norton. Devylder, S. (2018), ‘Diagrammatic Iconicity Explains Assymetries in Possessive Conctructions: Evidence from Paamese’, Cognitive Linguistics, 29 (2); 313–48. ­Devylder, S. (2019), Review of Barbara Dancygier (ed.), ‘The Cambridge Handbook of Cognitive Linguistics’ (2017), English Text Construction, 12 (1): 144–54. Devylder, S. (2020), ‘Polysemiotic Metaphors of Intertribal Exchanges in Paamese Culture’, Researching and Applying Metaphor (Virtual Conference), Hamar, Norway. Devylder, S. and J. Zlatev (2020), ‘Cutting and Breaking Metaphors of the Self and the Motivation and Sedimentation Model’, in A. Baicchi and G. Radden (eds), Figurative Meaning Construction in Thought and Language, Amsterdam: John Benjamins. Diget, I. S. K. (2019), Intersemiotic Translation from Film to Audio Description: A Cognitive Semiotic Approach (MA Thesis), Lund University, Lund. Dingemanse, M. (2012), ‘Advances in the Cross-linguistic Study of Ideophones’, Language and Linguistics Compass, 6: 654–72. Dingemanse, M., M. Perlman and P. Perniss (2020), ‘Construals of Iconicity: Experimental Approaches to Form–Meaning Resemblances in Language’, Language and Cognition, 12 (1): 1–14. Donald, M. (2001), A Mind So Rare: The Evolution of Human Consciousness, New York: Norton. Dowty, D. R., R. Wall and S. Peters (2012), Introduction to Montague Semantics, Voume. 11, Berlin: Springer Science and Business Media. Edelman, G. M. (1992), Bright Air, Brilliant Fire: On the Matter of the Mind, Volume 14, New York: Basic books New York. Erben Johansson, N., J. W. Carr and S. Kirby (2020), ‘Cultural Evolution Leads to Vocal Iconicity in an Experimental Iterated Learning Task’, in N. Erben Johansson (ed.), The Building Blocks of Sound Symbolism, 126–72, Lund: Media Tryck.

COGNITIVE LINGUISTICS AND SEMIOTICS

289

Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending and the Mind’s Hidden Complexities, New York: Basic Books. Forceville, C. (2017), ‘Visual and Multimodal Metaphor in Advertising’, Styles of Communication, 9 (2): 26–41. Gärdenfors, P. (1999), ‘Some Tenets of Cognitive Semantics’, in J. Allwood, P. Gärdenfors, A. Jucker, J. Mey, H. Parret and J. Verschueren (eds), Cognitive Semantics: Meaning and Cognition, 19–36, Amsterdam: John Benjamins. Geeraerts, D. (2010), Theories of Lexical Semantics, Oxford: Oxford University Press Geeraerts, D. and H. Cuyckens (2007), The Oxford Handbook in Cognitive Linguistics, Oxford: Oxford University Press. Gibbs, R. W. (2006), Embodiment and Cognitive Science, Cambridge: Cambridge University Press. Gibbs, R. W. (2017), Metaphor Wars, Cambridge: Cambridge University Press. Giraldo, V. (2019), ‘Referential Iconicity in Music and Speech’, Public Journal of Semiotics, 9 (1): 41–77. Grady, J. E. (1997), Foundations of Meaning: Primary Metaphors and Primary Scenes (PhD Dissertation), University of California, Berkeley. Grady, J. E. (2007), ‘Metaphor’, in D. Geeraerts and H. Cuyckens (eds), Handbook of Cognitive Linguistics, Oxford: Oxford University Press. Green, J. (2014), Drawn from the Ground: Sound, Sign and Inscription in Central Australian Sand Stories, Cambridge: Cambridge University Press. Haiman, J. (1983), ‘Iconic and Economic Motivation’, Language, 59 (4): 781–819. Haspelmath, M. (2008), ‘Frequency vs Iconicity in Explaining Grammatical Asymmetries’, Cognitive Linguistics, 19 (1): 1–33. Heine, B. (1997), Cognitive Foundations of Grammar. Oxford: Oxford University Press. Husserl, E. ([1900] 1970), Logical Investigations, New York: Humanities Press. ­Imai, M., M. Asano, M. Miyazaki, H. Okada, H. Yeung, K. Kitanjo, G. Thierry and S. Kita (2012), ‘Sound Symbolism Helps Infants’ Word Learning’, in E. Cartmill, T. C. Scott, M. Tamariz and J. Hurford (eds), The Evolution of Language: Proceedings of the 9th International Conference on the Evolution of Language, 456–7, Singapore: World Scientific. Itkonen, E. (2005), Analogy as Structure and Process: Approaches in Linguistics, Cognitive Psychology and Philosophy of Science, Amsterdam: John Benjamins. Itkonen, E. (2008), ‘The Central Role of Normativity in Language and Linguistics’, in J. Zlatev, T. P. Racine, C. Sinha and E. Itkonen (eds), The Shared Mind: Perspectives on Intersubjectivity, 279–306, Amsterdam: John Benjamins. Johansson, N. and J. Zlatev (2013), ‘Motivations for Sound Symbolism in Spatial Deixis: A Typological Study of 101 Languages’, Public Journal of Semiotics, 5 (1): 3–20. Johnson, M. (1987), The Body in the Mind, Chicago: University of Chicago Press. Johnson, M. (2010), ‘Metaphor and Cognition’, in D. Schmicking and S. Gallagher (eds), Handbook of Phenomenology and Cognitive Sciences, 401–14, Dordrecht: Springer. Jakobson, R. (1965), ‘Quest for the Essence of Language’, Diogenes, 13: 21–38. Jääskeläinen, A. (2016), ‘Mimetic Schemas and Shared Perception through Imitatives’, Nordic Journal of Linguistics, 39 (2): 15983. Kendon, A. (2004), Gesture: Visible Action as Utterance, Cambridge: Cambridge University Press. Kolter, A., S. H. Ladewig, M. Summa, C. Müller, S. C. Koch and T. Fuchs (2012), ‘Body Memory and the Emergence of Metaphor in Movement and Speech’, in S. C. Koch, C. Müller, M. Summa and T. Fuchs (eds.), Body, Metaphor and Movement (Advances in Consciousness Research 84), 201–26, Amsterdam: Benjamins.

290

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Konderak, P. (2018), Mind, Cognition, Semiosis: Ways to Cognitive Semiotics, Lublin: UMCS Press. Kövecses, Z. (2000), Metaphor and Emotion, Cambridge: Cambridge University Press. Kress, G. (2009), Multimodality: A Social Semiotic Approach to Contemporary Communication, London: Routledge. Lakoff, G. (1987), Women, Fire and Dangerous Things: What Categories Reveal about the Mind, Chicago: University of Chicago Press. Lakoff, G. and M. Johnson (1980), Metaphors We Live by, Chicago: University of Chicago Press. Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh: The Embodied Mind and Its Challenge to Western Thought, New York: Basic Books. Lakoff, G. and M. Johnson (2003), Metaphors We Live By, 2nd edn., Chicago: University of Chicago Press. Langacker, R. W. (1987), Foundations of Cognitive Grammar: Theoretical Prerequisites, Stanford: Stanford University Press. Langacker, R. W. (1990), Concept, Image, and Symbol: The Cognitive Basis of Grammar, Berlin: de Gruyter. Langacker, R. W. (2006), ‘Subjectification, Grammaticization, and Conceptual Archetypes’, in A. Athansiadou, C. Canakis and B. Cornillie (eds), Subjectification: Various Paths to Subjectivity, 17–40, Berlin: de Gruyter. Langacker, R. (2008), Cognitive Grammar: A Basic Introduction, Oxford: Oxford University Press. Langacker, R. W. (2016), ‘Working Toward a Synthesis’, Cognitive Linguistics, 27 (4): 465–77. ­Langacker, R. W. (2017), ‘Entrenchment in Cognitive Grammar’, in H. J. Schmid (ed.), Entrenchment and the Psychology of Language Learning: How We Reorganize and Adapt Linguistic Knowledge, 39–56, Berlin: Walter de Gruyter GmbH. Levinson, S. C. (2003), Space in Language and Cognition: Explorations in Cognitive Diversity, Cambridge: Cambridge University Press. Louhema, K., J. Zlatev, M. Graziano and J. V. D. Weijer (2019), ‘Translating from Monosemiotic to Polysemiotic Narratives: A Study of Finnish Speech and Gestures’, Sign Systems Studies, 47 (3–4): 480–525. Merleau-Ponty, M. (1962), Phenomenology of Perception, trans. C. Smith, London: Routledge. Moskaluk, K. (2020), Dizziness of Freedom: The Influence of Maladaptive Anxiety on Metaphorical Meaning-Making and the Motivation and Sedimentation Model (MA Thesis), Lund University, Lund. Möttönen, T. (2016a), Construal in Expression: Intersubjective Approach to Cognitive Grammar, Helsinki: University of Helsinki. Möttönen, T. (2016b), ‘Dependence of Construal on Linguistic and Pre-linguistic Intersubjectivity’, Nordic Journal of Linguistics, 39 (2): 209–29. Möttönen, T. (2019), ‘The Normative Basis of Construal’, in A. Mäkilähde, V. Leppänen and E. Itkonen (eds), Normativity in Language and Linguistics, 125–50, Amsterdam: John Benjamins. Mouratidou, A. (2019), ‘Choice Awareness and Manipulation Blindness: A Cognitive Semiotic Exploration of Choice-making’, Public Journal of Semiotics, 9 (1): 1–40. Müller, C. (2016), ‘Why Mixed Metaphors Make Sense’, in R. Gibbs (ed.), Mixing Metaphor, 31–56, Amsterdam: John Benjamins. Müller, C. and H. Kappelhoff (2018), Cinematic Metaphor: Experience–Affectivity– Temporality, Volume 4, Berlin: Walter de Gruyter GmbH and Co KG.

COGNITIVE LINGUISTICS AND SEMIOTICS

291

Munhall, K. G., P. Gribble, L. Sacco and M. Ward (1996), ‘Temporal Constraints on the McGurk Effect’, Perception and Psychophysics, 58 (3): 351–62. Nöth, Winfried (1999), ‘Peircean Semiotics in the Study of Iconicity in Language’, Transactions of the Charles S. Peirce Society, 35 (3): 613–19. Nöth, Winfried (2008), ‘Semiotic Foundations of Natural Linguistics and Diagrammatic Iconicity’, in K. Willems and L. De Cuypere (eds), Naturalness and Iconicity in Language (Iconicity in Language and Literature 7), 73–100, Amsterdam: John Benjamins. Oakley, T. (2009), From Attention to Meaning: Explorations in Semiotics, Linguistics, and Rhetoric, Bern: Peter Lang. Peirce, C. S. ([1866–1913] 1931–58) The Collected Papers of Charles Sanders Peirce, vols. 1–6, C. Hartshorne and P. Weiss (eds), vols. 7–8, A. Burks (ed.), Cambridge: Harvard University Press. Cited as CP. Pelkey, J. (2017), The Semiotics of X: Chiasmus, Cognition, and Extreme Body Memory, London: Bloomsbury Academic. Pelkey, J. (2018), ‘Upright Posture and the Meaning of Meronymy: A Synthesis of Metaphoric and Analytic Accounts’, Cognitive Semiotics, 11 (1): 1–18. Perniss, P., T. Thompson and G. Vigliocco (2010), ‘Iconicity as a General Property of Language: Evidence from Spoken and Signed Languages’, Frontiers in Psychology, 1: 1–15. Pragglejaz Group (2007), ‘MIP: A Method for Finding Metaphorically-used Words in Discourse’, Metaphor and Symbol, 22 (1): 1–40. Saeed, J. I. (2009), Semantics, Oxford: Oxford University Press. Sandler, W. (2012), ‘Dedicated Gestures and the Emergence of Sign Language’, Gesture, 12: 265–307. ­Saussure, F. D. ([1916] 1960), Cours de Linguistique Générale/ Course in General Linguistics, Paris and London: Payot and Duckworth. Schmid, H. J. (2020). The Dynamics of the Linguistic System: Usage, Conventionalization, and Entrenchment, Oxford: Oxford University Press. Sonesson, G. (2007), ‘From the Meaning of Embodiment to the Embodiment of Meaning: A Study in Phenomenological Semiotics’, in T. Ziemke, J. Zlatev and R. Frank (eds), Body, Language and Mind, Vol. 1: Embodiment, 85–128, Berlin: Mouton de Gruyter. Sonesson, G. (2009), ‘View from Husserl’s Lectern’, Cybernetics and Human Knowing, 16 (3–4): 107–48. Stampoulidis, G. and M. Bolognes (October 2019), ‘Bringing Metaphors Back to the Streets: A Corpus-based Study for the Identification and Interpretation of Rhetorical Figures in Street Art’, Visual Communication, 1470357219877538: 1–35 Stampoulidis, G. (2021). Street Artivism on Athenian Walls: A Cognitive Semiotic Analysis of Metaphor and Narrative in Street Art, Published PhD Dissertation, Lund University. Stampoulidis, G., M. Bolognesi and J. Zlatev (2019), ‘A Cognitive Semiotic Exploration of Metaphors in Greek Street Art’, Cognitive Semiotics, 12 (1): 1–20. Stjernfelt, F. (2014), Natural Propositions: The Actuality of Peirce’s Doctrine of Decisigns, Boston, MA: Docent Press. Talmy, L. (2000), Toward a Cognitive Semantics, Vol. 1, Cambridge, MA: MIT Press. Thompson, E. (2007), Mind in Life: Biology, Phenomenology and the Sciences of Mind, Cambridge: Harvard University Press. Torstensson, B. (2019), Metaphors and their Making: Bodily, Conventionally and Contextually Motivated Metaphors in Inter- and Intra-Generational Conversations (MA Thesis), Lund University, Lund.

292

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Tylén, K., R. Fusaroli, P. F. Bundgaard and S. Østergaard (2013), ‘Making Sense Together: A Dynamical Account of Linguistic Meaning-making’, Semiotica, 194: 39–62. van Langendonck, W. (2007), Iconicity the Oxford Handbook of Cognitive Linguistics, Oxford: Oxford University Press. Vandelanotte, L. (2017), ‘Viewpoint’, in B. Dancygier (ed.), The Cambridge Handbook of Cognitive Linguistics, 157–71, Cambridge: Cambridge University Press. Verhagen, A. (2007), ‘Construal and Perspectivisation’, in D. Geeraerts and H. Cuyckens (eds), The Oxford Handbook of Cognitive Linguistics, 48–81, Oxford: Oxford University Press. Vigliocco, G., P. Perniss and D. Vinson (2014), ‘Language as a Multimodal Phenomenon: Implications for Language Learning, Processing and Evolution’, Philosophical Transactions of the Royal Society B, 369 (1651): 1–7. Wilcox, S. (2004), ‘Cognitive Iconicity: Conceptual Spaces, Meaning, and Gesture in Signed Language’, Cognitive Linguistics, 15 (2): 119–47. Zinken, J. (2007), ‘Discourse Metaphors: The Link between Figurative Language and Habitual Analogies’, Cognitive Linguistics, 18: 445–66. Zlatev, J. (1997), Situated Embodiment: Studies in the Emergence of Spatial Meaning, Stockholm: Gotab. Zlatev, J. (2012), ‘Cognitive Semiotics: An Emerging Field for the Transdisciplinary Study of Meaning’, Public Journal of Semiotics, 4: 2–24. Zlatev, J. (2015), ‘Cognitive Semiotics’, in P. Trifonas (ed.), International Handbook of Semiotics, 1043–67, Dordrecht: Springer. Zlatev, J. (2016), ‘Turning Back to Experience in Cognitive Linguistics via Phenomenology’, Cognitive Linguistics, 27 (4): 559–72. ­Zlatev, J. (2019), ‘Mimesis Theory, Learning and Polysemiotic Communication’, in M. Peters (ed.), Encylcopedia of Educational Philosophy and Theory, Dordrecht: Springer. Zlatev, J. and J. Blomberg (2016), ‘Embodied Intersubjectivity, Sedimentation and Non-actual Motion Expressions’, Nordic Journal of Linguistics, 39 (2): 185–208. Zlatev, J. and J. Blomberg (2019), ‘Norms of Language: What Kinds and Where From? Insights from Phenomenology’, in A. Mäkilähde, V. Leppänen and E. Itkonen (eds), Normativity in Language and Linguistics, 69–101, Amsterdam: John Benjamins. Zlatev, J., P. Zywiczynski and S. Wacewicz (2020), ‘Pantomime as the Original Human-specific Communicative System’, Journal of Language Evolution, 5 (2): 1–19.

CHAPTER FOURTEEN

Cognitive Science and Semiotics GÖRAN SONESSON

For the purpose of this chapter, semiotics and cognitive science will be considered to be two distinct research traditions, each of which forms a temporal sequence made up of problems formulated from a particular perspective, solutions proposed to these problems, as well as new issues resulting from these solutions, and so on indeterminately. In this view, a research tradition can sometimes, at some places, be consolidated into a discipline, but before that, and after that, and at other places, it continues to exist as a network of (generations of) scholars, who are connected because of sustaining these issues and implementing some of these solutions. The perspective defining semiotics is the nature of meaning, broadly conceived, which may involve how meaning is produced and conserved by means of different vehicles (some or all of which may be signs, depending on how this term is defined; see Sonesson 2010), and how it develops in children, and in the evolution of human beings and other animals. The perspective of cognitive science is best described as being the nature of consciousness; although, as is well known, many of its exponents hoped to get rid of this pesky notion, substituting something easier to apprehend, such as the computer or the brain. It is also often interested in the uses to which consciousness (or whatever stands in for it) is put in the real world, as well as its history of evolution and development. As a research tradition, semiotics is very old, at least going back to Greek Antiquity, having reached a second apogee in Medieval Scholastics, and a third one during the Enlightenment, both periods of which abound in ‘treatises of signs’, before Peirce and Saussure toiled with it, in their separate ways, during the penultimate turn of the century, and the French structuralists made it fashionable during a short time span about seventy years ago. Cognitive science is much younger, but it is made up of disciplines, some of which, as research traditions, have rich pedigrees, which is true of psychology, philosophy, biology, linguistics and anthropology, and some which are very recent, such as artificial intelligence and neuroscience. Even more recent is cognitive semiotics, which aims to put together the perspectives of semiotics and cognitive science, and to resolve their separate, but (according to this tradition) connected, problems, by means of adding up their solutions and pooling their resources.

294

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

THE ENCOUNTER BETWEEN COGNITIVE SCIENCE AND SEMIOTICS Given that both semiotics and cognitive science are diversified research traditions, we need to go into some details about them both, before we can start untangling the events leading up to their encounter and the ensuing results. In this section, I suggest that three stages of development can be found within the constellation of disciplines termed cognitive science. They are termed ‘stages’, because they emerged in a temporal sequence, but this is not denying that many representatives of cognitive science have made halt at stage 2, let alone stage 1. Since this whole set of books is about semiotics, less need be said about (classical) semiotics (see vol. 1: History and Semiosis). Still, it is necessary to underscore where possible points of contact with cognitive science could be expected to occur. The third subsection will give a primary sketch of the ways this encounter has played out so far. Some of these scenarios will then be fleshed out in subsequent sections.

The stages of cognitive science Although the first ‘popular’ presentation of cognitive science bore the title, The Mind’s New Science (Gardner 1985), the original purpose of this conglomerate of disciplines was actually to do away with the mind, in the sense of consciousness, and instead substitute the computer. This first stage comprised in the tripartite scheme for the development of cognitive science delineated by Sonesson (2009: 115 ff.) corresponds to the good old days of artificial intelligence, from ‘symbolic computation’ as initiated by Herbert Simon to connectionism, originally associated with the work of David Rumelhart and James McClelland. Without any essential change to the basic epistemology, the computer metaphor was later exchanged for that of the brain, epitomized by Daniel Dennett’s ‘brain-in-the-vat’. The hardiest exponents of this neurological reductionism might be Paul and Patricia Churchland, but there can be no doubt the most influential thinkers of this stripe, at least in relation to the humanities (including semiotics), have been George Lakoff and Mark Johnson. In this scheme, the second stage starts out from the idea that, to function and to make sense, the computer, and by implication the mind, has to be ‘situated’: in other terms, it needs a context. If we restrict this context to the human body (or what is equivalent to it for the computer), we arrive at the notion of embodiment according to Lakoff and Johnson (1999). If then we take into account that, to human beings (and as it can be simulated in a computer), the representation of the body in the brain can be extended to the body in the world of real human experience, it is not too difficult to take the next step allowing the context to comprehend much more than the body. When Andy Clark and David Chalmers (1998) proceeded to include notebooks and everything else which might conserve information outside of the human organism in the notion of extended mind, their argument was exactly the same as that originally used to identify the computer and the mind: they are supposed to be functionally equivalent, in other words, substitutable. Pursuing the computer-brain metaphor to the end, it is possible to conclude, with Hutto and Myin (2013), that no content, or as they sometimes say, no meaning, is necessary, for neither the extended nor the un-extended mind, except (perhaps) in the case of language. So far, this historical account may roughly correspond to the distinction, made by Lakoff and Johnson (1999: 75ff.), between first- and secondgeneration cognitive science.

COGNITIVE SCIENCE AND SEMIOTICS

295

Starting out from Clark and Chalmers, we will arrive, by way of several intermediaries which we will neglect at present, in Mark Rowlands’ (2010) terms, to ‘the mind embodied, embedded, enacted, and extended’. Rowlands is out to establish whether or not all these addenda really are required, and to what extent they overlap. That, however, is not our present concern. From our point of view, the real issue, as we will see, is whether any of the ‘4E’ are really functionally equivalent to the un-extended mind. The notion of embodiment, in a more complicated sense than that of Lakoff and Johnson, epitomized by one hand feeling the other, was suggested much earlier, by Edmund Husserl, and later, as is more well known, by Maurice Merleau-Ponty. As for the extended mind more generally, it has much more of a pedigree, although it might only have been given a distinct status by Friedrich Hegel and his followers, and later by Husserl and the NeoKantians. Within recent semiotics, this theme has also been very much the focus of Jurij Lotman’s (1979) work. This is not to say that the current discussion of extended mind has not contributed new perspectives on this old issue, which are worthwhile exploring. The problem is, however, that no kind of extended mind can ever be equivalent to the un-extended mind, because they all need the mind, not only to be produced, but also to be interpreted. In the example given by Clark and Chalmers (1998), Otto’s notebook is meaningless, if he (or somebody else) does not write down the information, and if he does not remember (in particular since he is supposedly an Alzheimer’s patient) to look it up. The extended mind has to be extended from somewhere (Sonesson, 2021a). Although it is not pursued by most exponents of cognitive science, the third stage is of particular interest to us, because of its overlap with recent development in (cognitive) semiotics. It has resulted in a renovation of the study of consciousness, where Husserlinspired phenomenology has been prominent, although there have also been contributions from within the tradition of British-style linguistic philosophy. In spite of being anathema to much of mainstream cognitive science, notions such as agency, intentionality, consciousness, empathy and intersubjectivity are typical of ‘consciousness studies’, such as practised, for instance, by Evan Thompson (2007), Shaun Gallagher (2005) and Dan Zahavi (2005). A special mention should be made of Anthony Chemero (see Käufer and Chemero 2015) for bridging 4E cognitive science and phenomenology, including in the latter, the ecological psychology of James Gibson (as also suggested in Sonesson 1989). Since we are interested, in this chapter, in the encounter of semiotics and cognitive science, it is important to note two other characteristics of cognitive science, which results from it being, unlike semiotics so far, a constellation of already-existing sciences: first of all, thanks, notably to psychology, cognitive science employs experimental methods, which has very rarely been the case in semiotics. In the second place, because of the inclusion of linguistics and anthropology, cognitive science has posed questions about the origin of language and, marginally, some other semiotic resources, with reference to both child development and the origin of the human species, which have been a challenge to semiotic theory. This is particularly true of the work of two scholars whose basis is in neuroscience: Terrence Deacon (1997), who has had explicit recourse to terms borrowed from Peircean semiotics, and Merlin Donald (1991, 2001), who doesn’t use any semiotic terms, but who broaches issues in other ways familiar to anyone involved with semiotics.

Semiotics as a research tradition Since semiotics is the main subject of this book, we will not try to cover that ground here. Suffice it to say that, at the last mid-century, Saussure gained a prodigious following, in

296

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

particular as his notions were relayed by Louis Hjelmslev, in the French structuralists, whose most prominent representative turned out to be the Italian Umberto Eco. Focusing on one of the contradictory strands in Saussure’s posthumously published lectures, linguistics was described as an autonomous science, thus being enjoined to ignore any knowledge gained from other disciplines. Widening the scope to all kinds of semiosis, the French structuralists, and, more insistently, the Greimas school, claimed the same privilege for semiotics. ­Sonesson (1978) suggested in his dissertation that rather than an advantage, this amounted to a handicap. Unsurprisingly, his voice was not taken into account at the time, and there is no reason to think that many other protests did not go unheard in the same period. In Germany, nevertheless, such an extension of semiotics was realized concretely in a number of books by Walter Koch (1971) and his then disciple Winfried Nöth (1977). Koch (1986) even broached the subject of semiotic evolution. Although this cannot be considered an encounter of semiotics with cognitive science, which was hardly thought of at the time, it certainly constituted one of the failed attempts of semiotics to go into the interdisciplinary mode. In view of the importance of experiments to contemporary cognitive semiotics, it is worthwhile noting some early experiments, realized within a specifically defined semiotic framework. Thus, René Lindekens (1971) carried out experiments aiming to show that even photographic images were, to some extent, noniconic, since the shading of the picture approximated the real-world situation to the same extent that its contrast departed from that situation, and vice versa. In a similar vein, Hartmut Espe (1983) realized experiments which purported to show that the same photograph differently contrasted carries different affective import transferred to the subject of the picture: the girl appears more or less beautiful, the landscape more or less melancholic, and so on. But these were isolated instances of experiments being realized within semiotics. Among those protesters whose voices were more widely heard, Paul Bouissac may have been the first to give a wider scope to semiotics. In his interpretation, semiotics is a kind of ‘meta-analysis’, which ‘consists in reading through a large number of specialized scientific publications, selected among the published literature in one or several domains of inquiry, and of relating the partial results within a more encompassing model than the ones that are held by the various specialists concerned’ (Bouissac 1999: 4). While this is a very good description of interdisciplinary approaches in general, it is insufficient to spell out what is specific to this particular interdisciplinary approach. Sonesson (2010, 2016) suggested that the particular point of view taken by semiotics on these different domains of study pertains to the nature of meaning: more specifically, how meaning is produced, conveyed and experienced in different cases.

Approaches to cognitive semiotics Those of us who were present at the congresses of the IASS during the 1990s, have a distinct memory of Thomas Sebeok, who loomed large at those congresses, repeatedly pointing out to us that the difference between semiotics and cognitive science basically consisted in the latter, but not the former, having discovered the required formula to receive founding for its research. In his review of semiotics in the United States, Sebeok (1991: 2) claims that cognitive science is (at worst) ‘a historically untutored and – dare I use so pejorative a word – pretentious relabelling of an ancient multidisciplinary enterprise’ corresponding to what we have described above as the semiotic tradition of

COGNITIVE SCIENCE AND SEMIOTICS

297

thought. Ironically, Sebeok goes on to claim that the central notion of both traditions is that of representation. There is no space here to explore the polysemy of this term, also within the two traditions: suffice it to say that, at the time when Sebeok was writing his paper, cognitive science used the term in order to avoid talking about meaning; and since then, major trends within both semiotics and cognitive science have rejected the notion of representation, mostly without attending to the polysemy of the term, in favour of some version of active meaning-making, termed sign process, enaction or the like. The term ‘cognitive semiotics’ may first have been used, notably by Juan Magariños de Morentin (at least since 1999)1 as a label for Peircean semiotics. According to Claudio Paolucci (2021: 2), Umberto Eco (1999) also used this term with respect to Peirce’s semiotics. As far as I can tell, however, Eco (1999: 13) only claims that Peirce was the sole person whose theory was ‘semiotic, cognitive, and metaphysical all at the same time’. Such a usage is beyond our present concern, since it has nothing to do with the encounter between semiotics and cognitive science. Nor will it be possible, in the following, to account for all those scholars who have used the term in the required sense. In what follows, we will concentrate on contributions to the epistemological issues arising from such an encounter (see Konderak 2018). It is intriguing, nevertheless, that a least two scholars who want to marry cognitive science with semiotics, Nathan Houser (1996) and Claudio Paolucci (2021), understand the latter in terms of the Peircean brand. Perhaps the first explicit proposal for organizing an encounter between semiotics and cognitive science was formulated by Thomas Daddesio (1995), which, at the time, seems to have had zero impact. In Daddesio’s view, semiotics tends to restrict its attention to static structures, whereas cognitive science is involved with the way structures are processed by consciousness. As pointed out elsewhere (Sonesson 2016), this is a gross oversimplification. At the time, Daddesio had no means of knowing that, some years later, beginning with Clark and Chambers, cognitive science would trade very much into structures; but his description of semiotics suggests that he basically identified it with French structuralism, ignoring, most notably, the Prague School of semiotics, which was very much inspired by phenomenology, while also insisting on the social character of semiosis. Nevertheless, Daddesio was quite right in pinpointing the correlation between intersubjective structures (the Saussurean langue) and subjective access (language as ‘performance’, not in the sense of Chomsky but in that of psycholinguistics). It was probably Per Aage Brandt who first used the term ‘cognitive semiotics’ (dated by Brandt 2020 to the 1990s) to designate an approach fundamentally preoccupied with questions of linguistic semantics, which, from the very beginning, had been conceived within the framework of Greimasean semiotics (including the later contributions of René Thom and Jean Petitot), but which took on new inspiration from cognitive linguistics, notably as practised by Lakoff and Johnson (e.g. 1999) as well as Fauconnier and Turner (e.g. 2002). At the time, Brandt was active at Aarhus University in Denmark, but he later went to Case Western Reserve University in Ohio, where he joined Merlin Donald and Todd Oakley. Remaining in Aarhus, Peer F. Bundgaard may have been the first to publish an anthology bearing the title, in Danish, ‘Cognitive Semiotics’ (Bundgaard et al. 2003), which, however, apart from texts by Brandt, Bundgaard, Stjernfelt, Thom and Petitot, contained translations of papers mostly by exponents of cognitive linguistics. Frederik Stjernfelt (2007), who spent some time in Aarhus, but later returned to Copenhagen, is a formidable scholar, who has made important contributions to semiotics and phenomenology, both in the sense of Peirce and that of Husserl. In recent years, Aarhus

298

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

has been more notable for realizing many experimental studies, partly with a semiotic agenda (see the section below ‘The Contribution of Experimental Semiotics’). Quite independently of the Aarhus connection, another approach to cognitive semiotics emerged from the convergence of the lifetime work of two scholars at Lund University, who only began to collaborate at the turn of the last century: Göran Sonesson, who, after his linguistics studies in Lund, spent ten years working in France and in Mexico, after which he returned to Sweden to establish semiotics as a subject of doctoral studies; and Jordan Zlatev, who, after studies in cognitive science and linguistics, was employed at the Department of Linguistics in Lund. Zlatev and Sonesson initiated their collaboration with a faculty-internal project concerned with language, gesture and pictures in a developmental and evolutionary perspective, then proceeding to form the Swedish part in the EU project ‘Stages in the Evolution and Development of Sign Use’. Subsequently, they together the lead of a six-year-long programme, entitled ‘Centre for Cognitive Semiotics’, involving around thirty scholars (not counting the students), many from linguistics, but also from several other human sciences. From these experiences emerged an idea of cognitive semiotics as a specific way of merging semiotics and cognitive science which will be the main focus of the following sections.

SOME ISSUES OCCASIONED BY THE ENCOUNTER According to the first issue of the journal Cognitive Semiotics, published in 2007, cognitive semiotics aims to integrate ‘methods and theories developed in the disciplines of cognitive science with methods and theories developed in semiotics and the humanities, with the ultimate aim of providing new insights into the realm of human signification and its manifestation in cultural practices’. The ambition to take on the legacies of both semiotics and cognitive science is unproblematic, to the extent that they overlap or complement each other. The case is different when there seem to be contradictions between these two heritages, or perhaps between different strands within them. In the latter case, methodological and epistemological resolutions are called for. Since Zlatev and Sonesson seem to have been the only scholars actively concerned in sketching an epistemology for cognitive semiotics, the paragraphs which follow will attend to their proposals. This is not to deny that, in the end, cognitive semiotics may develop in quite different ways.

The part played by linguistics According to Jordan Zlatev (2015), linguistics should be considered, along with cognitive science and semiotics, to be the third element making up cognitive semiotics. It may be objected that linguistics is already one of the established disciplines incorporated into cognitive science, and that it is also, in principle, though rarely as an academic fact, a part of semiotics. It will be remembered that Saussure reinvented semiotics (his ‘semiology’) as a wider domain of research which should not only include the study of language as a part, but which was also called upon to take the latter at its template. This precept was unfortunately taken by most representatives of French structuralism as a pretext for projecting models and patterns which might be adequate for describing language to other semiotic means. The result could only be a serious distortion of the description of other semiotic resources, as was notably the case in pictorial semiotics (see Sonesson 1989, 2016). There are, however, other reasons for singling out linguistics. One of these reasons, and an important one, has to do with the epistemological status of (cognitive) semiotics.

COGNITIVE SCIENCE AND SEMIOTICS

299

The problematic position of the humanities, in a context in which the natural sciences already were the touchstone of scientificity, has been debated many times, but two such occasions are particularly pertinent in the present context, because they allowed, to a certain extent, the human (and social) sciences to gain a certain acceptance. The first of these rounds, the so-called Methodenstreit, which took place at the end of the nineteenth century, involved Heinrich Rickert, Wilhelm Windelband, Max Weber and Wilhelm Dilthey (see Beiser 2011), while the second round, which occurred in the middle of the twentieth century, was initiated by Hans-Georg Gadamer, Jürgen Habermas and Paul Ricoeur (see Radnitzky 1970; Ricœur 1983–5; Habermas 1985, 1995). The outcome of the first round was to describe the natural sciences as being nomothetic, that is, aiming to establish general laws, while the humanities were supposed to be ideographic, that is, their goal was to describe particular (historically situated) facts. The main contribution of the second round was to complicate the initial conditions, by including the social sciences, which, in the meantime, stood out as a different kind of science. In the intermission between the two rounds, Ernest Cassirer (1945: 89) observed that, during the first debate, ‘the fact that there is such a thing as human speech and that there is such a thing as linguistics was never mentioned’. He did not hesitate to qualify this as ‘a very regrettable fact, a sin of omission that could not fail to have its consequences’. It is easy to agree with this judgement. While linguistics is clearly, in this taxonomy, one of the human sciences, it is basically involved with laws and regularities, synchronically and, at least in part, diachronically (Sonesson 2018b). Cassirer did not live to discover that, also in the second round of the Methodenstreit, the peculiarity of linguistics was ignored. Without at the time being acquainted with Cassirer’s pertinent observation, Sonesson (1989, 2016, 2018b), taking semiotics to be a generalization of linguistics as a type of science, suggested both were fundamentally nomothetic, just as the natural sciences, but that, on the other hand, they were focused on qualities, like the traditional humanities, unlike the natural sciences, which were basically interested in quantities. This is a sense in which linguistics may serve as a paragon for the other semiotic sciences: in being predominantly nomothetic and qualitative. According to this conception, pictorial semiotics has to construct a model for the analysis of pictures, and then apply this model in the same way to a whole corpus of pictures (which is supposedly defined by some explicit criteria as being one and the same corpus), and if you observe discrepancies between the model and the corpus, you have to change the model. This has never seriously been done, no doubt in part because it would be a tedious work, which can only properly be realized with the help of computer algorithms (see Sonesson 1989; Reyes and Sonesson 2019). There is another sense in which linguistics may well play a fundamental part in the development of cognitive semiotics. There is a rich tradition in linguistics for studying the content side of the sign, that is, semantics, even if this part of the tradition was in part curtailed already by structuralism, and then more decisively by generative linguistics. Semantics writ large, it could be said, is semiotics. The pioneers of this kind of study were no doubt Jost Trier and Leo Weisgerber, whose studies of the semantic field were adapted to structuralist premises by Roman Jakobson, Louis Hjelmslev, Helmut Gipper, Eugenio Coseriu, Bernard Pottier, the early A. J. Greimas and others (see Diodato 2019; Coseriu 1977; Greimas 1966). After the hibernation of semantics within the framework of generative grammar, George Lakoff’s (1969) first attempt to strike it out on his own, under the label of ‘generative semantics’, establishing connections with psychology and fuzzy logic, as well as Charles Fillmore (1983) studies in case grammar, constituted, in

300

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

retrospect, a break with the Chomskyan paradigm, as well as a return to an earlier linguistic tradition. The later work by Lakoff, and his collaborators, as well as the contributions by Ronald Langacker and Leonard Talmy, while perhaps not taking much account of the earlier tradition, has certainly set up models which have been inspiring, not only within linguistics proper, but also for many scholars aiming to do cognitive semiotics (see Geeraerts and Cuyckens 2007). It is important to take into account these pioneer works within semantics, but, at the same time, there is a danger of generalizing too much from models possibly valid in linguistics to models for other semiotic resources.

With a little help from phenomenology Phenomenology, as we have seen (in an earlier section entitled ‘The Stages of Cognitive Science’), became an important contributor to cognitive science in its third stage. Although the basic tenets of phenomenology and much of contemporary semiotics may seem to be emphatically opposed, there have always been many connections between them, starting out with Husserl’s early text bearing the title, ‘Semiotik’. The entire model elaborated by the Prague semiotic circle was based on phenomenology (Mukařovský 1974; Sonesson, in press). Within Peirce’s semiotics, a distinct branch bore the name phenomenology, later rebaptized ‘phaneroscopy’, although it continued to carry almost the same definition as Husserlean phenomenology (Sonesson 2013; 2017). There is an obvious substratum of phenomenology, or more exactly, existentialism in much of French semiotics, already in the early work of Greimas (1966). Apart from being a domain where some parts of cognitive science and some parts of semiotics overlap, phenomenology has a very important part to play in cognitive semiotics. Since this overlap is otherwise rather limited, phenomenology is already required to mediate between the constructs of cognitive science and semiotics. It has to investigate whether both research traditions are really involved with the same thing, even though they use the same term, and whether they may be attending to the same subject, although they use different terms. Either way, phenomenology serves to explore the potentialities of enriching one conceptual framework using concepts and findings of the other. Since there are many brands of semiotics as well as of cognitive science, phenomenology should also take on the hard task of negotiating between their different claims and assumptions. In experimental studies, furthermore, it has to help formulating the tasks set up in such a way that they are relevant to the behaviour of real life, as well as connecting the results of the experiment to the situation in the experienced Lifeworld. Before more can be said about this issue, we have to consider the importance of experiments to cognitive semiotics, as well as to delve deeper into the nature of the semiotic sciences, more commonly known as the human and social sciences.

The contribution of experimental semiotics Even those, within semiotics, such as Koch, the early Nöth, and later Sonesson, who considered semiotics to be in the business of bridging traditional disciplines, taking into account concepts and findings from different disciplines, reviewed from an overarching semiotic perspective, did not realize any experiments of their own. Those semioticians who did, like Lindekens and Espe (see the section ‘Semiotics as a Research Tradition’) did not offer any rationale for doing so. Experiments have turned out to be an important

COGNITIVE SCIENCE AND SEMIOTICS

301

ingredient in cognitive semiotics, at least partly resulting from the encounter with cognitive science, and have played an important part in the approaches to cognitive semiotics, both in Aarhus and in Lund (see the section ‘Approaches to Cognitive Semiotics’). The term, ‘experimental semiotics’ is now normally applied to an experimental paradigm, which, according to Gallantucci and Garrod (2011: 1), proposes to study ‘novel forms of communication which people develop when they cannot use preestablished communication systems’. In spite of Galantucci’s claim to the contrary, this label has been used before, at least twice, to designate, on the one hand, the experimental study of perceptual meaning, as opposed to conventional meaning-making, and, at the other extreme, the study of complex social situations. In relation to Galantucci’s bare bone approach, both the minimalist procedure centring on perception and the maximalist manner including all of human sociability clearly involve human meaning-making, although subjecting it to experimental study in different ways (see Sonesson 2019). In the Galantucci-style bare bone approach, subjects are placed in a situation in which they are required to communicate, without using language, and in which they are, depending on the particular experimental situation, allowed the use of pictures or gestures, with or without some particular filter being applied to the employment of these means. This is no doubt a promising experimental procedure, and it has been used, not only by Galantucci and his close collaborators, but also by Kristian Tylén and his collaborators in Aarhus (e.g. Christensen et al. 2016). It should not be forgotten, however, that all experimentally manufactured situations are artificial in relation to the world of our experience, the Lifeworld, and this applies even more to such stripped-down situations. Cognitive semiotics in Lund has so far mostly pursued a more classical psychological approach, although starting out from features of more obvious relevance to semiotic theory. This involves creating a much richer experimental situation, which is, relatively speaking, closer to Lifeworld experience, and which thus may be used to mimic real-world situations in evolution and child development (e.g. Zlatev et al. 2013; Hribar et al. 2014; Sonesson and Lenninger 2015; Lenninger et al., 2020). Even though these experiments are no doubt closer to ordinary life, they remain very distant from it, since they aim to control a small number of features in the environment, in order to study the ensuing outcome. Even in this case, then, experiments have to be complemented by other approaches to bridge the space between the artifice of experimental situations and the Lifeworld.

Cognitive semiotics in relation to the human and social sciences If we except the third phase of cognitive science, which is influenced by phenomenology, the domineering ideology defining cognitive science is physicalist; that is, it takes for granted that the world as described by natural sciences is more real and/or basic that the one experienced by human beings. Many semioticians have no doubt shared in such a physicalist worldview, but, since semiotics is concerned with meaning, which cannot, as such, be described in physical terms, this must, in the end, be a self-defeating approach. Giambattista Vico ([1710] 1988: 45) famously formulated the Verum-Factum principle, according to which we, as human beings, have access to the truth about anything we, or other human beings, have created, that is, culture in general, and all specific cultures, whereas knowledge of other things, such as nature, can only be more indirectly, and laboriously, gained. Without any reference to Vico, the semiotician Luis Prieto (1975: 215 ff.) has suggested that, contrary to the reigning positivist epistemology, our access to the subject matter

302

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

of the human and social sciences (which he terms the semiotic sciences) is direct, and thus objective, while our access to the subject matter of the natural sciences can only be indirect and therefore subjective. Prieto observes that the sound, as a material object, may have innumerable properties, but the phoneme, which is the subject matter of linguistics, is defined by its functional relations to the other plane of the sign, and thus has a limited number of clearly defined features. Nevertheless, there is a problem with this claim, not only because our knowledge even of what we ourselves have created may be somewhat obscure, but more importantly, because, even in linguistics, we will have to go below the level of what we as language users are ordinarily aware of, beyond the phoneme to the phonological features, and so on (Sonesson 1989: 28 ff.). Put in the traditional terms of hermeneutics, we may say that, after coinciding with the user in his or her understanding of the phoneme, the semiotician goes on to explain the conditions of possibility of this understanding on the level of distinctive features. If so, semiotics involves the knowledge of the user but also something more. There are basically two procedures for going beyond what is directly given to human beings as users of meaning (see Sonesson, 2021b). One of these procedures, which can be characterized as an iconic operation, is particular to the semiotic sciences, as suggested by Vico and Prieto. The second one, which can be described as being indexical, is fundamentally the same as that employed by the natural sciences. Here it must be clear that these terms are not used to describe different kinds of signs, and not even, in the first case, a kind of ground (see Table 14.1). According to Sonesson’s (2013) meta-analysis of Peirce’s different phenomenological characterizations of grounds, iconicity basically corresponds to the idea of ‘something being there’, that is, a direct perceptual experience. Indexicality, according to the same meta-analysis, involves something being added to what was first perceived, by some more indirect means. Along the lines of what will henceforth be called the Vichean-Prietean supermaxim, it may be supposed that, since nature is opaque to us, we can only gain knowledge about it using indexical operations, but that the semiotic sciences, however paradoxical that may seem, can use iconic operations to go beyond first appearances, while also needing, in specific cases, to have recourse to indexical operations. The first procedure is only possible because of what Husserl has described as the sedimentation of acts in the stream of consciousness. According to Husserl (1901, 1966), all our acts of consciousness (including those which have an external manifestation) can only have a meaning because they take for granted the ‘passive synthesis’ of earlier experiences, building on numerous other acts of consciousness, some of which took place in our earlier life (genetic sedimentation), and many others which resulted from the experience of earlier generations of human

TABLE 14.1  The relations between principles, grounds and signs in the present interpretation of Peirce (Sonesson 2010) Firstness

Secondness

Thirdness

Principle (Firstness)

Iconicity

—

—

Ground (Secondness)

Iconic ground

Indexicality = indexical ground

Symbolicity = symbolic ground =

Sign (Thirdness)

Iconic sign (icon)

Indexical sign (index)

Symbolic sign (symbol)

COGNITIVE SCIENCE AND SEMIOTICS

303

beings (generative sedimentation). Since the sedimented results of these acts are what help make sense of our current interaction, we may be able to reactivate these acts, using the procedures of phenomenology. Combining Husserlean and Peircean phenomenology, we could say that this is a way of delving deeper into iconicity. There are two, very different, reasons why indexical procedures are needed. In the first case, some facts are simply not accessible to the human mind, and we are reduced to proposing relatively arbitrary hypotheses, which can be more or less well founded, to the extent that they serve to connect a greater or lesser extent of nodes of observations, whether these observations are made with the naked eye, with some, relatively speaking, simple extension of the eye such as microscopes and telescopes, or more sophisticated devices such as PET-scanning, fMRI and the like. In any case, these are probes into a matter which remains basically inaccessible to us, which is not to deny that the history of the (natural) sciences of the last 500 years has been the story of the extension of our ability to delve deeper into this black matter. The second case is different, but it leads to a similar conundrum. In spite of all our efforts to reanimate the sedimented layers of our consciousness, whether they were laid down genetically and generatively, our access to their meaning is often blocked. Using a term which has a history, since, after being coined by Ludwig Feuerbach, it was adopted and adapted by Karl Marx, and thus by all his followers, we might say that some kinds of sedimentations resist reanimation, because they have become reified. This is why some approaches to the study of human, and in particular, social, life have adopted the indexical procedures characteristic of the natural sciences. A case in point is economics, which is certainly opaque to human consciousness, although it is clearly a human creation. Although the term was coined for different reasons, economics deserves to be called ‘the dismal science’, because its hypotheses most of the time turn out to be wrong. The difference between natural science and economics is that, in the latter case, we do not have to wait for aeons to pass, because the predictions are rapidly falsified by close-time historical events. That is why economics and comparable approached to social life are, in Prieto’s sense, as subjective as the natural sciences.

COGNITIVE SEMIOTICS ON ITS OWN The purpose of the present section is to draw some conclusions from the preceding section, going beyond the encounter between semiotics and cognitive science, and taking into account the offer, more or less clearly assumed, by both these approaches to serve as a way of integrating the diverse human and the social sciences, and thus, by the same token, clarifying their relation to the scientific enterprise as a whole.

The domains of study Jordan Zlatev (2008) has suggested that the domain of study covered by cognitive semiotics can be divided into several levels, each with its corresponding subjects, worlds and signification. Meaning, in the broader sense of the term (well beyond the notion of sign), can be understood as the relationship between a subject and his/her environment which is determined by a value. On a biological level, this value is life, more specifically the maintenance and reproduction of life, and the subject is the organism. This is what was described by Jakob von Uexküll (1973) as the ‘functional cycle’. The next level, which Zlatev calls the natural Lifeworld (in the sense of what pertains to nature), has as its subject the minimal ego and is governed by the value of sentience. As Zlatev identified this

304

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

latter value with intentionality, in Husserl’s sense, as it originally appears in perception, it can be understood as consciousness, in the broadest sense of the word. Above the level of the minimal Ego is situated the cultural Ego, the subject of the cultural Lifeworld, which is determined by a value that Zlatev calls meaning. As we have already used this term to describe the relationship between a subject, an environment and a value, we will call culture the specific value of the third level. What is new here is life in a community, also known as sociability. Zlatev originally introduced the sign at this level, in the narrow sense of Sonesson (1989): a unit of two items which are differentiated from each other, one of which is directly present but not in focus, while the other is in focus, but not directly experienced. Sonesson has long argued, mostly orally (but see Sonesson 2015a) that there must be an intermediate level, which is not made up of signs, but which is still intersubjective and contains behaviours that set one society apart from another.2 Zlatev’s fourth level, which is for Sonesson the fifth (and now also of Zlatev 2015), consists for Zlatev of language, but it could also be generalized to include all kinds of semiotic resources which are in any way systemic, whether they depend mostly on rules of substitution, which is the case with language and formalized gesture systems, or on rules of transformation, as is the case with pictures and some kinds of gestures (see Table 14.2). Many animals, notably primates, but also some birds, clearly have cultures, in the minimal sense of the term. It remains difficult to determine whether some animals are capable of using signs or not, but systems of signs seem to be the exclusive property of human beings. In any case, there are many other reasons at present for conceiving sociability as being a particular rung on the semiotic hierarchy. It seems very probable that group selection is at least as important in evolutionary biology as individual selection, and that the specificity of human culture depends largely on the combination of natural and cultural selection rendered possible by group selection (see Richerson and Boyd 2005; Dunér and Sonesson 2016).

Evolution and development and anything in between According to a scenario for human evolution designed by Merlin Donald (1991), which has been very influential in cognitive semiotics, episodic memory, the memory for single situated happenings, is something that human beings share with many other animals. Mimetic memory, however, involves using the body to record behaviour, and is restricted to human beings and their antecessor species. Donald’s third stage, mythic memory, corresponds to language, so designated because, according to Donald, it involves the

TABLE 14.2  ‘The semiotic hierarchy’: Five types of selves, worlds and significations (adapted from Sonesson 2015a, as an elaboration of Zlatev 2009) ­ Level

Subjects

Worlds

Significations

Value

1

Organism

Umwelt

Biological

Life

2

Minimal Ego

Natural Lebenswelt

Intentional

Sentience

3

Cultural Ego

Cultural Lebenswelt

Cultural

Culture

4

Signic Ego

Universe of signs

Signic

Sign

5

Semiotically specific ego

Universe of discourse

Systemic

Sign system

COGNITIVE SCIENCE AND SEMIOTICS

305

construction of narratives, no doubt initially used to recount myths, and thought by Donald to be at least one of the reasons why language evolved. In this scenario, language would seem to be a limiting case: the capacity for language might well have an evolutionary origin, but the development of particular languages is part of history. It is in the fourth stage, however, which Donald calls theoretic culture, that history comes of its own. Pictures, writing and theories are the three human products typically manifesting this stage according to Donald. In later texts, Donald (2010) has used the term ‘exograms’ to pinpoint the specificity of this stage, which consists in making memory available outside, and relatively independently, of human beings, both as to their body (as in mimetic memory) and as to their consciousness (as in mythic memory). Within Donald’s mimetic stage, a number of important things, not properly distinguished in his scheme, seem to happen. First there is the use of tools in a systematic way, which may be thought of as memory embodied in the limbs. If tool use is going to gain currency in a community, imitation must take place, and imitation would seem to require the capacity to pick up (cognize) the type, given a number of tokens – or, in other words, extracting it from a sequence of episodes. Thus, in order to imitate an observed use of a hammer, which is a token of the act of hammering, one must be able to abstract the type from this individual act and the particular physical and motivational context in which they take place. It is only once you conceive the act of hammering as a type that you can transpose hammer usage to other contexts, in this way giving rise to a second token, and a third, etc. (see Dunér and Sonesson 2016). According to Donald, this is also the stage in which gestures begin to be used, but if so, this requires something more than the mere fitting of the token to the type, the discovery of signs, in the sense of a unit consisting of two items which are differentiated from each other, one of which is directly present but not in focus, while the other is in focus, but not directly experienced. In fact, in terms of the domains of meaning distinguished above (see the section ‘The Domains of Study’), already the episodic stage seems to go well beyond the mere Umwelt, entering the natural Lifeworld, but the mimetic stage appears to cover all the distance from the natural Lifeworld to the cultural Lifeworld and further to the universe of signs. On the other hand, many animals clearly live in a pre-episodic Umwelt, because they do not need to have any experience of the here-and-now: the tick, for instance, has to realize its acts in a certain temporal sequence, but not going back and forth in time (see Sonesson 2015b). The import of the third stage clearly is sociability becoming an important factor in the creation of meaning. This is not to suppose, along the lines of Rousseau’s ‘philosophical history’, that human beings once lived alone, like the orangutans, or even were the orangutans, but only that this factor gained more importance, and was more decisive, once cultural evolution came to predominate over natural evolution. The fourth stage also has its problems, which concern, in particular, the case of pictures. There is no obvious reason to suppose that the advent of picture production had to attend the capacity to preserve memory outside of human bodies and minds. If pictures were created earlier in history, they may have employed more perishable media, such as sand or human skin. In fact, there are historically and ethnographically testified examples of drawings customarily being made on the ground (for a recent example, see Green 2014; cf. Sonesson 2015b). Not even the tentative to reanimate generative sedimentations is of any help here. All approaches to evolution are bound to encounter the hurdle of memory reification. Studies of evolution are fundamentally reduced to using indexical procedures, in a situation in which the indexicalities available for making the connection are very sparse. Although we

306

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

have long since been warned to project scales of evolution and child development onto each other, this is basically our only extant recourse. Even if the analogy doesn’t hold, the projection of Donald’s scale onto child development, independently made by Kathrine Nelson (2007) and Zlatev (2008), may still be of interest, and, while it also depends largely on indexical procedures, it nevertheless has many more data points to rely on. Something similar is true about the study of the behaviour in animals similar to human beings, most notably the great apes. Both these approaches have been used at the Centre for Cognitive Semiotics at Lund University (e.g. Zlatev et al. 2013; Hribar et al. 2014; Sonesson and Lenninger 2015; Lenninger et al. 2020). Whether or not these analogies hold, the results should be interesting in their own right. Studies of ontogenesis and psychogenesis should not make us forget about microgenesis, so termed by Heinz Werner, but first studied, using the term ‘Aktualgenese’, by the Gestalt psychologist Friedrich Sander (see Sonesson 1989: 149; Rosenthal 2004). The term microgenesis designates the dynamic unfolding and differentiation on a brief time scale of a percept, a thought, an object of imagination or an expression. Most studies have, from the beginning, involved perception, but there is no reason to limit the notion to this domain of experience. Although Sander’s and Werner’s original studies were experimental, microgenesis also lends itself easily to a phenomenological approach, as testified by the pioneering work of Aron Gurwitsch (1964), which largely overlaps with earlier Gestalt psychology.

Methodologies In many papers, Sonesson has used the phrase ‘the dialectics of phenomenology and semiotics’ to describe how phenomenology is first called upon to construct an experimental situation, in such a way that it is relevant to the behaviour of real people, and then is required to relate the results of the experiment to the situation in the experienced Lifeworld, all of which forms a circle which may be repeated many times over (see the section ‘With a Little Help from Phenomenology’). More broadly, Zlatev (2015: 1059) describes ‘methodological triangulation’ as a characteristic of cognitive semiotics, which involves the integration of what he calls first-, second- and third-person methods as applied to any domain, where, however, first- and second-person methods have epistemological priority. Here the types of methods should, naturally, be understood in analogy to the three persons of the personal pronouns in Indo-European languages, also construed by Zlatev as being, respectively, subjective, intersubjective and objective.3 In subordinating third-person methods, in this sense, to first- and second-person methods, cognitive semiotics goes against the grain of cognitive science, except in what was characterized above as its third stage. This is, of course, not subjectivity and objectivity, as these terms are understood by Prieto (see the section ‘Cognitive Semiotics in Relation to the Human and Social Sciences’). To avoid confusion, the terms ipseity, dialogicity and neutrality will be used in the following (see Table 14.3). More importantly, Sonesson has suggested that we need to go a step further by distinguishing the modes of access employed and the phenomena on which they operate, both of which may pertain to the first, second and third person (see Table 14.3). Both in the case of introspection and in that of the phenomenological reduction, the mode of access is necessarily (in spite of Dennett 2003) the first person; but in introspection, we try to discover something about the particular person realizing the operation, whereas phenomenology is concerned with mapping out the invariant structures of

COGNITIVE SCIENCE AND SEMIOTICS

307

TABLE 14.3  The distinction of ipseity, dialogicity and neutrality in the phenomena accessed and the modes of access (adapted from Sonesson [submitted], as an extension of Zlatev 2015: 1059)

Modes of access

Phenomena accessed Ipseity (First person)

Dialogicity (Second person)

Neutrality (Third person)

Ipseity (First person)

Introspection

Empathy

Phenomenology

Dialogicity (Second person)

Participant observation

Dialogue

Interactive description

Neutrality (Third person)

Behaviouristic description (‘heterophenomenology’)

Interview

Experimentation

consciousness, which is to say that it aims to establish some neutral structures. From the point of view of their modes of access, then, both operations rely on ipseity, but only the first operation simply aims to access a phenomenon of ipseity. We can have third-person reports of introspection, which is what Daniel Dennett (2003), confusingly, calls ‘heterophenomenology’, but a more fruitful third-person approach which pertains to the third-person domain is experimentation. Both phenomenology and experimentation aim to attain neutral phenomena, but experimental approaches opt for a neutral mode of access, while phenomenology uses an access pertaining to ipseity. In part, experimentation and phenomenology may not be able to attain the same neutral structures. In part, however, they do, but then phenomenology uses an iconic approach (in the sense of the section ‘Cognitive Semiotics in Relation to the Human and Social Sciences’), while experimentation is reduced to an indexical modus operandi. Since both methods have their intrinsic problems, it is certainly worthwhile employing both approaches.

STATE-OF-THE-ART AND CURRENT PRIORITIES In the preface to a Conference volume, Zlatev, Sonesson and Konderek (2016: 9) claimed that cognitive semiotics could no longer be considered merely an ‘emerging’ discipline, but that, on the contrary, it already existed. Let us look at the evidence. On the positive side, we have two institutional facts. An international journal on cognitive semiotics, (Journal of) Cognitive Semiotics – Multidisciplinary Journal on Meaning and Mind, was initiated by scholars mostly connected to Aarhus semiotics in 2007. Several issues were published by Peter Lang Verlag, and, after an interregnum consisting of online publications, the journal was founded anew in 2014, in collaboration between scholars from Lund University, Aarhus University and Case Western Reserve University. It has since then been published by Mouton de Gruyter. The current board consists of Peer F. Bundgaard, Göran Sonesson, Todd Oakley, Merlin Donald and Bruno Galantolucci. This revival is connected to a second institutional fact, the creation of the International Association for Cognitive Semiotics (IACS), founded in 2013, and which has since then

308

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

held three successful conferences, in Lund (2014), Lublin (2016) and Toronto (2018), while a fourth one, which was planned to occur in Aachen in 2020, had to be postponed because of the coronavirus, but was finally held online in 2022. On the negative side, experience from the papers submitted to the journal and presentations at the conferences suggest that the specificity of cognitive semiotics, as delineated above, and more broadly as an encounter between cognitive science and semiotics, has still not been widely assimilated by most scholars. It goes without saying that the epistemology of cognitive semiotics does not have to be exactly the one sketched above. But many scholars still have to understand that cognitive semiotics is not identical to classical semiotics or traditional cognitive science, let alone to linguistics. This brings us to the first priority for this kind of research, which is precisely for those who aspire to be part of it to assume the whole intricacy of the conditions attending on the encounter between the two research traditions. Another current priority is no doubt to work for a full-scale employment of findings, models and methods from both interdisciplinary approaches and the traditional disciplines which they both strive to integrate. There are still models, methods and findings, both in semiotics and in cognitive science, which have not been fully assumed within the new research tradition. Finally, another priority should be to achieve a better cross-penetration of the tool kit and other paraphernalia offered by both traditions. This observation is in need of a gloss: as it is presented above, cognitive semiotics, unlike most of cognitive science, and like some parts of semiotics, phenomenology and the traditional human sciences, privileges an approach from within the Lifeworld, e.g. iconic operations, or, in other terms, employing first- and second-person modes of access. An encounter between semiotics and cognitive science may be organized, and to some extent has been done so, by taking one’s point of departure in traditional cognitive science; but, so far, there do not seem to exist any epistemological considerations attending to such an encounter.

CONCLUSION In this chapter, we have presented cognitive science and semiotics as two research traditions, the encounter of which has resulted in cognitive semiotics. The goal of the latter is to assimilate the methods, models and findings of both these research traditions onto a single pool of knowledge. With this aim in view, we have considered a few milestones in the history of semiotics as a tradition of thought and of cognitive science as a rather recent, still variegated, paradigm, set up to unify a number of already-existing sciences. With this point of departure, we have examined what cognitive semiotics is, and could be, considered as a way of banking on the successes and failures of both traditions of thought, thus pulling the resources of both traditions together. Any results of such an exploration can of course only have a preliminary value. Such an encounter as that between cognitive science and semiotics cannot be realized without certain decisions having been taken as to the overall epistemology of the ensuing research tradition. Just as epistemological determinations have come out differently in the history of semiotics as well as that of cognitive science, as we have seen above, these issues have been settled in different ways by different advocates of cognitive semiotics. Since they seem to have been the only scholars to have explicitly and pointedly set up (partly overlapping) epistemological stakes, Jordan Zlatev and Göran Sonesson have been our guides in the exposition of cognitive semiotics above. This is not to deny that other scholars, who have recently published books carrying the mention ‘cognitive semiotics’

COGNITIVE SCIENCE AND SEMIOTICS

309

in their title, have chosen a somewhat different path: thus, Konderak (2018), while accepting much of the Sonesson-Zlatev lore, in the end puts in a plea for computational modelling, and Paolucci (2021) bases his cognitive semiotics on Peircean semiotics, with a touch of late Eco, combined with radical enactivism. As for Brandt (2020), his cognitive semiotics remains very much a child of French structuralism (even though Brandt, at the end of his book, claims cognitive semiotics is the very opposite of structuralism). The justification for focusing on the notion of cognitive semiotics as conceived by Zlatev and Sonesson (apart from the fact that this entry is written by one of these scholars) is that, instead of simply importing an epistemology from some pre-existing doctrine, such as that of Peirce or Greimas, they have attempted to elaborate an epistemology specifically for cognitive semiotics, taking into account the emphasis on meaning in semiotics and the plurality of methods in cognitive science. According to this conception, phenomenology is presumed to be the link between the two earlier research traditions, since it could be considered the philosophical approach most concerned with meaning, which is central to semiotics and, also, although often repressed, to cognitive science. According to our pacesetters, the trajectory of meaning and thus the domains studied by cognitive semiotics run from the organism to the conjunction of sign systems in society. Cognitive semiotics uses methods taking their departure in the first, second and third person, which it tries to combine; though, given its point of departure in meaning, thirdperson methods only have an ancillary function. Unlike Zlatev, Sonesson does not want to mention linguistics as a third ingredient of cognitive semiotics, because it is already included in the former two. Still, also in his conception, linguistics has a fundamental part to play in cognitive semiotics. It is the epitome of cognitive semiotics, not for the reason suggested by Saussure and the French structuralists, but because it fails to conform to the pattern indicated for scientific disciplines delineated during the classical Methodenstreiten, according to which the natural sciences are dedicated to setting up rules and regularities, which they preferably formulate in terms of quantities, while the cultural sciences are concerned with individual facts, which they study in qualitative approaches. Breaking this pattern from the start, linguistics is involved with setting up rules, which are mainly of a qualitative nature. In this respect, linguistics is a paragon for some approaches to semiotics, and certainly for cognitive semiotics.

NOTES 1 As testified by his website, still persisting after his untimely death in 2010: http://www. magarinos.com.ar 2 We can certainly argue whether consciousness can appear before living in society, and whether we should not identify level 2 and 3, but I believe that, in the sense of a perceptual consciousness, consciousness is much more elementary than society. 3 These three kinds of methods should not be confused with Peirce’s notions of Firstness, Secondness and Thirdness. In fact, there is only a partial overlap (see Sonesson 2018b).

REFERENCES Beiser, F. C. (2011), The German Historicist Tradition, Oxford: Oxford University Press. Bouissac, P. (1999), ‘Le savoir sémiotiques’, Degrés, 25 (100): c1–c15. ­Brandt, P. A. (2020), Cognitive Semiotics: Signs, Mind and Meaning, London: Bloomsbury Academic.

310

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Bundgaard, P., J. Egholm and M. Skov, eds (2003), Kognitiv semiotik: en antologi om sprog, betydning og erkendelse, København: Haase. Cassirer, E. ([1945] 1972), ‘Structuralism in Modern Linguistics’, in Word 1: 99–120. Quoted from B. Malmberg (ed.), Readings in Modern Linguistics, 8th edn, 78–98, Stockholm: Scandinavian University Books. Christensen, P., R. Fusaroli and K. Tylén (2016), ‘Environmental Constraints Shaping Constituent Order in Emerging Communication Systems’, Cognition, 146: 67–80. Clark, A. and D. Chalmers (1998), ‘The Extended Mind’, Analysis, 58 (1): 7–19. Coseriu, E. (1977), Principios de semántica estructural, Madrid: Gredos. Daddesio, T. (1995), Of Minds and Symbols, Berlin: Mouton de Gruyter. Deacon, T. (1997), The Symbolic Species, New York: Norton. Dennett, D. (2003), ‘Who’s on First? Heterophenomenology Explained’, Journal of Consciousness Studies, 10 (9–10): 19–30. Diodato, F. (2019), ‘The Neo-Humboldtian Lexical Field Theory. Origin, Reception and Perspectives’, Paradigmi, 2: 259–74. Donald, M. (1991), Origins of the Modern Mind. Three Stages in the Evolution of Culture and Cognition, Cambridge: Harvard University Press. Donald, M. (2001), A Mind So Rare, New York: Norton. Donald, M. (2010), ‘The Exographic Revolution: Neuropsychological Sequelae’, in L. Malafouris and C. Renfrew (eds), The Cognitive Life of Things: Recasting the Boundaries of the Mind, 71–9, Cambridge: McDonald Institute Monographs. Dunér, D. and G. Sonesson, eds (2016), Human Lifeworlds: The Cognitive Semiotics of Cultural Evolution, Frankfurt am Main: Peter Lang. Eco, U. (1999), Kant and the Platypus: Essays on Language and Cognition, trans. A. McEwen, London: Secker & Warburg. Espe, H. (1983), ‘Empirische Analyse visueller Zeichen’, in M. Krampen (ed.), Visuelle Kommunikation und/oder verbale Kommunikation, 92–121, Hildesheim and Berlin: Olms Verlag. Fauconnier, G. and M. Turner (2002), The Way We Think: Conceptual Blending And The Mind’s Hidden Complexities, New York: Basic Books. Fillmore, C. (1968), ‘The Case for Case’, in W. Bach and R. Harms (eds), Universals in Linguistic Theory, 1–88, New York: Holt, Rinehart, and Winston. Galantucci, B. and S. Garrod (2011), ‘Experimental Semiotics: A Review’, Frontiers in Human Neuroscience, 5 (11): 1–15. Gallagher, S. (2005), How the Body Shapes the Mind, Oxford: Clarendon. Gardner, H. (1985), The Mind’s New Science, New York: Basic books. Geeraerts, D. and H. Cuyckens (2007), The Oxford Handbook of Cognitive Linguistics, Oxford: Oxford University Press. Green, J. (2014). Drawn from the Ground, Cambridge: Cambridge University Press. Greimas, A. J. (1966), Sémantique structurale: Recherchede méthode, Paris: Larousse. Gurwitsch, A. (1964), The Field of Consciousness, Pittsburgh: Duquesne University Press. Habermas, J. (1985), Zur Logik der Sozialwissenschaften: Erweiterte Ausgabe, Frankfurt am Main: Suhrkamp. Habermas, J. (1995), Theorie des kommunikativen Handelns, Frankfurt am Main: Suhrkamp. Houser, N. (1996), ‘Semiotic as Cognitive Science’, Cruzeiro Semiotico, 22/25: 139–50. ­Heis, J. (2018), ‘Neo-Kantianism’, The Stanford Encyclopedia of Philosophy (Summer 2018 edn), ed. E. N. Zalta, https://plato.stanford.edu/archives/sum2018/entries/neokantianism/

COGNITIVE SCIENCE AND SEMIOTICS

311

Hribar, A., J. Call and G. Sonesson (2014), ‘From Sign to Action. Studies in Chimpanzee Pictorial Competence’, Semiotica, 198: 205–40. Husserl, E. (1901), Logische Untersuchungen, Tübingen: Niemeyer. Fünfte Auflage 1968. Husserl, E. (1966), Analysen zur passiven Synthesis: aus Vorlesungs- und Forschungsmanuskripten 1918–1926, Husserliana: Gesammelte Werke. Bd 11. Haag: Nijhoff. Hutto, D. and E. Myin (2013), Radicalizing Enactivism: Basic Minds without Content, Cambridge, MA: MIT Press. Käufer, S. and A. Chemero (2015), Phenomenology: An Introduction, Cambridge: Polity Press. Koch, W. (1971), Varia Semiotica, Hildesheim: Olms Verlag. Koch, W. (1986), Evolutionäre Kultursemiotik, Bochum: Brockmeyer. Konderak, P. (2018), Mind, Cognition, Semiosis, Lublin: Maria Curie-Skłodowska University Press. Lakoff, G. (1969), On Generative Semantics, Bloomington: Indiana University Linguistics Club. Lakoff, G. and M. Johnson (1999), Philosophy in the Flesh, New York: Basic Books. Lenninger, S., T. Persson, J. van de Weijer and G. Sonesson (2020), ‘Mirror, Peephole and Video: The Role of Contiguity in Children’s Perception of Reference in Iconic Signs’, Front, Psychol., 14 July 2020. https: //doi.org/10.3389/fpsyg.2020.01622 Lindekens, R. (1971), Eléments pour une sémiotique de la photographie, Paris: Didier/Aimav. Lotman, J. (1979), ‘Culture as Collective Intellect and Problems of Artificial Intelligence’, Russian Poetics in translation, 6: 84–96. Mukařovský, J. (1974), Studien zur strukturalistischen Ästhetik und Poetik, München: Hanser. Nelson, K. (2007), Young Minds in Social Worlds, Cambridge: Harvard University Press. Nöth, W. (1977), Dynamik semiotischer Systeme, Stuttgart: Metzlersche Verlagsbuchhandling. Paolucci, C. (2021), Cognitive Semiotics: Integrating Signs, Minds, Meaning and Cognition, Berlin and New York: Springer. Prieto, L. (1975), Essai de linguistique et sémiologie générales, Genève: Droz. Radnitzky, G. (1970), Contemporary Schools of Metascience, Gothenburg: Akademiförlaget. Richerson, P. J. and R. Boyd (2005), Not by Genes Alone, Chicago: University of Chicago Press. Ricœur, P. (1983–5), Temps et récit, 3 vols, Paris: Seuil. Rosenthal, V. (2004), ‘Microgenesis, Immediate Experience and Visual Processes in Reading’, in A. Carsetti (ed.), Seeing, Thinking and Knowing, 221–43, Dordrecht: Kluwer Academic Publishers. Rowlands, M. (2010), The New Science of the Mind: From Extended Mind to Embodied Phenomenology, Cambridge, MA: MIT Press. Sebeok, T. A. (1991), Semiotics in the United States, Bloomington: Indiana University Press. Sonesson, G. (1978), Tecken och Handling, Lund: Doxa. Sonesson, G. (1989), Pictorial Concepts, Lund: Lund University Press. Sonesson, G. (2009), ‘The View from Husserl’s Lectern: Considerations on the Role of Phenomenology in Cognitive Semiotics’, Cybernetics and Human Knowing, 16 (3–4): 107–48. Sonesson, G. (2010), ‘Semiosis and the Elusive Final Interpretant of Understanding’, Semiotica, 179 (1–4): 145–258. ­Sonesson, G. (2013), ‘The Natural History of Branching: Approaches to the Phenomenology of Firstness, Secondness, and Thirdness’, Signs and Society, 1 (2): 297–326. Sonesson, G. (2015a), ‘Le jeu de la valeur et du sens’, in A. Biglari (ed.), Valeur. Aux fondements de la sémiotique, 85–100, Paris: L’Harmattan.

312

BLOOMSBURY SEMIOTICS: SEMIOTIC MOVEMENTS

Sonesson, G. (2015b), ‘From Remembering to Memory by Way of Culture: A Study in Cognitive Semiotics’, Southern Journal of Semiotics, 5 (1): 25–52. Sonesson, G. (2016), ‘Epistemological Prolegomena to the Cognitive Semiotics of Evolution and Development’, Language and Semiotic Studies, 2 (4): 46–99. Sonesson, G. (2017), ‘Mastering Phenomenological Semiotics with Husserl and Peirce’, in K. Bankov and P. Cobley (eds), Semiotics and Its Masters, vol. 1, 83–104, Berlin: de Gruyter. Sonesson, G. (2018a), ‘Meaning Redefined: Reflections on the Scholastic Heritage Conveyed by John Deely to Contemporary Semiotics’, The American Journal of Semiotics, 34 (1–2): 65–86. Sonesson, G. (2018b), ‘Beyond the “Tragedy of culture”: In-between Epistemology and Communication’, The American Journal of Semiotics, 33 (3–4): 141–80. Sonesson, G. (2019), ‘The Psammetichus Syndrome and Beyond: Five Experimental Approaches to Meaning-making’, The American Journal of Semiotics, 35 (1–2): 11–32. Sonesson, G. (2021a), ‘The Relevance of the Encyclopaedia: From Semiosis to Sedimentation and Back Again’, in N. S. Drăgan (ed.), Differences, Similarities and Meanings. The Interplay of Differences and Similarities in Communication and Semiotics, Berlin: de Gruyter. Sonesson, G. (2021b), ‘Nothing Human Is Foreign to Me – Except Some of It. From the Semiotic Tradition to Cognitive Semiotics by way of Phenomenology’, Degrés, 186–7: b1–b23. Sonesson, G. and S. Lenninger (2015), ‘The Psychological Development of Semiotic Competence: From the Window to the Movie by way of the Mirror’, Cognitive Development, 36: 191–201. Stjernfelt, F. (2007), Diagrammatology: An Investigation on the Borderlines of Phenomenology, Ontology, and Semiotics, Dordrecht: Springer. Thompson, E. (2007), Mind in Life, Cambridge: The Belnap Press. Vico, G. ([1710] 1988), On the Most Ancient Wisdom of the Italians, Unearthed from the Origins of the Latin Language, Ithaca and London: Cornell University Press. Von Uexküll, J. (1973), Theoretische Biologie, Frankfurt/M: Suhrkamp. Zahavi, D. (2005), Subjectivity and Selfhood, Cambridge: The MIT Press. Zlatev, J. (2008), ‘The Co-evolution of Intersubjectivity and Bodily Mimesis’, in J. Zlatev, T. Racine, C. Sinha and E. Itkonen (eds), The Shared Mind: Perspectives on Intersubjectivity, 215–44, Amsterdam: John Benjamins. Zlatev, J. (2009), ‘The Semiotic Hierarchy: Life, Consciousness, Signs and Language’, Cognitive Semiotics, 4: 169–200. Zlatev, J. (2015), ‘Cognitive Semiotics’, in P. Trifonas (ed.), International Handbook of Semiotics, 1043–67, Dordrecht: Springer. Zlatev, J., E. Alenkaer Madsen, S. Lenninger, T. Persson, S. Sayehli, G. Sonesson and J. van de Weijer (2013), ‘Understanding Communicative Intentions and Semiotic Vehicles by Children and Chimpanzees’, Cognitive Development, 28: 312–29. Zlatev, J., G. Sonesson and P. Konderak (eds) (2016), Meaning, Mind and Communication: Explorations in Cognitive Semiotics, Frankfurt am Main: Peter Lang.

INDEX

abduction 57–8, 128–9, 134, 139 n.7 Actor Network Theory (Latour) 52 adaptation 156 Adorno, T. W. 50 agenda-setting theory 54 Aktualgenese 306 American Sign Language (ASL) 278 Anderson, B. 74–5 anticipatory system 95–6 Aquiar, D. 150 Arnheim, R. 193 Arnold, M. 50 artificial intelligence (AI) 57–9, 204, 294 artificial semiotic system 96–7 augmented reality (AR) 59 Augustine, St. 10, 126 Austin, J. L. 116 authority 49, 169, 174, 260 autocommunication 155 Bakhtin, M. 49, 59, 152–3, 232 Barthes, R. 5–6, 47, 49, 55–7, 152, 154, 163, 170, 217, 239 base space 173–4 Bastin, G. 156 Bateson, G. 17–19, 25 Baudrillard, J. 59 Becoming a Translator (Robinson) 150 Bildung (Gadamer) 130 binary analogue model 37–8 biosemiotics 2, 9, 68, 74, 78–80, 150, 156, 261–4 blending mode 173–4, 176–8 Boehm, G. 73 bonding icons 239–40 Botev, H. 277 Bouissac, P. 296 Bounce 225–6 Boutet, D. 192 boyd, d. 67 Brandt, L. 180 n.28 Brandt, P. A. 7, 10, 179 n.12, 180 n.27, 297, 309

Bresciani, F. E. 91 Buber, M. 123 Bühler, K. 187, 190 Bulwer, J. 185 Bundgaard, P. F. 10, 119 n.10 Bunge, M. 91 ­Byrne, T. 104 Calbris, G. 186 Cannizzaro, S. 4 canon of use 218–20 Cantril, H. 47–8, 53 Capozzi, R. 137 Carnap, R. 5 Casasanto, D. 119 n.10 Cassirer, E. 15–16, 19, 299 categorial signs 110–11 Centre for Contemporary Cultural Studies (CCCS) 50–1 Centre for Mass Communication Studies (CECMAS) 47, 49 Chalmers, D. 294–5 Cheney-Lippold, J. 53 Chesterman, A. 147 chiasm model 28–9, 31–3, 35 Chomsky, N. 51 Cienki, A. 192, 198 Ciula, A. 69, 72, 74 Clark, A. 294–5 Clark, H. H. 193 classic media semiotics 55–7 closed system 92 Cobley, P. 78 cognition cognitive assemblages 55 cognitive psychology 107, 116–17, 232, 283 cognitive revolution 7 and iconicity 270, 278 cognitive linguistics (CL) 7, 116, 187, 269–71, 286–7 construal 271–4 iconicity 274–9

314

metaphor 279–83. See also conceptual metaphor theory (CMT) multimodality and polysemiosis 283–6 cognitive science 293, 307–9 approach 296–8 domain of study 303–4 evolution and development 304–6 and experimental semiotics 300–1 human and social sciences 301–3 linguistics 298–300 methodologies 306–7 phenomenology 300 semiotic tradition 295–6 stages of 294–5 cognitive semiotics 7, 12, 195, 270–1, 280, 282, 284, 286, 297–8, 300–9 Cohen, J. 222 Colino, S. 222 Comay, R. 20 communication 15–16, 258–9 function 110, 117, 190, 233 Gusdorf discourse model 22–4 ­human discourse model 19–22 human observer model 17–19 and information 16 matrix 28–31, 34–5, 38–40 scholastic trivium discourse model 24–5 trope model 27–33, 36, 39–40 competence 15, 37, 60, 75–6, 78–9 complex system 93–4 compositionality 93, 96–7, 108, 112, 233, 237–8 computational approach 96–7 conceptual meaning 164, 166, 172, 274 conceptual metaphor theory (CMT) 279, 282 consciousness 7, 9–10, 12, 15–17, 59, 91, 113–14, 117–18, 119 n.12, 125, 128, 130–3, 170, 273, 293–5, 297, 302–5, 307 construals 11, 70, 72, 75, 104, 108, 113, 115, 203–4, 270–6, 278, 281, 286 contiguity 198–200 conventionality 201 conventional system 279 Cornips, L. 263 Courtés, J. 151–2 Crisis of the European Sciences and Transcendental Phenomenology, The (Husserl) 135 critical discourse analysis (CDA) 6 Croce, B. 254 cultivation theory 54

INDEX

cultural Lifeworld 304–5 cultural text 154 Culture and Anarchy (Arnold) 50 curvature 224 cyberculture 57–8 cyberspace 49, 53, 58–9, 61–2 Daddesio, T. 297 Danesi, M. 2, 9, 72, 74 Dawkins, R. 60 Deacon, T. 295 Debrun, M. 94 decodification 128 decoding 23, 27, 37, 55 deduction 139 n.7 Deely, J. 33, 127, 129, 140 n.12, 261–2 De Kerckhove, D. 61 Deledalle-Rhodes, J. 149 Deleuze, G. 52, 55 Dennett, D. 307 depiction 17, 28, 37, 169, 175, 284–6 Derrida, J. 56, 126 deterministic system 93 Devylder, S. 274, 276, 283 diagrammatic iconicity 197–8, 275–7, 282 diagrams 68, 73, 167–8, 170, 172, 179 n.14, 179 n.17, 197–8 dialogicity 306–7 dialogue functions 129 digital humanities (DH) 67–70 ­biosemiotic approach 78–9 ecosemiotics 79–80 embodiment 74 existential graphs 73–4 iconicity 72–3 industry 77 literacy 74–6 multimodality 70–1 translation 71–2 digitalization 67, 71, 73–7 digital literacy 59, 75 Dilthey, W. 132 discourse analysis 231–3 discursive practices 221 dismal science 303 Dockhorn, K. 130 Domani 137–8 Donald, M. 52, 55, 295, 304–6 D’Ottaviano, I. M. L. 91 double articulation 284 Duteil-Mougel, C. 152 dynamical systems 92, 97

INDEX

ecosemiotics 68, 79–80 Eco, U. 10, 22, 59, 136, 139 n.4, 139 n.10, 153–4, 156 idiolect 136 modes of sign production 135 Numero Zero 123, 137–8 paramount 136–7 postmodernity 129 Theory of Semiotics, A 133–7 Efron, D. 185 Eide, Ø. 69, 72, 74 Ekman, P. 185 ELAN 202 Elleström, L. 72–4 Elliot, A. 222 emblematic gestures 189–90 embodied intersubjectivity 282 embodied level 280–2 embodiment 17, 74, 131, 295 encoding 27, 37 enunciation 111, 164, 169–70, 174, 176, 178, 179 nn.14, 18–19 Espe, H. 296 ‘Evolutionary Love’ (Peirce) 139 n.9 evolutionary system 94–5 existential graphs 71, 73–4, 76, 78 expansion 224 experimental semiotics 300–1 expressions 105–6, 109 Fabbri, P. 153, 157 Fairclough, N. 6 Ferraris, M. 124–7, 139 n.4 ­Fetzer, J. H. 87 Feuerbach, L. 303 Fillmore, C. 299 Fiore, Q. 48 Floreano, D. 87 Floridi, L. 77 Foucault, M. 6, 23, 25, 32, 56, 220 framing 237 Frankfurt School 50–1 Friesen, W. 185 functional cycle 303 fusion 152 Gadamer 124–7, 129–35, 137–8, 139 n.3 play, notion of 131–2 Galantucci, B. 301 Gamer, E. -C. 156 Garrod, S. 301 general systems theory 90–2

315

genuine signs 104, 106, 111–12 Gerbner, G. 54 gesture 183–5 co-speech 190–2, 194–5 coverbal 191 empirical research 202–4 first wave 185 kinesic, visual and imagined facets 192–3 multi-functionality and polysemy 190–1 Peircean approaches to 195–202 pointing 190–1 second wave 185–6 and semiotic theory 195 sign formation 193–4 sign language 194–5 third wave 186–7 verbo-gestural sign process 188–90 Gibson, J. 295 global discourse 60 global semiotics 261–2, 264 goal-based behaviour 96 Goethals, G. 150 Gorlée, D. 149–50, 157 Gottlieb, H. 145 Grady, J. E. 279 Gramsci, A. 51 Greimas, A. J. 10, 33, 40, 103, 118, 147, 151–2, 163, 165, 167 Grondin, J. 4, 124–6, 139 n.3 Gross, L. 54 Guattari, F. 52, 55 Gurwitsch, A. 306 Gusdorf, G. 22–4 Hadlington, L. 53 Halliday, M. A. K. 11, 146, 215, 217, 223, 231–4, 248 ­hardcore reality 262, 264 Harris, R. 254–65 Hartama-Heinonen, R. 147, 150 Hartley, J. 67, 75, 77, 80 hashtag 51, 60 Haviland, J. B. 186–7 Hawreliak, J. 56–7 Hayles, K. 55 Hegel, F. 295 hegemony theory 51 Heidegger, M. 4, 22, 24, 34–5, 37–8, 40, 58, 124–6, 130–2 Heine, B. 287 n.4 Herman, E. 51

316

hermeneutics 10, 123–7, 129–30, 132–3, 137, 302 heterocommunication 155 heterophenomenology 307 Hinnell, J. 202–3 Hippocrates 8–9 History of Hermeneutics (Ferraris) 126 Hjelmslev, L. 103, 163, 178 n.6, 239, 296 Hodge, B. 215 Hoffmeyer, J. 80 Holenstein, E. 103, 115 Horkheimer, M. 50 Houser, N. 297 human beings 15–17, 32, 34, 40, 48, 59, 74, 95, 105, 110, 126, 128, 131, 258–9, 262, 264, 293–4, 301–2, 304–6 human discourse model 19–22 humanism 126, 256 humanity 50, 77, 264–5 human observer model 17–19 Husserl 4, 10, 21–4, 27, 31–3, 37, 103–7, 109–15, 118 nn.2–3, 119 nn.11–12, 125, 135, 273, 295, 297, 300, 302–4 phenomenological semiotics 115–18 Hutton, C. 260, 294 hypertextuality 129 Hypodermic Needle Theory (HNT) 53–4 iconicity 57, 71–4, 78–9, 196, 203, 270, 274–80, 282, 284, 286, 287 n.2 of distance 276 of quantity 276 iconic operation 302 iconization 169, 239 icons 16, 25, 72, 88–9, 128, 139 n.8, 149, 167–9, 196–7, 199, 239–40 ideational meaning 233, 239 Iedema, R. 241 imagistic iconicity 275 indexicality 199–200, 203, 284 indications 105–6, 110–11, 113 Information and Communication Technologies (ICTs) 77 information value 237 Ingarden, R. 119 n.13 Ingram, H. J. 240 ­integral linguistics 280 integrationism 6, 11, 254, 256, 261–5 intelligibility 37–8, 40 inter-group communication 18, 27 intericonicity 156 International Association for Cognitive Semiotics (IACS) 307–8

INDEX

inter-personal communication 17, 27 interpersonal meaning 233, 237, 246 interpictorial translation 156 intersemiotic translation 155–7 intertextuality 151–2 intimating function 106, 109–10, 115–16 intimation 106, 111 intra-group communication 17, 27 intra-personal communication 17, 27 intrapictorial translation 156 Invasion from Mars, The: A Study in the Psychology of Panic (Cantril) 47 ipseity 306–7 isomorphism 135, 196, 275, 278 Itkonen, E. 275, 278 Jakobson, R. 2, 10, 15, 17, 22–3, 25–6, 71, 103, 143–7, 150–7, 199, 201, 270, 274–5, 287 n.2. See also communication chiasm model 28–9, 31–3 elements and functions 27–31 logic and semiotic squares 33–4 scientifically cautious 151 Jenkins, H. 58 Johnson, M. 279, 294–5 Jones, P. 6 Jourdain effect 1–3, 7–8 Kant, I. 130–1 Katz, E. 53–4 Keane, W. 255–6 Kees, W. 7 Keller, R. 253 Kendon, A. 186, 189, 192–3 Koch, W. 296 Konderak, P. 309 Kourdis, E. 156 Kress, G. 215, 217, 231–3, 237–9 Kristeva, J. 32, 151–2 Kukkonen, P. 148 Kull, K. 79, 150, 156 Lacković, N. 75 Lakoff, G. 279, 294–5, 299–300 Landowski, E. 165, 167 Langacker, R. W. 272–3, 275, 278 Langlois, G. 52 Language of Displayed Art, The (O’Toole) 217 language-processing 107 Lanigan, R. L. 9, 29, 32–3, 38, 133 Lasswell, H. D. 50 ­Latour, B. 51–2

INDEX

Lazarsfeld, P. F. 53–4 Lazzarato, M. 52 Leavis, F. R. 50 Le Bourgeois gentilhomme (The Bourgeois Gentleman, Molière) 1 Lederer, M. 152 Lemke, J. L. 248 Levinas, E. 125 Lévi-Strauss, C. 58 Levy, A. 51 Lindekens, R. 296, 300 linguistic metaphors 280, 283 linguistics 298–300 linguistic signs 69, 105, 111–15, 118, 119 n.10, 119 n.12, 156, 184, 188, 190, 201, 253–5, 257, 260, 265, 274 Lippmann, W. 50 liquidity 225 literacy 9, 74–7, 80 local discourse 60 Logical Investigations (Husserl) 103–4, 110, 113, 118 n.1 Lotman, J. 153–5, 295 Lotman, Y. 20, 77, 79 Lupyan, G. 119 n.10 McCarty, W. 79 McCombs, M. 54 McGurk effect 286 machinic phylum 52 McLuhan, M. 9, 48–9, 52, 59, 61–2 McNeill, D. 186, 196–7 Marais, K. 150 Marx, K. 303 mass culture 49–50 meaning-fulfilling acts 106–11, 116–17 meaning-giving acts 105–6, 108–10, 115–16 Mechanical Bride (McLuhan) 49 media 47–8, 61–2 classic 55–7 convergent 58 mass 47–51, 53–5, 57–8 new media 51–3, 55, 57–61 origins 48–9 psychological approach 53–5 Medium is the Massage, The: An Inventory of Effects (McLuhan and Fiore) 48–9 memes 60–1 mental software 55 Merleau-Ponty, M. 22, 25, 32, 34–5, 37, 104, 117, 295 meta-analysis 273, 296, 302 metafunctions 233–5, 242, 246–8

317

metaphor 20, 22, 32–3, 56, 137, 157, 173–5, 187, 190–1, 197–201, 269, 271, 274, 279–86 Methodenstreit 299, 309 metonymy 20, 22–3, 33, 174–5, 199–200, 203 mimetic memory 304 ­Mind’s New Science, The (Gardner) 294 Minsky, M. 57 Mitchell, W. J. T. 73 Mittelberg, I. 192, 196, 199, 201, 203 Molière 1 monomodal model 74, 198, 215, 241 Morgenstern, A. 192 Morris, C. 5, 7 Moscow-Tartu School. See Tartu-Moscow School Mossop, B. 145 motivation 281 Motivation & Sedimentation Model (MSM) 280, 282–3 Möttönen, T. 273 Müller, C. 187, 193, 198 multimodal discourse analysis (MDA) 231–5 Clarifai algorithm 242, 247–8 digital approaches 241–8 IBM Watson algorithm 241–2, 247–8 Just Terror 236–8, 240–1, 243–5 K-modes algorithm 242, 245 semiotic sciences 248–9 multimodality 4–5, 11, 68, 70–1, 73–5, 188–91, 215–17, 227, 231–2. See also polysemiosis affordances 219–20 canon of use 218–20 discourses 220–1 discursive practices 221 food packaging 221–3 inventory 223–4 language 192–5 and polysemiosis 283–6 resources and choices 217–18 semiotic materials 218–20 textures 224–7 music 171–2 Myin, E. 294 mythic memory 304–5 Mythologies (Barthes) 55 Nadin, M. 95 natural-artificial 225 natural Lifeworld 303, 305 Nelson, K. 306

318

INDEX

Nesselroth, P. W. 56 neutrality 306–7 new media 51–3, 55, 57–61 Noelle-Neumann, E. 54 Nolfi, S. 87 non-categorial sign 111 non-deterministic system 93 non-linguistic sign 10, 111, 151, 157, 255, 257, 265, 275, 278, 280–1, 286 Norris, C. 2 Nöth, W. 296

postmodernity 129 post-structuralism 56 pragmatics 163 of meaning 166–9 of music 171–2 semiosis and diegesis 165–6 and semiotics 163–5 of tropes 172–7 Prieto, L. 301–2, 306 propaganda theory 51 Public Opinion (Lippmann) 50

objective construal 272–3 ­Observer Model of Human Communication, The (Ruesch and Bateson) 17 O’Halloran, K. 68–9, 71, 77, 234 Ong, W. 60 open system 92 organization 25, 91–2, 94–6, 98 n.1 orientation 224 O’Toole, M. 217, 231–4

Queiroz, J. 150

Palmer, R. E. 124–5, 130, 139 n.2 Paolucci, C. 297, 309 Papert, S. 57 Pareyson, L. 133, 139 n.4, 140 n.14 Paris School of Semiotics 10, 103, 144, 151–3, 232 Pasquinelli, M. 57–8 Peirce, C. S. 10–11, 15, 196, 199, 201, 261 Firstness 95, 128, 302 to gesture 195–202 ground 195–6 interpretant 25, 89, 149, 195, 261 logic typology 22 process semiotics 88–9 representamen 25, 129, 140 n.13, 195 Secondness 95–6, 128, 302 Thirdness 95–6, 128, 302 translation 143–4, 148–50 Pennycook, A. 259 Perspectives Model of Communication (Lanigan) 29 Petrilli, S. 127–8, 139 nn.5–6, 11, 157 phenomenology 300 phonetics 163–4, 171 phonic iconicity 275 polysemiosis 283–6 polysemiotic approach 145, 285 Ponzio, A. 127–8, 139 nn.5, 11 Popovič, A. 158 n.1 p-organization 91–2, 94

radical indeterminacy 260 Rastier, F. 152 Reading Images (Kress and Van Leeuwen) 217 referential meaning 164, 166, 172 regularity 224–5 ­relevance 37–8, 40, 72, 104, 110–12, 116, 133, 280 relief 225 Ricoeur, P. 127 Rifkin, J. 79 rigidity 225 Robinson, D. 148–50 Rosenblatt, F. 57 Rosen, R. 94–5 Rowlands, M. 295 Ruda, F. 20 Ruesch, J. 7, 17–19, 25 salience 237 transfer 54 Sander, F. 306 Saussure, F. de communication model 263 différence 55–6 human capacity 254 idealism 254, 263 integrational linguistics 255–6 schemata 201 Schleiermacher, F. 132 Schmid, H. J. 287 n.4 scholastic trivium discourse model 21, 24–5 Schütz, A. 37 SCI-MX 225–6 Sebeok, T. A. 2, 7, 33, 78–9, 139 n.5, 157, 261, 263–5, 296–7 secondary orality 60 sedimentation 273, 280–2 Selfish Gene, The (Dawkins) 60

INDEX

self-maintaining system 90 self-organized system 92, 94–5 Semantics of Science, The (Harris) 257, 262–3 Semio-linguistic Research Group. See Paris School of Semiotics semiosis 2–6, 92, 96–7, 123–4, 128–30, 132–8, 149–56, 163–4, 204, 248, 296. See also polysemiosis agency in 8, 134 and cognition 118 connotative 167 diagrammatic 168 dialogic, dialogue and 124, 137–8 ethics in 128 hermeneutics and 130 human acts of 58 intersemiosis 151–2, 154–6, 231 linguistic 164 as meaning making 69 mediated 49, 52 meta-semiosis 178 n.6 multimodal 183, 188, 283, 285 and music 172, 231 patterns of 52 Peircean 87–8, 143, 149, 261, 269 ­and process 89, 150 recursion of 177 restricted 116 rhetorical 164 social 297 and translation 149–50 unlimited, infinite 134, 139 n.1, 261 urban 67 Semiotic Animal (Deely) 129 ‘Semiotic Modalities in Translation Causality’ (Chesterman) 147 semiotic(s) 146. See also biosemiotics; cognitive semiotics; ecosemiotics; semiosis; signs; social semiotics generations 147–8 ideologies 253–7 materials 218–20 reflexivity 255 systems 87 Semiotics and the Problem of Translation (Gorlée) 149 Semiotics Unbounded (Petrilli and Ponzio) 127 semiotranslation 149–50 sensus communis (Gadamer) 130–1 Shaw, D. 54 signal systems 284

319

signs 4, 11, 87–8, 104–10. See also linguistic signs and activities 256–61 categorial 110–11 co-speech gesture and 194–5 definition 88 genuine 104, 106, 111–12 gestural 167, 184, 186–94 humanity of 264–5 meaning formation 117–18 non-categorial 111 non-linguistic 10, 111, 151, 157, 255, 257, 265, 275, 278, 280–1, 286 ontologies 253–6 Peircean 195–202, 258, 261, 264 picture and 131 production 128, 134–8, 149 signal 110–13 surrogational 260 system 284–5 verbo-gestural 188–90 Simon, H. 294 simulation 96–8, 107–9, 116–18 situated level 280–1 Snell-Hornby, M. 145, 147 social media 60–1, 248–9 social semiotics 68, 70–1, 74, 215–17, 220–1, 231–5, 239, 248–9, 283 Sonesson, G. 155–6, 196, 294, 296, 298, 302, 304, 306, 308–9 s-organization 91, 94 Sources of Hermeneutics (Grondin) 125 Sowa, J. F. 73 ­spiral of silence theory 54 Stecconi, U. 150 Stjernfelt, F. 118, 297 Streeck, J. 187, 193 subjective construal 273 supercategories 262–3 Sütiste, E. 145, 155 Sweetser, E. 180 n.27 symbols 89, 96, 131, 135, 139 n.8, 167–70, 201, 239, 274–5 System Functional Theory (SFT) 233–4, 239 systemic functional multimodal discourse analysis perspective (SF-MDA) 234 systems theory 89–92 Tan, S. 234 Tarasti, E. 147 Tartu-Moscow School 10, 144, 153–5, 232 Tawheed gesture 238, 240

320

textual meaning 154, 233, 246, 248 theoretic culture 305 Theory of Linguistic Signs, A (Keller) 253 3D motion capture (MoCap) 203–4 Torop, P. 155 Torres-Martínez, S. 150 total translation 155 Toury, G. 143, 145 transduction 156–7 translatability 151 translation 71–2, 143–4 intersemiotic 155–7 Moscow-Tartu School 153–5 Paris School 151–3 Peircean account of 148–50 research 146–8 scholars 144–6 transposition 151 trivium model 21, 24–5, 33 trope model 27–33, 36, 39–40, 172–7 Truth and Method (Gadamer) 126, 130–3 Turns of Translation Studies, The: New Paradigms or Shifting Viewpoints? (Snell-Hornby) 145 Uexküll, J. von 79, 262, 303 uses and gratifications theory 54 Uspenskij, B. -A. 154

INDEX

utterance 136, 153, 164–7, 169, 176–7, 180 n.21, 185–6, 188, 190–2, 196, 198, 202–4 Vandelanotte, L. 283–4 van Langendonck, W. 275, 280 Van Leeuwen, T. 215, 217, 224, 231–3, 237–9 Veale, T. 204 Verhagen, A. 273 Vichean-Prietean supermaxim 302 Vico, G. 130, 136, 301–2 Vimercate 137–8 Visual Rhetoric (VR) 56–7 von Bertalanffy, L. 89–90 ­weight 224 Werner, H. 306 Wicke, P. 204 Wiener, N. 19 Wignell, P. 234, 239–40 Wilcox, S. 270, 278 Wundt, W. 185 Yoka, C. 156 Zethsen, K. 145 Zlatev, J. 201, 273, 298, 303–4, 306, 308–9 Zwaan, R. 107–8



321

322



323

324